NavList:

   

Reply

On 2014-10-10 7:46, Samuel L wrote:
> Gary,
> I never said my sextant reads to seconds. Read the original post.

You asked how to round off an Ho of 25°37′50.4″. Writing the angle with
that much precision implies Ho is known to better than a second of arc.
It's not surprising someone would ask how you'd accomplish such a
measurement.

Good practice is to omit digits that contribute nothing useful. For
instance, suppose I find the circumference of a tree trunk is 25 1/2
inches by tape measure. Then I divide by pi and get a diameter of
8.1169020976866621242130719319982 inches.

But it's a good guess that I rounded off the tape measure reading to the
nearest half inch. Roundoff error is therefore up to 1/4 inch, or about
1% of circumference. It follows that the diameter computation can be off
by 1% too, or roughly .1 inch. Furthermore, consider the rough surface
and non circular cross section of the trunk. To real world precision, we
have an 8-inch tree trunk. I can calculate the diameter to far higher
precision, but the additional digits are just noise.

Similar considerations apply in an angle format conversion. The end
result can be no better than the original data. Say a sextant reads
57°29.9′. What is that in decimal degrees, to appropriate precision?

First consider the precision of the original data. It's .1′, which is
about .0017°. So .001° and .1′ are approximately equivalent precision,
and 57°29.9′ can be expressed as 57.498°.

Now convert 57.498° to DMS format. One second is about .0003°, so tenth
seconds are not justified. Even single seconds are about three times
more precise than the decimal angle. But that's the most reasonable
match, so we round the conversion to one second precision: 57°29′53″.

Note the original sextant reading would have been 57°29′54″ in that
format, so the "round trip" missed its target by 1″. That's more than
good enough, but it does show how small errors accumulate in the
conversions from DDMM.m to DD.ddd to DDMMSS.

Choice of an appropriate precision is often a judgement call. If you're
testing a sextant with an indexing table and collimators, the .03′ max
roundoff error that can occur at .001° precision may be an uncomfortable
fraction of your error budget. In practical navigation the same error is
negligible.


> What I needed to know was when any readings or figures be rounded up.
> I'm always using an AH so once IE is removed during the reduction and that 
figure is divided by 2 the majority of figures end up with seconds of arc at 
the end.
> For example;
> Hs= 43d 43.4min
> IE   -.1
> Result divided by 2=  21d 51min 39sec (that's what the calculator says)

Sam, errors are subtracted and corrections added. The -.1′ index error
ought to be subtracted (not added) to obtain 43°43.5′, which you then
divide by 2 to obtain 21°51.8′.

To avoid confusion between errors and corrections, remember the goal is
to remove error from observations. Remove = subtract.

   

Reply