NavList:

From: Geoffrey Kolbe <geoffreykolbe@compuserve.com>
Subject: [NavList] Re: Why are NA sight reduction tables not popular?
To: NavList@fer3.com
Date: Monday, April 9, 2012, 12:00 AM

At 23:50 08/04/2012, Gary Lapook wrote:

> Now that I have looked at those pages in my N.A. I am again left scratching my head as to how somebody can come up with unnecessarily complicated ways to express simple things.
>
> The N.A.  implementation of he law of cosines formula is:
> S= sin Dec
> C = cos Dec cos LHA
> Hc = arc sin (S sin Lat + C cos Lat)
> Hunh?
>
> What's wrong with the way we all learned it:
>
> sin Hc = (sin Lat sin Dec + cos Lat cos Dec cos LHA)
>
> And the N.A. azimuth formula implementation:
>
> X = (S cos Lat - C sin Lat)/cos Hc
> If  X > +1 set X = +1
> If X < -1  set X = -1
> A = arc cos X
>
> Which is the equivalent of :
>
> cos A = (sin Dec cos Lat - cos Lat cos LHA)/cos Hc
>
> instead the standard:
>
> sin A = sec Hc cos Dec sin LHA
>
> Where do they find those people at the N.A. office?
>
> gl

Paraphrasing Herbert Prinz, in his posting of many years ago and which I regret I no longer have a date for :-

The problem is that calculators have rounding errors. Consider the NA expression for the azimuth A

(sinDec cosLat - cosDec sin Lat cosLHA)/cos Hc

The theoretical range of values for this expression is -1 to +1. However, due to rounding errors, the actual value that will be computed will almost always differ from the correct value. In the limits, these rounding errors may actually take the returned value for this expression outside the permitted range for which the arc cos algorithm in the calculator is defined. On trying to compute the inverted cosine, most calculators or computers will fail at this point. Sillier calculators, like the HP48, will happily continue in the domain of complex numbers, which is of questionable benefit to the average navigator.

Hence the check which the good people at the NA office put in as an extra step to ensure that whatever calculating machine you may be using - and the NA people have to consider that you may be using anything from a cheap scientific calculator from Walmart to a Cray super computer - the algorithm does not fail or become unstable as this expression approaches its limits.

You might argue that observing celestial bodies at the zenith is highly unlikely, but as the Arctic ocean increasingly becomes navigable during the summer, it is increasingly likely that navigators may try to use this expression in an Arctic scenario. In such circumstances, it is quite likely that the North Pole be used for an assumed position and an azimuth for Polaris be calculated, for example. The algorithm must be stable in such circumstances.

I should note here that the Power Squadron formula for the azimuth would certainly fail with the Pole as an assumed position and if the formula is used in a programmable calculator or computer which did not return Hc until all computations were done, you would get no useful output at all.

The main problem with the formula you propose, sin A = sec Hc cos Dec sin LHA, is the selection of quadrant. You need four rules to select the correct quadrant for the inverted sine, whereas you only need two for the inverted cosine. Too, I think the expression is not as accurate if used with the assumed position in polar regions and with high altitude stars such as Polaris.

I think that after a moment's thought, you will conclude - as Herbert Prinz did - that the people at the NA had actually made a carefully considered choice in the formulae on which they based their algorithm and also carefully considered the wide range of circumstances in which their algorithm may be used.

Geoffrey