NavList:

Harri Ojanen, you wrote:
"If you don't want to worry about when you need the extra precision in intermediate steps, it's easy to use double precision always."

Very true, and I think we can go one better than that. It's actually hard NOT to use so-called "double precision" on intermediate results. Nearly every computing device available does real number math with 16 digits of precision (known as "double precision" from, when, the 1970s? earlier?). That's default for almost everything. And really that's why you don't have to worry about "precision" anymore for computer-based celestial navigation calculations. Everything is worked out internally far beyond the precision requirements of the subject. The issue that you described is the problem of getting an inverse cosine of a number near 1. If the internal results were calculated to 8 digits precision (the now almost extinct "single precision"), then very small angles of a few minutes of arc and smaller would yield inaccurate results. An easy way to see this is to calculate the cosine of 0.1' and then take the inverse cosine of the result (try this on a handheld calculator or in a spreadsheet or in Javascript). You should get back 0.1 minutes of arc. Many devices/calculating systems will show some tiny difference in the result but well beyond the numerical significance of the result (if the process returns, e.g., 0.1000004, there's no problem). All calculations have "edge cases". You can imagine someone coding the great circle formula for distance between two points and then applying it to two points separated by a few centimeters. This would be an extreme "edge case" for the cosine problem described above, since the angle subtended at the center of the Earth by an arc on the surface a few centimeters in length is only a few thousandths of a second of arc (much smaller than the precision of the angles we deal with in celestial navigation). One other "edge case" involves nominal or apparent precision. Suppose you want to calculate the celestial coordinates, SHA and Dec, of the North Star to 0.1' of precision in order to "match" the numbers in an official almanac. It's easy to get the Declination to that accuracy/precision, but there's an illusion with the SHA. Since the longitudinal coordinate lines converge at the north celestial pole, the SHA of the North Star can be accurate to 0.1' in terms of actual angular position, but it might 10' off as "printed". This doesn't mean it's wrong or less accurate than it should be. But for a consumer of data comparing your numbers with official numbers, it can create the illusion of an imprecise or inaccurate number.

Frank Reed