NavList:

Sorry to take a while getting back to this...

David Iwancio, you wrote:
"I think it was Yallop who pointed out that floating point errors can result in trying to determine an angle with a cosine greater than 1 or less than -1, which required a check that Pepperday complains about existing."

It's an interesting concern. I don't think he actually complained about the right problem though! In his paper, Pepperday wrote:
"X has been computed from spherical formulae so how could X possibly exceed 1? Normally it cannot."

That's all true! Now back in the before time, the years around 1989 when the rules were being spelled out for the Nautical Almanac calculation article, it's also true, as Yallop (or whoever it was) contended that floating point errors might lead to an argument for an inverse cosine with an absolute value very slightly greater than one. But the trouble is that's not what the instructions say. They don't say, if |X|>1 by some wee bit, DX<0.00001, then knock it down to 1. They just say, if X is outside the range from -1 to +1, well get out the sledgehammer and bang if back in there! If a navigator adopts that brute force strategy, bad things happen. As I see it the problem with that rule is that it magically hides major keystroke errors, which are the greatest concern with instructions like these. If we skip that range-forcing step and hit inverse cosine on a value of, let's say, 2.4, any calculator or other computing system will promptly spit out an error message as it should. And in that case, the human navigator would presumably go back and try the work again. The typo would be discovered, and the error would be fixed. But if the algorithmic instructions tell you to force it back to 1, no matter what, then the error propagates in a weird and possibly dangerous way.

Yet another problem with this rule to hammer the argument back into range is that it's fundamentally an antique. The "floating point errors" that may have been a concern back then are generally gone. One can find some pathological cases, but it takes real effort, and the probability that they would occur in a practical case strikes me as vanishingly small. But are there cases where a reasonable, but small, error in an observation might blow out a "blind" trig computation on a calculator? Could there still be a case where a test might be employed to catch cases that are navigationally useful but slightly out-of-bounds from a pure math perspective?

Such cases do, in fact, occur now and then in lunars. It's possible to measure the two altitudes of the bodies and then the lunar distance when, for example, the two bodies are on opposite sides of the sky nearly in the same vertical circle, and yet the altitudes plus the distance add up to a total across the zenith that's greater than 180°. This would, by the pure geometry of it, be impossible, and could lead to an "explosive" inverse cosine. The trick here is that we don't have to pull the pin on the grenade. Alhtough we calculate the quantity cos(DZ) where DZ is supposed to be the difference in azimuth of the two bodies, we don't in fact need DZ. If we get to a point in the computation and see cos(DZ)=-1.0002, it's tempting to call "error!" and stop. But it's no problem. If we don't "pull the pin" by unwrapping that cosine to get DZ, nothing bad happens (at least in most cases). It all works because the math is robust against small errors in the inputs. Are there any cases like that in common Hc and Zn computations for the intercept method? I haven't thought it through...

You also wrote:
"the user would have had to "store" some intermediate results with pencil and paper"

In my opinion, that has some positive value. In the modern world, if we have an app (written for a real programmable computing device, smartphone or better) then it should do all the work with as little input from the navigator as possible. But if we're using a cheap and solar-powered indestructible calculator, then I think it makes sense to split the work up into manageable slices and write down intermediate results on paper. The paper records provide an audit trail, a debugging log.

Frank Reed