NavList:

Mike:

I do not have answers to your questions but I do have an observation:

The length of a arc-min angle across the surface of the earth varies by about 0.5% (10m/1852m = 0.0054).

If CelNav was using the distances rather than angles the accumulating error could become significant.  For example shooting the sun at the horizon the error would be 5400nm x 0.0054 = 29.2nm.  That's significantly larger than the expected 5-10nmm error for small navigation.

But the intercept method generally used sticks with angles almost to the end and only uses a relatively very small distance.  This method uses angles for both the calculated and apparent altitudes to they are both using the same basis  which is a perfect sphere.  Distance in nautical miles only appears when you determine the altitude intercept distance.  If you have a big intercept it would be a degree.  The distance error from 1 degree on intercept distance would be 60nm X 0.0054 = 0.324nm.  That is significantly better than the normally expected error.

So the varying length of a nautical mile only creates a relatively small error for celestial navigation as it is normally practiced.

That said I do not wish to discourage your from pursuing the answers to your questions, they are of interest in and of themselves and look forward to your discoveries.

I have found an online calculator that claims up to 130 place precision.  Precision is user defined.  I have not tested it and have no opinion about it's accuracy.

https://keisan.casio.com/calculator

Bob II

The Nautical Mile?
From: Mike Freeman
Date: 2020 Oct 30, 00:59 -0700

I am having difficulty correlating all the information I read about the nautical mile and hope you can help. Either confirm my understanding or correct me.

I have a book - The Oxford Companion to Ships and the Sea. Peter Kemp and copy a sentence from it............

A nautical mile is the distance on the earths surface subtended by one minute of latitude at the earths centre.

If the earth was a perfect sphere we could make divisions 5,400 of 1 minute at the earth centre. equator to pole and extend/subtend each minute to the earth surface and we would have 5,400 perfect nautical miles of +/- 1852m.

However with the earth being an oblate spheroid and having a greater radius at the equator if we subtend 1 minute of latitude at/near the equator the radius lines (radians?) travel further and are therefore diverging for longer which when calculated yields a nautical mile a few metres greater than 1852m. Conversely if the same calculation is performed for the pole a nautical mile of less than 1852m is determined.

As we know a nautical mile at the pole is greater (1862m?) and at the equator less (1843?) therefore the above calculation is obviously incorrect.