NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
From: Frank Reed
Date: 2025 Oct 11, 13:45 -0700
David Pike, you wrote:
"I looked for the tables you talked of, but I struggled to find them online."
Sorry. :) I don't know exactly where to find the tables I was talking about either. I may have hallucinated that. But what we need is just a calculation for the length of a degree. We need the length in km or nautical miles (n.m. below) of a degree of latitude and the corresponding length of a degree of longitude. At the equator:
- degree of latitude: 59.71 n.m.
- degree of longitude: 60.11 n.m.
And at 45° latitude:
- degree of latitude: 60.01 n.m.
- degree of longitude: 42.57 n.m. [or 60.20·cos(lat)]
And at 89° for a final comparison:
- degree of latitude: 60.31 n.m.
- degree of longitude: 1.05 n.m. [or 60.31·cos(lat)]
In the latter cases, it can be useful to pull out a factor of cos(lat) since that's the obvious, big piece of the pie. I've done that above in brackets [...] since it shows that the leading factors both approach 60.31 at the poles.
As I see it, the key feature here is that the paralles of latitude are slightly closer together near the equator and farther apart towards the poles. Not by much. It's +/-0.3 miles out of 60. Longitude size varies, too, but in that case the usual scaling by cos(lat) swamps the small changes due to the ellipsoidal coordinate definitions.
There's a short set of general case equations that we can apply. Let's call "dy" the separation in nautical miles in the North/South "Latitude" direction, while "dx" is the separation East/West. Suppose I have some two points separated by a small-ish difference in latitude and longitude, up to as much as 5° (300 nautical miles in Lat). If I have the coordinate differences, dLat and dLon, and the mean latitude, L, all in degrees I can calculate the offsets in nautical miles to a good approximation from:
- dy = 60 · dLat·[1 - (3/2)ε·cos(2L)]
- dx = 60 · dLon·[1 + ε - (1/2)ε·cos(2L)]·cos(L)
where ε is just the usual inverse flattening. That makes ε nearly equal to 1/298 and you can use 1/300 with negligible impact. Note that the cos(L) factor at the end of the dx formula is just the ordinary convergence of longitude lines as we move toward the poles. Both dx and dy have terms proportional to ε and proportional to cos(2L). As noted, ε is nearly 1/300. The factor cos(2L) varies smoothly from +1 at the equator, passing through zero at 45° latitude, and ending at -1 at the poles.
If we set ε to zero for the perfect spherical case, the above equations become:
- dy = 60 · dLat
- dx = 60 · dLon·cos(L)
which should look awfully familiar... and obvious.
The difference in the values of dx and dy and "Pythagorean" distances calculated from the equations for the ellipsoidal case (non-zero ε) is quite small in most cases --half a percent or less-- when compared with the spherical case. Nothing much to worry about... But when we look at things like sextant angles between objects in piloting range of 5 or 10 miles, the difference can be measured without difficulty. This is a real thing. Critically, I think, it's important to emphasize that these differences, dx and day, are small and local, and they determine effectively straight lines. They have nothing to do with drawing and parametrizing global "geodesic" curves stretching across the globe (no Vincenty). These differences arise from the way we define our coordinates of latitude and longitude on our slightly flattened globe. The parallels of latitude are closer together near the equator by a fraction of a percent.
Frank Reed
Clockwork Mapping / ReedNavigation.com
Conanicut Island USA






