NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
From: Frank Reed
Date: 2025 Oct 22, 09:11 -0700
I feel that my post with the key detail on this topic got lost in the sidebar discussions. So I repeat:
There's a short set of general case equations that we can apply. Let's call "dy" the separation in nautical miles in the North/South "Latitude" direction, while "dx" is the separation East/West. Suppose I have some two points separated by a small-ish difference in latitude and longitude, up to as much as 5° (300 nautical miles in Lat). If I have the coordinate differences, dLat and dLon, and the mean latitude, L, all in degrees I can calculate the offsets in nautical miles to a good approximation from:
- dy = 60 · dLat·[1 - (3/2)ε·cos(2L)]
- dx = 60 · dLon·[1 + ε - (1/2)ε·cos(2L)]·cos(L)
where ε is just the usual inverse flattening. That makes ε nearly equal to 1/298 and you can use 1/300 with negligible impact. Note that the cos(L) factor at the end of the dx formula is just the ordinary convergence of longitude lines as we move toward the poles. Both dx and dy have terms proportional to ε and proportional to cos(2L). As noted, ε is nearly 1/300. The factor cos(2L) varies smoothly from +1 at the equator, passing through zero at 45° latitude, and ending at -1 at the poles.
The two factors in square brackets above are all you need in some moderate range of latitude. They adjust the scale of latitude and longitude differences for the slight variations due to the definitions we use on the ellipsoidal globe.
Frank Reed






