NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
From: Frank Reed
Date: 2025 May 15, 07:10 -0700
Suppose you want a quick assessment of the refraction in a sextant sight. How does it change with altitude? First let's switch to zenith distance, z=(90°-h). For altitudes higher than 15° or z less than 75°, the refraction is accurately calculated by
r = 0.97'·tan(z),
which is no more than a way of writing Snell's Law of refraction for very slight refraction through a flat layer (the atmosphere is near enough to flat for z<75°). This "tan(z) rule" is excellent for sights that are well away from the horizon, and if we could carry tangent tables in our brains, we would need nothing more (*). Now since tan(z) is approximately equal to z (expressed as a pure number, or "in radians"), for small-ish angles the "tan(z) rule" is approximately equivalent to
r = 0.97·z / 57.3,
with z in degrees. But it begins to fall apart near z=45°, so we have the option of optimizing the factors for better results limited to that range (z<45°). And it turns out that we get very good results with a delightfully simple short rule:
r' = z° / 50.
That is, the refraction in minutes of arc for any sextant sight with zenith distance less than 45° is equal to the zenith distance in degrees divided by 50. I find this extremely useful for quick estimates... Example: the star Capella is 55° high. That's z=35°. The refraction then is 35/50 or 70/100 or 0.7'. And that's right.
Some of you will recognize this rule. I have previously pointed out that the refraction of any angle between points A and B on the celestial sphere (between any pair of "stars") is approximately "one part in 3000" so long as A and B are bother higher than 45°, and you should be able to see that the short rule, above, is the same thing. But I do think it's nice to have the short rule floating around separately in your head! :)
Frank Reed
* I said we don't have tangent tables in our memories, but if you want to apply the "tan(z) rule" for cases extending down from z≅45° to z=15°, you can still manage without a tangent table (or calculator, or slide rule). You can draw it... As long as you have a ruler (or some ruled paper?) and a "protractor" (or compass rose from an old chart?), you can draw a triangle to read off values for 0.97'·tan(z).






