# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: Atmospheric refraction high above Sea Level**

**From:**Frank Reed

**Date:**2024 Jul 5, 10:24 -0700

Antoine, you asked about the change in the *Nautical Almanac* refraction tables from 2004. After 2004 the refraction tables effectively push the prior table down to lower altitude by three minutes of arc for altitudes below about 4° above the horizon. This is reflected in the actual table at the front of the volume, and they also updated the Bennett-like formula in the explanation in the back. Above 4° altitude, the changes in the tables are insignificant, but there is an illusion to be aware of: the table for refraction at altitudes above 10° is a critical table, and when the dependent quantity (in this case the refraction) changes very slowly with the independent quantity (the angular altitude), the critical values separating one entry from the next can jump even from insignificant changes in the dependent quantity. In the Nautical Almanac refraction tables, even at 20° altitude and higher, the critical altitudes seem to change by a minute of arc in various places. This is not significant and nothing to worry about.

At the lowest altitudes in the refraction tables, you'll find:

Alt : r1998 : r2021

0°00': -34.5': -33.8'

0°03': 33.8 : 33.2

0°06': 33.2 : 32.6

0°09': 32.6 : 32.0

0°12': 32.0 : 31.5

0°15': 31.4 : 30.9

When we turn to the explanation, in the section of "methods and formulae for direct computation", the variant of Bennett's formula in 1998 was:

R₀ = 0.0167° / tan(H + 7.31/(H + 4.4)),

and in the 2021 edition, it is:

R₀ = 0.0167° / tan(H + 7.32/(H + 4.32)).

Notice that 0.0167° is a close approximation to one minute of arc, and that is probably how it was fitted originally. Also note that these formulae as written assume that the argument H is in degrees and also assume that the tangent function knows how to use arguments in degrees. Traditionally, this was a given, but in modern usage, one must be aware that end users frequently don't know about the issue of degrees versus pure angles (angles in radians) and how this can affect the calculations.

These variants of the old Bennett formula, like the original, were designed and intended for an era when lines of code or lines of a computational algorithm had to be highly efficient. We're talking 1980s here!! The line of code was designed to replicate the table as listed above and at all higher altitudes, too. It is not an independent result in physics. It's a hack, and that's all. The hack was also designed to be usable at all altitudes from 15° to 90° since in that ancient era, two different formulas wrapped in a conditional were considered far too much code. I don't recommend that at all today. Use the well-known --and physics-based-- result 1'/tan(H) for altitudes above 15°. This eliminates an otherwise pointless problem for altitudes close to 90° in the Bennett-type formula, and it also allow better fine-tuning of parameters if a Bennett-type formula is used at lower altitudes.

If we want to use my "below zero" formula and make it consistent with the later post-2004 refraction formula in the Nautical Almanac, a couple of the constants will require adjustment. As a quick cheat, these can be manually adjusted to get the value at 0° altitude to match the new table (33.8') and also to get the rate of change matching. And, Antoine, since your simple extrapolation may have been based on the post-2004 tables, it's even possible that your version ends up closer to my (supposedly) more accurate approximation (accurate because it was generated from the physics-based Auer-Standish integration method).

One concluding thought: years ago, generating a single refraction by running a full Auer-Standish integration would have seemed insane. That's why we leaned on short approximate formulas, like the ones we've been discussing. But today? An integration that formerly took a second or three would require an insignificant fraction of a second today. So maybe do that!

Frank Reed