NavList:

Mark, yes, you've raised some important points. 

The physicists who study atmospheric refraction long ago realized that we can rather cleanly divide the issue of refraction and its effect on observations in navigation and surveying into two nearly disjoint categories: astronomical refraction and terrestrial refraction. Astronomical refraction refers to the refraction that affects celestial bodies whos angular altitudes are "some safe distance" above the horizon. Historically, this safe distance was counted as 5° or more, but today we know it's "safe" down as low as 1°30' so long as we have proper standard refraction tables and we make the necessary temperature and pressure compensations. Terrestrial refraction applies to the observations of distant objects on the Earth so typically we're talking about elevations within a degree of the horizon at almost any distance from the observer. 

There is some overlap between terrestrial and astronomical refraction, but it's perhaps surprisingly small. Very few astronomical bodies are visible really close to the horizon. The Sun is clearly the most important case, and when we view the Sun right at the horizon, it's in the realm of terrestrial refraction, which is variable in a complex way, and of course it is also strongly refracted by the usual, dependable laws of astronomical refraction.

Why the difference? First, we need to recognize that our ordinary everyday refraction correction in navigation has a miraculous property: it doesn't depend on what's going on "way up there" in the atmosphere. The total refraction is proportional to the amount of air that a ray of light passes through. Looking straight up is the thinnest path. Tilting down toward the horizon we pass through progressively greater quantities of air, so the refraction is larger. And if the air pressure at sea level is higher or the absolute temperature is lower, that means there's more air (the air is "thicker") and the refraction is increased in proportion. But notice that this doesn't depend on the details of the temperatures in the atmosphere at altitude. It doesn't matter whether it gets warmer as we go up (that's considered a "temperature inversion"). All that counts are the temperature and pressure at sea level. The "weight" of the atmosphere above the observer is fully determined by those two local parameters. And we're very lucky that this is the case! This property implies that the astronomical refraction is predictable and reliable and fully determined to high accuracy by only three parameters which are all directly measurable: the altitude of the celestial body, and the local temperature and pressure.

In contrast with the behavior of light rays that are passing through the atmosphere at relatively high angles, light rays that are skimming the Earth's surface, near the observer's horizon, are significantly and strongly affected by the local structure of the atmosphere, in particular, the local lapse rate, the rate at which the temperature of the atmosphere changes with height above the ground. But the lapse rate depends on the weather! And this implies that the terrestrial refraction depends on the weather rendering it as fickle and unpredictable as morning fog.

You mentioned arctic explorers like Shackleton encountering extreme deviations in refraction at the horizon of several degrees. These occur because the temperature of the air above the surface of the sea might change from 5° above freezing to 40° below freezing in a change of height of just a few meters. If we limit ourselves to "normal" variability around "mean" conditions, then we're looking at variations in the horizon of around half a minute of arc with occasional excursions as large as two or three minutes of arc.

The variable density of air low in the atmosphere (quantified by the lapse rate) bends light rays downward normally. The amount of bending can be measured in minutes of arc per nautical mile. Typically this rotation, this downward bending is much less than a minute of arc, roughly 0.1' per nautical mile. Since this rotation is proportional to distance, mathematically we can represent it by changing the curvature of the Earth! Un-curve the Earth by 10% and make the light rays straight. The big advantage of this trick is that it applies to every problem that we can imagine: dip, angular height of objects beyond the horizon, dip short, distance to the horizon, etc. You do the geometry to solve these problems as if light rays travel in straight lines using the normal radius of the Earth, and then you replace the radius with a slightly adjusted value at the end. This is the origin of the various formulae in 1970s/80s editions of Bowditch that include a "beta" parameter. The big catch is that this parameter is not a constant; it's weather-dependent.

Copy of a little "cartoon" I posted a few years ago showing this procedure of un-curving the Earth:

Over the past 200 years, astronomers and others who worked on the theory of celestial navigation observations hoped that they could eliminate this uncertainty, this "weather-dependence", in the terrestrial refraction. They hoped that by measuring the right quantities they could make the uncertainty in dip and the other observations that occur near the horizontal as amenable to correction tables as the low-altitude astronomical refraction is fixed by temperature/pressure corrections. Frequent suggestions have appeared for measuring the sea temperature and the air temperature since this should govern the lapse rate in the lowest layers of the atmosphere. Unfortunately, these suggestions have proved inadequate in practice. We're stuck with it: it's weather.

Frank Reed