NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
Re: Refraction, Cassini model
From: Paul Hirose
Date: 2018 Nov 27, 14:04 -0800
From: Paul Hirose
Date: 2018 Nov 27, 14:04 -0800
I added another version of the Cassini model so now there are three to
evaluate. In addition, my refraction tables have been expanded to
1-degree steps from 10° to the horizon. To save readers the bother of
referring to my original message, I'll restate the formulas here and
include some new material.
CORRECTIONS FOR TEMPERATURE AND PRESSURE
For standard sea level conditions I assume 50% humidity, 10° Celsius (50
F), and 1010 millibars (29.83 inches Hg). For different temperature or
pressure, compute correction factor f. If P is pressure (millibars) and
T is temperature (Celsius),
f = .280 * P / (273 + T)
For inches Hg and Fahrenheit,
f = 17.1 * P / (460 + T)
In those formulas P is the actual air pressure at the observer. It can
be calculated from altimeter setting A for an observer h meters above
sea level:
P = A * (1 - h / 44 300) ^ 5.26
For h in feet,
P = A * (1 - h / 145 000) ^ 5.26
In both formulas A can be millibars or inches of mercury. The result
will have the same unit of measure. For example, at 5000 feet and
altimeter setting 29.8 inches, P = 24.78 inches. A precise calculation
says 24.80.
I use the term "altimeter setting" instead of "barometric pressure"
because when the latter is publicly distributed (in the US at least)
it's really the former. Unless best accuracy is important, it's enough
to use a fixed nominal altimeter setting since the normal range is
small. A 2% change, say from 29.9 to 30.5 inches, is unusual. A
corresponding change in absolute temperature, say from 60 to 70 F, is
easily exceeded every day.
REFRACTION MODELS
"Refco" is the routine from the International Astronomical Union SOFA
(Standards of Fundamental Astronomy) library. Its documentation says its
accuracy is better than 1″ at 10° altitude, compared to a ray trace
through a model atmosphere. Performance below 10° is not stated.
http://www.iausofa.org/2018_0130_C/sofa/refco.html
"Low alt" is the formula in the Astronomical Almanac section B for
altitude less than 15°. I have factored its polynomials into a format
convenient for a computer program:
refraction = P * (.1594 + (.0196 + 2e-5 * a) * a) / (273 + T) / (1.0 +
(.505 + .0845 * a) * a))
where "a" is altitude (degrees) and the result is degrees. At the
horizon in standard conditions it gives a reasonable 34.1′.
Cassini's refraction model is explained by astronomer Bruce Young:
https://aty.sdsu.edu/explain/atmos_refr/understanding.html
In standard conditions:
R = Earth radius = 6,371,000 meters
h = height of homogeneous atmosphere = 8307 m
n = air index of refraction = 1.0002816
Z = zenith distance = 90° - altitude
First compute a quantity I call "sinZ1":
sinZ1 = R / (R + h) * sin Z
sinZ1 = .9987 * sin Z
Then compute refraction from sinZ1 and the index of refraction:
refraction = arcsin (n * sinZ1) - arcsin (sinZ1)
To be strictly correct, R is the distance from the observer to Earth
center and therefore should include height above sea level. However, at
.01′ accuracy and 2000 meters (6600 feet) above sea level the small
adjustment has no noticeable effect.
In the tables I evaluate three Cassini versions.
"Cas1" is refraction from the basic formula, multiplied by correction
factor f. In other words, Cas1 assumes refraction is directly
proportional to air density.
"Cas2" applies separate corrections to homogeneous atmosphere height h
and refractive index n. It assumes refractivity (the fractional part of
index of refraction) is proportional to air density:
h = 8.22 * P / f (P is millibars)
h = 279 * P / f (P is inches Hg)
n = 1.0 + f * .0002816
"Cas3" is a formally correct implementation of the Cassini model. Height
of observer is applied to R, h is computed as in Cas2, and refractive
index n is computed from temperature, pressure, and humidity with the
Ciddor formulas, which are too long to quote. See this NIST page:
https://emtoolbox.nist.gov/Wavelength/Documentation.asp
"Ben1" is the basic Bennett formula in the Nautical Almanac:
R = 1 / tan (H + 7.32 / (H + 4.32))
where H is refracted altitude (degrees) and R is refraction (minutes).
"Ben2" adds the correction from "Astronomical Algorithms" by Meeus:
-0.06′ * sin (14.7 * R + 13)
where R is the refraction (minutes) from the basic Bennett formula but —
CAUTION — the result of the expression in parentheses is degrees. The
mixture of degrees, minutes, and radians requires careful coding on a
computer.
RESULTS
At 10° and up, the SOFA Refco routine is claimed to be highly accurate.
Therefore, it is the standard, and refraction values from the other
models are the difference from Refco. Below 10° I use the Astronomical
Almanac low altitude formula as the reference.
The first table is at sea level standard conditions of 10 C (50 F), 1010
millibars (29.83 inches Hg) and 50% humidity. Altitude is the refracted
altitude.
Refco low alt Cas1 Cas2 Cas3 Ben1 Ben2
45 0.97 0.22 0.00 0.00 0.00 0.03 0.00
40 1.15 0.18 0.00 0.00 0.00 0.03 0.00
35 1.38 0.15 0.00 0.00 0.00 0.04 0.01
30 1.67 0.11 0.00 0.00 0.00 0.05 0.01
25 2.06 0.07 0.00 0.00 0.00 0.06 0.01
20 2.63 0.03 0.00 0.00 0.00 0.07 0.02
15 3.55 0.00 0.00 0.00 0.00 0.08 0.03
10 5.29 -0.01 0.00 0.00 0.00 0.10 0.04
9 0.00 5.83 0.00 0.00 0.01 0.11 0.05
8 -0.02 6.51 0.00 0.00 0.00 0.11 0.05
7 -0.05 7.35 -0.02 -0.02 -0.01 0.10 0.05
6 -0.14 8.41 -0.05 -0.05 -0.05 0.09 0.05
5 -0.36 9.80 -0.12 -0.12 -0.11 0.07 0.06
4 -1.00 11.66 -0.28 -0.28 -0.28 0.05 0.07
3 -3.35 14.28 -0.70 -0.70 -0.69 0.02 0.10
2 -15.86 18.12 -1.87 -1.87 -1.86 0.00 0.14
1 -172.34 24.12 -5.24 -5.24 -5.22 -0.02 0.18
0 --- 34.13 -14.05 -14.05 -14.03 -0.33 0.30
Cassini is practically a perfect match to the reference models — better
than either Bennett formula — down to 7°. (In standard atmosphere
conditions the three Cassini versions should be identical, but tiny
differences occur due to my rounded constants.) At very low altitude
Bennett is remarkably close to the low altitude formula. However, we
must remember the latter is only another model and not the absolute
truth. Accuracy of a few hundredths of a minute in this part of the
table is an illusion.
In the second table air density is 116% of standard: pressure 1070 mb
(31.60 inches) and temperature -15 C (5 F).
Refco low alt Cas1 Cas2 Cas3 Ben1 Ben2
45 1.12 0.26 0.00 0.00 0.00 0.03 0.00
40 1.34 0.21 0.00 0.00 0.00 0.04 0.00
35 1.60 0.17 0.00 0.00 0.00 0.05 0.01
30 1.94 0.12 0.00 0.00 0.00 0.05 0.01
25 2.40 0.08 -0.01 0.00 0.00 0.06 0.02
20 3.07 0.03 -0.01 -0.01 0.00 0.07 0.02
15 4.14 -0.01 -0.02 -0.01 0.00 0.09 0.03
10 6.18 -0.04 -0.03 -0.01 0.00 0.09 0.03
9 0.04 6.78 0.00 0.04 0.05 0.13 0.07
8 0.04 7.56 0.00 0.04 0.06 0.13 0.08
7 0.03 8.54 -0.02 0.05 0.06 0.12 0.08
6 -0.03 9.77 -0.06 0.04 0.06 0.11 0.09
5 -0.18 11.38 -0.14 0.02 0.04 0.09 0.09
4 -0.71 13.55 -0.33 -0.07 -0.04 0.06 0.11
3 -2.83 16.59 -0.81 -0.36 -0.32 0.03 0.13
2 -14.88 21.06 -2.18 -1.39 -1.33 0.00 0.13
1 -171.99 28.03 -6.09 -4.82 -4.75 -0.02 0.17
0 --- 39.66 -16.32 -14.77 -14.69 -0.38 0.42
Down to 10° Cas3 is a perfect match to Refco, Cas2 practically as good,
and Cas1 only slightly less so. All three are better than Ben1 and Ben2,
though any advantage over the latter is marginal. At 9° the change of
reference model accounts for the discontinuity in the table. The
simplest Cassini version, Cas1, begins with the lead, but soon loses to
Cas2. The complex refractive index computation in Cas3 is no advantage here.
In the third table air density is 84% of standard. Pressure is 896 mb
(29.46 inches), equivalent to 1000 meters (3300 feet) above sea level
and temperature 25 C (77 F).
Refco low alt Cas1 Cas2 Cas3 Ben1 Ben2
45 0.81 0.19 0.00 0.00 0.00 0.03 0.00
40 0.97 0.16 0.00 0.00 0.00 0.03 0.00
35 1.16 0.13 0.00 0.00 0.00 0.04 0.01
30 1.40 0.09 0.00 0.00 0.00 0.04 0.01
25 1.73 0.06 0.00 0.00 0.00 0.05 0.01
20 2.21 0.03 0.00 0.00 0.00 0.06 0.02
15 2.98 0.01 0.01 0.00 0.00 0.08 0.03
10 4.43 0.01 0.02 0.01 0.00 0.11 0.05
9 -0.03 4.91 0.00 -0.01 -0.02 0.09 0.03
8 -0.05 5.48 0.00 -0.03 -0.03 0.09 0.03
7 -0.10 6.19 -0.02 -0.05 -0.06 0.09 0.03
6 -0.19 7.08 -0.04 -0.09 -0.10 0.08 0.03
5 -0.43 8.25 -0.10 -0.18 -0.19 0.06 0.03
4 -1.07 9.82 -0.24 -0.36 -0.38 0.04 0.03
3 -3.34 12.03 -0.59 -0.79 -0.82 0.02 0.06
2 -15.07 15.26 -1.58 -1.92 -1.95 0.00 0.12
1 -158.61 20.32 -4.41 -4.95 -4.99 -0.01 0.21
0 --- 28.75 -11.83 -12.47 -12.53 -0.28 0.21
Cas1 is practically as good as its siblings down to 10°. Below 10° the
simplest Cassini version is best and the most complex is worst! But all
are within 2% of the low altitude formula down to 5°.
Based on my results, I don't think the complexity of a precise
refractive index calculation in the Cas3 model is worthwhile. The
difference from the simple corrections of Cas1 and Cas2 are
insignificant for navigation. It's difficult a choose the winner between
the latter two, so I suppose it should be Cas1 due to its simplicity.
Since it applies the air density correction to refraction itself, the
expression R / (R + h) in the formula for sinZ1 is a constant.
All three Cassini versions have superior accuracy to even the improved
Bennett model except at very low altitude where no formula is reliable.
However, the Cassini model in any variant requires evaluation of three
trig functions, vs. none in Ben1 and one in Ben2.
Though it's more than 300 years old, Cassini's refraction model with
simple corrections for nonstandard conditions has amazing accuracy —
more than enough for celestial navigation.
