# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Re: refraction formula**

**From:**George Huxtable

**Date:**2000 Sep 28, 4:33 AM

Paul Hirose wrote- ====================================== >The tables I posted showing the effect of refraction on angular >separation used the refraction values from the table in the Nautical >Almanac. For some reason it never occurred to me that the Almanac >would have a formula I could program into my calculator, but it does: > >R = 1 / tan (H + 7.31 / (H + 4.4)) > >H is the non-refracted altitude. R is the number of minutes to add to >H to obtain apparent (refracted) altitude. The formula is for standard >conditions of 10 degrees C and 1010 millibars. Be sure to observe >correct order of arithmetic operations: divide 7.31 by (H + 4.4), then >add H and take the tangent. > >Of course the formula is an approximation; note that refraction is not >exactly zero at H = 90! > >I rewrote my calculator program to use this formula and recomputed my >tables. The differences were trifling. ===================================== My comments: The formula quoted above by Paul can be found in several texts and is a good and simple approximation to observed mean refraction. It's worth pointing out that it uses two different units of angular measure. The altitude H must be given in degrees, the refraction correction being in minutes: very convenient (but needs to be kept in mind). If H is the observed altitude, then R gives the correction in minutes as a positive quantity to subtract from it (which was what the expression was intended for). It can also be used the other way round, with only a little resulting error. This is how Paul was using it. If H is a calculated altitude, then R gives the positive correction in minutes to add to it to show the altitude an observer would measure with his sextant. For this latter purpose, the accuracy is slightly reduced, but is restored if an amended version by Saemundssen, quoted in Meeus, is used, of R = 1.02 / tan ( H + 10.3/ (H + 5.11)) where H is the CALCULATED altitude. I don't expect that there would be sufficient divergence between these two expressions to affect Paul's conclusions. -------------------- However, two minor snags are worth noting. It's quite likely that mariners might wish to incorporate the formula within a program for sight-reduction, on a computer or programmable calculator. Many such devices object to being asked to work out the tangent of an angle of 90 degrees (which comes out as infinity). For the expression shown above, if you plug in a value for altitude of 89.9225 degrees, this results in a tangent of 90 degrees having to be calculated. The end-result of the correction R should of course be zero, because as the next step, the reciprocal of the tan is taken. However, before getting to the reciprocal the calculator may have bombed beforehand when asked for tan 90! Who, you might well ask, is going to bother about asking for a refraction correction for an angle so near to 90 degrees, which we all know is as near to zero as dammit. Well, if it's part of a larger program which includes refraction correction, there's no knowing the angle for which it might be called up. This trap at 89.9225 degrees is within the working range of 0 to 90, and is worth avoiding, if only by the purists. It's easy to do so, because for any angle A, tan (90-A) is exactly the same as 1 / (tan A). This is obvious if you remember the definition of tan in terms of the sides of a right-angled triangle, being the ratio of the two sides that embrace the right-angle. So instead of writing R = 1/ tan (H + 7.31 / (H + 4.4)) , one can substitute R = tan (90 - H - 7.31 / (H + 4.4)) , which is exactly the same thing, and no more complex. This alternative version has nothing to embarrass a computer over the whole working range of altitudes of 0 to 90 degrees: an infinity occurs in the calculation only at an absurd altitude of minus 4.4 degrees. Paul touched on another aspect of the expression that might disturb a purist, in that its end result, for an altitude of 90 degrees, is a refraction of -0.00135 minutes. From simple symmetry arguments, however, it is clear that (except in quite extraordinary atmospheric conditions) when you look straight up vertically, there can be no refraction at all. So if anyone is disturbed by that tiny residual error at 90 degrees (far smaller than anyone could measure), it can be reduced toward zero by a small additional adjustment term, as shown below- R = tan (90 - H + .00086H - 7.31 / (H + 4.4)) or more simply, R = tan (90 - .99914H - 7.31 / (H + 4.4)) ------------------------- In the past, when all navigational calculations were made without electronic aid by means of lookup tables, a tabulation of refractions to accord with measurement was satisfactory. Algebraic expressions to approximate the data were proposed, but as refraction increases so markedly near the horizon, different expressions were used, one to cover the range 0 to (say) 20 degrees, another from 20 to 90. It is clearly more satisfactory if a single expression can be devised which covers the whole range, 0 to 90 degrees. The simple expression that Paul Hirose quoted was suggested by C G Bennett, "The calculation of astronomical refraction in marine navigation.", Journal of the institute for Navigation, vol 35, 255-259 (1982), and quoted in Meeus. It has been widely adopted. But it isn't the only such expression. In my own "sight-reduction with longterm almanac" programs, written years ago on a programmable pocket calculator, I used the following expression for refraction- R = .0162 tan (H - arctan (12H + 36)) Here, everything is in DEGREES and the result is always a negative quantity which must be ADDED to the observed altitude to correct it. I found the expression above in Keys, "Celestial navigation by calculator", Stanford, p.118. I have found it perfectly satisfactory in use but haven't compared it in detail with the Bennett formula or with refraction tables. Again, it gives a small non-zero result when H = 90 degrees. It should be appreciated that for small altitudes, refraction is notoriously variable and depends critically on layers of air at different temperatures through which the light passes near the horizon on its way to the observer's eye. So great reliance should not be placed on calculated or tabulated values of refraction at altitudes of a few degrees. Navigators are recommended to confine their observations to altitudes of 15 degrees or more, if possible. Finally if anyone interested in these matters hasn't come across "Astronomical Algorithms", by Jean Meeus, they have a real treat in store. It's a wonderful goldmine of information. My own programs were written using his first-edition paperback, 1991; now I have the second edition, 1998. George Huxtable. ------------------------------ george@huxtable.u-net.com George Huxtable, 1 Sandy Lane, Southmoor, Abingdon, Oxon OX13 5HX, UK. Tel. 01865 820222 or (int.) +44 1865 820222. ------------------------------