Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    Name or NavList Code:
    Re: refraction formula
    From: George Huxtable
    Date: 2000 Sep 28, 4:33 AM

    Paul Hirose wrote-
    >The tables I posted showing the effect of refraction on angular
    >separation used the refraction values from the table in the Nautical
    >Almanac. For some reason it never occurred to me that the Almanac
    >would have a formula I could program into my calculator, but it does:
    >R = 1 / tan (H + 7.31 / (H + 4.4))
    >H is the non-refracted altitude. R is the number of minutes to add to
    >H to obtain apparent (refracted) altitude. The formula is for standard
    >conditions of 10 degrees C and 1010 millibars. Be sure to observe
    >correct order of arithmetic operations: divide 7.31 by (H + 4.4), then
    >add H and take the tangent.
    >Of course the formula is an approximation; note that refraction is not
    >exactly zero at H = 90!
    >I rewrote my calculator program to use this formula and recomputed my
    >tables. The differences were trifling.
    My comments:
    The formula quoted above by Paul can be found in several texts and is a
    good and simple approximation to observed mean refraction. It's worth
    pointing out that it uses two different units of angular measure. The
    altitude H must be given in degrees, the refraction correction being in
    minutes: very convenient (but needs to be kept in mind).
    If H is the observed altitude, then R gives the correction in minutes as a
    positive quantity to subtract from it (which was what the expression was
    intended for).
    It can also be used the other way round, with only a little resulting
    error. This is how Paul was using it. If H is a calculated altitude, then R
    gives the positive correction in minutes to add to it to show the altitude
    an observer would measure with his sextant. For this latter purpose, the
    accuracy is slightly reduced, but is restored if an amended version by
    Saemundssen, quoted in Meeus, is used, of
    R = 1.02 / tan ( H + 10.3/ (H + 5.11)) where H is the CALCULATED altitude.
    I don't expect that there would be sufficient divergence between these two
    expressions to affect Paul's conclusions.
    However, two minor snags are worth noting.
    It's quite likely that mariners might wish to incorporate the formula
    within a program for sight-reduction, on a computer or programmable
    calculator. Many such devices object to being asked to work out the tangent
    of an angle of 90 degrees (which comes out as infinity). For the expression
    shown above, if you plug in a value for altitude of 89.9225 degrees, this
    results in a tangent of 90 degrees having to be calculated. The end-result
    of the correction R should of course be zero, because as the next step, the
    reciprocal of the tan is taken. However, before getting to the reciprocal
    the calculator may have bombed beforehand when asked for tan 90!
    Who, you might well ask, is going to bother about asking for a refraction
    correction for an angle so near to 90 degrees, which we all know is as near
    to zero as dammit. Well, if it's part of a larger program which includes
    refraction correction, there's no knowing the angle for which it might be
    called up.
    This trap at 89.9225 degrees is within the working range of 0 to 90, and is
    worth avoiding, if only by the purists. It's easy to do so, because for any
    angle A, tan (90-A) is exactly the same as 1 / (tan A). This is obvious if
    you remember the definition of tan in terms of the sides of a right-angled
    triangle, being the ratio of the two sides that embrace the right-angle.
    So instead of writing
    R = 1/ tan (H + 7.31 / (H + 4.4)) ,
     one can substitute
    R = tan (90 - H - 7.31 / (H + 4.4)) ,
    which is exactly the same thing, and no more complex. This alternative
    version has nothing to embarrass a computer over the whole working range of
    altitudes of 0 to 90 degrees: an infinity occurs in the calculation only at
    an absurd altitude of minus 4.4 degrees.
    Paul touched on another aspect of the expression that might disturb a
    purist, in that its end result, for an altitude of 90 degrees, is a
    refraction of -0.00135 minutes. From simple symmetry arguments, however, it
    is clear that (except in quite extraordinary atmospheric conditions) when
    you look straight up vertically, there can be no refraction at all. So if
    anyone is disturbed by that tiny residual error at 90 degrees (far smaller
    than anyone could measure), it can be reduced toward zero by a small
    additional adjustment term, as shown below-
    R = tan (90 - H + .00086H - 7.31 / (H + 4.4))
    or more simply,
    R = tan (90 - .99914H - 7.31 / (H + 4.4))
    In the past, when all navigational calculations were made without
    electronic aid by means of lookup tables, a tabulation of refractions to
    accord with measurement was satisfactory. Algebraic expressions to
    approximate the data were proposed, but as refraction increases so markedly
    near the horizon, different expressions were used, one to cover the range 0
    to (say) 20 degrees, another from 20 to 90.
    It is clearly more satisfactory if a single expression can be devised which
    covers the whole range, 0 to 90 degrees.
    The simple expression that Paul Hirose quoted was suggested by C G Bennett,
    "The calculation of astronomical refraction in marine navigation.",
    Journal of the institute for Navigation, vol 35, 255-259 (1982), and quoted
    in Meeus. It has been widely adopted.
    But it isn't the only such expression. In my own "sight-reduction with
    longterm almanac" programs, written years ago on a programmable pocket
    calculator, I used the following expression for refraction-
    R = .0162 tan (H - arctan (12H + 36))
    Here, everything is in DEGREES and the result is always a negative quantity
    which must be ADDED to the observed altitude to correct it.
    I found the expression above in Keys, "Celestial navigation by calculator",
    Stanford, p.118. I have found it perfectly satisfactory in use but haven't
    compared it in detail with the Bennett formula or with refraction tables.
    Again, it gives a small non-zero result when H = 90 degrees.
    It should be appreciated that for small altitudes, refraction is
    notoriously variable and depends critically on layers of air at different
    temperatures through which the light passes near the horizon on its way to
    the observer's eye. So great reliance should not be placed on calculated or
    tabulated values of refraction at altitudes of a few degrees. Navigators
    are recommended to confine their observations to altitudes of 15 degrees or
    more, if possible.
    Finally if anyone interested in these matters hasn't come across
    "Astronomical Algorithms", by Jean Meeus, they have a real treat in store.
    It's a wonderful goldmine of information. My own programs were written
    using his first-edition paperback, 1991; now I have the second edition,
    George Huxtable.
    George Huxtable, 1 Sandy Lane, Southmoor, Abingdon, Oxon OX13 5HX, UK.
    Tel. 01865 820222 or (int.) +44 1865 820222.

    Browse Files

    Drop Files


    What is NavList?

    Get a NavList ID Code

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    A NavList ID Code guarantees your identity in NavList posts and allows faster posting of messages.

    Retrieve a NavList ID Code

    Enter the email address associated with your NavList messages. Your NavList code will be emailed to you immediately.

    Email Settings

    NavList ID Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site