Welcome to the NavList Message Boards.


A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

Compose Your Message

Add Images & Files
    Name or NavList Code:
    Re: On potential error introduced by rounded values
    From: George Huxtable
    Date: 2005 Jan 12, 01:32 +0000

    There are two separate topics merged into Peter Fogg's recent pair of
    mailings, for both of which George Bennett's "Complete on-board Celestial
    Navigator has provided examples.
    The topics are
    1. The effect of multiple roundings on precision of the result, which I
    will respond to here under the threadname "On potential error introduced by
    rounded values."
    2. The defects of the formula for obtaining azimuth from its sine and the
    errors that it gives rise to in Bennett's lookup table for azimuth, which
    I'll deal with elsewhere under the threadname "Sight Reduction Formula
    I wrote-
    >"If we take 6 such roundings, then
    >in theory the maximum amount they could possibly (but most unlikely)
    >contribute to an error in position is ?3 miles..."
    and Peter Fogg replied-
    >And the chance of that most unlikely event is one in six to the power of
    >six, or 0.00002, or 0.002%?
    I don't see how he has calculated that number, and perhaps Peter will
    explain. The probability of the rounding errors combining to exceed ?3
    miles is precisely zero. And the probability of the error being exactly 3
    miles is also zero.
    I think that there's a conceptual problem here. For a contiuous
    distribution such as we're discussing, the probability of the error being
    ANY exact number is always zero. You have to express it as the probability
    of being in some bracket between two values, or exceeding some limit, for
    the probability to have meaning.
    >If this is correct, the results from models
    >similar to the Excel sample are unlikely to show an error result of more
    >than 2 minutes of arc in a significant number of cases.
    I don't think it is correct, but even so, I agree with that conclusion.
    > When a
    >fix is expressed to whole minutes of arc it doesn't mean the boat is at that
    >intersection (it is almost certainly not, even if the fix is entirely
    >accurate). It means that the position is somewhere within a rectangle (a
    >square at the equator) bounded by the halfway points to the next
    >intersection of minutes of arc of lat/long.
    I don't follow what Peter is saying here (or if I do, then I disagree). The
    true position isn't necessarily in that one rectangle. You could only state
    with certainty that the true position was in that one 1' rectangle if you
    knew that the fix was in itself entirely accurate, in terms of both
    observation and calculation, except for some final single rounding
    operation that expressed the result to the nearest whole minute.
    We have just agreed that the multiple roundings in the calculation we have
    considered are unlikely to exceed 2'. Even so, that could imply that the
    true position may not be in the unique 1' rectangular box that was the
    stated result of the calculation, but could quite likely be in one of the
    four adjacent 1' boxes that surround it, and with a smaller chance, in one
    of the adjacent 1' boxes that surround that. That was just the result of
    the agreed rounding errors in the calculation process, and errors in the
    observation itself haven't even been considered yet.
    contact George Huxtable by email at george@huxtable.u-net.com, by phone at
    01865 820222 (from outside UK, +44 1865 820222), or by mail at 1 Sandy
    Lane, Southmoor, Abingdon, Oxon OX13 5HX, UK.

    Browse Files

    Drop Files


    What is NavList?

    Get a NavList ID Code

    (please, no nicknames or handles)
    Do you want to receive all group messages by email?
    Yes No

    A NavList ID Code guarantees your identity in NavList posts and allows faster posting of messages.

    Retrieve a NavList ID Code

    Enter the email address associated with your NavList messages. Your NavList code will be emailed to you immediately.

    Email Settings

    NavList ID Code:

    Custom Index

    Start date: (yyyymm dd)
    End date: (yyyymm dd)

    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site
    Visit this site