NavList:

   

Reply

Alexandre, Peter, Jim, and all the others,

While I was in the middle of compiling my answer to Alexandre's last two
messages, the discussion has gone into the very wrong direction that I was
afraid it would. We are proposing methods to average before having justified
that averaging is appropriate at all. It seems that I have not made sufficiently
clear why I said from the outset that it is wrong to average the observations
before reducing them. Let me therefore address this point once more.

Peter Fog wrote

> > Getting back to nav, the process for averaging sights is simple
> > and effective. As many sights as possible
> > taken over about 5 minutes of time are plotted. Time is the
> > horizontal axis, observed altitude on the
> > vertical. The slope of this group of sights either rises; obs to
> > the east, or descends; to the west. This slope
> > is then compared to a calculated line, which is then best fitted
> > to the slope of sights. Any extreme outliers
> > are disregarded (probably best, although it goes against ideal
> > statistical practice).
> > Simple and effective. I'm always vaguely surprised its a
> > technique not more widely known.

To which Jim Thompson replied:

> Microsoft Excel makes it drop-dead easy to draw such a graph and, with a
> mouse click, to generate the equation for the line of best fit.  If a laptop
> is at hand.

Peter and Jim are shooting with guns at birds, and at the wrong birds at that.
If Peter has in mind a simple visual approach with paper and pencil when he
speaks of "best fit", I can accept that. But bringing Microsoft into this stage
of the sight reduction game is a severe faux pas. To see where all this is
coming from, let's look into what some would consider the semi-official current
practice of pleasure boaters, namely the doctrine of the USPS.

The US Power Squadron recommends (in fact demands for the sight folder to be
submitted for graduation from the JN course) that every altitude sight be
repeated at least three times and be checked for 'consistency'. Such a group of
sights the call a 'run'. (Junior Navigation, 99/01, p. 2-11 and Appendix G)
They specify what 'consistency' means: A rising body must show a steady growth
in altitude, a setting body a steady decrease. The consistency rule is waved for
sights near the meridian. (N.B. Alexandre: Altitudes above 75 deg are
discouraged, but admitted!).

In the USPS course, averaging the sights within a run is an option. One is
supposed to record all sights in a log (Form ED-SL (98)) and enumerate them. The
instructions on the back of the form say that for the reduction you can either
pick one sight from a run, or average several ones. The only guidance given is
to dismiss obviously 'inconsistent' sights.

Now, for the purpose of checking consistency, some people will find it easier to
sketch out the altitude versus time diagram in the manner Peter describes,
rather than to inspect the raw numbers. Nothing wrong with that. But then
inevitably the idea crops up that a best fitted line would give us the perfect
answer to what sextant altitude to use for a chosen time. This is not so.

First, the altitude grows in a linear fashion near the prime vertical. There,
you would use linear regression. Near the meridian you would have to use a
parabolic fit. That's also easy. But what kind of a fit do use in-between? The
correct answer is obvious: You fit the altitudes to the cosine formula that you
always use to obtain the altitudes from dec, lat and lha. But if you do this you
end up implementing the core of  the algorithm in the N.A. right then and there
at a place which is supposed to be only a preliminary step to remove
'inconsistencies'. So, what kind of fit is simple enough and still adequate?

Second, let's assume you observed two bodies, so you have two runs, and you have
figured out the answer to the above. The idea of a best fit is that you allow
for a random unknown observation error which varies slightly for each individual
observation. In the absence of any knowledge whatsoever about this error you
HOPE that you did the best, i.e the error is minimal. Hence the minimum least
squares. When you fit a line to each run, you treat the observational errors in
each run individually. Instead of minimizing the square sum of all observations
together, you minimize within each run. What is the justification for this? (I
am not saying it is impossible to argue for it, but I would like to see an
explicit statement of the underlying reasons. Otherwise we might be unaware of
what we are doing and why.) [*see footnote at the end]

Third, let's assume we have sorted out the above, but this time we have three or
more runs. Now we are in the awkward position that we have just done three or
more LSQ fits and we still have an overdetermined fix, to which the only correct
answer is - you guessed it - another LSQ fit. I leave it to you to work out the
details. It's not going to be easy to show that there might in fact be rare
situations where the fix obtained in such a round about manner is almost as
accurate as the one that would have been obtained if a least square solution as
proposed in the Nautical Almanac would have been used properly in the first
place.

Forth. Most people will shy away from solving the overdetermined system of three
or more linear equations and will therefore resort to some simpler method. I can
already see the writing on the wall: The most common mistake is to draw the fix
at the intersection of the angle bisectors of the cocked hat, instead of where
it really belongs. Therefore, some of our best fitting navigators will go
through the labour of three or more least square fits only to throw away the
fruits of their labour by marking an arbitrary spot in the cocked hat and claim
"This is where I am, with utmost mathematical precision!"

In short: If you want to be practical, follow Power Squadron procedures all the
way through. If you want to use the modern statistical apparatus, follow the
procedures in the N.A. p. 277-283. This is still very practical, but only if you
have a programmable pocket calculator.

(We have had many previous discussions on the subject of why the N.A. procedure
is the only rigorously correct solution to the overdetermined fix and also about
the location of the most probable position within the cocked hat. Therefore, I
won't repeat the arguments.)

Herbert Prinz

*) My main argument against splitting observations in this manner is a practical
one. If you have two sights of one body and another one or two sights of another
body nearby (< 15 degrees), you cannot check for inconsistencies individually,
but taking them together you can.

After sunset, when the first bright star comes out in an easterly direction I
may spend the time to take a run of 3 or 4 sights, since there may be nothing
else to do. A little later, when many dimmer stars come out all around me and
all at once, I rather go round and round trying to get a good spread in azimuth
before I loose the horizon, instead of getting hung up with a prolonged run of
one star. When I am done, it may turn out that I end up with several
observations of the same stars. I treat them just the same as if they were
observations of different stars. That is, when I have my HP-48 with me.

If I were to reduce manually, I would carefully select TWO bodies with a good
azimuth angle between them  (e.g. sun and half-moon), or at the most three stars
separated 120 deg, and aim for ONE quality observation of each body so that I
feel really good about it. One mile accuracy is what I am aiming for, three to
five is what I often get.

   

Reply