NavList:

You say that you have a method for detecting and eliminating an outlier. This can be done, but you didn't say what standard you use. How far out-of-line qualifies as an outlier worthy of being deleted? Is this ever justified? Did you know that this used to be a question of great statistical controversy in science with an "American School" in opposition to a "European School"? Imagine the arguments! This was mostly pre-transatlantic telegraphy, so you would have to follow the heated exchange of letters and essays over the course of many years... :)

Looking at your graph, your sights range from 9.54 to about 9.96 hours EDT (about 9:32 to 9:58 or an interval of 26 minutes) with raw sextant altitudes ranging from about 38.3° to 42.9°. Because manual observations are imperfect, there's some "noise" in your sights, a little up and down wobble, and it would be nice to eliminate that, as you suggest. If we had perfect measurements, there would be a smooth "arc" of altitudes plotted against time. This would be almost a straight line, but slightly convex from above (refraction can work for opposite curvature but only at very low altitudes, and at these altitudes, nothing much to worry about in terms of the shape of the arc). It seems to me that you're asking how much deviation there would be from a perfect straight line. If we have sights that can be well-approximated by a straight line, then you can run a line through them, either "by eyeball" or by some analytic fitting procedure, and this should cancel out a considerable fraction of the noise in the individual sights.

How convex is the perfect curve? This we can answer. The true altitude of the Sun's center at 9:32 from Belmar would have been 38°22.3'.  At the end of your sights, 26 minutes later, the true altitude would have been 43°08.8'. Given the uncertainty limits on reading your graph and assuming that your sights are LL, these numbers are consistent with what you have. But we don't really care. The goal here is not to match your numbers, but to see how perfect numbers would vary in the interval during this time interval from a location that is relatively close to your actual location.

With the endpoints of the interval in hand, we need to check the midpoint to assess how much non-linearity there would be in perfect observations. The middle time is 9:45 EDT and we should compare two numbers: the direct linear interpolation (in this case, the simple average) of the starting altitude and the ending altitude, and the exact altitude that can de calculated from the changing coordinates of the Sun and the geometry of the sky (spherical trig) at this moment. The former, the simple average , is 40°45.55'. The latter, the expected true altitude, is 40°46.5'. That bumps up by 0.95'. Call it one minutes of arc of convexity. As I see it, given the context, that's too much to ignore. Your sights probably have noise on the order of +/-1' (in the standard deviation sense). 

So what can you do? First, maybe simple averaging still works? Does it? Would it be close enough? Another simple approach would be to use a shorter run, fewer total minutes from start to finish. The size of that convexity bump is going to be approximately proportional to the square of the total time interval (I assume). If you reduce your time from start to finish from 26 minutes to 13 minutes, then the bump should decrease from 1' to a quarter of a minute --half the time, one-quarter of the offset. And that might be small enough to ignore.

Keeping the full range of 26 minutes, you could do better by looking at your sights relative to a perfect curve of the predicted altitudes. That is, you would plot Ho versus Hc values and then look at the differences. Hmmm... Ho-Hc values?? Wait a minute...  Suddenly we are looking at ordinary "intercepts". If we could assume that the azimuths are nearly identical (they're close enough), then you could create an "averaged" sight by averaging the intercepts. And from here we're enjoying a fun ride down the slippery slope to a complete statistical solution for a fix from sights over time, like the method described in the explanation pages in The Nautical Almanac. And that's a good thing. It's always an option, and we can see now how we would arrive there.

Incidentally, there are ways of calculating and tabulating a "convexity" error. We might want to know when the scale of that convexity offset for a standard time interval of, let's say, ten minutes exceeds some minimum. It's a measure of the second derivative of altitude versus time or hour angle (∂²h/∂t²). While this can be tabulated and probably exists in various collections of tables, in the modern world it's no big deal to take the direct route: just calculate Hc values and see how they change.

Frank Reed