NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
From: Rafael C. Caruso
Date: 2023 Apr 14, 06:18 -0700
Frank, thank you very much for your detailed response; it was most informative. I'll address your comments in the next two paragraphs, but first I would like to return to my original question, which was whether I should account for the fact that the Sun’s azimuth and declination were changing during the interval in which I collected a set of Sun altitude measurements (they were lower limb, as you deduced). Since you do not mention that any compensation is required, I interpret this as meaning that you think that the effect these changes is irrelevant for this relatively short interval of less than 30 minutes. My question was prompted by Lars Bergman’s critique of an article in the Journal of Navigation (http://fer3.com/arc/m2.aspx/Journal-Navigation-latitude-with-Bris-sextant-Bergman-sep-2022-g53240).
Second, the main reason why I prefer to collect more than one single altitude measurement is to do what you mentioned, to reduce the “noise” in my sights by including more measurements. I use a standard approach to analyze these data points: fitting them with a line (a straight line or a curve) that approximates their behavior, by means of a standard graphing software package. Although the line that describes the sun’s apparent motion in the sky is slightly convex, as you point out correctly, in a summer morning at my mid-latitude this convexity was very modest, and a straight line seemed good enough for the practical purpose of minimizing the “wobble” due to my imperfect measurements. I do understand that a straight line isn’t an accurate physical model of the sun’s apparent motion; this becomes quite obvious as the sun approaches the meridian.
Third, I’m afraid that my mention of outlier values wasn’t clear enough. Another reason why I like to collect a set of altitude measurements is just to make sure I haven't made any outrageous mistakes, such as reading the wrong degree value off the sextant’s arc, or jotting down a wrong number. This would show up as a flagrant outlier jut using the “eyeball test”. I did not mention any formal way to eliminate outlier values, because I haven't used one. I recall that a formal method to identify outliers in linear regression is to use Cook’s distances. A statistician friend (American school, no doubt) introduced me to this approach, which I haven’t used since my retirement. If memory serves, it involves calculating a regression line with and without each point of the data set in succession. A rule of thumb, according to this statistician, is that a point which has a Cook’s distance three times the mean of all distances may be labeled as an outlier. But, as I’ve said, this is beyond the scope of my far more simple-minded approach.
Best, Rafael