# NavList:

## A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding

**Instrument Error estimation: An Experiment**

**From:**Phil Sherrod

**Date:**1999 Mar 13, 21:08 EST

I enjoyed the message posted by Daniel K. Allen on 12 Mar 1999 regarding his experiments to determine the accuracy of his sextant observations. I performed a similar experiment on 20 Feb 1999, but for a somewhat different purpose. I own a surplus, Kollsman, periscope bubble sextant of the type that was used by the Air Force up until the 1980's. The device has about a one-foot-long periscope tube where the light enters near the top, then it is reflected by a tilting mirror down into the body of the device where the image is superimposed on a bubble and then projected out the eyepiece. It has a knob with an attached mechanical, digital dial to set the angle of tilt of the mirror, and it has another knob which selects one of 8 internal sun shades of different colors and densities. The digital readout is sufficient to provide an altitude measurement to about the nearest 1/2 minute of arc. There is also a built in mechanical averaging device that allows you to take 2-minute readings which are automatically (mechanically) averaged over the interval. The sextant has connections for a battery pack to illuminate the internal bubble, but other than illumination (which isn't required during daylight use), the device is totally mechanical, requiring no electrical power. It is a very nice piece of mechanical engineering. The purpose of my experiment was to determine the best estimate of the index error of the instrument. To do so, I took 29 observations of the lower limb of the Sun over the course of a day. I simultaneously took GPS position readings and averaged them to determine a good estimate of my true position. For maximum accuracy, I had the sextant resting on a solid platform while making the measurements. The timing was done with a digital watch synchronized to NBS standard time obtained through a high-speed Internet connection. The method that I used to determine the best estimate of the Instrument Error (IE) is somewhat unusual. What I did was perform a non-linear regression analysis to fit the calculated altitudes with the observed altitudes after correction for refraction and semi-diameter. For those of you who are not familiar with non-linear regression, it is very similar to the more common linear regression, except a non-linear function is being fitted to the data rather than a linear (straight line) function. The "best estimate" of the parameters of the regression are those values that minimize the sum of the squared deviations, where 'deviation' in this case is the difference between the observed altitude (after corrections for refraction and semi-dimameter) and the calculated altitude. For my regression problem, the only parameter whose value was being estimated was the Instrument Error (IE). Using this technique, I was able to calculate that the best estimate of the instrument error for my sextant is +2.3' (i.e., you must add 2.3' to the observed altitude, so I believe this would be "off the arc"). Applying this correction to my observed values, the worst observation was off by 8.8', and the average of the absolute values of the errors was 3.07'. This average error is about twice as large as what Daniel Allen reported when using his Tamaya Jupiter sextant, but not bad for a novice using a $90 surplus unit. For those of you who are interested in studying the analysis in more detail, I am attaching the source code and data for the non-linear regression analysis. Although this program looks a lot like a C program, it is actually the programming language used to describe a function to be fitted by the NLREG nonlinear regression program. If anyone cares to run the program, you can download a shareware version of the NLREG program from http://www.sandh.com/sherrod/nlreg.htm Phil Sherrod /* * NLREG nonlinear regression analysis to estimate Instrument Error. * Sights of the LL of the Sun taken 20 Feb 1999 using a bubble sextant. * The location was 36 deg. 00.12' N, 86 deg. 50.70' W */ Title "Sun sights 20 Feb 1999"; Variables Time, H; /* Input variables whose values are read as data */ Parameter IE; /* IE (Instrument Error) is parameter to be calculated */ Double Ho,Ha,Hc,refract,dec,GHA,LHA; /* Work variables */ /* latitude and longitude (determined from averaged GPS reading) */ Constant latitude = 36.0029; /* (North) */ Constant longitude = 86.8453; /* (West) */ /* Apply Index Error (note, IE is being computed by analysis) */ Ho = H + IE; /* Value of IE will be computed by NLREG to minimize errors */ /* Apply correction for refraction */ refract = (0.0167/tan(Ho+7.31/(Ho+4.4)))/60.; Ha = Ho - refract; /* Apply correction for semi-diameter */ Ha += 16.2/60; /* Estimate Declination and GHA around 18:00 GMT */ dec = -10.88833 + 0.015*(time-18.00); GHA = 86.565 + 15*(time-18.00); /* Compute Local Hour Angle */ LHA = GHA - longitude; /* Compute altitude */ Hc = asin(sin(latitude)*sin(dec) + cos(latitude)*cos(dec)*cos(LHA)); /* Here is the function whose error is to be minimized by adjusting IE */ Function Hc = Ha; /* Plot the observed and calculated altitudes */ Splot xvar=time,yvar=Ha,yvar2=Hc,connect; /* Write the values to a file */ Output to "iecalc.out" Ha,Hc,residual; /* Here comes the data: * Time (GMT, decimal hours), Observed altitude (decimal degrees) */ Data; 14.54 21.653 15.21 28.130 15.87 33.710 16.52 38.078 16.73 39.333 17.63 42.467 17.86 42.677 17.90 42.753 17.96 42.883 17.99 42.803 18.01 42.833 18.03 42.817 18.05 42.850 18.08 42.783 18.13 42.838 18.19 42.820 18.24 42.717 18.37 42.513 18.50 42.342 19.71 37.150 19.75 36.833 20.55 30.800 21.27 24.233 21.32 23.610 21.35 23.317 21.49 21.750 21.53 21.500 21.56 21.217 21.62 20.530 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-= =-= TO UNSUBSCRIBE, send this message to majordomo@XXX.XXX: =-= =-= navigation =-= =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--=-=