NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
From: Bill Lionheart
Date: 2021 Apr 28, 09:05 0000
Bob
I think it might have been Bunsen who said "lack of mathematical education reveals itself nowhere better than in exaggerated arithmetic accuracy". However I am not sure if it is true as you claim that there is no point in reading measurements to a greater accuracy than the overall accuracy of a reading.
Suppose you have a measurement process such as measuring an altitude by sextant and the random errors, due for example judging the sun touching the horizon and reading the time without delay etc are from some known distribution (for which Frank has an interesting suggestion). Suppose you then also truncate the digits of this measurement. This introduces roughly (again as Frank mentions some bias) uniformly distributed error. This changes the distribution of overall errors and I suspect will generally make the variance greater. By how much and is it enough to bother I am not sure in this case. But in some cases it matters.
The (English) Wikipedia article on False Precision is interesting although far from comprehensive. It makes the point that you should round after you have done your calculations not during the measurement. In our case if you do a series of sights and want to use eg regression to improve the accuracy (this assumes the times were right but the altitudes have errors), then you would use all the digits you can read.
Bill
