NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
Re: Round-off
From: Frank Reed
Date: 2009 May 22, 12:33 -0700
From: Frank Reed
Date: 2009 May 22, 12:33 -0700
John, you wrote: "But remember, that's only one of many different probability estimates. The max error is still L*N/2." You're right that there are many different ways of describing probability limits and estimating the odds of extreme outcomes. But the key is that you should always compare apples with apples. If we're looking at ONE standard deviation error in one variable, then we should never be comparing that to a TWO standard deviation measure of error, or yet some other measure of error (like max error), in another variable. So what about that maximum potential error in round-off? If we're dropping tenths, each round-off could be as large as 0.5 minutes of arc, and if there are a dozen round-offs, then that might lead to an error as large as +6.0 or -6.0, equal to the L*N/2 you gave. But this just isn't relevant in practice because we have no means of distinguishing round-off error from observational error (except, of course, by the obvious approach of running the calculation again without rounding off). To put it differently, we already should consider the possibility, rather low in probability, that our observations are off by +/-6.0 minutes of arc, and unlike round-off error, this is not an upper limit. Then we should ask whether the probability of being that far off is significantly increased by the possibility of round-off error. And that all depends on how the standard deviation of observational error compares with the standard deviation of round-off error. As noted in my post a little earlier today, if the standard deviation of the observational error is just a little above 1 minute of arc, the increase in the net standard deviation from rounding off (dropping the tenths) is not that great, less than 33%. It is nearly identical to assuming that your observational error is moderately higher and the round-off error is zero. -FER --~--~---------~--~----~------------~-------~--~----~ Navigation List archive: www.fer3.com/arc To post, email NavList@fer3.com To , email NavList-@fer3.com -~----------~----~----~----~------~----~------~--~---