NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
Re: Rejecting outliers: was: Kurtosis.
From: Peter Hakel
Date: 2010 Dec 31, 11:40 -0800
From: George Huxtable <george@hux.me.uk>
To: NavList@fer3.com
Sent: Fri, December 31, 2010 6:54:56 AM
Subject: [NavList] Rejecting outliers: was: Kurtosis.
[parts deleted by PH]
I am aware that a Gaussian distribution is no more than a convenient
approximation, representing observed scatter in measurements of many types,
that seems to work well in practice. And there are many reasons why some
observations might well lie outside an expected Gaussian error-band: they
are commonly ascribed to some sort of "blunder". Such blunders can come in
all sorts of unpredictable shapes and sizes, and it would seem impossible
to predict any frequency-distribution for errors of that type. They would
certainly corrupt any set of otherwise-valid measurements, and need to be
detected and discarded, to the extent that is possible. That is the
challenge that mariners face, to somehow distinguish the good from the bad.
Frank writes-"In the real world, at least from every practical set of
observations that I have seen, the probability of points "in the tails" of
the distribution are much higher. For example, you might get a 3.6 minute
of arc error one time out of a hundred observations or even one in fifty,
or in other words, with hundreds of times greater frequency than the
standard normal distribution would imply."
Is that comment intended to apply just to sextant altitudes, or generally,
to other fields of measurement as well, as seems to be implied? He appears
to be challenging the very basis of error-theory, rooted as it is in the
Gaussian distribution, which has provided a useful model for statisticians
for many years. He is perfectly entitled to do so, but to be taken
seriously will need to offer much firmer evidence than the anecdotal
statements provided above.
From: Peter Hakel
Date: 2010 Dec 31, 11:40 -0800
The issue may be partly of terminology; what George calls a "blunder" may be an unavoidable occurrence in a process whose probability distribution inherently has a "fat tail." I am not an expert in this area but long ago I crossed paths with a mathematician for whom this was the primary area of research.
The following link:
http://en.wikipedia.org/wiki/Fat_tail
contains information that is consistent with my recollections of presentations that I had attended all those years ago.
I don't think that Frank is "challenging the very basis of error-theory," he is just mentioning more recent developments in it. The forays into non-normal distributions are an active and useful area of research. I think Frank's point is that they may even be applicable to imperfect human navigators on small boats in bad weather attempting to do their best with sextants of varying mechanical quality. I would agree with that; Frank may correct me, if I interpreted his idea incorrectly. This may be one of those things that just might lead to intriguing results upon further study.
Peter Hakel
The following link:
http://en.wikipedia.org/wiki/Fat_tail
contains information that is consistent with my recollections of presentations that I had attended all those years ago.
I don't think that Frank is "challenging the very basis of error-theory," he is just mentioning more recent developments in it. The forays into non-normal distributions are an active and useful area of research. I think Frank's point is that they may even be applicable to imperfect human navigators on small boats in bad weather attempting to do their best with sextants of varying mechanical quality. I would agree with that; Frank may correct me, if I interpreted his idea incorrectly. This may be one of those things that just might lead to intriguing results upon further study.
Peter Hakel
From: George Huxtable <george@hux.me.uk>
To: NavList@fer3.com
Sent: Fri, December 31, 2010 6:54:56 AM
Subject: [NavList] Rejecting outliers: was: Kurtosis.
[parts deleted by PH]
I am aware that a Gaussian distribution is no more than a convenient
approximation, representing observed scatter in measurements of many types,
that seems to work well in practice. And there are many reasons why some
observations might well lie outside an expected Gaussian error-band: they
are commonly ascribed to some sort of "blunder". Such blunders can come in
all sorts of unpredictable shapes and sizes, and it would seem impossible
to predict any frequency-distribution for errors of that type. They would
certainly corrupt any set of otherwise-valid measurements, and need to be
detected and discarded, to the extent that is possible. That is the
challenge that mariners face, to somehow distinguish the good from the bad.
Frank writes-"In the real world, at least from every practical set of
observations that I have seen, the probability of points "in the tails" of
the distribution are much higher. For example, you might get a 3.6 minute
of arc error one time out of a hundred observations or even one in fifty,
or in other words, with hundreds of times greater frequency than the
standard normal distribution would imply."
Is that comment intended to apply just to sextant altitudes, or generally,
to other fields of measurement as well, as seems to be implied? He appears
to be challenging the very basis of error-theory, rooted as it is in the
Gaussian distribution, which has provided a useful model for statisticians
for many years. He is perfectly entitled to do so, but to be taken
seriously will need to offer much firmer evidence than the anecdotal
statements provided above.
