NavList:

RE : Linear-regression-other-tools-Cutting-oct-2018-g43062

Dear Bruce,

When you observe heights in a "noisy" environment - i.e. with quite high random observation errors (SDEV exceeding 1' or even 2') e.g. when observing close from the sea level with high waves or swell - I have experienced that the (x²) 2nd order coefficient derived from the data set may be quite wrong, especially when you have only a limited number of observations at hand.

Hence the idea of replacing it by its theoretical value which can be derived from the Observer's "approximate environment" whenever feasible.

The Navigator generally knows his position to better than 1° (60 NM). Hence starting from the well known formula :

sin h = sin φ * sin D + cos φ * cos D * cos T , with :

h = Geocentric Height

φ = D.R. Latitude

D = Body Declination

T = Body Local Hour Angle computed from D.R. Longitude and Greenwich Hour Angle.

it is possible to compute 1/2 * d²h/dUT₁²  which is quite close to the theoretical value of this (x²) 2nd order coefficient. Hint: in the course of the computation dT/dUT₁ should take in account your East-West speed especially on a fast boat.

When the effects of the second order terms have been removed, it is only necessary to perform 1st order linear regression on the remaining data.

Sometimes one iteration may improve your final observed fix. From your computed fix, compute again 1/2 * d²h/dUT₁²  and process your data again into a "refined" fix.

However, as you can guess, this is starting to become a bit complicated in terms of computation (do not mess with the various units and 1st/2nd derivatives coefficients !), much more complex actually than simply deriving this (x²) 2nd order coefficient directly from your data set.

From personal experience, I can say that if the (x²) 2nd order coefficient derived from the data set may and will at times spoil your end results - i.e. your computed fix - under extreme conditions, on the other hand using its theoretical value - even computed from some "approximate" environment - has always yielded more stable computed fixes equal or better than the fixes derived from the data set (x²) 2nd order coefficient.

In other words, and from personal experience: the theoretical  (x²) 2nd order coefficient never performs worse than its data derived counterpart, but it often performs better especially under a noisy data environment.

Best Regards,

Antoine