NavList:

Paul Hirose wrote:
"Yes, the topocenter is computed with the GRS80 ellipsoid. The program's output displays this as a unit vector and its magnitude. For example, at Kermit's actual position on the 15th (+46°41.20' -2°19.90'), it says
0.687854 -0.028008 0.725309 ITRS unit vector
6.36687e+003 km vector magnitude"

Yikes. This is an exquisite example of "jargon cosplay". It's no way to communicate. So let's get out the decoder ring. 

First, GRS80 ellipsoid? Synonymous with WGS84, in other words, "the" standard ellipsoid, the standard basis for coordinate on the globe. What about ITRS? That surely is something that every practical navigator should know. Well, no, it's just another junk acronym, critical to grad students trying to show off at conferences and otherwise just another name for "standard x,y,z coordinates". In this case, it's the set of x,y,z values pointing at Antoine's location where the z-axis points "north" parallel to the Earth's axis, the x-axis points toward the point south of Ghana at the equator/prime meridian crossing point, and the y-axis points towards the equator and 90°E longitude.

The coordinates as given here are precise to about 10m or 30 feet. This is "numerology cosplay" --a fascination with numerical precision for the sake of the show. It's outside the bounds of traditional navigation. How can you determine their precision? The x and z coordinates here are nearly in the plane of the prime meridian (because the longitude was near zero, and that's why the y value in these coordinates is a small number, about -0.028, while the others are closer to 1.0). Since we can ignore y, we can get a handle on the precision of the numbers by doing the prime meridian case on a spherical globe (oblateness is largely irrelevant to the precision issue). This is the simplest spherical coordinates to x,y,z coordinates you can get, really basic trig: 
  x = cos(Lat)
  y = 0
  z = sin(Lat)
Draw a picture of that if you need to. It's a simple circle --a cross-section of the globe cut along the prime meridian. If you try this with a latitude of 46.7° (near enough for this purpose) you will get values close to the x and z numbers above. They're not quite a perfect match because the longitude isn't exactly zero, and they don't include oblateness. But they do capture the sensitivity to latitude quite well! The x,y,z values above are given to six decimal places. How much do you have to change the latitude to see a single-digit change in that last decimal place? That's something you can punch up on a common scientific calculator. Does a celestial navigator working a "lunar" really need a position on the globe exact to 30 feet?! Not at all.

Over-cooked precision is a problem distinct from accuracy in celestial navigation. The mere fact that we can calculate a number to some number of digits is no measure of the quality and accuracy of that number.

Similarly in another recent post in this thread, Paul H. included a value for the dip of the horizon listed as 3.91 minutes of arc. This is just awful on many levels. The computation of dip is an approximation that incorporates a good (but not great) adjustment for average atmospheric refraction. As many NavList members know, the standard equation for dip in minutes of arc is:
   dip = 0.97'×sqrt(ht in feet).
This is an approximation, and the variability around this average value is critical to the systematic error of celestial navigation. Dip values computed to two decimal places are nonsensical. This again is something that you can try on a common calculator. Try the equation for 16 feet. Then try it for 16+x feet until you get a single digit change in the second decimal place. It's a difference in height of about an inch. Note that this impact of that increment "x" is reduced (less sensitivity in the final dip value) at higher heights of eye, so at 50 feet, you get a change of 0.01' for a two-inch change in height of eye. In an imaginary world, we might know our "height of eye" to the nearest inch or two. That is not the real world. Precision in dip at this level is pointless. 

Frank Reed
Clockwork Mapping / ReedNavigation.com
Conanicut Island USA

File: