NavList:

Robin, you wrote:
"The inflight display in the first attachment shows 2,780km to Honolulu and 6,841km to New York. At that time the interpolated Flightaware data gives a latitude, longitude of 30.3629°, -134.5397°."

So what would produce those anomalous distances? They are most certainly wrong. To resolve this, you have to think like a contract programmer working in some cubicle somewhere. You're supposed to produce a little app for in-flight general information that shows an aircraft silhouette on the map and displays distance from origin airport and distance to destination airport. The graphics part is fun and interesting, but the number stuff is a bit of a bore. You look up great-circle distance... your mind begins to wander... you think about how under-paid you are... and you say to yourself, "...I'm not gonna bother with that!" Then you spend three minutes thinking about calculating distances, realizing you need to finish this project by the end of the day, and you remember your "high school" analytic geometry:

  Distance = sqrt(x²+y²).

For y, you use difference in latitude degrees, and for x, you use difference in longitude degrees. Calculate distance in degrees as "square root of the sum of the squares" and then multiply by 60 to get nautical miles, and last multiply by 100/54 to convert to kilometers.

The numbers from this (flawed) algorithm match the ones you saw, Robin, on the in-flight display exactly. And if we do test cases strictly in the tropics, the results are reasonably good. Indeed within 10° of the equator and for differences of longitude that are 90° or less, this is a good approximation to the great circle distance. I would also suggest that this explains the pointing error on those arrows in the display. Their directions were probably computed from atan(y/x).

For anyone reading along who's unfamiliar with it, a map projection like this that treats latitude and longitude as equal is called a rectangular projection, or equirectangular, or plate carrée projection. Read more here. This type of map projection can be quite useful, but distance as calculated from x,y points in a rectangular projection has no value in the general case. It's not a meaningful measure of distance between points on the Earth's surface. Those numbers in the in-flight display Robin was watching have the property that they normally trend like distances --they get bigger as we get farther from our starting point, but in middle and high latitudes they are a long way from representing actual distances.

There is one other flavor of distance worth checking when you're puzzling over numbers like this, and that's the "tunneling distance" through the bulk of the Earth. It's the truest distance from a certain point of view: the number of miles on a straight line passing directly through the Earth from one point to another. This path, incidentally, lies directly beneath the great circle for any pair of points so the courses are always the same at the same spot on the track (relative to the Earth's center) while the distance is always less than the great circle on this path. This is the case because the plane containing a great circle contains the line connecting any two points on that circle.

Frank Reed