NavList:

Norman B, you wrote:
"dividing all measured angles by 1.002 is doing a pretty good job for me"

Yes, that's interesting. It may imply an error in mirror perpendicularity which cannot be adjusted away on your instrument. But is there more to it? Why not just make an ordinary table on the pattern that sextants have had for generations? You have to read the instrument in degrees and minutes, so listing the correction in minutes every 10 or 20° seems pretty straight-forward and, more importantly, it does not confine you to a theoretical model of a perfectly linear, proportional error.

Let's see... your factor, 1.002, implies an error of one part in 500. That's only an approximate number, so I'm going to modify it to one part in 480 just for the moment. That's equivalent to a factor of 1.00208, which is almost certainly indistinguishable (a difference of only half a minute of arc at 90° arc measure). The advantage of the latter is just that it makes it easier to remember the correction in minutes: one in 480 means one minute of arc every 8°. So if the sextant reads 24°, the correction would be 3'. By the way, is it up or down? I can't recall from your prior posts... And if the sextant reads 33°30' (just as a random example), that value is close to 32 and every 8° adds a minute of arc, so the correction would be 4'. Finally, for a reading of 87°, the correction would be 11'.

You can do that correction in your head. You could also make a little printed table with values every 15°, maybe? Doing this in a spreadsheet is needless over-kill, and it is forcing you to think that you require a linear factor. This tosses out variability that could make sense. Decades ago, some sextant calibration labs insisted on a polynomial model for sextant error, which was based on a particular model of sextant arc "eccentricity". Evrey case of sextant error was represented by only two numbers, which were combined in the polynomial to produce a table that only approximately represented the practical error. They, too, were limited by a theoretical model.

You wrote:
"I'm getting fixes that are less than 2nm away from the actual location with my Davis Mark 3"

Wow! That's right at the reading precision limit! The Davis Mk 3 can only be read to 2' precision, so you're done. You may now retire to an island in the South Pacific. :) Heh. 

The trouble, of course, in situations like this is that it's very hard to be "scientific". I'm writing here from my own personal experience as well as from speaking with other navigators about their attempts to calibrate their own sextants. You can't double-blind your experiments in any practical way. And you already mentioned that you're throwing out outliers, if I understood you correctly. So you really must ask yourself: is it really that good?? If yes, then fantastic! But how do you know? Please understand: I'm not telling you you're wrong. But keep in my mind that you are the easiest person to fool. No wait. It's me! I am the easiest person to fool. Hmm... I remember now... The "you" in this suggestion from Richard Feynman back in 1974 is each of us: 

"The first principle is that you must not fool yourself—and you are the easiest person to fool."

There's more in the full essay/address here: https://calteches.library.caltech.edu/51/2/CargoCult.htm. On first pass, it may seem obvious. But is it??

Frank Reed