NavList:

It is a bit tricky.  I tried it myself with a Davis Mark 3, in parallel with Greg's test, and got the expected result.  Executing the rigid-body motions on the sextant, though, seems to be best done in several steps: first move the eye to the edge of the sight tube, then line up with the center of the horizon mirror (the junction between mirror and clear part, or empty air in the case of the Mark 3), then get the horizon object in the center of the horizon mirror as well, and then finally rock the sextant to get the index-path object.

(I used two corners of buildings a few hundred meters distant.)  I'm not exactly sure why it feels more difficult than a normal centered sight, which after all needs the same steps applied, either serially or all at once with practice.

You're right that if you move the eye sufficiently out of plane, you eventually run out of index mirror.  The ray from the horizon mirror will just miss it entirely.

I don't know the whole story here.  Certainly a small hole establishes the line of sight more definitively.  It does need the eye to be quite close to the small hole to get any reasonable field, which can be uncomfortable.  A larger sight tube would seem to trade off comfort against a little more care needed to center the eye accurately.

Here are links to the relevant sections of Chauvenet's book and Simms's book.  To the modern eye, the treatments are a bit, shall we say, overly geometric, but that's OK.

Cheers,

Peter