NavList:

   

Reply

On 8/6/2022 10:30 AM, Herman Dekker wrote:
> I wondered why the conversion of minutes of arc to degrees, mostly gaves
> values with a repetitive figure.

That happens because a fraction does not convert to a terminating
decimal unless the denominator (after the fraction is "reduced") has
prime factors of 2 and 5 only.

For example, 1/60 does not terminate because 60 = 2 * 2 * 3 * 5. But
3/60 terminates because the reduced fraction is 1/20, and 20 = 2 * 2 * 5.

A similar rule applies to non-integer numbers in computers. The
fractional part is expressed as a binary fraction in the machine, so a
decimal value is not exact unless the denominator of its fractional form
is a power of 2.

For example, .25 is exact inside a computer because .25 = 1/4, and 4 = 2
* 2. But .1 is approximate because .1 = 1/10, and 10 = 2 * 5.

That was a problem in the Gulf War when a Patriot missile failed to
intercept a Scud, with fatal results. The value in an internal counter
(which incremented 10 times per second) was multiplied by .1 to obtain
time. But the system had been running continuously for 4 days, so the
error in .1 was magnified to about 1/3 second (I think).

Some programming languages have a number format for financial
computation where decimal values must be exact. However, this format
doesn't have the great range of magnitude that you need for scientific
and engineering work.

--
Paul Hirose
sofajpl.com

   

Reply