NavList:
A Community Devoted to the Preservation and Practice of Celestial Navigation and Other Methods of Traditional Wayfinding
Rounding decimal fractions
From: Paul Hirose
Date: 2013 Jul 18, 12:28 -0700
From: Paul Hirose
Date: 2013 Jul 18, 12:28 -0700
Stan K wrote: > A couple of years ago a friend of mine and I were discussing how to implement Increments and Corrections in my Celestial Tools program and his Navigation Calculator Workbook spreadsheet. Looking at the output of his spreadsheet, we noticed that the Almanac values apparently did not use what we call "standard" rounding, in this case, for instance, 0.00 through 0.04 would round down to 0.0 and 0.05 through 0.09 would round up to 0.1. I thought the convention, when a decimal can be rounded up or down with equal accuracy, is "round to even." E.g., 1.05 rounded to the nearest tenth is 1.0, but 1.15 is 1.2. But I see that my HP 49G calculator rounds up in these borderline cases. With 2 decimal point precision selected, 1/8 = .13, 3/8 = .38, 5/8 = .63, 7/8 = .88. The composite formatting feature in Microsoft's .NET Framework does the same thing as the calculator. But I discovered the .NET Math.Round class gives the programmer a "round to even" option. Results from a test program: 0.125 0.13 0.12 0.375 0.38 0.38 0.625 0.63 0.62 0.875 0.88 0.88 The first column is the fraction to full accuracy. In the second and third columns I used composite formatting with precision of two decimal places. The difference is that in the third column the value was first rounded with Math.Round. The sum of the second column is a little high (1/8 + 3/8 + 5/8 + 7/8 = 2 exactly) because rounding all borderline cases up introduces a small systematic error. I have to admit I never pay attention to these fine points in my own software. Since Tinyac and Lunar3 both have selectable precision, I assume the user will use, say, .01 precision if "tenths" are critical. --