What is precision of a number representation?

Terry Reedy tjreedy at udel.edu
Tue Jul 12 00:17:37 EDT 2016


On 7/11/2016 6:17 PM, Ethan Furman wrote:
> On 07/11/2016 02:51 PM, Chris Angelico wrote:

>> This is why it's important to be able to record precisions of
>> arbitrary numbers. If I then measure the width of this corridor with a
>> laser, I could get an extremely precise answer - say, 2,147
>> millimeters, with a precision of four significant digits, and
>> excellent accuracy. But if I multiply those numbers together to
>> establish the floor area of the corridor, the result does NOT have
>> four significant figures. It would be 64 square meters (not 64.41),
>> and the accuracy would be pretty low (effectively, the *in*accuracies
>> of both measurements get combined). But on the other hand, if you want
>> to know whether your new fridge will fit, you could measure it with
>> the same laser and come up with a figure of 1,973 mm (four sig fig),
>> which would mean your clearance is 174mm (four sig fig). How do you
>> record this? Is it 174.0? 0174? "174 with four significant figures"?
>
> 174.0,

A common fallacy.  There are only 3 significant figures because one, the 
first, was cancelled by subtracting close numbers.  The same is true of 
adding numbers of opposite signs.

-- 
Terry Jan Reedy




More information about the Python-list mailing list