What is precision of a number representation?

[Ben Finney]

Ethan Furman <ethan at stoneleaf.us> writes:

> On 07/11/2016 01:56 PM, Ben Finney wrote:
>
> > Precision is not a property of the number. It is a property of the
> > *representation* of that number.
> >
> > The representation “1×10²” has a precision of one digit.
> > The representation “100” has a precision of three digits.
> > The representation “00100” has a precision of five digits.
> > The representation “100.00” also has a precision of five digits.
>
> […] you haven't explained how 00100 is more precise than 100.

I haven't made any claim about “more precise” in the above.

But if you want such claims, here are comparable representations in
increasing order of precision:

* “0 ×10²” has one digit of precision.
* “0.0 ×10²” has two digits of precison.
* “0.01 ×10²” has three digits of precision.
* “0.010 ×10²” has four digits of precision.
* “0.0100 ×10²” has five digits of precision.
* “0.01000 ×10²” has six digits of precision.

Again, these might all be representations of the same number or of
different numbers; the representation claims only to represent a number
within a range of tolerance.


In decimal notation, some of the above can't be written with the same
precision. I could invent a representation — e.g. show “?” as a
place-holder for a digit not within the precision level. In that system,
the equivalent representations would be:

* “0????”
* “00???”
* “001??”
* “0010?”
* “00100”
* “00100.0”

Each of these claims to represent a quantity beginning at the
ten-thousands place, with increasing levels of precision in the claim.

So the comparison that invites is to show where the range of tolerance
for the representation ends. The fewer digits of precision, the less
precise the representation claims to be.

-- 
 \      “I find the whole business of religion profoundly interesting. |
  `\     But it does mystify me that otherwise intelligent people take |
_o__)                                    it seriously.” —Douglas Adams |
Ben Finney