Bullet proof passing numeric values from NMEA data stream.

[Steven D'Aprano]

On Tue, 20 Mar 2007 12:09:29 +0000, Doug Gray wrote:

> Folks,
> I am looking for a fast but most importantly a bullet proof method to pass
> and NMEA data stream (GPS output) ascii numeric strings. The best I can
> offer is:
> 
> def fint(a):
>  try: return int(float(a))
>  except: return 0


Will your application calculate the wrong results if it starts getting a
whole lot of spurious zeroes? Wouldn't it better to signal "this value is
invalid" rather than a false zero?

Do you actually want ints? It seems to me that if your data stream is
delivering floats, you're throwing away a lot of data. For example, if the
data stream is: 

2.4, 5.7, 3.9, 5.1, ...

you're getting:

2, 5, 3, 5, ...

which is possibly not even the right way to convert to ints. Shouldn't you
be rounding to nearest (i.e. 2, 6, 4, 5, ...)?

[snip]

> For example, each of the following throw the exception so do not return
> the correct value:
[snip examples]

All your examples include spurious whitespace. If that is the only
problem, here's a simple fix:

def despace(s):
    """Remove whitespace from string s."""
    return 

def fix_data(value):
    """Fix a GPS value string and return as a float."""
    return float(''.join(value.split()))


If only a few values are malformed, you might find this is faster:

def fix_data2(value):
    try:
        return float(value)
    except ValueError:
        return float(''.join(value.split()))

Only measurement with actual realistic data will tell you which is faster.

If you expect to get random non-numeric characters, then here's another
solution:

import string
# initialize some global data
table = string.maketrans("", "") # all 8 bit characters
keep = "1234567890.-+"
dontkeep = ''.join([c for c in table if c not in keep])

def fix_data3(value):
    try: # a fast conversion first
        return float(value)
    except ValueError: # fall-back conversion
        return float(string.translate(value, table, don'tkeep))

Once you've built the character tables, the translate function itself is
executed in C and is very fast.


> Also, why should I consider the string module?  Is it faster/better?

Most of the time you should use string methods, e.g.:

"hello world".upper() 

instead of 

string.upper("hello world")

The only time you should use the string module is when you need one of the
functions (or data objects) that don't exist as string methods (e.g.
translate).



-- 
Steven.