[SciPy-User] fromstring and int64

[Chris Barker]

On Wed, Oct 21, 2015 at 9:23 AM, Carlos Medrano <ctmedra at unizar.es> wrote:

>  I am having an issue with function fromstring and long integers (int64).
> It seems it cannot read properly long numbers. I use scipy version 0.14.0
> and numpy 1.8.2. I have the problem also if I use numpy directly.
>
> import scipy
> s='0 1445367600061 -35960 39671 79230'
> scipy.fromstring(s, sep=' ', dtype=scipy.int64)
>
> I get this
>
> array([ 0, 2147483647, -35960, 39671, 79230], dtype=int64)
>
> it works for me:
In [1]: %paste
import scipy
s='0 1445367600061 -35960 39671 79230'
scipy.fromstring(s, sep=' ', dtype=scipy.int64)

## -- End pasted text --
Out[1]:
array([            0, 1445367600061,        -35960,         39671,
               79230])

numpy version 1.9.3

However, I suspect it's a platform thing -- I'm on 64 bit OS-X, which used
64 bit integers for a long -- 32 bit platforms and Windows64 don't.

scipy.fromstring is numpy.fromstring, and numpy.fromstring is kludgy, ugly
and pretty broken. It also punts the string parsing to the C atoi(), with a
bit of standard python plugged in there. That's why I think it's a C long
problem.

Even if you are running on on a "proper" 64 bit platfrom, it may be broken,
but I can guarantee you it will be a pain to fix.

So the short answer is -- don't use it.

scipy.loadtxt('dumf.txt', dtype=scipy.int64)
>

yes, loadtxt is a lot smarter.

However, fromstring is a lt faster, so it's too bad.

If you really need very fast reading of numbers from text from files, I'd
look at panda's CSV reader -- I hear it's pretty sweet.

Also -- I have some Cython code that's blazingly fast -- only floats right
now, but it wouldn't be hard to adapt to integers...

I can send it to you if you want.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20151022/3b275329/attachment.html>