Solution for architecure dependence in Numeric ?
Michael Spencer
mahs at telcopartners.com
Fri Feb 18 09:27:15 EST 2005
"Johannes Nix|Johannes.Nix"@uni-oldenburg.de wrote:
> Hi,
>
> I have a tricky problem with Numeric. Some time ago, I have generated
> a huge and complex data structure, and stored it using the cPickle
> module. Now I want to evaluate it quickly again on a workstation
> cluster with 64-Bit Opteron CPUs - I have no more than three days to
> do this. Compiling Python and running Numeric has been no problem at
> all. However, I get an error message when accessing the data pickled
> before. (I can load it regularly on 32 bit computers, but it is a
> quite complex data object, so I really don't want to store every
> element as ASCII data). The problem seems to be with 64 Bit integers
> (with 32-bit-floats, no problem was observed).
>
> This looks like that (from the Unix command shell):
>
>
> jnix at 32bithost:~> python ~/python/test_npickle.py -dump test.pck
> jnix at 32bithost:~> python ~/python/test_npickle.py test.pck
> [0 1 2 3 4 5 6 7 8 9]
> jnix at 32bithost:~> ssh 64bithost python ~/python/test_npickle.py test.pck
> Traceback (most recent call last):
> File "/home/jnix/python/test_npickle.py", line 16, in ?
> a = cPickle.load(file(filename))
> File "/home/jnix/lib/python2.4/SuSE-9.0/x86_64/Numeric/Numeric.py", line 520, in array_constructor
> x.shape = shape
> ValueError: ('total size of new array must be unchanged', <function array_constructor at 0x2a960b0b18>, ((10,), 'l', '\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\x07\x00\x00\x00\x08\x00\x00\x00\t\x00\x00\x00', 1))
>
>
> also I get:
> jnix at 32bithost:~> python -c "import Numeric; print Numeric.arange(0).itemsize()"
> 4
> jnix at 64bithost:~> python -c "import Numeric; print Numeric.arange(0).itemsize()"
> 8
>
>
> The script used to produce the example above is:
>
> -------------------------------------------------------------------------
> #/usr/bin/python
> # -*- coding: latin1 -*-
>
> import Numeric
> import cPickle
> import sys
>
> if len(sys.argv) > 1 and sys.argv[1] == '-dump':
> filename = sys.argv[2]
> binary=1
> a = Numeric.arange(10)
> cPickle.dump(a, file(filename,'w',binary))
>
> else:
> filename = sys.argv[1]
> a = cPickle.load(file(filename))
> print a
>
>
> ---------------------------------------------------------------------
>
>
> So what would you suggest ? Can I hack Numeric to assume non-native
> 32 bit integer numbers ?
>
> Many thanks for any help,
>
> Johannes
It might be worth posting to the Numeric mailing list, mirrored at
http://news.gmane.org/gmane.comp.python.numeric.general
More information about the Python-list
mailing list