[Numpy-discussion] Memmap with multiprocessing
Brent Pedersen
bpederse at gmail.com
Wed Apr 27 18:19:50 EDT 2011
On Wed, Apr 27, 2011 at 4:07 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:
> I don't think this was working correctly in numpy 1.4 either. The
> underlying problem seems to be that instance attributes of ndarray
> subtypes get lost during pickling:
>
> import pickle
> import numpy as np
> class aarray(np.ndarray):
> def __new__(subtype):
> self = np.ndarray.__new__(subtype, (1,))
> self.attr = 'attr'
> return self
> def __array_finalize__(self, obj):
> self.attr = getattr(obj, 'attr', None)
> a = aarray()
> b = pickle.loads(a.dumps())
> assert a.attr == b.attr, (a.attr, b.attr)
>
> AssertionError: ('attr', None)
>
> Christoph
>
possibly related to this ticket:
http://projects.scipy.org/numpy/ticket/1452
>
> On 4/26/2011 2:21 PM, Ralf Gommers wrote:
>> On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes
>> <totonixsame at gmail.com> wrote:
>>> Hi,
>>>
>>> Has anyone confirmed if this is a bug? Should I post this in the bug tracker?
>>
>> I see the same thing with recent master. Something very strange is
>> going on in the memmap.__array_finalize__ method under Windows. Can
>> you file a bug?
>>
>> Ralf
>>
>>
>>>
>>> Thanks!
>>>
>>> On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes
>>> <totonixsame at gmail.com> wrote:
>>>> Hi all,
>>>>
>>>> I'm having a error using memmap objects shared among processes created
>>>> by the multprocessing module. This error only happen in Windows with
>>>> numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac
>>>> OS X it doesn't happen. This error is demonstrated by this little
>>>> example script here https://gist.github.com/929168 , and the traceback
>>>> is bellow (between<traceback> tags):
>>>>
>>>> <traceback>
>>>> Process Process-1:
>>>> Traceback (most recent call last):
>>>> File "C:\Python26\Lib\multiprocessing\process.py", line 232, in _bootstrap
>>>> self.run()
>>>> File "C:\Python26\Lib\multiprocessing\process.py", line 88, in run
>>>> self._target(*self._args, **self._kwargs)
>>>> File "C:\Documents and Settings\phamorim\Desktop\test.py", line 7, in
>>>> print_ma
>>>> trix
>>>> print matrix
>>>> File "C:\Python26\Lib\site-packages\numpy\core\numeric.py", line 1379, in
>>>> arra
>>>> y_str
>>>> return array2string(a, max_line_width, precision, suppress_small, ' ',
>>>> "", s
>>>> tr)
>>>> File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 309, in
>>>> ar
>>>> ray2string
>>>> separator, prefix)
>>>> File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 189, in
>>>> _a
>>>> rray2string
>>>> data = _leading_trailing(a)
>>>> File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 162, in
>>>> _l
>>>> eading_trailing
>>>> min(len(a), _summaryEdgeItems))]
>>>> File "C:\Python26\Lib\site-packages\numpy\core\memmap.py", line 257, in
>>>> __arra
>>>> y_finalize__
>>>> self.filename = obj.filename
>>>> AttributeError: 'memmap' object has no attribute 'filename'
>>>> Exception AttributeError: AttributeError("'NoneType' object has no attribute
>>>> 'te
>>>> ll'",) in<bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> 0,
>>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> 0, 0, 0, 0, 0, 0, 0, 0], dtype=int16)> ignored
>>>> </traceback>
>>>>
>>>> I don't know if it's a bug, but I thought it's is import to report
>>>> because the version 1.4.1 was working and 1.5.0 and above was not.
>>>>
>>>> Thanks!
>>>>
>>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
More information about the NumPy-Discussion
mailing list