[concurrency] map error returning object

[Jay L.]

List,

I am working with the multiprocessing module attempting to get parallel
reads working.  Currently, if I open the file in each child process and
read from an offset I achieve concurrency.  If I attempt to open the file
in the parent (via a class) and pass the class object to the children I get
an overflow error.

Here is a snippet:

def get_object(row):
    return file.get(row) #Where get seeks to the offset and returns
the data object.

rows = range(len(file)) #This gets the row ids.
pool = mp.Pool()
results = pool.map(get_object, rows)print results #Should be a list of
objects, so not human readable

The error: OverflowError: Python int too large to convert to C long

The error is being generated in line 528 of pool.py (Enthought Python
7.3, which is Python 2.7).

Digging into that source this line is the final line in the get method
of the ApplyResult class:

def get(self, timeout=None)
    self.wait(timeout)
    if not self._ready:
        raise TimeoutError
    if self._success:
        return self._value
    else:
        raise self._value

Am I correct in interpreting this to mean that the code is passing the
first self._ready check, but the failing the self._success check (that
appears to be just another self._ready check wrapped in the successful
function)

def successful(self):
    assert self._ready
    return self._success

Thanks for any info or insight.  The error is quite cryptic (to me) as
I would expect to see if I was misusing range or working with gigantic
numbers, not class objects.

Jay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/concurrency-sig/attachments/20130605/5014d359/attachment.html>