[IPython-dev] Run IPython Parallel from custom module?

[Ryan Nelson]

Hello everyone,

I'm developing a module for processing some data files. There are several
steps to this process, which can be time consuming, so I would like to
write a helper function in my module that uses IPython.parallel to spread
out the work over several processors. Unfortunately, I'm having trouble
getting things to work properly. Note: I can get something similar to work
when it is not written as a module...

I tried my best to make a simple example that fails with the same error
that I'm seeing in the larger case (below); however, it is still a little
complicated.

After running the test script, I get a "NameError" on the cluster nodes
(traceback below). If I check the nodes manually, the objects are defined.
I can also define new functions from the IPython terminal that act on those
objects, and those work as expected.

Any help you can provide is most appreciated. Sorry if this is a duplicate,
but I couldn't find it anywhere else.

Python 3.4 and 2.7 and IPython 2.3.1

Ryan

________________________________________

Here's the approximate structure of my module:

mymod/
    __init__.py
    functs.py : helper functions with Parallel support
    objects.py : contains my custom processing objects

# functs.py #
-------------
from IPython.parallel import Client

import mymod.objects as objects

def process(files):
    client = Client()
    dview = client[:]
    dview.block = True

    # Do this to make module path known
    # However, my mod will eventually be installed via pip, so this
    # isn't necessary to reproduce error in that case
    with dview.sync_imports():
        import sys
    sys.path.append(['/home/nelson/code/testing/',])

    dview['sys.path'] = sys.path
    File = objects.File1
    dview['File'] = File
    result = dview.map_sync(_do_this, files)
    return result

def _do_this(fname):
    f = File(fname)
    f.extra = 'hello'
    return f
-------------

# objects.py #
-------------
class File1(object):
    def __init__(self, fname):
        self.fname = fname
-------------

I also have a test script ("test.py") with the following:
-------------
from mymod.functs import process
result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
[print(i.fname, i.extra) for i in result]
-------------

When I run this script, I get the following errors:

importing sys on engine(s)
Traceback (most recent call last):
  File "test2.py", line 3, in <module>
    result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
  File "/home/nelson/code/testing/mymod/functs.py", line 16, in process
    result = dview.map_sync(_do_this, files)
  File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
366, in map_sync
    return self.map(f,*sequences,**kwargs)
  File "<string>", line 2, in map
  File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
66, in sync_results
    ret = f(self, *args, **kwargs)
  File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
624, in map
    return pf.map(*sequences)
  File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
line 271, in map
    ret = self(*sequences)
  File "<string>", line 2, in __call__
  File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
line 78, in sync_view_results
    return f(self, *args, **kwargs)
  File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
line 254, in __call__
    return r.get()
  File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
line 118, in get
    raise self._exception
  File
"/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
line 153, in wait
    results = error.collect_exceptions(results, self._fname)
  File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py", line
233, in collect_exceptions
    raise e
  File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py", line
231, in collect_exceptions
    raise CompositeError(msg, elist)
IPython.parallel.error.CompositeError: one or more exceptions from call to
method: _do_this
[0:apply]: NameError: name 'File' is not defined
[1:apply]: NameError: name 'File' is not defined
[2:apply]: NameError: name 'File' is not defined
[3:apply]: NameError: name 'File' is not defined

However, if I jump into an interactive Python terminal, these objects are
defined on the cluster nodes.
In [1]: from IPython.parallel import Client

In [2]: client = Client()

In [3]: dview = client[:]

In [4]: dview['File']


Out[4]:
[mymod.objects.File1,
 mymod.objects.File1,
 mymod.objects.File1,
 mymod.objects.File1]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20150117/e3dc9b1a/attachment.html>