[IPython-dev] Run IPython Parallel from custom module?
Ryan Nelson
rnelsonchem at gmail.com
Sat Jan 17 20:23:07 EST 2015
This worked well for the example that I posted. Thanks again.
However, it brings up another question. What are the restrictions on using
this on methods inside a custom object? For example, if I change
"functs.py" into an object with the following methods:
@interactive
def _do_this(self, fname):
f = File(fname)
f.extra = 'hello'
self._do_something_else(f)
return f
@interactive
def _do_something_else(self, f):
f.junk = 'World'
I don't get any errors, but "_do_something_else" doesn't seem to be called,
but "_do_this" works fine. (Actually, in my larger code, I'm getting a
pickling error... But that's tough to reproduce.)
Ryan
On Sat, Jan 17, 2015 at 4:17 PM, Ryan Nelson <rnelsonchem at gmail.com> wrote:
> Thanks!
>
> Searching for that decorator brings up a bunch of search hits. However, it
> looks like it isn't really documented in the main document yet (?).
>
> I'll try this out.
>
> Ryan
>
>
>
> On Sat, Jan 17, 2015 at 3:15 PM, MinRK <benjaminrk at gmail.com> wrote:
>
>> This is one of the most common sources of confusion with
>> IPython.parallel. When you call a function, it’s globals are resolved to
>> its own module. So when you call _do_this, it is looking for File in
>> mymod, not in the interactive namespace. If you had defined _do_this
>> interactively (in __main__), it would have found File. Similarly, if File
>> were defined in the same module as _do_this, it would also be found.
>>
>> IPython provides a decorator that lets you define functions in modules
>> that will be treated as if they were defined in __main__, and will thus
>> have access to the interactive namespace.
>>
>> from IPython.parallel import interactive
>> @interactivedef _do_this(fname):
>> f = File(fname)
>> ...
>>
>> -MinRK
>>
>>
>> On Sat, Jan 17, 2015 at 11:14 AM, Ryan Nelson <rnelsonchem at gmail.com>
>> wrote:
>>
>>> Hello everyone,
>>>
>>> I'm developing a module for processing some data files. There are
>>> several steps to this process, which can be time consuming, so I would like
>>> to write a helper function in my module that uses IPython.parallel to
>>> spread out the work over several processors. Unfortunately, I'm having
>>> trouble getting things to work properly. Note: I can get something similar
>>> to work when it is not written as a module...
>>>
>>> I tried my best to make a simple example that fails with the same error
>>> that I'm seeing in the larger case (below); however, it is still a little
>>> complicated.
>>>
>>> After running the test script, I get a "NameError" on the cluster nodes
>>> (traceback below). If I check the nodes manually, the objects are defined.
>>> I can also define new functions from the IPython terminal that act on those
>>> objects, and those work as expected.
>>>
>>> Any help you can provide is most appreciated. Sorry if this is a
>>> duplicate, but I couldn't find it anywhere else.
>>>
>>> Python 3.4 and 2.7 and IPython 2.3.1
>>>
>>> Ryan
>>>
>>> ________________________________________
>>>
>>> Here's the approximate structure of my module:
>>>
>>> mymod/
>>> __init__.py
>>> functs.py : helper functions with Parallel support
>>> objects.py : contains my custom processing objects
>>>
>>> # functs.py #
>>> -------------
>>> from IPython.parallel import Client
>>>
>>> import mymod.objects as objects
>>>
>>> def process(files):
>>> client = Client()
>>> dview = client[:]
>>> dview.block = True
>>>
>>> # Do this to make module path known
>>> # However, my mod will eventually be installed via pip, so this
>>> # isn't necessary to reproduce error in that case
>>> with dview.sync_imports():
>>> import sys
>>> sys.path.append(['/home/nelson/code/testing/',])
>>>
>>> dview['sys.path'] = sys.path
>>> File = objects.File1
>>> dview['File'] = File
>>> result = dview.map_sync(_do_this, files)
>>> return result
>>>
>>> def _do_this(fname):
>>> f = File(fname)
>>> f.extra = 'hello'
>>> return f
>>> -------------
>>>
>>> # objects.py #
>>> -------------
>>> class File1(object):
>>> def __init__(self, fname):
>>> self.fname = fname
>>> -------------
>>>
>>> I also have a test script ("test.py") with the following:
>>> -------------
>>> from mymod.functs import process
>>> result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
>>> [print(i.fname, i.extra) for i in result]
>>> -------------
>>>
>>> When I run this script, I get the following errors:
>>>
>>> importing sys on engine(s)
>>> Traceback (most recent call last):
>>> File "test2.py", line 3, in <module>
>>> result = process(['a', 'b', 'c', 'd', 'e', 'f', 'g'])
>>> File "/home/nelson/code/testing/mymod/functs.py", line 16, in process
>>> result = dview.map_sync(_do_this, files)
>>> File
>>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
>>> 366, in map_sync
>>> return self.map(f,*sequences,**kwargs)
>>> File "<string>", line 2, in map
>>> File
>>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
>>> 66, in sync_results
>>> ret = f(self, *args, **kwargs)
>>> File
>>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/view.py", line
>>> 624, in map
>>> return pf.map(*sequences)
>>> File
>>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
>>> line 271, in map
>>> ret = self(*sequences)
>>> File "<string>", line 2, in __call__
>>> File
>>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
>>> line 78, in sync_view_results
>>> return f(self, *args, **kwargs)
>>> File
>>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/remotefunction.py",
>>> line 254, in __call__
>>> return r.get()
>>> File
>>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
>>> line 118, in get
>>> raise self._exception
>>> File
>>> "/usr/lib64/python3.4/site-packages/IPython/parallel/client/asyncresult.py",
>>> line 153, in wait
>>> results = error.collect_exceptions(results, self._fname)
>>> File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py",
>>> line 233, in collect_exceptions
>>> raise e
>>> File "/usr/lib64/python3.4/site-packages/IPython/parallel/error.py",
>>> line 231, in collect_exceptions
>>> raise CompositeError(msg, elist)
>>> IPython.parallel.error.CompositeError: one or more exceptions from call
>>> to method: _do_this
>>> [0:apply]: NameError: name 'File' is not defined
>>> [1:apply]: NameError: name 'File' is not defined
>>> [2:apply]: NameError: name 'File' is not defined
>>> [3:apply]: NameError: name 'File' is not defined
>>>
>>> However, if I jump into an interactive Python terminal, these objects
>>> are defined on the cluster nodes.
>>> In [1]: from IPython.parallel import Client
>>>
>>> In [2]: client = Client()
>>>
>>> In [3]: dview = client[:]
>>>
>>> In [4]: dview['File']
>>>
>>>
>>> Out[4]:
>>> [mymod.objects.File1,
>>> mymod.objects.File1,
>>> mymod.objects.File1,
>>> mymod.objects.File1]
>>>
>>>
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
>>>
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20150117/7f178703/attachment.html>
More information about the IPython-dev
mailing list