From williamj at tenbase2.com Tue Jul 19 14:49:34 2011 From: williamj at tenbase2.com (williamj at tenbase2.com) Date: Tue, 19 Jul 2011 08:49:34 -0400 Subject: [execnet-dev] execnet and IronPython Message-ID: <4FF64BC3F6CD4F02A07887604876247D@GAMINGMACHINE> Hello, Can execnet be used with WCF or IronPython? I have a WCF Web service that uses embedded python scripts. However, I cannot get the Web service to work on new hardware and would like to upgrade to execnet. Thanks. Sincerely, William Johnston -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Tue Jul 19 19:20:29 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 19 Jul 2011 17:20:29 +0000 Subject: [execnet-dev] execnet and IronPython In-Reply-To: <4FF64BC3F6CD4F02A07887604876247D@GAMINGMACHINE> References: <4FF64BC3F6CD4F02A07887604876247D@GAMINGMACHINE> Message-ID: <20110719172029.GZ12151@merlinux.eu> Hi William, i don't think execnet currently works with IronPython. but i haven't tested for a long time and am not sure how difficult it still is to get it working with recent versions. best, holger On Tue, Jul 19, 2011 at 08:49 -0400, williamj at tenbase2.com wrote: > Hello, > > Can execnet be used with WCF or IronPython? > > I have a WCF Web service that uses embedded python scripts. However, I cannot get the Web service to work on new hardware and would like to upgrade to execnet. > > Thanks. > > Sincerely, > William Johnston > _______________________________________________ > execnet-dev mailing list > execnet-dev at codespeak.net > http://codespeak.net/mailman/listinfo/execnet-dev From tshead at sandia.gov Thu Jul 28 23:11:55 2011 From: tshead at sandia.gov (Shead, Timothy) Date: Thu, 28 Jul 2011 21:11:55 +0000 Subject: [execnet-dev] Internal errors calling channel.send Message-ID: <43D2F479-6897-49FD-8904-91F081A37AAB@sandia.gov> Hello, List! I'm using exenet 1.0.9 with Python 2.7 on a Mac OSX workstation. Execnet has generally been very reliable and easy to use, but I'm having trouble using it to send large "jobs" to worker processes. The size of the serialized data for each job varies, but using EXECNET_DEBUG=2 I see that they're all around 2500000 bytes. I get an intermittent (doesn't happen for the same job) error in the receiving thread: [68436] RECEIVERTHREAD Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/execnet-1.0.9-py2.7.egg/execnet/gateway_base.py", line 613, in _thread_receiver msg = Message(*self._unserializer.load()) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/execnet-1.0.9-py2.7.egg/execnet/gateway_base.py", line 801, in load loader(self) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/execnet-1.0.9-py2.7.egg/execnet/gateway_base.py", line 801, in load loader(self) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/execnet-1.0.9-py2.7.egg/execnet/gateway_base.py", line 819, in load_int i = self._read_int4() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/execnet-1.0.9-py2.7.egg/execnet/gateway_base.py", line 843, in _read_int4 return struct.unpack("!i", self.stream.read(4))[0] error: unpack str size does not match format [68436] RECEIVERTHREAD entering finalization [68436] RECEIVERTHREAD leaving finalization Typically, it takes between 5 - 30 jobs for this error to manifest. Thinking that maybe I was trying to work with a data structure that couldn't be serialized by execnet, I tried modifying my code to pickle the job data and send it as a string; this causes things to fail in a different way: [68335] RECEIVERTHREAD Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/execnet-1.0.9-py2.7.egg/execnet/gateway_base.py", line 613, in _thread_receiver msg = Message(*self._unserializer.load()) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/execnet-1.0.9-py2.7.egg/execnet/gateway_base.py", line 804, in load raise UnserializationError("internal unserialization error") UnserializationError: internal unserialization error [68335] RECEIVERTHREAD entering finalization [68335] RECEIVERTHREAD leaving finalization [68335] execution finished Again, the size of the pickled string is roughly 2500000 bytes, and things don't always fail on the same send. Any thoughts? Am I running into a practical limitation on the size of execnet messages? Thanks in advance, Tim Timothy M. Shead Sandia National Laboratories 1461, Scalable Analysis and Visualization From tshead at sandia.gov Fri Jul 29 00:14:27 2011 From: tshead at sandia.gov (Shead, Timothy) Date: Thu, 28 Jul 2011 22:14:27 +0000 Subject: [execnet-dev] Internal errors calling channel.send In-Reply-To: <43D2F479-6897-49FD-8904-91F081A37AAB@sandia.gov> References: <43D2F479-6897-49FD-8904-91F081A37AAB@sandia.gov> Message-ID: On Jul 28, 2011, at 3:11 PM, Shead, Timothy wrote: > Hello, List! > > I'm using exenet 1.0.9 with Python 2.7 on a Mac OSX workstation. Execnet has generally been very reliable and easy to use, but I'm having trouble using it to send large "jobs" to worker processes. The size of the serialized data for each job varies, but using EXECNET_DEBUG=2 I see that they're all around 2500000 bytes. I get an intermittent (doesn't happen for the same job) error in the receiving thread: Forgot to mention that my main application is running in Python, while the worker processes are running in Jython. Are there any known issues connecting to Jython? Many thanks, Tim Timothy M. Shead Sandia National Laboratories 1461, Scalable Analysis and Visualization From tshead at sandia.gov Fri Jul 29 00:26:41 2011 From: tshead at sandia.gov (Shead, Timothy) Date: Thu, 28 Jul 2011 22:26:41 +0000 Subject: [execnet-dev] Internal errors calling channel.send In-Reply-To: References: <43D2F479-6897-49FD-8904-91F081A37AAB@sandia.gov> Message-ID: <9418D43F-C247-4CC7-B3E5-38BA0F97D0E0@sandia.gov> On Jul 28, 2011, at 4:14 PM, Shead, Timothy wrote: > > On Jul 28, 2011, at 3:11 PM, Shead, Timothy wrote: > >> Hello, List! >> >> I'm using exenet 1.0.9 with Python 2.7 on a Mac OSX workstation. Execnet has generally been very reliable and easy to use, but I'm having trouble using it to send large "jobs" to worker processes. The size of the serialized data for each job varies, but using EXECNET_DEBUG=2 I see that they're all around 2500000 bytes. I get an intermittent (doesn't happen for the same job) error in the receiving thread: > > Forgot to mention that my main application is running in Python, while the worker processes are running in Jython. Are there any known issues connecting to Jython? Sure 'nuff, here's a test case that always hangs in the same spot: def stress_server(channel): for request in channel: channel.send(request) gateway = execnet.makegateway("popen//python=jython") channel = gateway.remote_exec(stress_server) for size in range(31): sys.stderr.write("Sending string length %d\n" % 2**size) original = "a" * (2**size) channel.send(original) response = channel.receive() self.assertEqual(original, response) ... running it with EXECNET_DEBUG=2 yields the following: [68941] RECEIVERTHREAD: starting to run [68941] sent Sending string length 1 [68941] sent [68947] creating slavegateway on [68947] RECEIVERTHREAD: starting to run [68947] received [68947] execution starts[1]: 'def stress_server(channel):\n for request in cha [68947] received [68947] calling stress_server(** {}) [68947] sent [68941] received Sending string length 2 [68941] sent [68947] received [68947] sent [68941] received Sending string length 4 [68941] sent [68947] received [68941] received [68947] sent Sending string length 8 [68941] sent [68947] received [68941] received [68947] sent Sending string length 16 [68941] sent [68947] received [68947] sent [68941] received Sending string length 32 [68941] sent [68947] received [68941] received [68947] sent Sending string length 64 [68941] sent [68947] received [68941] received [68947] sent Sending string length 128 [68941] sent [68947] received [68941] received [68947] sent Sending string length 256 [68941] sent [68947] received [68941] received [68947] sent Sending string length 512 [68941] sent [68947] received [68941] received [68947] sent Sending string length 1024 [68941] sent [68947] received [68941] received [68947] sent Sending string length 2048 [68941] sent [68947] received [68941] received Sending string length 4096 [68941] sent [68947] sent [68947] received [68941] received [68947] sent Sending string length 8192 [68941] sent [68947] received [68941] received [68947] sent Sending string length 16384 [68941] sent Any thoughts? Cheers, Tim Timothy M. Shead Sandia National Laboratories 1461, Scalable Analysis and Visualization