[execnet-dev] Internal errors calling channel.send

[Shead, Timothy]

On Jul 28, 2011, at 4:14 PM, Shead, Timothy wrote:

> 
> On Jul 28, 2011, at 3:11 PM, Shead, Timothy wrote:
> 
>> Hello, List!
>> 
>> I'm using exenet 1.0.9 with Python 2.7 on a Mac OSX workstation.  Execnet has generally been very reliable and easy to use, but I'm having trouble using it to send large "jobs" to worker processes.  The size of the serialized data for each job varies, but using EXECNET_DEBUG=2 I see that they're all around 2500000 bytes.  I get an intermittent (doesn't happen for the same job) error in the receiving thread:
> 
> Forgot to mention that my main application is running in Python, while the worker processes are running in Jython.  Are there any known issues connecting to Jython?

Sure 'nuff, here's a test case that always hangs in the same spot:

def stress_server(channel):
  for request in channel:
    channel.send(request)

gateway = execnet.makegateway("popen//python=jython")
channel = gateway.remote_exec(stress_server)

for size in range(31):
  sys.stderr.write("Sending string length %d\n" % 2**size)
  original = "a" * (2**size)
  channel.send(original)
  response = channel.receive()
  self.assertEqual(original, response)

... running it with EXECNET_DEBUG=2 yields the following:

[68941] RECEIVERTHREAD: starting to run
[68941] sent <Message.CHANNEL_EXEC channelid=1 len=108>
Sending string length 1
[68941] sent <Message.CHANNEL_DATA channelid=1 a>
[68947] creating slavegateway on <execnet.gateway_base.Popen2IO instance at 0x2>
[68947] RECEIVERTHREAD: starting to run
[68947] received <Message.CHANNEL_EXEC channelid=1 len=108>
[68947] execution starts[1]: 'def stress_server(channel):\n  for request in cha
[68947] received <Message.CHANNEL_DATA channelid=1 a>
[68947] calling stress_server(**                                                          {})
[68947] sent <Message.CHANNEL_DATA channelid=1 a>
[68941] received <Message.CHANNEL_DATA channelid=1 a>
Sending string length 2
[68941] sent <Message.CHANNEL_DATA channelid=1 aa>
[68947] received <Message.CHANNEL_DATA channelid=1 aa>
[68947] sent <Message.CHANNEL_DATA channelid=1 aa>
[68941] received <Message.CHANNEL_DATA channelid=1 aa>
Sending string length 4
[68941] sent <Message.CHANNEL_DATA channelid=1 aaaa>
[68947] received <Message.CHANNEL_DATA channelid=1 aaaa>
[68941] received <Message.CHANNEL_DATA channelid=1 aaaa>
[68947] sent <Message.CHANNEL_DATA channelid=1 aaaa>
Sending string length 8
[68941] sent <Message.CHANNEL_DATA channelid=1 aaaaaaaa>
[68947] received <Message.CHANNEL_DATA channelid=1 aaaaaaaa>
[68941] received <Message.CHANNEL_DATA channelid=1 aaaaaaaa>
[68947] sent <Message.CHANNEL_DATA channelid=1 aaaaaaaa>
Sending string length 16
[68941] sent <Message.CHANNEL_DATA channelid=1 aaaaaaaaaaaaaaaa>
[68947] received <Message.CHANNEL_DATA channelid=1 aaaaaaaaaaaaaaaa>
[68947] sent <Message.CHANNEL_DATA channelid=1 aaaaaaaaaaaaaaaa>
[68941] received <Message.CHANNEL_DATA channelid=1 aaaaaaaaaaaaaaaa>
Sending string length 32
[68941] sent <Message.CHANNEL_DATA channelid=1 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa>
[68947] received <Message.CHANNEL_DATA channelid=1 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa>
[68941] received <Message.CHANNEL_DATA channelid=1 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa>
[68947] sent <Message.CHANNEL_DATA channelid=1 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa>
Sending string length 64
[68941] sent <Message.CHANNEL_DATA channelid=1 len=66>
[68947] received <Message.CHANNEL_DATA channelid=1 len=66>
[68941] received <Message.CHANNEL_DATA channelid=1 len=66>
[68947] sent <Message.CHANNEL_DATA channelid=1 len=66>
Sending string length 128
[68941] sent <Message.CHANNEL_DATA channelid=1 len=130>
[68947] received <Message.CHANNEL_DATA channelid=1 len=130>
[68941] received <Message.CHANNEL_DATA channelid=1 len=130>
[68947] sent <Message.CHANNEL_DATA channelid=1 len=130>
Sending string length 256
[68941] sent <Message.CHANNEL_DATA channelid=1 len=258>
[68947] received <Message.CHANNEL_DATA channelid=1 len=258>
[68941] received <Message.CHANNEL_DATA channelid=1 len=258>
[68947] sent <Message.CHANNEL_DATA channelid=1 len=258>
Sending string length 512
[68941] sent <Message.CHANNEL_DATA channelid=1 len=514>
[68947] received <Message.CHANNEL_DATA channelid=1 len=514>
[68941] received <Message.CHANNEL_DATA channelid=1 len=514>
[68947] sent <Message.CHANNEL_DATA channelid=1 len=514>
Sending string length 1024
[68941] sent <Message.CHANNEL_DATA channelid=1 len=1026>
[68947] received <Message.CHANNEL_DATA channelid=1 len=1026>
[68941] received <Message.CHANNEL_DATA channelid=1 len=1026>
[68947] sent <Message.CHANNEL_DATA channelid=1 len=1026>
Sending string length 2048
[68941] sent <Message.CHANNEL_DATA channelid=1 len=2050>
[68947] received <Message.CHANNEL_DATA channelid=1 len=2050>
[68941] received <Message.CHANNEL_DATA channelid=1 len=2050>
Sending string length 4096
[68941] sent <Message.CHANNEL_DATA channelid=1 len=4098>
[68947] sent <Message.CHANNEL_DATA channelid=1 len=2050>
[68947] received <Message.CHANNEL_DATA channelid=1 len=4098>
[68941] received <Message.CHANNEL_DATA channelid=1 len=4098>
[68947] sent <Message.CHANNEL_DATA channelid=1 len=4098>
Sending string length 8192
[68941] sent <Message.CHANNEL_DATA channelid=1 len=8194>
[68947] received <Message.CHANNEL_DATA channelid=1 len=8194>
[68941] received <Message.CHANNEL_DATA channelid=1 len=8194>
[68947] sent <Message.CHANNEL_DATA channelid=1 len=8194>
Sending string length 16384
[68941] sent <Message.CHANNEL_DATA channelid=1 len=16386>

Any thoughts?

Cheers,
Tim

Timothy M. Shead
Sandia National Laboratories
1461, Scalable Analysis and Visualization