Problem with copy.deepcopy and multiprocessing

[James DeVincentis]

I've got a bit of a problem with copy.deepcopy and using
multiprocessing.Queue. 

 

I have an HTTPAPI that gets exposed to add objects to a
multiprocessing.Qeue. Source code here:
https://github.com/jmdevince/cifpy3/blob/master/lib/cif/api/handler.py#L283

 

The trouble is, even using deepcopy, my debugging shows that it keeps
re-using the same address space on every iteration of the loop (and/or fork
since it uses the Forking TCP server mixin). 

 

This is reflected when the reference to that address space gets removed from
the Queue. Source code:
https://github.com/jmdevince/cifpy3/blob/master/lib/cif/worker/worker.py

 

So I submit a few objects very rapidly to the HTTP server. It places them in
the queue. For some reason on the second iteration copy.deepcopy stops using
new address spaces for deepcopied objects. As such the queue processes that
object only once (even though Queue.put() is called repeatedly for the same
address space). 

 

I've tried 'del' the object before reusing it but it still reallocates to
the same address space. I'm at a complete loss. 

 

2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb2e8>

2015-10-14 17:03:30 THREAD #1-1  WARNING  Got Observable with Object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbdb978>

2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>

2015-10-14 17:03:30 THREAD #2-1  WARNING  Got Observable with Object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbdab70>

2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>

2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>

2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>

2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>

2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>

2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>

2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>

 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20151014/8444907e/attachment.html>