Problem with copy.deepcopy and multiprocessing.Queue

[MRAB]

On 2015-10-14 18:41, James DeVincentis wrote:
> I’ve got a bit of a problem with copy.deepcopy and using
> multiprocessing.Queue.
>
> I have an HTTPAPI that gets exposed to add objects to a
> multiprocessing.Qeue. Source code here:
> https://github.com/jmdevince/cifpy3/blob/master/lib/cif/api/handler.py#L283
>
> The trouble is, even using deepcopy, my debugging shows that it keeps
> re-using the same address space on every iteration of the loop (and/or
> fork since it uses the Forking TCP server mixin).
>
> This is reflected when the reference to that address space gets removed
> from the Queue. Source code:
> https://github.com/jmdevince/cifpy3/blob/master/lib/cif/worker/worker.py
>
> So I submit a few objects very rapidly to the HTTP server. It places
> them in the queue. For some reason on the second iteration copy.deepcopy
> stops using new address spaces for deepcopied objects. As such the queue
> processes that object only once (even though Queue.put() is called
> repeatedly for the same address space).
>
> I’ve tried ‘del’ the object before reusing it but it still reallocates
> to the same address space. I’m at a complete loss.
>
> 2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
> <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb2e8>
>
> 2015-10-14 17:03:30 THREAD #1-1  WARNING  Got Observable with Object:
> <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbdb978>
>
> 2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
> <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
>
> 2015-10-14 17:03:30 THREAD #2-1  WARNING  Got Observable with Object:
> <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbdab70>
>
> 2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
> <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
>
> 2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
> <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
>
> 2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
> <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
>
> 2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
> <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
>
> 2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
> <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
>
> 2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
> <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
>
> 2015-10-14 17:03:30 APIHTTP      WARNING  Storing object:
> <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
>
>
Could it just be re-using space that it has recently freed?