array.array()'s memory shared with multiprocessing.Process()

[MRAB]

On 2017-09-10 12:40, gerlando.falauto at gmail.com wrote:
>> 
>> I suspect it's down to timing.
>> 
>> What you're putting into the queue is a reference to the array, and it's 
>> only some time later that the array itself is pickled and then sent (the 
>> work being done in the 'background').
>> 
>> Modifying the array before (or while) it's actually being sent would 
>> explain the problem you're seeing.
> 
> That would also have been my guess. However, according to documentation:
> 
>> When an object is put on a queue, the object is pickled and a background 
>> thread later flushes the pickled data to an underlying pipe.
> 
> In my understanding this means the object is pickled *before* the background thread takes care of flushing the data to the pipe. Is that a mistake in the documentation then?
> 
> Any suggestion for a way to work around this limitation?
> Or perhaps a different approach altogether I could use to reduce CPU load?
> What the main thread actually does is dequeue data from a high-speed USB-to-serial (2.000.000 bps), that's why I came up with the array.array() solution to store collected data, hoping for the smallest possible overhead.
> Thanks!
> 
I've had a quick look at the source code.

When an object is put into the queue, it's actually put into an internal 
buffer (a deque), and then the method returns.

An internal thread works through the buffer and pickles the objects.

Therefore, just because the method has returned, it doesn't mean that 
it's now safe to modify the object.