Threading plus multiprocessing plus cv2 error

[Python]

On Sat, Aug 29, 2020 at 06:24:10PM +1000, John O'Hagan wrote:
> Dear list
> 
> Thanks to this list, I haven't needed to ask a question for
> a very long time, but this one has me stumped.
> 
> Here's the minimal 3.8 code, on Debian testing:
> 
> -----
> from multiprocessing import Process
> from threading import Thread
> from time import sleep
> import cv2
> 
> def show(im, title, location):
>     cv2.startWindowThread()
>     cv2.namedWindow(title)
>     cv2.moveWindow(title, *location)
>     cv2.imshow(title, im)
>     sleep(2) #just to keep window open
> 
> im1 = cv2.imread('/path/to/image1')
> im2 = cv2.imread('/path/to/image2')
> 
> Thread(target=show, args=(im1, 'im1', (600,0))).start()
> sleep(1)
> Process(target=show, args=(im2, 'im2', (0, 0))).start()
> -----
> 
> Here's the error:
> 
> -----
> [xcb] Unknown sequence number while processing queue
> [xcb] Most likely this is a multi-threaded client and XInitThreads has
> not been called 
> [xcb] Aborting, sorry about that.
> python3: ../../src/xcb_io.c:260: poll_for_event: Assertion
> `!xcb_xlib_threads_sequence_lost' failed.
> -----

It's hard to say EXACTLY what the nature of the error is, without
knowing what the underlying libraries are doing.  But from this one
thing is clear:  You are starting a new thread, and then forking a new
process.  This is very likely to break, because contrary to what
several people have said in this thread, the new thread WILL NOT be
copied to the new program, so in the new process, part of your program
is literally missing.  This can have all kinds of consequences.

For that reason, you can't spawn a new thread and then fork a new
process, in that order, and expect your program to function correctly.
Whatever your program does that depends on that new thread won't work
correctly, because that thread does not exist in the child (new
process).  You can, however, do those things in the reverse order and
it should be fine... new threads will be started in both processes
(unless you take steps to ensure only one of the processes creates the
thread).  Both processes retain their integrity and should run fine.

> There's no error without the sleep(1), nor if the Process is started
> before the Thread, nor if two Processes are used instead, nor if two
> Threads are used instead. IOW the error only occurs if a Thread is
> started first, and a Process is started a little later.

Hopefully my explanation above makes it clear why all of those things
are true, other than the sleep() issue.  That one is most likely just
a timing issue:  Whatever resource is causing the problem hasn't been
set up yet or the critical thread or process has already finished
execution before the issue can arise, or something of the sort.  When
you start new threads or processes, usually there's some delay as your
OS schedules each process/thread to run, which is somewhat random
based on how the scheduler works and how loaded the system is.  Such
timing problems (bugs that seem to happen randomly with each run of
the program, or over time in a long-executing program) are common in
multi-threaded programs that are written incorrectly, especially when
the state of one thread depends on the state of the other thread, and
the two don't synchronize correctly.  This is another way that the
first problem above can manifest, too: The threads can't synchronize
because one of them does not exist!

Hope that helps.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20200903/9fab11a9/attachment.sig>