Python child process in while True loop blocks parent

[Barry Scott]

> On 8 Dec 2021, at 17:11, Jen Kris <jenkris at tutanota.com> wrote:
> 
> I started this post on November 29, and there have been helpful comments since then from Barry Scott, Cameron Simpson, Peter Holzer and Chris Angelico.  Thanks to all of you.  
> 
> I've found a solution that works for my purpose, and I said earlier that I would post the solution I found. If anyone has a better solution I would appreciate any feedback. 
> 
> To recap, I'm using a pair of named pipes for IPC between C and Python.  Python runs as a child process after fork-execv.  The Python program continues to run concurrently in a while True loop, and responds to requests from C at intervals, and continues to run until it receives a signal from C to exit.  C sends signals to Python, then waits to receive data back from Python.  My problem was that C was blocked when Python started. 
> 
> The solution was twofold:  (1) for Python to run concurrently it must be a multiprocessing loop (from the multiprocessing module), and (2) Python must terminate its write strings with \n, or read will block in C waiting for something that never comes.  The multiprocessing module sidesteps the GIL; without multiprocessing the GIL will block all other threads once Python starts. 
> 
> Originally I used epoll() on the pipes.  Cameron Smith and Barry Scott advised against epoll, and for this case they are right.  Blocking pipes work here, and epoll is too much overhead for watching on a single file descriptor. 
> 
> This is the Python code now:
> 
> #!/usr/bin/python3
> from multiprocessing import Process

You already have feedback that multiprocessing is not required.

> import os
> 
> print("Python is running")
> 
> child_pid = os.getpid()
> print('child process id:', child_pid)
> 
> def f(a, b):
> 
>     print("Python now in function f")
> 
>     pr = os.open('/tmp/Pipe_01', os.O_RDONLY)
>     print("File Descriptor1 Opened " + str(pr))
>     pw = os.open('/tmp/Pipe_02', os.O_WRONLY)
>     print("File Descriptor2 Opened " + str(pw))
> 
>     while True:
> 
>         v = os.read(pr,64)
>         print("Python read from pipe pr")
>         print(v)
> 
>         if v == b'99':
>             os.close(pr)
>             os.close(pw)
>             print("Python is terminating")
>             os._exit(os.EX_OK)
> 
>         if v != "Send child PID":
>             os.write(pw, b"OK message received\n")

The \n should not be required as UDS (unix domain sockets) are message based not a stream of bytes.

>             print("Python wrote back")
> 
> if __name__ == '__main__':
>     a = 0
>     b = 0
>     p = Process(target=f, args=(a, b,))
>     p.start()
>     p.join()
> 
> The variables a and b are not currently used in the body, but they will be later. 
> 
> This is the part of the C code that communicates with Python:
> 
>     fifo_fd1 = open(fifo_path1, O_WRONLY);
>     fifo_fd2 = open(fifo_path2, O_RDONLY);
> 
>     status_write = write(fifo_fd1, py_msg_01, sizeof(py_msg_01));
>     if (status_write < 0) perror("write");
You continue on after the error, exit would be better.
> 
>     status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
>     if (status_read < 0) perror("read");
>     printf("C received message 1 from Python\n");
>     printf("%.*s",(int)buf_len, fifo_readbuf);

The length of the data read is in status_read not buf_len.

> 
>     status_write = write(fifo_fd1, py_msg_02, sizeof(py_msg_02));

How much data did you put into py_msg_01 buffer?
Is it a C string? If so you want to write sizeof(py_msg_02)-1 to avoid sending the trailing NUL (0)
of the C string.

>     if (status_write < 0) perror("write");
If it failed exit until you have figured out the error handling.
> 
>     status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
>     if (status_read < 0) perror("read");
>     printf("C received message 2 from Python\n");
>     printf("%.*s",(int)buf_len, fifo_readbuf);

The length of the data read is in status_read not buf_len.

At no point do I see in this C code the need for a \n in the data sent between python and C.

> 
>     // Terminate Python multiprocessing
>     printf("C is sending exit message to Python\n");
>     status_write = write(fifo_fd1, py_msg_03, 2);

Would be better to not be using "2" for the length to write here.
If py_msg_03 only has the exit msg in it the use sizeof(py_msg_03).
If its a C string the subtract 1 for the trailing NUL (0).

> 
>     printf("C is closing\n");
>     close(fifo_fd1);
>     close(fifo_fd2);
> 
> Screen output:
> 
> Python is running
> child process id: 5353
> Python now in function f
> File Descriptor1 Opened 6
> Thread created 0
> File Descriptor2 Opened 7
> Process ID: 5351
> Parent Process ID: 5351
> I am the parent
> Core joined 0
> I am the child
> Python read from pipe pr
> b'Hello to Python from C\x00\x00'
The \x00 is because the length you used is wrong.

> Python wrote back
> C received message 1 from Python
> OK message received
> Python read from pipe pr
> b'Message to Python 2\x00\x00'
The \x00 is because the length you used is wrong.

> Python wrote back
> C received message 2 from Python
> OK message received
> C is sending exit message to Python
> C is closing
> Python read from pipe pr
> b'99'
> Python is terminating
> 
> Python runs on a separate thread (created with pthreads) because I want the flexibility of using this same basic code as a stand-alone .exe, or for a C extension from Python called with ctypes.  If I use it as a C extension then I want the Python code on a separate thread because I can't have two instances of the Python interpreter running on one thread, and one instance will already be running on the main thread, albeit "suspended" by the call from ctypes. 
> 
> So that's my solution:  (1) Python multiprocessing module; (2) Python strings written to the pipe must be terminated with \n. 
> 
> Thanks again to all who commented. 
> 
> 
> 
> Dec 6, 2021, 13:33 by barry at barrys-emacs.org:
> 
> 
>> On 6 Dec 2021, at 21:05, Jen Kris <jenkris at tutanota.com <mailto:jenkris at tutanota.com>> wrote:
>> 
>> Here is what I don't understand from what you said.  "The child process is created with a single thread—the one that called fork()."  To me that implies that the thread that called fork() is the same thread as the child process.  I guess you're talking about the distinction between logical threads and physical threads.  
> 
> The thread that called fork is cloned into a new thread in the new process.
> It has a clone of the processor registers of the thread, stack memory segment of that thread,
> which means that it has all the stack variables and stack frames from the parent process's thread.
> 
> 
>> But the main issue is your suggestion that I should call fork-execv from the thread that runs the main C program, not from a separate physical pthread.  That would certainly eliminate the overhead of creating a new pthread. 
> 
> Forget about physical threads, they only matter when you are performance tuning your code.
> 
>> I am working now to finish this, and I will try your suggestion of calling fork-execv from the "main" thread.  When I reply back next I can give you a complete picture of what I'm doing. 
>> 
>> Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated. 
> 
> Barry
> 
> 
>