Python child process in while True loop blocks parent

[Jen Kris]

I started this post on November 29, and there have been helpful comments since then from Barry Scott, Cameron Simpson, Peter Holzer and Chris Angelico.  Thanks to all of you.  

I've found a solution that works for my purpose, and I said earlier that I would post the solution I found. If anyone has a better solution I would appreciate any feedback. 

To recap, I'm using a pair of named pipes for IPC between C and Python.  Python runs as a child process after fork-execv.  The Python program continues to run concurrently in a while True loop, and responds to requests from C at intervals, and continues to run until it receives a signal from C to exit.  C sends signals to Python, then waits to receive data back from Python.  My problem was that C was blocked when Python started. 

The solution was twofold:  (1) for Python to run concurrently it must be a multiprocessing loop (from the multiprocessing module), and (2) Python must terminate its write strings with \n, or read will block in C waiting for something that never comes.  The multiprocessing module sidesteps the GIL; without multiprocessing the GIL will block all other threads once Python starts. 

Originally I used epoll() on the pipes.  Cameron Smith and Barry Scott advised against epoll, and for this case they are right.  Blocking pipes work here, and epoll is too much overhead for watching on a single file descriptor. 

This is the Python code now:

#!/usr/bin/python3
from multiprocessing import Process
import os

print("Python is running")

child_pid = os.getpid()
print('child process id:', child_pid)

def f(a, b):

    print("Python now in function f")

    pr = os.open('/tmp/Pipe_01', os.O_RDONLY)
    print("File Descriptor1 Opened " + str(pr))
    pw = os.open('/tmp/Pipe_02', os.O_WRONLY)
    print("File Descriptor2 Opened " + str(pw))

    while True:

        v = os.read(pr,64)
        print("Python read from pipe pr")
        print(v)

        if v == b'99':
            os.close(pr)
            os.close(pw)
            print("Python is terminating")
            os._exit(os.EX_OK)

        if v != "Send child PID":
            os.write(pw, b"OK message received\n")
            print("Python wrote back")

if __name__ == '__main__':
    a = 0
    b = 0
    p = Process(target=f, args=(a, b,))
    p.start()
    p.join()

The variables a and b are not currently used in the body, but they will be later. 

This is the part of the C code that communicates with Python:

    fifo_fd1 = open(fifo_path1, O_WRONLY);
    fifo_fd2 = open(fifo_path2, O_RDONLY);

    status_write = write(fifo_fd1, py_msg_01, sizeof(py_msg_01));
    if (status_write < 0) perror("write");

    status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_01));
    if (status_read < 0) perror("read");
    printf("C received message 1 from Python\n");
    printf("%.*s",(int)buf_len, fifo_readbuf);

    status_write = write(fifo_fd1, py_msg_02, sizeof(py_msg_02));
    if (status_write < 0) perror("write");

    status_read = read(fifo_fd2, fifo_readbuf, sizeof(py_msg_02));
    if (status_read < 0) perror("read");
    printf("C received message 2 from Python\n");
    printf("%.*s",(int)buf_len, fifo_readbuf);

    // Terminate Python multiprocessing
    printf("C is sending exit message to Python\n");
    status_write = write(fifo_fd1, py_msg_03, 2);

    printf("C is closing\n");
    close(fifo_fd1);
    close(fifo_fd2);

Screen output:

Python is running
child process id: 5353
Python now in function f
File Descriptor1 Opened 6
Thread created 0
File Descriptor2 Opened 7
Process ID: 5351
Parent Process ID: 5351
I am the parent
Core joined 0
I am the child
Python read from pipe pr
b'Hello to Python from C\x00\x00'
Python wrote back
C received message 1 from Python
OK message received
Python read from pipe pr
b'Message to Python 2\x00\x00'
Python wrote back
C received message 2 from Python
OK message received
C is sending exit message to Python
C is closing
Python read from pipe pr
b'99'
Python is terminating

Python runs on a separate thread (created with pthreads) because I want the flexibility of using this same basic code as a stand-alone .exe, or for a C extension from Python called with ctypes.  If I use it as a C extension then I want the Python code on a separate thread because I can't have two instances of the Python interpreter running on one thread, and one instance will already be running on the main thread, albeit "suspended" by the call from ctypes. 

So that's my solution:  (1) Python multiprocessing module; (2) Python strings written to the pipe must be terminated with \n. 

Thanks again to all who commented. 



Dec 6, 2021, 13:33 by barry at barrys-emacs.org:

>
>
>
>> On 6 Dec 2021, at 21:05, Jen Kris <>> jenkris at tutanota.com>> > wrote:
>>
>> Here is what I don't understand from what you said.  "The child process is created with a single thread—the one that called fork()."  To me that implies that the thread that called fork() is the same thread as the child process.  I guess you're talking about the distinction between logical threads and physical threads.  
>>
>
> The thread that called fork is cloned into a new thread in the new process.
> It has a clone of the processor registers of the thread, stack memory segment of that thread,
> which means that it has all the stack variables and stack frames from the parent process's thread.
>
>
>
>> But the main issue is your suggestion that I should call fork-execv from the thread that runs the main C program, not from a separate physical pthread.  That would certainly eliminate the overhead of creating a new pthread. 
>>
>
> Forget about physical threads, they only matter when you are performance tuning your code.
>
>
>> I am working now to finish this, and I will try your suggestion of calling fork-execv from the "main" thread.  When I reply back next I can give you a complete picture of what I'm doing. 
>>
>> Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated. 
>>
>
> Barry
>
>
>