server bootstrapping upon connection (WARNING: LONG)

[Benjamin Han]

On 2004-02-10 19:59:47 -0500, Benjamin Han <this at is.for.spambot> said:

> On 2004-02-10 18:22:30 -0500, ralf at brainbot.com said:
> 
>> Benjamin Han <this at is.for.spambot> writes:
>> 
>>> On 2004-02-10 13:46:37 -0500, ralf at brainbot.com said:
>>> 
>>>> fortepianissimo at yahoo.com.tw (Fortepianissimo) writes:

> Here is the situation: I want my server started up upon connection.
> When the first connection comes in, the server is not running. The
> client realizes the fact, and then starts up the server and tries to
> connect again. This of course all happens on the same machine (local
> connection only).
> 
> The connections can come in as fast as 30+/sec, so the server is
> threaded (using SocketServer.ThreadingTCPServer). Further, the server
> initialization is threaded into two: one is to do the real, lengthy
> setup, the other is start lisenting ASAP.

>>>>> The problem: I need to prevent multiple copies of the server being
>>>>> started. I did this by using file locking (fcntl.lockf()). However,
>>>>> not every time the code successfully prevented the server being
>>>>> started up more than one time. Here is the relevant code:
> 
> ... from the "ps" log during the stress test, I still get two servers 
> started up, listed by ps like this:
> 
> 27308  ??  S      0:01.25 python -OO fooServer.py
> 27465  ??  SV     0:01.25 python -OO fooServer.py
> 
> In this case the one with pid 27308 was the first server process 
> started. Some time later, with unknown reason, another server entry 
> will show up in ps log, but the status is always "RV" or "SV" (e.g., 
> pid 27465). The way I started the server in the code is os.system() 
> call, and from the man page of ps, 'V' means "The process is suspended 
> during a vfork."
> 
> The weird thing is I also have log statements in the server script, and 
> from the log file only ONE copy is actually started. I start to suspect 
> that I might have misunderstood the ps log - could it be there really 
> is only ONE copy running, despite there're (up to) 2 entries shown in 
> the ps log?

Because of this discrepancy between the ps log and the log produced by 
running the server script, I started to suspect that the second server 
instance shown in the ps log was a forked process, especially since it 
always came with a 'V' (vforked) state. Adding a "-o flags" to the ps 
command showed the following (exerpt);

8896 Fri Feb 13 03:47:25 2004     R    2004004      -      - 2a567f8 
python -OO fooServer.py
8997 Fri Feb 13 03:47:34 2004     RV   8000014      -      - 2a54028 
python -OO fooServer.py

After checking sys/proc.h:

#define P_VFORK         0x2000000       /* process has vfork children */
#define P_INVFORK       0x8000000       /* proc in vfork */

Basically confirmed that the server process was vforked. I then did a 
"grep -r fork *" but none showed in any of my code. My question then 
is, where could such a "hidden" vfork happen? I did use the following 
in my code:

1. pyDNS (http://pydns.sourceforge.net/)
2. os.popen(), os.popen2(), os.system()

Any hint is extremely welcome!