Forking and Threads

[Paul Robinson]

OK, before I start, I realise that threading and forking are not
necessarily well known for playing well with each other. But given that,
and assuming that this is the road I have chosen to go down can anyone
explain why the attached program (also available at:
http://cobweb.quantisci.co.uk/pub/english.cgi/d10030658/forkingtest2.py
) will run happily for several minutes (upto about 10mins so far) and
then just sit there with the main process running with as much CPU time
as the OS will give it!

This is the "distilled down" version of the problem that I'm seeing in a
much larger application.

I've only tested this on:
Python 1.5.2
Red Hat Linux release 6.2 (Zoot)
Kernel 2.2.14-5.0smp on a 2-processor i686

and would very interested in how (and if) it performs on other operating
systems - obviously only those with both threading and forking need
apply.

I find that "watch -n 1 "pstree -c -p -a <user name> | grep python |
grep -v grep"" gives a overview of the running of the program as you can
see the child processes appearing and disappearing.

A large number of child processes and a short run time for the children
seems to cause the crash most quickly. These parameters can be set with
-MC (MAX_CHILDREN) and -MS (MAX_SECONDS the child will wait for).

forkingtest2.py -MC 20 -MS 3 crashes after a while on my dual 350MHz
PII.

Any insight into this behaviour would be gratefully appreciated.

Paul Robinson
Business Collaborator Development Manager
Enviros Software Solutions
WWW: http://www.businesscollaborator.com
-------------- next part --------------
#!/usr/bin/python

import os, sys
import marshal

from threading import Thread, currentThread
from time import sleep, time
from random import choice, seed
from string import atoi

CHILD = 0
MAX_CHILDREN = 5
MAX_SLEEP = 5

def log(*args):
	print '\t'*CHILD, os.getpid(),' : ', currentThread(), \
		' : ', reduce(lambda x, y: x + ' '+ y, map(str,args))

def processLoop():
	children = []
	while 1:

		while len(children) < MAX_CHILDREN:
			pid = os.fork()
			if pid:
				#parent
				children.append(pid)
			else:
				#child
				try:
					global CHILD
					CHILD = CHILD + 1
					#Do stuff
					seed(time())
					pause = choice(range(1,MAX_SLEEP))
					log('Sleeping', pause)
					sleep(pause)
					log('Exitting')
				finally:
					os._exit(0)
		else:
			log('Too many children')

		sleep(3)
		# Wait on children
		for i in range(len(children)-1, -1, -1):
			# Call waitpid (os.WNOHANG - non-blocking)
			status = os.waitpid(children[i], os.WNOHANG)
			if status[0] > 0:
				# Child has been "deaded"
				del children[i]

if __name__ == '__main__':
	try:
		i = sys.argv.index('-MC')
		global MAX_CHILDREN
		MAX_CHILDREN = atoi(sys.argv[i+1])
	except:
		pass

	print 'Set MAX_CHILDREN to', MAX_CHILDREN, ' - use "-MC n" to change'

	try:
		i = sys.argv.index('-MS')
		global MAX_SLEEP
		MAX_SLEEP = atoi(sys.argv[i+1])
		MAX_SLEEP = ((MAX_SLEEP >= 2) and MAX_SLEEP) or 2
	except:
		pass
	print 'Set MAX_SLEEP to', MAX_SLEEP, ' - use "-MS m" to change (where m >= 2)'

	pl = Thread(target=processLoop, name='processLoop')
	pl.start()
	currentThread().setName('Main Loop')
	f = open('/tmp/temp','a')
	while 1:
		log('Looping')
		sleep(5)