waste of resources ?

[Arne Mueller]

Carey Evans wrote:
> 
> Arne Mueller <a.mueller at icrf.icnet.uk> writes:
> 
> [snip]
> 
> > > def sigchild_handler(signum, frame):
> > >     os.wait()
> 
> [snip]
> 
> > Hm, that doesn't work properly  with my program. THe number of zombie
> > processes is steadily at about 20, which means there are 20 unhandled
> > died children.
> 
> Unix doesn't guarantee you'll get exactly one signal delivered per
> event.  To be sure to get all the children, you need to loop over the
> processes with waitpid().
> 
> I think something like this should do it:
> 
> ----------
> def sigchld_handler(signum, frame):
>     print "Checking for processes..."
>     try:
>         while 1:
>             pid, sts = os.waitpid(-1, os.WNOHANG)
>             if pid == 0:
>                 break
>             print "Process %d ended with status %04x." % (pid, sts)
>     except OSError, detail:
>         if detail.args[0] != errno.ECHILD:
>             raise
>         else:
>             print "No more children."
>     else:
>         print "Some children yet to finish."
> ----------

Hm, that's similar what I do in my program but I always have as many
zombies as I fork processes, here's a program fragment:

cpu = 4
used = 0
for fn in files:

        if cpu:
            if used < cpu: 
                r, w = os.pipe()
                pid = os.fork()
                used = used + 1
                if pid: # parent
                    os.close(w)
                    chfds.append(r)
                    chpids[r] = pid # dict. with filedescriptor as key
	            # fork children for available processors
                    if used < cpu and fn != last: continue 
                else:   # child
                    os.close(r)
                    param['files'] = [fn]
		    # some application specific stuff
                    msa = ProcessMSA(param)
                    msa.of = None
                    msa.dbkeys = {}
                    msa.param = {}
	            # pickle objecty and send it to parent
                    p = mpickle.dumps(msa)
                    r, w, o = select.select([], [w], [])
                    os.write(w[0], p)
                    os._exit(0)
	    # parent reads from ready file descriptors 
            while used >= cpu or (fn == last and used):
                r, w, o = select.select(chfds, [], [])
                for i in r:
                    msa = mpickle.loads(readPickleStream(i))
                    os.close(i)
                    stderr.write('number %d\n' % len(stats.ids.keys()))
                    stats.add(msa)  # do something with the received
object
                    used = used - 1 # adjust number of used processors
                    chfds.remove(i)
                    os.waitpid(chpids[i], os.WNOHANG)
                    del chpids[i]

Why does that code produce zombies? Do I've to send a signal to the
child that it has to die? I mean maybe ther child exits when the parent
hasn't reached the os.waitpid?

	Thanks for discussion,

	Arne