No subject

#! rnews 4339
Newsgroups: comp.lang.python
Path: news.xs4all.nl!newsspool.news.xs4all.nl!transit.news.xs4all.nl!news-spur1.maxwell.syr.edu!news.maxwell.syr.edu!nntp.abs.net!attws2!ip.att.net!NetNews1!xyzzy!nntp
From: Harry George <harry.g.george at boeing.com>
Subject: Re: Multiple instances of a python program
X-Nntp-Posting-Host: cola2.ca.boeing.com
Content-Type: text/plain; charset=us-ascii
Message-ID: <xqx4qbtnmin.fsf at cola2.ca.boeing.com>
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.4
Lines: 83
Sender: hgg9140 at cola2.ca.boeing.com
Organization: The Boeing Company
References: <1118947630.385547.231370 at g43g2000cwa.googlegroups.com> <pan.2005.06.17.16.53.10.43909 at REMOVETHIScyber.com.au> <1119043472.013942.233980 at f14g2000cwb.googlegroups.com>
Mime-Version: 1.0
Date: Mon, 20 Jun 2005 14:14:24 GMT
Xref: news.xs4all.nl comp.lang.python:382535

"Rahul" <codedivine at gmail.com> writes:

> Steven D'Aprano wrote:
> > On Thu, 16 Jun 2005 11:47:10 -0700, Rahul wrote:
> >
> > > Hi.
> > > I am part of a group in my univ where we organize a programming
> > > contest. In this contest we have a UDP based server. The server
> > > simulates a game and each contestant is to develop a team of virtual
> > > players. Each team is composed of 75 similar bots...i.e. governed by
> > > the same logic. Thus the contestant submits a single copy of the client
> > > and we instantiate the same program 75 times at the same time.
> > > The problem is that while executables from C source files are small and
> > > we can make 75 processes but we dont know what to do with python.
> > >
> > > If you have a python script and you want that 75 copies of the script
> > > be run simultaneously how will you do it? Is there anyway to do so
> > > without running 75 copies of the python interpreter simultaneously?
> >
> > Have you actually tested the performance of 75 instances of Python
> > running? Do you know that it will be too slow for your server, or are you
> > trying to optimize before testing?
> >
> > I wrote a short Python script, then launched 115 instances of Python
> > executing the script. There was no detectable slowdown of my system, which
> > is far from a high-end PC.
> >
> > The details of the script aren't important. It may even be that what I
> > tested is not even close to the load your server needs to deal with. But
> > you may be surprised at just how easily even a low-end PC copes 75
> > instances of Python. Or perhaps not -- but the only way to tell is to try.
> 
> Well...i havent tried (yes i hear "Premature optimization is evil evil
> evil i say") but the point was that if we can find a solution consuming
> less memory than we can even increase the number from 75 to lets say
> 200. as for hardware we have machines with 1.7 Ghz P4 and 128 mb ram.
> and i cant run them right now...since i am currently not in my
> univ...but asked now since we are planning right now and wanted to see
> which languages we can support. Probably c,c++,python and lisp using
> clisp...and java if we can find a way to avoid running 75 jvms...this
> year we supported c,c++,java and actually ran 75 jvms using 3 machines
> and it was horrible so we are looking for better ways for the 2006
> contest. And we may port our server from java to python too but it
> seems not many people had python installed but most had java installed.
> 
> rahul
> 

As others have said, start by testing.  Given 1.7 MHz and 128MB, I'm
guessing your performance problems are coming from swapping.  Simply
getting more RAM in the box would help.

A quick check on process load just for the instances themselves can be
done by gradually increasing max_processes (see below) until you swap.
On a box with 256MB total, 150MB free, I was getting over 150 children
before swap was noticeable.  By watching "top" I found each child was
using about 3MB, of which about 2MB is "shared" (basically, the python
library).  In other words, each instance was costing 1MB.

Of course, your actual apps will require more RAM than this little
test program.

---main program---
    max_processes=10
    progname="child.py"
    for i in xrange(max_processes):
        os.spawnv(os.P_NOWAIT,progname,[progname,str(i)])

---child program---
    def msg(txt):
        sys.stdout.write(txt)
        sys.stdout.flush()
    ....
    #get childnum from comline parameters
    ....
    for i in xrange(10):
        msg('%i ' % childnum)
        time.sleep(1)

-- 
harry.g.george at boeing.com
6-6M21 BCA CompArch Design Engineering
Phone: (425) 294-4718