scaling problems

[Graham Dumpleton]

On May 20, 2:00 pm, James A. Donald <jam... at echeque.com> wrote:
> > > 2.  It is not clear to me how a python web application scales.  Python
> > > is inherently single threaded, so one will need lots of python
> > > processes on lots of computers, with the database software handling
> > > parallel accesses to the same or related data.  One could organize it
> > > as one python program for each url, and one python process for each
> > > http request, but that involves a lot of overhead starting up and
> > > shutting down python processes.  Or one could organize it as one
> > > python program for each url, but if one gets a lot of http requests
> > > for one url, a small number of python processes will each sequentially
> > > handle a large number of those requests.  What I am really asking is:
> > > Are there python web frameworks that scale with hardware and how do
> > > they handle scaling?
>
> Reid Priedhorsky
>
> > This sounds like a good match for Apache withmod_python.
>
> I would hope that it is, but the question that I would like to know is
> how does mod_python handle the problem - how do python programs and
> processes relate to web pages and http requests when one is using mod_python, and what happens when one has quite a lot of web pages and
> a very large number of http requests?

Read:

  http://blog.dscpl.com.au/2007/09/parallel-python-discussion-and-modwsgi.html
  http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading

They talk about multi process nature of Apache and how GIL is not as
big a deal when using it.

The latter document explains the various process/threading modes when
using Apache/mod_wsgi. The embedded modes described in that
documentation also apply to mod_python.

The server is generally never the bottleneck, but if you are paranoid
about performance, then also look at relative comparison of mod_wsgi
and mod_python in:

  http://code.google.com/p/modwsgi/wiki/PerformanceEstimates

Graham