Python does not play well with others

[Graham Dumpleton]

On Feb 6, 8:57 am, "sjdevn... at yahoo.com" <sjdevn... at yahoo.com> wrote:
> On Feb 5, 12:52 pm, John Nagle <n... at animats.com> wrote:
>
>
>
> > sjdevn... at yahoo.com wrote:
> > > John Nagle wrote:
>
> > >>Graham Dumpleton wrote:
>
> > >>>On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.invalid> wrote:
>
> > >>>>"Paul Boddie" <p... at boddie.org.uk> writes:
> > >>     Realistically,mod_pythonis a dead end for large servers,
> > >>because Python isn't really multi-threaded.  The Global Python
> > >>Lock means that a multi-core CPU won't help performance.
>
> > > The GIL doesn't affect seperate processes, and any large server that
> > > cares about stability is going to be running a pre-forking MPM no
> > > matter what language they're supporting.
>
> >    Pre-forking doesn't reduce load; it just improves responsiveness.
> > You still pay for loading all the modules on every request.
>
> No, you don't.  Each server is persistent and serves many requests--
> it's not at all like CGI, and it reuses the loaded Python image.
>
> So if you have, say, an expensive to load Python module, that will
> only be executed once for each server you start...e.g. if you have
> Apache configured to accept up to 50 connections, the module will be
> run at most 50 times; once each of the 50 processes has started up,
> they stick around until you restart Apache, unless you've configured
> apache to only serve X requests in one process before restarting it.
> (The one major feature thatmod_python_is_ missing is the ability to
> do some setup in the Python module prior to forking.  That would make
> restarting Apache somewhat nicer).

There would be a few issues with preloading modules before the main
Apache child process performed the fork.

The first is whether it would be possible for code to be run with
elevated privileges given that the main Apache process usually is
started as root. I'm not sure at what point it switches to the special
user Apache generally runs as and whether in the main process the way
this switch is done is enough to prevent code getting back root
privileges in some way, so would need to be looked into.

The second issue is that there can be multiple Python interpreters
ultimately created depending on how URLs are mapped, thus it isn't
just an issue with loading a module once, you would need to create all
the interpreters you think might need it and preload it into each. All
this will blow out the memory size of the main Apache process.

There is also much more possibility for code, if it runs up extra
threads, to interfere with the operation of the Apache parent process.
One particular area which could be a problem is where Apache wants to
do a restart, as it will attempt to unload the mod_python module and
reload it. Right now this may not be an issue as mod_python does the
wrong thing and doesn't shutdown Python allowing it to be
reinitialised when mod_python is reloaded, but in mod_wsgi (when
mod_python isn't also being loaded), it will shutdown Python. If there
is user code executing in a thread within the parent process this may
actually stop mod_wsgi from cleanly shutting down Python thus causing
Apache to hang.

All up, the risks of loading extra modules in the parent process
aren't worth it and could just result in things being less stable.

Graham