[SciPy-Dev] import scipy.stats hangs in mod_python application

[Robert Kern]

On Sun, Aug 28, 2011 at 18:01, fork <forkandwait at gmail.com> wrote:
> Hi all,
>
> This may be really tricky to diagnose, given all the rather weird parameters,
> but here goes:
>
> I have a mod_python application that takes a bunch of mortality rates via a
> form, and processes them with scipy, numpy, and matplotlib. It is running on a
> Gentoo box.  (Gentoo + mod_python seemed like a good idea at the time, but so
> did the Edsel... anyway, I still have to maintain this beast...)
>
> In the multitude of files that get loaded when a mod_python based page is
> called, there is a line "import scipy.stats as ST".  That line works just great
> if I am running python at the command line, but it hangs forever when in the
> modpython application; I know this because I have run it both with an exception
> before it (barfs on the exception) and after it (just sits there, no errors to
> the apache log, etc).
>
> scipy.__version__ 0.9.0
>
> python is 2.7.1
>
> Anybody have any ideas how to fix this or even diagnose it?

So, has your program ever worked under mod_python? Is this an
intermittent problem, or is it reliably repeatable? In your program,
do you import numpy before the scipy.stats import?

IIRC, under certain configurations mod_python tries to instantiate
multiple interpreters in the same process. For pure Python code, this
is usually not too atrocious, even though it is officially
unsupported. However, numpy maintains some global pointer tables, and
people have run into problems trying to run multiple Python
interpreters in the same process when the programs use numpy. There
may be mod_python configuration options that control this behavior.

First, I would convert the entry point into your application into a
WSGI application. Since it looks like there is only one entry point,
doing basic form processing before passing the inputs to the
computational code, this should be trivial. It's really just a matter
of converting the mod_python idioms into WSGI idioms.

  http://wsgi.org/wsgi/

Now that you have a WSGI application, there are a *lot* of different
ways to deploy it. For Apache, mod_wsgi is much preferred over
mod_python these days, although since it was written by the same
author under very similar constraints as mod_python, he may have used
the same multiple interpreters trick. You can use mod_fcgid or
mod_scgi to communicate with a separate process running your scipy
code fronted by flup:

  http://trac.saddi.com/flup

Using mod_proxy, you can have Apache sit on the front end and
communicate with any number of WSGI-specific HTTP servers. gunicorn is
a nice one:

  http://gunicorn.org/

You can look at other options here:

  http://wsgi.org/wsgi/Servers

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco