[SciPy-User] scipy import problem in apache-mod_wsgi environment

Myunghwa Hwang mhwang4 at gmail.com
Sat Nov 5 15:52:54 EDT 2011 

Hello, list!

First of all, this will be a long message, due to the complexity of our
environment.
So, please be patient with my question.

I am trying to run a simple django application in a cluster environment.
And, my application hangs while it imports scipy.linalg, and both scipy and
apache do not write out error messages.
When I run my application in my local python shell, it imports
scipy.linalg. But, somehow it does not when it is run by apache.
So, after reading this message, please share any ideas about how to debug
this problem or new solutions to address this issue or deploy my
application.

Now, let me explain our current setup.
1. OS
-- The server is a compute cluster where each node runs centos 6 that was
installed from a clean version of centos6 minimal.
2. Apache
-- Apache 2.2 is also manually installed from one of default linux
repository.
  To be specific, it is installed from source code together with httpd-dev.
3. Python
-- Python 2.7.2 is also installed from source code across all nodes in the
cluster. Its source code is downloaded from python.org's ftp.
4. Python packages: nose, numpy, scipy
-- Nose 1.1.2 was downloaded from pypi.python.org and installed from its
source code.
-- numpy 1.6.1 was downloaded and installed from a linux repository. When
building numpy, gnu95 fortran complier was used.
-- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from
their source code.
----- atlas was from sourceforge's 3.8.4 stable version. To compile altas,
gcc was used.
----- lapack and blas was obtained from netlib.org's repository. To compile
the package of lapack and blas, gforan was used.
----- Finally, after exporting paths to blas, lapack, and atlas,
scipy-0.9.0 was installed from its source code.
      scipy was obtained from sourceforge.net's repository.

All of the above were installed in the same way across all nodes in our
cluster.
Since I am the only user of the cluster who needs to run python web
applications,
I installed python virtualenv package in my local directory.
Within my virtual environment, django-1.3 and pysal-1.2 (our own package)
were installed.
To deploy my web applications, we used mod_wsgi.
mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2.
My application is attached. Basically, it is 'hello world' application that
tests if numpy, scipy, and pysal can be imported.
In the attached file, lines 4-9 are just adding paths to django and pysal
so that apache knows where to find these packages.
Also, to let apache know where to find atlas-related packages, the path to
those packages was added to the LD_LIBRARY_PATH environment variable in the
/etc/sysconfig/httpd file.

When I first ran my application, it just hung and wrote no message.
So, across scipy.linalg modules, I added print out statements to figure out
at which point the import was broken.
Here is the messages I got when I imported scipy.linalg in my local python
shell.

   1. ########################
   2. starting linalg.__init__
   3. pre __init__.__doc__
   4. pre __init__.__version__
   5. pre __init__.misc
   6. pre __init__.basic
   7. #######################
   8. Starting basic
   9. pre basic.flinalg
   10. pre basic.lapack
   11. pre basic.misc
   12. pre basic.scipy.linalg
   13. pre basic.decomp_svd
   14. pre __init__.decomp
   15. ################
   16. starting decomp
   17. pre decomp.array et al.
   18. pre decomp.calc_lwork
   19. pre decomp.LinAlgError
   20. pre decomp.get_lapack_funcs
   21. pre decomp.get_blas_funcs
   22. ####################
   23. Starting blas
   24. pre blas.scipy.linalg.fblas
   25. pre blas.scipy.linalg.cblas
   26. pre __init__.decomp_lu
   27. pre __init__.decomp_cholesky
   28. pre __init__.decomp_qr
   29. #################
   30. Starting special_matrices
   31. pre special_matrices.math
   32. pre special_matrices.np
   33. pre __init__.decomp_svd
   34. pre __init__.decomp_schur
   35. ##################
   36. starting schur...
   37. pre decomp_schur.misc
   38. pre decomp_schur.LinAlgError
   39. pre decomp_schur.get_lapack_funcs
   40. pre decomp_schur.eigvals:1320454147.23Fri Nov  4 17:49:07 2011
   41. schur testing
   42. pre __init__.matfuncs
   43. #####################
   44. Starting matfuncs
   45. pre matfuncs. asarray et al
   46. pre matfuncs.matrix
   47. pre matfuncs.np
   48. pre matfuncs.misc
   49. pre matfuncs.basic
   50. pre matfuncs.special_matrices
   51. pre matfuncs.decomp
   52. pre matfuncs.decomp_svd
   53. pre matfuncs.decomp_schur
   54. pre __init__.blas
   55. pre __init__.special_matrices

When scipy.linalg is successfully imported, I should get these messages.
But, when my web application tried to import scipy.linalg, the output
messages stop at line 41.
At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was
already imported at line 16, scipy ignores it and continues to import other
modules in my local shell.
But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or
reload decomp.py and seems to kill my web application.
This is really odd, because python does not give any message about this
error and neither does apache. apache just hangs without sending out any
response.
Since lapack and blas functions were imported successfully, the problem
seems not related to path setup.

If anyone in the list has any insights into or experience into this kind of
symptom,
please share your insights and experience. In particular, debugging
techniques or less-known installation/compilation problems would be helpful.
I feel like I am at a dead end. So, please help me.

Thanks for reading this post.
I will look forward to yo


-- 
Myung-Hwa Hwang
GeoDa Center
School of Geographical Sciences and Urban Planning
Arizona State University
mhwang4 at gmail.com or Myunghwa.Hwang at asu.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20111105/248ea428/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: installation_info
Type: application/octet-stream
Size: 2790 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20111105/248ea428/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hello.py
Type: application/octet-stream
Size: 1624 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20111105/248ea428/attachment-0001.obj>

More information about the SciPy-User mailing list