[SciPy-dev] distutils, mtrand, Monte Carlo

[Ed Schofield]

Pearu Peterson wrote:
> Ed Schofield wrote:
>   
>> I've now modified the Monte Carlo package (still in the sandbox) to use
>> the RNG from RandomKit that comes with numpy.  It builds and works fine
>> for me, but only with two directories hard-coded into the setup.py
>> file.  I have some questions (e.g. for Pearu) on how to use numpy
>> distutils to do this portably.
>>
>> 1. Currently the add_headers method of distutils.misc_util.Configuration
>> installs headers to $PREFIX/include/python2.4/numpy.  This doesn't
>> bother me, but do we want this?  All other header files are installed
>> into $PREFIX/lib/python2.4/site-packages/numpy or scipy/.
>>     
>
> add_headers method behaves in a way python installs header files. 
> The reasons for not using add_headers in numpy/scipy cases have been 
> discussed earlier in this list (we need a FAQ item, see numpy ticket #69). 
> numpy uses add_data_files for installing numpy header files.
>   
Ah, thanks.  I've added a brief warning about this to the add_headers
section in DISTUTILS.txt, which is what lead me astray ...

>> 2. The add_library method creates a static library under
>> build/temp.linux-etc/.  Is it possible to use distutils to install this
>> (e.g. librandomkit.a) to a system-wide location?  (This is necessary to
>> prevent scipy having a build dependency on the numpy source files.)
>>     
>
> That would be a desirable feature. For example, when scipy would ship 
> blas/lapack source codes then it would be desirable to install blas/lapack
> libraries so that various scipy subpackages could use them.  
> I have started implementing the support for this feature about half a year 
> ago but I haven't finished it yet, mostly because of I haven't had time
> to figure out what would be good (portable,etc) locations of such 
> libraries and how would subpackages access them in different situations 
> (building whole scipy, building a subpackage without building scipy, 
> etc..).
>
>   

I've been playing around with it some more, and it actually seems that
config.add_extension already supports building and installing shared
libraries.  Normally this is used for building Python extension modules,
but the symbol table seems to be independent of Python unless the source
files explicitly use Python symbols.  So, unless I'm very mistaken, this
works already -- by using the machinery of add_extension rather than
add_library.  Here's an example patch (to NumPy) that builds a beautiful
shared library for randomkit:

Index: numpy/random/setup.py
===================================================================
--- numpy/random/setup.py       (revision 2374)
+++ numpy/random/setup.py       (working copy)
@@ -32,6 +32,10 @@
                         )

     config.add_data_files(('.', join('mtrand', 'randomkit.h')))
+
+    config.add_extension('librandomkit',
+                         sources=[join('mtrand', 'randomkit.c')],
+                         depends=[join('mtrand', 'randomkit.h')])

     return config


I've also committed a new sandbox/montecarlo/setup.py file to scipy that
uses this.  The relevant lines are:

    random_lib_dir = os.path.dirname(numpy.random.__file__)

    config.add_extension('_intsampler',
              include_dirs = [numpy.get_numpy_include(), random_lib_dir],
              libraries=['randomkit'],
              library_dirs=[random_lib_dir],
              runtime_library_dirs=[random_lib_dir],
              sources = [join('src', f) for f in
                        ['_intsamplermodule.c', 'compact5table.c']])

This works fine under Linux.  I'll also test it out with Win32/MinGW. 
(By the way, does distutils provide its own way of retrieving the
site-packages/numpy/random directory?)


-- Ed