[SciPy-dev] mlabwrap scikit [Was: Re: scikits project]

[Alexander Schmolck]

Robert Kern <robert.kern at gmail.com> writes:

Thanks for the feedback!

> > I assume this question reflects my svn-newbie status, but why doesn't the
> > scikits structure look something like this, given that as I understand it
> > scikits are meant to be essentially independent (and hence independely
> > versioned) projects under a common namespace?
> > 
> >       mlabwrap/
> >         branches/
> >         tags/
> >         trunk/
> >           setup.py
> >           scikits/
> >             __init__.py
> >             mlabwrap/
> >               __init__.py
> >       some_other_scikit/
> >          branches/
> >          ...
> > 
> > It somehow seems to me that tags and branches should apply to individual
> > projects and that one could still do convenient releases and svn-checkouts of
> > scikits/ as a whole by using e.g. svn:externals to the individual projects.
> 
> branches/ and tags/ directories can be shared between projects. There's nothing
> special about a branch or a tag in SVN; they are just copies. Just make sure you
> name your branches and tags appropriately (mlabwrap-1.0, etc.). I would like all
> of the scikits to be under one trunk, though, for conveniently grabbing the
> current source of all of the scikits without also grabbing every branch and
> tag.

This is actually exactly what I thought svn:externals would solve (but maybe I
don't understand the problems associated with that solution correctly):

 mlabwrap/
    branches/
    tags/
    trunk/
      ...
 some_other_scikit/
    branches/
    ...

 scikits
    trunk
      scikits
        -> mlabwrap/trunk/            # -> = external
        -> some_other_scikit/trunk/
        ...
    tags
        ...

Thus, if you want all scikits, you just checkout above scikits sub-dir. (I
presumably haven't got the directory-structure quite right, but I hope it
doesn't matter for getting the point across)

On the other hand, if the tags/ and branches/ directories are shared between
all projects, you need to prefix all tags/branches with the project name
(which would otherwise not be necessary) and have potentially quite a lot of
dirs too look at that don't interest you in the least (e.g. when browsing
tags/ or branches/ with trac), and, AFAICT, no particularly easy way to just
check out everything related to your project. I assume this could also be
fixed by 'symlinking' via externals, but you'd need many, many more (per
project: one for the trunk and one for each branch and tag, vs. one for the
trunk of each project -- and possibly the odd 'whole' scikits release tag).

Maybe the structure I proposed would also make importing and exporting of
projects slightly easier (because it mirrors the typical layout of an
individual svn project). Speaking of which -- is there something I can do with
the existing CVS so that it can be easily imported in the scikits svn (in
which case we can get rid of what's already checked in), or would importing
the CVS involve a hassle in any case, because then I'll just archive it on
sourceforge.

The other thing I've been wondering is if such a setup couldn't also be made
to accomodate something like Stefan van der Walt's layout proposal, which as
far as I can see would allow for the most convenient way possible to grab all
scikits and build them:

       setup.py
       scikits/
         __init__.py
         -> mlabwrap/
             mlabwrap_setup.py
             __init__.py
             awmstools.py
             ...
         -> some_other_scikit/ 
             some_other_scikit_setup.py

           ...
Couldn't one have a toplevel setup.py that just runs all
scikits/DIRNAME/DIRNAME_setup.py's it can find, passing through the command line
options (or something along those lines[1])? I.e. the distribution of each scikit
would contain the same ``scikits/setup.py`` which just looks for subdirs with
*setup.py's which it then calls (the XXX_setup.py's could also move one dir
up). To install any subset of scikits from svn one could then just do
something like:


 svn co ...trunk/scikits/{setup.py,scikits{_mlabwrap,some_other_scikit,...}}
 cd scikits; python setup.py install

Just an idea (possibly a bad one :).



> > 
> >       mlabwrap/
> >           setup.py
> >           scikits/
> >             __init__.py
> >             README.txt # etc.
> 
> This should be one level up.

OK

> 
> >             mlabraw.py # dummy importing mlabwrap/mlabraw, for backwards comp.
> 
> This shouldn't be here. I'd just put it into the toplevel mlabwrap/ directory
> and not install it. Just have it there for people to use if they need backwards
> compatibility.

OK

> 
> >             mlabwrap/
> >               __init__.py
> >               -> awmstools.py
> >               -> awmsmeta.py
> 
> This seems appropriate.

Good.

> 
> >               mlabwraw.cpp
> >             test/
> >               test_mlabwrap.py # etc.
> 
> This directory should move into mlabwrap/scikits/mlabwrap/ or even
> mlabwrap/.

OK, mlabwrap/test it is then.

> Leave scikits/__init__.py empty. Do what you like with
> scikits/mlabwrap/__init__.py . For a package as small as yours, importing
> everything from mlabwrap.py is reasonable. Then people will only have to import
> scikits.mlabwrap instead of scikits.mlabwrap.mlabwrap .

OK, I think I'll have __init__.py just do an import * from ./_mlabwrap.py then.

> > Actually, one more thing: distutils vs. scipy distutils vs.
> > setuptools -- which one should mlabwrap-1.0 final use? I'm only really
> > fully familiar with the first one.
> 
> We need setuptools to handle the scikits namespace package. Does your extension
> module use numpy? 

Yes, but optionally -- it also still works with Numeric (before someone
grumbles, Numeric support will be ripped out as soon as 1.0 is released).

> If so it should also use numpy.distutils. Just make sure to import
> setuptools first.
> 
>   import setuptools
>   from numpy.distutils.core import setup
>   ...

Is there a recipe/template for this somewhere? Googling "scipy setuptools"
comes up with

 <http://projects.scipy.org/ipython/ipython/wiki/SetupTools>

as the first hit, which seems to indicate that setuptools is still a bit alpha
and the docs can't be trusted if one wants something that actually works. I've
currently got a distutils setup.py that in common scenarios builds out of the
box with ``python setup.py install`` (it automatically detects if numpy or
Numeric is installed and which matlab version it needs to build for). What I'd
hope setuptool will add (apart from scikits namespace support) is the ability
to easy_install mlabwrap from PyPI, also downloading numpy if required.

I'd assume the same applies to most other projects that will live under
scikits, so it would be good to establish a standard way to do this and
document it somewhere, if there isn't anything around already. Should people
think that setuptools is still a bit problematic, I'll just release 1.0final
on sourceforge and distutils-based (and outside the scikits hierachy),
reserving the scikits treatment for subsequent versions.

thanks,

'as

Footnotes: 
[1] Obviously this would need to be fleshed out a bit and maybe it's more
    hassel than it's worth.