[SciPy-dev] mlabwrap scikit [Was: Re: scikits project]

[Alexander Schmolck]

Robert Kern <robert.kern at gmail.com> writes:

[on the question whether the svn-layout {branch,tags,trunk}/subproject
(Robert) or subproject/{branch,tags,trunk} (me) is preferable for scikits]

> By and large, it simply doesn't matter to "get everything related to your
> project". Believe me, you don't want all of branches/ and tags/ in a
> checkout.

I have done so in order to get some overview over a project that I was
unfamiliar with, but I agree that that's hardly an important use case.
Migration convenience seems more relevant, but unless rearranging dir
structure in svn repositories is really hard, that's not a big factor either.

> On the other hand, "getting all of the packages in scikits" does matter. 

Sure, but not that often either. Why would many people want to frequently
update the svn versions of all of several unrelated and fairly special purpose
scientific packages? I would assume the typical case is that someone hacks
e.g. mlabwrap and updates that from time to time rather than a dozen other
porjects.

> IMO, the inconvenience of prefixing your branches and tags is secondary to
> the performance problems of svn:external, which slows down all checkouts and
> updates for everyone 

Only dirs with svn:externals would be affected, right? Since the only such dir
would be /scikits, which most people won't checkout, I don't think there would
be much of an impact (see above).

> (although I'll have to double-check that claim for svn:external links within
> the same repository).

I think you're right: I added an external 'ext' on mlabwrap to mlabwrap/test,
and it appears to slow down checkout; I've got a very old svn client, but you
can try yourself:

svn co http://scipy.org/svn/scikits/trunk/mlabwrap/
 
> Also, it appears that Trac doesn't like svn:external to other repositories;
> I'm not sure if that extends to svn:external within the same repository.
> 
> http://trac.edgewall.org/wiki/TracFaq#DoesTracsupportsvn:externalsubversionrepositories

For this purpose it seems to work fine:

<https://projects.scipy.org/scipy/scikits/browser/trunk/mlabwrap>

It just displays the svn:externals property, which is all you want here, I
think.

Anyway, the choice between either layout (i.e. subproject internal/external
{tags,branches,trunk}) seems not terribly important. Each approach has
advantages and disadvantages, and apparently both are common; the 'official'
SVN book authors prefer internal, as do I, but write about the toplevel
{tags,branches,trunk} approach:

<http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.reposadmin.projects.chooselayout>

    There's nothing particularly incorrect about such a layout, but it may or
    may not seem as intuitive for your users. Especially in large,
    multi-project situations with many users, those users may tend to be
    familiar with only one or two of the projects in the repository. But the
    projects-as-branch-siblings tends to de-emphasize project individuality
    and focus on the entire set of projects as a single entity. That's a
    social issue though. We like our originally suggested arrangement for
    purely practical reasons: it's easier to ask about (or modify, or migrate
    elsewhere) the entire history of a single project when there's a single
    repository path that holds the history entirepast, present, tagged,
    branched andfor that project and that project alone.
      

> > Maybe the structure I proposed would also make importing and exporting of
> > projects slightly easier (because it mirrors the typical layout of an
> > individual svn project). Speaking of which -- is there something I can do with
> > the existing CVS so that it can be easily imported in the scikits svn (in
> > which case we can get rid of what's already checked in), or would importing
> > the CVS involve a hassle in any case, because then I'll just archive it on
> > sourceforge.
> 
> I don't know. You'll have to read the cvs2svn documentation.

Sorry, I should have phrased this better. I have already read the cvs2svn docs
and converted the CVS repository (just a single trunk, no branches or tags).
If I put a tar.bz2 of that repository on the web, can someone with admin
rights easily install it in lieu of the existing repository? Is Berkeley db
fine as backend? If it's not easy and quick to do, I'm also happy to loose the
revision history and abandon conversion attempts.

> > The other thing I've been wondering is if such a setup couldn't also be made
> > to accomodate something like Stefan van der Walt's layout proposal, which as
> > far as I can see would allow for the most convenient way possible to grab all
> > scikits and build them:
> > 
> >        setup.py
> >        scikits/
> >          __init_xg_.py
> >          -> mlabwrap/
> >              mlabwrap_setup.py
> >              __init__.py
> >              awmstools.py
> >              ...
> >          -> some_other_scikit/ 
> >              some_other_scikit_setup.py
 >
> Having two ways to install something is just begging for trouble.

I wasn't advocating two ways; you always call scikits/setup.py and it just
installs different amounts of stuff depending on how many subdirs you've
checked out.

> > Couldn't one have a toplevel setup.py that just runs all
> > scikits/DIRNAME/DIRNAME_setup.py's it can find, passing through the command line
> > options (or something along those lines[1])?
> 
> That's unworkable. I've tried.

I suspected that it might turn out to be. Pitty.

> >> If so it should also use numpy.distutils. Just make sure to import
> >> setuptools first.
> >>
> >>   import setuptools
> >>   from numpy.distutils.core import setup
> >>   ...
> > 
> > Is there a recipe/template for this somewhere? Googling "scipy setuptools"
> > comes up with
> > 
> >  <http://projects.scipy.org/ipython/ipython/wiki/SetupTools>
> > 
> > as the first hit, which seems to indicate that setuptools is still a bit alpha
> > and the docs can't be trusted if one wants something that actually works.
> 
> Fernando's a curmudgeon, and that page is old. Ignore him.  :-)

OK, I'm going the``ez_setup.py`` way.
 
> Like I said, just import setuptools before you import numpy.distutils. Then use
> numpy.distutils as normal to handle all of the building and stuff. setuptools
> adds some keywords to setup() that you should also provide, namely
> 
>   namespace_packages=['scikits'],
>
> That's all that's necessary. There's no particular magic to combining setuptools
> and numpy.distutils. 

Well, it's not quite obvious how to fully take advantage of setuptools,
though. One of the main reasons for using it is that it's meant to download
and install depenedencies automatically, but that can't work if my setup.py
import something from its sole dependency (numpy) to start with. Surely there
must be some way to write packages that depend on numpy but can be installed
automatically (and download numpy if required)? And why do I need to use
numpy.distutils in the first place? I find
<http://www.scipy.org/Documentation/numpy_distutils> rather unhelpful and I
didn't find any other documentation.

Another thing: should ``scikits/__init__.py`` be really empty? From

<http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages>

it looks a bit to me like it should contain:

  __import__('pkg_resources').declare_namespace(__name__)

?

Finally, what is the preferred download url for scikits projects? Should I
continue to host the file-release on SF, or should they go somewhere else?
Same for the webpage.

So I think some kind of template for scikit authors would be useful and I'd
suggest that once I've got setup.py etc. ironed out I put some info for other
prospective scikit authors on a wikipage on scipy.org -- what would be a good
place?

Finally, just to double check, does this directory structure look good to you: 

mlabwrap/
   setup.py
   README.txt
   tests/                # N.B. renamed from 'test'
     test_mlabwrap.py
     ...
   scikits/
     __init__.py         # empty
     mlabwrap/
        __init__.py
        _mlabwrap.py
        mlabraw.py
        -> awmstools.py
        -> awmsmeta.py

?

thanks,

'as