[Distutils] EasyInstall: svn support

[Phillip J. Eby]

At 04:26 PM 5/29/2005 -0500, Ian Bicking wrote:
>Phillip J. Eby wrote:
>>>   Basically it detects the svn index page (for http repositories, which 
>>> can't be detected based on the URL), or the svn: URL type, and 
>>> downloads from there.  I'd like for it to fix up the version based on 
>>> the svn revision, but I haven't looked closely enough at that part yet, 
>>> or if it's even possible since setup.py typically has a version 
>>> hardcoded in it.  I'm not even sure what the version number should look 
>>> like, so that it sorts properly with released versions (or even if it 
>>> should sort with released versions at all; should versions also 
>>> indicate branches, like stable vs. development?)
>>
>>Yeah, I think sticking with the setup.py version is best; EasyInstall and 
>>pkg_resources use the generated PKG-INFO to find out stuff like that.
>
>Well, at least for my own packages I'll try to make the version better, by 
>dynamically generating it in setup.py;

Note that it needs to be robust if the build machine doesn't have svn.


>  I tried to do this a little with Paste, but there's some improvement to 
> be done there as well.  What form should it take?
>
>E.g., if the last revision was 0.1, what should the trunk version be? 
>0.2alpha-svn2822?  Does the released version need to be 0.2final then? Or 
>do are version numbers sorted based on a better algorithm than string 
>comparison?

See pkg_resources.parse_version() for the version translator.  But since 
the algorithm is a bit obscure, here are the basic ideas:

* The string is converted to a tuple of strings
* Runs of numbers are split from non-numeric runs
* The string "*final" is added to the end of the tuple
* Dots are not kept in the string; only '-' and non-numerics
* Certain alpha strings are tweaked; pre, rc, and so on.
* Numeric runs are zero-padded for numeric comparision
* Elements that equal zero are dropped from each tuple-level numeric run

In other words, the revision system knows that '2.0' and '2.0.0' are 
equivalent version numbers.  It knows that alpha, beta, and candidate 
releases come *before* final releases, but that 'nb', 'pl' releases are 
*after*, as are '-' releases.  So, it knows that '2.0-1' is an "older" 
release than '2.0.1'.

Thus, the best and simplest way to add build/date/revision tags is just to 
pop a '-' and the extra data.  At this point, I've added '--tag-date', 
'--tag-build=TAG' and '--tag-svn-revision' options to bdist_egg that can be 
used individually or in combination.  If you did something like:

     setup.py bdist_egg --tag-build=svn --tag-date --tag-svn-revision

then today you'd get an egg with a revision like 
'0.2a-svn-2822-20050529'.  In practice, I suspect most people will use 
either tag-date or tag-svn-revision, because using both is kind of 
redundant unless the package is also integrating external components.

>>I was a bit puzzled by the diff at first; looks like you created a 
>>reversed diff.
>
>Oops.  I have a hard time with diff for some reason.  Like some kind of 
>chronological dyslexia.

difflexia, perhaps?  :)  Actually, difflexia sounds like a problem 
*reading* diffs.  Ah well.


>It seems like a match object is a little odd to pass around.

That's because it was an error.  It should've read 
'self._download_url(scheme.group(1),...' instead.


>I was getting an error when calling self.samefile(filename, 
>filename_that_doesnt_exist).  Maybe self.samefile needs to check 
>os.path.exists(destination).

Aha.  That's a platform-specific issue that I'll need to fix.  Windows 
doesn't have 'os.samefile', so I didn't see the problem.  I'll fix 
self.samefile() to check whether both paths exist before using os.samefile.


>>Before doing this, I think it would be a good idea to check the headers 
>>for a text/html Content-Type, so as not to be readline() a big chunk of 
>>binary.  :)
>
>Ah, yes, indeed.

I've done this in my version now.


>>>-        # Actually, this probably doesn't do anything; but somehow this
>>>-        # information should get put into the version string...
>>
>>It indeed doesn't do anything; you'd be better off munging the setup.py 
>>to change the setup version.
>>Maybe what I can do is have the build stuff look for a .svn dir in the 
>>build area, and if one is present, grab the revision and add it to the 
>>end of the version string supplied to setup().  Or simpler still, I could 
>>add a '--tag-svn-revision' option to bdist_egg, that would make *it* do 
>>that.  I've been thinking it would be nice to have a '--tag-build=NUMBER' 
>>or '--tag-date' option for bdist_egg anyway, to support daily builds and 
>>such.  So adding an option to "get the tag value from 'svn info'" would 
>>just be a minor variation on the theme.
>
>Yes, that would work well.

I've decided I don't want this to happen automatically for EasyInstall egg 
builds, though.    If I later let you pass build options through to the 
setup.py, you'll be able to do it that way.  I just don't think it should 
invoke --tag-svn-revision by default.  For one thing, I don't have a way to 
know if subversion is installed, and you could've downloaded a source .zip 
with .svn dirs accidentally included.  (I've seen this from time to time 
with CVS dirs.)



>It should really look for a scheme of 'svn+ssh' as well, but those are a 
>pain to make readable ;)

I'm actually checking for 'scheme=="svn" or scheme.startswith("svn+")', so 
they ought to work too.


>   And technically file: should work for svn too, which I suppose you 
> could detect because it points to a directory with a db/fs-type 
> file.  Eh, those can wait.

I'd need more details before I could implement that.

In the meantime, I'm going to go ahead and build/upload an 
EasyInstall/setuptools-0.3a2 release with the new subversion support, 
revision tagging, and bug fixes.  Thanks for your help!