[Distutils] Splitting distribute in several distributions

[Sridhar Ratnakumar]

On Tue, 11 Aug 2009 17:44:01 -0700, Tarek Ziadé <ziade.tarek at gmail.com>
wrote:

> 2009/8/12 Sridhar Ratnakumar <SridharR at activestate.com>:
>> I am glad that you have started thinking about the module split.
>>
>> 1) Why are the distributions named in CamelCase? Why not
>> 'distribute.resources' instead of 'DistributeResources' (like, for  
>> instance,
>> zope.interface instead of ZopeInterface).
> Because the distribution name is not the packages names.
> See for instance the Paste project : PasteScript, Paste, PasteDeploy.

Ok. So there are apparently two conventions for naming distributions.
However, zope.*, zc.*, z3c.* fall under more wide-namespaces (zope is a
large namespace with many sub-packages).


>> 2) In your proposal below, version comparison is part of
>> distribute.installer; this means distribute.resources (and other modules
>> needing version comparison) will be depending on distribute.installer,
>> correct? Is this intended, or should version comparison go to something  
>> like
>> distribute.core?
> When would distribute.resources need version comparisons if its sole
> role is to provide query APIs
> for installed packages ? (pre-PEP 376, then PEP 376-compliant)

When multiple versions of a same package are installed, then perhaps
distribute.resources might have to return them in sorted order .. but,
well, I am not aware about the querying API yet .. so cannot say for sure.

Further, if you implement distribute.pypi .. then that too would require
version comparison, would it not? In that case, distribute.pypi would
depend upon distribute.installer .. and vice versa, I suppose, leading to
circular dependencies.  Eg:
distribute.pypi.Distribution.find('PasteScript').get_releases() may have
to return the releases in sorted order (by version number).

There must be a reason why PJE kept parts of setuptools in
pkg_resources.py. What is it? We may have to consider it before splitting
pkg_resources.py itself.


>> 3) PyPM's backend uses a) pkg_resources' version comparison, b)
>> package_index's download logic (not API-friendly). I'd be interested to  
>> see
>> distribute.installer provide this download logic (finding URLs,  
>> tarballs and
>> fetching them) as an API. I believe pip and zc.buildout too relies on  
>> this
>> download logic.
> pakage_index would be included in distribute.installer, and we could  
> probably
> add simpler APIs to use it to drive a download

Note that package_index is not API-friendly. A few violations I can
think of: 1) it prints warning messages to stderr without programmer
control, 2) throws random network exceptions that should be handled in a  
custom exception (see the issues I reported in distribute tracker) 3) no  
ability to control URL types (eg: I want to fetch only tarballs/zipfiles,  
not svn/hg/etc..) and so on.

This is roughly the code I currently use in PyPM (that uses setuptools):

(...)
class PackageIndex2(PackageIndex):
     def _download_svn(self, url, filename):
         raise SVNDisabled, 'svn not allowed at the moment'
(...)
def download_latest_sdist(req, directory):
     (...)
     # TODO: we need timeout on network operations (not just socket  
timeouts)
     try:
         sdist = pi.fetch_distribution(req, directory, source=True)
     except (DistutilsError, httplib.BadStatusLine, socket.timeout), e:
         # catch BadStatusLine:  
http://bitbucket.org/tarek/distribute/issue/18/
         raise PackageError, e
     if sdist == None:
         raise SdistMissing, 'cannot find a sdist for %s' % req
     if not exists(sdist.location):
         raise SdistLocationMissing, 'downloaded sdist <%s> missing' %  
sdist.location
     return sdist

-srid