[Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals

[Tarek Ziadé]

Hello

I have followed most of the threads from the past days, and we talked
a lot on IRC with people from Fedora, Debian, Enthought, TG2 on
possible enhancements
While the other threads are continuing in deeper details, I would like
to start a fresh thread were people don't have to re-read everything
to be able to give their opinions
on very precise points,

This thread is focusing on shouting out the current problems and the
solutions that can be adopted.

I'd like to have "+1" and "-1" on each proposal, with at most one sentence.
or fix a mistake if there is. That could help us speed up the work.

let's try to keep this thread concise, if you want to discuss deeply
on a problem, start another thread, and
i'll follow it to fix my summary.


The problems
===========

1/ the dependencies of a package are not expressed in the Require
metadata of the package most of the time.
   adding a dependency to a module is not really done, developer add
dependencies to packages.

   Furthermore, developer tend to use setuptools "install_requires"
and "tests_require" arguments to express dependencies.

   So basically, you have to run an egg_info to get them, because the
info files are generated by commands.

2/ the existence of PyPI had a side-effect: people tend to push the
entire doc of the package in one single field (long_description)
   to display them at PyPI. The documentation of the package is not
cleary pointed to others.

3/ the metadata infos cannot be expressed in a static file by the
developer, because sometimes they are calculated by code.
    while this very permissive, that is how it works but they are
tighted to argument passing to setup().

4/ PyPI lacks of direct information about dependencies.

    In the meantime, the DOAP project is working on a way to express
dependencies, but it is a work in progress.

5/ ideally, they should be one and only one version of a given package
in an OS-based installation

6/ packagers would like to have more compatibility information to work
out on security upgrades or version conflicts

7/ developers should be able to have more options when they define
version dependencies in their packages, things like:
      A depends on B>=1.2 and B<=2.0  but with a preference to B 1.4
or "avoid B 1.7"

   they give tips to packagers !

8/ the requires-python field is rarely used by people, so unless you
try the package, you don't know when it is a source
    distribution, if it is going to run on various python versions.

9/ unless the developer has a strong comitment to an OS, he will never
create and use a file that is located in /etc

10/ you can't possibily have a complete knowledge of the dependency
graph and possible conflicts when
     you introduce a versioned dependency in your package.

    packages at given versions are known by some people to work well
together or not in a set of versioned packages,
    Let's call this a "known good set" (KGS)

    - OS packager know and maintain the KGS for their distribution.
    - Web framework packagers does it for their application

   you don't. unless you work in a "KGS" environment. But if you want
your package to be a regular python
   package at PyPI, packagers should be able to change its
dependencies to make it fit their own KGS,
  and to build their knowledge on it.

  The developer dependencies infos is a tip and a help for a packager,
not an enforcement. see [7]

 11/ people should always upload the sdist version at PyPI,  they
don't do it always. otherwise it is a pain for packagers.

Proposals
========

this is also a synthezis of what I hurd, and some elements I have
added to respect the needs that were expressed.

0/ a lot of work can be done to clean distutils, no matter what is
decided (another PEP is built for that) cleanning, removing old-style
code, testing

1/ let's change the Python Metadata , in order to introduce a better
dependency system, by

 - officialy introduce "install requires" and "test requires" metadata in there
 - mark "requires" as deprecated

2/ Let's move part of setuptools code in distutils, to respect those changes.

3/ let's create a simple convention : the metadata should be expressed
in a python module called 'pkginfo.py'
   where each metadata is a variable.

   that can be used by setup.py and therefore by any tool that work
with it, even if it does not run
   a setup.py command.

   This is simpler, this is cleaner. you don't have to run some setup
magic to read them.
   at least some magic introduces by commands

4/ let's change PyPI to make it work with the new metadata and to
enforce a few things

Enforcements:
    - a binary distribution cannot be uploaded if a source distrbution
has not been previously provided for the version
    - the requires-python need to be present. : come on, you know what
python versions your package work with !

New features:
   - we should be able to download the metadata of a package without
downloading the package
   - PyPI should display the install and test dependencies in the UI
   - The XML-RPC should provide this new metadata as well.
    - a commenting system should allow developers and packagers to
give more infos on a package at PyPI
     to make the work easier

Open question
============

(please if you want to react on those, open another thread, with a
clean cut, otherwise it is hard to follow directly)

- what about the documentation ? can't we express it better in the
Metadata ? I think we can structurize it a bit

- what about the configuration ? can't we find a way to interact with
a config ini-like file for instance
   and don't care if it is located at /etc/package.cfg or at /Volumes/..etc ?


Regards
Tarek

-- 
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/