[Distutils] PEP 390 - new format from setup.cfg

[Ian Bicking]

On Mon, Oct 12, 2009 at 4:45 AM, Tarek Ziadé <ziade.tarek at gmail.com> wrote:
> On Mon, Oct 12, 2009 at 2:29 AM, Ian Bicking <ianb at colorstudy.com> wrote:
>> The grammar in "Context-dependant sections" indicates possible EXPR
>> values.  Because the "in" operator is supported, I would assume that
>> tuples should also be allowed.
>
> "in" here is restricted to string. It was added so we could write things like:
>
> 'linux' in sys_platform  (where sys_platform can be linux2)

If you don't have tuples or <, >, etc, it seems like something like
"Python version 2.6 or higher" is hard to express.  You'd have to
enumerate 2.6, 2.7, and speculate on 2.8 and 2.9.

> I'll add a note on that.
>
>>
>> One aspect of the current setup.cfg is that it supports multiple
>> sections, for different setup.py commands or components.  This gives a
>> kind of namespacing.  I assume, but it isn't that specified, that any
>> section (not just "metadata") will be parsed the same way?
>
> I guess yes, even if I don't see a use case yet for that.

One use case for the current setup.cfg is for extensions, like
generating Sphynx documentation.  Those extensions can sometimes take
quite a lot of options, so they are best when partitioned into their
own section.  I'm also not sure whether [metadata] is intended to have
extensible variables, or a fixed set of variables.

>> Presumably setup.py will just contain an empty call to setup()?  (I
>> assume at least setup.py will be allowed for, for backward
>> compatibility, even if it is not required.)
>
> No because we might need to define extra options in setup()
> that are not part of the metadata, like what is required the for the
> sdist command
> (package_data, scripts, package, etc)

OK, so setup.cfg is for generating PKG-INFO, but installing a package
still involves running setup.py and some maybe-declarative code in
there.

>> I believe this does not include the concept of extra requirements.
>> Possibly it could as a side-effect of some variable available to the
>> language.  Like:
>>
>>  [metadata:'xmlrpc' in features]
>>  requires = lxml
>>
>> Sets and the & operator could be useful for this.
>
> How would you define/provide "features" here ?

I'm not sure.  With Setuptools the equivalent is "extras", like:

setup(name='foo',
    extras_require={'xmlrpc': ['lxml']})

Then if you do "easy_install foo[xmlrpc]" (or somehow require
foo[xmlrpc]) it will also install/require lxml.

If I was to say why this is a problem, I think it's because the
default is that there are no features for a package.  So someone
naively wants your package and installs it, but they don't get all the
features they thought the package provided (and which the package
actually *does* provide, but just doesn't have the prerequisite
libraries for -- since the package ships with the xmlrpc code, it's
just not working xmlrpc code).  There's also not a good way of seeing
what extras are provided, or what their purpose is.  So library
authors avoid the issue entirely and don't factor out requirements
into specific extras/features.  If there was a set of "default" extras
maybe it would be more workable.  I.e., "easy_install foo" installs
foo with all its default extras, and "easy_install foo[]" installs foo
without any extras (or you put the specific extras you want in []).

Anyway, the way extra requirements are serialized to disk is requires.txt, with:

  default_req1
  default_req2

  [extra]
  extra_req1
  ...

So, since the result involves multiple sections it wouldn't naturally
map to what you are proposing.

>> The way variables are handled is unclear.  Presumably all variables
>> are cumulative.  But given:
>>
>>  requires = Foo
>>  requires = Bar
>>
>> What is the value of "requires"?  A list, a value with newlines?
>> Presumably you could also do:
>>
>>  requires = Foo
>>      Bar
>>
>> Anyway, we're diverging from INI semantics at this point, so it needs
>> to be specified how it works.
>
> Right, this needs to be defined. I would be in favor of the latter, to
> stay ConfigParser
> compatible.

You still have to define how options are combined from multiple
sections, and what the resulting value is from the API.  That is, if
you have:

    [metadata]
    requires = Foo
        Bar

    [metadata:python_version == '2.4' or python_version== '2.3' or
python_version=='2.2']
    requires = BackwardCompat

Then what is the ultimate value of "requires"?  Is it
"Foo\nBar\nBackwardCompat" or ["Foo", "Bar", "BackwardCompat"], or
[Requirement.parse("Foo"), Requirement.parse('Bar"),
Requirement.parse("BackwardCompat")].

And given other variables (ones that perhaps distribute doesn't even
know about) how are they combined?

>> Is there a way to eliminate values?  Right now, for instance, if you
>> have "[build] home = /foo" in a distutils.cfg, there's no way to unset
>> that.  I'd like to see this functionality included.  Perhaps to delete
>> a specific value, but the simpler case of deleting a variable is
>> really all I want.
>
> Do you have a syntax in mind for this ?

Well, the way I added more meta-operations in Paste Deploy (which uses
ConfigParser) was to mess with the variable names.  So maybe:

    [metadata:python_version=="2.5"]
    del requires = Foo

It might have a value to delete a specific value (or line) from that
"requires" variable, or if empty it would clear that variable.  Since
ConfigParser doesn't order the variables, it would delete them from
"earlier" sections, not the current section.  This requires defining
what makes a section "earlier".  Or maybe it would delete "Foo" from
whatever the value of requires ends up being, or if you do "del
requires =" would make that section provide the only value for
requires.


Another thought: one thing distutils does but sometimes confuses me is
that it has boolean options (options which take no values, sometimes
with an option and an anti-option, e.g., --feature-foo and
--no-feature-foo) which turn into a single configuration variable with
a boolean value.  How that is supposed to work seems apropos (it's not
strictly syntax, but I don't feel like I can judge the syntax without
considering the full semantics of the configuration file, because the
semantics drive the actual use cases which should in turn be used to
judge the syntax).

-- 
Ian Bicking  |  http://blog.ianbicking.org  |  http://topplabs.org/civichacker