[Distutils] Environment marker expression types

[Nick Coghlan]

On Fri, Apr 26, 2013 at 10:12 AM, PJ Eby <pje at telecommunity.com> wrote:
> I was just fiddling with an experimental environment marker
> implementation for setuptools, and ran across a bit of a quirk in the
> spec.  It says that the value of 'extra' is 'None', but that
> comparisons can only be done on strings.
>
> This leads to two problems: first, the syntax doesn't have a way to
> spell 'None', so you can't actually compare it to something.

It's not explained well in the current PEP, but there's an implied
"extra is None" if extra isn't mentioned in the environment marker (or
there's no environment marker at all)

> Second,
> if you compare it to a string using the equality operators, IIUC
> you'll get a TypeError under Python 3.

Note that == and != don't emit TypeError with non-comparable types -
if both sides return NotImplemented from the corresponding magic
methods, then the comparison is just False for == and True for !=.
It's only the ordering comparisons that now emit TypeError rather than
attempting to guess an appropriate answer.

> Should it actually be defaulting 'extra' to an empty string?  That
> would actually make a lot more sense, and avoid the issue of
> implementing non-string comparisons, None literals, etc.

Once you account for the "extras not mentioned implies extras is
None", the current system is at least self-consistent.

The main advantage of combining the systems is that it allows the
extras to *also* participate in the environment marker system (for
example, only adding certain dependencies if you're on Windows *and*

> (Doing extras in this way actually has another problem, btw, which is
> that it's insane to do a != comparison on an 'extra'.  And there are
> probably other insane operations on extras, because unlike everything
> else, the extra would need to be dynamically evaluated.  I think it
> would probably be an improvement to remove extras from the environment
> marker system altogether and just have a mapping of extras instead,
> ala setuptools.  But that can be treated as a separate issue from the
> 'None' problem.)

While working on the latest PEP 426 update, I've actually been
pondering the problem of how to handle the environment marker system
in general. Specifically, I *don't* really want anyone to need to
check any environment markers in the metadata passed to the
post-install hook, or even in the metadata recorded in the
installation database. The installer should be resolving all of those
at install time. However, then you get the interesting question of
what happens if you share an installation database across
heterogeneous machines via a network file system, implying that either
these things *have* to be evaluated at run time, solely in order to
accommodate that niche use case, or else we need to constrain that use
case by saying that it is up to the people setting it up to ensure
that the environments are *sufficiently* similar for things to keep
working.

Anyway, I have a draft design that I plan to use for the initial
rewrite, but it's one of the areas where I'll be most interested in
robust feedback:

1. All unconditional metadata goes in the top level fields
2. A new "conditional" top level field is added
3. The new field points to a mapping that uses environment markers as keys
4. Assuming that the only fields that participate in the environment
marker system are represented as lists (as is currently the case),
then the values in the conditional mapping will themselves be mappings
from field names to lists that describe the additional entries to be
appended to create the full list.
5. The installer evaluates the conditional metadata as appropriate
before writing the metadata out to the installation database and
invoking the post-install hook. Post-installation, the top level
metadata includes both the unconditional metadata *and* any
conditional metadata that applies to the current system.

For example, if the project metadata looked like this:

    "requires": ["projectA", "projectB"]
    "conditional": {
        "sys.platform == 'win32'": {
            "requires": ["pywin32 (>1.0)"]
        }
    }

Then the installed metadata would be unchanged on Linux or Mac OS X,
but would look like this on Windows:

    "requires": ["projectA", "projectB", "pywin32 (>1.0)"]
    "conditional": {
        "sys.platform == 'win32'": {
            "requires": ["pywin32 (>1.0)"]
        }
    }

This *does* mean I am inclined to split out the "extras" system as its
own top-level field that recapitulates this structure in miniature:
the extras fields would always remain separate from the top-level
ones, but could also include a set of conditional entries that were
evaluated and potentially added at install time:

    "extras" : {
        "test" : {
            "requires": ["projectA", "projectB"]
            "conditional": {
                "sys.platform == 'win32'": {
                    "requires": ["pywin32 (>1.0)"]
                }
            }
        }
    }

Anyway, this kind of problem is part of the reason the rewrite is
taking a while. Structured metadata provides the opportunity to make
things cleaner and more consistent compared to the relatively ad hoc
approaches in metadata 1.2, but it's still an ongoing challenge to
avoid runaway complexity.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia