[Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Donald Stufft]

On October 2, 2015 at 10:27:36 PM, Nathaniel Smith (njs at pobox.com) wrote:
> 
> So I'm extremely reluctant to give up on standardizing how to handle
> VCS checkouts. And if we're going to have a standard for that, then
> would sure be nice if we could share the work between this standard
> and the one for sdists, given how similar they are.

Mentioned in another email, but I don't think we should give up on the VCS
handling, I just think we should defer it to another PEP. I think this PEP is
suffering from trying to use the same mechanism for VCS and sdists when they
are different things and have different considerations. The Python packaging
toolchain has a long history of suffering because it tried to reuse the same
thing across multiple "phases" of the packaging life cycle. The problem with
that is different phases have different needs, and when you're trying to use
the same thing across multiple you suffer from something that is a sort of
"Jack of All Trades, Master of None" kind of thing.

> 
> What prevents it in the current draft is that there's no way for
> foobar to say any such thing :-). If you ask for Django, then the only
> sdist it will look at is the one in the Django segment. This is an
> intentionally limited solution, based on the intuition that multiple
> wheels from a single sdist will tend to be a relatively rare case,
> when they do occur then there will generally be one "main" wheel that
> people will want to depend on, and that people should be uploading
> wheels anyway rather than relying on sdists.

There *must* be a 1:1 mapping in name/version between sdist and wheels. This
assumption is baked into basically every layer of the toolchain.

> 
> I feel like this idea of "source wheels" makes some sense if we want
> something that looks like a wheel, but without the ABI compatibility
> issues of wheels. I'm uncertain how well it can be made to work in
> practice, or how urgent it is once we have a 95% solution in place for
> linux wheels, but it's certainly an interesting idea. To me it feels
> rather different from a traditional sdist, and obviously there's still
> the problem of having a standard way to build from a VCS checkout.

I feel like you have some sort of "a sdist is jsut a tarball of a VCS"
mentality and I don't think that idea of a sdist is generally useful.

> > I don't think there's ever going to be a world where pip depends on virtualenv
> > or pyvenv.
> 
> Huh, really? Can you elaborate on why not? The standard doesn't have
> to require the use of clean build environments (I was thinking of the
> text in the standard as applying with the "as if rule" -- a valid
> sdist is one that can be built the way described, if you have some way
> that will work to build such sdists then your way is valid too). But
> using clean environments by default is really the only way that we're
> going to get a world where most packages have accurate build
> requirements.
> 

We'll most likely use a clean environment by explicitly emptying out the
sys.path or something similar. We won't depend on virtual environments because
it is a layering violation.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA