[Distutils] Where should I put tests when packaging python modules?

[Donald Stufft]

On October 6, 2015 at 6:33:10 AM, David Cournapeau (cournape at gmail.com) wrote:
> On Tue, Oct 6, 2015 at 11:21 AM, Donald Stufft wrote:
>  
> > On October 6, 2015 at 6:18:32 AM, David Cournapeau (cournape at gmail.com)
> > wrote:
> > > > The significant number is not so relevant if you buy the argument
> > > that it is useful to downstream packagers: it may be a few users,
> > > but those are crucial.
> > >
> > > I also forgot to mention that the ability to test something without
> > > building is crucial when you want to distribute binaries.
> >
> > Is it actually useful to them? None of the Linux downstreams I know of have
> > ever mentioned a preference for it. As far as I know, the only preference
> > they've ever expressed to me is that the tests are included in the sdist.
> >
>  
> It is at least useful to me, and I am packaging quite a few binaries.
>  
>  
> >
> > FreeBSD relies on ``python setup.py test`` as it's preferred test
> > invocation,
> > so it apparently doesn't find it useful either.
> >
>  
> I would like to hear their rationale before guessing. It is hard for me to
> imagine they would not rather test the binaries rather than from sources.
> Something as simple as making sure you have not forgotten runtime
> dependencies becomes much easier this way.
>  

I'm able to test runtime dependencies just fine without needing to put my tests
inside of the package. To the extent anyone can actually test runtime
dependencies in a test framework without actually depending on your test tools
at runtime [1].

I really don't think either way is "better" to be honest. I think any attempts
that one way is better than the other relies on nebulous edge cases that don't
really happen much in reality. You'll tell me that you've found it useful to
run tests against an installed distribution without having to fetch the
original tarball, I'll tell you that I've found it useful to run a newer
version of the test suite against an older installed copy of the library.

In practice, I think that you'll be able to manage finding the tarball and
running those tests against your installed distribution while I'd be able to
manage to copy just the tests I care about running out of the test suite and
run them manually. IOW, I don't think it really matters and people should just
do whatever they want.

[1] By this I mean, you'll never detect if you're missing a runtime dependency
    on something that your test dependencies include as part of their
    dependencies, since installing your test tools to run your tests would then
    trigger this missing dependency to be installed.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA