[Distutils] Where should I put tests when packaging python modules?

[Erik Bray]

On Tue, Oct 6, 2015 at 12:04 PM, Ionel Cristian Mărieș
<contact at ionelmc.ro> wrote:
>
> On Tue, Oct 6, 2015 at 6:33 PM, Erik Bray <erik.m.bray at gmail.com> wrote:
>>
>> Okay, though, so maybe if there is nothing to offer here but anecdata
>> then maybe we should stop acting like there's "one right way here".  I
>> have projects that install their test suite and test dependencies
>> because it is frequently useful to ask users to run a self-test (and
>> users themselves want to be able to do it for a variety of reasons).
>>
>> There are other projects where it doesn't make sense, and those don't
>> have to install the tests (I still think in those cases that the tests
>> should live in the package instead of outside it, but simply not
>> installed).
>
>
> To be honest, I like the idea of providing the tests in the installed
> package. Using the test suite as a debugging/diagnostic tool is obviously
> desirable. It's just that it's unpractical for general use. Another anecdote
> I know, but humour these two concerns:

Those are all fair questions, and I can't say I have great answers for
all of them.  But that's a fair point that if installable tests *are*
recommended as a practice then they should be addressed.

Before continuing on there are really three options being discussed here:

1) Tests totally outside the package.
2) Tests in the package but not installed (only in source dists / VCS checkout)
3) Tests in the package and some or all installed.

Skimming back through the rest of the thread I don't see too much
support for 1).  The only argument against it is the need for
specifying dependencies, etc., which really only impacts developers so
long as the tests aren't *installed*, I think.  But there's also the
question of what kinds of tests we're talking about.  I think unit
tests should live in the <packagename>.tests for a library.  Other
kinds of tests I don't have a strong opinion about.

So in any case if we did recommend putting tests in a subpackage we
could still do so without making a decree as to whether or not they
should be installed.  Maybe by default suggest that they not be
installed, and only have them installable if the developer has a plan
for addressing the questions below.  I think each of these questions
may have different answers depending on the needs of different
projects.

> * How to deal with dependencies?

What dependencies exactly?  Dependencies just needed for running the
tests (as opposed to running the code being tested?  Some of my code
has optional dependencies, and the tests for that code are only run if
those optional dependencies are satisfied, but that's not what we're
talking about right?)

> ** Should we use extras? Installing test deps has the disadvantage of
> version conflicts. Unless we make some sort of "virtualenv-for-tests" tool?

I'm rather fond of mktmpenv in virtualenv-wrapper, but I admit that
may be out of scope here...

Not to point to using easy_install as best practice, but something
resembling the way setup_requires could actually work
here--dependencies are downloaded into a temp dir and installed there
as eggs; added to sys.path.  No need for something as heavy-weight as
a virtualenv.  This still has some potential for VersionConflict
errors if not handled carefully though.  But I might try out something
like this and see where I get with it, because I think it would be
useful.

> ** Should we vendor the deps? But isn't that maybe too hard to do (or plain
> wrong for other reasons)?
> ** Should we avoid having deps? That can be too limiting in some situations,
> unittest is very bare compared to pytest or nose.

py.test comes with a command to generate a bundled copy of py.test for
vendoring: https://pytest.org/latest/goodpractises.html#deprecated-create-a-pytest-standalone-script
 (for reasons I've lost track of it's apparently deprecated now
though?)  I've used this with quite a bit of success to support
installed tests.  It allows me to ship a version of py.test that works
with my tests along with the package, and there is no issue with
version conflicts or anything.  In principle the same trick could be
used to bundle other dependencies.

In practice it's a little more complicated because downstream
packagers don't like this, but it's easy to remove the bundled py.test
and use the one provided by the system instead (as in on Debian).  In
that case we do have to work with the downstream packagers to make
sure all the tests are working for them.  I don't think this will be
be much of an issue for the average pure-Python package.

> * What sort of tests should be included? Integration tests are special case,
> if they need external services, temporary storage or what not. Would we then
> need to have clear separation for different types of tests?

I try to separate out tests like this and disable them by default.  I
think the answer to that question is going to vary a lot on a case by
case basis, but it is a good question to pose to anyone considering
having installed tests.

> I'm not saying tests inside package is bad at all. But to flaunt it around
> as a "best practice" requires at least some recommendations for the two
> concerns illustrated above, and probably better tooling.

Fair point, I agree!

Best,
Erik