[Distutils] Using build products instead of the source package during development

[Dimitri Tcaciuc]

Hello everyone,

I have a project with a python package and a compiled component inside
of it. Current directory layout is:

    <project>
      foo/
      foo/__init__.py
      foo/...

      src/
      src/c_foo.c

      tests/
      tests/test_foo.py

      setup.py

When the project is built, distutils create a `build/lib` directory
which I either add to `PYTHONPATH` or install into a virtual
environment. Resulting structure is the following:

    <project>
      build/lib
      build/lib/foo/__init__.py
      build/lib/foo/c_foo.so


The issue with that is that if I start a python interpreter session
from the project root, run tests from the project root etc., it picks
up the source tree instead of the built tree. This is a problem
because in that case the compiled extensions are not found since they
reside with build products.

I found several existing solutions to this in use:

1. Place python sources under a separate directory, eg. `lib/foo`,
`modules/foo` etc. Downside to that is an extra directory level to all
of the source files and inconsistency with projects that don't have
compiled extensions and therefore have their python packages in the
root directory.

2. Keep the packages in the root, which means an inconvenience having
to `chdir` out of the project root (eg. into tests/ directory) so that
python interpreter does not see the source packages (via build script
or manually).

3. Keep the packages in the root under a different name (eg. `foo-
module` or `foo-lib`) with an appropriate `package_dir={'foo':'lib-
foo'}` line in `setup.py`. This is a variation of pt. 1 without an
extra level of directory hierarchy, which is pretty much the same
thing, I suppose.

4. Keep the packages in the root and use `setup.py build_ext --
inplace`, however this contaminates the source tree. In addition, this
does not go well if I use another


Either case introduces an overhead vs. a plain python project where
one can modify/run code right out of the source tree. I'd very much
like to hear if there's a preferred way of dealing with this and/or
which particular method you use for you projects.


Dimitri.