What's the latest best practice on Python project directory layouts?

[Cameron Simpson]

On 14Jun2019 09:23, Malcolm Greene <python at bdurham.com> wrote:
>I have a collection of command line scripts that share a collection of common modules. This code collection is for internal use and will run under a single version of Python 3.6+ and a single OS. My understanding of best practice is to organize this collection of Python files into a folder structure like this:
>
># common files
>.gitignore
>readme.md
>requirements.txt
>setup.py  <--- what is the advantage of this file for internally distributed code bases?
>
># app specific package folders
>app-1
>    __init__.py (optional; if needed)
>    __main__.py
>    app-1-module-1.py
>    app-1-module-2.py
>    app-1-module-N.py
>
>app-2
>    __init__.py (optional; if needed)
>    __main__.py
>    app-2-module-1.py
>    app-2-module-2.py
>    app-2-module-N.py
>
># modules shared across multiple apps
>common
>    common-module-1.py
>    common-module-2.py
>    common-module-N.py
>
># tests - place at package level with sub-packages for each package -OR- underneath each app package?
>tests
>    app-1
>         test_app-1-module-1.py
>         test_app-1-module-2.py
>         test_app-1-module-N.py
>    app-2
>         test_app-2-module-1.py
>         test_app-2-module-2.py
>         test_app-2-module-N.py
>
># virtual env folder placed at same level as packages ???
>venv
>    <virtual-env files go here>
>
>And execute each app via the following ...
>
>python -m app-1 <optional-parameters ...>
>
>Questions
>
>1. Does the above structure sound reasonable?

Yes. Though I like to get them out of the top directory, details below.

>2. Where to place virtual env files and what to call this folder? venv, .env, etc?

I use "venv" myself.

>3. Where to put tests (pytest)? In a tests folder or under each package?

Personally, I'd do it however you would do them if the apps and the 
common modules were standalone. I use a foo_tests.py beside my foo.py 
module file myself, but a naming scheme adhering to the discoverability 
of your test running tool would also be a good choice.

>4. Use a src folder or not? If so, where to put above files relative to 
>the src folder?

Well, yeah. Always a subdirectory, I hate littering the top level.

Here is how I lay out a project, based on my current one (nonPython bits 
elided):

  project/
    bin/
    lib/python/all-modules-here
    venv/

Various points:

- I dislike using Python's "search for modules in the current 
  directory"; I would _always_ rather set $PYTHONPATH for specific 
  control.

- The lib/python depth is to accomodate lib/other-languages according to 
  the mix in the project.

Some context: I'm making a full stack app for a client at present. The 
client's code is like this:

  lib/python/clientname/appname/*.py

It happens that all the "common" code is in the appname subdirectory 
because it is self contained, but were it not it would be in

  lib/python/clientname/*.py

or possibly

  lib/python/clientname/util/*.py

(or "common" if you prefer).

One important aspect of this is that it lets me keep the client code 
away from conflicts with other library names, so the "clientname" module 
path component is important for this, _and_ it better labels the purpose 
of the module.

So the environment setup looks like this:

  project=/path/to/project  # or $(dirname "$0") if $0 is useful
  PYTHONPATH=$project/lib/python
  PATH=$project/bin:$project/venv/bin:$PATH
  export PYTHONPATH PATH

and the app is run:

  python -m clientname.appname args...

The app does relative imports for itself:

  from .util import SomeAppClass

and would import common code absolutely:

  from clientname.util import SomeCommonClass

In my current project I've actually got a small shell script which does 
the environment setup above and invokes the python app (or various other 
utility tasks like "init the database" etc). I find this approach 
generally useful.

Anyway, this might inform what choices you make. Happy to elaborate on 
specifics, though they get more personal and idiosyncratic the more fine 
grained we get.

Cheers,
Cameron Simpson <cs at cskk.id.au>