How to fix my imports/file structure

[Steven D'Aprano]

On Thursday 21 January 2016 12:26, Travis Griggs wrote:

> I wrote a simple set of python3 files for emulating a small set of mongodb
> features on a 32 bit platform. I fired up PyCharm and put together a
> directory that looked like:
> 
> minu/
>     client.py
>     database.py
>     collection.py
>     test_client.py
>     test_database.py
>     test_client.py

This will only work so long as you cd into the minu directory first.

To fix that, you can either:

(1) Add minu to your path. You can put this at the start of your script:

import sys
if "/path/to/minu" not in sys.path:
    sys.path.append("/path/to/minu")


(where "/path/to/minu" is the absolute path to the actual directory).

You need this to occur before you start importing from minu.

Or you can adjust the path using an environment variable:

export PYTHONPATH="/path/to/minu"

before you launch Python. (If you're using Linux or Unix, you could put that 
in your .bashrc, or equivalent.)

Or you can create a .pth file that points to your minu directory. 

(I haven't tried this, I might have some of the details wrong.)

In your Python site-packages directory (usually found somewhere like 
/usr/local/lib/python3.4/site-packages/) drop a file called "minu.pth" 
containing a single line of text:

/absolute/path/to/minu

then that path will be automatically added to your python path and you'll be 
able to import any .py file in minu from anywhere.

https://docs.python.org/3/library/site.html



(2) Alternatively, turn minu into a package, rather than a directory of 
unrelated modules.

- Create a file __init__.py and put it in the minu directory.

- Place the minu directory somewhere in your Python path. Or use a .pth 
file, as above.


Now minu is a *package*, and the modules "client.py", "collection.py" etc. 
are assumed to collaborate rather than be a random collection of arbitrary 
modules. From *outside* of minu, you can import submodules of the package:

import minu.collection
from minu.client import Spam

and from submodules *inside* minu, you can either use the same absolute 
imports as above, or you can use relative imports:

# inside minu.client
from .collection import Collection


Google for "relative and absolute imports" for more info.


> My imports are simple. For example, client.py has the following at the
> top:
> 
>     from collection import Collection
> 
> Basically, client has a Client class, collection has a Collection class,
> and database has a Database class. Not too tough.

Python isn't Java, you don't have to force each class to live in its own 
file. Sounds like you might simplify the whole job by putting everything in 
a single minu.py file.



> As long as I cd into the minu directory, I can fire up a python3
> interpreter and do things like:
> 
>     >>> from client import Client
>     >>> c = Client(pathstring='something’)
> 
> And everything just works. I can run the test_files as well, which use the
> same sorts of imports.
> 
> I'd like to modularize this, so I can use it another project by just
> dropping the minu directory alongside my application's .py files and just
> have everything work. E.g.
> 
>     SomeDirectory/
>         application.py
>         minu/
>             …
> 
> and application.py does something like:
> 
>     from minu.client import Client

Sounds like you want minu to be a package. See (2) above.


> When I try this though, and am running python3 from another directory, the
> local imports don't work. I placed an empty init.py in the minu directory.
> That made it so I could import minu. But the others broke. I tried using
> things like

You still need minu to be somewhere that the main script can see, that is, 
in one of the locations listed in sys.path, but that's not hard to set up. 
If you run:

    python3 SomeDirectory/application.py

SomeDirectory will be automatically added to the path and in theory it 
should all Just Work.

If not, start by printing sys.path and see what you have.


>     from .collection import Collection #added the dot
> 
> but then I can't run things in the original directory anymore, like I
> could before. What is the simple/right way to do this?

I would say that the right solution is to modify the imports in the minu 
submodules to use package absolute and/or relative import syntax. In theory 
you can probably get it to work by adding sufficient directories to the 
path, but that gets ugly and messy quickly.

Packages were invented to solve your problem, but the submodules do have to 
be written as if they are inside a package, and not as stand alone modules.



-- 
Steve