Package organization: where to put 'common' modules?

[fortepianissimo]

Paul Boddie wrote:
> fortepianissimo wrote:
> > Hm this doesn't work. Say I have the following directory structure:
> >
> > A
> > |--- util
> > |    |--- foo.py
> > |
> > |--- B
> >      |--- bar.py
> >
> >
> > And bar.py has this line
> >
> > from util import foo
>
> This would only work with A in sys.path/PYTHONPATH. However...
>
> > I then run
> >
> > python B/bar.py
> >
> > in directory A. Still got error
> >
> > ImportError: No module named util
>
> Yes, Python does this - it puts the directory of bar.py (B in this
> case) in sys.path, but not the directory in which you're sitting when
> you run the program from the shell (A in this case). I always get round
> this by running programs like this:
>
> PYTHONPATH=. python B/bar.py
>
> For me, this is a minor inconvenience since I know that ultimately the
> imported packages will be installed and available via site-packages.
>
> > A check of sys.path in bar.py when running from A revealed that
> > sys.path has A/B, not A.
> >
> > Am I missing something here?
>
> Well, I imagine that you want some kind of sys.path modification to
> take place, and you can certainly do this in bar.py, taking care only
> to do so in the "main program" section of the module - you don't
> usually want sys.path magic going on when just importing from the bar
> module.

Placing code modifying sys.path in main would be a problem if the
utility loading line is at the beginning of the module - error would
occur before sys.path can be modified.

It is especially true for me since I need to load these "common
functionality" not just for testing - they are necessary to implement
the functionality of the module.

Besides, adding code to manipulate sys.path in *every* module that
needs the common functionality module seems to be quite a hassle...

> However, my advice with regard to package structure and testing would
> be to put common functionality either in modules below those that need
> such functionality in the package structure, or in a separate hierarchy
> in the package structure, to use absolute references to such packages
> (A.util.foo) rather than relative ones (util.foo or stuff soon to be
> permitted in Python 2.5), and to put your test programs outside the
> package and have them use the package code like any other piece of
> software.

I need to import this common functionality not just for testing code,
so it probably doesn't matter if I separate the testing code from the
module itself.

In addition, the "common functionality" is "common" only for the system
I'm building - they are probably not that useful for general purposes.
So placing them in an absolute path (like site packages) doesn't make a
lot of sense.

> Despite the presumed benefits to the upcoming relative import support
> (less typing, perhaps), it's good to see that the existing import rules
> will be tightened up with absolute importing being enforced. Here's an
> example of import misbehaviour in today's Python:
>
> ---x
>    |---y ...print "x"
>    |---z ...import y.a # we want this to be absolute!
>
> ---y
>    |---a ...print "y"
>
> Now try and import x.z in Python 2.4! It won't even get to printing
> "y".

Thanks for pointing this out - I can see this would be a problem if not
careful.