Module Structure/Import Design Problem

[Gabriel Genellina]

En Fri, 21 Nov 2008 17:39:14 -0200, Stef Mientki <stef.mientki at gmail.com>
escribió:
> Gabriel Genellina wrote:
>> En Thu, 20 Nov 2008 17:36:11 -0200, Stef Mientki  
>> <stef.mientki at gmail.com> escribió:
>>
>>>>> I'm not an expert, I even don't fully understand your problem,
>>>>> but having struggled with imports in the past,
>>>>> I've a solution now, which seems to work quit well.
>>> http://mientki.ruhosting.nl/data_www/pylab_works/pw_importing.html
>>
>> May I reiterate my criticism to your "solution" posted last week?
> You're welcome,
> if there are better (even more important simpler) solutions, I'm in.
>> I don't think extending sys.path to include every directory under your  
>> project is a good idea. Basically, you're flattening the directory  
>> layout, removing any structure, like it was before Python 1.5 added  
>> package support.
>> It is like dumping all your modules in a single directory, with no  
>> hierarchy and lots of name conflicts.
> Well I started at 2.4 and now arrived at 2.5, so I don't know what it  
> was at 1.5.
> And I'm probably going to stay at 2.5 for a long time, upgrading is one  
> of the few (maybe the only) disadvantages of Python ;-)
> For Python, I guess you're right, all files are in one flat directory,  
> ...
> ... but it's just 1 program, so what's the objection ?

(note that it's *you* who wrote a single program.)

> For the program itself, which is highly dynamic, it can find sets of  
> python-files in certain subpaths,
> so that's quit ordered I think ?

(again, you're talking of *your* program.)
On the contrary, you're destroying the directory hierarchy on disk, making
modules live in a flat namespace, like a big bag without structure.

>> Worse: because your proposal makes the same file reachable under many  
>> different names, the *same* source module will generate *different*  
>> module objects stored as *different* entries in sys.modules. Globals  
>> don't work anymore, subclasses aren't subclasses... lots of problems.
> Could you explain this a little more ...
> ... I'm just adding every subpath to the Pythonpath,
> so in my ignorant believe, every module is imported the same way, but I  
> might be wrong.

The import mechanism in Python is not perfect. It uses a mapping "module
name" -> "module object" (sys.modules). If a module is not found by name
in sys.modules, it is loaded from disk. If you have two different ways to
find the *same* module, Python won't notice, and will create two different
copies of it.
So it is important to have only *one* way to locate a module; that is,
entries in sys.path should not overlap (uhm, well, not exactly, but it's
hard to explain right)

Consider this structure:

main/
        +--- PkgA/
            +--- PkgB/
                     foo.py

Without using your recipe, only main is is sys.path. The normal way to
reach module "foo" from "main", is to import it as PkgA.PkgB.foo
After using your recipe, sys.path will contain .../main/PkgA and
..../main/PkgA/PkgB. You may import foo using: PkgA.PkgB.foo, PkbB.foo, and
even foo directly.
All of these different ways to reach the same file create different module
objects and different entries in sys.modules. If you initialize something
when the module is loaded, by example, you'll run the initialization three
times.

There is a well known case: the main module. Even if the file is named
"app.py", when the program runs, it is known as "__main__". If another
module executes "import app" it will get a different module object. (I
think this case is listed in the FAQ.) Under your proposal, this aliasing
could happen with virtually any module.

> " Globals not working" , a good program doesn't have any globals ( said  
> a non-programmer ;-)

Uh? Classes are usually globals. Modules too. Global state is saved in
global variables...
A good programmer knows when to use global state - like the registry of
classes used by the factory method that someone suggested previously in
this thread.
In the 'foo' example above, there will be three copies of the module's
global variables - each with different values and evolving separately.

> Subclasses aren't subclasses ? ( I always derive subclasses in the  
> module itself, or in the same directory as the parentclass)

(again, *you* do it in that simple way.)
If you define a class X inside foo, you'll have three different classes,
all named X. A subclass Y of X is *not* a subclass of the "other" X's,
depending on how you import the foo module.

>> Relative imports (PEP328 [1]) were introduced -among other things- as  
>> an attempt to avoid such problems. You're going in the opposite  
>> direction. Please stop doing that - or at least keep us informed of  
>> where you work!
>>
>> [1] http://www.python.org/dev/peps/pep-0328
>>
> Sorry I don't understand all that pep-talk (I'm not a programmer ;-)
> I read it starts with " For the second problem, it is proposed that all  
> import statements be absolute by default (searching sys.path only) with  
> special syntax (leading dots) for accessing package-relative imports."
> I think that's exactly what I'm doing: absolute imports.

What you're doing is like using absolute paths in a filesystem, but with
every possible subdirectory symlinked to root. So /d is actually /a/b/c/d
-- and without additional effort (e.g. resolve symlinks) you can't
determine that they're the same.
Python doesn't do the equivalent to "resolve symlinks" in the module
hierarchy, and assumes that different names point to different things.
(Doing otherwise would be slow, and only justified when dealing with a
somewhat crazy setup like yours. But at least it should be clearly stated
in the documentation, which is rather scarce).

> Please explain in simple language what I'm all doing wrong,
> or how I get a better solution.

(A better solution for what? You didn't say what was your problem, and
your "solution" is not an answer to the OP question)
Note that what you are doing may work for *your* project with *your*
constraints and *your* way of doing things, but don't publish it as a
general recipe please.

-- 
Gabriel Genellina