Python packages - problems, pitfalls.

[Prabhu Ramachandran]

>>>>> "BH" == Bernhard Herzog <bh at intevation.de> writes:

    >> pkg_root/ __init__.py app.py # app is the application that is
    >> not part of the package

    BH> If it's not part of the package, why is it in pkg_root? Put it
    BH> somewhere outside of pkg_root, make sure pkg_root is in a
    BH> directory on Python's path and everything should be fine.

Easier said than done.  If I wanted to do this I'd have to re-nest
every blessed file in CVS.  This is something I want to avoid.

    >> Also, irrespective of whether people have run into such
    >> problems or not, don't you think it makes sense for modules to
    >> be searched the way I had suggested earlier??  Or is there some
    >> serious issue with this.

    BH> The name search rules for packages are more or less the same
    BH> as for local variables in functions, at least when packages
    BH> were introduced.  With nested scopes that similarity will be
    BH> gone, though.

    BH> Anyway, you may want to search a bit through Python's
    BH> history. Python 1.3 introduced the first stab a packages of
    BH> modules with a mechanism called "ni" which had a much more
    BH> elaborate name resolution scheme. One part of it was
    BH> "expanding search" which would automatically look in a
    BH> package's parent package and so forth and it had a way of
    BH> referring to a module's package (by the variable __) and to
    BH> package's parent (__.__).

Yes, unfortunately, this way of referring to parent packages was
removed!   From: http://python.org/doc/essays/packages.html

   "This feature was dropped because of its awkwardness; since most
   packages will have a relative shallow substructure, this is no big
   loss.)"

If you ask me that is a bad assumption.  "most packages will have a
relative shallow substructure" - where was that pulled out from??

    BH> The ni module is still there, but it's deprecated since the
    BH> new builtin package support was added in 1.5 and therefore now
    BH> lives in lib-old.

    BH> Guido's essay about the builtin package support has some
    BH> comments on the reasons behind the differences between ni and
    BH> the current scheme: http://python.org/doc/essays/packages.html

Yes I read that long before I posted to this list.  Sorry if I may
sound rude, but please understand that the problem is real and not
theoretical.  And I did do my homework and have my reasons and I'm not
the only one looking for a solution. :) Maybe I was not clear.

Let me try to be a little clearer.  This is what Python does now:

  (1) Check if module is in same directory (sibling module).

  (2) If not found - look at the global level for modules in any
  directories found in sys.path.

There is *absolutely* no in between here.  Its either local or global.
What I suggest.

  (1) Check if module is available in the same directory, if available
  - use it.

  (2) If not found walk up to parent dir.  check there, if not there
  go up until out of package.

  (3) If not found in (2) look at sys.path for module.

Here is pseudo code:

def load_module(module):
  # do stufff

def check_module_in_dir(module, dir):
  # do stuff

def find_parent_package_dir():
  # do stuff

def walk_package(module, dir):
  if check_module_in_dir(module, dir):
     return load_module(module)
  else:
     par = find_parent_package_dir()
     if par:
        if walk_package(module, par)
     else:
        # search in sys.path and try to load module from there.

dir = os.path.dirname(__file__)
walk_package(module, dir)

Now doesn't this make more sense than what is being done currently?
Forget about my silly examples. :)

Anyway, looks like I'm gonna have to look at knee.py and imputil and
figure out a way to implement this.

thanks,
prabhu

-- 
Prabhu Ramachandran			  MayaVi Data Visualizer
http://www.aero.iitm.ernet.in/~prabhu     http://mayavi.sf.net

Where there's no emotion, there's no motive for violence.
		-- Spock, "Dagger of the Mind", stardate 2715.1