[Distutils] namespace_package

Zvezdan Petkovic zvezdanpetkovic at gmail.com
Fri Dec 4 00:31:23 EST 2015 

Hi Kevin,

Sorry for not being able to respond to this earlier. I have time only in the evening.


> On Dec 3, 2015, at 3:57 AM, KP <patter001 at gmail.com> wrote:
> 
> Yes, the https://github.com/pypa/pip/issues/3 <https://github.com/pypa/pip/issues/3> definitely sounds like my issue. Seems there is some concern over the nspkg.pth there as well. It seems taht the nspkg.pth is a fine idea to replace the install of the __init__.py, but that it just doesn’t work to fully extend the locations in which a namespace can reside.


I now understand what the issue is and can help perhaps with some advice below.


> Either way, there a few other workarounds posted there as well, I will check them out to see if any of them are more palatable than the one I posted here.


You need to make foo.bar-1.0.0-py2.7-nspkg.pth look like this:

import sys, types, os, pkgutil; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('foo',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('foo',types.ModuleType('foo')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p); mp[:] = m and pkgutil.extend_path(mp, 'foo') or mp

Notice the difference from the default in use of pkgutil.extend_path() that helps with editable packages.

How to make this happen automatically is explained below.


> Thanks Lele for sending that link!
> 
> -Kevin


I also want to thank Lele for pointing out the possible cause of your issues.

Now, I’d like to separate two things here:

1. Defining namespaces correctly — that should be done as we talked before in the standard way.

2. Issue with the tooling — in this case pip is the tool that causes issues.

Let’s talk about it more. These are the reasons I didn’t recognize Kevin’s issue. :-)

- The namespaces work great when defined correctly (#1 above) with zc.buildout and other tools.
  There are good open-source build systems out there that take care of namespaces properly.

- When one makes a good custom build system, it can fix the pip deficiencies too.
  For example, the build system specifically used in a company, team, …

So, my advice is to:

1. Make a custom distutils Distribution extension, for example, MyDistribution (or some better name)

    (a) make custom install_egg_info class that overrides either:

          -  two templates for pth files (for newer versions of setuptools) or
          - install_namespaces method (for older versions of setuptools)

    (b) override get_command_class method in the MyDistribution class with your own install_egg_info

2. Build this package and install it.

3. Use distclass=MyDistribution in your foo package setup.py (import it from your package).

Using this approach everything will work for every of your packages automatically.

I attached your original code from foo_test.zip and added to it a minimal mydist package + a sample __init__.py file that I think you should use in your namespace packages (if you want).

Kevin, you can try this by doing the following:

- create a virtual environment
- source bin/activate
- go to mydist package and run “python setup.py sdist”
- pip install /path/to/mydist-1.0.0.tar.gz
- go to your foo_bar package and run “python setup.py sdist”
- pip install /path/to/foo.bar-1.0.0.tar.gz
- pip install -e /path/to/foo_more
- start the python interpreter in your virtual environment
- confirm that it works

	>>> import foo.bar
	foo.bar imported
	>>> import foo.more
	foo.more imported

I know that this is not a fix for pip, but it is a generic solution that I hope helps you.

Perhaps I should make a little better example (better names, better __init__.py, …) and post it as a little project on GitHub that people can refer to.

All the best,

Zvezdan





> 
> On Thu, Dec 3, 2015 at 3:19 AM, Lele Gaifax <lele at metapensiero.it <mailto:lele at metapensiero.it>> wrote:
> Zvezdan Petkovic <zvezdanpetkovic at gmail.com <mailto:zvezdanpetkovic at gmail.com>> writes:
> 
> > Hi KP,
> >
> >> On Dec 2, 2015, at 7:00 PM, KP <patter001 at gmail.com <mailto:patter001 at gmail.com>> wrote:
> >>
> >> >It looks like you are trying to find a workaround for the problem that perhaps is not a problem at all if you use the standard approach properly.
> >>
> >> I'm definitely _trying_ to use a standard approach...That is why I am here posting. Put simply this seems like a valid use case:
> >>
> >> >pip install foo.bar
> >> >pip install -e svn+http://<path-to-foo.more>
> >>
> >> Even if both tools have the namespace_package foo, the "foo.more" will not properly import.
> >>
> >> How is this going against standard approaches?
> >
> > I don’t know.
> > Without seeing the source code for these packages (which Marius already asked for) everything is hypothetical.
> > All I know is that I’m using namespaces successfully for over a decade now and never had the need to work around them.
> >
> > Show us the code.  We may be able to help better if we can try and reproduce.
> > Otherwise, it’s hard.
> 
> It seems that KP case is either https://github.com/pypa/pip/issues/3160 <https://github.com/pypa/pip/issues/3160> or
> https://github.com/pypa/pip/issues/3 <https://github.com/pypa/pip/issues/3>, isn't it? Both come with sample code
> that demonstrate the problem.
> 
> ciao, lele.
> --
> nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
> real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
> lele at metapensiero.it <mailto:lele at metapensiero.it>  |                 -- Fortunato Depero, 1929.
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20151203/d5d80c0d/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: foo-solution.zip
Type: application/zip
Size: 4247 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20151203/d5d80c0d/attachment-0001.zip>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20151203/d5d80c0d/attachment-0003.html>

More information about the Distutils-SIG mailing list