[Distutils] __init__.py files missing from my eggs

[Richard Cooper]

From: Phillip J. Eby [mailto:pje at telecommunity.com] 
> At 04:51 PM 11/11/2005 +0000, Richard Cooper wrote:
> >In the plugin's setup.py I'm doing the following for each plugin:
> >
> >setup(
> >     name = pluginName,
> >     packages = ['My.Project.plugin.'+pluginName],
> >     namespace_packages = ['My.Project.plugin'],
> >     package_dir = {'':'../../..'},
> >     package_data = ...
> >)
> I would guess that your problem is that you need to do:
> 
>      packages = [
>          'My', 'My.Project', 'My.Project.plugin',
>          'MyProject.plugin.'+pluginName
>      ]
> 
> Because it is otherwise not including those packages (and 
> their __init__.py files).

That pulls in too much. There are files in those packages other than the
__init__ files which I don't want included in the plugin. 

> On an unrelated note, I would caution you that your 
> unconventional project layout is likely to lead to other 
> problems, such as an inability to create usable source 
> distributions.  Neither setuptools nor the distutils are 
> designed to work with setup.py in any directory other than 
> the distribution root of the project.

Yeah. I thought that might be part of the problem. I've had no other
problems so far though. 

> What you probably want is a layout like this:
> 
> app/
>     setup.py
>     My/
>        Project/
>                app
> Plugin1/
>     setup.py
>     My/
>        Project/
>                plugin1/
> 
> etc.

My knee-jerk reaction is 'Ewww!' Is this what PEAK (or ant other large
setuptools based project) does? How do you handle My.Project.utils (for
example) and the half dozen or so other packages at different levels of
the hierarchy which are used in multiple "products"?

> Then, each project can have a source distribution built, be 
> managed by easy_install, etc.  Each project can also use 
> 'find_packages()' to get the list of packages to install, 
> without having to manually list them or the package directory 
> names.

My needs are a bit specialised, this is a commercial app so source
distribution, easy_install, etc will never be on the cards. Distribution
is strictly "frozen" products (py2exe and eggs) only.

> If you are working on these projects at the same 
> time, you need only run "setup.py develop" in each one to set 
> up a sys.path structure that merges all the packages into one.

We would probably end up with 20 or so "projects" if we did things that
way as opposed to the single code tree we have now. Given that,
"setup.py develop"*20 seems like more work than our current approach
which is "add the root of the code tree to PYTHONPATH" 

> Note, by the way, that you don't need namespace packages to 
> have plugins for an application.  Plugins can be in any 
> package structure you like, as eggs' "entry points" system 
> can be used by plugins to advertise the functions they 
> provide, and the importing can be done automatically for you.

Yeah I know. The entry point system is the main reason I started playing
with setuptools. However I still wanted the plugins we produce to all
live in the My.project.plugins package.



It looks like my problem is that I'm mingling separate products in the
same code tree and setuptools/distutils doesn't like that. Fair enough.
I don't think I'm quite ready to drink the "one tree per product"
kool-aid just yet. Firstly, it seems weird to me to split up the source
like that and secondly, my colleagues would probably kill me ;-)

So what I will probably end up doing is hacking up setuptools to insert
the __init__.py files I need. Which begs the question - Is this:

a) A dirty, dirty hack I should never speak of again OR
b) Not a bad idea and potentially useful to other people.

I have a feeling I know the answer to that ;-)

Thanks for the help.

Richard