[Pythonmac-SIG] py2app and nested packages

[Ronald Oussoren]

On 25 Apr, 2012, at 2:17, Chris Barker wrote:

> On Tue, Apr 24, 2012 at 4:29 AM, Ronald Oussoren
> 
>>> example: the "pubsub" package is delivered with wxPython, so it is
>>> commonly imported thusly:
>>> 
>>> from wx.lib.pubsub import Publisher
>> 
>> wx.lib.pubsub is a bad example, it manipulates __path__ which causes problems with py2app because it cannot detect this.  I added a recipe to py2app's repository a while ago that should fix this particular problem.
> 
> 
> I thought I had the latest -- I'll go check that out.

Knowing me I probably forgot to check in the updated recipe :-(.  I also lost track on whether or not all changes are released on PyPI.

> 
> Also, there is a bit of trick if you force it to import it's dyanamic
> module, but if you've solved that one, great!

IIRC that dynamic import is completely unnecessary and probably won't work when wx is in a zip file.

> 
> 
>> That's a bug, the code that implements the packages options assumes that all mentioned packages are toplevel packages.  It should be fairly easy to fix that, although this requires some path manipulation trickery because the subpackage will be in a different location that the parent: the parent is in site-packages.zip while the child is not.
> 
> right -- do you have an idea how to solve that? I don't!

The copying part is easy, just create the right directory structure (packages = ["wx.pubsub"] -> create "wx/pubsub" in the application bundle). Ensuring that __import__ finds the subpackage is harder, I'll have to experiment to find a working solution that won't cause nausea in anyone looking at the code. 

> 
>> The option "packages" currently always stores the included package outside of site-packages.zip to ensure that data files can be loaded in the old pre-pkgresources way (that is, by opening paths relative to somepackage.__file__).
> 
> which is useful, but another thought -- maybe there could be two options:
> 
> "packages" and "full_packages" or something -- so one would simple add
> the package to the modulegraph, to catch dynamic imports, and the
> other would do what is done now -- include the whole darn thing.
> 
> One other issue with the "whole darn thing" option, is that you get
> the *.py, *.pyc, Na *.pyo files -- so it's maybe three times as big as
> it could be -- I wonder if it's worth cleaning that up.

Adding "full_packages"  for the current behavior and giving "packages" a semantics that's simular to "modules" would be useful.

> 
>>> Is there a way to "turn off" a recipe (other than deleting it from the install)?
> 
>> No, and I'd prefer to fix recipes instead of having a way to disable them.
> 
> I understand that, and generally agree -- but it may be that there is
> no one recipe that works for everyone. For example, Scipy is weird
> enough that probably the only way to be sure it'll work in all cases
> is to include the whole darn thing -- but in my case at least, I'm
> using a couple functions in scipy.special, and it works fine if that
> is all I include.
> 
> Maybe if there is a way to pass options to recipes?

I guess you're right, there has to be a way to configure recipes and that includes disabling them completely.

Matplotlib is another example where this would be useful: it should be possible to specify that your app uses a particular backend without including the entire tree.

> 
>> The (scipy) recipe should get a way to specify which dynamic imports are present, simular to the hardcoded knowledge about the stdlib in modulegraph. That way py2app can include only the bits that are really necessary.
> 
> If you give me a pointer - maybe to the similar modulegraph code,
> perhaps I could work on that.

The modulegraph code I refer to is modulegraph.find_moduels.get_implies (<https://bitbucket.org/ronaldoussoren/modulegraph/src/tip/modulegraph/find_modules.py>).  The wx recipe in the py2app tree (<https://bitbucket.org/ronaldoussoren/py2app/src/tip/py2app/recipes/wx.py>) basicly does the same thing from the outside.

> 
>>> IIRC the matplotlib recipe includes the entire package because of the data files that are in the package, including only the code doesn't work.
> 
> I've poked into this more -- it turn out that there is special-case
> code in matplotlib itself for py2exe that looks for the data-files in
> a different place when sys.frozen -- I'm adding some code for py2app
> as well -- if I get that working, I can submit a patch to MPL and the
> recipe.
> 
> Is there a way to specify data files in a recipe?

Good question, and it seems the anwer is no (looking at <https://bitbucket.org/ronaldoussoren/py2app/src/00b275dbdb71/py2app/build_app.py#cl-603>).  Adding an option for that would be easy enough though.

> 
>>> - Is there a way to apply excludes after the recipe? As far as I can
>>> tell, if the recipe includes it, excludes doesn't remove it.
> 
>> That's correct, and is a bug: user specified actions should override > automatic actions, including the recipes.
> 
> OK -- I suspect the issue is there is no way to exclude stuff that is
> going to go outside of the zip file -- the full packages. It looks
> like what it does is remove it from modulegraph only.
> 
> Thanks for your work (and ideas) on this.


Speaking of ideas....   I'm pretty sure that py2app is not compatible with packaging/distutils2, and I don't want to maintain a codebase that works with all flavor of distutils (the current distutils, setuptools, distribute, distutils2 and packaging).   

In the medium term it would be better to make py2app  a standalone application instead of a distutils extension, with clear integration points for the packaging tools (such hooks for fetching dependencies and building extensions).  That will have to wait for the next release of pyobjc though, that process is moving  forward way too slow even without spending time on a py2app redesign.

Ronald

> 
> -Chris
> 
> 
> 
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> 
> Chris.Barker at noaa.gov
> _______________________________________________
> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
> http://mail.python.org/mailman/listinfo/pythonmac-sig
> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4788 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pythonmac-sig/attachments/20120425/db111853/attachment.bin>