[Pythonmac-SIG] py2app unable to find cprocessors.so

Paul Wiseman poalman at gmail.com
Fri Sep 14 01:13:29 CEST 2012 

On 13 September 2012 20:16, Chris Barker <chris.barker at noaa.gov> wrote:

> On Thu, Sep 13, 2012 at 10:36 AM, Michael McCracken
> <michael.mccracken at gmail.com> wrote:
> > FWIW, here's how I do something similar now, to avoid having many
> > copies of the Qt libraries.
>
> cool! Thanks for the description
>
> > There is one master app, and several sub-apps.
> >
> > * call setup() for each of the apps, generating full separate apps
> > with copies of the Qt libraries and other stuff
> > -- each call has a unique directory sent in the setup options
> > "bdist_base" and "dist_dir". This avoids sharing build state, as
> > mentioned earlier in the thread. This seems to work fine, calling
> > setup() multiple times in the same script.
>
> good to know -- I think the OP was right, the conflict over the
> sharing build dirs -- I didn't know you could override that.
>
> > * inside main app Contents/Resources, create empty sub.app/ directory
> > * for each sub app:
> > ** copy sub.app/Contents/MacOS into
> > main.app/Contents/Resources/sub.app/Contents/MacOS
> > ** copy sub.app/Contents/Info.plist into
> > main.app/Contents/Resources/sub.app/Contents/Info.plist
> > ** copy everything in sub.app/Contents/Resources/ that isn't include
> > or lib into main.app/Contents/Resources/sub.app/Contents/Resources/
> > ** create symlinks for sub.app/Contents/Resources/include ->
> > main.app/Contents/Resources/include (do the same with Resources/lib)
>
> OK -- here is where I"m confused -- with a sylink, you can do either
> relative or absoute path, yes? absolute wouldn't work, as who knows
> where the user will install it -- but relative could, as long as the
> users keeps them all together -- is that how you handle it?
>
> Alternatively I suppose you could require that main.app be installed
> in /Library/Frameworks or something, and then symlink to the absolute
> path.
>
> I actually kind of like that idea, 'cause it would keep the apps a bit
> more distinct, -- I have a base I want to use with multiple apps but
> don't necessarily want folks to have to install everything at once to
> use one.
>

You could create the multiple apps at run time when running the main app,
that's not too far away from what I'm doing at the moment.  I think it's
mainly just a few xml plist files that cant be symlinked because they wont
be the same as the main app. A slightly easier solution would be to zip the
apps up and drop them in place and relatively symlink the needed parts back
to the main app.


>
> > 3. merge the contents of
> > sub.app/Contents/Resources/lib/python*/site-packages.zip into the main
> > app's copy using zipfile
> > 4. merge the contents of the
> > sub.app/Contents/Resources/lib/python*/lib-dynload directory into the
> > main app's directory
>
>
> But anyway, this is a nice model for what py2app could so for us.
>
> Thanks,
>   -Chris
>
> > NOTE: this will ignore any packages in
> > sub.app/Contents/Resources/lib/python*/ that aren't in the
> > site-packages.zip.
> > That's fine for me, since I have the same "packages" option for all of
> the apps.
> >
> > -mike
> >
> >
> > On Thu, Sep 13, 2012 at 1:53 AM, Ronald Oussoren <ronaldoussoren at mac.com>
> wrote:
> >>
> >> On 13 Sep, 2012, at 10:47, Paul Wiseman <poalman at gmail.com> wrote:
> >>
> >> On 13 September 2012 07:18, Ronald Oussoren <ronaldoussoren at mac.com>
> wrote:
> >>>
> >>>
> >>> On 10 Sep, 2012, at 16:37, Paul Wiseman <poalman at gmail.com> wrote:
> >>>
> >>> Ah,
> >>>
> >>> I've found out how to recreate the error
> >>>
> >>> If I create a main.py with nothing but 'import sqlalchemy'
> >>>
> >>> then use the following setup.py:
> >>>
> >>> from setuptools import setup
> >>>
> >>> setup(
> >>>     version="1",
> >>>     name="TestApp1",
> >>>     app=["main.py"],
> >>>     setup_requires=["py2app"]
> >>> )
> >>>
> >>> setup(
> >>>     version="1",
> >>>     name="TestApp2",
> >>>     app=["main.py"],
> >>>     setup_requires=["py2app"]
> >>> )
> >>>
> >>> If it doesn't produce the error it's probably because of this: "The
> >>> "cprocessors" module in SQLAlchemy is written in C and compiles
> >>> conditionally, based on if the current platform supports compiling it.
>   If
> >>> not present, SQLAlchemy continues to work perfectly well as the hooks
> which
> >>> attempt to import it will fall back to pure-Python functions instead."
> So
> >>> you may have a cprocessors.py which I dont think you'd get the
> problem, only
> >>> if it compiled the .so when sqlalchemy installed.
> >>>
> >>>
> >>> I had the cprocessors extension in my build (that is, py2app mentioned
> in
> >>> copied the extension)
> >>>
> >>>
> >>> I get the error, but only when it builds the second app. In my main
> build
> >>> script I make a few apps in the same script (I make 3 apps which get
> moved
> >>> into the main app, any additional code in their site-packages.zip is
> moved
> >>> into the main apps zip, I remove the "sub-apps" Contents/Resources/lib
> >>> folder and symlink it at run time to the main apps lib folder.)
> >>>
> >>> Is this a bug or are you never supposed to run multiple setups in the
> same
> >>> build? If not how can I achieve the above?
> >>>
> >>>
> >>> Calling distutils.setup multiple times is at best untested with py2app,
> >>> and I wouldn't be surprised if it causes problems due to state leaking
> from
> >>> one build into the next.  A workaround would be to use the subprocess
> module
> >>> to run the setup jobs in separate processes.
> >>>
> >>
> >> Isn't the problem that they share dist folders, not a process? if not
> where
> >> does the state exist? Would I need to subprocess them from different
> >> directories?
> >>
> >>
> >> The py2app command itself has state and I haven't reviewed the code to
> know
> >> for sure that all state is cleaned up when the command is used twice in
> the
> >> same process. BTW. I also don't know if distutils.setup creates a new
> py2app
> >> command object every time it is called, if the second call to
> >> distutils.setup creates a new py2app command object there is no
> information
> >> leakage.
> >>
> >>
> >>>
> >>> BTW. I don't quite understand what you are trying to do with these 3
> apps.
> >>> Are you building 3 apps that share a lot of code/resources and where
> you
> >>> want two of the apps to link to the 3th one to reduce the amount of
> >>> disk-space used?
> >>>
> >>
> >> Yea exactly, I have some smaller apps which are used for specific
> separate
> >> jobs (one has a simple gui and generates and gathers log files from the
> main
> >> app and zips them up should the main app ever fail to open for
> instance),
> >> the jobs are all to do with the main app and all use a sub set of code
> to
> >> the main app, so I put the apps in the Resources folder and symlink the
> lib
> >> folder so I can include them with only using a little extra disk space,
> but
> >> more importantly keeping the installer size down.
> >>
> >>
> >> That sounds like something that would be useful to support directly.
> I'll
> >> add it to the list of nice-to-have featuers, but don't know when I'll
> get
> >> around to looking into this.
> >>
> >> Ronald
> >>
> >>
> >>>
> >>> Ronald
> >>>
> >>>
> >>> On 10 September 2012 13:18, Ronald Oussoren <ronaldoussoren at mac.com>
> >>> wrote:
> >>>>
> >>>>
> >>>> On 9 Sep, 2012, at 20:34, Paul Wiseman <poalman at gmail.com> wrote:
> >>>>
> >>>> Hey,
> >>>>
> >>>> When building an app that is using sqlalchemy I get this error:
> >>>>
> >>>> creating python loader for extension 'sqlalchemy.cprocessors'
> >>>> error:
> >>>>
> /Users/paul/Source/Python/build/bdist.macosx-10.6-intel/python2.7-standalone/app/temp/sqlalchemy/cprocessors.py:
> >>>> No such file or directory
> >>>>
> >>>> I took a look in site packages and there is no cprocessors.py, but a
> >>>> cprocessors.so - so maybe it is just looking for the wrong extension
> >>>>
> >>>> I tried adding "sqlalchemy.cprocessors" to the includes list in py2app
> >>>> but that hasn't helped.
> >>>>
> >>>> I was wondering if I can fool it by dropping an empty cprocessors.py
> so
> >>>> it will build, then swap it out afterwards for the so, but I'm sure
> there's
> >>>> a better way and I'm not convinced that could even work.
> >>>>
> >>>> Surely py2app doesn't assume every extension is .py, or if it does
> can it
> >>>> be changed?
> >>>>
> >>>>
> >>>> Py2app does not assume that every extension is a python file. Given
> the
> >>>> messasge I'd say that the error occurs in the code path that creates a
> >>>> helper python file that actually loads the exention.
> >>>>
> >>>> A little background information: when py2app creates the application
> >>>> bundle all modules are stored in a zipfile and loaded using python's
> >>>> zipimporter. Extensions cannot be stored in the zipfiles because the
> >>>> zipimporter doesn't support that. Py2app therefore creates a
> placeholder
> >>>> python module in the zipfile that loads the extensions from a
> directory in
> >>>> the application bundle.
> >>>>
> >>>> BTW. could you please create a sample project that demonstrates the
> >>>> problem? I've tried to reproduce your problem on my machine and
> failed to do
> >>>> so. I did run into another problem, py2app generated an incomplete
> bundle
> >>>> due to confusion between a _collections submodule in SQLAlchemy and
> the
> >>>> _collections extension in the stdlib; that's something I'm currently
> trying
> >>>> to fix.
> >>>>
> >>>> Ronald
> >>>>
> >>>> _______________________________________________
> >>>> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
> >>>> http://mail.python.org/mailman/listinfo/pythonmac-sig
> >>>> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >> _______________________________________________
> >> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
> >> http://mail.python.org/mailman/listinfo/pythonmac-sig
> >> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
> >>
> > _______________________________________________
> > Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
> > http://mail.python.org/mailman/listinfo/pythonmac-sig
> > unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
> http://mail.python.org/mailman/listinfo/pythonmac-sig
> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythonmac-sig/attachments/20120914/5cccd22f/attachment-0001.html>

More information about the Pythonmac-SIG mailing list