[Pythonmac-SIG] py2app unable to find cprocessors.so

[Paul Wiseman]

On 12 September 2012 18:13, Michael McCracken
<michael.mccracken at gmail.com>wrote:

> On Wed, Sep 12, 2012 at 4:08 AM, Paul Wiseman <poalman at gmail.com> wrote:
> > On 10 September 2012 13:18, Ronald Oussoren <ronaldoussoren at mac.com>
> wrote:
> >>
> >>
> >> On 9 Sep, 2012, at 20:34, Paul Wiseman <poalman at gmail.com> wrote:
> >>
> >> Hey,
> >>
> >> When building an app that is using sqlalchemy I get this error:
> >>
> >> creating python loader for extension 'sqlalchemy.cprocessors'
> >> error:
> >>
> /Users/paul/Source/Python/build/bdist.macosx-10.6-intel/python2.7-standalone/app/temp/sqlalchemy/cprocessors.py:
> >> No such file or directory
> >>
> >> I took a look in site packages and there is no cprocessors.py, but a
> >> cprocessors.so - so maybe it is just looking for the wrong extension
> >>
> >> I tried adding "sqlalchemy.cprocessors" to the includes list in py2app
> but
> >> that hasn't helped.
> >>
> >> I was wondering if I can fool it by dropping an empty cprocessors.py so
> it
> >> will build, then swap it out afterwards for the so, but I'm sure
> there's a
> >> better way and I'm not convinced that could even work.
> >>
> >> Surely py2app doesn't assume every extension is .py, or if it does can
> it
> >> be changed?
> >>
> >>
> >> Py2app does not assume that every extension is a python file. Given the
> >> messasge I'd say that the error occurs in the code path that creates a
> >> helper python file that actually loads the exention.
> >>
> >> A little background information: when py2app creates the application
> >> bundle all modules are stored in a zipfile and loaded using python's
> >> zipimporter. Extensions cannot be stored in the zipfiles because the
> >> zipimporter doesn't support that. Py2app therefore creates a placeholder
> >> python module in the zipfile that loads the extensions from a directory
> in
> >> the application bundle.
> >>
> >> BTW. could you please create a sample project that demonstrates the
> >> problem? I've tried to reproduce your problem on my machine and failed
> to do
> >> so. I did run into another problem, py2app generated an incomplete
> bundle
> >> due to confusion between a _collections submodule in SQLAlchemy and the
> >> _collections extension in the stdlib; that's something I'm currently
> trying
> >> to fix.
> >>
> >> Ronald
> >
> >
> > (Oops just realised I didn't reply to the mailing list before)
> >
> > Ah,
> >
> > I've found out how to recreate the error
> >
> > If I create a main.py with nothing but 'import sqlalchemy'
> >
> > then use the following setup.py:
> >
> > from setuptools import setup
> >
> > setup(
> >     version="1",
> >     name="TestApp1",
> >     app=["main.py"],
> >     setup_requires=["py2app"]
> > )
> >
> > setup(
> >     version="1",
> >     name="TestApp2",
> >     app=["main.py"],
> >     setup_requires=["py2app"]
> > )
> >
> > If it doesn't produce the error it's probably because of this: "The
> > "cprocessors" module in SQLAlchemy is written in C and compiles
> > conditionally, based on if the current platform supports compiling it.
> If
> > not present, SQLAlchemy continues to work perfectly well as the hooks
> which
> > attempt to import it will fall back to pure-Python functions instead." So
> > you may have a cprocessors.py which I dont think you'd get the problem,
> only
> > if it compiled the .so when sqlalchemy installed.
> >
> > I get the error, but only when it builds the second app. In my main build
> > script I make a few apps in the same script (I make 3 apps which get
> moved
> > into the main app, any additional code in their site-packages.zip is
> moved
> > into the main apps zip, I remove the "sub-apps" Contents/Resources/lib
> > folder and symlink it at run time to the main apps lib folder.)
> >
> > Is this a bug or are you never supposed to run multiple setups in the
> same
> > build? If not how can I achieve the above?
>
> Paul, I'm also doing multiple calls to setup and it works.
> Does the main app import sqlalchemy?
> I ask because just removing SubApp.app/Contents/Resources/lib and
> symlinking that whole thing will remove anything in
> SubApp.app/Contents/Resources/lib/pythonx.y/lib-dynload/ , and if the
> sub app needs something in there that isn't in the lib-dynload path of
> the main app, then you won't have it in the final package.
>
> In my case I have a step that copies files from sub-app lib-dynload
> folders into the main app's folder before replacing the whole /lib dir
> with a symlink.
> Sounds like you might have to do the same thing.
>
> Hope this helps,
> -mike
>

Ah I think I do need that step you're right. my sub apps are a subset of
the main app, so I think I've gotten away with not copying anything over.
I'll definitely add that into my build script though.

The error isn't related to the fact I symlink the lib folders together,
though, this is failing while building the second app. I just mentioned it
to explain why I want to call setup multiple times, I didn't know if it was
a common practice or not.

Yea the main.py is just "import sqlalchemy" in my example, so the first and
second app will be the same in the example (both import sqlalchemy). Could
you check if you have "cprocessors.so" or "cprocessors.py" in your
site-packages/sqlalchemy folder? I think the above error only occurs if you
have the .so
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythonmac-sig/attachments/20120912/7a3e0782/attachment-0001.html>