[Pythonmac-SIG] py2app unable to find cprocessors.so

[Paul Wiseman]

On 12 September 2012 19:27, Michael McCracken
<michael.mccracken at gmail.com>wrote:

> On Wed, Sep 12, 2012 at 10:54 AM, Paul Wiseman <poalman at gmail.com> wrote:
> > On 12 September 2012 18:13, Michael McCracken <
> michael.mccracken at gmail.com>
> > wrote:
> >>
> >> On Wed, Sep 12, 2012 at 4:08 AM, Paul Wiseman <poalman at gmail.com>
> wrote:
> >> > On 10 September 2012 13:18, Ronald Oussoren <ronaldoussoren at mac.com>
> >> > wrote:
> >> >>
> >> >>
> >> >> On 9 Sep, 2012, at 20:34, Paul Wiseman <poalman at gmail.com> wrote:
> >> >>
> >> >> Hey,
> >> >>
> >> >> When building an app that is using sqlalchemy I get this error:
> >> >>
> >> >> creating python loader for extension 'sqlalchemy.cprocessors'
> >> >> error:
> >> >>
> >> >>
> /Users/paul/Source/Python/build/bdist.macosx-10.6-intel/python2.7-standalone/app/temp/sqlalchemy/cprocessors.py:
> >> >> No such file or directory
> >> >>
> >> >> I took a look in site packages and there is no cprocessors.py, but a
> >> >> cprocessors.so - so maybe it is just looking for the wrong extension
> >> >>
> >> >> I tried adding "sqlalchemy.cprocessors" to the includes list in
> py2app
> >> >> but
> >> >> that hasn't helped.
> >> >>
> >> >> I was wondering if I can fool it by dropping an empty cprocessors.py
> so
> >> >> it
> >> >> will build, then swap it out afterwards for the so, but I'm sure
> >> >> there's a
> >> >> better way and I'm not convinced that could even work.
> >> >>
> >> >> Surely py2app doesn't assume every extension is .py, or if it does
> can
> >> >> it
> >> >> be changed?
> >> >>
> >> >>
> >> >> Py2app does not assume that every extension is a python file. Given
> the
> >> >> messasge I'd say that the error occurs in the code path that creates
> a
> >> >> helper python file that actually loads the exention.
> >> >>
> >> >> A little background information: when py2app creates the application
> >> >> bundle all modules are stored in a zipfile and loaded using python's
> >> >> zipimporter. Extensions cannot be stored in the zipfiles because the
> >> >> zipimporter doesn't support that. Py2app therefore creates a
> >> >> placeholder
> >> >> python module in the zipfile that loads the extensions from a
> directory
> >> >> in
> >> >> the application bundle.
> >> >>
> >> >> BTW. could you please create a sample project that demonstrates the
> >> >> problem? I've tried to reproduce your problem on my machine and
> failed
> >> >> to do
> >> >> so. I did run into another problem, py2app generated an incomplete
> >> >> bundle
> >> >> due to confusion between a _collections submodule in SQLAlchemy and
> the
> >> >> _collections extension in the stdlib; that's something I'm currently
> >> >> trying
> >> >> to fix.
> >> >>
> >> >> Ronald
> >> >
> >> >
> >> > (Oops just realised I didn't reply to the mailing list before)
> >> >
> >> > Ah,
> >> >
> >> > I've found out how to recreate the error
> >> >
> >> > If I create a main.py with nothing but 'import sqlalchemy'
> >> >
> >> > then use the following setup.py:
> >> >
> >> > from setuptools import setup
> >> >
> >> > setup(
> >> >     version="1",
> >> >     name="TestApp1",
> >> >     app=["main.py"],
> >> >     setup_requires=["py2app"]
> >> > )
> >> >
> >> > setup(
> >> >     version="1",
> >> >     name="TestApp2",
> >> >     app=["main.py"],
> >> >     setup_requires=["py2app"]
> >> > )
> >> >
> >> > If it doesn't produce the error it's probably because of this: "The
> >> > "cprocessors" module in SQLAlchemy is written in C and compiles
> >> > conditionally, based on if the current platform supports compiling it.
> >> > If
> >> > not present, SQLAlchemy continues to work perfectly well as the hooks
> >> > which
> >> > attempt to import it will fall back to pure-Python functions instead."
> >> > So
> >> > you may have a cprocessors.py which I dont think you'd get the
> problem,
> >> > only
> >> > if it compiled the .so when sqlalchemy installed.
> >> >
> >> > I get the error, but only when it builds the second app. In my main
> >> > build
> >> > script I make a few apps in the same script (I make 3 apps which get
> >> > moved
> >> > into the main app, any additional code in their site-packages.zip is
> >> > moved
> >> > into the main apps zip, I remove the "sub-apps" Contents/Resources/lib
> >> > folder and symlink it at run time to the main apps lib folder.)
> >> >
> >> > Is this a bug or are you never supposed to run multiple setups in the
> >> > same
> >> > build? If not how can I achieve the above?
> >>
> >> Paul, I'm also doing multiple calls to setup and it works.
> >> Does the main app import sqlalchemy?
> >> I ask because just removing SubApp.app/Contents/Resources/lib and
> >> symlinking that whole thing will remove anything in
> >> SubApp.app/Contents/Resources/lib/pythonx.y/lib-dynload/ , and if the
> >> sub app needs something in there that isn't in the lib-dynload path of
> >> the main app, then you won't have it in the final package.
> >>
> >> In my case I have a step that copies files from sub-app lib-dynload
> >> folders into the main app's folder before replacing the whole /lib dir
> >> with a symlink.
> >> Sounds like you might have to do the same thing.
> >>
> >> Hope this helps,
> >> -mike
> >
> >
> > Ah I think I do need that step you're right. my sub apps are a subset of
> the
> > main app, so I think I've gotten away with not copying anything over.
> I'll
> > definitely add that into my build script though.
> >
> > The error isn't related to the fact I symlink the lib folders together,
> > though, this is failing while building the second app. I just mentioned
> it
> > to explain why I want to call setup multiple times, I didn't know if it
> was
> > a common practice or not.
> >
> > Yea the main.py is just "import sqlalchemy" in my example, so the first
> and
> > second app will be the same in the example (both import sqlalchemy).
> Could
> > you check if you have "cprocessors.so" or "cprocessors.py" in your
> > site-packages/sqlalchemy folder? I think the above error only occurs if
> you
> > have the .so
>
> Ah, right - good point, the example wouldn't have the issue I mentioned.
>
> I can't reproduce your issue on OS X 10.7, with a recent trunk version
> of py2app.
>
> With a freshly installed sqlalchemy, building your sample runs fine, and I
> get
> dist/TestApp2.app/Contents/Resources/lib/python2.7/lib-dynload/sqlalchemy:
>   total used in directory 144 available 41745048
>   drwxr-xr-x   4 mmccrack  staff    136 Sep 12 11:11 .
>   drwxr-xr-x  49 mmccrack  staff   1666 Sep 12 11:11 ..
>   -rw-r--r--   1 mmccrack  staff  34716 Sep 12 11:11 cprocessors.so
>   -rw-r--r--   1 mmccrack  staff  35960 Sep 12 11:11 cresultproxy.so
>
> There is a sqlalchemy/cprocessors.py in the site-packages.zip.
>
> This appears correct to me, and if I add these lines to the main.py:
> from sqlalchemy import cprocessors
> print cprocessors.__file__
>
> I get this:
> 9/12/12 11:25:37.274 AM
> [0x0-0x317c179].org.pythonmac.unspecified.TestApp2:
>
> /Users/mmccrack/Documents/Canonical/Source/spikes/test-sqlalchemy-app/dist/TestApp2.app/Contents/Resources/lib/python2.7/lib-dynload/sqlalchemy/cprocessors.so
>
> So that seems to be working too...
>
> Not sure if this is helpful or not, I'm now wondering if there's a
> 10.6 issue lurking.
> -mike
>

I checked out the latest trunk (
http://svn.pythonmac.org/py2app/py2app/trunk/), installed that and the
problem has gone :)

Not sure which version I had before, but I only built this mac a week ago
or so, so think it's maybe the current version available for download
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythonmac-sig/attachments/20120913/155dd4d2/attachment-0001.html>