[Pythonmac-SIG] py2app unable to find cprocessors.so

[Michael McCracken]

On Wed, Sep 12, 2012 at 10:54 AM, Paul Wiseman <poalman at gmail.com> wrote:
> On 12 September 2012 18:13, Michael McCracken <michael.mccracken at gmail.com>
> wrote:
>>
>> On Wed, Sep 12, 2012 at 4:08 AM, Paul Wiseman <poalman at gmail.com> wrote:
>> > On 10 September 2012 13:18, Ronald Oussoren <ronaldoussoren at mac.com>
>> > wrote:
>> >>
>> >>
>> >> On 9 Sep, 2012, at 20:34, Paul Wiseman <poalman at gmail.com> wrote:
>> >>
>> >> Hey,
>> >>
>> >> When building an app that is using sqlalchemy I get this error:
>> >>
>> >> creating python loader for extension 'sqlalchemy.cprocessors'
>> >> error:
>> >>
>> >> /Users/paul/Source/Python/build/bdist.macosx-10.6-intel/python2.7-standalone/app/temp/sqlalchemy/cprocessors.py:
>> >> No such file or directory
>> >>
>> >> I took a look in site packages and there is no cprocessors.py, but a
>> >> cprocessors.so - so maybe it is just looking for the wrong extension
>> >>
>> >> I tried adding "sqlalchemy.cprocessors" to the includes list in py2app
>> >> but
>> >> that hasn't helped.
>> >>
>> >> I was wondering if I can fool it by dropping an empty cprocessors.py so
>> >> it
>> >> will build, then swap it out afterwards for the so, but I'm sure
>> >> there's a
>> >> better way and I'm not convinced that could even work.
>> >>
>> >> Surely py2app doesn't assume every extension is .py, or if it does can
>> >> it
>> >> be changed?
>> >>
>> >>
>> >> Py2app does not assume that every extension is a python file. Given the
>> >> messasge I'd say that the error occurs in the code path that creates a
>> >> helper python file that actually loads the exention.
>> >>
>> >> A little background information: when py2app creates the application
>> >> bundle all modules are stored in a zipfile and loaded using python's
>> >> zipimporter. Extensions cannot be stored in the zipfiles because the
>> >> zipimporter doesn't support that. Py2app therefore creates a
>> >> placeholder
>> >> python module in the zipfile that loads the extensions from a directory
>> >> in
>> >> the application bundle.
>> >>
>> >> BTW. could you please create a sample project that demonstrates the
>> >> problem? I've tried to reproduce your problem on my machine and failed
>> >> to do
>> >> so. I did run into another problem, py2app generated an incomplete
>> >> bundle
>> >> due to confusion between a _collections submodule in SQLAlchemy and the
>> >> _collections extension in the stdlib; that's something I'm currently
>> >> trying
>> >> to fix.
>> >>
>> >> Ronald
>> >
>> >
>> > (Oops just realised I didn't reply to the mailing list before)
>> >
>> > Ah,
>> >
>> > I've found out how to recreate the error
>> >
>> > If I create a main.py with nothing but 'import sqlalchemy'
>> >
>> > then use the following setup.py:
>> >
>> > from setuptools import setup
>> >
>> > setup(
>> >     version="1",
>> >     name="TestApp1",
>> >     app=["main.py"],
>> >     setup_requires=["py2app"]
>> > )
>> >
>> > setup(
>> >     version="1",
>> >     name="TestApp2",
>> >     app=["main.py"],
>> >     setup_requires=["py2app"]
>> > )
>> >
>> > If it doesn't produce the error it's probably because of this: "The
>> > "cprocessors" module in SQLAlchemy is written in C and compiles
>> > conditionally, based on if the current platform supports compiling it.
>> > If
>> > not present, SQLAlchemy continues to work perfectly well as the hooks
>> > which
>> > attempt to import it will fall back to pure-Python functions instead."
>> > So
>> > you may have a cprocessors.py which I dont think you'd get the problem,
>> > only
>> > if it compiled the .so when sqlalchemy installed.
>> >
>> > I get the error, but only when it builds the second app. In my main
>> > build
>> > script I make a few apps in the same script (I make 3 apps which get
>> > moved
>> > into the main app, any additional code in their site-packages.zip is
>> > moved
>> > into the main apps zip, I remove the "sub-apps" Contents/Resources/lib
>> > folder and symlink it at run time to the main apps lib folder.)
>> >
>> > Is this a bug or are you never supposed to run multiple setups in the
>> > same
>> > build? If not how can I achieve the above?
>>
>> Paul, I'm also doing multiple calls to setup and it works.
>> Does the main app import sqlalchemy?
>> I ask because just removing SubApp.app/Contents/Resources/lib and
>> symlinking that whole thing will remove anything in
>> SubApp.app/Contents/Resources/lib/pythonx.y/lib-dynload/ , and if the
>> sub app needs something in there that isn't in the lib-dynload path of
>> the main app, then you won't have it in the final package.
>>
>> In my case I have a step that copies files from sub-app lib-dynload
>> folders into the main app's folder before replacing the whole /lib dir
>> with a symlink.
>> Sounds like you might have to do the same thing.
>>
>> Hope this helps,
>> -mike
>
>
> Ah I think I do need that step you're right. my sub apps are a subset of the
> main app, so I think I've gotten away with not copying anything over. I'll
> definitely add that into my build script though.
>
> The error isn't related to the fact I symlink the lib folders together,
> though, this is failing while building the second app. I just mentioned it
> to explain why I want to call setup multiple times, I didn't know if it was
> a common practice or not.
>
> Yea the main.py is just "import sqlalchemy" in my example, so the first and
> second app will be the same in the example (both import sqlalchemy). Could
> you check if you have "cprocessors.so" or "cprocessors.py" in your
> site-packages/sqlalchemy folder? I think the above error only occurs if you
> have the .so

Ah, right - good point, the example wouldn't have the issue I mentioned.

I can't reproduce your issue on OS X 10.7, with a recent trunk version
of py2app.

With a freshly installed sqlalchemy, building your sample runs fine, and I get
dist/TestApp2.app/Contents/Resources/lib/python2.7/lib-dynload/sqlalchemy:
  total used in directory 144 available 41745048
  drwxr-xr-x   4 mmccrack  staff    136 Sep 12 11:11 .
  drwxr-xr-x  49 mmccrack  staff   1666 Sep 12 11:11 ..
  -rw-r--r--   1 mmccrack  staff  34716 Sep 12 11:11 cprocessors.so
  -rw-r--r--   1 mmccrack  staff  35960 Sep 12 11:11 cresultproxy.so

There is a sqlalchemy/cprocessors.py in the site-packages.zip.

This appears correct to me, and if I add these lines to the main.py:
from sqlalchemy import cprocessors
print cprocessors.__file__

I get this:
9/12/12 11:25:37.274 AM
[0x0-0x317c179].org.pythonmac.unspecified.TestApp2:
/Users/mmccrack/Documents/Canonical/Source/spikes/test-sqlalchemy-app/dist/TestApp2.app/Contents/Resources/lib/python2.7/lib-dynload/sqlalchemy/cprocessors.so

So that seems to be working too...

Not sure if this is helpful or not, I'm now wondering if there's a
10.6 issue lurking.
-mike