[Pythonmac-SIG] py2app unable to find cprocessors.so

[Michael McCracken]

On Wed, Sep 12, 2012 at 4:08 AM, Paul Wiseman <poalman at gmail.com> wrote:
> On 10 September 2012 13:18, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>>
>>
>> On 9 Sep, 2012, at 20:34, Paul Wiseman <poalman at gmail.com> wrote:
>>
>> Hey,
>>
>> When building an app that is using sqlalchemy I get this error:
>>
>> creating python loader for extension 'sqlalchemy.cprocessors'
>> error:
>> /Users/paul/Source/Python/build/bdist.macosx-10.6-intel/python2.7-standalone/app/temp/sqlalchemy/cprocessors.py:
>> No such file or directory
>>
>> I took a look in site packages and there is no cprocessors.py, but a
>> cprocessors.so - so maybe it is just looking for the wrong extension
>>
>> I tried adding "sqlalchemy.cprocessors" to the includes list in py2app but
>> that hasn't helped.
>>
>> I was wondering if I can fool it by dropping an empty cprocessors.py so it
>> will build, then swap it out afterwards for the so, but I'm sure there's a
>> better way and I'm not convinced that could even work.
>>
>> Surely py2app doesn't assume every extension is .py, or if it does can it
>> be changed?
>>
>>
>> Py2app does not assume that every extension is a python file. Given the
>> messasge I'd say that the error occurs in the code path that creates a
>> helper python file that actually loads the exention.
>>
>> A little background information: when py2app creates the application
>> bundle all modules are stored in a zipfile and loaded using python's
>> zipimporter. Extensions cannot be stored in the zipfiles because the
>> zipimporter doesn't support that. Py2app therefore creates a placeholder
>> python module in the zipfile that loads the extensions from a directory in
>> the application bundle.
>>
>> BTW. could you please create a sample project that demonstrates the
>> problem? I've tried to reproduce your problem on my machine and failed to do
>> so. I did run into another problem, py2app generated an incomplete bundle
>> due to confusion between a _collections submodule in SQLAlchemy and the
>> _collections extension in the stdlib; that's something I'm currently trying
>> to fix.
>>
>> Ronald
>
>
> (Oops just realised I didn't reply to the mailing list before)
>
> Ah,
>
> I've found out how to recreate the error
>
> If I create a main.py with nothing but 'import sqlalchemy'
>
> then use the following setup.py:
>
> from setuptools import setup
>
> setup(
>     version="1",
>     name="TestApp1",
>     app=["main.py"],
>     setup_requires=["py2app"]
> )
>
> setup(
>     version="1",
>     name="TestApp2",
>     app=["main.py"],
>     setup_requires=["py2app"]
> )
>
> If it doesn't produce the error it's probably because of this: "The
> "cprocessors" module in SQLAlchemy is written in C and compiles
> conditionally, based on if the current platform supports compiling it.   If
> not present, SQLAlchemy continues to work perfectly well as the hooks which
> attempt to import it will fall back to pure-Python functions instead." So
> you may have a cprocessors.py which I dont think you'd get the problem, only
> if it compiled the .so when sqlalchemy installed.
>
> I get the error, but only when it builds the second app. In my main build
> script I make a few apps in the same script (I make 3 apps which get moved
> into the main app, any additional code in their site-packages.zip is moved
> into the main apps zip, I remove the "sub-apps" Contents/Resources/lib
> folder and symlink it at run time to the main apps lib folder.)
>
> Is this a bug or are you never supposed to run multiple setups in the same
> build? If not how can I achieve the above?

Paul, I'm also doing multiple calls to setup and it works.
Does the main app import sqlalchemy?
I ask because just removing SubApp.app/Contents/Resources/lib and
symlinking that whole thing will remove anything in
SubApp.app/Contents/Resources/lib/pythonx.y/lib-dynload/ , and if the
sub app needs something in there that isn't in the lib-dynload path of
the main app, then you won't have it in the final package.

In my case I have a step that copies files from sub-app lib-dynload
folders into the main app's folder before replacing the whole /lib dir
with a symlink.
Sounds like you might have to do the same thing.

Hope this helps,
-mike