[Pythonmac-SIG] py2app unable to find cprocessors.so

[Paul Wiseman]

On 13 September 2012 10:33, Paul Wiseman <poalman at gmail.com> wrote:

> On 12 September 2012 19:27, Michael McCracken <michael.mccracken at gmail.com
> > wrote:
>
>> On Wed, Sep 12, 2012 at 10:54 AM, Paul Wiseman <poalman at gmail.com> wrote:
>> > On 12 September 2012 18:13, Michael McCracken <
>> michael.mccracken at gmail.com>
>> > wrote:
>> >>
>> >> On Wed, Sep 12, 2012 at 4:08 AM, Paul Wiseman <poalman at gmail.com>
>> wrote:
>> >> > On 10 September 2012 13:18, Ronald Oussoren <ronaldoussoren at mac.com>
>> >> > wrote:
>> >> >>
>> >> >>
>> >> >> On 9 Sep, 2012, at 20:34, Paul Wiseman <poalman at gmail.com> wrote:
>> >> >>
>> >> >> Hey,
>> >> >>
>> >> >> When building an app that is using sqlalchemy I get this error:
>> >> >>
>> >> >> creating python loader for extension 'sqlalchemy.cprocessors'
>> >> >> error:
>> >> >>
>> >> >>
>> /Users/paul/Source/Python/build/bdist.macosx-10.6-intel/python2.7-standalone/app/temp/sqlalchemy/cprocessors.py:
>> >> >> No such file or directory
>> >> >>
>> >> >> I took a look in site packages and there is no cprocessors.py, but a
>> >> >> cprocessors.so - so maybe it is just looking for the wrong extension
>> >> >>
>> >> >> I tried adding "sqlalchemy.cprocessors" to the includes list in
>> py2app
>> >> >> but
>> >> >> that hasn't helped.
>> >> >>
>> >> >> I was wondering if I can fool it by dropping an empty
>> cprocessors.py so
>> >> >> it
>> >> >> will build, then swap it out afterwards for the so, but I'm sure
>> >> >> there's a
>> >> >> better way and I'm not convinced that could even work.
>> >> >>
>> >> >> Surely py2app doesn't assume every extension is .py, or if it does
>> can
>> >> >> it
>> >> >> be changed?
>> >> >>
>> >> >>
>> >> >> Py2app does not assume that every extension is a python file. Given
>> the
>> >> >> messasge I'd say that the error occurs in the code path that
>> creates a
>> >> >> helper python file that actually loads the exention.
>> >> >>
>> >> >> A little background information: when py2app creates the application
>> >> >> bundle all modules are stored in a zipfile and loaded using python's
>> >> >> zipimporter. Extensions cannot be stored in the zipfiles because the
>> >> >> zipimporter doesn't support that. Py2app therefore creates a
>> >> >> placeholder
>> >> >> python module in the zipfile that loads the extensions from a
>> directory
>> >> >> in
>> >> >> the application bundle.
>> >> >>
>> >> >> BTW. could you please create a sample project that demonstrates the
>> >> >> problem? I've tried to reproduce your problem on my machine and
>> failed
>> >> >> to do
>> >> >> so. I did run into another problem, py2app generated an incomplete
>> >> >> bundle
>> >> >> due to confusion between a _collections submodule in SQLAlchemy and
>> the
>> >> >> _collections extension in the stdlib; that's something I'm currently
>> >> >> trying
>> >> >> to fix.
>> >> >>
>> >> >> Ronald
>> >> >
>> >> >
>> >> > (Oops just realised I didn't reply to the mailing list before)
>> >> >
>> >> > Ah,
>> >> >
>> >> > I've found out how to recreate the error
>> >> >
>> >> > If I create a main.py with nothing but 'import sqlalchemy'
>> >> >
>> >> > then use the following setup.py:
>> >> >
>> >> > from setuptools import setup
>> >> >
>> >> > setup(
>> >> >     version="1",
>> >> >     name="TestApp1",
>> >> >     app=["main.py"],
>> >> >     setup_requires=["py2app"]
>> >> > )
>> >> >
>> >> > setup(
>> >> >     version="1",
>> >> >     name="TestApp2",
>> >> >     app=["main.py"],
>> >> >     setup_requires=["py2app"]
>> >> > )
>> >> >
>> >> > If it doesn't produce the error it's probably because of this: "The
>> >> > "cprocessors" module in SQLAlchemy is written in C and compiles
>> >> > conditionally, based on if the current platform supports compiling
>> it.
>> >> > If
>> >> > not present, SQLAlchemy continues to work perfectly well as the hooks
>> >> > which
>> >> > attempt to import it will fall back to pure-Python functions
>> instead."
>> >> > So
>> >> > you may have a cprocessors.py which I dont think you'd get the
>> problem,
>> >> > only
>> >> > if it compiled the .so when sqlalchemy installed.
>> >> >
>> >> > I get the error, but only when it builds the second app. In my main
>> >> > build
>> >> > script I make a few apps in the same script (I make 3 apps which get
>> >> > moved
>> >> > into the main app, any additional code in their site-packages.zip is
>> >> > moved
>> >> > into the main apps zip, I remove the "sub-apps"
>> Contents/Resources/lib
>> >> > folder and symlink it at run time to the main apps lib folder.)
>> >> >
>> >> > Is this a bug or are you never supposed to run multiple setups in the
>> >> > same
>> >> > build? If not how can I achieve the above?
>> >>
>> >> Paul, I'm also doing multiple calls to setup and it works.
>> >> Does the main app import sqlalchemy?
>> >> I ask because just removing SubApp.app/Contents/Resources/lib and
>> >> symlinking that whole thing will remove anything in
>> >> SubApp.app/Contents/Resources/lib/pythonx.y/lib-dynload/ , and if the
>> >> sub app needs something in there that isn't in the lib-dynload path of
>> >> the main app, then you won't have it in the final package.
>> >>
>> >> In my case I have a step that copies files from sub-app lib-dynload
>> >> folders into the main app's folder before replacing the whole /lib dir
>> >> with a symlink.
>> >> Sounds like you might have to do the same thing.
>> >>
>> >> Hope this helps,
>> >> -mike
>> >
>> >
>> > Ah I think I do need that step you're right. my sub apps are a subset
>> of the
>> > main app, so I think I've gotten away with not copying anything over.
>> I'll
>> > definitely add that into my build script though.
>> >
>> > The error isn't related to the fact I symlink the lib folders together,
>> > though, this is failing while building the second app. I just mentioned
>> it
>> > to explain why I want to call setup multiple times, I didn't know if it
>> was
>> > a common practice or not.
>> >
>> > Yea the main.py is just "import sqlalchemy" in my example, so the first
>> and
>> > second app will be the same in the example (both import sqlalchemy).
>> Could
>> > you check if you have "cprocessors.so" or "cprocessors.py" in your
>> > site-packages/sqlalchemy folder? I think the above error only occurs if
>> you
>> > have the .so
>>
>> Ah, right - good point, the example wouldn't have the issue I mentioned.
>>
>> I can't reproduce your issue on OS X 10.7, with a recent trunk version
>> of py2app.
>>
>> With a freshly installed sqlalchemy, building your sample runs fine, and
>> I get
>> dist/TestApp2.app/Contents/Resources/lib/python2.7/lib-dynload/sqlalchemy:
>>   total used in directory 144 available 41745048
>>   drwxr-xr-x   4 mmccrack  staff    136 Sep 12 11:11 .
>>   drwxr-xr-x  49 mmccrack  staff   1666 Sep 12 11:11 ..
>>   -rw-r--r--   1 mmccrack  staff  34716 Sep 12 11:11 cprocessors.so
>>   -rw-r--r--   1 mmccrack  staff  35960 Sep 12 11:11 cresultproxy.so
>>
>> There is a sqlalchemy/cprocessors.py in the site-packages.zip.
>>
>> This appears correct to me, and if I add these lines to the main.py:
>> from sqlalchemy import cprocessors
>> print cprocessors.__file__
>>
>> I get this:
>> 9/12/12 11:25:37.274 AM
>> [0x0-0x317c179].org.pythonmac.unspecified.TestApp2:
>>
>> /Users/mmccrack/Documents/Canonical/Source/spikes/test-sqlalchemy-app/dist/TestApp2.app/Contents/Resources/lib/python2.7/lib-dynload/sqlalchemy/cprocessors.so
>>
>> So that seems to be working too...
>>
>> Not sure if this is helpful or not, I'm now wondering if there's a
>> 10.6 issue lurking.
>> -mike
>>
>
> I checked out the latest trunk (
> http://svn.pythonmac.org/py2app/py2app/trunk/), installed that and the
> problem has gone :)
>
> Not sure which version I had before, but I only built this mac a week ago
> or so, so think it's maybe the current version available for download
>
>
Hm maybe I spoke too soon, it builds ok now but when I run the app I get a
error dialog with Open console and Terminate options, and this stack trace
in the console

Traceback (most recent call last):
  File "/Applications/ZipCloud.app/Contents/Resources/__boot__.py", line
103, in <module>
    _argv_emulation()
  File "/Applications/ZipCloud.app/Contents/Resources/__boot__.py", line
101, in _argv_emulation
    _get_argvemulator().mainloop()
  File "/Applications/ZipCloud.app/Contents/Resources/__boot__.py", line
40, in mainloop
    stoptime = Evt.TickCount() + timeout
AttributeError: 'module' object has no attribute 'TickCount'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythonmac-sig/attachments/20120913/eadd5fd1/attachment.html>