[Pythonmac-SIG] py2app unable to find cprocessors.so

Michael McCracken michael.mccracken at gmail.com
Fri Sep 14 00:54:41 CEST 2012 

On Thu, Sep 13, 2012 at 12:16 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Thu, Sep 13, 2012 at 10:36 AM, Michael McCracken
> <michael.mccracken at gmail.com> wrote:
>> FWIW, here's how I do something similar now, to avoid having many
>> copies of the Qt libraries.
>
> cool! Thanks for the description
>
>> There is one master app, and several sub-apps.
>>
>> * call setup() for each of the apps, generating full separate apps
>> with copies of the Qt libraries and other stuff
>> -- each call has a unique directory sent in the setup options
>> "bdist_base" and "dist_dir". This avoids sharing build state, as
>> mentioned earlier in the thread. This seems to work fine, calling
>> setup() multiple times in the same script.
>
> good to know -- I think the OP was right, the conflict over the
> sharing build dirs -- I didn't know you could override that.
>
>> * inside main app Contents/Resources, create empty sub.app/ directory
>> * for each sub app:
>> ** copy sub.app/Contents/MacOS into
>> main.app/Contents/Resources/sub.app/Contents/MacOS
>> ** copy sub.app/Contents/Info.plist into
>> main.app/Contents/Resources/sub.app/Contents/Info.plist
>> ** copy everything in sub.app/Contents/Resources/ that isn't include
>> or lib into main.app/Contents/Resources/sub.app/Contents/Resources/
>> ** create symlinks for sub.app/Contents/Resources/include ->
>> main.app/Contents/Resources/include (do the same with Resources/lib)
>
> OK -- here is where I"m confused -- with a sylink, you can do either
> relative or absoute path, yes? absolute wouldn't work, as who knows
> where the user will install it -- but relative could, as long as the
> users keeps them all together -- is that how you handle it?

Yes, they're relative paths - all the sub-apps stay inside the main
.app wrapper.
If someone moves things around inside there, that's "unsupported" :)

> Alternatively I suppose you could require that main.app be installed
> in /Library/Frameworks or something, and then symlink to the absolute
> path.
>
> I actually kind of like that idea, 'cause it would keep the apps a bit
> more distinct, -- I have a base I want to use with multiple apps but
> don't necessarily want folks to have to install everything at once to
> use one.

Yes, that'd probably work, but it has the inconvenience of requiring
an installer...
I'm not sure what the best way to go in your situation is.
An installer isn't all that bad, I guess.

-mike

>> 3. merge the contents of
>> sub.app/Contents/Resources/lib/python*/site-packages.zip into the main
>> app's copy using zipfile
>> 4. merge the contents of the
>> sub.app/Contents/Resources/lib/python*/lib-dynload directory into the
>> main app's directory
>
>
> But anyway, this is a nice model for what py2app could so for us.
>
> Thanks,
>   -Chris
>
>> NOTE: this will ignore any packages in
>> sub.app/Contents/Resources/lib/python*/ that aren't in the
>> site-packages.zip.
>> That's fine for me, since I have the same "packages" option for all of the apps.
>>
>> -mike
>>
>>
>> On Thu, Sep 13, 2012 at 1:53 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>>>
>>> On 13 Sep, 2012, at 10:47, Paul Wiseman <poalman at gmail.com> wrote:
>>>
>>> On 13 September 2012 07:18, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>>>>
>>>>
>>>> On 10 Sep, 2012, at 16:37, Paul Wiseman <poalman at gmail.com> wrote:
>>>>
>>>> Ah,
>>>>
>>>> I've found out how to recreate the error
>>>>
>>>> If I create a main.py with nothing but 'import sqlalchemy'
>>>>
>>>> then use the following setup.py:
>>>>
>>>> from setuptools import setup
>>>>
>>>> setup(
>>>>     version="1",
>>>>     name="TestApp1",
>>>>     app=["main.py"],
>>>>     setup_requires=["py2app"]
>>>> )
>>>>
>>>> setup(
>>>>     version="1",
>>>>     name="TestApp2",
>>>>     app=["main.py"],
>>>>     setup_requires=["py2app"]
>>>> )
>>>>
>>>> If it doesn't produce the error it's probably because of this: "The
>>>> "cprocessors" module in SQLAlchemy is written in C and compiles
>>>> conditionally, based on if the current platform supports compiling it.   If
>>>> not present, SQLAlchemy continues to work perfectly well as the hooks which
>>>> attempt to import it will fall back to pure-Python functions instead." So
>>>> you may have a cprocessors.py which I dont think you'd get the problem, only
>>>> if it compiled the .so when sqlalchemy installed.
>>>>
>>>>
>>>> I had the cprocessors extension in my build (that is, py2app mentioned in
>>>> copied the extension)
>>>>
>>>>
>>>> I get the error, but only when it builds the second app. In my main build
>>>> script I make a few apps in the same script (I make 3 apps which get moved
>>>> into the main app, any additional code in their site-packages.zip is moved
>>>> into the main apps zip, I remove the "sub-apps" Contents/Resources/lib
>>>> folder and symlink it at run time to the main apps lib folder.)
>>>>
>>>> Is this a bug or are you never supposed to run multiple setups in the same
>>>> build? If not how can I achieve the above?
>>>>
>>>>
>>>> Calling distutils.setup multiple times is at best untested with py2app,
>>>> and I wouldn't be surprised if it causes problems due to state leaking from
>>>> one build into the next.  A workaround would be to use the subprocess module
>>>> to run the setup jobs in separate processes.
>>>>
>>>
>>> Isn't the problem that they share dist folders, not a process? if not where
>>> does the state exist? Would I need to subprocess them from different
>>> directories?
>>>
>>>
>>> The py2app command itself has state and I haven't reviewed the code to know
>>> for sure that all state is cleaned up when the command is used twice in the
>>> same process. BTW. I also don't know if distutils.setup creates a new py2app
>>> command object every time it is called, if the second call to
>>> distutils.setup creates a new py2app command object there is no information
>>> leakage.
>>>
>>>
>>>>
>>>> BTW. I don't quite understand what you are trying to do with these 3 apps.
>>>> Are you building 3 apps that share a lot of code/resources and where you
>>>> want two of the apps to link to the 3th one to reduce the amount of
>>>> disk-space used?
>>>>
>>>
>>> Yea exactly, I have some smaller apps which are used for specific separate
>>> jobs (one has a simple gui and generates and gathers log files from the main
>>> app and zips them up should the main app ever fail to open for instance),
>>> the jobs are all to do with the main app and all use a sub set of code to
>>> the main app, so I put the apps in the Resources folder and symlink the lib
>>> folder so I can include them with only using a little extra disk space, but
>>> more importantly keeping the installer size down.
>>>
>>>
>>> That sounds like something that would be useful to support directly. I'll
>>> add it to the list of nice-to-have featuers, but don't know when I'll get
>>> around to looking into this.
>>>
>>> Ronald
>>>
>>>
>>>>
>>>> Ronald
>>>>
>>>>
>>>> On 10 September 2012 13:18, Ronald Oussoren <ronaldoussoren at mac.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> On 9 Sep, 2012, at 20:34, Paul Wiseman <poalman at gmail.com> wrote:
>>>>>
>>>>> Hey,
>>>>>
>>>>> When building an app that is using sqlalchemy I get this error:
>>>>>
>>>>> creating python loader for extension 'sqlalchemy.cprocessors'
>>>>> error:
>>>>> /Users/paul/Source/Python/build/bdist.macosx-10.6-intel/python2.7-standalone/app/temp/sqlalchemy/cprocessors.py:
>>>>> No such file or directory
>>>>>
>>>>> I took a look in site packages and there is no cprocessors.py, but a
>>>>> cprocessors.so - so maybe it is just looking for the wrong extension
>>>>>
>>>>> I tried adding "sqlalchemy.cprocessors" to the includes list in py2app
>>>>> but that hasn't helped.
>>>>>
>>>>> I was wondering if I can fool it by dropping an empty cprocessors.py so
>>>>> it will build, then swap it out afterwards for the so, but I'm sure there's
>>>>> a better way and I'm not convinced that could even work.
>>>>>
>>>>> Surely py2app doesn't assume every extension is .py, or if it does can it
>>>>> be changed?
>>>>>
>>>>>
>>>>> Py2app does not assume that every extension is a python file. Given the
>>>>> messasge I'd say that the error occurs in the code path that creates a
>>>>> helper python file that actually loads the exention.
>>>>>
>>>>> A little background information: when py2app creates the application
>>>>> bundle all modules are stored in a zipfile and loaded using python's
>>>>> zipimporter. Extensions cannot be stored in the zipfiles because the
>>>>> zipimporter doesn't support that. Py2app therefore creates a placeholder
>>>>> python module in the zipfile that loads the extensions from a directory in
>>>>> the application bundle.
>>>>>
>>>>> BTW. could you please create a sample project that demonstrates the
>>>>> problem? I've tried to reproduce your problem on my machine and failed to do
>>>>> so. I did run into another problem, py2app generated an incomplete bundle
>>>>> due to confusion between a _collections submodule in SQLAlchemy and the
>>>>> _collections extension in the stdlib; that's something I'm currently trying
>>>>> to fix.
>>>>>
>>>>> Ronald
>>>>>
>>>>> _______________________________________________
>>>>> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
>>>>> http://mail.python.org/mailman/listinfo/pythonmac-sig
>>>>> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
>>> http://mail.python.org/mailman/listinfo/pythonmac-sig
>>> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
>>>
>> _______________________________________________
>> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
>> http://mail.python.org/mailman/listinfo/pythonmac-sig
>> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
> http://mail.python.org/mailman/listinfo/pythonmac-sig
> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

More information about the Pythonmac-SIG mailing list