[Pythonmac-SIG] py2app unable to find cprocessors.so

Thu Sep 13 21:16:49 CEST 2012

On Thu, Sep 13, 2012 at 10:36 AM, Michael McCracken
<michael.mccracken at gmail.com> wrote:
> FWIW, here's how I do something similar now, to avoid having many
> copies of the Qt libraries.

cool! Thanks for the description

> There is one master app, and several sub-apps.
>
> * call setup() for each of the apps, generating full separate apps
> with copies of the Qt libraries and other stuff
> -- each call has a unique directory sent in the setup options
> "bdist_base" and "dist_dir". This avoids sharing build state, as
> mentioned earlier in the thread. This seems to work fine, calling
> setup() multiple times in the same script.

good to know -- I think the OP was right, the conflict over the
sharing build dirs -- I didn't know you could override that.

> * inside main app Contents/Resources, create empty sub.app/ directory
> * for each sub app:
> ** copy sub.app/Contents/MacOS into
> main.app/Contents/Resources/sub.app/Contents/MacOS
> ** copy sub.app/Contents/Info.plist into
> main.app/Contents/Resources/sub.app/Contents/Info.plist
> ** copy everything in sub.app/Contents/Resources/ that isn't include
> or lib into main.app/Contents/Resources/sub.app/Contents/Resources/
> ** create symlinks for sub.app/Contents/Resources/include ->
> main.app/Contents/Resources/include (do the same with Resources/lib)

OK -- here is where I"m confused -- with a sylink, you can do either
relative or absoute path, yes? absolute wouldn't work, as who knows
where the user will install it -- but relative could, as long as the
users keeps them all together -- is that how you handle it?

Alternatively I suppose you could require that main.app be installed
in /Library/Frameworks or something, and then symlink to the absolute
path.

I actually kind of like that idea, 'cause it would keep the apps a bit
more distinct, -- I have a base I want to use with multiple apps but
don't necessarily want folks to have to install everything at once to
use one.

> 3. merge the contents of
> sub.app/Contents/Resources/lib/python*/site-packages.zip into the main
> app's copy using zipfile
> 4. merge the contents of the
> sub.app/Contents/Resources/lib/python*/lib-dynload directory into the
> main app's directory

But anyway, this is a nice model for what py2app could so for us.

Thanks,
  -Chris

> NOTE: this will ignore any packages in
> sub.app/Contents/Resources/lib/python*/ that aren't in the
> site-packages.zip.
> That's fine for me, since I have the same "packages" option for all of the apps.
>
> -mike
>
>
> On Thu, Sep 13, 2012 at 1:53 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>>
>> On 13 Sep, 2012, at 10:47, Paul Wiseman <poalman at gmail.com> wrote:
>>
>> On 13 September 2012 07:18, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>>>
>>>
>>> On 10 Sep, 2012, at 16:37, Paul Wiseman <poalman at gmail.com> wrote:
>>>
>>> Ah,
>>>
>>> I've found out how to recreate the error
>>>
>>> If I create a main.py with nothing but 'import sqlalchemy'
>>>
>>> then use the following setup.py:
>>>
>>> from setuptools import setup
>>>
>>> setup(
>>>     version="1",
>>>     name="TestApp1",
>>>     app=["main.py"],
>>>     setup_requires=["py2app"]
>>> )
>>>
>>> setup(
>>>     version="1",
>>>     name="TestApp2",
>>>     app=["main.py"],
>>>     setup_requires=["py2app"]
>>> )
>>>
>>> If it doesn't produce the error it's probably because of this: "The
>>> "cprocessors" module in SQLAlchemy is written in C and compiles
>>> conditionally, based on if the current platform supports compiling it.   If
>>> not present, SQLAlchemy continues to work perfectly well as the hooks which
>>> attempt to import it will fall back to pure-Python functions instead." So
>>> you may have a cprocessors.py which I dont think you'd get the problem, only
>>> if it compiled the .so when sqlalchemy installed.
>>>
>>>
>>> I had the cprocessors extension in my build (that is, py2app mentioned in
>>> copied the extension)
>>>
>>>
>>> I get the error, but only when it builds the second app. In my main build
>>> script I make a few apps in the same script (I make 3 apps which get moved
>>> into the main app, any additional code in their site-packages.zip is moved
>>> into the main apps zip, I remove the "sub-apps" Contents/Resources/lib
>>> folder and symlink it at run time to the main apps lib folder.)
>>>
>>> Is this a bug or are you never supposed to run multiple setups in the same
>>> build? If not how can I achieve the above?
>>>
>>>
>>> Calling distutils.setup multiple times is at best untested with py2app,
>>> and I wouldn't be surprised if it causes problems due to state leaking from
>>> one build into the next.  A workaround would be to use the subprocess module
>>> to run the setup jobs in separate processes.
>>>
>>
>> Isn't the problem that they share dist folders, not a process? if not where
>> does the state exist? Would I need to subprocess them from different
>> directories?
>>
>>
>> The py2app command itself has state and I haven't reviewed the code to know
>> for sure that all state is cleaned up when the command is used twice in the
>> same process. BTW. I also don't know if distutils.setup creates a new py2app
>> command object every time it is called, if the second call to
>> distutils.setup creates a new py2app command object there is no information
>> leakage.
>>
>>
>>>
>>> BTW. I don't quite understand what you are trying to do with these 3 apps.
>>> Are you building 3 apps that share a lot of code/resources and where you
>>> want two of the apps to link to the 3th one to reduce the amount of
>>> disk-space used?
>>>
>>
>> Yea exactly, I have some smaller apps which are used for specific separate
>> jobs (one has a simple gui and generates and gathers log files from the main
>> app and zips them up should the main app ever fail to open for instance),
>> the jobs are all to do with the main app and all use a sub set of code to
>> the main app, so I put the apps in the Resources folder and symlink the lib
>> folder so I can include them with only using a little extra disk space, but
>> more importantly keeping the installer size down.
>>
>>
>> That sounds like something that would be useful to support directly. I'll
>> add it to the list of nice-to-have featuers, but don't know when I'll get
>> around to looking into this.
>>
>> Ronald
>>
>>
>>>
>>> Ronald
>>>
>>>
>>> On 10 September 2012 13:18, Ronald Oussoren <ronaldoussoren at mac.com>
>>> wrote:
>>>>
>>>>
>>>> On 9 Sep, 2012, at 20:34, Paul Wiseman <poalman at gmail.com> wrote:
>>>>
>>>> Hey,
>>>>
>>>> When building an app that is using sqlalchemy I get this error:
>>>>
>>>> creating python loader for extension 'sqlalchemy.cprocessors'
>>>> error:
>>>> /Users/paul/Source/Python/build/bdist.macosx-10.6-intel/python2.7-standalone/app/temp/sqlalchemy/cprocessors.py:
>>>> No such file or directory
>>>>
>>>> I took a look in site packages and there is no cprocessors.py, but a
>>>> cprocessors.so - so maybe it is just looking for the wrong extension
>>>>
>>>> I tried adding "sqlalchemy.cprocessors" to the includes list in py2app
>>>> but that hasn't helped.
>>>>
>>>> I was wondering if I can fool it by dropping an empty cprocessors.py so
>>>> it will build, then swap it out afterwards for the so, but I'm sure there's
>>>> a better way and I'm not convinced that could even work.
>>>>
>>>> Surely py2app doesn't assume every extension is .py, or if it does can it
>>>> be changed?
>>>>
>>>>
>>>> Py2app does not assume that every extension is a python file. Given the
>>>> messasge I'd say that the error occurs in the code path that creates a
>>>> helper python file that actually loads the exention.
>>>>
>>>> A little background information: when py2app creates the application
>>>> bundle all modules are stored in a zipfile and loaded using python's
>>>> zipimporter. Extensions cannot be stored in the zipfiles because the
>>>> zipimporter doesn't support that. Py2app therefore creates a placeholder
>>>> python module in the zipfile that loads the extensions from a directory in
>>>> the application bundle.
>>>>
>>>> BTW. could you please create a sample project that demonstrates the
>>>> problem? I've tried to reproduce your problem on my machine and failed to do
>>>> so. I did run into another problem, py2app generated an incomplete bundle
>>>> due to confusion between a _collections submodule in SQLAlchemy and the
>>>> _collections extension in the stdlib; that's something I'm currently trying
>>>> to fix.
>>>>
>>>> Ronald
>>>>
>>>> _______________________________________________
>>>> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
>>>> http://mail.python.org/mailman/listinfo/pythonmac-sig
>>>> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
>>>>
>>>>
>>>
>>>
>>
>>
>>
>> _______________________________________________
>> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
>> http://mail.python.org/mailman/listinfo/pythonmac-sig
>> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
>>
> _______________________________________________
> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
> http://mail.python.org/mailman/listinfo/pythonmac-sig
> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov