[Pythonmac-SIG] py2app unable to find cprocessors.so

[Ronald Oussoren]

On 13 Sep, 2012, at 10:47, Paul Wiseman <poalman at gmail.com> wrote:

> On 13 September 2012 07:18, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> 
> On 10 Sep, 2012, at 16:37, Paul Wiseman <poalman at gmail.com> wrote:
> 
>> Ah,
>> 
>> I've found out how to recreate the error
>> 
>> If I create a main.py with nothing but 'import sqlalchemy'
>> 
>> then use the following setup.py:
>> 
>> from setuptools import setup
>> 
>> setup(
>>     version="1",
>>     name="TestApp1",
>>     app=["main.py"],
>>     setup_requires=["py2app"]
>> )
>> 
>> setup(
>>     version="1",
>>     name="TestApp2",
>>     app=["main.py"],
>>     setup_requires=["py2app"]
>> )
>> 
>> If it doesn't produce the error it's probably because of this: "The "cprocessors" module in SQLAlchemy is written in C and compiles conditionally, based on if the current platform supports compiling it.   If not present, SQLAlchemy continues to work perfectly well as the hooks which attempt to import it will fall back to pure-Python functions instead." So you may have a cprocessors.py which I dont think you'd get the problem, only if it compiled the .so when sqlalchemy installed.
> 
> I had the cprocessors extension in my build (that is, py2app mentioned in copied the extension)
> 
>> 
>> I get the error, but only when it builds the second app. In my main build script I make a few apps in the same script (I make 3 apps which get moved into the main app, any additional code in their site-packages.zip is moved into the main apps zip, I remove the "sub-apps" Contents/Resources/lib folder and symlink it at run time to the main apps lib folder.)
>> 
>> Is this a bug or are you never supposed to run multiple setups in the same build? If not how can I achieve the above?
> 
> Calling distutils.setup multiple times is at best untested with py2app, and I wouldn't be surprised if it causes problems due to state leaking from one build into the next.  A workaround would be to use the subprocess module to run the setup jobs in separate processes.
> 
> 
> Isn't the problem that they share dist folders, not a process? if not where does the state exist? Would I need to subprocess them from different directories?

The py2app command itself has state and I haven't reviewed the code to know for sure that all state is cleaned up when the command is used twice in the same process. BTW. I also don't know if distutils.setup creates a new py2app command object every time it is called, if the second call to distutils.setup creates a new py2app command object there is no information leakage.

>  
> BTW. I don't quite understand what you are trying to do with these 3 apps.  Are you building 3 apps that share a lot of code/resources and where you want two of the apps to link to the 3th one to reduce the amount of disk-space used? 
> 
> 
> Yea exactly, I have some smaller apps which are used for specific separate jobs (one has a simple gui and generates and gathers log files from the main app and zips them up should the main app ever fail to open for instance), the jobs are all to do with the main app and all use a sub set of code to the main app, so I put the apps in the Resources folder and symlink the lib folder so I can include them with only using a little extra disk space, but more importantly keeping the installer size down.

That sounds like something that would be useful to support directly. I'll add it to the list of nice-to-have featuers, but don't know when I'll get around to looking into this.

Ronald

>  
> Ronald
>> 
>> On 10 September 2012 13:18, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>> 
>> On 9 Sep, 2012, at 20:34, Paul Wiseman <poalman at gmail.com> wrote:
>> 
>>> Hey,
>>> 
>>> When building an app that is using sqlalchemy I get this error:
>>> 
>>> creating python loader for extension 'sqlalchemy.cprocessors'
>>> error: /Users/paul/Source/Python/build/bdist.macosx-10.6-intel/python2.7-standalone/app/temp/sqlalchemy/cprocessors.py: No such file or directory
>>> 
>>> I took a look in site packages and there is no cprocessors.py, but a cprocessors.so - so maybe it is just looking for the wrong extension
>>> 
>>> I tried adding "sqlalchemy.cprocessors" to the includes list in py2app but that hasn't helped.
>>> 
>>> I was wondering if I can fool it by dropping an empty cprocessors.py so it will build, then swap it out afterwards for the so, but I'm sure there's a better way and I'm not convinced that could even work.
>>> 
>>> Surely py2app doesn't assume every extension is .py, or if it does can it be changed?
>> 
>> Py2app does not assume that every extension is a python file. Given the messasge I'd say that the error occurs in the code path that creates a helper python file that actually loads the exention.
>> 
>> A little background information: when py2app creates the application bundle all modules are stored in a zipfile and loaded using python's zipimporter. Extensions cannot be stored in the zipfiles because the zipimporter doesn't support that. Py2app therefore creates a placeholder python module in the zipfile that loads the extensions from a directory in the application bundle. 
>> 
>> BTW. could you please create a sample project that demonstrates the problem? I've tried to reproduce your problem on my machine and failed to do so. I did run into another problem, py2app generated an incomplete bundle due to confusion between a _collections submodule in SQLAlchemy and the _collections extension in the stdlib; that's something I'm currently trying to fix.
>> 
>> Ronald
>>> _______________________________________________
>>> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
>>> http://mail.python.org/mailman/listinfo/pythonmac-sig
>>> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pythonmac-sig/attachments/20120913/3486dc7d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4788 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pythonmac-sig/attachments/20120913/3486dc7d/attachment.bin>