[Pythonmac-SIG] Python threading.Thread vs PyObjC performOnMainThread

[Bob Ippolito]

On May 16, 2005, at 2:53 PM, gandreas at gandreas.com wrote:

>
> On May 16, 2005, at 1:22 PM, Bob Ippolito wrote:
>
>
>>
>> On May 16, 2005, at 2:01 PM, gandreas at gandreas.com wrote:
>>
>>> The debugged app works like that, but the way that the "remote
>>> object" interface works is also designed to use sync TCP on the host
>>> side. Since it's pure python, I didn't want to start adding in
>>> NSFileHandles, and the obvious solution was "just use threads", and
>>> the whole "block on reads" model was much cleaner than any sort of
>>> polling.
>>
>> This isn't a very good argument since you're already using plenty  
>> of Objective-C code in there.
>
> My goal was to try to not make it worse... (since part of the goal  
> was to be able to provide a debugging stub that could be included  
> in anything that embedded python, regardless if it was PyObjC based  
> or not - or even Mac OS X based).

The important part is to make what sits inside the running program  
pure python with no non-stdlib dependencies.  The part that lives in  
the IDE is going to have so much IDE specific code in it that it  
doesn't really matter whether you're using pure python or not.

>>>> If you need a way to interrupt the debugged app asynchronously
>>>> (i.e. ctrl-C in gdb), then set up a signal handler in the debugged
>>>> app that installs a trace hook from the debugging bootstrap code
>>>> that makes it go into debugging mode and send it a signal from the
>>>> host app... but that's not something that's ever very clean in  
>>>> Python.
>>>>
>>>>
>>>
>>> I haven't quite figured out a good way to handle that one either,  
>>> but
>>> since the other app is a child, sending a signal should work as good
>>> as anything.
>>>
>>>
>>
>> There's a "you probably shouldn't use this" function in the C API  
>> that lets you write an extension module that sends an exception to  
>> another thread, so in theory you could start a background thread  
>> that sits on this and waits for a "signal" over TCP or the like  
>> from the parent and then uses that to break into the main thread.   
>> Don't do this though, there's a lot of reasons not to.
>>
>> Alternatively, you could start up a second socket on another  
>> thread and use it to send a signal to its own pid if it hears from  
>> the host.  This would facilitate true remote debugging.
>>
>>
>
> I  should be able to directly send the signal from the IDE to the  
> targeted app (since it's a child of the IDE), and the standard  
> "catch a unix signal invokes the debugger" action should work  
> (since the debugger interface on the client talks back to the IDE).

Yeah, this will work fine.  I'm just saying, that if you wanted  
something even more decoupled (to implement remote debugging, plugin  
debugging, objc.inject debugging, etc.), then that's how you could do  
it.

>>>> Update to PyObjC from svn and see if that makes a difference.
>>>>
>>>>
>>>
>>> Can't do that - the goal is to provide something that, at worse,
>>> requires the user to install a given binary package (PyOXIDE is
>>> designed to directly call packman if it detects PyObjC not being
>>> installed, to make things as transparent as possible).  If I require
>>> the user to download the latest sources and compile it, that becomes
>>> a nightmare.  If I compile it myself  from "latest beta" and  
>>> included
>>> it with PyOXIDE, that prevents bug fixed releases (since PyOXIDE
>>> would be hardcoded to run on the older, buggier, pre-release  
>>> version).
>>>
>>> There's already far too much "well, it will all work great, but  
>>> first
>>> you need to download this (and then compile that, and install this
>>> third thing)..." in the MacPython world that it makes me cringe (and
>>> certainly the choice of versions of Python included with the OS  
>>> don't
>>> make the problem less either).  Unfortunately, I can't think up a
>>> good way to solve that problem so I just cringe a lot and try to not
>>> make anything worse...
>>
>> Well that's too bad.  What you're suggesting will make things much  
>> worse.
>>
>> The Python interpreter used by your application should be private  
>> anyway, so you can and should include PyObjC (probably even Python  
>> itself) inside your application.
>>
>
> Including an entire "custom" version of Python just seems like a  
> bad solution to me.  It may be the only solution, depending on  
> various other circumstances, but it seems kind of sad to say "well,  
> OS X ships with Python.framework, but it can't be used to write a  
> Python IDE".

Mac OS X 10.3 and 10.4 ship with a Python 2.3 framework.  Mac OS X  
10.5 (hopefully) will not.  Do you want to have to maintain separate  
versions of your app for 10.3-10.4 and 10.5, so as to use the  
standard Python?

Additionally, do you want to test everything on Python 2.3.0 and  
Python 2.3.5?  What if you run into a bug that can't be monkeypatched  
for Python 2.3.0?  There's so many reasons NOT to use the framework  
that Apple provides, because they do a really poor job maintaining  
it, and provide no guarantee of backwards or forward compatibility of  
any sort... unlike their own frameworks.

>> I've said this a ton of times, but the ONLY time you should EVER  
>> run user Python code from an IDE is from separate processes,  
>> unless it's simply a plugin for the IDE.
>
> I personally don't agree with this.  There are plenty of times when  
> I just need something to "tinker" with, and the separate process  
> model doesn't work well because I find the ability to have a semi- 
> persistent state extremely useful (run the script, examine the  
> results using an interactive window, manually call some of the  
> routines, fix a routine and run that source file again, etc... -  
> not to mention that examining the state of objects within the  
> application is _much_ faster than having to proxy everything  
> through a socket).  Now if there were a "persistent external" mode,  
> that would be another story (but that's more a "future feature").

Semi-persistent state can be had just as easily with an external  
interpreter using the same methodology you're using for a debugging  
stub.  If the interpreter is already open, then just call execfile in  
it instead of tearing it down and starting another one.  No problem  
there, and you don't have to worry about forcing the user to use the  
IDE's version of Python nor do you have to worry about the IDE  
crashing, hanging, becoming unstable, blowing up, etc. because the  
user decided to call NSApp().terminate_(None) or something.

As far as speed goes.  If it's noticeably slower than you're doing  
something wrong.  Safe and correct is better than fast, anyway,  
ESPECIALLY for an IDE.  The last thing you want is the IDE to crash  
in the middle of doing something.  As far as I've seen here, the  
major complaint with your PyOXIDE is that it is not stable, so  
perhaps you should change your viewpoint on issues like this.

Speed is easy enough to fix anyway, there are plenty of ways you  
could put more code in the stub or use mmap to transfer memory or  
whatever else as an *optimization after it already works correctly*  
without compromising the integrity of the host application.


>>> After a good burrito for lunch, I'm thinking that adding an
>>> unlockPythonAndPerformSelectorOnMainThread(andRelockAfterwards)
>>> method might be the best way to handle this - at least it's  
>>> something
>>> easy that I can add as part of PyOXIDE...
>>
>> NO!  That is really really horribly bad to do.  Working around a  
>> bug in PyObjC by writing code that will break in correct versions  
>> of PyObjC is absolutely horrible.  You should NEVER release the  
>> lock unless you own it.
>
> Here, then, is probably the heart of the problem - the interactions  
> between python threads spawned by threading.Thread and the main  
> (objc) thread and performSelectorOnMainThread are "undefined", and  
> the more I look at it, the more it appears that using  
> threading.Thread in a PyObjC app is a bad idea and should probably  
> just be chalked up as "unsupported and probably won't work".  It  
> appears that I'd need to build something on top of NSThread (in  
> Objective-C) with explicit thread state handling to get this to  
> work consistently and correctly.

Well, that's simply not true.  PyObjC svn + Python 2.4.1 definitely  
does not have this problem.  There was a thread about this (aptly  
named "Threading") on pyobjc-dev in March/April.  I suggest you  
subscribe there if you haven't already.

The issue you're experiencing is a bug in Python 2.3.0 (another  
reason not to use Apple's Python framework on Panther).  This is in  
the interpreter of course, so it can't be fixed in-place and Apple  
sure isn't going to update it.  Python 2.4.1 and reportedly Python  
2.3.5 do not have this bug.  Yet another /really/ good reason not to  
use the Python framework that ships with Mac OS X 10.3!

The only edge case, of course, is that Foundation must be in  
"threaded" mode before you start calling into objc from the second  
thread, which generally happens behind the scenes anyway because  
you're using AppKit which will start a thread or two off the bat.   
This is a general restriction of using "raw" pthreads from any  
Foundation app.  PyObjC should probably spawn a NSThread during  
initialization, I'll go ahead and do that before the 1.3.5 release.

-bob