[Pythonmac-SIG] Python threading.Thread vs PyObjC performOnMainThread

[Bob Ippolito]

On May 16, 2005, at 2:01 PM, gandreas at gandreas.com wrote:

> On May 16, 2005, at 11:59 AM, Bob Ippolito wrote:
>
>
>> On May 16, 2005, at 12:35 PM, gandreas at gandreas.com wrote:
>>
>>> So I'm reworking how debugging is handled in PyOXIDE and I've come
>>> to a problem (several, but this is the biggy for the morning).
>>>
>>> My goal is to separate interacting with the (remote) debugged
>>> client in a separate thread, which forwards messages to the main
>>> thread as needed to run the debugger UI (since all UI must be done
>>> on the main thread).
>>>
>>>
>> ---
>>
>>
>>> When the user says "debug", we launch the separate processes,
>>> which includes a stub to open up a TCP connection back to the IDE,
>>> which we use to interact with on the main event loop (which was
>>> ugly, since it would then nest, well, it was ugly).  So now I call
>>> the above "spawnRemoteDebugger", it talks with the targeted
>>> process, and then at some point that process says "hey, let's
>>> start debugging" (and transfers over a proxy object to allow the
>>> IDE to interact with something that looks like a pdb object).
>>> That's all more or less fine.
>>>
>>>
>>
>> I don't understand why you need to use threads from the host app.
>> The best way I've found to do this is:
>>
>> Host app (debugger/interpreter UI) uses async TCP interleaved with
>> the main runloop (i.e. NSFileHandle, or something else)
>> Debugged app (application or interpreter) uses sync TCP that blocks
>> waiting for interaction from the debugger
>>
>>
>
> The debugged app works like that, but the way that the "remote
> object" interface works is also designed to use sync TCP on the host
> side. Since it's pure python, I didn't want to start adding in
> NSFileHandles, and the obvious solution was "just use threads", and
> the whole "block on reads" model was much cleaner than any sort of
> polling.

This isn't a very good argument since you're already using plenty of  
Objective-C code in there.

Anyway, there's Twisted, asyncore/medusa, etc. for non-blocking  
Python IO... but NSFileHandle or the like is going to be the simplest  
thing that's going to work.

>> If you need a way to interrupt the debugged app asynchronously
>> (i.e. ctrl-C in gdb), then set up a signal handler in the debugged
>> app that installs a trace hook from the debugging bootstrap code
>> that makes it go into debugging mode and send it a signal from the
>> host app... but that's not something that's ever very clean in  
>> Python.
>
> I haven't quite figured out a good way to handle that one either, but
> since the other app is a child, sending a signal should work as good
> as anything.

There's a "you probably shouldn't use this" function in the C API  
that lets you write an extension module that sends an exception to  
another thread, so in theory you could start a background thread that  
sits on this and waits for a "signal" over TCP or the like from the  
parent and then uses that to break into the main thread.  Don't do  
this though, there's a lot of reasons not to.

Alternatively, you could start up a second socket on another thread  
and use it to send a signal to its own pid if it hears from the  
host.  This would facilitate true remote debugging.

>>> The problem is that we then allocate and init the debugger
>>> controller (PyDebuggerController).  Since that needs to be done on
>>> the main thread, we use performSelectorOnMainThread.  That works
>>> fine except that when this debugger controller object gets
>>> created, it eventually wants to call the python code associated
>>> said object.  Of course, since that's on the main thread (and not
>>> this other debugger handler thread) it calls
>>> PyThread_aquire_lock.  But the lock is stuck in the thread that
>>> executing the performSelectorOnMainThread line, and everything
>>> just stops (the main thread waits on the semaphore which will
>>> never be released).
>>>
>>> So, it appears that performSelectorOnMainThread needs some way to
>>> release the lock to let Python code in the main thread be
>>> executed.  Any suggestions for workarounds?
>>
>> Update to PyObjC from svn and see if that makes a difference.
>
> Can't do that - the goal is to provide something that, at worse,
> requires the user to install a given binary package (PyOXIDE is
> designed to directly call packman if it detects PyObjC not being
> installed, to make things as transparent as possible).  If I require
> the user to download the latest sources and compile it, that becomes
> a nightmare.  If I compile it myself  from "latest beta" and included
> it with PyOXIDE, that prevents bug fixed releases (since PyOXIDE
> would be hardcoded to run on the older, buggier, pre-release version).
>
> There's already far too much "well, it will all work great, but first
> you need to download this (and then compile that, and install this
> third thing)..." in the MacPython world that it makes me cringe (and
> certainly the choice of versions of Python included with the OS don't
> make the problem less either).  Unfortunately, I can't think up a
> good way to solve that problem so I just cringe a lot and try to not
> make anything worse...

Well that's too bad.  What you're suggesting will make things much  
worse.

The Python interpreter used by your application should be private  
anyway, so you can and should include PyObjC (probably even Python  
itself) inside your application.  I've said this a ton of times, but  
the ONLY time you should EVER run user Python code from an IDE is  
from separate processes, unless it's simply a plugin for the IDE.   
That said, PyObjC 1.3.5 is just about ready to release anyway.

If PyOXIDE itself needs a bug fix from a newer version of PyObjC  
(which in this case, I'm almost 100% positive it does), then you  
should update PyOXIDE with a new version of PyObjC.

> After a good burrito for lunch, I'm thinking that adding an
> unlockPythonAndPerformSelectorOnMainThread(andRelockAfterwards)
> method might be the best way to handle this - at least it's something
> easy that I can add as part of PyOXIDE...

NO!  That is really really horribly bad to do.  Working around a bug  
in PyObjC by writing code that will break in correct versions of  
PyObjC is absolutely horrible.  You should NEVER release the lock  
unless you own it.

-bob