[IPython-dev] Macros don't work in threaded shells

[Fernando Perez]

On Mon, Mar 10, 2008 at 11:53 AM, Ville M. Vainio <vivainio at gmail.com> wrote:
> It appears recursive execution of _ip.runlines() is not possible with
>  threaded shells (and so e.g. macros hang ipython) - noticed this
>  through ticket #238, which is now a blocker.
>
>  Try:
>
>  ipython -q4thread
>
>  > _ip.runlines('1')
>
>  This will hang IPython.
>
>  First, we come to MTInteractiveShell.runsource(""" _ip.runlines('1')""")
>
>  This is given to runcode, which then does runsource( "1"). This never
>  gets to runcode, because the thread is waiting for  _ip.runlines('1')
>  to complete.
>
>  According to this comment, it seems this scenario has been considered,
>  but the logic does not seem to work:
>
>        # Case 3
>         # Store code in queue, so the execution thread can handle it.
>
>         # Note that with macros and other applications, we MAY re-enter this
>         # section, so we have to acquire the lock with non-blocking semantics,
>         # else we deadlock.
>         got_lock = self.thread_ready.acquire()
>         self.code_queue.put(code)
>         if got_lock:
>            self.thread_ready.wait()  # Wait until processed in timeout interval
>         else:
>             print "nolock",source
>         self.thread_ready.release()
>
>  (and no, using got_lock = self.thread_ready.acquire(blocking=False)
>  won't fix it).
>
>  Fernando? You remember whether this used to work?

Mmh, honestly I don't.  This is the problem with not having had a
solid test suite ever, we can't even track regressions reliably.

That code did change recently:

http://projects.scipy.org/ipython/ipython/ticket/210
http://projects.scipy.org/ipython/ipython/ticket/212

and those changes did fix existing deadlocks, but they may have
introduced new ones, or simply exposed other latent ones.  I'm afraid
by now I've shifted my attention enough away from this code that I'd
rather just push on the new architecture and leave these in as is,
unless someone (marc?) can contribute a good fix.  All new code we're
writing is nicely factored out and properly tested, and I've finally
made a mental shift to the new codebase to the point where I prefer
not to work on back fixes unless they are quick and easy.
Unfortunately 'quick and easy' does most definitely NOT apply to the
threaded logic for all the MT shells :(

In summary, if a good samaritan pitches in  with a fix I'd be very
grateful, but my focus now is 99% on the new code (quick fixes like
last night's l?? one aside).

Cheers,

f