Python + Shoutpy + Twisted Locks

[John Nagle]

Chris Mellon wrote:
> On 10/7/07, Michel Albert <exhuma at gmail.com> wrote:
>> On Oct 6, 4:21 am, "Gabriel Genellina" <gagsl-... at yahoo.com.ar> wrote:
>>> En Fri, 05 Oct 2007 04:55:55 -0300, exhuma.twn <exh... at gmail.com> escribi?:
>>>
>>>> [...] What I found
>>>> is that "libshout" is blocking, which should be fine as the whole
>>>> thing runs in it's separate thread. But the application hangs
>>>> nevertheless while streaming. This effectively blocks out the other
>>>> thread that checks the player status, which then fails to append new
>>>> songs to the queue. So only one song is played when streaming.
>>>> The other threads in my application run fine and don't block the rest
>>>> of the app. So I guess, that the main problem is that blocking occurs
>>>> "outside" the python world and "inside" the libshout world.
>>> Only one thread at a time may be executing Python code; the Global
>>> Interpreter Lock (GIL) ensures the mutual exclusion. Extension modules
>>> (written in C) may use the macros
>>> Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to release/acquire the GIL
>>> before/after an external blocking call.
>>> I don't know libshout, or how you are doing the binding python-libshout,
>>> but if your analysis is correct it means that the code is not releasing
>>> the GIL at the appropiate points.
>>>
>>> --
>>> Gabriel Genellina
>> Hmmm... ok. I suppose rewriting the whole thing using twisted's
>> deferreds could then solve the problem. Which are basically nothing
>> more than callbacks with a weird name ;) Unfortunately this means that
>> I have to re-think a lot. But in the end I suppose it will pay off.
>>
>> Thanks for taking the time and reading my little essay Gabriel ;)
>>
> 
> Using Twisted won't help if the libshout calls are really blocking the
> main thread.

     Right.

     The whole point of Twisted is that you can't do anything that takes
very long in a Twisted thread, or the whole server will stall.  If you
try to do something ongoing, like streaming, in a Twisted server, you
have to have long-running threads of your own and a scheduling system
to coordinate them with the Twisted threads.

     libshout's "nonblocking" mode isn't very nonblocking.  See

https://trac.xiph.org/browser/icecast/trunk/libshout/examples/nonblocking.c

There's a thread in a loop during playout.  All "nonblocking" seems to do is
defer the waiting from shout_write() to shout_sync().

     It looks like you'll need a separate thread for each stream being played
out, on top of the Twisted machinery.  Or a completely event-driven version
of "libshout" designed for Twisted.

				John Nagle