Event Driven programming - Doubts

[James Mills]

On Tue, Dec 23, 2008 at 12:57 AM, Kottiyath <n.kottiyath at gmail.com> wrote:
> Hi,
>    I have been looking at Twisted and lately Circuits as examples for
> event driven programming in Python.

Wonderful! :) "circuits" that is :)

>    Even though I understood how to implement the code in these and
> what is deferred etc, I have not yet understood the implementation of
> deferred. I went through a lot of tutorials, but I guess most places
> they expect that the user already understands how events are
> generated. The tutorials mention that there is no more threads once
> twisted is used.

deferred to me is nothing more than
scheduling "things" to happen "later".

ie: Timers, or Timed Events.

I clearly biased here as I'm the developer
of circuits, so I can't comment on Twisted's
design or architecture, but the way to
achieve Twisted's so-called deferred (events?)
is to use the Timer component, Timed Events.

>    My question is as follows:
>    I have not understood how the callbacks are hit without (a)
> blocking the code or (b) having new threads.

Again speaking in terms of circuits,
but also in the event-driven paradigm:

a) Event Handlers _can_ block - but ideally shouldn't (ihmo).
b) Forking new threads/processes per event is mostly not necessary.
(There are exceptions).

> The usual example given is that of a program waiting for data coming
> through a socket. In the tutorials, it is mentioned that -in an event
> driven program, we schedule the code to hit when the remote server
> gets back to us - .
> Now, my question is - somebody has to still wait on that socket and
> check whether the data is received, and once all the data is received,
> call the appropriate callbacks.

In the case of circuits' TCPServer component
you simply poll it in your main event-loop.
The pattern looks like this:

from circuits.lib.sockets import TCPServer

server = TCPServer(8000)
while True:
   server.poll()
   server.flush()

The TCPServer component has various builtin
event handlers that do all the work for you
and expose other more useful events to
the application, such as:
 * connect
 * disconnect
 * read
 * error

> Is twisted creating a micro-thread which just waits on the socket and
> once the data is received, calls callFromThread for it to run on the
> main loop?

I can't comment - I tend to avoid threads
where ever possible.

> If so, Even though data locking etc is not a problem, are we not still
> having threads? Will it not still cause scalability problems in high
> traffic?

Regarding scalability btw ... (in case you're interested)
circuits comes with circuits.lib.web and several
sets of Web Components. A lot of code was
borrowed from the BaseHTTPServer from the
python standard library and bits and pieces
from CherryPy and the cgi module.... In terms
of performance, it has a "raw" performance
~3k req/s on good hardware.

Do have fun in your endeavours :)

cheers
James

Circuits: http://trac.softcircuit.com.au/circuits/