[Twisted-Python] Re-working a synchronous iterator to use Twisted

[Jean-Paul Calderone]

On Tue, 17 Jun 2008 23:10:48 +0200, Terry Jones <terry at jon.es> wrote:
>For the record, here's a followup to my own posting, with working code.
>The earlier untested code was a bit of a mess. The below runs fine.
>
>In case it wasn't clear before, you're pulling "results" (e.g., from a
>search engine) in off the web. Each results pages comes with an indicator
>to tell you whether there are more results. I wanted to write a function
>(see processResults below) that, when called, would call the process
>function below on each result, all done asynchronously.
>
>This solution feels cumbersome, but it does work (aka prints the expected
>output).
>
>Comments welcome (just don't tell me to use version control :-))
>

I suspect the implementation could be slightly simplified, but it doesn't
look too bad as it is now (and I'm feeling slightly too lazy to back that
hunch up with code).  I did want to point out that you're missing one
tiny detail.

> [snip]
>
># ASYNCHRONOUS calling
>def processResults(uri):
>    def cb((resultIterator, deferred)):
>        for result in resultIterator:
>            process(result)
>        if deferred is not None:
>            deferred.addCallback(cb)
>    return getResults(uri).addCallback(cb)
>

Here, the returned Deferred will fire as soon as `cb´ has a result.  The
return value of `cb´ is always `None´ though, so `cb´ will have a result
synchronously in all cases.  This is incorrect for the case where there
are more results coming.  Your example produced the correct output anyway,
since all of your Deferreds are created already having results.  If you
had an "asynchronous" Deferred, you'd see your "finished" message before
process had been called on all results.

There are two possible solutions to this.  The simpler one is to return
`deferred´ from `cb´.  The problem this has is that it builds up a chain
of unbounded length which may ultimately encounter a limitation in the
implementation of Deferred, hitting the Python stack depth limit and then
failing with a RuntimeError.

The only slightly more complex one is to create a new Deferred in
`processResults´ and return it.  Then, fire it inside `cb´ when `deferred´
is None.  You also need to be slightly careful to hook up the errback
chain in this case, so that if there's some problem with getting an
iterator the app-facing Deferred gets errbacked.

Jean-Paul