[IPython-dev] question about decision to use twisted

[Darren Dale]

Hi Brian,

On Tuesday 08 July 2008 03:57:39 pm Brian Granger wrote:
> > I have a data acquisition and analysis project that currently uses
> > Parallel Python for distributed processing and PyQt4's for some threading
> > and event handling. I am concidering the processing module scheduled for
> > inclusion in the python standard library starting with version 2.6, and
> > also I am trying to get my head around twisted.
>
> Twisted is a much lower level solution than parallel python or
> processing.  Are you thinking about using Twisted directly or just
> using it through IPython (ipython1 is now in trunk IPython). 
(typo corrected)

I was thinking of using twisted directly, but I guess I really need to dig 
into ipython1. I don't have a sense of how I would use ipython1 in a 
non-interactive way, is there some discussion in the docs or some examples of 
scripting with ipython1? Is there a callback mechanism in ipython1 to respond 
to non-blocking events?

> Unless 
> you really want the treading API that processing provides, I think
> IPython is by far the best solution [I fully acknowledge my bias here]
> for high-level parallelism.
>
> > I would like to know what the ipython developers concerns were at the
> > time the decision was made to use twisted in ipython1, was there some
> > discussion on the ipython mailing lists? Any advice or comments would be
> > greatly appreciated.
>
> I don't think we really discussed this on the mailing list, it was
> probably private discussion between Fernando and myself.  We went with
> Twisted as:
>
> 1) we didn't want to reinvent Twisted
>
> 2) we really need what Twisted provides, so the temptation to reinvent
> it was great.
>
> 3) Twisted has the right abstractions that force you to write correct
> an robust networking code.  Of course you can do this without Twisted,
> but you are going to work much harder and probably get it wrong in
> subtle ways that are hard to figure out.
>
> We have been extremely happy about going with Twisted - it is our secret
> sauce.
>
> Are there any specific questions you have about Twisted?

After reading some positive things about twisted, I went looking for a couple 
negative reviews. I've read that twisted is a bit of a moving target, and the 
twisted jargon is pretty unfamiliar to me. So finally I was hoping to temper 
all this information with the informed opinion from a group I know and 
respect. This is just the kind of feedback I was hoping to get.

Thank you,
Darren