[Baypiggies] Web Crawler/Backend Engineer - San Francisco, CA

[Jeff Kunce]

OK. It's not a simple messaging problem, and it's not primarily an
async IO problem.  If it were me, I'd spend a few hours researching
Twisted.  Read some of the online reviews/tutorials about it, then dig
into the real thing.  It's a pretty impressive framework.  The most
common complaint is that it has its own (somewhat "twisted")
culture/terminology that makes it hard to learn.

Spread is cool.  I've tried several times to find an excuse to use it :)

  -- Jeff

On Thu, Feb 4, 2010 at 10:28 AM, K. Richard Pixley <rich at noir.com> wrote:
> Jeff Kunce wrote:
>
> Actually, I'm wondering about the scope of the original question about
> an event-based framework:
>  - Just wanting objects to communicate with events? Something simple
> like circuits is good.
>  - async IO? asyncore, medusa et al are tried and true
>  - A comprehensive framework with everything included (even peanut
> butter and jelly)? Twisted
>
> The scope really matters.  If you just want to send simple events
> between objects in your app, you could write your own circuits-like
> framework before you could even figure out how to read the
> documentation for Twisted.
>
> I'm building a moderately complex automated builder based primarily on
> incremental builds.  Multiple servers, each parallel in their own right,
> building multiple code branches against multiple target machines.  So I've
> got three or four distinct levels of parallelism going on in the builds
> already.  And there are several types of checking going on, including
> precommit speculative testing, post commit "continuous builds", a form of
> partial building used for dependency checking, and a sort of pre-pre-commit
> checking, (essentially branched recurrences of the pre-commit checker).  The
> builder monitors a source code repository building when necessary as well as
> taking speculative build requests from users and building those.
>
> Most of this parallelism doesn't happen in python.  The python pieces of the
> builder coordinate the various builders, partition and aggregate very high
> level parallelism, manage working directories, and alerting mechanisms.
>
> My plan is a distributed, dynamic, asynchronous, fault tolerant architecture
> based on virtual synchrony, (http://en.wikipedia.org/wiki/Virtual_synchrony
> - probably the spread toolkit http://spread.org).
>
> There's also a set of status and input web pages, (javascript based on
> ext-js grid widget), and my plan is to produce their data using pylons,
> perhaps tg2, which pulls data out of the virtual synchrony "cloud", probably
> via spread, but potentially via traditional socket calls, maybe even
> jsonrpc.
>
> So the "daemon"/cloud part needs to be capable of managing distributed,
> shared state, including the work queue, through spread, as well as managing
> a few subtasks like builds.  I'm not big on threads, and they don't really
> help much in the virtual synchrony paradigm anyway.
>
> It's not a huge project, but I'm expecting it to take me at least a few
> months to implement.
>
> --rich
>
> _______________________________________________
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:
> http://mail.python.org/mailman/listinfo/baypiggies
>