[Python-Dev] Pythonic concurrency

[Shane Hathaway]

Antoine Pitrou wrote:
>>I'd be happy to explain how 
>>ZODB solves those problems, if you're interested.
> 
> 
> Well, yes, I'm interested :)
> (I don't anything about Zope internals though, and I've never even used
> it)

Ok.  Quoting your list:

 > To apply the same thing to Python you would at least need :
 >   1. a way to define a subset of the current bag of reachable objects
 > which has to stay consistent w.r.t. transactions that are applied
 > to it (of course, you would have several such subsets in any
 > non-trivial application)

ZODB holds a tree of objects.  When you add an attribute to an object 
managed by ZODB, you're expanding the tree.  Consistency comes from 
several features:

   - Each thread has its own lazy copy of the object tree.

   - The application doesn't see changes to the object tree except at 
transaction boundaries.

   - The ZODB store keeps old revisions, and the new MVCC feature lets 
the application see the object system as it was at the beginning of the 
transaction.

   - If you make a change to the object tree that conflicts with a 
concurrent change, all changes to that copy of the object tree are aborted.

 >   2. a way to start and end a transaction on a bag of objects (begin /
 > commit / rollback)

ZODB includes a transaction module that does just that.  In fact, the 
module is so useful that I think it belongs in the standard library.

 >   3. a precise definition of the semantics of "consistency" here : for
 > example, only one thread could modify a bag of objects at any given
 > time, and other threads would continue to see the frozen,
 > stable version of that bag until the next version is committed by the
 > writing thread

As mentioned above, the key is that ZODB maintains a copy of the objects 
per thread.  A fair amount of RAM is lost that way, but the benefit in 
simplicity is tremendous.

You also talked about the risk that applications would accidentally pull 
a lot of objects into the tree just by setting an attribute.  That can 
and does happen, but the most common case is already solved by the 
pickle machinery: if you pickle something global like a class, the 
pickle stores the name and location of the class instead of the class 
itself.

Shane