[python-uk] [pyconuk] Minimalistic software transactional memory

[Michael Sparks]

On Wednesday 12 December 2007 08:43:14 Richard Taylor wrote:
> I like to use explicit read locks for shared data structures, mainly
> because it makes it much safer when someone comes along and adds
> functionality to the methods later on. 

I've had a think about this - especially with regard to reading locks - and
thought in more detail about how the functions operate and what API is
likely to get used.  In the end I came to the conclusion that the problem
areas are (naturally) these parts:

class Store(object):
    def usevar(self, key):
    def set(self, key, value):
    def using(self, *keys):
    def set_values(self, D):

The thing here all of these functions have in common is they are all
read-write on the state (eg creating non-pre-existing vars). So rather than
having separate read & write locks, I've decided on a single lock for
read-write.

The API for store has also changed such that it has the following methods as 
well:
    def __get(self, key):
    def __make(self, key):
    def __do_update(self, key, value):
    def __can_update(self,key, value):

These however assume the store is locked by the caller. (incidentally, I split 
a conceptual __try_update into __can & __do since that makes __do_update 
reusable. (DRY principle) ) They're marked private to indicate to a user that 
using these outside the class could easily cause problems.

I agree generally speaking about using decorators, however, this is intended 
to be used in Kamaelia's core concurrency framework "Axon". At present that 
code is largely python 2.2.late compatible, meaning that it should be 
portable to Nokia Series 60 phones as well as Jython. The other aspect is we 
have very very minimal reimplementations of the core in Ruby & C++ (as well 
as a shedskin compiled version), and I've even seen a Java reimplementation 
of the core as well.... 

So decorators - nice as they are (and relevant here) - are something I try and 
avoid - in the core at least. Using just the one lock though simplfies the 
boiler plating, and also I feel dramatically reduces the chances of any form 
of lock up based on trying to access the STM.

> > It's interesting though, after having developed large amounts of code
> > based on no-shared-data & read-only/write-only pipes with data handoff
> > and not having had any major concurrency issues (despite mixing threads
> > and non threads) switching to a shared data model instantly causes
> > problems.
> >
> > The difference is really stark. One is simple, natural and easy and the
> > other is subtle & problematic. I'm not shocked, but find it amusing :-)
>
> I agree. Where possible we use 'pipe and filter' or 'message queue'
> patterns for inter-thread communication. We have a library based around the
> MASCOT design method that uses concepts of 'queues' and 'pools', which has
> proved very powerful for large multi-threaded applications. 

I'd not heard of MASCOT - I'll take a look into it. It sounds very similar to 
what we currently do in Kamaelia. I suspect pools maybe similar to what we 
call backplanes.

> We have found 
> that the best way to avoid thread related problems is to indulge in a
> little design work to explicitly limit all thread communication to a few
> well thought through mechanisms and then police their use rigidly.

Indeed. That's the main reason for slowly progressing this STM code :-)

I think the latest iteration is now minimally sufficiently safe. (ie safe from 
user code for the 4 main calls a user can/will do) The current iteration of 
the code can be found here:

https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_Scratch/Bindings/STM/Axon/STM.py

Many thanks for the feedback - much appreciated!

Regards,


Michael.
--
http://yeoldeclue.com/blog
http://kamaelia.sourceforge.net/Developers/