[Python-Dev] STM and python

[Armin Rigo]

Hi,

On Thu, Dec 1, 2011 at 07:06, Matt Joiner <anacrolix at gmail.com> wrote:
> I saw this, I believe it just exposes an STM primitive to user code.
> It doesn't make use of STM for Python internals.

That's correct.

> Explicit STM doesn't seem particularly useful for a language that
> doesn't expose raw memory in its normal usage.

In my opinion, that sentence could not be more wrong.

It is true that, as I discuss on the blog post cited a few times in
this thread, the first goal I see is to use STM to replace the GIL as
an internal way of keeping the state of the interpreter consistent.
This could quite possibly be achieved using the new GCC
__transaction_atomic keyword, although I see already annoying issues
(e.g. the keyword can only protect a _syntactically nested_ piece of
code as a transaction).

However there is another aspect: user-exposed STM, which I didn't
explore much.  While it is potentially even more important, it is a
language design question, so I'm happy to delegate it to python-dev.
In my opinion, explicit STM (like Clojure) is not only *a* way to
write multithreaded Python programs, but it seems to be *the only* way
that really makes sense in general, for more than small examples and
more than examples where other hacks are enough (see
http://en.wikipedia.org/wiki/Software_transactional_memory#Composable_operations
).  In other words, locks are low-level and should not be used in a
high-level language, like direct memory accesses, just because it
forces the programmer to think about increasingly complicated
situations.

And of course there is the background idea that TM might be available
in hardware someday.  My own guess is that it will occur, and I bet
that in 5 to 10 years all new Intel and AMD CPUs will have Hybrid TM.
On such hardware, the performance penalty mostly disappears (which is
also, I guess, the reasoning behind GCC 4.7, offering a future path to
use Hybrid TM).

If python-dev people are interested in exploring the language design
space in that direction, I would be most happy to look in more detail
at GCC 4.7.  If we manage to make use of it, then we could get a
version of CPython using STM internally with a very minimal patch.  If
it seems useful we can then turn that patch into #ifdefs into the
normal CPython.  It would of course be off by default because of the
performance hit; still, it would give an optional alternate
"CPythonSTM" to play with in order to come up with good user-level
abstractions.  (This is what I'm already trying to do with PyPy
without using GCC 4.7, and it's progressing nicely.)  (My existing
patch to CPython emulating user-level STM with the GIL is not really
satisfying, also for the reason that it cannot emulate some other
potentially useful user constructs, like abort_and_retry().)


A bientôt,

Armin.