[pypy-commit] extradoc extradoc: Expand here and there.

[arigo]

Author: Armin Rigo <arigo at tunes.org>
Branch: extradoc
Changeset: r4202:7f5d0a8f1a66
Date: 2012-04-27 09:46 +0200
http://bitbucket.org/pypy/extradoc/changeset/7f5d0a8f1a66/

Log:	Expand here and there.

diff --git a/blog/draft/stm-apr2012.rst b/blog/draft/stm-apr2012.rst
--- a/blog/draft/stm-apr2012.rst
+++ b/blog/draft/stm-apr2012.rst
@@ -1,47 +1,75 @@
 STM update (and thanks everybody)
 =================================
 
-A short update on the Software Transactional Memory (STM) side.  I have
-now reached the point where the basics seem to work, and the integration
-with the Garbage Collection subsystem is --- not done by far, but at
-least "not crashing in my simple tests and not leaking memory too
-quickly".  (It is leaking a bit of memory though, and it is never
-calling ``__del__`` so far.)
+A short update on the Software Transactional Memory (STM) side.  Let me
+remind you that the work is to add STM internally into PyPy, with the
+goal of letting the user's programs run on multiple cores after a minor
+adaptation.  (The goal is not to expose STM to the user's program.)  I
+will soon write some official documentation that explains in more
+details exactly what you get.  For now you can read the previous blog__
+posts__, and you can also find technical details in the `call for
+donation`_ itself; or directly look at how I adapted the examples linked
+to later in this post.
+
+.. _`call for donation`: http://pypy.org/tmdonate.html
+.. __: http://morepypy.blogspot.com/2012/03/call-for-donations-for-software.html
+.. __: http://morepypy.blogspot.com/2012/01/transactional-memory-ii.html
+
+I have now reached the point where the basics seem to work.  There is no
+integration with the JIT so far; moreover the integration with the
+Garbage Collection subsystem is not finished right now, but at least
+"not crashing in my simple tests and not leaking memory too quickly".
+(It means that it is never calling ``__del__`` so far, although it
+releases memory; and when entering transactional mode or when going to
+the next transaction, all live objects become immortal.  This should
+still let most not-too-long-running programs work.)
 
 If you want to play with it, you can download `this binary`_ (you need to
 put it in a place with the paths ``lib-python`` and ``lib_pypy``, for
 example inside the main directory from a regular `nightly tarball`_
-or from a full checkout).  It
-is for Linux x86 32-bit.
-This version was compiled from the `stm-gc`_ branch on the 25th of April.
-It runs e.g. the `modified version of richards`_.
-This branch could also be translated for Linux x86-64,
-but I didn't do fixes for other platforms for now.
+or from a full checkout).
+This version was compiled for Linux x86 32-bit from the `stm-gc`_ branch
+on the 25th of April.  It runs e.g. the modified version of richards_.
+This branch could also be translated for Linux x86-64, but not for
+other OSes nor other CPUs for now.
 
 .. _`this binary`: http://wyvern.cs.uni-duesseldorf.de/~arigo/pypy-stm-22fccf3c9b5e.tar.bz2
 .. _`nightly tarball`: http://buildbot.pypy.org/nightly/trunk/
 .. _`stm-gc`: https://bitbucket.org/pypy/pypy/src/stm-gc
-.. _`modified version of richards`: https://bitbucket.org/pypy/pypy/raw/stm-gc/pypy/translator/stm/test/richards.py
+.. _richards: https://bitbucket.org/pypy/pypy/raw/stm-gc/pypy/translator/stm/test/richards.py
 
-It exposes the same interface as the pure Python transaction_ module
-(except of course that it's not using the naive implementation linked
-above, but its own).  A difference is that it
-doesn't support epoll right now, so it cannot be used to play with `a
-branch of Twisted`_; but that's coming soon.  For now you can use it to
-get multi-core usage on purely computational programs, like PyPy's own
-``translate.py`` --- for that I did a tweak `in rpython/rtyper.py`_ (lines
-273-281 are all that I needed to add), but there are a few more places
-in the whole ``translate.py`` that could be similarly enhanced.
+The resulting ``pypy-stm`` exposes the same interface as the pure Python
+transaction_ module, which is an emulator (running on CPython or any
+version of PyPy) which can be used to play around and prepare your
+programs.  See the comments in there.  A difference is that the real
+``pypy-stm`` doesn't support epoll right now, so it cannot be used yet
+to play with `a branch of Twisted`_ that was already adapted (thanks
+Jean-Paul Calderone); but that's coming soon.  For now you can use it to
+get multi-core usage on purely computational programs.
+
+I did for example adapt PyPy's own ``translate.py``: see the tweak `in
+rpython/rtyper.py`_.  Lines 273-281 are all that I needed to add, and
+they are mostly a "simplification and parallelization" of the lines
+above.  There are a few more places in the whole ``translate.py`` that
+could be similarly modified, but overall it is just that: a few places.
+I did not measure performance, but I checked that it is capable of using
+multiple cores in the RTyping step of translation, with --- as expected
+--- some still-reasonable number of conflicts, particularly at the
+beginning when shared data structures are still being built.
 
 .. _transaction: https://bitbucket.org/pypy/pypy/raw/stm-gc/lib_pypy/transaction.py
 .. _`a branch of Twisted`: svn://svn.twistedmatrix.com/svn/Twisted/branches/stm-5526
 .. _`in rpython/rtyper.py`: https://bitbucket.org/pypy/pypy/src/stm-gc/pypy/rpython/rtyper.py#cl-249
 
-The performance is not great, even taking into account that it has no
-JIT so far, but it seems to scale; at least, it does scale on my
-2-real-cores, 4-hyperthreaded-cores laptop, roughly as expected.
+On a few smaller, more regular examples like richards_, I did measure
+the performance.  It is not great, even taking into account that it has
+no JIT so far; but it is well within the 2-to-5-times slower goal, and
+it does seem to scale.  At least, it scales roughly as expected on my
+2-real-cores, 4-hyperthreaded-cores laptop (which means that using two
+cores is not quite twice as fast as using only one, and using all four
+cores is again not twice faster but only 20-30%).
 
-And...
+And finally...
 
 ...a big thank you to everyone who contributed some money to support
 this!  As you see on the PyPy_ site, we got more than 6700$ so far in