[Python-ideas] Fwd: Concurrent safety?

[Mike Meyer]

Sigh. Sent this only to bruce by accident. Sorry about the duplicates Bruce.

---------- Forwarded message ----------
From: Mike Meyer <mwm at mired.org>
Date: Mon, Oct 31, 2011 at 10:30 PM
Subject: Re: [Python-ideas] Concurrent safety?
To: Bruce Leban <bruce at leapyear.org>


On Mon, 31 Oct 2011 18:07:19 -0700
Bruce Leban <bruce at leapyear.org> wrote:
> On Mon, Oct 31, 2011 at 4:19 PM, Mike Meyer <mwm at mired.org> wrote:
> >> That would mean that all of my current code is broken.
> > Pretty much, yes. It's like adding garbage collection and removing
> > alloc*/free. It's going to break a *lot* of code. That's why I said
"not in
> > 3. and possibly never in cPython."
> In order to make concurrent code slightly safer, you're going to break all
> existing programs that don't use concurrency. That seems to me like a new
> language, not Python. You've been on this list long enough to see the
> attention that's paid to backward compatibility.

The way adding automatic memory management just made pointer
arithmetic slightly safer.

But yeah, the first thing I said was "never in 3.x, possibly never in
cPython." I've been on this list long enough to know that, while the
community pays a lot of attention to backwards compatibility, it is
willing to throw it out when there's enough benefit.

As you point out, this is a hard problem. I know I haven't covered all
the issues. That's why the second thing I said was that I'm hoping to
get people smarter than me to look at things.

The cpu manufacturers have switched to improving performance by adding
more cores instead of cranking clock speeds. As time goes by, more and
more programmers are going to want to leverage that in serious ways. I
already do, and find that Python makes some design choices unavailable
or very fragile(*). I'd like to make those choices available, and help
Python get ready for the time when that desire is the norm instead of
the exception.

> > Furthermore, merely *reading* an object that isn't locked can cause
> >> problems. This code is not thread-safe:
> >>     if element in dictionary: return dictionary[element]
> >> so you have to decide how much safety you want and what cost we're
> >> willing to pay for this.
> > You're right - it's not thread safe. However, it also doesn't suffer
from
> > the problem I'm trying to deal with, where you mutate an object in a way
> > that leaves things broken, but won't be detected at that point. If it
> > breaks because someone mutates the object underneath it, it'll throw an
> > exception at that point. I know you can construct cases where that isn't
> > so.
> I think the cases where non-thread-safe code won't throw an exception are
> numerous, for example, the equally trivial:

Again, I said such cases can be built. I *didn't* say they were
exceptions, I proposed a change to deal with them.

> If you're going to tackle thread safety, it should address more of the
> problem. These bugs are in many ways worse than mutating "an object in a
> way that leaves things broken, but won't be detected at that point." The
> above bugs may *never* be detected. I've come across bugs like that that
> were in code for many years before I found them (and I'm sure that's
> happened to others on this list as well).

Like me. That's part of why I want to get the interpreter to help find
them.

> The first thing to do is identify the problems you want to solve and make
> sure that the problems are well understood. Then design some solutions.
> Starting with a bad solution to a fraction of the problem isn't a good
> start.

I've identified the problem I want to solve: I want to make concurrent
use of python objects "safe by default", so that doing unsafe things
causes the programmer to have to do something explicit about making
things safe.  I believe this can be done at the mutation points
(again, clojure shows that it can be done). I also want to preserve as
much of Python's existing code as possible. It may be that Python's
existing data structures mean my believe about mutation points is
wrong. This may be the wrong solution. It may be that such a change is
to large to be acceptable.  But the only way to find out is to
investigate it.

This discussion has already generated significant changes in the
original proposal, plus some implementation ideas.

  <mike

*) I'll note that some of the alternatives make the choices available
in Python unavailable in a similar way.
--
Mike Meyer <mwm at mired.org>              http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111031/30ed1676/attachment.html>