[Python-ideas] Fwd: Concurrent safety?

[Mike Meyer]

On Thu, Nov 3, 2011 at 6:51 PM, Bruce Leban <bruce at leapyear.org> wrote:
>
> On Thu, Nov 3, 2011 at 9:38 AM, Mike Meyer <mwm at mired.org> wrote:
>>
>> Attributes are names. You don't lock names, you lock objects. To
>> prevent binding of an attribute, you lock the object it's an attribute
>> of. If you want to prevent mutating the value of the object bound to
>> the attribute, you lock that object.
>
> You conveniently ignore my comment about the importance of database row
> locking. :-)
>

I assure you it was unintentional, but there's been a lot of text
flowing here. I believe I've answered every comment, though I know
some of them were answered when first brought up, and not every time
thereafter. And some of the answers were "Yes, that's an issue" or
"Yes, but we can't keep people from writing bad code". I don't recall
anything specifically about database row locking, but if you feel it's
important enough to reiterate it, I'll treat it as equally important.

>>
>> > (3) What happens if one thread does:
>> >
>> > locking d, x: # stuff
>> >
>> > and another does
>> >
>> > locking x, d: # stuff
>>
>> That one I dealt with in the specification. There's an implementation
>> requirement that if two locking statements share objects, the shared
>> objects have to be locked in the same order. So one of the two will
>> lock things in the opposite of the order they appear in the statement.
> You're handwaving. How would that work? There's no way to know at compile
> time what the correct order is. So locking just got a whole lot more
> complicated. But it's worse than complicated: it's unworkable.

How is a requirement just handwaving? There is no "correct" order.
Given two objects, any locking statement will always lock them in the
same order. That's sufficient to prevent deadlocks. Sorting the list
passed to each value with a key of id will do the trick, but may not
be the best way to do it.

>> No, avoiding deadlock is one of the goals. That's why that requirement
>> exists, and there's a further requirement that you can only nest
>> locking statements if the inner one locks a subset of the outer one.
>
> In other words, this code is not allowed:
>
> def f():
>     locking x:
>         for y in x:
>             locking y:
>                 y.append(1)
>
> And neither is this:
>
> def f(x):
>     locking x:
>         x.append(1)
>
> if it is called inside code that has something locked. Which virtually all
> code will do since it's required everywhere. So your proposal is that I must
> lock everything before I change it but I can't lock it unless it's already
> locked. That's ridiculous. You might as well have a single semaphore to lock
> everything.

Yup, it's pretty bad. For that reason - among others - the proposal
has mutated. The original required listing objects, and either started
an STM transaction, or did explicit locks (undecided).  Someone
pointed out that the STM variant couldn't deal with IO, causing me to
note a change to the proposal wherein listing objecs caused explicit
locking, and having an empty list started an STM transaction. This is
not only easier to use, but solves several problems - including this
one. The STM variant hasn't gotten a lot of attention.

> We're wasting our time. You said above "You don't lock names, you lock
> objects." Why not? Because that was your original plan and your mind is
> closed to other ideas?

No, because I couldn't think of a way to make locking names work. You
presented a problem with locking names. Since the proposal doesn't
lock names, I figured I wasn't clear enough in the first place, and
tried fix that.

I've repeatedly said this wasn't meant to be a finished idea, and
asked for alternatives.  If you have a proposal that works by locking
names, by all means tell us about it!

> You want to ignore legitimate issues that aren't
> convenient to your proposal.

No, I don't. I want to collect them all. But I'm not perfect -
sometimes two comments may look similar when they in fact aren't.

> I hate to say things like "this idea sucks,"
> but sometimes that's the only way to put it. I think that this proposal is a
> terrible idea.

That may well be the case, and wouldn't particularly disappoint me.

> A healthy discussion about how to make concurrency better might be
> interesting but as long as all you want to do is defend your proposal, this
> thread isn't sparking useful discussion

I'm doing a lot more than defending the proposal. I'm tweaking this
proposal to solve problems people point out, collecting ideas for
alternative approaches, implementations, and tools that might help -
which are what make this discussion interesting and useful. I do
defend the proposal, especially when people point out issues that were
dealt with in the original post, which isn't interesting, but the
alternative (ignoring them) seems less useful.

     <mike