Adding a Par construct to Python?

Wed May 20 23:10:29 EDT 2009

Quoting Carl Banks <pavlovevidence at gmail.com>:

> On May 20, 4:07 pm, Luis Zarrabeitia <ky... at uh.cu> wrote:
> > On Wednesday 20 May 2009 06:16:39 pm Aahz wrote:
> 
> The designers of Python made a design decision(**) that extension
> writers would not have to take care of locking.  They could have made
> a different decision, they just didn't.

Well, then, maybe that's the only python's decision so far I may not agree with.
And I'm not criticizing it... but I'm questioning it, because I honestly don't
understand it.

> > There must be another reason (i.e, the refcounts) to argue _for_ the GIL,
> 
> Why?

1- refcounts is a _very_ strong reason to argue for the GIL. Add to that
simplifying CPython implementation, and you don't really need another one on top
of that, at least not to convince me, but
2- in [almost] every other language, _you_ have to be aware of the critical
sections when multithreading. Even in pure python, you have use locks. Don't you
find at least a bit odd the argument that in the particular case you are writing
a C extension from python, you should be relieved of the burden of locking?

> > because this one just seems to be just an attempt to fix unsafe code when
> > called from python.
> 
> I think you are being unfair in calling it unsafe.

I think I was unfair calling it a "fix". It sounds like it was broken, my bad. I
should have used 'not multithread-ready'.

> Suppose if I were to call PyList_Append from a C extension.  It's not
> necessary for me to guard the list I'm calling it on with a lock,
> because only the GIL thread is allowed to call most Python API or
> otherwise access objects.  But you seem to be suggesting that since I
> didn't guard the list with a lock it is "unsafe", even though the GIL
> is sufficient?

Certainly not. If you program under the assumption that you have a GIL, of
course it is not unsafe. Not your code, anyway. But, why is PyList_Append not
multithread-ready? (or rather, why does PyList_Append would requiere a _global_
lock to be multithread ready, instead of a more local lock?) Of course, given
that the GIL exists, to use it is the easier solution, but for this particular
case, it feels like discarding multithreading just to avoid locking.

> No, I totally disagree.  The code is not "unsafe" and the GIL doesn't
> "fix" it.  The code is jsut

[I think there was a word missing from that sentence...)

> > And that was my point. Refcounts + interpreter simplicity
> > seem to imply the need for a GIL, but to make unsafe code safe without
> fixing
> > said code (or even thinking about it) is a weird goal...
> 
> Why?  Do you seriously not see the benefit of simplifying the work of
> extention writers and core maintainers?  You don't have to agree that
> it's a good trade-off but it's a perfectly reasonable goal.

I do agree that, at least for the core maintainers, it is a good trade-off and a
reasonable goal to keep CPython simple. And if that has, as a positive side
efect for extensions writers, that their code becomes easier, so be it. What I
don't agree - rather, what I don't understand, is why it is presented in the
opposite direction.

> I highly suspect Aahz here would argue for a GIL even without the
> refcount issue, and even though I wouldn't agree, there's nothing
> weird or unreasonable about the argument, it's just a different
> viewpoint.

I consider it weird, at least. If I were to say that python should not allow
multithreading to simplify the lives of pure-python programmers, I hope I would
be shot down and ignored. But somehow I must consider perfectly natural the idea
of not allowing[1] multithreading when building C extensions, to simplify the
lives of extension programmers.

> > specially if it
> > became the only reason for a GIL. After all, one could argue for that goal
> in
> > almost all languages.
> 
> "B-B-B-But other languages do it that way!" is not a big factor in
> language decisions in Python.

That's not what I said. We are not talking about the _language_, but about one
very specific implementation detail. Not even that, I'm talking about one of the
reasons presented in favor of that specific implementation detail (while
agreeing with the others). The fact that the reason I'm having trouble with is
valid for almost any other language, and none of them have a GIL-like construct
(while still being successful, and not being exceptionally hard to build native
modules for) just suggests that _that_ particular reason for that particular
implementation detail is not a very strong one, even if all other reasons are.

> (**) - To be fair, Python didn't originally support threads, so there
> was a lot of code that would have become unsafe had threading been
> added without the GIL, and that probably influenced the decision to
> use a GIL, but I'm sure that wasn't only reason.  Would Python have a
> GIL if threading had been there from the start?  Who knows.

(This is a very good remmark. Maybe here lies the whole answer to my question.
We may be dragging the heavy chain of backward compatibility with existing
extension modules, that is just too costly to break.)

Regards,

Luis

[1] Ok, I know that is not exactly what the GIL does.

-- 
Luis Zarrabeitia
Facultad de Matemática y Computación, UH
http://profesores.matcom.uh.cu/~kyrie

-- 
Participe en Universidad 2010, del 8 al 12 de febrero de 2010
La Habana, Cuba 
http://www.universidad2010.cu