Future of Pypy?

[Steven D'Aprano]

Paul Rubin wrote:

> Steven D'Aprano <steve+comp.lang.python at pearwood.info> writes:
>> I'm sorry, but the instant somebody says "eliminate the GIL", they lose
>> credibility with me. Yes yes, I know that in *your* specific case you've
>> done your research and (1) multi-threaded code is the best solution for
>> your application and (2) alternatives aren't suitable.
> 
> I don't see what the big deal is.  I hear tons of horror stories about
> threads and I believe them, but the thing is, they almost always revolve
> around acquiring and releasing locks in the wrong order, forgetting to
> lock things, stuff like that.

Deadlocks, livelocks, and the simple fact that debugging threaded code is 
much, much, much harder than debugging single-thread code.

You'll notice that nowhere in my post did I say "Don't use threads!". But 
they aren't a panacea, and there are other concurrency models available 
which don't have the same disadvantages as threads (they have their own pros 
and cons instead).


[...]
>> For removal of the GIL to really make a difference: ...
[...]
>> - the task must be one that moving to some other multi-processing model
>> (such as greenlets, multiprocess, etc.) is infeasible;
> 
> I don't understand this--there can be multiple ways to solve a problem.

Yes, but my point is that if some other way solves the problem, then you 
should *use that other technique* rather than complain about the GIL. The 
GIL is not a bottleneck if you can bypass it.

I can sympathize with somebody who says "I have this problem that can only 
be solved with threads, anything else is too difficult, but the GIL makes it 
too slow".

But I don't sympathize with somebody who says "I have this problem that can 
be solved with threads or multiprocessing, but I insist that only threads 
can be used. And now the GIL makes it too slow. Python sux."

No. You [generic you] sux, for not investigating whether multiprocessing 
will do the job. It even has the same public API as threads.


>> - your threading bottleneck must be primarily CPU-bound, not I/O bound
>> (CPython's threads are already very effective at parallelising I/O
>> tasks);
> 
> If your concurrent program's workload makes it cpu-bound even 1% of the
> time, then you gain something by having it use your extra cores at those
> moments, instead of having those cores always do nothing.

Not necessarily. Threading has overhead too. Even in GIL-less Python, two 
threads aren't twice as fast as one thread. So there comes a point of 
diminishing returns, where the benefit you get from threading is less than 
the cost of threading. Now your code will actually run slower with threads, 
GIL or no GIL.


>> - and you must be using libraries and tools which prevent you moving to
>> Jython or IronPython or some other alternative.
> 
> I don't get this at all.  Why should I not want Python to have the same
> capabilities?

Python does have the same capabilities. Jython and IronPython aren't 
different languages, they are Python.

If you want *CPython* to work without a GIL, well, are you volunteering to 
do the work? It is a massive job, and the core devs aren't terribly 
interested. Probably because they understand that the GIL is not often an 
unsurmountable problem in real life. That brings us to the point I was 
making: I believe, and the core devs appear to think the same, that the 
*actual* number of people who would benefit from CPython losing the GIL is 
not large enough to justify the work it would require.


-- 
Steve