ESR "Waning of Python" post

[dieter]

Ben Finney <ben+python at benfinney.id.au> writes:
> ...
> Is it your position that the described behaviour is not a problem? Do
> you hold that position because you think multi-core machines are not a
> sector that Python needs to be good at? Or that the described behaviour
> doesn't occur? Or something else?

Every system you use has its advantages and its drawbacks.
Depending on the specific context (problem, resources, knowledge, ...),
you must choose an appropriate one.

Python uses the GIL mainly because it uses reference counting
(with almost constant changes to potentially concurrently used objects)
for memory management.
Dropping the GIL would mean dropping reference counting
likely in favour of garbage collection.

I work in the domain of web applications. And I made there a nasty
experience with garbage collection: occasionally, the web application
stopped to respond for about a minute. A (quite difficult) analysis
revealed that some (stupid) component created in some situations
(a search) hundreds of thousands of temporary objects and thereby
triggered a complete garbage collection. The garbage collector
started its mark and sweep phase to detect unreachable objects - traversing
a graph of millions of objects.
As garbage collection becomes drastically more complex
if the object graph can change during this phase (and this was Python),
a global look prevented any other activity -- leading to the observed
latencies.

When I remember right, there are garbage collection schemes that
can operate safely without stopping other concurrent work.
Nevertheless, even those garbage collectors have a significant
impact on performance when they become active (at apparently
non-deterministic times) and that may be inacceptable for
some applications.