[Python-Dev] "Fixing" the new GIL

[Antoine Pitrou]

Le Sun, 14 Mar 2010 23:11:44 +0200,
Nir Aides <nir at winpdb.org> a écrit :
> 
> I first disabled the call to spin() but client running time remained
> around 30 seconds.
> I then added TCP_NODELAY and running time dropped to a few dozen
> milliseconds for the entire no-spin run.

You don't want the benchmark to be dependent on the TCP stack's
implementation specifics. Therefore, you should use the UDP version
of Dave Beazley's benchmark.
I have posted it on http://bugs.python.org/issue7946, and included a
variant of it in ccbench.

> The special macro for socket code is one of the alternatives proposed
> by Antoine above.
> 
> However, thinking about it again, with this approach as soon as the
> new incoming request tries to read a file, query the DB, decompress
> some data or do anything which releases the GIL, it goes back to
> square one. no?

Indeed. This approach involves using the macro in every place where
releasing the GIL is meant to cover OS- or IO-dependent latencies.
Even then, though, there may be cases (such as doing a very small
amount of zlib compression) where the background CPU-bound thread will
get "too much time" compared to the "mostly IO" thread. But the other
approach isn't perfect either. These are heuristics anyway.

> I remember there was reluctance in the past to repeat the OS
> scheduling
> functionality and for a good reason.

I agree with this. Hence my posting on this list, about whether we
should go with an additional amount of complexity to solve what
may be considered an annoying performance problem.

Regards

Antoine.