[Tutor] Multi-threading IO and the GIL

[Tim Kelsey]

----- Original Message ----- 
From: "Kent Johnson" <kent37 at tds.net>
To: "Tim Kelsey" <tk at eutechnyx.com>
Cc: <tutor at python.org>
Sent: Tuesday, October 28, 2008 12:45 PM
Subject: Re: [Tutor] Multi-threading IO and the GIL


> On Tue, Oct 28, 2008 at 7:27 AM, Tim Kelsey <tk at eutechnyx.com> wrote:
>
>> Im not sure if this is the correct list to post this sort of thing
>
> We'll try to answer pretty much any Python question.
Thanks very much for your response, im very confused by this and greatfull 
for the input.


>
>> but I
>> have a question about python threading and the GIL. I have a 
>> multi-threaded
>> application that is hitting a MySQL database for some info from within 
>> the
>> various threads. When increase the size of my thread pool I see little or 
>> no
>> improvement in the application performance but the CPU usage is not 
>> maxed.
>> This suggests to me that the app is IO bound on the database, surly more
>> threads should help in this case.
>
> I can think of a few other possibilities...
>
> Does the app actually create more threads? If you only create two
> threads, the size of the thread pool doesn't much matter.
>

The app was tested with 5 to 20 threads in the pool, this is running on a 
Linux box and 'top -H' shows the threads using CPU time. The odd behaviour 
is that adding more threads has little impact on performance, suggesting 
that the app is processor bound. However, the total CPU time is less than 
100% ( ~60 - 80% ) seemingly indicating that IO is the bottleneck.

> It's possible that all the threads are blocking on the database, for
> example if they are all trying to modify the same row. Or the app
> could truly be I/O bound, if it is reading large amounts of data from
> the DB then more threads trying to read more data from the same source
> may not help.
>

The SQL in this case is simply a very SELECT opperations returning only a 
few K of data each time. The database is not under stress ( aprox 30-50% CPU 
use )


>> Is it possible that the GIL is not being release by a thread during the 
>> time
>> its blocking on the DB connection? Im using MySQLdb module and the
>> application is purely written in python.
>
> I don't know but it seems unlikely. Is there a MySQLdb list where you 
> could ask?
>

Ill ask about and see what I can find

> Kent
>
> This email has been scanned for all known viruses by the Email Protection 
> Agency.
>

Many thanks,
Tim Kelsey





This e-mail is confidential and may be privileged. It may be read, copied and used only by the intended recipient. No communication sent by e-mail to or from Eutechnyx is intended to give rise to contractual or other legal liability, apart from liability which cannot be excluded under English law. 

This email has been scanned for all known viruses by the Email Protection Agency.


www.eutechnyx.com Eutechnyx Limited. Registered in England No: 2172322