Please help with Threading

[Dave Angel]

On 05/18/2013 04:58 AM, Jurgens de Bruin wrote:
> This is my first script where I want to use the python threading module. I have a large dataset which is a list of dict this can be as much as 200 dictionaries in the list. The final goal is a  histogram for each dict 16 histograms on a page ( 4x4 ) - this already works.
> What I currently do is a create a nested list [ [ {}  ], [ {} ] ] each inner list contains 16 dictionaries, thus each inner list is a single page of 16 histograms. Iterating over the outer-list  and creating the graphs takes to long. So I would like multiple inner-list to be processes simultaneously and creating the graphs in "parallel".
> I am trying to use the python threading for this. I create 4 threads loop over the outer-list and send a inner-list to the thread. This seems to work if my nested lists only contains 2 elements - thus less elements than threads. Currently the scripts runs and then seems to get hung up. I monitor the resource  on my mac and python starts off good using 80% and when the 4-thread is created the CPU usages drops to 0%.
>
> My thread creating is based on the following : http://www.tutorialspoint.com/python/python_multithreading.htm
>
> Any help would be create!!!
>

CPython, and apparently (all of?) the other current Python 
implementations, uses a GIL to prevent multi-threaded applications from 
shooting themselves in the foot.

However the practical effect of the GIL is that CPU-bound applications 
do not multi-thread efficiently;  the single-threaded version usually 
runs faster.

The place where CPython programs gain from multithreading is where each 
thread spends much of its time waiting for some external trigger.

(More specifically, if such a wait is inside well-written C code, it 
releases the GIL so other threads can get useful work done.  Example is 
a thread waiting for internet activity, and blocks inside a system call)


-- 
DaveA