How to queue functions

[Dhananjay]

Dear all,

I am trying to use multiprocessing module.
I have 5 functions and 2000 input files.

First, I want to make sure that these 5 functions execute one after the
other.
Is there any way that I could queue these 5 functions within the same
script ?


Next, as there are 2000 input files.
I could queue them by queue.put() and get back to run one by one using
queue.get() as follows:

for file in files:
        if '.dat.gz' in file:
            q.put(file)

while True:
        item = q.get()
        x1 = f1(item)
        x2 = f2(x1)
        x3 = f3(x2)
        x4 = f4(x3)
        final_output = f5(x4)


However, how can I input them on my 8 core machine, so that at a time 8
files will be processed (to the set of 5 functions; each function one after
the other) ?

I am bit confused with the multiprocessing, please suggest me some
directions ....

Thank you


-- Joshi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20120918/f971ed8c/attachment.html>