submit jobs on multi-core

[Oscar Benjamin]

On 2012-09-11, Dhananjay <dhananjay.c.joshi at gmail.com> wrote:
> --===============0316394162==
> Content-Type: multipart/alternative; boundary=20cf30776bd309ffd004c96557e2
>
> --20cf30776bd309ffd004c96557e2
> Content-Type: text/plain; charset=ISO-8859-1
>
> Dear all,
>
> I have a python script in which I have a list of files to input one by one
> and for each file I get a number as an output.
> I used for loop to submit the file to script.
> My script uses one file at a time and returns the output.
>
> My computers has 8 cores.
> Is there any way that I could submit 8 jobs at a time and get all the
> output faster ?
> In other words, how can I modify my script so that I could submit 8 jobs
> together on 8 different processors ?
>
> I am bit new to this stuff, please suggest me some directions.
>
> Thank you.

The simplest way I've found to do this is to use something like GNU parallel.
I don't know if there's a Windows equivalent but it works well for me on linux
and you can use it for any program (not just python scripts).

>From the wikipedia page:
http://en.wikipedia.org/wiki/GNU_parallel

"""
The most common usage is to replace the shell loop, for example

    (for x in `cat list` ; do
          do_something $x
             done) | process_output

to the form of

    cat list | parallel do_something | process_output
"""

Note that there are two basic types of parallel execution depending on whether
or not your parallel processes need to communicate with one another. I'm
assuming that you really just want to run independent jobs simultaneously.
Otherwise the other suggestions may be more relevant.

Oscar