Multicore-programming?

[Paul Boddie]

On 7 Sep, 00:06, cnb <circularf... at yahoo.se> wrote:
> If I buy a multicore computer and I have really intensive program. How
> would that be distributed across the cores?

It typically depends on how the work done by the program is performed.

> Will algorithms always have to be programmed and told specifically to
> run on several cores so if not told it will only utilize one core?

Some algorithms lend themselves to parallelisation; others do not.
Sometimes tools and runtimes can help by executing some instructions
in parallel.

> So is the free lunch really over or is this just an overhyped
> phenomena?

The free lunch ended a few minutes ago. ;-)

> Is threading with Python hard? Can you start several processes with
> Python or just threads?

You can start both processes and threads with Python, although the
effect of starting many threads - the actual concurrency - will depend
on which implementation of Python you're using and where the bulk of
the work is performed.

If you're spending a lot of CPU time in processing data, and if that
processing is taking place in Python code, then for the most effective
threading you should consider an implementation like Jython or
IronPython which supports free-threading. If most of the work happens
in extension code (where you'd probably have little choice about using
CPython, anyway), then it might be the case that the extension
releases the global interpreter lock in CPython and you might then be
able to benefit from having many threads doing work simultaneously,
although I imagine that the extension would itself need to be thread-
safe, too.

If you're spending a lot of time moving data around, performing
communication, and so on, then multiple threads may still be effective
in CPython, since some of them might be getting a system call to read
or write data, thus freeing the CPU for the others. These kinds of
situations lend themselves to other approaches such as asynchronous
processing of data, however. It doesn't sound like this is like your
program, if by "intensive" you mean high levels of CPU activity.

As you note, the alternative to threads is processes, and many people
advocate multi-process, "shared nothing" solutions. Here's a list
which covers multicore and SMP-related solutions as well as high-end
clustering solutions:

http://wiki.python.org/moin/ParallelProcessing

Although the processing module is part of Python 2.6/3.0 as the
multiprocessing module, you might want to at least look at the pp,
pprocess and papyros solutions. My aim with pprocess was to target
multicore UNIX-like systems with an unintrusive API; pp and papyros,
on the other hand, seek to cover larger scale systems as well, and I
think that the way papyros has been done has some merit, mostly
because if you wanted to combine convenience with distributed
processing, you'd want to choose distributed object technologies as
the foundation (CORBA would have been good for this, too, at least for
multi-language support, but its APIs can also seem quite
intimidating).

Paul