[Python-ideas] Python and Concurrency

[Josiah Carlson]

"Richard Oudkerk" <r.m.oudkerk at googlemail.com> wrote:
> Josiah wrote:
> > Without work, #2 isn't "fast" using processes in Python.  It is trivial
> > using threads.  But here's the thing: with work, #2 can be made fast.
> > Using unix domain sockets (on linux, 3.4 ghz P4 Xeons, DDR2-PC4200
> > memory (you can get 50% faster memory nowadays)), I've been able to push
> > 400 megs/second between processes.  Maybe anonymous or named pipes, or
> > perhaps a shared mmap with some sort of synchronization would allow for
> > IPC to be cross platform and just about as fast.
> 
> The IPC uses sockets or (on Windows) named pipes.  Linux and Windows
> are roughly equal in speed.  On a P4 2.5Ghz laptop one can retreive an
> element from a shared dict about 20,000 times/sec.  Not sure if that
> qualifies as fast enough.

Depends on what the element is, but I suspect it isn't fast enough. 
Fairly large native dictionaries seem to run on the order of 1.3 million
fetches/second on my 2.8 ghz machine.

    >>> import time
    >>> d = dict.fromkeys(xrange(65536))
    >>> if 1:
    ...     t = time.time()
    ...     for j in xrange(1000000):
    ...             _ = d[j&65535]
    ...     print 1000000/(time.time()-t)
    ...
    1305482.97346
    >>>

But really, transferring little bits of data back and forth isn't what
is of my concern in terms of speed.  My real concern is transferring
nontrivial blocks of data; I usually benchmark blocks of sizes: 1k, 4k,
16k, 64k, 256k, 1M, 4M, 16M, and 64M. Those are usually pretty good to
discover the "sweet spot" for a particular implementation, and also
allow a person to discover whether or not their system can be used for
nontrivial processor loads.


 - Josiah