Python Performance vs. C++ in a Complex System

[David Jeske]

On Sun, Apr 22, 2001 at 08:45:19PM +0200, Gabriel Ambuehl wrote:
> Yup. FreeBSD does Pthreads in userspace (which can be fucking slow
> even when used from C) but also offers kthreads. I just couldn't yet
> figure out how to use those.

I've never seen Python thread reasonably on FreeBSD. Even FreeBSD 4.x
which supports threads in the kernel, does NOT have a kernel
preemptive pthreads implementation. Since Python threading is based on
pthreads, you're stuck with user-space select() threads.

Just use fork()ed processes, or write it with Medusa and use async
I/O. Doing async I/O with Medusa is not that hard.

>> Going multi-process does not have to mean using lots more
>> resources. In Linux, a thread is pretty close to a process. If you
>> load up all the code and then fork(), you'll have something which
>> is pretty damn close to the efficiency of threading, without the
>> locking verhead.
> 
> This depends a bit. As long as you go on the CPU usage, it isn't such
> a problem to work with fork(), trouble starts when you go with high
> number of concurrent, blocking processes (which don't much CPU during
> select()) and you run out of RAM and the system begins to swap.
> Python's threading takes care of the RAM usage but sacrifices enormous
> amounts of CPU time.

You really should NOT be creating that many simultaneous processes. If
they are blocked doing nothing, they shouldn't exist. Change the way
your program works so that it just spawns a set of workers when they
are needed. When you are not polling a service, you don't need to have
a thread (or process) dedicated to it.

-- 
David Jeske (N9LCA) + http://www.chat.net/~jeske/ + jeske at chat.net