running functions

[sjdevnull at yahoo.com]

Tom Anderson wrote:
> > If you program threads with shared nothing and communication over Queues
> > you are, in effect, using processes.  If all you share is read-only
> > memory, similarly, you are doing "easy" stuff and can get away with it.
> > In all other cases you need to know things like "which operations are
> > indivisible" and "what happens if I read part of this from before an
> > update and the other after the update completes, .....
>
> Right, but you have exactly the same problem with separate processes -
> except that with processes, having that richness of interaction is so
> hard, that you'll probably never do it in the first place!

It's not all that hard, but you do have to think about it and really
engineer your communications (and SHM segments) ahead of time.  Which
is a good thing, IMO.

Really the only difference between threads and processes is that
threads share all your memory.  When you consider that OS designers
spent a lot of time and effort to implement protected memory, maybe
that's not often the right thing.  Certainly threads have their place,
but I avoid them whenever there's an equally easy multiprocess solution
(which is the overwhelming majority of the time, but certainly not
always).

To diverge from the Python world for a moment, imagine a network server
that services each connection with a thread (possibly from a thread
pool rather than creating a new one for each connection).  And imagine
another that uses multiple process.  Now suppose there's a bug in some
extension that segfaults (for instance, when serving large media
files).  It's very easy for such a bug to bring down the whole server
in the multithreaded server; in the multiprocess server it naturally
brings down only the one connection, so the rest of the site continues
to run happily.  Many other stability and security holes are easy to
introduce when you eliminate memory protection.

I don't really understand the "process are really hard" thing, either.
It's a 5 line wrapper or so to make it look just like the threads
interface if you really want to, and you only have to do that once.
You do need to worry about IPC, but that is generally simpler than just
mucking about in process-wide memory, especially when it comes to
maintenance.  But...

> I've done a fair amount of threads programming, although in java rather
> than python (and i doubt very much that it's less friendly in python than
> java!), and i found it really fairly straightforward.

Java certainly doesn't have any real support for multiprocess
programming.  I view the lack of good multiprocess solutions in Java to
be one of the few major design flaws therein.  Indeed, I view the poor
performance of processes on Windows (and the accompanying need to use
threads for reasons other than memory sharing) to be the biggest single
problem with Windows.