Are threads bad? - was: Future of Pypy?

[Ryan Stuart]

On Tue Feb 24 2015 at 3:32:47 PM Paul Rubin <no.email at nospam.invalid> wrote:

> Ryan Stuart <ryan.stuart.85 at gmail.com> writes:
> Sure, the shared memory introduces the possibility of some bad errors,
> I'm just saying that I've found that by staying with a certain
> straightforward style, it doesn't seem difficult in practice to avoid
> those errors.
>

And all I'm saying is that it doesn't matter how careful you are, all it
takes is someone up the stack to not be as careful and you will be bitten.
And even if they are careful, it's still very easy to be bitten because
everything runs in shared memory.


> I don't understand what you mean about malloc.
>

My point is malloc, something further up (down?) the stack, is making
modifications to shared state when threads are involved. Modifying shared
state makes it infinitely more difficult to reason about the correctness of
your software.


> That article was interesting in some ways but confused in others.  One
> way it was interesting is it said various non-thread approaches (such as
> coroutines) had about the same problems as threads.  Some ways it
> was confused were:
>

We clearly got completely different things from the article. My
interpretation was that it was making *the exact opposite* point to what
you stated mainly because non-threading approaches don't share state. It
states that quite clearly. For example "it is – literally – exponentially
more difficult to reason about a routine that may be executed from an
arbitrary number of threads concurrently".


>   1) thinking Haskell threads were like processes with separate address
>   spaces.  In fact they are in the same address space and programming
>   with them isn't all that different from Python threads, though the
>   synchronization primitives are a bit different.  There is also an STM
>   library available that is ingenious though apparently somewhat slow.
>

I don't know Haskell, so I can't comment too much, except to say that by
default Haskell looks to use lightweight threads where only 1 thread can be
executing at a time [1] (sounds similar to the GIL to me). That doesn't
seem to be shared state multithreading, which is what the article is
referring to. But again, they will undoubtedly be someone sharing state
higher up the stack.


>   2) it has a weird story about the brass cockroach, that basically
>   signified that they didn't have a robust enough testing system to be
>   able to reproduce the bug.  That is what they should have worked on.
>

The point was that it wasn't feasible to have a robust testing suite
because, you guessed it, shared state and concurrent threads producing n*n
complexity.


>   3) It goes into various hazards of the balance transfer example not
>   mentioning that STM (available in Haskell and Clojure) completely
>   solves it.
>

This is probably correct. Is there any STM implementations out that that
don't significantly compromise performance?

Anyway, I got one thing out of this, which is that the multiprocessing
> module looks pretty nice and I should try it even when I don't need
> multicore parallelism, so thanks for that.
>

It's 1 real advantage is that it side-steps the GIL. So, if you need to
utilise multiple cores for CPU bound tasks, then it might well be the only
option.

Cheers


> In reality though, Python is still my most productive language for
> throwaway, non-concurrent scripting, but for more complicated concurrent
> programs, alternatives like Haskell, Erlang, and Go all have significant
> attractions.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20150225/a95e93e3/attachment.html>