Processing large CSV files - how to maximise throughput?

[Stefan Behnel]

Chris Angelico, 25.10.2013 08:13:
> On Fri, Oct 25, 2013 at 2:57 PM, Dave Angel wrote:
>> But I would concur -- probably they'll both give about the same speedup.
>> I just detest the pain that multithreading can bring, and tend to avoid
>> it if at all possible.
> 
> I don't have a history of major pain from threading. Is this a Python
> thing, or have I just been really really fortunate

Likely the latter. Threads are ok if what they do is essentially what you
could easily use multiple processes for as well, i.e. process independent
data, maybe from/to independent files etc., using dedicated channels for
communication.

As soon as you need them to share any state, however, it's really easy to
get it wrong and to run into concurrency issues that are difficult to
reproduce and debug.

Basically, with multiple processes, you start with independent systems and
add connections specifically where needed, whereas with threads, you start
with completely shared state and then prune away interdependencies and
concurrency until it seems to work safely. That approach makes it
essentially impossible to prove that threading is safe in a given setup,
except for the really trivial cases.

Stefan