Too many open files

[AMD]

Thank you every one,

I ended up using a solution similar to what Gary Herron suggested : 
Caching the output to a list of lists, one per file, and only doing the 
IO when the list reaches a certain treshold.
After playing around with the list threshold I ended up with faster 
execution times than originally and while having a maximum of two files 
open at a time! Its only a matter of trading memory for open files.
It could be that using this strategy with asynchronous IO or threads 
could yield even faster times, but I haven't tested it.
Again, much appreciated thanks for all your suggestions.

Andre M. Descombes

> Hello,
> 
> I need to split a very big file (10 gigabytes) into several thousand 
> smaller files according to a hash algorithm, I do this one line at a 
> time. The problem I have is that opening a file using append, writing 
> the line and closing the file is very time consuming. I'd rather have 
> the files all open for the duration, do all writes and then close them 
> all at the end.
> The problem I have under windows is that as soon as I get to 500 files I 
> get the Too many open files message. I tried the same thing in Delphi 
> and I can get to 3000 files. How can I increase the number of open files 
> in Python?
> 
> Thanks in advance for any answers!
> 
> Andre M. Descombes