Too many open files

[Duncan Booth]

Steven D'Aprano <steve at REMOVE-THIS-cybersource.com.au> wrote:

> On Mon, 04 Feb 2008 13:57:39 +0100, AMD wrote:
> 
>> The problem I have under windows is that as soon as I get to 500 files I
>> get the Too many open files message. I tried the same thing in Delphi
>> and I can get to 3000 files. How can I increase the number of open files
>> in Python?


> Windows XP has a limit of 512 files opened by any process, including 
> stdin, stdout and stderr, so your code is probably failing after file 
> number 509.

No, the C runtime has a limit of 512 files, the OS limit is actually 2048. 
See http://msdn2.microsoft.com/en-us/library/6e3b887c(VS.71).aspx

> I don't know how Delphi works around that issue. Perhaps one of the 
> Windows gurus can advise if there's a way to increase that limit from
> 512? 
> 

Call the C runtime function _setmaxstdio(n) to set the maxmimum the number 
of open files to n up to 2048. Alternatively os.open() and os.write() 
should bypass the C runtime limit.

It would probably be better though to implement some sort of caching scheme 
in memory and avoid having to mess with the limits at all. Or do it in two 
passes: creating 100 files on the first pass and splitting each of those in 
a second pass.