regex over files

[Robin Becker]

Gerald Klix wrote:
> Map the file into RAM by using the mmap module.
> The file's contents than is availabel as a seachable string.
> 

that's a good idea, but I wonder if it actually saves on memory? I just tried 
regexing through a 25Mb file and end up with 40Mb as working set (it rose 
linearly as the loop progessed through the file). Am I actually saving anything 
by not letting normal vm do its thing?

> HTH,
> Gerald
> 
> Robin Becker schrieb:
> 
>> Is there any way to get regexes to work on non-string/unicode objects. 
>> I would like to split large files by regex and it seems relatively 
>> hard to do so without having the whole file in memory. Even with 
>> buffers it seems hard to get regexes to indicate that they failed 
>> because of buffer termination and getting a partial match to be 
>> resumable seems out of the question.
>>
>> What interface does re actually need for its src objects?
> 
> 


-- 
Robin Becker