High memory usage - program mistake or Python feature?

[Ben S]

Gerald Klix wrote:
> It is difficult to diagnose if we can't see the whole program.
> But I suppose the re module makes some (superflous) copies.
> I would like to see the figures form LoadLogFile alone.

That is the whole program, pretty much. The rest is just reading a query
string from an HTTP request so that it can get a filename to pass to
LoadLogFile. Then there are several lines that use 'for' over the
results returned from GetLinesContainingCommand, which only returns 20
or 30 lines at most.

> Some performance hints:
> You can gain a little by using xreadlines. It does
> not read the whole file ant once. But if you map
> the sequence to another sequence you gain almost nothing.

If I use xreadlines (and drop the string stripping) I suppose I would
gain a lot on memory but lose out on speed (due to multiple passes over
the file), right? This isn't necessarily a bad thing since it's running
as a CGI script, and therefore the bottleneck is usually going to be the
network, I expect.

Thanks for your other ideas, too.

--
Ben Sizer
http://pages.eidosnet.co.uk/kylotan