python disk i/o speed

[nnes]

Peter Hansen <peter at engcorp.com> wrote in message news:<3D525996.AA4F55EF at engcorp.com>...
to suggestions.
> 
> Why don't you run the profiler on it and see where the time is
> being spent?  Random arbitrary optimizations is a waste of developer
> time, and generally makes the code less maintainable.
> 
> (And Duncan's comments are still highly relevant.)
> 
> -Peter

well, I am not as proficient using the profiler as you may asume :-).
Maybe you can explain what the output of it means. As far as I can see
it tells me what I already know: there is a script called bench3.py
with a function filterfile2() which does some i/o using strings.

Fri Aug 09 09:27:05 2002    filtprof
Fri Aug 09 09:27:05 2002    filtprof

         6 function calls in 65.583 CPU seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall
filename:lineno(function)
        2    0.005    0.003   65.583   32.792
profile:0(filterfile2(sys.argv[1],
sys.argv[2]))
        2    0.000    0.000   65.578   32.789 <string>:1(?)
        2   65.578   32.789   65.578   32.789 bench3.py:3(filterfile2)
        0    0.000             0.000          profile:0(profiler)

And regarding Duncans comment. This was a just a test to be able to
estimate just that. My current numbers are that processing raw
logfiles in C is about 10 times faster and takes me about 25 times
more time to programm than Python.  Since I got several examples about
it with about the same speed, these estimates should hopefully be
accurate.

If the logfile gets bigger than 500MB a day I might bite the bullet
and write that stuff in C or Java. There is at least a psychological
difference of waiting 6 minutes or one hour for the result. One might
be tempted to just start it interactively instead of waiting for batch
time.

I have cases where dictionaries and regex would fit nicely though. I
should really test these but I shudder just of thinking doing that in
C, or are there nice furry libraries in gcc for something like that?
In Java there are at least some classes for it.

Nestor