Python "compiler" is too slow for processing large data files???

[Ron Horn]

Thanks for trying it out, Martin and Wolfgang.

I re-ran the test on a different machine (still Windows 2000, but faster CPU
and more RAM), and can conclude only that the standard prebuilt install
(from www.python.org) for python 2.2.1, for Win32 is slower on compiles than
the linix and mac version you are using. I still got over 9 seconds on the
8000x10 test.   Here are my numbers for the 8000x10 (and also an 8000x20)
test on my Windows 2000 machine, 1.5GHz, Pentium 4:

C:\test>python compiletest.py 8000 10 comp.py
C:\test>timer r "python -c \"import comp\""
8000
Time passed: 00:00:09.39

C:\test>python compiletest.py 8000 20 comp2.py
C:\test>timer r "python -c \"import comp2\""
8000
Time passed: 00:00:26.47

The second case, 8000x20, is more realistic as to the amount of data I'm
working with, and I also did a quick check of the memory used to compile
that amount of data.  It was not scientific (I just watched the windows task
manager for the python process while it ran), but it seemed to need about
80MB to compile it.  Could not even run 8000x20 on my other mahine (python
ran out of memory).

Thanks for looking at it.  I'm glad the python community is out there!  For
now, I've changed my data format so that I can "exec" each line of the file
on its own, so I'm doing 8000 execs (one line of code each) , instead of
just 1 exec (with 8000 lines of data).  Seems unintuitive, but this runs in
about 3 seconds, instead of 26 seconds on the machine above.

Thanks again, maybe someone with more time and interest than me can
determine why the win32 compile function is slow and and eating so much
memory on larger-sized compiles?

r

"Martin Franklin" <mfranklin1 at gatwick.westerngeco.slb.com> wrote in message
news:mailman.1030614584.8635.python-list at python.org...
>
> Are you sure there is nothing else going on I generated a test module
comp.py
> with 8000 x 10 and the result of importing it is :-
>
>
> [scripts]$ python compiletest.py 8000 10 comp.py
> [scripts]$ time python -c "import comp"
> 8000
>
> real    0m2.658s
> user    0m2.490s
> sys     0m0.140s
>
> OK so I am not using Windows I am running on a 700Mh PIII laptop running
> RedHat 7.2, but even so 15 seconds sounds way to long.....
>
> -------------------------------------------------------
>
>