Python "compiler" is too slow for processing large data files???

[Martin Franklin]

On Wednesday 28 Aug 2002 5:35 pm, Ron Horn wrote:
> Hi all:
>
> I'm writing a fairly simple app that loads (text) data files, lets you edit
> the data, and then saves them back out.  Rather than parsing the data
> myself when loading a new file, and building data structures for the
> program's data that way, I decided to try to 'import' (or 'exec', actually)
> the data into the app.  The idea was to be able to format my data files as
> python code (see below), and then let the python compiler do the parsing.
> In the future, I could actually put 'def' and 'class' statements right into
> the data file to capture some behavior along with the data.
>
> Simple example - I can import or exec this file to load my data (my real
> app has int, float, and string data):
> ------ try5a3.py --------
> list1 = [
>     (323, 870, 46, ),
>     (810, 336, 271, ),
>     (572, 55, 596, ),
>     (337, 256, 629, ),
>     (31, 702, 16, ),
> ]
> print len(list1)
> ---------------------------
>
> Anyway, as my data files went from just a few lines, up to about 8000 lines
> (with 10 values in each line for total of about 450KB of text), the time to
> 'exec' the file became too slow (e.g. 15 seconds) and used too much memory
> (e.g. 50MB) (using ms-windows, python 2.2.1).  It is the "compile" phase,
> because if I re-run, and there is *.pyc file available, the import goes
> very fast (no compilation required).

Are you sure there is nothing else going on I generated a test module comp.py
with 8000 x 10 and the result of importing it is :-


[scripts]$ python compiletest.py 8000 10 comp.py
[scripts]$ time python -c "import comp"
8000

real    0m2.658s
user    0m2.490s
sys     0m0.140s

OK so I am not using Windows I am running on a 700Mh PIII laptop running
RedHat 7.2, but even so 15 seconds sounds way to long.....

-------------------------------------------------------