ASCII delimited files

[Mike Fletcher]

Okay, here's an entirely different approach, extremely memory intensive,
fairly fast.  Gives pretty good speed up to the point where you start
hitting memory effects (my 256 MB PIII 500 loads 10MB in 45 seconds, but
takes 7.5 minutes for a 36MB file).

For even greater speed, using an index-counting scheme (instead of expensive
front-of-list-deletions) might be tried.

Note: this does none of the fancy delimiter-determination of delimited.py,
it's just a loading engine.

Cheers,
Mike

Time for file of 365 bytes
 (5 * testset, 11 records found):
 0.0
Time for file of 3650 bytes
 (50 * testset, 101 records found):
 0.00999999046326
Time for file of 36500 bytes
 (500 * testset, 1001 records found):
 0.120000004768
Time for file of 365000 bytes
 (5000 * testset, 10001 records found):
 1.25199997425
next test may take 15 seconds or so
Time for file of 3650000 bytes
 (50000 * testset, 100001 records found):
 13.5299999714
next test may take 45 seconds or so
Time for file of 10950000 bytes
 (150000 * testset, 300001 records found):
 50.4729999304


-----Original Message-----
From: python-list-admin at python.org
[mailto:python-list-admin at python.org]On Behalf Of Darrell
Sent: November 11, 1999 1:20 PM
To: python-list at python.org
Subject: Re: ASCII delimited files


Oops.
That fix pushed into or beyond exponential runtime.
It won't finish the 3.6meg file anytime soon.
String slicing and pasting is dirt slow.

E:\ace\dtd\q\delim1>delimited.py
3600 bytes
0.0310000181198 time

E:\ace\dtd\q\delim1>delimited.py
36000 bytes
3.04700005054 time

Got some improvement with the attached version at
the cost of memory.

Now I'm thinking about the 'C' version again.

E:\ace\dtd\q\delim1>delimited.py
3600
0.0310000181198

E:\ace\dtd\q\delim1>delimited.py
36000
0.327999949455

E:\ace\dtd\q\delim1>delimited.py
360000
3.21899998188

E:\ace\dtd\q\delim1>delimited.py
3600000
48.2030000687

--
--Darrell




--
http://www.python.org/mailman/listinfo/python-list


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: csv.py
URL: <http://mail.python.org/pipermail/python-list/attachments/19991111/78894619/attachment.ksh>