Database in a simple ascii file

[Cameron Laird]

In article <3C04F826.CB408C9D at engcorp.com>,
Peter Hansen  <peter at engcorp.com> wrote:
>Rafael Mentz Aquino wrote:
>> 
>> How many lines can I have in a simple ascii file before
>> I start to loose performance when I open this file and
>> put all the lines in a dict, turning the first column into the key
>> and the rest of the line into a list associated to that key.
>> 
>> (my file has just 100 lines, but that question starts to bother me ;-)
>
>You will start to lose performance, relatively to no lines,
>with even one line.  If you start with ten lines, you will
>begin losing performance after you switch to a file of 11 
>lines...
>
>Why focus on performance when you have no idea how fast
>you want your program to be?  Recent advances in software
>development are teaching programmers *never* to optimize
>until the program works, and only when there is a clear
>requirement for more performance.
>
>If you are really concerned about performance, you will 
>always be able to get more than a 10 times speedup by 
>switching to a compiled language over Python.  Use of
>Python should imply you are more interested in readability
>and ease of use, and thus speed of *development*, over
>speed of execution.
>
>(Shorter answer: don't grow any grey hairs over it.)
			.
			.
			.
Peter and I agree on so many large conclusions,
and differ on so many details.

My own suspicion about the situation the original
poster describes is that I/O time dominates, and
that neither "switching to a compiled language" nor
a fancier persistence method will help by even a
factor of two (maybe not 10%) in comparison with
judiciously straightforward Python coding.  I know
that's true for many things I do:  it just takes a
long time to read in data.  Once it's in memory,
Python manages it plenty swiftly.

Peter's entirely correct in his general advice.
-- 

Cameron Laird <Cameron at Lairds.com>
Business:  http://www.Phaseit.net
Personal:  http://starbase.neosoft.com/~claird/home.html