For loop searching takes too long!

[Jonathan Gardner]

On Jan 29, 7:07 pm, Steven D'Aprano <st... at REMOVE-THIS-
cybersource.com.au> wrote:
> On Fri, 29 Jan 2010 14:49:06 -0800, Jonathan Gardner wrote:
> > On Jan 28, 3:52 pm, elsa <kerensael... at hotmail.com> wrote:
>
> >> I've got a problem with my program, in that the code just takes too
> >> long to run. Here's what I'm doing. If anyone has any tips, they'd be
> >> much appreciated!
>
> > First of all, don't play with large lists. Large lists have a tendency
> > to grow larger over time, until they get absurdly large.
>
> > If you're dealing with a long list, work with it like you'd work with a
> > file. If you need random access, stick it in some form of database, such
> > as BDB or PostgreSQL. If you need an index, stick it in some form of DB.
> > Eventually, large lists end up like that anyway. Don't fool yourself
> > early on---prepare for the worst and solve the problems of tomorrow
> > today.
>
> Talk about premature optimization. The OP probably has a few hundreds of
> thousands of items, millions at most, which is trivial on a modern PC.
> Your way of thinking is what has led to Firefox using a database to
> manage a few thousand bookmarks and cookies, which in turn leads to
> consistent data corruption problems. (I've been keeping note, and so far
> my installation of Firefox 3 has had database corruption at startup one
> time out of five.)
>

I think there's a difference between writing your code so that you
have the option of using a database, or a flat file, or any other type
of serial data source, and actually using a database (or whatever.)

I prefer to write my code so that I don't have to re-write it 6 months
later.

Having worked with datasets that are truly enormous, whenever I see a
list that is unbounded (as this appears to be), I freak out because
unbounded can be a really, really big number. Might as well write
software that works for the truly enormous case and the really large
case and be done with it once and for all.