Pop a list from beginning ? and memory saving...

[Shagshag13]

"Peter Hansen" <peter at engcorp.com> a écrit dans le message de news: 3CFA0676.68F80DE6 at engcorp.com...
> Shagshag13 wrote:
> >
> > I had a huge list containing consuming data, as i only need a one pass in a
> > for statement, i'm wondering if by doing this with a pop i could save memory ?
>
> Maybe, maybe not.  The implementation _probably_ makes a copy of
> the entire list at some point as you shrink it.  That would mean that
> at least at some interval, you would temporarily consume almost twice
> the memory of the original list as the original list is copied, then
> freed.  You might end up with less memory used, but with double the
> memory needed.  Of course, the same would have held true as you built
> the list in the first place...
>
> Why are you trying to save memory?  Are you really working with such
> a large amount of data in one chunk that you will run out?  Maybe
> there is some other approach you haven't thought of.  This effort
> almost smells like premature optimization (the root of all evil).
>
> -Peter

in fact i had huge memory troubles !!! i'm a python newbie, but i use it because i needed a
language which allow fast development (and can't remind me of things like C or Java...)

and yes i'm working with large amount of data : i'm trying to index something like 1 go of
plain text by using conventionnal text retrieval structures (i mean without compression
tricks, because i *don't* have time to implement theses, but with stoplist and so on)
and wishing to do it in one pass without relying on hdd temp files (i'm starting to
think that i were dreaming...)

the most i think i could simply do is this : i use integer each time as possible, i free
everything i don't need anymore with = None, and what could i do more ? Advices
are ***greatly*** welcome !!!

s13.

ps : do you know if any python databases could handle without trouble more than
1 or 2 go of data ???