Python Database Query Size

[Thomas Wouters]

On Sun, Nov 12, 2000 at 04:31:47AM +0000, ryzam at my-deja.com wrote:

[ snip problems stuffing lots of data from a database into a python list ]

> I'm trying to check to another programme like php and asp to see it
> performance. I think python list '[]' size cannot hold large data in
> one time.. it will crash...

A Python list has only one limitation: its length cannot exceed the maximum
value of an int on your system (typically about 2 milion.) Additionally,
what it contains are just pointers to other Python objects, which have to be
seperately allocated. And since the maximum accessible memory is typically
also constrained by the size of an int, memory is going to be a problem long
before the maximum size of a list is.

So, the problem is not howmany rows you try to load into the list, but how
large those rows are. Each row is transformed into a Python string object
(or possibly Unicode object) which require a few bytes more than the length
of the string they contain, and the addresses to those strings are pushed
onto the Python list. IDLE and/or Zope shouldn't interfere, except in sofar
that they require some memory themselves, or perhaps if you are running
profilers/debuggers at the same time. Try running your script outside of
Zope and IDLE, just by starting it as a script -- possibly removing
everything but the 'extract data from DB' code.

-- 
Thomas Wouters <thomas at xs4all.net>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!