64 bit Python

[Jeff Epler]

There's not enough information to guess the "real problem", but it could
be this:

"variable size" objects (declared with PyObject_VAR_HEAD) are limited to
INT_MAX items since the ob_size field is declared as 'int'.

This means that a Python string, tuple, or list (among other types) may
be limited to about 2 billion items on ILP32 and LP64 architectures.

Dicts and probably sets are also limited to INT_MAX elements, because
the "ma_fill" and "ma_used" fields are ints.

If you don't mind recompiling all your extensions, you could change the
type of ob_size to long in the "#define PyObject_VAR_HEAD".  I don't
know what breaks when you do this, but maybe a few google or google
groups searches could help you find others who have tried this.

Jeff
PS the limit of 500MB of "real data" in the 32-bit system may be because
a realloc may temporarily require (old size + new size) storage when it
does the equivalent of
    new_ptr = malloc(new_size)
    memcpy(new_ptr, old_ptr, old_size)
    free(old_size)
which will temporarily use >900MB of data when resizing a ~450MB object.
Python frequently uses realloc() to grow structures like lists.  If you
are working with strings, then
    s = s + t
doesn't ever use realloc (in 2.3.x anyway) but always allocates and
fills the result 's+t', only freeing the old value of s later when it is
no longer reacahable (which could be as soon as the assignment statement
completes)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20050214/3bc77515/attachment.sig>