[issue13555] cPickle MemoryError when loading large file (while pickle works)

[Antoine Pitrou]

Antoine Pitrou <pitrou at free.fr> added the comment:

I think there's a problem here:

+    self->data = realloc(self->data, self->size * sizeof(PyObject *));
+    if (self->data == NULL)
         goto nomemory;

If realloc() fails, the old data pointer is lost and therefore will never get free()ed.

Same for:

+        self->buf = (char *)realloc(self->buf, self->buf_size);

Here:

-        int *marks;
-        s=self->marks_size+20;
-        if (s <= self->num_marks) s=self->num_marks + 1;
+        size_t alloc;
+        Py_ssize_t *marks;
+
+        /* Use the size_t type to check for overflow. */
+        alloc = ((size_t)self->num_marks << 1) + 20;

It seems you are changing the overallocation algorithm (from additive to multiplicative). I'm not sure it should be in the scope of the patch, although multiplicative overallocation is generally better.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13555>
_______________________________________