marshal.dumps quadratic growth and marshal.dump not allowing file-like objects

[Peter Otten]

John Machin wrote:

>> Here's how marshal resizes the string:
>>
>>         newsize = size + size + 1024;
>>         if (newsize > 32*1024*1024) {
>>                 newsize = size + 1024*1024;
>>         }
>>
>> Maybe you can split your large objects and marshal multiple objects to
>> keep the size below the 32MB limit.
>>
> 
> But that change went into the svn trunk on 11-May-2008; perhaps the OP
> is using a production release which would have the previous version,
> which is merely "newsize = size + 1024;".

That is indeed much worse. Depending on what the OP means by "large objects"
the problem may be fixed in subversion then.
 
> Do people really generate 32MB pyc files, or is stopping doubling at
> 32MB just a safety valve in case someone/something runs amok?

A 32MB pyc would correspond to a module of roughly the same size. So
someone/something runs amok in either case.

Peter