[Python-Dev] cgi.py and huge uploads problem

[Guido van Rossum]

> there is a problem with the cgi.py implementation of Python 1.5.2
> and uploading of huge files.
> (found out by Joachim Rudolph, Virtual Photonics)
> 
> Class FieldStorage of cgi.py has a couple of methods which
> add accumulated lines to a self.lines array. This array fills and
> fills until the whole upload is done, with the side effect of
> loading the whole file into memory. The memory is freed
> after the whole upload is done.
> 
> This is no problem, until a company like VPI uses cgi.py to
> upload whole distributions of 100 MB and more, via Zope. :-)
> 
> Looking into cgi.py, I can't find a reason why this happens.
> Is this possibly just a debugging feature which is no longer
> needed?
> While cgi.py was modified for cosmetic resons, I didn't find
> changes for Python 2.0 on this topic.
> 
> Does it make sense to use a debug flag for this, or should
> the feature vanish completely?
> Do you want a patch?

You know, I have *no* idea why this is.  I have looked through various
revisions (this feature is as old as cgi.py:1.8) and cannot find any
use of or need for self.lines!  It just gets appended to.

So I think it's safe to toss all the references to self.lines and see
who complains.

--Guido van Rossum (home page: http://www.python.org/~guido/)