[Python-bugs-list] The Python 1.5.2 cgi module chews up memory (PR#435)

[guenter@ubka.uni-karlsruhe.de]

Hello,

I have a problem with one of my Python CGIs and am not sure what to do
about this.  I have found a related message in the bug reporting system
but do not know how to use this system to enter my problem so I will
describe it in this mail.

URL of related problem:
http://www.python.org/python-bugs/notabug?id=396;expression=FieldStorage;user=guest

My application grows to about twice the size of an uploaded file when
executing the statement

    self.form = cgi.FieldStorage()

I have taken great care not to use form['something'].value to make it
possible to upload large files but this does not help because the
problem occurs much earlier.

Referring to the related problem mentioned above:  I do not use Zope,
only the CGI module in the Python distribution.  The file I upload does
contain a lot of new line characters.  The mime encoded version of the
file should also contain new line characters.  I have not (yet) checked
where in cgi.py the memory is used and what it is used for.

I have been logging this with the following code on Linux:

    def log(self, info):
	# get the process size from the proc fs - will work only on linux:
	pid = os.getpid()
	pstat = open(os.path.join('/proc', '%s' % pid, 'stat')).read()
	size = string.atoi(string.split(pstat)[22]) / 1024
	# append some information to the log file:
	where = string.rstrip(traceback.format_stack()[-2])
	where = string.split(where, '\n')[:1]
	self.logfile.write('pid=%s, size=%s, At %s: %s\n'
			   % (pid, size, where, info))
	self.logfile.flush()

only to realize the problem must be in cgi.py.  Is this a known problem
and is there a fix for it?  Can I do anything to get this fixed in
future distributions of Python?

Thanks in advance for Your help.
-- 
Guenter Radestock, Universitaetsbibliothek Karlsruhe
guenter@ubka.uni-karlsruhe.de
http://www.ubka.uni-karlsruhe.de/~guenter