[New-bugs-announce] [issue32541] cgi.FieldStorage constructor assumes all lines terminate with \n

[Ian Craggs]

New submission from Ian Craggs <icraggs at gmail.com>:

Using cgi.FieldStorage in an HTTP server in a subclass of BaseHTTPRequestHandler, parsing the request with:

form = cgi.FieldStorage(fp=self.rfile,
                        headers=self.headers,
                        environ={"REQUEST_METHOD":op.upper(),
                              "CONTENT_TYPE":self.headers['Content-Type'],})

This has been working fine with clients using the Python requests library. Now processing requests from a Java library (org.apache.cxf.jaxrs.client.WebClient), the final line in a multipart request does not include the (\r)\n, which causes the final read to hang until a socket timeout.  The read in question is in cgi.py, read_lines_to_outerboundary:

line = self.fp.readline(1<<16) # bytes

(line 824 in Python 3.6.2).  I changed this read to not assume the termination of the final line with \n:

    def read_line(self, last_boundary):
        line = self.fp.readline(len(last_boundary))
        if line != last_boundary and not line.endswith(b"\n"):
            line += self.fp.readline((1<<16) - len(last_boundary))
        return line


and the request worked.  The Java library is being used in tests against our production web server so I assume that is working correctly.  

Perhaps I am misusing the FieldStorage class, I don't know, I'm not expert on this.

----------
components: Library (Lib), macOS
messages: 309868
nosy: Ian Craggs, ned.deily, ronaldoussoren
priority: normal
severity: normal
status: open
title: cgi.FieldStorage constructor assumes all lines terminate with \n
type: behavior
versions: Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue32541>
_______________________________________