[New-bugs-announce] [issue29353] Incorrect handling of HTTP response with "Content-Type: message/rfc822" header

[brokenenglish]

New submission from brokenenglish:

Hello.

I found a bug that causes incorrect handling of some values of "Content-Type" header. When you retrieve any URL with "Content-Type: message/rfc822" header, additional payload is added to HTTPMessage. In some cases it causes annoing warnings. Here is a code from packages requests and urllib3 that can cause warning output:
1. Checking and raising excpection in urllib3: https://github.com/shazow/urllib3/blob/0fb5e083b2adf7618db8c26e8e50206de09dd845/urllib3/util/response.py#L61-L66
2. The same code packaged in requests: https://github.com/kennethreitz/requests/blob/362da46e9a46da6e86e1907f03014384ab210151/requests/packages/urllib3/util/response.py#L61-L66
3. Logging the error to output: https://github.com/kennethreitz/requests/blob/362da46e9a46da6e86e1907f03014384ab210151/requests/packages/urllib3/connectionpool.py#L402-L407

The issue arises from the class email.feedparser.FeedParser that handles HTTP and email headers in the same way. So when it handles headers with the message with content-type "message/*" and some other, method _parsegen() tries to parse it like a message with another message (see comments to this condition: https://hg.python.org/cpython/file/tip/Lib/email/feedparser.py#l295). If headers-only argument is set to True, we can avoid some redundant checks on HTTP headers (see this condition: https://hg.python.org/cpython/file/tip/Lib/email/feedparser.py#l244).

I read the comment above (https://hg.python.org/cpython/file/tip/Lib/email/feedparser.py#l241) but I am a newbie and I don't see any other solution to my problem for now. I can rewrite the patch if you show me a better way to fix this issue.

A patch with the unit test is atached.

----------
files: rfc822_httpmessage_payload.patch
keywords: patch
messages: 286106
nosy: brokenenglish
priority: normal
severity: normal
status: open
title: Incorrect handling of HTTP response with "Content-Type: message/rfc822" header
versions: Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7
Added file: http://bugs.python.org/file46395/rfc822_httpmessage_payload.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue29353>
_______________________________________