[New-bugs-announce] [issue44647] Non-ASCII characters in os.environ cause silent failures in test_httpservers

[Łukasz Langa]

New submission from Łukasz Langa <lukasz at langa.pl>:

GH-23638 introduced a new test for Accept: headers in CGI HTTP servers. This test serializes all of `os.environ` on the server side.  For non-UTF8 locales this can fail for some Unicode characters found in environment variables.

This started failing this week on Azure Pipelines with their rollout of a new Windows 2019 image version that included a "BUILD_SOURCEVERSIONAUTHOR" env variable. For me specifically it includes a leading Unicode character so all my PRs started failing on Azure Pipelines Windows 2019 alone.

The result was truncated output from the CGI HTTP server, like:

======================================================================
FAIL: test_accept (test.test_httpservers.CGIHTTPServerTestCase) [OrderedDict([('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8')])]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\1\s\lib\test\test_httpservers.py", line 860, in test_accept
    self.assertIn(expected.encode('ascii'), res.read())
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b"'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'" not found in b''


The root cause is that the CGI script in question (`cgi_file6` in test_httpservers.py) is crashing on server-side reaching the `print(repr(os.environ))` line. However, this exception isn't visible on the client side where the test is running. The only visible issue is truncated output.

I suggest adding ENSURE_UNICODE_WORKS=Łukasz to the testing env so that this never regresses.

----------
messages: 397572
nosy: lukasz.langa
priority: normal
severity: normal
status: open
title: Non-ASCII characters in os.environ cause silent failures in test_httpservers
type: behavior
versions: Python 3.10, Python 3.11, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue44647>
_______________________________________