Strange UnicodeEncodeError in Windows image on Azure DevOps and Github

[Jessica Smith]

Hello,

Weird issue I've found on Windows images in Azure Devops Pipelines and
Github actions. Printing Unicode characters fails on these images because,
for some reason, the encoding is mapped to cp1252. What is particularly
weird about the code page being set to 1252 is that if you execute "chcp"
it shows that the code page is 65001.

At the end of this email are the cleaned up logs from GH actions. The
actions are very simple - print out unicode characters using echo to prove
the characters can be printed to the console. The rest of the commands are
in Python, and they include printing out the "encoding" variable of
sys.stdout, as well as printing sys.flags and sys.getfilesystemencoding.
Then print the same unicode character using print, which causes a
UnicodEncodeError because the character isn't in the cp1252 charmap.

I've also uploaded the logs to pastebin here: https://pastebin.com/ExzGRHav
I also uploaded a screenshot to imgur, since the logs are not the easiest
to read. https://imgur.com/a/dhvLWOJ

I'm trying to determine why this issue only happens on these images - I can
replicate it on multiple versions of Python (from 3.9 to 3.7 at least,
haven't tried more), but I can't replicate this on my own machines.

There are a few issues on GH regarding this issue but they seem to stay
open since they are hard to replicate. Here are the ones I have stumbled
upon while researching this.

https://github.com/databrickslabs/dbx/issues/455
https://github.com/PrefectHQ/prefect/issues/5754
https://github.com/pallets/click/issues/2121

Any insight or ideas on how to test and validate the cause would be great.
I'm pulling my hair out trying to find the root cause of this - not because
it really matters to any of my processes but because it is weird and broken.

Thanks for any help,

Jessica

Begin Logs:

2022-11-10T23:54:51.7272453Z Requested labels: windows-latest
2022-11-10T23:54:51.7272494Z Job defined at:
NodeJSmith/wsl_home/.github/workflows/blank.yml at refs/heads/main
2022-11-10T23:54:51.7272514Z Waiting for a runner to pick up this job...
2022-11-10T23:54:52.3387510Z Job is waiting for a hosted runner to come
online.
2022-11-10T23:55:04.8574435Z Job is about to start running on the hosted
runner: Hosted Agent (hosted)
2022-11-10T23:55:15.8332600Z Current runner version: '2.298.2'

2022-11-10T23:55:15.8366947Z ##[group]Operating System
2022-11-10T23:55:15.8367650Z Microsoft Windows Server 2022
2022-11-10T23:55:15.8367954Z 10.0.20348
2022-11-10T23:55:15.8368389Z Datacenter
2022-11-10T23:55:15.8368696Z ##[endgroup]

2022-11-10T23:55:15.8369023Z ##[group]Runner Image
2022-11-10T23:55:15.8369654Z Image: windows-2022
2022-11-10T23:55:15.8369931Z Version: 20221027.1
2022-11-10T23:55:15.8370539Z Included Software:
https://github.com/actions/runner-images/blob/win22/20221027.1/images/win/Windows2022-Readme.md
2022-11-10T23:55:15.8371174Z Image Release:
https://github.com/actions/runner-images/releases/tag/win22%2F20221027.1
2022-11-10T23:55:15.8371622Z ##[endgroup]

2022-11-10T23:55:15.8371955Z ##[group]Runner Image Provisioner
2022-11-10T23:55:15.8372277Z 2.0.91.1
2022-11-10T23:55:15.8372514Z ##[endgroup]

2022-11-10T23:55:16.3619998Z ##[group]Run echo  " └── ID:"
2022-11-10T23:55:16.3620626Z echo  " └── ID:"
2022-11-10T23:55:16.3927292Z shell: C:\Program Files\PowerShell\7\pwsh.EXE
-command ". '{0}'"
2022-11-10T23:55:16.3927894Z ##[endgroup]
2022-11-10T23:55:32.9958751Z  └── ID:

2022-11-10T23:55:34.0835652Z ##[group]Run chcp
2022-11-10T23:55:34.0836104Z chcp
2022-11-10T23:55:34.0878901Z shell: C:\Program Files\PowerShell\7\pwsh.EXE
-command ". '{0}'"
2022-11-10T23:55:34.0879350Z ##[endgroup]
2022-11-10T23:55:34.4878247Z Active code page: 65001

2022-11-10T23:55:34.7917219Z ##[group]Run python -c "import sys;
print('sys.stdout.encoding', sys.stdout.encoding);
print('sys.flags',sys.flags);print('sys.getfilesystemencoding',sys.getfilesystemencoding())"
2022-11-10T23:55:34.7918148Z python -c "import sys;
print('sys.stdout.encoding', sys.stdout.encoding);
print('sys.flags',sys.flags);print('sys.getfilesystemencoding',sys.getfilesystemencoding())"
2022-11-10T23:55:34.7960873Z shell: C:\Program Files\PowerShell\7\pwsh.EXE
-command ". '{0}'"
2022-11-10T23:55:34.7961202Z ##[endgroup]
2022-11-10T23:55:36.2324642Z sys.stdout.encoding cp1252
2022-11-10T23:55:36.2325910Z sys.flags sys.flags(debug=0, inspect=0,
interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0,
no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0,
hash_randomization=1, isolated=0, dev_mode=False, utf8_mode=0)
2022-11-10T23:55:36.2327055Z sys.getfilesystemencoding utf-8

2022-11-10T23:55:36.4553957Z ##[group]Run python -c "print('└── ID:')"
2022-11-10T23:55:36.4554395Z python -c "print('└── ID:')"
2022-11-10T23:55:36.4595413Z shell: C:\Program Files\PowerShell\7\pwsh.EXE
-command ". '{0}'"
2022-11-10T23:55:36.4595740Z ##[endgroup]
2022-11-10T23:55:36.8739309Z Traceback (most recent call last):
2022-11-10T23:55:37.1316425Z   File "<string>", line 1, in <module>
2022-11-10T23:55:37.1317452Z   File
"C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\encodings\cp1252.py",
line 19, in encode
2022-11-10T23:55:37.1324632Z     return
codecs.charmap_encode(input,self.errors,encoding_table)[0]
2022-11-10T23:55:37.1325913Z UnicodeEncodeError: 'charmap' codec can't
encode characters in position 0-2: character maps to <undefined>