[issue37387] test_compileall fails randomly on Windows when tests are run in parallel

[Eryk Sun]

Eryk Sun <eryksun at gmail.com> added the comment:

> I don't know enough about when and why CPython decides to 
> replace .pyc files 

It's straight-forward in the case of py_compile.compile():

    >>> pyc = py_compile.compile('test.py')
    >>> f = open(pyc)
    >>> try: py_compile.compile('test.py')
    ... except OSError as e: print(e)
    ...
    [WinError 5] Access is denied: '__pycache__\\test.cpython-310.pyc.1914201873840' -> '__pycache__\\test.cpython-310.pyc'

Rewriting the file uses `importlib._bootstrap_external._write_atomic()`, which writes the PYC to a temporary name and then tries to replace the existing file via os.replace(). 

Replacing an existing filename will fail with access denied in Windows if the target filename exists and is already open, mapped as image/data, readonly, or a directory. Notably it's not a sharing violation in the case of an open file, which means it doesn't matter whether the open(s) share delete access or not. In Windows 10 the NT API has a new file rename operation that supports a flag to replace an open file, but in this case we're still stuck with the problem that existing opens have to share delete access. Most opens could share delete access without a problem, but most don't.

----------
components: +Windows
nosy: +eryksun, paul.moore, steve.dower, tim.golden, zach.ware
versions: +Python 3.10, Python 3.11

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue37387>
_______________________________________