Permission denied when opening a file that was created concurrently by os.rename (Windows)

[Alexey Izbyshev]

On 2015-04-05 20:45, Alexey Izbyshev wrote:
> Hello!
> 
> I've hit a strange problem that I reduced to the following test case:
> * Run several python processes in parallel that spin in the following 
> loop:
> while True:
>   if os.path.isfile(fname):
>     with open(fname, 'rb') as f:
>       f.read()
>     break
> * Then, run another process that creates a temporary file and then
> renames it to the name than other processes are expecting
> * Now, some of the reading processes occasionally fail with
> "Permission denied" OSError
> 
> I was able to reproduce it on two Windows 7 64-bit machines. It seems
> when the file appears on the filesystem it is still unavailable to
> reading, but I have no idea how it can happen. Both source and
> destination files are in the same directory, and the destination
> doesn't exist before calling os.rename. Everything I could find
> indicates that os.rename should be atomic under this conditions even
> on Windows, so nobody should be able to observe the destination in
> unaccessible state.
> 
> I know that I can workaround this problem by removing useless
> os.path.isfile() check and wrapping open() with try-except, but I'd
> like to know the root cause of the problem. Please share you thoughts.
> 
> The test case is attached, the main file is test.bat. Python is
> expected to be in PATH. Stderr of readers is redirected to *.log. You
> may need to run several times to hit the issue.
> 
> Alexey Izbyshev,
> research assistant,
> ISP RAS
The attachment was rejected because of .bat files, so here is the code:
test.bat:
======================
@echo off
del test.txt
del *.log

for /L %%i in (1, 1, 10) do (
   start /b run-with-log.bat %%i
)

pythonw test-rename.py
======================
run-with-log.bat:
======================
@echo off
pythonw test-read.py %1 >nul 2>%1.log
exit
======================
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.zip
Type: application/zip
Size: 501 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20150405/3d0cd807/attachment.zip>