Permission denied when opening a file that was created concurrently by os.rename (Windows)

[Dave Angel]

On 04/05/2015 01:45 PM, Alexey Izbyshev wrote:
> Hello!
>
> I've hit a strange problem that I reduced to the following test case:
> * Run several python processes in parallel that spin in the following loop:
> while True:
>    if os.path.isfile(fname):
>      with open(fname, 'rb') as f:
>        f.read()
>      break
> * Then, run another process that creates a temporary file and then
> renames it to the name than other processes are expecting
> * Now, some of the reading processes occasionally fail with "Permission
> denied" OSError
>
> I was able to reproduce it on two Windows 7 64-bit machines. It seems
> when the file appears on the filesystem it is still unavailable to
> reading, but I have no idea how it can happen. Both source and
> destination files are in the same directory, and the destination doesn't
> exist before calling os.rename. Everything I could find indicates that
> os.rename should be atomic under this conditions even on Windows, so
> nobody should be able to observe the destination in unaccessible state.
>
> I know that I can workaround this problem by removing useless
> os.path.isfile() check and wrapping open() with try-except, but I'd like
> to know the root cause of the problem. Please share you thoughts.
>
> The test case is attached, the main file is test.bat. Python is expected
> to be in PATH. Stderr of readers is redirected to *.log. You may need to
> run several times to hit the issue.
>
> Alexey Izbyshev,
> research assistant,
> ISP RAS
>

The attachment is missing;  please just include it inline, after 
reducing it to a reasonably minimal sample.

My guess is that the process that does the os.rename is not closing the 
original file before renaming it.  So even though the rename is atomic, 
the file is still locked by the first process.



-- 
DaveA