[Python-Dev] windows file closing race condition?

[Nick Coghlan]

On 6 September 2013 15:50, Chris Withers <chris at simplistix.co.uk> wrote:
> Hi All,
>
> Continuous testing is a wonderful thing when it comes to finding weird edge
> case problems, like this one:
>
> http://jenkins.simplistix.co.uk/job/testfixtures-tox/COMPONENTS=zc,PYTHON=3.3,label=windows/149/testReport/junit/testfixtures.tests.test_tempdirectory/TempDirectoryTests/test_check_all_tuple/
>
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py",
> line 323, in __exit__
>     self.cleanup()
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py",
> line 78, in cleanup
>     rmtree(self.path)
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py",
> line 460, in rmtree
>     return _rmtree_unsafe(path, onerror)
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py",
> line 362, in _rmtree_unsafe
>     _rmtree_unsafe(fullname, onerror)
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py",
> line 371, in _rmtree_unsafe
>     onerror(os.rmdir, path, sys.exc_info())
>   File
> "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py",
> line 369, in _rmtree_unsafe
>     os.rmdir(path)
> OSError: [WinError 145] The directory is not empty:
> 'c:\\users\\jenkins\\appdata\\local\\temp\\tmpkeg4d7\\a'
>
> I'm 99% certain my code is correct here, the only place I open files for
> writing in that directory is here:
>
> https://github.com/Simplistix/testfixtures/blob/master/testfixtures/tempdirectory.py#L275
>
> So, from my perspective, I'm either looking at a bug in shutil.rmtree (which
> would be trying to delete a directory before deleting its content or failing
> to delete a file but ignoring an error) or the file object, when being used
> as a context manager, going through __exit__ without closing the file and
> releasing the handle.
>
> This happens very infrequently, the OS is Windows 7 and the filesystem is
> NTFS, if that helps...
>
> Any ideas?

This feels a lot like an issue we were seeing on the Windows
buildbots, which we ended up working around in the test support
library: http://bugs.python.org/issue15496

That would be some awfully ugly code to upgrade from "hack in the test
support library" to "this is just how Python unlinks files on
Windows", though :P

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia