[Python-Dev] Coernic Desktop Search versus shutil.rmtree

[Tim Peters]

It took a while to make this connection!  Last night I downloaded this
new free (beer) app for Windows:

    http://www.copernic.com/en/products/desktop-search/index.html

Just a note to say that it's fantastic.  It builds index files for all
the documents on your drive, including PDFs, HTMLs, and Outlook email
stores.  Then you do can seriously fast (sub-second) Boolean searches.
 It's indexed about 10GB of data on my drive with about a million
keywords, with 0 errors and 0 glitches, and the search quality is very
good.

Anyway, today I saw a really weird failure in the Zope X3 test suite,
shutil.rmtree() complaining that it couldn't remove a directory. 
Studying the test didn't turn up any plausible cause for this. 
Tonight I was running the Python CVS test suite, and it failed once in
the same mysterious way.  Then it failed again that way, but in
another test.  I eventually reduced it to this:

"""
import os
import shutil

LOCALEDIR = os.path.join('xx', 'LC_MESSAGES')
MOFILE  = os.path.join(LOCALEDIR, 'gettext.mo')
UMOFILE = os.path.join(LOCALEDIR, 'ugettext.mo')
MMOFILE = os.path.join(LOCALEDIR, 'metadata.mo')


class Drive:
    def setUp(self):
        if os.path.isdir(LOCALEDIR):
            shutil.rmtree(os.path.split(LOCALEDIR)[0])
        os.makedirs(LOCALEDIR)
        fp = open(MOFILE, 'wb');  fp.write('a'); fp.close()
        fp = open(UMOFILE, 'wb'); fp.write('b'); fp.close()
        fp = open(MMOFILE, 'wb'); fp.write('c'); fp.close()
        shutil.rmtree(os.path.split(LOCALEDIR)[0])

d = Drive()
while True:
    d.setUp()
    print '.',
"""

That failed every time, after printing from 0 to 100 dots, while
trying to rmdir xx/LC_MESSAGES.

The cause:  Windows has low-level hooks for apps that want to monitor
changes to the filesystem.  For example, virus scanners use those
heavily.  Coernic also uses them, to reindex changed files in the
background.  So it can keep a file open beyond the time Python thinks
it deleted it, and then trying to rmdir its parent directory fails
(because the directory isn't really empty yet).

Stopping the Desktop Search process makes these problems go away.  It
also appears to cure a range of incomprehensible complaints from large
CVS updates that starting showing up last night <wink>.  Ah, there's
an option to keep the search app running but to turn off the
filesystem hooking -- that cures it too.

Anyway, this is worth sharing because this has got to be the next PC
Killer App genre:  finding info on a 120GB disk has become impossible,
and I switched most of my email to a gmail account because I can't
even find "important" email from last week using Outlook anymore.  If
you don't run an app like Coernic yet, you will soon <0.5 wink>.