[Python-Dev] Windows buildbot test_bsddb3 problems (was RE: Buildbots for trunk are all red)

[Trent Nelson]

Spent some time on my buildbot (x86 2k8 trunk) this morning trying to track down why test_bsddb3 is failing (trunk with db-4.4.20).  The first test that fails is this:

test01_GetsAndPuts (bsddb.test.test_basics.BasicBTreeWithEnvTestCase) ... ERROR

That's slightly misleading though as the test runs fine -- the actual exception is being thrown in test_basics.BasicTestCase.tearDown() when os.remove() is called against the first db file (i.e. '__db.001'):

WindowsError: [Error 5] Access is denied: 'c:\users\trent~1.nel\appdata\local\temp\2\db_home2808\__db.001

This isn't surprising, given that the python_d.exe process still seems to have __db.001, __db.002 and __db.003 open at the time os.remove() is called.  The aforementioned tearDown() method looks like this:

    def tearDown(self):
        self.d.close()
        if self.env is not None:
            test_support.rmtree(self.homeDir)
            self.env.close()
            ## Make a new DBEnv to remove the env files from the home dir.
            ## (It can't be done while the env is open, nor after it has been
            ## closed, so we make a new one to do it.)
            #e = db.DBEnv()
            #e.remove(self.homeDir)
            #os.remove(os.path.join(self.homeDir, self.filename))
        else:
            os.remove(self.filename)

If I switch the order of statements such that self.env.close() is called before test_suppot.rmtree(self.homeDir), this particular test and a host of others that were also failing now pass (a runtime abort is no longer raised by the CRT half way into the tests either).  (Note that the order was switched to that shown above by Neal in r61052 on Feb 24th, which is when these issues started occurring.)

That said, there are still a lot more test failures down the track once this change has been made, either via the access denied WindowsError, or a DBInvalidArgError, e.g.:

ERROR: test02_WithSource (bsddb.test.test_recno.SimpleRecnoTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\svn\svn.python.org\projects\python\trunk\lib\bsddb\test\test_recno.py", line 33, in tearDown
    test_support.rmtree(self.homeDir)
  File "S:\src\svn\svn.python.org\projects\python\trunk\lib\test\test_support.py", line 70, in rmtree
    shutil.rmtree(path)
  File "S:\src\svn\svn.python.org\projects\python\trunk\lib\shutil.py", line 184, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "S:\src\svn\svn.python.org\projects\python\trunk\lib\shutil.py", line 182, in rmtree
    os.remove(fullname)
WindowsError: [Error 5] Access is denied: 'c:\\users\\trent~1.nel\\appdata\\local\\temp\\2\\db_home4656\\tmp04_knk'
======================================================================
ERROR: test01_1WriterMultiReaders (bsddb.test.test_thread.BTreeConcurrentDataStore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\svn\svn.python.org\projects\python\trunk\lib\bsddb\test\test_thread.py", line 62, in setUp
    self.env.open(homeDir, self.envflags | db.DB_CREATE)
DBInvalidArgError: (22, 'Invalid argument -- configured environment flags incompatible with existing environment')

The DBInvalidArgError exception is only raised after a previous WindowsError is encountered, so I assume it's a side-effect of the tearDown() method not cleaning the environment correctly.

It seems this comment in tearDown() is quite pertinent to our situation:
            ## Make a new DBEnv to remove the env files from the home dir.
            ## (It can't be done while the env is open, nor after it has been
            ## closed, so we make a new one to do it.)
            #e = db.DBEnv()
            #e.remove(self.homeDir)
            #os.remove(os.path.join(self.homeDir, self.filename))

Not sure why this is commented out -- quick search of svn history indicates it's been like that for at least the last year and a half.

Will have some more time this evening to spend on this, however, work calls at the moment.

Regards,

    Trent.

________________________________________
From: python-dev-bounces+tnelson=onresolve.com at python.org [python-dev-bounces+tnelson=onresolve.com at python.org] On Behalf Of Facundo Batista [facundobatista at gmail.com]
Sent: 26 February 2008 06:22
To: Thomas Hervé
Cc: python-dev at python.org
Subject: Re: [Python-Dev] Buildbots for trunk are all red

2008/2/25, Thomas Hervé <therve at free.fr>:

>  I've worked on that problem during the bug day. I've open a ticket with
>  a patch at http://bugs.python.org/issue2168.

Most of the buildbots are green now!!!

Thank you all! This community is as awesome as Python itself, ;)

Three remains in red, though:

- Alpha Tru64: test_smtplib.py is flaky, and _ssl.c is not compiled
correctly. Neil is hunting this, I think.

- X86 XP-3: seems to crash after test_bsddb3.py.

- X86 XP-4: idem. For this two, how can be tried if the bsddb lib in
those windows is correctly installed?

Thanks again.

--
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com