[Python-Dev] windows buildbot failures

[Tim Peters]

[Neal Norwitz]
> The windows buildbot slaves (cygwin too) are still having problems
> with the DLL being in use when we start compiling so the compile
> fails.  clean.bat is not called afterwards based on the buildbot log.
> I don't know if clean fixes the problem.  If it does, would this patch
> fix the problem:
>
> Index: Tools/buildbot/build.bat
> ===================================================================
> --- Tools/buildbot/build.bat    (revision 45475)
> +++ Tools/buildbot/build.bat    (working copy)
> @@ -1,4 +1,5 @@
>   @rem Used by the buildbot "compile" step.
> +cmd /c Tools\buildbot\clean.bat
>   cmd /c Tools\buildbot\external.bat
>  call "%VS71COMNTOOLS%vsvars32.bat"
>   devenv.com /useenv /build Debug PCbuild\pcbuild.sln

I doubt that will solve it, but you can check it in to give it a try
(and revert the checkin if it doesn't help).

> If the patch won't fix the problem, is there something else we can do
> to ensure the python DLL is no longer used regardless of whether the
> previous test passed or not?  If we can get the process handle, can we
> can subprocess.TerminateProcess()?

I don't know.  Typically (but not always, and in IME), we get into this like so:

1. The buildbot slave terminates the test run "early" for some reason.
   Maybe because the slave lost its connection to the master, or
   maybe because too much time has elapsed since the buildbot
   saw any output from the test process.

2. The buildbot code tries to kill the process itself.  It appears (to judge
   from the buildbot messges) that this never works on Windows.

3. For reasons that are still unknown, python_d.exe keeps running,
   and forever.

So long as the DLL is in use, Windows will not allow it to be deleted
or renamed, so all subsequent attempts to compile fail.

Now I'm not really sure _which_ process #2's "the process" may be.  If
it's the python_d.exe process, then I conclude that the programmatic
way the buildbot uses to try to kill python_d.exe doesn't work in this
situation.  I never have any trouble killing it "by hand" using Task
Manager, and that's all I know for sure.