[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

[STINNER Victor]

STINNER Victor <vstinner at redhat.com> added the comment:

> In Unix, the zombie remains visible in the task list (marked as <defunct> in Linux), but in Windows an exited process is removed from the Process Manager's active list, so it's no longer visible to users. Also, a Process object is reaped as soon as the last reference to it is closed, since clearly no one needs it anymore. 
> (...)
> We could call self._handle.Close() in _wait(), right after calling GetExitCodeProcess(self._handle). With this change, __exit__ will ensure that _handle gets closed in a deterministic context.

I created bpo-37410 and PR 14391 to close the handle when the process completes. IMHO this change is easy and safe, it can be backported to Python 3.7 and 3.8.

> Code that needs the handle indefinitely can call _handle.Detach() before exiting the with-statement context, but that should rarely be necessary.
> (...)
> There should be a way to indicate a Popen instance is intended to continue running detached from our process, so scripts don't have to ignore an irrelevant resource warning.

My point is that if you want to "detach" a process, it must be *explicit*. In my experience, processes are "leaked" implicitly: by mistake. In multiprocessing, it can be subtle, you don't manipulate subprocess.Popen directly.

Users must not access private attributes (Popen._handle). After I added the ResourceWarning, I created bpo-27068 "Add a detach() method to subprocess.Popen". But I only created to reply to a request of Martin Panter, I didn't use this feature myself. At some point, I decided to just close the issue. Maybe it's time to reopen it.


> I don't understand emitting a resource warning in Popen.__del__ if a process hasn't been waited on until completion beforehand (i.e. self.returncode is None). If a script wants to be strict about this, it can use a with statement, which is documented to wait on the process.

The purpose is to help developers to find bugs in their bugs. Many are now aware that programs continue to run in the background and that leaking zombie processes is an issue on Unix.

The process is not polled to be able to emit the warning in more cases, again, to ease debug.

The warning is ignored by default.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue37380>
_______________________________________