[New-bugs-announce] [issue11284] slow close file descriptors in subprocess, popen2, os.pepen*

[s7v7nislands]

New submission from s7v7nislands <s7v7nislands at gmail.com>:

when use popen*() and close_fds is True, python will close unused fds. but the MAXFD is not the real max. especially in freebsd, subprocess.MAXFD=655000. so python will try to close to many fd, it's too slow, in my test on freebsd, using about 3 seconds.

poor english.

patch for 2.7 trunck:
jiangxiaobing at s7v7nislands ~/source/svn/python27 $ svn diff
Index: Lib/subprocess.py
===================================================================
--- Lib/subprocess.py	(revision 88499)
+++ Lib/subprocess.py	(working copy)
@@ -1065,11 +1065,16 @@
 
 
         def _close_fds(self, but):
+            maxfd = MAX_FD
+            try:
+                maxfd = os.dup(0) + 1
+            except:
+                pass
             if hasattr(os, 'closerange'):
                 os.closerange(3, but)
-                os.closerange(but + 1, MAXFD)
+                os.closerange(but + 1, maxfd)
             else:
-                for i in xrange(3, MAXFD):
+                for i in xrange(3, maxfd):
                     if i == but:
                         continue
                     try:
Index: Lib/popen2.py
===================================================================
--- Lib/popen2.py	(revision 88499)
+++ Lib/popen2.py	(working copy)
@@ -82,8 +82,13 @@
     def _run_child(self, cmd):
         if isinstance(cmd, basestring):
             cmd = ['/bin/sh', '-c', cmd]
-        os.closerange(3, MAXFD)
+        maxfd = MAXFD
         try:
+            maxfd = os.dup(0) + 1
+        except:
+            pass
+        os.closerange(3, maxfd)
+        try:
             os.execvp(cmd[0], cmd)
         finally:
             os._exit(1)



patch for 3.3 truck:
diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index c02fb52..98a25b3 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -1112,8 +1112,14 @@ class Popen(object):
                 if fd >= start_fd:
                     os.closerange(start_fd, fd)
                     start_fd = fd + 1
-            if start_fd <= MAXFD:
-                os.closerange(start_fd, MAXFD)
+            maxfd = MAXFD
+            try:
+                maxfd = os.dup(0) + 1
+            except:
+                pass
+
+            if start_fd <= maxfd:
+                os.closerange(start_fd, maxfd)
 
 
         def _execute_child(self, args, executable, preexec_fn, close_fds,

----------
components: Library (Lib)
files: py3k.patch
keywords: patch
messages: 129043
nosy: s7v7nislands
priority: normal
severity: normal
status: open
title: slow close file descriptors in subprocess, popen2, os.pepen*
type: performance
versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file20834/py3k.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11284>
_______________________________________