Question about unreasonable slowness

[allenjo5 at mail.northgrum.com]

allenjo5 at mail.northgrum.com wrote:
> James Antill wrote:
> > On Fri, 17 Nov 2006 12:39:16 -0800, allenjo5 wrote:
> >
> > >> allenjo5 at mail.northgrum.com:
> > >> > Ok, I built Python 2.5 (same AIX 5.1 machine).  With the "for line in
> > >> > shellOut" loop in, it now takes "only" 7 secs instead of the 11 secs in
> > >> > python 2.4.3.   So, that's better, but still unreasonably slow.  And to
> > >> > answer another's question, I'm using the ksh builtin 'time' command to
> > >> > time the overall script.
> > >> >
> > >> > BTW, I don't think the AIX /bin/sh (actually ksh) is inherently slow.
> > >> > This naively translated pure shell version of my python test script
> > >> > completes in .1 secs:
> > >> >
> > >> > i=1
> > >> > while ((i<20))
> > >> > do ((i+=1))
> > >> >    print next
> > >> >    print "$shellIn" | /bin/sh -c ':' |
> > >> >            while read line
> > >> >            do print $line
> > >> >            done
> > >> > done
> > >>
> > > Yeah, apparently this is an AIX specific issue - perhaps the python
> > > implementation of popen4() needs to do something special for AIX?
> >
> >  This seems likely a more general issue, rather than just a python issue
> > (although the huge speed up from moving to 2.5.x). A
> > couple of things I'd try:
>
> With help from c.u.aix, I've discovered the problem.  Python (in
> popen2.py) is attempting to close filedescriptors 3 through 32767
> before running the /bin/sh.   This is because os.sysconf('SC_OPEN_MAX')
> is returning 32767.   So far, it looks like SC_OPEN_MAX is being set
> correctly to 4 in posixmodule.c, and indeed, os.sysconf_names seems to
> also have SC_OPEN_MAX set to 4:
>
> python -c 'import os;  print os.sysconf_names'
>
> ...
> 'SC_XOPEN_XCU_VERSION': 109, 'SC_OPEN_MAX': 4, 'SC_PRIORITIZED_IO': 91,
> ...
>
> In fact, none of the values that sysconf_names has set for the various
> constants are being returned by os.sysconf().  For example, the 2
> others I just listed:
>
> $ ./python -c 'import os; print os.sysconf("SC_XOPEN_XCU_VERSION")'
> 4
>
> $ ./python -c 'import os; print os.sysconf("SC_PRIORITIZED_IO")'
> -1
>
> This makes no sense to me... unless there is some memory alignment or
> endian issue going on here?

More info: clearly I had no idea what I was talking about :-)

The numbers associated with the names returned by os.sysconf_names are
the indices to an array that the C sysconf() function uses to return
the value of the name.   So, the fact that os.sysconf("SC_OPEN_MAX")
was returning 32767 on AIX is correct.   However, the slowness this
causes is still an issue.   This is because python is closing all these
file descriptors in python code, not C code - specifically, in
popen2.py:

try:
    MAXFD = os.sysconf('SC_OPEN_MAX')
except (AttributeError, ValueError):
    MAXFD = 256

...

    def _run_child(self, cmd):
        if isinstance(cmd, basestring):
            cmd = ['/bin/sh', '-c', cmd]
        for i in range(3, MAXFD):
            try:
                os.close(i)
            except OSError:
                pass
        try:
            os.execvp(cmd[0], cmd)
        finally:
            os._exit(1)


Any chance the "for i in range(3, MAXFD):" loop could be done in C
instead?  Even having, say, an os.rclose(x,y) low level function to
close all file descriptors in range [x,y] would be great.

John.