Python popenX() slowness on AIX?

[allenjo5 at mail.northgrum.com]

Stefaan A Eeckels wrote:
> On 24 Nov 2006 09:03:41 -0800
> allenjo5 at mail.northgrum.com wrote:
>
> > Stefaan A Eeckels wrote:
> > > On 21 Nov 2006 13:02:14 -0800
> > > allenjo5 at mail.northgrum.com wrote:
> > >
> > > > The fact that it does this in Python code instead of C is the main
> > > > cause of the slowness.  So, unless Python is changed to do this
> > > > in C, it's always going to be slow on AIX :-(
> > >
> > > I guess that the reason it's slow is that there are many
> > > descriptors to try and close. Reducing them using ulimit -n could
> > > improve the speed.
> > >
> > > AIX has a fcntl command to close all open file descriptors from a
> > > descriptor onwards:
> > >
> > > fcntl(3, F_CLOSEM);
> > >
> > > This of course should be used instead of the loop:
> > >
> > > 10 happens to be the value of F_CLOSEM (from /usr/include/fcntl.h).
> > > I've currently no access to an AIX system with Python, but it could
> > > be worth trying.
> >
> > Yes, very much worth it.  F_CLOSEM is _so_ much better than the loop,
> > even in C.   Using your brilliant suggestion, I now have a simple
> > patch to the python source that implements it for any OS that happens
> > to have the fcntl F_CLOSEM option.
>
> The *BSDs and Solaris have "closefrom(3)" which does the same as
> F_CLOSEM in AIX. As with AIX, the speedup is dramatic when there are a
> lot of file descriptors to try and close.
>
> > It is below in its entirety.  I believe I got the try: stuff correct,
> > but since I'm new to Python, I'd appreciate any comments.
>
> I'm no great Python specialist myself. I'll leave it to those better
> qualified to comment.
>
> > I have another patch to implement my os.rclose(x,y) method, which
> > would improve the speed of popenX() for the OSes that don't have
> > F_CLOSEM, by doing the close() loop in C instead of Python,  but I
> > don't know if it would be as likely to be accepted as this probably
> > would be.
> >
> > Now, where do I send my proposed patch for consideration?
>
> The README file in the Python distribution has the following to say:
>
> Patches and contributions
> -------------------------
>
> To submit a patch or other contribution, please use the Python Patch
> Manager at http://sourceforge.net/patch/?group_id=5470.  Guidelines
> for patch submission may be found at http://www.python.org/patches/.

Thanks.  I joined sourceforge, and submitted this patch:

http://sourceforge.net/tracker/index.php?func=detail&aid=1607087&group_id=5470&atid=305470

For some reason, I could not submit it using Firefox 1.5, so I had to
relent and use IE.

John.