should these be fixed for python 2.4?

[Alexander Schmolck]

Donn Cave <donn at u.washington.edu> writes:

> spawnv() function, and presumably Perl's system
> as shown above, are distinctly different from
> system(3), in that they execute the command directly
> with the indicated parameters, where system()
> executes the shell to interpret shell command text.
>
> In my opinion Python got this much righter than Perl
> when it used a different name.  

I think who got it righter depends the respective ratios of buggy code; I'd be
not be surprised if only a fraction of python code that simply attempts to run
an external command with certain arguments (and no shell preprocessing) worked
correctly.

I appreciate your point but I suspect that at the moment many people are
extremely likely to do the wrong thing, because it's much more obvious than
the right thing.

So even if changing system is the wrong step, I think there really ought to be
an obvious and simple way to accomplish what perl's system($cmd, @args) does.


> Unlike the case with Popen3, where the same function represents 2 different
> APIs, both the string and the list - I see later in your message you may be
> among the people who have been led astray by that.

BTW what is one supposed to use in lieu of the various popen* commands which
all only accept a cmd string rather than?

>> P.S. speaking of running external processes -- I'm also unaware of a good,
>> platform-independent (or even just unix) way to run a process that possibly
>> accepts input, but will just continue on EOF from stdin and getting back
>> stdout and stderr (if supported) as strings and the exit status.
>> 
>> I currently use this to run e.g. latex, but it doesn't seem to work 
>> absolutely
>> reliably and is unix only. I'm pretty sure there must be a better way.
>
> Yes, but either you use disk files or it's more work.
>
> The unreliability, I imagine is due to limitations of pipes'
> device buffer space.  You're trying to execute the whole
> thing within those limits - you wait() for process exit
> before you have read a byte.  It would be better to read
> output first, since that will terminate when the process
> exits, and then wait() to retrieve the status.  

Hm, I think that might be what I tried first, but that resulted in hangs.

> A fully reliable version needs to select on the two pipes to make sure that
> the error pipe doesn't fill up while you're trying to read all the output.
> Use os.read on the file descriptor so you're on the same page with select().

Based on your suggestion I tried this, but it hangs because the files remain
ready for reading. Do I have to check if nothing has been read from both
stderr and stdout? (I'm not really particularly clued about low-level system
stuff):

def readProcess(cmd, *args):
    r"""Like `os.popen3`, but returns 2 strings (stdin, stdout) and the exit
    code (unlike popen2, exit is 0 if no problems occured (for some bizzarre
    reason popen2 returns None... <sigh>). FIXME: only works for UNIX!"""
    BUFSIZE=1024**2
    import select
    cmdStr = _fixupCmdStr(cmd, args)
    popen = popen2.Popen3(cmdStr, capturestderr=True)
    which = {id(popen.fromchild): [],
             id(popen.childerr):  []}
    try:
        popen.tochild.close() # XXX make sure we're not waiting forever
        while True:
            ready = select.select([popen.fromchild, popen.childerr],[],[])
            for f in ready[0]:
                while True:
                    read = os.read(f.fileno(), BUFSIZE)
                    if not read:
                        break
                    which[id(f)].append(read)
            if not ready:
                break
        out, err = ["".join(which[id(f)])
                    for f in [popen.fromchild, popen.childerr]]
        exit = popen.wait()
    finally:
        try:
            popen.fromchild.close()
        finally:
            popen.childerr.close()
    return out or "", err or "", exit


> Or on the other hand, temporary disk files don't have these problems and are
> of course more portable.

Can you elucidate  a bit further? How do I portably redirect stdout and
stderr to tempfiles, or is that not what you meant?

thanks,

'as