subprocess call is not waiting.

[Chris Rebert]

On Fri, Sep 14, 2012 at 5:22 AM,  <paulstaten at gmail.com> wrote:
> os.system worked fine, and I found something in another section of code that was causing the "Too many open errors." (I was fooled, because output from subprocess call didn't seem to be coming out until the open files error.
>
> I'll go back and play with subprocess.call more, since os.system works. That's interesting about using shlex at run time. Is that just for the sake of computational cost?

No, like I said, you'll also get incorrect results. shlex isn't magic.
If the exact command line it's given wouldn't work in the shell, then
it won't magically fix things. Many (most?) dynamic invocations of
shlex.split() are naive and flawed:

>>> import shlex
>>> filename = "my summer vacation.txt"
>>> # the following error is less obvious when the command is more complex
>>> # (and when the filename isn't hardcoded)
>>> cmd = "cat " + filename
>>> shlex.split(cmd)
['cat', 'my', 'summer', 'vacation.txt']
>>> # that's wrong; the entire filename should be a single list element

Equivalent bash error:
chris at mbp ~ $ cat my summer vacation.txt
cat: my: No such file or directory
cat: summer: No such file or directory
cat: vacation.txt: No such file or directory

The right way, in bash:
chris at mbp ~ $ cat my\ summer\ vacation.txt
Last summer, I interned at a tech company and...
chris at mbp ~ $ cat 'my summer vacation.txt'
Last summer, I interned at a tech company and…

And indeed, shlex will get that right too:
>>> shlex.split("cat my\ summer\ vacation.txt")
['cat', 'my summer vacation.txt']
>>> shlex.split("cat 'my summer vacation.txt'")
['cat', 'my summer vacation.txt']

BUT that presumes that your filenames are already pre-quoted or have
had backslashes added, which very seldom is the case in reality. So,
you can either find an escaping function and hope you never forget to
invoke it (cf. SQL injection), or you can figure out the general
tokenization and let `subprocess` handle the rest (cf. prepared
statements):

>>> split('cat examplesimplefilename')
['cat', 'examplesimplefilename']
>>> # Therefore…
>>> def do_cat(filename):
...     cmd = ['cat', filename] # less trivial cases would be more interesting
...     call(cmd)
...
>>> filename = "my summer vacation.txt"
>>> # remember that those quotes are Python literal syntax and aren't in the string itself
>>> print filename
my summer vacation.txt
>>> do_cat(filename)
Last summer, I interned at a tech company and…
>>>

Generally, use (a) deliberately simple test filename(s) with shlex,
then take the resulting list and replace the filename(s) with (a)
variable(s).

Or, just figure out the tokenization without recourse to shlex; it's
not difficult in most cases!
The Note in the Popen docs covers some common tokenization mistakes people make:
http://docs.python.org/library/subprocess.html#subprocess.Popen

Cheers,
Chris