how to simulate tar filename substitution across piped subprocess.Popen() calls?

[Hans Mulder]

On 12/11/12 18:22:44, jkn wrote:
> Hi Hans
> 
> On Nov 12, 4:36 pm, Hans Mulder <han... at xs4all.nl> wrote:
>> On 12/11/12 16:36:58, jkn wrote:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> slight followup ...
>>
>>> I have made some progress; for now I'm using subprocess.communicate to
>>> read the output from the first subprocess, then writing it into the
>>> secodn subprocess. This way I at least get to see what is
>>> happening ...
>>
>>> The reason 'we' weren't seeing any output from the second call (the
>>> 'xargs') is that as mentioned I had simplified this. The actual shell
>>> command was more like (in python-speak):
>>
>>> "xargs -I {} sh -c \"test -f %s/{} && md5sum %s/{}\"" % (mydir, mydir)
>>
>>> ie. I am running md5sum on each tar-file entry which passes the 'is
>>> this a file' test.
>>
>>> My next problem; how to translate the command-string clause
>>
>>>     "test -f %s/{} && md5sum %s/{}" # ...
>>
>>> into s parameter to subprocss.Popen(). I think it's the command
>>> chaining '&&' which is tripping me up...
>>
>> It is not really necessary to translate the '&&': you can
>> just write:
>>
>>     "test -f '%s/{}' && md5sum '%s/{}'" % (mydir, mydir)
>>
>> , and xargs will pass that to the shell, and then the shell
>> will interpret the '&&' for you: you have shell=False in your
>> subprocess.Popen call, but the arguments to xargs are -I {}
>> sh -c "....", and this means that xargs ends up invoking the
>> shell (after replacing the {} with the name of a file).
>>
>> Alternatively, you could translate it as:
>>
>>     "if [ -f '%s/{}' ]; then md5sum '%s/{}'; fi" % (mydir, mydir)
>>
>> ; that might make the intent clearer to whoever gets to
>> maintain your code.
> 
> Yes to both points; turns out that my problem was in building up the
> command sequence to subprocess.Popen() - when to use, and not use,
> quotes etc. It has ended up as (spelled out in longhand...)
> 
> 
>         xargsproc = ['xargs']
> 
>         xargsproc.append('-I')
>         xargsproc.append("{}")
> 
>         xargsproc.append('sh')
>         xargsproc.append('-c')
> 
>         xargsproc.append("test -f %s/{} && md5sum %s/{}" % (mydir,
> mydir))

This will break if there are spaces in the file name, or other
characters meaningful to the shell.  If you change if to

        xargsproc.append("test -f '%s/{}' && md5sum '%s/{}'"
                             % (mydir, mydir))

, then it will only break if there are single quotes in the file name.

As I understand, your plan is to rewrite this bit in pure Python, to
get rid of any and all such problems.

> As usual, breaking it all down for the purposes of clarification has
> helpd a lot, as has your input. Thanks a lot.

You're welcome.

-- HansM