calling python scripts as a sub-process

[Catherine Moroney]

Dan Upton wrote:
> On Wed, Nov 19, 2008 at 2:38 PM, Catherine Moroney
> <Catherine.M.Moroney at jpl.nasa.gov> wrote:
>> Dan Upton wrote:
>>> On Wed, Nov 19, 2008 at 2:13 PM, Philip Semanchuk <philip at semanchuk.com>
>>> wrote:
>>>> On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:
>>>>
>>>>> The command (stored as an array of strings) that I'm executing is:
>>>>>
>>>>> ['python ../src_python/Match1.py ',
>>>>> '--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
>>>>> '--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
>>>>> '--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
>>>>> ",
>>>>> "--use_textid='true '"]
>>>>>
>>>> [snip]
>>>>
>>>>> I get the error below.  Does anybody know what this error refers
>>>>> to and what I'm doing wrong?  Is it even allowable to call another
>>>>> script as a sub-process rather than calling it directly?
>>>>>
>>>>> File "../src_python/Match4.py", line 24, in RunMatch4
>>>>>  sub1 = subprocess.Popen(command1)
>>>>>  File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
>>>>>  errread, errwrite)
>>>>>  File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
>>>>>  raise child_exception
>>>>> OSError: [Errno 2] No such file or directory
>>>> Try supplying a fully-qualified path to your script, e.g.:
>>>> ['python /home/catherine/src_python/Match1.py ',
>>>> '--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf ',
>>>> '--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf ',
>>>> '--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
>>>> ",
>>>> "--use_textid='true '"]
>>> I think when I came across this error, I added shell=True, e.g.
>>>
>>> sub1 = subprocess.Popen(command, shell=True)
>> I added the shell=True and this time it got into Match1 (hurrah!),
>> but it then opened up an interactive python session, and didn't
>> complete until I manually typed 'exit' in the interactive session.
>>
>> Match1 looks like:
>>
>> if __name__ == "__main__":
>> <<< parse arguments >>>
>>
>>    RunMatch1(file_ref, file_cmp, iblock_start, iblock_end, \
>>              nlinep, nsmpp, mindispx, maxdispx, mindispl,  \
>>              maxdispl, istep, chmetric, use_textid)
>>
>>    exit()
>>
>> where the routine RunMatch1 does all the actual processing.
>>
>> How do I get Match1 to run and exit normally without opening up an
>> interactive session, when called as a subprocess from Match4?
>>
> 
> Alternately, rather than using a list of arguments, have you tried
> just using a string?  (Again, that's the way I do it and I haven't
> been having any problems recently, although I'm running shell scripts
> or binaries with arguments rather than trying to invoke python on a
> script.)
> 
> command = "python ../src_python/Match1.py
> --file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF_F03_0024.hdf
> --file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF_F03_0024.hdf
> --block_start=62 --block_end=62 --istep=16 --chmetric='M2'
> --use_textid=true"
> 
> proc = subprocess.Popen(command, shell=True)

Thanks - that did the trick.  I just passed in one long string
and everything actually works.  Wow!  I had no idea if this was
even do-able.

This is so cool, and saves me a lot of code duplication.  I can
spawn off half a dozen jobs at once and then just wait for them
to finish.  It's great that python can function both as a
scripting language and also a full-blown programming language
at the same time.

Catherine