check whether a process is still running on remote hosts

[Cameron Simpson]

On 16Oct2018 14:14, tina_zy_qian at yahoo.com <tina_zy_qian at yahoo.com> wrote:
>I newly learned Python, and I need to wrap up a script to do following.
>
>Env:
>1: One launcher Linux VM (from where to run the Python script)
>2. 100+ Linux VM
>
>requirement:
>In general, run a remote_script on remote 100 VMs and get the log files generated to remote hosts back to launcher.
>
>steps
>1. cp (pexpect.spawn('scp  ')) supporting files including the remote_script to remotehost:/remote_folder
>2. run remote_script on remote host  ( pexpect.spawn('ssh %s@%s "%s"' % (user,host,cmd))
>3. wait and check until the remote_script was run on remote host. (to check "ps -ef" result on remote hosts)
>4. collect data from remote hosts -- reverse to step 1.
>
>
>I briefly implemented other steps expect step 3. Two options are below.
>
>option 1: run another script "ps -ef|grep remote_script and output result to a local file, then collected the files to launcher.
>
>option 2: run pexect.spawn('ssh' ....'ps -e') to get the result directly to laucher console.
>But how I can get only the output for "ps -ef" command only?
>
>I must use the (users, passwords, and hosts) way to do ssh and scp 
>because I may not be allowed to use ssh key on some hosts. Any 
>suggestions or sample code for step 3 or the whole script are 
>appreciated. Thanks.

Have you looked at ansible? It is a Python/ssh library/tool for remote 
admin. It is almost designed for this kind of thing. Run local script on 
remote hosts in parallel and wait for them?  Tick.

Ansible can prompt (once) for a remote ssh password and apply it to all 
the remote ssh invocations.

I _strongly_ recommend you do this with keys though. Passwords are 
something of a security nightmare.

You can be sure that the remote script has run if the ssh which invoked 
it returned with a zero exit status. If the ssh itself aborts, or the 
script fails, you get a nonzero status back from ssh.

Personally I would be (a) trying to do this without passwords - install 
keys and (b) using threads to spawn all the remote sshes, have each 
thread wait for its own ssh, and wait for all the threads.

Or reaching for ansible, which has this kind of logic built in.

Cheers,
Cameron Simpson <cs at cskk.id.au>