subprocess and stdin.write(), stdout.read()

[Nobody]

On Tue, 24 Mar 2015 12:08:24 -0700, Tobiah wrote:

> But if I want to send a string to stdin, how can I do that without
> stdin.write()?

	p.communicate(string)

> This seems to work:

Only because the amounts of data involved are small enough to avoid
deadlock.

If both sides write more data in one go than will fit into a pipe buffer,
you will get deadlock. The parent will be blocked waiting for the child to
consume the input, which doesn't happen because the child will be blocked
waiting for the parent to consume its output, which doesn't happen because
he parent will be blocked waiting for the child to consume the input, ...

That's a textbook example of deadlock: each side waiting forever for the
other side to make the first move.

This is exactly why POSIX' popen() function lets you either write to stdin
(mode=="w") or read from stdout (mode=="r") but not both.

> Will this always avoid the deadlock problem?

No.

> This also works:

Again, only because the amounts of data involved are small enough to avoid
deadlock.

> Is that vulnerable to deadlock?

Yes.

> Is there a better way to write to and read from the same process?

Use threads; one for each descriptor (stdin, stdout, stderr).

Non-blocking I/O is an alternative (and that's what .communicate() uses on
Unix), but threads will work on all common desktop and server platforms.

If you need to support platforms which lack threads, either

a) have the parent first write to a file (instead of .stdin), then have
the child read from the file while the parent reads .stdout and .stderr,
or

b) have the parent write to .stdin while the child writes its
stdout/stderr to files (or a file). Once the child completes, have
the parent read the file(s).

Using files allows for potentially gigabytes of data to be buffered. With
pipes, the amount may be as low as 512 bytes (the minimum value allowed by
POSIX) and will rarely be much more (a typical value is 4096 bytes, i.e.
one "page" on x86).