ANN: Sarge, a library wrapping the subprocess module, has been released.

[Vinay Sajip]

On Feb 12, 3:35 pm, Anh Hai Trinh <anh.hai.tr... at gmail.com> wrote:

> I think most users like to use Python, or they'd use Bash. I think people prefer not another language that is different from both, and having little benefits. My own opinion of course.
>

I have looked at pbs and clom: they Pythonify calls to external
programs by making spawning those look like function calls. There's
nothing wrong with that, it's just a matter of taste. I find that e.g.

wc(ls("/etc", "-1"), "-l")

is not as readable as

call(“ls /etc –1 | wc –l”)

and the attempt to Pythonify doesn't buy you much, IMO. Of course, it
is a matter of taste - I understand that there are people who will
prefer the pbs/clom way of doing things.

> Re. threads & fork():http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-befo...
>
> For a careful impl of fork-exec with threads, seehttp://golang.org/src/pkg/syscall/exec_unix.go

Thanks for the links. The first seems to me to be talking about the
dangers of locking and forking; if you don't use threads, you don't
need locks, so the discussion about locking only really applies in a
threading+forking scenario.

I agree that locking+forking can be problematic because the semantics
of what happens to the state of the locks and threads in the child
(for example, as mentioned in http://bugs.python.org/issue6721).
However, it's not clear that any problem occurs if the child just
execs a new program, overwriting the old - which is the case here. The
link you pointed to says that

"It seems that calling execve(2) to start another program is the only
sane reason you would like to call fork(2) in a multi-threaded
program."

which is what we're doing in this case. Even though it goes on to
mention the dangers inherent in inherited file handles, it also
mentions that these problems have been overcome in recent Linux
kernels, and the subprocess module does contain code to handle at
least some of these conditions (e.g. preexec_fn, close_fds keyword
arguments to subprocess.Popen).

Hopefully, if there are race conditions which emerge in the subprocess
code (as has happened in the past), they will be fixed (as has
happened in the past).

> Hmm, if the extra "envelop" is the async code with threads that may deadlock, I would say "thanks but no thanks" :p

That is of course your privilege. I would hardly expect you to drop
extproc in favour of sarge. But there might be people who need to
tread in these dangerous waters, and hopefully sarge will make things
easier for them. As I said earlier, one doesn't *need* to use
asynchronous calls.

I agree that I may have to review the design decisions I've made,
based on feedback based on people actually trying the async
functionality out. I don't feel that shying away from difficult
problems without even trying to solve them is the best way of moving
things forward. What are the outcomes?

* Maybe people won't even try the async functionality (in which case,
they won't hit problems)

* They'll hit problems and just give up on the library (I hope not -
if I ever have a problem with a library I want to use, I always try
and engage with the developers to find a workaround or fix)

* They'll report problems which, on investigation, will turn out to be
fixable bugs - well and good

* The reported bugs will be unfixable for some reason, in which case
I'll just have to deprecate that functionality.

Remember, this is version 0.1 of the library, not version 1.0. I
expect to do some API and functionality tweaks based on feedback and
bugs which show up.

> I do think that IO redirection is much nicer with extproc.

Again, a matter of taste. You feel that it's better to pass dicts
around in the public API where integer file handles map to other
handles or streams; I feel that using a Capture instance is less
fiddly for the user. Let a thousand flowers bloom, and all that.

I do thank you for the time you've taken to make these comments, and I
found the reading you pointed me to interesting. I will update the
sarge docs to point to the link on the Linux Programming blog, to make
sure people are informed of potential pitfalls.

Regards,

Vinay Sajip