Python open a named pipe == hanging?

[Alex Martelli]

Rochester <rochester1976 at gmail.com> wrote:

> Thank you for your advise.  So, it turns out that fifos are quite useless
> in Python programming then, which is quite disappointing to me :-(

Uh?  How so?  They're exactly as useful (or useless) as in any other
language: if you want the open-for-writing to NOT block, you must start
the separate process that open-for-reads _before_ you open-for-write.

> I am not saying that I _have to_ use fifo, afterall it is a rather odd
> thingy not in fasion since the last iceage... I am just disappointed by
> the fact that the old plain Bash seems to excel Python in this special
> aspect.

If your purpose is to launch a large number of separate processes,
that's what bash does for every step, while in Python you have to ask
for separate processes _explicitly_ (in any of many ways, such as
os.system, os.spawn, os.popen, subprocess, and so forth).  If you mostly
want to do processing in your "main process", then the "fork and exec a
new process for every small simple thing" approach of Bash is a minus.

But as long as you're happy with just starting new processes, go ahead,
you can easily do it in Python -- just start them in the right order!


> I am new to Python and much more comfortable in Bash programming.  A
> simple Bash script like this would take the advantage of a fifo, hence
> reduce the overhead of unneccesarry temporary files creation:
> 
> #!/bin/bash
> 
> mkfifo my_fifo
> echo "this is a string in my fifo!" > my_fifo &
> cat my_fifo
> rm my_fifo
> 
> Isn't it neat?

Not at all -- in fact, I find it rather silly.  Still, if you want the
same (silly) functionality in Python, it's trivial to do, without all
that many superfluous external processes and backgrounding -- just start
the processes in the right order, e.g., assuming 'cat' is something
potentially important that MUST run in an external process:


import os

fifoname = 'my_fifo'

os.mkfifo(fifoname)
op = os.popen('cat ' + fifoname)
print >> open(fifoname, 'w'), "some string"

of = open(fifoname, 'w')
print >>of, "some string"
of.close()
os.unlink(fifoname)

print op.read(),


As long as you start cat (which does the open-for-reading) before trying
to do the open-for-writing, everything will be hunky-dory -- just as the
manpage I selectively quoted clearly implied.  This is not an exact
equivalent to your bash script (it only runs cat in an external process,
nothing else) but it would be easier to make it closer (forking more
useless extra processes) if that was somehow a requirement.

> Anyway, I think every scripting language has its pros and cons.  Bash is
> probably more flexible in dealing with fifos and multiway pipes (through
> the magic menchanism of process substitution).

What's "process substitution" -- fork and exec?  Note the '&' you need
in your echo command to tell bash "and don't wait() for this process",
otherwise it would also "hang" (since the default is fork-exec-wait). In
Python, I have chosen to phrase "start an external process but don't
wait for it" as the popen line, and finish things off with the last
print statement; but of course there are abundant alternatives.

Basically, if just about all you want is to run external processes,
shell languages such as bash offer neater syntax, because fork, exec and
wait is what they do by default, so you save some "syntax overhead" made
necessary by languages which require this to be made explicit.  But in
practice "proper" shell scripts (where, say, over half the lines are
usefully running external processes) far too often degenerate to the
point where most of the logic is in fact internal to the script, pushing
the fraction of lines usefully running external processes to 20% or
less, as you add logic, error-handling, and other glue functionality to
the script.  In such situations, languages such as Perl, Python or Ruby
offer far better facilities (and the fact that they don't _by default_
fork, exec and wait for external processes helps them out here).


> Thank you!

You're welcome, even though I fell the post you're responding to may not
have deserved thanks since it appears to not have been as useful and
clear as I had meant it to be...


Alex