Python open a named pipe == hanging?

[Antoon Pardon]

On 2006-08-07, Rochester <rochester1976 at gmail.com> wrote:
> Thanks Alex, now I think I understand much better the fifo/pipe mechanism  
> and how Python treats them.
>
> For those who are interested, I would like to restate the problem I was  
> tring to solve and a working solution (inspired by Alex Martelli's code),  
> feel free to criticize it:
>
> The problem:
>
> I have an external program which takes two matrices (two text files) as  
> input, and produces an output to stdout, you can take diff as an example.   
> I want to call this program from within Python, without writing any  
> temporary file (for the performance reason).
>
> In Bash, this task would be best written as:
>
> #!/bin/bash
>
> diff <(step1) <(step2) | step3
>
> Where step1, step2 and step3 have to be independent external programs.
>
> Now in Python, since there is no exact equivalence of <() magic a.k.a.  
> process substitution, I figured taht the best solution should be to create  
> a pair of fifos and do something like this:
>
> #!/bin/bash
>
> mkfifo fifo1 fifo2
> step1 > fifo1 &
> step2 > fifo2 &
> diff step1 step2 | step3
>
> And a working Python equivalent code is:

I think your code only works because of an artefacts that may not work
in general.

> #!/usr/bin/python
>
> import os
>
> # do step1 and step2 in Python, say we end up with something like these:
>
> s1 = "some string\n second line."      # results from step1
> s2 = "some string\n a different line." # results from step2
>
> os.mkfifo('fifo1')
> os.mkfifo('fifo2')
>
> op = os.popen(' '.join(['diff', 'fifo1', 'fifo2'])) # this step is crucial!
> print >> open('fifo1', 'w'), s1
> print >> open('fifo2', 'w'), s2

This will not work in general. Suppose diff would open the two files
simultaneously and read the files in parralel. Since you first feed
the whole first file before you start the second, a deadlock could
occur if s1 was sufficiently large.

Something like the following instead of the two print statements
would be better IMO (not tested):

def cat(fn, st):
  fl = file(fn, 'w')
  fl.write(st)
  fl.close()

Threading.Thread(target = cat, args = ('fifo1', s1)).start()
Threading.Thread(target = cat, args = ('fifo2', s2)).start()

-- 
Antoon Pardon