Question about StringIO

[Frank Millman]

Benjamin Niemann wrote:
> Frank Millman wrote:
>
> > I will try to explain my experience with popen() briefly.
> >
> > I run through all the scripts and create a StringIO object with the
> > string I want to pass. It is about 250 000 bytes long. If I run psql
> > using popen(), and pass it the string via stdin, it works fine, but I
> > get all the messages on the screen. If I do the same, but end the
> > command with ' > fjm 2>&1' it works correctly and the messages end up
> > in the file fjm, which is about 40 000 bytes long. If I run it with
> > popen4(), it starts ok, but then hangs about 1/4 of the way through.
> > Exactly the same happens on MSW. It seems to be hitting a limit on the
> > size of the stdout file - is that possible?
> >
>
> That's probably a deadlock as described in
> <http://docs.python.org/lib/popen2-flow-control.html>
>

Thanks for this pointer. I have read it, but I don't think it applies
to my situation, as it talks about 'reading' from the child's stdout
while the child is 'writing' to stderr. I am not doing that, or at
least not consciously. Here is a code snippet. 's' is a StringIO object
that contains my input. It is about 6000 lines/250000 bytes long.

-------------

sql_stdin,sql_stdout = os.popen4('psql -U %s -d %s' % (user,database))

sql_stdin.writelines(s.readlines())
s.close()
sql_stdin.close()

-------------

It starts, and then hangs, after processing about 6% of my input.

If I add ' > fjm 2>&1' to the command, it works, so it is definitely
connected with the child writing to stdout/stderr.

I tried storing my input in a list, and passing ''.join(s), but it had
the same result. I also looped over the list and wrote one line at a
time to sql_stdin - same result.

I can work around this, so a solution is not critical. However, it
would be nice to know if this is a limitation of popen4(), or if I am
doing something wrong.

> > BTW, is there an equivalent of /dev/null on MSW?
>
> Dunno - but as a last resort, you could create a tempfile with a unique name
> (to be sure, not to override any existing data), dump your output there and
> later os.unlink() it...
>

A quick google revealed the answer - there is a device called NUL which
achieves the same purpose.

Thanks

Frank