[python-uk] Question from uniq dojo about fileinput

[Stestagg]

So, I was running this in python2.7, hence the tracebacks and decoding
issues.  However I think the blocking issue is still to do with not closing
the slave half of the pty from the parent process.

On Wed Jan 07 2015 at 11:30:00 PM Stestagg <stestagg at gmail.com> wrote:

> When I try to run it, I get a hang too, in my case (there may be other
> factors for you) what's happening is because of the openpty stuff.
>
> In the parent process, you're creating two file handles, master and
> slave.  Subprocess then forks, meaning the child inherits them.  At this
> point, both the parent and child processes have the same PTY handles open.
>
> In my version of the code, there's a traceback in uniq.py that's
> preventing making the child bail out early.  At this point the child's FDs
> are closed, but because the master still has the 'slave' FD open, the pty
> isn't being shut down.
>
> Add an os.close() after the subprocess call:
>
> process = Popen("python -u uniq.py", shell=True, stdin=PIPE, stdout=slave)
>     os.close(slave)
>
> This will cause the pty to collapse when the child exits, which should
> stop the hanging read().
>
> After adding some UTF-8 decoding to the uniq.py file, I managed to get
> this:
>
> >       assert expected == result
> E       assert ['bar\n', 'fo...o\n', 'end\n'] == ['bar\r\n', 'f...n',
> 'end\r\n']
> E         At index 0 diff: 'bar\n' != 'bar\r\n'
> E         Full diff:
> E         - ['bar\n', 'foo\n', 'bar\n', 'foo\n', 'end\n']
> E         + ['bar\r\n', 'foo\r\n', 'bar\r\n', 'foo\r\n', 'end\r\n']
> E         ?       ++         ++         ++         ++         ++
>
> Which seems quite close :)
>
> Steve
>
>
> On Wed Jan 07 2015 at 10:10:45 PM Tom Viner <tom at viner.tv> wrote:
>
>> Hi all,
>>
>> So before the London dojo meets again tomorrow night I wanted to resolve
>> a question from last month.
>>
>> The task last month was to make our own GNU uniq commands. After the dojo
>> I got our team's code working with the fileinput module
>> <https://docs.python.org/2/library/fileinput.html>. This allowed the
>> flexibility of seamlessly reading from either stdin or taking filenames as
>> arguments.
>>
>> With fileinput the code
>> <https://github.com/tomviner/pydojo-uniq-s6e4/blob/master/team1/uniq.py#L11>
>> can just say:
>> for line in fileinput.input():
>>
>> and that gives you all these usages:
>>
>> python uniq.py my_filename.txt other_file.txt
>> echo -e "hello\nhello2" | python uniq.py
>> python uniq.py <(python print_sleep_print.py)
>>
>> With the print_sleep_print.py script
>> <https://github.com/tomviner/pydojo-uniq-s6e4/blob/master/team1/print_sleep_print.py>
>> I can even see about 8Kb at a time will be buffered into uniq.py, made
>> unique and printed, and then more data will be passed in.
>>
>> The problem came when testing this feature. I found a way to connect to
>> both the input and output of a subprocess running the command:
>>
>>     master, slave = pty.openpty()
>>     process = Popen("python uniq.py", shell=True, stdin=PIPE,
>> stdout=slave)
>>     stdin_handle = process.stdin
>>     stdout_handle = os.fdopen(master)
>>
>> I then write some data in, and read data out. It all works except for one
>> thing: even if I close the stdin_handle the stdout_handle will just block
>> once I've read all the output. Full test here
>> <https://github.com/tomviner/pydojo-uniq-s6e4/blob/master/team1/test_uniq.py#L37>
>> .
>>
>> Appreciate any insight!
>> Tom
>> _______________________________________________
>> python-uk mailing list
>> python-uk at python.org
>> https://mail.python.org/mailman/listinfo/python-uk
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-uk/attachments/20150107/64bc62c7/attachment.html>