creating size-limited tar files

[Kushal Kumaran]

Ian Kelly <ian.g.kelly at gmail.com> writes:

> On Tue, Nov 13, 2012 at 3:31 AM, andrea crotti
> <andrea.crotti.0 at gmail.com> wrote:
>> but it's a bit ugly.  I wonder if I can use the subprocess PIPEs to do
>> the same thing, is it going to be as fast and work in the same way??
>
> It'll look something like this:
>
>>>> p1 = subprocess.Popen(cmd1, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>>> p2 = subprocess.Popen(cmd2, shell=True, stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>>> p1.communicate()
> ('', '')
>>>> p2.communicate()
> ('', '')
>>>> p1.wait()
> 0
>>>> p2.wait()
> 0
>
> Note that there's a subtle potential for deadlock here.  During the
> p1.communicate() call, if the p2 output buffer fills up, then it will
> stop accepting input from p1 until p2.communicate() can be called, and
> then if that buffer also fills up, p1 will hang.  Additionally, if p2
> needs to wait on the parent process for some reason, then you end up
> effectively serializing the two processes.
>
> Solution would be to poll all the open-ended pipes in a select() loop
> instead of using communicate(), or perhaps make the two communicate
> calls simultaneously in separate threads.

Or, you could just change the p1's stderr to an io.BytesIO instance.
Then call p2.communicate *first*.

-- 
regards,
kushal