is their any limit on the size of the data passed to a progra m via popen2?

[Steve Holden]

"husam" <h.jehadalwan at student.kun.nl> wrote in message
news:3C77AE2E.3020301 at student.kun.nl...
> Gustavo Cordova wrote:
>
> > That doing a read() on pipes hangs, so you have to
> > define a timeout or something, so that you can take
> > control back.
> >
> > But I dinnae kno how ta do such a thang.
> >
> > :-/
> >
> > -gus
> >
> > pd: I *think* it was jeff.
> >
> >
>
> Ya, I red that one. But, now the problem is that running the code
> succefully really depends on the number of the files passed to the
> application in one "popen2" session. I found that the number of the
> files must not exceed 250. I worked around this by splitting the 10000
> files into 50 lists each of them contains 200 files and for each list
> opening a "popen2" session and close it at the end in a "for" loop. Now
> the code works fine. I have searched the internet for informations
> related to the size of the pipe and how to modify it, but I found
> nothing except "select.select()". At the moment I really don't have time
> to implement this.
>
> any way, thanks for the contribution.
>
I'm coming in to this conversation a little late, but is there a possibilty
that you could actually transmit the contents of multiple files via a single
stream, allowing you to feed files into the popen()'d subprocesses as each
consumes input? This might save a lot a process- and file-creation overhead
if all you really need is the computaional output. You might use a netstring
representation, for example.

Even if all the above is true, don't be surprised to hit hard limits on
other annoying resources like threads-per-process or processes per subgroup.

That "250", by they way? Might it be 254 or 255, really? Or possibly
(255-number of other command line arguments)? This would be a typical result
of some element of the pipeline limiting the length  of the argument vector
so a byte could be used to select each element. So, you are probably the
victim of thoughtless design, or of some historical features of the code you
use.

Even so there are always limitations, whether it's long arithmetic slowdown
or inappropriate representation, and TheOnlyWayForward is to
MercilesslyRefactor. Every environment has limits, and sometimes it's a
question of learning what limits you can live with. Onward and downward, as
the Pascal standardization committee might have declared.

wondering-what-happened-to-tony-addyman-ly y'rs  - steve
--
Consulting, training, speaking: http://www.holdenweb.com/
Author, Python Web Programming: http://pydish.holdenweb.com/pwp/

"This is Python.  We don't care much about theory, except where it
intersects with useful practice."  Aahz Maruch on c.l.py