pipes.py and its limits

[Oren Tirosh]

On Wed, Jul 03, 2002 at 09:52:05AM -0400, val wrote:
> Hi All,
>    i'm looking at pipes.py, a great pipeline implementation
>    in python (BTW, who is the author?), and trying to figure
>    out its limits.  The description says that the commands
>    to be appended to the pipeline template have to be
>    shell functions.  I'd like to try to build the dynamic pipelines
>    using the python functions or callable objects.
> 
>    In other words, i'd like to modify the pipeline -
>    its functional components on-the-fly depending on
>    the results of running the pipeline (its output).
> 
>    I'm sure the real Pythoneers know how to do that.
>    Any pointers/help would be highly appreciated.

Take a look at http://tothink.com/python/dataflow for a library that
implements pipelines (I call them flows) using Python iterators and 
generators. 

I also have a library for intergrating external commands more easily
into Python scripts. http://tothink.com/python/shell

I'm working on merging the two so it will be possible to create flows
with mixed Python functions and external commands.

I do not intend to support modification of the flow while it is running
because its elements may be delicate stateful components that have 
different input and output rates, buffering, etc.  Changing it on the
fly could lose data.  But one of the elements in the flow can change its
functionality or even embed a separate flow object inside it if you 
want.

	Oren