Sending changed parameters into nested generators
John O'Hagan
research at johnohagan.com
Fri Jan 21 05:20:29 EST 2011
On Fri, 21 Jan 2011, cbrown wrote:
> On Nov 12, 10:52 pm, "John O'Hagan" <resea... at johnohagan.com> wrote:
> > On Sat, 13 Nov 2010, Steven D'Aprano wrote:
> > > On Fri, 12 Nov 2010 09:47:26 +0000, John O'Hagan wrote:
> > > > I have a generator function which takes as arguments another
> > > > generator and a dictionary of other generators like this:
> > > >
> > > > def modgen(gen, gendict):
> > > > for item in gen():
> > > > for k, v in gendict:
> > > > do_something_called_k(item, v.next())
> > > >
> > > > yield item
> > >
> > > [snip]
> > >
> > > > If anyone's still reading :) , how can I send new values to arbitrary
> > > > sub- generators?
> > >
> > > I have a headache after reading your problem :(
> > >
> > > I think it's a good time to point you at the Zen, particularly these
> > > five maxims:
> > >
> > > Beautiful is better than ugly.
> > > Simple is better than complex.
> > > Complex is better than complicated.
> > > Flat is better than nested.
> > > If the implementation is hard to explain, it's a bad idea.
> > >
> > > I'm afraid that your nested generators inside another generator idea
> > > fails all of those... it's not elegant (beautiful), it's complicated,
> > > it's nested, and the implementation is hard to explain.
> > >
> > > You could probably replace generators with full-blown iterators, but I
> > > wonder what you're trying to accomplish that is so complicated that it
> > > needs such complexity to solve it. What are you actually trying to
> > > accomplish? Can you give a simple example of what practical task you
> > > hope to perform? I suspect there's probably a more elegant way to
> > > solve the problem.
> >
> > I hope there is!
> >
> > The project not practical but artistic; it's a real-time musical
> > composition program.
> >
> > A (simplified) description: one module contains number-list generating
> > functions, others contain functions designed to filter and modify the
> > number lists produced, according to various parameters. Each such stream
> > of number lists is assigned a musical meaning (e.g. pitch, rhythm,
> > volume, etc) and they are combined to produce representations of musical
>>phrases, which are sent to a backend which plays the music
> >as it is produced, and makes PDF scores.
>
> >Each such "instrument" runs as a separate thread, so several can play
>>together in acoordinated fashion.
> >
> > All the compositional interest lies in the selection of number-list
>>generators and how their output is modified. For example, if I say "Play
>>every third note up an octave" it's not very interesting, compared to "Play
>>every nth note up an interval of m", where n and m vary according to some
>>pattern. It gets even more interesting when that pattern is a function of x
>>and y, which also vary according to another pattern, and so on.
> >
> > To that end, I have each parameter of each modifier set by another
> > generator, such that the value may change with each iteration. This may
> > continue recursively, until at some level we give a parameter a simple
> > value.
> >
> > That's all working, but I also want it to be interactive. Each thread
> > opens a terminal where new options can be entered, but so far it only
>>works, as I mentioned, for changing the values in a top-level mutable
>>object.
>
> I might first suggest this, although I have some caveats to add:
>
> def genfilter(evaluator, **param_sources):
> while True:
> params = {}
> for param, gen in param_sources.iteritems():
> params[param] = gen.next()
> yield evaluator(**params)
>
> You can then do things like:
> >>> def concat(in1, in2):
> >>> return str(in1)+"|"+str(in2)
> >>>
> >>> a = (i for i in range(1,5)) # generator based on a list
> >>> b = (2*i for i in xrange(1,5)) # 'pure' generator
> >>> c = genfilter(concat, in1=a, in2=b)
[...]
> or, more relevant to your original question regarding modifying things
>
> mid-stream:
> >>> class Mult():
> >>> def __init__(self, multiplier):
> >>> self.mulitplier = multiplier
> >>>
> >>> def multi(self, val):
> >>> return val*self.multiplier
> >>>
> >>> m = Mult(2)
> >>> a = (i for i in range(1,10))
> >>> b = (i for i in range(1,10))
> >>> c = genfilter(m.multi, val=b)
> >>> d = genfilter(concat, in1=a, in2=c)
> >>> d.next()
[...]
> But a real problem with this whole strategy is that a generator's
> next() function is called every time it is evaluated. If the
> relationship between your various generators forms a rooted tree,
> that's not a problem, but I would think the relationships form a
> directed acyclic graph, and in that case, you end up 'double
> incrementing' nodes in a way you don't want:
[...]
> To solve that problem, you need a somewhat more complex solution: a
> class that ensures that each previous stage is only invoked once per
> 'pass'. I've got an idea for that, if that is of interest.
Going for the record for pregnant pauses, I've taken on board the replies to
my post (from Nov 12!), for which I thank you, and have come up with a
solution. A simplified version follows.
Chas's mention of trees made me realise that my program _is_ actually a tree
of iterators, with each node receiving arguments from its children, so I made
this class:
class IterNode(object):
"""Iterator wrapper to give access to arguments
to allow recursive updating"""
def __init__(self, iterobj, arg):
"""Takes an iterator class or generator
function, and its single arg"""
self.arg = arg
self.iterobj = iterobj
self.iterator = iterobj(arg)
def __iter__(self):
return self
def next(self):
return self.iterator.next()
def __eq__(self, other):
return self.iterobj == other.iterobj and self.arg == other.arg
def __ne__(self, other):
return not self == other
def deepdate(self, other):
"""Deep-update a nested IterNode"""
if self != other:
arg1, arg2 = self.arg, other.arg
if arg1 != arg2:
if isinstance(arg1, dict) and isinstance(arg2, dict):
for k in arg1.copy().iterkeys():
if k not in arg2:
del arg1[k]
for k, v in arg2.iteritems():
if k in arg1:
arg1[k].deepdate(v)
else:
arg1[k] = v
else:
self.__init__(other.iterobj, other.arg)
...And two custom iterator classes: for the root of the tree, the PhraseMaker
iterator class is the only part really specific to music, in that it combines
sequences from iterators into representations of musical phrases, assigning
them to pitch, duration, volume etc.:
class PhraseMaker(object):
def __init__(self, iterdict):
self.iterdict = iterdict
self.iterator = self.phriterate()
def __iter__(self):
return self
def next(self):
return self.iterator.next()
##Omitting for brevity methods to combine
##sequences into musical phrases
def phriterate(self):
"Generate phrases"
while True:
phrase = []
for k, v in self.iterdict.iteritems():
getattr(self, k)(phrase, v.next())
yield phrase
and for the branches, "SeqGen" is a fairly generic iterator which takes any
sequence generators, and filters and modifies their output according to simple
functions, whose arguments come from iterators themselves, like this:
class SeqGen(object):
def __init__(self, iterdict):
self.iterdict = iterdict
self.iterator = self.squiterate()
def next(self):
return self.iterator.next()
def __iter__(self):
return self
def squiterate(self):
generators = self.iterdict.pop('generator')
genargs = self.iterdict.pop('genargs')
for gen in generators:
#Where "gens" is a module containing sequence generators:
generator = getattr(gens, gen)(genargs.next())
current_values = [(k, v.next()) for k, v in
self.iterdict.iteritems()]
for seq in generator:
for k, v in current_values:
#Where "seqmods" is a module containing functions
#which test or modify a sequence:
if getattr(seqmods, k)(seq, v) is False:
#Test for False because in-place modifiers return None
break
else:
current_values = [(k, v.next()) for k, v in
self.iterdict.iteritems()]
yield seq
The leaves of the tree are IterNodes whose arg is a list and which use this
simple generator:
def cycle(iterable):
while True:
for i in iterable:
yield i
I can build a tree of IterNodes from a nested dictionary like this:
def itertree(dic, iterator=PhraseMaker):
for k, v in dic.iteritems() :
if isinstance(v, dict):
dic[k] = itertree(v, SeqGen)
else:
dic[k] = IterNode(cycle, v)
return IterNode(iterator, dic)
d = {'a':[1,2,3], 'b':{'a':[4,5,6]}, 'c':{'a':{'a':[7,8,9], 'b':{'c':[10]}}}}
itnod = itertree(d)
and update it at any time, even during iteration, like this:
new_d = {'a':[1,2,3], 'b':{'a':[4,5,6]}, 'c':{'a':{ 'b':[4]}}, 'd':{}}
new_itnod = itertree(new_d)
itnod.deepdate(new_itnod)
(In real life the dictionary keys are the names of attributes of PhraseMaker,
gens, or seqmods.)
The problem Chas mentioned of consuming next() multiple times still applies to
a tree because if a sequence fails a test on a particular value, we want to
stay on that value till the test succeeds - which is why the SeqGen class has
the "current_values" list in the squiterate method. (The real version actually
has to keep count for each key in iterdict.) I would of course be interested
in any other ideas.
It works and is relatively easy to understand, but I still have feeling that
I'm breathing my own exhaust and this is too complex...
Regards,
John
More information about the Python-list
mailing list