Turning f(callback) into a generator

[Peter Otten]

Diez B. Roggisch wrote:

>> However, I did not succeed in turning the old os.path.walk(), i. e. a
>> function taking a callback, into a generator. Is there a general way to
>> do it without having to store all intermediate results first?
> 
> This works for me:
> 
> import os.path
> 
> def path_gen(start):
>     res = []
>     def cb(r, dir, names):
>         for n in names:
>             r.append(n)
> 
>     os.path.walk(start, cb, res)

At this point, you have stored all results in res, and thus did not play by
the rules :-)

>     for n in res:
>         yield n
> 
> 
> g = path_gen("/etc")
> 
> for n in g:
>     print n

Both os.path.walk() and os.walk() basically need memory for the names in
*one* directory; when you take os.walk() to model os.path.walk() that
doesn't change, but the other way round - bang, memory usage explosion.

This is a strange asymmetry, and I was wondering if I've overlooked a simple
way to transform a callback into a generator, but I now tend to the
assumption that you *need* threads: One thread with the callback puts names
or name lists into the queue until it's full (some arbitrary limit that
also determines the total memory needed), another thread (the generator)
consumes names from the queue.

Peter