dealing with infinite generators

[Avi Gross]

Morten,

 

I did not mean that programming languages don’t already have lots of infinities on board. There are plenty of infinite loops that have no terminating conditions and the process continues until it is actively killed. There are recursive function calls that may miss the call to stop because .0000001 is not exactly equal to zero and so on.

 

The issue here is not really new.

 

Any language allows you to create a loop that calls some function repeatedly and will only stop when the function returns nothing or perhaps throws an error.

 

The issue is that python has paradigms that are effectively somewhat hidden loops. A list comprehension is a hidden loop as are its cousins like a dictionary comprehension and of course generators in general. But unlike explicit loops, these constructs may not easily allow you to specify additional code. Yes, you probably can using various techniques. But in real loops you can set flags and increment variables and test for these conditions.

 

I don’t see any easy way to ask that this terminate after a while perhaps by including a modified iterator as the last entry:

 

[ p for p in primes_generator() ]

 

[ p for p in primes_generator() if p < 1000 ]

 

Even if you extended this to check for primes larger than 1,000 it would keep running and just not add additional values. Infinite CPU use and no return to the user of what they asked for.

 

But for the unpacking, example:

 

a, b, c = iterator

 

I do see how it might make sense to use a notation like ***c, as discussed. 

 

Actually, you made me thing about whether the iterator protocol might be amended to deal with infinities.

 

The list comprehension for example takes an “if” clause. I suspect it could also be expanded to take a “max” clause after which it stops generating new values. Or, since iterators are objects, you might be able to set a value by setting iterator.max=99 and have the protocol look for and enforce it so you only get the first 99 bottles of beer off the wall.

 

I am curious as to what happens in python if it is trying to expand an iterator and runs out of memory. I can imagine it stopping and throwing an exception but would it unwind whatever it was trying to do or leave a mess?

 

Because python has lots of flexibility and can potentially add any number of dunder hooks to objects, I wonder if there can be a way to designate something as potentially infinite when iterated. Any python facet that implements the iterator protocol might then be warned and choose not to run the protocol an infinite number of times, or perhaps at all. You might have to handle such objects more carefully using iter() and next() yourself in your own carefully crafted loop that does exit in finite time.

 

Just thinking. If more recent versions of python have done something I am not aware of, I would love to know.

 

From: Morten W. Petersen <morphex at gmail.com> 
Sent: Monday, December 3, 2018 7:14 AM
To: avigross at verizon.net
Cc: python-list at python.org
Subject: Re: dealing with infinite generators

 

On Mon, Dec 3, 2018 at 8:11 AM Avi Gross <avigross at verizon.net <mailto:avigross at verizon.net> > wrote:

[SPECULATION ALERT]

I found it interesting as people discussed  how one gets the length of
something set up to follow the iterator protocol and especially anything
that is effectively infinite.

[...]

Just some speculation. Adding new features can have odd side effects if you
go to extremes like infinities.

 

Well I guess one can't argue with that. 😁

 

-Morten


-- 

Videos at https://www.youtube.com/user/TheBlogologue
Twittering at http://twitter.com/blogologue

Blogging at http://blogologue.com
Playing music at https://soundcloud.com/morten-w-petersen

Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/

On Google+ here https://plus.google.com/107781930037068750156

On Instagram at https://instagram.com/morphexx/