keeping state in an iterator object by rebinding next()

[Terry Reedy]

"Wilbert Berendsen" <wbsoft at xs4all.nl> wrote in message 
news:200803192036.22626.wbsoft at xs4all.nl...
| Hi,
|
| i am writing a simple parser, that generates tokens. The parser needs to
| maintain some state, because some parts of the file consist of different
| tokens. I thought the object could simply remember its state by assigning
| it's next() method to the method that is currently parsing. When the 
state
| changes, the called method rebinds next() and the next token will be 
returned
| by that function. Here's an example, proving that this indeed works.

For some undisclosed version.

| >>> class A:

I stronly suspect that if you make this a new-style class by adding 
'(object)',
this will not work.  See below.

| ...  def a(self):
| ...   self.next = self.b

This attaches the class attribute (method) b to the instance

| ...   return 1
| ...  def b(self):
| ...   self.next = self.a
| ...   return 2
| ...  def __iter__(self):
| ...   return self

Since A does not have a 'next' (or __next__, in 3.0), instances of A are 
not really iterators, under the new iterator protocol, and hence __iter__ 
above is not valid.  See below.

| ...
| >>> a=A()
| >>> a.a()
| 1
| >>> a.next()
| 2
| >>> a.next()
| 1
| >>> j=0
| >>> for i in a:

In 3.0a3, which uses new-style classes and iterator protocol, this croaks 
with
TypeError: iter() returned non-iterator of type 'A'

| ...  j += 1
| ...  if j > 10: break # prevent from running endlessly
| ...  print i

I suspect that this only works because the for-loop, not finding A.next, 
used the old iterate-by-index protocol and calls a.next with 0, 1, 2, .... 
To find out, put 'print self' inside methods a and b.

 ...
| 2
| 1
| 2
| 1
| 2
| 1
| 2
| 1
| 2
| 1
| >>>
|
| my question is: is this legal Python?

In whatever version you used, it seems to be, but not in the future.

| An iterator could save the next() method object,

You mean, the iterator user.

| and in that case it could stop working.... It works now, because
| apparently the for- construct resolves 'next' each time for the object 
before
| calling it.

A standard idiom for explicit iteration with a new style iterator and 
'while' would do just what you suggest.

itnext = iter(a).next
try:
    while True
    print itnext()
except StopIteration:
    pass

| The other solution would be just jumping to the correct method from 
within the | next() method. But that gives an extra call...

Or, make a and b staticmethods, remove the self param, and make __iter__ a 
generator
  def __iter__(self):
    while True:
      yield self.next()

The speed penalty of the indirection thru the generator is minimal.

Terry Jan Reedy