generator/coroutine terminology

[Steven D'Aprano]

Marko Rauhamaa wrote:

> Your 'factory' is a:
> 
>     generator
>         A function which returns an iterator.
>     <URL: https://docs.python.org/3/glossary.html>

That glossary entry is misleading, or at least incomplete, and it fails to
match the way "generator" is used by actual Python programmers.

Here is a function which returns an iterator. According to the docs, it's a
generator, but that's simply wrong. In no way, shape or form should it be
considered to be a generator:

def not_a_generator():
    return iter([1, 2, 3])


A generator (function) may be a function which returns an iterator, but not
all functions that return iterators are generators, and in some ways
returning an iterator is the *least* interesting part of what makes a
generator a generator.

What distinguishes a generator from a regular function is the use of
`yield`. Any definition which fails to mention that fact is useless.

"Generator" is used to describe both functions with the `yield` statement,
and the return result of calling such functions. Where it is necessary to
distinguish the two, we call the function-with-yield a "generator
function". This terminology comes straight from the PEP introducing
generators:

https://www.python.org/dev/peps/pep-0255/


Such functions-with-yield *are* functions (or methods):

py> def gen():
...     yield 1
...
py> type(gen)
<type 'function'>


but they're a special kind of function, distinguished by a flag on the
__code__ object (also known as func_code in Python 2). The inspect module
has a function to check that flag, which looks like this:

def isgeneratorfunction(object):
    return bool((isfunction(object) or ismethod(object)) and
                object.func_code.co_flags & CO_GENERATOR)


The result of calling `gen` is an instance of the generator type, that is to
say, an instance of a type which considers its own name to be "generator":

py> x = gen()
py> type(x)
<type 'generator'>

Although this type is built-in, it is not available in the builtins
namespace, but it is bound to the name "GeneratorType" in the types module.
The inspect module has a function for this too:

def isgenerator(object):
    return isinstance(object, types.GeneratorType)


For brevity, I've deleted the docstring, but it is very informative to read
it. Run `import inspect; help(inspect.isgenerator)` for more details.


Like many English words, we have two meanings for "generator":

(1) A function containing the `yield` keyword, or "generator-function".

(2) The result of calling such a function, an instance of
types.GeneratorType. PEP 255 calls that a "generator-iterator", but that
name doesn't appear to have caught on anywhere.

If people can cope with the difference between a TV program and a computer
program, they can cope with "generator" having two meanings, especially
since we have ways to disambiguate between the two when needed.

 
> Your 'generator-instance' is an:
> 
>     iterator
>         An object representing a stream of data.
>     <URL: https://docs.python.org/3/glossary.html>

Generator instances are iterators, but not all iterators are generator
instances. Generator instances are special: they are subroutines which can
be suspended and resumed, with multiple exit points (each yield is an exit
point). 

Generators are a subset of coroutines, which are a generalization of
generators. Coroutines have multiple entry points and exit points (each
yield is both an entry point and exit point). CPython uses the same
internal mechanism for both, and as far as I know, there is no programmatic
way to distinguish a coroutine from a generator. Or at least no obvious
way -- there's no `inspect.iscoroutine` function that I know of.


> I don't think you should read much into what str(obj) returns. It's not
> much more than a random printable some CPython coder has cooked up in
> the heat of the moment.

I think that is completely wrong. The repr() and str() of generator
instances is hardly "random", it reflects the consensus of the PEP authors
and the Python core developers, in particular the BDFL Guido who approved
the PEP, that the type of object it is should be called "generator".



-- 
Steven