[Tutor] How to use introspection to discover parameters?

[Steven D'Aprano]

David wrote:
> Please help me understand the following (I have python 2.6.2) ...
> 
> Because I want to write nice docstrings, I read PEP257 where it says:
> "The one-line docstring should NOT be a "signature" reiterating the
> function/method parameters (which can be obtained by introspection)."

As a general rule, yes, that is right -- your doc strings should not merely 
repeat the function signature in the first line. The help() function 
automatically determines the signature for you. For example, if I define this 
function with a docstring, and then call help() on it:

py> def spam(n):
...     """Return Spam Spam Spam Glorious SPAM!!!"""
...     return "spam"*n
...
py> help(spam)


I get this help text displayed:


     Help on function spam in module __main__:

     spam(n)
         Return Spam Spam Spam Glorious SPAM!!!


Notice that the function signature, "spam(n)", is displayed by help() even 
though I didn't include it in the docstring. That is because help() uses 
introspection to discover the signature.

See more detail below.


> I am understanding this sentence to mean: don't put function parameters in my
> own docstrings, because they are readily obtained some other way. This other
> way is apparently called "introspection", but how to actually do it is a
> mystery that I cannot find documented. If the following guess is wrong, I will
> be grateful if someone would just show a simple example of how this is done.

You don't have to do a thing unless you are writing your own help system. The 
standard help() function already does it.

In case it isn't obvious, "introspection" in programming refers to the ability 
of your program to inspect its own components (in this case, functions) to 
find out what they are and what they can do. Python has a couple of built-in 
functions for introspection, plus an entire module of extra functions.

Built-ins include: dir, vars, globals, locals, hasattr, type, help

[Note: technically, help is not a built-in. It is added to the built-ins at 
runtime. Some systems may disable that, e.g. on embedded systems with limited 
memory.]

Plus the inspect module, which contains many other advanced introspection 
tools. You can read about the inspect module here:

http://docs.python.org/library/inspect.html
http://hg.python.org/cpython/file/2.7/Lib/inspect.py


[...]
> So I find no clue there how to do "introspection" to obtain function/method
> parameters, except that there is an API for it, somewhere.


Using the same function spam() I defined above:

py> import inspect
py> inspect.getargspec(spam)
ArgSpec(args=['n'], varargs=None, keywords=None, defaults=None)


And here is one way to use it to print the function signature. First we grab 
the argument spec, then we concatenate it after the function name.

py> spec = inspect.getargspec(spam)
py> spam.__name__ + inspect.formatargspec(*spec)
'spam(n)'


> So I look at that and guess: is PEP257 actually attempting to say
> "dont reiterate the function/method parameters in the docstring, because help()
> displays them in line 3 of its output".

Right!

> If it does mean this, it would be a lot clearer if it just said so!

Perhaps so. You might consider putting in a request for a documentation change 
on the bug tracker.


> A related question:
> help() seems to do the introspection for me. Does python allow me to do it in
> my own code? Specifically, how might I write my own function to mimic line 3 of
> help(), appearing like so:
> 
>>>> my_function(logging.log)
> "log(level, msg, *args, **kwargs)"
> 
> If I knew how to do that, it might help me understand how to do "introspection"
> better.


The help() function is written in pure Python. It's a big, powerful function, 
but if you want to read it, you can do so:

This is where it is added to the built-ins:

http://hg.python.org/cpython/file/2.7/Lib/site.py

And this is where the actual help functionality is made:

http://hg.python.org/cpython/file/2.7/Lib/pydoc.py


> One last thing. When looking for answers, I found this page which seems related:
> http://stackoverflow.com/questions/2536879/python-introspection-how-to-get-varnames-of-class-methods
> 
> There is a comment by S Lott who possibly is the author of "Building Skills In
> Python": "Please ... explain why you need introspection and why you can't
> simply read the source."

What an asinine comment. Where shall we start?

1) Source code is not always available. Even when available, it can sometimes 
be covered by a license which makes it professional suicide to read the code, 
because then anything you write afterwards may be legally deemed to be 
"copied" from the source code you read.

2) Even when you have the source code to read, reading the source can be much, 
much harder than using introspection. The pydoc module is over 2000 lines of 
code, the decimal module is over 4000 lines of code. I shudder to think how 
many thousands of lines of code, spread over who knows how many files, the 
numpy library would be. Ten? Twenty? And it is written in a mix of Python, C 
and Fortran. I wouldn't even know where to begin reading the source code for 
information on (say) numpy.ufunc. But with introspection, it's easy:

py> dir(numpy.ufunc)
['__call__', '__class__', '__delattr__', '__doc__', '__format__', 
'__getattribute__', '__hash__', '__init__', '__name__', '__new__', 
'__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', 
'__str__', '__subclasshook__', 'accumulate', 'identity', 'nargs', 'nin', 
'nout', 'ntypes', 'outer', 'reduce', 'reduceat', 'signature', 'types']

And now I know what methods and attributes ufunc has.

3) Even if the source code is short and sweet and easy to understand, this is 
not always useful. You still have to get that knowledge into your program, 
otherwise it is useless.

Consider our friend the help() function again. You can call it on a class, and 
it will show you help for each method in the class, one after the other. How 
does S Lott think that help() knows what methods the class has? Does he think 
that the person who programmed help() simply wrote up a giant list of every 
existing class and their methods, generated from reading the source code?

Of course not. That would be idiotic. Not only is it time-consuming and 
wasteful, not only does it duplicate work already done, but it is fragile: the 
slightest change to the class, and the list of methods may become out of date.

Instead, help() uses introspection to inspect the class programmatically, 
generate a list of methods on the fly, and display help for each one. You 
simply cannot do anything even remotely similar by reading the source code.


> Ummm ... I'm not sure what to make of that!! Because I've read that this
> instrospection thing is supposed to be an important feature of Python.


If you are ever unlucky enough to use a programming language without 
introspection, you'll soon come to miss it.

Introspection is a powerful technique. For example, I'm currently writing an 
experimental module which uses introspection to *automatically* create 
__repr__ methods for classes. (Well, mostly automatically -- at worst, the 
caller can drop a few extra hints.)





-- 
Steven