[Doc-SIG] Python Reader module parser now usable

[David Goodger]

The first part of the Docutils Python Source Reader component is in a
usable form:
<http://docutils.sf.net/docutils/readers/python/moduleparser.py>.  It
takes a module's text (a string) and converts it into a
documentation-oriented tree.  Assignments/attributes, functions,
classes, and methods are all converted.  Arbitrarily complex
right-hand sides of assignments (including default parameter values)
are supported by parsing tokens from tokenize.py.  Comments are not
handled yet, and namespaces are not computed.  There's a list of open
issues at the end of the module docstring.

I've also added a "showdoc" script to test/test_readers/test_python
which processes input from the test_parser.py module, or stdin,
depending on how it's called.  Please play with these; any input is
welcome.

Here's a sample.  Given this module as input::

    # comment

    """Docstring"""

    """Additional docstring"""

    __docformat__ = 'reStructuredText'

    a = 1
    """Attribute docstring"""

    class C(Super):

        """C's docstring"""

        class_attribute = 1
        """class_attribute's docstring"""

        def __init__(self, text=None):
            """__init__'s docstring"""

            self.instance_attribute = (text * 7
                                       + ' whaddyaknow')
            """instance_attribute's docstring"""


    def f(x,                            # parameter x
          y=a*5,                        # parameter y
          *args):                       # parameter args
        """f's docstring"""
        return [x + item for item in args]

    f.function_attribute = 1
    """f.function_attribute's docstring"""

Here's the output tree, with objects converted to the pseudo-XML form
I'm fond of::

    <Module filename="<stdin>">
        <Docstring>
            Docstring
        <Docstring lineno="5">
            Additional docstring
        <Attribute lineno="7" name="__docformat__">
            <Expression lineno="7">
                'reStructuredText'
        <Attribute lineno="9" name="a">
            <Expression lineno="9">
                1
            <Docstring lineno="10">
                Attribute docstring
        <Class bases="Super" lineno="12" name="C">
            <Docstring lineno="12">
                C's docstring
            <Attribute lineno="16" name="class_attribute">
                <Expression lineno="16">
                    1
                <Docstring lineno="17">
                    class_attribute's docstring
            <Method lineno="19" name="__init__">
                <Docstring lineno="19">
                    __init__'s docstring
                <ParameterList lineno="19">
                    <Parameter lineno="19" name="self">
                    <Parameter lineno="19" name="text">
                        <Default lineno="19">
                            None
                <Attribute lineno="22" name="self.instance_attribute">
                    <Expression lineno="22">
                        (text * 7 + ' whaddyaknow')
                    <Docstring lineno="24">
                        instance_attribute's docstring
        <Function lineno="27" name="f">
            <Docstring lineno="27">
                f's docstring
            <ParameterList lineno="27">
                <Parameter lineno="27" name="x">
                <Parameter lineno="27" name="y">
                    <Default lineno="27">
                        a * 5
                <ExcessPositionalArguments lineno="27" name="args">
        <Attribute lineno="33" name="f.function_attribute">
            <Expression lineno="33">
                1
            <Docstring lineno="34">
                f.function_attribute's docstring

-- 
David Goodger  <goodger@python.org>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/