From ping@lfw.org  Tue May 30 10:43:54 2000
From: ping@lfw.org (Ka-Ping Yee)
Date: Tue, 30 May 2000 02:43:54 -0700 (PDT)
Subject: [Doc-SIG] htmldoc.py and inspect.py
Message-ID: <Pine.LNX.4.10.10005300207110.2697-100000@localhost>

Hi everyone.

I'm writing to announce that i've posted my current work on
a documentation generator.  Those of you who were in the
doc-sig room at IPC8 may remember the example i showed.  Quite
a few people seemed to like it and i promised to produce the
script -- so here i am trying to make good on my promise.

Done:
    - documentation of user modules, classes, methods, and functions
    - documentation of built-in functions
    - automatic hyperlinking when classes and methods are mentioned
    - detection of leading comments when no docstring is present
    - overview of class inheritance tree within a module
    - generation of HTML files from the command line
    - all generated files are SGML compliant! :) :) (HTML 4.0 Transitional)

To do:
    - no directory/package indexing yet
    - simple HTTP server (so you can get fresh docs on your own modules)

As there are still items on the to-do list it's not entirely
done yet -- but there is enough done to work fairly well, and
i wanted to give you a chance to look at it and let me know
what you think.

The modules are:

    http://www.lfw.org/python/htmldoc.py
    http://www.lfw.org/python/inspect.py

Fred: i would like to seriously propose this to go with 1.6.
Will you consider it?

The "htmldoc" module is actually quite small (only about 300
lines) as most of the hard work has been factored out into
the "inspect" module -- a non-HTML-specific collection of
routines for getting all kinds of information out of your
Python objects.  My favourite routine in "inspect" is
inspect.getsource(object), which can get you the source code
for a function, method, or class.

The IPC8 example i showed is at

    http://www.lfw.org/python/SocketServer.html

That output was produced by a script called "manpy.cgi" that
i wrote at work, and unfortunately that meant the script could
not be distributed outside of my company.  The new script
produces very similar output, but it has been rewritten from
scratch on my own time -- which means (a) it's completely
unencumbered by any IP claims; and (b) i got to make it a
little bit cleaner.

I have run the new script over the Python 1.5.2 standard
library.  Please browse the results at

    http://www.lfw.org/python/htmldoc/

I welcome your feedback!

Thanks,



-- ?!ng

"To be human is to continually change.  Your desire to remain as you are
is what ultimately limits you."
    -- The Puppet Master, Ghost in the Shell




From larsga@garshol.priv.no  Tue May 30 11:50:15 2000
From: larsga@garshol.priv.no (Lars Marius Garshol)
Date: 30 May 2000 12:50:15 +0200
Subject: [Doc-SIG] htmldoc.py and inspect.py
In-Reply-To: <Pine.LNX.4.10.10005300207110.2697-100000@localhost>
References: <Pine.LNX.4.10.10005300207110.2697-100000@localhost>
Message-ID: <m3g0r0qplk.fsf@lambda.garshol.priv.no>

* Ka-Ping Yee
| 
| I welcome your feedback!

Initial reaction: wild joy that someone is working on this!

I think it looks very good, and would be very happy if this could be
expanded into a full solution. (Sorry, my stack is full in the near
term.)

Initial comments:

 - it seems to document a number of things that should not be
   documented (ideally there should be some way of avoiding this,
   for example test functions and _*/__* members should not be
   documented, except for __init__, __getitem__ and friends)

 - the HTML output should perhaps use a stylesheet instead of
   hardcoding the presentation

 - it would be nice if docstrings could use <p> instead of &nbsp; and
   <BR>, but that's probably a lot of work...

 - also, package/directory indexing would make a big difference

--Lars M.



From Juergen Hermann" <jhe@webde-ag.de  Tue May 30 14:09:50 2000
From: Juergen Hermann" <jhe@webde-ag.de (Juergen Hermann)
Date: Tue, 30 May 2000 14:09:50 +0100
Subject: [Doc-SIG] htmldoc.py and inspect.py
Message-ID: <20000530120952247.AAA531.499@hermes.cinetic.de>

On Tue, 30 May 2000 02:43:54 -0700 (PDT), Ka-Ping Yee wrote:

>I welcome your feedback!

I'm currently using pdoc.py, and I will definitely will give your script=
 
a look. What I would like VERY much would be a Python doc system that 
shares as much as possible with the JavaDoc/DOC++/Doxygen world, 
especially the "commands" to provide semantic markup (like \param spam).=



Ciao, J=FCrgen

--
J=FCrgen Hermann (jhe@webde-ag.de)
WEB.DE AG, Amalienbadstr.41, D-76227 Karlsruhe
Tel.: 0721/94329-0, Fax: 0721/94329-22




From walter@livinglogic.de  Tue May 30 15:35:38 2000
From: walter@livinglogic.de (Walter =?iso-8859-1?Q?D=F6rwald?=)
Date: Tue, 30 May 2000 16:35:38 +0200
Subject: [Doc-SIG] htmldoc.py and inspect.py
In-Reply-To: <20000530120952247.AAA531.499@hermes.cinetic.de>
Message-ID: <4.3.1.0.20000530163421.00ae5710@mail.tmt.de>

At 15:09 30.05.00, Juergen Hermann wrote:

>On Tue, 30 May 2000 02:43:54 -0700 (PDT), Ka-Ping Yee wrote:
>
> >I welcome your feedback!
>
>I'm currently using pdoc.py, and I will definitely will give your script
>a look. What I would like VERY much would be a Python doc system that
>shares as much as possible with the JavaDoc/DOC++/Doxygen world,
>especially the "commands" to provide semantic markup (like \param spam).

I would prefer if the semantic markup would use XML. XML was designed
to be used for stuff like that

Bye,
         Walter D=F6rwald



From willguaraldi@mediaone.net  Tue May 30 17:03:44 2000
From: willguaraldi@mediaone.net (will)
Date: Tue, 30 May 2000 12:03:44 -0400
Subject: [Doc-SIG] htmldoc.py and inspect.py
In-Reply-To: <Pine.LNX.4.10.10005300207110.2697-100000@localhost>
Message-ID: <NDBBJAKILMEBBFEDCMKKGECPCCAA.willguaraldi@mediaone.net>

If you change the arg-reading code in htmldoc.py to something along
these lines, then you can do ool documentation on modules that import
modules in other (non-standard) directories.

if __name__ == "__main__":
    import os
    modnames = []
    argn = 1
    while argn < len(sys.argv):
        arg = sys.argv[argn]
        if arg == "-m":
            sys.path.append(sys.argv[argn + 1])
            argn = argn + 1
        elif os.path.isdir(arg):
            for file in os.listdir(arg):
                if file[-3:] == ".py":
                    modnames.append(file[:-3])
                elif file[-9:] == "module.so":
                    modnames.append(file[:-9])
        else:
            if arg[-3:] == ".py":
                modnames.append(arg[:-3])
            else:
                modnames.append(arg)
        argn = argn + 1

I'm looking into building a function to generate a toc as well.  So if
you call:

   htmldoc.py -m /mud/server/ /mud/server/*.py

then it will build a toc with all the py files in /mud/server/.  Maybe
have a command-line flag to toggle whether or not to generate a toc
and what to call it.

I'm impressed though.  So far it's been very easy to use.  Took me
about 10 minutes to get it working with our project which right now
spans a few directories and consists of almost a hundred different
modules.

Also, I really like that it uses &nbsp; and <br> instead of <p>.  This
keeps the outputted html looking exactly like the doc-string did.
Makes it easier to build ascii diagrams and such.

I like that it doesn't use XML as a semantic mark-up.  Whoever (I
deleted the email--sorry bout that) mentioned that should refer to the
archives on lengthy discussion on the topic of what to use for markup
and see if that covers their arguments for/against using XML in this
fashion.

So, in my opinion, apart from a few minor changes:
 - needs to be able to add other dirs to sys.path for importing
purposes
 - needs to generate a toc for files it's documenting
 - needs to generate an index of functions/classes/modules
 - maybe a few other flags to toggle some of the things folks dislike
   like including __blah__ type functions

Other than that, it's great!  Thank you muchly Ping!
/will

> -----Original Message-----
> From: doc-sig-admin@python.org [mailto:doc-sig-admin@python.org]On
> Behalf Of Ka-Ping Yee
> Sent: Tuesday, May 30, 2000 5:44 AM
> To: doc-sig@python.org; fdrake@acm.org
> Subject: [Doc-SIG] htmldoc.py and inspect.py
>
>
> Hi everyone.
>
> I'm writing to announce that i've posted my current work on
> a documentation generator.  Those of you who were in the
> doc-sig room at IPC8 may remember the example i showed.  Quite
> a few people seemed to like it and i promised to produce the
> script -- so here i am trying to make good on my promise.
>
> Done:
>     - documentation of user modules, classes, methods, and functions
>     - documentation of built-in functions
>     - automatic hyperlinking when classes and methods are mentioned
>     - detection of leading comments when no docstring is present
>     - overview of class inheritance tree within a module
>     - generation of HTML files from the command line
>     - all generated files are SGML compliant! :) :) (HTML
> 4.0 Transitional)
>
> To do:
>     - no directory/package indexing yet
>     - simple HTTP server (so you can get fresh docs on your
> own modules)
>
> As there are still items on the to-do list it's not entirely
> done yet -- but there is enough done to work fairly well, and
> i wanted to give you a chance to look at it and let me know
> what you think.
>
> The modules are:
>
>     http://www.lfw.org/python/htmldoc.py
>     http://www.lfw.org/python/inspect.py
>
> Fred: i would like to seriously propose this to go with 1.6.
> Will you consider it?
>
> The "htmldoc" module is actually quite small (only about 300
> lines) as most of the hard work has been factored out into
> the "inspect" module -- a non-HTML-specific collection of
> routines for getting all kinds of information out of your
> Python objects.  My favourite routine in "inspect" is
> inspect.getsource(object), which can get you the source code
> for a function, method, or class.
>
> The IPC8 example i showed is at
>
>     http://www.lfw.org/python/SocketServer.html
>
> That output was produced by a script called "manpy.cgi" that
> i wrote at work, and unfortunately that meant the script could
> not be distributed outside of my company.  The new script
> produces very similar output, but it has been rewritten from
> scratch on my own time -- which means (a) it's completely
> unencumbered by any IP claims; and (b) i got to make it a
> little bit cleaner.
>
> I have run the new script over the Python 1.5.2 standard
> library.  Please browse the results at
>
>     http://www.lfw.org/python/htmldoc/
>
> I welcome your feedback!
>
> Thanks,
>
>
>
> -- ?!ng
>
> "To be human is to continually change.  Your desire to
> remain as you are
> is what ultimately limits you."
>     -- The Puppet Master, Ghost in the Shell
>
>
>
> _______________________________________________
> Doc-SIG maillist  -  Doc-SIG@python.org
> http://www.python.org/mailman/listinfo/doc-sig



From ping@lfw.org  Tue May 30 17:54:51 2000
From: ping@lfw.org (Ka-Ping Yee)
Date: Tue, 30 May 2000 09:54:51 -0700 (PDT)
Subject: [Doc-SIG] htmldoc.py and inspect.py
In-Reply-To: <4.3.1.0.20000530163421.00ae5710@mail.tmt.de>
Message-ID: <Pine.LNX.4.10.10005300945320.2697-100000@localhost>

At 15:09 30.05.00, Juergen Hermann wrote:
> I'm currently using pdoc.py, and I will definitely will give your script
> a look. What I would like VERY much would be a Python doc system that
> shares as much as possible with the JavaDoc/DOC++/Doxygen world,
> especially the "commands" to provide semantic markup (like \param spam).

On Tue, 30 May 2000, Walter [iso-8859-1] Dörwald wrote:
> I would prefer if the semantic markup would use XML. XML was designed
> to be used for stuff like that

This was a decision i made about the philosophy of the doc
generation script.  I chose to have it display the docstrings
exactly as they are in the code.  This makes the script small, 
predictable, and incapable of screwing up in any significant way. [1]

Also this avoids the entire issue of exactly what particular
structured-text formatting conventions you're going to use.
I do like structured text (indeed, WikiWiki is WwWonderful)
but you should see just how long discussions can go on and on
about how best to do this sort of thing.

Better to just get something done and working and out there
than continue debating, i think.

By the way, that's a very unusual middle name, Walter. :)



-- ?!ng

"To be human is to continually change.  Your desire to remain as you are
is what ultimately limits you."
    -- The Puppet Master, Ghost in the Shell

[1] The only "screw-up" i have noticed is in nturl2path.py and
reconvert.py, which mention "\b", "\f" etc. in their docstrings.
So for example the mention of "C:\foo" in nturl2path.py becomes
"C:", a form feed, and "oo".  I have submitted a little patch to
put an "r" in front of those docstrings, fixing this problem.



From ping@lfw.org  Tue May 30 17:59:33 2000
From: ping@lfw.org (Ka-Ping Yee)
Date: Tue, 30 May 2000 09:59:33 -0700 (PDT)
Subject: [Doc-SIG] htmldoc.py and inspect.py
In-Reply-To: <NDBBJAKILMEBBFEDCMKKGECPCCAA.willguaraldi@mediaone.net>
Message-ID: <Pine.LNX.4.10.10005300954570.2697-100000@localhost>

On Tue, 30 May 2000, will wrote:
> If you change the arg-reading code in htmldoc.py to something along
> these lines, then you can do ool documentation on modules that import
> modules in other (non-standard) directories.

Right -- the "__main__" part of htmldoc.py was really just for
quick testing -- there needs to be a little more smarts there
about knowing the location of the modules and where to put the
generated files, etc.

> I'm looking into building a function to generate a toc as well.  So if
> you call:
> 
>    htmldoc.py -m /mud/server/ /mud/server/*.py

The "multicolumn" function is very handy for making the lists
in the TOC.  I've got stuff for making indexes that i'm working on.

> I'm impressed though.  So far it's been very easy to use.  Took me
> about 10 minutes to get it working with our project which right now
> spans a few directories and consists of almost a hundred different
> modules.

This is very good news.  Thanks for testing.

> So, in my opinion, apart from a few minor changes:

Your suggestions are well taken.  Thanks.


-- ?!ng

"To be human is to continually change.  Your desire to remain as you are
is what ultimately limits you."
    -- The Puppet Master, Ghost in the Shell



From paul@prescod.net  Tue May 30 17:51:06 2000
From: paul@prescod.net (Paul Prescod)
Date: Tue, 30 May 2000 11:51:06 -0500
Subject: [Doc-SIG] htmldoc.py and inspect.py
References: <Pine.LNX.4.10.10005300945320.2697-100000@localhost>
Message-ID: <3933F17A.17DFBB80@prescod.net>

Ka-Ping Yee wrote:
> 
> ...
> 
> Also this avoids the entire issue of exactly what particular
> structured-text formatting conventions you're going to use.
> I do like structured text (indeed, WikiWiki is WwWonderful)
> but you should see just how long discussions can go on and on
> about how best to do this sort of thing.

Insofar as there are virtues to structured text, JavaDOC, POD, XML and
other stuff, I wonder if it would be useful to have a plug-in
architecture that allowed you to choose your own syntax. Eventually we
would want to standardize, of course, but in the short term we could
experiment and have fun holy wars and so forth.

Your code would pass the docstring to the plugin and get back a string
that it would splat into the HTML.

The correct plugin would be chosen using some kind of per-module pragma.
The default would be the current behavior.

-- 
 Paul Prescod  - ISOGEN Consulting Engineer speaking for himself
"Hardly anything more unwelcome can befall a scientific writer than 
having the foundations of his edifice shaken after the work is 
finished.  I have been placed in this position by a letter from 
Mr. Bertrand Russell..." 
 - Frege, Appendix of Basic Laws of Arithmetic (of Russell's Paradox)


From ping@lfw.org  Tue May 30 18:13:40 2000
From: ping@lfw.org (Ka-Ping Yee)
Date: Tue, 30 May 2000 10:13:40 -0700 (PDT)
Subject: [Doc-SIG] htmldoc.py and inspect.py
In-Reply-To: <Pine.LNX.4.10.10005300207110.2697-100000@localhost>
Message-ID: <Pine.LNX.4.10.10005301005580.2697-100000@localhost>

On Tue, 30 May 2000, Ka-Ping Yee wrote:
> 
> To do:
>     - no directory/package indexing yet
>     - simple HTTP server (so you can get fresh docs on your own modules)

Two more items:
    
    - command-line program ("pydoc"?) for showing a plain-text
      "man"-style page for a module
    - better rule for deciding when a function belongs to a module

Current rule is:
    - if function is built-in, always include it
    - if function's func_code.co_filename matches module, include it

Proposed new rule:
    - if function is built-in, include it only if module does not
      come from a .py or .pyc
    - if function's func_code.co_filename matches module, include it

(see http://www.lfw.org/python/htmldoc/calendar.html for goof:
all the time.*() functions are visible)

It would probably be nice to show a list of the functions and
classes that have been imported from other modules, but not
generate redundant documentation for them.  Sound right?


-- ?!ng

"To be human is to continually change.  Your desire to remain as you are
is what ultimately limits you."
    -- The Puppet Master, Ghost in the Shell



From ping@lfw.org  Tue May 30 18:16:16 2000
From: ping@lfw.org (Ka-Ping Yee)
Date: Tue, 30 May 2000 10:16:16 -0700 (PDT)
Subject: [Doc-SIG] htmldoc.py and inspect.py
In-Reply-To: <3933F17A.17DFBB80@prescod.net>
Message-ID: <Pine.LNX.4.10.10005301015040.2697-100000@localhost>

On Tue, 30 May 2000, Paul Prescod wrote:
> 
> Your code would pass the docstring to the plugin and get back a string
> that it would splat into the HTML.

Good idea.  I like it.  There's already a place for this, since
what you've described is the "markup" function's job.


-- ?!ng

"To be human is to continually change.  Your desire to remain as you are
is what ultimately limits you."
    -- The Puppet Master, Ghost in the Shell



From willguaraldi@mediaone.net  Tue May 30 18:18:42 2000
From: willguaraldi@mediaone.net (will)
Date: Tue, 30 May 2000 13:18:42 -0400
Subject: [Doc-SIG] htmldoc.py and inspect.py
In-Reply-To: <3933F17A.17DFBB80@prescod.net>
Message-ID: <NDBBJAKILMEBBFEDCMKKCEDACCAA.willguaraldi@mediaone.net>

If you look at the way it's structured right now, all the HTML is
localized into functions in htmldoc.py.  So you could easily take
that, copy, paste, munge, mangle and coerce into xmldoc.py or any
other output.

inspect.py is doing most of the work and htmldoc.py is doing the
formatting and output.  This is rather nice setup.  Pretty easy to
build other output formats by replacing htmldoc.py.

/will

> -----Original Message-----
> From: doc-sig-admin@python.org [mailto:doc-sig-admin@python.org]On
> Behalf Of Paul Prescod
> Sent: Tuesday, May 30, 2000 12:51 PM
> To: doc-sig@python.org
> Subject: Re: [Doc-SIG] htmldoc.py and inspect.py
>
> Ka-Ping Yee wrote:
> >
> > ...
> >
> > Also this avoids the entire issue of exactly what particular
> > structured-text formatting conventions you're going to use.
> > I do like structured text (indeed, WikiWiki is WwWonderful)
> > but you should see just how long discussions can go on and on
> > about how best to do this sort of thing.
>
> Insofar as there are virtues to structured text, JavaDOC,
> POD, XML and
> other stuff, I wonder if it would be useful to have a plug-in
> architecture that allowed you to choose your own syntax.
> Eventually we
> would want to standardize, of course, but in the short term we could
> experiment and have fun holy wars and so forth.
>
> Your code would pass the docstring to the plugin and get
> back a string
> that it would splat into the HTML.
>
> The correct plugin would be chosen using some kind of
> per-module pragma.
> The default would be the current behavior.
>
> --
>  Paul Prescod  - ISOGEN Consulting Engineer speaking for himself
> "Hardly anything more unwelcome can befall a scientific writer than
> having the foundations of his edifice shaken after the work is
> finished.  I have been placed in this position by a letter from
> Mr. Bertrand Russell..."
>  - Frege, Appendix of Basic Laws of Arithmetic (of
> Russell's Paradox)
>
> _______________________________________________
> Doc-SIG maillist  -  Doc-SIG@python.org
> http://www.python.org/mailman/listinfo/doc-sig



From paul@prescod.net  Wed May 31 06:54:08 2000
From: paul@prescod.net (Paul Prescod)
Date: Wed, 31 May 2000 00:54:08 -0500
Subject: [Doc-SIG] htmldoc.py and inspect.py
References: <NDBBJAKILMEBBFEDCMKKCEDACCAA.willguaraldi@mediaone.net>
Message-ID: <3934A900.8FB214B1@prescod.net>

will wrote:
> 
> If you look at the way it's structured right now, all the HTML is
> localized into functions in htmldoc.py.  So you could easily take
> that, copy, paste, munge, mangle and coerce into xmldoc.py or any
> other output.

That's a slightly different issue. I'm talking about allowing varying
*inputs*: docstrings that use various internal syntaxes.

-- 
 Paul Prescod  - ISOGEN Consulting Engineer speaking for himself
"Hardly anything more unwelcome can befall a scientific writer than 
having the foundations of his edifice shaken after the work is 
finished.  I have been placed in this position by a letter from 
Mr. Bertrand Russell..." 
 - Frege, Appendix of Basic Laws of Arithmetic (of Russell's Paradox)


From walter@livinglogic.de  Wed May 31 18:12:29 2000
From: walter@livinglogic.de (Walter Doerwald)
Date: Wed, 31 May 2000 19:12:29 +0200
Subject: [Doc-SIG] htmldoc.py and inspect.py
In-Reply-To: <Pine.LNX.4.10.10005300945320.2697-100000@localhost>
References: <4.3.1.0.20000530163421.00ae5710@mail.tmt.de>
Message-ID: <4.3.1.0.20000531190035.00af19f0@mail.tmt.de>

At 18:54 30.05.00, Ka-Ping Yee wrote:
>At 15:09 30.05.00, Juergen Hermann wrote:
> > I'm currently using pdoc.py, and I will definitely will give your script
> > a look. What I would like VERY much would be a Python doc system that
> > shares as much as possible with the JavaDoc/DOC++/Doxygen world,
> > especially the "commands" to provide semantic markup (like \param spam).
>
>On Tue, 30 May 2000, Walter [iso-8859-1] D=F6rwald wrote:
> > I would prefer if the semantic markup would use XML. XML was designed
> > to be used for stuff like that
>
>This was a decision i made about the philosophy of the doc
>generation script.  I chose to have it display the docstrings
>exactly as they are in the code.  This makes the script small,
>predictable, and incapable of screwing up in any significant way. [1]
>
>Also this avoids the entire issue of exactly what particular
>structured-text formatting conventions you're going to use.

But eventually someone has to implement some structured text docstring.
Plain ASCII dumps just don't cut it anymore.

And when you use XML you can convert this to practically any format
that is out there (PDF, TeX, ASCII, ...)

The display of docstring inside Python does not have to be
a plain dump of the docstring. How about a man-style display
build into Python? This could be done either via man or via a
ASCII web browser or ...

>I do like structured text (indeed, WikiWiki is WwWonderful)

WwWhat?

>but you should see just how long discussions can go on and on
>about how best to do this sort of thing.
>
>Better to just get something done and working and out there
>than continue debating, i think.

Well I started to implement an XML documentation package
as an example from my HTML generator XIST
(ftp://titan.bnbt.de/pub/livinglogic/xist/), but as this
was only meant to be an example, I'll probably cancel the
project. You might take a look at it. Perhaps it's possible
to combine both.

>By the way, that's a very unusual middle name, Walter. :)

50 years of computer science and I still can't type my name
into an email. I'm really impressed! :-(

Bye,
         Walter D=F6rwald

>[...]