[Doc-SIG] RE: Docstring grammar: a very revised proposal

[Ken Manheimer]

David Ascher wrote:

> Assuming this analysis is correct, I am inclined to scrap my original
> proposal, shelve my current prototype code, and work instead with Jim
Fulton
> and whoever else to discuss whether and how to modify StructuredText's
format
> and code to extend it to the needs which the DOC-SIG expressed.  The
value
> in this approach is:

Cool! David, we would be glad to have this happen.  We like
StructuredText, and would love to see it widely used and available.  And
we have very limited time right now, which means we would be very happy
to have someone like you/the python community making harmonious changes.
I think jim and/or i could field your questions/suggestions, to try to
keep us in agreement.

Re the specifics in your message:

> StructuredText as XML without a DTD:
> 
>     Unlike the grammar which was discussed on the doc-sig,
StructuredText
>     does not allow us to specify special syntax for special kinds of
text,
>     such as: signature blocks/tooltip descriptions, doctest.py code,
>     and keyworded paragraphs.  This is a lack of feature rather than a
flaw,
>     and could maybe be fixed by adding a postprocessor which would be
>     Python-internal-doc specific.

I'm not entirely clear about what you're referring to, but we're
thinking about a slightly different structure for the StructuredText
object that may admit this sort of thing.

A base class would just be responsible for parsing paragraphs (including
distinguishing bullet paragraphs that aren't newline separated - yay!),
then a document-formatting class, maybe abstract, that recognizes
generic document features like headers, emphasis, links, etc.  The
document formatting class could be subclassed by a renderering class, eg
an html-formatter or a docstring-formatter.

It so happens Jim has an intern looking at doing this restructuring as a
fun-ish (relief from C coding) side project.  It's not clear we'll get
it done, but it is clear we'd be happy to see it, and i think it would
allow a whole lot more customizability.

>     The use of single quotes to markup inline code (as in 'x') can be
>     surprising.  Many current docstrings use 'x' to refer to the
>     *string* containing the character x, not the variable x.  In
>     StructuredText, the quotes would dissappear in the rendering. With
>     practice, the current scheme could be used but users would have to
>     learn to write '"x"' to have their intent carry through to the
>     renderer.  This is probably my biggest problem with StructuredText
>     because I don't know how to fix it while maintaining
>     compatibility. Related questions for Jim or Ken are 1) How does
>     StructuredText parse ''?  2) How can one have a single quote in
>     verbatim text?

I can't answer all your questions.  We'd be interested to know what you
suggest as an alternative - how would you indicate brief <code>
passages?  It's possible the generic formatter would be settable to one
or the other, or it'd recognize both ticked-code and whatever you want
and the renderer would choose which to transform to code, or we'd have
an almost generic formatter which could be derived into different
flavors of generic formatters, each of which did the respective thing
for recognizing short <code> passages.

>     The tagging of underlined text with _'s is suboptimal.  Underlines
>     shouldn't be used from a typographic perspective (underlines were
>     designed to be used in manuscripts to communicate to the
>     typesetter that the text should be italicized -- no well-typeset
>     book ever uses underlines), and conflict with double-underscored
>     Python variable names (__init__ and the like), which would get
>     truncated and underlined when that effect is not desired.  Note
>     that while *complete* markup would prevent that truncation
>     ('__init__'), I think of docstring markups much like I think of
>     type annotations -- they should be optional and above all do no
>     harm. In this case the underline markup does harm.

Durn - i did that, and regret it.  We're really not wedded to use of
underscores for underlined text.

>     The requirement that a paragraph end with the word example or
>     examples or :: goes against my natural style, as I often do not
>     want such word or punctuation before a "displayed" paragraph.
>     Furthermore, the spec currently doesn't say how the renderer is
>     supposed to process the :: -- is it displayed as two colons, one,
>     or none?  If the two colons are not displayed by the renderer,
>     then my objection is diminished, although I would have preferred a
>     markup which is local to the paragraph which is affected, not the
>     previous one (cut and paste errors follow too easily). In some
>     versions in the past at least, both colons were displayed.  I'll
>     leave that as an open question, as additional markup could provide
>     an alternative which would suit me.r, which does the actual
rendering.  

I think we agree that it'd be nice to have the example cue in the text,
rather than before it, but we don't have a suggestion of a
natural/uncluttered way to do that.  As for the current :: format, to
double colon is supposed to be rendered as a double colon.

> Missing features:

> [references to entities other than URLs; define namespaces for
reference targets]

Sounds cool.  I think.-)

> [list items without requring blank lines]

Yay!

> [tagged paragraphs]

Might '::' example-style paragraphs fit into this category?  Maybe
provisions for special handling of the tag cue (E.g., throwing away
"Example:") would enable this to solve our examples problem.

> Technical points:

> [regex has been deprecated]

I think we would be ok with switchover to re if we had some kind of
caching FormattedDocument, which may be in the works.  But we use
StructuredText enough in zope that performance is an issue - and re
doesn't have the performance yet!

Whoops - gotta run!  Looking forward to more on this...

Ken Manheimer
klm@digicool.com