[Doc-SIG] scope of the parser

[David Goodger]

Alan:
> > > 95% of the time the desired behavior for directives will be to
> > > process the block argument as RST, and not do any other RST
> > > processing. In those cases, you can decouple. If you want
> > > something else, you have to muck with the inside of the parser.

David:
> > I don't follow. Can you give an example of a directive in the
> > other 5%?

Alan:
> A directive indicating an indexed and captioned code fragment::
>
>    .. example:: index=2.2 caption=`Recognizing inline markup`
>       inline.openers = '\'"([{<'
>       inline.closers = '\'")]}>'
>       inline.start_string_prefix = (r'(?:(?<=^)|(?<=[ \n%s]))'
>                                     % re.escape(inline.openers))
>       inline.end_string_suffix = (r'(?:(?=$)|(?=[- \n.,:;!?%s]))'
>                                   % re.escape(inline.closers))

I don't see how this is an example of a non-RST (or post-RST)
directive. It seems like a simple variant of a literal block. Perhaps
it depends on what the "index" and "caption" do?

However, the second example is easier to understand:

> A directive indicating raw output for a given format::
>
>    .. raw:: format=latex
>       \documentclass[twocolumn]{article}

I assume this is to be passed through to the LaTeX writer untouched.
Easy to accomplish. The "raw" directive should produce something like
this::

    <raw format="latex" xml:space="preserve">
        \documentclass[twocolumn]{article}

Even better would be if the element name was "raw-latex". Either way,
there are two possibilities here:

1. A directive becomes widespread enough to warrant built-in support.
   The DTD, dps.nodes, and all writers learn about it (some may
   learn to ignore it).

2. It's of limited application-specific use. The application's code
   needs a dps.nodes.Node subclass (probably subclass of TextElement)
   to handle it, the application may need a DTD extension [#]_, and
   the application's writer variants need support.

.. [#] Currently, the DTD is used only as a formalized notation of the
   DPS doc tree implemented in dps.nodes. In future, there might be
   direct doc tree validation support added to the code, which may use
   the DTD or some other notation (such as TREX or RELAX).

> > What do you mean by "decouple"?
>
> Do directive processing outside the parser, and in fact outside the
> entire DPS/RST codebase.

Do you mean "processing using code outside the DPS/RST project
codebase, but still run interleaved with it" or "processing after the
DPS/RST codebase has finished its work"? The former can be done now.
The latter, I'm not sure about.

> If a user comes along and decides they want a new directive, they
> ought to be able to do this while treating RST as a black box.

I don't agree. You need a degree of understanding of the inner
workings of a system in order to extend it. Adding a directive might
require a less than complete understanding of the internals, but not
none.

> User feeds in ``.. foo:: bar``, gets back ``<directive name="foo"
> args="bar"/>``, and can then process the XML into whatever they
> want, or treat it as a literal, or output a warning, or whatever.

The problem with this is that error checking goes out the window. A
directive with a misspelled type name would trigger no warnings in the
parser, and possibly not even in the writer.

If application-specific directives appear that cannot or should not be
added to the core parser, the current code can't handle it. In order
to add a directive, you have to put the directive implementation
module into the restructuredtext/directives directory, and edit
project source files: restructuredtext/directives/__init__.py for a
canonical directive name to (module name, function name) mapping, and
restructuredtext/languages/en.py for a language-specific to canonical
name mapping. This shortcoming could be remedied to allow runtime
additions to the "directive registry", with implementation modules
outside of the parser's package tree.

-- 
David Goodger    goodger@users.sourceforge.net    Open-source projects:
 - Python Docstring Processing System: http://docstring.sourceforge.net
 - reStructuredText: http://structuredtext.sourceforge.net
 - The Go Tools Project: http://gotools.sourceforge.net