rest-mode.el and more (was: Re: [Doc-SIG] reStructuredText: What a cool idea!!)

[Stefan Merten]

-----BEGIN PGP SIGNED MESSAGE-----

Hi David and all!

Sorry for the delay but lately I have been too busy with other things
to code :-( .

Last month (28 days ago) David Goodger wrote:
> Stefan Merten wrote:
>> * Is there a Perl implementation already? Does anybody plan such a
>>   beast?
>=20
> Mark Nodine <nodine@somerset.sps.mot.com> is working on a Perl
> implementation of the reStructuredText parser.  He hasn't released any
> code yet, though.

Ok, so it's probably better to wait for some code from Mark while not
holding breath ;-) .

>> * Is there a Emacs mode already?
>=20
> Not yet.  There are a few functions in
> tools/editors/emacs/restructuredtext.el, available from CVS or the
> snapshot (<http://docutils.sf.net/docutils-snapshot.tgz>).

Yes. This could and should be integrated with my work then.

>>   Particularly it there `font-lock'
>>   support for reStructuredText documents available?
>=20
> No.  It's an interesting idea, although I'm not sure how it would
> work.

I coded a font-lock mode which recognizes most reST constructs. I
checked it with

	http://docutils.sourceforge.net/spec/rst/reStructuredText.txt

and it worked pretty good.

At the moment blocks are not handled. For the most part this is no
problem, but comments, section headers, and literal blocks should
really be font-locked correctly. Also definition lists are not
font-locked yet. Apart from that it is not clear whether it makes
sense to font-lock tables. Everything else should work fine.
Everything missing is marked in the code by ``FIXME``.

I already played around a bit with font-locking for comment blocks,
but this is a bit difficult.

>>   If no such thing is available I could give it a try.
>=20
> Please do!

:-)

You can download it from

	http://www.merten-home.de/FreeSoftware/rest-mode/index.html

I tried it with Emacs 20 and Emacs 21. XEmacs 21 did not like
everything as it seems :-( . But for sure it's a start.

If you want to include it in an official CVS tree that's fine.
However, please contact me so I can continue to maintain it or we can
look for other arrangements.

Please note that in the file I'm suggesting file name extensions
``.rest`` and ``.reST`` for reStructuredText documents. I don't know
whether there is a widely accepted extension already. However, I think
there should be one.

Also take care about the mode settings at the end of for instance

	http://docutils.sourceforge.net/spec/rst/reStructuredText.txt

This conflicts with automated settings depending on the file name.

>> * Are there transformation tools from OpenOffice format to
>>   reStructuredText?
>=20
> There is work being done on OpenOffice.org XML *writers*, but nothing
> to my knowledge the other way.
>=20
>>   For SDF lately I created a suite of XSLT style sheets which
>>   transform the `styles.xml' and `content.xml' contained in `.sxw'
>>   and `.sxc' to SDF. Though this involves some post processing by
>>   hand it is *much* easier than anything I did before.
>=20
> Sounds like a good approach.  I've heard of people setting up
> semi-automated systems to assist with conversion.
>=20
>>   Of course I'd like to see this for reStructuredText as well. If
>>   no-one did this until know, I could rewrite my XSLT scripts a bit
>>   to support reStructuredText.
>=20
> Such code would be welcome.

:-)

Now font-locking turned the light on ;-) I'll work on that.

>> * Is there a transformation for Docutils Document Tree documents in
>>   XML to a reStructuredText source form? Or following PEP 258: Is
>>   there a writer for reStructuredText?
>=20
> There has been some discussion about a generic to-reStructuredText
> conversion system, perhaps a Docutils "Writer", but no implementation
> yet.

I'd find it useful because then one could use the XML format as a
general intermediate format.

>> * Are there plans to support embedded code in reStructuredText
>>   documents?
>=20
> I believe Mark Nodine's Perl implementation has it.

Perl people seem to like that ;-) .

> There are notes
> about at <http://docutils.sf.net/spec/notes.html#misc-exec>.  Such a
> feature is inherently dangerous though, and there have to be
> safeguards in place to prevent reStructuredText/Docutils being used as
> a vector for viruses & trojans.

Yes, security will be an issue then. May be it could help to have any
code execution switched off by default and code is only executed when
this is switched on explicitly.

>> * Are there writers for plain text and/or manual pages?
>=20
> Not that I know of.  I recall some discussion, but nothing concrete.

I'd find that great - particularly for manual pages.

Hmm... While thinking about it: It would suffice to have a writer for
POD. This way you'll indirectly have a writer for manual pages for
free.

>> Finally I'd like to draw your attention to
>>=20
>>     http://template-toolkit.org/
>=20
> Looks useful, but templating systems are plentiful.
>=20
>> One thing I particularly liked with SDF is the feature to embed SDF
>> or other stuff in anything which supports comments. This way you
>> were able to embed for instance POD documentation in a C++ file. One
>> of the nice features is, that it rearranges the embedded code
>> allowing you to write your documentation at the place of the
>> definition and nonetheless get every piece of documentation at the
>> right place in the final document.
>=20
> Sounds like "literate programming".  Do you have a link to a
> description of how SDF accomplishes this rearranging?

This was solved fairly easy. The extractor (``sdfget``) used a table
like this::

  Key           Format
  NAME          "=3Dhead1 $key\n\n$data\n\n"
  SYNOPSIS      "=3Dhead1 $key\n\n$data\n\n"
  DESCRIPTION   "=3Dhead1 $key\n\n$data\n\n"
  OPTIONS       "=3Dhead1 $key\n\n$data\n\n"
  FEATURES      "=3Dhead1 $key\n\n$data\n\n"
  EXAMPLE       "=3Dhead1 $key\n\n$data\n\n"
  EXAMPLES      "=3Dhead1 $key\n\n$data\n\n"
  RETURN VALUES "=3Dhead1 $key\n\n$data\n\n"
  EXIT STATUS   "=3Dhead1 $key\n\n$data\n\n"
  DIAGNOSTICS   "=3Dhead1 $key\n\n$data\n\n"
  ENVIRONMENT   "=3Dhead1 $key\n\n=3Dover 4\n\n$data\n\n=3Dback\n\n"
  FILES         "=3Dhead1 $key\n\n=3Dover 4\n\n$data\n\n=3Dback\n\n"
  PREREQUISITES "=3Dhead1 $key\n\n$data\n\n"
  SEE ALSO      "=3Dhead1 $key\n\n$data\n\n"
  AUTHOR        "=3Dhead1 $key\n\n$data\n\n"
  LICENSE       "=3Dhead1 $key\n\n$data\n\n"
  AVAILABILITY  "=3Dhead1 $key\n\n$data\n\n"
  NOTES         "=3Dhead1 $key\n\n$data\n\n"
  BUGS          "=3Dhead1 $key\n\n$data\n\n"

The table is given in an extra report file given as an argument to
``sdfget``. The ``Key`` column gives some keys to be recognized in
your source code while the ``Format`` column gave the format to be
used as a Perl string. ``$key`` was the ``Key`` from the first column
while ``$data`` was the data contained in *all* sections tagged with
the respective key. As you may have recognized the above table
transforms to POD snippets forming a manual page.

Then in the source you can write things like that::

  #>>NAME::
  #
  # rest-mode - An Emacs mode for reStructuredText documents

  #>>AUTHOR:: Stefan Merten <smerten@oekonux.de>

  #>>SYNOPSIS::
  #
  # B<someprogram> I<option>...

  [Some Perl code here]

  #>>FILES::
  #
  # =3Ditem F<somefile>
  #
  # F<somefile> is useful for...

  [More Perl code here]

  #>>ENVIRONMENT::
  #
  # =3Ditem I<TMPDIR>
  #
  # This is the standard directory where temporary files are created. It =
is
  # overwritten by the option B<--tmpdir>.

  [More Perl code here]

  #>>FILES::
  #
  # =3Ditem F<anotherfile>
  #
  # F<anotherfile> can be used to...

As you may have recognized the triggering markup is::

  >>key::

and it can be ended by a non-comment line (though there is explicit
end markup, too).

I.e. you can distribute documentation snippets over the whole source
code - i.e. where things are used in the code. All snippets are
concatenated to a single ``$data``. The sequence of sections is given
by the key sequence given in the table. This results (in this case) in
a clean POD file which can then be processed further.

>> In the free software project I started lately
>> [http://savannah.nongnu.org/projects/mamo/] I created a plug-in for
>> the Template Toolkit which exactly does this sort of gathering of
>> snippets of documentation from an arbitrary source file

This works similar to what I showed above - however without a table
but with a template file which determines the resulting structure.

>> Thinking reStructuredText and Docutils it might qualify as the input
>> part of a reader.
>=20
> I don't understand the implications.  Can you describe this for us?

There seems to be a general trend to integrate documentation in the
source code. This applies to any language and I think this is a *very*
good thing. However, there is the problem, that the structure of a
program rarely is the best structure for its documentation. But if you
have some sort of gathering mechanism like the one described above you
can scatter documentation over the source(s) and collect it later.
Then all you need is a program for that.

Though this might be a useful feature in general of course it is for
people who want to use reStructuredText for their program
documentation. The approach using ``sdfget`` was one option. However,
SDF is dead. The approach using the Template Toolkit and the
``Gather`` plugin mentioned above would be a second option. However,
this would need the Template Toolkit in addition to the Docutils code
to work with reStructuredText. A third option would be to have such a
functionality in the reStructuredText / Docutils context. That is what
I'm trying to suggest.


						Mit Freien Gr=FC=DFen

						Stefan

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3in
Charset: noconv
Comment: Processed by Mailcrypt 3.5.7, an Emacs/PGP interface

iQCVAwUBPoNZqAnTZgC3zSk5AQF1YgQAo7FHkd30RKxS4u1qa2NQXiguO/n9smuj
H8G7JH7MNnqgKKY702aqeFg5OeNP0gLa0VPOASutAXxBbVDda/rUmPUF5WrlI+ER
YRhI+E2ZR8Y1QoUiE/9bUcl9+u4pddMuj+Dl/7YZzgw0uyhpkm66U43YkZ/K+Oow
jpgA3k8BL7M=3D
=3DkgrP
-----END PGP SIGNATURE-----