From luc.saffre at gmx.net  Wed Nov 19 18:21:17 2003
From: luc.saffre at gmx.net (Luc Saffre)
Date: Wed Nov 19 18:21:26 2003
Subject: [Doc-SIG] New concept of "commands"?
Message-ID: <3FBBFAED.9060301@gmx.net>

Hello,

there is a general problem with docutils: if you have a collection of
hyperlinked pages (for example a website), then the only possibility
to refer to other pages of your site is to hard-code the extension 
(usually `.html`) in the reStructuredText source. But if you hard-coded 
html links and then render your site in pdf (for example) then the 
intra-site links would also be broken since the other pages would have 
another extension than `.html`.

Solution could be to introduce a new docutils command, let's call it
"lref" (for "local reference"), whose syntax would be something like::

   [lref index Main Page]

and which would get expanded by the writer (or translator?) to::

   <a href="index.html">Main page</a>

when in static html mode, but to something else when in pdf mode. (I 
suppose that it is possible to link from one pdf file to another?)

The syntax `[cmd param1...paramN]` would be a new concept for
reStructuredText:  if a `[` is followed immediately by a recognized
command, then the parser collects the "parameters" (the remaining
text until the closing ']'), then it executes a function associated
with this command. The function should probably return a list of
parsed nodes (not to be parsed again).  If a `[` is not followed by any 
known command, then there is no special action.

Another example would be a `filref` command, to be used by authors
who often talk about individual files of a CVS tree::

     [fileref src/lino/__init__.py]

This could (now for both html or pdf) expand automatically to the 
following extra-site link::

   <a 
href="http://cvs.sourceforge.net/viewcvs.py/lino/lino/src/lino/__init__.py?rev=1.4&view=auto">src/lino/__init__.py</a>

In general, "commands" would be something similar to directives, but 
without requiring to be in a separate paragraph.


Questions:

- Am I missing something? Can my problem be solved without requiring a 
new syntax concept?

- Who can give me hints about how to implement this?


Luc Saffre
Author of Lino:
http://lino.sourceforge.net/


From goodger at python.org  Wed Nov 19 21:55:17 2003
From: goodger at python.org (David Goodger)
Date: Wed Nov 19 21:55:19 2003
Subject: [Doc-SIG] New concept of "commands"?
In-Reply-To: <3FBBFAED.9060301@gmx.net>
References: <3FBBFAED.9060301@gmx.net>
Message-ID: <3FBC2D15.1040506@python.org>

Luc Saffre wrote:
 > there is a general problem with docutils: if you have a collection
 > of hyperlinked pages (for example a website), then the only
 > possibility to refer to other pages of your site is to hard-code the
 > extension (usually `.html`) in the reStructuredText source. But if
 > you hard-coded html links and then render your site in pdf (for
 > example) then the intra-site links would also be broken since the
 > other pages would have another extension than `.html`.

This issue has come up before.  A summary of the last idea is at
<http://docutils.sf.net/spec/notes.html#adaptable-file-extensions>.

 > Solution could be to introduce a new docutils command, let's call it
 > "lref" (for "local reference"), whose syntax would be something
 > like::
 >
 >   [lref index Main Page]

I'd rather not add syntax unless absolutely necessary.  If the earlier
idea isn't sufficient, perhaps some variation on the existing
"`reference`_" or "`reference <URL>`_" syntax?

 > The syntax `[cmd param1...paramN]` would be a new concept for
 > reStructuredText:  if a `[` is followed immediately by a recognized
 > command, then the parser collects the "parameters" (the remaining
 > text until the closing ']'), then it executes a function associated
 > with this command.

It seems fragile to me.  [Brackets] are very commonly used in ordinary
text, and even more so in technical text.

 > Another example would be a `filref` command, to be used by authors
 > who often talk about individual files of a CVS tree::
 >
 >     [fileref src/lino/__init__.py]
 >
 > This could (now for both html or pdf) expand automatically to the
 > following extra-site link::
 >
 >   <a href="http://cvs.sourceforge.net/viewcvs.py/lino/lino/src/
lino/__init__.py?rev=1.4&view=auto">src/lino/__init__.py</a>

There's another old idea related to this:
<http://docutils.sf.net/spec/notes.html#targets-in-other-documents>.

 > In general, "commands" would be something similar to directives, but
 > without requiring to be in a separate paragraph.

That concept is what "substitutions" are now.  The directive
parameters are outside of the text, which I consider a major advantage
to readability.

 > - Who can give me hints about how to implement this?

First, study the spec
(http://docutils.sf.net/spec/rst/reStructuredText.html) and the parser
code (overview starts in docutils/parsers/rst/__init__.py's module
docstring).  Next, ask some specific questions.  Let's firm up the
proposal first though.

-- 
David Goodger    http://starship.python.net/~goodger
For hire: http://starship.python.net/~goodger/cv
Docutils: http://docutils.sourceforge.net/
(includes reStructuredText: http://docutils.sf.net/rst.html)


From luc.saffre at gmx.net  Thu Nov 20 04:24:22 2003
From: luc.saffre at gmx.net (Luc Saffre)
Date: Thu Nov 20 04:24:44 2003
Subject: [Doc-SIG] New concept of "commands"?
In-Reply-To: <3FBC2D15.1040506@python.org>
References: <3FBBFAED.9060301@gmx.net> <3FBC2D15.1040506@python.org>
Message-ID: <3FBC8846.5040204@gmx.net>

Thank you, David.

I agree that a syntax change is something to avoid if possible, and you 
mentioned two alternatives:

- Substitutions : are not a candidate for my problem because there is no 
possibility to specify parameters.

- Some variation on the existing "`reference`_" or "`reference <URL>`_" 
syntax... : This seems promising!

Idea 1 : provide a possibility to declare a function that decides what 
to do with URLs. The parser would call this function each time that it 
found an URL. The default function would do the same as what is done 
now. A framework-defined function would call the default function if the 
URL is a normal one, but if it starts with a keyword (for example 
"lref:" instead of the normal "http:") then it would apply its own 
processing (and return a list of nodes, I guess). Then I could write::

   Return to `Main page <lref:index>`_.

or::

   See file `__init__.py <fileref:src/lino/__init.py>`_ for details.

Idea 2 : what about  `Python expression`!_ or something similar? The 
python expression would not be allowed to contain backticks, but imho 
that's not a big problem. Problem could rather be that a text like "She 
shouted "_No!_" and ran away." must not be processed this way. The rule 
would be that something which ends with "!_" and does not start with "_" 
is to be considered a Python expression to evaluate.

   Return to `url("index", "Main page")`!_.

or::

   See file `fileref("src/lino/__init__.py")`!_ for details.


I hope for some feedback about these ideas.

Luc Saffre


On 20/11/2003 4:55, David Goodger wrote:

> Luc Saffre wrote:
>  > there is a general problem with docutils: if you have a collection
>  > of hyperlinked pages (for example a website), then the only
>  > possibility to refer to other pages of your site is to hard-code the
>  > extension (usually `.html`) in the reStructuredText source. But if
>  > you hard-coded html links and then render your site in pdf (for
>  > example) then the intra-site links would also be broken since the
>  > other pages would have another extension than `.html`.
> 
> This issue has come up before.  A summary of the last idea is at
> <http://docutils.sf.net/spec/notes.html#adaptable-file-extensions>.
> 
>  > Solution could be to introduce a new docutils command, let's call it
>  > "lref" (for "local reference"), whose syntax would be something
>  > like::
>  >
>  >   [lref index Main Page]
> 
> I'd rather not add syntax unless absolutely necessary.  If the earlier
> idea isn't sufficient, perhaps some variation on the existing
> "`reference`_" or "`reference <URL>`_" syntax?
> 
>  > The syntax `[cmd param1...paramN]` would be a new concept for
>  > reStructuredText:  if a `[` is followed immediately by a recognized
>  > command, then the parser collects the "parameters" (the remaining
>  > text until the closing ']'), then it executes a function associated
>  > with this command.
> 
> It seems fragile to me.  [Brackets] are very commonly used in ordinary
> text, and even more so in technical text.
> 
>  > Another example would be a `filref` command, to be used by authors
>  > who often talk about individual files of a CVS tree::
>  >
>  >     [fileref src/lino/__init__.py]
>  >
>  > This could (now for both html or pdf) expand automatically to the
>  > following extra-site link::
>  >
>  >   <a href="http://cvs.sourceforge.net/viewcvs.py/lino/lino/src/
> lino/__init__.py?rev=1.4&view=auto">src/lino/__init__.py</a>
> 
> There's another old idea related to this:
> <http://docutils.sf.net/spec/notes.html#targets-in-other-documents>.
> 
>  > In general, "commands" would be something similar to directives, but
>  > without requiring to be in a separate paragraph.
> 
> That concept is what "substitutions" are now.  The directive
> parameters are outside of the text, which I consider a major advantage
> to readability.
> 
>  > - Who can give me hints about how to implement this?
> 
> First, study the spec
> (http://docutils.sf.net/spec/rst/reStructuredText.html) and the parser
> code (overview starts in docutils/parsers/rst/__init__.py's module
> docstring).  Next, ask some specific questions.  Let's firm up the
> proposal first though.
> 


From goodger at python.org  Fri Nov 21 01:01:17 2003
From: goodger at python.org (David Goodger)
Date: Fri Nov 21 01:01:17 2003
Subject: [Doc-SIG] New concept of "commands"?
In-Reply-To: <3FBC8846.5040204@gmx.net>
References: <3FBBFAED.9060301@gmx.net> <3FBC2D15.1040506@python.org>
	<3FBC8846.5040204@gmx.net>
Message-ID: <3FBDAA2D.8030409@python.org>

Luc Saffre wrote:
> Thank you, David.

You're welcome.

 > - Substitutions : are not a candidate for my problem because there
 > is no possibility to specify parameters.

Perhaps not.  Not directly anyhow.  Did you look at the proposal at
<http://docutils.sf.net/spec/notes.html#adaptable-file-extensions>?

 > - Some variation on the existing "`reference`_" or "`reference
 > <URL>`_" syntax... : This seems promising!
 >
 > Idea 1 : provide a possibility to declare a function that decides
 > what to do with URLs. The parser would call this function each time
 > that it found an URL. The default function would do the same as what
 > is done now. A framework-defined function would call the default
 > function if the URL is a normal one, but if it starts with a keyword
 > (for example "lref:" instead of the normal "http:") then it would
 > apply its own processing (and return a list of nodes, I guess). Then
 > I could write::
 >
 >   Return to `Main page <lref:index>`_.
 >
 > or::
 >
 >   See file `__init__.py <fileref:src/lino/__init.py>`_ for details.

Interesting idea: custom URI schemes that trigger internal behavior.
We'd have to be careful not to duplicate any existing schemes (see
docutils/urischemes.py); perhaps a custom scheme syntax that cannot
conflict with public schemes (i.e., an invalid URI syntax)?  The
syntax for URI schemes (from RFC 2396) is::

     scheme        = alpha *( alpha | digit | "+" | "-" | "." )

Perhaps a role-like syntax would suffice?  I don't think this is valid
URI syntax::

     Return to `Main page <:lref:index>`_.

 > Idea 2 : what about  `Python expression`!_ or something similar?

There is a proposal for an "exec" directive at
<http://docutils.sf.net/spec/notes.html#misc-exec>, followed by
"system" and "eval".  All of these are potentially dangerous, which is
one reason they haven't been implemented up to now.  The notes file
has some ideas for reducing the danger.  But do we want to leave the
door open to potentially malicious *documents*?  Alternatively, it
ought to be easy to use one of the many templating systems with
reStructuredText files, chained together with Docutils.

In general, for text that isn't directly part of the document I'd much
rather use substitutions rather than adding inline syntax, to keep the
non-textual details outside of the text.  Python expressions can be
quite readable, but still, only to a programmer.  Not to a
non-technical writer.

-- 
David Goodger    http://starship.python.net/~goodger
For hire: http://starship.python.net/~goodger/cv
Docutils: http://docutils.sourceforge.net/
(includes reStructuredText: http://docutils.sf.net/rst.html)


From luc.saffre at gmx.net  Fri Nov 21 05:55:02 2003
From: luc.saffre at gmx.net (Luc Saffre)
Date: Fri Nov 21 05:55:53 2003
Subject: [Doc-SIG] Adaptable file extensions (was: New concept of
	"commands"?)
In-Reply-To: <3FBDAA2D.8030409@python.org>
References: <3FBBFAED.9060301@gmx.net> <3FBC2D15.1040506@python.org>
	<3FBC8846.5040204@gmx.net> <3FBDAA2D.8030409@python.org>
Message-ID: <3FBDEF06.3060600@gmx.net>

Yes, <http://docutils.sf.net/spec/notes.html#adaptable-file-extensions>
seems a good solution for the problem of "intra-site" or "local" links 
(i mean links who stay on the same site).

I just made a first release of WebMan which is a candidate for such a 
change, and which gives a concrete application:
<http://lino.sourceforge.net/webman.html>
When WebMan generates static html, then the file extension for 
intra-site links is ".html", but when WebMan runs as a Medusa/Quixote 
server, there must be no extension at all. Though currently the links 
are hard-coded with ".html". That's why intra-site links currently fail 
when you serve a WebMan module dynamically.

 From WebMan's point of view I can say that support for file types other 
than documents is not necessary and makes things complicated. The rule 
should simply be: if an URL ends with ".*", then ask the Writer for the 
file extension and replace the ".*" by this.

For someone who has Python and docutils installed it should be 
relatively painless to get the sources from CVS,
(see http://lino.sourceforge.net/install.html#get-the-sources)
insert them manually to your Python path (don't run setup.py!) and run 
the scripts webman.py and webmandemo.py. (Oops: for webmandemo.py you 
will also need medusa and quixote installed:
http://www.amk.ca/python/code/medusa.html
http://www.mems-exchange.org/software/quixote/)

Luc Saffre


On 21/11/2003 8:01, David Goodger wrote:
> Luc Saffre wrote:
>  > - Substitutions : are not a candidate for my problem because there
>  > is no possibility to specify parameters.
> 
> Perhaps not.  Not directly anyhow.  Did you look at the proposal at
> <http://docutils.sf.net/spec/notes.html#adaptable-file-extensions>?


From luc.saffre at gmx.net  Fri Nov 21 07:17:28 2003
From: luc.saffre at gmx.net (Luc Saffre)
Date: Fri Nov 21 07:17:43 2003
Subject: [Doc-SIG] New concept of "commands"?
In-Reply-To: <3FBDAA2D.8030409@python.org>
References: <3FBBFAED.9060301@gmx.net> <3FBC2D15.1040506@python.org>
	<3FBC8846.5040204@gmx.net> <3FBDAA2D.8030409@python.org>
Message-ID: <3FBE0258.1010103@gmx.net>

David,

yes, custom URI schemes that trigger internal behavior would enable me 
to write a solution for the "fileref" problem.

And I agree with you that an 'exec' (or 'python') directive should not 
be integrated into standard reStructuredText for security reasons.  This 
is no problem for me or WebMan, since WebMan can declare its own 
directive (as done in 
http://cvs.sourceforge.net/viewcvs.py/lino/lino/src/lino/webman/xdocutils.py?view=markup)

But I would like to come back to my original idea of "commands". After 
all these discussions I still believe that this would be an elegant 
extension slot for reStructuredText.  It would also make custom URI 
schemes unnecessary.

Your problem was that the `[cmd param1...paramN]` syntax seems fragile. 
Okay, we could require an underscore after the "]" and say that commands 
are a special case of citation references: if the label starts with "=", 
then its not a citation reference but a command.

Consider also that this "feature" can be disabled in standard rst as 
long as there is no declared command.  It is just an extension slot for 
tools as WebMan. Maybe there are authors who wrote reStructured texts 
containing citations whose labels start with "=".

It's true that WebMan would create its own dialect of "extended rst", 
making webman files unreadable by standard docutils tools. But this is 
WebMan's problem, not your's.

Consider also that commands would enhance readability for specific 
contexts::

   Don't execute [=fileref /setup.py]_ because it does not work yet.

is much more readable than::

   Don't execute `setup.py <:fileref:/setup.py>`_ because it does
   not work yet.


I did not yet find time to study the parser's code in order to check how 
expensive my idea would be. But in any case I hope for your blessing 
before diving into this project.

Luc


On 21/11/2003 8:01, David Goodger wrote:

> Luc Saffre wrote:
> 
>  > - Some variation on the existing "`reference`_" or "`reference
>  > <URL>`_" syntax... : This seems promising!
>  >
>  > Idea 1 : provide a possibility to declare a function that decides
>  > what to do with URLs. The parser would call this function each time
>  > that it found an URL. The default function would do the same as what
>  > is done now. A framework-defined function would call the default
>  > function if the URL is a normal one, but if it starts with a keyword
>  > (for example "lref:" instead of the normal "http:") then it would
>  > apply its own processing (and return a list of nodes, I guess). Then
>  > I could write::
>  >
>  >   Return to `Main page <lref:index>`_.
>  >
>  > or::
>  >
>  >   See file `__init__.py <fileref:src/lino/__init.py>`_ for details.
> 
> Interesting idea: custom URI schemes that trigger internal behavior.
> We'd have to be careful not to duplicate any existing schemes (see
> docutils/urischemes.py); perhaps a custom scheme syntax that cannot
> conflict with public schemes (i.e., an invalid URI syntax)?  The
> syntax for URI schemes (from RFC 2396) is::
> 
>     scheme        = alpha *( alpha | digit | "+" | "-" | "." )
> 
> Perhaps a role-like syntax would suffice?  I don't think this is valid
> URI syntax::
> 
>     Return to `Main page <:lref:index>`_.
> 
>  > Idea 2 : what about  `Python expression`!_ or something similar?
> 
> There is a proposal for an "exec" directive at
> <http://docutils.sf.net/spec/notes.html#misc-exec>, followed by
> "system" and "eval".  All of these are potentially dangerous, which is
> one reason they haven't been implemented up to now.  The notes file
> has some ideas for reducing the danger.  But do we want to leave the
> door open to potentially malicious *documents*?  Alternatively, it
> ought to be easy to use one of the many templating systems with
> reStructuredText files, chained together with Docutils.
> 
> In general, for text that isn't directly part of the document I'd much
> rather use substitutions rather than adding inline syntax, to keep the
> non-textual details outside of the text.  Python expressions can be
> quite readable, but still, only to a programmer.  Not to a
> non-technical writer.
>