WANTED: Regular expressions for breaking TeX/LaTeX document into tokens

[Jonathan Fine]

Wes James wrote:
> On Wed, Feb 24, 2010 at 5:03 AM, Jonathan Fine <J.Fine at open.ac.uk> wrote:
>> Hi
>>
>> Does anyone know of a collection of regular expressions that will break a
>> TeX/LaTeX document into tokens?  Assume that there is no verbatim or other
>> category code changes.
> 
> I'm not sure how this does it, but it might help:
> 
> http://plastex.sourceforge.net/plastex/sect0025.html

Thanks, Wes.  I'm already using PlasTeX

It handles changes of category codes, which makes it over the top for 
what I want to do.  In addition it is a  fairly large complex 
application, and sadly it's not at all easy to use just a part of the 
code base.

There's been more discussion of this thread on comp.text.tex (which is 
where I set the follow-up to).

-- 
Jonathan