[Python-Dev] Better text processing support in py2k?

[M.-A. Lemburg]

Tim Peters wrote:
> 
> [Skip Montanaro, wants nicer text facilities]
> > While I don't want to turn Python into Perl, I would like to see
> > it do a better job of what most people probably use the language
> > for.  Here is a very short list of things I think need attention:
> >
> >     1. [*A* clear way to do memory- and time-efficient textfile
> >         input]
>
> ...
> 
> The Python QIO extension module is much easier to port but less compatible
> (it doesn't use stdio, so QIO-opened files don't play well with others) and
> slower (although that's likely repairable -- he's got two passes over the
> buffer where one hairier pass should suffice).

What is QIO ?
 
> > Depending how far people want to go with things, adding some
> > language syntax to support regular expressions might be in order.
> > ...
> >     3. I've not yet used it, but I am told the pattern matching in
> >        Marc-Andre Lemburg's mxTextTools
> >       (http://starship.python.net/crew/lemburg/)
> >        is both powerful and efficient (though it certainly appears
> >        complex).  Perhaps it deserves consideration for
> >        incorporation into the core Python distribution.
> 
> It's not complex, it's complicated -- and *that's* what makes it un-Pythonic
> <wink>.  Tony Ibbs has written a friendly wrapper around mxTextTools that
> suppresses much of the non-essential complication.  OTOH, if you go into
> this with a regexp mindset, it will run much slower than a real regexp
> package, because the bulk of the latter is devoted to doing optimization;
> mxTextTools is WYSIWYG (it screams if you code to its strengths, but crawls
> if you e.g. try to implement naive backtracking).

All true. mxTextTools provides the tools, not the magic. But this
is also its strength: you can optimize the hell out of your particular
parsing requirement without having to think about how the RE optimizer
works.

> You should go to the REBOL site and look at the description of REBOL's PARSE
> verb in the FAQ ... mumble, mumble ... at
> 
>     http://www.rebol.com/faq.html#11550948
> 
> Here's an example pulled from that page (this is a REBOL code fragment):
> 
>     digit: charset "0123456789"
>     expr: [term ["+" | "-"] expr | term]
>     term: [factor ["*" | "/"] term | factor]
>     factor: [primary "**" factor | primary]
>     primary: [value | "(" expr ")"]
>     value: [digit value | digit]
> 
>     parse "1 + 2 ** 9" expr
> 
> There hasn't been a pattern scheme this clean, convenient or powerful since
> SNOBOL4.  It exploits REBOL's Forth-like (lack of!) syntax, and
> Smalltalk-like penchant for passing around thunks (anonymous closures --
> "[...]" in REBOL builds a lexically-scoped entity called "a block", which
> can be treated as code (executed) or data (manipulated like a Python list)
> at will).

Looks nice indeed, but how does executable code fit into
that definition ? (mxTextTools allows you to write your own
parsing elements in Python, BTW; it should be possible to
use those mechanisms to achieve a similar intergration.)
 
> ...
>
> BTW, the mxTextTools engine could be used to get blazing implementations of
> the primary Searcher methods (it excels at simple analysis).  OTOH, making
> lots of calls to analyze short strings is slow.

That's why mxTextTools converts these search idioms into byte codes
which it executes at C level. Some future version will even "precompile"
the tuple input and then omit the type checks during the search...
that should give another noticeable speedup. Note that recursion
etc. can be done at C level too -- Python function calls are not
needed.

> The only clean solutions to
> that are Perl's and Icon's (build everyting into one language so the
> compiler can optimize stuff away), and REBOL's (make no distinction between
> code and data, so that code can be analyzed & optimized at runtime -- and
> build the entire implementation around making closures and calls
> supernaturally fast).

Just for kicks, here is the mysplit() function using mxTextTools:

from mx.TextTools import *

table = (
    # Match all whitespace
    (None,AllInSet,whitespace_set,+1),
    # Match and tag all non-whitespace
    ('text',AllInSet + AppendMatch,nonwhitespace_set,+1),
    # Loop until EOF
    (None,EOF,Here,-2),
    )

def mysplit(text):

    return tag(text,table)[1]

The timings:
 mysplit: 5.84 sec.
 string.split: 3.62 sec.

Note that you can customize the above to split text at any
character set you like, not just whitespace... without
compiling or writing C code. The function mx.TextTools.setsplit()
provides this functionality as pure C function.

-- 
Marc-Andre Lemburg
______________________________________________________________________
Y2000:                                            Get ready to party !
Business:                                      http://www.lemburg.com/
Python Pages:                           http://www.lemburg.com/python/