[New-bugs-announce] [issue7594] shlex refactoring

[Brian Harring]

New submission from Brian Harring <ferringb at gmail.com>:

Currently, shlex.shlex's internal read_token is a bit of a nasty mess to
follow and pretty poorly performant.

The refactoring I'll be attaching essentially inverts the state machine
approach read_token uses- instead, converting it over to a more
procedural set of loops.  The benefits of this is that it's far, far
easier to track what the code is actually doing and the performance goes
up fairly drastically via tightening the various for loops (for example,
if doing quoting, you only care about escapes and the terminating quote-
thus a loop there).

End result, the refactored code actually has whitespace in it and still
is shorter- specifically the tokenenizer was converted into a generator
to preserve state internally, thus simplifying the code flow immensely.

Beyond that, this should still be api compatible... and is 2x faster,
passing tests.

Note this patch is cut against 2.6.4; assuming folks are happy w/ the
general approach, I'll rebase it to py3k trunk.

>From there, I'd be willing to tackle the other shlex issues where
applicable, but personally... not w/out the refactoring base, since that
state machine is a pita to trace out.

----------
components: Library (Lib)
files: shlex-py2.6.4-20091229.patch
keywords: patch
messages: 96987
nosy: ferringb
severity: normal
status: open
title: shlex refactoring
type: performance
versions: Python 2.6, Python 2.7, Python 3.2
Added file: http://bugs.python.org/file15692/shlex-py2.6.4-20091229.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7594>
_______________________________________