Python IS slow ! [was] Re: Python too slow for real world

[Tim Peters]

[Michael Hudson, writes some seriously disturbed code for capturing
 current bindings of non-local names]

Ooh -- that is cute!

> ...
> A word of warning: this code is nasty, *nasty*, NASTY.  Possibly the
> most horrible thing you will see perpetrated in Python this year.

Watch out, or Gordon will revive a metaclass thread <wink>.

> It applies regular expressions to strings of bytecode...

Didn't need to, though.  That is, string.replace would have worked as well
here.  Regardless, it's vulnerable to picking up an inappropriate
bit-pattern by unlikely accident in either case, so should be made
bulletproof in that respect.  Believe the attached does that (and is a bit
more frugal by appending values to the consts only if they're actually
referenced, and only once when they are).

> I made Python core repeatedly when debugging it.

That's why "new" is an undocumented module <0.1 wink>.

> However, it works. The returned functions are very fast. I wrote this
> package because I wanted to avoid both the tackiness of the `default
> argument hack' and the performance penalty of using classes to fake
> closures.

Not to mention writing functions to write functions that get dynamically
compiled -- this is a reasonable hack for expert use, and with any luck at
all Guido will make it completely useless in Python2 <wink>.

> ... [getting more extreme] ...
> The thing is, this would replace two argumented bytecode with one,
> changing the length of the codestring and you'd need to recompute
> jumps. I haven't had the bloody-mindedness to get this to work yet.

At least Skip Montanaro has written a bytecode optimizer with reusable code
for all this kind of manipulation; offhand I don't have a URL, though.

rewriting-from-scratch-is-a-noble-python-tradition<wink>-ly y'rs  - tim


A safer replacement for munge_code and hardcoded VM constants:

from dis import opname, HAVE_ARGUMENT
LOAD_CONST = opname.index("LOAD_CONST")
LOAD_GLOBAL = opname.index("LOAD_GLOBAL")
del opname

def munge_code(code, vars):
    codelist = list(code.co_code)
    names = list(code.co_names)
    consts = list(code.co_consts)
    nameindex2value = {}
    for name, value in vars.items():
        try:
            index = names.index(name)
        except ValueError:
            pass
        else:
            nameindex2value[index] = value

    # Replace LOAD_GLOBAL instructions referring to names in vars w/
    # LOAD_CONST instructions referring to the name's current binding.
    nameindex2constindex = {}
    i, n = 0, len(codelist)
    while i < n:
        c = codelist[i]
        i = i+1
        op = ord(c)
        if op < HAVE_ARGUMENT:
            continue
        i = i+2
        if op != LOAD_GLOBAL:
            continue
        oparg = 256 * ord(codelist[i-1]) + ord(codelist[i-2])
        if not nameindex2value.has_key(oparg):
            # this name wasn't in vars
            continue
        if nameindex2constindex.has_key(oparg):
            index = nameindex2constindex[oparg]
        else:
            # first time this name is being replaced; create a slot
            # for it in the consts vector
            nameindex2constindex[oparg] = index = len(consts)
            consts.append(nameindex2value[oparg])
        codelist[i-3] = chr(LOAD_CONST)
        codelist[i-2] = chr(index & 0xff)
        codelist[i-1] = chr(index >> 8)
    assert i == n

    return copy_code_with_changes(
        code,
        consts=tuple(consts),
        code=string.join(codelist, ''))