Python too slow for real world

[Christian Tismer]

Just did a little more cleanup to the code.
This it is:

def read_write_bulk(input, output, exclude):
    bufsize = 1 << 16 ; splitter = ">"
    ignore=exclude.has_key ; split=string.split ; No = None

    buffer = input.read(bufsize) 
    got = len(buffer)
    while 1 :
        pieces = split(buffer, splitter)
        idx = 0
        inner = pieces[1:-1]
        for piece in inner:
            idx = idx+1 ; key = split(piece, No, 1)[0]
            if ignore(key): 
                del inner[idx] ; idx = idx-1
        output.write(splitter)
        output.write(string.join(inner, splitter))
        if got==0:
            break
        chunk = input.read(bufsize)
        buffer = splitter+pieces[-1] + chunk
        got = len(chunk)
        if got==0:
            buffer = buffer+splitter # spill last one

Also, I think with this I/O layout, buffering of the
files doesn't count any more at all.
Let me know if it is still much slower than Perl.

ciao - chris

-- 
Christian Tismer             :^)   <mailto:tismer at appliedbiometrics.com>
Applied Biometrics GmbH      :     Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101   :    *Starship* http://starship.python.net
10553 Berlin                 :     PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint       E182 71C7 1A9D 66E9 9D15  D3CC D4D7 93E2 1FAE F6DF
     we're tired of banana software - shipped green, ripens at home