speeding up reading files (possibly with cython)

[skip at pobox.com]

    >> about 800 MB in size containing tab separated data... my program
    >> parses this file and stores its fields in a dictionary of lists.

    ...

    >> currently, this is very slow in python, even if all i do is break up
    >> each line using split() and store its values in a dictionary,
    >> indexing by one of the tab separated values in the file.

Why not use the csv module and specify TAB as your delimiter?

    reader = csv.reader(open(fname, "rb"))
    for row in reader:
        ...

-- 
Skip Montanaro - skip at pobox.com - http://www.smontanaro.net/