How to make this unpickling/sorting demo faster?

[Paul Rubin]

Steve Bergman <sbergman27 at gmail.com> writes:
> I started at about 7 seconds to just read in and sort, and at 20
> seconds to read, sort, and write.  I can now read in, sort, and write
> back out in almost exactly 5 seconds, thanks to marshal and your
> suggestions.

That is not bad if you aren't using a raid disk.  If you require that
the output actually be written to the disk though, make sure to type
"sync" after running your program and count that as part of the
elapsed time.  If those are 64-bit floats then a million of them is 16
MB and you're doing pretty good if you can get that much through a
file system in 1 second, so reading and writing the data is about 2
seconds all by itself.  Therefore the likelihood of a C or asm program
being 20x faster including disk i/o is dim.  But realistically,
counting just CPU time, you might get a 20x speedup with assembler if
you're really determined, using x86 SSE (128-bit vector) instructions,
cache prefetching, etc.