Jeremy Hylton : weblog : 2003-11-24

ZEO Cache Effectiveness

Monday, November 24, 2003, 11:19 p.m.

I just wanted to confirm that loading objects from the ZEO cache is faster than loading them from the ZEO server. With a fast network, you might think we could read data from the ZEO server as fast as we could read it off disk. Not close, though. The cache looks to be 40x faster.

We have probably made a lot of decision decisions that favor simplicity over performance for the ZEO server. If an object is sitting in memory on the server, couldn't we send it to the server faster than we could read it from disk? The protocol is one problem. The data gets pickled as part of the encoding process, then it is chunked up by asyncore. As a result, we make several copies of the data as it is sent. The receiver has to do the same. Neither cPickle nor asyncore is that fast compared to copying bytes over the network.

I did a simple test with a 1.5GB Data.fs, which means that the whole file can't sit in the machine's buffer cache. (It's only got 1GB of RAM.) I chose 250 random oids and loaded each object 11 times. I compared the median times of cached lookup and a direct zeoLoad() call to the storage server. Note that zeoLoad has a counter-productive optimization for locked versions, causing it to read the object's header twice. It will hit in the cache the second time, but the struct.unpack() call is probably still slow.

For the 250 oids, the cache lookup is about 40 times faster. A read from the cache takes about 80 microseconds. A read from the server takes about 3 milliseconds. The server was completely idle when I made these measurements. If the server was heavily loaded, reads would take longer.