Performance of cPickle module

[Holger Türk]

sh at defuze.org wrote:
 > [...]
> If I had to use a database, the database would keep track of my users and I
> would only need to do a SQL statement. Would the cPickle more efficient in my
> case than a database ?
> 
> To give a bit of code let's say that I have something like :
> 
> import cPickle
> 
> class UserData:
> def __init__(self,name,email):
> self.name = name
> self.email = email
> 
> class Users:
> def __init__(self):
> self.users = {}
> self.hasChanged = false
> 
> def _deserialize(self):
> if self.hasChanged == false:
> self.users = cPickle.load('users.dat')
> else:
> #parse the xml file...
> 
> Is it an efficient method ?
> 
> Thanks
> - Sylvain

Hi,

this may be interesting for others, too.
I modified the example given above a little, entered
1000 users and saved and loaded the Users object
1000 times using cPickle on an Athlon 1GHz.
The results are:

[...]
995
996
997
998
999

real    5m26.115s
user    3m59.570s
sys     0m5.060s

That are 0.326s per save/load-roundtrip.

-rw-r--r--    1 holger   users      173972 2004-05-11 15:23 test.pickle


For 10 users:

[...]
995
996
997
998
999

real    0m5.148s
user    0m2.710s
sys     0m0.740s

0.005s per roundtrip.

-rw-r--r--    1 holger   users        1708 2004-05-11 15:25 test.pickle


That should be fast enough for a weblog application.
The http/cgi-overhead and the concurrent access on the
pickled objects when writing them will probably be
the harder problems.

Greetings,

Holger


Here's the program:
#!/usr/bin/python

import string, random, cPickle

def randString (l):
	return "".join ([string.letters [random.randrange (l)] for i in range (l)])

class UserData:
	def __init__(self,name,email):
		self.name = name
		self.email = email

class Users:
	def __init__(self):
		self.users = {}

u = Users ()

for a in range (1000):
	u.users [randString (20)] = (UserData (randString (40), randString (40)))

for a in range (1000):
	print a
	
	f = open ("test.pickle", "w")
	p = cPickle.Pickler (f)
	p.dump (u)
	f.close ()

	f = open ("test.pickle", "r")
	p = cPickle.Unpickler (f)
	u = p.load ()
	f.close ()