cPickle alternative?

[Michael Peuser]

Hi Drochem,

(1) Your dataset seems to break the binary cPickle mode ;-) (I tried it with
the "new Pickle" in 2.3 - same result: "EOF error" when loading back...) May
be there is someone interested in fixing this ....


(2) I run your code and - as you noticed - it takes some time to *generate*
the datastructure. To be fair pickle has to do the same so it cannot be
*significantly* faster!!!
The size of the file was 5,5 MB

(3) Timings (2.2):
Generation of data:   18 secs
Dunping:  3,2 secs
Loading: 19,4 sec

(4) I couldn't refrain from running it under 2.3
Generation of data: 8,5 secs !!!!
Dumping: 6,4 secs  !!!!
Loading: 5,7 secs


So your programming might really improve when changing to 2.3 - and if
anyone can fix the cPickle bug, protocol mode 2 will be even more efficient.

Kindly
Michael

"Drochom" <pedrosch at gazeta.pl> schrieb im Newsbeitrag
news:bhjn6v$pi8$1 at nemesis.news.tpi.pl...
>
[....]
> TRY THIS:
>
> import cPickle as Pickle
> from time import clock
> from random import randrange
>
>
> x=[]
>
> for i in xrange(20000):
>  c = []
>  for j in xrange(randrange(2,25)):
>   c.append((chr(randrange(33,120)),randrange(1,100000),randrange(1,3)))
>  c = tuple(c)
>  x.append(c)
>  if i%1000==0: print i #it will help you to survive waiting...
> print len(x), "records"
>
> t0=clock()
> f=open ("test","w")
> Pickle.dump(x,f,0)
> f.close()
> print "out=", clock()-t0
>
>
> t0=clock()
> f=open ("test")
> x=Pickle.load(f)
> f.close()
> print "in=", clock()-t0
>
> Thanks once again:)
>
>
>