pickle, cPickle, & HIGHEST_PROTOCOL

[A.B., Khalid]

I wonder if someone can explain what is wrong here. I am pickling a
list of dictionaries (see code attached) and unpickling it back using
the HIGHEST_PROTOCOL of pickle and cPickle. I am getting an error
message and trace backs if the list exceeds eight items. Whether I use
pickle or cPickle does not matter, i.e., the eight number causes a
problem in both modules, although the trace backs are of course
dissimilar.

This pickling and unpickling of the list of dictionaries worked when I
stopped using the HIGHEST_PROTOCOL in both modules (pickle and
cPickle), which got Python to use the ASCII format (I suppose) as I can
read the pickled data.

This behavior was observed in Python 2.3.4 (final), and 2.4 (final) on
Win98.

Any comments?


Regards,
Khalid



# Sample program tester.py begin
!
! import pickle as pkl
! import os
! #-----------------------------
! def test_pickle():
!    fn = 'rkeys.txt'
!    f = file(fn, 'r')
!    lines = f.readlines()
!    f.close()
!    _test_list = []
!    for line in lines:
!        sline = line.split(',')
!        #print sline
!        key, value = sline[0], sline[1].strip()
!        _test_dict = {}
!        _test_dict[key] = value
!        _test_list.append(_test_dict)
!
!    # Let's see the contents of our object
!    print _test_list
!
!    # Then pickle it
!    f = file('pkl_' + fn, 'w')
!    pkl.dump(_test_list, f, pkl.HIGHEST_PROTOCOL)
!    f.close()
!
!    # Empty it
!    _test_list = []
!    print _test_list
!
!    # Unpickling object here:
!    f = file('pkl_' + fn, 'r')
!    _test_list = pkl.load(f)
!    f.close()
!
!    # See contents after loading
!    print _test_list
!#-----------------------------
!if __name__ == '__main__':
!	test_pickle()
!
!# Sample program end


# Contents of file rkeys.txt (without the triple quotes):
"""
'1','v1'
'2','v2'
'3','v3'
'4','v4'
'5','v5'
'6','v6'
'7','v7'
'8','v8'
'9','v9'
"""


# Output (without the triple quotes)
# Using "import pickle as pkl":
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
!Traceback (most recent call last):
!  File "tester.py", line 41, in ?
!    test_pickle()
!  File "tester.py", line 34, in test_pickle
!    _test_list = pkl.load(f)
!  File "D:\PY23\PYTHON\DIST\SRC\lib\pickle.py", line 1390, in load
!    return Unpickler(file).load()
!  File "D:\PY23\PYTHON\DIST\SRC\lib\pickle.py", line 872, in load
!    dispatch[key](self)
!  File "D:\PY23\PYTHON\DIST\SRC\lib\pickle.py", line 1189, in
load_binput
!    i = ord(self.read(1))
!TypeError: ord() expected a character, but string of length 0 found
"""


# Output (without the triple quotes)
# Using "import cPickle as pkl":
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
!Traceback (most recent call last):
!  File "tester.py", line 41, in ?
!    test_pickle()
!  File "tester.py", line 34, in test_pickle
!    _test_list = pkl.load(f)
!EOFError
"""


# Output (without the triple quotes)
# Using "import cPickle as pkl", or "import pickle as pkl"
# but _not_ using the HIGHEST_PROTOCOL:
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
"""