Load a list subset with pickle?

[Robert Kern]

On 2009-10-15 11:05 AM, Peng Yu wrote:
> On Tue, Oct 13, 2009 at 1:23 PM, Robert Kern<robert.kern at gmail.com>  wrote:
>> On 2009-10-13 13:00 PM, Peng Yu wrote:
>>>
>>> I use pickle to dump a long list. But when I load it, I only want to
>>> load the first a few elements in the list. I am wondering if there is
>>> a easy way to do so? Thank you!
>>
>> Not by pickling the list. However, you can concatenate pickles, so you could
>> just pickle each item from the list in order to the same file and only
>> unpickle the first few.
>>
>> In [1]: import cPickle
>>
>> In [2]: from cStringIO import StringIO
>>
>> In [3]: very_long_list = range(10)
>>
>> In [4]: f = StringIO()
>>
>> In [5]: for item in very_long_list:
>>    ...:     cPickle.dump(item, f)
>>    ...:
>>    ...:
>>
>> In [6]: f.seek(0,0)
>>
>> In [7]: cPickle.load(f)
>> Out[7]: 0
>>
>> In [8]: cPickle.load(f)
>> Out[8]: 1
>>
>> In [9]: cPickle.load(f)
>> Out[9]: 2
>>
>> In [10]: cPickle.load(f)
>> Out[10]: 3
>
> How do I determine if I have loaded all the elements? I use the
> following code. I'm wondering if there is any better solution than
> this.
>
>
> ###############
> import pickle
>
> alist = [1, 2.0, 3, 4+6j]
>
> output=open('serialize_list.output/serialize_list.pkl', 'wb')
> for e in alist:
>    pickle.dump(e, output)
> output.close()
>
> input=open('serialize_list.output/serialize_list.pkl', 'rb')
>
> try:
>    while 1:
>      e = pickle.load(input)
>      print e
> except EOFError:
>    pass

You could write out an integer with the number of expected elements at the very 
beginning.

In [1]: import cPickle

In [2]: alist = [1, 2.0, 3, 4+6j]

In [3]: output = open('foo.pkl', 'wb')

In [4]: cPickle.dump(len(alist), output)

In [5]: for item in alist:
    ...:     cPickle.dump(item, output)
    ...:
    ...:

In [6]: output.close()

In [7]: input = open('foo.pkl', 'rb')

In [8]: n = cPickle.load(input)

In [9]: n
Out[9]: 4

In [10]: for i in range(n):
    ....:     print cPickle.load(input)
    ....:
    ....:
1
2.0
3
(4+6j)

In [11]: assert input.read(1) == ''

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco