pickle.load() on an dictionary of dictionaries doesn't load full data structure on first call

[Albert Hopkins]

On Sun, 2009-02-22 at 16:15 -0800, James Pearson wrote:
> I've been using irclib to write a simple irc bot, and I was running
> into some difficulties with pickle.  Upon some experimentation with
> pdb, I found that pickle.load() doesn't load *all* of the data the
> _first_ time it's called.
> 
> For instance, I have this dictionary pickled:
> {'xiong_chiamiov': {'name': 'James Pearson', 'distro': 'Arch'}}
> The first time I pickle.load() it, I get this:
> {'xiong_chiamiov': {'name': 'James Pearson'}}
> The 2nd time, I get the full thing.
> 
> Is this a bug in pickle, something wrong with the way I'm approaching
> this, or what?  Full code is available at:
> http://github.com/xiongchiamiov/mpu/
> 

In general it's a bad idea to point the list to a full.  It's usually
wiser to deduce the issue to a relatively small, easily runnable,
snippet of code and then paste that snippet in your post.  Readers
usually don't want to have to read through your entire program to
understand what you are asking them.

Having said that, pickle is for serializing data.  You shouldn't be
opening a file in read/write/append mode and dumping and loading to the
same file object.  It's not a multi-access database like a DBMS is.  The
basic idea is:

     1. open (empty) file write-only.
     2. dump object(s) to file.
     3. close file

or 

     1. open file read-only
     2. load object(s) from file
     3. close file

Anything other than the two may lead to undesired/undefined behavior.

But the likely answer to your question is that your pickle file has (at
least) 2 objects stored in it.  And so on each load you get subsequent
objects.  That's probably the "undesired" affect of opening the file in
append mode.