Removing Duplicate entries in a file...

[Mike Meyer]

"sri2097" <srikar2097 at gmail.com> writes:
> Hi all, I'm storing number of dictionary values into a file using the
> 'cPickle' module and then am retrieving it. The following is the code
> for it  -
>
> # Code for storing the values in the file
> import cPickle
>
> book = {raw_input("Name: "): [int(raw_input("Phone: ")),
> raw_input("Address: ")] }
> file_object = file(database, 'w+')
> cPickle.dump(book, file_object)
> file_object.close()
>
> # Code for retrieving values and modifiing them.
>     tobe_modified_name = raw_input("Enter name to be modified: ")
>     file_object = file(database)
>
>     while file_object.tell() != EOFError:
>         try:
>             stored_dict = cPickle.load(file_object)
>             if stored_dict.has_key(tobe_modified_name):
>                 print ("Entry found !")
>             # I want to modify the values retrieved from the file and
> then put it back to the file       without duplicate entry.
>                 file_object = file(database, 'a+')
>         except EOFError:
>             break
>     file_object.close()
>
>
> Now, my problem is after finding the entry in the file, I want to make
> changes to the 'values' under the searched 'key' and then insert it
> back to the file. But in doing so I'm having duplicate entries for the
> same key. I want to remove the previous key and value entry in the file
> and key the latest one.  How to solve this problem ?

First, file_object.tell won't return EOFError. Nothing should return
EOFError - it's an exception. It should be raised.

As you noticed, cPickle.load will raise EOFError when called on a file
that you've reached the end of. However, you want to narrow the
try clause as much as possible:

      try:
          stored_dict = cPickle.load(file_object)
      except EOFError:
          break
       
       # Work with stored dict here.

If you weren't doing a break in the except clause, you'd work with the
dictionary in an else clause.

> I actually thought of 2 ways -
>
> 1) In Java there is something called 'file_pointer' concept where in
> after you find the entry you are looking for you move all the entries
> below this entry. Then you get the searched entry at the bottom of the
> file. After this truncate the file by a certain bytes to remove the old
> entry. Can we do this in Python using the file.truncate([size]) method
> ?

Yup, this would work. You'd have to save the value from
file_object.tell() before calling cPickle.load, so you could go back
to that point to write the next object. You'd either have to load all
the following objects into memory, or shuttle back and forth between
the read and write positions. The latter sounds "really crappy" to me.

> 2) Although this is a really crappy way but nevertheless I'll put it
> across. First after finding the entry you are looking for in the file,
> make a copy of this file without the entry found in the previous file.
> Make the changes to the 'values' under this key and insert this into
> the second file what you have created. Before exiting delete the first
> file.

Actually, there's a good reason for doing it that way. But first,
another alternative.

Unless your file is huge (more than a few hundred megabytes), you
might consider loading the entire thing into memory. Instead of
calling cPickle.dump multiple times, put all the dictionaries in a
list, then call cPickle.dump on the list. When you want to update the
list, cPickle.load will load the entire list, so you can use Python to
work on it.

As for saving the file, best practice for updating a file is to write
it to a temporary file, and then rename the new file to the old name
after the write has successfully finished. This way, if the write
fails for some reason, your working file isn't corrupted. Doing it
this way also makes dealing with the case of the the list being to big
load into memory easy:

# Warning, untested code

while 1:
    try:
        stored_dict = cPickle.load(input_file)
    except EOFError:
        break
    if stored_dict.has_key(tobe_modified_name):
       print "Entry found !"
       # Modify stored_dict here
    cPickle.dump(stored_dict, output_file)

output_file.close()
os.unlink(database)	# May not be required; depends on your os
os.rename(datebase_temp, database)


You'll probably want to handle exceptions from cPickle.dump and
output_file.close cleanly as well.

        <mike
-- 
Mike Meyer <mwm at mired.org>			http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.