Eliminating duplicates entries from a list efficiently

[Paul Rubin]

Paul <paul at oz.net> writes:
> Can anyone suggest an efficient way to eliminate duplicate entries
> in a list?  The naive approach below works fine, but is very slow 
> with lists containing tens of thousands of entries:

I generally do it by sorting the list and then making a pass through
it moving stuff down:

    def uniq(a):
        "Remove duplicate elements from a.  a must be sorted."
        p=0
        for i in xrange(1,len(a)):
            if a[i] != a[p]:
                p=p+1
                a[p]=a[i]
        del a[p+1:]