Lazy-evaluation lists/dictionaries

[Terry Reedy]

On 10/26/2014 10:14 AM, Jon Ribbens wrote:
> I have a need, in a Python C extension I am writing, for lists and
> dictionaries with "lazy evaluation" - by which I mean that at least
> some of the values in the lists/dictionaries are "proxy objects"
> which, rather than returning as themselves, should return the thing
> they are a proxy for when retrieved. This is because retrieving
> the proxied objects is expensive and only a small minority of them
> will actually be accessed, so retrieving them all before they are
> actually accessed is massively inefficient.
>
> With object attributes, this can be easily done with the descriptor
> protocol and having properties which look like simple objects but
> actually cause a method to be invoked when they are accessed. However
> there doesn't seem to be any equivalent way of doing this for
> retrieving items from lists or dictionaries - unfortunately, the
> method implementations of list() and dict() are full of direct
> accesses to the underlying data pointers rather than indirecting
> through obj->tp_as_mapping->mp_subscript or whatever.
>
> Is there any better way to do this other than simply re-implementing
> these types from scratch, emulating all their methods and operations?
> (i.e. using UserList/UserDict). I was under the impression that that
> sort of thing was supposed to have gone since Python 2.2 or so.

We considered dropping UserDict and UserList for 3.0 but kept them in 
collections for cases in which subclassing does not work.

-- 
Terry Jan Reedy