using __getitem()__ correctly

[Steven D'Aprano]

Hmmm, you seem to be pasting in text from multiple messages, and jumping
around in time ("Ian had said, to which I answered") which may get a bit
confusing. Hopefully I can answer without getting lost :-)


On Thu, 31 Dec 2015 10:17 pm, Charles T. Smith wrote:

> On Thu, 31 Dec 2015 10:50:53 +1100, Steven D'Aprano wrote:
> 
>> I'm not sure what distinction you're referring to, can you explain?

I'm pretty sure I was directing that question to Ben, who has explained his
position.


> Ian Kelly had said:
> 
>>> How precisely are you trying to store these: as an attribute, or as a
>>> dict item? If it's supposed to be in the dict, then why is your
>>> __getitem__ trying to look up an attribute to begin with?
> 
> to which I answered:
> 
>>> In any case, I thought that class attributes were, in fact, items of
>>>__dict__?

Ah, but *which* __dict__? There could be many.

Let's put aside some of the technical details which add complexity, and
consider just a simple case: we have an instance "x" of some class X, and
we write "x.spam" to look up the attribute "spam". What happens?

This is equivalent to the function call getattr(x, "spam"), which looks for
an instance attribute, then a class attribute, then in each of the
superclasses (if any). What does getattr do?

We can imagine that it looks something like this simplified version:

def getattr(obj, name):
    try:
        return obj.__dict__[name]
    except KeyError:
        for cls in type(obj).__mro__:
            try:
                return cls.__dict__[name]
            except KeyError:
                pass
        # Still here?
        raise AttributeError('not found')


I've skipped a lot of detail -- calling of __getattribute__ and __getattr__
methods, the possibility of __slots__, the possibility that the instance
doesn't have a __dict__, descriptors, the influence of the metaclass,
differences between classic and new-style classes -- but the above shows
the essentials: an attribute lookup may look in multiple dicts to find the
attribute.

The astute reader will notice that inside getattr() I'm doing an attribute
lookup "obj.__dict__". How does that not trigger an endless series of
recursive calls to getattr?

The answer is that a handful of special attributes, including __dict__, are
built into the structure of the object itself, not part of the __dict__,
and so the Python interpreter can find them without looking in the dict.
Don't worry about it -- it's not relevant except in the senses:

(1) the problem of endless recursive calls to getattr() is solved; and
(2) objects can have attributes which aren't actually stored in a dict, 
    such as __dict__ itself, and __slots__, and probably others.

What those attributes are is, I think, an implementation detail and
irrelevant. So long as you use the existing mechanisms for doing lookups:

x.spam
getattr(x, "spam")

it will just work.


I wrote to Ben:
>> Obviously there is a syntax difference between x.attr and x['key'], but
>> attributes *are* items in a dictionary (ignoring __slots__ and
>> __getattr__ for the time being). Either the instance __dict__, the class
>> __dict__, or a superclass __dict__.
> 
> 
> That was my understanding but I wasn't sure what Ian meant when he went on
> to say:
> 
>>> If it's supposed to be in the dict, then why is your __getitem__
>>> trying to look up an attribute to begin with?
> 
> Which raises the question of where they would be if not in a dictionary.

They could be created on the fly by __getattr__ or __getattribute__, they
could be in a __slot__, or they could be a special attribute built into the
object structure. But I don't think that is relevant to Ian's question.


> This brings up a fundamental unclarity of mine: an object has attributes,
> one of which is __dict__, which has attributes.
> 
> - how does one access the attributes in e.g. self

self.attribute_name
getattr(self, "attribute_name")


> - where do I find the attribute 'mcc' by cruising around in pdb, given
>   the objs below?

What is the object "self" below? 

What makes you think that 'mcc' is an attribute? The evidence suggests that
it is not. You write:

dir(self)

and the result does *not* include 'mcc', which is very strong evidence
that 'mcc' is not an attribute.



> (PDB)pp dir (self)
[...]
>  'clear',
>  'copy',
>  'fromkeys',
>  'get',
>  'has_key',

etc. This suggests that "self" is a dict, or specifically a subclass of
dict. So you would have:

class MyDict(dict):
    ...


x = MyDict()
x.some_method()

and then dropped into the debugger. Now you're looking at the instance x
from inside one of the methods, where it is known as "self". Being a
(subclass of a) dict, it has all the usual dict attributes, like methods
clear, copy, etc., plus whatever extra attributes you give it. There is no
evidence that you have given it any extra attributes.


> (PDB)pp dir (self.__dict__)
[...]
>  'clear',
>  'copy',
>  'fromkeys',
>  'get',
>  'has_key',
etc.

Being an instance, x will usually have an instance dict. (There are
exceptions, but this is not one of those cases.) You're looking at the
attributes of that dict. Being a dict, it has all the usual dict
attributes, like methods clear, copy, etc. 


> (PDB)pp (self.keys())
> ['mcc']

self is a dict, and dicts have keys:values. This particular dict has a
single key, 'mcc'. That's not an attribute. What makes you think it is an
attribute?


> It was recommended that I use the obj[name] syntax in __getattr__()
> but that then invoked __getitem__(), complicating the matter.

It was recommended by whom? For what purpose? There's something which either
you haven't told us, or I've missed. What are you trying to accomplish?


> To be specific, if an attribute is not available, I want to assume
> it's a graph node and create it.  If the attribute has an index,
> I want to create and access an array of those nodes.

What's a graph node? Create it where? What do you mean by "attribute has an
index"? Where are you creating "an array of those nodes"?

What does any of this business got to do with self.__dict__?


> It seems to fall perfectly within the definition of __getattr__()
> and __getitem__(), but I suspect that slight programming errors
> (i.e. mine) are hiding the proper functionality of these methods.

I think you have gotten lost in the details and are over-complicating
matters greatly. Can you start with a HIGH LEVEL overview of what this
class is supposed to represent? It seems to be a subclass of dict. Why?




-- 
Steven