[Python-checkins] Add doctests to the descriptor HowTo (GH-23500)
rhettinger
webhook-mailer at python.org
Tue Nov 24 23:57:21 EST 2020
https://github.com/python/cpython/commit/2d44a6bc4fea742e3215101a2c4c3fb18b70a4c6
commit: 2d44a6bc4fea742e3215101a2c4c3fb18b70a4c6
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: rhettinger <rhettinger at users.noreply.github.com>
date: 2020-11-24T20:57:02-08:00
summary:
Add doctests to the descriptor HowTo (GH-23500)
files:
M Doc/howto/descriptor.rst
diff --git a/Doc/howto/descriptor.rst b/Doc/howto/descriptor.rst
index 8c6e90319a7f3..e94f0ef88416e 100644
--- a/Doc/howto/descriptor.rst
+++ b/Doc/howto/descriptor.rst
@@ -43,21 +43,26 @@ Simple example: A descriptor that returns a constant
----------------------------------------------------
The :class:`Ten` class is a descriptor that always returns the constant ``10``
-from its :meth:`__get__` method::
+from its :meth:`__get__` method:
+.. testcode::
class Ten:
def __get__(self, obj, objtype=None):
return 10
-To use the descriptor, it must be stored as a class variable in another class::
+To use the descriptor, it must be stored as a class variable in another class:
+
+.. testcode::
class A:
x = 5 # Regular class attribute
y = Ten() # Descriptor instance
An interactive session shows the difference between normal attribute lookup
-and descriptor lookup::
+and descriptor lookup:
+
+.. doctest::
>>> a = A() # Make an instance of class A
>>> a.x # Normal attribute lookup
@@ -83,7 +88,9 @@ Dynamic lookups
---------------
Interesting descriptors typically run computations instead of returning
-constants::
+constants:
+
+.. testcode::
import os
@@ -131,7 +138,9 @@ the public attribute is accessed.
In the following example, *age* is the public attribute and *_age* is the
private attribute. When the public attribute is accessed, the descriptor logs
-the lookup or update::
+the lookup or update:
+
+.. testcode::
import logging
@@ -201,7 +210,9 @@ variable name was used.
In this example, the :class:`Person` class has two descriptor instances,
*name* and *age*. When the :class:`Person` class is defined, it makes a
callback to :meth:`__set_name__` in *LoggedAccess* so that the field names can
-be recorded, giving each descriptor its own *public_name* and *private_name*::
+be recorded, giving each descriptor its own *public_name* and *private_name*:
+
+.. testcode::
import logging
@@ -236,7 +247,9 @@ be recorded, giving each descriptor its own *public_name* and *private_name*::
An interactive session shows that the :class:`Person` class has called
:meth:`__set_name__` so that the field names would be recorded. Here
-we call :func:`vars` to look up the descriptor without triggering it::
+we call :func:`vars` to look up the descriptor without triggering it:
+
+.. doctest::
>>> vars(vars(Person)['name'])
{'public_name': 'name', 'private_name': '_name'}
@@ -307,7 +320,9 @@ restrictions. If those restrictions aren't met, it raises an exception to
prevent data corruption at its source.
This :class:`Validator` class is both an :term:`abstract base class` and a
-managed attribute descriptor::
+managed attribute descriptor:
+
+.. testcode::
from abc import ABC, abstractmethod
@@ -347,7 +362,7 @@ Here are three practical data validation utilities:
user-defined `predicate
<https://en.wikipedia.org/wiki/Predicate_(mathematical_logic)>`_ as well.
-::
+.. testcode::
class OneOf(Validator):
@@ -400,10 +415,12 @@ Here are three practical data validation utilities:
)
-Practical use
--------------
+Practical application
+---------------------
+
+Here's how the data validators can be used in a real class:
-Here's how the data validators can be used in a real class::
+.. testcode::
class Component:
@@ -418,11 +435,26 @@ Here's how the data validators can be used in a real class::
The descriptors prevent invalid instances from being created::
- Component('WIDGET', 'metal', 5) # Allowed.
- Component('Widget', 'metal', 5) # Blocked: 'Widget' is not all uppercase
- Component('WIDGET', 'metle', 5) # Blocked: 'metle' is misspelled
- Component('WIDGET', 'metal', -5) # Blocked: -5 is negative
- Component('WIDGET', 'metal', 'V') # Blocked: 'V' isn't a number
+ >>> Component('Widget', 'metal', 5) # Blocked: 'Widget' is not all uppercase
+ Traceback (most recent call last):
+ ...
+ ValueError: Expected <method 'isupper' of 'str' objects> to be true for 'Widget'
+
+ >>> Component('WIDGET', 'metle', 5) # Blocked: 'metle' is misspelled
+ Traceback (most recent call last):
+ ...
+ ValueError: Expected 'metle' to be one of {'metal', 'plastic', 'wood'}
+
+ >>> Component('WIDGET', 'metal', -5) # Blocked: -5 is negative
+ Traceback (most recent call last):
+ ...
+ ValueError: Expected -5 to be at least 0
+ >>> Component('WIDGET', 'metal', 'V') # Blocked: 'V' isn't a number
+ Traceback (most recent call last):
+ ...
+ TypeError: Expected 'V' to be an int or float
+
+ >>> c = Component('WIDGET', 'metal', 5) # Allowed: The inputs are valid
Technical Tutorial
@@ -526,7 +558,9 @@ If a descriptor is found for ``a.x``, then it is invoked with:
``desc.__get__(a, type(a))``.
The logic for a dotted lookup is in :meth:`object.__getattribute__`. Here is
-a pure Python equivalent::
+a pure Python equivalent:
+
+.. testcode::
def object_getattribute(obj, name):
"Emulate PyObject_GenericGetAttr() in Objects/object.c"
@@ -546,9 +580,108 @@ a pure Python equivalent::
return cls_var # class variable
raise AttributeError(name)
+
+.. testcode::
+ :hide:
+
+ # Test the fidelity of object_getattribute() by comparing it with the
+ # normal object.__getattribute__(). The former will be accessed by
+ # square brackets and the latter by the dot operator.
+
+ class Object:
+
+ def __getitem__(obj, name):
+ try:
+ return object_getattribute(obj, name)
+ except AttributeError:
+ if not hasattr(type(obj), '__getattr__'):
+ raise
+ return type(obj).__getattr__(obj, name) # __getattr__
+
+ class DualOperator(Object):
+
+ x = 10
+
+ def __init__(self, z):
+ self.z = z
+
+ @property
+ def p2(self):
+ return 2 * self.x
+
+ @property
+ def p3(self):
+ return 3 * self.x
+
+ def m5(self, y):
+ return 5 * y
+
+ def m7(self, y):
+ return 7 * y
+
+ def __getattr__(self, name):
+ return ('getattr_hook', self, name)
+
+ class DualOperatorWithSlots:
+
+ __getitem__ = Object.__getitem__
+
+ __slots__ = ['z']
+
+ x = 15
+
+ def __init__(self, z):
+ self.z = z
+
+ @property
+ def p2(self):
+ return 2 * self.x
+
+ def m5(self, y):
+ return 5 * y
+
+ def __getattr__(self, name):
+ return ('getattr_hook', self, name)
+
+
+.. doctest::
+ :hide:
+
+ >>> a = DualOperator(11)
+ >>> vars(a).update(p3 = '_p3', m7 = '_m7')
+ >>> a.x == a['x'] == 10
+ True
+ >>> a.z == a['z'] == 11
+ True
+ >>> a.p2 == a['p2'] == 20
+ True
+ >>> a.p3 == a['p3'] == 30
+ True
+ >>> a.m5(100) == a.m5(100) == 500
+ True
+ >>> a.m7 == a['m7'] == '_m7'
+ True
+ >>> a.g == a['g'] == ('getattr_hook', a, 'g')
+ True
+
+ >>> b = DualOperatorWithSlots(22)
+ >>> b.x == b['x'] == 15
+ True
+ >>> b.z == b['z'] == 22
+ True
+ >>> b.p2 == b['p2'] == 30
+ True
+ >>> b.m5(200) == b['m5'](200) == 1000
+ True
+ >>> b.g == b['g'] == ('getattr_hook', b, 'g')
+ True
+
+
Interestingly, attribute lookup doesn't call :meth:`object.__getattribute__`
directly. Instead, both the dot operator and the :func:`getattr` function
-perform attribute lookup by way of a helper function::
+perform attribute lookup by way of a helper function:
+
+.. testcode::
def getattr_hook(obj, name):
"Emulate slot_tp_getattr_hook() in Objects/typeobject.c"
@@ -650,7 +783,9 @@ be used to implement an `object relational mapping
The essential idea is that the data is stored in an external database. The
Python instances only hold keys to the database's tables. Descriptors take
-care of lookups or updates::
+care of lookups or updates:
+
+.. testcode::
class Field:
@@ -665,8 +800,11 @@ care of lookups or updates::
conn.execute(self.store, [value, obj.key])
conn.commit()
-We can use the :class:`Field` class to define "models" that describe the schema
-for each table in a database::
+We can use the :class:`Field` class to define `models
+<https://en.wikipedia.org/wiki/Database_model>`_ that describe the schema for
+each table in a database:
+
+.. testcode::
class Movie:
table = 'Movies' # Table name
@@ -687,12 +825,41 @@ for each table in a database::
def __init__(self, key):
self.key = key
-An interactive session shows how data is retrieved from the database and how
-it can be updated::
+To use the models, first connect to the database::
>>> import sqlite3
>>> conn = sqlite3.connect('entertainment.db')
+An interactive session shows how data is retrieved from the database and how
+it can be updated:
+
+.. testsetup::
+
+ song_data = [
+ ('Country Roads', 'John Denver', 1972),
+ ('Me and Bobby McGee', 'Janice Joplin', 1971),
+ ('Coal Miners Daughter', 'Loretta Lynn', 1970),
+ ]
+
+ movie_data = [
+ ('Star Wars', 'George Lucas', 1977),
+ ('Jaws', 'Steven Spielberg', 1975),
+ ('Aliens', 'James Cameron', 1986),
+ ]
+
+ import sqlite3
+
+ conn = sqlite3.connect(':memory:')
+ conn.execute('CREATE TABLE Music (title text, artist text, year integer);')
+ conn.execute('CREATE INDEX MusicNdx ON Music (title);')
+ conn.executemany('INSERT INTO Music VALUES (?, ?, ?);', song_data)
+ conn.execute('CREATE TABLE Movies (title text, director text, year integer);')
+ conn.execute('CREATE INDEX MovieNdx ON Music (title);')
+ conn.executemany('INSERT INTO Movies VALUES (?, ?, ?);', movie_data)
+ conn.commit()
+
+.. doctest::
+
>>> Movie('Star Wars').director
'George Lucas'
>>> jaws = Movie('Jaws')
@@ -724,7 +891,9 @@ triggers a function call upon access to an attribute. Its signature is::
property(fget=None, fset=None, fdel=None, doc=None) -> property
-The documentation shows a typical use to define a managed attribute ``x``::
+The documentation shows a typical use to define a managed attribute ``x``:
+
+.. testcode::
class C:
def getx(self): return self.__x
@@ -733,7 +902,9 @@ The documentation shows a typical use to define a managed attribute ``x``::
x = property(getx, setx, delx, "I'm the 'x' property.")
To see how :func:`property` is implemented in terms of the descriptor protocol,
-here is a pure Python equivalent::
+here is a pure Python equivalent:
+
+.. testcode::
class Property:
"Emulate PyProperty_Type() in Objects/descrobject.c"
@@ -772,6 +943,57 @@ here is a pure Python equivalent::
def deleter(self, fdel):
return type(self)(self.fget, self.fset, fdel, self.__doc__)
+.. testcode::
+ :hide:
+
+ # Verify the Property() emulation
+
+ class CC:
+ def getx(self):
+ return self.__x
+ def setx(self, value):
+ self.__x = value
+ def delx(self):
+ del self.__x
+ x = Property(getx, setx, delx, "I'm the 'x' property.")
+
+ # Now do it again but use the decorator style
+
+ class CCC:
+ @Property
+ def x(self):
+ return self.__x
+ @x.setter
+ def x(self, value):
+ self.__x = value
+ @x.deleter
+ def x(self):
+ del self.__x
+
+
+.. doctest::
+ :hide:
+
+ >>> cc = CC()
+ >>> hasattr(cc, 'x')
+ False
+ >>> cc.x = 33
+ >>> cc.x
+ 33
+ >>> del cc.x
+ >>> hasattr(cc, 'x')
+ False
+
+ >>> ccc = CCC()
+ >>> hasattr(ccc, 'x')
+ False
+ >>> ccc.x = 333
+ >>> ccc.x == 333
+ True
+ >>> del ccc.x
+ >>> hasattr(ccc, 'x')
+ False
+
The :func:`property` builtin helps whenever a user interface has granted
attribute access and then subsequent changes require the intervention of a
method.
@@ -780,7 +1002,9 @@ For instance, a spreadsheet class may grant access to a cell value through
``Cell('b10').value``. Subsequent improvements to the program require the cell
to be recalculated on every access; however, the programmer does not want to
affect existing client code accessing the attribute directly. The solution is
-to wrap access to the value attribute in a property data descriptor::
+to wrap access to the value attribute in a property data descriptor:
+
+.. testcode::
class Cell:
...
@@ -791,6 +1015,9 @@ to wrap access to the value attribute in a property data descriptor::
self.recalc()
return self._value
+Either the built-in :func:`property` or our :func:`Property` equivalent would
+work in this example.
+
Functions and methods
---------------------
@@ -804,7 +1031,9 @@ prepended to the other arguments. By convention, the instance is called
*self* but could be called *this* or any other variable name.
Methods can be created manually with :class:`types.MethodType` which is
-roughly equivalent to::
+roughly equivalent to:
+
+.. testcode::
class MethodType:
"Emulate Py_MethodType in Objects/classobject.c"
@@ -821,7 +1050,9 @@ roughly equivalent to::
To support automatic creation of methods, functions include the
:meth:`__get__` method for binding methods during attribute access. This
means that functions are non-data descriptors that return bound methods
-during dotted lookup from an instance. Here's how it works::
+during dotted lookup from an instance. Here's how it works:
+
+.. testcode::
class Function:
...
@@ -833,13 +1064,17 @@ during dotted lookup from an instance. Here's how it works::
return MethodType(self, obj)
Running the following class in the interpreter shows how the function
-descriptor works in practice::
+descriptor works in practice:
+
+.. testcode::
class D:
def f(self, x):
return x
-The function has a :term:`qualified name` attribute to support introspection::
+The function has a :term:`qualified name` attribute to support introspection:
+
+.. doctest::
>>> D.f.__qualname__
'D.f'
@@ -867,7 +1102,7 @@ Internally, the bound method stores the underlying function and the bound
instance::
>>> d.f.__func__
- <function D.f at 0x1012e5ae8>
+ <function D.f at 0x00C45070>
>>> d.f.__self__
<__main__.D object at 0x1012e1f98>
@@ -919,20 +1154,26 @@ It can be called either from an object or the class: ``s.erf(1.5) --> .9332`` o
``Sample.erf(1.5) --> .9332``.
Since static methods return the underlying function with no changes, the
-example calls are unexciting::
+example calls are unexciting:
+
+.. testcode::
class E:
@staticmethod
def f(x):
print(x)
+.. doctest::
+
>>> E.f(3)
3
>>> E().f(3)
3
Using the non-data descriptor protocol, a pure Python version of
-:func:`staticmethod` would look like this::
+:func:`staticmethod` would look like this:
+
+.. doctest::
class StaticMethod:
"Emulate PyStaticMethod_Type() in Objects/funcobject.c"
@@ -949,27 +1190,31 @@ Class methods
Unlike static methods, class methods prepend the class reference to the
argument list before calling the function. This format is the same
-for whether the caller is an object or a class::
+for whether the caller is an object or a class:
+
+.. testcode::
class F:
@classmethod
def f(cls, x):
return cls.__name__, x
- >>> print(F.f(3))
+.. doctest::
+
+ >>> F.f(3)
('F', 3)
- >>> print(F().f(3))
+ >>> F().f(3)
('F', 3)
This behavior is useful whenever the method only needs to have a class
reference and does rely on data stored in a specific instance. One use for
class methods is to create alternate class constructors. For example, the
classmethod :func:`dict.fromkeys` creates a new dictionary from a list of
-keys. The pure Python equivalent is::
+keys. The pure Python equivalent is:
- class Dict:
- ...
+.. testcode::
+ class Dict(dict):
@classmethod
def fromkeys(cls, iterable, value=None):
"Emulate dict_fromkeys() in Objects/dictobject.c"
@@ -978,13 +1223,17 @@ keys. The pure Python equivalent is::
d[key] = value
return d
-Now a new dictionary of unique keys can be constructed like this::
+Now a new dictionary of unique keys can be constructed like this:
+
+.. doctest::
>>> Dict.fromkeys('abracadabra')
- {'a': None, 'r': None, 'b': None, 'c': None, 'd': None}
+ {'a': None, 'b': None, 'r': None, 'c': None, 'd': None}
Using the non-data descriptor protocol, a pure Python version of
-:func:`classmethod` would look like this::
+:func:`classmethod` would look like this:
+
+.. testcode::
class ClassMethod:
"Emulate PyClassMethod_Type() in Objects/funcobject.c"
@@ -999,9 +1248,31 @@ Using the non-data descriptor protocol, a pure Python version of
return self.f.__get__(cls)
return MethodType(self.f, cls)
+.. testcode::
+ :hide:
+
+ # Verify the emulation works
+ class T:
+ @ClassMethod
+ def cm(cls, x, y):
+ return (cls, x, y)
+
+.. doctest::
+ :hide:
+
+ >>> T.cm(11, 22)
+ (<class 'T'>, 11, 22)
+
+ # Also call it from an instance
+ >>> t = T()
+ >>> t.cm(11, 22)
+ (<class 'T'>, 11, 22)
+
The code path for ``hasattr(obj, '__get__')`` was added in Python 3.9 and
makes it possible for :func:`classmethod` to support chained decorators.
-For example, a classmethod and property could be chained together::
+For example, a classmethod and property could be chained together:
+
+.. testcode::
class G:
@classmethod
@@ -1009,6 +1280,12 @@ For example, a classmethod and property could be chained together::
def __doc__(cls):
return f'A doc for {cls.__name__!r}'
+.. doctest::
+
+ >>> G.__doc__
+ "A doc for 'G'"
+
+
Member objects and __slots__
----------------------------
@@ -1017,11 +1294,15 @@ fixed-length array of slot values. From a user point of view that has
several effects:
1. Provides immediate detection of bugs due to misspelled attribute
-assignments. Only attribute names specified in ``__slots__`` are allowed::
+assignments. Only attribute names specified in ``__slots__`` are allowed:
+
+.. testcode::
class Vehicle:
__slots__ = ('id_number', 'make', 'model')
+.. doctest::
+
>>> auto = Vehicle()
>>> auto.id_nubmer = 'VYE483814LQEX'
Traceback (most recent call last):
@@ -1029,7 +1310,9 @@ assignments. Only attribute names specified in ``__slots__`` are allowed::
AttributeError: 'Vehicle' object has no attribute 'id_nubmer'
2. Helps create immutable objects where descriptors manage access to private
-attributes stored in ``__slots__``::
+attributes stored in ``__slots__``:
+
+.. testcode::
class Immutable:
@@ -1047,7 +1330,19 @@ attributes stored in ``__slots__``::
def name(self): # Read-only descriptor
return self._name
- mark = Immutable('Botany', 'Mark Watney') # Create an immutable instance
+.. doctest::
+
+ >>> mark = Immutable('Botany', 'Mark Watney')
+ >>> mark.dept
+ 'Botany'
+ >>> mark.dept = 'Space Pirate'
+ Traceback (most recent call last):
+ ...
+ AttributeError: can't set attribute
+ >>> mark.location = 'Mars'
+ Traceback (most recent call last):
+ ...
+ AttributeError: 'Immutable' object has no attribute 'location'
3. Saves memory. On a 64-bit Linux build, an instance with two attributes
takes 48 bytes with ``__slots__`` and 152 bytes without. This `flyweight
@@ -1055,7 +1350,9 @@ design pattern <https://en.wikipedia.org/wiki/Flyweight_pattern>`_ likely only
matters when a large number of instances are going to be created.
4. Blocks tools like :func:`functools.cached_property` which require an
-instance dictionary to function correctly::
+instance dictionary to function correctly:
+
+.. testcode::
from functools import cached_property
@@ -1067,17 +1364,21 @@ instance dictionary to function correctly::
return 4 * sum((-1.0)**n / (2.0*n + 1.0)
for n in reversed(range(100_000)))
+.. doctest::
+
>>> CP().pi
Traceback (most recent call last):
...
TypeError: No '__dict__' attribute on 'CP' instance to cache 'pi' property.
-It's not possible to create an exact drop-in pure Python version of
+It is not possible to create an exact drop-in pure Python version of
``__slots__`` because it requires direct access to C structures and control
over object memory allocation. However, we can build a mostly faithful
simulation where the actual C structure for slots is emulated by a private
``_slotvalues`` list. Reads and writes to that private structure are managed
-by member descriptors::
+by member descriptors:
+
+.. testcode::
null = object()
@@ -1114,7 +1415,9 @@ by member descriptors::
return f'<Member {self.name!r} of {self.clsname!r}>'
The :meth:`type.__new__` method takes care of adding member objects to class
-variables::
+variables:
+
+.. testcode::
class Type(type):
'Simulate how the type metaclass adds member objects for slots'
@@ -1129,7 +1432,9 @@ variables::
The :meth:`object.__new__` method takes care of creating instances that have
slots instead of an instance dictionary. Here is a rough simulation in pure
-Python::
+Python:
+
+.. testcode::
class Object:
'Simulate how object.__new__() allocates memory for __slots__'
@@ -1161,7 +1466,9 @@ Python::
super().__delattr__(name)
To use the simulation in a real class, just inherit from :class:`Object` and
-set the :term:`metaclass` to :class:`Type`::
+set the :term:`metaclass` to :class:`Type`:
+
+.. testcode::
class H(Object, metaclass=Type):
'Instance variables stored in slots'
@@ -1174,8 +1481,8 @@ set the :term:`metaclass` to :class:`Type`::
At this point, the metaclass has loaded member objects for *x* and *y*::
- >>> import pprint
- >>> pprint.pp(dict(vars(H)))
+ >>> from pprint import pp
+ >>> pp(dict(vars(H)))
{'__module__': '__main__',
'__doc__': 'Instance variables stored in slots',
'slot_names': ['x', 'y'],
@@ -1183,8 +1490,20 @@ At this point, the metaclass has loaded member objects for *x* and *y*::
'x': <Member 'x' of 'H'>,
'y': <Member 'y' of 'H'>}
+.. doctest::
+ :hide:
+
+ # We test this separately because the preceding section is not
+ # doctestable due to the hex memory address for the __init__ function
+ >>> isinstance(vars(H)['x'], Member)
+ True
+ >>> isinstance(vars(H)['y'], Member)
+ True
+
When instances are created, they have a ``slot_values`` list where the
-attributes are stored::
+attributes are stored:
+
+.. doctest::
>>> h = H(10, 20)
>>> vars(h)
@@ -1193,9 +1512,30 @@ attributes are stored::
>>> vars(h)
{'_slotvalues': [55, 20]}
-Misspelled or unassigned attributes will raise an exception::
+Misspelled or unassigned attributes will raise an exception:
+
+.. doctest::
>>> h.xz
Traceback (most recent call last):
...
AttributeError: 'H' object has no attribute 'xz'
+
+.. doctest::
+ :hide:
+
+ # Examples for deleted attributes are not shown because this section
+ # is already a bit lengthy. We still test that code here.
+ >>> del h.x
+ >>> hasattr(h, 'x')
+ False
+
+ # Also test the code for uninitialized slots
+ >>> class HU(Object, metaclass=Type):
+ ... slot_names = ['x', 'y']
+ ...
+ >>> hu = HU()
+ >>> hasattr(hu, 'x')
+ False
+ >>> hasattr(hu, 'y')
+ False
More information about the Python-checkins
mailing list