[Python-Dev] PEP 520: Preserving Class Attribute Definition Order (round 4)

Eric Snow ericsnowcurrently at gmail.com
Mon Jun 20 22:17:04 EDT 2016 

I've updated PEP 520 to reflect a clearer focus on the definition
order and less emphasis on OrderedDict.

-eric

=======================================


PEP: 520
Title: Preserving Class Attribute Definition Order
Version: $Revision$
Last-Modified: $Date$
Author: Eric Snow <ericsnowcurrently at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 7-Jun-2016
Python-Version: 3.6
Post-History: 7-Jun-2016, 11-Jun-2016, 20-Jun-2016


Abstract
========

When a class is defined using a ``class`` statement, the class body is
executed within a namespace.  After the execution completes, that
namespace is copied into new ``dict`` and the original definition
namespace is discarded.  The new copy is stored away as the class's
namespace and is exposed as ``__dict__`` through a read-only proxy.

This PEP preserves the order in which the attributes in the definition
namespace were added to it, before that namespace is discarded.  This
means it reflects the definition order of the class body.  That order
will now be preserved in the ``__definition_order__`` attribute of the
class.  This allows introspection of the original definition order,
e.g. by class decorators.

Additionally, this PEP changes the default class definition namespace
to ``OrderedDict``.  The long-lived class namespace (``__dict__``) will
remain a ``dict``.


Motivation
==========

Currently Python does not preserve the order in which attributes are
added to the class definition namespace. The namespace used during
execution of a class body defaults to ``dict``.  If the metaclass
defines ``__prepare__()`` then the result of calling it is used.  Thus,
before this PEP, to access your class definition namespace you must
use ``OrderedDict`` along with a metaclass. Then you must preserve the
definition order (from the ``OrderedDict``) yourself.  This has a
couple of problems.

First, it requires the use of a metaclass.  Metaclasses introduce an
extra level of complexity to code and in some cases (e.g. conflicts)
are a problem.  So reducing the need for them is worth doing when the
opportunity presents itself.  PEP 422 and PEP 487 discuss this at
length.  Given that we now have a C implementation of ``OrderedDict``
and that ``OrderedDict`` is the common use case for ``__prepare__()``,
we have such an opportunity by defaulting to ``OrderedDict``.

Second, only classes that opt in to using the ``OrderedDict``-based
metaclass will have access to the definition order. This is problematic
for cases where universal access to the definition order is important.
One of the original motivating use cases for this PEP is generic class
decorators that make use of the definition order.


Specification
=============

Part 1:

* the order in which class attributes are defined is preserved in the
  new ``__definition_order__`` attribute on each class
* "dunder" attributes (e.g. ``__init__``, ``__module__``) are ignored
* ``__definition_order__`` is a ``tuple`` (or ``None``)
* ``__definition_order__`` is a read-only attribute
* ``__definition_order__`` is always set:

  1. if ``__definition_order__`` is defined in the class body then it
     must be a ``tuple`` of identifiers or ``None``; any other value
     will result in ``TypeError``
  2. classes that do not have a class definition (e.g. builtins) have
     their ``__definition_order__`` set to ``None``
  3. classes for which `__prepare__()`` returned something other than
     ``OrderedDict`` (or a subclass) have their ``__definition_order__``
     set to ``None`` (except where #1 applies)

Part 2:

* the default class *definition* namespace is now ``OrderdDict``

The following code demonstrates roughly equivalent semantics for the
default behavior::

   class Meta(type):
       @classmethod
       def __prepare__(cls, *args, **kwargs):
           return OrderedDict()

   class Spam(metaclass=Meta):
       ham = None
       eggs = 5
       __definition_order__ = tuple(k for k in locals()
                                    if not (k.startswith('__') and
                                            k.endswith('__')))

Note that [pep487_] proposes a similar solution, albeit as part of a
broader proposal.

Why a tuple?
------------

Use of a tuple reflects the fact that we are exposing the order in
which attributes on the class were *defined*.  Since the definition
is already complete by the time ``__definition_order__`` is set, the
content and order of the value won't be changing.  Thus we use a type
that communicates that state of immutability.

Why a read-only attribute?
--------------------------

As with the use of tuple, making ``__definition_order__`` a read-only
attribute communicates the fact that the information it represents is
complete.  Since it represents the state of a particular one-time event
(execution of the class definition body), allowing the value to be
replaced would reduce confidence that the attribute corresponds to the
original class body.

If a use case for a writable (or mutable) ``__definition_order__``
arises, the restriction may be loosened later.  Presently this seems
unlikely and furthermore it is usually best to go immutable-by-default.

Note that ``__definition_order__`` is centered on the class definition
body.  The use cases for dealing with the class namespace (``__dict__``)
post-definition are a separate matter.  ``__definition_order__`` would
be a significantly misleading name for a feature focused on more than
class definition.

See [nick_concern_] for more discussion.

Why ignore "dunder" names?
--------------------------

Names starting and ending with "__" are reserved for use by the
interpreter.  In practice they should not be relevant to the users of
``__definition_order__``.  Instead, for nearly everyone they would only
be clutter, causing the same extra work for everyone.

Why None instead of an empty tuple?
-----------------------------------

A key objective of adding ``__definition_order__`` is to preserve
information in class definitions which was lost prior to this PEP.
One consequence is that ``__definition_order__`` implies an original
class definition.  Using ``None`` allows us to clearly distinquish
classes that do not have a definition order.  An empty tuple clearly
indicates a class that came from a definition statement but did not
define any attributes there.

Why None instead of not setting the attribute?
----------------------------------------------

The absence of an attribute requires more complex handling than ``None``
does for consumers of ``__definition_order__``.

Why constrain manually set values?
----------------------------------

If ``__definition_order__`` is manually set in the class body then it
will be used.  We require it to be a tuple of identifiers (or ``None``)
so that consumers of ``__definition_order__`` may have a consistent
expectation for the value.  That helps maximize the feature's
usefulness.

We could also also allow an arbitrary iterable for a manually set
``__definition_order__`` and convert it into a tuple.  However, not
all iterables infer a definition order (e.g. ``set``).  So we opt in
favor of requiring a tuple.

Why is __definition_order__ even necessary?
-------------------------------------------

Since the definition order is not preserved in ``__dict__``, it is
lost once class definition execution completes.  Classes *could*
explicitly set the attribute as the last thing in the body.  However,
then independent decorators could only make use of classes that had done
so.  Instead, ``__definition_order__`` preserves this one bit of info
from the class body so that it is universally available.


Support for C-API Types
=======================

Arguably, most C-defined Python types (e.g. built-in, extension modules)
have a roughly equivalent concept of a definition order. So conceivably
``__definition_order__`` could be set for such types automatically. This
PEP does not introduce any such support. However, it does not prohibit
it either.


Compatibility
=============

This PEP does not break backward compatibility, except in the case that
someone relies *strictly* on ``dict`` as the class definition namespace.
This shouldn't be a problem since ``issubclass(OrderedDict, dict)`` is
true.


Changes
=============

In addition to the class syntax, the following expose the new behavior:

* builtins.__build_class__
* types.prepare_class
* types.new_class


Other Python Implementations
============================

Pending feedback, the impact on Python implementations is expected to
be minimal.  If a Python implementation cannot support switching to
`OrderedDict``-by-default then it can always set ``__definition_order__``
to ``None``.


Implementation
==============

The implementation is found in the tracker. [impl_]


Alternatives
============

cls.__dict__ as OrderedDict
-------------------------------

Instead of storing the definition order in ``__definition_order__``,
the now-ordered definition namespace could be copied into a new
``OrderedDict``.  This would then be used as the mapping proxied as
``__dict__``.  Doing so would mostly provide the same semantics.

However, using ``OrderedDict`` for ``__dict__`` would obscure the
relationship with the definition namespace, making it less useful.
Additionally, doing this would require significant changes to the
semantics of the concrete ``dict`` C-API.

A "namespace" Keyword Arg for Class Definition
----------------------------------------------

PEP 422 introduced a new "namespace" keyword arg to class definitions
that effectively replaces the need to ``__prepare__()``. [pep422_]
However, the proposal was withdrawn in favor of the simpler PEP 487.

A stdlib Metaclass that Implements __prepare__() with OrderedDict
-----------------------------------------------------------------

This has all the same problems as writing your own metaclass.  The
only advantage is that you don't have to actually write this
metaclass.  So it doesn't offer any benefit in the context of this
PEP.

Set __definition_order__ at Compile-time
----------------------------------------

Each class's ``__qualname__`` is determined at compile-time.
This same concept could be applied to ``__definition_order__``.
The result of composing ``__definition_order__`` at compile-time
would be nearly the same as doing so at run-time.

Comparative implementation difficulty aside, the key difference
would be that at compile-time it would not be practical to
preserve definition order for attributes that are set dynamically
in the class body (e.g. ``locals()[name] = value``).  However,
they should still be reflected in the definition order.  One
posible resolution would be to require class authors to manually
set ``__definition_order__`` if they define any class attributes
dynamically.

Ultimately, the use of ``OrderedDict`` at run-time or compile-time
discovery is almost entirely an implementation detail.


References
==========

.. [impl] issue #24254
   (https://bugs.python.org/issue24254)

.. [nick_concern] Nick's concerns about mutability
   (https://mail.python.org/pipermail/python-dev/2016-June/144883.html)

.. [pep422] PEP 422
   (https://www.python.org/dev/peps/pep-0422/#order-preserving-classes)

.. [pep487] PEP 487
   (https://www.python.org/dev/peps/pep-0487/#defining-arbitrary-namespaces)

.. [orig] original discussion
   (https://mail.python.org/pipermail/python-ideas/2013-February/019690.html)

.. [followup1] follow-up 1
   (https://mail.python.org/pipermail/python-dev/2013-June/127103.html)

.. [followup2] follow-up 2
   (https://mail.python.org/pipermail/python-dev/2015-May/140137.html)


Copyright
===========
This document has been placed in the public domain.

More information about the Python-Dev mailing list