Namespaces are one honking great idea

[Steven D'Aprano]

The Zen of Python says:

    Namespaces are one honking great idea -- let's do more of those!



Proposal
=========

Add a new "namespace" object to Python.


Rationale
==========

Sometimes we have a group of related functions and variables that belong
together, but are not sufficiently unrelated to the rest of the module that
we want to split them out into another file.

Python allows us to group related chunks of code in three levels of
grouping:

- the class, containing one of more variable and/or function (method);
- the module, containing one or more classes and/or functions;
- the package, containing one or more modules.

But classes are not like the others: they must be instantiated before they
can be used, and they are more than just a mere namespace grouping related
entities. Classes support inheritance. Classes should be used for "is-a"
relationships, not "has-a" relationships. Although classes (and instances)
are namespaces, they provide fundamentally different kind of behaviour than
modules and packages.

Sometimes we have a group of related functions which belongs together, but
they are more closely related to each other than they are to the rest of
the module, but not so distantly unrelated that they should be split off
into a separate module. Nor do they belong as a class -- there's nothing
analogous to an instance, and no need for inheritance.

Perhaps the functions have some name clashes, e.g. two different functions
that deserve the same name, or perhaps they're just conceptually related.

What is needed is a module inside a module, but without the necessity of
refactoring the code into a package and splitting the code out into a
separate file. In other words, something rather like C++ namespaces:

https://msdn.microsoft.com/en-us/library/5cb46ksf.aspx


Proof of concept
=================

Here's a proof of concept. I use a class with a custom metaclass like this:


# Python 3 version
class ExampleNS(metaclass=Namespace):
    x = 1
    y = []

    def spam(n):
        return 'spam'*n

    z = spam(3).upper()

    def ham(n):
        return spam(n).replace('sp', 'H')

    def test():
        global x
        x += 1
        y.append(1)
        return x, y


And a demonstration of its use:

py> Example.spam(5)
'spamspamspamspamspam'
py> Example.ham(5)
'HamHamHamHamHam'
py> Example.test()
(2, [1])
py> Example.test()
(3, [1, 1])
py> print(Example.x, Example.y, Example.z)
3 [1, 1] SPAMSPAMSPAM

Despite the "class" statement (a limitation of Python's lack of dedicated
syntax for namespaces), the Example namespace behaves like (in fact, *is*)
a module embedded inside a module.

Notice that the functions are not methods and don't require a "self"
parameter. Being functions, they can be used inside the body of the
namespace, including inside other functions.

Outside the namespace, access is via dotted attribute: Example.spam. But
inside it, functions can refer to each other without knowing the name of
their namespace.

Especially note that inside the namespace, the global statement makes a name
refer to the namespace scope.


Implementation
===============

Here's a simplified implementation:


from types import FunctionType, ModuleType

class NamespaceType(ModuleType):
    def __repr__(self):
        return 'namespace <%s>' % self.__name__

class Namespace(type):
    def __new__(meta, name, bases, ns):
        if bases != ():
            raise RuntimeError("namespaces cannot inherit")
        module = NamespaceType(name)
        globals_ = module.__dict__
        if '__builtins__' not in globals_:
            globals_['__builtins__'] = globals()['__builtins__']
        for name, obj in ns.items():
            if type(obj) is FunctionType:
                ns[name] = meta._copy_and_inject(obj, globals_)
        module.__dict__.update(ns)
        return module

    @classmethod
    def _copy_and_inject(meta, func, globals):
        """Return a new function object with the given globals."""
        assert type(func) is FunctionType
        f = FunctionType(
                func.__code__, globals, func.__name__,
                 func.__defaults__, func.__closure__)
        f.__dict__ = func.__dict__
        # Next two lines are Python 3 only.
        f.__kwdefaults__ = func.__kwdefaults__
        f.__annotations__ = func.__annotations__
        return f



The namespace object itself is actually a module (to be precise, a subclass
of ModuleType). The Namespace metaclass returns a new module object,
containing all the names I defined inside the class, after modifying any
function objects to treat the new module object as their globals.


Unresolved questions
=====================

Would a decorator be better than a metaclass?

    @namespace
    class Example:
        ...


How about a mechanism to exclude functions from having their globals
changed?


Some limitations
=================

The simplified implementation shown doesn't allow functions inside the
namespace to access names outside of the namespace easily. In practice, the
user might want the name resolution order inside the namespace to be:

 local variables
 nonlocal variables
 namespace globals
 module globals
 builtins


The biggest limitation is that I have to abuse the class statement to do
this. In an ideal world, there would be syntactic support and a keyword:

    namespace Example:
        x = 0
        y = []
        def test(n): ...

although others might argue that *not* needing a dedicated keyword is, in
fact, better.


Disadvantages
==============

Those unfamiliar with the namespace concept may be confused by a class that
doesn't get instantiated.


Alternatives
=============

At the moment, Python gives us four existing solutions:

(1) Split the functions into a separate file, whether they deserve it or
not. That's the smallest grouping Python supports so you're stuck with it.

The obvious disadvantage is that what is conceptually a single file must be
split into two, making more work for the developer.


(2) Don't split the functions into their own namespace, keep them in the
enclosing module namespace, but give them a "virtual namespace" by
prefixing their names:

    def foods_spam(a, b): ...

    def foods_eggs(a): ...

    def advertising_spam(a, b, c): ...

This requires the reader to see the "_" as a "virtual dot", rather than as a
word separator as underscores are normally used as. It also fails to
visually group the functions together, and requires functions to refer to
each other using their full "virtual" dotted name.


(3) Pretend you're in the Kingdom of Nouns[1]. Put the functions in a class,
turn them into methods, and create an instance just so you can use them:

    class Foods:
        def spam(self, a, b): ...
        def eggs(self, a): ...

    class Advertising:
        def spam(self, a, b, c): ...

    foods = Foods()
    advertising = Advertising()

optionally making each instance a singleton. Sometimes the class is deleted
to discourage subclassing.

This solution is often an anti-pattern, as it violates expectations about
classes. The class should not be subclassed, nor should it be instantiated
more than once. The class doesn't represent an "is-a" relationship, nor
does the instance represent a real-world object.


(4) Put them in a class, but use staticmethod (or classmethod) to avoid
needing an instance:

    class foods:
        @staticmethod
        def spam(a, b): ...
        @staticmethod
        def eggs(a): ...

    class advertising:
        @staticmethod
        def spam(a, b, c): ...


This has the disadvantage that functions in the class cannot call each other
directly, but must use the dotted names. This is the Python equivalent of
the "Java utility class" pattern, or arguably anti-pattern[2], where you
create a class with little or no state and populate it with a number of
static methods.







[1]
http://steve-yegge.blogspot.com/2006/03/execution-in-kingdom-of-nouns.html

[2]
http://www.yegor256.com/2014/05/05/oop-alternative-to-utility-classes.html

-- 
Steven