Notice: While Javascript is not essential for this website, your interaction with the content will be limited. Please turn Javascript on for the full experience.

PEP 499 -- python -m foo should bind sys.modules['foo'] in addition to sys.modules['__main__']

PEP:499
Title:python -m foo should bind sys.modules['foo'] in addition to sys.modules['__main__']
Author:Cameron Simpson <cs at zip.com.au>
Status:Draft
Type:Standards Track
Created:07-Aug-2015
Python-Version:3.6

Abstract

When a module is used as a main program on the Python command line, such as by:

python -m module.name ...

it is easy to accidentally end up with two independent instances of the module if that module is again imported within the program. This PEP proposes a way to fix this problem.

When a module is invoked via Python's -m option the module is bound to sys.modules['__main__'] and its .__name__ attribute is set to '__main__'. This enables the standard "main program" boilerplate code at the bottom of many modules, such as:

if __name__ == '__main__':
    sys.exit(main(sys.argv))

However, when the above command line invocation is used it is a natural inference to presume that the module is actually imported under its official name module.name, and therefore that if the program again imports that name then it will obtain the same module instance.

That actuality is that the module was imported only as '__main__'. Another import will obtain a distinct module instance, which can lead to confusing bugs.

Proposal

It is suggested that to fix this situation all that is needed is a simple change to the way the -m option is implemented: in addition to binding the module object to sys.modules['__main__'], it is also bound to sys.modules['module.name'].

Nick Coghlan has suggested that this is as simple as modifying the runpy module's _run_module_as_main function as follows:

main_globals = sys.modules["__main__"].__dict__

to instead be:

main_module = sys.modules["__main__"]
sys.modules[mod_spec.name] = main_module
main_globals = main_module.__dict__

Considerations and Prerequisites

Pickling Modules

Nick has mentioned issue 19702 [1] which proposes (quoted from the issue):

  • runpy will ensure that when __main__ is executed via the import system, it will also be aliased in sys.modules as __spec__.name
  • if __main__.__spec__ is set, pickle will use __spec__.name rather than __name__ to pickle classes, functions and methods defined in __main__
  • multiprocessing is updated appropriately to skip creating __mp_main__ in child processes when __main__.__spec__ is set in the parent process

The first point above covers this PEP's specific proposal.

Background

I tripped over this issue [2] while debugging a main program via a module which tried to monkey patch a named module, that being the main program module. Naturally, the monkey patching was ineffective as it imported the main module by name and thus patched the second module instance, not the running module instance.

However, the problem has been around as long as the -m command line option and is encountered regularly, if infrequently, by others.

In addition to issue 19702 [1], the discrepancy around __main__ is alluded to in PEP 451 and a similar proposal (predating PEP 451) is described in PEP 395 under Fixing dual imports of the main module [3].

Source: https://github.com/python/peps/blob/master/pep-0499.txt