[Python-checkins] r70256 - in python/branches/py3k: Doc/library/importlib.rst Lib/importlib/NOTES Lib/importlib/_bootstrap.py Lib/importlib/abc.py Lib/importlib/test/abc.py Lib/importlib/test/source/test_abc_loader.py Lib/importlib/test/source/test_file_loader.py Lib/importlib/test/source/test_loader.py Lib/importlib/test/source/util.py Lib/importlib/test/test_abc.py
brett.cannon
python-checkins at python.org
Mon Mar 9 04:35:51 CET 2009
Author: brett.cannon
Date: Mon Mar 9 04:35:50 2009
New Revision: 70256
Log:
Introduce importlib.abc. The module contains various ABCs related to imports
(mostly stuff specified by PEP 302). There are two ABCs, PyLoader and
PyPycLoader, which help with implementing source and source/bytecode loaders by
implementing load_module in terms of other methods. This removes a lot of
gritty details loaders typically have to worry about.
Added:
python/branches/py3k/Lib/importlib/abc.py
python/branches/py3k/Lib/importlib/test/source/test_abc_loader.py
python/branches/py3k/Lib/importlib/test/source/test_file_loader.py
- copied, changed from r70247, /python/branches/py3k/Lib/importlib/test/source/test_loader.py
python/branches/py3k/Lib/importlib/test/test_abc.py
Removed:
python/branches/py3k/Lib/importlib/test/source/test_loader.py
Modified:
python/branches/py3k/Doc/library/importlib.rst
python/branches/py3k/Lib/importlib/NOTES
python/branches/py3k/Lib/importlib/_bootstrap.py
python/branches/py3k/Lib/importlib/test/abc.py
python/branches/py3k/Lib/importlib/test/source/util.py
Modified: python/branches/py3k/Doc/library/importlib.rst
==============================================================================
--- python/branches/py3k/Doc/library/importlib.rst (original)
+++ python/branches/py3k/Doc/library/importlib.rst Mon Mar 9 04:35:50 2009
@@ -82,6 +82,175 @@
occuring from to already be imported (i.e., *package* must already be
imported).
+:mod:`importlib.abc` -- Abstract base classes related to import
+---------------------------------------------------------------
+
+.. module:: importlib.abc
+ :synopsis: Abstract base classes related to import
+
+The :mod:`importlib.abc` module contains all of the core abstract base classes
+used by :keyword:`import`. Some subclasses of the core abstract base classes
+are also provided to help in implementing the core ABCs.
+
+
+.. class:: Finder
+
+ An abstract base class representing a :term:`finder`.
+
+ ..method:: find_module(fullname, path=None)
+
+ An abstract method for finding a :term:`loader` for the specified
+ module. If the :term:`finder` is found on :data:`sys.meta_path` and the
+ module to be searched for is a subpackage or module then *path* is set
+ to the value of :attr:`__path__` from the parent package. If a loader
+ cannot be found, :keyword:`None` is returned.
+
+ The exact definition of a :term:`finder` can be found in :pep:`302`.
+
+
+.. class:: Loader
+
+ An abstract base class for a :term:`loader`.
+
+ ..method:: load_module(fullname)
+
+ An abstract method for loading a module. If the module cannot be
+ loaded, :exc:`ImportError` is raised, otherwise the loaded module is
+ returned.
+
+ If the requested module is already exists in :data:`sys.modules`, that
+ module should be used and reloaded.
+ Otherwise a new module is to be created by the loader and inserted into
+ :data:`sys.modules`before any loading begins to prevent recursion from
+ the import. If the loader inserted into a module and the load fails it
+ must be removed by the loader from :data:`sys.modules`; modules already
+ in :data:`sys.modules` before the loader began execution should be left
+ alone. The :func:`importlib.util.module_for_loader` decorator handles
+ all of these details.
+
+ The loader is expected to set several attributes on the module when
+ adding a new module to :data:`sys.modules`.
+
+ - :attr:`__name__`
+ The name of the module.
+
+ - :attr:`__file__`
+ The path to where the module data is stored (not set for built-in
+ modules).
+
+ - :attr:`__path__`
+ Set to a list of strings specifying the search path within a
+ package. This attribute is not set on modules.
+
+ - :attr:`__package__`
+ The parent package for the module/package. If the module is
+ top-level then it has a value of the empty string. The
+ :func:`importlib.util.set_package` decorator can handle the details
+ for :attr:`__package__`.
+
+ - :attr:`__loader__`
+ Set to the loader used to load the module.
+
+ See :pep:`302` for the exact definition for a loader.
+
+
+.. class:: ResourceLoader
+
+ An abstract base class for a :term:`loader` which implements the optional
+ :pep:`302` protocol for loading arbitrary resources from the storage
+ back-end.
+
+ ..method:: get_data(path)
+
+ An abstract method to return the bytes for the data located at *path*.
+ Loaders that have a file-like storage back-end can implement this
+ abstract method to give direct access
+ to the data stored. :exc:`IOError` is to be raised if the *path* cannot
+ be found. The *path* is expected to be constructed using a module's
+ :attr:`__path__` attribute or an item from :attr:`__path__`.
+
+
+.. class:: InspectLoader
+
+ An abstract base class for a :term:`loader` which implements the optional
+ :pep:`302` protocol for loaders which inspect modules.
+
+ ..method:: is_package(fullname)
+
+ An abstract method to return a true value if the module is a package, a
+ false value otherwise. :exc:`ImportError` is raised if the
+ :term:`loader` cannot find the module.
+
+ ..method:: get_source(fullname)
+
+ An abstract method to return the source of a module. It is returned as
+ a string with universal newline support. Returns :keyword:`None` if no
+ source is available (e.g. a built-in module). Raises :exc:`ImportError`
+ if the loader cannot find the module specified.
+
+ ..method:: get_code(fullname)
+
+ An abstract method to return the :class:`code` object for a module.
+ :keyword:`None` is returned if the module does not have a code object
+ (e.g. built-in module). :exc:`ImportError` is raised if loader cannot
+ find the requested module.
+
+
+.. class:: PyLoader
+
+ An abstract base class inheriting from :class:`importlib.abc.InspectLoader`
+ and :class:`importlib.abc.ResourceLoader` designed to ease the loading of
+ Python source modules (bytecode is not handled; see
+ :class:`importlib.abc.PyPycLoader` for a source/bytecode ABC). A subclass
+ implementing this ABC will only need to worry about exposing how the source
+ code is stored; all other details for loading Python source code will be
+ handled by the concrete implementations of key methods.
+
+ ..method:: source_path(fullname)
+
+ An abstract method that returns the path to the source code for a
+ module. Should return :keyword:`None` if there is no source code.
+ :exc:`ImportError` if the module cannot be found.
+
+ ..method:: load_module(fullname)
+
+ A concrete implementation of :meth:`importlib.abc.Loader.load_module`
+ that loads Python source code.
+
+ ..method:: get_code(fullname)
+
+ A concrete implementation of
+ :meth:`importlib.abc.InspectLoader.get_code` that creates code objects
+ from Python source code.
+
+
+.. class:: PyPycLoader
+
+ An abstract base class inheriting from :class:`importlib.abc.PyLoader`.
+ This ABC is meant to help in creating loaders that support both Python
+ source and bytecode.
+
+ ..method:: source_mtime(fullname)
+
+ An abstract method which returns the modification time for the source
+ code of the specified module. The modification time should be an
+ integer. If there is no source code, return :keyword:`None. If the
+ module cannot be found then :exc:`ImportError` is raised.
+
+ ..method:: bytecode_path(fullname)
+
+ An abstract method which returns the path to the bytecode for the
+ specified module. :keyword:`None` is returned if there is no bytecode.
+ :exc:`ImportError` is raised if the module is not found.
+
+ ..method:: write_bytecode(fullname, bytecode)
+
+ An abstract method which has the loader write *bytecode* for future
+ use. If the bytecode is written, return :keyword:`True`. Return
+ :keyword:`False` if the bytecode could not be written. This method
+ should not be called if :data:`sys.dont_write_bytecode` is true.
+
+
:mod:`importlib.machinery` -- Importers and path hooks
------------------------------------------------------
@@ -93,44 +262,27 @@
.. class:: BuiltinImporter
- :term:`Importer` for built-in modules. All known built-in modules are
- listed in :data:`sys.builtin_module_names`.
+ An :term:`importer` for built-in modules. All known built-in modules are
+ listed in :data:`sys.builtin_module_names`. This class implements the
+ :class:`importlib.abc.Finder` and :class:`importlib.abc.Loader` ABCs.
Only class methods are defined by this class to alleviate the need for
instantiation.
- .. classmethod:: find_module(fullname, path=None)
-
- Class method that allows this class to be a :term:`finder` for built-in
- modules.
-
- .. classmethod:: load_module(fullname)
-
- Class method that allows this class to be a :term:`loader` for built-in
- modules.
-
.. class:: FrozenImporter
- :term:`Importer` for frozen modules.
+ An :term:`importer` for frozen modules. This class implements the
+ :class:`importlib.abc.Finder` and :class:`importlib.abc.Loader` ABCs.
Only class methods are defined by this class to alleviate the need for
instantiation.
- .. classmethod:: find_module(fullname, path=None)
-
- Class method that allows this class to be a :term:`finder` for frozen
- modules.
-
- .. classmethod:: load_module(fullname)
-
- Class method that allows this class to be a :term:`loader` for frozen
- modules.
-
.. class:: PathFinder
- :term:`Finder` for :data:`sys.path`.
+ :term:`Finder` for :data:`sys.path`. This class implements the
+ :class:`importlib.abc.Finder` ABC.
This class does not perfectly mirror the semantics of :keyword:`import` in
terms of :data:`sys.path`. No implicit path hooks are assumed for
@@ -142,15 +294,15 @@
.. classmethod:: find_module(fullname, path=None)
Class method that attempts to find a :term:`loader` for the module
- specified by *fullname* either on :data:`sys.path` or, if defined, on
+ specified by *fullname* on :data:`sys.path` or, if defined, on
*path*. For each path entry that is searched,
:data:`sys.path_importer_cache` is checked. If an non-false object is
- found then it is used as the :term:`finder` to query for the module
- being searched for. For no entry is found in
+ found then it is used as the :term:`finder` to look for the module
+ being searched for. If no entry is found in
:data:`sys.path_importer_cache`, then :data:`sys.path_hooks` is
searched for a finder for the path entry and, if found, is stored in
:data:`sys.path_importer_cache` along with being queried about the
- module.
+ module. If no finder is ever found then :keyword:`None` is returned.
:mod:`importlib.util` -- Utility code for importers
@@ -166,10 +318,11 @@
A :term:`decorator` for a :term:`loader` which handles selecting the proper
module object to load with. The decorated method is expected to have a call
- signature of ``method(self, module_object)`` for which the second argument
- will be the module object to be used by the loader (note that the decorator
+ signature taking two positional arguments
+ (e.g. ``load_module(self, module)``) for which the second argument
+ will be the module object to be used by the loader. Note that the decorator
will not work on static methods because of the assumption of two
- arguments).
+ arguments.
The decorated method will take in the name of the module to be loaded as
expected for a :term:`loader`. If the module is not found in
Modified: python/branches/py3k/Lib/importlib/NOTES
==============================================================================
--- python/branches/py3k/Lib/importlib/NOTES (original)
+++ python/branches/py3k/Lib/importlib/NOTES Mon Mar 9 04:35:50 2009
@@ -3,42 +3,12 @@
* Public API left to expose (w/ docs!)
- + abc
+ + abc.PyLoader.get_source
+ + util.set_loader
- - Finder
+* Implement InspectLoader for BuiltinImporter and FrozenImporter.
- * find_module
-
- - Loader
-
- * load_module
-
- - ResourceLoader(Loader)
-
- * get_data
-
- - InspectLoader(Loader)
-
- * is_package
- * get_code
- * get_source
-
- - PyLoader(ResourceLoader)
-
- * source_path
-
- - PyPycLoader(PyLoader)
-
- * source_mtime
- * bytecode_path
- * write_bytecode
-
- + test (Really want to worry about compatibility with future versions?)
-
- - abc
-
- * FinderTests [doc]
- * LoaderTests [doc]
+ + Expose function to see if a frozen module is a package.
* Remove ``import *`` from importlib.__init__.
@@ -68,3 +38,4 @@
+ imp
+ py_compile
+ compileall
+ + zipimport
Modified: python/branches/py3k/Lib/importlib/_bootstrap.py
==============================================================================
--- python/branches/py3k/Lib/importlib/_bootstrap.py (original)
+++ python/branches/py3k/Lib/importlib/_bootstrap.py Mon Mar 9 04:35:50 2009
@@ -383,14 +383,8 @@
def load_module(self, module):
"""Load a module from source or bytecode."""
name = module.__name__
- try:
- source_path = self.source_path(name)
- except ImportError:
- source_path = None
- try:
- bytecode_path = self.bytecode_path(name)
- except ImportError:
- bytecode_path = None
+ source_path = self.source_path(name)
+ bytecode_path = self.bytecode_path(name)
# get_code can worry about no viable paths existing.
module.__file__ = source_path or bytecode_path
return self._load_module(module)
Added: python/branches/py3k/Lib/importlib/abc.py
==============================================================================
--- (empty file)
+++ python/branches/py3k/Lib/importlib/abc.py Mon Mar 9 04:35:50 2009
@@ -0,0 +1,110 @@
+"""Abstract base classes related to import."""
+from . import _bootstrap
+from . import machinery
+import abc
+import types
+
+
+class Loader(metaclass=abc.ABCMeta):
+
+ """Abstract base class for import loaders.
+
+ See PEP 302 for details.
+
+ """
+
+ def load_module(self, fullname:str) -> types.ModuleType:
+ raise NotImplementedError
+
+Loader.register(machinery.BuiltinImporter)
+Loader.register(machinery.FrozenImporter)
+
+
+class Finder(metaclass=abc.ABCMeta):
+
+ """Abstract base class for import finders.
+
+ See PEP 302 for details.
+
+ """
+
+ @abc.abstractmethod
+ def find_module(self, fullname:str, path:[str]=None) -> Loader:
+ raise NotImplementedError
+
+Finder.register(machinery.BuiltinImporter)
+Finder.register(machinery.FrozenImporter)
+Finder.register(machinery.PathFinder)
+
+
+class Importer(Finder, Loader):
+
+ """Abstract base class for importers."""
+
+
+
+class ResourceLoader(Loader):
+
+ """Abstract base class for loaders which can return data from the back-end
+ storage.
+
+ This ABC represents one of the optional protocols specified by PEP 302.
+
+ """
+
+ @abc.abstractmethod
+ def get_data(self, path:str) -> bytes:
+ raise NotImplementedError
+
+
+class InspectLoader(Loader):
+
+ """Abstract base class for loaders which supports introspection.
+
+ This ABC represents one of the optional protocols specified by PEP 302.
+
+ """
+
+ @abc.abstractmethod
+ def is_package(self, fullname:str) -> bool:
+ return NotImplementedError
+
+ @abc.abstractmethod
+ def get_code(self, fullname:str) -> types.CodeType:
+ return NotImplementedError
+
+ @abc.abstractmethod
+ def get_source(self, fullname:str) -> str:
+ return NotImplementedError
+
+
+class PyLoader(_bootstrap.PyLoader, InspectLoader):
+
+ """Abstract base class that implements the core parts needed to load Python
+ source code."""
+
+ # load_module and get_code are implemented.
+
+ @abc.abstractmethod
+ def source_path(self, fullname:str) -> object:
+ raise NotImplementedError
+
+
+class PyPycLoader(_bootstrap.PyPycLoader, PyLoader):
+
+ """Abstract base class that implements the core parts needed to load Python
+ source and bytecode."""
+
+ # Implements load_module and get_code.
+
+ @abc.abstractmethod
+ def source_mtime(self, fullname:str) -> int:
+ raise NotImplementedError
+
+ @abc.abstractmethod
+ def bytecode_path(self, fullname:str) -> object:
+ raise NotImplementedError
+
+ @abc.abstractmethod
+ def write_bytecode(self, fullname:str, bytecode:bytes):
+ raise NotImplementedError
Modified: python/branches/py3k/Lib/importlib/test/abc.py
==============================================================================
--- python/branches/py3k/Lib/importlib/test/abc.py (original)
+++ python/branches/py3k/Lib/importlib/test/abc.py Mon Mar 9 04:35:50 2009
@@ -65,10 +65,11 @@
Attributes to verify:
- * __file__
- * __loader__
* __name__
+ * __file__
+ * __package__
* __path__
+ * __loader__
"""
pass
Added: python/branches/py3k/Lib/importlib/test/source/test_abc_loader.py
==============================================================================
--- (empty file)
+++ python/branches/py3k/Lib/importlib/test/source/test_abc_loader.py Mon Mar 9 04:35:50 2009
@@ -0,0 +1,390 @@
+import importlib
+from importlib import abc
+from .. import abc as testing_abc
+from .. import util
+from . import util as source_util
+import imp
+import marshal
+import os
+import sys
+import types
+import unittest
+
+
+class PyLoaderMock(abc.PyLoader):
+
+ # Globals that should be defined for all modules.
+ source = ("_ = '::'.join([__name__, __file__, __package__, "
+ "repr(__loader__)])")
+
+ def __init__(self, data):
+ """Take a dict of 'module_name: path' pairings.
+
+ Paths should have no file extension, allowing packages to be denoted by
+ ending in '__init__'.
+
+ """
+ self.module_paths = data
+ self.path_to_module = {val:key for key,val in data.items()}
+
+ def get_data(self, path):
+ if path not in self.path_to_module:
+ raise IOError
+ return self.source.encode('utf-8')
+
+ def is_package(self, name):
+ try:
+ return '__init__' in self.module_paths[name]
+ except KeyError:
+ raise ImportError
+
+ def get_source(self, name): # Should not be needed.
+ raise NotImplementedError
+
+ def source_path(self, name):
+ try:
+ return self.module_paths[name]
+ except KeyError:
+ raise ImportError
+
+
+class PyPycLoaderMock(abc.PyPycLoader, PyLoaderMock):
+
+ default_mtime = 1
+
+ def __init__(self, source, bc={}):
+ """Initialize mock.
+
+ 'bc' is a dict keyed on a module's name. The value is dict with
+ possible keys of 'path', 'mtime', 'magic', and 'bc'. Except for 'path',
+ each of those keys control if any part of created bytecode is to
+ deviate from default values.
+
+ """
+ super().__init__(source)
+ self.module_bytecode = {}
+ self.path_to_bytecode = {}
+ self.bytecode_to_path = {}
+ for name, data in bc.items():
+ self.path_to_bytecode[data['path']] = name
+ self.bytecode_to_path[name] = data['path']
+ magic = data.get('magic', imp.get_magic())
+ mtime = importlib._w_long(data.get('mtime', self.default_mtime))
+ if 'bc' in data:
+ bc = data['bc']
+ else:
+ bc = self.compile_bc(name)
+ self.module_bytecode[name] = magic + mtime + bc
+
+ def compile_bc(self, name):
+ source_path = self.module_paths.get(name, '<test>') or '<test>'
+ code = compile(self.source, source_path, 'exec')
+ return marshal.dumps(code)
+
+ def source_mtime(self, name):
+ if name in self.module_paths:
+ return self.default_mtime
+ elif name in self.module_bytecode:
+ return None
+ else:
+ raise ImportError
+
+ def bytecode_path(self, name):
+ try:
+ return self.bytecode_to_path[name]
+ except KeyError:
+ if name in self.module_paths:
+ return None
+ else:
+ raise ImportError
+
+ def write_bytecode(self, name, bytecode):
+ self.module_bytecode[name] = bytecode
+ return True
+
+ def get_data(self, path):
+ if path in self.path_to_module:
+ return super().get_data(path)
+ elif path in self.path_to_bytecode:
+ name = self.path_to_bytecode[path]
+ return self.module_bytecode[name]
+ else:
+ raise IOError
+
+ def is_package(self, name):
+ try:
+ return super().is_package(name)
+ except TypeError:
+ return '__init__' in self.bytecode_to_path[name]
+
+
+class PyLoaderTests(testing_abc.LoaderTests):
+
+ """Tests for importlib.abc.PyLoader."""
+
+ mocker = PyLoaderMock
+
+ def eq_attrs(self, ob, **kwargs):
+ for attr, val in kwargs.items():
+ self.assertEqual(getattr(ob, attr), val)
+
+ def test_module(self):
+ name = '<module>'
+ path = 'path/to/module'
+ mock = self.mocker({name: path})
+ with util.uncache(name):
+ module = mock.load_module(name)
+ self.assert_(name in sys.modules)
+ self.eq_attrs(module, __name__=name, __file__=path, __package__='',
+ __loader__=mock)
+ self.assert_(not hasattr(module, '__path__'))
+ return mock, name
+
+ def test_package(self):
+ name = '<pkg>'
+ path = '/path/to/<pkg>/__init__'
+ mock = self.mocker({name: path})
+ with util.uncache(name):
+ module = mock.load_module(name)
+ self.assert_(name in sys.modules)
+ self.eq_attrs(module, __name__=name, __file__=path,
+ __path__=[os.path.dirname(path)], __package__=name,
+ __loader__=mock)
+ return mock, name
+
+ def test_lacking_parent(self):
+ name = 'pkg.mod'
+ path = 'path/to/pkg/mod'
+ mock = self.mocker({name: path})
+ with util.uncache(name):
+ module = mock.load_module(name)
+ self.assert_(name in sys.modules)
+ self.eq_attrs(module, __name__=name, __file__=path, __package__='pkg',
+ __loader__=mock)
+ self.assert_(not hasattr(module, '__path__'))
+ return mock, name
+
+ def test_module_reuse(self):
+ name = 'mod'
+ path = 'path/to/mod'
+ module = imp.new_module(name)
+ mock = self.mocker({name: path})
+ with util.uncache(name):
+ sys.modules[name] = module
+ loaded_module = mock.load_module(name)
+ self.assert_(loaded_module is module)
+ self.assert_(sys.modules[name] is module)
+ return mock, name
+
+ def test_state_after_failure(self):
+ name = "mod"
+ module = imp.new_module(name)
+ module.blah = None
+ mock = self.mocker({name: 'path/to/mod'})
+ mock.source = "1/0"
+ with util.uncache(name):
+ sys.modules[name] = module
+ self.assertRaises(ZeroDivisionError, mock.load_module, name)
+ self.assert_(sys.modules[name] is module)
+ self.assert_(hasattr(module, 'blah'))
+ return mock
+
+ def test_unloadable(self):
+ name = "mod"
+ mock = self.mocker({name: 'path/to/mod'})
+ mock.source = "1/0"
+ with util.uncache(name):
+ self.assertRaises(ZeroDivisionError, mock.load_module, name)
+ self.assert_(name not in sys.modules)
+ return mock
+
+
+class PyLoaderInterfaceTests(unittest.TestCase):
+
+
+ def test_no_source_path(self):
+ # No source path should lead to ImportError.
+ name = 'mod'
+ mock = PyLoaderMock({})
+ with util.uncache(name):
+ self.assertRaises(ImportError, mock.load_module, name)
+
+ def test_source_path_is_None(self):
+ name = 'mod'
+ mock = PyLoaderMock({name: None})
+ with util.uncache(name):
+ self.assertRaises(ImportError, mock.load_module, name)
+
+
+class PyPycLoaderTests(PyLoaderTests):
+
+ """Tests for importlib.abc.PyPycLoader."""
+
+ mocker = PyPycLoaderMock
+
+ @source_util.writes_bytecode
+ def verify_bytecode(self, mock, name):
+ assert name in mock.module_paths
+ self.assert_(name in mock.module_bytecode)
+ magic = mock.module_bytecode[name][:4]
+ self.assertEqual(magic, imp.get_magic())
+ mtime = importlib._r_long(mock.module_bytecode[name][4:8])
+ self.assertEqual(mtime, 1)
+ bc = mock.module_bytecode[name][8:]
+
+
+ def test_module(self):
+ mock, name = super().test_module()
+ self.verify_bytecode(mock, name)
+
+ def test_package(self):
+ mock, name = super().test_package()
+ self.verify_bytecode(mock, name)
+
+ def test_lacking_parent(self):
+ mock, name = super().test_lacking_parent()
+ self.verify_bytecode(mock, name)
+
+ def test_module_reuse(self):
+ mock, name = super().test_module_reuse()
+ self.verify_bytecode(mock, name)
+
+ def test_state_after_failure(self):
+ super().test_state_after_failure()
+
+ def test_unloadable(self):
+ super().test_unloadable()
+
+
+class SkipWritingBytecodeTests(unittest.TestCase):
+
+ """Test that bytecode is properly handled based on
+ sys.dont_write_bytecode."""
+
+ @source_util.writes_bytecode
+ def run_test(self, dont_write_bytecode):
+ name = 'mod'
+ mock = PyPycLoaderMock({name: 'path/to/mod'})
+ sys.dont_write_bytecode = dont_write_bytecode
+ with util.uncache(name):
+ mock.load_module(name)
+ self.assert_((name in mock.module_bytecode) is not
+ dont_write_bytecode)
+
+ def test_no_bytecode_written(self):
+ self.run_test(True)
+
+ def test_bytecode_written(self):
+ self.run_test(False)
+
+
+class RegeneratedBytecodeTests(unittest.TestCase):
+
+ """Test that bytecode is regenerated as expected."""
+
+ @source_util.writes_bytecode
+ def test_different_magic(self):
+ # A different magic number should lead to new bytecode.
+ name = 'mod'
+ bad_magic = b'\x00\x00\x00\x00'
+ assert bad_magic != imp.get_magic()
+ mock = PyPycLoaderMock({name: 'path/to/mod'},
+ {name: {'path': 'path/to/mod.bytecode',
+ 'magic': bad_magic}})
+ with util.uncache(name):
+ mock.load_module(name)
+ self.assert_(name in mock.module_bytecode)
+ magic = mock.module_bytecode[name][:4]
+ self.assertEqual(magic, imp.get_magic())
+
+ @source_util.writes_bytecode
+ def test_old_mtime(self):
+ # Bytecode with an older mtime should be regenerated.
+ name = 'mod'
+ old_mtime = PyPycLoaderMock.default_mtime - 1
+ mock = PyPycLoaderMock({name: 'path/to/mod'},
+ {name: {'path': 'path/to/mod.bytecode', 'mtime': old_mtime}})
+ with util.uncache(name):
+ mock.load_module(name)
+ self.assert_(name in mock.module_bytecode)
+ mtime = importlib._r_long(mock.module_bytecode[name][4:8])
+ self.assertEqual(mtime, PyPycLoaderMock.default_mtime)
+
+
+class BadBytecodeFailureTests(unittest.TestCase):
+
+ """Test import failures when there is no source and parts of the bytecode
+ is bad."""
+
+ def test_bad_magic(self):
+ # A bad magic number should lead to an ImportError.
+ name = 'mod'
+ bad_magic = b'\x00\x00\x00\x00'
+ mock = PyPycLoaderMock({}, {name: {'path': 'path/to/mod',
+ 'magic': bad_magic}})
+ with util.uncache(name):
+ self.assertRaises(ImportError, mock.load_module, name)
+
+ def test_bad_bytecode(self):
+ # Bad code object bytecode should elad to an ImportError.
+ name = 'mod'
+ mock = PyPycLoaderMock({}, {name: {'path': '/path/to/mod', 'bc': b''}})
+ with util.uncache(name):
+ self.assertRaises(ImportError, mock.load_module, name)
+
+
+def raise_ImportError(*args, **kwargs):
+ raise ImportError
+
+class MissingPathsTests(unittest.TestCase):
+
+ """Test what happens when a source or bytecode path does not exist (either
+ from *_path returning None or raising ImportError)."""
+
+ def test_source_path_None(self):
+ # Bytecode should be used when source_path returns None, along with
+ # __file__ being set to the bytecode path.
+ name = 'mod'
+ bytecode_path = 'path/to/mod'
+ mock = PyPycLoaderMock({name: None}, {name: {'path': bytecode_path}})
+ with util.uncache(name):
+ module = mock.load_module(name)
+ self.assertEqual(module.__file__, bytecode_path)
+
+ # Testing for bytecode_path returning None handled by all tests where no
+ # bytecode initially exists.
+
+ def test_all_paths_None(self):
+ # If all *_path methods return None, raise ImportError.
+ name = 'mod'
+ mock = PyPycLoaderMock({name: None})
+ with util.uncache(name):
+ self.assertRaises(ImportError, mock.load_module, name)
+
+ def test_source_path_ImportError(self):
+ # An ImportError from source_path should trigger an ImportError.
+ name = 'mod'
+ mock = PyPycLoaderMock({}, {name: {'path': 'path/to/mod'}})
+ with util.uncache(name):
+ self.assertRaises(ImportError, mock.load_module, name)
+
+ def test_bytecode_path_ImportError(self):
+ # An ImportError from bytecode_path should trigger an ImportError.
+ name = 'mod'
+ mock = PyPycLoaderMock({name: 'path/to/mod'})
+ bad_meth = types.MethodType(raise_ImportError, mock)
+ mock.bytecode_path = bad_meth
+ with util.uncache(name):
+ self.assertRaises(ImportError, mock.load_module, name)
+
+
+def test_main():
+ from test.support import run_unittest
+ run_unittest(PyLoaderTests, PyLoaderInterfaceTests,
+ PyPycLoaderTests, SkipWritingBytecodeTests,
+ RegeneratedBytecodeTests, BadBytecodeFailureTests,
+ MissingPathsTests)
+
+
+if __name__ == '__main__':
+ test_main()
Copied: python/branches/py3k/Lib/importlib/test/source/test_file_loader.py (from r70247, /python/branches/py3k/Lib/importlib/test/source/test_loader.py)
==============================================================================
--- /python/branches/py3k/Lib/importlib/test/source/test_loader.py (original)
+++ python/branches/py3k/Lib/importlib/test/source/test_file_loader.py Mon Mar 9 04:35:50 2009
@@ -102,102 +102,6 @@
self.assert_('_temp' not in sys.modules)
-class DontWriteBytecodeTest(unittest.TestCase):
-
- """If sys.dont_write_bytcode is true then no bytecode should be created."""
-
- def tearDown(self):
- sys.dont_write_bytecode = False
-
- @source_util.writes_bytecode
- def run_test(self, assertion):
- with source_util.create_modules('_temp') as mapping:
- loader = importlib.PyPycFileLoader('_temp', mapping['_temp'], False)
- loader.load_module('_temp')
- bytecode_path = source_util.bytecode_path(mapping['_temp'])
- assertion(bytecode_path)
-
- def test_bytecode_written(self):
- fxn = lambda bc_path: self.assert_(os.path.exists(bc_path))
- self.run_test(fxn)
-
- def test_bytecode_not_written(self):
- sys.dont_write_bytecode = True
- fxn = lambda bc_path: self.assert_(not os.path.exists(bc_path))
- self.run_test(fxn)
-
-
-class BadDataTest(unittest.TestCase):
-
- """If the bytecode has a magic number that does not match the
- interpreters', ImportError is raised [bad magic]. The timestamp can have
- any value. And bad marshal data raises ValueError.
-
- """
-
- # [bad magic]
- def test_bad_magic(self):
- with source_util.create_modules('_temp') as mapping:
- py_compile.compile(mapping['_temp'])
- os.unlink(mapping['_temp'])
- bytecode_path = source_util.bytecode_path(mapping['_temp'])
- with open(bytecode_path, 'r+b') as file:
- file.seek(0)
- file.write(b'\x00\x00\x00\x00')
- loader = importlib.PyPycFileLoader('_temp', mapping['_temp'], False)
- self.assertRaises(ImportError, loader.load_module, '_temp')
- self.assert_('_temp' not in sys.modules)
-
-
-class SourceBytecodeInteraction(unittest.TestCase):
-
- """When both source and bytecode are present, certain rules dictate which
- version of the code takes precedent. All things being equal, the bytecode
- is used with the value of __file__ set to the source [basic top-level],
- [basic package], [basic sub-module], [basic sub-package].
-
- """
-
- def import_(self, file, module, *, pkg=False):
- loader = importlib.PyPycFileLoader(module, file, pkg)
- return loader.load_module(module)
-
- def run_test(self, test, *create, pkg=False):
- create += (test,)
- with source_util.create_modules(*create) as mapping:
- for name in create:
- py_compile.compile(mapping[name])
- if pkg:
- import_name = test.rsplit('.', 1)[0]
- else:
- import_name = test
- loader = importlib.PyPycFileLoader(import_name, mapping[test], pkg)
- # Because some platforms only have a granularity to the second for
- # atime you can't check the physical files. Instead just make it an
- # exception trigger if source was read.
- loader.get_source = lambda self, x: 42
- module = loader.load_module(import_name)
- self.assertEqual(module.__file__, mapping[name])
- self.assert_(import_name in sys.modules)
- self.assertEqual(id(module), id(sys.modules[import_name]))
-
- # [basic top-level]
- def test_basic_top_level(self):
- self.run_test('top_level')
-
- # [basic package]
- def test_basic_package(self):
- self.run_test('pkg.__init__', pkg=True)
-
- # [basic sub-module]
- def test_basic_sub_module(self):
- self.run_test('pkg.sub', 'pkg.__init__')
-
- # [basic sub-package]
- def test_basic_sub_package(self):
- self.run_test('pkg.sub.__init__', 'pkg.__init__', pkg=True)
-
-
class BadBytecodeTest(unittest.TestCase):
"""But there are several things about the bytecode which might lead to the
Deleted: python/branches/py3k/Lib/importlib/test/source/test_loader.py
==============================================================================
--- python/branches/py3k/Lib/importlib/test/source/test_loader.py Mon Mar 9 04:35:50 2009
+++ (empty file)
@@ -1,271 +0,0 @@
-import importlib
-from .. import abc
-from . import util as source_util
-
-import imp
-import os
-import py_compile
-import sys
-import unittest
-
-
-class SimpleTest(unittest.TestCase):
-
- """Should have no issue importing a source module [basic]. And if there is
- a syntax error, it should raise a SyntaxError [syntax error].
-
- """
-
- # [basic]
- def test_module(self):
- with source_util.create_modules('_temp') as mapping:
- loader = importlib.PyPycFileLoader('_temp', mapping['_temp'], False)
- module = loader.load_module('_temp')
- self.assert_('_temp' in sys.modules)
- check = {'__name__': '_temp', '__file__': mapping['_temp'],
- '__package__': ''}
- for attr, value in check.items():
- self.assertEqual(getattr(module, attr), value)
-
- def test_package(self):
- with source_util.create_modules('_pkg.__init__') as mapping:
- loader = importlib.PyPycFileLoader('_pkg', mapping['_pkg.__init__'],
- True)
- module = loader.load_module('_pkg')
- self.assert_('_pkg' in sys.modules)
- check = {'__name__': '_pkg', '__file__': mapping['_pkg.__init__'],
- '__path__': [os.path.dirname(mapping['_pkg.__init__'])],
- '__package__': '_pkg'}
- for attr, value in check.items():
- self.assertEqual(getattr(module, attr), value)
-
-
- def test_lacking_parent(self):
- with source_util.create_modules('_pkg.__init__', '_pkg.mod')as mapping:
- loader = importlib.PyPycFileLoader('_pkg.mod', mapping['_pkg.mod'],
- False)
- module = loader.load_module('_pkg.mod')
- self.assert_('_pkg.mod' in sys.modules)
- check = {'__name__': '_pkg.mod', '__file__': mapping['_pkg.mod'],
- '__package__': '_pkg'}
- for attr, value in check.items():
- self.assertEqual(getattr(module, attr), value)
-
- def fake_mtime(self, fxn):
- """Fake mtime to always be higher than expected."""
- return lambda name: fxn(name) + 1
-
- def test_module_reuse(self):
- with source_util.create_modules('_temp') as mapping:
- loader = importlib.PyPycFileLoader('_temp', mapping['_temp'], False)
- module = loader.load_module('_temp')
- module_id = id(module)
- module_dict_id = id(module.__dict__)
- with open(mapping['_temp'], 'w') as file:
- file.write("testing_var = 42\n")
- # For filesystems where the mtime is only to a second granularity,
- # everything that has happened above can be too fast;
- # force an mtime on the source that is guaranteed to be different
- # than the original mtime.
- loader.source_mtime = self.fake_mtime(loader.source_mtime)
- module = loader.load_module('_temp')
- self.assert_('testing_var' in module.__dict__,
- "'testing_var' not in "
- "{0}".format(list(module.__dict__.keys())))
- self.assertEqual(module, sys.modules['_temp'])
- self.assertEqual(id(module), module_id)
- self.assertEqual(id(module.__dict__), module_dict_id)
-
- def test_state_after_failure(self):
- # A failed reload should leave the original module intact.
- attributes = ('__file__', '__path__', '__package__')
- value = '<test>'
- name = '_temp'
- with source_util.create_modules(name) as mapping:
- orig_module = imp.new_module(name)
- for attr in attributes:
- setattr(orig_module, attr, value)
- with open(mapping[name], 'w') as file:
- file.write('+++ bad syntax +++')
- loader = importlib.PyPycFileLoader('_temp', mapping['_temp'], False)
- self.assertRaises(SyntaxError, loader.load_module, name)
- for attr in attributes:
- self.assertEqual(getattr(orig_module, attr), value)
-
- # [syntax error]
- def test_bad_syntax(self):
- with source_util.create_modules('_temp') as mapping:
- with open(mapping['_temp'], 'w') as file:
- file.write('=')
- loader = importlib.PyPycFileLoader('_temp', mapping['_temp'], False)
- self.assertRaises(SyntaxError, loader.load_module, '_temp')
- self.assert_('_temp' not in sys.modules)
-
-
-class DontWriteBytecodeTest(unittest.TestCase):
-
- """If sys.dont_write_bytcode is true then no bytecode should be created."""
-
- def tearDown(self):
- sys.dont_write_bytecode = False
-
- @source_util.writes_bytecode
- def run_test(self, assertion):
- with source_util.create_modules('_temp') as mapping:
- loader = importlib.PyPycFileLoader('_temp', mapping['_temp'], False)
- loader.load_module('_temp')
- bytecode_path = source_util.bytecode_path(mapping['_temp'])
- assertion(bytecode_path)
-
- def test_bytecode_written(self):
- fxn = lambda bc_path: self.assert_(os.path.exists(bc_path))
- self.run_test(fxn)
-
- def test_bytecode_not_written(self):
- sys.dont_write_bytecode = True
- fxn = lambda bc_path: self.assert_(not os.path.exists(bc_path))
- self.run_test(fxn)
-
-
-class BadDataTest(unittest.TestCase):
-
- """If the bytecode has a magic number that does not match the
- interpreters', ImportError is raised [bad magic]. The timestamp can have
- any value. And bad marshal data raises ValueError.
-
- """
-
- # [bad magic]
- def test_bad_magic(self):
- with source_util.create_modules('_temp') as mapping:
- py_compile.compile(mapping['_temp'])
- os.unlink(mapping['_temp'])
- bytecode_path = source_util.bytecode_path(mapping['_temp'])
- with open(bytecode_path, 'r+b') as file:
- file.seek(0)
- file.write(b'\x00\x00\x00\x00')
- loader = importlib.PyPycFileLoader('_temp', mapping['_temp'], False)
- self.assertRaises(ImportError, loader.load_module, '_temp')
- self.assert_('_temp' not in sys.modules)
-
-
-class SourceBytecodeInteraction(unittest.TestCase):
-
- """When both source and bytecode are present, certain rules dictate which
- version of the code takes precedent. All things being equal, the bytecode
- is used with the value of __file__ set to the source [basic top-level],
- [basic package], [basic sub-module], [basic sub-package].
-
- """
-
- def import_(self, file, module, *, pkg=False):
- loader = importlib.PyPycFileLoader(module, file, pkg)
- return loader.load_module(module)
-
- def run_test(self, test, *create, pkg=False):
- create += (test,)
- with source_util.create_modules(*create) as mapping:
- for name in create:
- py_compile.compile(mapping[name])
- if pkg:
- import_name = test.rsplit('.', 1)[0]
- else:
- import_name = test
- loader = importlib.PyPycFileLoader(import_name, mapping[test], pkg)
- # Because some platforms only have a granularity to the second for
- # atime you can't check the physical files. Instead just make it an
- # exception trigger if source was read.
- loader.get_source = lambda self, x: 42
- module = loader.load_module(import_name)
- self.assertEqual(module.__file__, mapping[name])
- self.assert_(import_name in sys.modules)
- self.assertEqual(id(module), id(sys.modules[import_name]))
-
- # [basic top-level]
- def test_basic_top_level(self):
- self.run_test('top_level')
-
- # [basic package]
- def test_basic_package(self):
- self.run_test('pkg.__init__', pkg=True)
-
- # [basic sub-module]
- def test_basic_sub_module(self):
- self.run_test('pkg.sub', 'pkg.__init__')
-
- # [basic sub-package]
- def test_basic_sub_package(self):
- self.run_test('pkg.sub.__init__', 'pkg.__init__', pkg=True)
-
-
-class BadBytecodeTest(unittest.TestCase):
-
- """But there are several things about the bytecode which might lead to the
- source being preferred. If the magic number differs from what the
- interpreter uses, then the source is used with the bytecode regenerated.
- If the timestamp is older than the modification time for the source then
- the bytecode is not used [bad timestamp].
-
- But if the marshal data is bad, even if the magic number and timestamp
- work, a ValueError is raised and the source is not used [bad marshal].
-
- """
-
- def import_(self, file, module_name):
- loader = importlib.PyPycFileLoader(module_name, file, False)
- module = loader.load_module(module_name)
- self.assert_(module_name in sys.modules)
-
- # [bad magic]
- @source_util.writes_bytecode
- def test_bad_magic(self):
- with source_util.create_modules('_temp') as mapping:
- py_compile.compile(mapping['_temp'])
- bytecode_path = source_util.bytecode_path(mapping['_temp'])
- with open(bytecode_path, 'r+b') as bytecode_file:
- bytecode_file.seek(0)
- bytecode_file.write(b'\x00\x00\x00\x00')
- self.import_(mapping['_temp'], '_temp')
- with open(bytecode_path, 'rb') as bytecode_file:
- self.assertEqual(bytecode_file.read(4), imp.get_magic())
-
- # [bad timestamp]
- @source_util.writes_bytecode
- def test_bad_bytecode(self):
- zeros = b'\x00\x00\x00\x00'
- with source_util.create_modules('_temp') as mapping:
- py_compile.compile(mapping['_temp'])
- bytecode_path = source_util.bytecode_path(mapping['_temp'])
- with open(bytecode_path, 'r+b') as bytecode_file:
- bytecode_file.seek(4)
- bytecode_file.write(zeros)
- self.import_(mapping['_temp'], '_temp')
- source_mtime = os.path.getmtime(mapping['_temp'])
- source_timestamp = importlib._w_long(source_mtime)
- with open(bytecode_path, 'rb') as bytecode_file:
- bytecode_file.seek(4)
- self.assertEqual(bytecode_file.read(4), source_timestamp)
-
- # [bad marshal]
- def test_bad_marshal(self):
- with source_util.create_modules('_temp') as mapping:
- bytecode_path = source_util.bytecode_path(mapping['_temp'])
- source_mtime = os.path.getmtime(mapping['_temp'])
- source_timestamp = importlib._w_long(source_mtime)
- with open(bytecode_path, 'wb') as bytecode_file:
- bytecode_file.write(imp.get_magic())
- bytecode_file.write(source_timestamp)
- bytecode_file.write(b'AAAA')
- self.assertRaises(ValueError, self.import_, mapping['_temp'],
- '_temp')
- self.assert_('_temp' not in sys.modules)
-
-
-def test_main():
- from test.support import run_unittest
- run_unittest(SimpleTest, DontWriteBytecodeTest, BadDataTest,
- SourceBytecodeInteraction, BadBytecodeTest)
-
-
-if __name__ == '__main__':
- test_main()
Modified: python/branches/py3k/Lib/importlib/test/source/util.py
==============================================================================
--- python/branches/py3k/Lib/importlib/test/source/util.py (original)
+++ python/branches/py3k/Lib/importlib/test/source/util.py Mon Mar 9 04:35:50 2009
@@ -1,5 +1,6 @@
from .. import util
import contextlib
+import functools
import imp
import os
import os.path
@@ -9,11 +10,24 @@
def writes_bytecode(fxn):
+ """Decorator to protect sys.dont_write_bytecode from mutation."""
+ @functools.wraps(fxn)
+ def wrapper(*args, **kwargs):
+ original = sys.dont_write_bytecode
+ sys.dont_write_bytecode = False
+ to_return = fxn(*args, **kwargs)
+ sys.dont_write_bytecode = original
+ return to_return
+ return wrapper
+
+
+def writes_bytecode_files(fxn):
"""Decorator that returns the function if writing bytecode is enabled, else
a stub function that accepts anything and simply returns None."""
if sys.dont_write_bytecode:
return lambda *args, **kwargs: None
else:
+ @functools.wraps(fxn)
def wrapper(*args, **kwargs):
to_return = fxn(*args, **kwargs)
sys.dont_write_bytecode = False
Added: python/branches/py3k/Lib/importlib/test/test_abc.py
==============================================================================
--- (empty file)
+++ python/branches/py3k/Lib/importlib/test/test_abc.py Mon Mar 9 04:35:50 2009
@@ -0,0 +1,31 @@
+from importlib import abc
+from importlib import machinery
+import unittest
+
+
+class SubclassTests(unittest.TestCase):
+
+ """Test that the various classes in importlib are subclasses of the
+ expected ABCS."""
+
+ def verify(self, ABC, *classes):
+ """Verify the classes are subclasses of the ABC."""
+ for cls in classes:
+ self.assert_(issubclass(cls, ABC))
+
+ def test_Finder(self):
+ self.verify(abc.Finder, machinery.BuiltinImporter,
+ machinery.FrozenImporter, machinery.PathFinder)
+
+ def test_Loader(self):
+ self.verify(abc.Loader, machinery.BuiltinImporter,
+ machinery.FrozenImporter)
+
+
+def test_main():
+ from test.support import run_unittest
+ run_unittest(SubclassTests)
+
+
+if __name__ == '__main__':
+ test_main()
More information about the Python-checkins
mailing list