Import Problem

[John Roth]

I've found a case where it seems that Python is importing two copies of a
module without any reason or indication. It took me a while to verify that
this is what is occuring: I had to write a __import__ hook to trace the
actual activity. The source code for the hook is below.

There are several examples of the problem in the trace; this is one
particularly juicy one.

-----------Part 1 ----------------
'0059' Import 'TypeAdapter' global: 'fit.Fixture' local: '9547968' 'None'

'0060' Import 're' global: 'fit.TypeAdapter' local: '9965904' 'None'
'0060' path:  're' module: '9387920' dict: '9415248'
'0060' result 're' is at '9387920'
'0060' path:  're' module: '9387920' dict: '9415248'
----------- end of part 1 ------------------------

What this shows is import # 59 importing 'TypeAdapter' from
'fit.Fixture.'

TypeAdapter is then importing 're'. Before the import, re is
already in the sys.modules; the import returns the expected
address, and the sys.modules entry remains the same. This
isn't a problem, it's simply an example of trace output for
a correct action.

-------------- Part 2 --------------------------
'0113' Import 'compiler.pyassem' global: 'compiler.pycodegen' local:
'10439248' '('TupleArg',)'
'0113' path:  'compiler.pyassem' module: '10508624' dict: '10515056'
'0113' result 'compiler.pyassem' is at '10508624'
'0113' path:  'compiler.pyassem' module: '10508624' dict: '10515056'
'0081' result 'pycodegen' is at '10396368'
'0081' path:  'compiler.pycodegen' module: '10396368' dict: '10439248'
'0066' result 'compiler' is at '10006320'
'0066' path:  'compiler' module: '10006320' dict: '9968496'
'0059' result 'TypeAdapter' is at '9973392'
'0059' path:  'fit.TypeAdapter' module: '9973392' dict: '9965904'
--------------- End of Part 2 ------------------------------

This shows the final unwinding of the TypeAdapter import that occurs
about 54 imports later! Note the address of the TypeAdapter module.
The fact that there's no path entry for 59 before the result of the
import shows that it was not in the sys.modules dictionary at that
point.

--------------- Part 3 ----------------------------------------
'0118' Import 'fit.eg.Division' global: 'None' local: 'None' 'None'

'0119' Import 'fit.ColumnFixture' global: 'fit.eg.Division' local:
'10636576' '('ColumnFixture',)'
------------------- End of Part 3 ------------------------------

This simply shows the start of the traces for imports 118 and 119,
which are in the next entry.

--------------- Part 4 ----------------------------------------
'0125' Import 'fit.TypeAdapter' global: 'fit.ColumnFixture' local:
'10637008' 'None'
'0125' path:  'fit.TypeAdapter' module: '9973392' dict: '9965904'
'0125' result 'fit.TypeAdapter' is at '8976560'
'0125' path:  'fit.TypeAdapter' module: '9973392' dict: '9965904'
'0119' result 'fit.ColumnFixture' is at '10650256'
'0119' path:  'fit.ColumnFixture' module: '10650256' dict: '10637008'
'0118' result 'fit.eg.Division' is at '8976560'
'0118' path:  'fit.eg.Division' module: '10603024' dict: '10636576'
--------------- End of Part 4 ------------------------------------

Entry 125 is the next import of TypeAdapter, in module ColumnFixture
which is in module Division. Note that sys.modules (the two lines that
start with 'path') still shows the same address for TypeAdapter as the
previous trace entries: that hasn't changed. However, the third line
shows that the result of the import is at a different address!

I'm baffled about why it's occurring. System is ActiveState Python
2.3.3. on Windows XP.

The import hook is:

---------------------- Beginning of module ImportSpike ------------
# Override for Import statement

print "in ImportSpike"
import __builtin__
import sys
from types import ModuleType

seqNum = [0]

def identifyGlobalDict(aDict, seqNum):
    if aDict is None:
        return "None"
    for key, value in sys.modules.items():
        if value is None: continue
        try:
            if value.__dict__ == aDict:
                return key
        except:
            pass
    return id(aDict)

def findModule(name, seqNum):
    for key, value in sys.modules.items():
        if value is None: continue
        if key.endswith(name):
            if len(key) == len(name) or key[-(len(name)+1)] == '.':
                print "'%04i' path:  '%s' module: '%s' dict: '%s'" % (
                    seqNum, key, id(value), id(value.__dict__))
                if type(value) != ModuleType:
                    print "'%04i' ***** '%s' is '%s'" % (
                        seqNum, key, type(value))

def newImport(path, globals = None, locals = None, nameList = None):
    seqNum[0] += 1
    localSeqNum = seqNum[0]
    localObj = locals and id(locals)
    globalObj = globals and id(globals)
    print " "
    print "'%04i' Import '%s' global: '%s' local: '%s' '%s'" % (
        localSeqNum, path, identifyGlobalDict(globals, localSeqNum),
        localObj, nameList)
    findModule(path, localSeqNum)
    result = oldImport(path, globals, locals, nameList)
    print "'%04i' result '%s' is at '%s'" % (localSeqNum, path, id(result))
    findModule(path, localSeqNum)
    return result

oldImport = __import__

__builtin__.__import__ = newImport
------------------------- End of module ImportSpike -------------------

Any idea of what's going on?

John Roth