[Patches] [ python-Patches-1553314 ] Fix inspect.py 2.5 slowdown

[SourceForge.net]

Patches item #1553314, was opened at 2006-09-06 20:51
Message generated for change (Comment added) made by ncoghlan
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1553314&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Library (Lib)
Group: Python 2.5
Status: Open
Resolution: None
Priority: 9
Submitted By: Nick Coghlan (ncoghlan)
Assigned to: Neal Norwitz (nnorwitz)
Summary: Fix inspect.py 2.5 slowdown

Initial Comment:
Every call to inspect.findsource() in Python 2.5 is
calling both os.path.abspath() and os.path.normpath(),
even if the file name is already absolute and
normalized, and the relevant entry is in the module
name cache.

This patch changes getmodule() to first check the
modulesbyfile cache earlier, so these calls are skipped
when the module name for the file is already known.

----------------------------------------------------------------------

>Comment By: Nick Coghlan (ncoghlan)
Date: 2006-09-07 20:53

Message:
Logged In: YES 
user_id=1038590

I beefed up the tests to pick up the error Ralf spotted in
the previous version of the patch, and added more comments
to getmodule() to explain why some things are the way they
are (specifically the use of sys.modules.items() and the use
of module.__name__).

Committed to the trunk - I'll backport tomorrow night unless
Neal objects.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2006-09-07 16:58

Message:
Logged In: YES 
user_id=33168

I had the same question as pje wrt to his second comment
about module.__name__ (assuming this is the same patch
posted to python-dev).  I would rather use the version
currently checked in to minimize change and maximize the
testing.  Please add a comments describing why one is chosen
rather than the other.  Also adding a comment in the code
about pje's items comment below would be nice.

Is this the attached version the same one Ralf (or whoever)
said fixed the problem?

----------------------------------------------------------------------

Comment By: Phillip J. Eby (pje)
Date: 2006-09-06 23:49

Message:
Logged In: YES 
user_id=56214

I just looked at the current patch; please don't use
sys.modules.iteritems() in your search loop; it'll break if
touching a module causes another module to be imported. 
This can happen if somebody is using one of the many "lazy
import" tools out there.  Use .items() instead.

Second, there was a reason that the loop used
module.__name__ instead of the sys.modules keys; a module
can be listed under more than one sys.modules key, so the
loop was storing the *canonical* name of the module.  I'm
not 100% positive this is a critical thing, but I'm
suspicious of the idea of just randomly choosing which of
the possible keys should be used.


----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2006-09-06 23:04

Message:
Logged In: YES 
user_id=1038590

Bah. The linecache version has the same problem as the
previous version - it's the failure case that searches the
whole of sys.modules that is slow, not the success case.

Patch attached which stores a mapping from module names to
module.__file__ values and skips the absolute lookup if the
value of module.__file__ hasn't changed.



----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2006-09-06 22:48

Message:
Logged In: YES 
user_id=1038590

Low impact approach that simply checks the linecache
directly before doing anything.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1553314&group_id=5470