[ python-Bugs-1695718 ] PEP 302 broken

[SourceForge.net]

Bugs item #1695718, was opened at 2007-04-06 09:32
Message generated for change (Comment added) made by bcannon
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1695718&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.5
Status: Closed
Resolution: Wont Fix
Priority: 6
Private: No
Submitted By: phil (philipdumont)
Assigned to: Georg Brandl (gbrandl)
Summary: PEP 302 broken

Initial Comment:
The product I'm working on uses a PEP 302 importer hook.  It's a home-grown ltihooks clone.  (We don't use ltihooks because it's GPLed, and we don't want to be.)

Our importer worked on 2.4.3, is broken on 2.5.0.

What's broken is that if the hook's find_module() method returns None for a given module name, say 'spam', then that is supposed to cause the import machinery to fall back on the regular unhooked import behavior -- that is, find 'spam.py' (or 'spam.pyc', 'spam.pyo') in the directory in question.  This used to be what happened, but no longer.

Tracing through the code, the problem seems to be occurring due to the 'continue' at line 1289 (in the 2.5 tarball) of Python/import.c.

Slogging through SVN (aside: this would have been easier if your ViewSVN supported annotate/blame -- any chance you could add that?), it appears that the offending continue statement was added in revision 46372, whose checkin comment claims that it was done for performance reasons.  I'm all for performance improvements, but not at the expense of breakage.

Attached is a tarball with some files that reproduce the problem.  (The LibtoolImporter.py file is a stripped-down toy version of what we are really using.)  Unwind the tarball, cd to the directory, and run script.py.  Here's what I get:



shell prompt> pwd
/home/phil/pep302_bug
shell prompt> ls -CF
eggs.la  LibtoolImporter.py  script.py*  spam.py
shell prompt> python2.4.3 script.py
.la file(s) found in /home/phil/pep302_bug, LibtoolImporter will be used.
LibtoolImporter.find_module() couldn't find spam.la or spammodule.la
in /home/phil/pep302_bug.  Returning None.  This is *supposed* to cause a
fallback to the default import code looking for spam.py
in /home/phil/pep302_bug
module spam loaded
shell prompt> python2.5 script.py
.la file(s) found in /home/phil/pep302_bug, LibtoolImporter will be used.
LibtoolImporter.find_module() couldn't find spam.la or spammodule.la
in /home/phil/pep302_bug.  Returning None.  This is *supposed* to cause a
fallback to the default import code looking for spam.py
in /home/phil/pep302_bug
Traceback (most recent call last):
  File "script.py", line 4, in <module>
    import spam
ImportError: No module named spam
shell prompt>

----------------------------------------------------------------------

>Comment By: Brett Cannon (bcannon)
Date: 2007-04-09 10:42

Message:
Logged In: YES 
user_id=357491
Originator: NO

I am planning to work on the import machinery in general for this year so
hopefully things will improve.

----------------------------------------------------------------------

Comment By: phil (philipdumont)
Date: 2007-04-09 06:40

Message:
Logged In: YES 
user_id=1364034
Originator: YES

Ok.  I concede that I misread the PEP.  And I won't dispute the NO CHANGE
resolution.  But before I drop it entirely, I wish to make one more
comment.

>From PEP 302: "The hooks proposed in this PEP...would allow the
__import__-based tools to get rid of most of their import.c emulation
code."

This implementation change causes hooks that handle directories containing
both custom modules and "normal" modules to have to do some of that
import.c emulation.  Which is what I guess I will do.

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2007-04-08 05:58

Message:
Logged In: YES 
user_id=849994
Originator: NO

You're correct, Brett.

I found the implicit fallback at the Iceland sprint and we decided to
change
that in 2.5:

http://mail.python.org/pipermail/python-dev/2006-May/065174.html

PEP 302 has even been edited to say:

    [...] Note
    that if the callable returns an importer object for a specific
    sys.path entry, the builtin import machinery will not be invoked
    to handle that entry any longer, even if the importer object later
    fails to find a specific module.

Closing as "Won't fix", if more need for discussion arises, please take
this to python-dev.

----------------------------------------------------------------------

Comment By: Brett Cannon (bcannon)
Date: 2007-04-07 21:55

Message:
Logged In: YES 
user_id=357491
Originator: NO

I don't agree with this interpretation of PEP 302 for this instance.  If
you read the PEP there is no explicit mention that if an importer for an
entry on sys.path fails that it falls back on the default import behaviour.
 The only mention of using the default behaviour is if a value of None is
stored in sys.path_importer_cache (which also occurs when no entry on
sys.path_hooks returns a usable importer).

In my interpretation of PEP 302 (and how I implemented it for my pure
Python import implementation), if an importer exists for an entry on
sys.path then it essentially "owns" that entry.  The default import
semantics only kick in for unclaimed sys.path entries in my view.

Now I could be wrong and if I am I hope Phil can point out where in PEP
302 I am wrong.  Otherwise we have either undocumented behaviour that
changed (and that is always messy) or implied semantics that got fixed.

In other words, this probably needs to go to python-dev if Phil can't
point out exactly where PEP 302 says the semantics he is expecting is
supposed to be that way.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2007-04-06 12:21

Message:
Logged In: YES 
user_id=80475
Originator: NO

Georg, I believe this was your checkin.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1695718&group_id=5470