[Import-SIG] My objections to implicit package directories

[Calvin Spealman]

On Thu, Mar 15, 2012 at 1:50 PM, Guido van Rossum <guido at python.org> wrote:
> On Thu, Mar 15, 2012 at 4:26 AM, Calvin Spealman <ironfroggy at gmail.com> wrote:
>>
>> On Mar 14, 2012 11:56 PM, "Guido van Rossum" <guido at python.org> wrote:
>>>
>>> On Wed, Mar 14, 2012 at 8:33 PM, PJ Eby <pje at telecommunity.com> wrote:
>>> > On Mon, Mar 12, 2012 at 11:49 PM, Guido van Rossum <guido at python.org>
>>> > wrote:
>>> >>
>>> >> So the only backwards incompatibility is that "import foo" may succeed
>>> >> where it previously failed if there is a directory foo/ somewhere on
>>> >> sys.path but no foo.py and no foo/__init__.py anywhere. I don't think
>>> >> this is a big deal.
>>> >
>>> >
>>> > Actually, it *is* a big deal.  An early draft of PEP 402 was like the
>>> > newly-proposed approach, and it was shot down by an *actual code sample
>>> > from
>>> > a real package* that somebody posted to show the problem.  The
>>> > motivating
>>> > example is actually presented in the PEP: trying to "import json" with a
>>> > json/ directory present -- and the "json" package *not* present -- and
>>> > then
>>> > trying to use its contents.  The PEP works around this by taking the
>>> > namespace concept to its logical extreme and saying you can only import
>>> > the
>>> > *contents* of the namespace, because the namespace itself doesn't exist.
>>> >  (i.e., it's virtual.)
>>> >
>>> > tl;dr version: the killer problem with allowing "import foo" to succeed
>>> > is
>>> > that code which does try/except ImportError to test for a package's
>>> > presence
>>> > will get false positives.  That's the motivating reason for only
>>> > allowing
>>> > sub-namespace package imports to succeed: it prevents the immediate
>>> > breakage
>>> > of real code, the moment the feature is introduced.  :-(
>>>
>>> Well, too bad. It's too much of a wart. Such try/except clauses make
>>> me sad anyway -- and we're trying to get rid of them by e.g.
>>> encouraging the pattern where heapq.py tries to import _heapq rather
>>> than having user code try first _heapq and then heapq. (And the idiom
>>> used in heapq.py doesn't mind if _heapq exists as an empty package.)
>>
>> While I agree it is a wart, the heapq.py example might not be the best. A
>> more real-world example might be the situation is PEP 417, where we'll be
>> seeing people write
>>
>> try:
>>     import unittest.mock
>> except ImportError:
>>     import mock
>>
>> And owning neither the two modules, the _ trick is not available.
>>
>> I like the idea that if a package is only an implied namespace package, with
>> no contents of it's own, it cannot itself be directly imported. This still
>> allows the simple check, but without seriously modifying the new feature
>> itself.
>
> How would you implement that anyway? The import logic always tries to
> import the parent module before importing the child module. So the
> import attempt for "foo" has no idea whether it is imported as *part*
> of "import foo.bar", or as plain "import foo", or perhaps as part of
> "from foo import bar".
>
> It would also be odd to find that
>
>  import foo
>  import foo.bar
>
> would fail, whereas
>
>  import foo.bar
>  import foo
>

I would prefer neither `import foo` succeed, in fact, if foo is only
an implied namespace package. The import machinery could treat it the
same, up to the point that it would bind the name, where it would
check if it was such an implied package and raise an appropriate
exception. Maybe some ImportError subclass? So, this would not be in
an error in the loading portion of the import, but in adding it to the
namespace where the import statement was executed.

> would succeed, because as a side effect of "import foo.bar", a module
> object for foo is created and inserted as sys.modules['foo'].
>
> I really don't want to have to maintain and explain such complexity --
> nor the complexity that follows from attempts to "fix" this objection.
>
> Finally, in your example, why on earth would unittest/mock/ exist as
> an empty directory???

It was only meant as an example where the try/except for import
fallbacks is natural and wise. It seemed your previous statements were
calling all such things a wart.

> --
> --Guido van Rossum (python.org/~guido)



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy