From fredrik@effbot.org  Thu Feb  1 16:54:04 2001
From: fredrik@effbot.org (Fredrik Lundh)
Date: Thu, 1 Feb 2001 17:54:04 +0100
Subject: [Import-sig] Imputil
References: <00c301c08af2$19e85870$e000a8c0@thomasnotebook> <3A782042.B845D04E@lemburg.com> <007d01c08b95$77d241f0$e000a8c0@thomasnotebook> <3A7829C3.488DE50A@lemburg.com>
Message-ID: <01aa01c08c6f$98ce0170$e46940d5@hagrid>

M.-A. Lemburg wrote:
> Changing cPickle.c doesn't help here: the PyImport_ImportModule()
> API would have to be made __import__ aware to fix this problem
> because a lot of Python code including the core itself uses
> this more direct import API.

any reason cPickle.c cannot just use PyImport_Import
instead of PyImport_ImportModule?

</F>



From mal@lemburg.com  Thu Feb  1 17:11:30 2001
From: mal@lemburg.com (M.-A. Lemburg)
Date: Thu, 01 Feb 2001 18:11:30 +0100
Subject: [Import-sig] Imputil
References: <00c301c08af2$19e85870$e000a8c0@thomasnotebook> <3A782042.B845D04E@lemburg.com> <007d01c08b95$77d241f0$e000a8c0@thomasnotebook> <3A7829C3.488DE50A@lemburg.com> <01aa01c08c6f$98ce0170$e46940d5@hagrid>
Message-ID: <3A7998C2.32BE87B@lemburg.com>

Fredrik Lundh wrote:
> 
> M.-A. Lemburg wrote:
> > Changing cPickle.c doesn't help here: the PyImport_ImportModule()
> > API would have to be made __import__ aware to fix this problem
> > because a lot of Python code including the core itself uses
> > this more direct import API.
> 
> any reason cPickle.c cannot just use PyImport_Import
> instead of PyImport_ImportModule?

Sure, but what about all the other instances where 
PyImport_ImportModule() is used instead of PyImport_Import() ?

Just fixing cPickle.c won't help much -- we'd need a more generic
change to make Python fully imputil aware. Perhaps we could
redirect PyImport_ImportModule() to PyImport_Import() with some
extra logic to make it work during the interpreter startupo phase
too.

--

BTW, while looking around in Modules/ I found some really old
code in cStringIO.c which should probably be updated to the
new string methods:

static char O_writelines__doc__[] =
"writelines(sequence_of_strings): write each string";
static PyObject *
O_writelines(Oobject *self, PyObject *args) {
        PyObject *tmp = 0;
        static PyObject *string_joinfields = 0;

        UNLESS (PyArg_ParseTuple(args, "O:writelines", &args)) return NULL;

        if (!string_joinfields) {
                UNLESS (tmp = PyImport_ImportModule("string")) return NULL;
                string_joinfields=PyObject_GetAttrString(tmp, "joinfields");
                Py_DECREF(tmp);
                UNLESS (string_joinfields) return NULL;
        }

        if (PyObject_Size(args) < 0) return NULL;

        tmp = PyObject_CallFunction(string_joinfields, "Os", args, "");
        UNLESS (tmp) return NULL;

        args = Py_BuildValue("(O)", tmp);
        Py_DECREF(tmp);
        UNLESS (args) return NULL;

        tmp = O_write(self, args);
        Py_DECREF(args);
        return tmp;
}

-- 
Marc-Andre Lemburg
______________________________________________________________________
Company:                                        http://www.egenix.com/
Consulting:                                    http://www.lemburg.com/
Python Pages:                           http://www.lemburg.com/python/


From gmcm@hypernet.com  Thu Feb  1 17:25:07 2001
From: gmcm@hypernet.com (Gordon McMillan)
Date: Thu, 1 Feb 2001 12:25:07 -0500
Subject: [Import-sig] Imputil
In-Reply-To: <3A7829C3.488DE50A@lemburg.com>
Message-ID: <3A7955A3.18861.7A8B63D9@localhost>

[Thomas Heller]
> > > > cPickle uses PyImport_ImportModule(name) instead of
> > > > PyImport_Import(name) which would then in turn use the
> > > > import hook function. cPickle imports copy_reg, string,
> > > > types.
[MAL]
> > > The various uses of PyImport_ImportModule() in cPickle are
> > > only used for standard modules -- I don't think these pose
> > > much of a problem w/r to imputil and other import hooks,
> > > since these are either available using the standard import
> > > mechanism or as frozen modules compiled into the core.
[TH]
> > No, these are exactly the problem (maybe only my problem?). If
> > the standard import mechanism is NOT available (think of
> > Gordon's installer or my py2exe project) because everything is
> > loaded from an archive, importing cPickle will fail if you do:
[MAL]
> Perhaps you should add some "boot" code to py2exe then or
> freeze the standard lib into the interpreter itself ?!
> (mxCGIPython works this way.)

Freezing them all in bloats the exe by a meg or more. Putting 
them in a compressed archive gets it down to 500K or so. 
Just archiving the ones you need makes it even smaller.
[MAL]
> Changing cPickle.c doesn't help here: the PyImport_ImportModule()
> API would have to be made __import__ aware to fix this problem
> because a lot of Python code including the core itself uses this
> more direct import API.

PyImport_Import() is already __import__ aware. Other than a 
(misguided) thirst for speed there's very few excuses for using 
PyImport_ImportModule() instead.
 
> OTOH, it may sometimes be necessary to explicitly use the
> direct and builtin import mechanism (e.g. during Python
> startup)... 

This can be a valid reason, but not for cPickle...

- Gordon


From guido@digicool.com  Fri Feb  9 05:58:33 2001
From: guido@digicool.com (Guido van Rossum)
Date: Fri, 09 Feb 2001 00:58:33 -0500
Subject: [Import-sig] Imputil
In-Reply-To: Your message of "Thu, 01 Feb 2001 12:25:07 EST."
 <3A7955A3.18861.7A8B63D9@localhost>
References: <3A7955A3.18861.7A8B63D9@localhost>
Message-ID: <200102090558.AAA16511@cj20424-a.reston1.va.home.com>

OK, I see that there's some convergence here.  It appears that most
uses of PyImport_ImportModule() should really be using
PyImport_Import() instead.  There are two solutions: change the body
of PyImport_ImportModule() to call PyImport_Import(), or change all
(most?) calls to PyImport_ImportModule() to use PyImport_Import()
instead.

I'd vote for the former -- it's less work and more effective.  Note
that those places that really need to use the built-in import
primitive (e.g. the default built-in __import__ function) can use
PyImport_ImportModuleEx().  __import__ already does this.

Somebody submit a patch (or just check it in if you're feeling very
confident :-), please...

--Guido van Rossum (home page: http://www.python.org/~guido/)


From thomas.heller@ion-tof.com  Fri Feb  9 13:57:29 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 9 Feb 2001 14:57:29 +0100
Subject: [Import-sig] Imputil
References: <3A7955A3.18861.7A8B63D9@localhost>  <200102090558.AAA16511@cj20424-a.reston1.va.home.com>
Message-ID: <017501c092a0$3ded04e0$e000a8c0@thomasnotebook>

Guido writes:
> OK, I see that there's some convergence here.  It appears that most
> uses of PyImport_ImportModule() should really be using
> PyImport_Import() instead.  There are two solutions: change the body
> of PyImport_ImportModule() to call PyImport_Import(), or change all
> (most?) calls to PyImport_ImportModule() to use PyImport_Import()
> instead.
> 
> I'd vote for the former -- it's less work and more effective.  Note
> that those places that really need to use the built-in import
> primitive (e.g. the default built-in __import__ function) can use
> PyImport_ImportModuleEx().  __import__ already does this.
> 
> Somebody submit a patch (or just check it in if you're feeling very
> confident :-), please...
> 
Done so (submitted the patch, not checked in)

http://sourceforge.net/patch/?func=detailpatch&patch_id=103710&group_id=5470

Thomas



From thomas.heller@ion-tof.com  Fri Feb  9 20:12:37 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 9 Feb 2001 21:12:37 +0100
Subject: [Import-sig] Another imputil problem
Message-ID: <045b01c092d4$a584cdc0$e000a8c0@thomasnotebook>

Dear import-sig, I encountered two cases where imputil behaves
different than 'normal python'.

xml/__init__.py contains the following code fragment:

try:
    import _xmlplus
except ImportError:
    pass
else:
    try:
        v = _xmlplus.version_info
    except AttributeError:
        # _xmlplue is too old; ignore it
        pass
    else:
        if v >= _MINIMUM_XMLPLUS_VERSION:
            import sys
            sys.modules[__name__] = _xmlplus
        else:
            del v

when you have PyXML installed and execute
>>>import xml; print xml
you get
  <module '_xmlplus' from 'c:\python20\_xmlplus\__init__.pyc'>

If you try
>>>import imputil; imputil._test_revamp()
>>>import xml; print xml
you get
  <module 'xml' from 'c:\python20\lib\xml\__init__.pyc'>
and
>>>del sys; import sys; print sys.modules['xml']
prints
  <module '_xmlplus' from 'c:\python20\_xmlplus\__init__.pyc'>

IMO this should be fixed. I can try to prepare a patch for this.

Thomas



From thomas.heller@ion-tof.com  Fri Feb  9 21:04:45 2001
From: thomas.heller@ion-tof.com (Thomas Heller)
Date: Fri, 9 Feb 2001 22:04:45 +0100
Subject: [Import-sig] Another imputil problem
References: <045b01c092d4$a584cdc0$e000a8c0@thomasnotebook>
Message-ID: <04cf01c092db$f25fc670$e000a8c0@thomasnotebook>

This seems to fix the problem:

*** imputil.py~ Thu Sep 28 05:10:32 2000
--- imputil.py Fri Feb 09 21:02:53 2001
***************
*** 280,286 ****
          if not is_module:
              exec code in module.__dict__
  
!         return module
  
      def _load_tail(self, m, parts):
          """Import the rest of the modules, down from the top-level module.
--- 280,287 ----
          if not is_module:
              exec code in module.__dict__
  
!         # fetch from sys.modules instead of returning module directly.
!         return sys.modules[fqname]
  
      def _load_tail(self, m, parts):
          """Import the rest of the modules, down from the top-level module.




From fdrake@acm.org  Fri Feb  9 21:03:14 2001
From: fdrake@acm.org (Fred L. Drake, Jr.)
Date: Fri, 9 Feb 2001 16:03:14 -0500 (EST)
Subject: [Import-sig] Another imputil problem
In-Reply-To: <04cf01c092db$f25fc670$e000a8c0@thomasnotebook>
References: <045b01c092d4$a584cdc0$e000a8c0@thomasnotebook>
 <04cf01c092db$f25fc670$e000a8c0@thomasnotebook>
Message-ID: <14980.23314.490871.258808@cj42289-a.reston1.va.home.com>

Thomas Heller writes:
 > This seems to fix the problem:

  This is also what the built-in import machinery does; I see no
reason not to check it in.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>
PythonLabs at Digital Creations



From guido@digicool.com  Fri Feb  9 21:26:11 2001
From: guido@digicool.com (Guido van Rossum)
Date: Fri, 09 Feb 2001 16:26:11 -0500
Subject: [Import-sig] Another imputil problem
In-Reply-To: Your message of "Fri, 09 Feb 2001 16:03:14 EST."
 <14980.23314.490871.258808@cj42289-a.reston1.va.home.com>
References: <045b01c092d4$a584cdc0$e000a8c0@thomasnotebook> <04cf01c092db$f25fc670$e000a8c0@thomasnotebook>
 <14980.23314.490871.258808@cj42289-a.reston1.va.home.com>
Message-ID: <200102092126.QAA23751@cj20424-a.reston1.va.home.com>

> Thomas Heller writes:
>  > This seems to fix the problem:
> 
>   This is also what the built-in import machinery does; I see no
> reason not to check it in.

Agreed.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From gstein@lyra.org  Sat Feb 10 00:39:58 2001
From: gstein@lyra.org (Greg Stein)
Date: Fri, 9 Feb 2001 16:39:58 -0800
Subject: [Import-sig] Another imputil problem
In-Reply-To: <04cf01c092db$f25fc670$e000a8c0@thomasnotebook>; from thomas.heller@ion-tof.com on Fri, Feb 09, 2001 at 10:04:45PM +0100
References: <045b01c092d4$a584cdc0$e000a8c0@thomasnotebook> <04cf01c092db$f25fc670$e000a8c0@thomasnotebook>
Message-ID: <20010209163958.N26044@lyra.org>

Makes sense.

+1

[ damn self-modifying modules... :-) ]

Cheers,
-g

On Fri, Feb 09, 2001 at 10:04:45PM +0100, Thomas Heller wrote:
> This seems to fix the problem:
> 
> *** imputil.py~ Thu Sep 28 05:10:32 2000
> --- imputil.py Fri Feb 09 21:02:53 2001
> ***************
> *** 280,286 ****
>           if not is_module:
>               exec code in module.__dict__
>   
> !         return module
>   
>       def _load_tail(self, m, parts):
>           """Import the rest of the modules, down from the top-level module.
> --- 280,287 ----
>           if not is_module:
>               exec code in module.__dict__
>   
> !         # fetch from sys.modules instead of returning module directly.
> !         return sys.modules[fqname]
>   
>       def _load_tail(self, m, parts):
>           """Import the rest of the modules, down from the top-level module.
> 
> 
> 
> _______________________________________________
> Import-sig mailing list
> Import-sig@python.org
> http://mail.python.org/mailman/listinfo/import-sig

-- 
Greg Stein, http://www.lyra.org/