From barry@zope.com  Mon Jul  8 21:14:33 2002
From: barry@zope.com (Barry A. Warsaw)
Date: Mon, 8 Jul 2002 16:14:33 -0400
Subject: [Import-sig] SIG charters
Message-ID: <15657.62121.520364.556758@anthem.wooz.org>

Many of the sig deadlines have expired.  See www.python.org/sigs

Of these, the following appear to have no (or not much) meaningful
recent activity:

    do-sig
    import-sig
    types-sig

I propose to retire these sigs and extend all others for another year.

Objections?  <ducking>
-Barry



From guido@python.org  Tue Jul  9 21:50:59 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 09 Jul 2002 16:50:59 -0400
Subject: [Import-sig] Re: [meta-sig] SIG charters
In-Reply-To: Your message of "Mon, 08 Jul 2002 16:14:33 EDT."
 <15657.62121.520364.556758@anthem.wooz.org>
References: <15657.62121.520364.556758@anthem.wooz.org>
Message-ID: <200207092050.g69KoxW04101@odiug.zope.com>

> Many of the sig deadlines have expired.  See www.python.org/sigs
> 
> Of these, the following appear to have no (or not much) meaningful
> recent activity:
> 
>     do-sig
>     import-sig
>     types-sig
> 
> I propose to retire these sigs and extend all others for another year.

+1

--Guido van Rossum (home page: http://www.python.org/~guido/)



From anthony@computronix.com  Fri Jul 12 22:00:23 2002
From: anthony@computronix.com (Anthony Tuininga)
Date: 12 Jul 2002 15:00:23 -0600
Subject: [Import-sig] Imports with hooks not thread safe?
Message-ID: <1026507627.20941.146.camel@chl0151.edmonton.computronix.com>

I just joined this list so my apologies if this has been discussed
before. The list (from the archives) seems close to dead with a
suggestion to kill it permanently.... :-)

I noticed that in PyImport_ImportModuleEx, that a lock is held until the
import is completed or an exception is raised. The only problem is that
the higher level interface PyImport_Import does __NOT__ hold the lock
and this is the method that is used for hooks. The builtin function
__import__ calls PyImport_ImportModuleEx but if that function is
replaced with a call to a hook, no locking takes place which makes the
import mechanism not thread safe.

My question is: would there be a problem with holding the lock during
PyImport_Import? If there is, would there be a problem with exposing a
method that would do the import lock? The only thing that is currently
available is imp.lock_held() but that simply states whether or not the
import lock is held and nothing that I could see provides any mechanism
for acquiring the import lock.

I am using Installer (from Gordon McMillan) and it is using its own
locking mechanism which seems to break inside PythonCOM (from Mark
Hammond) if a large enough quantity of threads are executing. All of my
imports are done within the import of the main module which is done
using the normal method but I would like to solve this problem in a more
permanent way as otherwise I will have to fork Installer for our own
internal use. Blech.

-- 
Anthony Tuininga
anthony@computronix.com
 
Computronix
Distinctive Software. Real People.
Suite 200, 10216 - 124 Street NW
Edmonton, AB, Canada  T5N 4A3
Phone:	(780) 454-3700
Fax:	(780) 454-3838
http://www.computronix.com




From gmcm@hypernet.com  Sat Jul 13 14:17:10 2002
From: gmcm@hypernet.com (Gordon McMillan)
Date: Sat, 13 Jul 2002 09:17:10 -0400
Subject: [Import-sig] Imports with hooks not thread safe?
In-Reply-To: <1026507627.20941.146.camel@chl0151.edmonton.computronix.com>
Message-ID: <3D2FF016.2985.99BDA4A@localhost>

Anthony,

I've created a bug report on SF:
 https://sourceforge.net/tracker/index.php?func=detail&aid=580952&group_id=5470&atid=105470

and assigned it to Tim for initial review. I'm
suggesting that the imp module should grow
acquire_lock and release_lock so hooks can
do this right.

-- Gordon
http://www.mcmillan-inc.com/



On 12 Jul 2002 at 15:00, Anthony Tuininga wrote:

> I just joined this list so my apologies if this has
> been discussed before. The list (from the archives)
> seems close to dead with a suggestion to kill it
> permanently.... :-)
> 
> I noticed that in PyImport_ImportModuleEx, that a lock
> is held until the import is completed or an exception
> is raised. The only problem is that the higher level
> interface PyImport_Import does __NOT__ hold the lock
> and this is the method that is used for hooks. The
> builtin function __import__ calls
> PyImport_ImportModuleEx but if that function is
> replaced with a call to a hook, no locking takes place
> which makes the import mechanism not thread safe.
> 
> My question is: would there be a problem with holding
> the lock during PyImport_Import? If there is, would
> there be a problem with exposing a method that would do
> the import lock? The only thing that is currently
> available is imp.lock_held() but that simply states
> whether or not the import lock is held and nothing that
> I could see provides any mechanism for acquiring the
> import lock.
> 
> I am using Installer (from Gordon McMillan) and it is
> using its own locking mechanism which seems to break
> inside PythonCOM (from Mark Hammond) if a large enough
> quantity of threads are executing. All of my imports
> are done within the import of the main module which is
> done using the normal method but I would like to solve
> this problem in a more permanent way as otherwise I
> will have to fork Installer for our own internal use.
> Blech.
> 
> -- 
> Anthony Tuininga
> anthony@computronix.com
> 
> Computronix
> Distinctive Software. Real People.
> Suite 200, 10216 - 124 Street NW
> Edmonton, AB, Canada  T5N 4A3
> Phone:	(780) 454-3700
> Fax:	(780) 454-3838
> http://www.computronix.com
> 
> 
> 
> _______________________________________________
> Import-sig mailing list
> Import-sig@python.org
> http://mail.python.org/mailman/listinfo/import-sig





From tuiningas@shaw.ca  Sun Jul 14 03:55:48 2002
From: tuiningas@shaw.ca (Anthony & Janet Tuininga)
Date: Sat, 13 Jul 2002 20:55:48 -0600
Subject: [Import-sig] Imports with hooks not thread safe?
In-Reply-To: <3D2FF016.2985.99BDA4A@localhost>
References: <3D2FF016.2985.99BDA4A@localhost>
Message-ID: <1026615349.3786.4.camel@localhost>

Thanks. But wouldn't it make more sense to acquire the lock around
__any__ import, regardless of whether it was hooked or not? Or would
that become too inflexible? Or would that cause potential problems with
existing code? I am really all that familiar with the core Python code
but that makes the most sense to me. Would you be able to give me some
insight? I agree that the exposing of the import lock would also solve
the problem but that would mean that all the import hooks would have to
do this in order to be thread safe. Or could it be placed in the core
imputils.py in the base class? Comments?

Anthony

On Sat, 2002-07-13 at 07:17, Gordon McMillan wrote:
> Anthony,
> 
> I've created a bug report on SF:
>  https://sourceforge.net/tracker/index.php?func=detail&aid=580952&group_id=5470&atid=105470
> 
> and assigned it to Tim for initial review. I'm
> suggesting that the imp module should grow
> acquire_lock and release_lock so hooks can
> do this right.
> 
> -- Gordon
> http://www.mcmillan-inc.com/
> 
> 
> 
> On 12 Jul 2002 at 15:00, Anthony Tuininga wrote:
> 
> > I just joined this list so my apologies if this has
> > been discussed before. The list (from the archives)
> > seems close to dead with a suggestion to kill it
> > permanently.... :-)
> > 
> > I noticed that in PyImport_ImportModuleEx, that a lock
> > is held until the import is completed or an exception
> > is raised. The only problem is that the higher level
> > interface PyImport_Import does __NOT__ hold the lock
> > and this is the method that is used for hooks. The
> > builtin function __import__ calls
> > PyImport_ImportModuleEx but if that function is
> > replaced with a call to a hook, no locking takes place
> > which makes the import mechanism not thread safe.
> > 
> > My question is: would there be a problem with holding
> > the lock during PyImport_Import? If there is, would
> > there be a problem with exposing a method that would do
> > the import lock? The only thing that is currently
> > available is imp.lock_held() but that simply states
> > whether or not the import lock is held and nothing that
> > I could see provides any mechanism for acquiring the
> > import lock.
> > 
> > I am using Installer (from Gordon McMillan) and it is
> > using its own locking mechanism which seems to break
> > inside PythonCOM (from Mark Hammond) if a large enough
> > quantity of threads are executing. All of my imports
> > are done within the import of the main module which is
> > done using the normal method but I would like to solve
> > this problem in a more permanent way as otherwise I
> > will have to fork Installer for our own internal use.
> > Blech.
> > 
> > -- 
> > Anthony Tuininga
> > anthony@computronix.com
> > 
> > Computronix
> > Distinctive Software. Real People.
> > Suite 200, 10216 - 124 Street NW
> > Edmonton, AB, Canada  T5N 4A3
> > Phone:	(780) 454-3700
> > Fax:	(780) 454-3838
> > http://www.computronix.com
> > 
> > 
> > 
> > _______________________________________________
> > Import-sig mailing list
> > Import-sig@python.org
> > http://mail.python.org/mailman/listinfo/import-sig
> 






From gmcm@hypernet.com  Sun Jul 14 14:58:26 2002
From: gmcm@hypernet.com (Gordon McMillan)
Date: Sun, 14 Jul 2002 09:58:26 -0400
Subject: [Import-sig] Imports with hooks not thread safe?
In-Reply-To: <1026615349.3786.4.camel@localhost>
References: <3D2FF016.2985.99BDA4A@localhost>
Message-ID: <3D314B42.16550.EE7FF7B@localhost>

On 13 Jul 2002 at 20:55, Anthony Tuininga wrote:

> Thanks. But wouldn't it make more sense to acquire the
> lock around __any__ import, regardless of whether it
> was hooked or not? 

You can't. Hooks typically *replace*
builtin.__import__. Imports from Python code
(where import has been hooked) don't necessarily touch
anything in the builtin Python import machinery.

[...]

> I agree that the exposing of the import lock would
> also solve the problem but that would mean that all
> the import hooks would have to do this in order to
> be thread safe. 

Correct. Writing a good import hook isn't easy.
Most import hooks are specializations or extensions
of behavior in some little area, so can get away with
it.

> Or could it be placed in the core
> imputils.py in the base class? Comments? 

iu is a replacement for imputils. imputils has a
number of flaws - no import lock; it does not
put None in sys.modules when the "is this a
relative import?" test fails. Can't recall what else,
but that's why I wrote iu.

-- Gordon
http://www.mcmillan-inc.com/




From anthony@computronix.com  Mon Jul 15 15:47:11 2002
From: anthony@computronix.com (Anthony Tuininga)
Date: 15 Jul 2002 08:47:11 -0600
Subject: [Import-sig] Imports with hooks not thread safe?
In-Reply-To: <3D314B42.16550.EE7FF7B@localhost>
References: <3D2FF016.2985.99BDA4A@localhost>
 <3D314B42.16550.EE7FF7B@localhost>
Message-ID: <1026744435.32240.15.camel@chl0151.edmonton.computronix.com>

On Sun, 2002-07-14 at 07:58, Gordon McMillan wrote:
> On 13 Jul 2002 at 20:55, Anthony Tuininga wrote:
> 
> > Thanks. But wouldn't it make more sense to acquire the
> > lock around __any__ import, regardless of whether it
> > was hooked or not? 
> 
> You can't. Hooks typically *replace*
> builtin.__import__. Imports from Python code
> (where import has been hooked) don't necessarily touch
> anything in the builtin Python import machinery.

Ok but please bear with me a moment while I struggle to understand
this....

If I look at the C code for Python (Python/import.c), any code written
in C uses PyImport_Import if it wants to invoke the import hooks that
have been set up and PyImport_ImportModuleEx if it wants to "go direct"
so to speak. Technically, the lock could be acquired and released within
PyImport_Import and this would solve __ONE__ method for importing
modules. PyImport_Import calls the __import__ method of the current
globals which is how hooks work, right?

I then found __ANOTHER__ location (Python/ceval.c) where imports are
done. This one is what is actually used for import statements found
within Python itself. It calls the __import__ method of the current
globals directly without bothering to call anything in import.c. Now
perhaps these could be reconciled, but if not, the import lock could be
acquired and released here as well. Right? Or absolutely wrong???

> [...]
> 
> > I agree that the exposing of the import lock would
> > also solve the problem but that would mean that all
> > the import hooks would have to do this in order to
> > be thread safe. 
> 
> Correct. Writing a good import hook isn't easy.
> Most import hooks are specializations or extensions
> of behavior in some little area, so can get away with
> it.

So wouldn't it be better to make this simpler? Or would making it
simpler just cause more grief in the long run....? :-)

> > Or could it be placed in the core
> > imputils.py in the base class? Comments? 
> 
> iu is a replacement for imputils. imputils has a
> number of flaws - no import lock; it does not
> put None in sys.modules when the "is this a
> relative import?" test fails. Can't recall what else,
> but that's why I wrote iu.

So wouldn't it be better to fix imputils.py to do this right? Or is it
that much of a bother to get changes to core Python?

> 
> -- Gordon
> http://www.mcmillan-inc.com/
> 
> 
> 
> _______________________________________________
> Import-sig mailing list
> Import-sig@python.org
> http://mail.python.org/mailman/listinfo/import-sig
-- 
Anthony Tuininga
anthony@computronix.com
 
Computronix
Distinctive Software. Real People.
Suite 200, 10216 - 124 Street NW
Edmonton, AB, Canada  T5N 4A3
Phone:	(780) 454-3700
Fax:	(780) 454-3838
http://www.computronix.com




From anthony@computronix.com  Mon Jul 15 17:25:11 2002
From: anthony@computronix.com (Anthony Tuininga)
Date: 15 Jul 2002 10:25:11 -0600
Subject: [Import-sig] Bug Fix for Installer
Message-ID: <1026750316.32209.29.camel@chl0151.edmonton.computronix.com>

Well, after a little more sleuthing I discovered the source of the
problem with the locking within the import hooks in Installer.

The _release function needs to have one line added:

    def _acquire(self):
        if self.rlock.locked():
            if self.locker == self._get_ident():
                self.lockcount = self.lockcount + 1
##                print "_acquire incrementing lockcount to",
self.lockcount
                return
        self.rlock.acquire()
        self.locker = self._get_ident()
        self.lockcount = 0
##        print "_acquire first time!"


    def _release(self):
        if self.lockcount:
            self.lockcount = self.lockcount - 1
##            print "_release decrementing lockcount to", self.lockcount
        else:
#---------------------------------------------------------------------------
# THIS LINE WAS ADDED (BELOW)
            self.locker = None
# THIS LINE WAS ADDED (ABOVE)
#---------------------------------------------------------------------------
            self.rlock.release()
##            print "_release releasing lock!"

Without that line, the following scenario can occur.....

1) Thread 1 releases the lock
2) Thread 2 acquires the lock but does __NOT__ set who the waiter is (it
loses control before it can set the locker)
3) Thread 1 sees that the lock is acquired and thinks that it still owns
the lock so simply increments the lock count
4) Thread 2 sets the locker and the lockcount to 0
5) Thread 1 releases the lock -- OOPS!

If this doesn't make any sense to you, fire away! This (in my opinion)
is a really good reason for why the import lock needs to be held in core
Python in order to avoid this sort of problem. Multi-threaded
applications are a pain to deal with!

On another note, a slight improvement in performance could likely be
seen by checking sys.modules __before__ acquiring the lock and simply
returning. Note that sys.modules will have to be checked again after the
acquisition of the lock since some other thread may have imported it
while this thread was waiting for the lock to be acquired. Any comments?

Gordon, any thoughts on when this could be integrated into Installer?

-- 
Anthony Tuininga
anthony@computronix.com
 
Computronix
Distinctive Software. Real People.
Suite 200, 10216 - 124 Street NW
Edmonton, AB, Canada  T5N 4A3
Phone:	(780) 454-3700
Fax:	(780) 454-3838
http://www.computronix.com




From jwaterman@lcs.mit.edu  Fri Jul 26 20:28:10 2002
From: jwaterman@lcs.mit.edu (Jason Waterman)
Date: Fri, 26 Jul 2002 15:28:10 -0400
Subject: [Import-sig] imputil with only .pyc files
Message-ID: <15681.41674.734898.786114@catfish.lcs.mit.edu>

I've been using imputil as the base for importing python modules over
http.  I've noticed that the builtin importer imputil.BuiltinImporter
doesn't import .pyc files if there is no corresponding .py file.  I've
added my own suffix handler for just .pyc files, but I'm wondering if
this is a bug or a feature?

We're using python on the ipaqs, so we've deleted the .py files to
save space.

Thanks,
Jason



From guido@python.org  Fri Jul 26 21:52:01 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 26 Jul 2002 16:52:01 -0400
Subject: [Import-sig] imputil with only .pyc files
In-Reply-To: Your message of "Fri, 26 Jul 2002 15:28:10 EDT."
 <15681.41674.734898.786114@catfish.lcs.mit.edu>
References: <15681.41674.734898.786114@catfish.lcs.mit.edu>
Message-ID: <200207262052.g6QKq1W23965@pcp02138704pcs.reston01.va.comcast.net>

> I've been using imputil as the base for importing python modules over
> http.  I've noticed that the builtin importer imputil.BuiltinImporter
> doesn't import .pyc files if there is no corresponding .py file.  I've
> added my own suffix handler for just .pyc files, but I'm wondering if
> this is a bug or a feature?

Sounds like a feature that's become a bug. :-(

> We're using python on the ipaqs, so we've deleted the .py files to
> save space.

Of course.

Can you submit a patch to sourceforge?

--Guido van Rossum (home page: http://www.python.org/~guido/)