From Prabhu Ramachandran Sat Nov 10 08:30:25 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Sat, 10 Nov 2001 14:00:25 +0530 Subject: [Import-sig] Proposal for a modified import mechanism. Message-ID: <15340.58785.94412.122093@monster.linux.in> Hi, Sorry about the cross posting. Over the last couple of weeks I described a problem and inconsistency with the way Python imports modules. For more details look at this thread: http://mail.python.org/pipermail/python-list/2001-November/070719.html In short, currently, import allows one to use non-absolute module names for modules that are in the current directory. If a module is not in the same directory, import then looks for modules in sys.path. Consequently, dealing with packages that are re-nested is a pain. Complex package structure also causes problems. I'd like to note that I was also not the only person who suffered from this issue -- four others on mentioned similar problems and some asked me to let them know if I found a solution. Subsequently, I proposed another approach that first looks in the local directory and then walks up the current package tree looking for modules before looking at sys.path. I also modified knee.py to obtain a working solution. More information is here: http://mail.python.org/pipermail/python-list/2001-November/071212.html the threading is messed up and starts here: http://mail.python.org/pipermail/python-list/2001-November/071218.html You can find the new module and a simple test package here: http://av.stanford.edu/~prabhu/download/ There is also a slightly enhanced version of knee.py included that supports caching module lookup failures suggested by Rainer and Gordon in: http://mail.python.org/pipermail/python-list/2001-November/071218.html it also fixes a bug where the parent package is an extension module. I therefore have a working import style that seems to handle importing packages in a more natural(?) and consistent(?) manner. I've also tested it out with a large package like scipy (http://www.scipy.org) with no trouble or significant performance problems: http://mail.python.org/pipermail/python-list/2001-November/071325.html I'd like to ask the Python developers if they'd consider (a) changing the way the current import works to do what I proposed, or, (b) add a new keyword like 'rimport' (or something) that does this recursive search through parent packages and loads modules. This was actually suggested by Gordon McMillan. Gordon actually suggested something stronger -- import only supports absolute names, rimport is relative import and rrimport is a recursive relative import. But this would break the current import since import currently aupports some relative lookup. So maybe import and rimport is a workable solution? (c) patch the existing knee.py with my fixes. Note: these fixes have nothing to do with the recirsive module lookup stuff -- knee.py is merely an improved version of the older one. Thanks for listening patiently and sorry again for all the cross posting. prabhu From frederic.giacometti@arakne.com Sat Nov 10 18:13:47 2001 From: frederic.giacometti@arakne.com (Frederic Giacometti) Date: Sat, 10 Nov 2001 10:13:47 -0800 Subject: [Import-sig] Re: Proposal for a modified import mechanism. References: Message-ID: <3BED6E5B.8E142057@arakne.com> > > I'd like to ask the Python developers if they'd consider > > (a) changing the way the current import works to do what I > proposed, or, > > (b) add a new keyword like 'rimport' (or something) that does this > recursive search through parent packages and loads modules. This > was actually suggested by Gordon McMillan. Gordon actually > suggested something stronger -- import only supports absolute > names, rimport is relative import and rrimport is a recursive > relative import. But this would break the current import since > import currently aupports some relative lookup. So maybe import > and rimport is a workable solution? I'd rather introduce a __parent__ module attribute (in addition to the existing __name__) so that, for instance, the following would do your job: from __parent__.__parent__.toto import something In its spirit, this is similar to the '..' of the file systems. For top-level modules, __parent__ would be None. I'm personnally against anything that enlarges the search path uselessly; because the obvious reason of increased name space collision, increased run-time overhead etc... Frederic Giacometti From ej@ee.duke.edu Sat Nov 10 19:03:02 2001 From: ej@ee.duke.edu (eric) Date: Sat, 10 Nov 2001 14:03:02 -0500 Subject: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED6E5B.8E142057@arakne.com> Message-ID: <033301c16a1a$59c1c690$c300a8c0@ericlaptop> I have to agree with Prabhu on this one. The current behavior of import, while fine for standard modules and even simple packages with a single level, is sub-optimal for packages that contain sub-packages. The proposed behavior solves the problem. Handling the packaging issues in SciPy was difficult, and even resulted in a (not always popular) decision to build and overwrite the Numeric package on machines that install SciPy. Prabhu's import doesn't resolve all the issues (I think packages may just be difficult...), but it would have solved this one. The proposed import allows us to put our own version of Numeric in the top SciPy directory. Then all SciPy sub-packages would grab this one instead of an existing site-packages/Numeric. That makes SciPy self-contained and allows people to try it out without worrying that it might break their current installation. There are other solutions to this problem, but Prabhu's fix is by far the easiest and most robust. Prabhu's import also has some other nice benefits. Some of the sub-packages in SciPy are useful outside of SciPy. Also sometimes it is easier to develop a packages outside of the SciPy framework. It would be nice to be able to develop a module or package 'foo' outside of SciPy and then move it into SciPy at a later date. However, every SciPy sub-package that referred to foo prior to its inclusion in SciPy now has to be updated from 'import foo' to 'import scipy.foo'. These kind of issues make it very painful and time consuming to rearrange package structures or move modules and sub-packages in and out of the package. Simplifying this will improves package development. > I'm personnally against anything that enlarges the search path uselessly; Hopefully I've explained why it is useful for complex packages. > because the obvious reason of increased name space collision, increased > run-time overhead etc... I'm missing something here because I don't understand why this increases name space collision. If the objection is to the fact that SciPy can have a version of Numeric in it that masks a Numeric installed in site-packages, I guess I consider this a feature, not a bug. Afterall, this is already the behavior for single level packages, extending it to multi-level packages seems natural. If this isn't your objection, please explain. The current runtime overhead isn't so bad. Prabhu sent me a few numbers on the SciPy import (which contains maybe 10-15 nested packages). I attached them below -- the overhead is less than 10%. It should be negligible for standard modules as only packages are really affected (right Prabhu?). $ python >>> import time >>> s = time.time (); import scipy; print time.time()-s 1.37971198559 >>> $ python >>> import my_import >>> import time >>> s = time.time (); import scipy; print time.time()-s 1.48667407036 There may be technical issues under the covers that make this hairier than it appears, but, from the standpoint of someone working on a large multi-level package, it looks like a good idea. see ya, eric ----- Original Message ----- From: "Frederic Giacometti" To: Cc: ; ; Sent: Saturday, November 10, 2001 1:13 PM Subject: Re: Proposal for a modified import mechanism. > > > > > > I'd like to ask the Python developers if they'd consider > > > > (a) changing the way the current import works to do what I > > proposed, or, > > > > (b) add a new keyword like 'rimport' (or something) that does this > > recursive search through parent packages and loads modules. This > > was actually suggested by Gordon McMillan. Gordon actually > > suggested something stronger -- import only supports absolute > > names, rimport is relative import and rrimport is a recursive > > relative import. But this would break the current import since > > import currently aupports some relative lookup. So maybe import > > and rimport is a workable solution? > > I'd rather introduce a __parent__ module attribute (in addition to the > existing __name__) so that, for instance, the following would do your job: > > from __parent__.__parent__.toto import something > > In its spirit, this is similar to the '..' of the file systems. > > For top-level modules, __parent__ would be None. > > I'm personnally against anything that enlarges the search path uselessly; > because the obvious reason of increased name space collision, increased > run-time overhead etc... > > Frederic Giacometti > > > From frederic.giacometti@arakne.com Sat Nov 10 21:43:54 2001 From: frederic.giacometti@arakne.com (Frederic Giacometti) Date: Sat, 10 Nov 2001 13:43:54 -0800 Subject: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED6E5B.8E142057@arakne.com> <033301c16a1a$59c1c690$c300a8c0@ericlaptop> Message-ID: <3BED9F9A.30F77DB3@arakne.com> eric wrote: > I have to agree with Prabhu on this one. The current behavior of import, > while fine for standard modules and even simple packages with a single > level, is sub-optimal for packages that contain sub-packages. The proposed > behavior solves the problem. > > Handling the packaging issues in SciPy was difficult, and even resulted in a > (not always popular) decision to build and overwrite the Numeric package on > machines that install SciPy. Prabhu's import doesn't resolve all the issues > (I think packages may just be difficult...), but it would have solved this > one. The proposed import allows us to put our own version of Numeric in the > top SciPy directory. Then all SciPy sub-packages would grab this one > instead of an existing site-packages/Numeric. But then, this is not an import problem. If you use Numeric, you call Numeric. If you call something other than Numeric, just give a different name, and all the confusion will go away. If you're worried that you've already encoded the Numeric name 50 times into 300 files; run a python script over these 300 files; this will do the renaming of the 15.000 occurences of the Numeric name. > That makes SciPy > self-contained and allows people to try it out without worrying that it > might break their current installation. There are other solutions to this > problem, but Prabhu's fix is by far the easiest and most robust. And then, in maintenance/integration phase, sometimes 'Numeric' will call Numeric, some other times it will your package ? What if somebody, for some reason I know nothing of (e.g. probably some integration) wants to call Numeric and your Numeric package in the same module ? Wish them tough luck to sort out this poisoned gift.... > Prabhu's import also has some other nice benefits. Some of the sub-packages > in SciPy are useful outside of SciPy. Also sometimes it is easier to > develop a packages outside of the SciPy framework. It would be nice to be > able to develop a module or package 'foo' outside of SciPy and then move it > into SciPy at a later date. However, every SciPy sub-package that referred > to foo prior to its inclusion in SciPy now has to be updated from 'import > foo' to 'import scipy.foo'. These kind of issues make it very painful and > time consuming to rearrange package structures or move modules and > sub-packages in and out of the package. There are basic python scripts which do this painlessly. If you're really working on a large project, there's a project architect which normally would take care of such things, and for whom this should not be a too much of a problem. > Simplifying this will improves > package development. > > > I'm personnally against anything that enlarges the search path uselessly; > > Hopefully I've explained why it is useful for complex packages. Python helps in many areas, but expecting it to palliate for the package design and architecture flaws that inexorably surface anytimes something non-trivial is developped, might be somehow at the edge. Python has not yet replaced the need for relevant software architects. > > > because the obvious reason of increased name space collision, increased > > run-time overhead etc... > > I'm missing something here because I don't understand why this increases > name space collision. If the objection is to the fact that SciPy can have a > version of Numeric in it that masks a Numeric installed in site-packages, I > guess I consider this a feature, not a bug. Actually, it is normally worse than a bug: it is a source of bug tomorrow in your application - of all the bugs you'll have when your programmer will be confusing the two Numeric packages, as well as all the mainteance and integration problems you'll have down the line -. But by then, hopefully for you, you'll be somewhere else... The sad reality of most projects :(( > Afterall, this is already the > behavior for single level packages, extending it to multi-level packages > seems natural. If this isn't your objection, please explain. > > The current runtime overhead isn't so bad. Prabhu sent me a few numbers on > the SciPy import (which contains maybe 10-15 nested packages). I attached > them below -- the overhead is less than 10%. It should be negligible for > standard modules as only packages are really affected (right Prabhu?). And that's how, when you cumulate of the overheads for all new features, you get potenially +100-200% overhead on the new releases. Albeit all the efforts of the Python team, Python 2.0 is up to 70% slower than python 1.5.2; Python 2.1.1 is up to 30% slower than python 2.0, and so on... So, +10% on only such a minor features is anything but negligible :((( FG From gmcm@hypernet.com Sat Nov 10 21:50:32 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Sat, 10 Nov 2001 16:50:32 -0500 Subject: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <033301c16a1a$59c1c690$c300a8c0@ericlaptop> Message-ID: <3BED5AD8.24350.78407262@localhost> eric wrote: [Frederic Giacometti] > > because the obvious reason of increased name space collision, > > increased run-time overhead etc... > > I'm missing something here because I don't understand why this > increases name space collision. Currently, os.py in a package masks the real one from anywhere inside the package. This would extend that to anywhere inside any nested subpackage. Whether that's a "neat" or a "dirty" trick is pretty subjective. The wider the namespace you can trample on, the more it tends to be "dirty". > If the objection is to the fact > that SciPy can have a version of Numeric in it that masks a > Numeric installed in site-packages, I guess I consider this a > feature, not a bug. Afterall, this is already the behavior for > single level packages, extending it to multi-level packages seems > natural. If this isn't your objection, please explain. Well, it's a feature that can crash Python. If the package (which the user has, and you have a hijacked, incompatible copy of) contains an extension module, all kinds of nasty things can happen when both are loaded. Submit patches to the package authors, or require a specific version, or write a wrapper that adapts to different versions or fork or do without. This is definitely a dirty trick. > The current runtime overhead isn't so bad. Under anything near normal usage, no - packages structures are nearly always shallow. It wouldn't be much work to construct a case where time spent in import doubled, however. When the "try relative, then try absolute" strategy was introduced with packages, it added insignificant overhead. It's not so insignificant now. When (and if) the standard library moves to a package structure, it's possilbe it will be seen as a burden. - Gordon From gmcm@hypernet.com Sat Nov 10 23:06:48 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Sat, 10 Nov 2001 18:06:48 -0500 Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism. In-Reply-To: <15340.58785.94412.122093@monster.linux.in> Message-ID: <3BED6CB8.27669.78864436@localhost> Prabhu wrote: [Just singling out one section, since I've said plenty on this subject at other points in these threads] [Prabhu works on knee.py] > it also fixes a bug where the parent package is an extension > module. Python provides no support for an extension module being a package parent module. More precisely, I think the fact that an extension module can be made to behave like a package parent module is an accident. There is special code in import for modules named __init__, and the code is bypassed for extension modules. I suspect you'd have to provide a pretty strong justification before this would become supported behavior. - Gordon From Prabhu Ramachandran Sun Nov 11 04:35:13 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Sun, 11 Nov 2001 10:05:13 +0530 Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism. In-Reply-To: <3BED6CB8.27669.78864436@localhost> References: <15340.58785.94412.122093@monster.linux.in> <3BED6CB8.27669.78864436@localhost> Message-ID: <15342.1.830165.25074@monster.linux.in> >>>>> "GMcM" == Gordon McMillan writes: GMcM> [Prabhu works on knee.py] >> it also fixes a bug where the parent package is an extension >> module. GMcM> Python provides no support for an extension module being a GMcM> package parent module. More precisely, I think the fact that GMcM> an extension module can be made to behave like a package GMcM> parent module is an accident. There is special code in GMcM> import for modules named __init__, and the code is bypassed GMcM> for extension modules. GMcM> I suspect you'd have to provide a pretty strong GMcM> justification before this would become supported behavior. I guess this was unclear. My addition is extremely simple and does not do anything new. Here is an illustration >>> import knee >>> import Numeric.array Traceback (most recent call last): File "", line 1, in ? File "/usr/local/lib/python2.1/knee.py", line 17, in import_hook m = load_tail(q, tail) File "/usr/local/lib/python2.1/knee.py", line 68, in load_tail m = import_module(head, mname, m) File "/usr/local/lib/python2.1/knee.py", line 97, in import_module parent and parent.__path__) AttributeError: 'Numeric' module has no attribute '__path__' >>> Point is, there is a line in knee.py (line 97 that assumes that there is a __path__ attribute for the passed parent. However, if parent is an extension module there is none. So I simply modified it. Here is the diff. $ diff knee.py /usr/local/lib/python2.1/knee.py 98,101c98 < except (ImportError, AttributeError): < # extension modules dont have a __path__ attribute. < # caching failures. < sys.modules[fqname] = None --- > except ImportError: In fact that is all I changed in knee.py! Which is why I said the changes are very small. Maybe I should have shown a patch but the mail was already long. prabhu From ej@ee.duke.edu Sun Nov 11 04:44:44 2001 From: ej@ee.duke.edu (eric) Date: Sat, 10 Nov 2001 23:44:44 -0500 Subject: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED5AD8.24350.78407262@localhost> Message-ID: <037a01c16a6b$993421a0$c300a8c0@ericlaptop> Hey Gordon, > eric wrote: > > [Frederic Giacometti] > > > > because the obvious reason of increased name space collision, > > > increased run-time overhead etc... > > > > I'm missing something here because I don't understand why this > > increases name space collision. > > Currently, os.py in a package masks the real one from > anywhere inside the package. This would extend that to > anywhere inside any nested subpackage. Whether that's a > "neat" or a "dirty" trick is pretty subjective. The wider the > namespace you can trample on, the more it tends to be "dirty". Yeah, I guess I come down on the "neat" side in this one. If I have a module or package called 'common' at the top level of a deep hierarchy, I'd like all sub-packages to inherit it. That seems intuitive to me and inline with the concept of a 'package'. Perhaps the hijacking of the Numeric example strikes a nerve, but inheriting the 'common' module shouldn't be so contentious. Also, if someone has the gall to hijack os.py at the top of your package directory structure, it seems very likely you want this new behavior everywhere within your package. I have a feeling this discussion has been around the block a few times when packages were first being developed... > > > If the objection is to the fact > > that SciPy can have a version of Numeric in it that masks a > > Numeric installed in site-packages, I guess I consider this a > > feature, not a bug. Afterall, this is already the behavior for > > single level packages, extending it to multi-level packages seems > > natural. If this isn't your objection, please explain. > > Well, it's a feature that can crash Python. If the package > (which the user has, and you have a hijacked, incompatible > copy of) contains an extension module, all kinds of nasty > things can happen when both are loaded. This I need to know about. How does this happen? So you have two extension modules, with the same name, one living in a package and the other living in site-packages. If you import both of these, their namespaces don't conflict in some strange way do they? Or are you talking about passing a structure (like a numeric array) generated in one ext module into a routine in the other ext module (expecting a different format of a numeric array) and then getting some strange (seg-fault even) kind of behavior? Anyway, I'd like a few more details for reasons orthogonal to this discussion. > Submit patches to the package authors, or require a specific > version, or write a wrapper that adapts to different versions or > fork or do without. This is definitely a dirty trick. We've done the "require specific version" option here, and "conveniently" upgraded the user's Numeric package for them. The problem is that some people use old versions of Numeric in production code, and don't want to risk an upgrade -- but still want to try out SciPy. I consider our solution a dirtier trick than encapsulating things completely within SciPy. I also don't think any nasty things could happen in this specific situation of having two relatively recent Numerics loaded up, but I could be wrong. > > > The current runtime overhead isn't so bad. > > Under anything near normal usage, no - packages structures > are nearly always shallow. It wouldn't be much work to > construct a case where time spent in import doubled, however. > > When the "try relative, then try absolute" strategy was > introduced with packages, it added insignificant overhead. It's > not so insignificant now. When (and if) the standard library > moves to a package structure, it's possilbe it will be seen as a > burden. I haven't followed the discussion as to whether the standard library will move to packages, but I'll be surpised if it does. I very much like the encapsulation offered by packages, but have found their current incarnation difficult to develop compared to simple modules. The current discussion concerns only one of the issues. As for overhead, I thought I'd get a couple more data points from distutils and xml since they are standard packages. The distutils import is pretty much a 0% hit. However, the xml import is *much* slower -- a factor of 3.5. Thats a huge hit and worth complaining about. I don't know if this can be optimized or not. If not, it may be a show stopper, even if the philosophical argument was uncontested. eric import speed numbers below: C:\temp>python ActivePython 2.1, build 210 ActiveState) based on Python 2.1 (#15, Apr 23 2001, 18:00:35) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import time >>> t1 = time.time();import distutils.command.build; t2 = time.time();print t2-t 1 0.519999980927 >>> C:\temp>python ActivePython 2.1, build 210 ActiveState) based on Python 2.1 (#15, Apr 23 2001, 18:00:35) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import my_import >>> import time >>> t1 = time.time();import distutils.command.build; t2 = time.time();print t2-t 1 0.511000037193 >>> C:\temp>python ActivePython 2.1, build 210 ActiveState) based on Python 2.1 (#15, Apr 23 2001, 18:00:35) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import my_import >>> import time >>> t1 = time.time();import xml.sax.saxutils; t2 = time.time();print t2-t1 1.35199999809 >>> C:\temp>python ActivePython 2.1, build 210 ActiveState) based on Python 2.1 (#15, Apr 23 2001, 18:00:35) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import time >>> t1 = time.time();import xml.sax.saxutils; t2 = time.time();print t2-t1 0.381000041962 >>> From ej@ee.duke.edu Sun Nov 11 05:40:54 2001 From: ej@ee.duke.edu (eric) Date: Sun, 11 Nov 2001 00:40:54 -0500 Subject: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED6E5B.8E142057@arakne.com> <033301c16a1a$59c1c690$c300a8c0@ericlaptop> <3BED9F9A.30F77DB3@arakne.com> Message-ID: <038301c16a73$70c5f790$c300a8c0@ericlaptop> Hey Frederic, > But then, this is not an import problem. > If you use Numeric, you call Numeric. If you call something other than Numeric, > just give a different name, and all the confusion will go away. This is certainly an option, but not a good one in my opinion. The main issue is we want to force a specific version of Numeric for SciPy while allowing people to keep their old standard version of Numeric available for their production code. Single level packages provide a handy way of doing this. Multi-level packages (like SciPy) do not. I guess I just don't see why making a package multi-level should inherently make it harder to do things. > If you're worried that you've already encoded the Numeric name 50 times into > 300 files; run a python script over these 300 files; this will do the renaming > of the 15.000 occurences of the Numeric name. Sure, but this is inconvenient, and something I think should be handled by the packaging facility, not by running a renaming script. > > > That makes SciPy > > self-contained and allows people to try it out without worrying that it > > might break their current installation. There are other solutions to this > > problem, but Prabhu's fix is by far the easiest and most robust. > > And then, in maintenance/integration phase, sometimes 'Numeric' will call > Numeric, some other times it will your package ? In the integration phase, users would need to change the "from Numeric import *" to "from scipy import *" (or search and replace Numeric with scipy) in their code as Numeric is completely subsumed into scipy. So, when not using SciPy, their legacy code can continue using an old version of Numeric. When switching to SciPy, the make the replacement. As I said, its mainly a version issue (with a few minor changes). > > What if somebody, for some reason I know nothing of (e.g. probably some > integration) wants to call Numeric and your Numeric package in the same module ? > Wish them tough luck to sort out this poisoned gift.... > > > Prabhu's import also has some other nice benefits. Some of the sub-packages > > in SciPy are useful outside of SciPy. Also sometimes it is easier to > > develop a packages outside of the SciPy framework. It would be nice to be > > able to develop a module or package 'foo' outside of SciPy and then move it > > into SciPy at a later date. However, every SciPy sub-package that referred > > to foo prior to its inclusion in SciPy now has to be updated from 'import > > foo' to 'import scipy.foo'. These kind of issues make it very painful and > > time consuming to rearrange package structures or move modules and > > sub-packages in and out of the package. > > There are basic python scripts which do this painlessly. If you're really > working on a large project, there's a project architect which normally would > take care of such things, and for whom this should not be a too much of a > problem. Hmmm. I guess the "project architect" in this case is jointly held by Travis Oliphant and yours truely. Neither of us are packaging guru's, but do have a fair amount of experience with Python. We worked quite a while on (and are still working on) packaging issues. Incidently, I have know idea what Travis O.'s opinion is on this specific topic. > > > > Simplifying this will improves > > package development. > > > > > I'm personnally against anything that enlarges the search path uselessly; > > > > Hopefully I've explained why it is useful for complex packages. > > Python helps in many areas, but expecting it to palliate for the package design > and architecture flaws that inexorably surface anytimes something non-trivial is > developped, might be somehow at the edge. Python has not yet replaced the need > for relevant software architects. Them thars fightin' words. ; ) I'm biased, but don't thinking scipy's architecture is flawed. It is simply a *very* large package of integrated sub-packages that also relies heavily on a 3rd evolving group of modules (Numeric). As such, it reveals the difficult issues that arise when trying to build large packages of integrated sub-packages that rely on a 3rd evolving group of modules... > > > > > > because the obvious reason of increased name space collision, increased > > > run-time overhead etc... > > > > I'm missing something here because I don't understand why this increases > > name space collision. If the objection is to the fact that SciPy can have a > > version of Numeric in it that masks a Numeric installed in site-packages, I > > guess I consider this a feature, not a bug. > > Actually, it is normally worse than a bug: it is a source of bug tomorrow in > your application - of all the bugs you'll have when your programmer will be > confusing the two Numeric packages, as well as all the mainteance and > integration problems you'll have down the line -. I disagree and don't think that is true in this (and many other) situations. People who want to use SciPy will migrate completely to it since it includes Numeric. What the sub-package option offers is a way to test SciPy and optionally use it while keeping their standard Numeric around for their production code. > > But by then, hopefully for you, you'll be somewhere else... The sad reality of > most projects :(( > > > Afterall, this is already the > > behavior for single level packages, extending it to multi-level packages > > seems natural. If this isn't your objection, please explain. > > > > The current runtime overhead isn't so bad. Prabhu sent me a few numbers on > > the SciPy import (which contains maybe 10-15 nested packages). I attached > > them below -- the overhead is less than 10%. It should be negligible for > > standard modules as only packages are really affected (right Prabhu?). > > And that's how, when you cumulate of the overheads for all new features, you get > potenially +100-200% overhead on the new releases. > Albeit all the efforts of the Python team, Python 2.0 is up to 70% slower than > python 1.5.2; Python 2.1.1 is up to 30% slower than python 2.0, and so on... > So, +10% on only such a minor features is anything but negligible :((( The computational cost of additional functionality is always a question of what portion of a program is impacted. If we were talking about 10% hit on looping structures or dictionary lookups or local variable lookups, then yes it needs extreme scrutiny. Adding 10% to a rare event is not worthy of note. I expect (and see) 0% overhead for importing standard modules (by far the most common case). Adding 10% overhead to importing a very large package with 10-15 nested sub-packages is just not a big deal. The 350% cost I saw (noted in a response to Gordon) is a *huge* deal and would need to be solved (moving to C would help) before this became standard. eric ----- Original Message ----- From: "Frederic Giacometti" To: "eric" Cc: ; ; ; Sent: Saturday, November 10, 2001 4:43 PM Subject: Re: Proposal for a modified import mechanism. > > > eric wrote: > > > I have to agree with Prabhu on this one. The current behavior of import, > > while fine for standard modules and even simple packages with a single > > level, is sub-optimal for packages that contain sub-packages. The proposed > > behavior solves the problem. > > > > Handling the packaging issues in SciPy was difficult, and even resulted in a > > (not always popular) decision to build and overwrite the Numeric package on > > machines that install SciPy. Prabhu's import doesn't resolve all the issues > > (I think packages may just be difficult...), but it would have solved this > > one. The proposed import allows us to put our own version of Numeric in the > > top SciPy directory. Then all SciPy sub-packages would grab this one > > instead of an existing site-packages/Numeric. > > But then, this is not an import problem. > If you use Numeric, you call Numeric. If you call something other than Numeric, > just give a different name, and all the confusion will go away. > If you're worried that you've already encoded the Numeric name 50 times into > 300 files; run a python script over these 300 files; this will do the renaming > of the 15.000 occurences of the Numeric name. > > > That makes SciPy > > self-contained and allows people to try it out without worrying that it > > might break their current installation. There are other solutions to this > > problem, but Prabhu's fix is by far the easiest and most robust. > > And then, in maintenance/integration phase, sometimes 'Numeric' will call > Numeric, some other times it will your package ? > > What if somebody, for some reason I know nothing of (e.g. probably some > integration) wants to call Numeric and your Numeric package in the same module ? > Wish them tough luck to sort out this poisoned gift.... > > > Prabhu's import also has some other nice benefits. Some of the sub-packages > > in SciPy are useful outside of SciPy. Also sometimes it is easier to > > develop a packages outside of the SciPy framework. It would be nice to be > > able to develop a module or package 'foo' outside of SciPy and then move it > > into SciPy at a later date. However, every SciPy sub-package that referred > > to foo prior to its inclusion in SciPy now has to be updated from 'import > > foo' to 'import scipy.foo'. These kind of issues make it very painful and > > time consuming to rearrange package structures or move modules and > > sub-packages in and out of the package. > > There are basic python scripts which do this painlessly. If you're really > working on a large project, there's a project architect which normally would > take care of such things, and for whom this should not be a too much of a > problem. > > > > Simplifying this will improves > > package development. > > > > > I'm personnally against anything that enlarges the search path uselessly; > > > > Hopefully I've explained why it is useful for complex packages. > > Python helps in many areas, but expecting it to palliate for the package design > and architecture flaws that inexorably surface anytimes something non-trivial is > developped, might be somehow at the edge. Python has not yet replaced the need > for relevant software architects. > > > > > > because the obvious reason of increased name space collision, increased > > > run-time overhead etc... > > > > I'm missing something here because I don't understand why this increases > > name space collision. If the objection is to the fact that SciPy can have a > > version of Numeric in it that masks a Numeric installed in site-packages, I > > guess I consider this a feature, not a bug. > > Actually, it is normally worse than a bug: it is a source of bug tomorrow in > your application - of all the bugs you'll have when your programmer will be > confusing the two Numeric packages, as well as all the mainteance and > integration problems you'll have down the line -. > > But by then, hopefully for you, you'll be somewhere else... The sad reality of > most projects :(( > > > Afterall, this is already the > > behavior for single level packages, extending it to multi-level packages > > seems natural. If this isn't your objection, please explain. > > > > The current runtime overhead isn't so bad. Prabhu sent me a few numbers on > > the SciPy import (which contains maybe 10-15 nested packages). I attached > > them below -- the overhead is less than 10%. It should be negligible for > > standard modules as only packages are really affected (right Prabhu?). > > And that's how, when you cumulate of the overheads for all new features, you get > potenially +100-200% overhead on the new releases. > Albeit all the efforts of the Python team, Python 2.0 is up to 70% slower than > python 1.5.2; Python 2.1.1 is up to 30% slower than python 2.0, and so on... > So, +10% on only such a minor features is anything but negligible :((( > > FG From Prabhu Ramachandran Sun Nov 11 07:42:03 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Sun, 11 Nov 2001 13:12:03 +0530 Subject: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <037a01c16a6b$993421a0$c300a8c0@ericlaptop> References: <3BED5AD8.24350.78407262@localhost> <037a01c16a6b$993421a0$c300a8c0@ericlaptop> Message-ID: <15342.11211.377711.175571@monster.linux.in> Hi, >>>>> "ej" == "eric" writes: >> Currently, os.py in a package masks the real one from anywhere >> inside the package. This would extend that to anywhere inside >> any nested subpackage. Whether that's a "neat" or a "dirty" >> trick is pretty subjective. The wider the namespace you can >> trample on, the more it tends to be "dirty". ej> Yeah, I guess I come down on the "neat" side in this one. If ej> I have a module or package called 'common' at the top level of ej> a deep hierarchy, I'd like all sub-packages to inherit it. ej> That seems intuitive to me and inline with the concept of a ej> 'package'. Perhaps the hijacking of the Numeric example ej> strikes a nerve, but inheriting the 'common' module shouldn't ej> be so contentious. Also, if someone has the gall to hijack ej> os.py at the top of your package directory structure, it seems ej> very likely you want this new behavior everywhere within your ej> package. I agree on this. Also each package is kind of isolated. Any module like os.py inside a sub package won't affect _every_ other sub package and will only affect packages that are nested inside this particular package. So there is some kind of safety net and its not like sticking everything inside sys.path. :) Also, right now, what prevents someone from sticking an os.py somewhere in sys.path and completely ruining standard behaviour. So, its not asif this new approach to importing package makes things dirty, you can very well do 'bad' things right now. [snip] ej> As for overhead, I thought I'd get a couple more data points ej> from distutils and xml since they are standard packages. The ej> distutils import is pretty much a 0% hit. However, the xml ej> import is *much* slower -- a factor of 3.5. Thats a huge hit ej> and worth complaining about. I don't know if this can be ej> optimized or not. If not, it may be a show stopper, even if ej> the philosophical argument was uncontested. >>>> import my_import import time t1 = time.time();import >>>> xml.sax.saxutils; t2 = time.time();print t2-t1 1.35199999809 >>>> import time t1 = time.time();import xml.sax.saxutils; t2 = >>>> time.time();print t2-t1 0.381000041962 IMHO, this is an unfair/wrong comparison. (0) I suspect that you did not first clean things up by doing a plain import xml.sax.saxutils a few times and then start testing. (1) import itself is implemented in C. my_import is pretty much completely in Python. Here is a fairer comparison (done after a few imports). >>> import time >>> s = time.time (); import xml.sax.saxutils; print time.time()-s 0.0434629917145 >>> import my_import >>> import time >>> s = time.time (); import xml.sax.saxutils; print time.time()-s 0.0503059625626 Which is still not bad at all and nothing close to 350% slowdown. But to see if the presently measured slowdown is not the parent lookup we really need to compare things against the modified (to cache failures) knee.py: >>> import knee >>> import time >>> s = time.time (); import xml.sax.saxutils; print time.time()-s 0.0477709770203 >>> import my_import >>> import time >>> s = time.time (); import xml.sax.saxutils; print time.time()-s 0.0501489639282 Which is really not very bad since its just a 5% slowdown. Here are more tests for scipy: >>> import time >>> s = time.time (); import scipy; print time.time()-s 1.36110007763 >>> import knee, time >>> s = time.time (); import scipy; print time.time()-s 1.48176395893 >>> import my_import, time >>> s = time.time (); import scipy; print time.time()-s 1.5150359869 Which means that doing the parent lookup stuff in this case is really not so bad and the biggest slowdown is mostly thanks to knee being implemented in Python. And there is no question of a 350% slowdown!! :) prabhu From Prabhu Ramachandran Sun Nov 11 08:03:59 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Sun, 11 Nov 2001 13:33:59 +0530 Subject: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <033301c16a1a$59c1c690$c300a8c0@ericlaptop> References: <3BED6E5B.8E142057@arakne.com> <033301c16a1a$59c1c690$c300a8c0@ericlaptop> Message-ID: <15342.12527.94615.833898@monster.linux.in> >>>>> "ej" == ej writes: [snip] ej> The current runtime overhead isn't so bad. Prabhu sent me a ej> few numbers on the SciPy import (which contains maybe 10-15 ej> nested packages). I attached them below -- the overhead is ej> less than 10%. It should be negligible for standard modules ej> as only packages are really affected (right Prabhu?). It depends on how you do it. If you have a sub-package that tries to import a standard module it will go through all the parent packages searching for the module and when it doesn't find one it will check in sys.path. There are a few things to note: (1) For a module in a package, the first import will be naturally the slowest. (2) Subsequent imports will be faster since failures are cached and the package is already imported. (3) If the module in question is not inside a package there will be no slowdown whatsoever since there is no parent package at all. I've timed this with vtk and it seems to be correct. >>> import my_import, time # NOTE: I am not inside any package. >>> s = time.time (); import vtkpython; print time.time()-s 1.06130003929 >>> import time >>> s = time.time (); import vtkpython; print time.time()-s 1.06413698196 Its slower with standard import you may say - but that might just be my kernel's scheduling affecting things. I think its fair to conclude that there is no slowdown if you are not inside a package and based on my earlier timings, that recursive searching thru package parents is not too expensive either. There is one issue. lets say we have two sub-packages that have modules of the same name. Then if we explicitly want the other sub-packages module to be imported there is currently no way of doing it. In such a case maybe adding a __parent__ or using (__ as ni did) might be a good idea too. prabhu From Prabhu Ramachandran Sun Nov 11 08:08:18 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Sun, 11 Nov 2001 13:38:18 +0530 Subject: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <3BED5AD8.24350.78407262@localhost> References: <033301c16a1a$59c1c690$c300a8c0@ericlaptop> <3BED5AD8.24350.78407262@localhost> Message-ID: <15342.12786.547708.67405@monster.linux.in> >>>>> "GMcM" == Gordon McMillan writes: [snipped off other issues raised] >> The current runtime overhead isn't so bad. GMcM> Under anything near normal usage, no - packages structures GMcM> are nearly always shallow. It wouldn't be much work to GMcM> construct a case where time spent in import doubled, GMcM> however. But that can be said of almost anything. A nicer question to ask is -- for most circumstances (99%) is the import mechanism fast enough? GMcM> When the "try relative, then try absolute" strategy was GMcM> introduced with packages, it added insignificant GMcM> overhead. It's not so insignificant now. When (and if) the GMcM> standard library moves to a package structure, it's possilbe GMcM> it will be seen as a burden. Yes, which is why maybe adding an 'rimport' keyword (which you suggested) would be a more conservative option? prabhu From mal@lemburg.com Sun Nov 11 14:05:50 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 11 Nov 2001 15:05:50 +0100 Subject: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED6E5B.8E142057@arakne.com> <033301c16a1a$59c1c690$c300a8c0@ericlaptop> Message-ID: <3BEE85BE.130B8B3C@lemburg.com> [Please leave this on the import-sig; severe cross-posting won't get you many friends...] eric wrote: > > I have to agree with Prabhu on this one. The current behavior of import, > while fine for standard modules and even simple packages with a single > level, is sub-optimal for packages that contain sub-packages. The proposed > behavior solves the problem. I have been doing business in the package and subpackage area for years now and can't really say that I have serious problems with the current situation. If all you need is relative imports, then writing a small helper for this should do the trick. The needed information is readily available now. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From gmcm@hypernet.com Sun Nov 11 15:52:20 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Sun, 11 Nov 2001 10:52:20 -0500 Subject: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <037a01c16a6b$993421a0$c300a8c0@ericlaptop> Message-ID: <3BEE5864.23998.7C1EDB5A@localhost> > Hey Gordon, > > > eric wrote: > > > > [Frederic Giacometti] > > > > > > because the obvious reason of increased name space > > > > collision, increased run-time overhead etc... > > > > > > I'm missing something here because I don't understand why > > > this increases name space collision. > > > > Currently, os.py in a package masks the real one from > > anywhere inside the package. This would extend that to > > anywhere inside any nested subpackage. Whether that's a > > "neat" or a "dirty" trick is pretty subjective. The wider the > > namespace you can trample on, the more it tends to be "dirty". > > Yeah, I guess I come down on the "neat" side in this one. If I > have a module or package called 'common' at the top level of a > deep hierarchy, I'd like all sub-packages to inherit it. I have no objection to having an easy way of getting to common from subpackages. [snip] > Also, if someone has the gall to hijack os.py at the top of your > package directory structure, it seems very likely you want this > new behavior everywhere within your package. Perhaps. The trouble is that under this proposal there is no way to get to the real os.py, even when you know what's going on. The proposal doesn't fix a flaw in today's import, it extends the flaw. If there were different ways of spelling "absolute" and "relative", you would get what you want, and I would get what I want. > I have a feeling this discussion has been around the block a few > times when packages were first being developed... Not really. I think it was Mark Hammond that convinced Guido of the usefulness of packages. It got hacked into import.c. It worked for the small existing population of packages and it basically hasn't been touched since then. Package authors certainly haven't ended up following Java's model. > > > > > If the objection is to the fact > > > that SciPy can have a version of Numeric in it that masks a > > > Numeric installed in site-packages, I guess I consider this a > > > feature, not a bug. Afterall, this is already the behavior > > > for single level packages, extending it to multi-level > > > packages seems natural. If this isn't your objection, please > > > explain. > > > > Well, it's a feature that can crash Python. If the package > > (which the user has, and you have a hijacked, incompatible copy > > of) contains an extension module, all kinds of nasty things can > > happen when both are loaded. > > This I need to know about. How does this happen? So you have > two extension modules, with the same name, one living in a > package and the other living in site-packages. If you import > both of these, their namespaces don't conflict in some strange > way do they? > > Or are you talking about passing a structure (like a numeric > array) generated in one ext module into a routine in the other > ext module (expecting a different format of a numeric array) and > then getting some strange (seg-fault even) kind of behavior? > Anyway, I'd like a few more details for reasons orthogonal to > this discussion. Much depends on platform. Windows will happily load 2, so errors are likely to be of the 2nd sort (although it may not take explicitly passing something around - the extension may be relying on TLS data, for example). On platforms that do a real dynamic link, it's going to be C symbols that get stomped on. [snip] > I haven't followed the discussion as to whether the standard > library will move to packages, but I'll be surpised if it does. It's been brought up (and delayed) with each Python release since packages became official. > I very much like the encapsulation offered by packages, but have > found their current incarnation difficult to develop compared to > simple modules. The current discussion concerns only one of the > issues. And it's my contention that the problem this proposal "fixes" is a symptom of a deeper problem - the fact that the import namespace has a concept of "relative" but no way to spell it. The proposal makes some things work, and makes others worse. > As for overhead, I thought I'd get a couple more data points from > distutils and xml since they are standard packages. The > distutils import is pretty much a 0% hit. However, the xml > import is *much* slower -- a factor of 3.5. Prabhu objects to your numbers. Perhaps he's using the standard xml package, and you're using _xmlplus? The latter is vastly more complex. - Gordon From gmcm@hypernet.com Sun Nov 11 15:52:20 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Sun, 11 Nov 2001 10:52:20 -0500 Subject: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <15342.11211.377711.175571@monster.linux.in> References: <037a01c16a6b$993421a0$c300a8c0@ericlaptop> Message-ID: <3BEE5864.21151.7C1EDCCD@localhost> Prabhu wrote: > Also, right now, what prevents someone from sticking an os.py > somewhere in sys.path and completely ruining standard behaviour. If they stick it after the real one, it has no effect. If the user sticks it in front, it's an installation issue, and the user's responsibility. If an author sticks it in front, it's a dirty trick that may or may not work. > So, its not asif this new approach to importing package makes > things dirty, you can very well do 'bad' things right now. Of course. The problem here is that it's all implicit and there's no way around it. sys.modules['os'] = mydirtyhack is pretty obvious. Putting an os.py (or os/__init__.py) inside a package is not. And once it's done, there's no workaround. - Gordon From gmcm@hypernet.com Sun Nov 11 15:52:19 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Sun, 11 Nov 2001 10:52:19 -0500 Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism. In-Reply-To: <15342.1.830165.25074@monster.linux.in> References: <3BED6CB8.27669.78864436@localhost> Message-ID: <3BEE5863.11711.7C1ED9A1@localhost> Prabhu wrote: > >>>>> "GMcM" == Gordon McMillan writes: > > GMcM> [Prabhu works on knee.py] > > >> it also fixes a bug where the parent package is an > extension >> module. > > GMcM> Python provides no support for an extension module > being a GMcM> package parent module. More precisely, I think > the fact that GMcM> an extension module can be made to behave > like a package GMcM> parent module is an accident. [snip] > I guess this was unclear. My addition is extremely simple and > does not do anything new. Here is an illustration And I wasn't clear. I wasn't complaining about your implementation. I was complaining that the rules of import are ill-specified. There are many import hacks in use today that rely on undocumented features, some of which are accidental. When package support was introduced, no one knew how packages would end up being used. I would hate to see Python stuck supporting some of this crap under the guise of "backwards compatibility". - Gordon From Prabhu Ramachandran Sun Nov 11 18:02:57 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Sun, 11 Nov 2001 23:32:57 +0530 Subject: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <3BEE5864.23998.7C1EDB5A@localhost> References: <037a01c16a6b$993421a0$c300a8c0@ericlaptop> <3BEE5864.23998.7C1EDB5A@localhost> Message-ID: <15342.48465.531579.718808@monster.linux.in> >>>>> "GMcM" == Gordon McMillan writes: [snip] >> Also, if someone has the gall to hijack os.py at the top of >> your package directory structure, it seems very likely you want >> this new behavior everywhere within your package. GMcM> Perhaps. The trouble is that under this proposal there is no GMcM> way to get to the real os.py, even when you know what's GMcM> going on. The proposal doesn't fix a flaw in today's import, GMcM> it extends the flaw. If there were different ways of GMcM> spelling "absolute" and "relative", you would get what you GMcM> want, and I would get what I want. I agree. My proposal was simply a way to clear up my problem. I'll admit I didn't think of the bigger picture. What do you propose? You mentioned that import should be narrower in its search, rimport a hybrid and rrimport do what i asked for. This does, somewhat atleast address the problem. As regards getting the real os.py I think the best possible way would be to put all standard modules into a package called std or something. This need not necessarily require a package structure for standard modules. import could simply be modified to see if someone has asked for std.foo and it will only provide the os.py that is found on sys.path. Of course a terrorist package could stick in a nasty .pth file and expose its os.py as the _real_ one which is a problem but I guess its unlikely someone would want to create such a bad package. [snip] GMcM> And it's my contention that the problem this proposal GMcM> "fixes" is a symptom of a deeper problem - the fact that the GMcM> import namespace has a concept of "relative" but no way to GMcM> spell it. The proposal makes some things work, and makes GMcM> others worse. I understand but apart from concerns on speed (for which I really haven't seen real examples) and concerns on issues on things like exposing os.py, I really don't see anything becoming 'worse'. However, I agree wholeheartedly that the import mechanism needs a re-think and am willing to help in my limited capacity. :) GMcM> Prabhu objects to your numbers. Perhaps he's using the GMcM> standard xml package, and you're using _xmlplus? The latter GMcM> is vastly more complex. Indeed, I am using the standard xml package. I don't have the _xmlplus package installed. So I might be wrong but I really can't believe that there is a 3.5 fold speed reduction. I'd like to hear a confirmation from Eric. Anyway, is there some other large package that you'd like me to test with. prabhu From Prabhu Ramachandran Mon Nov 12 07:34:01 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Mon, 12 Nov 2001 13:04:01 +0530 Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism. In-Reply-To: <3BEE5863.11711.7C1ED9A1@localhost> References: <3BED6CB8.27669.78864436@localhost> <3BEE5863.11711.7C1ED9A1@localhost> Message-ID: <15343.31593.765575.923037@monster.linux.in> >>>>> "GMcM" == Gordon McMillan writes: GMcM> [snip] >> I guess this was unclear. My addition is extremely simple and >> does not do anything new. Here is an illustration GMcM> And I wasn't clear. I wasn't complaining about your GMcM> implementation. I was complaining that the rules of import GMcM> are ill-specified. There are many import hacks in use today GMcM> that rely on undocumented features, some of which are GMcM> accidental. GMcM> When package support was introduced, no one knew how GMcM> packages would end up being used. I would hate to see Python GMcM> stuck supporting some of this crap under the guise of GMcM> "backwards compatibility". Okay, so what do you propose? What do you think is the best solution?? import, rimport, rrimport?? prabhu p.s. I am not on import-sig so please cc me in on replies. Thanks. From gmcm@hypernet.com Mon Nov 12 14:06:28 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 12 Nov 2001 09:06:28 -0500 Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism. In-Reply-To: <15343.31593.765575.923037@monster.linux.in> References: <3BEE5863.11711.7C1ED9A1@localhost> Message-ID: <3BEF9114.31305.80E44B37@localhost> Prabhu wrote: > Okay, so what do you propose? What do you think is the best > solution?? import, rimport, rrimport?? If I had access to Guido's time machine, I would probably change things so: import is "absolute import" rimport is "relative import" rrimport is "recursive relative import" I don't, so I tend towards: aimport is "absolute import" rimport is "relative import" rrimport is "recursive relative import" and import is try: rimport except ImportError: aimport I'm not delighted with these names (and avoid syntax debates like the plague), but they seem much clearer than grafting relative-path syntax onto package / module names. For one thing, all the familiar (to Windows & *nix users) punctuation characters are unusable. For another, mac reverses the spellings (so is relative and is absolute). My rather unreliable crystal ball tells me the number one objection to this will be "Why should I have to spell it out - I just want to import it". My reply to that would be the same as my reply to those who don't want to be bothered by having to choose a spelling for division: at the time you write the code, you know exactly what you want. When your code runs, we can only guess what you wanted. - Gordon From jim@interet.com Mon Nov 12 14:52:57 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 12 Nov 2001 09:52:57 -0500 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED5AD8.24350.78407262@localhost> Message-ID: <3BEFE249.371EA5FD@interet.com> Gordon McMillan wrote: > Currently, os.py in a package masks the real one from > anywhere inside the package. This would extend that to What??? When Python starts, it imports site.py which imports os.py. So os.py gets loaded, and won't normally get re-loaded. The os.py that gets loaded depends on sys.path. So if os.py is in package1, it won't get loaded for "import os", but it would get loaded for "import package1.os". Are you saying that "import package1.package2.os" will load package1/os.py? I hope that "import os" will not load package1/os.py, will it? Or am I totally confused. Jim Ahlstrom From jim@interet.com Mon Nov 12 15:02:59 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 12 Nov 2001 10:02:59 -0500 Subject: [Import-sig] Re: [Python-Dev] Re: Proposal for a modified import mechanism. References: <3BED6E5B.8E142057@arakne.com> <033301c16a1a$59c1c690$c300a8c0@ericlaptop> <3BED9F9A.30F77DB3@arakne.com> <038301c16a73$70c5f790$c300a8c0@ericlaptop> Message-ID: <3BEFE4A3.63DF1A21@interet.com> eric wrote: > (and see) 0% overhead for importing standard modules (by far the most > common case). Adding 10% overhead to importing a very large package > with 10-15 nested sub-packages is just not a big deal. The 350% cost I > saw (noted in a response to Gordon) is a *huge* deal and would need to be > solved (moving to C would help) before this became standard. I have code which caches directory contents, and related benchmarks. This might help, and could be combined with a new Python module for importing packages, say, as a new method "importer" in __init__.py. Please see python-dev, "Caching directory files in import.c". JimA From Prabhu Ramachandran Mon Nov 12 15:42:05 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Mon, 12 Nov 2001 21:12:05 +0530 Subject: [Import-sig] Re: [Python-Dev] Proposal for a modified import mechanism. In-Reply-To: <3BEF9114.31305.80E44B37@localhost> References: <3BEE5863.11711.7C1ED9A1@localhost> <3BEF9114.31305.80E44B37@localhost> Message-ID: <15343.60877.693648.637512@monster.linux.in> >>>>> "GMcM" == Gordon McMillan writes: GMcM> Prabhu wrote: >> Okay, so what do you propose? What do you think is the best >> solution?? import, rimport, rrimport?? GMcM> If I had access to Guido's time machine, I would probably GMcM> change things so: import is "absolute import" rimport is GMcM> "relative import" rrimport is "recursive relative import" Is borrowing the time machine an option? GMcM> I don't, so I tend towards: aimport is "absolute import" GMcM> rimport is "relative import" rrimport is "recursive relative GMcM> import" and import is try: rimport except ImportError: GMcM> aimport It sounds good but I dont see the point with introducing rimport and rrimport. Afterall, import is going to do rimport so maybe all we need is: import - "same old" aimport - "absolute import" rimport - "recursive relative import" GMcM> My rather unreliable crystal ball tells me the number one GMcM> objection to this will be "Why should I have to spell it out GMcM> - I just want to import it". My reply to that would be the GMcM> same as my reply to those who don't want to be bothered by GMcM> having to choose a spelling for division: at the time you GMcM> write the code, you know exactly what you want. When your GMcM> code runs, we can only guess what you wanted. Makes sense. prabhu From Prabhu Ramachandran Mon Nov 12 15:53:20 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Mon, 12 Nov 2001 21:23:20 +0530 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <3BEFE249.371EA5FD@interet.com> References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> Message-ID: <15343.61552.414932.44100@monster.linux.in> >>>>> "JCA" == James C Ahlstrom writes: JCA> When Python starts, it imports site.py which imports os.py. JCA> So os.py gets loaded, and won't normally get re-loaded. The JCA> os.py that gets loaded depends on sys.path. JCA> So if os.py is in package1, it won't get loaded for "import JCA> os", but it would get loaded for "import package1.os". Are JCA> you saying that "import package1.package2.os" will load JCA> package1/os.py? I hope that "import os" will not load JCA> package1/os.py, will it? Or am I totally confused. Ummm doing an 'import os' will import the package1/os.py and *not* the standard one. This will happen even though os.py was imported earlier by site.py. This is what Gordon was objecting to in the first place and why he proposes using rimport, rrimport etc. to make things more explicit. prabhu From jeremy@zope.com Mon Nov 12 16:33:19 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Mon, 12 Nov 2001 11:33:19 -0500 (EST) Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <15343.61552.414932.44100@monster.linux.in> References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> Message-ID: <15343.63951.137972.487852@slothrop.digicool.com> >>>>> "PR" == Prabhu Ramachandran writes: PR> Ummm doing an 'import os' will import the package1/os.py and PR> *not* the standard one. This will happen even though os.py was PR> imported earlier by site.py. This is what Gordon was objecting PR> to in the first place and why he proposes using rimport, PR> rrimport etc. to make things more explicit. Of course, you can use the existing mechanism to do this: 'from package1 import os'. The use of an explicit name seems like the clearest route when you have a package-local module that shadows a top-level module -- no need to understand details of relative imports, no question about what is intended by the code. I haven't followed this thread closely. Is there some reason that explicit names in imports is not sufficient? Jeremy From fredrik@pythonware.com Mon Nov 12 17:05:30 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 12 Nov 2001 18:05:30 +0100 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> Message-ID: <006a01c16b9d$c31fc040$ced241d5@hagrid> James C. Ahlstrom wrote: > > Currently, os.py in a package masks the real one from > > anywhere inside the package. This would extend that to > > What??? > > When Python starts, it imports site.py which imports os.py. > So os.py gets loaded, and won't normally get re-loaded. a python system doesn't necessarily load site.py, but nevermind... > So if os.py is in package1, it won't get loaded for "import os", > but it would get loaded for "import package1.os". Are you saying > that "import package1.package2.os" will load package1/os.py? no, but an "import os" from inside the "spam.py" module installed as "package1.spam" will attempt to import "package1.os" before it looks for "os" in the path. consider this (somewhat simplified) directory structure: package1/__init__.py package1/spam.py package1/os.py and this interpreter session: $ python -vvS Python 2.1.1 Type "copyright", "credits" or "license" for more information. >>> import package1 import package1 # directory pythonware # trying package1/__init__.py # package1/__init__.pyc matches package1/__init__.py import pythonware # precompiled from package1/__init__.pyc >>> import package1.spam # trying package1/spam.py # package1/spam.pyc matches package1/spam.py import pythonware.spam # precompiled from package1/spam.pyc # trying package1/os.py # package1/os.pyc matches package1/os.py import pythonware.os # precompiled from package1/os.pyc (note the last line) > I hope that "import os" will not load package1/os.py, will it? depends on who's calling it. From Prabhu Ramachandran Mon Nov 12 18:20:33 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Mon, 12 Nov 2001 23:50:33 +0530 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <15343.63951.137972.487852@slothrop.digicool.com> References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> Message-ID: <15344.4849.71643.868791@monster.linux.in> >>>>> "JH" == Jeremy Hylton writes: >>>>> "PR" == Prabhu Ramachandran writes: PR> Ummm doing an 'import os' will import the package1/os.py and PR> *not* the standard one. This will happen even though os.py PR> was imported earlier by site.py. This is what Gordon was PR> objecting to in the first place and why he proposes using PR> rimport, rrimport etc. to make things more explicit. JH> Of course, you can use the existing mechanism to do this: JH> 'from package1 import os'. The use of an explicit name seems JH> like the clearest route when you have a package-local module JH> that shadows a top-level module -- no need to understand JH> details of relative imports, no question about what is JH> intended by the code. JH> I haven't followed this thread closely. Is there some reason JH> that explicit names in imports is not sufficient? Yes indeed there is. I've already explained my reasons twice. Eric also explained why this was important for Scipy. Anyway, in short, its a big pain re-nesting packages. Also for any package that has a deep enough structure its a real pain accessing packages. from pkg import subpkg is also not the best way to do imports. I personally prefer import pkg.subpkg and I believe this is the recommended way of doing imports. prabhu From jeremy@alum.mit.edu Mon Nov 12 18:35:47 2001 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Mon, 12 Nov 2001 13:35:47 -0500 (EST) Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <15344.4849.71643.868791@monster.linux.in> References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> Message-ID: <15344.5763.557118.787529@walden.zope.com> >>>>> "PR" == Prabhu Ramachandran writes: >>>>> "JH" == Jeremy Hylton writes: JH> I haven't followed this thread closely. Is there some reason JH> that explicit names in imports is not sufficient? PR> Yes indeed there is. I've already explained my reasons twice. PR> Eric also explained why this was important for Scipy. I've gone back through the messages on python-dev, but don't seem a clear summary of the issues that lead to your proposed change. The best I can come up with are: 1) packages that are re-nested are a pain and 2) complex package structures also cause problems. Eric has a specific set of issues with SciPy that involve packages that are developed and used externally but also included in SciPy. I have had a hard time trying to figure out precisely what the problems are. PR> Anyway, in short, its a big pain re-nesting packages. Also for PR> any package that has a deep enough structure its a real pain PR> accessing packages. What does "re-nesting" mean? It get the impression you mean putting one package inside another after it was developed and pacakged for use as a top-level package. If so, it doesn't seem like a problem that occurs that often, right? I'd be hesitant to add features to the import mechanism to cater to an infrequent case. I'd rather see the imports be explicit "import root.a.b.c" than "import b.c". Then re-nesting requires all the import statements to be edited. It's more typing and might even require a simple script to do search-and-replace, but it doesn't sound like a prohibitive burden. I expect there is more to the issue than just wanting to avoid some extra typing. A short PEP that describes the specific problems being solved and discussing alternatives would help. PR> from pkg import subpkg is also not the best way to do imports. I PR> personally prefer import pkg.subpkg and I believe this is the PR> recommended way of doing imports. Why do you think this is the recommended way of doing imports? I use both in my code and haven't been able to come up with a clear rationale for doing one or the other. The from ... import form seems useful when the name of the package/module is long or when it's only one or two names I'm using. Jeremy From gmcm@hypernet.com Mon Nov 12 22:17:19 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 12 Nov 2001 17:17:19 -0500 Subject: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <15344.5763.557118.787529@walden.zope.com> References: <15344.4849.71643.868791@monster.linux.in> Message-ID: <3BF0041F.517.82A5AD5B@localhost> Jeremy wrote: [relative and recursively-relative imports] > I'd rather see the imports be explicit "import root.a.b.c" than > "import b.c". Then re-nesting requires all the import statements > to be edited. It's more typing and might even require a simple > script to do search-and-replace, but it doesn't sound like a > prohibitive burden. As a (minor) data point, if "b.c" is resolved as a relative import, it will be faster than the absolute form ("import a.b.c"). Having re-arranged a number of packages, I have some sympathy for Prabhu's complaint. OTOH, this is a feature which only helps package authors (not package users, who are likely to have a somewhat harder time finding their way around the package). [Prabhu] > PR> from pkg import subpkg is also not the best way to do > imports. I PR> personally prefer import pkg.subpkg and I > believe this is the PR> recommended way of doing imports. [Jeremy] > Why do you think this is the recommended way of doing imports? I > use both in my code and haven't been able to come up with a clear > rationale for doing one or the other. The from ... import form > seems useful when the name of the package/module is long or when > it's only one or two names I'm using. When you have circular imports, someone must use the "import a.b.c" form. This can show up in some surprising ways, especially when the package in question desparately needs re-arranging . - Gordon From barry@zope.com Mon Nov 12 22:32:27 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 12 Nov 2001 17:32:27 -0500 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> Message-ID: <15344.19963.181812.444906@anthem.wooz.org> >>>>> "JH" == Jeremy Hylton writes: JH> I'd rather see the imports be explicit "import root.a.b.c" JH> than "import b.c". Then re-nesting requires all the import JH> statements to be edited. It's more typing and might even JH> require a simple script to do search-and-replace, but it JH> doesn't sound like a prohibitive burden. Note that applications can achieve the same thing without editing code by doing sys.path manipulations. JH> I expect there is more to the issue than just wanting to avoid JH> some extra typing. A short PEP that describes the specific JH> problems being solved and discussing alternatives would help. Indeed. We've been here before (perhaps, several "befores" :). Every time this comes up I get the feeling like there are easy ways to accomplish what you want if you think of the problem differently, or I'm missing something fundamental about the problem, and/or the problem has never been specified identified, or people are trying to solve too many problems at once. Are the needs of application authors different than library authors? -Barry From michel@zope.com Mon Nov 12 23:10:18 2001 From: michel@zope.com (Michel Pelletier) Date: Mon, 12 Nov 2001 15:10:18 -0800 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> <15344.19963.181812.444906@anthem.wooz.org> Message-ID: <3BF056DA.74724E5C@zope.com> "Barry A. Warsaw" wrote: > > Are the needs of application authors different than library authors? This is the best place to start, almost everyone on this list plays both roles to one degree or another. I've read Prabhu's emails and I understand his problem. He's explained it a couple of times, but in general users and their needs have been unclear which I think spawned most of this discussion. I'm actually sort of interested in more about the idea of 'looking up' for packages that Prabhu mentioned and think it could be very useful. In Zope we call this "acquisition" and we use this pattern many, many times to override general site policies and objects with more specific ones the farther "down" you go in the object heirarchy. This not only gives us a nice customization model, but it also gives us a nice delegation model, ie, those responsible on high (that's all of you) can dictate what is and is not the standard library "policy" and users below you (that's me and all the other lusers) can specificly override that mandate at a lower level without interfering with other users. -Michel From Prabhu Ramachandran Tue Nov 13 05:08:34 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Tue, 13 Nov 2001 10:38:34 +0530 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <15344.5763.557118.787529@walden.zope.com> References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> Message-ID: <15344.43730.821724.769885@monster.linux.in> >>>>> "JH" == Jeremy Hylton writes: >>>>> "PR" == Prabhu Ramachandran writes: JH> I haven't followed this thread closely. Is there some reason JH> that explicit names in imports is not sufficient? PR> Yes indeed there is. I've already explained my reasons twice. PR> Eric also explained why this was important for Scipy. [snip] JH> I have had a hard time trying to figure out precisely what the JH> problems are. I think you got it mostly right. Let me try to elaborate on it. (1) Re-nesting a package is a pain. What I mean by re-nesting is that say I have a package, A, that is separate (and that has its own sub packages) and now I want it as part of another package, B. Lets further suppose that the module which re-nests the package, B, tracks the development of A and keeps their copy updated. In this case A is developed as a standalone package and B adds something to it that A cannot/refuses to use. With the current approach B would be forced to modify A every time A changes in some significant way simply because A was re-nested. Yes, this is contrived but such situations do occur. To make things clearer. My main objection is that the name of a package when one imports it depends on its parent packages name. This is IMHO absurd. foo/ sub/ sub1/ From sub1 if you had to import anything from sub you'd have to do import foo.sub.module. So if foo is now part of something else - you have to change all references to foo. (2) If you have a complex package with more than 2-3 nested sub directories it becomes a huge pain to use clean import statements and not have to type long lines just to get to different modules. (3) If you argue that import must always do only absolute imports then why are sibling packages allowed? i.e. if there are two modules in the same directory Python currently allows one to import them with a relative name rather than an absolute foo.sub.pkg kind of name. If this is valid, then its natural to expect that searching also be done in the local package structure. (4) Yes, its possible re-factoring code but sometimes this can be a pain if you have a CVS tree and you want to re-organize your package structure. Bernhard Herzog posted a solution for my specific problem, so that really is not the issue. In my case the current cvsroot for my sources is this: cvs.sourceforge.net/cgi-bin/viewcvs.cgi/mayavi/mayavi/ and Bernhard's solution would create a directory structure like so: cvs.sourceforge.net/cgi-bin/viewcvs.cgi/mayavi/mayavi/mayavi/ Which is pretty crazy if you ask me, its bad enough as it is. :) this will solve my particular problem but is dirty. JH> What does "re-nesting" mean? It get the impression you mean JH> putting one package inside another after it was developed and JH> pacakged for use as a top-level package. If so, it doesn't JH> seem like a problem that occurs that often, right? I'd be JH> hesitant to add features to the import mechanism to cater to JH> an infrequent case. I've had about 4 others mailing me about their related problems. So I wouldn't classify this as a rare problem that can be safely ignored. JH> I'd rather see the imports be explicit "import root.a.b.c" JH> than "import b.c". Then re-nesting requires all the import JH> statements to be edited. It's more typing and might even JH> require a simple script to do search-and-replace, but it JH> doesn't sound like a prohibitive burden. It all depends. I think Eric explained his position pretty clearly. I'm convinced that Python's import structure needs improvement. JH> I expect there is more to the issue than just wanting to avoid JH> some extra typing. A short PEP that describes the specific JH> problems being solved and discussing alternatives would help. Well, its all about convenience anyway - if not we'd all be talking to computers in binary! Why do we need 'high-level' programming languages? Yes, I'm digressing into te philosophy of computing but I dont think syntactic sugar is something to be ignored because its silly. PR> from pkg import subpkg is also not the best way to do PR> imports. I personally prefer import pkg.subpkg and I believe PR> this is the recommended way of doing imports. JH> Why do you think this is the recommended way of doing imports? JH> I use both in my code and haven't been able to come up with a JH> clear rationale for doing one or the other. The from JH> ... import form seems useful when the name of the JH> package/module is long or when it's only one or two names I'm JH> using. Well, the Python howto explains it much better than I could hope to: http://py-howto.sourceforge.net/doanddont/node8.html Since re-loading packages is important for me, I prefer using plain imports. prabhu From Prabhu Ramachandran Tue Nov 13 05:20:03 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Tue, 13 Nov 2001 10:50:03 +0530 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <15344.19963.181812.444906@anthem.wooz.org> References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> <15344.19963.181812.444906@anthem.wooz.org> Message-ID: <15344.44419.359241.47222@monster.linux.in> >>>>> "BAW" == Barry A Warsaw writes: JH> I'd rather see the imports be explicit "import root.a.b.c" JH> than "import b.c". Then re-nesting requires all the import JH> statements to be edited. It's more typing and might even JH> require a simple script to do search-and-replace, but it JH> doesn't sound like a prohibitive burden. BAW> Note that applications can achieve the same thing without BAW> editing code by doing sys.path manipulations. Its not the application that I'm concerned about - an application is typically a single/few file(s) and editing them to suit things is certainly not an issue. But editing 100 files inside a package each time the parent changes is nuts. There is another way to get around this by manipulating __path__ inside a sub package. But this leads to the same module being imported several times. This is what I use currently and its evil. :( JH> I expect there is more to the issue than just wanting to avoid JH> some extra typing. A short PEP that describes the specific JH> problems being solved and discussing alternatives would help. BAW> Indeed. We've been here before (perhaps, several "befores" BAW> :). Every time this comes up I get the feeling like there BAW> are easy ways to accomplish what you want if you think of the So do I need to write a PEP? Is there some special formality/format I need to keep in mind? BAW> problem differently, or I'm missing something fundamental BAW> about the problem, and/or the problem has never been BAW> specified identified, or people are trying to solve too many BAW> problems at once. BAW> Are the needs of application authors different than library BAW> authors? I would think so. prabhu From barry@zope.com Tue Nov 13 06:01:21 2001 From: barry@zope.com (Barry A. Warsaw) Date: Tue, 13 Nov 2001 01:01:21 -0500 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> <15344.43730.821724.769885@monster.linux.in> Message-ID: <15344.46897.223437.983547@anthem.wooz.org> >>>>> "PR" == Prabhu Ramachandran writes: PR> (1) Re-nesting a package is a pain. What I mean by PR> re-nesting is that say I have a package, A, that is separate PR> (and that has its own sub packages) and now I want it as part PR> of another package, B. Why would you want to do that? Why not just keep them separate top-level packages that cooperate? Or export A's names in B's modules? I think distutils helps out here because it's now easy to install A in a way that B could just use, or add to. FWIW, we knit things together as well, e.g. with StandaloneZODB. It's got a bunch of top-level packages that are treated as a single entity via a figment of CVS's imagination. So what if it installs a bunch of separate top-level package names that aren't all treed under a single package? PR> Lets further suppose that the module which re-nests the PR> package, B, tracks the development of A and keeps their copy PR> updated. Okay. PR> In this case A is developed as a standalone package and B adds PR> something to it that A cannot/refuses to use. Okay. PR> With the current approach B would be forced to modify A every PR> time A changes in some significant way simply because A was PR> re-nested. Yes, this is contrived but such situations do PR> occur. Why does B have to add packages to A's namespace? Why can't the B author simply use distutils to ensure that vanilla A is installed, import the bits and pieces of A that you want to expose, overriding what you want to change, and export an interface through B that clients can use instead of A? I.e. through the use of "from foo import bar" and "from foo import bar as baz", you can present whatever public interface you want, through B's namespace, and mimic as much or as little of A's as you want. PR> Its not the application that I'm concerned about - an PR> application is typically a single/few file(s) and editing them PR> to suit things is certainly not an issue. Well, not /all/ applications! JH> I expect there is more to the issue than just wanting to avoid JH> some extra typing. A short PEP that describes the specific JH> problems being solved and discussing alternatives would help. BAW> Indeed. We've been here before (perhaps, several "befores" BAW> :). Every time this comes up I get the feeling like there BAW> are easy ways to accomplish what you want if you think of the PR> So do I need to write a PEP? Is there some special PR> formality/format I need to keep in mind? PEP 1 and PEP 9 are your guidelines to proper PEP form and procedure. BAW> Are the needs of application authors different than library BAW> authors? PR> I would think so. That would be good to outline in your PEP then . -Barry From fredrik@pythonware.com Tue Nov 13 06:44:41 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Tue, 13 Nov 2001 07:44:41 +0100 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED5AD8.24350.78407262@localhost><3BEFE249.371EA5FD@interet.com><15343.61552.414932.44100@monster.linux.in><15343.63951.137972.487852@slothrop.digicool.com><15344.4849.71643.868791@monster.linux.in><15344.5763.557118.787529@walden.zope.com><15344.19963.181812.444906@anthem.wooz.org> <15344.44419.359241.47222@monster.linux.in> Message-ID: <009501c16c0e$acd6e130$ced241d5@hagrid> Prabhu Ramachandran wrote: > Its not the application that I'm concerned about - an application is > typically a single/few file(s) and editing them to suit things is > certainly not an issue. But editing 100 files inside a package each > time the parent changes is nuts. I'd recommend that you don't put the above in your PEP. changing the language to cater to package owners who have more files than users (and don't appear to give a damn about the few users they have) isn't just nuts, it's total madness. if you do this a lot, write a 20-line script to do it for you. From Prabhu Ramachandran Tue Nov 13 07:39:31 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Tue, 13 Nov 2001 13:09:31 +0530 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <009501c16c0e$acd6e130$ced241d5@hagrid> References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> <15344.19963.181812.444906@anthem.wooz.org> <15344.44419.359241.47222@monster.linux.in> <009501c16c0e$acd6e130$ced241d5@hagrid> Message-ID: <15344.52787.673648.792323@monster.linux.in> >>>>> "FL" == Fredrik Lundh writes: >> Its not the application that I'm concerned about - an >> application is typically a single/few file(s) and editing them >> to suit things is certainly not an issue. But editing 100 files >> inside a package each time the parent changes is nuts. FL> I'd recommend that you don't put the above in your PEP. FL> changing the language to cater to package owners who have more FL> files than users (and don't appear to give a damn about the FL> few users they have) isn't just nuts, it's total madness. Ouch! Yes, I might be mad but I guess you are taking my statement totally out of context. (1) I was talking of _my_ particular application where I have a script that imports the package depending on whether the application is installed or frozen. I wasn't talking of every Python application! (2) How does modifying the behaviour of import or better still adding a couple of new keywords (aimport, rimport etc.) that do things a little differently affect the all the users of Python? What we proposed will only affect the package in question and not every user in the system. AFAIK, what was proposed would also not break backward compatibility in any way. (3) As to me not giving a 'damn' about the 'few users' that I may have, I think that is fairly untrue and all I have to say is that I wouldn't have considered posting to c.l.p* with this proposal if it wouldn't be of use to someone else. It would have been very easy for me to have solved my problem and left things at that. FL> if you do this a lot, write a 20-line script to do it for you. I'm afraid I already did better than that. I modified knee.py to do exactly what I wanted and it does work. Also, even without such a hack I can get my package running, thanks. However, the point is that Gordon, Eric, some others and myself think that it would be nice if the import mechanism were a little nicer to package developers. The problem is not something that I am personally desperate for. I'm just trying to see if Python's import mechanism can be improved (if necessary). Flaming me is pointless. :) prabhu From Prabhu Ramachandran Tue Nov 13 08:38:31 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Tue, 13 Nov 2001 14:08:31 +0530 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <3BF056DA.74724E5C@zope.com> References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> <15344.19963.181812.444906@anthem.wooz.org> <3BF056DA.74724E5C@zope.com> Message-ID: <15344.56327.337652.24737@monster.linux.in> >>>>> "MP" == Michel Pelletier writes: MP> I'm actually sort of interested in more about the idea of MP> 'looking up' for packages that Prabhu mentioned and think it MP> could be very useful. In Zope we call this "acquisition" and MP> we use this pattern many, many times to override general site MP> policies and objects with more specific ones the farther MP> "down" you go in the object heirarchy. This not only gives us MP> a nice customization model, but it also gives us a nice MP> delegation model, ie, those responsible on high (that's all of MP> you) can dictate what is and is not the standard library MP> "policy" and users below you (that's me and all the other MP> lusers) can specificly override that mandate at a lower level MP> without interfering with other users. Yes, this is a nicer and probably more convincing use/argument than I had proposed. prabhu From Prabhu Ramachandran Tue Nov 13 08:56:45 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Tue, 13 Nov 2001 14:26:45 +0530 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <15344.46897.223437.983547@anthem.wooz.org> References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> <15344.43730.821724.769885@monster.linux.in> <15344.46897.223437.983547@anthem.wooz.org> Message-ID: <15344.57421.641192.998739@monster.linux.in> >>>>> "BAW" == Barry A Warsaw writes: >>>>> "PR" == Prabhu Ramachandran writes: PR> (1) Re-nesting a package is a pain. What I mean by re-nesting [Re-nesting packages contrived example] BAW> Why would you want to do that? Why not just keep them BAW> separate top-level packages that cooperate? Or export A's BAW> names in B's modules? I think distutils helps out here BAW> because it's now easy to install A in a way that B could just BAW> use, or add to. Umm, that was a contrived example so might not be very sensible. For a more realistic one I think I'll pass the question on to Eric. I think Eric did mention his difficulty with SciPy here: http://mail.python.org/pipermail/python-list/2001-November/071794.html BAW> Why does B have to add packages to A's namespace? Why can't BAW> the B author simply use distutils to ensure that vanilla A is BAW> installed, import the bits and pieces of A that you want to BAW> expose, overriding what you want to change, and export an BAW> interface through B that clients can use instead of A? BAW> I.e. through the use of "from foo import bar" and "from foo BAW> import bar as baz", you can present whatever public interface BAW> you want, through B's namespace, and mimic as much or as BAW> little of A's as you want. Ture, its possible to do things and work around situations with the current scheme. I guess I need to come up with something that definitively proves my point. Will think about it. Maybe Gordon has a better/more convincing argument? I think Michel Pelletier also had a different point of view on this. PR> Its not the application that I'm concerned about - an PR> application is typically a single/few file(s) and editing them PR> to suit things is certainly not an issue. BAW> Well, not /all/ applications! Indeed. I guess I caused confusion here. I was talking of my particular application where I ran into problems with re-nesting and too much typing I was referring to that. I certainly don't intend changing every single application when there is no need for that. PR> So do I need to write a PEP? Is there some special PR> formality/format I need to keep in mind? BAW> PEP 1 and PEP 9 are your guidelines to proper PEP form and BAW> procedure. Thanks. Will look at them. prabhu From mal@lemburg.com Tue Nov 13 09:23:58 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 13 Nov 2001 10:23:58 +0100 Subject: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> <15344.19963.181812.444906@anthem.wooz.org> <3BF056DA.74724E5C@zope.com> Message-ID: <3BF0E6AE.89CD9B9E@lemburg.com> Michel Pelletier wrote: > > "Barry A. Warsaw" wrote: > > > > Are the needs of application authors different than library authors? > > This is the best place to start, almost everyone on this list plays both > roles to one degree or another. I've read Prabhu's emails and I > understand his problem. He's explained it a couple of times, but in > general users and their needs have been unclear which I think spawned > most of this discussion. Well, I've been there and done that. I also proposed an (implicit) relative import scheme some years ago, because I thought it might be worthwhile having to sort out the problems with naming conflicts, local installs etc. In the meantime I found that I can work well without relative imports by being careful about the import statements I use in software using the packages and in modules of the packages themselves. IMHO, it's a much better strategy to think about defining a proper absolute package name structure and using it throughout your code. Given such a structure, the need for relative imports which are hard to debug and can cause a whole variety of problems (e.g. pickled objects still carry the absolute package name, modules can be overridden without the user realizing this, etc.). > I'm actually sort of interested in more about the idea of 'looking up' > for packages that Prabhu mentioned and think it could be very useful. > In Zope we call this "acquisition" and we use this pattern many, many > times to override general site policies and objects with more specific > ones the farther "down" you go in the object heirarchy. This not only > gives us a nice customization model, but it also gives us a nice > delegation model, ie, those responsible on high (that's all of you) can > dictate what is and is not the standard library "policy" and users below > you (that's me and all the other lusers) can specificly override that > mandate at a lower level without interfering with other users. Then you probably also know that debugging an acqusition scenario can become a nightmare, simple due to the fact that each and every attribute has to be tracked to where it's defined in the acquisition hierarchy in order to tell where the problem originated. Acquisition is very nice if it works, but way to complicated for causal users manage in case something goes wrong. The same goes for relative imports which are essentially the same thing w/r to module lookups in packages. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From barry@zope.com Tue Nov 13 14:59:14 2001 From: barry@zope.com (Barry A. Warsaw) Date: Tue, 13 Nov 2001 09:59:14 -0500 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> <15344.43730.821724.769885@monster.linux.in> <15344.46897.223437.983547@anthem.wooz.org> <15344.57421.641192.998739@monster.linux.in> Message-ID: <15345.13634.801472.695445@anthem.wooz.org> >>>>> "PR" == Prabhu Ramachandran writes: PR> Umm, that was a contrived example so might not be very PR> sensible. For a more realistic one I think I'll pass the PR> question on to Eric. I think Eric did mention his difficulty PR> with SciPy here: PR> http://mail.python.org/pipermail/python-list/2001-November/071794.html I'm not wholly unsympathetic to the problem, but I'm trying to give some pushback because of just the reason Michel gives. Even though I can think of my own cool uses of an "acquisitional import", I think Python should be really careful here. One of the deep problems with implicit acquisition is that you often don't know where something really comes from. I'm worried that building this into the import mechanism will make it harder to figure out where something comes from. Explicit is better than implicit. But having said all that, it's clear to me that we've mined this vein enough on the list. It's PEP time! -Barry From jeremy@zope.com Tue Nov 13 17:08:33 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Tue, 13 Nov 2001 12:08:33 -0500 (EST) Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <15344.43730.821724.769885@monster.linux.in> References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> <15344.43730.821724.769885@monster.linux.in> Message-ID: <15345.21393.28230.859650@slothrop.digicool.com> >>>>> "PR" == Prabhu Ramachandran writes: PR> (1) Re-nesting a package is a pain. What I mean by re-nesting PR> is PR> that say I have a package, A, that is separate (and that has PR> its own sub packages) and now I want it as part of another PR> package, B. Lets further suppose that the module which PR> re-nests the package, B, tracks the development of A and keeps PR> their copy updated. In this case A is developed as a PR> standalone package and B adds something to it that A PR> cannot/refuses to use. With the current approach B would be PR> forced to modify A every time A changes in some significant PR> way simply because A was re-nested. Yes, this is contrived PR> but such situations do occur. I think we need to analyze the requirements here a lot more carefully. The specific question is what the packaging requirements are for A and B. I would like to find a solution for the problems that lead to re-nesting, rather than trying to figure out how to make re-nesting easier. Jeremy From Prabhu Ramachandran Tue Nov 13 18:07:24 2001 From: Prabhu Ramachandran (Prabhu Ramachandran) Date: Tue, 13 Nov 2001 23:37:24 +0530 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. In-Reply-To: <15345.13634.801472.695445@anthem.wooz.org> References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> <15344.43730.821724.769885@monster.linux.in> <15344.46897.223437.983547@anthem.wooz.org> <15344.57421.641192.998739@monster.linux.in> <15345.13634.801472.695445@anthem.wooz.org> Message-ID: <15345.24924.396195.958560@monster.linux.in> >>>>> "BAW" == Barry A Warsaw writes: [on the suggested package import improvements] BAW> I'm not wholly unsympathetic to the problem, but I'm trying BAW> to give some pushback because of just the reason Michel BAW> gives. Even though I can think of my own cool uses of an BAW> "acquisitional import", I think Python should be really BAW> careful here. One of the deep problems with implicit BAW> acquisition is that you often don't know where something BAW> really comes from. I'm worried that building this into the BAW> import mechanism will make it harder to figure out where BAW> something comes from. Explicit is better than implicit. BAW> But having said all that, it's clear to me that we've mined BAW> this vein enough on the list. It's PEP time! Great! I'm very happy and thankful that you folks have been so patient with me and are listening to this stuff. I have a few other things to wind up this week, so I'll try starting to write the PEP this weekend. I'll first run it by Eric and maybe pester Gordon a bit before I pass it on to the experts. Thanks again! prabhu From jim@interet.com Thu Nov 15 14:27:18 2001 From: jim@interet.com (James C. Ahlstrom) Date: Thu, 15 Nov 2001 09:27:18 -0500 Subject: [Python-Dev] Re: [Import-sig] Re: Proposal for a modified import mechanism. References: <3BED5AD8.24350.78407262@localhost> <3BEFE249.371EA5FD@interet.com> <15343.61552.414932.44100@monster.linux.in> <15343.63951.137972.487852@slothrop.digicool.com> <15344.4849.71643.868791@monster.linux.in> <15344.5763.557118.787529@walden.zope.com> <15344.43730.821724.769885@monster.linux.in> <15344.46897.223437.983547@anthem.wooz.org> <15344.57421.641192.998739@monster.linux.in> <15345.13634.801472.695445@anthem.wooz.org> <15345.24924.396195.958560@monster.linux.in> Message-ID: <3BF3D0C6.1892D49D@interet.com> Prabhu Ramachandran wrote: > I have a few other things to wind up this week, so I'll try starting > to write the PEP this weekend. I'll first run it by Eric and maybe > pester Gordon a bit before I pass it on to the experts. Please note that I am currently coding PEP 273 http://python.sourceforge.net/peps/pep-0273.html which also deals with imports. My changes are mostly confined to import.c. If you plan to change import.c we need to coordinate. My changes have nothing to do with the way packages are imported. The existing mechanism is extended to zip archives. Jim Ahlstrom From eo1426@ezlink.com.tw Tue Nov 27 12:09:46 2001 From: eo1426@ezlink.com.tw (¾_Ãv) Date: Tue, 27 Nov 2001 20:09:46 +0800 Subject: [Import-sig] TOP Computer Tables, Office Chair, Chair Components Manufacturer Message-ID: Everorient Enterprise Co. Ltd



Everorient Enterprise Co. Ltd , founded in 1984, is the pioneer and leader of computer tables, office chair and chair components in Taiwan. Our products are marketing worldwide with recognition as quality supplier from buyers as they are certified and conformed to SGS and ISO9002 quality standard. Our company products are mostly exported to Europe, South America, Canada and USA. We always insist our company's principle is superior quality, reasonable price, prompt delivery to our customers, so we surely you can be pretty facile to succeed in business among the competitive market.
Enclosed please find some photos of our products for your reference. Please try to find any items you will be interested in and if you have any further questions, please feel free to contact us. We are pleasured and glad to serve you.

Thanks for your kind attention and looking forward to hearing from you soon.




Best regards
International Trade Manger:
David Liao
Everorient Enterprise Co., Ltd.

CONTACT US:
Everorient Enterprise Co., Ltd.
ADDRESS: P.O.Box 3-36, Shulin city, Taipei Hsien, Taiwan
Tel : 886-2-8684-2955
Fax : 886-2-8686-8118
E-mail:ever336@ms2.hinet.net
Homepage:http://www.everorient.com.tw