From ralf at brainbot.com Wed Oct 1 14:21:57 2003 From: ralf at brainbot.com (Ralf Schmitt) Date: Wed, 01 Oct 2003 20:21:57 +0200 Subject: [IPython-dev] replacement for deep_reload.py Message-ID: <3F7B1B45.5040907@brainbot.com> Hi, I've just hacked my own version of deep_reload.py. It works by replacing sys.modules with an empty hash and installing it's __builtin__.__import__, which just calls __builtin__.reload and the old __builtin__.__import__. The only problem I've discovered so far, is that dreload(os) doesn't work (dreload(some module which imports os) however does work). I've tested it with python 2.3.1 and IPython from cvs. The code is very simple, so maybe you can save yourself some headache :) I'll use that version in the next few days, and will report on other problems I found.. - Ralf import sys import __builtin__ # Save the original hooks builtin_import = None old_modules = {} reloaded = {} def my_import_hook(name, globals=None, locals=None, fromlist=None): global reloaded ## if fromlist: ## print 'Importing', fromlist, 'from module', name ## else: ## print 'Importing module', name if old_modules.has_key(name) and not reloaded.has_key(name): reloaded[name]=1 print "Reloading", name __builtin__.reload(old_modules[name]) return builtin_import(name, globals, locals, fromlist) # Replacement for reload() def reload(module, exclude=['sys', '__builtin__', '__main__']): """Recursively reload all modules used in the given module. Optionally takes a list of modules to exclude from reloading. The default exclude list contains sys, __main__, and __builtin__, to prevent, e.g., resetting display, exception, and io hooks. """ global builtin_import global old_modules global reloaded reloaded = {} old_modules = sys.modules sys.modules = {} for ex in exclude+list(sys.builtin_module_names): if old_modules.has_key(ex): sys.modules[ex] = old_modules[ex] reloaded[ex]=1 builtin_import = __builtin__.__import__ __builtin__.__import__ = my_import_hook try: return __builtin__.reload(module) finally: __builtin__.__import__ = builtin_import for m in old_modules: if not sys.modules.has_key(m): #print "RESTORING module", m sys.modules[m] = old_modules[m] else: #print "NOT RESTORING", m pass From fperez at colorado.edu Wed Oct 1 15:44:31 2003 From: fperez at colorado.edu (Fernando Perez) Date: Wed, 01 Oct 2003 13:44:31 -0600 Subject: [IPython-dev] replacement for deep_reload.py In-Reply-To: <3F7B1B45.5040907@brainbot.com> References: <3F7B1B45.5040907@brainbot.com> Message-ID: <3F7B2E9F.1050701@colorado.edu> Ralf Schmitt wrote: > Hi, > I've just hacked my own version of deep_reload.py. > It works by replacing sys.modules with an empty hash and installing > it's __builtin__.__import__, which just calls __builtin__.reload and the old > __builtin__.__import__. > The only problem I've discovered so far, is that dreload(os) > doesn't work (dreload(some module which imports os) however does work). > I've tested it with python 2.3.1 and IPython from cvs. > The code is very simple, so maybe you can save yourself some headache :) > I'll use that version in the next few days, and will report on other > problems I found.. > - Ralf Great! Many thanks for this. Please give it a bit more pounding, and I'd encourage other users of dreload to also try it out. The code is definitely far simpler than the original dreload, but since I don't understand that code too well, I'd like to tiptoe a bit on this issue. If it survives a bit of pounding and discussion, I'll definitely be glad to put it in. Obviously, I'll also test it myself and think about it a bit. Best, f. From jcollins at boulder.net Wed Oct 1 17:53:22 2003 From: jcollins at boulder.net (Jeffery D. Collins) Date: Wed, 01 Oct 2003 15:53:22 -0600 Subject: [IPython-dev] Re: ipython improvements In-Reply-To: <3F79ECF2.8060901@colorado.edu> References: <3F79CCCD.5070303@boulder.net> <3F79ECF2.8060901@colorado.edu> Message-ID: <3F7B4CD2.1090204@boulder.net> Fernando Perez wrote: > Jeffery D. Collins wrote: > >> Hey Fernando, >> >> I've been thinking of a few improvements for IPython and thought I >> would share a few with you. > > > I'mm cc-ing the ipyhton-dev list here so the discussion gets archived, > and also b/c others may also have useful contributions to make. > >> 1. match on the contents of __all__ if present in a module >> >> The problem I'm attempting to address is the large number of module >> attributes when that module imports lots of other objects. One >> example is: from Numeric import *. When completing on names in the >> given module, the names the module defines are lost among the imported >> names. We need some way to complete only those names that the module >> actually defines. Ideally, there would exist a parameter that would >> contain this information, but (IIRC) there is none. However, we could >> use the user-definable __all__ attribute to determine the attribute >> matches. The disadvantage is that it must be created by the user. >> >> I have implemented this feature. If the __all__ attribute is present, >> those names are used for completion; otherwise, it defaults to the >> original behavior. > > > Great! Is this implemented in the patches you sent me? This is > something which has irked me always, but I never really looked into > fixing it. I think your approach of using __all__ as a marker of what > to complete on is the right one. If it was not in your original > patches, perhaps you can unify it and send it my way. I still haven't > looked at the others, so I could work on the whole thin in one pass. The patch is on its way... > >> 2. configuration >> What I need at the moment (if it doesn't exist) is a 'from x import y >> as z' mechanism. As an alternative, I could just import x.y and >> assign that to z. This should be possible, correct? > > > Yes. But you can also simply 'execute from x import y as z' in the > config file, or even 'execfile my_extra_configthings.py'. The 'execute' > option is for single lines of plain python code you want to run at > startup, and the 'execfile' option gives you full freedom to have an > arbitrarily complex startup file which is regular python code. Thanks. I really should read the manual more carefully about this:) >> We have discussed configuration briefly in the past. You noted at the >> time that it would have been better to use python itself instead of >> specialized configuration files. How difficult would it be to make >> such changes? > > > Difficult, not really. But time-consuming, quite. The big problem is > the startup ipmaker routine, which is a single, very long function which > carefully tiptoes through bringing ipython up with zero modularity. > This stuff should probably all go into the constructor, or perhaps > partly into __init__ and partly into an explicitly callable initialize() > method, I'm not sure. I keep the 2nd option open, because it might be > useful to have the constructor be reasonably simple, with an initializer > which must be called before running providing fine-tuning, for the > purposes of embedders. > > In cleaning up ipmaker, one of the first things to go would be the > baroque config system currently in place. The only slightly delicate > problem is how to handle recursive inclusions, so that one config file > can include another, and in the end have it all work transparently for > the user. That requirement is key, because it is what makes profiles in > ipython a powerful customization tool. Initially I thought it couldn't > be solved if the config files were real python code, but that was just a > sign of my inexperience with the language. With a bit of care that's not > a problem, and removing the current config manager would vastly simplify > a lot of code. I agree that this is quite feasible with python. There may already exist an implementation somewhere on which to model it. > >> I guess that leads to the bigger question of how to reformulate >> IPython given the lessons-learned. I am pretty satisfied with the >> current behavior, use it all of the time and in fact am now unable to >> use the default interactive environment! Just looking quickly among >> the modules, has python been sufficiently patched to allow the removal >> of FlexCompleter? Is DPyGetOpt really needed? Python already has a >> fairly complete version of getopt, now that optparse has made it into >> the distribution. For these, I realize these changes may introduce be >> backwards compatibility problems that would need to be resolved. In >> all fairness, these are pretty minor issues. I'll get back with you >> when I've had a chance to look through the code. BTW, which is the >> appropriate forum for discussing IPython development? scipy-dev/user? >> Is there a thread on this; if not, should one be started? :) > > > Well, I just started it on ipython-dev with this reply to your mail :) > If you're not subscribed to the list, don't be afraid, it's very low > traffic. > > Your other comments are on the nose, and basically mimic what I've been > thinking lately. I'm leaning strongly towards making 2.3 a requirement > for ipython when the internal cleanup starts. The reasons: > > - DPyGetOpt: replace it with getoptparse. > - Logger: replace it with the standard logging module (I think). > - FlexCompleter: that's the new rlcompleter.py (it was my patch :) > - Itpl: remove altogether, we can do without. > > These changes would mean yanking a fair bit of code out of ipython, > which is always a good thing in my book. Since this rewrite will likely > take some time, probably by the moment it's production-ready, most > people won't see the 2.3 requirement as a big burden. And I'd still > keep the 0.5x branch around for critical fixes for users of py2.1. > > If the 2.3 requirement is a big problem for many people, one could > always package getoptparse and logger from 2.3 along with ipython, I > suppose. But I'll worry about that when the time comes. I agree with the 2.3 requirement here. I only occasionally refer to 2.2 for debugging purposes when some odd or unexpected behavior occurs. If there is a need to backport standard packages, a subdirectory could contain these modules. > Basically the cleanup would entail: > > - Cleaning ipmaker, with removing the current config handler as part of > this. > > - Making Magic.py NOT be a mixin, but rather an attribute of the ipython > shell itself (self.MagicSystem = Magic() or somesuch). > > - The 4 module removals mentioned above (requiring 2.3) > > Please also take a look at the new_design document, which is where I try > to keep notes on this whole problem so everyone has access to them. I > may even port this thread back to that document later. Where can I find this document? SF (seems to be down at the moment)? -- Jeffery Collins (http://www.boulder.net/~jcollins) From fperez at colorado.edu Wed Oct 1 18:09:19 2003 From: fperez at colorado.edu (Fernando Perez) Date: Wed, 01 Oct 2003 16:09:19 -0600 Subject: [IPython-dev] Re: ipython improvements In-Reply-To: <3F7B4CD2.1090204@boulder.net> References: <3F79CCCD.5070303@boulder.net> <3F79ECF2.8060901@colorado.edu> <3F7B4CD2.1090204@boulder.net> Message-ID: <3F7B508F.8040702@colorado.edu> Jeffery D. Collins wrote: >>Please also take a look at the new_design document, which is where I try >>to keep notes on this whole problem so everyone has access to them. I >>may even port this thread back to that document later. > > > Where can I find this document? SF (seems to be down at the moment)? It's a pdf called new_design.pdf included with every distribution of ipython, in the doc/ directory. So regardless of how you got ipyton (rpm, .tar.gz, CVS, etc) you should have that file already. It gets put in the same directory as the manual. Let me know if you have any problem with this. And by the way, I don't use SF :) Ipython is all hosted (including CVS) at scipy.org. Cheers, f From hupp at cs.wisc.edu Sat Oct 4 19:19:30 2003 From: hupp at cs.wisc.edu (Adam Hupp) Date: Sat, 4 Oct 2003 18:19:30 -0500 Subject: [IPython-dev] Problem when sys.argv does not exist Message-ID: <20031004231929.GA4669@upl.cs.wisc.edu> I'm running into a problem embedding when sys.argv is not defined. I'm running ipython from a C extension that does not set sys.argv. The root of the problem is that DPyGetOpt unconditionally checks the passed arguments against sys.argv. With no sys.argv this raises an exception and aborts the whole program. Checked against 0.5.0. The following change to DPyGetOpt.py should fix it: < if args == sys.argv: < args = sys.argv[1:] --- > if hasattr(sys, "argv"): > if args == sys.argv: > args = sys.argv[1:] -Adam From ralf at brainbot.com Sun Oct 5 08:55:44 2003 From: ralf at brainbot.com (Ralf Schmitt) Date: Sun, 05 Oct 2003 14:55:44 +0200 Subject: [IPython-dev] replacement for deep_reload.py In-Reply-To: <3F7B2E9F.1050701@colorado.edu> References: <3F7B1B45.5040907@brainbot.com> <3F7B2E9F.1050701@colorado.edu> Message-ID: <3F8014D0.6070102@brainbot.com> Fernando Perez wrote: > > Great! Many thanks for this. Please give it a bit more pounding, and > I'd encourage other users of dreload to also try it out. The code is > definitely far simpler than the original dreload, but since I don't > understand that code too well, I'd like to tiptoe a bit on this > issue. If it survives a bit of pounding and discussion, I'll > definitely be glad to put it in. One of the things where it differ's from the old deep_reload is that when importing a submodule, say ipbug.vm, it will not reload ipbug/__init__.py. I've attached another version, which tries to do just that and I'm using that version currently without problems. However I think there must be an even less complicated version. Clearing sys.modules and 'reimporting' the module like in the following code seems to work ok. ==== import sys, bbutils.textproc.spellcheck m=sys.modules.copy() sys.modules.clear() sys.modules['sys'] = sys import bbutils.textproc.spellcheck ==== Currently I don't have the time to investigate this further, but in a week or two, I'll have another look at this. - Ralf -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: deep_reload.py URL: From fperez at colorado.edu Mon Oct 6 13:28:00 2003 From: fperez at colorado.edu (Fernando Perez) Date: Mon, 06 Oct 2003 11:28:00 -0600 Subject: [IPython-dev] Problem when sys.argv does not exist In-Reply-To: <20031004231929.GA4669@upl.cs.wisc.edu> References: <20031004231929.GA4669@upl.cs.wisc.edu> Message-ID: <3F81A620.5050802@colorado.edu> Adam Hupp wrote: > I'm running into a problem embedding when sys.argv is not defined. > I'm running ipython from a C extension that does not set sys.argv. > The root of the problem is that DPyGetOpt unconditionally checks the > passed arguments against sys.argv. With no sys.argv this raises an > exception and aborts the whole program. Checked against 0.5.0. The > following change to DPyGetOpt.py should fix it: > > < if args == sys.argv: > < args = sys.argv[1:] > --- > >> if hasattr(sys, "argv"): >> if args == sys.argv: >> args = sys.argv[1:] Cool! Ipython _can_ run from within C extensions :) A while back a user asked about it and said he had some problems, but I never heard back from him. So I had no idea whether it worked or not. I applied your change as: if hasattr(sys, "argv") and args == sys.argv: args = sys.argv[1:] Given the short-circuiting behavior of 'and', this should be fully equivalent to your suggestion and slightly cheaper than a nested 'if'. Thanks for the contribution, it's already in CVS. Regards, Fernando. From fperez at colorado.edu Mon Oct 6 13:35:29 2003 From: fperez at colorado.edu (Fernando Perez) Date: Mon, 06 Oct 2003 11:35:29 -0600 Subject: [IPython-dev] replacement for deep_reload.py In-Reply-To: <3F8014D0.6070102@brainbot.com> References: <3F7B1B45.5040907@brainbot.com> <3F7B2E9F.1050701@colorado.edu> <3F8014D0.6070102@brainbot.com> Message-ID: <3F81A7E1.5060806@colorado.edu> Ralf Schmitt wrote: >>Great! Many thanks for this. Please give it a bit more pounding, and >>I'd encourage other users of dreload to also try it out. The code is >>definitely far simpler than the original dreload, but since I don't >>understand that code too well, I'd like to tiptoe a bit on this >>issue. If it survives a bit of pounding and discussion, I'll >>definitely be glad to put it in. > > > One of the things where it differ's from the old deep_reload is that > when importing a submodule, say ipbug.vm, it will not reload > ipbug/__init__.py. I've attached another version, which tries to do just > that and I'm using that version currently without problems. > However I think there must be an even less complicated version. > Clearing sys.modules and 'reimporting' the module like in the following > code seems to work ok. > ==== > import sys, bbutils.textproc.spellcheck > m=sys.modules.copy() > sys.modules.clear() > sys.modules['sys'] = sys > import bbutils.textproc.spellcheck > ==== > > Currently I don't have the time to investigate this further, but in a > week or two, I'll have another look at this. Great. And don't worry: I have a trip this week which will shut me off from ipython for at least the rest of the week. So take your time, and 0.5.1 will have quite a few nice goodies in it :) Regards, Fernando. From anders at bruun-olsen.net Tue Oct 7 04:28:38 2003 From: anders at bruun-olsen.net (Anders Bruun Olsen) Date: Tue, 7 Oct 2003 10:28:38 +0200 Subject: [IPython-dev] Bugtracker? Message-ID: <20031007082838.GE28465@elmer.skumleren.net> Hi, I filed a bug in the bugtracker almost 2 weeks ago but nobody seems to have noticed it and since there seems to be no activity in the bugtracker I just want to checkup on the state of affairs here. Have I filed a bug the wrong place or what? -- Anders -----BEGIN GEEK CODE BLOCK----- Version: 3.12 GCS/O d--@ s:+ a-- C++ UL+++$ P++ L+++ E- W+ N(+) o K? w O-- M- V PS+ PE@ Y+ PGP+ t 5 X R+ tv+ b++ DI+++ D+ G e- h !r y? ------END GEEK CODE BLOCK------ PGPKey: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x8BFECB41 From fperez at colorado.edu Tue Oct 7 11:34:22 2003 From: fperez at colorado.edu (Fernando Perez) Date: Tue, 07 Oct 2003 09:34:22 -0600 Subject: [IPython-dev] Bugtracker? In-Reply-To: <20031007082838.GE28465@elmer.skumleren.net> References: <20031007082838.GE28465@elmer.skumleren.net> Message-ID: <3F82DCFE.9040607@colorado.edu> Anders Bruun Olsen wrote: > Hi, > > I filed a bug in the bugtracker almost 2 weeks ago but nobody seems to > have noticed it and since there seems to be no activity in the > bugtracker I just want to checkup on the state of affairs here. Have I > filed a bug the wrong place or what? > My bad. I just checked it and assigned it to myself, I'll look into it as soon as I have a chance (next week, I'm leaving for a work trip soon). You did the right thing with the bug tracker. It's just that I haven't figured out how to make it email me when new bugs are filed. I thought it was done, but obviously not. I'll contact the bugtracker admin to find out how to do that. Best regards, Fernando From anders at bruun-olsen.net Tue Oct 7 15:18:05 2003 From: anders at bruun-olsen.net (Anders Bruun Olsen) Date: Tue, 7 Oct 2003 21:18:05 +0200 Subject: [IPython-dev] Bugtracker? In-Reply-To: <3F82DCFE.9040607@colorado.edu> References: <20031007082838.GE28465@elmer.skumleren.net> <3F82DCFE.9040607@colorado.edu> Message-ID: <20031007191809.GA15419@elmer.skumleren.net> On Tue, Oct 07, 2003 at 09:34:22AM -0600, Fernando Perez wrote: > >I filed a bug in the bugtracker almost 2 weeks ago but nobody seems to > >have noticed it and since there seems to be no activity in the > >bugtracker I just want to checkup on the state of affairs here. Have I > >filed a bug the wrong place or what? > My bad. I just checked it and assigned it to myself, I'll look into it as > soon as I have a chance (next week, I'm leaving for a work trip soon). > You did the right thing with the bug tracker. It's just that I haven't > figured out how to make it email me when new bugs are filed. I thought it > was done, but obviously not. I'll contact the bugtracker admin to find out > how to do that. Twisted (twistedmatrix.com) uses Roundup also and it sends out mails, so they Twisted admins might be able to help you if the Roundup admins are unresponsive. -- Anders -----BEGIN GEEK CODE BLOCK----- Version: 3.12 GCS/O d--@ s:+ a-- C++ UL+++$ P++ L+++ E- W+ N(+) o K? w O-- M- V PS+ PE@ Y+ PGP+ t 5 X R+ tv+ b++ DI+++ D+ G e- h !r y? ------END GEEK CODE BLOCK------ PGPKey: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x8BFECB41 From fperez at colorado.edu Tue Oct 7 17:16:07 2003 From: fperez at colorado.edu (Fernando Perez) Date: Tue, 07 Oct 2003 15:16:07 -0600 Subject: [IPython-dev] Bugtracker? In-Reply-To: <20031007191809.GA15419@elmer.skumleren.net> References: <20031007082838.GE28465@elmer.skumleren.net> <3F82DCFE.9040607@colorado.edu> <20031007191809.GA15419@elmer.skumleren.net> Message-ID: <3F832D17.6030206@colorado.edu> Anders Bruun Olsen wrote: > Twisted (twistedmatrix.com) uses Roundup also and it sends out mails, so > they Twisted admins might be able to help you if the Roundup admins are > unresponsive. No problem, Joe Cooper (the Scipy admin) just told me he'd enable email notifications later. I simply had never asked him about it. So this incident shouldn't happen again. Thanks for reporting (both, the ipython bug and the email thingie), f