From ralf at brainbot.com  Mon Sep 22 10:30:47 2003
From: ralf at brainbot.com (Ralf Schmitt)
Date: Mon, 22 Sep 2003 16:30:47 +0200
Subject: [IPython-dev] bug report: dreload not working with 'from module import'
Message-ID: <3F6F0797.7040504@brainbot.com>

Hi,
sorry for posting to the developers mailing list, but using the 
bugtracker wasn't possible (there seems to be no dns entry for 
www.scipy.net).

I have the following package:

ipbug/__init__.py
<empty>
---
ipbug/version.py:
from ipbug.vm import Version

---
ipbug/vm.py:
class Version:
      pass


Starting IPython, importing ipbug.version, and trying to dreload it, 
fails with 'ImportError: No module named ipbug.Version'.
I also wasn't able to checkout the cvs version ('Unknown host 
scipy.org.' after I entered the password), so I have only included a 
traceback for 0.5.0.

Thanks for the fantastic work on IPython,
- Ralf






Python 2.3 (#1, Sep  1 2003, 16:57:54)
Type "copyright", "credits" or "license" for more information.

IPython 0.5.0 -- An enhanced Interactive Python.
?       -> Introduction to IPython's features.
@magic  -> Information about IPython's 'magic' @ functions.
help    -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import ipbug.version                       

In [2]: dreload(ipbug.version)
Reloading ipbug.version
Reloading ipbug.ipbug
Reloading ipbug
========== Version ipbug.Version <module 'ipbug' from 'ipbug/__init__.pyc'>
Reloading ipbug.Version
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/usr/home/ralf/<console>

/usr/local/lib/python2.3/site-packages/IPython/deep_reload.py in 
reload(module,
exclude)
    190     finally:
    191         __builtin__.__import__ = original_import
--> 192         found_now = {}
    193     return ret
    194

/usr/local/lib/python2.3/site-packages/IPython/deep_reload.py in 
deep_reload_hoo
k(module)
    169     pname = name[:i]
    170     parent = sys.modules[pname]
--> 171     return import_module(name[i+1:], name, parent)
    172
    173 # Save the original hooks

/usr/local/lib/python2.3/site-packages/IPython/deep_reload.py in 
import_module(p
artname, fqname, parent)
    155         m = imp.load_module(fqname, fp, pathname, stuff)
    156     finally:
--> 157         if fp: fp.close()
    158        
    159     if parent:

/usr/home/ralf/ipbug/version.py
     -2 from ipbug.vm import Version
     -1

/usr/local/lib/python2.3/site-packages/IPython/deep_reload.py in 
deep_import_hoo
k(name, globals, locals, fromlist)
     50         return q
     51     if hasattr(m, "__path__"):
---> 52         ensure_fromlist(m, fromlist)
     53     return m
     54

/usr/local/lib/python2.3/site-packages/IPython/deep_reload.py in 
ensure_fromlist
(m, fromlist, recursive)
    129             submod = import_module(sub, subname, m)
    130             if not submod:
--> 131                 raise ImportError, "No module named " + subname
    132
    133 # Need to keep track of what we've already reloaded to prevent 
cyclic ev
il

ImportError: No module named ipbug.Version

In [3]:



From fperez at colorado.edu  Tue Sep 23 20:28:59 2003
From: fperez at colorado.edu (Fernando Perez)
Date: Tue, 23 Sep 2003 18:28:59 -0600
Subject: [IPython-dev] bug report: dreload not working with 'from module
 import'
In-Reply-To: <3F6F0797.7040504@brainbot.com>
References: <3F6F0797.7040504@brainbot.com>
Message-ID: <3F70E54B.5030903@colorado.edu>

Sorry for the late reply, I was on vacation.

Ralf Schmitt wrote:
> Hi,
> sorry for posting to the developers mailing list, but using the 
> bugtracker wasn't possible (there seems to be no dns entry for 
> www.scipy.net).

Mmh, works for me now.  But this list is ok too, don't worry.

> I have the following package:
> 
> ipbug/__init__.py
> <empty>
> ---
> ipbug/version.py:
> from ipbug.vm import Version
> 
> ---
> ipbug/vm.py:
> class Version:
>       pass

> Starting IPython, importing ipbug.version, and trying to dreload it, 
> fails with 'ImportError: No module named ipbug.Version'.
> I also wasn't able to checkout the cvs version ('Unknown host 
> scipy.org.' after I entered the password), so I have only included a 
> traceback for 0.5.0.

Weird.  I'm having no problem accessing the scipy.org cvs.  Please let me know 
if this problem persists.

On to your bug:  indeed, the dreload() code is far from perfect. 
Unfortunately, it's also extremely tricky, it has zero comments, and I didn't 
originally write it (it was one of the pieces from Nathan Gray I picked up 
when I started ipython).  I'll put your report on the todo list, but this one 
may take a bit to get fixed.  Every time I've had to deal with that code I've 
ended up with a headache :)  But I promise to look into it.

> Thanks for the fantastic work on IPython,

Glad you like it, and thanks for the feedback!

Best,

f



From ralf at brainbot.com  Wed Sep 24 07:05:14 2003
From: ralf at brainbot.com (Ralf Schmitt)
Date: Wed, 24 Sep 2003 13:05:14 +0200
Subject: [IPython-dev] bug report: dreload not working with 'from module
 import'
In-Reply-To: <3F70E54B.5030903@colorado.edu>
References: <3F6F0797.7040504@brainbot.com> <3F70E54B.5030903@colorado.edu>
Message-ID: <3F717A6A.6050407@brainbot.com>

Fernando Perez wrote:

> Sorry for the late reply, I was on vacation.
>
> Ralf Schmitt wrote:
>
>> Hi,
>> sorry for posting to the developers mailing list, but using the 
>> bugtracker wasn't possible (there seems to be no dns entry for 
>> www.scipy.net).
>
>
> Mmh, works for me now.  But this list is ok too, don't worry.
>
>> Starting IPython, importing ipbug.version, and trying to dreload it, 
>> fails with 'ImportError: No module named ipbug.Version'.
>> I also wasn't able to checkout the cvs version ('Unknown host 
>> scipy.org.' after I entered the password), so I have only included a 
>> traceback for 0.5.0.
>
>
> Weird.  I'm having no problem accessing the scipy.org cvs.  Please let 
> me know if this problem persists.

cvs access did work without any problems this morning..

>
> On to your bug:  indeed, the dreload() code is far from perfect. 
> Unfortunately, it's also extremely tricky, it has zero comments, and I 
> didn't originally write it (it was one of the pieces from Nathan Gray 
> I picked up when I started ipython).  I'll put your report on the todo 
> list, but this one may take a bit to get fixed.  Every time I've had 
> to deal with that code I've ended up with a headache :)  But I promise 
> to look into it.

When using deep_reload.py from IPython 0.4.0 and an otherwise unchanged 
cvs version, the above dreload seems to work..diff's between those 2 
files aren't that big :)

On to another bug:

Python 2.3 (#1, Sep  1 2003, 16:57:54)
Type "copyright", "credits" or "license" for more information.

IPython 0.5.1.cvs -- An enhanced Interactive Python.
?       -> Introduction to IPython's features.
@magic  -> Information about IPython's 'magic' @ functions.
help    -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import sys

In [2]: sys.modules
[......]
/usr/home/ralf/excvs/ipython/scripts/IPython/FakeModule.py in 
__getattr__(self,
key)
     23
     24     def __getattr__(self,key):
---> 25         return self.__dict__[key]
     26
     27     def __str__(self):

KeyError: '__repr__'
Out[2]:
In [3]:



Seems like FakeModule is missing a __repr__ method. Here's a 2 line diff:



Index: FakeModule.py
===================================================================
RCS file: /home/cvsroot/world/ipython/IPython/FakeModule.py,v
retrieving revision 1.3
diff -u -r1.3 FakeModule.py
--- FakeModule.py       22 Aug 2003 16:04:33 -0000      1.3
+++ FakeModule.py       24 Sep 2003 10:49:15 -0000
@@ -27,3 +27,5 @@
     def __str__(self):
         return "<IPython.FakeModule instance>"
 
+    def __repr__(self):
+        return str(self)





From fperez at colorado.edu  Wed Sep 24 17:18:58 2003
From: fperez at colorado.edu (Fernando Perez)
Date: Wed, 24 Sep 2003 15:18:58 -0600
Subject: [IPython-dev] bug report: dreload not working with 'from module
 import'
In-Reply-To: <3F717A6A.6050407@brainbot.com>
References: <3F6F0797.7040504@brainbot.com> <3F70E54B.5030903@colorado.edu> <3F717A6A.6050407@brainbot.com>
Message-ID: <3F720A42.9080208@colorado.edu>

Ralf Schmitt wrote:

> When using deep_reload.py from IPython 0.4.0 and an otherwise unchanged 
> cvs version, the above dreload seems to work..diff's between those 2 
> files aren't that big :)

Does this mean that the dreload() from 0.4.0 works ok for you?  As it turns 
out, those changes in dreload were necessary to fix another bug reported 
earlier on the list.

The dreload code is really tricky, and unfortunately it has almost no 
comments.  Since I didn't originally write it, fixing anything in there is 
like pulling teeth :)  I'll try to see if I can find a way to fix both your 
newly reported problem while not breaking the previous fix.  Fun...

> On to another bug:

[snip]

> Seems like FakeModule is missing a __repr__ method. Here's a 2 line diff:

Thanks!  Applied, will go to CVS in a few minutes.

Regards,

f.



From ralf at brainbot.com  Wed Sep 24 17:39:30 2003
From: ralf at brainbot.com (Ralf Schmitt)
Date: Wed, 24 Sep 2003 23:39:30 +0200
Subject: [IPython-dev] bug report: dreload not working with 'from module
 import'
In-Reply-To: <3F720A42.9080208@colorado.edu>
References: <3F6F0797.7040504@brainbot.com> <3F70E54B.5030903@colorado.edu> <3F717A6A.6050407@brainbot.com> <3F720A42.9080208@colorado.edu>
Message-ID: <3F720F12.6070607@brainbot.com>

Fernando Perez wrote:

> Ralf Schmitt wrote:
>
>> When using deep_reload.py from IPython 0.4.0 and an otherwise 
>> unchanged cvs version, the above dreload seems to work..diff's 
>> between those 2 files aren't that big :)
>
>
> Does this mean that the dreload() from 0.4.0 works ok for you?  As it 
> turns out, those changes in dreload were necessary to fix another bug 
> reported earlier on the list.

Yes:
Python 2.3 (#1, Sep  1 2003, 16:57:54)
Type "copyright", "credits" or "license" for more information.

IPython 0.5.1.cvs -- An enhanced Interactive Python.
?       -> Introduction to IPython's features.
@magic  -> Information about IPython's 'magic' @ functions.
help    -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import ipbug.version; dreload(ipbug.version)
Reloading version
Reloading ipbug
Reloading ipbug
Reloading vm
Out[1]: <module 'ipbug.version' from 'ipbug/version.pyc'>


Regards,
- Ralf

-- 
brainbot technologies ag
boppstrasse 64 . 55118 mainz . germany
fon +49 6131 211639-1 . fax +49 6131 211639-2
http://brainbot.com/  mailto:ralf at brainbot.com




From fperez at colorado.edu  Wed Sep 24 17:45:27 2003
From: fperez at colorado.edu (Fernando Perez)
Date: Wed, 24 Sep 2003 15:45:27 -0600
Subject: [IPython-dev] bug report: dreload not working with 'from module
 import'
In-Reply-To: <3F720F12.6070607@brainbot.com>
References: <3F6F0797.7040504@brainbot.com> <3F70E54B.5030903@colorado.edu> <3F717A6A.6050407@brainbot.com> <3F720A42.9080208@colorado.edu> <3F720F12.6070607@brainbot.com>
Message-ID: <3F721077.4080801@colorado.edu>

Ralf Schmitt wrote:
> Fernando Perez wrote:
>>Does this mean that the dreload() from 0.4.0 works ok for you?  As it 
>>turns out, those changes in dreload were necessary to fix another bug 
>>reported earlier on the list.
> 
> 
> Yes:

Gargh!!!!

Ok, thanks.  I'll either rip my hair out or find a way to fix this somehow. 
I've been inches from just removing dreload() altogether, but apparently 
enough people use it that I suspect that wouldn't go down very well :)

Cheers,

f.



From fperez at colorado.edu  Tue Sep 30 16:52:02 2003
From: fperez at colorado.edu (Fernando Perez)
Date: Tue, 30 Sep 2003 14:52:02 -0600
Subject: [IPython-dev] Re: ipython improvements
In-Reply-To: <3F79CCCD.5070303@boulder.net>
References: <3F79CCCD.5070303@boulder.net>
Message-ID: <3F79ECF2.8060901@colorado.edu>

Jeffery D. Collins wrote:
> Hey Fernando,
> 
> I've been thinking of a few improvements for IPython and thought I would 
> share a few with you.

I'mm cc-ing the ipyhton-dev list here so the discussion gets archived, and 
also b/c others may also have useful contributions to make.

> 1.  match on the contents of __all__ if present in a module
> 
> The problem I'm attempting to address is the large number of module 
> attributes when that module imports lots of other objects.  One example 
> is: from Numeric import *.  When completing on names in the given 
> module, the names the module defines are lost among the imported names. 
>   We need some way to complete only those names that the module actually 
> defines.  Ideally, there would exist a parameter that would contain this 
> information, but (IIRC) there is none.  However, we could use the 
> user-definable __all__ attribute to determine the attribute matches. 
> The disadvantage is that it must be created by the user.
> 
> I have implemented this feature.  If the __all__ attribute is present, 
> those names are used for completion; otherwise, it defaults to the 
> original behavior.

Great!  Is this implemented in the patches you sent me?  This is something 
which has irked me always, but I never really looked into fixing it.  I think 
your  approach of using __all__ as a marker of what to complete on is the 
right one.  If it was not in your original patches, perhaps you can unify it 
and send it my way.  I still haven't looked at the others, so I could work on 
the whole thin in one pass.

> 2.  configuration
> What I need at the moment (if it doesn't exist) is a 'from x import y as 
> z' mechanism.  As an alternative, I could just import x.y and assign 
> that to z.  This should be possible, correct?

Yes.  But you can also simply 'execute from x import y as z' in the config 
file, or even 'execfile my_extra_configthings.py'.  The 'execute' option is 
for single lines of plain python code you want to run at startup, and the 
'execfile' option gives you full freedom to have an arbitrarily complex 
startup file which is regular python code.

> We have discussed configuration briefly in the past.  You noted at the 
> time that it would have been better to use python itself instead of 
> specialized configuration files.  How difficult would it be to make such 
> changes?

Difficult, not really.  But time-consuming, quite.  The big problem is the 
startup ipmaker routine, which is a single, very long function which carefully 
tiptoes through bringing ipython up with zero modularity.  This stuff should 
probably all go into the constructor, or perhaps partly into __init__ and 
partly into an explicitly callable initialize() method, I'm not sure.  I keep 
the 2nd option open, because it might be useful to have the constructor be 
reasonably simple, with an initializer which must be called before running 
providing fine-tuning, for the purposes of embedders.

In cleaning up ipmaker, one of the first things to go would be the baroque 
config system currently in place.  The only slightly delicate problem is how 
to handle recursive inclusions, so that one config file can include another, 
and in the end have it all work transparently for the user.  That requirement 
is key, because it is what makes profiles in ipython a powerful customization 
tool.  Initially I thought it couldn't be solved if the config files were real 
python code, but that was just a sign of my inexperience with the language. 
With a bit of care that's not a problem, and removing the current config 
manager would vastly simplify a lot of code.

> I guess that leads to the bigger question of how to reformulate IPython 
> given the lessons-learned.  I am pretty satisfied with the current 
> behavior, use it all of the time and in fact am now unable to use the 
> default interactive environment!  Just looking quickly among the 
> modules, has python been sufficiently patched to allow the removal of 
> FlexCompleter?  Is DPyGetOpt really needed?  Python already has a fairly 
> complete version of getopt, now that optparse has made it into the 
> distribution.  For these, I realize these changes may introduce be 
> backwards compatibility problems that would need to be resolved.  In all 
> fairness, these are pretty minor issues.  I'll get back with you when 
> I've had a chance to look through the code.  BTW, which is the 
> appropriate forum for discussing IPython development?  scipy-dev/user? 
> Is there a thread on this; if not, should one be started? :)

Well, I just started it on ipython-dev with this reply to your mail :)  If 
you're not subscribed to the list, don't be afraid, it's very low traffic.

Your other comments are on the nose, and basically mimic what I've been 
thinking lately.  I'm leaning strongly towards making 2.3 a requirement for 
ipython when the internal cleanup starts.  The reasons:

- DPyGetOpt: replace it with getoptparse.
- Logger: replace it with the standard logging module (I think).
- FlexCompleter: that's the new rlcompleter.py (it was my patch :)
- Itpl: remove altogether, we can do without.

These changes would mean yanking a fair bit of code out of ipython, which is 
always a good thing in my book.  Since this rewrite will likely take some 
time, probably by the moment it's production-ready, most people won't see the 
2.3 requirement as a big burden.  And I'd still keep the 0.5x branch around 
for critical fixes for users of py2.1.

If the 2.3 requirement is a big problem for many people, one could always 
package getoptparse and logger from 2.3 along with ipython, I suppose.  But 
I'll worry about that when the time comes.
Basically the cleanup would entail:

- Cleaning ipmaker, with removing the current config handler as part of this.

- Making Magic.py NOT be a mixin, but rather an attribute of the ipython shell 
itself (self.MagicSystem = Magic() or somesuch).

- The 4 module removals mentioned above (requiring 2.3)

Please also take a look at the new_design document, which is where I try to 
keep notes on this whole problem so everyone has access to them.  I may even 
port this thread back to that document later.

If you consider doing any of this for real, as a timetable I'd consider:

1. Incorporate your recent patches for completion, as a 0.5.1 bugfix release.

2. I need to finish another 0.5.x with Gary Bishop's patches for Windows color 
handling.  Those require some discussion and design (under Windows), which is 
why I've fallen way behind on them.  But they are important for Windows users, 
which is a lot of people out there.

3. Start with the cleanup, with no more work going into the 0.5.x branch 
except for bugfixes.

Many thanks for your input, and let's see where this goes,

Fernando.

ps.  Since we're both in Boulder, if one day you want to meet over coffee to 
talk about this, drop me a call.  If you are considering any of this for real, 
it may be a lot more efficient to meet in person, with a laptop going over code.