From ipython at ml.schieke.net Mon Aug 2 19:00:59 2004 From: ipython at ml.schieke.net (Jaco Schieke) Date: Tue, 03 Aug 2004 01:00:59 +0200 Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython Message-ID: <410EC7AB.3040806@ml.schieke.net> I had some issues trying to get tab-completion working with COM objects' properties in Window$ using the win32com module & IPython and have been able to get things working with help from the python-win32 list. I'm reposting the solution on here for inclusion if it makes sense into IPython. The best place to fix this would however be in python's rlcompleter.py and not IPython's FlexCompleter.py, but previous requests to fix rlcompleter.py have gone unheeded - and this one's got an even lower chance as it is win32 specific. (Patch follows at the end of email) Regards, Jaco Schieke Thread from python-win32 follows: -------- Original Message -------- Subject: Re: [python-win32] Tab-completion of data/properties and win32com Date: Tue, 03 Aug 2004 00:37:49 +0200 Mark Hammond wrote: >>I am trying to use IPython interactively to do some win32com >>programming >>and want to use the completion feature. The problem is that >>tab-completion only completes the object methods, and not the >>associated >>properties, for instance: >> >> > >IDLE doesn't have special knowledge of COM objects. Pythonwin does though - >it shows the Excel properties in that example. > >You could try and find the place in IDLE where the list is built, and copy >the code from pywin\scintilla\view.py - look for the comment "# The object >may be a COM object with typelib support - lets see if we can get its >props." Then submit a patch for IDLE, and it may well be accepted. > >Another cute idea would be to hack makepy to use "properties" for 2.3 and >later - this should make them look more "real" - but IDLE may not support >that yet (and it would be non-trivial anyway) > >Mark. > > > > > Mark, Thanks for the advice. We misunderstood each other, but I got it working. What I ended up doing is to patch rlcompleter.py to allow it to get all the attributes of the COM object (as is done in pywin's view.py). I'm currently using IPython (ipython.scipy.org) which is using a modified rlcompleter.py due to limitations of the exising rlcompleter.py implementation. I'll submit a patch to IPython and see if it gets accepted there. Would there be any point in submitting a patch against 2.3's rlcompleter? This is a win32 specific feature, and if I understand you correctly, it can be solved in other ways by fixing makepy. Regards & tx for the assistance, Jaco Index: ipython/IPython/FlexCompleter.py =================================================================== RCS file: /home/cvsroot/world/ipython/IPython/FlexCompleter.py,v retrieving revision 1.3 diff -r1.3 FlexCompleter.py 163d162 < 170a170,175 > def list2dict(l): > ret={} > for i in l: > ret[i] = None > return ret > 185a191,194 > matches_dict = {} > matches_dict.update(list2dict(matches)) > matches = matches_dict.keys() > matches.sort() 188a198 > 192a203,208 > try: > ret=ret+klass._prop_map_get_.keys() > ret=ret+klass._prop_map_put_.keys() > pass > except AttributeError: > pass From Fernando.Perez at colorado.edu Mon Aug 2 19:09:52 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 02 Aug 2004 17:09:52 -0600 Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython In-Reply-To: <410EC7AB.3040806@ml.schieke.net> References: <410EC7AB.3040806@ml.schieke.net> Message-ID: <410EC9C0.9030606@colorado.edu> Jaco Schieke wrote: > I had some issues trying to get tab-completion working with COM objects' > properties in Window$ using the win32com module & IPython and have been > able to get things working with help from the python-win32 list. I'm > reposting the solution on here for inclusion if it makes sense into > IPython. The best place to fix this would however be in python's > rlcompleter.py and not IPython's FlexCompleter.py, but previous requests > to fix rlcompleter.py have gone unheeded - and this one's got an even > lower chance as it is win32 specific. Well, unfortunately there's no point in accepting your patch: FlexCompleter is slated for removal from ipython soon. FlexCompleter was really just a copy of rlcompleter from python 2.1, back when rlcompleter did not support arbitrary namespaces. But my patch to rlcompleter got accepted into mainline, so there really is no need anymore for FlexCompleter, which is now a copy of the official rlcompleter. It's actually just a matter of my having been lazy that it's still around, since I should have removed it a while back (when 2.2 became a requirement for ipython -- I don't really remember when this happened). Sorry to break these bad news to you. As you say, the proper place for this fix is really in rlcompleter proper, so perhaps you could continue pounding on their door until at least you get a response. That's how I got the patch which made FlexCompleter accepted into the official rlcompleter. Regards, f From ipython at ml.schieke.net Mon Aug 2 19:37:43 2004 From: ipython at ml.schieke.net (Jaco Schieke) Date: Tue, 03 Aug 2004 01:37:43 +0200 Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython In-Reply-To: <410EC9C0.9030606@colorado.edu> References: <410EC7AB.3040806@ml.schieke.net> <410EC9C0.9030606@colorado.edu> Message-ID: <410ED047.7030500@ml.schieke.net> Fernando, I see though that the regexp line: m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text) in FlexCompleter is not incorporated into rlcompleter for python 2.3. The original reason for changing this from the vanilla rlcompleter was to allow for completions on arrays and other funny things, e.g. "".. I was involved in this but took a 2 year leave of absence ;-). I read some comments (after posting here) on sourceforge on the original patch submitted for rlcompleter though and there are issues with arbitrary code execution (https://sourceforge.net/tracker/?func=detail&aid=547176&group_id=5470&atid=305470). Having said all of this ... I not quite sure what the solution is. For the while being, I like typing as little as possible & living on the edge - so I'll stick with FlexCompleter. Regards, Jaco Fernando Perez wrote: > Jaco Schieke wrote: > >> I had some issues trying to get tab-completion working with COM >> objects' properties in Window$ using the win32com module & IPython >> and have been able to get things working with help from the >> python-win32 list. I'm reposting the solution on here for inclusion >> if it makes sense into IPython. The best place to fix this would >> however be in python's rlcompleter.py and not IPython's >> FlexCompleter.py, but previous requests to fix rlcompleter.py have >> gone unheeded - and this one's got an even lower chance as it is >> win32 specific. > > > Well, unfortunately there's no point in accepting your patch: > FlexCompleter is slated for removal from ipython soon. FlexCompleter > was really just a copy of rlcompleter from python 2.1, back when > rlcompleter did not support arbitrary namespaces. But my patch to > rlcompleter got accepted into mainline, so there really is no need > anymore for FlexCompleter, which is now a copy of the official > rlcompleter. > > It's actually just a matter of my having been lazy that it's still > around, since I should have removed it a while back (when 2.2 became a > requirement for ipython -- I don't really remember when this happened). > > Sorry to break these bad news to you. As you say, the proper place > for this fix is really in rlcompleter proper, so perhaps you could > continue pounding on their door until at least you get a response. > That's how I got the patch which made FlexCompleter accepted into the > official rlcompleter. > > Regards, > > f > > From Fernando.Perez at colorado.edu Tue Aug 3 00:10:30 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 02 Aug 2004 22:10:30 -0600 Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython In-Reply-To: <410ED047.7030500@ml.schieke.net> References: <410EC7AB.3040806@ml.schieke.net> <410EC9C0.9030606@colorado.edu> <410ED047.7030500@ml.schieke.net> Message-ID: <410F1036.5080803@colorado.edu> Jaco Schieke wrote: > Fernando, > > I see though that the regexp line: > m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text) > in FlexCompleter is not incorporated into rlcompleter for python 2.3. > > The original reason for changing this from the vanilla rlcompleter was > to allow for completions on arrays and other funny things, e.g. > "".. I was involved in this but took a 2 year leave of absence > ;-). I read some comments (after posting here) on sourceforge on the > original patch submitted for rlcompleter though and there are issues > with arbitrary code execution > (https://sourceforge.net/tracker/?func=detail&aid=547176&group_id=5470&atid=305470). Thanks for pointing this to me. Well, I guess then my solution will then be to keep FlexCompleter around, but I'll strip it to the bare essentials, overriding only the necessary methods from rlcompleter. Cheers, f From Fernando.Perez at colorado.edu Tue Aug 3 00:20:05 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 02 Aug 2004 22:20:05 -0600 Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython In-Reply-To: <410EC7AB.3040806@ml.schieke.net> References: <410EC7AB.3040806@ml.schieke.net> Message-ID: <410F1275.5070505@colorado.edu> Jaco Schieke wrote: > (Patch follows at the end of email) I have some questions about this. I'll also make some general comments about patches in a separate mail, please heed them for the future. Since I may keep FlexCompleter around, I want to clarify a few things about your patch. >> def list2dict(l): >> ret={} >> for i in l: >> ret[i] = None >> return ret >> > > 185a191,194 > >> matches_dict = {} >> matches_dict.update(list2dict(matches)) >> matches = matches_dict.keys() >> matches.sort() What exactly is all of this for? If you want to return the matches sorted, a simple matches.sort() before returning will do just fine, no? Maybe I'm just tired now, but the list2dict contortions seem unnecessary to me. Besides, I think readline sorts itself always, since I always see the proposed completions come out sorted. Could you clarify exactly what this is doing? As far as I can tell, all of it is completely unnecessary. > 188a198 > > 192a203,208 > >> try: >> ret=ret+klass._prop_map_get_.keys() >> ret=ret+klass._prop_map_put_.keys() >> pass >> except AttributeError: >> pass This could be acceptable, I imagine, for the benefit of win32 users. But not in this form: all Win-specific code should be isolated at definition time, I have no intention of making Unix users pay at runtime for windows-only stuff. So you'd need to write instead something like: def get_class_members(): --- current code --- if on_windows: # redefine with win-specific hacks def get_class_members(): --- your windows code --- or something similar. Be mindful of the details about cygwin vs non-cygwin environments when writing windows-specific code, please. Also make sure you test it very well, since I can't test it myself. Best, f From Fernando.Perez at colorado.edu Tue Aug 3 01:56:44 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 02 Aug 2004 23:56:44 -0600 Subject: [IPython-dev] A simple patch policy for IPython contributions Message-ID: <410F291C.901@colorado.edu> Hi everyone, as ipython's userbase grows, more and more people are sending me patches. I greatly appreciate this, and I hope the process continues. However, more patches mean more alien code for me to study, so I want to specify a minimal 'policy' on patch creation to try and make the process as smooth as possible. This is just a set of simple guidelines aimed at ensuring that we all spend our limited time on the code itself and not struggling with patch management. I've said some of this before, but I want it in a separate message for easy reference and archiving. I don't want to sound patronizing or to anger anyone, it's just a matter of making the process as simple and efficient as possible to all involved (with my time counting double or triple, because I am the bottleneck). Any comments on this are very much welcome (except for any discussion on #1). Ipython's minimal patch policy: 1. ***NEVER, EVER, EVER*** USE HARD TABS. USE 4 SPACES FOR INDENTATION. I want to make sure that this one is as loudly and clearly spelled out as possible. It is absolutely not open to discussion. I've already had bugs creep in because of patches with hard tabs in them, so in the future I'll simply reject any patch with a single tab in it. IPython is indented (following python.org policy) with 4 spaces per level. 2. Please send patches as attachments, with a .diff extension. This way, there are no issues with manual copy/pasting from a mail client's window. And they load them into XEmacs with syntax highlighting. 3. Make your patches against current CVS. 4. Use -u in diff, I find unified patches easier to visually parse. That's it. For those interested in more info about contributing to an open source project, Here are some links I've found. They are probably no better or worse than other similar documents: http://www.linux.com/howtos/Software-Release-Practice-HOWTO/index.shtml http://www.kegel.com/academy/opensource.html Best regards, and thanks again to all who have contributed in the past. Fernando. From ipython at ml.schieke.net Tue Aug 3 19:22:01 2004 From: ipython at ml.schieke.net (Jaco Schieke) Date: Wed, 04 Aug 2004 01:22:01 +0200 Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython In-Reply-To: <410F1275.5070505@colorado.edu> References: <410EC7AB.3040806@ml.schieke.net> <410F1275.5070505@colorado.edu> Message-ID: <41101E19.50803@ml.schieke.net> Fernando, Some clarifications: 1. The dictionary hack is simply to keep "matches" unique. It took me a while to realize this, but pyreadline actually expects an unique array (see lib\site-packages\readline\pyreadline.py(788): if r and r not in completions:), otherwise it stops reading from the matches returned and simply exits. This is not win32 specific and should be a usefull addition should be incorporated into rlcompleter.py. An easy way to test this is to add matches.sort() just before you return matches in FlexCompleter.py. Then try []. ... the results are disappointing. The reason for this is obviously that you first dir() the object and then the base classes, so you'd not notice this normally. However sorting the array (or adding COM attributes after you dir()'ed the base classes) causes the issues, as duplicates appear very quickly. It took me quite some time to figure this out. - so in short: matches must be unique'd and list2dict is the easiest way I know (I may be rusty.) Let me know if you concur then I'll submit a bug report for rlcompleter.py. 2. I've rewritten the function to check for sys.platform and only run win32 hacks if on the right platform - patch follows.. Regards, Jaco Index: ipython/IPython/FlexCompleter.py =================================================================== RCS file: /home/cvsroot/world/ipython/IPython/FlexCompleter.py,v retrieving revision 1.3 diff -B -r1.3 FlexCompleter.py 78a79 > import sys 156c157 < instances, class members are are also considered.) --- > instances, class members are are also considered.) A unique array is returned 162c163,167 < import re --- > def list2dict(l): > ret={} > for i in l: > ret[i] = None > return ret 163a169 > import re 185a192,196 > > matches_dict = {} > matches_dict.update(list2dict(matches)) # ensures uniqueness > matches = matches_dict.keys() > matches.sort() 193a206,232 > > if sys.platform == "win32": > def get_class_members(klass): > """Obtains class and base class members through dir() > > Additional query is also performed in the case that the class is a > COM object instantiated by the win32 module of Mark Hammond, > as dir() does not reveal the properties associated with > the COM object. > > """ > ret = dir(klass) > > if hasattr(klass,'__bases__'): > for base in klass.__bases__: > ret = ret + get_class_members(base) > > #Win32 specific code follows > try: # in case klass is a COM object from win32 > ret=ret+klass._prop_map_get_.keys() > ret=ret+klass._prop_map_put_.keys() > pass > except AttributeError: > pass > #End of Win32 specific code > > return ret Fernando Perez wrote: > Jaco Schieke wrote: > >> (Patch follows at the end of email) > > > I have some questions about this. I'll also make some general > comments about patches in a separate mail, please heed them for the > future. > > Since I may keep FlexCompleter around, I want to clarify a few things > about your patch. > >>> def list2dict(l): >>> ret={} >>> for i in l: >>> ret[i] = None >>> return ret >>> >> >> 185a191,194 >> >>> matches_dict = {} >>> matches_dict.update(list2dict(matches)) >>> matches = matches_dict.keys() >>> matches.sort() >> > > What exactly is all of this for? If you want to return the matches > sorted, a simple > > matches.sort() > > before returning will do just fine, no? Maybe I'm just tired now, but > the list2dict contortions seem unnecessary to me. > > Besides, I think readline sorts itself always, since I always see the > proposed completions come out sorted. Could you clarify exactly what > this is doing? As far as I can tell, all of it is completely unnecessary. > >> 188a198 >> >> 192a203,208 >> >>> try: >>> ret=ret+klass._prop_map_get_.keys() >>> ret=ret+klass._prop_map_put_.keys() >>> pass >>> except AttributeError: >>> pass >> > > This could be acceptable, I imagine, for the benefit of win32 users. > But not in this form: all Win-specific code should be isolated at > definition time, I have no intention of making Unix users pay at > runtime for windows-only stuff. So you'd need to write instead > something like: > > def get_class_members(): > --- current code --- > > if on_windows: > # redefine with win-specific hacks > def get_class_members(): > --- your windows code --- > > or something similar. Be mindful of the details about cygwin vs > non-cygwin environments when writing windows-specific code, please. > Also make sure you test it very well, since I can't test it myself. > > Best, > > f > > > From Fernando.Perez at colorado.edu Tue Aug 3 20:07:26 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 03 Aug 2004 18:07:26 -0600 Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython In-Reply-To: <41101E19.50803@ml.schieke.net> References: <410EC7AB.3040806@ml.schieke.net> <410F1275.5070505@colorado.edu> <41101E19.50803@ml.schieke.net> Message-ID: <411028BE.2020909@colorado.edu> Hi Jaco, Jaco Schieke wrote: > Some clarifications: > 1. The dictionary hack is simply to keep "matches" unique. It took me a > while to realize this, but pyreadline actually expects an unique array > (see lib\site-packages\readline\pyreadline.py(788): if r and r not in > completions:), otherwise it stops reading from the matches returned and > simply exits. This is not win32 specific and should be a usefull > addition should be incorporated into rlcompleter.py. An easy way to > test this is to add matches.sort() just before you return matches in > FlexCompleter.py. Then try []. ... the results are disappointing. Mmh, I just tried this and I don't see any problem at all, at least not under Unix. If this is something which is needed by pyreadline, then that's where the patch should be directed. But I've never seen Unix readline needing any of this, so I don't want to add unnecessary functionality. Sorry, but until you can show me a case where the Unix readline has problems with this setup, I'd rather not include this patch (and I suspect a similar answer may come from the rlcompleter crowd). Obviously if you do have a usage case which shows how this is necessary on all platforms, then I'll gladly include it (and eventually it should work its way towards rlcompleter, where it probably really belongs; after all, FlexCompleter shouldn't even exist if rlcompleter had all we want). But if the patch belongs in pyreadline (win32), then that's where it should go, not in ipython. > 2. I've rewritten the function to check for sys.platform and only run > win32 hacks if on the right platform - patch follows.. This looks fine. I'll wait until we settle the whole thing before applying the patch: if it's only going to be the win32 stuff, then I might ask you to resend me a standalone one so I don't accidentally mess things up applying and removing unneeded parts. Also, please heed my comments about patches which I sent yesterday; your patch from yesterday had hard tabs, and I really prefer attachments created with -u. Thanks for your efforts, this will eventually go into 0.6.3 Best, f From gb at cs.unc.edu Wed Aug 4 18:33:25 2004 From: gb at cs.unc.edu (Gary Bishop) Date: Wed, 04 Aug 2004 18:33:25 EDT Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython Message-ID: <200408042233.i74MXQah000796@fafnir.cs.unc.edu> I agree with Fernando. If pyreadline is different from GNU readline we need to fix it. I have NEVER used GNU readline on Linux. I played with it a bit on cygwin but didn't really know that I was doing when writing it. I read the manual and tried to implement what I thought it does. Let's make pyreadline work as much like the GNU readline as we can. I'll be happy to accept patches to it. gb From ipython at ml.schieke.net Thu Aug 5 17:35:50 2004 From: ipython at ml.schieke.net (Jaco Schieke) Date: Thu, 05 Aug 2004 23:35:50 +0200 Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython In-Reply-To: <200408042233.i74MXQah000796@fafnir.cs.unc.edu> References: <200408042233.i74MXQah000796@fafnir.cs.unc.edu> Message-ID: <4112A836.2080607@ml.schieke.net> OK. I need to convince you guys (any myself) first that there is a problem though. Here is the patch for a proof of concept... I think my previous description of the POC may have been ambiguous, as the change could have been implemted in more than one place in FlexCompleter.py. Having applied this to FlexCompleter.py (or even rlcompleter.py at the right place with a vanilla python shell), things like sys. fail on my system (dont have access to a proper linux system to test.) Please apply and verify. My diagnosis is that the error lies in pyreadline at \lib\site-packages\readline\pyreadline.py(788)_get_completions() with the line "if r and r not in completions:", since this will break as soon as a duplicate entry is detected. But before we discuss this further, please verify that my POC is not due to a wacky configuration on my side. Regards, Jaco Proof of concept for duplicate entry failure. After applying, do in IPython: import sys; sys. $ cvs -z3 diff -ut IPython/FlexCompleter.py Index: IPython/FlexCompleter.py =================================================================== RCS file: /home/cvsroot/world/ipython/IPython/FlexCompleter.py,v retrieving revision 1.3 diff -u -t -r1.3 FlexCompleter.py --- IPython/FlexCompleter.py 22 Aug 2003 16:04:33 -0000 1.3 +++ IPython/FlexCompleter.py 5 Aug 2004 21:48:17 -0000 @@ -183,6 +183,7 @@ for word in words: if word[:n] == attr and word != "__builtins__": matches.append("%s.%s" % (expr, word)) + matches.sort() return matches def get_class_members(klass): Gary Bishop wrote: > I agree with Fernando. If pyreadline is different from GNU readline we > need to fix it. I have NEVER used GNU readline on Linux. I played with > it a bit on cygwin but didn't really know that I was doing when > writing it. > > I read the manual and tried to implement what I thought it does. Let's > make pyreadline work as much like the GNU readline as we can. I'll be > happy to accept patches to it. > > gb > > From Fernando.Perez at colorado.edu Thu Aug 5 17:59:20 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 05 Aug 2004 15:59:20 -0600 Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython In-Reply-To: <4112A836.2080607@ml.schieke.net> References: <200408042233.i74MXQah000796@fafnir.cs.unc.edu> <4112A836.2080607@ml.schieke.net> Message-ID: <4112ADB8.1020006@colorado.edu> Jaco Schieke wrote: > OK. I need to convince you guys (any myself) first that there is a > problem though. Here is the patch for a proof of concept... I think my > previous description of the POC may have been ambiguous, as the change > could have been implemted in more than one place in FlexCompleter.py. > Having applied this to FlexCompleter.py (or even rlcompleter.py at the > right place with a vanilla python shell), things like sys. fail on > my system (dont have access to a proper linux system to test.) Please > apply and verify. Works for me here (Linux). After applying the matches.sort() patch: In [1]: import sys In [2]: sys. sys._getframe sys.exit sys.path_hooks sys.api_version sys.exitfunc sys.path_importer_cache sys.argv sys.getcheckinterval sys.platform sys.builtin_module_names sys.getdefaultencoding sys.prefix sys.byteorder sys.getdlopenflags sys.setcheckinterval sys.call_tracing sys.getfilesystemencoding sys.setdlopenflags sys.callstats sys.getrecursionlimit sys.setprofile sys.copyright sys.getrefcount sys.setrecursionlimit sys.displayhook sys.hexversion sys.settrace sys.exc_clear sys.ipcompleter sys.stderr sys.exc_info sys.maxint sys.stdin sys.exc_type sys.maxunicode sys.stdout sys.excepthook sys.meta_path sys.version sys.exec_prefix sys.modules sys.version_info sys.executable sys.path sys.warnoptions So if this is indeed a problem under WinXP, the ball would be in Gary's court, I'm afraid. Best, f From ipython at ml.schieke.net Thu Aug 5 18:23:03 2004 From: ipython at ml.schieke.net (Jaco Schieke) Date: Fri, 06 Aug 2004 00:23:03 +0200 Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython In-Reply-To: <4112ADB8.1020006@colorado.edu> References: <200408042233.i74MXQah000796@fafnir.cs.unc.edu> <4112A836.2080607@ml.schieke.net> <4112ADB8.1020006@colorado.edu> Message-ID: <4112B347.50701@ml.schieke.net> Fernando Perez wrote: > Jaco Schieke wrote: > >> OK. I need to convince you guys (any myself) first that there is a >> problem though. Here is the patch for a proof of concept... I think >> my previous description of the POC may have been ambiguous, as the >> change could have been implemted in more than one place in >> FlexCompleter.py. Having applied this to FlexCompleter.py (or even >> rlcompleter.py at the right place with a vanilla python shell), >> things like sys. fail on my system (dont have access to a proper >> linux system to test.) Please apply and verify. > > > Works for me here (Linux). After applying the matches.sort() patch: > You're right. This is windows specific - due to the usage of pyreadline. I'll rework the original COM patch for FlexCompleter.py (if you're still up for applying it) and see what Gary has to say on pyreadline. Regards, Jaco From Fernando.Perez at colorado.edu Thu Aug 5 18:30:00 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 05 Aug 2004 16:30:00 -0600 Subject: [IPython-dev] Tab-completion of data/properties and win32com fix for IPython In-Reply-To: <4112B347.50701@ml.schieke.net> References: <200408042233.i74MXQah000796@fafnir.cs.unc.edu> <4112A836.2080607@ml.schieke.net> <4112ADB8.1020006@colorado.edu> <4112B347.50701@ml.schieke.net> Message-ID: <4112B4E8.3060605@colorado.edu> Jaco Schieke wrote: > Fernando Perez wrote: >> >>Works for me here (Linux). After applying the matches.sort() patch: >> > > You're right. This is windows specific - due to the usage of > pyreadline. I'll rework the original COM patch for FlexCompleter.py (if > you're still up for applying it) and see what Gary has to say on pyreadline. Sure, that one, I have nothing against (since it doesn't affect Unix in any way :). I'd like to hear the opinion of others more experienced with Windows, though. I have zero basis for judging whether it's a good idea or not. But in principle, unless someone objects on technical grounds (with an explanation), I can put that one in. Thanks for heeding my call for -u patches, but please send them as attachments. It's much easier for me to save a standalone diff file than to mess with either cut/paste (which always has problems with some email clients, and I need to use several different ones), or saving the whole email and then retrimming it. For various reasons I use multiple mail clients, so I have no integrated tools for handling patches directly embedded into the messages. Having them as attachments with a proper filename makes the process much smoother. I know there are others who prefer inlined patches, this depends a lot on how your toolchain is setup. Sorry to insist, but given that for contributors it's a small effort, but for me that extra time is multiplied by all patches sent, I really must. Best, f From Fernando.Perez at colorado.edu Sun Aug 22 18:56:38 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sun, 22 Aug 2004 16:56:38 -0600 Subject: [IPython-dev] ipython, gtk & matplotlib support; end of current ipython development Message-ID: <412924A6.1030003@colorado.edu> Hi everyone, sorry for the cross post to those of you who are on all these lists, but since this will affect ipython's future quite a bit, I want a significant heads-up to be seen by all potentially affected. 1. PyGTK & matplotlib --------------------- Thanks to Antoon Pardon and John Hunter, ipython has nearly ready full support for interactive matplotlib with all backends. In this process, we've also added GTK threading support, so you can now use ipython for pygtk development. This code is now in IPython's CVS, and the matplotlib features require matplotlib CVS (for matplotlib use only; matplotlib has NOT become a general ipython requirement). So those of you willing to bleed a little can use it, and now is your opportunity to let us know of any problems you find. Our solution is simpler, but more limited in scope, than scipy's gui_thread. However it currently does NOT work with the WX backends, only with Tk and GTK (-AGG or not). Help from any WX guru is welcome, the place to look is at the end of IPython/Shell.py. I hope that we can find, at least in our more limited context, a working solution for WX which doesn't require all the complexity of gui_thread. In order to use this, do an 'ipython -upgrade' after a cvs update; this will get the necessary support files into ~/.ipython. You will then need to use the new threading options; copying from the docs: -gthread, -mpthread: these are special options, only one of which can be given, and which can ONLY be given as the FIRST option passed to IPython (they will have no effect in any other position). If -gthread is given, IPython starts running a separate thread for GTK operation, so that pyGTK-based programs can open GUIs without blocking IPython. The -mpthread option adds special support for the matplotlib library (http://matplotlib.sourceforge.net), allowing interactive usage of the GTK and WX backends. This also modifies the @run command to correctly execute any matplotlib-based script which calls show() at the end, without blocking. The most convenient way to use this is the new pylab profile, which should be invoked as follows (aliasing this in your shell may be a good idea): $ ipython -mpthread -p pylab pylab will honor your choice of matplotlib backend, though currently it will revert (with a warning) WX to TkAgg, since WX is broken. This will go away once we figure out the WX problems. 2. IPython's future ------------------- Once this support for matplotlib is working to satisfaction, it will mean the end of the line for any more feature-related changes to ipython for quite a while. Once this is reasonably shaken (I hope with at most one more release beyond 0.6.3, which will officially include this), I plan on beginning the long-awaited internal cleanup of ipython. Given my very limited time, this will mean essentially ZERO new features on ipython for quite a while. It will also mean that the new ipython will: - require python 2.3: I want to deprecate as much redundant code as I can from the ipython distribution. I'll use optparse and any other new module from the stdlib which can help shrink ipython. - break backwards compatibility in many areas. In particular, the ipythonrc files will become true python files. - the internal class structure of ipython will drastically change. If you have code which uses ipython via IPython.Shell, you should be fine, as I'll try to keep that API stable. If you've been poking your dirty fingers into iplib or ipmaker directly, expect things to break badly, and don't even think about complaining :) Hopefully once this is over, it will mean having a much cleaner ipython, with an easy path for including it into GUI shells (such as pycrust), and a sane internal code structure. As the new design shakes down, we'll eventually have an ipython 1.0 at last ;) Because of these changes coming down the pipe, if you have any further patches or changes for the current ipython which you'd like to see included, please send them NOW to me. Once I shift gears to the cleanup project, I'll unfortunately have to drop most changes not going in that direction, simply for lack of time. I'd like to thank everybody who has contributed to ipython's development so far, and to encourage others to join in. The cleanup should not be too hard, and it will open the door for having ipython as the interactive core for very high quality python-based environments in the future. Best regards, f From vivainio at kolumbus.fi Mon Aug 23 11:51:02 2004 From: vivainio at kolumbus.fi (Ville Vainio) Date: Mon, 23 Aug 2004 18:51:02 +0300 Subject: [IPython-dev] Re: [IPython-user] ipython, gtk & matplotlib support; end of current ipython development In-Reply-To: <412924A6.1030003@colorado.edu> References: <412924A6.1030003@colorado.edu> Message-ID: <200408231851.02741.vivainio@kolumbus.fi> On Monday 23 August 2004 01:56, Fernando Perez wrote: > far, and to encourage others to join in. The cleanup should not be too > hard, and it will open the door for having ipython as the interactive core > for very high quality python-based environments in the future. Great. Keep us posted on your ideas as you proceed :-). From jdhunter at ace.bsd.uchicago.edu Mon Aug 23 11:40:23 2004 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Mon, 23 Aug 2004 10:40:23 -0500 Subject: [IPython-dev] potential pylab bug Message-ID: First of all, I'm loving the new interactive shell. For the first time in 2 years of using python every day, I'm trying to do almost everything interactively from the shell, including updating my package from ipython w/o restarting ipython! cd ~/some/dir !sudo python setup.py install reload somemodule very nice.... But I *may* have found a bug. I have a script that produces an errobar plot which I'm calling with run in the pylab config . The script imports matplotlib.matlab and nothing more. On a call to errorbar, the ipython session hung. I could not control-c it control control-z it. A ps revealed hunter:~/python/projects/cluster/examples> ps PID TTY TIME CMD 29177 pts/0 00:00:00 tcsh 29381 pts/0 00:28:51 ipython 29405 pts/0 00:00:00 sh 29406 pts/0 00:00:00 gnuplot 29411 pts/0 00:00:00 ps hunter:~/python/projects/cluster/examples> killall -9 ipython hunter:~/python/projects/cluster/examples> ps PID TTY TIME CMD 29177 pts/0 00:00:00 tcsh 29413 pts/0 00:00:00 ps Note that gnuplot was running, and it was killed when I killed ipython! Is it possible that there is some vestigial gnuplot support creaping in? Note I have not been able to reproduce this behavior, either with the original script or a minimal errorbar script. I only encountered it in the middle of a fairly long interactive session. I just wanted to mention FYI in case it triggers something in your mind about how gnuplot might creap in. I haven't used gnuplot in years so I would be surprised if it is creaping in through on of my modules, but I can't say for sure. JDH From Fernando.Perez at colorado.edu Thu Aug 26 23:17:00 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 26 Aug 2004 21:17:00 -0600 Subject: [IPython-dev] potential pylab bug In-Reply-To: References: Message-ID: <412EA7AC.5080109@colorado.edu> John Hunter wrote: > First of all, I'm loving the new interactive shell. For the first > time in 2 years of using python every day, I'm trying to do almost > everything interactively from the shell, including updating my package > from ipython w/o restarting ipython! > > cd ~/some/dir > !sudo python setup.py install > reload somemodule > > very nice.... Great! I'm glad you're liking it. In my group, the one person already using matplotlib full time is also loving it. I'll bring some comments he has to your attention next week. I think we have something very nice in our hands here. > But I *may* have found a bug. I have a script that produces an > errobar plot which I'm calling with run in the pylab config . The > script imports matplotlib.matlab and nothing more. On a call to > errorbar, the ipython session hung. I could not control-c it control > control-z it. A ps revealed > > hunter:~/python/projects/cluster/examples> ps > PID TTY TIME CMD > 29177 pts/0 00:00:00 tcsh > 29381 pts/0 00:28:51 ipython > 29405 pts/0 00:00:00 sh > 29406 pts/0 00:00:00 gnuplot > 29411 pts/0 00:00:00 ps > > hunter:~/python/projects/cluster/examples> killall -9 ipython > hunter:~/python/projects/cluster/examples> ps > PID TTY TIME CMD > 29177 pts/0 00:00:00 tcsh > 29413 pts/0 00:00:00 ps > > Note that gnuplot was running, and it was killed when I killed > ipython! Is it possible that there is some vestigial gnuplot support > creaping in? This is very strange. The only way ipython can start gnuplot is if you import IPython.GnuplotRuntime or IPython.GnuplotInteractive. The default numeric profile DOES load the gnuplot support, but pylab doesn't. Here's an example: [~]> pylab Using IPython.Shell.IPShellMatplotlibWX with the WXAgg backend. Python 2.2.3 (#1, Oct 15 2003, 23:33:35) [GCC 3.3.1 20030930 (Red Hat Linux 3.3.1-6)] on linux2 Type "copyright", "credits" or "license" for more information. (MatplotlibShell) In [1]: !ps aux | grep gnuplot fperez 20987 0.0 0.1 4928 572 pts/12 S 21:09 0:00 grep gnuplot Try the same thing and let me know what you see. Let me also know exactly how you are invoking ipython (which profile) and what your profile does. Are you using the pylab profile as supplied by me, or did you modify it? I'm also puzzled by the fact that ps shows 'ipython', instead of 'python'. ipython is not a binary, and on my system ps shows python. Are you on a BSD or linux box (I don't know if they report differently)? Best, f ps. I just got your mail now (Thursday evening), but the date on it is 3 days ago. I wonder if there's a problem with the scipy list server (I also just got a mail I sent to scipy-user earlier this afternoon, ~6 hours ago). From Fernando.Perez at colorado.edu Mon Aug 30 22:47:46 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 30 Aug 2004 20:47:46 -0600 Subject: [IPython-dev] ANN: IPython 0.6.3 is out, with matplotlib support. In-Reply-To: References: Message-ID: <4133E6D2.9070804@colorado.edu> Hi all, IPython 0.6.3 is out at: http://ipython.scipy.org/dist. Some highlights of this release: 1. Matplotlib support The major new feature this time is integrated support for matplotlib (http://matplotlib.sourceforge.net). Thanks to the help of John Hunter (matplotlib's author) and others, IPython now has a new option -pylab. With this flag, it loads and configures matplotlib for interactive use, automatically adapting thread handling to whatever backend is configured in your .matplotlibrc file. This means that you can use Tk, GTK, or WX and IPython automatically handles threading for you so that the matplotlib figure windows do NOT block the shell. The matplotlib features have been tested and work fairly well under Linux, though it's quite possible that bugs remain (threading code is particularly tricky to debug). However, under Mac OSX and Windows we have run into difficulties which we don't know how to solve yet. It appears that for these platforms, at this point only the Tk/TkAgg backends work well. Anyone who works on these systems and knows about threads is welcome to help; the place to look is towards the end of Ipython/Shell.py, where all the thread-handling classes reside. I'd like to thank John for all the time he spent helping me with this. I think for those interested in scientific computing, the combination of matplotlib and ipython makes for a very interesting environment. Please note that this requires matplotlib version 0.62.4 or newer, as John and I coordinated the development of these features, and it required changes to both ipython and matplotlib. 2. GTK & WXPython threading There is also experimental, but disabled by default, support for generic (i.e. not specific to matplotlib) GTK and WX threading. Ideally, IPython should allow you to run arbitrary GTK or WX programs without blocking, and new -gthread/-wthread options were introduced for this purpose. However, this code is not working correctly and my threading knowledge is very limited. If anyone is interested in this kind of functionality, and can help with debugging, simply modify the start() function at the end of IPython/Shell.py to re-enable those options (they are commented out). At that point the threading code will become active again. Any fixes for this will be most welcome. 3. As usual, a number of small bugfixes recently reported are also included. Enjoy, and please report any problems. Best, Fernando Perez.