From fperez at colorado.edu  Tue Jun  8 02:42:58 2004
From: fperez at colorado.edu (Fernando Perez)
Date: Tue, 08 Jun 2004 00:42:58 -0600
Subject: [IPython-dev] Windows fixes, new prompts in CVS, testers?
Message-ID: <40C55FF2.9090106@colorado.edu>

Hi all,

I've made a bunch of changes which are now in CVS, and I'd very much
appreciate some third-party feedback from the willing.

Some highlights:

(1) Windows support _should_ now work out-of-the box for those having Gary's
readline.  He helped me with this, and says it works for him now.  Please let
me know of any problems, with as much detail as possible (also say so if it
works!)

(2) New bash-like prompts, arbitrary coloring of prompts, and a 'shell'
profile.  Yes, this is ipython:

planck[~]> ipython -p shell
Python 2.3.3 (#1, May  7 2004, 10:31:40)
Type "copyright", "credits" or "license" for more information.

IPython 0.6.1.cvs -- An enhanced Interactive Python.
?       -> Introduction to IPython's features.
@magic  -> Information about IPython's 'magic' @ functions.
help    -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

IPython profile: shell

Welcome to pysh, a set of extensions to IPython for shell usage.
      help(shell)  -- help on the installed shell extensions.
fperez at planck[~]> cd test
fperez at planck[~/test]> !!ls d*
                     <2> ['die.py', 'divc', 'div.c', 'divf', 'div.f', 'div.py']
fperez at planck[~/test]> for f in _2:
....................:     info = getoutput('ls -l %s' % f)
....................:     print 'File details:',f,'->',info
....................:
File details: die.py -> -rw-r--r--  1 fperez wavelet 124 Oct  6  2003 die.py
File details: divc -> -rwxr-xr-x  1 fperez wavelet 9988 Jul  7  2003 divc
File details: div.c -> -rw-r--r--  1 fperez wavelet 196 Jul  7  2003 div.c
File details: divf -> -rwxr-xr-x  1 fperez wavelet 11671 Jul  7  2003 divf
File details: div.f -> -rw-r--r--  1 fperez wavelet 237 Jul  7  2003 div.f
File details: div.py -> -rw-r--r--  1 fperez wavelet 49 May 12  2002 div.py


This is just to give you a simple example of the functionality.  None of this
is in the manual yet, so you'll have to look into IPython/UserConfig for the
profile (ipythonrc-shell) and associated pysh.py files.  Just copy these to
your ~/.ipythonrc directory.

The allowed prompt escapes are mainly the ones in bash, with some additions.
See the Prompts.py code for details (I'll document it later).  This code stems
from a contribution by W.J. van der Laan <gnufnork at hetdigitalegat.nl>, many
thanks to him.

Best,

f




From Fernando.Perez at colorado.edu  Tue Jun  8 20:20:56 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 08 Jun 2004 18:20:56 -0600
Subject: [IPython-dev] Installers for current CVS (basically a release
 candidate). Testers, please?
Message-ID: <40C657E8.1060906@colorado.edu>

Hi all,

in http://ipython.scipy.org/dist/testing/ I've packaged the current CVS code as:

1. The usual .tgz
2. rpms tagged with the proper Python version
3. *** a proper windows executable installer! (thanks to Cory Dodt)

These now contain all the recent features properly documented, along with many 
fixes.  For those who missed the previous mail, the highlights are:

- windows color support is supposed to work out of the box (thanks to Gary Bishop)
- bash-like customizable prompts, and a new 'shell' profile included.  Yes, I 
now have 'pysh' as an alias to 'ipython -p shell' and I use it for on-the-fly 
system scripting (thanks to W.J. van der Laan).

I would greatly appreciate feedback from Windows users on whether (3) works or 
not, since I made it 'blindly' under Linux, and have no way to run it.  If it 
does, I'll stop shipping .zip files for Windows users and continue from now on 
with this executable installer.

*** WARNING --- BACKWARDS INCOMPATIBLE CHANGE***
If you update, you'll need to fix ALL your ipythonrc files to use, in your 
prompt_in2 strings, '\D' instead of '%n'.  Otherwise your prompts won't print 
properly.  Sorry about this, but trivial as it sounds, it is nearly impossible 
for me to keep compatibility with the old syntax while implementing the new, 
much more flexible prompt system.

Cheers,

f



From Fernando.Perez at colorado.edu  Thu Jun 10 03:51:08 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 10 Jun 2004 01:51:08 -0600
Subject: [IPython-dev] A cry for help: I give up with Windows installers
Message-ID: <40C812EC.5000406@colorado.edu>

Hi all,

I'm sorry to say this, but I just give up with getting a Windows installer to 
work.  I borrowed a windows laptop for the evening, with the best of 
intentions.  I should have remembered what my last try to do some development 
under this OS was like...

I've just wasted 6 hours trying to get a trivial script, which WORKS FINE on 
its own (ipython_win_postinstall), not to crash the entire Windows installer. 
  MY script runs to completion, but the controlling installer just dies 
spectacularly at the end (binary crash, not a python exception), leaving a 
mess of incomplete logfiles for deinstallation later.

If I run the same script manually (the way it used to be for windows users), 
it executes just fine.

Under the control of the installer, there are paths which change, but once I 
fix those (so the script completes running), the binary installer just blows 
up at the end.

I've attached a copy here of a hacked version of the post-install script, in 
case any windows guru out there is willing to get this working.  This one has 
the paths fixed so it runs under the control of the binary installer, and it 
logs to a file (since print statements aren't shown).

You can make a binary installer using this post-install script via (using 
python 2.3):

python setup.py bdist_wininst --install-script=ipython_win_post_install.py


Unless someone who has more experience and knowledge than me on Windows can 
help, I'll have to revert to the old method of just shipping .zip files for 
Windows users.  It's incomprehensible to me that doing something so simple 
(copying a few files, making a few shortcuts) should be this difficult.

I think my own time is more productively spent dealing with the parts of 
ipython I can actually make some progress on.  Over the next few weeks (after 
I release 0.6.1) I'll try to get a threaded version of IPython, so that GTK 
programs can be run within ipython (such as matplotlib with the GTK backend). 
  Back to Linux...

Best,

f
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipython_win_post_install.py
Type: text/x-python
Size: 6023 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20040610/986ff1dd/attachment.py>

From thorsten at thorstenkampe.de  Thu Jun 10 07:45:06 2004
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Thu, 10 Jun 2004 13:45:06 +0200
Subject: [IPython-dev] Re: Windows fixes, new prompts in CVS, testers?
References: <40C55FF2.9090106@colorado.edu>
Message-ID: <6l5z369fybig.dlg@thorstenkampe.de>

* Fernando Perez (2004-06-08 08:42 +0100)
> I've made a bunch of changes which are now in CVS, and I'd very much
> appreciate some third-party feedback from the willing.
> 
> Some highlights:
> 
> (1) Windows support _should_ now work out-of-the box for those having Gary's
> readline.  He helped me with this, and says it works for him now.  Please let
> me know of any problems, with as much detail as possible (also say so if it
> works!)

The coloured out prompt now works.

But there is now a problem only with the Windows version (not the
Cygwin):

WARNING: Couldn't start log: [Errno 17] File exists
Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)]

This corresponds to these settings in ipythonrc:
log 0
logfile ~/.ipython/ipython.log backup

Thorsten



From Fernando.Perez at colorado.edu  Thu Jun 10 11:54:30 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 10 Jun 2004 09:54:30 -0600
Subject: [IPython-dev] Re: Windows fixes, new prompts in CVS, testers?
In-Reply-To: <6l5z369fybig.dlg@thorstenkampe.de>
References: <40C55FF2.9090106@colorado.edu> <6l5z369fybig.dlg@thorstenkampe.de>
Message-ID: <40C88436.6030603@colorado.edu>

Thorsten Kampe wrote:
> * Fernando Perez (2004-06-08 08:42 +0100)

> The coloured out prompt now works.

Thanks for the confirmation.

WINDOWS USERS: please note that I'm thinking about making Gary's readline 
(which in turn requires ctypes) REQUIREMENTS for ipython.  It makes ipyhton 
under windows provide most of the linux functionality, and it finally seems to 
work pretty well.

If anyone objects, now is the time to say so.

> But there is now a problem only with the Windows version (not the
> Cygwin):
> 
> WARNING: Couldn't start log: [Errno 17] File exists
> Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)]
> 
> This corresponds to these settings in ipythonrc:
> log 0
> logfile ~/.ipython/ipython.log backup

Bummer.  I'll try to borrow again a windows computer to see if I can test 
this.  I suspect the problem is the expansion of '~' under Windows.  Please 
try to see if you can set this path explicitly, since I suspect that '~' 
doesn't expand to anything sensible under Windows.

Cheers,

f



From thorsten at thorstenkampe.de  Thu Jun 10 12:54:59 2004
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Thu, 10 Jun 2004 18:54:59 +0200
Subject: [IPython-dev] Re: Re: Windows fixes, new prompts in CVS, testers?
References: <40C55FF2.9090106@colorado.edu>
	<6l5z369fybig.dlg@thorstenkampe.de> <40C88436.6030603@colorado.edu>
Message-ID: <k7i4tkdr6ac3.dlg@thorstenkampe.de>

* Fernando Perez (2004-06-10 17:54 +0100)
> Thorsten Kampe wrote:
>> * Fernando Perez (2004-06-08 08:42 +0100)
>> The coloured out prompt now works.
> 
> Thanks for the confirmation.
> 
> WINDOWS USERS: please note that I'm thinking about making Gary's readline 
> (which in turn requires ctypes) REQUIREMENTS for ipython.  It makes ipyhton 
> under windows provide most of the linux functionality, and it finally seems to 
> work pretty well.
> 
> If anyone objects, now is the time to say so.

Gary Bishop's readline (which seems to me the best of the usable
Windows readline clones) unfortunately makes typing for me on my
Windows XP machine painfully slow (about one character per second)[1].

With his readline I can neither use IPython nor plain vanilla Python.
 
>> But there is now a problem only with the Windows version (not the
>> Cygwin):
>> 
>> WARNING: Couldn't start log: [Errno 17] File exists
>> Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)]
>> 
>> This corresponds to these settings in ipythonrc:
>> log 0
>> logfile ~/.ipython/ipython.log backup
> 
> Bummer.  I'll try to borrow again a windows computer to see if I can test 
> this.  I suspect the problem is the expansion of '~' under Windows.  Please 
> try to see if you can set this path explicitly, since I suspect that '~' 
> doesn't expand to anything sensible under Windows.

Sorry, I was too laconic. This only happens when
IPYTHONDIR/ipython.log and IPYTHONDIR/ipython.log~ exist( I'm using
the same directory for the Windows and Cygwin IPython). When I delete
the files Windows IPython shows no errors and creates those files.
When both exist IPython errors on the next IPython startup.

Thorsten

[1] No on-access virus scanner running



From gb at cs.unc.edu  Thu Jun 10 13:16:37 2004
From: gb at cs.unc.edu (Gary Bishop)
Date: Thu, 10 Jun 2004 13:16:37 EDT
Subject: [IPython-dev] Slow typing with windows readline?
Message-ID: <200406101716.i5AHGaii000604@fafnir.cs.unc.edu>

Thorsten reports really slow typing with my readline code. I've seen 
one other person reporting that.

Is that the experience of most? It flies for me. Looks just like typing 
at the command prompt or into python normally. I've used it on several 
computers always with great performance.

So, I'll need some help trying to track down what slows it down. I 
can't believe it is computing for all that time! 1 second is a billion 
instructions for these modern computers. Lets say you have a really 
only computer and it's only 100 million instructions, that is still one 
heck of a lot of computing for a single character. The code doesn't do 
much; it simply get the character and dispatches to a function.

Could it be some difference in the way folks have configured their 
console windows?

Any help appreciated.
gb



From Fernando.Perez at colorado.edu  Thu Jun 10 13:18:37 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 10 Jun 2004 11:18:37 -0600
Subject: [IPython-dev] Re: Re: Windows fixes, new prompts in CVS, testers?
In-Reply-To: <k7i4tkdr6ac3.dlg@thorstenkampe.de>
References: <40C55FF2.9090106@colorado.edu>
	<6l5z369fybig.dlg@thorstenkampe.de> <40C88436.6030603@colorado.edu>
	<k7i4tkdr6ac3.dlg@thorstenkampe.de>
Message-ID: <40C897ED.3020402@colorado.edu>

Thorsten Kampe wrote:
> * Fernando Perez (2004-06-10 17:54 +0100)
> 
>>Thorsten Kampe wrote:
>>
>>>* Fernando Perez (2004-06-08 08:42 +0100)
>>>The coloured out prompt now works.
>>
>>Thanks for the confirmation.
>>
>>WINDOWS USERS: please note that I'm thinking about making Gary's readline 
>>(which in turn requires ctypes) REQUIREMENTS for ipython.  It makes ipyhton 
>>under windows provide most of the linux functionality, and it finally seems to 
>>work pretty well.
>>
>>If anyone objects, now is the time to say so.
> 
> 
> Gary Bishop's readline (which seems to me the best of the usable
> Windows readline clones) unfortunately makes typing for me on my
> Windows XP machine painfully slow (about one character per second)[1].
> 
> With his readline I can neither use IPython nor plain vanilla Python.

Well, someone objected :)

Ok, I won't make it a requirement then.  I'll try to add in code to enable 
coloring by default if possible, I think I now know how to auto-detect Gary's 
readline for those who have it.  This will only work at the time of creating 
the user's .ipython directory, so those who already have it will need to 
enable coloring manually.

If you ever pin down the origin of this weird keyboard behavior, let us know. 
  Others may have the same problem, and if it's likely to be common I can add 
it to the ipython manual.  Note that since you see the problem with vanilla 
python, at least I'm off the hook on this one :)

For the record, yesterday in my futile attempts at getting a good windows 
installer working, at least I could confirm that with ctypes+gary_readline, 
ipython indeed gives nice out-of-the-box coloring and tab completion for me. 
And I didn't get any of Thorsten's keyboard slugishness problems.

>>>But there is now a problem only with the Windows version (not the
>>>Cygwin):
>>>
>>>WARNING: Couldn't start log: [Errno 17] File exists
>>>Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)]
>>>
>>>This corresponds to these settings in ipythonrc:
>>>log 0
>>>logfile ~/.ipython/ipython.log backup
>>
>>Bummer.  I'll try to borrow again a windows computer to see if I can test 
>>this.  I suspect the problem is the expansion of '~' under Windows.  Please 
>>try to see if you can set this path explicitly, since I suspect that '~' 
>>doesn't expand to anything sensible under Windows.
> 
> 
> Sorry, I was too laconic. This only happens when
> IPYTHONDIR/ipython.log and IPYTHONDIR/ipython.log~ exist( I'm using
> the same directory for the Windows and Cygwin IPython). When I delete
> the files Windows IPython shows no errors and creates those files.
> When both exist IPython errors on the next IPython startup.

well, from Logger.py, here's the code that does this:

         if self.LOGMODE == 'backup':
             if os.path.isfile(self.logfname):
                 os.rename(self.logfname,self.logfname+'~')
             self.logfile = open(self.logfname,'w')

I wonder if os.rename and open() work with different semantics under Windows. 
  If open('foo','w') is called and 'foo' exists, under posix it gets 
overwritten.  It seems that under windows an exception is raised.  Can you 
confirm that?  The joys of writing portable code...

Never mind... From the docs:

rename(  	src, dst)
     Rename the file or directory src to dst. If dst is a directory, OSError 
will be raised. On Unix, if dst exists and is a file, it will be removed 
silently if the user has permission. The operation may fail on some Unix 
flavors if src and dst are on different filesystems. If successful, the 
renaming will be an atomic operation (this is a POSIX requirement). On 
Windows, if dst already exists, OSError will be raised even if it is a file; 
there may be no way to implement an atomic rename when dst names an existing file.

Ok, I'll just enable a workaround for windows. Stupid windows, stupid.

Best,

f



From Fernando.Perez at colorado.edu  Thu Jun 10 13:23:05 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 10 Jun 2004 11:23:05 -0600
Subject: [IPython-dev] Slow typing with windows readline?
In-Reply-To: <200406101716.i5AHGaii000604@fafnir.cs.unc.edu>
References: <200406101716.i5AHGaii000604@fafnir.cs.unc.edu>
Message-ID: <40C898F9.3010201@colorado.edu>

Gary Bishop wrote:
> Thorsten reports really slow typing with my readline code. I've seen 
> one other person reporting that.
> 
> Is that the experience of most? It flies for me. Looks just like typing 
> at the command prompt or into python normally. I've used it on several 
> computers always with great performance.

Well, yesterday I tested it on a borrowed windows computer and it felt just 
fine for me.  But my vast expertise under windows is well-known around here, 
so I won't embarrass myself trying to guess any further...

Good luck tracking this one down, and let me know of anything I can do under 
ipython to contribute.

Best,

f



From Fernando.Perez at colorado.edu  Thu Jun 10 13:30:35 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 10 Jun 2004 11:30:35 -0600
Subject: [IPython-dev] Re: Re: Windows fixes, new prompts in CVS, testers?
In-Reply-To: <40C897ED.3020402@colorado.edu>
References: <40C55FF2.9090106@colorado.edu>
	<6l5z369fybig.dlg@thorstenkampe.de> <40C88436.6030603@colorado.edu>
	<k7i4tkdr6ac3.dlg@thorstenkampe.de> <40C897ED.3020402@colorado.edu>
Message-ID: <40C89ABB.1060307@colorado.edu>

Fernando Perez wrote:
> Thorsten Kampe wrote:

>>>>But there is now a problem only with the Windows version (not the
>>>>Cygwin):
>>>>
>>>>WARNING: Couldn't start log: [Errno 17] File exists
>>>>Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)]
>>>>
>>>>This corresponds to these settings in ipythonrc:
>>>>log 0
>>>>logfile ~/.ipython/ipython.log backup

[snip]

> Ok, I'll just enable a workaround for windows. Stupid windows, stupid.

Committed to CVS, let me know if it works (at least it doesn't break anything 
under posix).

Cheers,

f



From thorsten at thorstenkampe.de  Thu Jun 10 20:58:49 2004
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Fri, 11 Jun 2004 02:58:49 +0200
Subject: [IPython-dev] Re: Slow typing with windows readline?
References: <200406101716.i5AHGaii000604@fafnir.cs.unc.edu>
Message-ID: <of3txru45azu$.dlg@thorstenkampe.de>

* Gary Bishop (2004-06-10 19:16 +0100)
> Thorsten reports really slow typing with my readline code. I've seen 
> one other person reporting that.
> [...]
> Could it be some difference in the way folks have configured their 
> console windows?

It is this[1] setting in my cmd.exe startup file which sets the height
and width of the command window to 87x45 and also the scroll buffer
size of the Window to 87x9999 (which is the maximum for the horizontal
scroll buffer). When you make this setting through the GUI you get a
warning that a scroll buffer of 9999 might use a lot of RAM (but it
doesn't). For instance I have a scroll buffer in rxvt of 65535 lines
and it is no problem for Python.

Your simple patch works beautifully...

Thorsten

[1] reg add HKCU\Console /v ScreenBufferSize /t REG_DWORD /d 655294551 /f



From thorsten at thorstenkampe.de  Thu Jun 10 20:58:53 2004
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Fri, 11 Jun 2004 02:58:53 +0200
Subject: [IPython-dev] Re: Re: Re: Windows fixes, new prompts in CVS,
	testers?
References: <40C55FF2.9090106@colorado.edu>
	<6l5z369fybig.dlg@thorstenkampe.de> <40C88436.6030603@colorado.edu>
	<k7i4tkdr6ac3.dlg@thorstenkampe.de> <40C897ED.3020402@colorado.edu>
	<40C89ABB.1060307@colorado.edu>
Message-ID: <vokqjd1v6wo2.dlg@thorstenkampe.de>

* Fernando Perez (2004-06-10 19:30 +0100)
> Fernando Perez wrote:
>> Thorsten Kampe wrote:
>>>>>But there is now a problem only with the Windows version (not the
>>>>>Cygwin):
>>>>>
>>>>>WARNING: Couldn't start log: [Errno 17] File exists
>>>>>Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)]
>>>>>
>>>>>This corresponds to these settings in ipythonrc:
>>>>>log 0
>>>>>logfile ~/.ipython/ipython.log backup
> 
> [snip]
> 
>> Ok, I'll just enable a workaround for windows. Stupid windows, stupid.
> 
> Committed to CVS, let me know if it works (at least it doesn't break anything 
> under posix).

No, I made a new install[1] but the error message is still the same.
At least it doesn't break Cygwin IPython.

Thorsten

[1] C:\Python23\python.exe setup.py install -O1 -f



From gb at cs.unc.edu  Thu Jun 10 21:08:23 2004
From: gb at cs.unc.edu (Gary Bishop)
Date: Thu, 10 Jun 2004 21:08:23 EDT
Subject: [IPython-dev] Re: Re: Re: Windows fixes, new prompts in CVS,
	testers?
Message-ID: <200406110108.i5B18MoJ024095@fafnir.cs.unc.edu>

Could it be the permissions on the file as created by cygwin? I've had 
occasional problems with permissions resulting from using both.
gb



From viktor.ransmayr at t-online.de  Fri Jun 11 05:14:01 2004
From: viktor.ransmayr at t-online.de (Viktor Ransmayr)
Date: Fri, 11 Jun 2004 11:14:01 +0200
Subject: [IPython-dev] Re: [IPython-user] A cry for help: I give up with
	Windows installers
In-Reply-To: <40C812EC.5000406@colorado.edu>
References: <40C812EC.5000406@colorado.edu>
Message-ID: <40C977D9.8000001@t-online.de>

Hi Fernando,

I've tried to build the windows-installer from source using 
"IPython-0.6.1.cvs.tar.gz".

Executing  "python setup.py bdist_wininst 
--install-script=ipython_win_post_install.py"
I get the following traceback:

removing 'build\bdist.win32\wininst' (and everything under it)
Traceback (most recent call last):
  File "setup.py", line 130, in ?
    ipython_win_post_install.run(wait=1)
  File 
"D:\Environments\Python\IPython\IPython-0.6.1.cvs\ipython_win_post_install.py", 
line 83, in run
    shutil.copy(sys.prefix+'\\Scripts\\ipython',ip_filename)
  File "D:\Python23\lib\shutil.py", line 71, in copy
    copyfile(src, dst)
  File "D:\Python23\lib\shutil.py", line 37, in copyfile
    fsrc = open(src, 'rb')
IOError: [Errno 2] No such file or directory: 
'D:\\Python23\\Scripts\\ipython'

The windows-installer is created however ...

The way I interprete it is that setup.py (?) doesn't create the 
subdirectory "ipython".

Since it's the first time that I play around w/ distutils I failed to 
provide a patch. -  I hope that I provided
enough info for you to fix the problem. - I'm happy to try out any new 
version from your side.

Regards,

    Viktor

PS: For completeness I'm attaching the stripped build log.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: bdist_wininst_stripped_log
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20040611/d4b1866b/attachment.ksh>

From gb at cs.unc.edu  Fri Jun 11 10:23:13 2004
From: gb at cs.unc.edu (Gary Bishop)
Date: Fri, 11 Jun 2004 10:23:13 EDT
Subject: [IPython-dev] Re: Slow typing with windows readline?
Message-ID: <200406111423.i5BENDHq030055@fafnir.cs.unc.edu>

I have posted a new release of my readline package at uncpythontools at 
SourceForge.

https://sourceforge.net/project/showfiles.php?group_id=82407&package_id=84552&release_id=245063

This fixes, I hope, the bug Thorsten and a few others noticed with slow 
typing. Also fixes, the bug with changing the console color to bright 
white; now it uses whatever the console is set for (I think).

gb



From Fernando.Perez at colorado.edu  Fri Jun 11 13:31:29 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 11 Jun 2004 11:31:29 -0600
Subject: [IPython-dev] A cry for help: I give up with Windows installers
In-Reply-To: <40C977D9.8000001@t-online.de>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
Message-ID: <40C9EC71.5050907@colorado.edu>

Viktor Ransmayr wrote:
> Hi Fernando,
> 
> I've tried to build the windows-installer from source using 
> "IPython-0.6.1.cvs.tar.gz".
> 
> Executing  "python setup.py bdist_wininst 
> --install-script=ipython_win_post_install.py"
> I get the following traceback:
> 
> removing 'build\bdist.win32\wininst' (and everything under it)
> Traceback (most recent call last):
>   File "setup.py", line 130, in ?
>     ipython_win_post_install.run(wait=1)
>   File 
> "D:\Environments\Python\IPython\IPython-0.6.1.cvs\ipython_win_post_install.py", 
> line 83, in run
>     shutil.copy(sys.prefix+'\\Scripts\\ipython',ip_filename)
>   File "D:\Python23\lib\shutil.py", line 71, in copy
>     copyfile(src, dst)
>   File "D:\Python23\lib\shutil.py", line 37, in copyfile
>     fsrc = open(src, 'rb')
> IOError: [Errno 2] No such file or directory: 
> 'D:\\Python23\\Scripts\\ipython'

OK, I get it.  It's my mistake, I never thought the installer would be _built_ 
under windows, so my setup.py is kind of hacked.  I'll try to fix it and 
provide an updated one tomorrow.  Can you get CVS code?

Many thanks for your help, hopefully with a bit of collaboration we can fix 
the windows issues.  Especially now that Gary's stuff is working so well, I'd 
love to provide full support for Windows users.

Best,

f



From Fernando.Perez at colorado.edu  Fri Jun 11 13:34:46 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 11 Jun 2004 11:34:46 -0600
Subject: [IPython-dev] Re: Re: Re: Windows fixes, new prompts in CVS,
	testers?
In-Reply-To: <200406110108.i5B18MoJ024095@fafnir.cs.unc.edu>
References: <200406110108.i5B18MoJ024095@fafnir.cs.unc.edu>
Message-ID: <40C9ED36.3010405@colorado.edu>

Gary Bishop wrote:
> Could it be the permissions on the file as created by cygwin? I've had 
> occasional problems with permissions resulting from using both.
> gb

Thorsten, do you have a feel for whether this could be the problem?  I really 
don't understand why it doesn't work.  Here's the code:

         if self.LOGMODE == 'backup':
             if os.path.isfile(self.logfname):
                 backup_logname = self.logfname+'~'
                 # Manually remove any old backup, since os.remove may fail
                 # under Windows.
                 if os.path.isfile(backup_logname):
                     os.remove(backup_logname)
                 os.rename(self.logfname,backup_logname)
             self.logfile = open(self.logfname,'w')

It _manually_ does the file removal of the backup first, before attempting the 
rename operation.  If this doesn't work, I simply don't know what else one can 
do to force windows to delete the backup and rename the current file as backup.

Cheers,

f



From viktor.ransmayr at t-online.de  Fri Jun 11 15:43:04 2004
From: viktor.ransmayr at t-online.de (Viktor Ransmayr)
Date: Fri, 11 Jun 2004 21:43:04 +0200
Subject: [IPython-dev] A cry for help: I give up with Windows installers
In-Reply-To: <40C9EC71.5050907@colorado.edu>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu>
Message-ID: <40CA0B48.8090709@t-online.de>

Hi Fernando,

You wrote:

> I wrote:
>
>> Hi Fernando,
>>
>> I've tried to build the windows-installer from source using 
>> "IPython-0.6.1.cvs.tar.gz".
>>
>> Executing  "python setup.py bdist_wininst 
>> --install-script=ipython_win_post_install.py"
>> I get the following traceback:
>>
>> removing 'build\bdist.win32\wininst' (and everything under it)
>> Traceback (most recent call last):
>>   File "setup.py", line 130, in ?
>>     ipython_win_post_install.run(wait=1)
>>   File 
>> "D:\Environments\Python\IPython\IPython-0.6.1.cvs\ipython_win_post_install.py", 
>> line 83, in run
>>     shutil.copy(sys.prefix+'\\Scripts\\ipython',ip_filename)
>>   File "D:\Python23\lib\shutil.py", line 71, in copy
>>     copyfile(src, dst)
>>   File "D:\Python23\lib\shutil.py", line 37, in copyfile
>>     fsrc = open(src, 'rb')
>> IOError: [Errno 2] No such file or directory: 
>> 'D:\\Python23\\Scripts\\ipython'
>
>
> OK, I get it.  It's my mistake, I never thought the installer would be 
> _built_ under windows, so my setup.py is kind of hacked.  I'll try to 
> fix it and provide an updated one tomorrow.  Can you get CVS code?
>
Yes, I can. - At the moment I'm mostly working w/ the CVS-Variant  that 
is distributed w/ MSYS.
- However should this turn out to cause problems I'd switch to whatever 
variant you suggest.

> Many thanks for your help, hopefully with a bit of collaboration we 
> can fix the windows issues.  Especially now that Gary's stuff is 
> working so well, I'd love to provide full support for Windows users.

I'm glad, if I can help.

Regards,

    Viktor



From thorsten at thorstenkampe.de  Sat Jun 12 06:42:10 2004
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Sat, 12 Jun 2004 12:42:10 +0200
Subject: [IPython-dev] Re: Re: Re: Re: Windows fixes, new prompts in CVS,
	testers?
References: <200406110108.i5B18MoJ024095@fafnir.cs.unc.edu>
	<40C9ED36.3010405@colorado.edu>
Message-ID: <unrv57wgb00w.dlg@thorstenkampe.de>

* Fernando Perez (2004-06-11 19:34 +0100)
> Gary Bishop wrote:
>> Could it be the permissions on the file as created by cygwin? I've had 
>> occasional problems with permissions resulting from using both.
>> gb
> 
> Thorsten, do you have a feel for whether this could be the problem?  I really 
> don't understand why it doesn't work.  Here's the code:
> [...]
> It _manually_ does the file removal of the backup first, before attempting the 
> rename operation.  If this doesn't work, I simply don't know what else one can 
> do to force windows to delete the backup and rename the current file as backup.

Sorry, it was my fault. I updated IPython cvs to ~/ipython but
reinstalled from $IPYTHONDIR/ipython (where the old cvs was). Your new
cvs version solved the problem with "logfile backup" (and there are no
problems with rotate/append/over).

Two things still irritate me about the Windows version of IPython:

The different name for ipythonrc (ipythonrc.ini) which makes it
impossible to use ipythonrc for Cygwin IPython and Windows IPython.

And the ^M in the history files that IPython doesn't ignore in the
history file (so you could use a common history file for Windows and
Cygwin IPython).

Thorsten



From Fernando.Perez at colorado.edu  Sat Jun 12 09:14:23 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sat, 12 Jun 2004 07:14:23 -0600
Subject: [IPython-dev] Re: Re: Re: Re: Windows fixes, new prompts in CVS, 
	testers?
In-Reply-To: <unrv57wgb00w.dlg@thorstenkampe.de>
References: <200406110108.i5B18MoJ024095@fafnir.cs.unc.edu>
	<40C9ED36.3010405@colorado.edu> <unrv57wgb00w.dlg@thorstenkampe.de>
Message-ID: <40CB01AF.7010105@colorado.edu>

Thorsten Kampe wrote:
> Sorry, it was my fault. I updated IPython cvs to ~/ipython but
> reinstalled from $IPYTHONDIR/ipython (where the old cvs was). Your new
> cvs version solved the problem with "logfile backup" (and there are no
> problems with rotate/append/over).

OK. Bug closed.

> Two things still irritate me about the Windows version of IPython:
> 
> The different name for ipythonrc (ipythonrc.ini) which makes it
> impossible to use ipythonrc for Cygwin IPython and Windows IPython.

Since windows manages all file associations for editing via extensions, I 
didn't want to leave the files extension-less.  This problem will ultimately 
disappear, since as I rewrite the config system, ipython will be only 
configured via proper python files instead of the fake rc syntax.

> And the ^M in the history files that IPython doesn't ignore in the
> history file (so you could use a common history file for Windows and
> Cygwin IPython).

As I said, this should be dealt with at the readline C level, via proper 
universal lineend support.  Feel free to follow it up with the Python team via 
SF's bug system.  It is not ipython's fault, and a workaround isn't 
particularly clean to implement (it's possible, but I only see ugly hacks as 
viable options).

f.



From thorsten at thorstenkampe.de  Sat Jun 12 11:55:54 2004
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Sat, 12 Jun 2004 17:55:54 +0200
Subject: [IPython-dev] 
	Re: Re: Re: Re: Re: Windows fixes, new prompts in CVS, testers?
References: <200406110108.i5B18MoJ024095@fafnir.cs.unc.edu>
	<40C9ED36.3010405@colorado.edu> <unrv57wgb00w.dlg@thorstenkampe.de>
	<40CB01AF.7010105@colorado.edu>
Message-ID: <27hbo7vssutt.dlg@thorstenkampe.de>

* Fernando Perez (2004-06-12 15:14 +0100)
> Thorsten Kampe wrote:
>> Sorry, it was my fault. I updated IPython cvs to ~/ipython but
>> reinstalled from $IPYTHONDIR/ipython (where the old cvs was). Your new
>> cvs version solved the problem with "logfile backup" (and there are no
>> problems with rotate/append/over).
> 
> OK. Bug closed.
> 
>> Two things still irritate me about the Windows version of IPython:
>> 
>> The different name for ipythonrc (ipythonrc.ini) which makes it
>> impossible to use ipythonrc for Cygwin IPython and Windows IPython.
> 
> Since windows manages all file associations for editing via extensions, I 
> didn't want to leave the files extension-less. This problem will ultimately 
> disappear, since as I rewrite the config system, ipython will be only 
> configured via proper python files instead of the fake rc syntax.

I hope in an "ordered" way. Comparing my pythonrc.py to my ipythonrc,
the .py looks like an unordered mess - and it has much less content.

Thorsten



From gb at cs.unc.edu  Sun Jun 13 18:12:52 2004
From: gb at cs.unc.edu (Gary Bishop)
Date: Sun, 13 Jun 2004 18:12:52 EDT
Subject: [IPython-dev] ^M in history file 
Message-ID: <200406132212.i5DMCpH8002446@fafnir.cs.unc.edu>

I have uploaded yet another release (1.4) of my readline module for 
windows. The only change is the elimination of the carriage-return 
(Ctrl-M) characters in the history file for compatibility with cygwin.

gb



From Fernando.Perez at colorado.edu  Mon Jun 14 01:58:07 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sun, 13 Jun 2004 23:58:07 -0600
Subject: [IPython-dev] A cry for help: I give up with Windows installers
In-Reply-To: <40CA0B48.8090709@t-online.de>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu> <40CA0B48.8090709@t-online.de>
Message-ID: <40CD3E6F.9090600@colorado.edu>

Viktor Ransmayr wrote:

>>>I've tried to build the windows-installer from source using 
>>>"IPython-0.6.1.cvs.tar.gz".
>>>
>>>Executing  "python setup.py bdist_wininst 
>>>--install-script=ipython_win_post_install.py"
>>>I get the following traceback:

[snip]

>>OK, I get it.  It's my mistake, I never thought the installer would be 
>>_built_ under windows, so my setup.py is kind of hacked.  I'll try to 
>>fix it and provide an updated one tomorrow.  Can you get CVS code?

Sorry for the delay.  I just committed a fixed setup.py which should allow 
building the installer under Windows.  I can't test it, so I hope I didn't 
screw up.

Let me know how it goes.

Best,

f



From viktor.ransmayr at t-online.de  Mon Jun 14 06:44:05 2004
From: viktor.ransmayr at t-online.de (Viktor Ransmayr)
Date: Mon, 14 Jun 2004 12:44:05 +0200
Subject: [IPython-dev] Windows installer - Round II
In-Reply-To: <40CD3E6F.9090600@colorado.edu>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu> <40CA0B48.8090709@t-online.de>
	<40CD3E6F.9090600@colorado.edu>
Message-ID: <40CD8175.7040705@t-online.de>

Hi Fernando,

You wrote:

> Viktor Ransmayr wrote:
>
>>>> I've tried to build the windows-installer from source using 
>>>> "IPython-0.6.1.cvs.tar.gz".
>>>>
>>>> Executing  "python setup.py bdist_wininst 
>>>> --install-script=ipython_win_post_install.py"
>>>> I get the following traceback:
>>>
>
> [snip]
>
>>> OK, I get it.  It's my mistake, I never thought the installer would 
>>> be _built_ under windows, so my setup.py is kind of hacked.  I'll 
>>> try to fix it and provide an updated one tomorrow.  Can you get CVS 
>>> code?
>>
>
> Sorry for the delay.  I just committed a fixed setup.py which should 
> allow building the installer under Windows.  I can't test it, so I 
> hope I didn't screw up.
>
> Let me know how it goes.
>
Good and bad news.

Good news first: I can build the windows installer from the latest 
CVS-Source
w/o problems.

Bad news: If I run the produced installer, I get the identical traceback as
previously reported.

### For the record, here's my work-log again:

2004-06-14 11:54:54

o Donloaded latest CVS-version of IPython into
  D:\Environments\Python\IPython\IPython-cvs\


2004-06-14 11:59:33

o Try to make a binary installer using the post-install script:
  "python setup.py bdist_wininst 
--install-script=ipython_win_post_install.py"
 - Everything seems fine, i.e. binary installer gets produced, no 
traceback get
   produced by the script. - Saved logfile as ipython_bdist_20040614.log.


2004-06-14 12:10:17

o Install the self-created IPython-0.6.1.cvs.win32.exe
 - Follow the instructions in "Upgrading from a previous version" and verify
   that the directory C:\Dokumente und Einstellungen\Viktor 
RANSMAYR\.ipython
   doesn't exist.

o Installation failed w/ the following traceback:

Traceback (most recent call last):
  File "D:\Python23\Scripts\ipython_win_post_install.py", line 126, in ?
    run()
  File "D:\Python23\Scripts\ipython_win_post_install.py", line 86, in run
    shutil.copytree('doc',doc_dir)
  File "D:\Python23\lib\shutil.py", line 100, in copytree
    names = os.listdir(src)
WindowsError: [Errno 3] Das System kann den angegebenen Pfad nicht 
finden: 'doc/*.*'
*** run_installscript: internal error 0xFFFFFFFF ***


2004-06-14 12:28:49

o De-Install Python-0.6.1.cvs
 - A similar traceback than above occured. Again it couldn't be copied, 
however
   I verified it line by line. - It is identical to the one above, 
beside the
   very last line "*** run_installscript: internal error 0xFFFFFFFF ***"
   which is missing.
 - De-installation succeeded partially, i.e. it removed the
   directory D:\Python23\Lib\site-packages\IPython\ but did not remove
   directory C:\Programme\IPython.
 - Delete the directories C:\Programme\IPython.

### End of work-log.

I'll try myself to dig deeper into this issue too, but don't hold your 
breath ...

Regards,

    Viktor




From Fernando.Perez at colorado.edu  Mon Jun 14 19:57:27 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 14 Jun 2004 17:57:27 -0600
Subject: [IPython-dev] Windows installer - Round II
In-Reply-To: <40CD8175.7040705@t-online.de>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu> <40CA0B48.8090709@t-online.de>
	<40CD3E6F.9090600@colorado.edu> <40CD8175.7040705@t-online.de>
Message-ID: <40CE3B67.2090909@colorado.edu>

Viktor Ransmayr wrote:

> Bad news: If I run the produced installer, I get the identical traceback as
> previously reported.
> 
> ### For the record, here's my work-log again:
> 
> 2004-06-14 11:54:54
> 
> o Donloaded latest CVS-version of IPython into
>   D:\Environments\Python\IPython\IPython-cvs\
> 
> 
> 2004-06-14 11:59:33
> 
> o Try to make a binary installer using the post-install script:
>   "python setup.py bdist_wininst 
> --install-script=ipython_win_post_install.py"
>  - Everything seems fine, i.e. binary installer gets produced, no 
> traceback get
>    produced by the script. - Saved logfile as ipython_bdist_20040614.log.
> 
> 
> 2004-06-14 12:10:17
> 
> o Install the self-created IPython-0.6.1.cvs.win32.exe
>  - Follow the instructions in "Upgrading from a previous version" and verify
>    that the directory C:\Dokumente und Einstellungen\Viktor 
> RANSMAYR\.ipython
>    doesn't exist.
> 
> o Installation failed w/ the following traceback:
> 
> Traceback (most recent call last):
>   File "D:\Python23\Scripts\ipython_win_post_install.py", line 126, in ?
>     run()
>   File "D:\Python23\Scripts\ipython_win_post_install.py", line 86, in run
>     shutil.copytree('doc',doc_dir)
>   File "D:\Python23\lib\shutil.py", line 100, in copytree
>     names = os.listdir(src)
> WindowsError: [Errno 3] Das System kann den angegebenen Pfad nicht 
> finden: 'doc/*.*'
> *** run_installscript: internal error 0xFFFFFFFF ***

Well, that's where it was getting stuck before, and all my attempts to fix 
things failed.  You may try to use the hacked ipython_win_post_install.py I 
posted.  That one has changes to the paths so that it runs to completion, 
avoiding the traceback above.  However, though MY script runs correctly 
(verified by an internally generated logfile which records its end), the 
Windows binary GUI installer dies spectacularly.

If you can debug that problem, then my hacked ipython_win_post_install.py 
could become the official one.

Cheers,

f



From thorsten at thorstenkampe.de  Wed Jun 16 17:10:55 2004
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Wed, 16 Jun 2004 23:10:55 +0200
Subject: [IPython-dev] IPython cvs crash when pasting text
Message-ID: <1gbbwed4b80i3.dlg@thorstenkampe.de>

The latest cvs build crashes each time when I try to paste multiline
text under platform Cygwin. From the crash report: "AttributeError:
Prompt2 instance has no attribute 'col_num'".

Thorsten



From Fernando.Perez at colorado.edu  Wed Jun 16 18:12:48 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 16 Jun 2004 16:12:48 -0600
Subject: [IPython-dev] IPython cvs crash when pasting text
In-Reply-To: <1gbbwed4b80i3.dlg@thorstenkampe.de>
References: <1gbbwed4b80i3.dlg@thorstenkampe.de>
Message-ID: <40D0C5E0.20002@colorado.edu>

Thorsten Kampe wrote:
> The latest cvs build crashes each time when I try to paste multiline
> text under platform Cygwin. From the crash report: "AttributeError:
> Prompt2 instance has no attribute 'col_num'".

 From the notice about the CVS code in my recent email:

*** WARNING --- BACKWARDS INCOMPATIBLE CHANGE***
If you update, you'll need to fix ALL your ipythonrc files to use, in your
prompt_in2 strings, '\D' instead of '%n'.  Otherwise your prompts won't print
properly.  Sorry about this, but trivial as it sounds, it is nearly impossible
for me to keep compatibility with the old syntax while implementing the new,
much more flexible prompt system.

**************

Sorry, but I really can't avoid making this change, incompatible as it is.

But I _did_ just commit to CVS a hack which at least will make it not crash. 
The prompt display for the continuations will be wrong, but it won't crash 
anymore.

Cheers,

f



From viktor.ransmayr at t-online.de  Sun Jun 20 07:12:19 2004
From: viktor.ransmayr at t-online.de (Viktor Ransmayr)
Date: Sun, 20 Jun 2004 13:12:19 +0200
Subject: [IPython-dev] Windows installer - Progress Report
In-Reply-To: <40CE3B67.2090909@colorado.edu>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu> <40CA0B48.8090709@t-online.de>
	<40CD3E6F.9090600@colorado.edu> <40CD8175.7040705@t-online.de>
	<40CE3B67.2090909@colorado.edu>
Message-ID: <40D57113.5070009@t-online.de>

Hi Fernando Perez

You wrote:

> Viktor Ransmayr wrote:
>
>> Bad news: If I run the produced installer, I get the identical 
>> traceback as
>> previously reported.
>>
>> ### For the record, here's my work-log again:
>>
>> 2004-06-14 11:54:54
>>
>> o Donloaded latest CVS-version of IPython into
>>   D:\Environments\Python\IPython\IPython-cvs\
>>
>>
>> 2004-06-14 11:59:33
>>
>> o Try to make a binary installer using the post-install script:
>>   "python setup.py bdist_wininst 
>> --install-script=ipython_win_post_install.py"
>>  - Everything seems fine, i.e. binary installer gets produced, no 
>> traceback get
>>    produced by the script. - Saved logfile as 
>> ipython_bdist_20040614.log.
>>
>>
>> 2004-06-14 12:10:17
>>
>> o Install the self-created IPython-0.6.1.cvs.win32.exe
>>  - Follow the instructions in "Upgrading from a previous version" and 
>> verify
>>    that the directory C:\Dokumente und Einstellungen\Viktor 
>> RANSMAYR\.ipython
>>    doesn't exist.
>>
>> o Installation failed w/ the following traceback:
>>
>> Traceback (most recent call last):
>>   File "D:\Python23\Scripts\ipython_win_post_install.py", line 126, in ?
>>     run()
>>   File "D:\Python23\Scripts\ipython_win_post_install.py", line 86, in 
>> run
>>     shutil.copytree('doc',doc_dir)
>>   File "D:\Python23\lib\shutil.py", line 100, in copytree
>>     names = os.listdir(src)
>> WindowsError: [Errno 3] Das System kann den angegebenen Pfad nicht 
>> finden: 'doc/*.*'
>> *** run_installscript: internal error 0xFFFFFFFF ***
>
>
> Well, that's where it was getting stuck before, and all my attempts to 
> fix things failed.  You may try to use the hacked 
> ipython_win_post_install.py I posted.  That one has changes to the 
> paths so that it runs to completion, avoiding the traceback above.  
> However, though MY script runs correctly (verified by an internally 
> generated logfile which records its end), the Windows binary GUI 
> installer dies spectacularly.

I found the reason why the Windows installer failed:

if os_name=='windows' and 'install' in sys.argv[1]:     # *not* argv

However, there is no need to call the post-install script at all in the 
setup-script.
This is done automatically by distutils.

With the attached versions of setup.py and 'ipython_win_post_install.py' 
I can
run a *complete* and *successful* installation/ deinstallation cycle.

For the moment I just want to document the facts and the current status. - I
attach portions of my work-log:

2004-06-20 12:31:28

o Prepare a *minimal* version to send to FP and the mailing list.

o Run 'bdist_wininst'.

o Install 'IPython-0.6.1.cvs.win32.exe'
 - Finished w/o a glitch; i.e.
 - C:\Programme\IPython\IPython.py created.
 - Menu 'Start > Alle Programme > IPython' created empty.

o Running C:\Programme\IPython\IPython.py in IDLE w/ 'Run Module' fails w/

### Begin of IDLE Output ###

 >>> ================================ RESTART 
================================
 >>>

Traceback (most recent call last):
  File "C:\Programme\IPython\IPython.py", line 26, in -toplevel-
    import IPython
  File "C:\Programme\IPython\IPython.py", line 27, in -toplevel-
    IPython.Shell.IPShell().mainloop()
AttributeError: 'module' object has no attribute 'Shell'
 >>>

### End of IDLE Output ###

o De-Install 'Python 2.3 IPython-0.6.1.cvs'
 - Finished w/o *any* glitch.
 - However it reported that 1 files or directories could not be removed.
  * C:\Programme\IPython\ - Most likely because I had the script 
'IPython.py'
    open in IDLE.

o Remove C:\Programme\IPython\ manually ...


2004-06-20 12:49:46

o Proofed my theory w/ another installation/ deinstallation cycle.
 - Everything got properly removed ...

### End of work-log ###

But the way it looks like, this is just the start of the road to a working
Windows Installer. - I'll start a follow-up discussion in a future mail.

Regards,
    Viktor


-------------- next part --------------
A non-text attachment was scrubbed...
Name: setup.py
Type: text/x-python
Size: 5206 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20040620/f5a16a32/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipython_win_post_install.py
Type: text/x-python
Size: 4437 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20040620/f5a16a32/attachment-0001.py>

From viktor.ransmayr at t-online.de  Sun Jun 20 17:07:42 2004
From: viktor.ransmayr at t-online.de (Viktor Ransmayr)
Date: Sun, 20 Jun 2004 23:07:42 +0200
Subject: [IPython-dev] Windows installer - Clarification
In-Reply-To: <40D57113.5070009@t-online.de>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu> <40CA0B48.8090709@t-online.de>
	<40CD3E6F.9090600@colorado.edu> <40CD8175.7040705@t-online.de>
	<40CE3B67.2090909@colorado.edu> <40D57113.5070009@t-online.de>
Message-ID: <40D5FC9E.8040808@t-online.de>

Hi Fernando,

I wrote:

> I found the reason why the Windows installer failed:
>
> if os_name=='windows' and 'install' in sys.argv[1]:     # *not* argv
>
> However, there is no need to call the post-install script at all in 
> the setup-script.
> This is done automatically by distutils.
>
> With the attached versions of setup.py and 
> 'ipython_win_post_install.py' I can
> run a *complete* and *successful* installation/ deinstallation cycle.
>
> For the moment I just want to document the facts and the current 
> status. - I
> attach portions of my work-log:
>
> 2004-06-20 12:31:28
>
> o Prepare a *minimal* version to send to FP and the mailing list.
>
> o Run 'bdist_wininst'.
>
> o Install 'IPython-0.6.1.cvs.win32.exe'
> - Finished w/o a glitch; i.e.
> - C:\Programme\IPython\IPython.py created.
> - Menu 'Start > Alle Programme > IPython' created empty.
>
> o Running C:\Programme\IPython\IPython.py in IDLE w/ 'Run Module' 
> fails w/
>
> ### Begin of IDLE Output ###
>
> >>> ================================ RESTART 
> ================================
> >>>
>
> Traceback (most recent call last):
>  File "C:\Programme\IPython\IPython.py", line 26, in -toplevel-
>    import IPython
>  File "C:\Programme\IPython\IPython.py", line 27, in -toplevel-
>    IPython.Shell.IPShell().mainloop()
> AttributeError: 'module' object has no attribute 'Shell'
> >>>
>
> ### End of IDLE Output ###
>
> o De-Install 'Python 2.3 IPython-0.6.1.cvs'
> - Finished w/o *any* glitch.
> - However it reported that 1 files or directories could not be removed.
>  * C:\Programme\IPython\ - Most likely because I had the script 
> 'IPython.py'
>    open in IDLE.
>
> o Remove C:\Programme\IPython\ manually ...
>
>
> 2004-06-20 12:49:46
>
> o Proofed my theory w/ another installation/ deinstallation cycle.
> - Everything got properly removed ...
>
> ### End of work-log ###
>
> But the way it looks like, this is just the start of the road to a 
> working
> Windows Installer. - I'll start a follow-up discussion in a future mail.

Before I can start this discussion I need to make sure that I start w/ the
right source-base, when I create the Windows Installer :-)

Is my understanding correct, that as a packager I have to start w/
a source distribution and *not* w/ a CVS snapshot if I stay solely in
a Windows environment ?

Regards,
    Viktor



From Fernando.Perez at colorado.edu  Sun Jun 20 19:13:32 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sun, 20 Jun 2004 17:13:32 -0600
Subject: [IPython-dev] Windows installer - Clarification
In-Reply-To: <40D5FC9E.8040808@t-online.de>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu> <40CA0B48.8090709@t-online.de>
	<40CD3E6F.9090600@colorado.edu> <40CD8175.7040705@t-online.de>
	<40CE3B67.2090909@colorado.edu> <40D57113.5070009@t-online.de>
	<40D5FC9E.8040808@t-online.de>
Message-ID: <40D61A1C.9060404@colorado.edu>

Viktor Ransmayr wrote:

> Before I can start this discussion I need to make sure that I start w/ the
> right source-base, when I create the Windows Installer :-)
> 
> Is my understanding correct, that as a packager I have to start w/
> a source distribution and *not* w/ a CVS snapshot if I stay solely in
> a Windows environment ?

No, you should be using CVS code always for development.  It's the closest 
thing in existence (and identical when I submit) to what's sitting on my 
computer.  The actual releases may lag by a lot with respect to what's in CVS.

A few more points, concerning development:

- Please make sure you follow my coding standard (which is the normal python 
one).  The only thing I really care about:

      *************   NO HARD TABS ANYWHERE IN THE FILE  **************

Use spaces for indenting, 4 spaces per level.  I've had bugs creep in from 
patches sent in which used tabs instead of spaces.  So now I scan all patches 
for tabs in them, and will outright reject any patch with a single tab in it. 
  Sorry but the potential for subtle bugs is just too high, it has already bit me.

- Please send unified diff patches for smoother integration, you can make them 
against CVS or against a local copy.  Here are some useful tutorials on 'best 
practices' for ensuring a smooth workflow:

http://www.kegel.com/academy/opensource.html
http://www.linux.com/howtos/Software-Release-Practice-HOWTO/index.shtml

It's ok to send patches as attachments instead of inline, just make sure you 
name them with a .diff extension so they load nicely highlighted into XEmacs 
right away for me to study them.

I don't want to sound patronizing, but I simply have no idea what your level 
of experience with collaborative code development is.  If you're already 
familiar with all of this, great.  If you are new to this kind of project, 
spending a little time reading these two tutorials will be a very good 
investment for the future.  We want to spend our time coding, not fighting 
diff and patch :)

Thanks again for your work.

Best,

Fernando.



From Fernando.Perez at colorado.edu  Sun Jun 20 19:17:15 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sun, 20 Jun 2004 17:17:15 -0600
Subject: [IPython-dev] Windows installer - Progress Report
In-Reply-To: <40D57113.5070009@t-online.de>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu> <40CA0B48.8090709@t-online.de>
	<40CD3E6F.9090600@colorado.edu> <40CD8175.7040705@t-online.de>
	<40CE3B67.2090909@colorado.edu> <40D57113.5070009@t-online.de>
Message-ID: <40D61AFB.6040607@colorado.edu>

Hi Viktor,

Viktor Ransmayr wrote:
> I found the reason why the Windows installer failed:
> 
> if os_name=='windows' and 'install' in sys.argv[1]:     # *not* argv
> 
> However, there is no need to call the post-install script at all in the 
> setup-script.
> This is done automatically by distutils.
> 
> With the attached versions of setup.py and 'ipython_win_post_install.py' 
> I can
> run a *complete* and *successful* installation/ deinstallation cycle.

That's great news.  If this works in the end, I'll remove that magic and leave 
setup.py alone, so that:

1. Users of a .zip download who are assumed to be familiar enough with 
handling source installs, can do the normal

$ python setup.py install

No 'magic' auto-adding of the install argument under Windows, which is a gross 
hack to begin with.

2. Users of the graphical installer will simply double-click on the executable 
and it will run to completion.

We're on the right track, great :)

Best,

f



From Fernando.Perez at colorado.edu  Sun Jun 20 20:30:54 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sun, 20 Jun 2004 18:30:54 -0600
Subject: [IPython-dev] Windows installer - Progress Report
In-Reply-To: <40D57113.5070009@t-online.de>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu> <40CA0B48.8090709@t-online.de>
	<40CD3E6F.9090600@colorado.edu> <40CD8175.7040705@t-online.de>
	<40CE3B67.2090909@colorado.edu> <40D57113.5070009@t-online.de>
Message-ID: <40D62C3E.8000104@colorado.edu>

Viktor Ransmayr wrote:

> For the moment I just want to document the facts and the current status. - I
> attach portions of my work-log:
> 
> 2004-06-20 12:31:28
> 
> o Prepare a *minimal* version to send to FP and the mailing list.
> 
> o Run 'bdist_wininst'.
> 
> o Install 'IPython-0.6.1.cvs.win32.exe'
>  - Finished w/o a glitch; i.e.
>  - C:\Programme\IPython\IPython.py created.
>  - Menu 'Start > Alle Programme > IPython' created empty.

This needs fixing: my old hacked one was able to create the proper shortcuts, 
and that's ultimately our goal.  We want to get shortcuts made _both_ by the 
graphic installer and by a manual 'python setup.py install'.

By the way, I think my old hack re-copied the documentation from the dir where 
the source got unpacked over to C:\Program...\IPython.  But those docs are 
already being copied by setup.py.  It would be much better to just make the 
shortcuts point to the right place instead of making a second copy.  How to 
actually find at runtime where distutils ends up putting them may take a bit 
of work, I've never understood 100% how distutils makes all of its path 
decisions.  And it irks me to no end that there's no way to get a 'return' 
value from setup.py with all the information about what ended up where.

Good luck on this front: distutils is probably the single most annoying Python 
technology in existence (as much as I love most of Python, distutils pisses me 
off every single time I use it).

To summarize the goal: we want to have

1. A source installer which when called from a command-line as 'setup.py 
install', works

2. A GUI-driven .exe installer which also works to completion

Both 1 and 2 should put ipython where it belongs along with the documentation, 
and leave StartMenu shortcuts for the HTML/PDF manual, and also for 
new_design.pdf (same doc/ directory).

2 should allow ipython to be de-installed with the normal windows process. 
Users of 1 will need to clean up manually, but that's fine.  That's also how 
it works under Unix if a source-based install is made, there's no automatic 
de-installation.

> o Running C:\Programme\IPython\IPython.py in IDLE w/ 'Run Module' fails w/
> 
> ### Begin of IDLE Output ###
> 
>  >>> ================================ RESTART 
> ================================
>  >>>
> 
> Traceback (most recent call last):
>   File "C:\Programme\IPython\IPython.py", line 26, in -toplevel-
>     import IPython
>   File "C:\Programme\IPython\IPython.py", line 27, in -toplevel-
>     IPython.Shell.IPShell().mainloop()
> AttributeError: 'module' object has no attribute 'Shell'
>  >>>
> 
> ### End of IDLE Output ###

No surprise here.  Ipyhton is _not_ meant to run inside IDLE, at least not 
yet.  Idle changes the environment way too much for ipython to run correctly, 
among other things it mangles stdin/out/err.

Current CVS actually _kind of_ runs inside idle, but this is a total hack. 
For proper testing, use IPython from a normal terminal, with Gary's readline 
installed (I added to CVS code to complain --but not crash-- if Gary's lib is 
missing).

If you _also_ have cygwin, even better: it would be great if you could test in 
both environments.

I greatly appreciate your help on this front.  My famously catastrophic 
Windows support only got worse recently:  the only computer with Windows I 
could occasionally borrow died, and my wife is replacing it with a Macintosh 
Powerbook.  While this means the joy of never dealing with Windows again 
anymore for me, it also means I don't have any kind of access to the platform 
anymore.  So now _all_ windows-specific testing and development will have to 
be done by users.

Best regards,

Fernando.



From Fernando.Perez at colorado.edu  Mon Jun 21 03:58:21 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 21 Jun 2004 01:58:21 -0600
Subject: [IPython-dev] Fairly deep changes in CVS, testers welcome
Message-ID: <40D6951D.2060709@colorado.edu>

Hi all,

I know I recently put out a 'release candidate', but please bear with me. 
I've committed a raft of pretty serious changes which do touch the critical 
inner loops in ipython, so I'm a bit worried and would really like some feedback.

I hope to lure you in because the offered functionality is somewhat 
interesting, IMHO.  I finally got around to rewriting the whole ipython alias 
subsystem.  The old one was based on a trick which felt neat when I was in my 
2nd week of python learning (dynamic code generation and runtime modification 
of the running instance method table).  In reality it was just a heavyweight 
hack, which could be replaced by a simple, light dictionary.

With the new code in place, we now have:

In [1]: len(__IP.alias_table)
Out[1]: 24

In [2]: rehash  # takes ~ 0.3 s to run on my laptop

In [3]: len(__IP.alias_table)
Out[3]: 3652

rehash is a new magic which works just like the one in tcsh.  Which means we 
can now do things like:

In [4]: python
Python 2.2.3 (#1, Oct 15 2003, 23:33:35)
[GCC 3.3.1 20030930 (Red Hat Linux 3.3.1-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>>

In [5]: latex
This is TeX, Version 3.14159 (Web2C 7.4.5)
**
! End of file on the terminal... why?

and:

In [6]: PATH='a normal python string'

In [7]: echo $PATH
a normal python string

In [8]: echo $$PATH
/usr/local/lf9560/bin:/usr/local/intel/compiler70/ia32/bin: ... [SNIP]

the last two rely on new escaping functionality I also introduced.  You can 
now call system commands (whether aliases or via !, !!, @sc and @sx) and 
expand any python expression you want (see PEP-215 at 
http://www.python.org/peps/pep-0215.html for details).  A double $$ will pass 
a single $ to the shell so you can also access normal shell vars.

All of this means that now pysh (my alias for 'ipython -p shell') is a much 
more realistic environment as a shell.  In particular, I rewrote the readline 
support so that it's now trivial to define the order of priorities for tab 
completion.  The orders are now:

- pysh matchers = ['file_matches','alias_matches','python_matches']

- mormal matchers = ['python_matches','file_matches','alias_matches']

Run pysh (well, 'ipython -p shell') and let me know how it feels, and how it 
actually performs.

The only really major thing missing from pysh to be usable as a light shell (I 
don't claim it to be in the same league as bash/tcsh) is that all of this 
magic only applies for single-line statements.  If you want to write 
multi-line code, you still need to call system() yourself and do your own 
variable expansions.

I actually don't see this as a big drawback:  this case is much less common, 
and if you are already typing multiple lines, being a bit explicit on exactly 
what you want is probably a good thing.  I only wanted the 'quick and dirty' 
single-line statements to be as expressive as possible.  As always, ipython 
tries to give as much 'punch per keystroke' as it can.

In summary, the new goodies are:

- Python variable expansion in all system access mechanisms (!, !!, @sc, @sx 
and aliases).

- Lightweight alias table and new @rehash/@rehashx magics for pulling all of 
$PATH as aliases.

- shell profile which uses these (it starts with a @rehash), and reorders 
tab-completion so files come first.

I'm going to let these ideas simmer in CVS for a while before pushing them 
further.  The changelog has more details, as usual.  And Viktor Ransmayr is 
working hard on the nasty distutils-windows battle, so I want to give that 
time to mature as well.

In particular, @rehash/x are very Unix-specific.  I don't know how to deal 
with $PATH and how to make executability decisions under Windows, which (I 
think) has a baroque extension-based mechanism for this.  If someone else 
wants to write this code, look at Magic.py for the Unix examples.


I would very much appreciate feedback, both of success and (hopefully not too 
much) of problems.  As I said, these ideas forced me to really cut in places I 
normally try to stay away from, so there's a chance I could have broken 
something big.

Best,

f



From Fernando.Perez at colorado.edu  Mon Jun 21 11:55:20 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 21 Jun 2004 09:55:20 -0600
Subject: [IPython-dev] Re: [IPython-user] Re: Fairly deep changes in CVS,
	testers welcome
In-Reply-To: <ddlr75pshcla$.dlg@thorstenkampe.de>
References: <40D6951D.2060709@colorado.edu>
	<ddlr75pshcla$.dlg@thorstenkampe.de>
Message-ID: <40D704E8.3050304@colorado.edu>

Thorsten Kampe wrote:
> Under Cygwin:

> 1>>> rehash
> ---------------------------------------------------------------------------
> OSError                                   Traceback (most recent call last)
> 
> /home/thorsten/<console> 
> 
> /usr/lib/python2.3/site-packages/IPython/Magic.py in magic_rehash(self=<IPython.iplib.InteractiveShell instance>, parameter_s='')
>    1755         alias_table = self.shell.alias_table
>    1756         for pdir in path:
> -> 1757             for ff in os.listdir(pdir):
>         ff = 'w9xpopen.exe', global os = <module 'os' from '/usr/lib/python2.3/os.pyc'>, global listdir = undefined, pdir = 'Z'
>    1758                 # each entry in the alias table must be
> (N,name), where
>    1759                 # N is the number of positional arguments of
> the alias.
> 
> OSError: [Errno 2] No such file or directory: 'Z'

Thanks, fixed in CVS.  Like tcsh's rehash, it now just ignores items in $PATH 
which aren't directoires.

Cheers,

f



From viktor.ransmayr at t-online.de  Mon Jun 21 15:53:57 2004
From: viktor.ransmayr at t-online.de (Viktor Ransmayr)
Date: Mon, 21 Jun 2004 21:53:57 +0200
Subject: [IPython-dev] Windows installer - Clarification
In-Reply-To: <40D61A1C.9060404@colorado.edu>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu> <40CA0B48.8090709@t-online.de>
	<40CD3E6F.9090600@colorado.edu> <40CD8175.7040705@t-online.de>
	<40CE3B67.2090909@colorado.edu> <40D57113.5070009@t-online.de>
	<40D5FC9E.8040808@t-online.de> <40D61A1C.9060404@colorado.edu>
Message-ID: <40D73CD5.8050807@t-online.de>

Hello Fernando,

You wrote:

> No, you should be using CVS code always for development.  It's the 
> closest thing in existence (and identical when I submit) to what's 
> sitting on my computer.  The actual releases may lag by a lot with 
> respect to what's in CVS.

Thanks for clarification. - The reason, why I brought this up, is that 
if I work from a
CVS snapshot on Windows, I miss the HTML- and PDF-Version of the 
documentation,
which was needed in the original version of your post-install script.

> I don't want to sound patronizing, but I simply have no idea what your 
> level of experience with collaborative code development is.  If you're 
> already familiar with all of this, great.  If you are new to this kind 
> of project, spending a little time reading these two tutorials will be 
> a very good investment for the future.  We want to spend our time 
> coding, not fighting diff and patch :)

No need for apologies and/or discussion. - I should have send *.diff's 
in the first place ...

Regards,
    Viktor



From viktor.ransmayr at t-online.de  Mon Jun 21 16:07:58 2004
From: viktor.ransmayr at t-online.de (Viktor Ransmayr)
Date: Mon, 21 Jun 2004 22:07:58 +0200
Subject: [IPython-dev] Windows installer - Progress Report
In-Reply-To: <40D62C3E.8000104@colorado.edu>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu> <40CA0B48.8090709@t-online.de>
	<40CD3E6F.9090600@colorado.edu> <40CD8175.7040705@t-online.de>
	<40CE3B67.2090909@colorado.edu> <40D57113.5070009@t-online.de>
	<40D62C3E.8000104@colorado.edu>
Message-ID: <40D7401E.9080609@t-online.de>

Hello Fernando,

You wrote:
[...]

> This needs fixing: my old hacked one was able to create the proper 
> shortcuts, and that's ultimately our goal.  We want to get shortcuts 
> made _both_ by the graphic installer and by a manual 'python setup.py 
> install'.

I was just interested in getting *any* binary version working. - The 
'other excuse' was stated
in my previous mail: I'm missing the HTML- and PDF-Version of the 
documentation, when
working from a CVS-Snapshot on Windows.
[...]

> No surprise here.  Ipyhton is _not_ meant to run inside IDLE, at least 
> not yet.  Idle changes the environment way too much for ipython to run 
> correctly, among other things it mangles stdin/out/err.
>
I'm not sure about this. - The only reason, why I used IDLE was to 
'save/copy' the traceback;
 i.e. executing IPython from C:\Programme\IPython\IPython.py didn't work 
either ...

> Current CVS actually _kind of_ runs inside idle, but this is a total 
> hack. For proper testing, use IPython from a normal terminal, with 
> Gary's readline installed (I added to CVS code to complain --but not 
> crash-- if Gary's lib is missing).
>
> If you _also_ have cygwin, even better: it would be great if you could 
> test in both environments.

I don't use Cygwin, but I do have a working MinGW/MSYS 'environment' running
on my Win XP Laptop.

I'll keep you informed about the progress I make ...

Regards,
    Viktor



From Fernando.Perez at colorado.edu  Mon Jun 21 16:40:40 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 21 Jun 2004 14:40:40 -0600
Subject: [IPython-dev] Windows installer - Clarification
In-Reply-To: <40D73CD5.8050807@t-online.de>
References: <40C812EC.5000406@colorado.edu> <40C977D9.8000001@t-online.de>
	<40C9EC71.5050907@colorado.edu> <40CA0B48.8090709@t-online.de>
	<40CD3E6F.9090600@colorado.edu> <40CD8175.7040705@t-online.de>
	<40CE3B67.2090909@colorado.edu> <40D57113.5070009@t-online.de>
	<40D5FC9E.8040808@t-online.de> <40D61A1C.9060404@colorado.edu>
	<40D73CD5.8050807@t-online.de>
Message-ID: <40D747C8.8060008@colorado.edu>

Viktor Ransmayr wrote:
> Hello Fernando,
> 
> You wrote:
> 
> 
>>No, you should be using CVS code always for development.  It's the 
>>closest thing in existence (and identical when I submit) to what's 
>>sitting on my computer.  The actual releases may lag by a lot with 
>>respect to what's in CVS.
> 
> 
> Thanks for clarification. - The reason, why I brought this up, is that 
> if I work from a
> CVS snapshot on Windows, I miss the HTML- and PDF-Version of the 
> documentation,
> which was needed in the original version of your post-install script.

Ah, I'd forgotten about this.  The build process really depends on a bunch of 
unix tools, specifically lyxport (which pulls in perl, lyx, latex, latex2html, 
pdflatex,...).  I don't know how feasible it is for you to get all of this 
working under Windows, I suspect it would be a nightmare.

lyxport (http://www-hep.colorado.edu/~fperez/lyxport) is something I wrote a 
while back and which I use to manage the IPython documentation.  You could get 
by with keeping a copy of the html/pdf docs from the release tarballs, and the 
actual code from CVS.  It won't be perfect, but it should let you make all 
functional tests.

Once the setup code is working, I can do the actual release builds on a unix 
box where everything will get built in the right order.

Sorry for this hack, I never thought development would be done under Windows.

Thanks,

f



From pythondev-dang at lazytwinacres.net  Thu Jun 24 09:15:22 2004
From: pythondev-dang at lazytwinacres.net (Daniel 'Dang' Griffith)
Date: Thu, 24 Jun 2004 18:15:22 +0500
Subject: [IPython-dev] rehash on Win*
Message-ID: <20040624131523.4712.qmail@server265.com>

   >  > In particular, @rehash/x are very Unix-specific.??I don't know how to
   >  deal
   >  > with $PATH and how to make executability decisions under Windows,
   >  which (I
   >  > think) has a baroque extension-based mechanism for this.??If someone else
   >  
   >  Why don't you just make some well-known extensions executable? exe, bat,
   >  cmd, py, pl? It's much better than nothing, and works for 99% of the cases.

You can get the extensions from os.environ['pathext']:

In [27]: import os
In [28]: print os.environ['pathext'].split(';')
['.PYC', '.PY', '.COM', '.EXE', '.BAT', '.CMD', '.VBS', '.VBE', '.JS', '.JSE', '.WSF', '.WSH']

This list is in the order that the shell searches for them.  (I've added pyc and py
to mine, so I can run python scripts without entering an extension (and regularly
reconsider whether I should have pyc before or after py).)

I've not had a chance to look at rehash, yet, but something else that might matter
is that the Win* shell is not case sensitive, so python.exe and PYTHON.EXE can
both be executed by entering either upper or lowercase 'python'.

    --dang



From Fernando.Perez at colorado.edu  Thu Jun 24 14:44:43 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 24 Jun 2004 12:44:43 -0600
Subject: [IPython-dev] Re: rehash on Win*
In-Reply-To: <20040624131523.4712.qmail@server265.com>
References: <20040624131523.4712.qmail@server265.com>
Message-ID: <40DB211B.9090709@colorado.edu>

Daniel 'Dang' Griffith wrote:
>    >  > In particular, @rehash/x are very Unix-specific.  I don't know how to
>    >  deal
>    >  > with $PATH and how to make executability decisions under Windows,
>    >  which (I
>    >  > think) has a baroque extension-based mechanism for this.  If someone else
>    >  
>    >  Why don't you just make some well-known extensions executable? exe, bat,
>    >  cmd, py, pl? It's much better than nothing, and works for 99% of the cases.
> 
> You can get the extensions from os.environ['pathext']:
> 
> In [27]: import os
> In [28]: print os.environ['pathext'].split(';')
> ['.PYC', '.PY', '.COM', '.EXE', '.BAT', '.CMD', '.VBS', '.VBE', '.JS', '.JSE', '.WSF', '.WSH']
> 
> This list is in the order that the shell searches for them.  (I've added pyc and py
> to mine, so I can run python scripts without entering an extension (and regularly
> reconsider whether I should have pyc before or after py).)

Thanks.  Right now this is a settable string by the user, should I instead use 
this $pathext variable?  Which is the more natural approach for Windows users? 
  If $pathext is universal under windows, I could then leave the 
windows-specific code as:

try:
	ext=os.environ['pathext'].replace(';','|').replace('.','')
except KeyError:
	ext='exe|com|bat'

How does this sound?  I need Windows users' feedback to make this decision.

Cheers,

f



From Fernando.Perez at colorado.edu  Thu Jun 24 14:57:43 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 24 Jun 2004 12:57:43 -0600
Subject: [IPython-dev] Re: [IPython-user] Fairly deep changes in CVS,
	testers welcome
In-Reply-To: <40DAC6E3.1050305@kolumbus.fi>
References: <40D6951D.2060709@colorado.edu> <40DAC6E3.1050305@kolumbus.fi>
Message-ID: <40DB2427.4040907@colorado.edu>

Ville Vainio wrote:
> On Mon, Jun 21, 2004 at 01:58:21AM -0600, Fernando Perez wrote:

>  > The only really major thing missing from pysh to be usable as a light 
> shell
>  > (I don't claim it to be in the same league as bash/tcsh) is that all of
>  > this magic only applies for single-line statements.  If you want to write
>  > multi-line code, you still need to call system() yourself and do your own
>  > variable expansions.
> 
> Is there a particular reason for this? Couldn't the same expansion routines
> be applied to all lines that start with "!" or "!!"? The need for this is
> very typical in cases like:
> 
>  >>>> !!find . | grep wav
> 
> (verify that files are correct)
> 
>  >>>> wavs = _
> 
>  >>>> for w in wavs:
> ....   !oggenc $w

Well, the time machine strikes again :)  In my private copy I actually have it 
working:

fperez[~]|1> cd test
fperez[~/test]|2> $$a=ls
fperez[~/test]|3> for f in a:
               |.>     wc -l $f
               |.>
4 argv.py
1 bar.py
2 bar.pyc
9 die.py
14 div.c
14 div.f
3 div.py
113 error.py

I need to clean some things before this goes into CVS, it's not ready even for 
that yet.

The $ syntax is a pysh-only extension (not for normal IPython) which does:

$var=cmd <=> @sc var=cmd and  $$var=cmd <=> @sc -l var=cmd

Very useful for rapidly building _python_ variables from shell output.

In multiline input, only aliases and ! will be allowed, not magics or !!. 
Magics are too tied to controlling ipython itself, so it doesn't make sense to 
run them in a python loop.  And !! is meant to return a value into a list, but 
I don't want to allow something like

	a=!!ls

that's just gross.  In case you need list-split output inside a loop, I added 
a 'split' keyword argument to getoutput(), so this works: 
getoutput('ls',split=1) will give you a newline-split list.

I made up

sout = lambda cmd: getoutput(cmd,split=1)

and stuck it into my ipythonrc file, so I can say inside a loop:

   alist = sout('ls')

and then operate on alist as needed.

>  > In particular, @rehash/x are very Unix-specific.  I don't know how to 
> deal
>  > with $PATH and how to make executability decisions under Windows, 
> which (I
>  > think) has a baroque extension-based mechanism for this.  If someone else
> 
> Why don't you just make some well-known extensions executable? exe, bat,
> cmd, py, pl? It's much better than nothing, and works for 99% of the cases.

See Dang's comment.  Together with some code sent privately by Gary Bishop, 
the issue for Win users should be ok, once I commit (in a couple of days).

> I'll try to check it out (literally) after midsummer routines, though my
> familiarity with ipython (codebase and in general) possibly isn't good
> enough to make informed judgements just yet. Apparently ipython is just now
> starting to address my needs, it's just good to see that you have
> interest/ambition to invade the shell domain after all :-).

Well, I wasn't planning on it.  But I recently realized that once the alias 
infrastructure was cleaned up, it really was much less work than I'd 
originally thought.  And now I'm using it as my shell, which is really nice :)

Cheers,

f



From pythondev-dang at lazytwinacres.net  Thu Jun 24 15:04:24 2004
From: pythondev-dang at lazytwinacres.net (Daniel 'Dang' Griffith)
Date: Fri, 25 Jun 2004 00:04:24 +0500
Subject: [IPython-dev] Re: Re: rehash on Win*
Message-ID: <20040624190424.30234.qmail@server265.com>

   >  >
   >  > You can get the extensions from os.environ['pathext']:
   >  >
   >  > In [27]: import os
   >  > In [28]: print os.environ['pathext'].split(';')
   >  > ['.PYC', '.PY', '.COM', '.EXE', '.BAT', '.CMD', '.VBS', '.VBE', '.JS', '.JSE', '.WSF', '.WSH']
   >  >
   >  Thanks.??Right now this is a settable string by the user, should I instead use
   >  this $pathext variable???Which is the more natural approach for Windows users?
   >  ??If $pathext is universal under windows, I could then leave the
   >  windows-specific code as:

I realized after I sent the above that I don't know if it works on Win9x.
I know it works on WinNt, Win2k and WinXp.  I don't know how much of
your user base that might affect.

I don't want to suggest too much, yet, as I've still not had a chance to
look at the rehash/alias Magic py file.  I know it doesn't seem to work
"as expected" (whatever that means) on my machine.  The 'echo' posted
earlier is not a good test on Windows, since 'echo' is a built-in, like 'dir',
and so isn't anywhere in the PATH.  (Well, it might be, if some has an
echo.exe or whatever, but it's ignored, because Win* processes built-ins
before searching the PATH.)

I'll try to get to the code tonight, but the weather's been nice for a few 
days, which means all of the yardwork I've been enjoying neglecting is 
now needing to be done.
    --dang



From Fernando.Perez at colorado.edu  Thu Jun 24 15:12:19 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 24 Jun 2004 13:12:19 -0600
Subject: [IPython-dev] Re: Re: rehash on Win*
In-Reply-To: <20040624190424.30234.qmail@server265.com>
References: <20040624190424.30234.qmail@server265.com>
Message-ID: <40DB2793.3030905@colorado.edu>

Daniel 'Dang' Griffith wrote:

> I'll try to get to the code tonight, but the weather's been nice for a few 
> days, which means all of the yardwork I've been enjoying neglecting is 
> now needing to be done.

Please wait a bit til I commit what I've done over the last few days.  That 
way I'll get feedback on the more 'final' version, which will incorporate 
windows extension handling and multiline specials.

But I'm busy now, so it may not be until the weekend.  I'll be sure to drop a 
line as soon as the commits are in on the list.

Cheers,

f



From Fernando.Perez at colorado.edu  Sat Jun 26 04:29:01 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sat, 26 Jun 2004 02:29:01 -0600
Subject: [IPython-dev] pysh is in CVS
Message-ID: <40DD33CD.8000506@colorado.edu>

Hi all,

I've put all of pysh's functionality (including @rehashx for Windows, based on 
the $pathext environment variable) into CVS.  Testers and feedback would be 
most welcome.  In a few days if all looks OK I'll make tarballs for those 
unwilling to play with raw CVS.

Rather than retype stuff, I'll paste below the pysh help docstring so you can 
get a feel for the functionality available.  Other than bugfixing, I don't 
plan on adding any significant features.

Don't even ask for job control, builting piping, etc.

To start pysh, simply type 'ipython -p pysh' (you may need to update your 
.ipython directory).

The ipythonrc-pysh profile is very compact, and the associated pysh.py file is 
also now very lightweight to speedup loading.  The bulk of the code has been 
put into InterpreterExec, an internal IPython extension (in IPython/Extensions).

Regards,

f.

========================================================================
     """Pysh is a set of modules and extensions to IPython which make shell-like
     usage with Python syntax more convenient.  Keep in mind that pysh is NOT a
     full-blown shell, so don't try to make it your /etc/passwd entry!

     In particular, it has no job control, so if you type Ctrl-Z (under Unix),
     you'll suspend pysh itself, not the process you just started.

     Since pysh is really nothing but a customized IPython, you should
     familiarize yourself with IPython's features.  This brief help mainly
     documents areas in which pysh differs from the normal IPython.

     ALIASES
     -------
     All of your $PATH has been loaded as IPython aliases, so you should be
     able to type any normal system command and have it executed.  See @alias?
     and @unalias? for details on the alias facilities.

     SPECIAL SYNTAX
     --------------
     Any lines which begin with '~', '/' and '.' will be executed as shell
     commands instead of as Python code. The special escapes below are also
     recognized.  !cmd is valid in single or multi-line input, all others are
     only valid in single-line input:

     !cmd      - pass 'cmd' directly to the shell
     !!cmd     - execute 'cmd' and return output as a list (split on '\\n')
     $var=cmd  - capture output of cmd into var, as a string
     $$var=cmd - capture output of cmd into var, as a list (split on '\\n')

     The $/$$ syntaxes make Python variables from system output, which you can
     later use for further scripting.  The converse is also possible: when
     executing an alias or calling to the system via !/!!, you can expand any
     python variable or expression by prepending it with $.  Full details of
     the allowed syntax can be found in Python's PEP 215.

     A few brief examples will illustrate these:

         fperez[~/test]|3> !ls *s.py
         scopes.py  strings.py

     ls is an internal alias, so there's no need to use !:
         fperez[~/test]|4> ls *s.py
         scopes.py*  strings.py

     !!ls will return the output into a Python variable:
         fperez[~/test]|5> !!ls *s.py
                       <5> ['scopes.py', 'strings.py']
         fperez[~/test]|6> print _5
         ['scopes.py', 'strings.py']

     $ and $$ allow direct capture to named variables:
         fperez[~/test]|7> $astr = ls *s.py
         fperez[~/test]|8> astr
                       <8> 'scopes.py\\nstrings.py'

         fperez[~/test]|9> $$alist = ls *s.py
         fperez[~/test]|10> alist
                       <10> ['scopes.py', 'strings.py']

     alist is now a normal python list you can loop over.  Using $ will expand
     back the python values when alias calls are made:
         fperez[~/test]|11> for f in alist:
                       |..>     print 'file',f,
                       |..>     wc -l $f
                       |..>
         file scopes.py     13 scopes.py
         file strings.py      4 strings.py

     IPython's input history handling is still active, which allows you to
     rerun a single block of multi-line input by simply using exec:
         fperez[~/test]|12> $$alist = ls *.eps
         fperez[~/test]|13> exec _i11
         file image2.eps    921 image2.eps
         file image.eps    921 image.eps

     While these are new special-case syntaxes, they are designed to allow very
     efficient use of the shell with minimal typing.  At an interactive shell
     prompt, conciseness of expression wins over readability.

     USEFUL FUNCTIONS AND MODULES
     ----------------------------
     The os, sys and shutil modules from the Python standard library are
     automatically loaded.  Some additional functions, useful for shell usage,
     are listed below.  You can request more help about them with '?'.

     shell   - execute a command in the underlying system shell
     system  - like shell(), but return the exit status of the command
     sout    - capture the output of a command as a string
     lout    - capture the output of a command as a list (split on '\\n')
     getoutputerror - capture (output,error) of a shell command

     sout/lout are the functional equivalents of $/$$.  They are provided to
     allow you to capture system output in the middle of true python code,
     function definitions, etc (where $ and $$ are invalid).

     DIRECTORY MANAGEMENT
     --------------------
     Since each command passed by pysh to the underlying system is executed in
     a shell which exits immediately, you can NOT use !cd to navigate the
     filesystem.

     Pysh provides its own builtin '@cd' magic command to move in the
     filesystem (the @ is not required with automagic on).  It also maintains a
     list of visited directories (use @dhist to see it) and allows direct
     switching to any of them.  Type 'cd?' for more details.

     @pushd, @popd and @dirs are provided for directory stack handling.



From pythondev-dang at lazytwinacres.net  Mon Jun 28 09:56:38 2004
From: pythondev-dang at lazytwinacres.net (Daniel 'Dang' Griffith)
Date: Mon, 28 Jun 2004 18:56:38 +0500
Subject: [IPython-dev] Re: [IPython-user] pysh is in CVS
Message-ID: <20040628135638.3657.qmail@server265.com>

    Wow!  I had a minor problems because there some executables
in my path with the same names as some Windows built-ins (e.g.,
dir.exe, mkdir.exe, rmdir.exe).  After running rehashx, IPython was
running them, rather than the built-in.  I was unaware of their 
presence until this time; I deleted them, and life is good again.
    Great update!  I'll keep you posted if I see any strangenesses.
It's really cool to be able to run 'jar', to access zip files, without 
needing two windows open (an IPython and a command prompt).
        --dang
p.s.
What do you mean about builtin piping?  I ask, because "ls /s | more" 
works just fine.  And if I re-aliasing ls to ls.exe, "ls -R | less" works 
just fine as well.

   -------Original Message-------
   > From: Fernando Perez <Fernando.Perez at colorado.edu>
   > Subject: [IPython-user] pysh is in CVS
   > Sent: 26 Jun 2004 13:29:01
   >
   >  Hi all,
   >  
   >  I've put all of pysh's functionality (including @rehashx for Windows, based on
   >  the $pathext environment variable) into CVS.??Testers and feedback would be
   >  most welcome.??In a few days if all looks OK I'll make tarballs for those
   >  unwilling to play with raw CVS.
   >  
   >  Rather than retype stuff, I'll paste below the pysh help docstring so you can
   >  get a feel for the functionality available.??Other than bugfixing, I don't
   >  plan on adding any significant features.
   >  
   >  Don't even ask for job control, builting piping, etc.
   >  
   >  To start pysh, simply type 'ipython -p pysh' (you may need to update your
   >  .ipython directory).
   >  
   >  The ipythonrc-pysh profile is very compact, and the associated pysh.py file is
   >  also now very lightweight to speedup loading.??The bulk of the code has been
   >  put into InterpreterExec, an internal IPython extension (in IPython/Extensions).
   >  
   >  Regards,
   >  
   >  f.



From Fernando.Perez at colorado.edu  Mon Jun 28 11:14:04 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 28 Jun 2004 09:14:04 -0600
Subject: [IPython-dev] Re: [IPython-user] pysh is in CVS
In-Reply-To: <20040628135638.3657.qmail@server265.com>
References: <20040628135638.3657.qmail@server265.com>
Message-ID: <40E035BC.2040406@colorado.edu>

Daniel 'Dang' Griffith wrote:
>     Wow!  I had a minor problems because there some executables
> in my path with the same names as some Windows built-ins (e.g.,
> dir.exe, mkdir.exe, rmdir.exe).  After running rehashx, IPython was
> running them, rather than the built-in.  I was unaware of their 
> presence until this time; I deleted them, and life is good again.

Yes, this is a bit of an annoyance.  It is in principle possible to have 
multiple internal namespaces for commands, I just didn't want to deal with 
such hassles for now.  A true shell (bash, tcsh, etc) keeps at least 3 
namespaces: $PATH commands, user aliases and shell builtins.  For ipython's 
shell functionality, I've lumped all these into one, since I already have to 
worry about managing the magic and python language namespaces as well.  But it 
might be a good idea to at least have two: one for commands and one for user 
aliases.

It's not that hard, just a bit time-consuming because the two dicts have to be 
managed carefully so it remains transparent to the user and doesn't slow 
things down noticeably.  Keep in mind that ipython has to remain responsive 
enough that you don't really notice any delay after pressing <Return>.  This 
means that whatever fancy magic it does, has to be finished in about 0.1s, 
which is roughly the human threshold for noticing 'slugishness'.  So I try 
very hard to keep the mainloop very tight so as to respect this constraint on 
reasonable hardware (my laptop is a 1.1GHz P-III, so I think it's a good 
middle-of-the-road benchmark these days).  If you actually look at the code, 
you'll see that I'm using a fairly stretchy definiton of 'tight' here, but 
that's a topic for another day.  Let's just say that I try my best :)

>     Great update!  I'll keep you posted if I see any strangenesses.
> It's really cool to be able to run 'jar', to access zip files, without 
> needing two windows open (an IPython and a command prompt).

Glad you like it!

> What do you mean about builtin piping?  I ask, because "ls /s | more" 
> works just fine.  And if I re-aliasing ls to ls.exe, "ls -R | less" works 
> just fine as well.

Yes, this works because the whole line is passed to the shell, which is the 
one handling the pipes.  What I meant was to write a bunch of _python_ 
utilities which simulate piping by overriding the '|' operator so you can 
simulate pipes in true python code.  There's an ASPN recipe on that if you 
feel like googling a bit.  I won't go that far, but as you correctly point 
out, 'regular' piping works just fine, courtesy of the underlying system shell.

Thanks for your report, and keep me posted on any issues you may encounter.

Cheers,

f



From Fernando.Perez at colorado.edu  Mon Jun 28 16:18:18 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 28 Jun 2004 14:18:18 -0600
Subject: [IPython-dev] Re: [IPython-user] pysh is in CVS
In-Reply-To: <40E05B9D.1050604@kolumbus.fi>
References: <40DD33CD.8000506@colorado.edu> <40E05B9D.1050604@kolumbus.fi>
Message-ID: <40E07D0A.5020808@colorado.edu>

Ville Vainio wrote:

>>Don't even ask for job control, builtin piping, etc.
> 
> 
> Don't dismiss job control so quickly ;-).
> 
> ipython might even be useful for doing much more "advanced" job control 
> that what you typically do with bash:
> 
>  >>> !& tail -f output.log
> 
> # launched job, "job object" is in jobs[0]
> 
>  >>> for line in jobs[0].output():
>              if "warning" in line: break
> 
>  >>> jobs[0].change_output(open("/dev/null"),"w")
> 
> # time passes
> 
>  >>> jobs[0].interact()  # output comes back to current terminal

Nifty idea indeed.  You may have to roll up your sleeves after all, heh? ;-)

Seriously, the current pysh functionality is about as far as I can afford to 
push it myself.  It works great for me, and I'm already enjoying easy looping 
with real python syntax for things with my filesystem.  Plus a shell which is 
itself a calculator is always handy.  I'll polish things a bit, fix problems 
as they appear, but that's about it.

My next item is a multithreaded ipython so I can use matplotlib interactively 
with a GTK frontend.  This will allow anyone to run GTK/Wx apps from within 
ipython without the GUI loop blocking ipython itself.  I had a stab at it 
Saturday evening but I failed, I'll try again over the next few weeks.  Since 
I've never written a line of multithreaded code, it will take me a bit of 
effort I'm sure.

After I finish the multithreaded ipython, so that a pylab profile can be 
shipped for a matlab-like environment (using matplotlib), I'll begin the slow 
cleanup of the internals.  This will set the stage for inclusion of ipython 
into IDLE, PyCrust, etc.  I already did a quick test and I can sort of run 
ipython inside IDLE, so it's definitely doable.  It will just take some time.

This is necessary for the kind of scientific computing environment (or coding 
in general, it's just that sci-comp is my field) that I envision and which we 
discussed a while back with Ero's Texmacs example.  Since that's the direction 
_I_ will be pushing things in, fancy shell functionality will have to be done 
by others.  I already broke my own promise of not writing pysh once, I won't 
do it twice :)

> Yup, pysh seemed to work alright. One thing that is still noticeably 
> slower than on bash is changing the directory with "cd" - on a slow 
> machine you have enough time to type "ls" and blink an eye before 
> ipython shows the prompt again. On a faster machine it's ok, obviously. 
> This is true for both "pysh" and plain unconfigured ipython.

I'm afraid there's not much I can do here.  All the niceties of ipython come 
at a price: the inner loop is a very complicated piece of code, with string 
analysis, regexp checks, multiple try/excepts, etc.  None of this is 
expendable, if you really want all the magic behavior which makes ipython so 
nice.  If you want a fast inner loop, implementing plain python behavior is 
trivial (see code.py in the stdlib).  As always, there's no free lunch.

It may be possible to optimize things a bit, but I doubt major improvements 
are possible.  Feel free to prove me wrong though :) Keep in mind that this 
code is often very subtle, and almost any change will break some corner case, 
so be careful before cutting out any seemingly 'dumb' codepath; chances are 
it's there for a very good reason.  The code is also hackish, dirty and 
uncommented.  Oh, and there are no unit tests for it.  Have fun :)

Cheers,

f



From Fernando.Perez at colorado.edu  Mon Jun 28 17:49:37 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 28 Jun 2004 15:49:37 -0600
Subject: [IPython-dev] Re: [IPython-user] pysh is in CVS
In-Reply-To: <40E08E37.1060801@kolumbus.fi>
References: <40DD33CD.8000506@colorado.edu> <40E05B9D.1050604@kolumbus.fi>
	<40E07D0A.5020808@colorado.edu> <40E08E37.1060801@kolumbus.fi>
Message-ID: <40E09271.6090104@colorado.edu>

Ville Vainio wrote:
> Fernando Perez wrote:
> 
> 
>>Seriously, the current pysh functionality is about as far as I can 
>>afford to push it myself.  It works great for me,
> 
> 
> Fair enough. I imagine other people might want to help too, esp. after 
> you do that codebase cleanup you've been talking about so that the 
> modifications don't need to be rewritten for the new architecture. And 
> you should really post a PR article on c.l.py about the newfound shell 
> functionality :).

I'll do so when 0.6.1 is out.  I skipped mentioning 0.6.0 b/c it didn't quite 
work for Windows users out-of-the-box.  And I'm not very good at PR :)

> Well, in some toolkits at least all the drawing commands need to be done 
> from the same thread (since X is single-threaded).  I imagine you 
> weren't trying to execute actual GUI code from the ipython thread?

I've actually included here interactive.py, the multithreaded mini-console 
which ships with matplotlib.  This is what I'll be using as my 'design 
reference', I already spoke to John Hunter (matplotlib's author) and he's OK 
with my blatant ripping code out of there.

Since I know zilch about multithreaded programming, I'd appreciate it if 
anyone sees any fundamental flaws in this code (it's very small) before I 
start really using it.  I am going to try to make a version of ipython (easy 
subclassing) whose inner loop is modeled after this code so that the gtk 
thread can draw without blocking user input/execution.

> As long as you see shell functionality as desirable (and will accept 
> patches to implement that), it's all well and good.

Patches will always be welcome, and I'd actually _like_ the kind of 
functionality you suggest.  It's just that my spare time is very finite.

> I could almost swear the delay was because of i/o. Executing normal 
> statements was instantaneous even on the slow machine, so I wouldn't be 
> so quick to blame the inner loop. I took a look at magic_cd, and the 
> call to parse_options catches my eye. Apparently parse_options does lots 
> of nasty stuff (like starts another python interpreter in a new shell). 
> That's a big hit on slow machines. IMO magic_cd should just do trivial 
> option parsing completely in python (current process), shell expansion 
> is not really needed .
> 
> I changed line 1865 of Magic.py (in cvs) to:
>             opts,ps = 
> self.parse_options(parameter_s,'q',mode='string',use_shell=False)
> 
> i.e. I added use_shell = False. It might break something, but boy, does 
> it speed it up on my slow machine.

Good catch!  In fact I started using the underlying shell by firing a 
wholesale python interpreter because I simply didn't want to deal with the 
subtleties of parsing a string into a proper sys.argv.  It's an ugly hack, and 
I'd love to get rid of it (plus, it's expensive as you noted).  If anyone can 
provide me with a pure python function which does:

argv_list = parse_string(argv_str)

which takes a string (with arbitrary quotes, parens, etc.) and returns a 
python list which would be the correct sys.argv seen by a command line 
consisting of this argv_str, I'd be very happy.

On this one I punted and decided to let the shell deal with it, but it's 
really an ugly solution.

What I'll do is at least leave this option off by default for most magics, and 
turn it on only for those that _really_ need to be able to reconstruct a 
sys.argv out of a string (basically @run).

This will go into CVS probably tonight.

Cheers,

f

ps.  You might want to subscribe to ipython-dev, I've moved this thread out of 
ipython-users since it's a bit technical for general users.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: interactive.py
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20040628/293f40b8/attachment.ksh>

From vivainio at kolumbus.fi  Mon Jun 28 19:51:33 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Tue, 29 Jun 2004 02:51:33 +0300
Subject: [IPython-dev] Re: [IPython-user] pysh is in CVS
In-Reply-To: <40E09271.6090104@colorado.edu>
References: <40DD33CD.8000506@colorado.edu> <40E05B9D.1050604@kolumbus.fi>
	<40E07D0A.5020808@colorado.edu> <40E08E37.1060801@kolumbus.fi>
	<40E09271.6090104@colorado.edu>
Message-ID: <40E0AF05.7000709@kolumbus.fi>


> get rid of it (plus, it's expensive as you noted).  If anyone can 
> provide me with a pure python function which does:
>
> argv_list = parse_string(argv_str)
>
> which takes a string (with arbitrary quotes, parens, etc.) and returns 
> a python list which would be the correct sys.argv seen by a command 
> line consisting of this argv_str, I'd be very happy.


Well, shlex.split(s) does arbitrary quotes, but not parens:


In [5]: shlex.split('hi "very much stuff" (a b c) hai')
Out[5]: ['hi', 'very much stuff', '(a', 'b', 'c)', 'hai']

Still, it's an ok solution for most cases, dont you think?



From Fernando.Perez at colorado.edu  Mon Jun 28 19:58:53 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 28 Jun 2004 17:58:53 -0600
Subject: [IPython-dev] Re: [IPython-user] pysh is in CVS
In-Reply-To: <40E0AF05.7000709@kolumbus.fi>
References: <40DD33CD.8000506@colorado.edu> <40E05B9D.1050604@kolumbus.fi>
	<40E07D0A.5020808@colorado.edu> <40E08E37.1060801@kolumbus.fi>
	<40E09271.6090104@colorado.edu> <40E0AF05.7000709@kolumbus.fi>
Message-ID: <40E0B0BD.5050702@colorado.edu>

Ville Vainio wrote:
>>get rid of it (plus, it's expensive as you noted).  If anyone can 
>>provide me with a pure python function which does:
>>
>>argv_list = parse_string(argv_str)
>>
>>which takes a string (with arbitrary quotes, parens, etc.) and returns 
>>a python list which would be the correct sys.argv seen by a command 
>>line consisting of this argv_str, I'd be very happy.
> 
> 
> 
> Well, shlex.split(s) does arbitrary quotes, but not parens:
> 
> 
> In [5]: shlex.split('hi "very much stuff" (a b c) hai')
> Out[5]: ['hi', 'very much stuff', '(a', 'b', 'c)', 'hai']
> 
> Still, it's an ok solution for most cases, dont you think?

Cool!  I didn't even know about shlex.  The stdlib is really quite the 
treasure trove (and I _did_ google around for a while when I faced this 
problem, but somehow I missed shlex).

I'll look into this tonight in detail, but it does sound quite reasonable. 
Esp. considering that parens don't typically belong naked on a shell line to 
begin with, and that shlex handles them gracefully if inside a string:

planck[~]|37> shlex.split('hi "very much (stuff)" (a b c) hai')
          <37> ['hi', 'very much (stuff)', '(a', 'b', 'c)', 'hai']

This means I'll most likely remove that ugly shell call to a full-blown python 
interpeter, which I hated since day 1 (but I really wanted to provide true 
sys.argv in @run).

Many thanks!

f



From vivainio at kolumbus.fi  Mon Jun 28 21:17:47 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Tue, 29 Jun 2004 04:17:47 +0300
Subject: [IPython-dev] Magics, aliases, namespaces
Message-ID: <40E0C33B.5050600@kolumbus.fi>

Current ipython seems to "special case" lots of stuff - aliases, magics 
etc. It would seem to yield itself to a simpler and more extensible 
architecture, consisting of multple dictionaries, where just the order 
of search is determined by profile/prefix. All the functionality would 
just be "pushed" to ipython core, i.e. the objects that represent the 
functionality are instantiated on "plugins" and they are registered with 
ipython core. i.e.

def magic_f(options, args):
  pass

m = MyMagic(magic_f)
m.arghandler = stockhandlers.OptParseArgHandler
m.options = .... # optparse options, for easy passing to magic_f and 
help display

# note that arghandler takes a peek at the instance, sees options, and 
proceeds
# accordingly. Various arg handlers can ask for arbitrary options.
# some handler might optionally glob the filenames, search data folders 
for filenames etc.

ipythonregistry.register(category = "magic", name = "mymagic", obj = m)

I bet most of the current functionality could easily be added to such a 
registry by just instantiating e.g. "Magic" class and registering all 
the functions with some sensible default args. Supporting legacy stuff 
in general would be easy - just implement new handlers (arg and 
otherwise) for new stuff, and have the old handlers around forever / as 
long as is deemed appropriate. The 'python' namespace (locals() + 
globals()) should be special, of course.

It should be easy to have arbitrary number of "categories" (like 
'magic', 'shellcommand','projectspecific_myproject', 
'projectspecific_otherproject'). It would only influence the order in 
which the completions are suggested, and in case of name collision which 
one is selected. Also, I think all completions should be shown, even the 
ones in "later" categories in the case of a match in an "earlier" category.

The "functionality object" might also have a callable object 
'expandarg', and ipython would run it (with the arg index) when tab 
completion is attempted after typing the function name. It would return 
the possible completions for that particular command... or None to 
"revert to default file completions". Such functionality could be useful 
in making ipython an "operating console" of a sort. I'm talking about:

 >> lscat reactor   # list commands in category - reactor is "project 
specific" category

r_operate


 >> r_operate <tab>
start   shutdown  notify_meltdown   evacuate

 >> r_operate shutdown <tab>

1 2 5 6     # note that 3 and 4 are already shut down

 >> r_operate shutdown 2

Shutting down..
ok.

 >> r_operate start <tab>

2 3 4      # 2 was shut down, can be started now

 >> r_operate start 2

Starting....
ok.

I'm talking about having a super-configurable man-machine language here! 
Like cmd.Cmd on steroids, as a free addition to ipython!

Sorry for the ramblings - it's 4:15 in Finland, and I had to get some 
codethink out of me. Recently started summer holiday has deprived me of 
the usual dose of code-excercise...

I guess it's time to hit the bed :-).



From Fernando.Perez at colorado.edu  Mon Jun 28 23:05:07 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 28 Jun 2004 21:05:07 -0600
Subject: [IPython-dev] Magics, aliases, namespaces
In-Reply-To: <40E0C33B.5050600@kolumbus.fi>
References: <40E0C33B.5050600@kolumbus.fi>
Message-ID: <40E0DC63.3030907@colorado.edu>

Ville Vainio wrote:
> Current ipython seems to "special case" lots of stuff - aliases, magics 
> etc. It would seem to yield itself to a simpler and more extensible 
> architecture, consisting of multple dictionaries, where just the order 
> of search is determined by profile/prefix. All the functionality would 
> just be "pushed" to ipython core, i.e. the objects that represent the 
> functionality are instantiated on "plugins" and they are registered with 
> ipython core. i.e.

[snip many good ideas]

All of this sounds very nice.  Now you know why I need to clean up the 
internals, don't you? :)

Keep in mind that ipython was born as an 'afternoon hack' in late 2001 because 
I was having fun learning python and was procrastinating finishing my thesis. 
  I glued together some code of mine with LazyPython and IPP.  IPP (by Janko 
Hauser) already had a bare-bones magics infrastructure and a basic logger, but 
both had been coded in as mixin classes (they still are).  Besides, I didn't 
know the code at all, so I started by nibbling at the sides, slowly going 
further into the core.

But the overall system is in bad need of a redesign, and much of what you said 
goes exactly in the direction I'd like to take it.  But before we get there, a 
continuosly working system must be kept while cleaning things up.  The 
new_design.pdf document included in the distro is used as a scratchpad for 
these kinds of ideas, I'll probably drop your email wholesale in there so it 
doesn't get lost.

But the order of the day will be more like:

- Make Magic and Logger not be mixins, but without further architectural 
cleanup.  Just pull them out of the main class and make sure everything 
continues to work.  It may be possible to use the new logging package from the 
stdlib, I'm not sure (our logging needs are really simple, so this is no big 
deal).

- Clean up the config system, so that ipythonrc files can be normal python 
code.  Maintain the ability to recursively nest profiles.

- Get rid of ipmaker and roll all that into a proper constructor.

Once this is done, then we can _start_ the kind of proper architectural 
redesign you have in mind, and which I'd love to implement.

If you have some time on your hands, I can obviously use all the help 
available.  IPython is only a nights/weekends project for me, and how much 
time I can devote to it varies a lot.  Hence the rather glacial pace of 
development (the 'famous cleanup' has been coming for a loooong time).

If you feel like you'd like to seriously pitch in, let me know and I can point 
you to specific starting points so we can proceed with some semblance of order.

Best,

f



From Fernando.Perez at colorado.edu  Tue Jun 29 02:22:11 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 29 Jun 2004 00:22:11 -0600
Subject: [IPython-dev] Re: [IPython-user] pysh is in CVS
In-Reply-To: <40E0B0BD.5050702@colorado.edu>
References: <40DD33CD.8000506@colorado.edu> <40E05B9D.1050604@kolumbus.fi>
	<40E07D0A.5020808@colorado.edu> <40E08E37.1060801@kolumbus.fi>
	<40E09271.6090104@colorado.edu> <40E0AF05.7000709@kolumbus.fi>
	<40E0B0BD.5050702@colorado.edu>
Message-ID: <40E10A93.9000001@colorado.edu>

Fernando Perez wrote:
> Ville Vainio wrote:
> 
>>>get rid of it (plus, it's expensive as you noted).  If anyone can 
>>>provide me with a pure python function which does:
>>>
>>>argv_list = parse_string(argv_str)
>>>
>>>which takes a string (with arbitrary quotes, parens, etc.) and returns 
>>>a python list which would be the correct sys.argv seen by a command 
>>>line consisting of this argv_str, I'd be very happy.
>>
>>
>>
>>Well, shlex.split(s) does arbitrary quotes, but not parens:

Argh.  I should have noticed that shlex.split is a python 2.3 feature.  It 
took some work to backport it (IPython maintains 2.2 compatibility), because 
the shlex module is a fair bit dumber under 2.2 than in 2.3.  I'm not sure I 
got it 100% right, there's a bit of regexp work to extract quotes correctly, 
and I always fear regexps.

Anyway, the fix is in (the nasty shell call was really bad enough that it 
warranted the effort), along with the gnuplot one and some other things.

In particular, the readline history files are now per-profile.  Instead of 
just having in ~/.ipython a 'history' file, you'll now have 'history', 
'history-pysh', 'history-numeric', etc.  I was getting really annoyed at 
pysh's history clobbering my numerical work history.

This is all in CVS, I'll let it simmer a bit longer before I put out RC 
tarballs.  Let me know how it behaves in real work please.

Cheers,

f

ps. There's a new magic, @system_verbose, which will make ipython print out 
all system calls it makes.  This may help debug things if you think that it's 
making incorrect expansions of some kind when going to the system. The 
function toggles the verbosity flag on/off on each call.



From vivainio at kolumbus.fi  Tue Jun 29 08:13:40 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Tue, 29 Jun 2004 15:13:40 +0300
Subject: [IPython-dev] Magics, aliases, namespaces
In-Reply-To: <40E0DC63.3030907@colorado.edu>
References: <40E0C33B.5050600@kolumbus.fi> <40E0DC63.3030907@colorado.edu>
Message-ID: <40E15CF4.2060200@kolumbus.fi>


> But the order of the day will be more like:
>
> - Make Magic and Logger not be mixins, but without further 
> architectural cleanup.  Just pull them out of the main class and make 
> sure everything continues to work.  It may be

Shouldn't this be rather simple, or am I misunderstanding something? 
Just implement only one function in Magic, e.g. magic_execute. Then the 
"magic handler" would return the command to be executed as

"magic_execute(real_magic_name, rest_of_the_line)".

magic_execute would then do whataver it wants to do, probably just 
lookup real_magic_name in some dictionary and pass the rest_of_the_line 
to it as args.

Obviously the first implementation of all the magics could be exactly 
the current ones via some portability bridge class...

> If you feel like you'd like to seriously pitch in, let me know and I 
> can point you to specific starting points so we can proceed with some 
> semblance of order.

Perhaps I might be able to put in some hours every now and again. I'm 
not sure how "serious" commitment I can make, but you could nevertheless 
put up some "needs to be done, and how to go about it" tasks to the 
ipython web page / wiki / whatever. I'll see if there's something I 
would like to do.

BTW, one thing that you could even do for the next release: magics for

@bookmark  <bookmarkname>   (associate <bookmarkname> with current dir)

@cdb <bookmarkname>     (cd bookmark, change to dir identified by 
<bookmarkname>

These bookmarks would of course be persistent, stored in the user's 
ipython dir on exit and read in on startup. In fact a lightweight 
persistence system might be implemented, where a dictionary (accessible 
by global persiststore() function) is pickled out on exit, and pickled 
in on startup. Bookmarks could be stored there, and other data that the 
users may want to store. Pickling is pretty safe as long as you use 
standard python types, so name changes won't break the compatibility.

BTW, I'm going to post a little bit of PR/advocacy brainstorming on 
c.l.py soon, probably today. I'm having some wild ideas again ;-).



From vivainio at kolumbus.fi  Tue Jun 29 10:09:59 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Tue, 29 Jun 2004 17:09:59 +0300
Subject: [IPython-dev] The hubrid rant I just posted c.l.py
Message-ID: <40E17837.4040204@kolumbus.fi>

I said I would do some advocacy, so here goes:

Pythonic Nirvana - towards a true Object Oriented Environment
=============================================================

IPython (by Francois Pinard) recently (next release - changes are
still in CVS) got the basic abilities of a system shell (like
bash). Actually, using it as a shell is rather comfortable. This all
made me think...

Why do we write simple scripts to do simple things? Why do we
serialize data to flat text files in order to process them? Everything
could be so much simpler and immensely more powerful if we operated on
data *directly* - we should operate on objects, lists, dictionaries,
functions, not files. We shouldn't write scripts to perform our
routines - and way too often, we don't even try, because moving the
data between scripts is too troublesome, involving file formats, argv
parsing and other tedious stuff.

If we use a shell with full Python capabilities, we can introduce new
funtionality for integrating our environments very easily. Consider a
project framework for managing a software project::

   >> import pf
   >> pf.projects
  
   --> [<project 'foo'>, <project 'bar'>]

   >> p = pf.projects[0]   # we want to work with project 'foo'
   >> headers = Set([f for f in p.files() if f.endswith(".h")])
   >> srcs = p.files - headers
   >> found = Set([f for f in srcs if find_any(f, headers)])

   >> r = findradio("kazoo classics")[0] # some diversion needed
   >> music.play   # manipulates our music player
   >> music.vol 100

   # back to work...
   >> notfound = srcs - found

   # who has made a header that is not used?
   >> jackasses = Set([p.author(file) for file in notfound])


Now we have the names of our victims in 'jackasses' variable. We want
to make that variable accessible to all in our project team, in a
persistent store. We use tab completion to find the databases::

   >> export jackasses <TAB>

   Completions: "fooproject_db" "barproject_db" "private"

Note how tab completions for "export" notices that we are trying to
enter the second parameter. It knows that it is the database name, and
the completion mechanism (written for export) dynamically queries the
list of databases for which we are allowed to export data. After
seeing the choices we choose one of them::

   >> export jackasses "fooproject_db" name="slackers"

Now the list of guys is accessible to everybody. We also compose a
volatile email to everybody::

   >> xemacs tmp  

   >> for m in [Mail(to=name, bodyfile="tmp") for name in jackasses]: m.send

And rat the guys to the management::

  >> xemacs tmp2

  # mail contents
  #  The following guys have not been doing their jobs:
  #  @("\n".join(jackasses))

  >> cont = open(tmp2).read().expand()   # runs it through EmPy template 
expansion system.

  >> Mail(to=p.boss, body=cont).send

Notice how jackasses variable was used inside the mail. We can also
schedule some extra hours for the guys to check if their headers are
needed, create a cron script to monitor that they have fixed the bugs,
etc.

The boss might want to fire them at once:

  >> l = import "slackers"
  >> [e.fire() for e in Employee(l)]

Or the boss might want to do some more extensive firing to invigorate
the company::

  >> ent = MyEnterprise()
 
  Auth needed!
  Password: ******

  >> st = stats(ent.allemployees())
  >> avgperf = st.average_performance


  >> def dead_weight(emp):
  ..   if emp.performance() < avgperf: return True
  ..   return False

  >> ent.fire_if(dead_weight)

Typing all that might seem like a lot of work. However, a lot of it
will probably be implemented as functions aggregating the
functionality. Most of the lines here could be factored out to
a function (note that I didn't say script)::
 
   def unused_headers(prj):
     """ returns the header files that are not used in the project """
     ... implementation ...


With conventional scripting techniques, nobody would want to do all
this. With the pythonic approach, creating this kind of business
intelligence is a breeze, eliminating tons of tedious routine!

Obviously this all can be done in specific scripts, which start doing
the thing "from the start" - open database connections, ask the user
to select the project, etc. However, writing such scripts is a lot of
work. With the interactive, much more dynamic approach, pieces of
functionality can be implemented one by one, and they become usable
immediately.

I can imagine that for power users and "knowledge workers", this type
of system would yield immense power. The payback is also incremental -
the whole system grows and gets refactored, streamlining the
process. In the end most of it will probably be driven by a simple
GUI. Especially the "fire below average employees" function, which
should not be run in a yearly cron job - only when needed. Various GUI
threads could be running in the same Python process, manipulating the
same namespace

What needs to be done
---------------------

Not surprisingly, "we're not there yet".

- IPython needs some refactoring (the codebase isn't quite scalable
  and extensible enough yet). Francois can use some help.

- Flexible persistence system needs to be itengrated. ZODB, perhaps?

- Domain specific modules (like project / "employee management"
  systems) need to be implemented. This luckily mostly writes itself.

- Flexible, but easy to use protocols need to be specified for
  documenting the functions, argument expansion, gui interaction etc. A
  gui module should display the documentation for the "current" function
  and possible arguments, so there's no need to press tab at all times.

Still, all in all, we're almost there. This has the same "feel" of
tight integration that I imagine the Lisp Macine guys were
experiencing, but Python is much more scripting-friendly and easier to
learn.

Ergo, world domination.




From Fernando.Perez at colorado.edu  Tue Jun 29 10:29:11 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 29 Jun 2004 08:29:11 -0600
Subject: [IPython-dev] Magics, aliases, namespaces
In-Reply-To: <40E15CF4.2060200@kolumbus.fi>
References: <40E0C33B.5050600@kolumbus.fi> <40E0DC63.3030907@colorado.edu>
	<40E15CF4.2060200@kolumbus.fi>
Message-ID: <40E17CB7.5050009@colorado.edu>

Ville Vainio wrote:
>>But the order of the day will be more like:
>>
>>- Make Magic and Logger not be mixins, but without further 
>>architectural cleanup.  Just pull them out of the main class and make 
>>sure everything continues to work.  It may be
> 
> 
> Shouldn't this be rather simple, or am I misunderstanding something? 
> Just implement only one function in Magic, e.g. magic_execute. Then the 
> "magic handler" would return the command to be executed as

The problem is that in many places, the Magic functions access shell 
attributes directly (since it's a mixin).  A boundary needs to be drawn 
between the two.  Same for logger.  This is easy, it just needs to be done 
carefully to make sure all codepaths are taken into account.

> Perhaps I might be able to put in some hours every now and again. I'm 
> not sure how "serious" commitment I can make, but you could nevertheless 
> put up some "needs to be done, and how to go about it" tasks to the 
> ipython web page / wiki / whatever. I'll see if there's something I 
> would like to do.
> 
> BTW, one thing that you could even do for the next release: magics for
> 
> @bookmark  <bookmarkname>   (associate <bookmarkname> with current dir)
> 
> @cdb <bookmarkname>     (cd bookmark, change to dir identified by 
> <bookmarkname>

There's your first project.  Send me a patch :)

> BTW, I'm going to post a little bit of PR/advocacy brainstorming on 
> c.l.py soon, probably today. I'm having some wild ideas again ;-).

I saw that, thanks.  But I (Fernando Perez) wrote ipython, not Francois Pinard 
(though he has sent me useful ideas and comments in the past). :)

Cheers,

f



From vivainio at kolumbus.fi  Tue Jun 29 10:38:23 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Tue, 29 Jun 2004 17:38:23 +0300
Subject: [IPython-dev] Magics, aliases, namespaces
In-Reply-To: <40E17CB7.5050009@colorado.edu>
References: <40E0C33B.5050600@kolumbus.fi> <40E0DC63.3030907@colorado.edu>
	<40E15CF4.2060200@kolumbus.fi> <40E17CB7.5050009@colorado.edu>
Message-ID: <40E17EDF.1040804@kolumbus.fi>

Fernando Perez wrote:

>> Shouldn't this be rather simple, or am I misunderstanding something? 
>> Just implement only one function in Magic, e.g. magic_execute. Then 
>> the "magic handler" would return the command to be executed as
>
>
> The problem is that in many places, the Magic functions access shell 
> attributes directly (since it's a mixin).  A boundary needs to be 
> drawn between the two.  Same for logger.  This is easy, it just needs 
> to be done carefully to make sure all codepaths are taken into account.

Ah. Perhaps the magic_execute could just pass the reference to shell 
environment then...

> There's your first project.  Send me a patch :)

Ok :).

> I saw that, thanks.  But I (Fernando Perez) wrote ipython, not 
> Francois Pinard (though he has sent me useful ideas and comments in 
> the past). :)

Ouch. Two south-european names with same initials - you get the idea ;-)



From Fernando.Perez at colorado.edu  Tue Jun 29 10:57:23 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 29 Jun 2004 08:57:23 -0600
Subject: [IPython-dev] Magics, aliases, namespaces
In-Reply-To: <40E17EDF.1040804@kolumbus.fi>
References: <40E0C33B.5050600@kolumbus.fi> <40E0DC63.3030907@colorado.edu>
	<40E15CF4.2060200@kolumbus.fi> <40E17CB7.5050009@colorado.edu>
	<40E17EDF.1040804@kolumbus.fi>
Message-ID: <40E18353.9020106@colorado.edu>

Ville Vainio wrote:

> Ah. Perhaps the magic_execute could just pass the reference to shell 
> environment then...

It's already setup like that.  In all magics, self.shell is a reference to the 
IPython shell, and that's what you should use.  But some of the original code 
doesn't use it, which is why I said that an easy but slow pass must be done 
over that code to clean those up.  I've been trying to at least not make the 
code worse over time, so all new code for a while has been written in this manner.

>>I saw that, thanks.  But I (Fernando Perez) wrote ipython, not 
>>Francois Pinard (though he has sent me useful ideas and comments in 
>>the past). :)
> 
> 
> Ouch. Two south-european names with same initials - you get the idea ;-)

I see: your hashing function has really bad collision properties :)

Cheers,

f



From Fernando.Perez at colorado.edu  Tue Jun 29 13:45:09 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 29 Jun 2004 11:45:09 -0600
Subject: [IPython-dev] Magics, aliases, namespaces
In-Reply-To: <40E17EDF.1040804@kolumbus.fi>
References: <40E0C33B.5050600@kolumbus.fi> <40E0DC63.3030907@colorado.edu>
	<40E15CF4.2060200@kolumbus.fi> <40E17CB7.5050009@colorado.edu>
	<40E17EDF.1040804@kolumbus.fi>
Message-ID: <40E1AAA5.9060504@colorado.edu>

Ville Vainio wrote:

>>I saw that, thanks.  But I (Fernando Perez) wrote ipython, not 
>>Francois Pinard (though he has sent me useful ideas and comments in 
>>the past). :)
> 
> 
> Ouch. Two south-european names with same initials - you get the idea ;-)

By the way, Francois is Canadian and I'm Colombian.  But yes, I guess we're 
both a by-product of the european colonization of America :)

Best,

f



From vivainio at kolumbus.fi  Tue Jun 29 15:16:26 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Tue, 29 Jun 2004 22:16:26 +0300
Subject: [IPython-dev] 
	Re: ipython: patch to implement @bookmark, @cdb magics
In-Reply-To: <40E1A7C3.8070803@colorado.edu>
References: <40E19655.1020301@kolumbus.fi> <40E1A7C3.8070803@colorado.edu>
Message-ID: <40E1C00A.5050806@kolumbus.fi>

Fernando Perez wrote:

> I'd actually prefer to implement the 'cd to bookmark' via a -b option 
> to @cd instead of as a wholesale new magic.  It keeps the 
> functionality nicely together, and it makes it more likely that people 
> will actually use it.

Hmm, I'd kinda like to have it as cdb. It's much faster to type, and I 
implemented it in anticipation of frequent directory switching back and 
forth, like cdb foos [foo source], do some stuff there, cdb fooh [foo 
header], do other stuff there.

Well, the new patch I've attached implements it that way anyway ;-). I 
also removed the persistent file as config option; I just couldn't get 
the config system working :-/. I guess it could be implemented as config 
option once the config system is fixed, i.e. python-based configuration 
files are implemented.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: second_patch_for_bookmarks.diff
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20040629/cfb9ae6d/attachment.ksh>

From Fernando.Perez at colorado.edu  Tue Jun 29 17:14:19 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 29 Jun 2004 15:14:19 -0600
Subject: [IPython-dev] Re: ipython: patch to implement @bookmark, @cdb
	magics
In-Reply-To: <40E1C00A.5050806@kolumbus.fi>
References: <40E19655.1020301@kolumbus.fi> <40E1A7C3.8070803@colorado.edu>
	<40E1C00A.5050806@kolumbus.fi>
Message-ID: <40E1DBAB.3090902@colorado.edu>

Ville Vainio wrote:
> Fernando Perez wrote:
> 
> 
>>I'd actually prefer to implement the 'cd to bookmark' via a -b option 
>>to @cd instead of as a wholesale new magic.  It keeps the 
>>functionality nicely together, and it makes it more likely that people 
>>will actually use it.
> 
> 
> Hmm, I'd kinda like to have it as cdb. It's much faster to type, and I 
> implemented it in anticipation of frequent directory switching back and 
> forth, like cdb foos [foo source], do some stuff there, cdb fooh [foo 
> header], do other stuff there.

-1

I really try to keep the number of builtin magics under control.  Especially 
since it's so trivial to write a one-line magic in your own config file that 
delegates (pseudo code below):

def magic_cdb(foo):
   self.magic_cd('-b %s' % foo)

I actually have a bunch of such 'private' magics to suit my own personal 
taste, but I don't include them in the distribution.  It's better to 
distribute a few flexible functions, and let the hacker/twiddler-minded ones 
(like myself) do the fine-tuning on their own.  The builtin magic list is 
large enough as it is.

But overall I like the patch quite a bit, along with the idea for a persistent 
storage pickle.  This can be extended later to many things.  I'll have to 
think about whether to have that name be tied to a profile or not, you might 
want your bookmarks to be general, or not.  I'm not really sure.

> Well, the new patch I've attached implements it that way anyway ;-). I 
> also removed the persistent file as config option; I just couldn't get 
> the config system working :-/. I guess it could be implemented as config 
> option once the config system is fixed, i.e. python-based configuration 
> files are implemented.

The config option has to be initialized in ipmaker.py.  That stuff is a mess, 
which is why I want to rewrite it.  The reason it's so ugly, is because it 
used to be my $PYTHOSTARTUP file, which eventually became an object initalizer 
routine.  But all of that needs to go into the underlying object itself.

There's no rush, I doubt I'll get to work on these patches before the weekend. 
    So feel free to play with them further.  But I think it's a really nice 
idea, so keep going.

Best,

f



From vivainio at kolumbus.fi  Tue Jun 29 18:34:44 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Wed, 30 Jun 2004 01:34:44 +0300
Subject: [IPython-dev] tab completion w/ spaces in filename?
In-Reply-To: <40E1DBAB.3090902@colorado.edu>
References: <40E19655.1020301@kolumbus.fi> <40E1A7C3.8070803@colorado.edu>
	<40E1C00A.5050806@kolumbus.fi> <40E1DBAB.3090902@colorado.edu>
Message-ID: <40E1EE84.2050209@kolumbus.fi>

Fernando Perez wrote:

> -1
>
> I really try to keep the number of builtin magics under control.  
> Especially since it's so trivial to write a one-line magic in your own 
> config file that delegates (pseudo code below):

Ack. I'll move the code from cdb to cd.

BTW, is tab completion supposed to work for files with spaces in their 
name? Completing a spaced file leaves the spaces as such in the file 
name (i.e. they are not escaped, quoted or anything), so that pressing 
tab a second time gives the list of all files, i.e. it doesn't consider 
what is already typed/completed. Adding backslashes manually to escape 
the spaces doesn't help.



From Fernando.Perez at colorado.edu  Tue Jun 29 18:41:28 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 29 Jun 2004 16:41:28 -0600
Subject: [IPython-dev] tab completion w/ spaces in filename?
In-Reply-To: <40E1EE84.2050209@kolumbus.fi>
References: <40E19655.1020301@kolumbus.fi> <40E1A7C3.8070803@colorado.edu>
	<40E1C00A.5050806@kolumbus.fi> <40E1DBAB.3090902@colorado.edu>
	<40E1EE84.2050209@kolumbus.fi>
Message-ID: <40E1F018.1010200@colorado.edu>

Ville Vainio wrote:
> Fernando Perez wrote:
> 
> 
>>-1
>>
>>I really try to keep the number of builtin magics under control.  
>>Especially since it's so trivial to write a one-line magic in your own 
>>config file that delegates (pseudo code below):
> 
> 
> Ack. I'll move the code from cdb to cd.

Thanks.  I didn't mean to discourage you, I'm just trying to keep the base 
design reasonably clean.  Tweakers like you and me can always go under the hood :)

> BTW, is tab completion supposed to work for files with spaces in their 
> name? Completing a spaced file leaves the spaces as such in the file 
> name (i.e. they are not escaped, quoted or anything), so that pressing 
> tab a second time gives the list of all files, i.e. it doesn't consider 
> what is already typed/completed. Adding backslashes manually to escape 
> the spaces doesn't help.

It's broken.  I'd never noticed, since I just don't use spaces in my filenames 
ever.  It should be easy to fix by putting in the escapes in the file 
completer method.  Look towards the beginning of iplib.py.

Best,

f



From Fernando.Perez at colorado.edu  Tue Jun 29 23:48:58 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 29 Jun 2004 21:48:58 -0600
Subject: [IPython-dev] Re: tab completion w/ spaces in filename?
In-Reply-To: <40E1EE84.2050209@kolumbus.fi>
References: <40E19655.1020301@kolumbus.fi> <40E1A7C3.8070803@colorado.edu>
	<40E1C00A.5050806@kolumbus.fi> <40E1DBAB.3090902@colorado.edu>
	<40E1EE84.2050209@kolumbus.fi>
Message-ID: <40E2382A.70204@colorado.edu>

Ville Vainio wrote:
> BTW, is tab completion supposed to work for files with spaces in their 
> name? Completing a spaced file leaves the spaces as such in the file 
> name (i.e. they are not escaped, quoted or anything), so that pressing 
> tab a second time gives the list of all files, i.e. it doesn't consider 
> what is already typed/completed. Adding backslashes manually to escape 
> the spaces doesn't help.

Fixed, I'll put it in CVS in a minute.

Thanks for the report.

Best,

f



From vivainio at kolumbus.fi  Wed Jun 30 07:56:24 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Wed, 30 Jun 2004 14:56:24 +0300
Subject: [IPython-dev] Python-based configuration system
Message-ID: <40E2AA68.4070009@kolumbus.fi>

Apparently replacing the current configuration system with a 
python-based one should not be too much work:

class Configuration:

   def parsefile(file):
     " execute the file in self.confdict namespace "

   def getdict():
     " return the underlying dict "

   def pushopts(optdict):
     self.confdict.update(optdict)

   class RcProxy
     " a Struct-like proxy for dict "

   def getrcstruct():
     " return RcProxy object "

The RcProxy instance given by getrcstruct would just implement 
getattr/setattr to return the corresponding item in confdict, and 
delegate non-existing key lookups to the confdict as such.

Implementation of Configuration methods is trivial, and RcProxy is 
mostly copy-paste from here (delegation recipe):

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52295
 
Then we just put the RcProxy instance to the rc attrbute of the 
interpreter, and nothing would need to be changed in interpreter class 
and other classes that might use interpreter.rc.

Configuration could/should really be that simple, I've implemented such 
a system in many seperate projects and there has never been any real 
problems (most importantly, there has never been a need for more 
sophisticated conflict resolution than "last evaluation wins". The 
"seed" configuration file containing all the defaults (that is executed 
first) could be in /etc/ipython-defaults (or whatever) and other 
user/system config files would just be executed in the same 
Configuration object after it.



From Fernando.Perez at colorado.edu  Wed Jun 30 11:26:11 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 30 Jun 2004 09:26:11 -0600
Subject: [IPython-dev] Python-based configuration system
In-Reply-To: <40E2AA68.4070009@kolumbus.fi>
References: <40E2AA68.4070009@kolumbus.fi>
Message-ID: <40E2DB93.9030304@colorado.edu>

Ville Vainio wrote:
> Apparently replacing the current configuration system with a 
> python-based one should not be too much work:

[snip]

The work isn't hard, it's just a matter of going through the hairball of the 
current code and ripping out all the deadwood.  The issue is just one of time, 
not of difficulty.  Nothing in ipython is actually difficult code by any 
stretch of the imagination, it's just a bunch of useful simple things strung 
together.  Something along the lines of what you suggest is pretty much what I 
had in mind.

If you want to actually get going with this kind of work, I'm happy to have a 
helping hand.  We can even set up a dev CVS branch for these changes, so that 
we can play safely with the code without breaking the mainline.

But I need to do a few things in order:

1. Get 0.6.1 out officially.  We can include your @bookmarks patches if you 
want.  This could be done relatively soon, since most issues seem more or less 
resolved.  The only ones I can see left are:

  a. http://www.scipy.net/roundup/ipython/issue6
  b. Cygwin readline problems.  I don't know what to do here, it will probably
     stay as it is.
  c. The windows GUI installer.  Viktor, do you have a feel for whether you
     think you can make it work for 0.6.1?

2. Write a multithreaded version of ipython like the interactive.py from 
matplotlib I recently posted.  Sorry about this, but I won't move forward 
until I can do this, since: a) I promised it already to others b) I also need 
it for my own work.

If you think you can/want to pitch in with 2, that would be _great_.  If not, 
and you don't want my working on that to hold you back, we can set up a CVS 
dev branch after 1. is done so you can start playing with it while I write the 
multithreaded code.  The danger of doing this is that we may break sync:  I 
may need to refactor ipython's read/eval loop a bit for threads.  So I'd 
rather do this in order with you.

3. By the time we put out 0.6.2 with the multithreaded ipython included, we'll 
flush out any bug reports which may have come from 0.6.1, esp. about pysh. 
This can then be considered 'end of the road' for the current codebase.

At this point we can start really tearing the code, chainsaw in hand.

How does that sound?

Cheers,

f



From vivainio at kolumbus.fi  Wed Jun 30 13:47:31 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Wed, 30 Jun 2004 20:47:31 +0300
Subject: [IPython-dev] Python-based configuration system
In-Reply-To: <40E2DB93.9030304@colorado.edu>
References: <40E2AA68.4070009@kolumbus.fi> <40E2DB93.9030304@colorado.edu>
Message-ID: <40E2FCB3.40803@kolumbus.fi>

Fernando Perez wrote:

> The work isn't hard, it's just a matter of going through the hairball 
> of the current code and ripping out all the deadwood.  The issue is 
> just one of time, not of difficulty.  Nothing in ipython is actually

I thought config rehaul would have been quite easy on the 
deadwood-ripping department, most of the work being localized on 
ipmaker.py. Though I admit it's rather tedious, because one would have 
to construct the new config files too. And there's that DPyGetOpt stuff 
as well, which, as you have said, should be replaced by optparse (or 
Optik if/when you insist on pre-2.3 backwards compatibility).

I would perhaps choose a more laissez-faire approach to configuring with 
command line parametes, though - just check that the var exist in the 
config dict, if it doesn't, complain. This would mean that "short" 
options as aliases for full names weren't supported, but it would ease 
maintenance a bit.

> 2. Write a multithreaded version of ipython like the interactive.py 
> from matplotlib I recently posted.  Sorry about this, but I won't move 
> forward until I can do this, since: a) I promised it already to others 
> b) I also need it for my own work.
>
> If you think you can/want to pitch in with 2, that would be _great_.  
> If not, and you don't want my

Well, 2. seems rather dependency-involved, which means more tedious 
testing, so I don't know whether I have the time/energy to directly help 
codewise (I'm supposed to have a summer holiday & all ;-). I agree about 
the need to do it first after 2.6.1.

The code you posted previously seemed to be ok, and not-hard to 
integrate with IPython. The way I would go about it (which would be 
usable in future too) would be to create a new class:

class ExecutionContext:
  run (cmd)  
  # blocks until cmd is done, returns the return value. Normal impl is 
trivial, in multithreaded case you
  # "feed" the cmd to that thread of yours and wait till it's done

  trycomplete(text)
  # asking the context for completions makes IMO more sense than asking 
for direct dicts


This would also be a better "one place to look" for the actual 
evaluation than separate funcs in the same class...

> 3. By the time we put out 0.6.2 with the multithreaded ipython 
> included, we'll flush out any bug reports which may have come from 
> 0.6.1, esp. about pysh. This can then be considered 'end of the road' 
> for the current codebase.
>
> At this point we can start really tearing the code, chainsaw in hand.
>
> How does that sound?

Reasonable :-). I guess I could help with some of the chainsawing, 
though I'm afraid my freetime will evaporate before that.

But now, off to finalizing that @bookmark system...



From vivainio at kolumbus.fi  Wed Jun 30 14:53:41 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Wed, 30 Jun 2004 21:53:41 +0300
Subject: [IPython-dev] tab completion w/ spaces in filename - still broken
In-Reply-To: <40E2382A.70204@colorado.edu>
References: <40E19655.1020301@kolumbus.fi> <40E1A7C3.8070803@colorado.edu>
	<40E1C00A.5050806@kolumbus.fi> <40E1DBAB.3090902@colorado.edu>
	<40E1EE84.2050209@kolumbus.fi> <40E2382A.70204@colorado.edu>
Message-ID: <40E30C35.10104@kolumbus.fi>

Fernando Perez wrote:

> Ville Vainio wrote:
>
>> BTW, is tab completion supposed to work for files with spaces in 
>> their name? Completing a spaced file leaves the spaces as such in the 
>> file name (i.e. they are not escaped, quoted or anything), so that 
>> pressing tab a second time gives the list of all files, i.e. it 
>> doesn't consider what is already typed/completed. Adding backslashes 
>> manually to escape the spaces doesn't help.
>
>
> Fixed, I'll put it in CVS in a minute.

The current version still doesn't work, albeit it malfunctions differently:


root at axp[~]|2> touch "a b c"
root at axp[~]|3> touch "a c d"
root at axp[~]|4> a
a\ b\ c  a\ c\ d
root at axp[~]|4> a\ b
b2m                          bonobo-text-plain
b2m.xemacs21                 bootlogd
badblocks                    browsercaps-updater
base-config                  brushtopbm
basename                     bsd-write
... (long list snipped) ...

Apparently space is a delimiter for readline even when escaped with \. 
Fix for this possibly won't be as easy...



From Fernando.Perez at colorado.edu  Wed Jun 30 16:23:30 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 30 Jun 2004 14:23:30 -0600
Subject: [IPython-dev] tab completion w/ spaces in filename - still broken
In-Reply-To: <40E30C35.10104@kolumbus.fi>
References: <40E19655.1020301@kolumbus.fi> <40E1A7C3.8070803@colorado.edu>
	<40E1C00A.5050806@kolumbus.fi> <40E1DBAB.3090902@colorado.edu>
	<40E1EE84.2050209@kolumbus.fi> <40E2382A.70204@colorado.edu>
	<40E30C35.10104@kolumbus.fi>
Message-ID: <40E32142.1000805@colorado.edu>

Ville Vainio wrote:
> The current version still doesn't work, albeit it malfunctions differently:
> 
> 
> root at axp[~]|2> touch "a b c"
> root at axp[~]|3> touch "a c d"
> root at axp[~]|4> a
> a\ b\ c  a\ c\ d
> root at axp[~]|4> a\ b
> b2m                          bonobo-text-plain
> b2m.xemacs21                 bootlogd
> badblocks                    browsercaps-updater
> base-config                  brushtopbm
> basename                     bsd-write
> ... (long list snipped) ...
> 
> Apparently space is a delimiter for readline even when escaped with \. 
> Fix for this possibly won't be as easy...

You're right.  I tried quickly playing with readline.set_completer_delims(), 
but I could not convince readline to keep the whole filename intact for 
completion.  So we have a partial fix, but a full solution will require a real 
readline guru (it may not be possible with python's readline, I don't know: 
I'm not sure if python's readline exposes all of GNU readline).

Best,

f



From vivainio at kolumbus.fi  Wed Jun 30 16:57:05 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Wed, 30 Jun 2004 23:57:05 +0300
Subject: [IPython-dev] tab completion w/ spaces in filename - still broken
In-Reply-To: <40E32142.1000805@colorado.edu>
References: <40E19655.1020301@kolumbus.fi> <40E1A7C3.8070803@colorado.edu>
	<40E1C00A.5050806@kolumbus.fi> <40E1DBAB.3090902@colorado.edu>
	<40E1EE84.2050209@kolumbus.fi> <40E2382A.70204@colorado.edu>
	<40E30C35.10104@kolumbus.fi> <40E32142.1000805@colorado.edu>
Message-ID: <40E32921.3080000@kolumbus.fi>

Fernando Perez wrote:

> readline to keep the whole filename intact for completion.  So we have 
> a partial fix, but a full solution will require a real readline guru 
> (it may not be possible with python's readline, I don't know: I'm not 
> sure if python's readline exposes all of GNU readline).

readline.get_line_buffer(), which gets the whole line, might be usable 
for this, together with readline.get_endidx(). I imagine that the 
completion function could move left enough to find the first "real", 
i.e. unescaped delimiter to find the real start position (and hence the 
real completable text).

I've never really used readline, but reading the module docs might 
suggest something like that...



From Fernando.Perez at colorado.edu  Wed Jun 30 17:03:46 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 30 Jun 2004 15:03:46 -0600
Subject: [IPython-dev] tab completion w/ spaces in filename - still broken
In-Reply-To: <40E32921.3080000@kolumbus.fi>
References: <40E19655.1020301@kolumbus.fi> <40E1A7C3.8070803@colorado.edu>
	<40E1C00A.5050806@kolumbus.fi> <40E1DBAB.3090902@colorado.edu>
	<40E1EE84.2050209@kolumbus.fi> <40E2382A.70204@colorado.edu>
	<40E30C35.10104@kolumbus.fi> <40E32142.1000805@colorado.edu>
	<40E32921.3080000@kolumbus.fi>
Message-ID: <40E32AB2.60806@colorado.edu>

Ville Vainio wrote:
> Fernando Perez wrote:
> 
> 
>>readline to keep the whole filename intact for completion.  So we have 
>>a partial fix, but a full solution will require a real readline guru 
>>(it may not be possible with python's readline, I don't know: I'm not 
>>sure if python's readline exposes all of GNU readline).
> 
> 
> readline.get_line_buffer(), which gets the whole line, might be usable 
> for this, together with readline.get_endidx(). I imagine that the 
> completion function could move left enough to find the first "real", 
> i.e. unescaped delimiter to find the real start position (and hence the 
> real completable text).

Except that readline.get_line_buffer() doesn't work.  It blocks in a truly 
bizarre manner (at least on my box).  I tried putting calls to it inside the 
file_complete() method, and they go into a black hole.  I can't play much with 
this now, but that was exactly my thought.  If you can get it to work, great, 
otherwise I might look at it again this weekend.

But the behavior was really strange, to the point that I'm thinking that 
routine may actually be buggy in the standard lib.  Try calling it and just 
printing its output, I couldn't even get _any_ i/o to work after calling it. 
I tried printing normally, to stderr, and even diverting out to a file on disk 
in case readline was hijacking stdout/err.   All to no avail: even my file 
ended up empty, regardless of what I printed to it.

Best,

f