From ellisonbg at gmail.com Mon May 2 23:11:46 2011 From: ellisonbg at gmail.com (Brian Granger) Date: Mon, 2 May 2011 20:11:46 -0700 Subject: [IPython-dev] ipython in a background thread In-Reply-To: <840FA957-B8F0-481D-8BC1-7B26921D90B1@yale.edu> References: <840FA957-B8F0-481D-8BC1-7B26921D90B1@yale.edu> Message-ID: I would check out the multiprocess Qt GUI that we have in master now. Cheers, Brian On Fri, Apr 29, 2011 at 10:03 AM, Zachary Pincus wrote: > Hello all, > > I find that ipython, specifically ipython running in a good terminal (as opposed to embedded in a GUI window), provides a really great interface (e.g. in my case, a microscope and camera hardware). Given this use, it's been important to also have a GUI thread running so I can throw acquired images rapidly up onto a GL canvas or something. For a while, I've used a hacked-up pyglet runloop that would work in a background thread, and set up a simple message-passing system. > > This got quite kluged quite fast though, and it turns out this approach won't work at all with cocoa GUIs on OS X, which seem to require running on thread-0 (a pity). I know people on this list have looked and thought about these issues a lot, so I'd be curious what seems like the best approach in general? (There used to be some code in IPython for doing this with various window toolkits, but I don't see that in the 0.10.2's codebase anymore...) > > - I could run the GUI from thread-0 and try message-passing to IPython on another thread. Does this work well at all, or is it a huge kluge to get all the readline etc. features working right? > > - I could run the GUI in a separate process entirely, which would force a much cleaner API, but I'm not sure if pumping images at video-rate could be done cleanly. Maybe with shared memmapped arrays? (I've seen some recipes for this sort of thing on the numpy list lately.) > > - Any other possibilities or thoughts? > > Thanks a lot, > Zach > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Brian E. Granger Cal Poly State University, San Luis Obispo bgranger at calpoly.edu and ellisonbg at gmail.com From zachary.pincus at yale.edu Tue May 3 07:36:59 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Tue, 3 May 2011 07:36:59 -0400 Subject: [IPython-dev] ipython in a background thread In-Reply-To: References: <840FA957-B8F0-481D-8BC1-7B26921D90B1@yale.edu> Message-ID: <640923AE-FBEE-4D68-894C-976FC20BF81B@yale.edu> > I would check out the multiprocess Qt GUI that we have in master now. I was looking at that -- unless I'm wrong, that throws up a window and emulates a terminal (or rather, runs ipython) inside of it? I was hoping to avoid that sort of approach, actually, and use a dedicated terminal emulator. However, if folks with more experience with this sort of thing than I have decided that the latter approach really isn't workable, I suppose the former will have to do. Zach > Cheers, > > Brian > > On Fri, Apr 29, 2011 at 10:03 AM, Zachary Pincus > wrote: >> Hello all, >> >> I find that ipython, specifically ipython running in a good terminal (as opposed to embedded in a GUI window), provides a really great interface (e.g. in my case, a microscope and camera hardware). Given this use, it's been important to also have a GUI thread running so I can throw acquired images rapidly up onto a GL canvas or something. For a while, I've used a hacked-up pyglet runloop that would work in a background thread, and set up a simple message-passing system. >> >> This got quite kluged quite fast though, and it turns out this approach won't work at all with cocoa GUIs on OS X, which seem to require running on thread-0 (a pity). I know people on this list have looked and thought about these issues a lot, so I'd be curious what seems like the best approach in general? (There used to be some code in IPython for doing this with various window toolkits, but I don't see that in the 0.10.2's codebase anymore...) >> >> - I could run the GUI from thread-0 and try message-passing to IPython on another thread. Does this work well at all, or is it a huge kluge to get all the readline etc. features working right? >> >> - I could run the GUI in a separate process entirely, which would force a much cleaner API, but I'm not sure if pumping images at video-rate could be done cleanly. Maybe with shared memmapped arrays? (I've seen some recipes for this sort of thing on the numpy list lately.) >> >> - Any other possibilities or thoughts? >> >> Thanks a lot, >> Zach >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> > > > > -- > Brian E. Granger > Cal Poly State University, San Luis Obispo > bgranger at calpoly.edu and ellisonbg at gmail.com From takowl at gmail.com Tue May 3 08:07:49 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Tue, 3 May 2011 13:07:49 +0100 Subject: [IPython-dev] ipython in a background thread In-Reply-To: <640923AE-FBEE-4D68-894C-976FC20BF81B@yale.edu> References: <840FA957-B8F0-481D-8BC1-7B26921D90B1@yale.edu> <640923AE-FBEE-4D68-894C-976FC20BF81B@yale.edu> Message-ID: On 3 May 2011 12:36, Zachary Pincus wrote: > I was looking at that -- unless I'm wrong, that throws up a window and > emulates a terminal (or rather, runs ipython) inside of it? I was hoping to > avoid that sort of approach, actually, and use a dedicated terminal > emulator. However, if folks with more experience with this sort of thing > than I have decided that the latter approach really isn't workable, I > suppose the former will have to do. > You can have the terminal running alongside GUI toolkits, and as far as I know the code should still be in 0.10.2 (I don't know where, but it's under IPython/lib in the dev version). I believe we do this by joining the terminal and GUI event loops, rather than threading, but I've not looked much at the relevant code. There isn't an implementation for Pyglet yet. With the Qt GUI, it's possible to display images inline, rather than just text. I don't know if it's what you're after, but it might be worth looking at. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Tue May 3 08:34:15 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Tue, 3 May 2011 13:34:15 +0100 Subject: [IPython-dev] ipython in a background thread In-Reply-To: References: <840FA957-B8F0-481D-8BC1-7B26921D90B1@yale.edu> <640923AE-FBEE-4D68-894C-976FC20BF81B@yale.edu> Message-ID: On 3 May 2011 13:10, Carl Joseph Younger wrote: > Is is possible to animate the images or just display a still? > I'm not sure. I believe we're using Qt webkit, but perhaps someone else on the list can give more details. Brian's also got an HTML notebook interface in the works, but it's not in master yet. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.hirschfeld at gmail.com Thu May 5 12:18:32 2011 From: dave.hirschfeld at gmail.com (Dave Hirschfeld) Date: Thu, 5 May 2011 16:18:32 +0000 (UTC) Subject: [IPython-dev] zmq/pyzmq Version Incompatibilities Message-ID: Not sure if this is an appropriate forum for pyzmq problems but since it's a dependency for the parallel stuff I thought I'd post here. This is more of a heads-up in case people weren't aware of possible problems running off the latest github checkouts. Running stock zmq 2.1.4 with the pyzmq 2.1.4 branch works except for the (benign?) error: ====================================================================== FAIL: test_create (zmq.tests.test_socket.TestSocket) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\dev\bin\Python26\Lib\site-packages\zmq\tests\test_socket.py", line 47, in test_create self.assertRaisesErrno(zmq.EPROTONOSUPPORT, s.bind, 'ftl://a') File "C:\dev\bin\Python26\Lib\site-packages\zmq\tests\__init__.py", line 104, in assertRaisesErrno got '%s'" % (zmq.ZMQError(errno), zmq.ZMQError(e.errno))) AssertionError: wrong error raised, expected 'Unknown error' got 'Protocol not supported' ---------------------------------------------------------------------- * With trunk pyzmq and trunk zmq I get the following error: C:\dev\src\pyzmq>python setup.py configure --zmq=C:\dev\src\zeromq-trunk running configure ****************************************** Configure: Autodetecting ZMQ settings... Custom ZMQ dir: C:\dev\src\zeromq-trunk c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\dev\src\zeromq-trunk\include -Izmq\utils -Izmq\core -Izmq\devices /Tcdetect\vers.c /Fodetect\vers.obj vers.c c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /nologo /INCREMENTAL:NO /LIBPATH:C:\dev\src\zeromq-trunk\lib libzmq.lib detect\vers.obj /OUT:detect\vers.exe /MANIFESTFILE:detect\vers.exe.manifest C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin\mt.exe -nologo -manifest detect\vers.exe.manifest -outputresource:detect\vers.exe;1 ZMQ version detected: 3.0.0 ****************************************** C:\dev\src\pyzmq>python setup.py build running build running build_py ... running build_ext building 'zmq.core.constants' extension creating build\temp.win32-2.6 creating build\temp.win32-2.6\Release creating build\temp.win32-2.6\Release\zmq creating build\temp.win32-2.6\Release\zmq\core c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\dev\src\zeromq-trunk\include -Izmq\utils -Izmq\core -Izmq\devices -IC:\dev\bin\Python26\include -IC:\dev\bin\Python26\PC /Tczmq\core\constants.c /Fobuild\temp.win32-2.6\Release\zmq\core\constants.obj constants.c zmq\core\constants.c(704) : error C2065: 'ZMQ_NOBLOCK' : undeclared identifier zmq\core\constants.c(824) : error C2065: 'ZMQ_UPSTREAM' : undeclared identifier zmq\core\constants.c(836) : error C2065: 'ZMQ_DOWNSTREAM' : undeclared identifier zmq\core\constants.c(916) : error C2065: 'ZMQ_HWM' : undeclared identifier zmq\core\constants.c(928) : error C2065: 'ZMQ_SWAP' : undeclared identifier zmq\core\constants.c(1012) : error C2065: 'ZMQ_MCAST_LOOP' : undeclared identifier zmq\core\constants.c(1108) : error C2065: 'ZMQ_STREAMER' : undeclared identifier zmq\core\constants.c(1120) : error C2065: 'ZMQ_FORWARDER' : undeclared identifier zmq\core\constants.c(1132) : error C2065: 'ZMQ_QUEUE' : undeclared identifier error: command '"c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe"' failed with exit status 2 C:\dev\src\pyzmq> * With trunk pyzmq and 2.1.6 zmq, pyzmq compiles but the tests segfault: C:\dev\bin\Python26\Lib\site-packages\zmq\tests>python Python 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. ->>> import zmq ->>> zmq.pyzmq_version() '2.1dev' ->>> zmq.zmq_version() '2.1.6' ->>> C:\dev\bin\Python26\Lib\site-packages\zmq\tests>nosetests -v test_fail_init (zmq.tests.test_context.TestContext) ... ok test_init (zmq.tests.test_context.TestContext) ... ok test_instance (zmq.tests.test_context.TestContext) ... ok test_term (zmq.tests.test_context.TestContext) ... ok test_term_hang (zmq.tests.test_context.TestContext) ... ok test_device_attributes (zmq.tests.test_device.TestDevice) ... ok test_device_types (zmq.tests.test_device.TestDevice) ... ok test_single_socket_forwarder_bind (zmq.tests.test_device.TestDevice) ... Assertion failed: Permission denied (..\..\..\src\err.cpp:228) * With trunk pyzmq and 2.1.4 zmq pyzmq won't import: ->>> import zmq Traceback (most recent call last): File "", line 1, in File "C:\dev\bin\Python26\lib\site-packages\zmq\__init__.py", line 35, in from zmq import core, devices File "C:\dev\bin\Python26\lib\site-packages\zmq\core\__init__.py", line 26, in from zmq.core import (constants, error, message, context, ImportError: DLL load failed: The specified procedure could not be found. ->>> HTH, Dave From benjaminrk at gmail.com Thu May 5 18:09:37 2011 From: benjaminrk at gmail.com (MinRK) Date: Thu, 5 May 2011 15:09:37 -0700 Subject: [IPython-dev] zmq/pyzmq Version Incompatibilities In-Reply-To: References: Message-ID: Hello, Thanks for the thorough reports! Are you using VS-2010? I understand there are various issues when you use VS-2010 (MSVC 10) to compile a library, but 2008 (MSVC 9, and the one used for all official Pythons so far) for Python extensions. We have one known issue caused by this, and it is the errno mismatch you see with 2.1.4+2.1.4, but it could be the cause of many of your problems. Note that libzmq has 3 repos right now on GitHub. zeromq2-1 can be considered 'maint', zeromq2-2 is the current dev, and libzmq (version 3.0) is the API-breaking "Don't worry about changing everything" (think Python 3) development repo. I should also note that I have no build issues, and 100% of tests passing on my Windows VM for pzmq-master with: zeromq-2.1.4, 2.1.6, 2.1dev (2.1.7), and 2.2dev (2.2.0). My VM is Windows 7 x86_64, official Python 2.7.1 from MSI, MSVC 9 (VS 2008 express). On Thu, May 5, 2011 at 09:18, Dave Hirschfeld wrote: > Not sure if this is an appropriate forum for pyzmq problems but since it's > a dependency for the parallel stuff I thought I'd post here. > > This is more of a heads-up in case people weren't aware of possible problems > running off the latest github checkouts. Running stock zmq 2.1.4 with the pyzmq > 2.1.4 branch works except for the (benign?) error: This error is Benign *unless* you are trying to handle ZMQ errors yourself. This errno mismatch only seems to occur when libzmq is built with VS-2010. It has no effect on IPython's use of libzmq. > > ====================================================================== > FAIL: test_create (zmq.tests.test_socket.TestSocket) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "C:\dev\bin\Python26\Lib\site-packages\zmq\tests\test_socket.py", > ? ?line 47, in test_create > ? ?self.assertRaisesErrno(zmq.EPROTONOSUPPORT, s.bind, 'ftl://a') > ?File "C:\dev\bin\Python26\Lib\site-packages\zmq\tests\__init__.py", > ? ?line 104, in assertRaisesErrno > ? ?got '%s'" % (zmq.ZMQError(errno), zmq.ZMQError(e.errno))) > AssertionError: wrong error raised, > ? ? ? ? ? ? ? ?expected 'Unknown error' got 'Protocol not supported' > > ---------------------------------------------------------------------- > > > * With trunk pyzmq and trunk zmq I get the following error: See above for multiply defined 'trunk' for zeromq. We do not yet have a branch that tracks 3.0 development (libzmq on github) > > C:\dev\src\pyzmq>python setup.py configure --zmq=C:\dev\src\zeromq-trunk > running configure > ****************************************** > Configure: Autodetecting ZMQ settings... > ? ?Custom ZMQ dir: ? ? ? C:\dev\src\zeromq-trunk > c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox > /MD /W3 /GS- /DNDEBUG -IC:\dev\src\zeromq-trunk\include -Izmq\utils -Izmq\core > -Izmq\devices /Tcdetect\vers.c /Fodetect\vers.obj > vers.c > c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /nologo > /INCREMENTAL:NO /LIBPATH:C:\dev\src\zeromq-trunk\lib libzmq.lib detect\vers.obj > /OUT:detect\vers.exe /MANIFESTFILE:detect\vers.exe.manifest > C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin\mt.exe -nologo -manifest > detect\vers.exe.manifest -outputresource:detect\vers.exe;1 > ? ?ZMQ version detected: 3.0.0 > ****************************************** > > C:\dev\src\pyzmq>python setup.py build > running build > running build_py > ... > running build_ext > building 'zmq.core.constants' extension > creating build\temp.win32-2.6 > creating build\temp.win32-2.6\Release > creating build\temp.win32-2.6\Release\zmq > creating build\temp.win32-2.6\Release\zmq\core > c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox > /MD /W3 /GS- /DNDEBUG -IC:\dev\src\zeromq-trunk\include -Izmq\utils -Izmq\core > -Izmq\devices -IC:\dev\bin\Python26\include -IC:\dev\bin\Python26\PC > /Tczmq\core\constants.c /Fobuild\temp.win32-2.6\Release\zmq\core\constants.obj > constants.c > zmq\core\constants.c(704) : > ? ?error C2065: 'ZMQ_NOBLOCK' : undeclared identifier > zmq\core\constants.c(824) : > ? ?error C2065: 'ZMQ_UPSTREAM' : undeclared identifier > zmq\core\constants.c(836) : > ? ?error C2065: 'ZMQ_DOWNSTREAM' : undeclared identifier > zmq\core\constants.c(916) : > ? ?error C2065: 'ZMQ_HWM' : undeclared identifier > zmq\core\constants.c(928) : > ? ?error C2065: 'ZMQ_SWAP' : undeclared identifier > zmq\core\constants.c(1012) : > ? ?error C2065: 'ZMQ_MCAST_LOOP' : undeclared identifier > zmq\core\constants.c(1108) : > ? ?error C2065: 'ZMQ_STREAMER' : undeclared identifier > zmq\core\constants.c(1120) : > ? ?error C2065: 'ZMQ_FORWARDER' : undeclared identifier > zmq\core\constants.c(1132) : > ? ?error C2065: 'ZMQ_QUEUE' : undeclared identifier > error: command '"c:\Program Files (x86)\Microsoft Visual Studio > 9.0\VC\BIN\cl.exe"' failed with exit status 2 > > C:\dev\src\pyzmq> > > > * With trunk pyzmq and 2.1.6 zmq, pyzmq compiles but the tests segfault: Permission denied is not a segfault. I cannot reproduce this, pyzmq master passes all tests with 2.1.6. It's possible this is due to using MSVC10 with libzmq, and MSVC 9 with pyzmq. If your checkout was before yesterday, this could have simply been a port conflict in the test (recently fixed). > > C:\dev\bin\Python26\Lib\site-packages\zmq\tests>python > Python 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) > [MSC v.1500 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > ->>> import zmq > ->>> zmq.pyzmq_version() > '2.1dev' > ->>> zmq.zmq_version() > '2.1.6' > ->>> > > C:\dev\bin\Python26\Lib\site-packages\zmq\tests>nosetests -v > test_fail_init (zmq.tests.test_context.TestContext) ... ok > test_init (zmq.tests.test_context.TestContext) ... ok > test_instance (zmq.tests.test_context.TestContext) ... ok > test_term (zmq.tests.test_context.TestContext) ... ok > test_term_hang (zmq.tests.test_context.TestContext) ... ok > test_device_attributes (zmq.tests.test_device.TestDevice) ... ok > test_device_types (zmq.tests.test_device.TestDevice) ... ok > test_single_socket_forwarder_bind (zmq.tests.test_device.TestDevice) ... > Assertion failed: Permission denied (..\..\..\src\err.cpp:228) > > * With trunk pyzmq and 2.1.4 zmq pyzmq won't import: > > ->>> import zmq > Traceback (most recent call last): > ?File "", line 1, in > ?File "C:\dev\bin\Python26\lib\site-packages\zmq\__init__.py", > ? ?line 35, in > ? ?from zmq import core, devices > ?File "C:\dev\bin\Python26\lib\site-packages\zmq\core\__init__.py", > ? ?line 26, in > ? ?from zmq.core import (constants, error, message, context, > ImportError: DLL load failed: The specified procedure could not be found. Do you have a different libzmq.dll on your PATH? Did you make sure pyzmq was clean when you rebuilt it against each new libzmq? I can run pyzmq master with libzmq 2.1.4 (built with either VS 2010 or 2008), with no problems aside from the errno mismatch on VS2010). I hope we can resolve these, because pzymq is known to work on Windows in all of these combinations (excluding libzmq-3.0). -MinRK > ->>> > > HTH, > Dave > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > From dave.hirschfeld at gmail.com Tue May 10 10:56:05 2011 From: dave.hirschfeld at gmail.com (David Hirschfeld) Date: Tue, 10 May 2011 15:56:05 +0100 Subject: [IPython-dev] zmq/pyzmq Version Incompatibilities In-Reply-To: References: Message-ID: On Thu, May 5, 2011 at 11:09 PM, MinRK wrote: > > Are you using VS-2010? ?I understand there are various issues when you > use VS-2010 (MSVC 10) to compile a library, but 2008 (MSVC 9, and the > one used for all official Pythons so far) for Python extensions. ?We > have one known issue caused by this, and it is the errno mismatch you > see with 2.1.4+2.1.4, but it could be the cause of many of your > problems. > > > I should also note that I have no build issues, and 100% of tests > passing on my Windows VM for pzmq-master with: zeromq-2.1.4, 2.1.6, > 2.1dev (2.1.7), and 2.2dev (2.2.0). > > My VM is Windows 7 x86_64, official Python 2.7.1 from MSI, MSVC 9 (VS > 2008 express). > > > Do you have a different libzmq.dll on your PATH? ?Did you make sure > pyzmq was clean when you rebuilt it against each new libzmq? > > I can run pyzmq master with libzmq 2.1.4 (built with either VS 2010 or > 2008), with no problems aside from the errno mismatch on VS2010). > > I hope we can resolve these, because pzymq is known to work on Windows > in all of these combinations (excluding libzmq-3.0). > > -MinRK > Hi Min, I'm on Win7 x64 running Python 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)] on win32. I had previously used VS2010 to compile zmq/pyzmq but to eliminate that in the following I've used VS2008. I'm using zmq-2.2 master and pyzmq master and I made sure to delete the build directory beforehand so it should be a clean install. There's no other libzmq.dll in my path. The results of the zmq/pyzmq build are attached - everything seems fine at first glance however when running the tests I get the "Permission denied" error occurring at random times: C:\dev\bin\Python26\Lib\site-packages\zmq\tests>nosetests -v test_no_events (zmq.tests.test_poll.TestPoll) ... Assertion failed: Permission denied (..\..\..\src\err.cpp:234) This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. C:\dev\bin\Python26\Lib\site-packages\zmq\tests> C:\dev\bin\Python26\Lib\site-packages\zmq\tests>nosetests -v test_single_socket_forwarder_bind (zmq.tests.test_device.TestDevice) ... Assertion failed: Permission denied (..\..\..\src\err.cpp:234) This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. C:\dev\bin\Python26\Lib\site-packages\zmq\tests> C:\dev\bin\Python26\Lib\site-packages\zmq\tests>nosetests -v test_monitor (zmq.tests.test_monqueue.TestMonitoredQueue) ... Assertion failed: Permission denied (..\..\..\src\err.cpp:234) This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. C:\dev\bin\Python26\Lib\site-packages\zmq\tests> I'm not sure if segfault is the right technical term but it does crash the python interpreter, bringing up a "python.exe has stopped working" dialog box. It seems like a tricky one to debug and could easily be something specific to my environment. Hopefully this helps narrow it down a bit though... -Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pyzmq-build.log Type: application/octet-stream Size: 25472 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pyzmq-configure.log Type: application/octet-stream Size: 793 bytes Desc: not available URL: From fperez.net at gmail.com Thu May 12 09:56:44 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 12 May 2011 08:56:44 -0500 Subject: [IPython-dev] Heads-up on htmlnotebook branch Message-ID: Hi folks, a couple of minor notes on the htmlnb branch... - I'm seeing this traceback when I have the qt console open as well: Traceback (most recent call last): File "/home/fperez/ipython/ipython/IPython/frontend/qt/base_frontend_mixin.py", line 102, in _dispatch handler(msg) File "/home/fperez/ipython/ipython/IPython/frontend/qt/console/rich_ipython_widget.py", line 98, in _handle_display_data if not self._hidden and self._is_from_this_session(msg): File "/home/fperez/ipython/ipython/IPython/frontend/qt/base_frontend_mixin.py", line 109, in _is_from_this_session return msg['parent_header']['session'] == session KeyError: 'session' - I tried to merge htmlnotebook into master (locally, obviously :) to see if it would go away, but right now there's a small conflict in setupbase.py. So we probably want to update htmlnotebook to merge cleanly. I won't get to this for the next couple of days, I'm giving a couple of talks in Austin about it though (and demoing it :) so I'll report on anything else I see. I gave one at the Austin Python Users Group last night that was very well received, and hopefully we'll be getting some more pull requests soon from the local hackers :) Cheers, f From ellisonbg at gmail.com Thu May 12 12:49:39 2011 From: ellisonbg at gmail.com (Brian Granger) Date: Thu, 12 May 2011 09:49:39 -0700 Subject: [IPython-dev] Heads-up on htmlnotebook branch In-Reply-To: References: Message-ID: Fernando, I think may be a regression in the qt app. Can you try multiple qt frontends with no htmlnotebook? Brian On Thu, May 12, 2011 at 6:56 AM, Fernando Perez wrote: > Hi folks, > > a couple of minor notes on the htmlnb branch... > > - I'm seeing this traceback when I have the qt console open as well: > > Traceback (most recent call last): > ?File "/home/fperez/ipython/ipython/IPython/frontend/qt/base_frontend_mixin.py", > line 102, in _dispatch > ? ?handler(msg) > ?File "/home/fperez/ipython/ipython/IPython/frontend/qt/console/rich_ipython_widget.py", > line 98, in _handle_display_data > ? ?if not self._hidden and self._is_from_this_session(msg): > ?File "/home/fperez/ipython/ipython/IPython/frontend/qt/base_frontend_mixin.py", > line 109, in _is_from_this_session > ? ?return msg['parent_header']['session'] == session > KeyError: 'session' > > - I tried to merge htmlnotebook into master (locally, obviously :) to > see if it would go away, but right now there's a small conflict in > setupbase.py. ?So we probably want to update htmlnotebook to merge > cleanly. > > I won't get to this for the next couple of days, I'm giving a couple > of talks in Austin about it though (and demoing it :) so I'll report > on anything else I see. > > I gave one at the Austin Python Users Group last night that was very > well received, and hopefully we'll be getting some more pull requests > soon from the local hackers :) > > Cheers, > > f > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Brian E. Granger Cal Poly State University, San Luis Obispo bgranger at calpoly.edu and ellisonbg at gmail.com From joonpyro at gmail.com Fri May 13 16:48:37 2011 From: joonpyro at gmail.com (Joon Ro) Date: Fri, 13 May 2011 15:48:37 -0500 Subject: [IPython-dev] run filename autocompletion "The kernel heartbeat has been inactive ... " error Message-ID: Hi, First of all let me say the qtconsole is awesome. Thanks for the brilliant work. I don't know if this is a bug, but when I press tab to autocomplete during the input of the argument of the run magic command: In [2]: %run 'work[TAB] ipython-qtconsole hangs for a while, and then gives me the following error: The kernel heartbeat has been inactive for 3.00 seconds. Do you want to restart the kernel? You may first want to check the network connection. And then if I press yes it tries to restart the python shell but it hangs after the initial messages (In [1] never shows up) In other cases (for example, with cd) the autocompletion of directories works well. I'm using openSUSE 11.4 64bit. I am not sure if this is place to report these problems, so if not please let me know. Joon -------------- next part -------------- An HTML attachment was scrubbed... URL: From epatters at enthought.com Fri May 13 17:38:55 2011 From: epatters at enthought.com (Evan Patterson) Date: Fri, 13 May 2011 16:38:55 -0500 Subject: [IPython-dev] run filename autocompletion "The kernel heartbeat has been inactive ... " error In-Reply-To: References: Message-ID: <069E9B21-354D-431D-869B-810644BA352F@enthought.com> Joon, Thanks for the report. You have indeed found a bug, although I was not able to immediately reproduce it myself. Could you please create an issue for this on the IPython issue tracker on GitHub? Also, if you run the Qt console from a terminal, you should see some output from the kernel. If there is a traceback after you execute the bad command, please attach it to the issue. Thanks, Evan On May 13, 2011, at 3:48 PM, Joon Ro wrote: > Hi, > > First of all let me say the qtconsole is awesome. Thanks for the brilliant work. > > I don't know if this is a bug, but when I press tab to autocomplete during the input of the argument of the run magic command: > > In [2]: %run 'work[TAB] > > ipython-qtconsole hangs for a while, and then gives me the following error: > > The kernel heartbeat has been inactive for 3.00 seconds. Do you want to restart the kernel? You may first want to check the network connection. > > And then if I press yes it tries to restart the python shell but it hangs after the initial messages (In [1] never shows up) > > In other cases (for example, with cd) the autocompletion of directories works well. > > I'm using openSUSE 11.4 64bit. > > I am not sure if this is place to report these problems, so if not please let me know. > > Joon > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From joonpyro at gmail.com Fri May 13 20:01:58 2011 From: joonpyro at gmail.com (Joon Ro) Date: Fri, 13 May 2011 19:01:58 -0500 Subject: [IPython-dev] run filename autocompletion "The kernel heartbeat has been inactive ... " error In-Reply-To: <069E9B21-354D-431D-869B-810644BA352F@enthought.com> References: <069E9B21-354D-431D-869B-810644BA352F@enthought.com> Message-ID: Thanks for the reply Evan. As you said I ran ipython-qtconsole --pylab inline in the terminal, and actually that did not happen. It happens only when I run it as an application - I made an application item with command: ipython-qtconsole --pylab inline (I'm using KDE) Now I tried it with different options, and when I check "Run in terminal" it does not happen. So it only happens when I run it as an application, with "Run in terminal" unchecked. I will create an issue for this. Thank you, Joon On Fri, 13 May 2011 16:38:55 -0500, Evan Patterson wrote: > Joon, > > Thanks for the report. You have indeed found a bug, although I was not > able to immediately reproduce it myself. > Could you please create an issue for this on the IPython issue tracker > on GitHub? Also, if you run the Qt console from a terminal, you should > see some output from the kernel. If there >is a traceback after you > execute the bad command, please attach it to the issue. > > Thanks, > Evan > > On May 13, 2011, at 3:48 PM, Joon Ro wrote: > >>>> Hi, >> >> First of all let me say the qtconsole is awesome. Thanks for the >> brilliant work. >> >> I don't know if this is a bug, but when I press tab to autocomplete >> during the input of the argument of the run magic command: >> In [2]: %run 'work[TAB] >> >> ipython-qtconsole hangs for a while, and then gives me the following >> error: >> >> The kernel heartbeat has been inactive for 3.00 seconds. Do you want to >> restart the kernel? You may first want to check the network connection. >> >> And then if I press yes it tries to restart the python shell but it >> hangs after the initial messages (In [1] never shows up) >> >> In other cases (for example, with cd) the autocompletion of directories >> works well. >> I'm using openSUSE 11.4 64bit. >> >> I am not sure if this is place to report these problems, so if not >> please let me know. >> >> Joon >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Using Opera's revolutionary email client: http://www.opera.com/mail/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjaminrk at gmail.com Mon May 16 01:10:13 2011 From: benjaminrk at gmail.com (MinRK) Date: Sun, 15 May 2011 22:10:13 -0700 Subject: [IPython-dev] zmq/pyzmq Version Incompatibilities In-Reply-To: References: Message-ID: My guess is that these are port conflict issues. I believe Permission Denied is what happens when you try to bind to an in-use port, but maybe that's not accurate. I don't know if this would change anything, but do you get any different output if you do the Windows 'run as administrator' trick? (I've never done such a thing with a command-line script). On Tue, May 10, 2011 at 07:56, David Hirschfeld wrote: > On Thu, May 5, 2011 at 11:09 PM, MinRK wrote: >> >> Are you using VS-2010? ?I understand there are various issues when you >> use VS-2010 (MSVC 10) to compile a library, but 2008 (MSVC 9, and the >> one used for all official Pythons so far) for Python extensions. ?We >> have one known issue caused by this, and it is the errno mismatch you >> see with 2.1.4+2.1.4, but it could be the cause of many of your >> problems. >> >> >> I should also note that I have no build issues, and 100% of tests >> passing on my Windows VM for pzmq-master with: zeromq-2.1.4, 2.1.6, >> 2.1dev (2.1.7), and 2.2dev (2.2.0). >> >> My VM is Windows 7 x86_64, official Python 2.7.1 from MSI, MSVC 9 (VS >> 2008 express). >> >> >> Do you have a different libzmq.dll on your PATH? ?Did you make sure >> pyzmq was clean when you rebuilt it against each new libzmq? >> >> I can run pyzmq master with libzmq 2.1.4 (built with either VS 2010 or >> 2008), with no problems aside from the errno mismatch on VS2010). >> >> I hope we can resolve these, because pzymq is known to work on Windows >> in all of these combinations (excluding libzmq-3.0). >> >> -MinRK >> > > Hi Min, > I'm on Win7 x64 running Python 2.6.5 (r265:79096, Mar 19 2010, > 21:48:26) [MSC v.1500 32 bit (Intel)] on win32. I had previously used > VS2010 to compile zmq/pyzmq but to eliminate that in the following > I've used VS2008. I'm using zmq-2.2 master and pyzmq master and I made > sure to delete the build directory beforehand so it should be a clean > install. There's no other libzmq.dll in my path. > > The results of the zmq/pyzmq build are attached - everything seems > fine at first glance however when running the tests I get the > "Permission denied" error occurring at random times: > > C:\dev\bin\Python26\Lib\site-packages\zmq\tests>nosetests -v > > test_no_events (zmq.tests.test_poll.TestPoll) ... Assertion failed: > Permission denied (..\..\..\src\err.cpp:234) > > This application has requested the Runtime to terminate it in an unusual way. > Please contact the application's support team for more information. > > C:\dev\bin\Python26\Lib\site-packages\zmq\tests> > > C:\dev\bin\Python26\Lib\site-packages\zmq\tests>nosetests -v > > test_single_socket_forwarder_bind (zmq.tests.test_device.TestDevice) > ... Assertion failed: Permission denied (..\..\..\src\err.cpp:234) > > This application has requested the Runtime to terminate it in an unusual way. > Please contact the application's support team for more information. > > C:\dev\bin\Python26\Lib\site-packages\zmq\tests> > > C:\dev\bin\Python26\Lib\site-packages\zmq\tests>nosetests -v > > test_monitor (zmq.tests.test_monqueue.TestMonitoredQueue) ... > Assertion failed: Permission denied (..\..\..\src\err.cpp:234) > > This application has requested the Runtime to terminate it in an unusual way. > Please contact the application's support team for more information. > > C:\dev\bin\Python26\Lib\site-packages\zmq\tests> > > > I'm not sure if segfault is the right technical term but it does crash > the python interpreter, bringing up a "python.exe has stopped working" > dialog box. > > It seems like a tricky one to debug and could easily be something > specific to my environment. Hopefully this helps narrow it down a bit > though... > > -Dave > From robert.kern at gmail.com Tue May 17 18:48:59 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 17 May 2011 17:48:59 -0500 Subject: [IPython-dev] Package-specific extensions: an idea Message-ID: There's a general problem for writing package-specific extensions to IPython. Namely, you want to be able to write and enable an extension to IPython that provides special behavior for particular objects, say numpy arrays, without requiring that a particular package be imported at startup. You can often get by in ad hoc ways. For example, in the pretty-printing code, I added the ability to register deferred type-checking. For magic functions, you can usually just do local imports inside the magics. However, I've had an idea for a general solution. Let's say you want to make an extension that does something special with numpy arrays. Perhaps it alters the ? introspection to print out array metadata like shape and dtype instead of the ndarray docstring when you use ? on arrays. You don't want it to modify anything until numpy is imported. You also don't want to import numpy at startup every time. You could implement a deferred isinstance() check the way I do it in pretty.py, but that's a complicated procedure that would need to be replicated everywhere. Instead, we could have an API for our configuration for deferred extensions, namely, a list of (module_name, extension_name) pairs. IPython would keep a registry mapping these module names to the extension names. IPython's execution loop can be modified to check sys.modules after every execution and activate every extension whose requisite modules appear in sys.modules. Then it removes those extensions from the list. So if my extension is named ipy_numpy_oinspect.py, then I'd have something like the following in my ipython_config.py file: c.Global.deferred_extensions = [ # (module_to_wait_for, extension) ('numpy', 'ipy_numpy_oinspect'), ] And somewhere in run_cell() after the run_ast() call: # Check for any deferred extensions. # FIXME: probably wrap this up in a method. loaded = [] for modname, extname in self.extension_manager.deferred_extensions: # Explicitly check for is not None since sometimes a missing module # will get a None entry to let the import mechanism fail faster. if sys.modules.get(modname, None) is not None: self.extension_manager.load_extension(extname) loaded.append((modname, extname)) for x in loaded: self.extension_manager.deferred_extensions.remove(x) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ipython at mspacek.mm.st Fri May 20 16:58:57 2011 From: ipython at mspacek.mm.st (Martin Spacek) Date: Fri, 20 May 2011 13:58:57 -0700 Subject: [IPython-dev] replacement for appstart_qt4(app) ? Message-ID: Motivated by ipython-qtconsole, I've taken the plunge and switched from 0.10 to git master. Apologies if this post is more appropriate for ipython-user. I have an app that used appstart_qt4(app) in its __name__ == '__main__' block to automatically drop into the IPython debugger on any error. This is still referenced in: docs/examples/lib/gui-qt.py appstart_qt4() is now gone from git master, and I can't seem to find any mention of what to use in its place. get_app_qt4() and start_event_loop_qt4() in IPython.lib.guisupport seem like good candidates, but AFAICT they don't automatically drop you into the ipdb on an error. I did however find this sys.excepthook method, which seems to work: http://ipython.scipy.org/doc/manual/html/interactive/reference.html#automatic-invocation-of-pdb-on-exceptions I want to both allow the user to drop into ipython at any point in time, as well as automatically drop into it on any error. After some messing around, I put together this minimal working example for git master: import sys from PyQt4 import QtCore, QtGui from IPython import embed from IPython.core import ultratb def set_excepthook(): """Drops us into IPython's debugger on any error""" sys.excepthook = ultratb.FormattedTB(mode='Verbose', call_pdb=1) class SimpleWindow(QtGui.QWidget): def __init__(self, parent=None): QtGui.QWidget.__init__(self, parent) self.setGeometry(300, 300, 300, 200) self.setWindowTitle('embed and ipdb test') shellButton = QtGui.QPushButton('drop into shell', self) shellButton.setGeometry(10, 10, 100, 35) self.connect(shellButton, QtCore.SIGNAL('clicked()'), self.shell) raiseErrorButton = QtGui.QPushButton('raise Error', self) raiseErrorButton.setGeometry(10, 50, 100, 35) self.connect(raiseErrorButton, QtCore.SIGNAL('clicked()'), self.raise_error) def shell(self): embed(display_banner=False) # "self" is accessible from the shell # embed() seems to override the excepthook, need to reset it: set_excepthook() def raise_error(self): raise RuntimeError if __name__ == "__main__": # prevents "The event loop is already running" errors: QtCore.pyqtRemoveInputHook() set_excepthook() app = QtGui.QApplication(sys.argv) window = SimpleWindow() window.show() sys.exit(app.exec_()) Hitting the shell button drops you into ipython, and hitting the raise Error button drops you into ipdb to inspect the error. I've noticed the embed() call clears the sys.excepthook setting, so I have to constantly reset it afterwards. Seems a bit messy. Is there a better way to do this? What I'd ultimately want is a second Qt window running ipython-qtconsole in tandem with the main window, so the user never has to hit the "shell" button to stop the Qt loop just to inspect all of the existing objects. I used to have this kind of functionality in wxPython with pycrust/pyshell. Is such a thing currently possible with IPython in Qt? Cheers, Martin From efiring at hawaii.edu Sat May 21 17:49:26 2011 From: efiring at hawaii.edu (Eric Firing) Date: Sat, 21 May 2011 11:49:26 -1000 Subject: [IPython-dev] tiny patch for sphinx version change Message-ID: <4DD83366.6000808@hawaii.edu> I updated to ipython master. When attempting to build the docs I ran into a problem associated with changes in Sphinx between version 0.6.7 and 1.x. The attached patch allows the build to proceed with the latter and should still work with the former, although I have not tested that. Eric -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: sphinx_version_patch URL: From pivanov314 at gmail.com Tue May 24 04:20:46 2011 From: pivanov314 at gmail.com (Paul Ivanov) Date: Tue, 24 May 2011 01:20:46 -0700 Subject: [IPython-dev] UI related: TermKit (a webkit-based terminal) Message-ID: <20110524082046.GE2795@ykcyc> Hey gang, thought I'd point you to this project, which looks like it has a bit of overlap and capitalizes on some of the capabilities we now have with the new Qt console stuff in ipython trunk: a nice writeup: http://acko.net/blog/on-termkit code: https://github.com/unconed/TermKit -- Paul Ivanov 314 address only used for lists, off-list direct email at: http://pirsquared.org | GPG/PGP key id: 0x0F3E28F7 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: From dave.hirschfeld at gmail.com Tue May 24 06:17:09 2011 From: dave.hirschfeld at gmail.com (David Hirschfeld) Date: Tue, 24 May 2011 11:17:09 +0100 Subject: [IPython-dev] zmq/pyzmq Version Incompatibilities In-Reply-To: References: Message-ID: On Mon, May 16, 2011 at 6:10 AM, MinRK wrote: > My guess is that these are port conflict issues. ?I believe Permission > Denied is what happens when you try to bind to an in-use port, but > maybe that's not accurate. > > I don't know if this would change anything, but do you get any > different output if you do the Windows 'run as administrator' trick? > (I've never done such a thing with a command-line script). > I am an administrator on my machine. I did try specifically opening a terminal as admin and running nosetests from there with the same results. If I open up the debugger on the exception I get the attached call stack where it can be seen that it's trying to bind to tcp://127.0.0.1:64385 If I use netstat to check what ports are being taken there's no mention of 127.0.0.1:64385 C:\Users\dhirschfeld>netstat -ao | grep LISTENING TCP 0.0.0.0:135 DavePC:0 LISTENING 160 TCP 0.0.0.0:445 DavePC:0 LISTENING 4 TCP 0.0.0.0:3389 DavePC:0 LISTENING 1292 TCP 0.0.0.0:8009 DavePC:0 LISTENING 7848 TCP 0.0.0.0:8081 DavePC:0 LISTENING 2588 TCP 0.0.0.0:9535 DavePC:0 LISTENING 1820 TCP 0.0.0.0:9593 DavePC:0 LISTENING 1804 TCP 0.0.0.0:9594 DavePC:0 LISTENING 1804 TCP 0.0.0.0:9595 DavePC:0 LISTENING 1804 TCP 0.0.0.0:33354 DavePC:0 LISTENING 1692 TCP 0.0.0.0:49152 DavePC:0 LISTENING 692 TCP 0.0.0.0:49153 DavePC:0 LISTENING 592 TCP 0.0.0.0:49154 DavePC:0 LISTENING 940 TCP 0.0.0.0:53638 DavePC:0 LISTENING 1524 TCP 0.0.0.0:63961 DavePC:0 LISTENING 816 TCP 0.0.0.0:64084 DavePC:0 LISTENING 780 TCP 0.0.0.0:64089 DavePC:0 LISTENING 4904 TCP 127.0.0.1:1434 DavePC:0 LISTENING 2224 TCP 127.0.0.1:9592 DavePC:0 LISTENING 1804 TCP 127.0.0.1:21584 DavePC:0 LISTENING 1072 TCP 127.0.0.1:55376 DavePC:0 LISTENING 6364 TCP 127.0.0.1:58862 DavePC:0 LISTENING 6364 TCP 127.0.0.1:58901 DavePC:0 LISTENING 6364 TCP 127.0.0.1:63963 DavePC:0 LISTENING 1692 TCP 127.0.0.1:63964 DavePC:0 LISTENING 1804 TCP [::]:135 DavePC:0 LISTENING 160 TCP [::]:445 DavePC:0 LISTENING 4 TCP [::]:3389 DavePC:0 LISTENING 1292 TCP [::]:49152 DavePC:0 LISTENING 692 TCP [::]:49153 DavePC:0 LISTENING 592 TCP [::]:49154 DavePC:0 LISTENING 940 TCP [::]:53638 DavePC:0 LISTENING 1524 TCP [::]:63961 DavePC:0 LISTENING 816 TCP [::]:64084 DavePC:0 LISTENING 780 TCP [::]:64089 DavePC:0 LISTENING 4904 TCP [::1]:1434 DavePC:0 LISTENING 2224 C:\Users\dhirschfeld>netstat -ao | grep 64385 C:\Users\dhirschfeld> ...so I'm stumped, sorry. I think I'll have to revert to 2.1.4 in the meantime. In case it helps in the Windows Event Viewer I get an Application Error with the following details: Faulting application name: python.exe, version: 0.0.0.0, time stamp: 0x4ba3e443 Faulting module name: MSVCR90D.dll, version: 9.0.30729.4148, time stamp: 0x4a594c7b Exception code: 0x80000003 Fault offset: 0x0003c355 Faulting process id: 0x2724 Faulting application start time: 0x01cc19fa9c5ac48d Faulting application path: C:\dev\bin\Python26\python.exe Faulting module path: C:\WINDOWS\WinSxS\x86_microsoft.vc90.debugcrt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_2a4cbfc25558bcd3\MSVCR90D.dll Report Id: df3ef44d-85ed-11e0-8d2a-d48564bd9523 -Dave -------------- next part -------------- A non-text attachment was scrubbed... Name: call_stack.png Type: image/png Size: 30004 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: terminal_run_under_debug.png Type: image/png Size: 49142 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: call_stack2.png Type: image/png Size: 26969 bytes Desc: not available URL: From takowl at gmail.com Tue May 24 06:18:58 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Tue, 24 May 2011 11:18:58 +0100 Subject: [IPython-dev] UI related: TermKit (a webkit-based terminal) In-Reply-To: <20110524082046.GE2795@ykcyc> References: <20110524082046.GE2795@ykcyc> Message-ID: On 24 May 2011 09:20, Paul Ivanov wrote: > Hey gang, > > thought I'd point you to this project, which looks like it has a > bit of overlap and capitalizes on some of the capabilities > we now have with the new Qt console stuff in ipython trunk: > I've left a comment on his blog about what we're doing, as well. He's certainly had similar ideas to those behind the new interactive IPython frontends. I like the idea. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From andresete.chaos at gmail.com Tue May 24 22:00:43 2011 From: andresete.chaos at gmail.com (=?UTF-8?Q?Omar_Andr=C3=A9s_Zapata_Mesa?=) Date: Tue, 24 May 2011 21:00:43 -0500 Subject: [IPython-dev] Zmq Terminal. Message-ID: Hi all. Thanks Thomas. Sorry by my english!! I am studying and I undertake to improve. I am working in implemend Crt+C but I am having a rare problem. Explain. I am trying to capture the exception when raw_input is getting the code, see code try: print() self._splitter.push(raw_input(' In[%i]: '%self.prompt_count+self.code)) while self._splitter.push_accepts_more(): self.code = raw_input(' .....: '+' '*self._splitter.indent_spaces) self._splitter.push(' '*self._splitter.indent_spaces+self.code) self._execute(self._splitter.source,False) self._splitter.reset() except KeyboardInterrupt: print('\nKeyboardInterrupt\n') pass but it produce an error in the ipkernel module In[1]: KeyboardInterrupt In[1]: --------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) /home/omazapa/MyProjects/GSoC/ipython/IPython/frontend/zmqterminal/ in () /home/omazapa/MyProjects/GSoC/ipython/IPython/zmq/ipkernel.pyc in main() 678 shell=kernel.shell) 679 --> 680 start_kernel(namespace, kernel) 681 682 /home/omazapa/MyProjects/GSoC/ipython/IPython/zmq/entry_point.pyc in start_kernel(namespace=Namespace(colors=None, hb=39548, ip='127.0.0.1',...ue, pub=34056, pylab=None, req=58834, xrep=59134), kernel=) 147 148 # Start the kernel mainloop. --> 149 kernel.start() 150 151 /home/omazapa/MyProjects/GSoC/ipython/IPython/zmq/ipkernel.pyc in start(self=) 168 """ 169 while True: --> 170 time.sleep(self._poll_interval) global stdin = undefined global stdout = undefined global stderr = undefined global optional = undefined global default = undefined global None = undefined 171 self.do_one_iteration() 172 KeyboardInterrupt: then the connection is lost. the idea is capture the exception KeyboardInterrupt in two differents times 1) in raw_input 2) when sub socket is is printing outputs to send signal SIGINT using kernel manager's method self.km.interrupt_kernel() The capture of second option is working but the first not for these unkown error you can see the code applying the attached diff file to apply in frontend.py file I prefer dont push this little change yet to the repo, before get some opinion. Very thanks Thomas You're a big help. -- Omar Andres Zapata Mesa User Linux #490962 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: frontend.diff Type: text/x-patch Size: 1022 bytes Desc: not available URL: From takowl at gmail.com Wed May 25 06:29:41 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 25 May 2011 11:29:41 +0100 Subject: [IPython-dev] Zmq Terminal. In-Reply-To: References: Message-ID: 2011/5/25 Omar Andr?s Zapata Mesa > but it produce an error in the ipkernel module I think the problem is that both the frontend and the kernel get to see the Ctrl-C, so there's a KeyboardInterrupt raised in each. I don't know much about how processes fit together, but a bit of fiddling suggests that setting independent to True when you start the kernel (add it to the kwargs dict on line 242) seems to work. I think that might have other effects, though - reading the code, it looks like the kernel will live on if the frontend crashes, which may not be what you want. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed May 25 10:45:35 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 25 May 2011 15:45:35 +0100 Subject: [IPython-dev] [IPython-User] ipython (0.10.1) -wthread and -pylab bug in ubuntu natty In-Reply-To: <201105251553.22500.vincefn@users.sourceforge.net> References: <201105251553.22500.vincefn@users.sourceforge.net> Message-ID: That reminds me - if I select the GTK backend in matplotlibrc, I can replicate the blocking behaviour in trunk as well, with: x = randn(1000) hist(x) # No window appears show() # Blocks even after closing the window Is this some quirk of my setup, or can anyone else replicate that? This is matplotlib 0.99.3. Thomas On 25 May 2011 14:53, Vincent Favre-Nicolin wrote: > Apologies for cross-posting - if you are not using ubuntu, don't read > ahead. > > If you are using ipython combined with '-wthread' (for mayavi's mlab) or > '-pylab' (for matplotlib), there is currently a bug in Ubunty 11.04 (natty) > with ipython 0.10.1, which prevents the correct thread handling: > e.g. with mayavi's mlab, it is necessary to use 'show()' to interact with > the window, which is not very practical (and is blocking). And for > '-pylab', > the GTK and wx (Aggg) backend do not work... > > The bug is known and already corrected in ipython 0.10.2: > https://github.com/ipython/ipython/issues/185 > > So the remaining issue is to get the new version as an update in ubuntu > 11.04/natty ! > => this is why I'm writing this email, so that other people concerned can > vote > for the bug and, if maintainers read this, if they can push this as stable- > release-update (SRU)... > > The bug to vote for is @: > https://bugs.launchpad.net/bugs/777420 > > Incidentally, it is easy to install ipython 0.10.2 'manually', as it is > already available for ubuntu oneiric (11.10): > > wget http://launchpadlibrarian.net/70857487/ipython_0.10.2-1_all.deb > sudo dpkg -i ipython_0.10.2-1_all.deb > > > Vincent > -- > Vincent Favre-Nicolin > > CEA / INAC http://inac.cea.fr > Universit? Joseph Fourier http://www.ujf-grenoble.fr > > http://vincefn.net > ObjCryst & Fox : http://objcryst.sourceforge.net > _______________________________________________ > IPython-User mailing list > IPython-User at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From epatters at enthought.com Wed May 25 11:06:20 2011 From: epatters at enthought.com (Evan Patterson) Date: Wed, 25 May 2011 10:06:20 -0500 Subject: [IPython-dev] Zmq Terminal. In-Reply-To: References: Message-ID: On May 25, 2011, at 5:29 AM, Thomas Kluyver wrote: > 2011/5/25 Omar Andr?s Zapata Mesa > but it produce an error in the ipkernel module > > I think the problem is that both the frontend and the kernel get to see the Ctrl-C, so there's a KeyboardInterrupt raised in each. I don't know much about how processes fit together, but a bit of fiddling suggests that setting independent to True when you start the kernel (add it to the kwargs dict on line 242) seems to work. I think that might have other effects, though - reading the code, it looks like the kernel will live on if the frontend crashes, which may not be what you want. Right, if you set 'independent' the kernel process will survive if the frontend process terminates, which usually isn't what you want. Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed May 25 11:54:08 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 25 May 2011 16:54:08 +0100 Subject: [IPython-dev] Zmq Terminal. In-Reply-To: References: Message-ID: 2011/5/25 Evan Patterson > Right, if you set 'independent' the kernel process will survive if the > frontend process terminates, which usually isn't what you want. I thought as much. I *think* the critical bit is calling os.setsid() as we launch the kernel. If we do that as well as specifying --parent, will ParentPollerUnix still be able to kill the kernel when the frontend dies? A quick test suggests that it does, but I don't know much about these things. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From epatters at enthought.com Wed May 25 12:20:45 2011 From: epatters at enthought.com (Evan Patterson) Date: Wed, 25 May 2011 11:20:45 -0500 Subject: [IPython-dev] Zmq Terminal. In-Reply-To: References: Message-ID: On May 25, 2011, at 10:54 AM, Thomas Kluyver wrote: > 2011/5/25 Evan Patterson > Right, if you set 'independent' the kernel process will survive if the frontend process terminates, which usually isn't what you want. > > I thought as much. I *think* the critical bit is calling os.setsid() as we launch the kernel. If we do that as well as specifying --parent, will ParentPollerUnix still be able to kill the kernel when the frontend dies? A quick test suggests that it does, but I don't know much about these things. I think that ParentPollerUnix would still be able to kill the kernel reliably, but I'm hardly an expert on the subtleties of Unix process management. I am wondering why this should be necessary at all, though. Shouldn't the kernel process always trap Ctrl+C signals at some point to prevent them from hosing the kernel? I've actually had a few problems with this myself. Sometimes when I interrupt the kernel from the Qt console, the whole kernel goes down. Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed May 25 12:47:25 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 25 May 2011 17:47:25 +0100 Subject: [IPython-dev] Zmq Terminal. In-Reply-To: References: Message-ID: On 25 May 2011 17:20, Evan Patterson wrote: > I am wondering why this should be necessary at all, though. Shouldn't the > kernel process always trap Ctrl+C signals at some point to prevent them from > hosing the kernel? I've actually had a few problems with this myself. > Sometimes when I interrupt the kernel from the Qt console, the whole kernel > goes down. It's when the frontend is accepting input - so Ctrl-C should just give you a clean prompt, without sending anything to the kernel. We could catch the exception in the kernel and ignore, but it doesn't feel like the right way to do it. I'd prefer to arrange it so that the signal from the user hitting Ctrl-C doesn't reach the kernel at all. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From pivanov314 at gmail.com Wed May 25 13:02:29 2011 From: pivanov314 at gmail.com (Paul Ivanov) Date: Wed, 25 May 2011 10:02:29 -0700 Subject: [IPython-dev] [IPython-User] ipython (0.10.1) -wthread and -pylab bug in ubuntu natty In-Reply-To: References: <201105251553.22500.vincefn@users.sourceforge.net> Message-ID: <20110525170229.GM2795@ykcyc> Hi Thomas, Thomas Kluyver, on 2011-05-25 15:45, wrote: > That reminds me - if I select the GTK backend in matplotlibrc, I can > replicate the blocking behaviour in trunk as well, with: > > x = randn(1000) > hist(x) # No window appears > show() # Blocks even after closing the window > > Is this some quirk of my setup, or can anyone else replicate that? This is > matplotlib 0.99.3. That's a pretty old version of matplotlib for someone running IPython trunk ;) It works for me on ~month old mpl trunk, and ipython trunk. I remember noting this kind of behavior when testing and fixing the GTK blocking behavior that ended up in the 0.10.2 release. IIRC, it occurred when you would try to use one backend for matplotlib, but another is implied by the parameters IPython was invoked with. In a comment on ipython/issues/185, I wrote ...users will see this issue if they use e.g "ipython -gthread" but their matplotlib is set to some other backend by default (say "WX"). > On 25 May 2011 14:53, Vincent Favre-Nicolin > wrote: > > If you are using ipython combined with '-wthread' (for mayavi's mlab) or > > '-pylab' (for matplotlib), there is currently a bug in Ubunty 11.04 (natty) > > with ipython 0.10.1, which prevents the correct thread handling: > > e.g. with mayavi's mlab, it is necessary to use 'show()' to interact with > > the window, which is not very practical (and is blocking). And for > > '-pylab', > > the GTK and wx (Aggg) backend do not work... In the example Vincent provided above, if you invoked ipython 0.10.2 with -wthread, but have matplotlib use GTK, I believe you will still see the blocking issue. (at least for a similar vintage matplotlib, it might not be an issue for ~month old mpl trunk, at least I couldn't replicate it there briefly just now). best, -- Paul Ivanov 314 address only used for lists, off-list direct email at: http://pirsquared.org | GPG/PGP key id: 0x0F3E28F7 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: From epatters at enthought.com Wed May 25 13:07:01 2011 From: epatters at enthought.com (Evan Patterson) Date: Wed, 25 May 2011 12:07:01 -0500 Subject: [IPython-dev] Zmq Terminal. In-Reply-To: References: Message-ID: On May 25, 2011, at 11:47 AM, Thomas Kluyver wrote: > On 25 May 2011 17:20, Evan Patterson wrote: > I am wondering why this should be necessary at all, though. Shouldn't the kernel process always trap Ctrl+C signals at some point to prevent them from hosing the kernel? I've actually had a few problems with this myself. Sometimes when I interrupt the kernel from the Qt console, the whole kernel goes down. > > It's when the frontend is accepting input - so Ctrl-C should just give you a clean prompt, without sending anything to the kernel. We could catch the exception in the kernel and ignore, but it doesn't feel like the right way to do it. I'd prefer to arrange it so that the signal from the user hitting Ctrl-C doesn't reach the kernel at all. The way interrupts work in the Qt console (in Unix) is precisely that, though: the frontend process sends a SIGINT signal to the kernel. The expectation is that the kernel will handle the signal in a reasonable way. We could try to devise some other scheme, like I did for Windows, but it would be difficult to get right. The kernel should just do the right thing. Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed May 25 13:30:43 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 25 May 2011 18:30:43 +0100 Subject: [IPython-dev] [IPython-User] ipython (0.10.1) -wthread and -pylab bug in ubuntu natty In-Reply-To: <20110525170229.GM2795@ykcyc> References: <201105251553.22500.vincefn@users.sourceforge.net> <20110525170229.GM2795@ykcyc> Message-ID: On 25 May 2011 18:02, Paul Ivanov wrote: > That's a pretty old version of matplotlib for someone running > IPython trunk ;) > I'm not a heavy matplotlib user, in fact ;). I tried easy_installing a newer version in the virtualenv, but it wasn't compiling for some reason. I'll try it again. > > It works for me on ~month old mpl trunk, and ipython trunk. > > I remember noting this kind of behavior when testing and fixing > the GTK blocking behavior that ended up in the 0.10.2 release. > IIRC, it occurred when you would try to use one backend for > matplotlib, but another is implied by the parameters IPython was > invoked with. In a comment on ipython/issues/185, I wrote > > ...users will see this issue if they use e.g "ipython -gthread" > but their matplotlib is set to some other backend by default > (say "WX"). > This was just with "ipython -pylab", so I assume it should be able to sort out getting the correct backend. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From epatters at enthought.com Wed May 25 14:05:34 2011 From: epatters at enthought.com (Evan Patterson) Date: Wed, 25 May 2011 13:05:34 -0500 Subject: [IPython-dev] Zmq Terminal. In-Reply-To: References: Message-ID: <558D4A18-DADA-4B78-A52D-5AC0608D36A1@enthought.com> On May 25, 2011, at 12:24 PM, Thomas Kluyver wrote: > On 25 May 2011 18:07, Evan Patterson wrote: > The way interrupts work in the Qt console (in Unix) is precisely that, though: the frontend process sends a SIGINT signal to the kernel. The expectation is that the kernel will handle the signal in a reasonable way. We could try to devise some other scheme, like I did for Windows, but it would be difficult to get right. The kernel should just do the right thing. > > That only applies while something is being executed, though (looking at the code, it checks self._executing). We can do the equivalent and call interrupt_kernel from the terminal frontend. The problem is that the kernel gets the SIGINT straight from the user, wherever it is in the code. In particular, if the user presses Ctrl-C at an input prompt, that shows up while the kernel is idling. We could catch it, but I feel that the right way to do it is to insulate the kernel from the user's keystrokes. But this is not robust. Frontends use SIGINT signals to tell to the kernel that it should it interrupt its execution. There is no guarantee that they will be polite about it, i.e. only send SIGINT when the kernel is actually executing. In fact, in the general case (where multiple frontends are connected to the same kernel), there is no way to even know whether the kernel is executing. I think it is good principle to ensure that the kernel is as robust as possible to misbehaving clients. This means that it should be prepared to handle SIGINT at any time. Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed May 25 14:06:57 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 25 May 2011 19:06:57 +0100 Subject: [IPython-dev] [IPython-User] ipython (0.10.1) -wthread and -pylab bug in ubuntu natty In-Reply-To: References: <201105251553.22500.vincefn@users.sourceforge.net> <20110525170229.GM2795@ykcyc> Message-ID: On 25 May 2011 18:30, Thomas Kluyver wrote: > I'm not a heavy matplotlib user, in fact ;). I tried easy_installing a > newer version in the virtualenv, but it wasn't compiling for some reason. > I'll try it again. OK, I've got matplotlib 1.0.1 installed. Now if I specify backend : GTK, no window appears, even when I call show(). It works as expected with GTKAgg, but not with GTK. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From epatters at enthought.com Wed May 25 14:20:56 2011 From: epatters at enthought.com (Evan Patterson) Date: Wed, 25 May 2011 13:20:56 -0500 Subject: [IPython-dev] Zmq Terminal. In-Reply-To: References: <558D4A18-DADA-4B78-A52D-5AC0608D36A1@enthought.com> Message-ID: <127B4FC6-B3C2-4A06-B5C2-FDBD03599BFF@enthought.com> On May 25, 2011, at 1:12 PM, Thomas Kluyver wrote: > On 25 May 2011 19:05, Evan Patterson wrote: > I think it is good principle to ensure that the kernel is as robust as possible to misbehaving clients. This means that it should be prepared to handle SIGINT at any time. > > That makes sense, although I still think that we should also insulate the kernel from the user when the terminal contains the frontend too. Now that I can get behind. The kernel shouldn't be exposed to random junk from the frontend terminal. os.setpid() may be the solution for that, but it's probably worth doing a little more investigation. > > Shall I add a try/except KeyboardInterrupt inside the loop in IPython.zmq.ipkernel.Kernel.start()? Yep, I think that's where it makes sense to have it. Thanks, Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From pivanov314 at gmail.com Wed May 25 14:41:28 2011 From: pivanov314 at gmail.com (Paul Ivanov) Date: Wed, 25 May 2011 11:41:28 -0700 Subject: [IPython-dev] [IPython-User] ipython (0.10.1) -wthread and -pylab bug in ubuntu natty In-Reply-To: References: <201105251553.22500.vincefn@users.sourceforge.net> <20110525170229.GM2795@ykcyc> Message-ID: <20110525184128.GO2795@ykcyc> Thomas Kluyver, on 2011-05-25 19:06, wrote: > On 25 May 2011 18:30, Thomas Kluyver wrote: > > > I'm not a heavy matplotlib user, in fact ;). I tried easy_installing a > > newer version in the virtualenv, but it wasn't compiling for some reason. > > I'll try it again. > > > OK, I've got matplotlib 1.0.1 installed. Now if I specify backend : GTK, no > window appears, even when I call show(). It works as expected with GTKAgg, > but not with GTK. odd. I get the same with -pylab, but not without it. Actually -- plt.figure() does make a window come up, but it doesn't render anything on there. With plain GTK backend, I get a window after show (I started with just -gthread), but the call to show blocks. Hitting ctrl-c after that returns me to the ipython prompt, and the figure appears to still update when I resize it, so I'm not sure why the call to show blocked, given that the threading seems to work. Actually, the above behavior is what I see even if I just launch ipython without -gthread -- it even works in just vanilla python shell, so it must be using the internal matplotlib threading code. -- Paul Ivanov 314 address only used for lists, off-list direct email at: http://pirsquared.org | GPG/PGP key id: 0x0F3E28F7 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: From takowl at gmail.com Wed May 25 14:42:02 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 25 May 2011 19:42:02 +0100 Subject: [IPython-dev] Zmq Terminal. In-Reply-To: <127B4FC6-B3C2-4A06-B5C2-FDBD03599BFF@enthought.com> References: <558D4A18-DADA-4B78-A52D-5AC0608D36A1@enthought.com> <127B4FC6-B3C2-4A06-B5C2-FDBD03599BFF@enthought.com> Message-ID: On 25 May 2011 19:20, Evan Patterson wrote: > Shall I add a try/except KeyboardInterrupt inside the loop in > IPython.zmq.ipkernel.Kernel.start()? > > > Yep, I think that's where it makes sense to have it. > Ok, great. Pull request: https://github.com/ipython/ipython/pull/472 Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed May 25 14:47:51 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 25 May 2011 19:47:51 +0100 Subject: [IPython-dev] [IPython-User] ipython (0.10.1) -wthread and -pylab bug in ubuntu natty In-Reply-To: <20110525184128.GO2795@ykcyc> References: <201105251553.22500.vincefn@users.sourceforge.net> <20110525170229.GM2795@ykcyc> <20110525184128.GO2795@ykcyc> Message-ID: On 25 May 2011 19:41, Paul Ivanov wrote: > Actually, the above behavior is what I see even if I just launch > ipython without -gthread -- it even works in just vanilla python > shell, so it must be using the internal matplotlib threading > code. > This matches what I see. Are IPython and matplotlib both trying to do clever things with threads, and tripping over each others toes? Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed May 25 15:23:00 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 25 May 2011 12:23:00 -0700 Subject: [IPython-dev] control is not returned to main program, when plotting In-Reply-To: <20110114100508.GK13983@ykcyc> References: <20110114094843.1b1361db@moon> <20110114100508.GK13983@ykcyc> Message-ID: On Fri, Jan 14, 2011 at 2:05 AM, Paul Ivanov wrote: > Hi Thomas, > > this was originally reported in issue 185, and I submitted a pull > request on Dec 28th [**cough** last year! ;) ] to fix > this, coming to the same conclusion as you, which Mark Voorhies > graciously tested and verified that it resolves the issue. > > https://github.com/ipython/ipython/pull/237 > > In the pull request message, I came to the same conclusion as > you: > > ?What isn't clear to me is the purpose of the function > ?check_gtk inside Shell.py, which was added in the problematic > ?commit. For some reason, it unconditionally returns mode > ?'tkthread' if gtk was safely imported and hasattr > ?'set_interactive', regardless of what mode was passed to it. > > Your patch is a subset of the lines to be removed by the pull > request. I additionally got rid of the entire check_gtk function > as it was no longer used anywhere. > > We just need someone with ipython privileges to put pull it into > the 0.10.2 branch. Anyone lurking out there? :) Just to note for the record that this did get merged in to 0.10.2 before release, so the problem is fixed in the released version. Thanks to those who did the actual work! Cheers, f From efiring at hawaii.edu Wed May 25 15:42:28 2011 From: efiring at hawaii.edu (Eric Firing) Date: Wed, 25 May 2011 09:42:28 -1000 Subject: [IPython-dev] [IPython-User] ipython (0.10.1) -wthread and -pylab bug in ubuntu natty In-Reply-To: References: <201105251553.22500.vincefn@users.sourceforge.net> <20110525170229.GM2795@ykcyc> <20110525184128.GO2795@ykcyc> Message-ID: <4DDD5BA4.1020804@hawaii.edu> On 05/25/2011 08:47 AM, Thomas Kluyver wrote: > On 25 May 2011 19:41, Paul Ivanov > wrote: > > Actually, the above behavior is what I see even if I just launch > ipython without -gthread -- it even works in just vanilla python > shell, so it must be using the internal matplotlib threading > code. > > > This matches what I see. Are IPython and matplotlib both trying to do > clever things with threads, and tripping over each others toes? No, mpl does not use threads. Ipython versions <= 0.10.x uses threads for gui interaction. The present development version does not, and is considered much more robust as a consequence. Eric > > Thomas From fperez.net at gmail.com Wed May 25 17:13:03 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 25 May 2011 14:13:03 -0700 Subject: [IPython-dev] UI related: TermKit (a webkit-based terminal) In-Reply-To: References: <20110524082046.GE2795@ykcyc> Message-ID: On Tue, May 24, 2011 at 3:18 AM, Thomas Kluyver wrote: > I've left a comment on his blog about what we're doing, as well. He's > certainly had similar ideas to those behind the new interactive IPython > frontends. I like the idea. Yup, I'd seen it recently too. There's a ton of similarity in the underlying approach, at the end of the day our display protocol and what he's doing are almost the same thing. His demos look pretty cool, not in that I would want an everyday shell to work like that precisely, but in highlighting what can be done with the tools. Cheers, f From fperez.net at gmail.com Wed May 25 21:24:26 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 25 May 2011 18:24:26 -0700 Subject: [IPython-dev] [IPython-User] ipython (0.10.1) -wthread and -pylab bug in ubuntu natty In-Reply-To: References: <201105251553.22500.vincefn@users.sourceforge.net> <20110525170229.GM2795@ykcyc> Message-ID: On Wed, May 25, 2011 at 11:06 AM, Thomas Kluyver wrote: > OK, I've got matplotlib 1.0.1 installed. Now if I specify backend : GTK, no > window appears, even when I call show(). It works as expected with GTKAgg, > but not with GTK. > > Thomas OK, that was just a plain bug, fixed here: https://github.com/ipython/ipython/pull/474 None of us uses plain gtk, so we'd simply missed the fact that it had never worked in 0.11 :) Cheers, f From fperez.net at gmail.com Wed May 25 23:26:39 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 25 May 2011 20:26:39 -0700 Subject: [IPython-dev] tiny patch for sphinx version change In-Reply-To: <4DD83366.6000808@hawaii.edu> References: <4DD83366.6000808@hawaii.edu> Message-ID: On Sat, May 21, 2011 at 2:49 PM, Eric Firing wrote: > I updated to ipython master. ?When attempting to build the docs I ran into a > problem associated with changes in Sphinx between version 0.6.7 and 1.x. > ?The attached patch allows the build to proceed with the latter and should > still work with the former, although I have not tested that. Thanks! Committed (with due authorship): https://github.com/ipython/ipython/commit/54f4afc320a579317694177f8ac0ef877876c333 Cheers, f From takowl at gmail.com Thu May 26 08:56:04 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Thu, 26 May 2011 13:56:04 +0100 Subject: [IPython-dev] %edit magic in new frontends Message-ID: Bringing this discussion onto the list, it started on a pull request here: https://github.com/ipython/ipython/pull/460#issuecomment-1214412 In the plain terminal frontend, calling %edit brings up an editor, typically something inside the terminal like nano or vi, and blocks until the editor closes. Then, depending on how it was called, it can execute the code that's just been edited, redefine a macro if you were editing one, or take no action. In addition, it may spit out the edited code as a string so you can reuse it. In the Qt console, calling %edit will normally open the target in a GUI editor.* Editors that can open multiple documents may not block the call, although apparently some of them have command-line flags for that behaviour. So as it stands, %edit in the Qt console will show you a file you can change and save, but won't take any further action on it. Then again, since the Qt console has decent editing capabilities itself, you can use the %recall and %loadpy macros to bring code to the next prompt, and edit it there before running it. * I say 'will' because in fact it currently does nothing by default, but that should be sorted soon. We should also consider how we will want %edit to work in other situations, including the HTML notebook which is being prepared, and IPython as an embedded widget in other software. Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu May 26 17:04:39 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 26 May 2011 14:04:39 -0700 Subject: [IPython-dev] %edit magic in new frontends In-Reply-To: References: Message-ID: On Thu, May 26, 2011 at 5:56 AM, Thomas Kluyver wrote: > In the Qt console, calling %edit will normally open the target in a GUI > editor.* Editors that can open multiple documents may not block the call, > although apparently some of them have command-line flags for that behaviour. > So as it stands, %edit in the Qt console will show you a file you can change > and save, but won't take any further action on it. Then again, since the Qt > console has decent editing capabilities itself, you can use the %recall and > %loadpy macros to bring code to the next prompt, and edit it there before > running it. It seems to me that %edit should, in the gui console, by default use the -x flag always. This way it's useful as a mechanism to quickly load the thing you want to work on, but you're still responsible for saving and doing %run yourself. That doesn't bother me too much, because I think that with the multiline capabilities of the Qt console, many of the use cases of %edit are now subsumed by simply recalling a previous cell with up-arrow and editing it right there. And once the amount of code in the cell becomes large, it's probably best to just keep a persistent editor open using the classic 'save, alt-tab, run' cycle. > We should also consider how we will want %edit to work in other situations, > including the HTML notebook which is being prepared, and IPython as an > embedded widget in other software. Applications that embed the console can always override %edit to synchronize with their own mechanisms; Evan, does that sound reasonable for your use cases? As for the html notebook, it would be really cool if edit would open the file in-browser, either using a full-tab Codemirror cell or something like ymacs (http://www.ymacs.org/). But the architectural decisions for magics with client-specific behavior are a bit tricky, since all code is actually executed in-kernel, and the same kernel can be simultaneously serving more than one client... Cheers, f From takowl at gmail.com Thu May 26 17:14:30 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Thu, 26 May 2011 22:14:30 +0100 Subject: [IPython-dev] %edit magic in new frontends In-Reply-To: References: Message-ID: On 26 May 2011 22:04, Fernando Perez wrote: > That doesn't bother me too much, because I think that with the > multiline capabilities of the Qt console, many of the use cases of > %edit are now subsumed by simply recalling a previous cell with > up-arrow and editing it right there. And once the amount of code in > the cell becomes large, it's probably best to just keep a persistent > editor open using the classic 'save, alt-tab, run' cycle. > The one thing I don't think you can easily do is editing a macro and have it redefined from the edited code. You'd have to do '%recall macro', edit, run, '%macro 52'. But I guess if you've got a better multi-line cell environment, macros are less important. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu May 26 17:40:09 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 26 May 2011 14:40:09 -0700 Subject: [IPython-dev] %edit magic in new frontends In-Reply-To: References: Message-ID: On Thu, May 26, 2011 at 2:14 PM, Thomas Kluyver wrote: > The one thing I don't think you can easily do is editing a macro and have it > redefined from the edited code. You'd have to do '%recall macro', edit, run, > '%macro 52'. But I guess if you've got a better multi-line cell environment, > macros are less important. Yes. I realize macros get trickier here, but when I first wrote the macro system, I thought of it as a poor man's solution to ease some of the pain of not having decent multiline editing. So I think for now it's not the end of the world if "%edit macro" isn't great in the Qt console, since I imagine it will be so rarely (if at all) used. And as you indicated, %recall provides pretty much that functionality in-console (since I don't expect anyone to be writing 100-line macros), so I think we're Ok there. Cheers, f From epatters at enthought.com Thu May 26 20:08:09 2011 From: epatters at enthought.com (Evan Patterson) Date: Thu, 26 May 2011 19:08:09 -0500 Subject: [IPython-dev] %edit magic in new frontends In-Reply-To: References: Message-ID: On Thu, May 26, 2011 at 4:04 PM, Fernando Perez wrote: > On Thu, May 26, 2011 at 5:56 AM, Thomas Kluyver wrote: > > In the Qt console, calling %edit will normally open the target in a GUI > > editor.* Editors that can open multiple documents may not block the call, > > although apparently some of them have command-line flags for that > behaviour. > > So as it stands, %edit in the Qt console will show you a file you can > change > > and save, but won't take any further action on it. Then again, since the > Qt > > console has decent editing capabilities itself, you can use the %recall > and > > %loadpy macros to bring code to the next prompt, and edit it there before > > running it. > > It seems to me that %edit should, in the gui console, by default use > the -x flag always. This way it's useful as a mechanism to quickly > load the thing you want to work on, but you're still responsible for > saving and doing %run yourself. > > That doesn't bother me too much, because I think that with the > multiline capabilities of the Qt console, many of the use cases of > %edit are now subsumed by simply recalling a previous cell with > up-arrow and editing it right there. And once the amount of code in > the cell becomes large, it's probably best to just keep a persistent > editor open using the classic 'save, alt-tab, run' cycle. > > > We should also consider how we will want %edit to work in other > situations, > > including the HTML notebook which is being prepared, and IPython as an > > embedded widget in other software. > > Applications that embed the console can always override %edit to > synchronize with their own mechanisms; Evan, does that sound > reasonable for your use cases? > > As for the html notebook, it would be really cool if edit would open > the file in-browser, either using a full-tab Codemirror cell or > something like ymacs (http://www.ymacs.org/). But the architectural > decisions for magics with client-specific behavior are a bit tricky, > since all code is actually executed in-kernel, and the same kernel can > be simultaneously serving more than one client... > > Yep, as long as %edit is still programmatically over-ridable, it's all good to me. BTW, the link to ymacs was neat. I had no idea there was such a thing. Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sat May 28 17:18:45 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 28 May 2011 14:18:45 -0700 Subject: [IPython-dev] ipython in a background thread In-Reply-To: <840FA957-B8F0-481D-8BC1-7B26921D90B1@yale.edu> References: <840FA957-B8F0-481D-8BC1-7B26921D90B1@yale.edu> Message-ID: Hi Zach, On Fri, Apr 29, 2011 at 10:03 AM, Zachary Pincus wrote: > I find that ipython, specifically ipython running in a good terminal (as opposed to embedded in a GUI window), provides a really great interface (e.g. in my case, a microscope and camera hardware). Given this use, it's been important to also have a GUI thread running so I can throw acquired images rapidly up onto a GL canvas or something. For a while, I've used a hacked-up pyglet runloop that would work in a background thread, and set up a simple message-passing system. > > This got quite kluged quite fast though, and it turns out this approach won't work at all with cocoa GUIs on OS X, which seem to require running on thread-0 (a pity). I know people on this list have looked and thought about these issues a lot, so I'd be curious what seems like the best approach in general? (There used to be some code in IPython for doing this with various window toolkits, but I don't see that in the 0.10.2's codebase anymore...) > > - I could run the GUI from thread-0 and try message-passing to IPython on another thread. Does this work well at all, or is it a huge kluge to get all the readline etc. features working right? > That's what we had always in the 0.10.x series, and it's horribly brittle. In master/0.11, we have compelely restructured how GUI event loop integration is handled, to work instead with PyOSInputHook and tie directly into the event loop of each toolkit. There's already support for Qt, WX and GTK (Tk works automatically because Python does the same internally). It should be possible to do something similar for OpenGL/pyglet, I think. Here are some pointers in the documentation: http://ipython.github.com/ipython-doc/dev/interactive/reference.html#gui-event-loop-support-support The implementation machinery lives here: https://github.com/ipython/ipython/blob/master/IPython/lib/inputhook.py https://github.com/ipython/ipython/blob/master/IPython/lib/inputhookwx.py https://github.com/ipython/ipython/blob/master/IPython/lib/inputhookgtk.py And these are the key parts of the code that use it: ./frontend/terminal/interactiveshell.py: 27 : from IPython.lib.inputhook import enable_gui ./frontend/terminal/ipapp.py: 39 : from IPython.lib import inputhook 513 : gui = inputhook.GUI_WX 515 : gui = inputhook.GUI_QT 517 : gui = inputhook.GUI_GTK 531 : activate = inputhook.enable_gui It would be great to have similar GL integration as we have for the others, so please let us know if any of this makes sense, and if you make any progress we'd love to integrate it. > - I could run the GUI in a separate process entirely, which would force a much cleaner API, but I'm not sure if pumping images at video-rate could be done cleanly. Maybe with shared memmapped arrays? (I've seen some recipes for this sort of thing on the numpy list lately.) > If the client is only going to consume the data read-only (or writing to it doesn't matter anymore for the sender), then you may consider our two-process architecture, used by the Qt console. It will require some more design, that's true, but you'll have something very robust and flexible in general, with possibilities for multi-client monitoring, remote monitoring, etc. The pyzmq code we use has the ability to do zero-copy sends of numpy arrays and is quite fast, so it might be possible to obtain video frame rates. Something like that would be a very cool application of this architecture, let us know if you decide to go down that road and we might be able to provide some advice. Best, f From fperez.net at gmail.com Sat May 28 21:18:19 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 28 May 2011 18:18:19 -0700 Subject: [IPython-dev] Continued pain with gui integration, Traits/Qt-specific, help welcome Message-ID: Hi folks, I directly pinged Robert and Evan on the bug, as they are the most likely to have the necessary expertise, but anyone willing to pitch in on what can be going on with Ctrl-C and Traits/Qt code, as explained here: https://github.com/ipython/ipython/issues/8#issuecomment-1255696 would do us a great favor. I have most of what's needed to close that bug in its original form, but this crash that I only noticed today while doing more interactive testing is truly nasty, since it affects --as best I can tell-- *any* Traits/Qt code. That means that as it stands today (at least on my system), ipython 0.11 is useless for using any Enthought/Traits app with the Qt backend (but OK with the WX one). Thanks for any help/info/etc. In particular, if anyone can find the examples given in the comment linked above work OK for them (meaning, Ctrl-C does *not* crash ipython for them), then details on your platform, Qt version and Enthought tools version/installation origin (EPD, distro, pythonxy, manual build, etc) would be enormously helpful. Thanks! f From fperez.net at gmail.com Sun May 29 02:41:38 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 28 May 2011 23:41:38 -0700 Subject: [IPython-dev] IPython 0.11@SciPy2011 (i.e. July) Message-ID: Hi all, [ sorry for the -user crosspost, but we're looking at the homestretch of a major release so the feedback of our more adventurous users would be vastly welcome ] I know we've talked about it before, but now it does finally need to happen. We actually need to make a release, and not of the 0.10.x codebase :) We have both a talk and a tutorial scheduled at SciPy'11 in mid-July around IPython, so we should really show up with a fresh-off-the-oven 0.11 with all the goodies we've been working on lately. The last few months have seen a massive amount of activity, and I actually think we're in pretty decent shape overall. This is a view of the issues tagged for 0.11 with critical priority, which I think is realistically all we can consider tackling: https://github.com/ipython/ipython/issues?labels=prio-critical&sort=created&direction=desc&state=open&page=1&milestone=5 Things don't look too bad, I think: #8: it has a nasty side, but I just begged for help earlier, and hopefully we'll figure out what's going on with Traits/Qt. The fact that matplotlib works perfectly with the Qt backend gives me hope there's a way out of this one. #66: doc updating, no problem. #79/175: it's a lot of work, but much progress has been made recently, so I think we'll be fine. #340: Brian, what's your thought on this one? #351/369: I think these are almost the same issue. Annoying, but doable. #378: doesn't look insurmountable. Honestly the only thing I'm really worried about is today's discovery of the disaster we have with Traits/Qt in #8, I think the rest is either not too bad or major (all the config work) but well tackled at this point. The Qt one worries me because it's a bit of a wildcard, I simply have no clue what's going on, despite spending most of the day on it, reading the Qt mailing lists, stracing processes, etc. But perhaps (hopefully) someone more knowledgeable than me in Qt matters can help us there, and it won't turn out to be such a big deal. If anyone sees any issue that you think should be promoted to 0.11/critical, please say so on this thread and we'll look into it. We can't guarantee that we'll promote them to that status (we can't delay the release forever, and there will be other releases), but we will certainly have a look. At this point, anyone who can help: - testing the git master branch in your normal usage patterns, and reporting any problems you encounter, from simple documentation confusions to things that don't work. - tackling any of the above bugs with a pull request - helping review any of our open pull requests will earn our undying gratitude. When we're a little closer to release we'll put test eggs/release candidates, but even if you aren't using git, github offers nicely updated single-click downloads as zip/tarballs without needing git on your part, at: https://github.com/ipython/ipython/archives/master Anything I've missed? Since SciPy is mid-July, I'd like to shoot for the actual release to happen early July, so people have a chance to download/install it before the conference. It will give us a chance to get useful feedback there, from which we can choose how to focus our energy at the sprint in Austin. Best, f From takowl at gmail.com Mon May 30 09:30:19 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Mon, 30 May 2011 14:30:19 +0100 Subject: [IPython-dev] Website Message-ID: Hi all, I think IPython could probably use a dedicated homepage. At the moment, we've got the wiki (http://ipython.scipy.org/moin/) and the documentation site (http://ipython.github.com/ipython-doc/), but I envisage something a bit more polished as a first port of call. It would probably just have a brief description of IPython and a set of links (downloads, documentation, development, and so on). I suggest we use github pages to host it at ipython.github.org. I'll have a shot at this, but I'm not particularly a designer, so if anyone else reckons they could do a good job of it, I'm happy to stand aside. Also, if there's any particular theme or style we try to maintain that I should incorporate, let me know. Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Mon May 30 15:46:05 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 30 May 2011 15:46:05 -0400 Subject: [IPython-dev] ipython in a background thread In-Reply-To: References: <840FA957-B8F0-481D-8BC1-7B26921D90B1@yale.edu> Message-ID: <78AE0AB6-3ADB-4993-AFFF-05E632558ADC@yale.edu> Thanks for the pointers into the code, Fernando! I saw that there were a lot of changes in 0.11, but I wasn't sure where to go to see what the new mechanics were. Most obliged. Zach On May 28, 2011, at 5:18 PM, Fernando Perez wrote: > Hi Zach, > > On Fri, Apr 29, 2011 at 10:03 AM, Zachary Pincus > wrote: >> I find that ipython, specifically ipython running in a good terminal (as opposed to embedded in a GUI window), provides a really great interface (e.g. in my case, a microscope and camera hardware). Given this use, it's been important to also have a GUI thread running so I can throw acquired images rapidly up onto a GL canvas or something. For a while, I've used a hacked-up pyglet runloop that would work in a background thread, and set up a simple message-passing system. >> >> This got quite kluged quite fast though, and it turns out this approach won't work at all with cocoa GUIs on OS X, which seem to require running on thread-0 (a pity). I know people on this list have looked and thought about these issues a lot, so I'd be curious what seems like the best approach in general? (There used to be some code in IPython for doing this with various window toolkits, but I don't see that in the 0.10.2's codebase anymore...) >> >> - I could run the GUI from thread-0 and try message-passing to IPython on another thread. Does this work well at all, or is it a huge kluge to get all the readline etc. features working right? >> > > That's what we had always in the 0.10.x series, and it's horribly > brittle. In master/0.11, we have compelely restructured how GUI event > loop integration is handled, to work instead with PyOSInputHook and > tie directly into the event loop of each toolkit. There's already > support for Qt, WX and GTK (Tk works automatically because Python does > the same internally). It should be possible to do something similar > for OpenGL/pyglet, I think. > > Here are some pointers in the documentation: > > http://ipython.github.com/ipython-doc/dev/interactive/reference.html#gui-event-loop-support-support > > The implementation machinery lives here: > > https://github.com/ipython/ipython/blob/master/IPython/lib/inputhook.py > https://github.com/ipython/ipython/blob/master/IPython/lib/inputhookwx.py > https://github.com/ipython/ipython/blob/master/IPython/lib/inputhookgtk.py > > And these are the key parts of the code that use it: > > ./frontend/terminal/interactiveshell.py: > 27 : from IPython.lib.inputhook import enable_gui > ./frontend/terminal/ipapp.py: > 39 : from IPython.lib import inputhook > 513 : gui = inputhook.GUI_WX > 515 : gui = inputhook.GUI_QT > 517 : gui = inputhook.GUI_GTK > 531 : activate = inputhook.enable_gui > > It would be great to have similar GL integration as we have for the > others, so please let us know if any of this makes sense, and if you > make any progress we'd love to integrate it. > >> - I could run the GUI in a separate process entirely, which would force a much cleaner API, but I'm not sure if pumping images at video-rate could be done cleanly. Maybe with shared memmapped arrays? (I've seen some recipes for this sort of thing on the numpy list lately.) >> > > If the client is only going to consume the data read-only (or writing > to it doesn't matter anymore for the sender), then you may consider > our two-process architecture, used by the Qt console. It will require > some more design, that's true, but you'll have something very robust > and flexible in general, with possibilities for multi-client > monitoring, remote monitoring, etc. The pyzmq code we use has the > ability to do zero-copy sends of numpy arrays and is quite fast, so it > might be possible to obtain video frame rates. > > Something like that would be a very cool application of this > architecture, let us know if you decide to go down that road and we > might be able to provide some advice. > > Best, > > f From jason-sage at creativetrax.com Tue May 31 13:42:49 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Tue, 31 May 2011 12:42:49 -0500 Subject: [IPython-dev] pyzmq authentication Message-ID: <4DE52899.90402@creativetrax.com> In the docs for ipython, there are big warnings about the insecurity of using ZMQ. Has anyone thought about adding authentication support to the pyzmq bindings, similar to what is done in the multiprocessing Connection objects? http://docs.python.org/dev/library/multiprocessing#authentication-keys (see also the examples in http://docs.python.org/dev/library/multiprocessing#multiprocessing.connection.Listener) I realize that it is not in the design scope of ZMQ itself to implement security, but the python bindings might be the right level for at least authentication. Thanks, Jason -- Jason Grout From benjaminrk at gmail.com Tue May 31 13:57:33 2011 From: benjaminrk at gmail.com (MinRK) Date: Tue, 31 May 2011 10:57:33 -0700 Subject: [IPython-dev] pyzmq authentication In-Reply-To: <4DE52899.90402@creativetrax.com> References: <4DE52899.90402@creativetrax.com> Message-ID: We did briefly have an encrypted socket, but the zeromq community rightly opposed that rather vehemently, largely because we aren't security experts, and the illusion of security provided by a poor implementation is really *less* secure than having no security at all. Our answer with IPython is that SSH provides our security. Typically the Controller listens on localhost, and the best way to connect to it from another machine is with an SSH tunnel (IPython does help create the tunnels) rather than listening on a public port. We do provide a small level of additional security by including an authentication key in all messages that is checked when receiving to determine if the sender is authorized to make a request. That said, the SSH tunnel tools (IPython.external.ssh) don't depend on anything else in IPython, so we have thought about moving them to pyzmq itself. -MinRK On Tue, May 31, 2011 at 10:42, Jason Grout wrote: > In the docs for ipython, there are big warnings about the insecurity of > using ZMQ. ?Has anyone thought about adding authentication support to > the pyzmq bindings, similar to what is done in the multiprocessing > Connection objects? > > http://docs.python.org/dev/library/multiprocessing#authentication-keys > > (see also the examples in > http://docs.python.org/dev/library/multiprocessing#multiprocessing.connection.Listener) > > I realize that it is not in the design scope of ZMQ itself to implement > security, but the python bindings might be the right level for at least > authentication. > > Thanks, > > Jason > > -- > Jason Grout > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > From jason-sage at creativetrax.com Tue May 31 14:13:23 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Tue, 31 May 2011 13:13:23 -0500 Subject: [IPython-dev] pyzmq authentication In-Reply-To: References: <4DE52899.90402@creativetrax.com> Message-ID: <4DE52FC3.1080408@creativetrax.com> On 5/31/11 12:57 PM, MinRK wrote: > We did briefly have an encrypted socket, but the zeromq community > rightly opposed that rather vehemently, largely because we aren't > security experts, and the illusion of security provided by a poor > implementation is really *less* secure than having no security at all. > > Our answer with IPython is that SSH provides our security. Typically > the Controller listens on localhost, and the best way to connect to it > from another machine is with an SSH tunnel (IPython does help create > the tunnels) rather than listening on a public port. We do provide a > small level of additional security by including an authentication key > in all messages that is checked when receiving to determine if the > sender is authorized to make a request. If I understand things correctly, if I have several frontends running code on a single backend server (with multiple kernels---the sage notebook is my usecase), then untrusted code from any of the kernels could connect to and mess with the other sessions, right? Is it correct to say that any user could connect with any kernel running on the same server? Thanks, Jason From benjaminrk at gmail.com Tue May 31 14:45:33 2011 From: benjaminrk at gmail.com (MinRK) Date: Tue, 31 May 2011 11:45:33 -0700 Subject: [IPython-dev] pyzmq authentication In-Reply-To: <4DE52FC3.1080408@creativetrax.com> References: <4DE52899.90402@creativetrax.com> <4DE52FC3.1080408@creativetrax.com> Message-ID: On Tue, May 31, 2011 at 11:13, Jason Grout wrote: > On 5/31/11 12:57 PM, MinRK wrote: >> >> We did briefly have an encrypted socket, but the zeromq community >> rightly opposed that rather vehemently, largely because we aren't >> security experts, and the illusion of security provided by a poor >> implementation is really *less* secure than having no security at all. >> >> Our answer with IPython is that SSH provides our security. ?Typically >> the Controller listens on localhost, and the best way to connect to it >> from another machine is with an SSH tunnel (IPython does help create >> the tunnels) rather than listening on a public port. ?We do provide a >> small level of additional security by including an authentication key >> in all messages that is checked when receiving to determine if the >> sender is authorized to make a request. > > If I understand things correctly, if I have several frontends running code > on a single backend server (with multiple kernels---the sage notebook is my > usecase), then untrusted code from any of the kernels could connect to and > mess with the other sessions, right? ?Is it correct to say that any user > could connect with any kernel running on the same server? Oh, you are talking about the *non* parallel kernel. Yes, that code has exactly zero security - anyone with access to the sockets can execute arbitrary code. We really do need to replace IPython.zmq.session with the one in the parallel code which does include simple key checking, which should be per-kernel (or per-cluster in the parallel code). -MinRK > > Thanks, > > Jason > From takowl at gmail.com Tue May 31 16:37:25 2011 From: takowl at gmail.com (Thomas Kluyver) Date: Tue, 31 May 2011 21:37:25 +0100 Subject: [IPython-dev] Website In-Reply-To: References: Message-ID: On 30 May 2011 14:30, Thomas Kluyver wrote: > I think IPython could probably use a dedicated homepage. At the moment, > we've got the wiki (http://ipython.scipy.org/moin/) and the documentation > site (http://ipython.github.com/ipython-doc/), but I envisage something a > bit more polished as a first port of call. It would probably just have a > brief description of IPython and a set of links (downloads, documentation, > development, and so on). I suggest we use github pages to host it at > ipython.github.org. > > I'll have a shot at this, but I'm not particularly a designer, so if anyone > else reckons they could do a good job of it, I'm happy to stand aside. Also, > if there's any particular theme or style we try to maintain that I should > incorporate, let me know. > This (attached) is what I have at present. The main focus is on the navigation box, which is supposed to be an easy route in to the main IPython resources. One thing I'm not sure about is whether to have a 'more links' section at the bottom, for things like the talks & presentations section, or the list of projects using IPython. Do we want to promote these from the homepage, or is a link from the docs front page sufficient? If we do want to promote them, I think we should make an effort to keep them up to date. Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ipython-website.zip Type: application/zip Size: 246254 bytes Desc: not available URL: From ellisonbg at gmail.com Tue May 31 23:07:15 2011 From: ellisonbg at gmail.com (Brian Granger) Date: Tue, 31 May 2011 20:07:15 -0700 Subject: [IPython-dev] pyzmq authentication In-Reply-To: References: <4DE52899.90402@creativetrax.com> <4DE52FC3.1080408@creativetrax.com> Message-ID: Jason, The other thing to consider is that the main frontend that will need security is the html notebook. For that, we do plan on having HTTPS enabled by default. Cheers, Brian On Tue, May 31, 2011 at 11:45 AM, MinRK wrote: > On Tue, May 31, 2011 at 11:13, Jason Grout wrote: >> On 5/31/11 12:57 PM, MinRK wrote: >>> >>> We did briefly have an encrypted socket, but the zeromq community >>> rightly opposed that rather vehemently, largely because we aren't >>> security experts, and the illusion of security provided by a poor >>> implementation is really *less* secure than having no security at all. >>> >>> Our answer with IPython is that SSH provides our security. ?Typically >>> the Controller listens on localhost, and the best way to connect to it >>> from another machine is with an SSH tunnel (IPython does help create >>> the tunnels) rather than listening on a public port. ?We do provide a >>> small level of additional security by including an authentication key >>> in all messages that is checked when receiving to determine if the >>> sender is authorized to make a request. >> >> If I understand things correctly, if I have several frontends running code >> on a single backend server (with multiple kernels---the sage notebook is my >> usecase), then untrusted code from any of the kernels could connect to and >> mess with the other sessions, right? ?Is it correct to say that any user >> could connect with any kernel running on the same server? > > Oh, you are talking about the *non* parallel kernel. ?Yes, that code > has exactly zero security - anyone with access to the sockets can > execute arbitrary code. ?We really do need to replace > IPython.zmq.session with the one in the parallel code which does > include simple key checking, which should be per-kernel (or > per-cluster in the parallel code). > > -MinRK > >> >> Thanks, >> >> Jason >> > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Brian E. Granger Cal Poly State University, San Luis Obispo bgranger at calpoly.edu and ellisonbg at gmail.com From ellisonbg at gmail.com Tue May 31 23:33:01 2011 From: ellisonbg at gmail.com (Brian Granger) Date: Tue, 31 May 2011 20:33:01 -0700 Subject: [IPython-dev] Website In-Reply-To: References: Message-ID: Thomas, Yes, the website definitely needs a refresh. Here are somethings we have talked about the following in the past: * Moving away from ipython.scipy.org to simply ipython.org * Using sphinx for the entire site as matplotlib does. * Moving the wiki stuff over to github. Cheers, Brian On Mon, May 30, 2011 at 6:30 AM, Thomas Kluyver wrote: > Hi all, > > I think IPython could probably use a dedicated homepage. At the moment, > we've got the wiki (http://ipython.scipy.org/moin/) and the documentation > site (http://ipython.github.com/ipython-doc/), but I envisage something a > bit more polished as a first port of call. It would probably just have a > brief description of IPython and a set of links (downloads, documentation, > development, and so on). I suggest we use github pages to host it at > ipython.github.org. > > I'll have a shot at this, but I'm not particularly a designer, so if anyone > else reckons they could do a good job of it, I'm happy to stand aside. Also, > if there's any particular theme or style we try to maintain that I should > incorporate, let me know. > > Thanks, > Thomas > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -- Brian E. Granger Cal Poly State University, San Luis Obispo bgranger at calpoly.edu and ellisonbg at gmail.com From ellisonbg at gmail.com Tue May 31 23:37:57 2011 From: ellisonbg at gmail.com (Brian Granger) Date: Tue, 31 May 2011 20:37:57 -0700 Subject: [IPython-dev] %edit magic in new frontends In-Reply-To: References: Message-ID: I apologize that I can't participate more fully in this discussion right now (last week of classes + finals). A few thoughts though: * The edit logic must be 100% frontend. Otherwise different frontends connected to the same kernel will get confused. * I vote that the -x flag be removed completely. Edit should just edit. Cheers, Brian On Thu, May 26, 2011 at 2:04 PM, Fernando Perez wrote: > On Thu, May 26, 2011 at 5:56 AM, Thomas Kluyver wrote: >> In the Qt console, calling %edit will normally open the target in a GUI >> editor.* Editors that can open multiple documents may not block the call, >> although apparently some of them have command-line flags for that behaviour. >> So as it stands, %edit in the Qt console will show you a file you can change >> and save, but won't take any further action on it. Then again, since the Qt >> console has decent editing capabilities itself, you can use the %recall and >> %loadpy macros to bring code to the next prompt, and edit it there before >> running it. > > It seems to me that %edit should, in the gui console, by default use > the -x flag always. ?This way it's useful as a mechanism to quickly > load the thing you want to work on, but you're still responsible for > saving and doing %run yourself. > > That doesn't bother me too much, because I think that with the > multiline capabilities of the Qt console, many of the use cases of > %edit are now subsumed by simply recalling a previous cell with > up-arrow and editing it right there. ?And once the amount of code in > the cell becomes large, it's probably best to just keep a persistent > editor open using the classic 'save, alt-tab, run' cycle. > >> We should also consider how we will want %edit to work in other situations, >> including the HTML notebook which is being prepared, and IPython as an >> embedded widget in other software. > > Applications that embed the console can always override %edit to > synchronize with their own mechanisms; Evan, does that sound > reasonable for your use cases? > > As for the html notebook, it would be really cool if edit would open > the file in-browser, either using a full-tab Codemirror cell or > something like ymacs (http://www.ymacs.org/). ?But the architectural > decisions for magics with client-specific behavior are a bit tricky, > since all code is actually executed in-kernel, and the same kernel can > be simultaneously serving more than one client... > > Cheers, > > f > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Brian E. Granger Cal Poly State University, San Luis Obispo bgranger at calpoly.edu and ellisonbg at gmail.com From ellisonbg at gmail.com Tue May 31 23:39:22 2011 From: ellisonbg at gmail.com (Brian Granger) Date: Tue, 31 May 2011 20:39:22 -0700 Subject: [IPython-dev] %edit magic in new frontends In-Reply-To: References: Message-ID: On Thu, May 26, 2011 at 2:04 PM, Fernando Perez wrote: > On Thu, May 26, 2011 at 5:56 AM, Thomas Kluyver wrote: >> In the Qt console, calling %edit will normally open the target in a GUI >> editor.* Editors that can open multiple documents may not block the call, >> although apparently some of them have command-line flags for that behaviour. >> So as it stands, %edit in the Qt console will show you a file you can change >> and save, but won't take any further action on it. Then again, since the Qt >> console has decent editing capabilities itself, you can use the %recall and >> %loadpy macros to bring code to the next prompt, and edit it there before >> running it. > > It seems to me that %edit should, in the gui console, by default use > the -x flag always. ?This way it's useful as a mechanism to quickly > load the thing you want to work on, but you're still responsible for > saving and doing %run yourself. +1 I would go a step further and say -x should be i) removed entirely or ii) made the default in all frontends. > That doesn't bother me too much, because I think that with the > multiline capabilities of the Qt console, many of the use cases of > %edit are now subsumed by simply recalling a previous cell with > up-arrow and editing it right there. ?And once the amount of code in > the cell becomes large, it's probably best to just keep a persistent > editor open using the classic 'save, alt-tab, run' cycle. > >> We should also consider how we will want %edit to work in other situations, >> including the HTML notebook which is being prepared, and IPython as an >> embedded widget in other software. > > Applications that embed the console can always override %edit to > synchronize with their own mechanisms; Evan, does that sound > reasonable for your use cases? > > As for the html notebook, it would be really cool if edit would open > the file in-browser, either using a full-tab Codemirror cell or > something like ymacs (http://www.ymacs.org/). ?But the architectural > decisions for magics with client-specific behavior are a bit tricky, > since all code is actually executed in-kernel, and the same kernel can > be simultaneously serving more than one client... > > Cheers, > > f > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Brian E. Granger Cal Poly State University, San Luis Obispo bgranger at calpoly.edu and ellisonbg at gmail.com