From fperez.net at gmail.com  Sun Aug  2 02:32:24 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Sat, 1 Aug 2009 23:32:24 -0700
Subject: [IPython-dev] News, refactoring and getting 0.10 out the door...
In-Reply-To: <db6b5ecc0907281150o781a7696sd2215e2cc8281983@mail.gmail.com>
References: <db6b5ecc0907281150o781a7696sd2215e2cc8281983@mail.gmail.com>
Message-ID: <db6b5ecc0908012332m26184374r8fb3857338eb0532@mail.gmail.com>

Hi all,

On Tue, Jul 28, 2009 at 11:50 AM, Fernando Perez<fperez.net at gmail.com> wrote:
> Well, now we should really try to finish up, especially so that the
> code that has been written can be merged as soon as possible for Brian
> to push the refactoring into trunk. ?Brian's refactoring is massive
> and touches everything, so initially there is going to be some
> breakage (we'll do our best to minimize it and Brian is super careful,
> but some pain will no doubt occur). ?For this reason, and to prevent
> stalling Brian's advance, I'd like to wrap up the 0.10 release work as
> soon as possible.

OK,  some updates...

I've done a lot of cleanup, and I think 0.10 is getting ready.  It
would be great if someone could have a look, I've uploaded a tarball,
eggs, rpms and a Windows installer of the release candidate:

http://ipython.scipy.org/dist/testing/

Unless a major problem is found, I think we'll be able to cut 0.10
final from this state of the code, and we can then start working on
reviewing all of Brian's work to merge it into the trunk.  We'll  put
up a big warning on the various webpages when that happens though,
since for a while it won't be a very good idea to run daily from
trunk, as the APIs will be in flux (we'll try to make sure that trunk
is always in test-passing state though).

Cheers,

f


From ondrej at certik.cz  Mon Aug  3 21:19:58 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 3 Aug 2009 19:19:58 -0600
Subject: [IPython-dev] IPython <-> PuDB integration
In-Reply-To: <200907092001.21523.lists@informa.tiker.net>
References: <200907092001.21523.lists@informa.tiker.net>
Message-ID: <85b5c3130908031819v240f5868i47f89ad08a872200@mail.gmail.com>

On Thu, Jul 9, 2009 at 6:01 PM, Andreas Kl?ckner<lists at informa.tiker.net> wrote:
> Hi all,
>
> I'm the author of PuDB [1], a full-screen, console-based visual debugger for
> Python. I've recently (0.92.6) added support for using IPython as an
> interactive shell, prompted by a user's request.
>
> [1] http://pypi.python.org/pypi/pudb
>
> Now that gave me an idea. The IPython web page mentions that IPython can
> optionally drop to Python's pdb debugger. If a user has PuDB installed, you
> could offer an option to drop into PuDB instead, resulting in an arguably
> nicer debugging experience (I'm biased, though). As of 0.92.7 (hot off the
> presses) PuDB offers the same programming interface as pdb--just replace 'pdb'
> with 'pudb'.


Nice project, only I don't like the default colors. :) Remind me old
turbo pascal.

It's easy to change them though in the pudb/theme.py.

Ondrej


From fperez.net at gmail.com  Mon Aug  3 21:52:32 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 3 Aug 2009 18:52:32 -0700
Subject: [IPython-dev] News, refactoring and getting 0.10 out the door...
In-Reply-To: <db6b5ecc0908012332m26184374r8fb3857338eb0532@mail.gmail.com>
References: <db6b5ecc0907281150o781a7696sd2215e2cc8281983@mail.gmail.com> 
	<db6b5ecc0908012332m26184374r8fb3857338eb0532@mail.gmail.com>
Message-ID: <db6b5ecc0908031852m54f99315m57c603622854b840@mail.gmail.com>

On Sat, Aug 1, 2009 at 11:32 PM, Fernando Perez<fperez.net at gmail.com> wrote:
> Unless a major problem is found, I think we'll be able to cut 0.10
> final from this state of the code, and we can then start working on
> reviewing all of Brian's work to merge it into the trunk. ?We'll ?put
> up a big warning on the various webpages when that happens though,
> since for a while it won't be a very good idea to run daily from
> trunk, as the APIs will be in flux (we'll try to make sure that trunk
> is always in test-passing state though).


Heads up: I've just pushed some small fixes to trunk so that the test
suite runs cleanly:

- with nose 0.11.x (there were problems there)
- if twisted isn't available at all

I've tested using both a fully installed Python with all the toys, and
'naked' virtualenvs that have only python + nose 0.10 and nose 0.11,
both with Python 2.5 and Python 2.6.  All combinations now pass the
test suite (though obviously without dependencies, many tests are
simply not run).

I've also run the test suite with Windows (though with all
dependencies, I don't have virtualenvs there) and it also passes for
me.

So I think that we're getting pretty close to putting 0.10 out...

Cheers,

f


From fperez.net at gmail.com  Tue Aug  4 06:56:05 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 03:56:05 -0700
Subject: [IPython-dev] Last checks on release notes for 0.10
Message-ID: <db6b5ecc0908040356g42e648f4jd211938ab67bee51@mail.gmail.com>

Hi all,

I just finished up and committed the "what's new" document for 0.10,
which is effectively the release notes.  I'll paste them below, feel
free to provide comments/feedback of any kind.

I am especially interested in hearing if I ommitted any contributor,
please let me know so I can correct that before the release is out.

Cheers,

f

###

Release 0.10
============

This release brings months of slow but steady development, and will be the last
before a major restructuring and cleanup of IPython's internals that is already
under way.  For this reason, we hope that 0.10 will be a stable and robust
release so that while users adapt to some of the API changes that will come
with the refactoring that will become IPython 0.11, they can safely use 0.10 in
all existing projects with minimal changes (if any).

IPython 0.10 is now a medium-sized project, with roughly (as reported by David
Wheeler's :command:`sloccount` utility) 40750 lines of Python code, and a diff
between 0.9.1 and this release that contains almost 28000 lines of code and
documentation.  Our documentation, in PDF format, is a 495-page long PDF
document (also available in HTML format, both generated from the same sources).

Many users and developers contributed code, features, bug reports and ideas to
this release.  Please do not hesitate in contacting us if we've failed to
acknowledge your contribution here.  In particular, for this release we have
contribution from the following people, a mix of new and regular names (in
alphabetical order by first name):

* Alexander Clausen: fix #341726.
* Brian Granger: lots of work everywhere (features, bug fixes, etc).
* Daniel Ashbrook: bug report on MemoryError during compilation, now fixed.
* Darren Dale: improvements to documentation build system, feedback, design
  ideas.
* Fernando Perez: various places.
* Ga?l Varoquaux: core code, ipythonx GUI, design discussions, etc. Lots...
* John Hunter: suggestions, bug fixes, feedback.
* Jorgen Stenarson: work on many fronts, tests, fixes, win32 support, etc.
* Laurent Dufr?chou: many improvements to ipython-wx standalone app.
* Lukasz Pankowski: prefilter, `%edit`, demo improvements.
* Matt Foster: TextMate support in `%edit`.
* Nathaniel Smith: fix #237073.
* Pauli Virtanen: fixes and improvements to extensions, documentation.
* Prabhu Ramachandran: improvements to `%timeit`.
* Robert Kern: several extensions.
* Sameer D'Costa: help on critical bug #269966.
* Stephan Peijnik: feedback on Debian compliance and many man pages.
* Steven Bethard: we are now shipping his :mod:`argparse` module.
* Tom Fetherston: many improvements to :mod:`IPython.demo` module.
* Ville Vainio: lots of work everywhere (features, bug fixes, etc).
* Vishal Vasta: ssh support in ipcluster.
* Walter Doerwald: work on the :mod:`IPython.ipipe` system.

Below we give an overview of new features, bug fixes and backwards-incompatible
changes.  For a detailed account of every change made, feel free to view the
project log with :command:`bzr log`.

New features
------------

* New `%paste` magic automatically extracts current contents of clipboard and
  pastes it directly, while correctly handling code that is indented or
  prepended with `>>>` or `...` python prompt markers.  A very useful new
  feature contributed by Robert Kern.

* IPython 'demos', created with the :mod:`IPython.demo` module, can now be
  created from files on disk or strings in memory.  Other fixes and
  improvements to the demo system, by Tom Fetherston.

* Added :func:`find_cmd` function to :mod:`IPython.platutils` module, to find
  commands in a cross-platform manner.

* Many improvements and fixes to Ga?l Varoquaux's :command:`ipythonx`, a
  WX-based lightweight IPython instance that can be easily embedded in other WX
  applications.  These improvements have made it possible to now have an
  embedded IPython in Mayavi and other tools.

* :class:`MultiengineClient` objects now have a :meth:`benchmark` method.

* The manual now includes a full set of auto-generated API documents from the
  code sources, using Sphinx and some of our own support code.  We are now
  using the `Numpy Documentation Standard`_  for all docstrings, and we have
  tried to update as many existing ones as possible to this format.

.. _Numpy Documentation Standard:
http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines#docstring-standard

* The new :mod:`IPython.Extensions.ipy_pretty` extension by Robert Kern
  provides configurable pretty-printing.

* Many improvements to the :command:`ipython-wx` standalone WX-based IPython
  application by Laurent Dufr?chou.  It can optionally run in a thread, and
  this can be toggled at runtime (allowing the loading of Matplotlib in a
  running session without ill effects).

* IPython includes a copy of Steven Bethard's argparse_ in the
  :mod:`IPython.external` package, so we can use it internally and it is also
  available to any IPython user.  By installing it in this manner, we ensure
  zero conflicts with any system-wide installation you may already have while
  minimizing external dependencies for new users.

* An improved and much more robust test suite, that runs groups of tests in
  separate subprocesses using either Nose or Twisted's :command:`trial` runner
  to ensure proper management of Twisted-using code.  The test suite degrades
  gracefully if optional dependencies are not available, so that the
  :command:`iptest` command can be run with only Nose installed and nothing
  else.  We also have more and cleaner test decorators to better select tests
  depending on runtime conditions, do setup/teardown, etc.

* The new ipcluster now has a fully working ssh mode that should work on
  Linux, Unix and OS X.  Thanks to Vishal Vatsa for implementing this!

* The wonderful TextMate editor can now be used with %edit on OS X.  Thanks
  to Matt Foster for this patch.

* The documentation regarding parallel uses of IPython, including MPI and PBS,
  has been significantly updated and improved.

* The developer guidelines in the documentation have been updated to explain
  our workflow using :command:`bzr` and Launchpad.

* Fully refactored :command:`ipcluster` command line program for starting
  IPython clusters.  This new version is a complete rewrite and 1) is fully
  cross platform (we now use Twisted's process management), 2) has much
  improved performance, 3) uses subcommands for different types of clusters, 4)
  uses argparse for parsing command line options, 5) has better support for
  starting clusters using :command:`mpirun`, 6) has experimental support for
  starting engines using PBS.  It can also reuse FURL files, by appropriately
  passing options to its subcommands.  However, this new version of ipcluster
  should be considered a technology preview.  We plan on changing the API in
  significant ways before it is final.

* The :mod:`argparse` module has been added to :mod:`IPython.external`.

* Full description of the security model added to the docs.

* cd completer: show bookmarks if no other completions are available.

* sh profile: easy way to give 'title' to prompt: assign to variable
  '_prompt_title'. It looks like this::

        [~]|1> _prompt_title = 'sudo!'
        sudo![~]|2>

* %edit: If you do '%edit pasted_block', pasted_block variable gets updated
  with new data (so repeated editing makes sense)

Bug fixes
---------

* Fix #368719, removed top-level debian/ directory to make the job of Debian
  packagers easier.

* Fix #291143 by including man pages contributed by Stephan Peijnik from the
  Debian project.

* Fix #358202, effectively a race condition, by properly synchronizing file
  creation at cluster startup time.

* `%timeit` now handles correctly functions that take a long time to execute
  even the first time, by not repeating them.

* Fix #239054, releasing of references after exiting.

* Fix #341726, thanks to Alexander Clausen.

* Fix #269966.  This long-standing and very difficult bug (which is actually a
  problem in Python itself) meant long-running sessions would inevitably grow
  in memory size, often with catastrophic consequences if users had large
  objects in their scripts.  Now, using `%run` repeatedly should not cause any
  memory leaks.  Special thanks to John Hunter and Sameer D'Costa for their
  help with this bug.

* Fix #295371, bug in `%history`.

* Improved support for py2exe.

* Fix #270856: IPython hangs with PyGTK

* Fix #270998: A magic with no docstring breaks the '%magic magic'

* fix #271684: -c startup commands screw up raw vs. native history

* Numerous bugs on Windows with the new ipcluster have been fixed.

* The ipengine and ipcontroller scripts now handle missing furl files
  more gracefully by giving better error messages.

* %rehashx: Aliases no longer contain dots. python3.0 binary
  will create alias python30. Fixes:
  #259716 "commands with dots in them don't work"

* %cpaste: %cpaste -r repeats the last pasted block.
  The block is assigned to pasted_block even if code
  raises exception.

* Bug #274067 'The code in get_home_dir is broken for py2exe' was
  fixed.

* Many other small bug fixes not listed here by number (see the bzr
log for more info).

Backwards incompatible changes
------------------------------

* `ipykit` and related files were unmaintained and have been removed.

* The :func:`IPython.genutils.doctest_reload` does not actually call
  `reload(doctest)` anymore, as this was causing many problems with the test
  suite.  It still resets `doctest.master` to None.

* While we have not deliberately broken Python 2.4 compatibility, only minor
  testing was done with Python 2.4, while 2.5 and 2.6 were fully tested.  But
  if you encounter problems with 2.4, please do report them as bugs.

* The :command:`ipcluster` now requires a mode argument; for example to start a
  cluster on the local machine with 4 engines, you must now type::

    $ ipcluster local -n 4

* The controller now has a ``-r`` flag that needs to be used if you want to
  reuse existing furl files.  Otherwise they are deleted (the default).

* Remove ipy_leo.py. You can use :command:`easy_install ipython-extension` to
  get it.  (done to decouple it from ipython release cycle)


From robert.kern at gmail.com  Tue Aug  4 11:44:08 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 04 Aug 2009 10:44:08 -0500
Subject: [IPython-dev] Last checks on release notes for 0.10
In-Reply-To: <db6b5ecc0908040356g42e648f4jd211938ab67bee51@mail.gmail.com>
References: <db6b5ecc0908040356g42e648f4jd211938ab67bee51@mail.gmail.com>
Message-ID: <h59l0a$l16$1@ger.gmane.org>

On 2009-08-04 05:56, Fernando Perez wrote:

> * IPython includes a copy of Steven Bethard's argparse_ in the
>    :mod:`IPython.external` package, so we can use it internally and it is also
>    available to any IPython user.  By installing it in this manner, we ensure
>    zero conflicts with any system-wide installation you may already have while
>    minimizing external dependencies for new users.

> * The :mod:`argparse` module has been added to :mod:`IPython.external`.

Dupe!

Also, argparse 1.0 was just released. It shouldn't break anything to upgrade 
IPython's copy. Of course, we're also not using any of the new features or bug 
fixes in IPython itself, but I would want to in my kernmagic library.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From ellisonbg.net at gmail.com  Tue Aug  4 14:14:22 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 4 Aug 2009 11:14:22 -0700
Subject: [IPython-dev] Problems with "trial IPython.kernel" on trunk
Message-ID: <6ce0ac130908041114y7bf04b55r2f98a1fd5ae61d6f@mail.gmail.com>

Hi,

I just ran the  test suite for trunk with all deps installed and I get a
huge number of errors and failures when trial runs on IPython.kernel.

This is due to a bug that I have fixed in my own module-reorg branch.
Basically, in ultratb, there are calls like this:

ipapi.get()

But, when the test suite for IPython.kernel is run using trial, no IP
instance every gets created so ipapi.get() returns None.  The fix is to do a
check for None before using the return value of ipapi.get().

What I don't understand is how other's haven't seen this - especially
Fernando in his running of the test suite over and over for the release.  I
can port my fix for this from my module-regorg branch to trunk, but I want
to first make sure that there is nothing wierd going on.

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090804/593fa789/attachment.html>

From ellisonbg.net at gmail.com  Tue Aug  4 14:14:22 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 4 Aug 2009 11:14:22 -0700
Subject: [IPython-dev] Problems with "trial IPython.kernel" on trunk
Message-ID: <6ce0ac130908041114y7bf04b55r2f98a1fd5ae61d6f@mail.gmail.com>

Hi,

I just ran the  test suite for trunk with all deps installed and I get a
huge number of errors and failures when trial runs on IPython.kernel.

This is due to a bug that I have fixed in my own module-reorg branch.
Basically, in ultratb, there are calls like this:

ipapi.get()

But, when the test suite for IPython.kernel is run using trial, no IP
instance every gets created so ipapi.get() returns None.  The fix is to do a
check for None before using the return value of ipapi.get().

What I don't understand is how other's haven't seen this - especially
Fernando in his running of the test suite over and over for the release.  I
can port my fix for this from my module-regorg branch to trunk, but I want
to first make sure that there is nothing wierd going on.

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090804/593fa789/attachment-0001.html>

From ellisonbg.net at gmail.com  Tue Aug  4 14:17:52 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 4 Aug 2009 11:17:52 -0700
Subject: [IPython-dev] Nose style skipping of tests in IPython.kernel
Message-ID: <6ce0ac130908041117s66022113uc2f535e218d36c57@mail.gmail.com>

In one of the recent commits to trunk:

http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/revision/1196

New logic was added in iptest and other places for twisted using modules to
not be tested if twisted is missing.

BUT, in three modules:

kernel.engineservice
kernel.error
kernel.newserialized

Nose specific logic was added that looks like this:

__test__ = {}

But, these tests should *never* be run with nose because of the bad
interactions between nose and twisteds reactor.  Why was this logic added to
these modules?

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090804/8fabc18c/attachment.html>

From ellisonbg.net at gmail.com  Tue Aug  4 14:17:52 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 4 Aug 2009 11:17:52 -0700
Subject: [IPython-dev] Nose style skipping of tests in IPython.kernel
Message-ID: <6ce0ac130908041117s66022113uc2f535e218d36c57@mail.gmail.com>

In one of the recent commits to trunk:

http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/revision/1196

New logic was added in iptest and other places for twisted using modules to
not be tested if twisted is missing.

BUT, in three modules:

kernel.engineservice
kernel.error
kernel.newserialized

Nose specific logic was added that looks like this:

__test__ = {}

But, these tests should *never* be run with nose because of the bad
interactions between nose and twisteds reactor.  Why was this logic added to
these modules?

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090804/8fabc18c/attachment-0001.html>

From fperez.net at gmail.com  Tue Aug  4 14:59:35 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 11:59:35 -0700
Subject: [IPython-dev] Last checks on release notes for 0.10
In-Reply-To: <h59l0a$l16$1@ger.gmane.org>
References: <db6b5ecc0908040356g42e648f4jd211938ab67bee51@mail.gmail.com> 
	<h59l0a$l16$1@ger.gmane.org>
Message-ID: <db6b5ecc0908041159k5839170fn2a5a28ea2e19aaed@mail.gmail.com>

On Tue, Aug 4, 2009 at 8:44 AM, Robert Kern<robert.kern at gmail.com> wrote:
> On 2009-08-04 05:56, Fernando Perez wrote:
>
>> * IPython includes a copy of Steven Bethard's argparse_ in the
>> ? ?:mod:`IPython.external` package, so we can use it internally and it is also
>> ? ?available to any IPython user. ?By installing it in this manner, we ensure
>> ? ?zero conflicts with any system-wide installation you may already have while
>> ? ?minimizing external dependencies for new users.
>
>> * The :mod:`argparse` module has been added to :mod:`IPython.external`.
>
> Dupe!

Thanks, fixed!

> Also, argparse 1.0 was just released. It shouldn't break anything to upgrade
> IPython's copy. Of course, we're also not using any of the new features or bug
> fixes in IPython itself, but I would want to in my kernmagic library.

Done.  We're barely making use of it now, all tests pass and the
ipcluster script which is the only one using it has no problems.  So
I'm glad we can ship 0.10 with argparse 1.0, thanks for catching that.

Cheers,

f


From fperez.net at gmail.com  Tue Aug  4 15:16:14 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 12:16:14 -0700
Subject: [IPython-dev] Problems with "trial IPython.kernel" on trunk
In-Reply-To: <6ce0ac130908041114y7bf04b55r2f98a1fd5ae61d6f@mail.gmail.com>
References: <6ce0ac130908041114y7bf04b55r2f98a1fd5ae61d6f@mail.gmail.com>
Message-ID: <db6b5ecc0908041216r15318475ma549e44a7738fde4@mail.gmail.com>

On Tue, Aug 4, 2009 at 11:14 AM, Brian Granger<ellisonbg.net at gmail.com> wrote:
> Hi,
>
> I just ran the? test suite for trunk with all deps installed and I get a
> huge number of errors and failures when trial runs on IPython.kernel.
>
> This is due to a bug that I have fixed in my own module-reorg branch.
> Basically, in ultratb, there are calls like this:
>
> ipapi.get()
>
> But, when the test suite for IPython.kernel is run using trial, no IP
> instance every gets created so ipapi.get() returns None.? The fix is to do a
> check for None before using the return value of ipapi.get().
>
> What I don't understand is how other's haven't seen this - especially
> Fernando in his running of the test suite over and over for the release.? I
> can port my fix for this from my module-regorg branch to trunk, but I want
> to first make sure that there is nothing wierd going on.

Mmh, I don't see that at all, no.  The trial part of the test suite
gives me typically ~400 tests on my linux boxes, a few skips at the
beginning, and that's it:

*****************************************************************************
IPython test group: trial
IPython.frontend.cocoa.tests.test_cocoa_frontend
  TestIPythonCocoaControler
    testControllerCompletesToken ...                                  [SKIPPED]
    testControllerExecutesCode ...                                    [SKIPPED]
    testControllerInstantiatesIEngine ...                             [SKIPPED]
    testControllerMirrorsUserNSWithValuesAsStrings ...                [SKIPPED]
    testCurrentIndent ...                                             [SKIPPED]
IPython.frontend.tests.test_asyncfrontendbase
  TestAsyncFrontendBase
    test_blockID_added_to_failure ...                                      [OK]
    test_blockID_added_to_result ...                                       [OK]

...


===============================================================================
[SKIPPED]: IPython.testing.tests.test_decorators_trial.TestDecoratorsTrial.test_linux

Skipping test: test_linux. This test does not run under Linux
-------------------------------------------------------------------------------
Ran 399 tests in 31.466s

PASSED (skips=8, successes=391)
*****************************************************************************

What do things look like for you?

Also, can you show me the commit with your fix?  I can try to apply
that same change on trunk and see what happens, but I'm quite puzzled
that you are getting errors I'm not...

Cheers,

f


From fperez.net at gmail.com  Tue Aug  4 15:16:14 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 12:16:14 -0700
Subject: [IPython-dev] Problems with "trial IPython.kernel" on trunk
In-Reply-To: <6ce0ac130908041114y7bf04b55r2f98a1fd5ae61d6f@mail.gmail.com>
References: <6ce0ac130908041114y7bf04b55r2f98a1fd5ae61d6f@mail.gmail.com>
Message-ID: <db6b5ecc0908041216r15318475ma549e44a7738fde4@mail.gmail.com>

On Tue, Aug 4, 2009 at 11:14 AM, Brian Granger<ellisonbg.net at gmail.com> wrote:
> Hi,
>
> I just ran the? test suite for trunk with all deps installed and I get a
> huge number of errors and failures when trial runs on IPython.kernel.
>
> This is due to a bug that I have fixed in my own module-reorg branch.
> Basically, in ultratb, there are calls like this:
>
> ipapi.get()
>
> But, when the test suite for IPython.kernel is run using trial, no IP
> instance every gets created so ipapi.get() returns None.? The fix is to do a
> check for None before using the return value of ipapi.get().
>
> What I don't understand is how other's haven't seen this - especially
> Fernando in his running of the test suite over and over for the release.? I
> can port my fix for this from my module-regorg branch to trunk, but I want
> to first make sure that there is nothing wierd going on.

Mmh, I don't see that at all, no.  The trial part of the test suite
gives me typically ~400 tests on my linux boxes, a few skips at the
beginning, and that's it:

*****************************************************************************
IPython test group: trial
IPython.frontend.cocoa.tests.test_cocoa_frontend
  TestIPythonCocoaControler
    testControllerCompletesToken ...                                  [SKIPPED]
    testControllerExecutesCode ...                                    [SKIPPED]
    testControllerInstantiatesIEngine ...                             [SKIPPED]
    testControllerMirrorsUserNSWithValuesAsStrings ...                [SKIPPED]
    testCurrentIndent ...                                             [SKIPPED]
IPython.frontend.tests.test_asyncfrontendbase
  TestAsyncFrontendBase
    test_blockID_added_to_failure ...                                      [OK]
    test_blockID_added_to_result ...                                       [OK]

...


===============================================================================
[SKIPPED]: IPython.testing.tests.test_decorators_trial.TestDecoratorsTrial.test_linux

Skipping test: test_linux. This test does not run under Linux
-------------------------------------------------------------------------------
Ran 399 tests in 31.466s

PASSED (skips=8, successes=391)
*****************************************************************************

What do things look like for you?

Also, can you show me the commit with your fix?  I can try to apply
that same change on trunk and see what happens, but I'm quite puzzled
that you are getting errors I'm not...

Cheers,

f


From fperez.net at gmail.com  Tue Aug  4 15:20:25 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 12:20:25 -0700
Subject: [IPython-dev] Nose style skipping of tests in IPython.kernel
In-Reply-To: <6ce0ac130908041117s66022113uc2f535e218d36c57@mail.gmail.com>
References: <6ce0ac130908041117s66022113uc2f535e218d36c57@mail.gmail.com>
Message-ID: <db6b5ecc0908041220p9d967aesf404591c2d967bd4@mail.gmail.com>

On Tue, Aug 4, 2009 at 11:17 AM, Brian Granger<ellisonbg.net at gmail.com> wrote:
> In one of the recent commits to trunk:
>
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/revision/1196
>
> New logic was added in iptest and other places for twisted using modules to
> not be tested if twisted is missing.
>
> BUT, in three modules:
>
> kernel.engineservice
> kernel.error
> kernel.newserialized
>
> Nose specific logic was added that looks like this:
>
> __test__?=?{}
>
> But, these tests should *never* be run with nose because of the bad
> interactions between nose and twisteds reactor.? Why was this logic added to
> these modules?

I had seen those before in some of the other twisted files, and in
adding the twisted-skipping logic elsewhere, I must have accidentally
added that in those files, sorry.  It is harmless, since nose will not
see those at all, but it was put in there by accident.  In fact, what
we should do is to remove that logic from *all* twisted-using modules
in your cleanup branch, since now we are sending those to trial
instead of nose.  But I didn't want to delete anything I wasn't 100%
sure about this late in the game, and instead I ended up accidentally
adding stuff :)

I suggest we:

- leave those as they are for now in 0.10, they are harmless
- remove them in your reorg branch altogether, so they don't cause
visual/mental clutter.  Twisted's trial should manage that code
anyway, not nose.

How does that sound?

f


From fperez.net at gmail.com  Tue Aug  4 15:20:25 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 12:20:25 -0700
Subject: [IPython-dev] Nose style skipping of tests in IPython.kernel
In-Reply-To: <6ce0ac130908041117s66022113uc2f535e218d36c57@mail.gmail.com>
References: <6ce0ac130908041117s66022113uc2f535e218d36c57@mail.gmail.com>
Message-ID: <db6b5ecc0908041220p9d967aesf404591c2d967bd4@mail.gmail.com>

On Tue, Aug 4, 2009 at 11:17 AM, Brian Granger<ellisonbg.net at gmail.com> wrote:
> In one of the recent commits to trunk:
>
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/revision/1196
>
> New logic was added in iptest and other places for twisted using modules to
> not be tested if twisted is missing.
>
> BUT, in three modules:
>
> kernel.engineservice
> kernel.error
> kernel.newserialized
>
> Nose specific logic was added that looks like this:
>
> __test__?=?{}
>
> But, these tests should *never* be run with nose because of the bad
> interactions between nose and twisteds reactor.? Why was this logic added to
> these modules?

I had seen those before in some of the other twisted files, and in
adding the twisted-skipping logic elsewhere, I must have accidentally
added that in those files, sorry.  It is harmless, since nose will not
see those at all, but it was put in there by accident.  In fact, what
we should do is to remove that logic from *all* twisted-using modules
in your cleanup branch, since now we are sending those to trial
instead of nose.  But I didn't want to delete anything I wasn't 100%
sure about this late in the game, and instead I ended up accidentally
adding stuff :)

I suggest we:

- leave those as they are for now in 0.10, they are harmless
- remove them in your reorg branch altogether, so they don't cause
visual/mental clutter.  Twisted's trial should manage that code
anyway, not nose.

How does that sound?

f


From ellisonbg.net at gmail.com  Tue Aug  4 15:27:32 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 4 Aug 2009 12:27:32 -0700
Subject: [IPython-dev] Nose style skipping of tests in IPython.kernel
In-Reply-To: <db6b5ecc0908041220p9d967aesf404591c2d967bd4@mail.gmail.com>
References: <6ce0ac130908041117s66022113uc2f535e218d36c57@mail.gmail.com>
	<db6b5ecc0908041220p9d967aesf404591c2d967bd4@mail.gmail.com>
Message-ID: <6ce0ac130908041227p1f08e3acw36a7abe798e1caab@mail.gmail.com>

That is completely fine.  I was just worried that you found that you
actually needed them to get the test suite to run.  That would have implied
something weird going on with the test suite.  I think we are fine then on
this one.

Brian

On Tue, Aug 4, 2009 at 12:20 PM, Fernando Perez <fperez.net at gmail.com>wrote:

> On Tue, Aug 4, 2009 at 11:17 AM, Brian Granger<ellisonbg.net at gmail.com>
> wrote:
> > In one of the recent commits to trunk:
> >
> > http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/revision/1196<http://bazaar.launchpad.net/%7Eipython-dev/ipython/trunk/revision/1196>
> >
> > New logic was added in iptest and other places for twisted using modules
> to
> > not be tested if twisted is missing.
> >
> > BUT, in three modules:
> >
> > kernel.engineservice
> > kernel.error
> > kernel.newserialized
> >
> > Nose specific logic was added that looks like this:
> >
> > __test__ = {}
> >
> > But, these tests should *never* be run with nose because of the bad
> > interactions between nose and twisteds reactor.  Why was this logic added
> to
> > these modules?
>
> I had seen those before in some of the other twisted files, and in
> adding the twisted-skipping logic elsewhere, I must have accidentally
> added that in those files, sorry.  It is harmless, since nose will not
> see those at all, but it was put in there by accident.  In fact, what
> we should do is to remove that logic from *all* twisted-using modules
> in your cleanup branch, since now we are sending those to trial
> instead of nose.  But I didn't want to delete anything I wasn't 100%
> sure about this late in the game, and instead I ended up accidentally
> adding stuff :)
>
> I suggest we:
>
> - leave those as they are for now in 0.10, they are harmless
> - remove them in your reorg branch altogether, so they don't cause
> visual/mental clutter.  Twisted's trial should manage that code
> anyway, not nose.
>
> How does that sound?
>
> f
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090804/5974878f/attachment.html>

From ellisonbg.net at gmail.com  Tue Aug  4 15:27:32 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 4 Aug 2009 12:27:32 -0700
Subject: [IPython-dev] Nose style skipping of tests in IPython.kernel
In-Reply-To: <db6b5ecc0908041220p9d967aesf404591c2d967bd4@mail.gmail.com>
References: <6ce0ac130908041117s66022113uc2f535e218d36c57@mail.gmail.com>
	<db6b5ecc0908041220p9d967aesf404591c2d967bd4@mail.gmail.com>
Message-ID: <6ce0ac130908041227p1f08e3acw36a7abe798e1caab@mail.gmail.com>

That is completely fine.  I was just worried that you found that you
actually needed them to get the test suite to run.  That would have implied
something weird going on with the test suite.  I think we are fine then on
this one.

Brian

On Tue, Aug 4, 2009 at 12:20 PM, Fernando Perez <fperez.net at gmail.com>wrote:

> On Tue, Aug 4, 2009 at 11:17 AM, Brian Granger<ellisonbg.net at gmail.com>
> wrote:
> > In one of the recent commits to trunk:
> >
> > http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/revision/1196<http://bazaar.launchpad.net/%7Eipython-dev/ipython/trunk/revision/1196>
> >
> > New logic was added in iptest and other places for twisted using modules
> to
> > not be tested if twisted is missing.
> >
> > BUT, in three modules:
> >
> > kernel.engineservice
> > kernel.error
> > kernel.newserialized
> >
> > Nose specific logic was added that looks like this:
> >
> > __test__ = {}
> >
> > But, these tests should *never* be run with nose because of the bad
> > interactions between nose and twisteds reactor.  Why was this logic added
> to
> > these modules?
>
> I had seen those before in some of the other twisted files, and in
> adding the twisted-skipping logic elsewhere, I must have accidentally
> added that in those files, sorry.  It is harmless, since nose will not
> see those at all, but it was put in there by accident.  In fact, what
> we should do is to remove that logic from *all* twisted-using modules
> in your cleanup branch, since now we are sending those to trial
> instead of nose.  But I didn't want to delete anything I wasn't 100%
> sure about this late in the game, and instead I ended up accidentally
> adding stuff :)
>
> I suggest we:
>
> - leave those as they are for now in 0.10, they are harmless
> - remove them in your reorg branch altogether, so they don't cause
> visual/mental clutter.  Twisted's trial should manage that code
> anyway, not nose.
>
> How does that sound?
>
> f
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090804/5974878f/attachment-0001.html>

From ellisonbg.net at gmail.com  Tue Aug  4 15:28:13 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 4 Aug 2009 12:28:13 -0700
Subject: [IPython-dev] Problems with "trial IPython.kernel" on trunk
In-Reply-To: <6ce0ac130908041225v5f8666bblab35f4cd20b624c@mail.gmail.com>
References: <6ce0ac130908041114y7bf04b55r2f98a1fd5ae61d6f@mail.gmail.com>
	<db6b5ecc0908041216r15318475ma549e44a7738fde4@mail.gmail.com>
	<6ce0ac130908041225v5f8666bblab35f4cd20b624c@mail.gmail.com>
Message-ID: <6ce0ac130908041228x34078182q151b0ba993e91b6d@mail.gmail.com>

>
> Mmh, I don't see that at all, no.  The trial part of the test suite
> gives me typically ~400 tests on my linux boxes, a few skips at the
> beginning, and that's it:
>

Hmm.  Now that is really odd.  Can you run this using:

trial IPython.kernel

and see if you get the same thing?

If your is passing, then *somehow* an IP instance is being created so that
when ipapi.get() is called, it gets that instance rather than none.  It is
possible for you to put some print statements into the places that create
the IP instances to see who is causing it to be created?

Just to make sure I am not crazy I am going to double check everything.


> What do things look like for you?
>

Probably 100 errors and failures, hangs, and reactor unclean errors.


> Also, can you show me the commit with your fix?  I can try to apply
> that same change on trunk and see what happens, but I'm quite puzzled
> that you are getting errors I'm not...
>

Here it is, but I don't think you should add this to trunk until we
understand what is going on.

http://bazaar.launchpad.net/~ipython-dev/ipython/module-reorg/revision/1225<http://bazaar.launchpad.net/%7Eipython-dev/ipython/module-reorg/revision/1225>

Cheers,

Brian



>
> Cheers,
>
> f
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090804/cb3ee2c5/attachment.html>

From ellisonbg.net at gmail.com  Tue Aug  4 15:28:13 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 4 Aug 2009 12:28:13 -0700
Subject: [IPython-dev] Problems with "trial IPython.kernel" on trunk
In-Reply-To: <6ce0ac130908041225v5f8666bblab35f4cd20b624c@mail.gmail.com>
References: <6ce0ac130908041114y7bf04b55r2f98a1fd5ae61d6f@mail.gmail.com>
	<db6b5ecc0908041216r15318475ma549e44a7738fde4@mail.gmail.com>
	<6ce0ac130908041225v5f8666bblab35f4cd20b624c@mail.gmail.com>
Message-ID: <6ce0ac130908041228x34078182q151b0ba993e91b6d@mail.gmail.com>

>
> Mmh, I don't see that at all, no.  The trial part of the test suite
> gives me typically ~400 tests on my linux boxes, a few skips at the
> beginning, and that's it:
>

Hmm.  Now that is really odd.  Can you run this using:

trial IPython.kernel

and see if you get the same thing?

If your is passing, then *somehow* an IP instance is being created so that
when ipapi.get() is called, it gets that instance rather than none.  It is
possible for you to put some print statements into the places that create
the IP instances to see who is causing it to be created?

Just to make sure I am not crazy I am going to double check everything.


> What do things look like for you?
>

Probably 100 errors and failures, hangs, and reactor unclean errors.


> Also, can you show me the commit with your fix?  I can try to apply
> that same change on trunk and see what happens, but I'm quite puzzled
> that you are getting errors I'm not...
>

Here it is, but I don't think you should add this to trunk until we
understand what is going on.

http://bazaar.launchpad.net/~ipython-dev/ipython/module-reorg/revision/1225<http://bazaar.launchpad.net/%7Eipython-dev/ipython/module-reorg/revision/1225>

Cheers,

Brian



>
> Cheers,
>
> f
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090804/cb3ee2c5/attachment-0001.html>

From fperez.net at gmail.com  Tue Aug  4 15:29:29 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 12:29:29 -0700
Subject: [IPython-dev] Nose style skipping of tests in IPython.kernel
In-Reply-To: <6ce0ac130908041227p1f08e3acw36a7abe798e1caab@mail.gmail.com>
References: <6ce0ac130908041117s66022113uc2f535e218d36c57@mail.gmail.com> 
	<db6b5ecc0908041220p9d967aesf404591c2d967bd4@mail.gmail.com> 
	<6ce0ac130908041227p1f08e3acw36a7abe798e1caab@mail.gmail.com>
Message-ID: <db6b5ecc0908041229p7d7974f7qe59592dc7beda037@mail.gmail.com>

On Tue, Aug 4, 2009 at 12:27 PM, Brian Granger<ellisonbg.net at gmail.com> wrote:
> That is completely fine.? I was just worried that you found that you
> actually needed them to get the test suite to run.? That would have implied
> something weird going on with the test suite.? I think we are fine then on
> this one.

OK, thanks.

f


From fperez.net at gmail.com  Tue Aug  4 15:29:29 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 12:29:29 -0700
Subject: [IPython-dev] Nose style skipping of tests in IPython.kernel
In-Reply-To: <6ce0ac130908041227p1f08e3acw36a7abe798e1caab@mail.gmail.com>
References: <6ce0ac130908041117s66022113uc2f535e218d36c57@mail.gmail.com> 
	<db6b5ecc0908041220p9d967aesf404591c2d967bd4@mail.gmail.com> 
	<6ce0ac130908041227p1f08e3acw36a7abe798e1caab@mail.gmail.com>
Message-ID: <db6b5ecc0908041229p7d7974f7qe59592dc7beda037@mail.gmail.com>

On Tue, Aug 4, 2009 at 12:27 PM, Brian Granger<ellisonbg.net at gmail.com> wrote:
> That is completely fine.? I was just worried that you found that you
> actually needed them to get the test suite to run.? That would have implied
> something weird going on with the test suite.? I think we are fine then on
> this one.

OK, thanks.

f


From fperez.net at gmail.com  Tue Aug  4 15:45:52 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 12:45:52 -0700
Subject: [IPython-dev] Fwd: [TIP] py.test 1.0 release / remarks
In-Reply-To: <20090804144335.GR31589@trillke.net>
References: <20090804144335.GR31589@trillke.net>
Message-ID: <db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com>

Howdy,

not that I'm advocating any changes *now*, but this might be worth
looking into.  I think nose and py.test share some ancestry, and it
seems that py.test is maturing nicely.  I especially like the
'funcargs' way of writing parametric tests without 'yield': I use
parametric tests *a lot*, but they are extremely hard to debug when
they break, because of how nose runs them.

Just a note for the scratchpad :)

Cheers,

f


---------- Forwarded message ----------
From: holger krekel <holger at merlinux.eu>
Date: Tue, Aug 4, 2009 at 7:43 AM
Subject: [TIP] py.test 1.0 release / remarks
To: Testing in Python <testing-in-python at lists.idyll.org>


Hi testing-in-python,

i am happy to announce the 1.0.0 py.test release:

? ?http://tetamap.wordpress.com/2009/08/04/pylib-1-0-0

the new features are reasonably stable now - let me summarize
a few distinctive ones maybe of interest to people
particularly interested into testing with python:

* test function arguments ("funcargs"): ?With this, python test
?functions can name arguments and one writes factory functions to
?provide instances for such arguments. ?These factory
?functions now have many interaction possibilities, for example
?they can use helper methods to efficiently manage the life cycle of
?such fixture objects across the whole testing process. ?Test function
?arguments also make test parametrization natural - one sinmply provides
?several different argument instances, no changes to the test
?function needed, no magic "yield" for generating tests anymore -
?although 1.0 still allows them and of course also allows
?traditional xUnit-style setup or even running unittest.py style tests.

* plugins: it is now easy to write plugins by implementing one
?or more of the 37 hooks which py.test calls to implement
?the testing process. There are many examples, among them
?the "oejskit" plugin which integrates testing of javascript
?code in real-life browsers into a regular test run.
?Apart from such separately distributed "cross-project" plugins
?you can also write per-project plugins/extensions that lives
?with your testing code.

* distributed testing: distributing test runs among Linux/OSX/Windows
?hosts and across python-2.4 till python-2.6 interpreters
?works reasonably stable now. ?This means that you can easily
?iron out test-module/function specific problems across a
?variety of platforms, accelerating the change & test feedback cycle.

* xfail: a new way to mark tests as "expected to fail" which
?means they run normally but are reported/counted specially.
?This "xfail" mark is meant to mark missing / wrong implementation
?rather than missing dependencies / wrong platforms for which one
?uses "skip". ?Especially for larger test suites making
?this distinction is very helpful.

* IO-capturing: output of test functions is captured per-test,
?by default including any output from sub processes.
?This works on all platforms and also (now) interacts well
?with the logging module without importing/using it itself so
?there are no interferences.

* pastebin: sends your test session output or individual test
?failures to the Pocoo pastebin service and prints out URLs.
?Convenient for seemless IRC/messaging communication.

Hope you find some of this interesting and let the issue tracker,
the mailing list or me know of any issues or comments.

cheers,
holger

--
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu

_______________________________________________
testing-in-python mailing list
testing-in-python at lists.idyll.org
http://lists.idyll.org/listinfo/testing-in-python


From fperez.net at gmail.com  Tue Aug  4 15:48:38 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 12:48:38 -0700
Subject: [IPython-dev] Problems with "trial IPython.kernel" on trunk
In-Reply-To: <db6b5ecc0908041248q725426c2l8797a82efafe2553@mail.gmail.com>
References: <6ce0ac130908041114y7bf04b55r2f98a1fd5ae61d6f@mail.gmail.com> 
	<db6b5ecc0908041216r15318475ma549e44a7738fde4@mail.gmail.com> 
	<6ce0ac130908041225v5f8666bblab35f4cd20b624c@mail.gmail.com> 
	<db6b5ecc0908041248q725426c2l8797a82efafe2553@mail.gmail.com>
Message-ID: <db6b5ecc0908041248w681e0a17s858df697e07a3f27@mail.gmail.com>

On Tue, Aug 4, 2009 at 12:48 PM, Fernando Perez<fperez.net at gmail.com> wrote:
> On Tue, Aug 4, 2009 at 12:25 PM, Brian Granger<ellisonbg.net at gmail.com> wrote:
>> Hmm.? Now that is really odd.? Can you run this using:
>>
>> trial IPython.kernel
>>
>> and see if you get the same thing?
>>
>
> OK, now I do see the errors...
>
> Mmmh, can you point me to the changeset you made?
>
> Also, is it clear to you why they were passing before? ?We are running
> trial in a *separate process*, so there's no way the parent's ipython
> instance could be leaking into the trial process.
>
> I'm a little confused right now.
>
> Cheers,
>
> f
>


From ondrej at certik.cz  Tue Aug  4 15:57:26 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 4 Aug 2009 13:57:26 -0600
Subject: [IPython-dev] Fwd: [TIP] py.test 1.0 release / remarks
In-Reply-To: <db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com>
References: <20090804144335.GR31589@trillke.net>
	<db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com>
Message-ID: <85b5c3130908041257x45154c62q291022ce0e223bcf@mail.gmail.com>

On Tue, Aug 4, 2009 at 1:45 PM, Fernando Perez<fperez.net at gmail.com> wrote:
> Howdy,
>
> not that I'm advocating any changes *now*, but this might be worth
> looking into.  I think nose and py.test share some ancestry, and it
> seems that py.test is maturing nicely.  I especially like the
> 'funcargs' way of writing parametric tests without 'yield': I use
> parametric tests *a lot*, but they are extremely hard to debug when
> they break, because of how nose runs them.
>
> Just a note for the scratchpad :)

Py.test is way better imho, if nothing, just the nice error reporting.
I will have a look if I can contribute the green [OK] results, I just
can't live without it. Otherwise I think it's pretty good.


Ondrej


From ellisonbg.net at gmail.com  Tue Aug  4 16:00:54 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 4 Aug 2009 13:00:54 -0700
Subject: [IPython-dev] Fwd: [TIP] py.test 1.0 release / remarks
In-Reply-To: <85b5c3130908041257x45154c62q291022ce0e223bcf@mail.gmail.com>
References: <20090804144335.GR31589@trillke.net>
	<db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com>
	<85b5c3130908041257x45154c62q291022ce0e223bcf@mail.gmail.com>
Message-ID: <6ce0ac130908041300s3661a46ckf37d6fc59df1522c@mail.gmail.com>

On Tue, Aug 4, 2009 at 12:57 PM, Ondrej Certik <ondrej at certik.cz> wrote:

> On Tue, Aug 4, 2009 at 1:45 PM, Fernando Perez<fperez.net at gmail.com>
> wrote:
> > Howdy,
> >
> > not that I'm advocating any changes *now*, but this might be worth
> > looking into.  I think nose and py.test share some ancestry, and it
> > seems that py.test is maturing nicely.  I especially like the
> > 'funcargs' way of writing parametric tests without 'yield': I use
> > parametric tests *a lot*, but they are extremely hard to debug when
> > they break, because of how nose runs them.
> >
> > Just a note for the scratchpad :)
>
> Py.test is way better imho, if nothing, just the nice error reporting.
> I will have a look if I can contribute the green [OK] results, I just
> can't live without it. Otherwise I think it's pretty good.
>

In what other ways is it better?  The error reporting of nose is horrible
though, that is for sure.

Cheers,

Brian



>
>
> Ondrej
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090804/df5cc1d7/attachment.html>

From gael.varoquaux at normalesup.org  Tue Aug  4 16:07:20 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 4 Aug 2009 22:07:20 +0200
Subject: [IPython-dev] Fwd: [TIP] py.test 1.0 release / remarks
In-Reply-To: <db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com>
References: <20090804144335.GR31589@trillke.net>
	<db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com>
Message-ID: <20090804200720.GG11772@phare.normalesup.org>

On Tue, Aug 04, 2009 at 12:45:52PM -0700, Fernando Perez wrote:
> not that I'm advocating any changes *now*, but this might be worth
> looking into.  I think nose and py.test share some ancestry, and it
> seems that py.test is maturing nicely.  I especially like the
> 'funcargs' way of writing parametric tests without 'yield': I use
> parametric tests *a lot*, but they are extremely hard to debug when
> they break, because of how nose runs them.

Well, with the discover module going in the standard library, I have been
wondering if we could do with only the standard library. Most of what
nose adds to test discovery (and nice assertion) has prooved to be too
much magic, and has bitten me in the past.

Right now, discover isn't much, but I am full of hope (eventhough it is
not PEP 8 compliant!!!).

Ga?l


From fperez.net at gmail.com  Tue Aug  4 16:14:22 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 13:14:22 -0700
Subject: [IPython-dev] Problems with "trial IPython.kernel" on trunk
In-Reply-To: <6ce0ac130908041257x61618c0ctb50df37e9a159807@mail.gmail.com>
References: <6ce0ac130908041114y7bf04b55r2f98a1fd5ae61d6f@mail.gmail.com> 
	<db6b5ecc0908041216r15318475ma549e44a7738fde4@mail.gmail.com> 
	<6ce0ac130908041225v5f8666bblab35f4cd20b624c@mail.gmail.com> 
	<db6b5ecc0908041248q725426c2l8797a82efafe2553@mail.gmail.com> 
	<6ce0ac130908041257x61618c0ctb50df37e9a159807@mail.gmail.com>
Message-ID: <db6b5ecc0908041314m4f4b9a2ax18447767e3f475d3@mail.gmail.com>

On Tue, Aug 4, 2009 at 12:57 PM, Brian Granger<ellisonbg.net at gmail.com> wrote:
>
>
> On Tue, Aug 4, 2009 at 12:48 PM, Fernando Perez <fperez.net at gmail.com>
> wrote:
>>
>> On Tue, Aug 4, 2009 at 12:25 PM, Brian Granger<ellisonbg.net at gmail.com>
>> wrote:
>> > Hmm.? Now that is really odd.? Can you run this using:
>> >
>> > trial IPython.kernel
>> >
>> > and see if you get the same thing?
>> >
>>
>> OK, now I do see the errors...
>
> But they really aren't there if you run the test suite using iptest?

Nope, I really get '[OK]' in those...

>> Mmmh, can you point me to the changeset you made?
>
> Here it is:
>
> http://bazaar.launchpad.net/~ipython-dev/ipython/module-reorg/revision/1225
>
> * I fixed the problem in core.ultratb
> * I removed kernel/core/ultraTB
> * Fixed the remaining import statements that were using kernel/core/ultraTB
> to use core/ultratb
>
>
>>
>> Also, is it clear to you why they were passing before? ?We are running
>> trial in a *separate process*, so there's no way the parent's ipython
>> instance could be leaking into the trial process.
>
> Do you have something weird going on with your paths and versions of
> ipython, twisted, etc.? I almost wonder if when the test suite is run using
> iptest, it is picking up a different version of trial and ipython?? Then
> they would pass.
>
>>
>> I'm a little confused right now.
>
> Me too.

OK, glad to know I'm not the only one :)  Thanks for your pointers
above, I'm on this one...  I'll ping back when I have something to
show.

f


From fperez.net at gmail.com  Tue Aug  4 19:18:38 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 16:18:38 -0700
Subject: [IPython-dev] Problems with "trial IPython.kernel" on trunk
In-Reply-To: <6ce0ac130908041257x61618c0ctb50df37e9a159807@mail.gmail.com>
References: <6ce0ac130908041114y7bf04b55r2f98a1fd5ae61d6f@mail.gmail.com> 
	<db6b5ecc0908041216r15318475ma549e44a7738fde4@mail.gmail.com> 
	<6ce0ac130908041225v5f8666bblab35f4cd20b624c@mail.gmail.com> 
	<db6b5ecc0908041248q725426c2l8797a82efafe2553@mail.gmail.com> 
	<6ce0ac130908041257x61618c0ctb50df37e9a159807@mail.gmail.com>
Message-ID: <db6b5ecc0908041618r1dab983cr4a7a4e21c3e5cece@mail.gmail.com>

On Tue, Aug 4, 2009 at 12:57 PM, Brian Granger<ellisonbg.net at gmail.com> wrote:
>
>>
>> Mmmh, can you point me to the changeset you made?
>
> Here it is:
>
> http://bazaar.launchpad.net/~ipython-dev/ipython/module-reorg/revision/1225
>
> * I fixed the problem in core.ultratb
> * I removed kernel/core/ultraTB
> * Fixed the remaining import statements that were using kernel/core/ultraTB
> to use core/ultratb
>
>
>>
>> Also, is it clear to you why they were passing before? ?We are running
>> trial in a *separate process*, so there's no way the parent's ipython
>> instance could be leaking into the trial process.
>
> Do you have something weird going on with your paths and versions of
> ipython, twisted, etc.? I almost wonder if when the test suite is run using
> iptest, it is picking up a different version of trial and ipython?? Then
> they would pass.


OK, I've understood the problem fully and fixed it basically by
applying manually your changes above, in appropriate form for the
current trunk.  I tried to do it in a way that will minimize the
chances of you getting a conflict when you merge, but forgive me if it
happens :)

The problem was that ipdoctest unconditionally starts an ipython
instance when imported.  I do this because I was never able to get it
to work right with nose if I delayed the start at all, so I punted and
did it this way.  But this means that if you run

trial IPython

the trial path search will cause this ipython instance to start up,
and later tests will find it, while if you run

trial IPython.kernel,

trial will only search below the kernel/ package and it will not
trigger ipdoctest (that lives under testing/).

Mystery solved :)

For now, applying your changes makes everything work fine, both with
the top-level suite and with any manner of running trial I could find.
 But since that unconditional ipython start is still a problem, I made
it a critical bug to me:

https://bugs.launchpad.net/ipython/+bug/409096

So this doesn't stay below the radar for long.

But I can run the test suite either from the top, or in any variation
of 'trial' that I can find, without problems.

I'm running out of available time now, and I think we've dug to the
bottom of this one pretty good, so I'm inching towards calling it
done.  I'm going to prep the release as-is, but will hold off on
sending out the email announcement in case anyone can spot any
last-second problems.

Thanks for all the help!

Cheers,

f


From fperez.net at gmail.com  Tue Aug  4 23:07:38 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 20:07:38 -0700
Subject: [IPython-dev] Fwd: [TIP] py.test 1.0 release / remarks
In-Reply-To: <20090804200720.GG11772@phare.normalesup.org>
References: <20090804144335.GR31589@trillke.net>
	<db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com> 
	<20090804200720.GG11772@phare.normalesup.org>
Message-ID: <db6b5ecc0908042007w5da06fdbp9207c2fabd07edec@mail.gmail.com>

On Tue, Aug 4, 2009 at 1:07 PM, Gael
Varoquaux<gael.varoquaux at normalesup.org> wrote:
>
> Well, with the discover module going in the standard library, I have been
> wondering if we could do with only the standard library. Most of what
> nose adds to test discovery (and nice assertion) has prooved to be too
> much magic, and has bitten me in the past.

I'd certainly *love* to ditch third-party dependencies for something
as basic as testing.  We'll have to wait still a while to see where
that goes, but I do think that flexible, robust, full testing should
be part of the stdlib, and I see the current need for third-party
tools like nose or py.test as a serious deficiency.

I recently wrote up my personal  list of  'warts' in python for
scientific computing work,  and this very problem was right there:

https://cirl.berkeley.edu/fperez/py4science/warts.html

So here's to hoping that they solve it :)

Cheers,

f


From ondrej at certik.cz  Tue Aug  4 23:19:02 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 4 Aug 2009 21:19:02 -0600
Subject: [IPython-dev] Fwd: [TIP] py.test 1.0 release / remarks
In-Reply-To: <db6b5ecc0908042007w5da06fdbp9207c2fabd07edec@mail.gmail.com>
References: <20090804144335.GR31589@trillke.net>
	<db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com>
	<20090804200720.GG11772@phare.normalesup.org>
	<db6b5ecc0908042007w5da06fdbp9207c2fabd07edec@mail.gmail.com>
Message-ID: <85b5c3130908042019v4464694fnaa3dbc1b590ae1f8@mail.gmail.com>

On Tue, Aug 4, 2009 at 9:07 PM, Fernando Perez<fperez.net at gmail.com> wrote:
> On Tue, Aug 4, 2009 at 1:07 PM, Gael
> Varoquaux<gael.varoquaux at normalesup.org> wrote:
>>
>> Well, with the discover module going in the standard library, I have been
>> wondering if we could do with only the standard library. Most of what
>> nose adds to test discovery (and nice assertion) has prooved to be too
>> much magic, and has bitten me in the past.
>
> I'd certainly *love* to ditch third-party dependencies for something
> as basic as testing. ?We'll have to wait still a while to see where
> that goes, but I do think that flexible, robust, full testing should
> be part of the stdlib, and I see the current need for third-party
> tools like nose or py.test as a serious deficiency.
>
> I recently wrote up my personal ?list of ?'warts' in python for
> scientific computing work, ?and this very problem was right there:
>
> https://cirl.berkeley.edu/fperez/py4science/warts.html
>
> So here's to hoping that they solve it :)

Nice list. This is what I like the most on your page:

Research: coming...

Haha. :) So maybe you can put in the warts list: Python is so
addictive, that one gets no real research done. :)

Ondrej


From gokhansever at gmail.com  Tue Aug  4 23:42:04 2009
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Tue, 4 Aug 2009 22:42:04 -0500
Subject: [IPython-dev] Fwd: [TIP] py.test 1.0 release / remarks
In-Reply-To: <85b5c3130908042019v4464694fnaa3dbc1b590ae1f8@mail.gmail.com>
References: <20090804144335.GR31589@trillke.net>
	<db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com>
	<20090804200720.GG11772@phare.normalesup.org>
	<db6b5ecc0908042007w5da06fdbp9207c2fabd07edec@mail.gmail.com>
	<85b5c3130908042019v4464694fnaa3dbc1b590ae1f8@mail.gmail.com>
Message-ID: <49d6b3500908042042t64dca040m945c5ffa2faa4df2@mail.gmail.com>

On Tue, Aug 4, 2009 at 10:19 PM, Ondrej Certik <ondrej at certik.cz> wrote:

> On Tue, Aug 4, 2009 at 9:07 PM, Fernando Perez<fperez.net at gmail.com>
> wrote:
> > On Tue, Aug 4, 2009 at 1:07 PM, Gael
> > Varoquaux<gael.varoquaux at normalesup.org> wrote:
> >>
> >> Well, with the discover module going in the standard library, I have
> been
> >> wondering if we could do with only the standard library. Most of what
> >> nose adds to test discovery (and nice assertion) has prooved to be too
> >> much magic, and has bitten me in the past.
> >
> > I'd certainly *love* to ditch third-party dependencies for something
> > as basic as testing.  We'll have to wait still a while to see where
> > that goes, but I do think that flexible, robust, full testing should
> > be part of the stdlib, and I see the current need for third-party
> > tools like nose or py.test as a serious deficiency.
> >
> > I recently wrote up my personal  list of  'warts' in python for
> > scientific computing work,  and this very problem was right there:
> >
> > https://cirl.berkeley.edu/fperez/py4science/warts.html
> >
> > So here's to hoping that they solve it :)
>
> Nice list. This is what I like the most on your page:
>
> Research: coming...
>
> Haha. :) So maybe you can put in the warts list: Python is so
> addictive, that one gets no real research done. :)
>
> Ondrej


That is very true Ondrej,

Right now I am thinking of writing a Python program to write my abstract due
by tomorrow :)



-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090804/5bfecab0/attachment.html>

From fperez.net at gmail.com  Wed Aug  5 00:25:31 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 21:25:31 -0700
Subject: [IPython-dev] Fwd: [TIP] py.test 1.0 release / remarks
In-Reply-To: <85b5c3130908042019v4464694fnaa3dbc1b590ae1f8@mail.gmail.com>
References: <20090804144335.GR31589@trillke.net>
	<db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com> 
	<20090804200720.GG11772@phare.normalesup.org>
	<db6b5ecc0908042007w5da06fdbp9207c2fabd07edec@mail.gmail.com> 
	<85b5c3130908042019v4464694fnaa3dbc1b590ae1f8@mail.gmail.com>
Message-ID: <db6b5ecc0908042125x76ceec59gd2c3cb81a3dcf599@mail.gmail.com>

On Tue, Aug 4, 2009 at 8:19 PM, Ondrej Certik<ondrej at certik.cz> wrote:
> Nice list. This is what I like the most on your page:
>
> Research: coming...
>
> Haha. :) So maybe you can put in the warts list: Python is so
> addictive, that one gets no real research done. :)

You're not kidding...

I need to update that site, it's pretty pathetic that I positively
hate web stuff, so  I hardly ever touch that.  One thing I want to do
is rebuild it using  sphinx, now that it's fairly clean.  And
hopefully put some actual content while I'm at it :)

Cheers,

f

ps - Ondrej, let's book some time at Scipy to look at your nb appspot
code.  Now that we're managing to clean up the internals, it may be
time to revisit those ideas...


From ondrej at certik.cz  Wed Aug  5 00:48:40 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 4 Aug 2009 22:48:40 -0600
Subject: [IPython-dev] Fwd: [TIP] py.test 1.0 release / remarks
In-Reply-To: <db6b5ecc0908042125x76ceec59gd2c3cb81a3dcf599@mail.gmail.com>
References: <20090804144335.GR31589@trillke.net>
	<db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com>
	<20090804200720.GG11772@phare.normalesup.org>
	<db6b5ecc0908042007w5da06fdbp9207c2fabd07edec@mail.gmail.com>
	<85b5c3130908042019v4464694fnaa3dbc1b590ae1f8@mail.gmail.com>
	<db6b5ecc0908042125x76ceec59gd2c3cb81a3dcf599@mail.gmail.com>
Message-ID: <85b5c3130908042148x6e464cc0j2cba328a3750e744@mail.gmail.com>

On Tue, Aug 4, 2009 at 10:25 PM, Fernando Perez<fperez.net at gmail.com> wrote:
> On Tue, Aug 4, 2009 at 8:19 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>> Nice list. This is what I like the most on your page:
>>
>> Research: coming...
>>
>> Haha. :) So maybe you can put in the warts list: Python is so
>> addictive, that one gets no real research done. :)
>
> You're not kidding...
>
> I need to update that site, it's pretty pathetic that I positively
> hate web stuff, so ?I hardly ever touch that. ?One thing I want to do
> is rebuild it using ?sphinx, now that it's fairly clean. ?And
> hopefully put some actual content while I'm at it :)
>
> Cheers,
>
> f
>
> ps - Ondrej, let's book some time at Scipy to look at your nb appspot
> code. ?Now that we're managing to clean up the internals, it may be
> time to revisit those ideas...

Sure. See also this thread:

http://groups.google.com/group/sage-devel/browse_thread/thread/442cccf3e60a2ff4

scipy 09 will be a busy conference for me. tutorial+2 lectures + lots
of other stuff with other people, like vtk/mayavi with Prabhu, traits
with ets guys, ...

Ondrej


From fperez.net at gmail.com  Wed Aug  5 01:25:07 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 22:25:07 -0700
Subject: [IPython-dev] Fwd: [TIP] py.test 1.0 release / remarks
In-Reply-To: <85b5c3130908042148x6e464cc0j2cba328a3750e744@mail.gmail.com>
References: <20090804144335.GR31589@trillke.net>
	<db6b5ecc0908041245v1e267bebpfd0282d39064944c@mail.gmail.com> 
	<20090804200720.GG11772@phare.normalesup.org>
	<db6b5ecc0908042007w5da06fdbp9207c2fabd07edec@mail.gmail.com> 
	<85b5c3130908042019v4464694fnaa3dbc1b590ae1f8@mail.gmail.com> 
	<db6b5ecc0908042125x76ceec59gd2c3cb81a3dcf599@mail.gmail.com> 
	<85b5c3130908042148x6e464cc0j2cba328a3750e744@mail.gmail.com>
Message-ID: <db6b5ecc0908042225h7f40ab1dg4770c0c732a02abc@mail.gmail.com>

On Tue, Aug 4, 2009 at 9:48 PM, Ondrej Certik<ondrej at certik.cz> wrote:
> Sure. See also this thread:
>
> http://groups.google.com/group/sage-devel/browse_thread/thread/442cccf3e60a2ff4
>
> scipy 09 will be a busy conference for me. tutorial+2 lectures + lots
> of other stuff with other people, like vtk/mayavi with Prabhu, traits
> with ets guys, ...

Get used to it, it's always busy :)

Cheers,

f


From fperez.net at gmail.com  Wed Aug  5 02:48:07 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 4 Aug 2009 23:48:07 -0700
Subject: [IPython-dev] [ANN] IPython 0.10 is out.
Message-ID: <db6b5ecc0908042348x370c50a2l23cf14e92b96b2cb@mail.gmail.com>

Hi all,

on behalf of the IPython development team, I'm happy to announce that
we've just put out IPython 0.10 final.  Many thanks to all those who
contributed ideas, bug reports and code.

You can download it from the usual location:

- http://ipython.scipy.org/moin/Download: direct links to various formats
- http://ipython.scipy.org/dist: all files are stored here.

The official documentation for this release can be found at:

- http://ipython.scipy.org/doc/rel-0.10/html: as HTML pages.
- http://ipython.scipy.org/doc/rel-0.10/ipython.pdf: as a single PDF.

In brief, this release gathers all recent work and in a sense closes a
cycle of the current useful-but-internally-messy structure of the
IPython code.  We are now well into the work of a major internal
cleanup that will inevitably change some APIs and will  likely take
some time to stabilize, so the 0.10 release should be used for a while
until the dust settles on the development branch.

The 0.10 release fixes many bugs, including some very problematic ones
(a major memory leak with repeated %run is closed), and also brings a
number of new features, stability improvements and improved
documentation.  Some highlights:

- Improved WX-based ipythonx and ipython-wx tools, suitable for
embedding  into other applications and standalone use.

- Better interactive demos with the IPython.demo module.

- Refactored ipcluster with support for local execution, MPI, PBS and
systems with SSH key access preconfigured.

- Integration with the TextMate editor in the %edit command.

The full release notes are available here with all the details:

http://ipython.scipy.org/doc/rel-0.10/html/changes.html#release-0-10

We hope you enjoy it, please report any problems as usual either on
the mailing list, or by filing a bug report at our Launchpad tracker:

https://bugs.launchpad.net/ipython

Cheers,

The IPython team.


From fperez.net at gmail.com  Wed Aug  5 04:29:21 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 5 Aug 2009 01:29:21 -0700
Subject: [IPython-dev] Cleaned up launchpad, trunk open for business!
Message-ID: <db6b5ecc0908050129n1f7301fcr3b712aa5f7817647@mail.gmail.com>

Hi folks,

after the release, I tried to clean things up a little and organize
better the situation in launchpad.  I still find some things take more
clicks than they should, but overall the site has improved a lot.  We
now have a nice overview of the code evolution visible as a timeline
with all series present:

https://launchpad.net/ipython/+series

I updated the pages for each series and retargetted a bunch of
milestones that had been lumped on trunk (I finally figured out where
to click for that to happen) into their own series, so the picture is
much clearer.

Brian has done a great job of marking all new work  with bugs, I went
through and cleaned up everything we had in progress/committed as
released.

As it stands now, Launchpad is actually getting to be pretty useful
for organizing and supporting the running of the project. So I guess
we can now focus again on trunk, reviewing the new reorganization
work, merging any small leftovers we still have that didn't make it
into 0.10, etc.

Have fun, and thanks again to all  who contribute to IPython!

f


From matthieu.brucher at gmail.com  Wed Aug  5 10:28:39 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 5 Aug 2009 16:28:39 +0200
Subject: [IPython-dev] Before a patch for LSF support
Message-ID: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>

Hi,

I'm trying to setup ipython 0.10 and test it before submitting a patch
for LSF support (which is what we are using in Pau instead of PBS).
This would be my first patch for Ipython (emotion).

So first, I'm trying to get the ipcluster work locally with:

> ipcluster local -n 4
And the log:
2009-08-05 16:09:15+0200 [-] Log opened.
2009-08-05 16:09:15+0200 [-] Process ['ipcontroller',
'--logfile=/users/brucher/.ipython/log/ipcontroller'] has started with
pid=8050
2009-08-05 16:09:15+0200 [-] Waiting for controller to finish starting...
2009-08-05 16:09:17+0200 [-] Controller started
2009-08-05 16:09:17+0200 [-] Process ['ipengine',
'--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
with pid=8051
2009-08-05 16:09:17+0200 [-] Process ['ipengine',
'--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
with pid=8052
2009-08-05 16:09:17+0200 [-] Process ['ipengine',
'--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
with pid=8053
2009-08-05 16:09:17+0200 [-] Process ['ipengine',
'--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
with pid=8054
2009-08-05 16:09:17+0200 [-] Engines started with pids: [8051, 8052, 8053, 8054]

Then, in the same location, I'm starting IPython and I try to connect
to the controller/engines but strangelly, it fails:

[6] mec = client.MultiEngineClient()
---------------------------------------------------------------------------
BananaError                               Traceback (most recent call last)

/users/brucher/<ipython console> in <module>()

/users/brucher/x86_64/lib/python2.5/site-packages/IPython/kernel/client.py
in get_multiengine_client(furl_or_file)
     67     """
     68     client = blockingCallFromThread(_client_tub.get_multiengine_client,
---> 69         furl_or_file)
     70     return client.adapt_to_blocking_client()
     71

/users/brucher/x86_64/lib/python2.5/site-packages/IPython/kernel/twistedutil.pyc
in blockingCallFromThread(f, *a, **kw)
     70         @raise: any error raised during the callback chain.
     71         """
---> 72         return
twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
     73
     74 else:

/users/brucher/x86_64/lib/python2.5/site-packages/twisted/internet/threads.pyc
in blockingCallFromThread(reactor, f, *a, **kw)
    112     result = queue.get()
    113     if isinstance(result, failure.Failure):
--> 114         result.raiseException()
    115     return result
    116

/users/brucher/x86_64/lib/python2.5/site-packages/twisted/python/failure.pyc
in raiseException(self)
    324         information if available.
    325         """
--> 326         raise self.type, self.value, self.tb
    327
    328

BananaError: BananaError: ('crypto for PB is not available, we cannot
handle encrypted PB-URLs like
pb://lhourelkbvqan3lmgrjcuhndtwiskows at 10.57.15.41:34415,127.0.0.1:34415/gjgrouve4cmkx43b65qydlkyln5xwbc6',)

I've tried without openssl with ipcluster -xy, but in this case, it
claimed it couldn't find furls in .ipython/security/..., and indeed,
there were none there or in the folder where I've started ipcluster.

Is there something I'm missing? I've tried following the guide, but no
success so far :|

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From dwf at cs.toronto.edu  Wed Aug  5 17:38:38 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 5 Aug 2009 17:38:38 -0400
Subject: [IPython-dev] Fwd: [patch] timeit unit select breaks with >=
	1000second duration
References: <0ED3A291-EC44-4536-8FC3-8CBE6AC99706@cs.toronto.edu>
Message-ID: <BCAC118A-EFF8-403C-9622-1A99074E24C9@cs.toronto.edu>

Hey guys,

I submitted this a while back, but I see there's a lot more activity  
on ipython-dev lately (congrats on 0.10, everyone!) so I'll try  
again. :)

David

Begin forwarded message:

> From: David Warde-Farley <dwf at cs.toronto.edu>
> Date: July 4, 2009 5:48:23 PM GMT-04:00
> To: ipython-dev at scipy.org
> Subject: [IPython-dev] [patch] timeit unit select breaks with >=  
> 1000second duration
> Delivery-Date: Sat, 04 Jul 2009 17:48:40 -0400
>
> This is a minor concern since I don't know how many people use  
> timeit for things that take more than a few seconds but I noticed  
> that if you have a function that runs for say, 1000 seconds, the  
> code in Magic.py fails since 'order' gets a negative number. Thus  
> you get something like 1.0e12 nanoseconds being printed, which is  
> kind of silly.
>
> Attached is a patch that fixes it in the simplest way possible: if  
> best is >= 1000 then just use seconds, e.g.
>
> In [16]: %timeit -n 1 -r 1 time.sleep(1000)
> 1 loops, best of 1: 1e+03 s per loop
>
> Regards,
>
> David
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: duration-patch.diff
Type: application/octet-stream
Size: 528 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090805/ab037c20/attachment.obj>
-------------- next part --------------
>
>
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev


From fperez.net at gmail.com  Wed Aug  5 18:10:14 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 5 Aug 2009 15:10:14 -0700
Subject: [IPython-dev] Fwd: [patch] timeit unit select breaks with >=
	1000second duration
In-Reply-To: <BCAC118A-EFF8-403C-9622-1A99074E24C9@cs.toronto.edu>
References: <0ED3A291-EC44-4536-8FC3-8CBE6AC99706@cs.toronto.edu> 
	<BCAC118A-EFF8-403C-9622-1A99074E24C9@cs.toronto.edu>
Message-ID: <db6b5ecc0908051510l2c5b1adcq189c40c0833915e8@mail.gmail.com>

Hi David,

On Wed, Aug 5, 2009 at 2:38 PM, David Warde-Farley<dwf at cs.toronto.edu> wrote:
> I submitted this a while back, but I see there's a lot more activity on
> ipython-dev lately (congrats on 0.10, everyone!) so I'll try again. :)
>

Yes, sorry we missed it, we could have easily applied this for 0.10.

At least now we won't have an excuse for forgetting:

https://bugs.launchpad.net/ipython/+bug/409566

Cheers,

f


From matthieu.brucher at gmail.com  Thu Aug  6 03:07:11 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 6 Aug 2009 09:07:11 +0200
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
Message-ID: <e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>

Hi again,

It seems that it works now, also nothing changed (perhaps a quota
issue in my home folder?).

I will try to setup the cluster stuff now.

Matthieu

2009/8/5 Matthieu Brucher <matthieu.brucher at gmail.com>:
> Hi,
>
> I'm trying to setup ipython 0.10 and test it before submitting a patch
> for LSF support (which is what we are using in Pau instead of PBS).
> This would be my first patch for Ipython (emotion).
>
> So first, I'm trying to get the ipcluster work locally with:
>
>> ipcluster local -n 4
> And the log:
> 2009-08-05 16:09:15+0200 [-] Log opened.
> 2009-08-05 16:09:15+0200 [-] Process ['ipcontroller',
> '--logfile=/users/brucher/.ipython/log/ipcontroller'] has started with
> pid=8050
> 2009-08-05 16:09:15+0200 [-] Waiting for controller to finish starting...
> 2009-08-05 16:09:17+0200 [-] Controller started
> 2009-08-05 16:09:17+0200 [-] Process ['ipengine',
> '--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
> with pid=8051
> 2009-08-05 16:09:17+0200 [-] Process ['ipengine',
> '--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
> with pid=8052
> 2009-08-05 16:09:17+0200 [-] Process ['ipengine',
> '--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
> with pid=8053
> 2009-08-05 16:09:17+0200 [-] Process ['ipengine',
> '--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
> with pid=8054
> 2009-08-05 16:09:17+0200 [-] Engines started with pids: [8051, 8052, 8053, 8054]
>
> Then, in the same location, I'm starting IPython and I try to connect
> to the controller/engines but strangelly, it fails:
>
> [6] mec = client.MultiEngineClient()
> ---------------------------------------------------------------------------
> BananaError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Traceback (most recent call last)
>
> /users/brucher/<ipython console> in <module>()
>
> /users/brucher/x86_64/lib/python2.5/site-packages/IPython/kernel/client.py
> in get_multiengine_client(furl_or_file)
> ? ? 67 ? ? """
> ? ? 68 ? ? client = blockingCallFromThread(_client_tub.get_multiengine_client,
> ---> 69 ? ? ? ? furl_or_file)
> ? ? 70 ? ? return client.adapt_to_blocking_client()
> ? ? 71
>
> /users/brucher/x86_64/lib/python2.5/site-packages/IPython/kernel/twistedutil.pyc
> in blockingCallFromThread(f, *a, **kw)
> ? ? 70 ? ? ? ? @raise: any error raised during the callback chain.
> ? ? 71 ? ? ? ? """
> ---> 72 ? ? ? ? return
> twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
> ? ? 73
> ? ? 74 else:
>
> /users/brucher/x86_64/lib/python2.5/site-packages/twisted/internet/threads.pyc
> in blockingCallFromThread(reactor, f, *a, **kw)
> ? ?112 ? ? result = queue.get()
> ? ?113 ? ? if isinstance(result, failure.Failure):
> --> 114 ? ? ? ? result.raiseException()
> ? ?115 ? ? return result
> ? ?116
>
> /users/brucher/x86_64/lib/python2.5/site-packages/twisted/python/failure.pyc
> in raiseException(self)
> ? ?324 ? ? ? ? information if available.
> ? ?325 ? ? ? ? """
> --> 326 ? ? ? ? raise self.type, self.value, self.tb
> ? ?327
> ? ?328
>
> BananaError: BananaError: ('crypto for PB is not available, we cannot
> handle encrypted PB-URLs like
> pb://lhourelkbvqan3lmgrjcuhndtwiskows at 10.57.15.41:34415,127.0.0.1:34415/gjgrouve4cmkx43b65qydlkyln5xwbc6',)
>
> I've tried without openssl with ipcluster -xy, but in this case, it
> claimed it couldn't find furls in .ipython/security/..., and indeed,
> there were none there or in the folder where I've started ipcluster.
>
> Is there something I'm missing? I've tried following the guide, but no
> success so far :|
>
> Matthieu
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From fperez.net at gmail.com  Thu Aug  6 03:10:49 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 6 Aug 2009 00:10:49 -0700
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com> 
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
Message-ID: <db6b5ecc0908060010i2c4d1c22s5d2db66c32117bc6@mail.gmail.com>

Hi Matthieu,

On Thu, Aug 6, 2009 at 12:07 AM, Matthieu
Brucher<matthieu.brucher at gmail.com> wrote:
> It seems that it works now, also nothing changed (perhaps a quota
> issue in my home folder?).

OK, keep us posted.  I'm sorry for not having replied today,  but I
honestly don't know that part of the code very well at all.

> I will try to setup the cluster stuff now.

Great, looking forward to the patch!

Take care,

f


From matthieu.brucher at gmail.com  Thu Aug  6 03:36:57 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 6 Aug 2009 09:36:57 +0200
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <db6b5ecc0908060010i2c4d1c22s5d2db66c32117bc6@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<db6b5ecc0908060010i2c4d1c22s5d2db66c32117bc6@mail.gmail.com>
Message-ID: <e76aa17f0908060036r5bdf1734we1ceda8e94ce33c4@mail.gmail.com>

2009/8/6 Fernando Perez <fperez.net at gmail.com>:
> Hi Matthieu,
>
> On Thu, Aug 6, 2009 at 12:07 AM, Matthieu
> Brucher<matthieu.brucher at gmail.com> wrote:
>> It seems that it works now, also nothing changed (perhaps a quota
>> issue in my home folder?).
>
> OK, keep us posted. ?I'm sorry for not having replied today, ?but I
> honestly don't know that part of the code very well at all.

I ran into another issue: on a cluster, the home folder may be
different than on the access box. In that case, the .ipython/security
does not exist and the engine will not start (I've just tested this).

>> I will try to setup the cluster stuff now.
>
> Great, looking forward to the patch!

Well, first I have to try to understand what is going on, as I don't
even get the log files after the engine submission !

Also, I've tried to extract the job id (it seems it is needed), but
the BatchEngineSet.parse_job_id extracts everything that is matched by
the regexp describing a job (it uses group()). I had to put "Job
<(\d+)>" as a regexp, so group() returns, for instance, "Job <1234>"
instead of "1234". I may submit a patch to get group(1) and modify the
PBS regexp accordingly.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From vivainio at gmail.com  Fri Aug  7 04:55:18 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Fri, 7 Aug 2009 11:55:18 +0300
Subject: [IPython-dev] Fwd:  ctrl-c bug using -gthread on windows,
	with 	possible solution
In-Reply-To: <4A7BE35A.2040101@tudelft.nl>
References: <4988D721.3060301@tudelft.nl> <4988D7AF.9090809@tudelft.nl>
	<498C0B82.2030404@tudelft.nl> <49941EC0.7010006@tudelft.nl>
	<46cb515a0902120554l28037c50i14d70f21f1833755@mail.gmail.com>
	<49949D16.6020100@tudelft.nl> <4A7BE35A.2040101@tudelft.nl>
Message-ID: <46cb515a0908070155u60e408eeg55b7e417f56c7575@mail.gmail.com>

Thanks Pieter, cc:ing to ipython-dev.

Someone might want to cherry-pick the following change from my
trunk-dev and put it to trunk:

http://bazaar.launchpad.net/~villemvainio/ipython/trunk-dev/revision/1160



---------- Forwarded message ----------
From: Pieter Cristiaan de Groot <p.c.degroot at tudelft.nl>
Date: Fri, Aug 7, 2009 at 11:18 AM
Subject: Re: [IPython-dev] ctrl-c bug using -gthread on windows, with
possible solution
To: "Ville M. Vainio" <vivainio at gmail.com>


Hello Ville,

it looks like the ctrl-c bugfix for gthread didn't make it to 0.10.
The error is still easily reproducable by starting ipython with
-ghread. Type sleep(10), en then ctrl-c (in windows).

The suggest change is:

before:

? ? ? update_tk(self.tk)
? ? ? self.IP.runcode()
? ? ? time.sleep(0.01)
? ? ? return True


after:

? ? ? try:
? ? ? ? ? update_tk(self.tk)
? ? ? ? ? self.IP.runcode()
? ? ? ? ? time.sleep(0.01)
? ? ? ? ? return True
? ? ? except IOError:
? ? ? ? ? return True

Best regards,

Pieter

Pieter Cristiaan de Groot wrote:
>
> Just to be sure I answer your question fully:
>
> case 1) the original 0.9.1 ipython code gives this error if I start it with -gthread and push ctrl-c
>
> ##########################
>
> CODE (in Shell.py, around linenr. 840):
>
> ? ? ? ?update_tk(self.tk)
> ? ? ? ?self.IP.runcode()
> ? ? ? ?time.sleep(0.01)
> ? ? ? ?return True
>
> ERROR:
>
> KeyboardInterrupt - Press <Enter> to continue.---------------------------------------------------------------------------
> IOError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Traceback (most recent call last)
>
> c:\python25\lib\site-packages\IPython\Shell.pyc in on_timer(self)
> ? ?840 ? ? ? ? update_tk(self.tk)
> ? ?841 ? ? ? ? self.IP.runcode()
> --> 842 ? ? ? ? time.sleep(0.01)
> ? ?843 ? ? ? ? return True
> ? ?844
>
> IOError: [Errno 4] Interrupted function call
>
> #################
>
> case 2) catching the exception (with some prints):
>
> ###############
>
> CODE:
>
> ? ? ? ?try:
> ? ? ? ? ? ?update_tk(self.tk)
> ? ? ? ? ? ?self.IP.runcode()
> ? ? ? ? ? ?time.sleep(0.01)
> ? ? ? ? ? ?return True
> ? ? ? ?except Exception, e:
> ? ? ? ? ? ?print type(e)
> ? ? ? ? ? ?print(e)
> ? ? ? ? ? ?return True
>
> ERROR:
>
> KeyboardInterrupt - Press <Enter> to continue. <type 'exceptions.IOError'>
> [Errno 4] Interrupted function call
>
> #########################
>
> case 3) if I understood correctly this is what you wanted to see:
>
> #####################
>
> CODE:
>
> ? ? ? ?update_tk(self.tk)
> ? ? ? ?self.IP.runcode()
> ? ? ? ?try:
> ? ? ? ? ? ?time.sleep(0.01)
> ? ? ? ? ? ?return True
> ? ? ? ?except Exception ,e:
> ? ? ? ? ? ?print type(e)
> ? ? ? ? ? ?print e
> ? ? ? ? ? ?return True
>
> ERROR:
>
> KeyboardInterrupt - Press <Enter> to continue. <type 'exceptions.IOError'>
> [Errno 4] Interrupted function call
>
> ##########################
>
> case 3a) occasionally (1 in ~20 times), if you interrupt the code while it is not in the sleep, then everything works fine. It gives this message:
>
> ############################
>
> CODE: (same as case 3, and probably the same for 1 and 2)
>
> ERROR:
>
> KeyboardInterrupt - Press <Enter> to continue.
>
> ###########################
>
> I hope this answers your question. If you want me to check any more things, please ask.
>
> Pieter
>
> Ville M. Vainio wrote:
>
>>
>> On Thu, Feb 12, 2009 at 3:06 PM, Pieter Cristiaan de Groot
>> <p.c.degroot at tudelft.nl> wrote:
>>
>>
>>>
>>> this is my last attempt to get the attention to this post. Even if this
>>> is the wrong place to post to,
>>> could somebody explain where it should be posted?
>>>
>>
>> This is the right place, you probably just didn't get lucky in drawing
>> people's attention. Perhaps because people don't use gthread on
>> windows all that much.
>>
>> Can you try catching the exception just for the sleep command?
>>
>>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>



-- 
Ville M. Vainio
http://tinyurl.com/vainio


From p.c.degroot at tudelft.nl  Fri Aug  7 05:01:13 2009
From: p.c.degroot at tudelft.nl (Pieter Cristiaan de Groot)
Date: Fri, 07 Aug 2009 11:01:13 +0200
Subject: [IPython-dev] Fwd:  ctrl-c bug using -gthread on windows,
 with 	possible solution
In-Reply-To: <46cb515a0908070155u60e408eeg55b7e417f56c7575@mail.gmail.com>
References: <4988D721.3060301@tudelft.nl>
	<4988D7AF.9090809@tudelft.nl>	<498C0B82.2030404@tudelft.nl>
	<49941EC0.7010006@tudelft.nl>	<46cb515a0902120554l28037c50i14d70f21f1833755@mail.gmail.com>	<49949D16.6020100@tudelft.nl>
	<4A7BE35A.2040101@tudelft.nl>
	<46cb515a0908070155u60e408eeg55b7e417f56c7575@mail.gmail.com>
Message-ID: <4A7BED59.6080603@tudelft.nl>

Hello Ville and the rest,

I'm reasonably sure that it is always an IOError, and the prints might 
not look so pretty when you're not debugging. Maybe this one is preferrable?

      try:
          update_tk(self.tk)
          self.IP.runcode()
          time.sleep(0.01)
          return True
      except IOError:
          return True

Best,
Pieter

Ville M. Vainio wrote:
> Thanks Pieter, cc:ing to ipython-dev.
>
> Someone might want to cherry-pick the following change from my
> trunk-dev and put it to trunk:
>
> http://bazaar.launchpad.net/~villemvainio/ipython/trunk-dev/revision/1160
>
>
>
> ---------- Forwarded message ----------
> From: Pieter Cristiaan de Groot <p.c.degroot at tudelft.nl>
> Date: Fri, Aug 7, 2009 at 11:18 AM
> Subject: Re: [IPython-dev] ctrl-c bug using -gthread on windows, with
> possible solution
> To: "Ville M. Vainio" <vivainio at gmail.com>
>
>
> Hello Ville,
>
> it looks like the ctrl-c bugfix for gthread didn't make it to 0.10.
> The error is still easily reproducable by starting ipython with
> -ghread. Type sleep(10), en then ctrl-c (in windows).
>
> The suggest change is:
>
> before:
>
>       update_tk(self.tk)
>       self.IP.runcode()
>       time.sleep(0.01)
>       return True
>
>
> after:
>
>       try:
>           update_tk(self.tk)
>           self.IP.runcode()
>           time.sleep(0.01)
>           return True
>       except IOError:
>           return True
>
> Best regards,
>
> Pieter
>
> Pieter Cristiaan de Groot wrote:
>   
>> Just to be sure I answer your question fully:
>>
>> case 1) the original 0.9.1 ipython code gives this error if I start it with -gthread and push ctrl-c
>>
>> ##########################
>>
>> CODE (in Shell.py, around linenr. 840):
>>
>>        update_tk(self.tk)
>>        self.IP.runcode()
>>        time.sleep(0.01)
>>        return True
>>
>> ERROR:
>>
>> KeyboardInterrupt - Press <Enter> to continue.---------------------------------------------------------------------------
>> IOError                                   Traceback (most recent call last)
>>
>> c:\python25\lib\site-packages\IPython\Shell.pyc in on_timer(self)
>>    840         update_tk(self.tk)
>>    841         self.IP.runcode()
>> --> 842         time.sleep(0.01)
>>    843         return True
>>    844
>>
>> IOError: [Errno 4] Interrupted function call
>>
>> #################
>>
>> case 2) catching the exception (with some prints):
>>
>> ###############
>>
>> CODE:
>>
>>        try:
>>            update_tk(self.tk)
>>            self.IP.runcode()
>>            time.sleep(0.01)
>>            return True
>>        except Exception, e:
>>            print type(e)
>>            print(e)
>>            return True
>>
>> ERROR:
>>
>> KeyboardInterrupt - Press <Enter> to continue. <type 'exceptions.IOError'>
>> [Errno 4] Interrupted function call
>>
>> #########################
>>
>> case 3) if I understood correctly this is what you wanted to see:
>>
>> #####################
>>
>> CODE:
>>
>>        update_tk(self.tk)
>>        self.IP.runcode()
>>        try:
>>            time.sleep(0.01)
>>            return True
>>        except Exception ,e:
>>            print type(e)
>>            print e
>>            return True
>>
>> ERROR:
>>
>> KeyboardInterrupt - Press <Enter> to continue. <type 'exceptions.IOError'>
>> [Errno 4] Interrupted function call
>>
>> ##########################
>>
>> case 3a) occasionally (1 in ~20 times), if you interrupt the code while it is not in the sleep, then everything works fine. It gives this message:
>>
>> ############################
>>
>> CODE: (same as case 3, and probably the same for 1 and 2)
>>
>> ERROR:
>>
>> KeyboardInterrupt - Press <Enter> to continue.
>>
>> ###########################
>>
>> I hope this answers your question. If you want me to check any more things, please ask.
>>
>> Pieter
>>
>> Ville M. Vainio wrote:
>>
>>     
>>> On Thu, Feb 12, 2009 at 3:06 PM, Pieter Cristiaan de Groot
>>> <p.c.degroot at tudelft.nl> wrote:
>>>
>>>
>>>       
>>>> this is my last attempt to get the attention to this post. Even if this
>>>> is the wrong place to post to,
>>>> could somebody explain where it should be posted?
>>>>
>>>>         
>>> This is the right place, you probably just didn't get lucky in drawing
>>> people's attention. Perhaps because people don't use gthread on
>>> windows all that much.
>>>
>>> Can you try catching the exception just for the sleep command?
>>>
>>>
>>>       
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>>
>>     
>
>
>
>   


From ellisonbg.net at gmail.com  Sun Aug  9 14:57:19 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sun, 9 Aug 2009 11:57:19 -0700
Subject: [IPython-dev] Cleaned up launchpad, trunk open for business!
In-Reply-To: <db6b5ecc0908050129n1f7301fcr3b712aa5f7817647@mail.gmail.com>
References: <db6b5ecc0908050129n1f7301fcr3b712aa5f7817647@mail.gmail.com>
Message-ID: <6ce0ac130908091157j39b98001gcb4f213f8ddcb799@mail.gmail.com>

Fernando,

Thanks for doing all of this!

after the release, I tried to clean things up a little and organize
>
better the situation in launchpad.  I still find some things take more
> clicks than they should, but overall the site has improved a lot.  We
> now have a nice overview of the code evolution visible as a timeline
> with all series present:
>
> https://launchpad.net/ipython/+series
>
> I updated the pages for each series and retargetted a bunch of
> milestones that had been lumped on trunk (I finally figured out where
> to click for that to happen) into their own series, so the picture is
> much clearer.
>
> Brian has done a great job of marking all new work  with bugs, I went
> through and cleaned up everything we had in progress/committed as
> released.
>
> As it stands now, Launchpad is actually getting to be pretty useful
> for organizing and supporting the running of the project. So I guess
> we can now focus again on trunk, reviewing the new reorganization
> work, merging any small leftovers we still have that didn't make it
> into 0.10, etc.
>
> Have fun, and thanks again to all  who contribute to IPython!
>
> f
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090809/b828543a/attachment.html>

From ellisonbg.net at gmail.com  Sun Aug  9 15:01:16 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sun, 9 Aug 2009 12:01:16 -0700
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
Message-ID: <6ce0ac130908091201n3d71f3e0mb168f774bf9a8836@mail.gmail.com>

Matthieu,

Sorry about the delay.  Here are the step I usually go through if I am
having trouble with ipcluster:

1.  First, clear out $HOME/.ipython/security
2.  Try to create a 1 engine cluster using ipcontroller and ipengine
manually (this is what ipcluster uses underneath the hood).  This will help
you debug any security related things.
3.  Then move on to ipcluster local and beyond.

Cheers,

Brian

On Thu, Aug 6, 2009 at 12:07 AM, Matthieu Brucher <
matthieu.brucher at gmail.com> wrote:

> Hi again,
>
> It seems that it works now, also nothing changed (perhaps a quota
> issue in my home folder?).
>
> I will try to setup the cluster stuff now.
>
> Matthieu
>
> 2009/8/5 Matthieu Brucher <matthieu.brucher at gmail.com>:
> > Hi,
> >
> > I'm trying to setup ipython 0.10 and test it before submitting a patch
> > for LSF support (which is what we are using in Pau instead of PBS).
> > This would be my first patch for Ipython (emotion).
> >
> > So first, I'm trying to get the ipcluster work locally with:
> >
> >> ipcluster local -n 4
> > And the log:
> > 2009-08-05 16:09:15+0200 [-] Log opened.
> > 2009-08-05 16:09:15+0200 [-] Process ['ipcontroller',
> > '--logfile=/users/brucher/.ipython/log/ipcontroller'] has started with
> > pid=8050
> > 2009-08-05 16:09:15+0200 [-] Waiting for controller to finish starting...
> > 2009-08-05 16:09:17+0200 [-] Controller started
> > 2009-08-05 16:09:17+0200 [-] Process ['ipengine',
> > '--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
> > with pid=8051
> > 2009-08-05 16:09:17+0200 [-] Process ['ipengine',
> > '--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
> > with pid=8052
> > 2009-08-05 16:09:17+0200 [-] Process ['ipengine',
> > '--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
> > with pid=8053
> > 2009-08-05 16:09:17+0200 [-] Process ['ipengine',
> > '--logfile=/users/brucher/.ipython/log/ipengine8050-'] has started
> > with pid=8054
> > 2009-08-05 16:09:17+0200 [-] Engines started with pids: [8051, 8052,
> 8053, 8054]
> >
> > Then, in the same location, I'm starting IPython and I try to connect
> > to the controller/engines but strangelly, it fails:
> >
> > [6] mec = client.MultiEngineClient()
> >
> ---------------------------------------------------------------------------
> > BananaError                               Traceback (most recent call
> last)
> >
> > /users/brucher/<ipython console> in <module>()
> >
> >
> /users/brucher/x86_64/lib/python2.5/site-packages/IPython/kernel/client.py
> > in get_multiengine_client(furl_or_file)
> >     67     """
> >     68     client =
> blockingCallFromThread(_client_tub.get_multiengine_client,
> > ---> 69         furl_or_file)
> >     70     return client.adapt_to_blocking_client()
> >     71
> >
> >
> /users/brucher/x86_64/lib/python2.5/site-packages/IPython/kernel/twistedutil.pyc
> > in blockingCallFromThread(f, *a, **kw)
> >     70         @raise: any error raised during the callback chain.
> >     71         """
> > ---> 72         return
> > twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
> >     73
> >     74 else:
> >
> >
> /users/brucher/x86_64/lib/python2.5/site-packages/twisted/internet/threads.pyc
> > in blockingCallFromThread(reactor, f, *a, **kw)
> >    112     result = queue.get()
> >    113     if isinstance(result, failure.Failure):
> > --> 114         result.raiseException()
> >    115     return result
> >    116
> >
> >
> /users/brucher/x86_64/lib/python2.5/site-packages/twisted/python/failure.pyc
> > in raiseException(self)
> >    324         information if available.
> >    325         """
> > --> 326         raise self.type, self.value, self.tb
> >    327
> >    328
> >
> > BananaError: BananaError: ('crypto for PB is not available, we cannot
> > handle encrypted PB-URLs like
> > pb://lhourelkbvqan3lmgrjcuhndtwiskows at 10.57.15.41:34415,
> 127.0.0.1:34415/gjgrouve4cmkx43b65qydlkyln5xwbc6',)
> >
> > I've tried without openssl with ipcluster -xy, but in this case, it
> > claimed it couldn't find furls in .ipython/security/..., and indeed,
> > there were none there or in the folder where I've started ipcluster.
> >
> > Is there something I'm missing? I've tried following the guide, but no
> > success so far :|
> >
> > Matthieu
> > --
> > Information System Engineer, Ph.D.
> > Website: http://matthieu-brucher.developpez.com/
> > Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
> >
>
>
>
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090809/d4bfdf3d/attachment.html>

From ellisonbg.net at gmail.com  Sun Aug  9 15:07:58 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sun, 9 Aug 2009 12:07:58 -0700
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <e76aa17f0908060036r5bdf1734we1ceda8e94ce33c4@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<db6b5ecc0908060010i2c4d1c22s5d2db66c32117bc6@mail.gmail.com>
	<e76aa17f0908060036r5bdf1734we1ceda8e94ce33c4@mail.gmail.com>
Message-ID: <6ce0ac130908091207k23920456w2d9c676224aa7c1c@mail.gmail.com>

> I ran into another issue: on a cluster, the home folder may be
> different than on the access box. In that case, the .ipython/security
> does not exist and the engine will not start (I've just tested this).
>

Currently our model for ipcluster is that:

.ipython/security is shared by all hosts and in the same location.  If you
don't have this situation, you will have to manually move the .furl files
around and tell ipengine where the .furl files are located.  You will also
need to use persistent furl files.  Docs on all this are here:

http://ipython.scipy.org/doc/stable/html/parallel/parallel_process.html

Let us know if you have other questions - this side of things can be very
subtle.  Another thing to watch out for.  Some batch systems *require* the
processes on compute nodes to call MPI_Init upon starting.  This can be
accomplished by using mpi4py.  See how we do this in the mpiexec/mpirun
versions of ipcluster.  But on some system (depends on which MPI) that is
not enough.  Some systems require that the *VERY FIRST* things a process
does is call MPI_Init.  On these systems you will need to build a custom
version of the python binary that handles this correctly.  Again, mpi4py
provides such a binary.  Hopefully you won't have to deal with these things!

Cheers,

Brian


>
> >> I will try to setup the cluster stuff now.
> >
> > Great, looking forward to the patch!
>
> Well, first I have to try to understand what is going on, as I don't
> even get the log files after the engine submission !
>
> Also, I've tried to extract the job id (it seems it is needed), but
> the BatchEngineSet.parse_job_id extracts everything that is matched by
> the regexp describing a job (it uses group()). I had to put "Job
> <(\d+)>" as a regexp, so group() returns, for instance, "Job <1234>"
> instead of "1234". I may submit a patch to get group(1) and modify the
> PBS regexp accordingly.
>

Yes, you will *very* likely have to modify the various regexps.



>
> Matthieu
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090809/0dab3893/attachment.html>

From matthieu.brucher at gmail.com  Sun Aug  9 15:27:17 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Sun, 9 Aug 2009 21:27:17 +0200
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <6ce0ac130908091201n3d71f3e0mb168f774bf9a8836@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<6ce0ac130908091201n3d71f3e0mb168f774bf9a8836@mail.gmail.com>
Message-ID: <e76aa17f0908091227y609e98a9h47be368c9f0a5548@mail.gmail.com>

2009/8/9 Brian Granger <ellisonbg.net at gmail.com>:
> Matthieu,
>
> Sorry about the delay.

No worries ;)

? Here are the step I usually go through if I am
> having trouble with ipcluster:
>
> 1.? First, clear out $HOME/.ipython/security
> 2.? Try to create a 1 engine cluster using ipcontroller and ipengine
> manually (this is what ipcluster uses underneath the hood).? This will help
> you debug any security related things.
> 3.? Then move on to ipcluster local and beyond.

I've tested again and it works with local and ssh (ssh on a remote
cluster that has a different $HOME).
I still have issues to get the log files from LSF, although the jobs
are submitted correctly (BTW, is there an explanation of how PBS
engine is set up?)

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From matthieu.brucher at gmail.com  Sun Aug  9 15:34:40 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Sun, 9 Aug 2009 21:34:40 +0200
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <6ce0ac130908091207k23920456w2d9c676224aa7c1c@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<db6b5ecc0908060010i2c4d1c22s5d2db66c32117bc6@mail.gmail.com>
	<e76aa17f0908060036r5bdf1734we1ceda8e94ce33c4@mail.gmail.com>
	<6ce0ac130908091207k23920456w2d9c676224aa7c1c@mail.gmail.com>
Message-ID: <e76aa17f0908091234q3657cf0j1a286ac0a2f21bc4@mail.gmail.com>

2009/8/9 Brian Granger <ellisonbg.net at gmail.com>:
>
>> I ran into another issue: on a cluster, the home folder may be
>> different than on the access box. In that case, the .ipython/security
>> does not exist and the engine will not start (I've just tested this).
>
> Currently our model for ipcluster is that:
>
> .ipython/security is shared by all hosts and in the same location.? If you
> don't have this situation, you will have to manually move the .furl files
> around and tell ipengine where the .furl files are located.? You will also
> need to use persistent furl files.? Docs on all this are here:
>
> http://ipython.scipy.org/doc/stable/html/parallel/parallel_process.html

With ssh-based ipcluster, I didn't need to copy the furls, as I
launched it from the host where I launched ipython as well.

> Let us know if you have other questions - this side of things can be very
> subtle.? Another thing to watch out for.? Some batch systems *require* the
> processes on compute nodes to call MPI_Init upon starting.? This can be
> accomplished by using mpi4py.? See how we do this in the mpiexec/mpirun
> versions of ipcluster.? But on some system (depends on which MPI) that is
> not enough.? Some systems require that the *VERY FIRST* things a process
> does is call MPI_Init.? On these systems you will need to build a custom
> version of the python binary that handles this correctly.? Again, mpi4py
> provides such a binary.? Hopefully you won't have to deal with these things!

I hope so! I don't think LSF requires to launch MPI_Init, but first, I
have to get access to the log files (I don't understand why they were
not copied, whereas the job was submitted).


>> Also, I've tried to extract the job id (it seems it is needed), but
>> the BatchEngineSet.parse_job_id extracts everything that is matched by
>> the regexp describing a job (it uses group()). I had to put "Job
>> <(\d+)>" as a regexp, so group() returns, for instance, "Job <1234>"
>> instead of "1234". I may submit a patch to get group(1) and modify the
>> PBS regexp accordingly.
>
> Yes, you will *very* likely have to modify the various regexps.

Is it needed to have the exact job ID ? Perhaps to kill the job?

Cheers,

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From ellisonbg.net at gmail.com  Mon Aug 10 13:09:18 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 10 Aug 2009 10:09:18 -0700
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <e76aa17f0908091234q3657cf0j1a286ac0a2f21bc4@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<db6b5ecc0908060010i2c4d1c22s5d2db66c32117bc6@mail.gmail.com>
	<e76aa17f0908060036r5bdf1734we1ceda8e94ce33c4@mail.gmail.com>
	<6ce0ac130908091207k23920456w2d9c676224aa7c1c@mail.gmail.com>
	<e76aa17f0908091234q3657cf0j1a286ac0a2f21bc4@mail.gmail.com>
Message-ID: <6ce0ac130908101009ie1e612ajebece1ab1ccf6866@mail.gmail.com>

> I hope so! I don't think LSF requires to launch MPI_Init, but first, I
> have to get access to the log files (I don't understand why they were
> not copied, whereas the job was submitted).
>

What log files are you referring to?  The IPython log files?  For the
engine?  The Controller?  Where do they need to be copied?  Could you
clarify?


> >> Also, I've tried to extract the job id (it seems it is needed), but
> >> the BatchEngineSet.parse_job_id extracts everything that is matched by
> >> the regexp describing a job (it uses group()). I had to put "Job
> >> <(\d+)>" as a regexp, so group() returns, for instance, "Job <1234>"
> >> instead of "1234". I may submit a patch to get group(1) and modify the
> >> PBS regexp accordingly.
> >
> > Yes, you will *very* likely have to modify the various regexps.
>
> Is it needed to have the exact job ID ? Perhaps to kill the job?
>

Yes, you definitely need the exact job ID to kill the job.

Brian


>
> Cheers,
>
> Matthieu
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090810/6828a91b/attachment.html>

From ellisonbg.net at gmail.com  Mon Aug 10 13:13:00 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 10 Aug 2009 10:13:00 -0700
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <e76aa17f0908091227y609e98a9h47be368c9f0a5548@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<6ce0ac130908091201n3d71f3e0mb168f774bf9a8836@mail.gmail.com>
	<e76aa17f0908091227y609e98a9h47be368c9f0a5548@mail.gmail.com>
Message-ID: <6ce0ac130908101013j466eb860yfb087edf96aff9fb@mail.gmail.com>

> I've tested again and it works with local and ssh (ssh on a remote
> cluster that has a different $HOME).
> I still have issues to get the log files from LSF, although the jobs
> are submitted correctly (BTW, is there an explanation of how PBS
> engine is set up?)
>

The only real explanation of how the PBS ipcluster works is the code
itself.  Basically, it goes through the following sequence:

1.  Start ipcontroller on the same node that ipcluster is being run.
2.  Construct a PBS batch script that calls ipengine on each node and then
submit the batch script.
3.  Get the job ID from the output of qsub.

You should be able to get LSF working by:

1.  Changing the submission syntax.
2.  Writing your own template for the batch script (test the batch script
without ipcluster first!)
3.  Write your own regexp for parsing the job ID
4.  Possibly add logic for copying the furl files around or for setting the
command line options to point to them is they are on different locations.

Keep us posted as it would be great to have LSF support.

Cheers,

Brian



>
> Matthieu
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090810/6870555f/attachment.html>

From matthieu.brucher at gmail.com  Mon Aug 10 14:18:18 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 10 Aug 2009 20:18:18 +0200
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <6ce0ac130908101009ie1e612ajebece1ab1ccf6866@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<db6b5ecc0908060010i2c4d1c22s5d2db66c32117bc6@mail.gmail.com>
	<e76aa17f0908060036r5bdf1734we1ceda8e94ce33c4@mail.gmail.com>
	<6ce0ac130908091207k23920456w2d9c676224aa7c1c@mail.gmail.com>
	<e76aa17f0908091234q3657cf0j1a286ac0a2f21bc4@mail.gmail.com>
	<6ce0ac130908101009ie1e612ajebece1ab1ccf6866@mail.gmail.com>
Message-ID: <e76aa17f0908101118m58f73a25u1ae3950ac6a9e752@mail.gmail.com>

2009/8/10 Brian Granger <ellisonbg.net at gmail.com>:
>
>> I hope so! I don't think LSF requires to launch MPI_Init, but first, I
>> have to get access to the log files (I don't understand why they were
>> not copied, whereas the job was submitted).
>
> What log files are you referring to?? The IPython log files?? For the
> engine?? The Controller?? Where do they need to be copied?? Could you
> clarify?

Sorry, I meant the LSF log files. Maybe the job is submitted but
couldn't be run. I'll try again tomorrow, the cluster is back online.

>> >> Also, I've tried to extract the job id (it seems it is needed), but
>> >> the BatchEngineSet.parse_job_id extracts everything that is matched by
>> >> the regexp describing a job (it uses group()). I had to put "Job
>> >> <(\d+)>" as a regexp, so group() returns, for instance, "Job <1234>"
>> >> instead of "1234". I may submit a patch to get group(1) and modify the
>> >> PBS regexp accordingly.
>> >
>> > Yes, you will *very* likely have to modify the various regexps.
>>
>> Is it needed to have the exact job ID ? Perhaps to kill the job?
>
> Yes, you definitely need the exact job ID to kill the job.

OK, this means I will modify the regexp and someone will have to check
the changes against PBS.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From matthieu.brucher at gmail.com  Mon Aug 10 14:22:18 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 10 Aug 2009 20:22:18 +0200
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <6ce0ac130908101013j466eb860yfb087edf96aff9fb@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<6ce0ac130908091201n3d71f3e0mb168f774bf9a8836@mail.gmail.com>
	<e76aa17f0908091227y609e98a9h47be368c9f0a5548@mail.gmail.com>
	<6ce0ac130908101013j466eb860yfb087edf96aff9fb@mail.gmail.com>
Message-ID: <e76aa17f0908101122k57f2300fo7fe26ae0938f5c7e@mail.gmail.com>

2009/8/10 Brian Granger <ellisonbg.net at gmail.com>:
>
>> I've tested again and it works with local and ssh (ssh on a remote
>> cluster that has a different $HOME).
>> I still have issues to get the log files from LSF, although the jobs
>> are submitted correctly (BTW, is there an explanation of how PBS
>> engine is set up?)
>
> The only real explanation of how the PBS ipcluster works is the code
> itself.? Basically, it goes through the following sequence:
>
> 1.? Start ipcontroller on the same node that ipcluster is being run.
> 2.? Construct a PBS batch script that calls ipengine on each node and then
> submit the batch script.
> 3.? Get the job ID from the output of qsub.
>
> You should be able to get LSF working by:
>
> 1.? Changing the submission syntax.

This is already done (not complicated, two names to change!)

> 2.? Writing your own template for the batch script (test the batch script
> without ipcluster first!)

Should be OK, I wrote several LSF scripts.

> 3.? Write your own regexp for parsing the job ID
> 4.? Possibly add logic for copying the furl files around or for setting the
> command line options to point to them is they are on different locations.

This may be the only thing that I couldn't check.

> Keep us posted as it would be great to have LSF support.

Of course I will! I will also try to get a patch against the trunk
(for th emoment, I only test against 0.10 due to proxy issues on my
side).

Cheers,

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From jdh2358 at gmail.com  Mon Aug 10 19:28:32 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Mon, 10 Aug 2009 18:28:32 -0500
Subject: [IPython-dev] possible cpaste bug
Message-ID: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com>

I just learned about cpaste today watching a showmedo video, and it
looks great.  Of course, I torture tested it with a complex example,
and it appears to fail on my system.  I don't know if cpaste is
expected to supoprt the full range of files you can "run" in ipython,
but here is one that failed for me:

In [7]: IPython.__version__
Out[7]: '0.11.bzr.r1205'

I can wget it and run it fine

In [10]: !wget http://matplotlib.sourceforge.net/plot_directive/mpl_examples/api/radar_chart.py
--18:27:10--  http://matplotlib.sourceforge.net/plot_directive/mpl_examples/api/radar_chart.py
           => `radar_chart.py'
Resolving matplotlib.sourceforge.net... 216.34.181.96
Connecting to matplotlib.sourceforge.net|216.34.181.96|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6,539 (6.4K) [text/plain]

100%[====================================>] 6,539         --.--K/s

18:27:10 (299.77 KB/s) - `radar_chart.py' saved [6539/6539]


In [11]: run radar_chart.py

but when I try and select it and paste it into ipython on OSX, I have problems

I was hoping cpaste would support more than the standard paste into an
ipython shell, eg the requirements of blank lines following indented
regions, etc, but maybe I am pushing it too hard.  Is there some
reason why cpaste  doesn't accept anything that "run" does?


From gokhansever at gmail.com  Mon Aug 10 19:35:33 2009
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Mon, 10 Aug 2009 18:35:33 -0500
Subject: [IPython-dev] possible cpaste bug
In-Reply-To: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com>
References: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com>
Message-ID: <49d6b3500908101635t705263b2me5b288b5e151be9f@mail.gmail.com>

On Mon, Aug 10, 2009 at 6:28 PM, John Hunter <jdh2358 at gmail.com> wrote:

> I just learned about cpaste today watching a showmedo video, and it
> looks great.  Of course, I torture tested it with a complex example,
> and it appears to fail on my system.  I don't know if cpaste is
> expected to supoprt the full range of files you can "run" in ipython,
> but here is one that failed for me:
>
> In [7]: IPython.__version__
> Out[7]: '0.11.bzr.r1205'
>
> I can wget it and run it fine
>
> In [10]: !wget
> http://matplotlib.sourceforge.net/plot_directive/mpl_examples/api/radar_chart.py
> --18:27:10--
> http://matplotlib.sourceforge.net/plot_directive/mpl_examples/api/radar_chart.py
>           => `radar_chart.py'
> Resolving matplotlib.sourceforge.net... 216.34.181.96
> Connecting to matplotlib.sourceforge.net|216.34.181.96|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 6,539 (6.4K) [text/plain]
>
> 100%[====================================>] 6,539         --.--K/s
>
> 18:27:10 (299.77 KB/s) - `radar_chart.py' saved [6539/6539]
>
>
> In [11]: run radar_chart.py
>
> but when I try and select it and paste it into ipython on OSX, I have
> problems
>
> I was hoping cpaste would support more than the standard paste into an
> ipython shell, eg the requirements of blank lines following indented
> regions, etc, but maybe I am pushing it too hard.  Is there some
> reason why cpaste  doesn't accept anything that "run" does?
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>

John,

It works here without any problems. Just make sure you copy the file content
correctly.

For some unknown reason, in the first try it has been copied erroneously as:

:    f3_base = [0.01, 0.02, 0.85, 0.19, 0.05, 0.10, 0.00, 0.00, 0.00]
:    f3_CO =   [0.01, 0.01, 0.79, 0.10, 0.00, 0.05, 0.00, 0.31, 0.00]
:    f3_O3 =   [0.01, 0.02, 0.86, 0.27, 0.16, 0.19, 0.
:    f3_both = [0.01, 0.02, 0.71, 0.24, 0.13, 0.16, 0.

later I manually copied the whole content and cpaste worked properly showing
the resulting figures.

PS: I use the same IPython rev.


-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090810/5c69811b/attachment.html>

From fperez.net at gmail.com  Mon Aug 10 22:36:18 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 10 Aug 2009 19:36:18 -0700
Subject: [IPython-dev] possible cpaste bug
In-Reply-To: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com>
References: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com>
Message-ID: <db6b5ecc0908101936s2f7c94d8n108e1cb09d6af7e4@mail.gmail.com>

Hey John,

On Mon, Aug 10, 2009 at 4:28 PM, John Hunter<jdh2358 at gmail.com> wrote:
> but when I try and select it and paste it into ipython on OSX, I have problems
>
> I was hoping cpaste would support more than the standard paste into an
> ipython shell, eg the requirements of blank lines following indented
> regions, etc, but maybe I am pushing it too hard. ?Is there some
> reason why cpaste ?doesn't accept anything that "run" does?

I'm actually also seeing %cpaste problems here, no idea why:

------------------------------------------------------------
   File "<string>", line 102
     f3_CO =   [0.01, 0.01, 0.79, 0.10, 0.00, 0.05,
         ^
SyntaxError: invalid syntax


The bad news is, I don't know what the problem is.  The good news is,
there's a new %paste that will paste straight from the in-memory
clipboard, and that runs fine.  Just copy to the clipboard anything,
type 'paste' in ipython (or %paste if you have a variable called paste
already), and you're done.  It's also faster than cpaste.

%paste doesn't allow you to keep inputting text into the area like
%cpaste does, so it's not a full replacement.  But in single
copy/paste/execute cases it's actually more convenient than %cpaste,
and apparently also more robust.

If you like it, you have R. Kern to thank for :)

Cheers,

f

ps - I'm filing a bug report on this one though, even if it's tricky
to reproduce.


From fperez.net at gmail.com  Mon Aug 10 22:50:43 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 10 Aug 2009 19:50:43 -0700
Subject: [IPython-dev] possible cpaste bug
In-Reply-To: <db6b5ecc0908101936s2f7c94d8n108e1cb09d6af7e4@mail.gmail.com>
References: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com> 
	<db6b5ecc0908101936s2f7c94d8n108e1cb09d6af7e4@mail.gmail.com>
Message-ID: <db6b5ecc0908101950u609ec112qecb09214a5e0f48a@mail.gmail.com>

On Mon, Aug 10, 2009 at 7:36 PM, Fernando Perez<fperez.net at gmail.com> wrote:
>
> ps - I'm filing a bug report on this one though, even if it's tricky
> to reproduce.

https://bugs.launchpad.net/ipython/+bug/411746

Note the possible workarounds indicated there (break up the paste
operation in smaller parts, or use the new %paste)

Cheers,

f


From robert.kern at gmail.com  Tue Aug 11 00:39:30 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 11 Aug 2009 00:39:30 -0400
Subject: [IPython-dev] possible cpaste bug
In-Reply-To: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com>
References: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com>
Message-ID: <h5qsm3$1f8$1@ger.gmane.org>

On 2009-08-10 19:28, John Hunter wrote:
> I just learned about cpaste today watching a showmedo video, and it
> looks great.  Of course, I torture tested it with a complex example,
> and it appears to fail on my system.  I don't know if cpaste is
> expected to supoprt the full range of files you can "run" in ipython,
> but here is one that failed for me:
>
> In [7]: IPython.__version__
> Out[7]: '0.11.bzr.r1205'
>
> I can wget it and run it fine
>
> In [10]: !wget http://matplotlib.sourceforge.net/plot_directive/mpl_examples/api/radar_chart.py
> --18:27:10--  http://matplotlib.sourceforge.net/plot_directive/mpl_examples/api/radar_chart.py
>             =>  `radar_chart.py'
> Resolving matplotlib.sourceforge.net... 216.34.181.96
> Connecting to matplotlib.sourceforge.net|216.34.181.96|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 6,539 (6.4K) [text/plain]
>
> 100%[====================================>] 6,539         --.--K/s
>
> 18:27:10 (299.77 KB/s) - `radar_chart.py' saved [6539/6539]
>
>
> In [11]: run radar_chart.py
>
> but when I try and select it and paste it into ipython on OSX, I have problems
>
> I was hoping cpaste would support more than the standard paste into an
> ipython shell, eg the requirements of blank lines following indented
> regions, etc, but maybe I am pushing it too hard.  Is there some
> reason why cpaste  doesn't accept anything that "run" does?

Are you sure that the correct text got fully pasted? I have had problems pasting 
moderate amounts of text, a couple of screenfuls, into a terminal on OS X, 
particularly with iTerm but also I think with Terminal.app. Some lines go 
missing or get repeated. The problem occurs with any program, not just IPython.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From fperez.net at gmail.com  Tue Aug 11 02:51:01 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 10 Aug 2009 23:51:01 -0700
Subject: [IPython-dev] possible cpaste bug
In-Reply-To: <h5qsm3$1f8$1@ger.gmane.org>
References: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com> 
	<h5qsm3$1f8$1@ger.gmane.org>
Message-ID: <db6b5ecc0908102351s7ead4aa0t314bc3df62bdb15f@mail.gmail.com>

On Mon, Aug 10, 2009 at 9:39 PM, Robert Kern<robert.kern at gmail.com> wrote:
> Are you sure that the correct text got fully pasted? I have had problems pasting
> moderate amounts of text, a couple of screenfuls, into a terminal on OS X,
> particularly with iTerm but also I think with Terminal.app. Some lines go
> missing or get repeated. The problem occurs with any program, not just IPython.

I should have pasted in the message the real input and not just the
traceback, it would have shown that indeed this seems to be, as you
suspect, a problem with the terminal itself.  I now tried it in
gnome-terminal, konsole and plain ole' xterm.  The first two both
cause the problem (in slightly different parts of the input,  but
always around the various f3* declarations),  while xterm is always
OK.

I suspect this is because xterm doesn't try to do anything fancy with
fonts (or anything else for that  matter) while g-t/konsole both use
pretty, anti-aliased fonts that take day and night to render.  The
overhead of rendering text and scrolling seems to be sufficient to
cause the  app to drop input from the paste stream somehow, in fact I
can see both 'stutter' as the pasted text accumulates.  Xterm paints
the text in ugly, old-style fonts but it does so blazingly fast, and
has no problems at all with the long input.

John, could you try using xterm on the mac as well? I think the X11
app ships with a plain old xterm, that could show a similar pattern to
what I'm seeing on linux.

Good catch, Robert, thanks.

Cheers,

f


From matthieu.brucher at gmail.com  Tue Aug 11 08:06:52 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 11 Aug 2009 14:06:52 +0200
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <e76aa17f0908101122k57f2300fo7fe26ae0938f5c7e@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<6ce0ac130908091201n3d71f3e0mb168f774bf9a8836@mail.gmail.com>
	<e76aa17f0908091227y609e98a9h47be368c9f0a5548@mail.gmail.com>
	<6ce0ac130908101013j466eb860yfb087edf96aff9fb@mail.gmail.com>
	<e76aa17f0908101122k57f2300fo7fe26ae0938f5c7e@mail.gmail.com>
Message-ID: <e76aa17f0908110506p1c6bb232nc1901fb09be1319@mail.gmail.com>

>> 4.? Possibly add logic for copying the furl files around or for setting the
>> command line options to point to them is they are on different locations.
>
> This may be the only thing that I couldn't check.

OK, I only have an issue with this at the moment. This is a log from the engine:

2009-08-11 14:04:44+0200 [-] Log opened.
2009-08-11 14:04:44+0200 [-] Using furl file:
/users/brucher/.ipython/security/ipcontroller-engine.furl
2009-08-11 14:04:44+0200 [Uninitialized] 'EngineConnector: engine
registration failed:'
2009-08-11 14:04:44+0200 [Uninitialized] Unhandled Error
	Traceback (most recent call last):
	Failure: twisted.internet.error.ConnectionRefusedError: Connection
was refused by other side: 111: Connection refused.
	
2009-08-11 14:04:44+0200 [Uninitialized] error connecting to
controller: Connection was refused by other side: 111: Connection
refused.
2009-08-11 14:04:44+0200 [-] Main loop terminated.

It is launch correctly by LSF, it is thus only a matter of setting the
connection correctly.

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From jdh2358 at gmail.com  Tue Aug 11 20:24:43 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Tue, 11 Aug 2009 19:24:43 -0500
Subject: [IPython-dev] possible cpaste bug
In-Reply-To: <db6b5ecc0908102351s7ead4aa0t314bc3df62bdb15f@mail.gmail.com>
References: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com>
	<h5qsm3$1f8$1@ger.gmane.org>
	<db6b5ecc0908102351s7ead4aa0t314bc3df62bdb15f@mail.gmail.com>
Message-ID: <88e473830908111724g22b5131fj7428b0f342a50133@mail.gmail.com>

On Tue, Aug 11, 2009 at 1:51 AM, Fernando Perez<fperez.net at gmail.com> wrote:

> John, could you try using xterm on the mac as well? I think the X11
> app ships with a plain old xterm, that could show a similar pattern to
> what I'm seeing on linux.

I tried to try on xterm, but goddam osx and xterms do not play nicely
together.  I could not paste from the desktop environment to the x11
environment since my buffer was not recognized, so I tried to cat the
example in one xterm to paste it into another but could not get the
whole thing on one screen and could not get the window to scroll as I
moved the mouse past the top.  I'm sure there is a way to do this
(Robert 3..2..1..) but I don't know how so I punted.  I am pretty sure
Robert's explanation is correct, though it is a major wart to simply
drop the input to the term, which appears to be what os x is doing,
IMO.

"paste" did work, so that is cool (thanks Robert!)  But for
tutorial/demo purposes, which is what is on my mind of late, the
screen echo is nice, kind of a "nothing up my sleaves" feeling, that
the paste with no echo does not have.  Could we do a paste -e (echo)
or -v (verbose) and ask paste to echo the paste to the stdout, even if
only for visual effect (and during the tutorial its nice to point to
the code in the terminal)

JDH


From fperez.net at gmail.com  Wed Aug 12 01:06:55 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 11 Aug 2009 22:06:55 -0700
Subject: [IPython-dev] possible cpaste bug
In-Reply-To: <88e473830908111724g22b5131fj7428b0f342a50133@mail.gmail.com>
References: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com> 
	<h5qsm3$1f8$1@ger.gmane.org>
	<db6b5ecc0908102351s7ead4aa0t314bc3df62bdb15f@mail.gmail.com> 
	<88e473830908111724g22b5131fj7428b0f342a50133@mail.gmail.com>
Message-ID: <db6b5ecc0908112206tc407eclf1065b8ada5759ce@mail.gmail.com>

On Tue, Aug 11, 2009 at 5:24 PM, John Hunter<jdh2358 at gmail.com> wrote:

> I tried to try on xterm, but goddam osx and xterms do not play nicely
> together. ?I could not paste from the desktop environment to the x11
> environment since my buffer was not recognized, so I tried to cat the
> example in one xterm to paste it into another but could not get the
> whole thing on one screen and could not get the window to scroll as I
> moved the mouse past the top. ?I'm sure there is a way to do this
> (Robert 3..2..1..) but I don't know how so I punted. ?I am pretty sure
> Robert's explanation is correct, though it is a major wart to simply
> drop the input to the term, which appears to be what os x is doing,
> IMO.

Yes, majorly annoying indeed (and it's awful that the two major linux
desktop terminal emulators both also have the bug, when 20+ years old
xterm gets it right).

> "paste" did work, so that is cool (thanks Robert!) ?But for
> tutorial/demo purposes, which is what is on my mind of late, the
> screen echo is nice, kind of a "nothing up my sleaves" feeling, that
> the paste with no echo does not have. ?Could we do a paste -e (echo)
> or -v (verbose) and ask paste to echo the paste to the stdout, even if
> only for visual effect (and during the tutorial its nice to point to
> the code in the terminal)

Your wish is our command, Dr. Hunter; bug and solution:

https://bugs.launchpad.net/ipython/+bug/412339
https://code.launchpad.net/~ipython-dev/ipython/cpaste-fixes

In action:

In [2]: paste -e
x = 1
y = 2
print 'x, y:',x,y
## -- End pasted text --
x, y: 1 2

In [3]: paste
x, y: 1 2

In [4]: paste -r
Re-executing 'x = 1...' (30 chars)
x, y: 1 2


1-800-pygeeks at your service!

Cheers,

f

ps - you can safely merge from that branch into your working one.
Since Brian is in the middle of major reorganizations, this may take a
few days to get merged, as I don't want to get in his way.  But it's
localized enough that you can merge it for next week's tutorials
locally while we get it in place.


From fperez.net at gmail.com  Wed Aug 12 01:55:40 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 11 Aug 2009 22:55:40 -0700
Subject: [IPython-dev] Fwd: ctrl-c bug using -gthread on windows,
	with 	possible solution
In-Reply-To: <4A7BED59.6080603@tudelft.nl>
References: <4988D721.3060301@tudelft.nl> <4988D7AF.9090809@tudelft.nl> 
	<498C0B82.2030404@tudelft.nl> <49941EC0.7010006@tudelft.nl> 
	<46cb515a0902120554l28037c50i14d70f21f1833755@mail.gmail.com> 
	<49949D16.6020100@tudelft.nl> <4A7BE35A.2040101@tudelft.nl> 
	<46cb515a0908070155u60e408eeg55b7e417f56c7575@mail.gmail.com> 
	<4A7BED59.6080603@tudelft.nl>
Message-ID: <db6b5ecc0908112255vd2bb990i5a78329614189d7a@mail.gmail.com>

Hi Pieter,

On Fri, Aug 7, 2009 at 2:01 AM, Pieter Cristiaan de
Groot<p.c.degroot at tudelft.nl> wrote:
> Hello Ville and the rest,
>
> I'm reasonably sure that it is always an IOError, and the prints might
> not look so pretty when you're not debugging. Maybe this one is preferrable?
>
> ? ? ?try:
> ? ? ? ? ?update_tk(self.tk)
> ? ? ? ? ?self.IP.runcode()
> ? ? ? ? ?time.sleep(0.01)
> ? ? ? ? ?return True
> ? ? ?except IOError:
> ? ? ? ? ?return True

I'm sorry this one slipped through the cracks with the recent 0.10
work. It was a small fix but I didn't see it and there was no bug
report assigned to the 0.10 milestone with this information to alert
me.  When you reported it originally I was simply too swamped and
unresponsive, and it slipped.

I've now made a bug for it so this doesn't happen again:

https://bugs.launchpad.net/ipython/+bug/412353

As it says there, it's possible we may not need it for 0.11, but we'll
see.  In any case, if we do put out an 0.10.1 bugfix release later,
this should go in there.

Regards,

f


From fperez.net at gmail.com  Wed Aug 12 01:57:57 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 11 Aug 2009 22:57:57 -0700
Subject: [IPython-dev] Fwd:  Last checks on release notes for 0.10
In-Reply-To: <db6b5ecc0908041211y22ada259k697be7a3a4033ade@mail.gmail.com>
References: <db6b5ecc0908040356g42e648f4jd211938ab67bee51@mail.gmail.com> 
	<6ce0ac130908041107q6714056cq3c124abbcbff90ca@mail.gmail.com> 
	<db6b5ecc0908041211y22ada259k697be7a3a4033ade@mail.gmail.com>
Message-ID: <db6b5ecc0908112257i5356ab01nc3bd81dd76f1aea5@mail.gmail.com>

I was cleaning my inbox and just noticed I'd sent this off-list when
it was meant for the list.  It's basically a wrapup of the 0.10
release notes process with some thoughts on the logs/changes.txt.
Nothing major, I just want it archived on-list, since it was meant to
go there.

---------- Forwarded message ----------
From: Fernando Perez <fperez.net at gmail.com>
Date: Tue, Aug 4, 2009 at 12:11 PM
Subject: Re: [IPython-dev] Last checks on release notes for 0.10
To: Brian Granger <ellisonbg.net at gmail.com>


On Tue, Aug 4, 2009 at 11:07 AM, Brian Granger<ellisonbg.net at gmail.com> wrote:
> Fantastic, that was a huge task for you to write the changes for all of our
> commits for the last year!
>
> We definitely need to come up with a good way of making sure that we all
> write our own changes.

I have to say, it actually wasn't too bad, though it was time
consuming. ?But I thought it was going to be worse. ?A few comments:

- Both you and Ville had been pretty good about adding your edits to
changes.txt, especially earlier. ?There wasn't that much that I had to
re-edit from the changelog.

- When big merges are done, the practice of putting a summary commit
message in the merge is *extremely* useful. ?It makes this kind of job
much nicer, because that summary log message can be almost copy/pasted
without changes, if it was well written, rather than dissecting the
next-level messages from the individual commits.

- It's important that we remember to always credit who gave us
something if it's not the committer. ?There was a lot of that
information in the commits, so it seems we're doing pretty good on
that front, but obviously I can't know if we missed someone (unless
they tell us). This is just a reminder to keep things up on that
front. ?As a reminder, if you are ever committing something that's
completely (or almost so) a third-party contribution, do the commit as

bzr commit --author="Someone Else"

This way it will show that name separately in the log, which makes it
even easier to spot. ?Obviously we often rework third party
contributions extensively, but this is still good to keep in mind for
cases when we don't touch the code too much.

- Overall, I just used

bzr log | less

and I was in the end quite happy that it's pretty good and painless.
The individual messages have all the relevant info, if --author is
used it's clear when they are third-party contributions, and the
indentation makes it very easy to read.

For all my unhappiness with bzr's slowness and its braindead behavior
when X11 connections aren't being tunneled, I have to say that for all
the recent flurry of work we've done here it has behaved very very
well. ?And I also think that we're reaching a good workflow with the
tool. ?Not perfect, but very decent. ?Which reminds me of this post by
Linus:

http://torvalds-family.blogspot.com/2009/06/happiness-is-warm-scm.html

Distributed version control is both a tool and a way of working, and I
think many of us (at least I did) at the beginning failed to grasp how
much more of a workflow planning it requires. ?The fact that bzr's
docs are woefully short on the workflow details didn't help. ?But it's
interesting to see that the Linux kernel crowd, arguably one of the
most technically savvy development communities in the world, even with
having a tailor-made DVCS they created themselves, still needed
several years to find their 'happy spot' in terms of the integration
between the tools and the team's workflow.


In any case, I think I am going to add some of these notes I just
typed up here to our dev documents, and then I'll get on with cutting
out the release.

Stop me now if you think it's a bad idea :)

Cheers,

f


From matthieu.brucher at gmail.com  Wed Aug 12 03:15:34 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 12 Aug 2009 09:15:34 +0200
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <e76aa17f0908110506p1c6bb232nc1901fb09be1319@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<6ce0ac130908091201n3d71f3e0mb168f774bf9a8836@mail.gmail.com>
	<e76aa17f0908091227y609e98a9h47be368c9f0a5548@mail.gmail.com>
	<6ce0ac130908101013j466eb860yfb087edf96aff9fb@mail.gmail.com>
	<e76aa17f0908101122k57f2300fo7fe26ae0938f5c7e@mail.gmail.com>
	<e76aa17f0908110506p1c6bb232nc1901fb09be1319@mail.gmail.com>
Message-ID: <e76aa17f0908120015n29478a18qbeb29f71ed7185e1@mail.gmail.com>

2009/8/11 Matthieu Brucher <matthieu.brucher at gmail.com>:
>>> 4.? Possibly add logic for copying the furl files around or for setting the
>>> command line options to point to them is they are on different locations.
>>
>> This may be the only thing that I couldn't check.
>
> OK, I only have an issue with this at the moment. This is a log from the engine:
>
> 2009-08-11 14:04:44+0200 [-] Log opened.
> 2009-08-11 14:04:44+0200 [-] Using furl file:
> /users/brucher/.ipython/security/ipcontroller-engine.furl
> 2009-08-11 14:04:44+0200 [Uninitialized] 'EngineConnector: engine
> registration failed:'
> 2009-08-11 14:04:44+0200 [Uninitialized] Unhandled Error
> ? ? ? ?Traceback (most recent call last):
> ? ? ? ?Failure: twisted.internet.error.ConnectionRefusedError: Connection
> was refused by other side: 111: Connection refused.
>
> 2009-08-11 14:04:44+0200 [Uninitialized] error connecting to
> controller: Connection was refused by other side: 111: Connection
> refused.
> 2009-08-11 14:04:44+0200 [-] Main loop terminated.
>
> It is launch correctly by LSF, it is thus only a matter of setting the
> connection correctly.
>
> Matthieu

Is there a simple way to test the connections with foolscape?

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From ellisonbg.net at gmail.com  Wed Aug 12 05:58:13 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 12 Aug 2009 02:58:13 -0700
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <e76aa17f0908120015n29478a18qbeb29f71ed7185e1@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<6ce0ac130908091201n3d71f3e0mb168f774bf9a8836@mail.gmail.com>
	<e76aa17f0908091227y609e98a9h47be368c9f0a5548@mail.gmail.com>
	<6ce0ac130908101013j466eb860yfb087edf96aff9fb@mail.gmail.com>
	<e76aa17f0908101122k57f2300fo7fe26ae0938f5c7e@mail.gmail.com>
	<e76aa17f0908110506p1c6bb232nc1901fb09be1319@mail.gmail.com>
	<e76aa17f0908120015n29478a18qbeb29f71ed7185e1@mail.gmail.com>
Message-ID: <6ce0ac130908120258y66547ee5x77b4453beae5fecc@mail.gmail.com>

This looks like a basic TCP/IP connection issue.  It could be related to a
number of things.  One thing to keep in mind is the direction of the
connections.  The controller need to start first - it listens on a port and
then the engines connect to it.  The host that the controller is on needs to
allow incoming TCP/IP connections.  The hosts with the engine need to allow
outgoing connections.

Have a look at the following:

* Firewall.  If a fire wall is blocking the engine from connecting to the
controller you will see this type of error.  A fire wall like this would be
unusual though (I have never seen one before).  To test this, start the
controller on the head node, ssh to a compute node and then just telnet (it
will fail) to the controller.  But you should see the connection start to
happen.  You could also run ipengine by hand on the compute node.
* If the controller hasn't been started or failed to start, you would also
see this.  Look at the controller logs to see if this is going on.
* If there is NAT (network address translation) on the cluster.  This is
pretty common.  Typically this would be that the head node has multiple
network interfaces, one for the outside world and one for talking to the
compute nodes.  In this case, you will need to use ifconfig to hunt down the
right ip address.  Then you will need to use the --engine-ip flag to
ipcontroller to set the ip address that the engines will connect to.  The
engines get this from the furl file that the controller writes.

I am betting that the 2nd or 3rd of these is going on.  Keep us posted as
these things can be pretty tough to debug because of how some clusters are
setup.  But, take heart, I have never encountered a system that we could get
working - and this includes some pretty crazy systems.

Cheers,

Brian


On Wed, Aug 12, 2009 at 12:15 AM, Matthieu Brucher <
matthieu.brucher at gmail.com> wrote:

> 2009/8/11 Matthieu Brucher <matthieu.brucher at gmail.com>:
> >>> 4.  Possibly add logic for copying the furl files around or for setting
> the
> >>> command line options to point to them is they are on different
> locations.
> >>
> >> This may be the only thing that I couldn't check.
> >
> > OK, I only have an issue with this at the moment. This is a log from the
> engine:
> >
> > 2009-08-11 14:04:44+0200 [-] Log opened.
> > 2009-08-11 14:04:44+0200 [-] Using furl file:
> > /users/brucher/.ipython/security/ipcontroller-engine.furl
> > 2009-08-11 14:04:44+0200 [Uninitialized] 'EngineConnector: engine
> > registration failed:'
> > 2009-08-11 14:04:44+0200 [Uninitialized] Unhandled Error
> >        Traceback (most recent call last):
> >        Failure: twisted.internet.error.ConnectionRefusedError: Connection
> > was refused by other side: 111: Connection refused.
> >
> > 2009-08-11 14:04:44+0200 [Uninitialized] error connecting to
> > controller: Connection was refused by other side: 111: Connection
> > refused.
> > 2009-08-11 14:04:44+0200 [-] Main loop terminated.
> >
> > It is launch correctly by LSF, it is thus only a matter of setting the
> > connection correctly.
> >
> > Matthieu
>
> Is there a simple way to test the connections with foolscape?
>
> Matthieu
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090812/6eb27151/attachment.html>

From ellisonbg.net at gmail.com  Wed Aug 12 06:01:55 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 12 Aug 2009 03:01:55 -0700
Subject: [IPython-dev] Fwd: ctrl-c bug using -gthread on windows,
	with 	possible solution
In-Reply-To: <db6b5ecc0908112255vd2bb990i5a78329614189d7a@mail.gmail.com>
References: <4988D721.3060301@tudelft.nl> <4988D7AF.9090809@tudelft.nl>
	<498C0B82.2030404@tudelft.nl> <49941EC0.7010006@tudelft.nl>
	<46cb515a0902120554l28037c50i14d70f21f1833755@mail.gmail.com>
	<49949D16.6020100@tudelft.nl> <4A7BE35A.2040101@tudelft.nl>
	<46cb515a0908070155u60e408eeg55b7e417f56c7575@mail.gmail.com>
	<4A7BED59.6080603@tudelft.nl>
	<db6b5ecc0908112255vd2bb990i5a78329614189d7a@mail.gmail.com>
Message-ID: <6ce0ac130908120301nb6c4fb6w3867bb06ad0fb8d3@mail.gmail.com>

Yes, once the inputhook branch is merged into trunk, this should be fixed.
Details will follow on list when this happens.

Cheers,

Brian

On Tue, Aug 11, 2009 at 10:55 PM, Fernando Perez <fperez.net at gmail.com>wrote:

> Hi Pieter,
>
> On Fri, Aug 7, 2009 at 2:01 AM, Pieter Cristiaan de
> Groot<p.c.degroot at tudelft.nl> wrote:
> > Hello Ville and the rest,
> >
> > I'm reasonably sure that it is always an IOError, and the prints might
> > not look so pretty when you're not debugging. Maybe this one is
> preferrable?
> >
> >      try:
> >          update_tk(self.tk)
> >          self.IP.runcode()
> >          time.sleep(0.01)
> >          return True
> >      except IOError:
> >          return True
>
> I'm sorry this one slipped through the cracks with the recent 0.10
> work. It was a small fix but I didn't see it and there was no bug
> report assigned to the 0.10 milestone with this information to alert
> me.  When you reported it originally I was simply too swamped and
> unresponsive, and it slipped.
>
> I've now made a bug for it so this doesn't happen again:
>
> https://bugs.launchpad.net/ipython/+bug/412353
>
> As it says there, it's possible we may not need it for 0.11, but we'll
> see.  In any case, if we do put out an 0.10.1 bugfix release later,
> this should go in there.
>
> Regards,
>
> f
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090812/ab77f97e/attachment.html>

From jdh2358 at gmail.com  Wed Aug 12 07:10:06 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Wed, 12 Aug 2009 06:10:06 -0500
Subject: [IPython-dev] possible cpaste bug
In-Reply-To: <db6b5ecc0908112206tc407eclf1065b8ada5759ce@mail.gmail.com>
References: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com>
	<h5qsm3$1f8$1@ger.gmane.org>
	<db6b5ecc0908102351s7ead4aa0t314bc3df62bdb15f@mail.gmail.com>
	<88e473830908111724g22b5131fj7428b0f342a50133@mail.gmail.com>
	<db6b5ecc0908112206tc407eclf1065b8ada5759ce@mail.gmail.com>
Message-ID: <88e473830908120410i296b5dcekb6cd7571b5d38fcc@mail.gmail.com>

On Wed, Aug 12, 2009 at 12:06 AM, Fernando Perez<fperez.net at gmail.com> wrote:

> Your wish is our command, Dr. Hunter; bug and solution:
>
> https://bugs.launchpad.net/ipython/+bug/412339
> https://code.launchpad.net/~ipython-dev/ipython/cpaste-fixes

Ahh this is working perfectly.  Many thanks. I wonder if the silent
mode in the original implementation was a design choice or a
convenience.  If the latter,  maybe you want the echo on by default
with a -q (quiet) or -s (silent) to silence it.  I certainly defer to
Robert's preference, but I'll normally be using this in echo mode,

Thanks all for the help!

JDH


From matthieu.brucher at gmail.com  Wed Aug 12 11:06:00 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 12 Aug 2009 17:06:00 +0200
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <6ce0ac130908120258y66547ee5x77b4453beae5fecc@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<6ce0ac130908091201n3d71f3e0mb168f774bf9a8836@mail.gmail.com>
	<e76aa17f0908091227y609e98a9h47be368c9f0a5548@mail.gmail.com>
	<6ce0ac130908101013j466eb860yfb087edf96aff9fb@mail.gmail.com>
	<e76aa17f0908101122k57f2300fo7fe26ae0938f5c7e@mail.gmail.com>
	<e76aa17f0908110506p1c6bb232nc1901fb09be1319@mail.gmail.com>
	<e76aa17f0908120015n29478a18qbeb29f71ed7185e1@mail.gmail.com>
	<6ce0ac130908120258y66547ee5x77b4453beae5fecc@mail.gmail.com>
Message-ID: <e76aa17f0908120806y2547f31fq52f5ba83043e3708@mail.gmail.com>

> * Firewall.? If a fire wall is blocking the engine from connecting to the
> controller you will see this type of error.? A fire wall like this would be
> unusual though (I have never seen one before).? To test this, start the
> controller on the head node, ssh to a compute node and then just telnet (it
> will fail) to the controller.? But you should see the connection start to
> happen.? You could also run ipengine by hand on the compute node.

No worries on this side. We do a lot of client/server stuff, it did
work with telnet.

> * If the controller hasn't been started or failed to start, you would also
> see this.? Look at the controller logs to see if this is going on.

It seems the controller was launched (and as I can telnet it, I think
it is online?):

2009-08-12 16:59:52+0200 [-] Log opened.
2009-08-12 16:59:52+0200 [-] Process ['ipcontroller',
'--logfile=/users/brucher/.ipython/log/ipcontroller'] has started with
pid=5001
2009-08-12 16:59:52+0200 [-] Waiting for controller to finish starting...
2009-08-12 16:59:55+0200 [-] Controller started
2009-08-12 16:59:55+0200 [-] Using template for batch script: lsf.template
2009-08-12 16:59:55+0200 [-] Writing instantiated batch script: lsf.template-run
2009-08-12 16:59:55+0200 [-] Job started with job id: '6166'

> * If there is NAT (network address translation) on the cluster.? This is
> pretty common.?Typically this would be that the head node has multiple
> network interfaces, one for the outside world and one for talking to the
> compute nodes.? In this case, you will need to use ifconfig to hunt down the
> right ip address.? Then you will need to use the --engine-ip flag to
> ipcontroller to set the ip address that the engines will connect to.? The
> engines get this from the furl file that the controller writes.

I don't think there is something like that here. I can connect to the
LSF nodes with ssh and then telnet the controller: it works with the
IP address indicated in the furl.

> I am betting that the 2nd or 3rd of these is going on.? Keep us posted as
> these things can be pretty tough to debug because of how some clusters are
> setup.? But, take heart, I have never encountered a system that we could get
> working - and this includes some pretty crazy systems.

I suppose you meant the contrary ;)
I still have hope to get it working in the near future :D

At least, I have also the LSF logs, but they do not show a thing, as
everything is output in the ipengine logs.

Cheers,

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From robert.kern at gmail.com  Wed Aug 12 11:38:03 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 12 Aug 2009 11:38:03 -0400
Subject: [IPython-dev] possible cpaste bug
In-Reply-To: <88e473830908120410i296b5dcekb6cd7571b5d38fcc@mail.gmail.com>
References: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com>	<h5qsm3$1f8$1@ger.gmane.org>	<db6b5ecc0908102351s7ead4aa0t314bc3df62bdb15f@mail.gmail.com>	<88e473830908111724g22b5131fj7428b0f342a50133@mail.gmail.com>	<db6b5ecc0908112206tc407eclf1065b8ada5759ce@mail.gmail.com>
	<88e473830908120410i296b5dcekb6cd7571b5d38fcc@mail.gmail.com>
Message-ID: <h5unkv$opb$1@ger.gmane.org>

On 2009-08-12 07:10 AM, John Hunter wrote:
> On Wed, Aug 12, 2009 at 12:06 AM, Fernando Perez<fperez.net at gmail.com>  wrote:
>
>> Your wish is our command, Dr. Hunter; bug and solution:
>>
>> https://bugs.launchpad.net/ipython/+bug/412339
>> https://code.launchpad.net/~ipython-dev/ipython/cpaste-fixes
>
> Ahh this is working perfectly.  Many thanks. I wonder if the silent
> mode in the original implementation was a design choice or a
> convenience.

I frequently consider "laziness" to be a design choice. But I do agree that 
echoing by default would be a good idea. It gives the user confidence that the 
pasted code is what the user thought it was.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From ellisonbg.net at gmail.com  Wed Aug 12 12:17:45 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 12 Aug 2009 09:17:45 -0700
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <e76aa17f0908120806y2547f31fq52f5ba83043e3708@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908060007s1f6448e3jcc5fb14bfb0441f4@mail.gmail.com>
	<6ce0ac130908091201n3d71f3e0mb168f774bf9a8836@mail.gmail.com>
	<e76aa17f0908091227y609e98a9h47be368c9f0a5548@mail.gmail.com>
	<6ce0ac130908101013j466eb860yfb087edf96aff9fb@mail.gmail.com>
	<e76aa17f0908101122k57f2300fo7fe26ae0938f5c7e@mail.gmail.com>
	<e76aa17f0908110506p1c6bb232nc1901fb09be1319@mail.gmail.com>
	<e76aa17f0908120015n29478a18qbeb29f71ed7185e1@mail.gmail.com>
	<6ce0ac130908120258y66547ee5x77b4453beae5fecc@mail.gmail.com>
	<e76aa17f0908120806y2547f31fq52f5ba83043e3708@mail.gmail.com>
Message-ID: <6ce0ac130908120917m6dd5a68fq67bd22fade7ce47@mail.gmail.com>

Matthieu,

Can you do the following test:

headnode> ipcontroller

# Then copy the engines furl file to the compute node and do in a separate
terminal:

computenode> ipengine --furl-file=[path to the furl file]

If that doesn't work, it is either:

* IP address related issue.  Play with ifconfig and ipcontroller --engine-ip
* Firewall.  But you said this wasn't an issue.

Hope this helps.

Cheers,

Brian

On Wed, Aug 12, 2009 at 8:06 AM, Matthieu Brucher <
matthieu.brucher at gmail.com> wrote:

> > * Firewall.  If a fire wall is blocking the engine from connecting to the
> > controller you will see this type of error.  A fire wall like this would
> be
> > unusual though (I have never seen one before).  To test this, start the
> > controller on the head node, ssh to a compute node and then just telnet
> (it
> > will fail) to the controller.  But you should see the connection start to
> > happen.  You could also run ipengine by hand on the compute node.
>
> No worries on this side. We do a lot of client/server stuff, it did
> work with telnet.
>
> > * If the controller hasn't been started or failed to start, you would
> also
> > see this.  Look at the controller logs to see if this is going on.
>
> It seems the controller was launched (and as I can telnet it, I think
> it is online?):
>
> 2009-08-12 16:59:52+0200 [-] Log opened.
> 2009-08-12 16:59:52+0200 [-] Process ['ipcontroller',
> '--logfile=/users/brucher/.ipython/log/ipcontroller'] has started with
> pid=5001
> 2009-08-12 16:59:52+0200 [-] Waiting for controller to finish starting...
> 2009-08-12 16:59:55+0200 [-] Controller started
> 2009-08-12 16:59:55+0200 [-] Using template for batch script: lsf.template
> 2009-08-12 16:59:55+0200 [-] Writing instantiated batch script:
> lsf.template-run
> 2009-08-12 16:59:55+0200 [-] Job started with job id: '6166'
>
> > * If there is NAT (network address translation) on the cluster.  This is
> > pretty common. Typically this would be that the head node has multiple
> > network interfaces, one for the outside world and one for talking to the
> > compute nodes.  In this case, you will need to use ifconfig to hunt down
> the
> > right ip address.  Then you will need to use the --engine-ip flag to
> > ipcontroller to set the ip address that the engines will connect to.  The
> > engines get this from the furl file that the controller writes.
>
> I don't think there is something like that here. I can connect to the
> LSF nodes with ssh and then telnet the controller: it works with the
> IP address indicated in the furl.
>
> > I am betting that the 2nd or 3rd of these is going on.  Keep us posted as
> > these things can be pretty tough to debug because of how some clusters
> are
> > setup.  But, take heart, I have never encountered a system that we could
> get
> > working - and this includes some pretty crazy systems.
>
> I suppose you meant the contrary ;)
> I still have hope to get it working in the near future :D
>
> At least, I have also the LSF logs, but they do not show a thing, as
> everything is output in the ipengine logs.
>
> Cheers,
>
> Matthieu
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090812/1428d457/attachment.html>

From fperez.net at gmail.com  Wed Aug 12 13:51:09 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 12 Aug 2009 10:51:09 -0700
Subject: [IPython-dev] possible cpaste bug
In-Reply-To: <h5unkv$opb$1@ger.gmane.org>
References: <88e473830908101628o6934aff4p323dde976cc340a@mail.gmail.com> 
	<h5qsm3$1f8$1@ger.gmane.org>
	<db6b5ecc0908102351s7ead4aa0t314bc3df62bdb15f@mail.gmail.com> 
	<88e473830908111724g22b5131fj7428b0f342a50133@mail.gmail.com> 
	<db6b5ecc0908112206tc407eclf1065b8ada5759ce@mail.gmail.com> 
	<88e473830908120410i296b5dcekb6cd7571b5d38fcc@mail.gmail.com> 
	<h5unkv$opb$1@ger.gmane.org>
Message-ID: <db6b5ecc0908121051m50cc64f8l956d8b7eefdd467d@mail.gmail.com>

On Wed, Aug 12, 2009 at 8:38 AM, Robert Kern<robert.kern at gmail.com> wrote:
> On 2009-08-12 07:10 AM, John Hunter wrote:
>> On Wed, Aug 12, 2009 at 12:06 AM, Fernando Perez<fperez.net at gmail.com> ?wrote:
>>
>>> Your wish is our command, Dr. Hunter; bug and solution:
>>>
>>> https://bugs.launchpad.net/ipython/+bug/412339
>>> https://code.launchpad.net/~ipython-dev/ipython/cpaste-fixes
>>
>> Ahh this is working perfectly. ?Many thanks. I wonder if the silent
>> mode in the original implementation was a design choice or a
>> convenience.
>
> I frequently consider "laziness" to be a design choice. But I do agree that
> echoing by default would be a good idea. It gives the user confidence that the
> pasted code is what the user thought it was.

As you wish, gentlemen:

In [3]: paste
x=1
y=2
print 'x,y',x,y
## -- End pasted text --
x,y 1 2

In [4]: paste -q
x,y 1 2


You can pull again from the branch.

Cheers,

f


From matthieu.brucher at gmail.com  Thu Aug 13 02:59:13 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 13 Aug 2009 08:59:13 +0200
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <6ce0ac130908120917m6dd5a68fq67bd22fade7ce47@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<6ce0ac130908091201n3d71f3e0mb168f774bf9a8836@mail.gmail.com>
	<e76aa17f0908091227y609e98a9h47be368c9f0a5548@mail.gmail.com>
	<6ce0ac130908101013j466eb860yfb087edf96aff9fb@mail.gmail.com>
	<e76aa17f0908101122k57f2300fo7fe26ae0938f5c7e@mail.gmail.com>
	<e76aa17f0908110506p1c6bb232nc1901fb09be1319@mail.gmail.com>
	<e76aa17f0908120015n29478a18qbeb29f71ed7185e1@mail.gmail.com>
	<6ce0ac130908120258y66547ee5x77b4453beae5fecc@mail.gmail.com>
	<e76aa17f0908120806y2547f31fq52f5ba83043e3708@mail.gmail.com>
	<6ce0ac130908120917m6dd5a68fq67bd22fade7ce47@mail.gmail.com>
Message-ID: <e76aa17f0908122359t85c684exac86efb1ad6d213f@mail.gmail.com>

2009/8/12 Brian Granger <ellisonbg.net at gmail.com>:
> Matthieu,
>
> Can you do the following test:
>
> headnode> ipcontroller
>
> # Then copy the engines furl file to the compute node and do in a separate
> terminal:
>
> computenode> ipengine --furl-file=[path to the furl file]
>
> If that doesn't work, it is either:
>
> * IP address related issue.? Play with ifconfig and ipcontroller --engine-ip
> * Firewall.? But you said this wasn't an issue.

OK, I have some improvements here:
- I can connect to the controller, but I have first to copy the furl:
$HOME on the headnode is not available on the computenode
- with ipcontroller-engine.furl, it seems to be working.

So the only issue is that I need to copy this furl file to the node,
as they can't access the file. I may have had some furl files left
inside that folder, which lead to the issue at hand. This should be
done with --engine-furl-file=... I suppose? Is there a way of exposing
it inside ipcluster, with kernel_config perhaps?

Cheers,

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From ellisonbg.net at gmail.com  Thu Aug 13 04:10:25 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 13 Aug 2009 01:10:25 -0700
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <e76aa17f0908122359t85c684exac86efb1ad6d213f@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<e76aa17f0908091227y609e98a9h47be368c9f0a5548@mail.gmail.com>
	<6ce0ac130908101013j466eb860yfb087edf96aff9fb@mail.gmail.com>
	<e76aa17f0908101122k57f2300fo7fe26ae0938f5c7e@mail.gmail.com>
	<e76aa17f0908110506p1c6bb232nc1901fb09be1319@mail.gmail.com>
	<e76aa17f0908120015n29478a18qbeb29f71ed7185e1@mail.gmail.com>
	<6ce0ac130908120258y66547ee5x77b4453beae5fecc@mail.gmail.com>
	<e76aa17f0908120806y2547f31fq52f5ba83043e3708@mail.gmail.com>
	<6ce0ac130908120917m6dd5a68fq67bd22fade7ce47@mail.gmail.com>
	<e76aa17f0908122359t85c684exac86efb1ad6d213f@mail.gmail.com>
Message-ID: <6ce0ac130908130110y7402edfbs18480f22c1fd4092@mail.gmail.com>

> OK, I have some improvements here:
> - I can connect to the controller, but I have first to copy the furl:
> $HOME on the headnode is not available on the computenode
> - with ipcontroller-engine.furl, it seems to be working.
>

Great, that is definitely progress!


> So the only issue is that I need to copy this furl file to the node,
> as they can't access the file. I may have had some furl files left
> inside that folder, which lead to the issue at hand. This should be
> done with --engine-furl-file=... I suppose? Is there a way of exposing
> it inside ipcluster, with kernel_config perhaps?
>

Yes, stale furl files can definitely cause problems if you are not using
persistent furl files.  Currently, ipcluster assumes that you will be using
a shared file system that all the engines and controller can read in the
same location.  Thus, we don't yet expose options like this to ipcluster.
BUT, we are in the middle of completely refactoring IPython's config
system.  Part of that work will be to make ipcluster more configurable.
This should happen sometime in the next 2 months.  In the meantime if you
have patches to add such an option to ipcluster, that would be fine.  Keep
us posted.

Cheers,

Brian



>
> Cheers,
>
> Matthieu
> --
> Information System Engineer, Ph.D.
> Website: http://matthieu-brucher.developpez.com/
> Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090813/63aa0a9f/attachment.html>

From matthieu.brucher at gmail.com  Thu Aug 13 04:19:57 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 13 Aug 2009 10:19:57 +0200
Subject: [IPython-dev] Before a patch for LSF support
In-Reply-To: <6ce0ac130908130110y7402edfbs18480f22c1fd4092@mail.gmail.com>
References: <e76aa17f0908050728g5b8a24edxf7e6e9e8ae3da1c6@mail.gmail.com>
	<6ce0ac130908101013j466eb860yfb087edf96aff9fb@mail.gmail.com>
	<e76aa17f0908101122k57f2300fo7fe26ae0938f5c7e@mail.gmail.com>
	<e76aa17f0908110506p1c6bb232nc1901fb09be1319@mail.gmail.com>
	<e76aa17f0908120015n29478a18qbeb29f71ed7185e1@mail.gmail.com>
	<6ce0ac130908120258y66547ee5x77b4453beae5fecc@mail.gmail.com>
	<e76aa17f0908120806y2547f31fq52f5ba83043e3708@mail.gmail.com>
	<6ce0ac130908120917m6dd5a68fq67bd22fade7ce47@mail.gmail.com>
	<e76aa17f0908122359t85c684exac86efb1ad6d213f@mail.gmail.com>
	<6ce0ac130908130110y7402edfbs18480f22c1fd4092@mail.gmail.com>
Message-ID: <e76aa17f0908130119l138b6170x783464a70746f473@mail.gmail.com>

2009/8/13 Brian Granger <ellisonbg.net at gmail.com>:
>
>> OK, I have some improvements here:
>> - I can connect to the controller, but I have first to copy the furl:
>> $HOME on the headnode is not available on the computenode
>> - with ipcontroller-engine.furl, it seems to be working.
>
> Great, that is definitely progress!
>
>>
>> So the only issue is that I need to copy this furl file to the node,
>> as they can't access the file. I may have had some furl files left
>> inside that folder, which lead to the issue at hand. This should be
>> done with --engine-furl-file=... I suppose? Is there a way of exposing
>> it inside ipcluster, with kernel_config perhaps?
>
> Yes, stale furl files can definitely cause problems if you are not using
> persistent furl files.? Currently, ipcluster assumes that you will be using
> a shared file system that all the engines and controller can read in the
> same location.? Thus, we don't yet expose options like this to ipcluster.
> BUT, we are in the middle of completely refactoring IPython's config
> system.? Part of that work will be to make ipcluster more configurable.
> This should happen sometime in the next 2 months.? In the meantime if you
> have patches to add such an option to ipcluster, that would be fine.? Keep
> us posted.

Excellent ! I saw a mail of Fernando on this. I will need some time to
help you with this.

Meanwhile, I will try to use ipcluster from a node to another LSF
node. I have still some issues with the IP address: when I'm starting
ipcluster from a compute node, the furl points to 127.0.0.1 instead of
the name of the host or at least its public IP address.

cheers,

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From ellisonbg.net at gmail.com  Thu Aug 13 20:01:48 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 13 Aug 2009 17:01:48 -0700
Subject: [IPython-dev] Status of summer work on IPython
Message-ID: <6ce0ac130908131701q284a8d2ai719ea736427c1616@mail.gmail.com>

Hello all,

My summer work on IPython is progressing nicely and I wanted to give
everyone a status update in case you aren't following the branches.  Here is
where we are today:

1.  IPython's modules and packages have been completely re-organized.  This
branch was just merged into trunk and should really help us to get our code
base in better shape.

2.  I just put up a branch "inputhook" for review.  This is a complete
refactor of IPython's integration with GUI event loop that uses the
PyOS_InputHook.  This is a huge improvement over the previous situation and
give us:

https://code.launchpad.net/~ipython-dev/ipython/inputhook

* Ctrl-C that just works
* You can dynamically select, turn-off, swtich the enabled GUI toolkit at
runtime.  It should not be possible to support these things in packages like
matplotlib.
* You can in some cases get multple GUI toolkits working.  On the Mac, I can
do Wx+Qt4 with no problems.  This (as expected) though is a bit subtle and
depends on lots of things.

We are still finalizing the API for all of this, but the core capabilties
are ready for people to being playing with.  My goal is to merge this
quickly so I can finalize the API with people at SciPy.

3.  In a new branch, I have been setting the new foundation for a refactored
IPython.  This is very exciting work and it should give us a very solid
framework for developing IPython forward.  Some of the highlights:
Traitlets, Component, Application, ConfigLoader, oh my!  This is not ready
for public consumption yet, but feel free to have a look at where I am
headed with this.

https://code.launchpad.net/~ipython-dev/ipython/config-refactor

Cheers,

Brian



Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090813/ea0afd0d/attachment.html>

From ellisonbg.net at gmail.com  Thu Aug 13 20:01:48 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 13 Aug 2009 17:01:48 -0700
Subject: [IPython-dev] Status of summer work on IPython
Message-ID: <6ce0ac130908131701q284a8d2ai719ea736427c1616@mail.gmail.com>

Hello all,

My summer work on IPython is progressing nicely and I wanted to give
everyone a status update in case you aren't following the branches.  Here is
where we are today:

1.  IPython's modules and packages have been completely re-organized.  This
branch was just merged into trunk and should really help us to get our code
base in better shape.

2.  I just put up a branch "inputhook" for review.  This is a complete
refactor of IPython's integration with GUI event loop that uses the
PyOS_InputHook.  This is a huge improvement over the previous situation and
give us:

https://code.launchpad.net/~ipython-dev/ipython/inputhook

* Ctrl-C that just works
* You can dynamically select, turn-off, swtich the enabled GUI toolkit at
runtime.  It should not be possible to support these things in packages like
matplotlib.
* You can in some cases get multple GUI toolkits working.  On the Mac, I can
do Wx+Qt4 with no problems.  This (as expected) though is a bit subtle and
depends on lots of things.

We are still finalizing the API for all of this, but the core capabilties
are ready for people to being playing with.  My goal is to merge this
quickly so I can finalize the API with people at SciPy.

3.  In a new branch, I have been setting the new foundation for a refactored
IPython.  This is very exciting work and it should give us a very solid
framework for developing IPython forward.  Some of the highlights:
Traitlets, Component, Application, ConfigLoader, oh my!  This is not ready
for public consumption yet, but feel free to have a look at where I am
headed with this.

https://code.launchpad.net/~ipython-dev/ipython/config-refactor

Cheers,

Brian



Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090813/ea0afd0d/attachment-0001.html>

From gokhansever at gmail.com  Sun Aug 16 20:48:43 2009
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Sun, 16 Aug 2009 19:48:43 -0500
Subject: [IPython-dev] Error building the docs
Message-ID: <49d6b3500908161748i5fddb91cu64dd29f30d4bf829@mail.gmail.com>

Hello,

Seemingly there is a failure somewhere in the IPy trunk which breaks the
documentation compilation

132 files written
Build API docs finished.
mkdir -p build/html build/doctrees
sphinx-build -b html -d build/doctrees   source build/html
Running Sphinx v0.6.2
WARNING: extension 'ipython_console_highlighting' has no setup() function;
is it really a Sphinx extension module?
loading pickled environment... not found
building [html]: targets for 168 source files that are out of date
updating environment: 168 added, 0 changed, 0 removed
reading sources... [  5%]
api/generated/IPython.core.fakemodule

Exception occurred:
  File
"/usr/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/environment.py",
line 656, in read_doc
    pickle.dump(doctree, f, pickle.HIGHEST_PROTOCOL)
PicklingError: Can't pickle <type 'module'>: attribute lookup
__builtin__.module failed
The full traceback has been saved in /tmp/sphinx-err-5zWYds.log, if you want
to report the issue to the author.
Please also report this if it was a user error, so that a better error
message can be provided next time.
Send reports to sphinx-dev at googlegroups.com. Thanks!
make: *** [html] Error 1


This is the full traceback log:

Traceback (most recent call last):
  File
"/usr/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/cmdline.py",
line 172, in main
    app.build(all_files, filenames)
  File
"/usr/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/application.py",
line 130, in build
    self.builder.build_update()
  File
"/usr/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/builders/__init__.py",
line 265, in build_update
    'out of date' % len(to_build))
  File
"/usr/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/builders/__init__.py",
line 285, in build
    purple, length):
  File
"/usr/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/builders/__init__.py",
line 131, in status_iterator
    for item in iterable:
  File
"/usr/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/environment.py",
line 519, in update_generator
    self.read_doc(docname, app=app)
  File
"/usr/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/environment.py",
line 656, in read_doc
    pickle.dump(doctree, f, pickle.HIGHEST_PROTOCOL)
PicklingError: Can't pickle <type 'module'>: attribute lookup
__builtin__.module failed



-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090816/fa9dad23/attachment.html>

From ellisonbg.net at gmail.com  Mon Aug 17 18:38:17 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 17 Aug 2009 15:38:17 -0700
Subject: [IPython-dev] Why do self.readline = readline
Message-ID: <6ce0ac130908171538w298dfa69ne4ea314d7603aa2e@mail.gmail.com>

Hi,

In a couple of places in IPython's code base (InteractiveShell,
IPCompleter), we do this:

import IPython.utils.rlineimpl as readline
self.readline = readline

Is there a reason for this?  The only reason I can think of is to make sure
that we point to the right readline always, even if a user imports the wrong
one later.

Is this thinking correct?  Can we get rid of this and just use readline as a
normal module?

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090817/1b63954b/attachment.html>

From ellisonbg.net at gmail.com  Mon Aug 17 18:38:17 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 17 Aug 2009 15:38:17 -0700
Subject: [IPython-dev] Why do self.readline = readline
Message-ID: <6ce0ac130908171538w298dfa69ne4ea314d7603aa2e@mail.gmail.com>

Hi,

In a couple of places in IPython's code base (InteractiveShell,
IPCompleter), we do this:

import IPython.utils.rlineimpl as readline
self.readline = readline

Is there a reason for this?  The only reason I can think of is to make sure
that we point to the right readline always, even if a user imports the wrong
one later.

Is this thinking correct?  Can we get rid of this and just use readline as a
normal module?

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090817/1b63954b/attachment-0001.html>

From ellisonbg.net at gmail.com  Mon Aug 17 23:32:16 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 17 Aug 2009 20:32:16 -0700
Subject: [IPython-dev] Limitations of log re-playing
Message-ID: <6ce0ac130908172032x683a5d52n442c929674b64c33@mail.gmail.com>

Hi,

Currently when an IPython log is saved, it contains the command line
arguments and options that were passed to IPython.  This allow a log that is
being re-played later to have be run with the same command line arguments
and options.  While this idea is quite nice, I think we have a *very* leaky
abstraction that needs to be plugged in some manner.  Why do I say this is
leaky:

* This assumes that the log is created by and re-played by the "ipython"
command line program.  As I refactor IPython, there are already becoming
*many* ways of running an InteractiveShell instance other than this.  In
these other contexts, the "command line arguments" simply don't make sense
anymore.

* The log re-playing code is currently in ipmaker.py.  This is going away
entirely.  Of course, the log replay capability doesn't have to go away.
The best place to put it is in InteractiveShell or some other helper
component that manages logs.  But, putting it there further removes it from
the traditional command line context.

* The command line information is not a complete specification of the state
of IPython that was used to create the log.  Namely config file or other
runtime config changes could have dramatically changed the behavior and
those things might not be reflected in the log.

So, what should we do with IPython's log replaying capability:

* Just remove the saving of the command line args/opts when a log is saved.
This is my preference.
* Explore some other, command line independent way of storing IPython's
state in the log file.
* Get rid of log replaying all together.
* Something else I haven't thought of.

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090817/70baaf00/attachment.html>

From stefan at sun.ac.za  Wed Aug 19 18:06:30 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 19 Aug 2009 15:06:30 -0700
Subject: [IPython-dev] IPython.ipapi -> IPython.core.ipapi
Message-ID: <9457e7c80908191506w5f3df8c8id9eedfd226119ba8@mail.gmail.com>

Hi all,

A patch for replacing

import IPython.ipapi

with

import IPython.core.ipapi

This gets rid of the warning currently displayed in trunk.

BTW, I noticed the unit tests are broken on dev trunk.  Is that usual,
or should I look into fixing it?

Cheers
St?fan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: import_ipapi_from_core.patch
Type: application/octet-stream
Size: 10399 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090819/d955c76d/attachment.obj>

From robert.kern at gmail.com  Wed Aug 19 19:39:29 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 19 Aug 2009 16:39:29 -0700
Subject: [IPython-dev] A plug for grin
Message-ID: <h6i2fi$ldl$1@ger.gmane.org>

[disclaimer: I wrote grin.]

Given Brian's current refactoring work, I would like to point out that my 
grep-alike tool, grin, has an example script that goes with it called 
grinimports which search normalized import statements in Python code. It might 
help people migrate their code to the new layout.

   http://pypi.python.org/pypi/grin

grinimports.py is in examples/.

For example, in my partially fixed ~/.ipython/:

[.ipython]$ grinimports.py ipapi
./home_ipy_user_conf.py:
     1 : import IPython.ipapi
./ipy_profile_mydoctest.py:
     2 : from IPython import ipapi
./ipy_profile_sh.py:
     1 : from IPython import ipapi
./ipy_user_conf.py:
     2 : import IPython.core.ipapi
./my_completers.py:
     1 : import IPython.ipapi
./my_ipy_traits_completer.py:
     1 : from IPython.core.ipapi import TryNext
     2 : from IPython.core.ipapi import get as ipget

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From ellisonbg.net at gmail.com  Fri Aug 21 14:16:56 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Fri, 21 Aug 2009 11:16:56 -0700
Subject: [IPython-dev] Dropping 2.4 support
Message-ID: <6ce0ac130908211116j7080ba39u55a746cde88547a@mail.gmail.com>

Hi,

Darren Dale and I were just talking about IPython's refactoring effort.  He
asked how committed we are to providing 2.4 support moving forward (post
0.10).  There is no doubt that we would gain *a lot* by giving up 2.4
support.  How do folks feel about this?

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090821/f4660b46/attachment.html>

From fperez.net at gmail.com  Fri Aug 21 14:30:07 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 21 Aug 2009 11:30:07 -0700
Subject: [IPython-dev] Dropping 2.4 support
In-Reply-To: <6ce0ac130908211116j7080ba39u55a746cde88547a@mail.gmail.com>
References: <6ce0ac130908211116j7080ba39u55a746cde88547a@mail.gmail.com>
Message-ID: <db6b5ecc0908211130l34d11d29q795f6373dcfdef9f@mail.gmail.com>

On Fri, Aug 21, 2009 at 11:16 AM, Brian Granger<ellisonbg.net at gmail.com> wrote:
>
> Darren Dale and I were just talking about IPython's refactoring effort.? He
> asked how committed we are to providing 2.4 support moving forward (post
> 0.10).? There is no doubt that we would gain *a lot* by giving up 2.4
> support.? How do folks feel about this?

I'm inclined to want to drop it, but I'd like to make it a reasoned
decision.  Let's try to make a brief list of:

- Specific things we gain from being 2.5-only (not the "what's new in
2.5" document, but which of those do *we* benefit from)

- Which major forms of python distribution (linux distros, defaults on
solaris or other big unix systems, versions of osx, etc) default to
2.4?

At least that way we'll be able to weigh the costs and benefits and
document, if we take the step, why we do it.

Cheers,

f


From gael.varoquaux at normalesup.org  Fri Aug 21 14:51:08 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 21 Aug 2009 20:51:08 +0200
Subject: [IPython-dev] Dropping 2.4 support
In-Reply-To: <db6b5ecc0908211130l34d11d29q795f6373dcfdef9f@mail.gmail.com>
References: <6ce0ac130908211116j7080ba39u55a746cde88547a@mail.gmail.com>
	<db6b5ecc0908211130l34d11d29q795f6373dcfdef9f@mail.gmail.com>
Message-ID: <20090821185108.GA5009@phare.normalesup.org>

On Fri, Aug 21, 2009 at 11:30:07AM -0700, Fernando Perez wrote:
> - Which major forms of python distribution (linux distros, defaults on
> solaris or other big unix systems, versions of osx, etc) default to
> 2.4?

Not the current RHEL, apparently:
https://www.redhat.com/archives/fedora-python-devel-list/2009-August/msg00001.html

Ga?l


From ellisonbg.net at gmail.com  Fri Aug 21 15:04:28 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Fri, 21 Aug 2009 12:04:28 -0700
Subject: [IPython-dev] Dropping 2.4 support
In-Reply-To: <db6b5ecc0908211130l34d11d29q795f6373dcfdef9f@mail.gmail.com>
References: <6ce0ac130908211116j7080ba39u55a746cde88547a@mail.gmail.com>
	<db6b5ecc0908211130l34d11d29q795f6373dcfdef9f@mail.gmail.com>
Message-ID: <6ce0ac130908211204l777ad9c7ma4a97fd2abe555f9@mail.gmail.com>

I'm inclined to want to drop it, but I'd like to make it a reasoned
> decision.  Let's try to make a brief list of:
>
> - Specific things we gain from being 2.5-only (not the "what's new in
> 2.5" document, but which of those do *we* benefit from)
>

* Absolute imports would allow us to ship enthought.traits in externals and
avoid 1) the performance problems of namespace packages and 2) conflicting
with other installed versions of enthought.traits.

* Certain parts of twisted use the new generator features in 2.5.  Being
able to use these features of twisted would probably dramatically simplify
all of our twisted using test code.  This would be a *massive* improvement
in maintainability.

* Obviously there are all the other minor things like with and
try/except/else/finally, but these are convenience things.

* The faster we drop support for python 2.4 and 2.5, the easier moving to
py3k will be.  I am not yet proposing that we drop 2.5 support though.




>
> - Which major forms of python distribution (linux distros, defaults on
> solaris or other big unix systems, versions of osx, etc) default to
> 2.4?
>
> At least that way we'll be able to weigh the costs and benefits and
> document, if we take the step, why we do it.
>
> Cheers,
>
> f
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090821/67275627/attachment.html>

From gael.varoquaux at normalesup.org  Fri Aug 21 15:10:24 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 21 Aug 2009 21:10:24 +0200
Subject: [IPython-dev] Dropping 2.4 support
In-Reply-To: <6ce0ac130908211204l777ad9c7ma4a97fd2abe555f9@mail.gmail.com>
References: <6ce0ac130908211116j7080ba39u55a746cde88547a@mail.gmail.com>
	<db6b5ecc0908211130l34d11d29q795f6373dcfdef9f@mail.gmail.com>
	<6ce0ac130908211204l777ad9c7ma4a97fd2abe555f9@mail.gmail.com>
Message-ID: <20090821191024.GD5009@phare.normalesup.org>

On Fri, Aug 21, 2009 at 12:04:28PM -0700, Brian Granger wrote:
>      I'm inclined to want to drop it, but I'd like to make it a reasoned
>      decision. ?Let's try to make a brief list of:

>      - Specific things we gain from being 2.5-only (not the "what's new in
>      2.5" document, but which of those do *we* benefit from)

>    * Absolute imports would allow us to ship enthought.traits in externals
>    and avoid 1) the performance problems of namespace packages and 2)
>    conflicting with other installed versions of enthought.traits.

Yes. IMHO this one is huge. I have been pushing people a lot to use
relative imports.

G.


From fperez.net at gmail.com  Fri Aug 21 17:58:12 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 21 Aug 2009 14:58:12 -0700
Subject: [IPython-dev] Dropping 2.4 support
In-Reply-To: <6ce0ac130908211204l777ad9c7ma4a97fd2abe555f9@mail.gmail.com>
References: <6ce0ac130908211116j7080ba39u55a746cde88547a@mail.gmail.com> 
	<db6b5ecc0908211130l34d11d29q795f6373dcfdef9f@mail.gmail.com> 
	<6ce0ac130908211204l777ad9c7ma4a97fd2abe555f9@mail.gmail.com>
Message-ID: <db6b5ecc0908211458l4a646c77jd4012ecfba84f12a@mail.gmail.com>

On Fri, Aug 21, 2009 at 12:04 PM, Brian Granger<ellisonbg.net at gmail.com> wrote:

>> - Specific things we gain from being 2.5-only (not the "what's new in
>> 2.5" document, but which of those do *we* benefit from)
>
> * Absolute imports would allow us to ship enthought.traits in externals and
> avoid 1) the performance problems of namespace packages and 2) conflicting
> with other installed versions of enthought.traits.
>
> * Certain parts of twisted use the new generator features in 2.5.? Being
> able to use these features of twisted would probably dramatically simplify
> all of our twisted using test code.? This would be a *massive* improvement
> in maintainability.
>
> * Obviously there are all the other minor things like with and
> try/except/else/finally, but these are convenience things.
>
> * The faster we drop support for python 2.4 and 2.5, the easier moving to
> py3k will be.? I am not yet proposing that we drop 2.5 support though.


This is a good list for me.  Just to provide some context for the
others, here at scipy we've been having extensive discussions with
Enthought and the matplotlib team, on the possibility of using traits
without making it an external dependency.  For now we'll continue with
Brian's Traitlets work because:

- it gives us a pure-python implementation

- it serves as a 'review' of the Traits model, which is great in many
ways, but has accumulated some things that can probably be looked at
again with the benefit of hindsight.

But we'll enable the running of  our test suite both agianst Traitlets
AND against true Traits, to ensure we spot deviations in Traitlets
from the Traits API.

Once our own dust settles, we can revisit the question of whether to
swap out Traitlets for Traits, especially if it becomes possible to
remove the pure C dependencies in it after a cleanup and some possible
Cython work.

With this rationale, unless someone has a strong objection, I'm
inclined to allow Trunk to move forward to 2.5.  We can always
backport critical bugfixes to 0.10 and put out a 0.10.1 that would be
2.4-fixed, if such issues are reported (hopefully with patches).

We don't have to make this decision *now* though, so let's wait for a
few days in case someone has a valid objection that may force us to
revisit the question, before making a final commitment.

Thanks for the  feedback!

Cheers,

f


From dsdale24 at gmail.com  Fri Aug 21 18:44:44 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 21 Aug 2009 18:44:44 -0400
Subject: [IPython-dev] Dropping 2.4 support
In-Reply-To: <db6b5ecc0908211458l4a646c77jd4012ecfba84f12a@mail.gmail.com>
References: <6ce0ac130908211116j7080ba39u55a746cde88547a@mail.gmail.com>
	<db6b5ecc0908211130l34d11d29q795f6373dcfdef9f@mail.gmail.com>
	<6ce0ac130908211204l777ad9c7ma4a97fd2abe555f9@mail.gmail.com>
	<db6b5ecc0908211458l4a646c77jd4012ecfba84f12a@mail.gmail.com>
Message-ID: <a08e5f80908211544s42f7f48btd02f466c76f6a17f@mail.gmail.com>

On Fri, Aug 21, 2009 at 5:58 PM, Fernando Perez<fperez.net at gmail.com> wrote:
> On Fri, Aug 21, 2009 at 12:04 PM, Brian Granger<ellisonbg.net at gmail.com> wrote:
>
>>> - Specific things we gain from being 2.5-only (not the "what's new in
>>> 2.5" document, but which of those do *we* benefit from)
>>
>> * Absolute imports would allow us to ship enthought.traits in externals and
>> avoid 1) the performance problems of namespace packages and 2) conflicting
>> with other installed versions of enthought.traits.
>>
>> * Certain parts of twisted use the new generator features in 2.5.? Being
>> able to use these features of twisted would probably dramatically simplify
>> all of our twisted using test code.? This would be a *massive* improvement
>> in maintainability.
>>
>> * Obviously there are all the other minor things like with and
>> try/except/else/finally, but these are convenience things.
>>
>> * The faster we drop support for python 2.4 and 2.5, the easier moving to
>> py3k will be.? I am not yet proposing that we drop 2.5 support though.
>
>
> This is a good list for me. ?Just to provide some context for the
> others, here at scipy we've been having extensive discussions with
> Enthought and the matplotlib team, on the possibility of using traits
> without making it an external dependency. ?For now we'll continue with
> Brian's Traitlets work because:
>
> - it gives us a pure-python implementation
>
> - it serves as a 'review' of the Traits model, which is great in many
> ways, but has accumulated some things that can probably be looked at
> again with the benefit of hindsight.
>
> But we'll enable the running of ?our test suite both agianst Traitlets
> AND against true Traits, to ensure we spot deviations in Traitlets
> from the Traits API.
>
> Once our own dust settles, we can revisit the question of whether to
> swap out Traitlets for Traits, especially if it becomes possible to
> remove the pure C dependencies in it after a cleanup and some possible
> Cython work.
>
> With this rationale, unless someone has a strong objection, I'm
> inclined to allow Trunk to move forward to 2.5. ?We can always
> backport critical bugfixes to 0.10 and put out a 0.10.1 that would be
> 2.4-fixed, if such issues are reported (hopefully with patches).
>
> We don't have to make this decision *now* though, so let's wait for a
> few days in case someone has a valid objection that may force us to
> revisit the question, before making a final commitment.

The two reasons I suggested dropping 2.4 support were absolute imports
and py3 transition. I would like to know what Eric Jones and Robert
Kern think about the following: if enthought.traits imported
absolute_import, and used relative imports for all intra-traits
imports, it would allow ipython to include enthought_traits as a
(non-namespace) subpackage in ipython  with a simple patch to make it
pure python. Ipython would *not* install it as top level package, we
learned from our problems doing this in matplotlib. This could replace
Traitlets entirely, and Ipython could first try to import traits from
the top level, and fall back on the ipython.enthought_traits
subpackage if it fails. Ipython could still test using both the pure
python alternative and the full top-level traits package, and it would
be much easier (trivial) to track traits development and offer patches
for upstream.

Darren


From dwf at cs.toronto.edu  Sat Aug 22 06:29:17 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Sat, 22 Aug 2009 03:29:17 -0700
Subject: [IPython-dev] Dropping 2.4 support
In-Reply-To: <20090821185108.GA5009@phare.normalesup.org>
References: <6ce0ac130908211116j7080ba39u55a746cde88547a@mail.gmail.com>
	<db6b5ecc0908211130l34d11d29q795f6373dcfdef9f@mail.gmail.com>
	<20090821185108.GA5009@phare.normalesup.org>
Message-ID: <3AF9453A-4E16-4FBB-8453-B8713EA29561@cs.toronto.edu>

  21-Aug-09, at 11:51 AM, Gael Varoquaux wrote:

>
> Not the current RHEL, apparently:
> https://www.redhat.com/archives/fedora-python-devel-list/2009-August/msg00001.html

Just my 2c, which should be worth an awful lot less than 2c given that  
I've contributed approximately 5 lines to IPython: given how common it  
is as a cluster computing environment, and how one of the most  
compelling use cases of IPython is cluster parallelism, RHEL should  
count for a lot IMHO.

I personally usually build my own Python and libraries and get them  
installed in /opt, but for less savvy users... Maybe freezing an 0.10  
branch in bugfix only mode and clearly indicating RHEL/py2.4 users  
should use 0.10.x?

Cheers,

David


From Dwf at cs.toronto.edu  Tue Aug 25 16:05:48 2009
From: Dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 25 Aug 2009 16:05:48 -0400
Subject: [IPython-dev] Dropping 2.4 support
In-Reply-To: <3AF9453A-4E16-4FBB-8453-B8713EA29561@cs.toronto.edu>
References: <6ce0ac130908211116j7080ba39u55a746cde88547a@mail.gmail.com>
	<db6b5ecc0908211130l34d11d29q795f6373dcfdef9f@mail.gmail.com>
	<20090821185108.GA5009@phare.normalesup.org>
	<3AF9453A-4E16-4FBB-8453-B8713EA29561@cs.toronto.edu>
Message-ID: <5BEABD23-413E-4CA0-8FE3-3EDF000177AF@cs.toronto.edu>

On 22-Aug-09, at 6:29 AM, David Warde-Farley wrote:

> Just my 2c, which should be worth an awful lot less than 2c given that
> I've contributed approximately 5 lines to IPython: given how common it
> is as a cluster computing environment, and how one of the most
> compelling use cases of IPython is cluster parallelism, RHEL should
> count for a lot IMHO.
>
> I personally usually build my own Python and libraries and get them
> installed in /opt, but for less savvy users... Maybe freezing an 0.10
> branch in bugfix only mode and clearly indicating RHEL/py2.4 users
> should use 0.10.x?

Fernando, Brian, Gael and I had a conversation a few nights ago in  
person and for the sake of documenting it I promised I'd recount it  
here.

Basically, since Red Hat isn't shipping recent versions of IPython or  
any of the scientific toolstack it becomes increasingly unlikely that  
people are using the stock distribution Python but are building the  
whole stack from source but are unwilling to build a new Python as  
well. Freezing 0.10 as a stable endpoint for 2.4 users is likely a  
more than reasonable compromise.

Cheers,

David


From fperez.net at gmail.com  Tue Aug 25 21:03:23 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 25 Aug 2009 18:03:23 -0700
Subject: [IPython-dev] Dropping 2.4 support
In-Reply-To: <5BEABD23-413E-4CA0-8FE3-3EDF000177AF@cs.toronto.edu>
References: <6ce0ac130908211116j7080ba39u55a746cde88547a@mail.gmail.com> 
	<db6b5ecc0908211130l34d11d29q795f6373dcfdef9f@mail.gmail.com> 
	<20090821185108.GA5009@phare.normalesup.org>
	<3AF9453A-4E16-4FBB-8453-B8713EA29561@cs.toronto.edu> 
	<5BEABD23-413E-4CA0-8FE3-3EDF000177AF@cs.toronto.edu>
Message-ID: <db6b5ecc0908251803g5f9a30d0y2af6ef528fb49609@mail.gmail.com>

On Tue, Aug 25, 2009 at 1:05 PM, David Warde-Farley<Dwf at cs.toronto.edu> wrote:

> Fernando, Brian, Gael and I had a conversation a few nights ago in
> person and for the sake of documenting it I promised I'd recount it
> here.
>
> Basically, since Red Hat isn't shipping recent versions of IPython or
> any of the scientific toolstack it becomes increasingly unlikely that
> people are using the stock distribution Python but are building the
> whole stack from source but are unwilling to build a new Python as
> well. Freezing 0.10 as a stable endpoint for 2.4 users is likely a
> more than reasonable compromise.

Thanks David for the recap, it was great seeing you at the conference.

We have in fact an example of the  above at Berkeley, where CentOS
machines get a manually built full python  stack, because everything
in the stock system is just too old to do anything with.

In reality, my main concern now is to find the least-painful way
possible for matplotlib and ETS to sync with our changes, but we
should be able to provide code for them (the examples in our docs
already do much of this) to update the core parts where  they do GUI
operations to be compatible with ipython, whether a pre- or post- 0.10
version is installed.

Cheers,

f


From glenn at tarbox.org  Tue Aug 25 23:31:53 2009
From: glenn at tarbox.org (Glenn Tarbox, PhD)
Date: Tue, 25 Aug 2009 20:31:53 -0700
Subject: [IPython-dev] PySide to replace PyQt?
Message-ID: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>

Sooner or later, something was gonna need to happen WRT Riverbank and the
PyQt licensing.  I had hoped that an entirely new project wasn't going to be
necessary.... but apparently it is.

PySide Released to the Wild:
http://labs.trolltech.com/blogs/2009/08/25/pyside-released-to-the-wild/

>From the PySide FAQ

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
What about PyQt?

Nokia?s initial research into Python bindings for Qt involved speaking with
Riverbank Computing, the makers of PyQt. We had several discussions with
them to see if it was possible to use PyQt to achieve our goals.
Unfortunately, a common agreement could not be found , so in the end we
decided to proceed with PySide.

We will however maintain API compatibility with PyQt (you can use the same
method names but can?t inter-operate with PyQt), at least for the initial
release. To import PySide you have to use ?import PySide? instead of ?import
PyQt4?.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

I didn't know where to post this.  PySide needs to mature a bit (support
more than Linux for example) but both Matplotlib and Enthought are
affected.  PyQt will likely need to be replaced in both packages once PySide
becomes more mature as the licensing of PyQt is problematic now that Qt is
LGPL.  Its also likely that with Nokia's backing, the PySide API will
eventually dominate.

Hopefully, the above statement regarding the similarity of the API will make
moving over easy.  Personally I'd like to see a focus on Qt vs Wx by
Enthought as I believe it to be much more powerful... but thats my personal
opinion and what I use.

As a side note, I've successfully nailed my C++ Qt code to IPython using a
Cython shim.  The PyQt event loop is available and all seems to work great.

-glenn

-- 
Glenn H. Tarbox, PhD ||  206-274-6919
http://www.tarbox.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090825/82790965/attachment.html>

From ellisonbg.net at gmail.com  Wed Aug 26 14:19:04 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 26 Aug 2009 11:19:04 -0700
Subject: [IPython-dev] Getting InteractiveShell to clean up after itself
Message-ID: <6ce0ac130908261119y5e84c419y5e7782b0520e9c32@mail.gmail.com>

Hi,

Currently in my ipython-app branch, InteractiveShell is a real object that
can be instantiated:

ip = InteractiveShell()

But, when you do:

del ip
gc.collect()

The InteractiveShell.__del__ does *not* get called.  This is not too
surprising given how garbage collection
works in python:

*  We have cycles to InteractiveShell all over the place (everybody points
to it!)
*  In Python cycles + __del__ methods => some objects can't be collected
even when gc.collect() is run.

I would like to resolve this problem and I see two ways of accomplishing it:

1.  Begin to use weakrefs everywhere.  This would break all our cycles and
would probably be a good idea
anyways.  If we do this with discipline, we might be able to get a __del__
method called to do cleanup
(removing sys.*hook, removing things injected into __builtin__, etc.).

2.  Add a .cleanup() method that does all the cleanup by hand.  This won't
break cycles, but it would ensure
that the InteractiveShell had "detached" from everything outside of itself.
If we go this way, and drop 2.4
support, we could make InteractiveShell a context manager and have cleanup
called when the context
exists.

My goal is that we could create and destroy InteractiveShell instances (for
testing, etc.) whenever we want
or need and they would always clean up after themselves.

If we don't go this route, then I think we should make InteractiveShell a
singleton.

Thoughts?

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090826/f32fdd7c/attachment.html>

From fperez.net at gmail.com  Wed Aug 26 14:40:44 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 26 Aug 2009 11:40:44 -0700
Subject: [IPython-dev] Getting InteractiveShell to clean up after itself
In-Reply-To: <6ce0ac130908261119y5e84c419y5e7782b0520e9c32@mail.gmail.com>
References: <6ce0ac130908261119y5e84c419y5e7782b0520e9c32@mail.gmail.com>
Message-ID: <db6b5ecc0908261140s4bc0bc1dhc06a2b72a02fa43@mail.gmail.com>

Howdy,

On Wed, Aug 26, 2009 at 11:19 AM, Brian Granger<ellisonbg.net at gmail.com> wrote:
> Hi,
>
> Currently in my ipython-app branch, InteractiveShell is a real object that
> can be instantiated:
>
> ip = InteractiveShell()
>
> But, when you do:
>
> del ip
> gc.collect()
>
> The InteractiveShell.__del__ does *not* get called.? This is not too
> surprising given how garbage collection
> works in python:
>
> *? We have cycles to InteractiveShell all over the place (everybody points
> to it!)
> *? In Python cycles + __del__ methods => some objects can't be collected
> even when gc.collect() is run.
>
> I would like to resolve this problem and I see two ways of accomplishing it:
>
> 1.? Begin to use weakrefs everywhere.? This would break all our cycles and
> would probably be a good idea
> anyways.? If we do this with discipline, we might be able to get a __del__
> method called to do cleanup
> (removing sys.*hook, removing things injected into __builtin__, etc.).
>
> 2.? Add a .cleanup() method that does all the cleanup by hand.? This won't
> break cycles, but it would ensure
> that the InteractiveShell had "detached" from everything outside of itself.
> If we go this way, and drop 2.4
> support, we could make InteractiveShell a context manager and have cleanup
> called when the context
> exists.
>
> My goal is that we could create and destroy InteractiveShell instances (for
> testing, etc.) whenever we want
> or need and they would always clean up after themselves.
>
> If we don't go this route, then I think we should make InteractiveShell a
> singleton.
>
> Thoughts?

I need to run now, so rather hastily:

1. While I'm all for using more weakrefs, they are a subtle tool and I
doubt we can get away with using them everywhere for everything.  I
don't want to 'bet the farm' on that strategy, because we can easily
find a situation where weakrefs don't work, we have way too much
user-facing  state managemnet we must accomplish robustly and weakrefs
may not be the tool for everything we need to do.

2. This is my favorite.  I wouldn't make the shell necessarily a
context manager by itself yet, I'm not sure it's the right interface
(I tend to prefer objects that have small, well-defined interfaces by
themselves), but it would be easy, and good, to have a context manager
that manages a shell instance for us and cleans it up on exit.

How does this sound?

Cheers,
f


From ellisonbg.net at gmail.com  Wed Aug 26 15:17:06 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 26 Aug 2009 12:17:06 -0700
Subject: [IPython-dev] Getting InteractiveShell to clean up after itself
In-Reply-To: <db6b5ecc0908261140s4bc0bc1dhc06a2b72a02fa43@mail.gmail.com>
References: <6ce0ac130908261119y5e84c419y5e7782b0520e9c32@mail.gmail.com>
	<db6b5ecc0908261140s4bc0bc1dhc06a2b72a02fa43@mail.gmail.com>
Message-ID: <6ce0ac130908261217m4c785c24u66d9aa58a243a044@mail.gmail.com>

> 1. While I'm all for using more weakrefs, they are a subtle tool and I
> doubt we can get away with using them everywhere for everything.  I
> don't want to 'bet the farm' on that strategy, because we can easily
> find a situation where weakrefs don't work, we have way too much
> user-facing  state managemnet we must accomplish robustly and weakrefs
> may not be the tool for everything we need to do.
>

Yes, I agree.  When I proposed to use them "everywhere" I was actually
thinking quite narrowly.  Here is what I am thinking:

* Components will track children and parents using weakrefs.
* Component.get_instances will have an interface for getting a weakref.
Component holds the instances as weakref anyways,
so this is trivial.
* When Components need to hold onto a ref to each other they should be a
weakref.

Because InteractiveShell is a Component, no-one should hold a strong ref to
it under
this approach.


> 2. This is my favorite.  I wouldn't make the shell necessarily a
> context manager by itself yet, I'm not sure it's the right interface
> (I tend to prefer objects that have small, well-defined interfaces by
> themselves), but it would be easy, and good, to have a context manager
> that manages a shell instance for us and cleans it up on exit.
>

Yes, I like this.  For now, I am just adding a .cleanup method.  We can add
the context manager later.

Also, it would be very nice to be able to manage lots of other things with
"with":

io traps
sys hooks
threading locks (we are going to have to make InteractiveShell thread safe
with locks)
etc.

More evidence that dropping 2.4 support makes sense.


> How does this sound?
>

Great!  Thanks for the feedback.



>
> Cheers,
> f
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090826/d6379d78/attachment.html>

From robert.kern at gmail.com  Wed Aug 26 16:00:40 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 26 Aug 2009 15:00:40 -0500
Subject: [IPython-dev] Getting InteractiveShell to clean up after itself
In-Reply-To: <6ce0ac130908261217m4c785c24u66d9aa58a243a044@mail.gmail.com>
References: <6ce0ac130908261119y5e84c419y5e7782b0520e9c32@mail.gmail.com>	<db6b5ecc0908261140s4bc0bc1dhc06a2b72a02fa43@mail.gmail.com>
	<6ce0ac130908261217m4c785c24u66d9aa58a243a044@mail.gmail.com>
Message-ID: <h74499$il8$1@ger.gmane.org>

On 2009-08-26 14:17 PM, Brian Granger wrote:
>
>     1. While I'm all for using more weakrefs, they are a subtle tool and I
>     doubt we can get away with using them everywhere for everything.  I
>     don't want to 'bet the farm' on that strategy, because we can easily
>     find a situation where weakrefs don't work, we have way too much
>     user-facing  state managemnet we must accomplish robustly and weakrefs
>     may not be the tool for everything we need to do.
>
>
> Yes, I agree.  When I proposed to use them "everywhere" I was actually
> thinking quite narrowly.  Here is what I am thinking:
>
> * Components will track children and parents using weakrefs.

Usually, you need strong references in one direction to keep the whole graph alive.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From robert.kern at gmail.com  Wed Aug 26 16:45:36 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 26 Aug 2009 15:45:36 -0500
Subject: [IPython-dev] Getting InteractiveShell to clean up after itself
In-Reply-To: <db6b5ecc0908261140s4bc0bc1dhc06a2b72a02fa43@mail.gmail.com>
References: <6ce0ac130908261119y5e84c419y5e7782b0520e9c32@mail.gmail.com>
	<db6b5ecc0908261140s4bc0bc1dhc06a2b72a02fa43@mail.gmail.com>
Message-ID: <h746tg$r7t$1@ger.gmane.org>

On 2009-08-26 13:40 PM, Fernando Perez wrote:

> 1. While I'm all for using more weakrefs, they are a subtle tool and I
> doubt we can get away with using them everywhere for everything.  I
> don't want to 'bet the farm' on that strategy, because we can easily
> find a situation where weakrefs don't work, we have way too much
> user-facing  state managemnet we must accomplish robustly and weakrefs
> may not be the tool for everything we need to do.

In [11]: from enthought.traits.api import HasTraits, WeakRef

In [12]: class A(HasTraits):
    ....:     b = WeakRef()
    ....:

In [14]: b = HasTraits()

In [15]: a = A(b=b)

In [16]: print a.b
<enthought.traits.has_traits.HasTraits object at 0x18a1cf0>

In [17]: del b

In [18]: print a.b
None


Just sayin'.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From ellisonbg.net at gmail.com  Wed Aug 26 16:59:41 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 26 Aug 2009 13:59:41 -0700
Subject: [IPython-dev] Getting InteractiveShell to clean up after itself
In-Reply-To: <h74499$il8$1@ger.gmane.org>
References: <6ce0ac130908261119y5e84c419y5e7782b0520e9c32@mail.gmail.com>
	<db6b5ecc0908261140s4bc0bc1dhc06a2b72a02fa43@mail.gmail.com>
	<6ce0ac130908261217m4c785c24u66d9aa58a243a044@mail.gmail.com>
	<h74499$il8$1@ger.gmane.org>
Message-ID: <6ce0ac130908261359k4cc7c026i6ff333d863c7fdf7@mail.gmail.com>

> > * Components will track children and parents using weakrefs.
>
> Usually, you need strong references in one direction to keep the whole
> graph alive.
>

Good point.  I will watch this carefully when I implement this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090826/a7110f18/attachment.html>

From ellisonbg.net at gmail.com  Wed Aug 26 17:36:39 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 26 Aug 2009 14:36:39 -0700
Subject: [IPython-dev] Should IPython be a singleton?
Message-ID: <6ce0ac130908261436h73034f19mccc14893552db7c5@mail.gmail.com>

Currently, IPython is not a singleton, but effectively it is.  We do a
number of things that essentially prevent
multiple InteractiveShells from being created:

1.  We muck with sys.excepthook, sys.displayhook, sys.ipcompleter,
sys.stdin, sys.stdout

2.  We push a number of thing into __builtin__ that point to the specific
InteractiveShell:

__IPYTHON__
exit
quit
reload

3.  Because of severe cycles in our object graph, an InterativeShell
instances can't be
garbage collected.

So, should we actually make InteractiveShell a singleton?  There are a
couple of different
options:

1.  One InteractiveShell per process - a true singleton.
2.  One InteractiveShell at a time, but multiple (serial) ones in a process.
3.  Multiple, sumultanous InteractiveShells

The only subtlety is that some level of nesting is possible.

Just a note: it will take a lot of refactoring to really implement any of
the above options.
But, I want to know what we are aiming for as this issue affects many design
choices.

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090826/55df48a2/attachment.html>

From gael.varoquaux at normalesup.org  Wed Aug 26 17:41:21 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 26 Aug 2009 23:41:21 +0200
Subject: [IPython-dev] Should IPython be a singleton?
In-Reply-To: <6ce0ac130908261436h73034f19mccc14893552db7c5@mail.gmail.com>
References: <6ce0ac130908261436h73034f19mccc14893552db7c5@mail.gmail.com>
Message-ID: <20090826214121.GB28820@phare.normalesup.org>

On Wed, Aug 26, 2009 at 02:36:39PM -0700, Brian Granger wrote:
>    1.? One InteractiveShell per process - a true singleton.
>    2.? One InteractiveShell at a time, but multiple (serial) ones in a
>    process.
>    3.? Multiple, sumultanous InteractiveShells

When embedding in GUIs, option 1 limits the use of IPython. We have
already encountered this problem for instance when scripting Mayavi, from
IPython.

Ga?l


From robert.kern at gmail.com  Wed Aug 26 17:55:09 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 26 Aug 2009 16:55:09 -0500
Subject: [IPython-dev] Should IPython be a singleton?
In-Reply-To: <6ce0ac130908261436h73034f19mccc14893552db7c5@mail.gmail.com>
References: <6ce0ac130908261436h73034f19mccc14893552db7c5@mail.gmail.com>
Message-ID: <h74avv$98g$1@ger.gmane.org>

On 2009-08-26 16:36 PM, Brian Granger wrote:
> Currently, IPython is not a singleton, but effectively it is.  We do a
> number of things that essentially prevent
> multiple InteractiveShells from being created:
>
> 1.  We muck with sys.excepthook, sys.displayhook, sys.ipcompleter,
> sys.stdin, sys.stdout

That doesn't have to be a problem. That's why I wrote all of those *Trap 
classes. And as far as I can tell, sys.ipcompleter is entirely from IPython, not 
Python itself. I have no idea why it's in the sys namespace in the first place.

> 2.  We push a number of thing into __builtin__ that point to the
> specific InteractiveShell:
>
> __IPYTHON__
> exit
> quit
> reload

Could be Trapped, as well.

> 3.  Because of severe cycles in our object graph, an InterativeShell
> instances can't be
> garbage collected.

Or rather, they *are* garbage collected and not __del__eted.

> So, should we actually make InteractiveShell a singleton?  There are a
> couple of different
> options:
>
> 1.  One InteractiveShell per process - a true singleton.
> 2.  One InteractiveShell at a time, but multiple (serial) ones in a process.
> 3.  Multiple, sumultanous InteractiveShells
>
> The only subtlety is that some level of nesting is possible.
>
> Just a note: it will take a lot of refactoring to really implement any
> of the above options.
> But, I want to know what we are aiming for as this issue affects many
> design choices.

#3 would be really, really desirable, and I think it is feasible.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From ellisonbg.net at gmail.com  Wed Aug 26 18:19:47 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 26 Aug 2009 15:19:47 -0700
Subject: [IPython-dev] Should IPython be a singleton?
In-Reply-To: <h74avv$98g$1@ger.gmane.org>
References: <6ce0ac130908261436h73034f19mccc14893552db7c5@mail.gmail.com>
	<h74avv$98g$1@ger.gmane.org>
Message-ID: <6ce0ac130908261519n2f32175cw1a356ab1bbdfc547@mail.gmail.com>

> > 1.  We muck with sys.excepthook, sys.displayhook, sys.ipcompleter,
> > sys.stdin, sys.stdout
>
> That doesn't have to be a problem. That's why I wrote all of those *Trap
> classes. And as far as I can tell, sys.ipcompleter is entirely from
> IPython, not
> Python itself. I have no idea why it's in the sys namespace in the first
> place.
>

Yes, your trap classs would help manage all of this for sure.  But we have
a long way to go in the core before the mucking with sys is done in a safe
manner.

As far as ipcompleter, I haven't tracked down exactly why it is there.  But
I do know it is used
from that location in the embedded shell.  But, a proper API should fix
that.

> 2.  We push a number of thing into __builtin__ that point to the
> specific InteractiveShell:
>
> __IPYTHON__
> exit
> quit
> reload

Could be Trapped, as well.
>

Yes.


>
> > 3.  Because of severe cycles in our object graph, an InterativeShell
> > instances can't be
> > garbage collected.
>
> Or rather, they *are* garbage collected and not __del__eted.
>

I have spent some time with gc and muppy/pympler and as of right now, it is
not even garbage collected.  I create and destroy an InteractiveShell
instance
from python (not IPython) and muppy/gc still list it in the live objects.

I haven't had a change to figure out who the offending ref holder is - there

are dozens of them.

> So, should we actually make InteractiveShell a singleton?  There are a
> couple of different
> options:
>
> 1.  One InteractiveShell per process - a true singleton.
> 2.  One InteractiveShell at a time, but multiple (serial) ones in a
process.
> 3.  Multiple, sumultanous InteractiveShells
>
> The only subtlety is that some level of nesting is possible.
>
> Just a note: it will take a lot of refactoring to really implement any
> of the above options.
> But, I want to know what we are aiming for as this issue affects many
> design choices.

#3 would be really, really desirable, and I think it is feasible.
>

I agree, I think it will take some time to refactor things in place and get
this to
be possible.

Cheers,

Brian



>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma
>  that is made terrible by our own mad attempt to interpret it as though it
> had
>  an underlying truth."
>   -- Umberto Eco
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090826/efcb4d49/attachment.html>

From robert.kern at gmail.com  Wed Aug 26 18:45:32 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 26 Aug 2009 17:45:32 -0500
Subject: [IPython-dev] Should IPython be a singleton?
In-Reply-To: <6ce0ac130908261519n2f32175cw1a356ab1bbdfc547@mail.gmail.com>
References: <6ce0ac130908261436h73034f19mccc14893552db7c5@mail.gmail.com>	<h74avv$98g$1@ger.gmane.org>
	<6ce0ac130908261519n2f32175cw1a356ab1bbdfc547@mail.gmail.com>
Message-ID: <h74duf$hvp$1@ger.gmane.org>

On 2009-08-26 17:19 PM, Brian Granger wrote:

>      > 3.  Because of severe cycles in our object graph, an InterativeShell
>      > instances can't be
>      > garbage collected.
>
>     Or rather, they *are* garbage collected and not __del__eted.
>
>
> I have spent some time with gc and muppy/pympler and as of right now, it is
> not even garbage collected.  I create and destroy an InteractiveShell
> instance
> from python (not IPython) and muppy/gc still list it in the live objects.
>
> I haven't had a change to figure out who the offending ref holder is -
> there
> are dozens of them.

When tracking down problems like this, one thing you have to be very careful 
about is not creating references via your instrumentation. It's disturbingly 
easy to do. :-)

__builtins__.__IPYTHON__ and __builtins__.__IPYTHON__active are two culprits, 
and the remaining one is a cell object. I'm not sure where he is coming from.


[~]$ python foo.py
<IPython.core.iplib.InteractiveShell object at 0x3aad90>
<IPython.core.iplib.InteractiveShell object at 0x3aad90>
40 references:
<type 'instancemethod'>
<bound method InteractiveShell.generate_output_prompt of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.shell_hook of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.show_in_pager of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.pre_prompt_hook of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.pre_runcode_hook of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.clipboard_get of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'dict'>
A dict with keys: ['app_name', 'IP', 'crash_report_fname', 
'user_message_template', 'show_crash_traceback', 'contact_email', 'bug_tracker', 
'contact_name']

<type 'instancemethod'>
<bound method InteractiveShell.dummy_handler of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.ipmagic of <IPython.core.iplib.InteractiveShell 
object at 0x3aad90>>

<type 'dict'>
A dict with keys: ['magic', 'IP', 'user_ns', 'user_global_ns', 'system', 'dbg', 
'meta', 'extensions', 'set_crash_handler', 'set_custom_exc', 'set_hook']

<type 'instancemethod'>
<bound method InteractiveShell.set_hook of <IPython.core.iplib.InteractiveShell 
object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.set_custom_exc of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.set_crash_handler of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.set_hook of <IPython.core.iplib.InteractiveShell 
object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.ipmagic of <IPython.core.iplib.InteractiveShell 
object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.ipalias of <IPython.core.iplib.InteractiveShell 
object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.ipsystem of <IPython.core.iplib.InteractiveShell 
object at 0x3aad90>>

<type 'list'>
A list of 19524 elements; probably gc.get_objects().

<type 'cell'>
A cell: <cell at 0x3aab90: InteractiveShell object at 0x3aad90>
<IPython.core.iplib.InteractiveShell object at 0x3aad90>

<type 'dict'>
A dict with keys: ['dir_stack', 'ESC_HELP', 'pycolorize', 'SyntaxTB', 
'indent_current_nsp', 'hooks', 'user_ns', 'user_global_ns', 'system', 
'_main_ns_cache', 'dir_hist', 'autoindent', 'tempfiles', 'BANNER', 
'code_to_run', 'loghead_tpl', 'ESC_PAREN', 'stdin_encoding', 'usage_min', 
'has_readline', 'ns_table', 'getoutput', 'no_alias', 'home_dir', 'filename', 
'ESC_SH_CAP', 'ns_refs_table', 'rc', 'usage', 'starting_dir', 'logger', 
'banner2', 'options_table', 'more', 'shell', 'jobs', 'auto_alias', 'api', 
'buffer', 'ESC_SHELL', 'alias_table', 'InteractiveTB', 'exit_now', 
'builtins_added', 'output_hist', 'internal_ns', 'user_config_ns', 'ESC_QUOTE2', 
'custom_exceptions', 'pager', 'sys_excepthook', 'getoutputerror', 
'esc_handlers', 'input_hist_raw', 'name', 'embedded', 'CACHELENGTH', 
'strdispatchers', 'CustomTB', '_magic_state', 'compile', 'input_hist', 'meta', 
'_user_main_module', 'ESC_MAGIC', 'ESC_QUOTE']

<type 'dict'>
A dict with keys: ['shell', 'name']

<type 'dict'>
A dict with keys: ['shell', 'name']

<type 'dict'>
A dict with keys: ['_dh', '_sh', '__builtins__', 'In', '_ip', '_ih', '__name__', 
'foo', '_oh', 'Out']

<type 'instancemethod'>
<bound method InteractiveShell.handle_auto of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.handle_auto of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.handle_auto of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.handle_magic of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.handle_help of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.handle_shell_escape of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.handle_shell_escape of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.generate_prompt of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'dict'>
A dict with keys: ['log_active', 'shell', '_i', 'log_raw_input', '_i00', '_iii', 
'loghead', 'log_output', 'logfname', '_ii', 'timestamp', '_logmode', 'logfile']

<type 'instancemethod'>
<bound method InteractiveShell.shutdown_hook of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.result_display of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.synchronize_with_editor of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.editor of <IPython.core.iplib.InteractiveShell 
object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.fix_error_editor of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.input_prefilter of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'instancemethod'>
<bound method InteractiveShell.late_startup_hook of 
<IPython.core.iplib.InteractiveShell object at 0x3aad90>>

<type 'dict'>
A dict with keys: ['obj', 'InteractiveShell', '__builtins__', '__file__', 'gc', 
'refs', '__name__', 'ref', '__doc__', 'types']


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From dwf at cs.toronto.edu  Wed Aug 26 19:11:24 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 26 Aug 2009 19:11:24 -0400
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>
Message-ID: <F1521FE2-6933-4F04-9145-A6F540DC8435@cs.toronto.edu>

This was heavily discussed at the conference last week.

It seems as though Phil Thompson might have painted himself into a  
corner.  At any rate, I for one welcome our new Finnish overlords.

David

On 25-Aug-09, at 11:31 PM, Glenn Tarbox, PhD wrote:

> Sooner or later, something was gonna need to happen WRT Riverbank  
> and the
> PyQt licensing.  I had hoped that an entirely new project wasn't  
> going to be
> necessary.... but apparently it is.
>
> PySide Released to the Wild:
> http://labs.trolltech.com/blogs/2009/08/25/pyside-released-to-the- 
> wild/
>
>> From the PySide FAQ
>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> What about PyQt?
>
> Nokia?s initial research into Python bindings for Qt involved  
> speaking with
> Riverbank Computing, the makers of PyQt. We had several discussions  
> with
> them to see if it was possible to use PyQt to achieve our goals.
> Unfortunately, a common agreement could not be found , so in the end  
> we
> decided to proceed with PySide.
>
> We will however maintain API compatibility with PyQt (you can use  
> the same
> method names but can?t inter-operate with PyQt), at least for the  
> initial
> release. To import PySide you have to use ?import PySide? instead  
> of ?import
> PyQt4?.
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>
> I didn't know where to post this.  PySide needs to mature a bit  
> (support
> more than Linux for example) but both Matplotlib and Enthought are
> affected.  PyQt will likely need to be replaced in both packages  
> once PySide
> becomes more mature as the licensing of PyQt is problematic now that  
> Qt is
> LGPL.  Its also likely that with Nokia's backing, the PySide API will
> eventually dominate.
>
> Hopefully, the above statement regarding the similarity of the API  
> will make
> moving over easy.  Personally I'd like to see a focus on Qt vs Wx by
> Enthought as I believe it to be much more powerful... but thats my  
> personal
> opinion and what I use.
>
> As a side note, I've successfully nailed my C++ Qt code to IPython  
> using a
> Cython shim.  The PyQt event loop is available and all seems to work  
> great.
>
> -glenn
>
> -- 
> Glenn H. Tarbox, PhD ||  206-274-6919
> http://www.tarbox.org
> _______________________________________________
> Enthought-Dev mailing list
> Enthought-Dev at enthought.com
> https://mail.enthought.com/mailman/listinfo/enthought-dev



From ellisonbg.net at gmail.com  Wed Aug 26 19:17:39 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 26 Aug 2009 16:17:39 -0700
Subject: [IPython-dev] Should IPython be a singleton?
In-Reply-To: <h74duf$hvp$1@ger.gmane.org>
References: <6ce0ac130908261436h73034f19mccc14893552db7c5@mail.gmail.com>
	<h74avv$98g$1@ger.gmane.org>
	<6ce0ac130908261519n2f32175cw1a356ab1bbdfc547@mail.gmail.com>
	<h74duf$hvp$1@ger.gmane.org>
Message-ID: <6ce0ac130908261617h35095f22u1ff18f2049a4cf89@mail.gmail.com>

> When tracking down problems like this, one thing you have to be very
> careful
> about is not creating references via your instrumentation. It's
> disturbingly
> easy to do. :-)
>

Yes, it is very tough to do right.


> __builtins__.__IPYTHON__ and __builtins__.__IPYTHON__active are two
> culprits,
> and the remaining one is a cell object. I'm not sure where he is coming
> from.
>

In my version, I have gotten rid of __IPYTHON__ and IPYTHON_active is an
int.

What is a cell object?  Could I see the foo.py script?

These are the game I have been playing, but it would be helpful to see what
you are doing here.

>
>
> [~]$ python foo.py
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>
> 40 references:
> <type 'instancemethod'>
> <bound method InteractiveShell.generate_output_prompt of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.shell_hook of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.show_in_pager of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.pre_prompt_hook of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.pre_runcode_hook of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.clipboard_get of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'dict'>
> A dict with keys: ['app_name', 'IP', 'crash_report_fname',
> 'user_message_template', 'show_crash_traceback', 'contact_email',
> 'bug_tracker',
> 'contact_name']
>
> <type 'instancemethod'>
> <bound method InteractiveShell.dummy_handler of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.ipmagic of
> <IPython.core.iplib.InteractiveShell
> object at 0x3aad90>>
>
> <type 'dict'>
> A dict with keys: ['magic', 'IP', 'user_ns', 'user_global_ns', 'system',
> 'dbg',
> 'meta', 'extensions', 'set_crash_handler', 'set_custom_exc', 'set_hook']
>
> <type 'instancemethod'>
> <bound method InteractiveShell.set_hook of
> <IPython.core.iplib.InteractiveShell
> object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.set_custom_exc of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.set_crash_handler of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.set_hook of
> <IPython.core.iplib.InteractiveShell
> object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.ipmagic of
> <IPython.core.iplib.InteractiveShell
> object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.ipalias of
> <IPython.core.iplib.InteractiveShell
> object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.ipsystem of
> <IPython.core.iplib.InteractiveShell
> object at 0x3aad90>>
>
> <type 'list'>
> A list of 19524 elements; probably gc.get_objects().
>
> <type 'cell'>
> A cell: <cell at 0x3aab90: InteractiveShell object at 0x3aad90>
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>
>
> <type 'dict'>
> A dict with keys: ['dir_stack', 'ESC_HELP', 'pycolorize', 'SyntaxTB',
> 'indent_current_nsp', 'hooks', 'user_ns', 'user_global_ns', 'system',
> '_main_ns_cache', 'dir_hist', 'autoindent', 'tempfiles', 'BANNER',
> 'code_to_run', 'loghead_tpl', 'ESC_PAREN', 'stdin_encoding', 'usage_min',
> 'has_readline', 'ns_table', 'getoutput', 'no_alias', 'home_dir',
> 'filename',
> 'ESC_SH_CAP', 'ns_refs_table', 'rc', 'usage', 'starting_dir', 'logger',
> 'banner2', 'options_table', 'more', 'shell', 'jobs', 'auto_alias', 'api',
> 'buffer', 'ESC_SHELL', 'alias_table', 'InteractiveTB', 'exit_now',
> 'builtins_added', 'output_hist', 'internal_ns', 'user_config_ns',
> 'ESC_QUOTE2',
> 'custom_exceptions', 'pager', 'sys_excepthook', 'getoutputerror',
> 'esc_handlers', 'input_hist_raw', 'name', 'embedded', 'CACHELENGTH',
> 'strdispatchers', 'CustomTB', '_magic_state', 'compile', 'input_hist',
> 'meta',
> '_user_main_module', 'ESC_MAGIC', 'ESC_QUOTE']
>
> <type 'dict'>
> A dict with keys: ['shell', 'name']
>
> <type 'dict'>
> A dict with keys: ['shell', 'name']
>
> <type 'dict'>
> A dict with keys: ['_dh', '_sh', '__builtins__', 'In', '_ip', '_ih',
> '__name__',
> 'foo', '_oh', 'Out']
>
> <type 'instancemethod'>
> <bound method InteractiveShell.handle_auto of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.handle_auto of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.handle_auto of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.handle_magic of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.handle_help of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.handle_shell_escape of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.handle_shell_escape of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.generate_prompt of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'dict'>
> A dict with keys: ['log_active', 'shell', '_i', 'log_raw_input', '_i00',
> '_iii',
> 'loghead', 'log_output', 'logfname', '_ii', 'timestamp', '_logmode',
> 'logfile']
>
> <type 'instancemethod'>
> <bound method InteractiveShell.shutdown_hook of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.result_display of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.synchronize_with_editor of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.editor of
> <IPython.core.iplib.InteractiveShell
> object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.fix_error_editor of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.input_prefilter of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'instancemethod'>
> <bound method InteractiveShell.late_startup_hook of
> <IPython.core.iplib.InteractiveShell object at 0x3aad90>>
>
> <type 'dict'>
> A dict with keys: ['obj', 'InteractiveShell', '__builtins__', '__file__',
> 'gc',
> 'refs', '__name__', 'ref', '__doc__', 'types']
>
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma
>  that is made terrible by our own mad attempt to interpret it as though it
> had
>  an underlying truth."
>   -- Umberto Eco
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090826/3ccf67c4/attachment.html>

From robert.kern at gmail.com  Wed Aug 26 19:27:00 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 26 Aug 2009 18:27:00 -0500
Subject: [IPython-dev] Should IPython be a singleton?
In-Reply-To: <6ce0ac130908261617h35095f22u1ff18f2049a4cf89@mail.gmail.com>
References: <6ce0ac130908261436h73034f19mccc14893552db7c5@mail.gmail.com>	<h74avv$98g$1@ger.gmane.org>	<6ce0ac130908261519n2f32175cw1a356ab1bbdfc547@mail.gmail.com>	<h74duf$hvp$1@ger.gmane.org>
	<6ce0ac130908261617h35095f22u1ff18f2049a4cf89@mail.gmail.com>
Message-ID: <h74gc5$ntc$1@ger.gmane.org>

On 2009-08-26 18:17 PM, Brian Granger wrote:
>
>     When tracking down problems like this, one thing you have to be very
>     careful
>     about is not creating references via your instrumentation. It's
>     disturbingly
>     easy to do. :-)
>
>
> Yes, it is very tough to do right.
>
>     __builtins__.__IPYTHON__ and __builtins__.__IPYTHON__active are two
>     culprits,
>     and the remaining one is a cell object. I'm not sure where he is
>     coming from.
>
>
> In my version, I have gotten rid of __IPYTHON__ and IPYTHON_active is an
> int.
>
> What is a cell object?

It is created when there are closures.

http://docs.python.org/c-api/cell.html

> Could I see the foo.py script?

Oops! Sorry. I thought I catted it. Here it is with a few more things cleaned up.

[~]$ cat foo.py
import gc
import sys
import types

from IPython.core.shell import InteractiveShell

real_main = sys.modules['__main__']

s = InteractiveShell('foo')
print s

# Clean up.
del s.shell
del s
del __builtins__.__IPYTHON__active
del __builtins__.__IPYTHON__
if hasattr(sys, 'ipcompleter'):
     del sys.ipcompleter
sys.stdin = sys.__stdin__
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__
sys.displayhook = sys.__displayhook__
sys.excepthook = sys.__excepthook__
sys.modules['__main__'] = real_main

for obj in gc.get_objects():
     if getattr(type(obj), '__name__', None) == 'InteractiveShell':
         print obj
         refs = gc.get_referrers(obj)
         print '%s references:' % len(refs)
         for ref in refs:
             print type(ref)
             if isinstance(ref, dict):
                 print 'A dict with keys: %r' % (ref.keys(),)
             elif isinstance(ref, list):
                 print 'A list of %s elements; probably gc.get_objects().' % 
len(ref)
             elif type(ref).__name__ == 'cell':
                 print 'A cell: %r' % ref
                 print ref.cell_contents
             else:
                 print ref
             print


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From ellisonbg.net at gmail.com  Wed Aug 26 19:34:38 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 26 Aug 2009 16:34:38 -0700
Subject: [IPython-dev] Should IPython be a singleton?
In-Reply-To: <h74gc5$ntc$1@ger.gmane.org>
References: <6ce0ac130908261436h73034f19mccc14893552db7c5@mail.gmail.com>
	<h74avv$98g$1@ger.gmane.org>
	<6ce0ac130908261519n2f32175cw1a356ab1bbdfc547@mail.gmail.com>
	<h74duf$hvp$1@ger.gmane.org>
	<6ce0ac130908261617h35095f22u1ff18f2049a4cf89@mail.gmail.com>
	<h74gc5$ntc$1@ger.gmane.org>
Message-ID: <6ce0ac130908261634h18f19b61n6febe6b2a0c3e465@mail.gmail.com>

Thanks! I will have a look at this to see how it differs from what I have
been doing.

Brian

On Wed, Aug 26, 2009 at 4:27 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On 2009-08-26 18:17 PM, Brian Granger wrote:
> >
> >     When tracking down problems like this, one thing you have to be very
> >     careful
> >     about is not creating references via your instrumentation. It's
> >     disturbingly
> >     easy to do. :-)
> >
> >
> > Yes, it is very tough to do right.
> >
> >     __builtins__.__IPYTHON__ and __builtins__.__IPYTHON__active are two
> >     culprits,
> >     and the remaining one is a cell object. I'm not sure where he is
> >     coming from.
> >
> >
> > In my version, I have gotten rid of __IPYTHON__ and IPYTHON_active is an
> > int.
> >
> > What is a cell object?
>
> It is created when there are closures.
>
> http://docs.python.org/c-api/cell.html
>
> > Could I see the foo.py script?
>
> Oops! Sorry. I thought I catted it. Here it is with a few more things
> cleaned up.
>
> [~]$ cat foo.py
> import gc
> import sys
> import types
>
> from IPython.core.shell import InteractiveShell
>
> real_main = sys.modules['__main__']
>
> s = InteractiveShell('foo')
> print s
>
> # Clean up.
> del s.shell
> del s
> del __builtins__.__IPYTHON__active
> del __builtins__.__IPYTHON__
> if hasattr(sys, 'ipcompleter'):
>     del sys.ipcompleter
> sys.stdin = sys.__stdin__
> sys.stdout = sys.__stdout__
> sys.stderr = sys.__stderr__
> sys.displayhook = sys.__displayhook__
> sys.excepthook = sys.__excepthook__
> sys.modules['__main__'] = real_main
>
> for obj in gc.get_objects():
>     if getattr(type(obj), '__name__', None) == 'InteractiveShell':
>         print obj
>         refs = gc.get_referrers(obj)
>         print '%s references:' % len(refs)
>         for ref in refs:
>             print type(ref)
>             if isinstance(ref, dict):
>                 print 'A dict with keys: %r' % (ref.keys(),)
>             elif isinstance(ref, list):
>                 print 'A list of %s elements; probably gc.get_objects().' %
> len(ref)
>             elif type(ref).__name__ == 'cell':
>                 print 'A cell: %r' % ref
>                 print ref.cell_contents
>             else:
>                 print ref
>             print
>
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma
>  that is made terrible by our own mad attempt to interpret it as though it
> had
>  an underlying truth."
>   -- Umberto Eco
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090826/feb91cd2/attachment.html>

From dsdale24 at gmail.com  Thu Aug 27 07:49:52 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Thu, 27 Aug 2009 07:49:52 -0400
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <F1521FE2-6933-4F04-9145-A6F540DC8435@cs.toronto.edu>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>
	<F1521FE2-6933-4F04-9145-A6F540DC8435@cs.toronto.edu>
Message-ID: <a08e5f80908270449y22c4a0ddt422628d6f2de262@mail.gmail.com>

Hi David,

On Wed, Aug 26, 2009 at 7:11 PM, David Warde-Farley<dwf at cs.toronto.edu> wrote:
> This was heavily discussed at the conference last week.
>
> It seems as though Phil Thompson might have painted himself into a
> corner.

I don't know. Who could have anticipated the sale of Qt and subsequent
license change by Nokia?

> At any rate, I for one welcome our new Finnish overlords.

I'll reserve judgment until they can build and distribute a stable
product across platforms.

Darren


> On 25-Aug-09, at 11:31 PM, Glenn Tarbox, PhD wrote:
>
>> Sooner or later, something was gonna need to happen WRT Riverbank
>> and the
>> PyQt licensing. ?I had hoped that an entirely new project wasn't
>> going to be
>> necessary.... but apparently it is.
>>
>> PySide Released to the Wild:
>> http://labs.trolltech.com/blogs/2009/08/25/pyside-released-to-the-
>> wild/
>>
>>> From the PySide FAQ
>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> What about PyQt?
>>
>> Nokia?s initial research into Python bindings for Qt involved
>> speaking with
>> Riverbank Computing, the makers of PyQt. We had several discussions
>> with
>> them to see if it was possible to use PyQt to achieve our goals.
>> Unfortunately, a common agreement could not be found , so in the end
>> we
>> decided to proceed with PySide.
>>
>> We will however maintain API compatibility with PyQt (you can use
>> the same
>> method names but can?t inter-operate with PyQt), at least for the
>> initial
>> release. To import PySide you have to use ?import PySide? instead
>> of ?import
>> PyQt4?.
>> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>>
>> I didn't know where to post this. ?PySide needs to mature a bit
>> (support
>> more than Linux for example) but both Matplotlib and Enthought are
>> affected. ?PyQt will likely need to be replaced in both packages
>> once PySide
>> becomes more mature as the licensing of PyQt is problematic now that
>> Qt is
>> LGPL. ?Its also likely that with Nokia's backing, the PySide API will
>> eventually dominate.
>>
>> Hopefully, the above statement regarding the similarity of the API
>> will make
>> moving over easy. ?Personally I'd like to see a focus on Qt vs Wx by
>> Enthought as I believe it to be much more powerful... but thats my
>> personal
>> opinion and what I use.
>>
>> As a side note, I've successfully nailed my C++ Qt code to IPython
>> using a
>> Cython shim. ?The PyQt event loop is available and all seems to work
>> great.
>>
>> -glenn
>>
>> --
>> Glenn H. Tarbox, PhD || ?206-274-6919
>> http://www.tarbox.org
>> _______________________________________________
>> Enthought-Dev mailing list
>> Enthought-Dev at enthought.com
>> https://mail.enthought.com/mailman/listinfo/enthought-dev
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>



-- 
"In our description of nature, the purpose is not to disclose the real
essence of the phenomena but only to track down, so far as it is
possible, relations between the manifold aspects of our experience" -
Niels Bohr

"It is a bad habit of physicists to take their most successful
abstractions to be real properties of our world." - N. David Mermin

"Once we have granted that any physical theory is essentially only a
model for the world of experience, we must renounce all hope of
finding anything like the correct theory ... simply because the
totality of experience is never accessible to us." - Hugh Everett III


From dwf at cs.toronto.edu  Thu Aug 27 14:46:58 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Thu, 27 Aug 2009 14:46:58 -0400
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <a08e5f80908270449y22c4a0ddt422628d6f2de262@mail.gmail.com>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>
	<F1521FE2-6933-4F04-9145-A6F540DC8435@cs.toronto.edu>
	<a08e5f80908270449y22c4a0ddt422628d6f2de262@mail.gmail.com>
Message-ID: <7B2DD705-BC1C-488F-9AE6-067102A2247A@cs.toronto.edu>

Hi Darren,

On 27-Aug-09, at 7:49 AM, Darren Dale wrote:

>> It seems as though Phil Thompson might have painted himself into a
>> corner.
>
> I don't know. Who could have anticipated the sale of Qt and subsequent
> license change by Nokia?

Oh, agreed. But once that happened (about 18-20 months ago IIRC) their  
agendas were pretty much guaranteed to collide. It's unfortunate that  
they couldn't reach an agreement.

>> At any rate, I for one welcome our new Finnish overlords.
>
> I'll reserve judgment until they can build and distribute a stable
> product across platforms.

Indeed, though with the amount of manpower they're dedicating to the  
PySide effort, I don't imagine it will be long.

One thing that is a pain in the neck is the boost::python dependency,  
which is sufficiently annoying to install on its own. Oh well.

David


From vivainio at gmail.com  Thu Aug 27 15:03:31 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 27 Aug 2009 22:03:31 +0300
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <7B2DD705-BC1C-488F-9AE6-067102A2247A@cs.toronto.edu>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>
	<F1521FE2-6933-4F04-9145-A6F540DC8435@cs.toronto.edu>
	<a08e5f80908270449y22c4a0ddt422628d6f2de262@mail.gmail.com>
	<7B2DD705-BC1C-488F-9AE6-067102A2247A@cs.toronto.edu>
Message-ID: <46cb515a0908271203k2ad1039dvaf80e94b47bfc030@mail.gmail.com>

On Thu, Aug 27, 2009 at 9:46 PM, David Warde-Farley<dwf at cs.toronto.edu> wrote:

> Indeed, though with the amount of manpower they're dedicating to the
> PySide effort, I don't imagine it will be long.

And in any case, it will have PyQt-compatible API - so starting to
leverage PyQt "for real" is not out of place even now, even if you
didn't dig the license earlier.

> One thing that is a pain in the neck is the boost::python dependency,
> which is sufficiently annoying to install on its own. Oh well.

That might change at some point (they are also doing a direct mapping
to CPython API in parallel).

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From basti.wiesner at gmx.net  Thu Aug 27 16:02:41 2009
From: basti.wiesner at gmx.net (Sebastian Wiesner)
Date: Thu, 27 Aug 2009 22:02:41 +0200
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <46cb515a0908271203k2ad1039dvaf80e94b47bfc030@mail.gmail.com>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>
	<7B2DD705-BC1C-488F-9AE6-067102A2247A@cs.toronto.edu>
	<46cb515a0908271203k2ad1039dvaf80e94b47bfc030@mail.gmail.com>
Message-ID: <200908272202.41983.basti.wiesner@gmx.net>

At Thursday 27 August 2009 21:03:31
> On Thu, Aug 27, 2009 at 9:46 PM, David Warde-Farley<dwf at cs.toronto.edu> 
wrote:
> > Indeed, though with the amount of manpower they're dedicating to the
> > PySide effort, I don't imagine it will be long.
>
> And in any case, it will have PyQt-compatible API - so starting to
> leverage PyQt "for real" is not out of place even now, even if you
> didn't dig the license earlier.

As a matter of fact, its API isn't API-compatible to PyQt4:  static methods 
have different names due to restrictions in boost.python.  It also doesn't seem 
to support any of the things introduced as of PyQt 4.5 (new signal-slot API, 
implicit QVariant conversion).

Moreover, PyQt-compatibility isn't guaranteed for future releases of PySide.  
The roadmap [1] says:

> In the future, PySide API may be modified to better support more Pythonic
> constructs and interfaces. This may break PyQt4 compatibility, and therefore
> community participation and acceptance is crucial.

For my part, I'm not really convinced, that this whole thing is going to be a 
success story.

[1] http://www.pyside.org/roadmap/

-- 
Freedom is always the freedom of dissenters.
                                      (Rosa Luxemburg)


From vivainio at gmail.com  Thu Aug 27 16:39:51 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 27 Aug 2009 23:39:51 +0300
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <200908272202.41983.basti.wiesner@gmx.net>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>
	<7B2DD705-BC1C-488F-9AE6-067102A2247A@cs.toronto.edu>
	<46cb515a0908271203k2ad1039dvaf80e94b47bfc030@mail.gmail.com>
	<200908272202.41983.basti.wiesner@gmx.net>
Message-ID: <46cb515a0908271339h3b2425c4v8e01189b4e0b08cf@mail.gmail.com>

On Thu, Aug 27, 2009 at 11:02 PM, Sebastian
Wiesner<basti.wiesner at gmx.net> wrote:

>
> As a matter of fact, its API isn't API-compatible to PyQt4: ?static methods
> have different names due to restrictions in boost.python. ?It also doesn't seem

Yeah, *some* static methods (when static method has same name as an
instance method).

> Moreover, PyQt-compatibility isn't guaranteed for future releases of PySide.
> The roadmap [1] says:

It's not guaranteed, but the probable route is that any new stuff is
added to the PyQt compatible api, instead of replacing it. They are
seeking community acceptance, and random api breakage is not the way
to foster that.


>> In the future, PySide API may be modified to better support more Pythonic
>> constructs and interfaces. This may break PyQt4 compatibility, and therefore
>> community participation and acceptance is crucial.
>
> For my part, I'm not really convinced, that this whole thing is going to be a
> success story.

Don't underestimate the lure of LGPL. People can tolerate quite a bit
of technical hurdles to get a more permissive license (c.f. Gnome/KDE
in the past - Gnome sucked quite a bit pre-2.0, yet it was the more
popular one among distributors).

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From robert.kern at gmail.com  Thu Aug 27 17:01:29 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 27 Aug 2009 16:01:29 -0500
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <46cb515a0908271339h3b2425c4v8e01189b4e0b08cf@mail.gmail.com>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>	<7B2DD705-BC1C-488F-9AE6-067102A2247A@cs.toronto.edu>	<46cb515a0908271203k2ad1039dvaf80e94b47bfc030@mail.gmail.com>	<200908272202.41983.basti.wiesner@gmx.net>
	<46cb515a0908271339h3b2425c4v8e01189b4e0b08cf@mail.gmail.com>
Message-ID: <h76s79$9oi$1@ger.gmane.org>

On 2009-08-27 15:39 PM, Ville M. Vainio wrote:
> On Thu, Aug 27, 2009 at 11:02 PM, Sebastian
> Wiesner<basti.wiesner at gmx.net>  wrote:

>> For my part, I'm not really convinced, that this whole thing is going to be a
>> success story.
>
> Don't underestimate the lure of LGPL. People can tolerate quite a bit
> of technical hurdles to get a more permissive license (c.f. Gnome/KDE
> in the past - Gnome sucked quite a bit pre-2.0, yet it was the more
> popular one among distributors).

Particularly since the new license restrictions on sip are GPL-incompatible. I 
don't see Linux distributions picking up PyQt4 4.5 anytime soon.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From vivainio at gmail.com  Thu Aug 27 17:15:03 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Fri, 28 Aug 2009 00:15:03 +0300
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <h76s79$9oi$1@ger.gmane.org>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>
	<7B2DD705-BC1C-488F-9AE6-067102A2247A@cs.toronto.edu>
	<46cb515a0908271203k2ad1039dvaf80e94b47bfc030@mail.gmail.com>
	<200908272202.41983.basti.wiesner@gmx.net>
	<46cb515a0908271339h3b2425c4v8e01189b4e0b08cf@mail.gmail.com>
	<h76s79$9oi$1@ger.gmane.org>
Message-ID: <46cb515a0908271415n1b70d098lb91e5e33c1fe9cf1@mail.gmail.com>

On Fri, Aug 28, 2009 at 12:01 AM, Robert Kern<robert.kern at gmail.com> wrote:

>> Don't underestimate the lure of LGPL. People can tolerate quite a bit
>> of technical hurdles to get a more permissive license (c.f. Gnome/KDE
>> in the past - Gnome sucked quite a bit pre-2.0, yet it was the more
>> popular one among distributors).
>
> Particularly since the new license restrictions on sip are GPL-incompatible. I
> don't see Linux distributions picking up PyQt4 4.5 anytime soon.

It's already on Ubuntu and Debian (Karmic has sip 4.8.1-1, pyqt
4.5.2-0ubuntu1). IIRC PyQT 4.5 was already on Jaunty.

Related bug (seems to be filed just a while ago):

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=543730

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From dsdale24 at gmail.com  Thu Aug 27 17:20:06 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Thu, 27 Aug 2009 17:20:06 -0400
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <h76s79$9oi$1@ger.gmane.org>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>
	<7B2DD705-BC1C-488F-9AE6-067102A2247A@cs.toronto.edu>
	<46cb515a0908271203k2ad1039dvaf80e94b47bfc030@mail.gmail.com>
	<200908272202.41983.basti.wiesner@gmx.net>
	<46cb515a0908271339h3b2425c4v8e01189b4e0b08cf@mail.gmail.com>
	<h76s79$9oi$1@ger.gmane.org>
Message-ID: <a08e5f80908271420t371adbf9t8112426656c29fea@mail.gmail.com>

On Thu, Aug 27, 2009 at 5:01 PM, Robert Kern<robert.kern at gmail.com> wrote:
> On 2009-08-27 15:39 PM, Ville M. Vainio wrote:
>> On Thu, Aug 27, 2009 at 11:02 PM, Sebastian
>> Wiesner<basti.wiesner at gmx.net> ?wrote:
>
>>> For my part, I'm not really convinced, that this whole thing is going to be a
>>> success story.
>>
>> Don't underestimate the lure of LGPL. People can tolerate quite a bit
>> of technical hurdles to get a more permissive license (c.f. Gnome/KDE
>> in the past - Gnome sucked quite a bit pre-2.0, yet it was the more
>> popular one among distributors).
>
> Particularly since the new license restrictions on sip are GPL-incompatible. I
> don't see Linux distributions picking up PyQt4 4.5 anytime soon.

It will be interesting to see how the kde folks and kde-oriented
distributions respond. pykde-4.3 has already made the transition to
PyQt-4.5, and I'm pretty sure several plasmoids use pykde.

Darren


From robert.kern at gmail.com  Thu Aug 27 17:34:58 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 27 Aug 2009 16:34:58 -0500
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <46cb515a0908271415n1b70d098lb91e5e33c1fe9cf1@mail.gmail.com>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>	<7B2DD705-BC1C-488F-9AE6-067102A2247A@cs.toronto.edu>	<46cb515a0908271203k2ad1039dvaf80e94b47bfc030@mail.gmail.com>	<200908272202.41983.basti.wiesner@gmx.net>	<46cb515a0908271339h3b2425c4v8e01189b4e0b08cf@mail.gmail.com>	<h76s79$9oi$1@ger.gmane.org>
	<46cb515a0908271415n1b70d098lb91e5e33c1fe9cf1@mail.gmail.com>
Message-ID: <h76u63$gg0$1@ger.gmane.org>

On 2009-08-27 16:15 PM, Ville M. Vainio wrote:
> On Fri, Aug 28, 2009 at 12:01 AM, Robert Kern<robert.kern at gmail.com>  wrote:
>
>>> Don't underestimate the lure of LGPL. People can tolerate quite a bit
>>> of technical hurdles to get a more permissive license (c.f. Gnome/KDE
>>> in the past - Gnome sucked quite a bit pre-2.0, yet it was the more
>>> popular one among distributors).
>>
>> Particularly since the new license restrictions on sip are GPL-incompatible. I
>> don't see Linux distributions picking up PyQt4 4.5 anytime soon.
>
> It's already on Ubuntu and Debian (Karmic has sip 4.8.1-1, pyqt
> 4.5.2-0ubuntu1). IIRC PyQT 4.5 was already on Jaunty.
>
> Related bug (seems to be filed just a while ago):
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=543730

The license change kind of slipped in unannounced. I should amend my statement: 
"I don't see Linux distributions pick up PyQt4 4.5 knowing the license change 
anytime soon."

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



From basti.wiesner at gmx.net  Fri Aug 28 09:27:59 2009
From: basti.wiesner at gmx.net (Sebastian Wiesner)
Date: Fri, 28 Aug 2009 15:27:59 +0200
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <46cb515a0908271203k2ad1039dvaf80e94b47bfc030@mail.gmail.com>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>
	<7B2DD705-BC1C-488F-9AE6-067102A2247A@cs.toronto.edu>
	<46cb515a0908271203k2ad1039dvaf80e94b47bfc030@mail.gmail.com>
Message-ID: <200908281528.00824.basti.wiesner@gmx.net>

On Thursday 27 August 2009 21:03:31 Ville M. Vainio wrote
> >> In the future, PySide API may be modified to better support more
> >> Pythonic constructs and interfaces. This may break PyQt4 compatibility,
> >> and therefore community participation and acceptance is crucial.
> >
> > For my part, I'm not really convinced, that this whole thing is going to
> > be a success story.
>
> Don't underestimate the lure of LGPL. People can tolerate quite a bit
> of technical hurdles to get a more permissive license (c.f. Gnome/KDE
> in the past - Gnome sucked quite a bit pre-2.0, yet it was the more
> popular one among distributors).

Yet, KDE didn't cease to exist, and remember, how look it to for both projects 
to work together, even for most basic things like menus or desktop icons :)

I spoke of "the whole thing", not just "PySide".   I do not doubt, that PySide 
will be successful, with a permissive license and Nokia standing behind it.

But I do doubt, that PyQt will step aside quickly, with many applications 
using and a libraries (PyKDE, PyQwt) building on it.   In the end, there will 
be two competing bindings for the same library, which only makes things more 
complicated for everyone, projects like PyKDE, PyQwt or matplotlib as well as 
other Python developers, who consider choosing Qt as their GUI toolkit.

But we'll see, what future brings ?

-- 
Freedom is always the freedom of dissenters.
                                      (Rosa Luxemburg)


From vivainio at gmail.com  Fri Aug 28 09:38:45 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Fri, 28 Aug 2009 16:38:45 +0300
Subject: [IPython-dev] [Enthought-Dev] PySide to replace PyQt?
In-Reply-To: <46cb515a0908271415n1b70d098lb91e5e33c1fe9cf1@mail.gmail.com>
References: <f049b35c0908252031l7b80e8b5ld8d8a3d04190233b@mail.gmail.com>
	<7B2DD705-BC1C-488F-9AE6-067102A2247A@cs.toronto.edu>
	<46cb515a0908271203k2ad1039dvaf80e94b47bfc030@mail.gmail.com>
	<200908272202.41983.basti.wiesner@gmx.net>
	<46cb515a0908271339h3b2425c4v8e01189b4e0b08cf@mail.gmail.com>
	<h76s79$9oi$1@ger.gmane.org>
	<46cb515a0908271415n1b70d098lb91e5e33c1fe9cf1@mail.gmail.com>
Message-ID: <46cb515a0908280638o4018a02eg392c2d3b2865776b@mail.gmail.com>

On Fri, Aug 28, 2009 at 12:15 AM, Ville M. Vainio<vivainio at gmail.com> wrote:

> It's already on Ubuntu and Debian (Karmic has sip 4.8.1-1, pyqt
> 4.5.2-0ubuntu1). IIRC PyQT 4.5 was already on Jaunty.
>
> Related bug (seems to be filed just a while ago):
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=543730

I contacted Phil about this, and apparently this licensing problem
will be resolved in the next version (dual licensed as proper GPL).

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From ellisonbg.net at gmail.com  Mon Aug 31 18:43:09 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 31 Aug 2009 15:43:09 -0700
Subject: [IPython-dev] New GUI integration in IPython
Message-ID: <6ce0ac130908311543o3cfc3ddbm9862d9523743fc00@mail.gmail.com>

Hello all,

This email is being sent out to to the lists of users+devs who regularly use
IPython's "pylab" mode or "-wthread", "-qthread", "-gthread", etc. threaded
shells.  As of today, in IPython's trunk, we have a completely new
implementation of our GUI event loop integration that dramatically improves
the stability of using the TERMINAL BASED IPython with GUI applications.
This does not affect attempts to embed IPython into GUI applications.

At this point, we need developers to begin to try out the new stuff and
adapt their projects to use the new capabilities.  Here are some things you
will get:

* Stability and robustness have been improved greatly.
* KeyboardInterrupts should work on all platforms reliably.
* No more command line flags - instead everything can be
activated/de-activated/switched at runtime.  This should allow projects like
matplotlib to enable reliable backend switching.  See the new %gui magic for
more information on this.
* We have a new developer module for working with these features
(IPython.lib.inputhook).
* Unless someone complains very loudly *and* steps up to the plate to
maintain them, the old threaded shells will be removed in the next release
of IPython.

Here are some starting points for documentation on the new features:

http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/docs/source/interactive/reference.txt#L1375
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/lib/inputhook.py
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/core/magic.py#L3542

Please let us know if you have questions - we are more than willing to help
you get started with all of this.

Cheers,

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090831/a4d09c76/attachment.html>