From hans_meine at gmx.net Mon Aug 1 08:53:42 2005 From: hans_meine at gmx.net (Hans Meine) Date: Mon, 1 Aug 2005 14:53:42 +0200 Subject: [IPython-dev] [FWD] An interesting take on the Notebook Problem In-Reply-To: <42E0FD1E.6000807@an.org> References: <42D8466E.1010509@colorado.edu> <42E0FD1E.6000807@an.org> Message-ID: <200508011453.43146.hans_meine@gmx.net> On Friday 22 July 2005 16:05, Toni Alatalo wrote: > [...] quick Googling now didn't tell me if the Python > random generator is guaranteed to give the same results with the same > seed and in what conditions (i know from working on procedural modelling > that it at least works on the same computer..), but that can be looked > at later. Confusingly, calling "random.seed(...)" is not enough. A too quick check once gave me the impression that it was (the first call of random.random() indeed returned the same value), but in order to store/restore the complete state you need to use the provided functions random.getstate() and random.setstate() which return the state as a a 3-tuple of an int, a tuple containing 625 ints and a None here. Greetings, Hans From rkern at ucsd.edu Mon Aug 1 08:58:39 2005 From: rkern at ucsd.edu (Robert Kern) Date: Mon, 01 Aug 2005 05:58:39 -0700 Subject: [IPython-dev] Re: [FWD] An interesting take on the Notebook Problem In-Reply-To: <200508011453.43146.hans_meine@gmx.net> References: <42D8466E.1010509@colorado.edu> <42E0FD1E.6000807@an.org> <200508011453.43146.hans_meine@gmx.net> Message-ID: Hans Meine wrote: > On Friday 22 July 2005 16:05, Toni Alatalo wrote: > >>[...] quick Googling now didn't tell me if the Python >>random generator is guaranteed to give the same results with the same >>seed and in what conditions (i know from working on procedural modelling >>that it at least works on the same computer..), but that can be looked >>at later. > > Confusingly, calling "random.seed(...)" is not enough. A too quick check once > gave me the impression that it was (the first call of random.random() indeed > returned the same value), but in order to store/restore the complete state > you need to use the provided functions random.getstate() and > random.setstate() which return the state as a a 3-tuple of an int, a tuple > containing 625 ints and a None here. random.seed() deterministically sets the whole state from an integer/long. Calling random.seed() should be enough. In [1]: import random In [2]: random.seed(123456789) In [3]: [random.random() for i in range(20)] Out[3]: [0.64140061618587263, 0.54218926809694945, 0.99317506628327212, 0.84325213668691656, 0.81173392833794056, 0.3971737100780004, 0.93709510791204254, 0.6891026531658162, 0.39711048852598374, 0.35102519242304475, 0.399603131718545, 0.54558174824433847, 0.207714227402927, 0.6001575513874905, 0.023680264289157371, 0.61140144008940833, 0.026211036291071466, 0.62015590923237174, 0.73967307701763618, 0.31412119286874374] In [4]: random.seed(123456789) In [5]: [random.random() for i in range(20)] Out[5]: [0.64140061618587263, 0.54218926809694945, 0.99317506628327212, 0.84325213668691656, 0.81173392833794056, 0.3971737100780004, 0.93709510791204254, 0.6891026531658162, 0.39711048852598374, 0.35102519242304475, 0.399603131718545, 0.54558174824433847, 0.207714227402927, 0.6001575513874905, 0.023680264289157371, 0.61140144008940833, 0.026211036291071466, 0.62015590923237174, 0.73967307701763618, 0.31412119286874374] -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From hans_meine at gmx.net Mon Aug 1 12:21:13 2005 From: hans_meine at gmx.net (Hans Meine) Date: Mon, 1 Aug 2005 18:21:13 +0200 Subject: [IPython-dev] Re: [FWD] An interesting take on the Notebook Problem In-Reply-To: References: <200508011453.43146.hans_meine@gmx.net> Message-ID: <200508011821.14279.hans_meine@gmx.net> On Monday 01 August 2005 14:58, Robert Kern wrote: > random.seed() deterministically sets the whole state from an > integer/long. Calling random.seed() should be enough. That's what I thought, too. However, I tried that with a unit-testing program which draws a lot more than 20 numbers, and there was a difference (fatal for me). That is the whole reason for my posting, because what you wrote was exactly my understanding of a "seed". *fiddling around* Strange, I can't reproduce it. I even tried hash(tuple([random.random() for i in range(4000)])) Even better so. Strange memory. Never mind, sorry for the noise. ;-| Maybe my program was non-deterministic in another way. Probably it was me calling it non-deterministic'ly though. ;-) Greetings, Hans From hans_meine at gmx.net Mon Aug 1 12:34:33 2005 From: hans_meine at gmx.net (Hans Meine) Date: Mon, 1 Aug 2005 18:34:33 +0200 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <42E71F48.6020205@colorado.edu> References: <1121681923.3210.18.camel@tzanko> <42E71F48.6020205@colorado.edu> Message-ID: <200508011834.34232.hans_meine@gmx.net> On Wednesday 27 July 2005 07:44, Fernando Perez wrote: > Here's an even worse example: > > In [3]: for i in range(3): > ...: print 'passing...' > ...: i > ...: > [...] I think the example above should simply print > 'passing...' three times, and NOT generate an Out[] cell at all. Right, think so too. > > As for having stdout cells interleaved with other types of output cells, > > I think that's going to be pretty hard. You'd have to collect the output > > from each statement separately. I'm of the opinion that you should just > > collect one each of stdout and stderr. > > Yup. I disagree. In my opinion, I want to preserve the order of interleaving stdout/stderr outputs. E.g. for i in range(10): myFunc(filenames[i]) Now let myFunc output either "File processed fine." to stdout or "Error processing file" to stderr. (Admitted, the messages suck. ;-) ) I would like to know exactly which i's failed. Greetings, Hans From Fernando.Perez at colorado.edu Mon Aug 1 15:41:12 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 01 Aug 2005 13:41:12 -0600 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <200508011834.34232.hans_meine@gmx.net> References: <1121681923.3210.18.camel@tzanko> <42E71F48.6020205@colorado.edu> <200508011834.34232.hans_meine@gmx.net> Message-ID: <42EE7AD8.6010407@colorado.edu> Hans Meine wrote: > On Wednesday 27 July 2005 07:44, Fernando Perez wrote: >>>As for having stdout cells interleaved with other types of output cells, >>>I think that's going to be pretty hard. You'd have to collect the output >>>from each statement separately. I'm of the opinion that you should just >>>collect one each of stdout and stderr. >> >>Yup. > > > I disagree. In my opinion, I want to preserve the order of interleaving > stdout/stderr outputs. E.g. > > for i in range(10): > myFunc(filenames[i]) > > Now let myFunc output either "File processed fine." to stdout or "Error > processing file" to stderr. (Admitted, the messages suck. ;-) ) > > I would like to know exactly which i's failed. Oh, I don't disagree with you that it would be _nice_ to have that interleaving. The problem is _how_ to do it. On the implementation end of things (where desires have to become reality), you have an object which can track each of the output streams (stdout/err), and that's about it. Those animals are buffered to begin with, so knowing in a reliable way when data came into each is not really possible, I think even for a plain shell. I seem to recall that even at the shell, quite often stdout and stderr come out in seemingly random interleaving order, depending on the (in practice random) details of buffer fill times. So I'm not really sure how this could be done in general. One possible conveninence feature we could add would be auto-timestamping. Since we have to trap stdout and reroute it to the gui, we could offer the possibility of prepending each write() call with a timestamp. This would be off by default, of course, but it could be useful in many circumstances. If I'm missing some obvious solution to the interleaving problem, feel free to point it out. Cheers, f From Fernando.Perez at colorado.edu Mon Aug 1 15:51:16 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 01 Aug 2005 13:51:16 -0600 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <42EE7AD8.6010407@colorado.edu> References: <1121681923.3210.18.camel@tzanko> <42E71F48.6020205@colorado.edu> <200508011834.34232.hans_meine@gmx.net> <42EE7AD8.6010407@colorado.edu> Message-ID: <42EE7D34.4020108@colorado.edu> Fernando Perez wrote: > Oh, I don't disagree with you that it would be _nice_ to have that > interleaving. The problem is _how_ to do it. On the implementation end of > things (where desires have to become reality), you have an object which can > track each of the output streams (stdout/err), and that's about it. Those > animals are buffered to begin with, so knowing in a reliable way when data > came into each is not really possible, I think even for a plain shell. mmh, scratch that. Upon thinking a bit more, it seems to me that since our output proxies have to trap all write() calls, they should be able to timestamp each one of them internally. These timestamps should correspond with the actuall call time, not with the results of output showing up on the buffered output, so they are indeed enough to reconstruct the call sequence. The implementation would be still a bit delicate, as you want to tag output to each stream as a separate kind, but multiple consecutive write() calls to one kind of stream should not create new cells. Instead, they should continue writing to the currently open one, unless the other stream has had a write() call made, time at which the code should open a new cell to switch streams. It seems possible, but a fair bit of work. Cheers, f From Fernando.Perez at colorado.edu Mon Aug 1 16:02:35 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 01 Aug 2005 14:02:35 -0600 Subject: [IPython-dev] [FWD] An interesting take on the Notebook Problem In-Reply-To: <42E0FD1E.6000807@an.org> References: <42D8466E.1010509@colorado.edu> <42E0FD1E.6000807@an.org> Message-ID: <42EE7FDB.1020002@colorado.edu> Toni Alatalo wrote: > Fernando Perez wrote: > > >>[Chris, note that ipytho-dev discards non-subscriber posts (too much >>spam). I've manually > [ paper about 'compendium' interactive documents, what we call 'notebooks' ] > > thanks for posting this - was quite a good read, and something that > makes the notebook project even more interesting > Besides the SciPy tutorial, perhaps that seminal paper Golub and the R > compendium should also be among the 'test cases' for Python notebooks > (hopefully the dataset is not too tricky to convert to whatever the > format in our system would be). As I mentioned before, don't worry about something that ambitious yet. We'll first concentrate on getting the basic system up and running, before we can load some monster genome analysis into it, which probably depends on a zillion other libraries. The paper was indeed interesting, and it shows there is actual interest in the scientific community for this kind of environment. Providing facilities to ease up the packaging of support data for public distribution, for example, would be a good future enhancement, so that a notebook can be bundled easily with all its supporting data files for others to execute, check, experiment. But first things first: let's cross that bridge when we get to it. > oh and one technical issue relating reproducibility (although that is > probably not among the top questions for us yet): as Gentleman notes, > some research methods use randomness, and to be able to reproduce the > exact same computations, and a heavy way is to include the random table > used in the compendium. quick Googling now didn't tell me if the Python > random generator is guaranteed to give the same results with the same > seed and in what conditions (i know from working on procedural modelling > that it at least works on the same computer..), but that can be looked > at later. I think this one was settled already by Robert's clarifications. But don't worry about random number generators at this point, that's not our initial goal. Cheers, f From hans_meine at gmx.net Tue Aug 2 05:58:45 2005 From: hans_meine at gmx.net (Hans Meine) Date: Tue, 2 Aug 2005 11:58:45 +0200 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <42EE7D34.4020108@colorado.edu> References: <1121681923.3210.18.camel@tzanko> <42EE7AD8.6010407@colorado.edu> <42EE7D34.4020108@colorado.edu> Message-ID: <200508021158.46447.hans_meine@gmx.net> On Monday 01 August 2005 21:51, Fernando Perez wrote: > Fernando Perez wrote: > > Oh, I don't disagree with you that it would be _nice_ to have that > > interleaving. The problem is _how_ to do it. On the implementation end > > of things (where desires have to become reality), you have an object > > which can track each of the output streams (stdout/err), and that's about > > it. Those animals are buffered to begin with, so knowing in a reliable > > way when data came into each is not really possible, I think even for a > > plain shell. > > mmh, scratch that. Upon thinking a bit more, it seems to me that since our > output proxies have to trap all write() calls, they should be able to > timestamp each one of them internally. These timestamps should correspond > with the actuall call time, not with the results of output showing up on > the buffered output, so they are indeed enough to reconstruct the call > sequence. That's exactly the way we did it in our python shell for colored ouput (stderr in red). > The implementation would be still a bit delicate, as you want to tag output > to each stream as a separate kind, but multiple consecutive write() calls > to one kind of stream should not create new cells. Instead, they should > continue writing to the currently open one, unless the other stream has had > a write() call made, time at which the code should open a new cell to > switch streams. It seems possible, but a fair bit of work. Indeed, since we had no "cells" (no notebook), we did not have that problem, but that does not seem too hard, no? e.g. sth. along the lines of class ErrorStream: def write(s): lastCell = {retrieve somehow} if lastCell.type() == ErrorOutput: lastCell.append(s) else: appendCell(ErrorOutputCell(s)) Greets, Hans From hans_meine at gmx.net Tue Aug 2 06:06:25 2005 From: hans_meine at gmx.net (Hans Meine) Date: Tue, 2 Aug 2005 12:06:25 +0200 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <42EE7D34.4020108@colorado.edu> References: <1121681923.3210.18.camel@tzanko> <42EE7AD8.6010407@colorado.edu> <42EE7D34.4020108@colorado.edu> Message-ID: <200508021206.26725.hans_meine@gmx.net> On Monday 01 August 2005 21:51, Fernando Perez wrote: > mmh, scratch that. Upon thinking a bit more, it seems to me that since our > output proxies have to trap all write() calls, [...] BTW: Did you find a way to catch stderr output from binary extension modules? In our terminal we just redirected sys.stderr, which does not effect our C++ extensions. :-( Maybe it's possible to redirect the file descriptors? In that case, would we have a problem with buffering, or does that happen after writing.. no. The iostreams library will already buffer output I guess. :-( An idea worth thinking about though. (Otherwise, extension module which containt printf's or cerr << "sth." will output to the terminal that started the IPython GUI. That's not optimal, but at least none of the standard modules do such things AFAICS.. ;-) Oh yes - the version mismatch warning from Python could be an example!) Have a nice day, Hans From Fernando.Perez at colorado.edu Tue Aug 2 12:56:57 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 02 Aug 2005 10:56:57 -0600 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <200508021206.26725.hans_meine@gmx.net> References: <1121681923.3210.18.camel@tzanko> <42EE7AD8.6010407@colorado.edu> <42EE7D34.4020108@colorado.edu> <200508021206.26725.hans_meine@gmx.net> Message-ID: <42EFA5D9.6070300@colorado.edu> Hans Meine wrote: > On Monday 01 August 2005 21:51, Fernando Perez wrote: > >>mmh, scratch that. Upon thinking a bit more, it seems to me that since our >>output proxies have to trap all write() calls, [...] > > BTW: Did you find a way to catch stderr output from binary extension modules? > In our terminal we just redirected sys.stderr, which does not effect our C++ > extensions. :-( > > Maybe it's possible to redirect the file descriptors? > > In that case, would we have a problem with buffering, or does that happen > after writing.. no. The iostreams library will already buffer output I > guess. :-( > > An idea worth thinking about though. (Otherwise, extension module which > containt printf's or cerr << "sth." will output to the terminal that started > the IPython GUI. That's not optimal, but at least none of the standard > modules do such things AFAICS.. ;-) Oh yes - the version mismatch warning > from Python could be an example!) I don't know how to trap 'cerr << foo' calls from python. I tried some simple tricks with file descriptors at some point, but failed. It may be possible to do it with the proper incantation, but my quick attempts have been so far unsucessful. Cheers, f From Fernando.Perez at colorado.edu Tue Aug 2 13:00:30 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 02 Aug 2005 11:00:30 -0600 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <200508021158.46447.hans_meine@gmx.net> References: <1121681923.3210.18.camel@tzanko> <42EE7AD8.6010407@colorado.edu> <42EE7D34.4020108@colorado.edu> <200508021158.46447.hans_meine@gmx.net> Message-ID: <42EFA6AE.4080402@colorado.edu> Hans Meine wrote: > On Monday 01 August 2005 21:51, Fernando Perez wrote: > >>Fernando Perez wrote: >>mmh, scratch that. Upon thinking a bit more, it seems to me that since our >>output proxies have to trap all write() calls, they should be able to >>timestamp each one of them internally. These timestamps should correspond >>with the actuall call time, not with the results of output showing up on >>the buffered output, so they are indeed enough to reconstruct the call >>sequence. > > > That's exactly the way we did it in our python shell for colored ouput (stderr > in red). OK, good to know that someone has tested the idea and it does indeed work. Thanks. >>The implementation would be still a bit delicate, as you want to tag output >>to each stream as a separate kind, but multiple consecutive write() calls >>to one kind of stream should not create new cells. Instead, they should >>continue writing to the currently open one, unless the other stream has had >>a write() call made, time at which the code should open a new cell to >>switch streams. It seems possible, but a fair bit of work. > > > Indeed, since we had no "cells" (no notebook), we did not have that problem, > but that does not seem too hard, no? > e.g. sth. along the lines of > > class ErrorStream: > def write(s): > lastCell = {retrieve somehow} > if lastCell.type() == ErrorOutput: > lastCell.append(s) > else: > appendCell(ErrorOutputCell(s)) That should work, yes. Nice and simple :) Best, f From rkern at ucsd.edu Wed Aug 3 01:13:16 2005 From: rkern at ucsd.edu (Robert Kern) Date: Tue, 02 Aug 2005 22:13:16 -0700 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <42EFA6AE.4080402@colorado.edu> References: <1121681923.3210.18.camel@tzanko> <42EE7AD8.6010407@colorado.edu> <42EE7D34.4020108@colorado.edu> <200508021158.46447.hans_meine@gmx.net> <42EFA6AE.4080402@colorado.edu> Message-ID: Fernando Perez wrote: > Hans Meine wrote: >> class ErrorStream: >> def write(s): >> lastCell = {retrieve somehow} >> if lastCell.type() == ErrorOutput: >> lastCell.append(s) >> else: >> appendCell(ErrorOutputCell(s)) > > That should work, yes. Nice and simple :) I still think that this should be a low priority. While recording the stdout and stderr outputs properly interleaved is nice and simple, it still complicates the document generation and UI. Any interleaving of stdout and stderr is an accident. The whole point of having two different streams is that they can be handled separately and piped to two different locations. stderr messages need to be written with that in mind for any piece of code. Usually, they are, so I don't believe that this is going to be a significant issue in practice. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From hans_meine at gmx.net Wed Aug 3 05:26:57 2005 From: hans_meine at gmx.net (Hans Meine) Date: Wed, 3 Aug 2005 11:26:57 +0200 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: References: <1121681923.3210.18.camel@tzanko> <42EFA6AE.4080402@colorado.edu> Message-ID: <200508031126.58744.hans_meine@gmx.net> On Wednesday 03 August 2005 07:13, Robert Kern wrote: > Any interleaving of stdout and stderr is an accident. This is an extreme point of view. ;-) > The whole point of > having two different streams is that they can be handled separately and > piped to two different locations. I agree. > stderr messages need to be written > with that in mind for any piece of code. Right, but.. > Usually, they are, [...] ..I don't agree 100% here. And it really *can* be misleading if the "order" is wrong (i.e. not chronological) - people are just too used to it from e.g. system shells. It's still a feature in my perspective that's not too unimportant and I don't see why it "complicates document generation". Just because there is not maximally one error, one output cell belonging to one input, but an ordered alternating sequence? Greetings, Hans From rkern at ucsd.edu Wed Aug 3 06:52:28 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed, 03 Aug 2005 03:52:28 -0700 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <200508031126.58744.hans_meine@gmx.net> References: <1121681923.3210.18.camel@tzanko> <42EFA6AE.4080402@colorado.edu> <200508031126.58744.hans_meine@gmx.net> Message-ID: Hans Meine wrote: > On Wednesday 03 August 2005 07:13, Robert Kern wrote: > >>Any interleaving of stdout and stderr is an accident. > > This is an extreme point of view. ;-) Then allow me to rephrase: "Any specific interleaving of stdout and stderr is an accident." >>The whole point of >>having two different streams is that they can be handled separately and >>piped to two different locations. > > I agree. > >>stderr messages need to be written >>with that in mind for any piece of code. > > Right, but.. > >>Usually, they are, [...] > > ..I don't agree 100% here. > > And it really *can* be misleading if the "order" is wrong (i.e. not > chronological) - people are just too used to it from e.g. system shells. The shells don't do any interleaving. When stdout and stderr are both output semi-immediately to the same terminal, the outputs get interleaved somewhat arbitrarily thanks to buffering. In the reasonably common case where the two streams are piped to different things, no interleaving takes place at all. It's the latter case that the separation of stdout and stderr is designed to support. Use of stderr that doesn't work well with separate destinations is missing the point. > It's still a feature in my perspective that's not too unimportant and I don't > see why it "complicates document generation". Just because there is not > maximally one error, one output cell belonging to one input, but an ordered > alternating sequence? Look at the code. It's dead simple (at least this bit) and works. At minimum, implementing an interleaved stdout and stderr means more work compared to doing nothing and using the current, working code. As currently implemented, to get the data, the document generator just looks up the number of the cell (the "NN" in "In [NN]" and "Out[NN]") and the type. To change that, you would have to look up the number of the cell, the type, and some other specifier to pick out the specific stdout/stderr datum. That extra information needs to be handled every time there's output, not just the few times that stdout and stderr are interleaved. Alternatively, we can give up fine-grained selection of which elements are shown for a given cell number, but I think that's going to be a feature used quite often. I often want to suppress the Out or the stdout elements because they're too long, but keep the In element. I hate being the killjoy all the time, and I'm not suggesting that this feature won't be worthwhile to implement eventually but that we have higher priorities right now. Use of stderr is relatively rare (outside of tracebacks which we catch by a different mechanism). Use of stderr that really needs a specific interleaving with stdout (which was never guaranteed anywhere else) to make sense is even rarer. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From rkern at ucsd.edu Thu Aug 4 02:29:12 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed, 03 Aug 2005 23:29:12 -0700 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <200508021206.26725.hans_meine@gmx.net> References: <1121681923.3210.18.camel@tzanko> <42EE7AD8.6010407@colorado.edu> <42EE7D34.4020108@colorado.edu> <200508021206.26725.hans_meine@gmx.net> Message-ID: Hans Meine wrote: > On Monday 01 August 2005 21:51, Fernando Perez wrote: > >>mmh, scratch that. Upon thinking a bit more, it seems to me that since our >>output proxies have to trap all write() calls, [...] > > BTW: Did you find a way to catch stderr output from binary extension modules? Yes! I didn't have any luck with doing the file descriptor magic in pure Python, but a little bit of Pyrex (attached) works wonders. It works for Python (without touching the sys.stdout or sys.stderr objects), C's printf and fprintf, C++'s cout and cerr, and FORTRAN's WRITE. It needs to be rewritten to carefully handle errors, but the functionality is there. It works on OS X; I presume it will work on most sane UNIX's. I think it will probably also work on Windows, but I wouldn't care to wager real money on that belief. Of course, redirecting stdout and stderr like this forgoes any interleaving. ;-) # test.py import sys import redirc rout = redirc.Redirector(redirc.STDOUT) rerr = redirc.Redirector(redirc.STDERR) import ctest import cxxtest import ftest rout.start() rerr.start() print "Python: stdout" print >>sys.stderr, "Python: stderr" ctest.stdout() ctest.stderr() cxxtest.stdout() cxxtest.stderr() ftest.stdo() out = rout.stop() err = rerr.stop() print 'stdout:' print repr(out) print print 'stderr:' print repr(err) # and the output: stdout: 'Python: stdout\nC: stdout\nC++: stdout\n FORTRAN: stdout\n' stderr: 'Python: stderr\nC: stderr\nC++: stderr\n' -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: redirc.pyx URL: From rkern at ucsd.edu Thu Aug 4 03:07:12 2005 From: rkern at ucsd.edu (Robert Kern) Date: Thu, 04 Aug 2005 00:07:12 -0700 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: References: <1121681923.3210.18.camel@tzanko> <42EE7AD8.6010407@colorado.edu> <42EE7D34.4020108@colorado.edu> <200508021206.26725.hans_meine@gmx.net> Message-ID: Robert Kern wrote: > Hans Meine wrote: >> BTW: Did you find a way to catch stderr output from binary extension >> modules? > > Yes! I didn't have any luck with doing the file descriptor magic in pure > Python, but a little bit of Pyrex (attached) works wonders. And some trivial modifications renders it working in pure Python. Joy! -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: redir.py URL: From hans_meine at gmx.net Thu Aug 4 08:35:51 2005 From: hans_meine at gmx.net (Hans Meine) Date: Thu, 4 Aug 2005 14:35:51 +0200 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: References: <1121681923.3210.18.camel@tzanko> Message-ID: <200508041435.51894.hans_meine@gmx.net> On Thursday 04 August 2005 09:07, Robert Kern wrote: > >> BTW: Did you find a way to catch stderr output from binary extension > >> modules? > > > > Yes! I didn't have any luck with doing the file descriptor magic in pure > > Python, but a little bit of Pyrex (attached) works wonders. > > And some trivial modifications renders it working in pure Python. Joy! Cool. That's indeed what I had in mind. However, isn't there a problem with os.pipe() buffering? I thought you'd probably have to read() at one point to prevent blocking of further write()s. I our shell, we have separate processes for the frontend and the backend, mainly to make Ctrl-C work. (Otherwise, the [Ctrl-C] X events are not handled before the current code stops.) Thus, it'd be easy to do the reading in time. Greetings, Hans From Fernando.Perez at colorado.edu Thu Aug 4 11:19:00 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 04 Aug 2005 09:19:00 -0600 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: References: <1121681923.3210.18.camel@tzanko> <42EE7AD8.6010407@colorado.edu> <42EE7D34.4020108@colorado.edu> <200508021206.26725.hans_meine@gmx.net> Message-ID: <42F231E4.3060508@colorado.edu> Robert Kern wrote: > Robert Kern wrote: > >>Hans Meine wrote: > > >>>BTW: Did you find a way to catch stderr output from binary extension >>>modules? >> >>Yes! I didn't have any luck with doing the file descriptor magic in pure >>Python, but a little bit of Pyrex (attached) works wonders. > > > And some trivial modifications renders it working in pure Python. Joy! My hat's off to you. Awesome. f From rkern at ucsd.edu Thu Aug 4 18:47:29 2005 From: rkern at ucsd.edu (Robert Kern) Date: Thu, 04 Aug 2005 15:47:29 -0700 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: References: <1121681923.3210.18.camel@tzanko> <42EE7AD8.6010407@colorado.edu> <42EE7D34.4020108@colorado.edu> <200508021206.26725.hans_meine@gmx.net> Message-ID: [This ought to be in reply to Hans's message, but it hasn't propogated to GMane, yet.] Hans Meine wrote: > However, isn't there a problem with os.pipe() buffering? I thought > you'd probably have to read() at one point to prevent blocking of > further write()s. You're right, and it starts blocking pretty early. We can resolve that by redirecting to a real file instead. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: redirfile.py URL: From hans_meine at gmx.net Fri Aug 5 06:35:43 2005 From: hans_meine at gmx.net (Hans Meine) Date: Fri, 5 Aug 2005 12:35:43 +0200 Subject: [IPython-dev] Preventing "Oops, IPython crashed" ? In-Reply-To: <429CB6A1.6010604@colorado.edu> References: <200505311445.23247.hans_meine@gmx.net> <429CB6A1.6010604@colorado.edu> Message-ID: <200508051235.44378.hans_meine@gmx.net> Hi again! On Tuesday 31 May 2005 21:10, Fernando Perez wrote: > Hans Meine wrote: > > I forgot to ask this question, related to my Qt experiments: I loaded > > some experimental GUI code, and noticed that exceptions resulting from > > GUI actions result in this very verbose IPython traceback / bugreporting > > output. > > > > Is there a way to get a similar traceback to the one I get when calling > > functions manually from within IPython? > > Not that I know. Unfortunately, exceptions raised in a different thread > propagate all the way through to sys.excepthook, which is the ipython > crash handler. And I don't know of a way at that point of identifying > whether the exception was caused by ipython code or by user code, so the > crash handler just does its job and prints that monster traceback. > > This really annoys me a lot, but from all the googling and reading of > maling lists which I did, it seemed to me that cross-thread exception > handling in python was just somewhat limited. I'd love to learn of a > way around this problem, though, so by all means let me know if you are > aware of a viable solution. Indeed, that seems to be a serious problem. > There is a workaround: you can manually reset sys.excepthook to the > ipython normal exception handler: > > sys.excepthook = __IPYTHON__.excepthook > > By running this line, you lose the real crash handler though, so if > ipython proper crashes, I will get a LOT less debug information. Caveat > emptor. I think this workaround should be documented somehow. When working with GUI callbacks that possibly result in exceptions in a different thread, the long traceback is much less helpful than the shiny "normal" IPython one. (Congratulations on that BTW.) Another workaround could be to color the long one, too (after all, it's mostly the missing color which bugs me), and either remove that from the report.txt or even send a colored report (although the usefulness depends on your mailer since it's not attached but in the mail body) which you can dump to a console? Ciao, / / /--/ / / ANS From Fernando.Perez at colorado.edu Thu Aug 11 15:49:57 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 11 Aug 2005 13:49:57 -0600 Subject: [IPython-dev] Preventing "Oops, IPython crashed" ? In-Reply-To: <200508051235.44378.hans_meine@gmx.net> References: <200505311445.23247.hans_meine@gmx.net> <429CB6A1.6010604@colorado.edu> <200508051235.44378.hans_meine@gmx.net> Message-ID: <42FBABE5.20304@colorado.edu> Hans Meine wrote: >>There is a workaround: you can manually reset sys.excepthook to the >>ipython normal exception handler: >> >>sys.excepthook = __IPYTHON__.excepthook >> >>By running this line, you lose the real crash handler though, so if >>ipython proper crashes, I will get a LOT less debug information. Caveat >>emptor. > > > I think this workaround should be documented somehow. When working with GUI > callbacks that possibly result in exceptions in a different thread, the long > traceback is much less helpful than the shiny "normal" IPython one. > (Congratulations on that BTW.) > > Another workaround could be to color the long one, too (after all, it's mostly > the missing color which bugs me), and either remove that from the report.txt > or even send a colored report (although the usefulness depends on your mailer > since it's not attached but in the mail body) which you can dump to a > console? The problem is that I have no way of knowing if that is being hit by a matplotlib exception, or an honest ipython crash. Cross-thread exception handling in python is a nightmare, so all those guys fall trhough into sys.excepthook, where the ipython crash handler sits. That one doesn't use color, because ansi escapes inside emails make a mess. So I'm afraid I don't really know how to solve this particular problem cleanly. Any ideas would be most welcome. Specifically, a way to trap the mpl exceptions only would do the trick, but all try/excepts I've tried to wrap around the mpl threads have so far failed. My reading of multiple posts on c.l.py on the exceptions/threads mix make it sound like it's a pretty explosive cocktail, so I really don't know what else to do. Cheers, f From hans_meine at gmx.net Fri Aug 12 04:33:07 2005 From: hans_meine at gmx.net (Hans Meine) Date: Fri, 12 Aug 2005 10:33:07 +0200 Subject: [IPython-dev] Preventing "Oops, IPython crashed" ? In-Reply-To: <42FBABE5.20304@colorado.edu> References: <200505311445.23247.hans_meine@gmx.net> <200508051235.44378.hans_meine@gmx.net> <42FBABE5.20304@colorado.edu> Message-ID: <200508121033.07962.hans_meine@gmx.net> On Thursday 11 August 2005 21:49, Fernando Perez wrote: > So I'm afraid I don't really know how to solve this particular problem > cleanly. Any ideas would be most welcome. Specifically, a way to trap the > mpl exceptions only would do the trick, but all try/excepts I've tried to > wrap around the mpl threads have so far failed. Yes, I understood the problem is hard-to-impossible to solve cleanly, but what I wanted to say is that it'd make sense to document the workaround sys.excepthook = __IPYTHON__.excepthook or even make it a configurable option. The problem I have is not with matplotlib, but with Qt: If you write an interactive Qt program (which is most probable if you're using Qt in the first place), your code will most probably be executed from the GUI thread and as such all small mistakes in your own code will result in this lengthy default exc.handler output. I hard-coded the above statement into IPShellQt.__init__ since IPShellQt is much less usable otherwise. So I propose a documented ipythonrc switch that activates this, possibly only for IPShellQt if other toolkits work differently. I hope this made my intention more clear. :-) Ciao, Hans From Fernando.Perez at colorado.edu Tue Aug 16 16:52:27 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 16 Aug 2005 14:52:27 -0600 Subject: [IPython-dev] [Fwd: [IPython SVN] New commit performed by fperez] Message-ID: <4302520B.4050103@colorado.edu> Heads up (esp. Frederic), I normally don't mention all SVN commits here, but in this case there is an API change which may impact some users. Please see the commit log below. It will only affect those who use custom exception handlers, and they can still have the same functionality as before. It's just that instead of using the self.code_to_run_src attribute, if they want to print/use source information they can either use the self.buffer object, which is a list of lines, or assemble a string via '\n'.join(self.buffer) I hadn't realized I was duplicating this attribute, and duplication like this is bugs waiting to happen. Cheers, f -------- Original Message -------- Subject: [IPython SVN] New commit performed by fperez Date: Tue, 16 Aug 2005 11:34:52 -0600 From: Fernando Perez To: fperez at colorado.edu Commit performed at: Tue Aug 16 11:34:37 2005 SVN arguments used: SVN Version: 687:703 This version probably points to this changeset: http://projects.scipy.org/ipython/ipython/changeset/687:703 Current SVN status after last commit: ------------------------------------------------------------------------ r703 | fperez | 2005-08-16 11:34:44 -0600 (Tue, 16 Aug 2005) | 9 lines * IPython/iplib.py (runsource): remove self.code_to_run_src attribute. I realized this is nothing more than '\n'.join(self.buffer), and having the same data in two different places is just asking for synchronization bugs. This may impact people who have custom exception handlers, so I need to warn ipython-dev about it (F. Mantegazza may use them). * Fix http://www.scipy.net/roundup/ipython/issue38 ------------------------------------------------------------------------ From hans_meine at gmx.net Thu Aug 18 11:53:21 2005 From: hans_meine at gmx.net (Hans Meine) Date: Thu, 18 Aug 2005 17:53:21 +0200 Subject: [IPython-dev] Threads and Ctrl-C Message-ID: <200508181753.22482.hans_meine@gmx.net> Hi! I don't know how many of you are actually working with IPython and GUI threads - from my traceback experiences, I seem to be rather alone with my strugglings. (I guess other toolkits than Qt - which I use - should have similar problems, but I might be wrong.) OK, try this: Start "ipython -qthread", enter: In [1]: import time In [2]: for i in range(60): time.sleep(1) ...: Now press Ctrl-C, you will get a nice message "KeyboardInterrupt - Press to continue." which is more or less meaning "I don't know what to do with your keypress, since I am totally unaware that the other thread might be executing some code". In the before-mentioned Qt Python shell we wrote at our lab, we had a similar situation (probably Toni will be interested in this), but we actually had two separate processes for the GUI and Python, mostly to solve exactly this single Ctrl-C problem: As long as Python is doing something, the Ctrl-C event would not get through to the GUI event handler, so the event handler could not interrupt the interpreter. Ullrich K?the's solution was to write a small "interrupt" module that is triggered from the other process via a socket and interrupts the interpreter by calling PyErr_SetInterrupt(). It's not too large; I attached it for your inspiration. IMHO having the possibility to interrupt some loop you accidentally started with wrong parameters is a crucial feature?! Greetings, Hans -------------- next part -------------- A non-text attachment was scrubbed... Name: interrupt.c Type: text/x-csrc Size: 4194 bytes Desc: not available URL: From Fernando.Perez at colorado.edu Fri Aug 19 14:58:36 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Fri, 19 Aug 2005 12:58:36 -0600 Subject: [IPython-dev] Threads and Ctrl-C In-Reply-To: <200508181753.22482.hans_meine@gmx.net> References: <200508181753.22482.hans_meine@gmx.net> Message-ID: <43062BDC.8060506@colorado.edu> Hans Meine wrote: > Hi! > > I don't know how many of you are actually working with IPython and GUI threads > - from my traceback experiences, I seem to be rather alone with my > strugglings. (I guess other toolkits than Qt - which I use - should have > similar problems, but I might be wrong.) > > OK, try this: > > Start "ipython -qthread", enter: > > In [1]: import time > > In [2]: for i in range(60): time.sleep(1) > ...: > > Now press Ctrl-C, you will get a nice message "KeyboardInterrupt - Press > to continue." which is more or less meaning "I don't know what to do > with your keypress, since I am totally unaware that the other thread might be > executing some code". > > In the before-mentioned Qt Python shell we wrote at our lab, we had a similar > situation (probably Toni will be interested in this), but we actually had two > separate processes for the GUI and Python, mostly to solve exactly this > single Ctrl-C problem: As long as Python is doing something, the Ctrl-C > event would not get through to the GUI event handler, so the event handler > could not interrupt the interpreter. > > Ullrich K?the's solution was to write a small "interrupt" module that is > triggered from the other process via a socket and interrupts the interpreter > by calling PyErr_SetInterrupt(). It's not too large; I attached it for your > inspiration. IMHO having the possibility to interrupt some loop you > accidentally started with wrong parameters is a crucial feature?! Thanks for the code, it will probably be useful in the future, as we move to a 2-process model. For a number of reasons (signal handling being only one), that is the right model in the long run, so I hope we can start moving in that direction sooner rather than later. Regards, f From mantegazza at ill.fr Mon Aug 22 11:07:01 2005 From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=) Date: Mon, 22 Aug 2005 17:07:01 +0200 Subject: [IPython-dev] Re: [IPython-user] ipython gui interpreters In-Reply-To: <4309E405.4060000@indiana.edu> References: <4309E405.4060000@indiana.edu> Message-ID: <200508221707.02130.mantegazza@ill.fr> Le Lundi 22 Ao?t 2005 16:41, Charles Moad a ?crit : > Out of curiosity, is there any working code out there for using ipython > in guis for Tk or Gtk? I know the notebook work is going on, but I am > interested in turning a textarea into an interactive interpreter. Hello, I started to port konsole, the terminal emulator of KDE, for pyqt, with the help of Lars Doelle, the konsole author, which sent me a qt-based only code. But as there are a lot of C++ specific stuff in the code (heavy usage of macros, lots of pointers...), I think it is better to start from scratch and make something more pythonic. As I won't have time to do that, I asked a french company to do the job for us. I'm waiting for an offer. In any case the code will be released under GPL. The architecture of the original C++ code is very nice, with a good isolation between emulation and view. I think we'll keep the same, so it will be easy to adapt it for another toolkit. In fact, the nasty part is the emulation itself (ESC sequences to decode, readline support, and so). Once it is written, the rest will be very easy. If you can't wait, I can send the code I wrote so far (only a few untested classes, and not the emulation one !). PS : this project will allow to run any kind of console-base application (shell, python..., and of course, ipython). -- Fr?d?ric From vivainio at gmail.com Mon Aug 22 12:37:58 2005 From: vivainio at gmail.com (Ville Vainio) Date: Mon, 22 Aug 2005 19:37:58 +0300 Subject: [IPython-dev] Re: [IPython-user] ipython gui interpreters In-Reply-To: <200508221707.02130.mantegazza@ill.fr> References: <4309E405.4060000@indiana.edu> <200508221707.02130.mantegazza@ill.fr> Message-ID: <46cb515a0508220937653a6edf@mail.gmail.com> On 8/22/05, Fr?d?ric Mantegazza wrote: > I started to port konsole, the terminal emulator of KDE, for pyqt, with the > help of Lars Doelle, the konsole author, which sent me a qt-based only > code. > > But as there are a lot of C++ specific stuff in the code (heavy usage of > macros, lots of pointers...), I think it is better to start from scratch > and make something more pythonic. Did you consider merely adding a series of hooks for injecting python code where it's needed, instead of rewriting everything? Or can't you embed konsole in a pyqt app as it stands? -- Ville Vainio - http://tinyurl.com/2prnb