From jorgen.stenarson at bostream.nu Thu Apr 6 14:26:22 2006 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Thu, 06 Apr 2006 20:26:22 +0200 Subject: [IPython-dev] memory leak? Message-ID: <44355D4E.10905@bostream.nu> hi, lately I have been using ipython to %run scripts that create large numeric arrays (in my case say 1000x1000 complex numbers). The problem I see is that when I rerun the script its like all variables created in that file are not freed even though if they have not been accessed in a way that would put them in the In or Out cache. For instance %running a file containing import Numeric a=Numeric.zeros((1000,1000),"D") repeatedly without any intervening commands will have memory usage go up with every invocation of %run. But using execfile from an ordinary python prompt on the same file this is not the case. Is this a known problem (I'm on windows)? If it is, is there some way to manually free this memory? /J?rgen From Fernando.Perez at colorado.edu Fri Apr 7 04:46:27 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Fri, 07 Apr 2006 10:46:27 +0200 Subject: [IPython-dev] memory leak? In-Reply-To: <44355D4E.10905@bostream.nu> References: <44355D4E.10905@bostream.nu> Message-ID: <443626E3.3050702@colorado.edu> J?rgen Stenarson wrote: > hi, > > lately I have been using ipython to %run scripts that create large > numeric arrays (in my case say 1000x1000 complex numbers). The problem I > see is that when I rerun the script its like all variables created in > that file are not freed even though if they have not been accessed in a > way that would put them in the In or Out cache. For instance %running a > file containing > > import Numeric > a=Numeric.zeros((1000,1000),"D") > > repeatedly without any intervening commands will have memory usage go up > with every invocation of %run. But using execfile from an ordinary > python prompt on the same file this is not the case. Is this a known > problem (I'm on windows)? If it is, is there some way to manually free > this memory? I suspect the difference comes from the fact that ipython updates the interactive namespace with the namespace where execfile() ran, so you do end up holding a reference to 'a' in your user namespace. However, on each run this reference gets updated, so at least all previous instances should be freed. IPython can't strictly speaking _leak_ memory in the C sense, given that we have no extension code of our own. What can happen is that we hold references to stuff which prevent garbage collection from doing its job. Just a technicality (since the end effect is the same, lost memory), but I wanted to clarify the language. Try running a garbage collection manually, that may help: import gc gc.collect() and let us know what you see. Cheers, f From jorgen.stenarson at bostream.nu Fri Apr 7 13:45:28 2006 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Fri, 07 Apr 2006 19:45:28 +0200 Subject: [IPython-dev] memory leak? In-Reply-To: <443626E3.3050702@colorado.edu> References: <44355D4E.10905@bostream.nu> <443626E3.3050702@colorado.edu> Message-ID: <4436A538.2050102@bostream.nu> Fernando Perez skrev: > J?rgen Stenarson wrote: >> hi, >> >> lately I have been using ipython to %run scripts that create large >> numeric arrays (in my case say 1000x1000 complex numbers). The problem >> I see is that when I rerun the script its like all variables created >> in that file are not freed even though if they have not been accessed >> in a way that would put them in the In or Out cache. For instance >> %running a file containing >> >> import Numeric >> a=Numeric.zeros((1000,1000),"D") >> >> repeatedly without any intervening commands will have memory usage go >> up with every invocation of %run. But using execfile from an ordinary >> python prompt on the same file this is not the case. Is this a known >> problem (I'm on windows)? If it is, is there some way to manually free >> this memory? > > I suspect the difference comes from the fact that ipython updates the > interactive namespace with the namespace where execfile() ran, so you do > end up holding a reference to 'a' in your user namespace. However, on > each run this reference gets updated, so at least all previous instances > should be freed. > > IPython can't strictly speaking _leak_ memory in the C sense, given that > we have no extension code of our own. What can happen is that we hold > references to stuff which prevent garbage collection from doing its > job. Just a technicality (since the end effect is the same, lost > memory), but I wanted to clarify the language. > > Try running a garbage collection manually, that may help: > > import gc > gc.collect() > > and let us know what you see. > > Cheers, > > f > Hi, The gc.collect() works as long as I have not done something that will put results in Out. I have tried Out.clear() and deleting all _x where x is a number. But that doesn't do the trick, is there some other way of clearing the output cache? /J?rgen From hans_meine at gmx.net Sat Apr 8 07:59:50 2006 From: hans_meine at gmx.net (Hans Meine) Date: Sat, 8 Apr 2006 13:59:50 +0200 Subject: [IPython-dev] memory leak? In-Reply-To: <4436A538.2050102@bostream.nu> References: <44355D4E.10905@bostream.nu> <443626E3.3050702@colorado.edu> <4436A538.2050102@bostream.nu> Message-ID: <200604081359.51145.hans_meine@gmx.net> On Friday 07 April 2006 19:45, J?rgen Stenarson wrote: > The gc.collect() works as long as I have not done something that will > put results in Out. I have tried Out.clear() and deleting all _x where x > is a number. But that doesn't do the trick, is there some other way of > clearing the output cache? Ah, yes! That would be a feature I have been looking for many times already! :-) Sth. like the following would be great: %clearout - clear whole cache %clearout 112 - clear one line %clearout :100 140:150 113 - clear multiple (sub)ranges /me looking forward to Fernandos' "That has been there for ages" comment. (Or to his time machine?) ;-) -- Ciao, / / .o. /--/ ..o / / ANS ooo -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From vivainio at gmail.com Sat Apr 8 19:45:53 2006 From: vivainio at gmail.com (Ville Vainio) Date: Sat, 8 Apr 2006 16:45:53 -0700 Subject: [IPython-dev] memory leak? In-Reply-To: <200604081359.51145.hans_meine@gmx.net> References: <44355D4E.10905@bostream.nu> <443626E3.3050702@colorado.edu> <200604081359.51145.hans_meine@gmx.net> Message-ID: <46cb515a0604081645y5b492116kcfe03c2b84a079b0@mail.gmail.com> On 4/8/06, Hans Meine wrote: > Sth. like the following would be great: > > %clearout - clear whole cache > %clearout 112 - clear one line > %clearout :100 140:150 113 - clear multiple (sub)ranges I'll take a look at this over the next week, I have winter holiday so I should have plenty of time & energy to play with stuff like this. -- Ville Vainio - vivainio.googlepages.com vainio.blogspot.com - g[mail | talk]='vivainio' From vivainio at gmail.com Sun Apr 9 14:54:55 2006 From: vivainio at gmail.com (Ville Vainio) Date: Sun, 9 Apr 2006 11:54:55 -0700 Subject: [IPython-dev] [IPython-user] running ipython from command line with pysh profile does not work In-Reply-To: <70ec82800604090929i6a4b9f78s60d86eeee5ff1551@mail.gmail.com> References: <70ec82800604090048h528e214u8a9051fcbfedf31c@mail.gmail.com> <46cb515a0604090232t1db4f6ddvc1ce05d2d6a9d781@mail.gmail.com> <70ec82800604090241p7bdb1f87ubd7d2bed441fec87@mail.gmail.com> <46cb515a0604090414w4544878eu6ea24efc8e8939f7@mail.gmail.com> <70ec82800604090929i6a4b9f78s60d86eeee5ff1551@mail.gmail.com> Message-ID: <46cb515a0604091154w50e5844eu5d7d19b48dcfdc55@mail.gmail.com> On 4/9/06, Krishna Mohan Gundu wrote: > I have a question for the developers. Is there any good reason, why > magic commands and system commands cannot span multiple lines using > escaping newline. I needed this, as my job has some pretty long sed > commands. I modified iplib.py to get the functionality I needed. I can > post the diff if you are interested. I will play with this feature a > bit more as I need it. The backslash character is "magical" for both python and the underlying mechanism (os.system/magic handling). In the name of simplicity, the magical behaviour on python side was sacrificed (and indeed, it was broken for a long time anyway) and \ is passed to underlying mechanism as-is. Do post the diff against the current svn, though, so we can take a look and discuss it. -- Ville Vainio - vivainio.googlepages.com vainio.blogspot.com - g[mail | talk]='vivainio' From Fernando.Perez at colorado.edu Mon Apr 10 03:40:37 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 10 Apr 2006 09:40:37 +0200 Subject: [IPython-dev] memory leak? In-Reply-To: <46cb515a0604081645y5b492116kcfe03c2b84a079b0@mail.gmail.com> References: <44355D4E.10905@bostream.nu> <443626E3.3050702@colorado.edu> <200604081359.51145.hans_meine@gmx.net> <46cb515a0604081645y5b492116kcfe03c2b84a079b0@mail.gmail.com> Message-ID: <443A0BF5.20407@colorado.edu> Ville Vainio wrote: > On 4/8/06, Hans Meine wrote: > > >>Sth. like the following would be great: >> >>%clearout - clear whole cache >>%clearout 112 - clear one line >>%clearout :100 140:150 113 - clear multiple (sub)ranges > > > I'll take a look at this over the next week, I have winter holiday so > I should have plenty of time & energy to play with stuff like this. Yes, something like this would be nice to have. Note that the cache object already has some basic flushing capabilities: __IP.outputcache.flush?? which would be good to reuse/extend/cleanup. In the meantime, the OP can use at least __IP.outputcache.flush() for a full flush of the output cache (if not a line-targetted one). Cheers, f From Fernando.Perez at colorado.edu Mon Apr 10 03:43:40 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 10 Apr 2006 09:43:40 +0200 Subject: [IPython-dev] memory leak? In-Reply-To: <443A0BF5.20407@colorado.edu> References: <44355D4E.10905@bostream.nu> <443626E3.3050702@colorado.edu> <200604081359.51145.hans_meine@gmx.net> <46cb515a0604081645y5b492116kcfe03c2b84a079b0@mail.gmail.com> <443A0BF5.20407@colorado.edu> Message-ID: <443A0CAC.5050405@colorado.edu> Fernando Perez wrote: > Ville Vainio wrote: > >>On 4/8/06, Hans Meine wrote: >> >> >> >>>Sth. like the following would be great: >>> >>>%clearout - clear whole cache >>>%clearout 112 - clear one line >>>%clearout :100 140:150 113 - clear multiple (sub)ranges >> >> >>I'll take a look at this over the next week, I have winter holiday so >>I should have plenty of time & energy to play with stuff like this. > > > Yes, something like this would be nice to have. Note that the cache object > already has some basic flushing capabilities: > > __IP.outputcache.flush?? > > which would be good to reuse/extend/cleanup. > > In the meantime, the OP can use at least > > __IP.outputcache.flush() > > for a full flush of the output cache (if not a line-targetted one). I should have added that a gc.collect() would probably be a good idea to have at the end of that flush() method. I also noticed that the Out[] object isn't really correctly flushed, only the _NN variables are deleted (that code was probably written before Out[] existed as a proper cache). Just some pointers... Cheers, f From vivainio at gmail.com Tue Apr 11 10:40:22 2006 From: vivainio at gmail.com (Ville Vainio) Date: Tue, 11 Apr 2006 07:40:22 -0700 Subject: [IPython-dev] memory leak? In-Reply-To: <200604081359.51145.hans_meine@gmx.net> References: <44355D4E.10905@bostream.nu> <443626E3.3050702@colorado.edu> <200604081359.51145.hans_meine@gmx.net> Message-ID: <46cb515a0604110740r773d018cgefaa6057a95487dd@mail.gmail.com> On 4/8/06, Hans Meine wrote: > Sth. like the following would be great: > > %clearout - clear whole cache > %clearout 112 - clear one line > %clearout :100 140:150 113 - clear multiple (sub)ranges Now %clear in %clear out works in SVN. At this point I didn't see it necessary to support specifically targeted clears, typically users will probably store valueable (non-throwaway) output cache data in variables anyway. -- Ville Vainio - vivainio.googlepages.com vainio.blogspot.com - g[mail | talk]='vivainio' From ravi at ati.com Tue Apr 11 12:50:39 2006 From: ravi at ati.com (Ravikiran Rajagopal) Date: Tue, 11 Apr 2006 12:50:39 -0400 Subject: [IPython-dev] Embedding threaded IPython without IPShellEmbed Message-ID: <200604111250.39047.ravi@ati.com> Hi, Background: I am trying to use python as a scripting language for a C++ application. The application has a lot of arrays that are exposed to python (via Boost.Python). For debugging and extension code development, I would like to be able to open up a python interpreter, and interactively manipulate the data structures (using new scipy) and visualize them (using matplotlib). Problem: After Py_Initialize, I load up scipy via PyRun_SimpleString. Then, from C++ code, I would like to call the interpreter. Without ipython, I could use PyRun_InteractiveLoop, which would return control to my application on EOF. I would like to be able to do the same with ipython. The trick is also very straightforward: create an IPShellEmbed instance via PyRun_SimpleString, and then call its __call__ method using PyRun_SimpleString whenever I needed an interpreter. However, IPShellEmbed does not support threading options, and that makes it a pain to use Matplotlib. As far as I can see, the only option with IPShellEmbed is to use a single threaded back end (such as Tk) for MPL. I would much prefer to use the GTKAgg back end for MPL, since I would be able to keep the plot windows responsive (in a separate thread) while my application goes about its number crunching. I realize that I only need one instance of the embedded shell. In this case, I would like to make ipython do the necessary thread setup options (see code later) which works perfectly fine when calling the mainloop directly. However, I need ipython to return control back to the application without exiting the GTK/Qt main loop. How do I do this? In other words, how do I get in and out of the ipython mainloop interactively? Code for simply using embedded ipython with pylab GTK backend support: const char *ipython_ini = "import sys\n" "sys.argv = ['','-pylab']\n" "import IPython.Shell\n" "IPython.Shell.start().mainloop()\n"; PyRun_SimpleString( ipythonini ); Code for embedding IPShellEmbed (no pylab support): const char *ipython_ini = "import sys\n" "sys.argv = ['','-pylab']\n" "import IPython.Shell\n" "ish = IPython.Shell.IPShellEmbed(sys.argv)\n"; const char *ipython_run = "ish()\n" PyRun_SimpleString( ipython_ini ); // Much later in the code PyRun_SimpleString( ipython_run ); Code example of what I would like to have but does NOT work: const char *ipython_ini = "import sys\n" "sys.argv = ['','-pylab']\n" "import IPython.Shell\n" "ish = IPython.Shell.start()\n"; const char *ipython_run = "ish.embed_mainloop()\n" PyRun_SimpleString( ipython_ini ); // Much later in the code PyRun_SimpleString( ipython_run ); Any help is greatly appreciated. Thanks, Ravi From jorgen.stenarson at bostream.nu Tue Apr 11 14:11:28 2006 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Tue, 11 Apr 2006 20:11:28 +0200 Subject: [IPython-dev] memory leak? In-Reply-To: <46cb515a0604110740r773d018cgefaa6057a95487dd@mail.gmail.com> References: <44355D4E.10905@bostream.nu> <443626E3.3050702@colorado.edu> <200604081359.51145.hans_meine@gmx.net> <46cb515a0604110740r773d018cgefaa6057a95487dd@mail.gmail.com> Message-ID: <443BF150.8060403@bostream.nu> Ville Vainio skrev: > On 4/8/06, Hans Meine wrote: > >> Sth. like the following would be great: >> >> %clearout - clear whole cache >> %clearout 112 - clear one line >> %clearout :100 140:150 113 - clear multiple (sub)ranges > > Now > > %clear in > %clear out > > works in SVN. At this point I didn't see it necessary to support > specifically targeted clears, typically users will probably store > valueable (non-throwaway) output cache data in variables anyway. > > -- > Ville Vainio - vivainio.googlepages.com > vainio.blogspot.com - g[mail | talk]='vivainio' > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.net > http://scipy.net/mailman/listinfo/ipython-dev > Ville, I think there is one more thing that needs clearing and that is the entries list in the outputcache object. In my attached patch I have removed it completely because I could not see that it was used for anything use for it. I have also attached a patch for another option to the clear command: %clear array which will delete all arrays from the user_ns namespace if ipython was started in -pylab mode. Arrays are determined by pylab.arraytype. There is still some strangeness in the garbage collection process though. If I run In[1] %run some_script.py In[2] a #array variable in script Out[2] array(...) In[3] %run some_script.py In[4] a #array variable in script Out[4] array(...) In[5] %run some_script.py In[5] a #array variable in script Out[5] array(...) In[6] %run some_script.py In[7]%clear out I get back some memory. If I then run gc.collect() a few times at the prompt I will get back some more for each call to gc.collect() this will stabilize after a few calls. I'm not sure what the reason is but I thought I should mention it. /J?rgen -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ipython_clear.patch URL: From hans_meine at gmx.net Sat Apr 15 10:18:45 2006 From: hans_meine at gmx.net (Hans Meine) Date: Sat, 15 Apr 2006 16:18:45 +0200 Subject: [IPython-dev] Missing Import in pspersistence.py Message-ID: <200604151618.51918.hans_meine@gmx.net> Hi! I could not start IPython because I got crash log; The reason is a missing import of sys in pspersistence.py: --- IPython/Extensions/pspersistence.py (Revision 1265) +++ IPython/Extensions/pspersistence.py (Arbeitskopie) @@ -12,7 +12,7 @@ import pickleshare -import inspect,pickle,os,textwrap +import inspect,pickle,os,sys,textwrap from IPython.FakeModule import FakeModule def restore_aliases(self): BTW: Isn't the %store per-profile, analoguous to the history? (I use to store objects of classes defined in extension modules loaded by the profile, but the error conceiled by the above crash was that it was trying to load these when starting w/o profile.) Greetings and a happy easter! Hans -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From vivainio at gmail.com Sun Apr 16 11:06:26 2006 From: vivainio at gmail.com (Ville Vainio) Date: Sun, 16 Apr 2006 08:06:26 -0700 Subject: [IPython-dev] Missing Import in pspersistence.py In-Reply-To: <200604151618.51918.hans_meine@gmx.net> References: <200604151618.51918.hans_meine@gmx.net> Message-ID: <46cb515a0604160806x35531a05ud0795403c71b7591@mail.gmail.com> On 4/15/06, Hans Meine wrote: > Hi! > > I could not start IPython because I got crash log; The reason is a missing > import of sys in pspersistence.py: Good catch, thank for the patch! > BTW: Isn't the %store per-profile, analoguous to the history? Not anymore, I overhauled the whole persistence system a while ago and thought it's better to share the variables across profiles. > Greetings and a happy easter! Indeed! -- Ville Vainio - vivainio.googlepages.com vainio.blogspot.com - g[mail | talk]='vivainio' From jorgen.stenarson at bostream.nu Wed Apr 19 15:32:23 2006 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Wed, 19 Apr 2006 21:32:23 +0200 Subject: [IPython-dev] Patch for wildcard Message-ID: <44469047.8090707@bostream.nu> Ville, attached is a patch to allow using ? as a wildcard for a single character (as in filename globbing). I have also included a file with some tests for the wildcard functionality. The test_wildcard.py is meant to be put in a directory such as IPython/tests. Run the test by using python test_wildcard.py /J?rgen -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ipython_wild_card.patch URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: test_wildcard.py URL: From jorgen.stenarson at bostream.nu Wed Apr 19 15:41:54 2006 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Wed, 19 Apr 2006 21:41:54 +0200 Subject: [IPython-dev] more patches for flush Message-ID: <44469282.80702@bostream.nu> Ville, in the CachedOutput cache_size_min is set to 20 which seems strange, why put in a hardcoded lower limit on the cache_size? Perhaps my change to 3 should be 1 instead but I thought the lower limit maybe should include _, __, and ___. Then there was a bug in the warning message that the cache is being flushed. It referenced a non existent attribute. This message can become very annoying if you have a low cache limit I would prefer to remove it or at least be able to disable it in the config file. /J?rgen -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ipython_flush_traceback.patch URL: From jorgen.stenarson at bostream.nu Wed Apr 19 16:12:11 2006 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Wed, 19 Apr 2006 22:12:11 +0200 Subject: [IPython-dev] pyreadline question In-Reply-To: <46cb515a0604030750l2f9c6f10k9a78afa8cdbf1bf9@mail.gmail.com> References: <442CFEE7.6020000@voidspace.org.uk> <442D0205.9020205@voidspace.org.uk> <442D9160.2090405@bostream.nu> <46cb515a0604030750l2f9c6f10k9a78afa8cdbf1bf9@mail.gmail.com> Message-ID: <4446999B.8080101@bostream.nu> Moved from a thread in ipython-user. I tried to add this to ipy_user_conf.py. But I believe that file is executed after readline is imported which won't work. But perhaps we could apply the patch and continue to think about how to best expose the configuration. /J?rgen Ville Vainio skrev: > On 3/31/06, J?rgen Stenarson wrote: > >> I had to make some backwards incompatible changes to how pyreadline is >> started. Now the recommended way is to "import readline" to initialize >> pyreadline, if you want to set the config_path do: >> >> import pyreadline.rlmain >> pyreadline.rlmain.config_path=r"c:\xxx\pyreadlineconfig.ini" >> >> before "import readline". >> >> Ville will have to decide on how he wants to expose this in ipython. A >> workaround for now is to add those two lines before "import IPython" in >> the ipython.py startup script. > > Perhaps we should put the lines you suggest in > UserConfig/ipy_user_conf.py, in comments? That's the first place the > user *should* look for personal config options... > > Extensions/ipy_system_conf.py is ok as well, considering that > pyreadline is only used on Windows ATM. > > -- > Ville Vainio - vivainio.googlepages.com > vainio.blogspot.com - g[mail | talk]='vivainio' > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ipython_config.patch URL: