From Michael.J.Culbertson at wheaton.edu Thu Sep 2 16:42:28 2004 From: Michael.J.Culbertson at wheaton.edu (Michael Culbertson) Date: Thu, 2 Sep 2004 15:42:28 -0500 (CDT) Subject: [SciPy-dev] optimize.fmin: xtol/ftol Message-ID: Hello, I think there is an error in the termination of the Simplex optimization algorithm implemented by scipy.optimize.fmin(). According to the doc string, xtol and ftol are _relative_ values: xtol -- acceptable relative error in xopt for convergence. ftol -- acceptable relative error in func(xopt) for convergence. Yet, as I read the termination condition: if (max(Num.ravel(abs(sim[1:]-sim[0]))) <= xtol \ and max(abs(fsim[0]-fsim[1:])) <= ftol): break the values are taken as absolute quantities. Shouldn't the condition include some division of the difference by one of the values or a multiplication of the relative tolerances by one of the values for a proper relative comparison? Or, have I misinterpreted the statements above? Thanks, Michael Culbertson From Fernando.Perez at colorado.edu Tue Sep 7 01:28:47 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 06 Sep 2004 23:28:47 -0600 Subject: [SciPy-dev] Python & Scientific computing links? Message-ID: <413D470F.90300@colorado.edu> Hi all, at the excellent BOF session run by Joe Harrington at scipy'04, one of the things that came out was the need for more organized and centralized documentation about using Python for various aspects of scientific computing. An effort is under way to tackle this problem, but I'll let Joe report in full on the whole idea in due time. As my tiny contribution to this, I offered to collect a simple page of links and short summaries of any project out on the web which is reasonably connected to scientific computing. Obviously I have my own bookmarks list which I'll use as a starting point, but I'm asking here for all of you to simply mail me your favorite list of links on this topic. Feel free to add extra info which may help me, but it's OK if it's just a list of URLs. I'll collect them, check that at least things are active, and write a short summary for each project (probably copy/paste from their own pages). In a couple of weeks, I'll mail this page to Joe so it can go into the relevant area of the Scipy.org website, where it will most likely end up as a Wiki page. From that point on, the community can continue to improve it, correct my mistakes, and keep it updated as new projects of interest appear or old ones vanish. Best, f. From bulatov at cs.orst.edu Wed Sep 8 18:13:26 2004 From: bulatov at cs.orst.edu (bulatov at cs.orst.edu) Date: Wed, 8 Sep 2004 15:13:26 -0700 Subject: [SciPy-dev] Python & Scientific computing links? In-Reply-To: <413D470F.90300@colorado.edu> References: <413D470F.90300@colorado.edu> Message-ID: <1094681606.413f8406c9c89@webmail.oregonstate.edu> Here's a list of Artificial Intelligence/Machine Learning links I compiled with help from comp.lang.python http://yaroslav.hopto.org/pubwiki/index.php/ai-python Yaroslav ??????? Fernando Perez : > Hi all, > > at the excellent BOF session run by Joe Harrington at scipy'04, one of the > things that came out was the need for more organized and centralized > documentation about using Python for various aspects of scientific > computing. > > An effort is under way to tackle this problem, but I'll let Joe report in > full > on the whole idea in due time. As my tiny contribution to this, I offered to > > collect a simple page of links and short summaries of any project out on the > > web which is reasonably connected to scientific computing. > > Obviously I have my own bookmarks list which I'll use as a starting point, > but > I'm asking here for all of you to simply mail me your favorite list of links > > on this topic. Feel free to add extra info which may help me, but it's OK if > > it's just a list of URLs. I'll collect them, check that at least things are > > active, and write a short summary for each project (probably copy/paste from > > their own pages). > > In a couple of weeks, I'll mail this page to Joe so it can go into the > relevant area of the Scipy.org website, where it will most likely end up as a > > Wiki page. From that point on, the community can continue to improve it, > correct my mistakes, and keep it updated as new projects of interest appear > or > old ones vanish. > > Best, > > f. > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From Fernando.Perez at colorado.edu Wed Sep 8 19:04:39 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed, 08 Sep 2004 17:04:39 -0600 Subject: [SciPy-dev] Python & Scientific computing links? In-Reply-To: <1094681606.413f8406c9c89@webmail.oregonstate.edu> References: <413D470F.90300@colorado.edu> <1094681606.413f8406c9c89@webmail.oregonstate.edu> Message-ID: <413F9007.4020205@colorado.edu> bulatov at cs.orst.edu wrote: > Here's a list of Artificial Intelligence/Machine Learning links I compiled with > help from comp.lang.python > > http://yaroslav.hopto.org/pubwiki/index.php/ai-python Great, thanks! f From eric at enthought.com Sun Sep 12 05:32:25 2004 From: eric at enthought.com (eric jones) Date: Sun, 12 Sep 2004 02:32:25 -0700 Subject: [SciPy-dev] SciPy WishList and Priorities Message-ID: <414417A9.6090403@enthought.com> Hey group, One of the request at the SciPy '04 conference was a list of priorities, etc. that the developers felt needed to be addressed. It is a target rich environment... :-) I've put together a stream-of-conciousness set of notes on the Wiki with my 0.02 worth. Travis )., Pearu, and others, feel free to add to this. http://www.scipy.org/wikis/featurerequests/PrioritiesAndWishlist I haven't really specified an priorities or actionable plans yet (feel free to add this yourself...), but we know have a place to start discussing. Thanks for all who came to the conference. I thought it was another great success and there was a lot of useful discussion about where we should go and how. It really feels like to community is becoming organized and mature enough to attack some of the issues that have been outstanding for a long time. Hopefully the community will make notable progress by the time SciPy '05 roles around. thanks, eric From Fernando.Perez at colorado.edu Mon Sep 13 16:00:24 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 13 Sep 2004 14:00:24 -0600 Subject: [SciPy-dev] SciPy WishList and Priorities In-Reply-To: <414417A9.6090403@enthought.com> References: <414417A9.6090403@enthought.com> Message-ID: <4145FC58.1090703@colorado.edu> eric jones wrote: > Hey group, > > One of the request at the SciPy '04 conference was a list of priorities, > etc. that the developers felt needed to be addressed. It is a target > rich environment... :-) > > I've put together a stream-of-conciousness set of notes on the Wiki with > my 0.02 worth. Travis )., Pearu, and others, feel free to add to this. > > http://www.scipy.org/wikis/featurerequests/PrioritiesAndWishlist > Sorry if I'm being dense, but how can I edit that page? I'd like to add some comments, and even though I logged in, the only wiki buttons I see are for printing and email. Am I missing something? Best, f From oliphant at ee.byu.edu Mon Sep 13 20:20:28 2004 From: oliphant at ee.byu.edu (Travis E. Oliphant) Date: Mon, 13 Sep 2004 18:20:28 -0600 Subject: [SciPy-dev] Re: [AstroPy] Python version of IDL Astron library In-Reply-To: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu> References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu> Message-ID: <4146394C.5090407@ee.byu.edu> Since Tom invited me into the discussion, I will comment, despite the fact that I am not an astronomer. > Hi folks- > > Sorry to have missed all of the interesting discussion on this; I was > at the HEAD meeting (on a panel on astrostatistics software, among > other things---lots of folks are interested in Python, by the way). > > Joe's summary of the main issues was very good. I agree on many > points; here are some possible disagreements. > > >>UNDERLYING PACKAGES >> >>1. Numarray for all new code, and to teach to new users. The reality is that Numarray is not ready as a drop-in replacement for Numeric but people keep introducing it to new users as if it were ready. This is a real problem in the community right now. These issues have been discussed before and are on the scipy.org wiki that Eric has started. Basically the issues preventing a drop-in replacement are 1) small arrays are too slow, 2) no backward-compatible ufunc C-API --- nothing like the compatibility API for arrays exists for ufuncs. 3) migrating legacy code is way too time consuming and effort filled for almost no gain. While not optimal the current best solution is to make Numarray more aware of Numeric arrays and vice versa, so that unnecessary memory copying is not happening. > > >>2. Matplotlib for all plotting. Try it. You'll like it! > Chaco is nearly there. In some ways I feel bad that John Hunter didn't use his energy to making easy plots using Chaco/ Tk instead of developing yet another infrastructure. It is true that matplotlib got to a functional state more quickly, but chaco has a much more sturdy foundation for development, I think. The Enable library that Enthought has put together allows amazing possibilities and improved portability. Their motives are completely pure, though their time is limited. It would be nice if Enthought published the location of their chaco SVN server so that users could see the current code and not the legacy stuff in SciPy. I think the reason they haven't is more of a logistic issue (not being ready to support inquiries) and perhaps separating it from other code bases. But, if Eric is reading this, I would suggest to him, that not publishing and promoting the SVN site creates more problems and discourages potential users in just the way Tom is expressing. > > Do either of these packages (matplotlib, Chaco) allow one to interact > with plots in a way that lets Python easily know where on a plot the > user is clicking? I.e., is there the basis for making "interactive" > plots, not merely in the sense of panning and zooming, but in actually > manipulating points on the plots as controls (maybe "dynamic" would be > a better adjective)? I have in mind something like SysQuake > (http://www.calerga.com/). > I believe chaco/enable makes this kind of thing very possible. > g77 is free and trivial to install on OS X. In my experience > Fortran on linux and the Mac isn't problematical. Is this a > Windows problem? Are there really many astronomers using Windows for > scientific calculation? g77 also works under Windows. This should not be a problem. SciPy is compiled under Windows using g77 constantly. > >>DOCS >> >>Documentation is a strong suit of IDL's and we need to be as good. I >>am not aware of a standard doc header for Python. > > > What little there is that is standard is outlined by Guido: > > http://www.python.org/doc/essays/styleguide.html > > >>The source language of docs is another issue. It has to be open and >>functional on all platforms. It has to handle simple markup and >>inline figures. It has to produce PDF and HTML. MSWord is out. >>Research Structured Text or LaTeX are my votes. Some like LyX and >>TeXmacs. We'll see what the community wants. > > > I'd love to see a format that allows including equations somehow, or > some "web"-like framework (referring to Knuth's WEB, not the world-wide > one) that allows one to jointly maintain fairly extensive technical > documentation with the code. I say, use LaTeX format in equations in doc strings if necessary and use LyX as a format for documentation. The file format is structured text that with the LyX editor becomes very easy to manipulate. > > I believe Bill Press and Saul Teukolsky wrote software (or had it > written) for maintaining Numerical Recipes code within LaTeX for > their book. Perhaps we can find out about what they did to see > if it might be adaptable to our purposes. > > >>COORDINATION > > > Regarding coordination with SciPy, it may be worth pointing out > that the astrostat project I have funded by AISR supports Travis > Oliphant for a month a year to support the project. He's offered > to host our Inference package at SciPy.org. Perhaps I can persuade > him to make that just a part of a larger set of packages, and get > his help behind hosting our effort at SciPy.org (in this last year of > funding!). > I hope that the AstroPy coordinates with SciPy. A lot of build issues have been worked out in SciPy. SciPy has been explicitly designed to work with subpackages. I see no reason that much of what astropy wants to have available can't be a sub-package of SciPy. If there are reasons, then we need to fix them on the SciPy end, or the stated purpose of SciPy is not being fulfilled. Now, I can also see the blossoming of very special-purpose packages, but the line between special-purpose and general is changing all the time. However, if there are general-purpose algorithms that apply beyond astronomy, they really should be in SciPy and not in a different location. SciPy is supposed to be a community effort. There is room for many more developers in the SciPy world. -Travis Oliphant From oliphant at ee.byu.edu Mon Sep 13 20:27:29 2004 From: oliphant at ee.byu.edu (Travis E. Oliphant) Date: Mon, 13 Sep 2004 18:27:29 -0600 Subject: [SciPy-dev] Re: [AstroPy] Python version of IDL Astron library In-Reply-To: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu> References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu> Message-ID: <41463AF1.1090902@ee.byu.edu> > > Regarding coordination with SciPy, it may be worth pointing out > that the astrostat project I have funded by AISR supports Travis > Oliphant for a month a year to support the project. He's offered > to host our Inference package at SciPy.org. Perhaps I can persuade > him to make that just a part of a larger set of packages, and get > his help behind hosting our effort at SciPy.org (in this last year of > funding!). You can easily persuade me to help making anything that is deemed acceptable a part of the scipy package. We want SciPy to be general purpose and easily available. I am anxious to hear about what I can do to help get your inference routines into SciPy. -Travis From perry at stsci.edu Mon Sep 13 20:43:20 2004 From: perry at stsci.edu (Perry Greenfield) Date: Mon, 13 Sep 2004 20:43:20 -0400 Subject: [SciPy-dev] Re: [AstroPy] Python version of IDL Astron library In-Reply-To: <4146394C.5090407@ee.byu.edu> References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu> <4146394C.5090407@ee.byu.edu> Message-ID: <131B68FF-05E7-11D9-B225-000393989D66@stsci.edu> On Sep 13, 2004, at 8:20 PM, Travis E. Oliphant wrote: > > The reality is that Numarray is not ready as a drop-in replacement for > Numeric but people keep introducing it to new users as if it were > ready. This is a real problem in the community right now. > > These issues have been discussed before and are on the scipy.org wiki > that Eric has started. Basically the issues preventing a drop-in > replacement are 1) small arrays are too slow, 2) no > backward-compatible ufunc C-API --- nothing like the compatibility API > for arrays exists for ufuncs. 3) migrating legacy code is way too > time consuming and effort filled for almost no gain. > > While not optimal the current best solution is to make Numarray more > aware of Numeric arrays and vice versa, so that unnecessary memory > copying is not happening. > Well, we want to go beyond that to the point that scipy and other Numeric packages can support numarray and Numeric simultaneously. We intend to do most of the work to make it so with scipy. f2py already has been adapted to work with numarray, and we would also take a look at the ufunc api issue to deal with that. I think we should start on this fairly soon (certainly within a month or so). This is pretty much our highest priority now that numarray 1.1 is out. Perry From eric at enthought.com Tue Sep 14 03:54:35 2004 From: eric at enthought.com (eric jones) Date: Tue, 14 Sep 2004 00:54:35 -0700 Subject: [SciPy-dev] Re: [AstroPy] Python version of IDL Astron library In-Reply-To: <4146394C.5090407@ee.byu.edu> References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu> <4146394C.5090407@ee.byu.edu> Message-ID: <4146A3BB.4020208@enthought.com> Travis E. Oliphant wrote: > Since Tom invited me into the discussion, I will comment, despite the > fact that I am not an astronomer. > >> Hi folks- >> >> Sorry to have missed all of the interesting discussion on this; I was >> at the HEAD meeting (on a panel on astrostatistics software, among >> other things---lots of folks are interested in Python, by the way). >> >> Joe's summary of the main issues was very good. I agree on many >> points; here are some possible disagreements. >> >> >>> UNDERLYING PACKAGES >>> >>> 1. Numarray for all new code, and to teach to new users. >> > > The reality is that Numarray is not ready as a drop-in replacement for > Numeric but people keep introducing it to new users as if it were > ready. This is a real problem in the community right now. > > These issues have been discussed before and are on the scipy.org wiki > that Eric has started. Basically the issues preventing a drop-in > replacement are 1) small arrays are too slow, 2) no > backward-compatible ufunc C-API --- nothing like the compatibility API > for arrays exists for ufuncs. 3) migrating legacy code is way too > time consuming and effort filled for almost no gain. > > While not optimal the current best solution is to make Numarray more > aware of Numeric arrays and vice versa, so that unnecessary memory > copying is not happening. > >> >> >>> 2. Matplotlib for all plotting. Try it. You'll like it! >> >> > > Chaco is nearly there. In some ways I feel bad that John Hunter > didn't use his energy to making easy plots using Chaco/ Tk instead of > developing yet another infrastructure. It is true that matplotlib got > to a functional state more quickly, but chaco has a much more sturdy > foundation for development, I think. While I agree, I also appreciate the amount of progress that matplotlib has made. In some sense, it takes the pressure off of Chaco. For overall development efficiency in the long run, I believe it would have been better if the matplotlib energy had gone into Chaco. Perry and the STSci team tried though, and didn't make the headway the wished for. So, in the short run, I don't think as usable of a tool would have resulted as soon -- we just didn't have the resources to coordinate with the community over the last year to fix the deficiencies that made it difficult for scientist to use. I am quite sure that many of these deficiencies will be addressed over the next year. I am also excited about the capabilities its underlying architecture provides advanced interactive plotting within applications -- our main use of it now. We will continue to put significant effort into it as we rely on it heavily. Both projects will continue to advance I'm sure, and also push each other to advance. Because their licensing allow them to incorporate features from the other toolset, the story isn't so bad. Matplotlib can evolve and solve one set of problems. Chaco can evolve to solve another. In the future the can borrow from each other where appropriate and, who knows, maybe even combine efforts. Everyone here is after the same goal -- a solid plotting toolkit for Python. With two viable toolkits under construction creating a little healthy competition, I think we are more likely to get there now than at any point in the past -- in fact, it sounds like matplotlib solves a majority of the scientists day-to-day plotting needs now. This is good news all around. > > The Enable library that Enthought has put together allows amazing > possibilities and improved portability. Their motives are completely > pure, though their time is limited. It would be nice if Enthought > published the location of their chaco SVN server so that users could > see the current code and not the legacy stuff in SciPy. I think the > reason they haven't is more of a logistic issue (not being ready to > support inquiries) and perhaps separating it from other code bases. > But, if Eric is reading this, I would suggest to him, that not > publishing and promoting the SVN site creates more problems and > discourages potential users in just the way Tom is expressing. Here it is: http://svn.enthought.com/svn/enthought/ I'm pretty sure anaonymous access is enabled, but I haven't checked. There is other stuff in here but not much in terms of documentation. And, as Travis says, there isn't much email support on it at the moment as it is not fully baked for release. It will get there though. There are a number of people working on this tree on a daily basis. > >> >> Do either of these packages (matplotlib, Chaco) allow one to interact >> with plots in a way that lets Python easily know where on a plot the >> user is clicking? I.e., is there the basis for making "interactive" >> plots, not merely in the sense of panning and zooming, but in actually >> manipulating points on the plots as controls (maybe "dynamic" would be >> a better adjective)? I have in mind something like SysQuake >> (http://www.calerga.com/). >> It is a little to much to talk about here, but yes Enable/Chaco are built from the ground up to support this. It is not exposed for this to be done as easily as it should be. see ya, eric From Fernando.Perez at colorado.edu Tue Sep 14 03:24:15 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 14 Sep 2004 01:24:15 -0600 Subject: [SciPy-dev] Re: [AstroPy] Python version of IDL Astron library In-Reply-To: <4146A3BB.4020208@enthought.com> References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu> <4146394C.5090407@ee.byu.edu> <4146A3BB.4020208@enthought.com> Message-ID: <41469C9F.8040707@colorado.edu> eric jones wrote: >>>>2. Matplotlib for all plotting. Try it. You'll like it! > While I agree, I also appreciate the amount of progress that matplotlib > has made. In some sense, it takes the pressure off of Chaco. For > overall development efficiency in the long run, I believe it would have > been better if the matplotlib energy had gone into Chaco. Perry and the > STSci team tried though, and didn't make the headway the wished for. > So, in the short run, I don't think as usable of a tool would have > resulted as soon -- we just didn't have the resources to coordinate with > the community over the last year to fix the deficiencies that made it > difficult for scientist to use. I've been one of those waiting for chaco since its first announcement, and my approach was to use Gnuplot2 (IPython's Gnuplot.py extensions) while Chaco evolved. But this has taken a while, and in the meantime for day to day scientific use matplotlib has rapidly matured. So I actually think it will work out for the best in the long run: those of us who need something better than gnuplot/grace now have a realistic option to switch. I hope it's clear that I'm NOT bashing Chaco in any way, but those of us who needed high quality EPS 6 months or a year ago simply couln't wait for it :) We should not underestimate that having these tools _today_ makes it much easier to 'sell' python to new scientific users. I've committed effort on ipython to coexist with matplotlib as efficiently as possible precisely because I believe this is an important piece of the puzzle. Matplotlib's existence also gives Chaco room to breathe and develop as it needs, at its pace. I'm sure that in the future, useful Chaco/matplotlib cross-feedback will occur when that makes sense. Perhaps we all would have wanted J. Hunter to hack on Chaco for the last year. But at least now we have two viable systems, one which works today and the other which is building a very solid foundation for the future. I can only see this as a good thing in the long run. Cheers, f From travis at enthought.com Tue Sep 14 11:39:02 2004 From: travis at enthought.com (Travis N. Vaught) Date: Tue, 14 Sep 2004 09:39:02 -0600 Subject: [SciPy-dev] SciPy WishList and Priorities In-Reply-To: <4145FC58.1090703@colorado.edu> References: <414417A9.6090403@enthought.com> <4145FC58.1090703@colorado.edu> Message-ID: <41471096.9050108@enthought.com> All wiki info should be available for editing for all scipy.org Members now. Apologies for the lock-down. Travis V. Fernando Perez wrote: > > Sorry if I'm being dense, but how can I edit that page? I'd like to > add some comments, and even though I logged in, the only wiki buttons > I see are for printing and email. Am I missing something? > > Best, > > f From eric at enthought.com Tue Sep 14 15:03:20 2004 From: eric at enthought.com (eric jones) Date: Tue, 14 Sep 2004 12:03:20 -0700 Subject: [SciPy-dev] Re: [AstroPy] Python version of IDL Astron library In-Reply-To: <41469C9F.8040707@colorado.edu> References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu> <4146394C.5090407@ee.byu.edu> <4146A3BB.4020208@enthought.com> <41469C9F.8040707@colorado.edu> Message-ID: <41474078.6090103@enthought.com> Fernando Perez wrote: > eric jones wrote: > >>>>> 2. Matplotlib for all plotting. Try it. You'll like it! >>>> > >> While I agree, I also appreciate the amount of progress that >> matplotlib has made. In some sense, it takes the pressure off of >> Chaco. For overall development efficiency in the long run, I believe >> it would have been better if the matplotlib energy had gone into >> Chaco. Perry and the STSci team tried though, and didn't make the >> headway the wished for. So, in the short run, I don't think as >> usable of a tool would have resulted as soon -- we just didn't have >> the resources to coordinate with the community over the last year to >> fix the deficiencies that made it difficult for scientist to use. > > > I've been one of those waiting for chaco since its first announcement, > and my approach was to use Gnuplot2 (IPython's Gnuplot.py extensions) > while Chaco evolved. But this has taken a while, and in the meantime > for day to day scientific use matplotlib has rapidly matured. So I > actually think it will work out for the best in the long run: those of > us who need something better than gnuplot/grace now have a realistic > option to switch. I hope it's clear that I'm NOT bashing Chaco in any > way, but those of us who needed high quality EPS 6 months or a year > ago simply couln't wait for it :) > > We should not underestimate that having these tools _today_ makes it > much easier to 'sell' python to new scientific users. I've committed > effort on ipython to coexist with matplotlib as efficiently as > possible precisely because I believe this is an important piece of the > puzzle. > > Matplotlib's existence also gives Chaco room to breathe and develop as > it needs, at its pace. I'm sure that in the future, useful > Chaco/matplotlib cross-feedback will occur when that makes sense. > > Perhaps we all would have wanted J. Hunter to hack on Chaco for the > last year. But at least now we have two viable systems, one which > works today and the other which is building a very solid foundation > for the future. I can only see this as a good thing in the long run. This is exactly what I was trying to say, but perhaps not as well. Having a solution sooner fixes many issues, and that is very beneficial for the reasons you mention. However, from a strictly person-hours required to reach the long term goal of a utopian plotting toolkit, it would be more efficient to have all hands working on the same project. There are alwasy tradeoffs, and I think that the current situation (two packages) is not bad at all. eric > > Cheers, > > f > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev From eric at enthought.com Tue Sep 14 15:04:28 2004 From: eric at enthought.com (eric jones) Date: Tue, 14 Sep 2004 12:04:28 -0700 Subject: [SciPy-dev] SciPy WishList and Priorities In-Reply-To: <41471096.9050108@enthought.com> References: <414417A9.6090403@enthought.com> <4145FC58.1090703@colorado.edu> <41471096.9050108@enthought.com> Message-ID: <414740BC.7000508@enthought.com> Fernando, Please confirm when you get a chance that this is indeed working correctly for you. eric Travis N. Vaught wrote: > All wiki info should be available for editing for all scipy.org > Members now. Apologies for the lock-down. > > Travis V. > > Fernando Perez wrote: > >> >> Sorry if I'm being dense, but how can I edit that page? I'd like to >> add some comments, and even though I logged in, the only wiki buttons >> I see are for printing and email. Am I missing something? >> >> Best, >> >> f > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev From jdhunter at ace.bsd.uchicago.edu Tue Sep 14 13:20:55 2004 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Tue, 14 Sep 2004 12:20:55 -0500 Subject: [SciPy-dev] Re: [AstroPy] Python version of IDL Astron library In-Reply-To: <41474078.6090103@enthought.com> (eric jones's message of "Tue, 14 Sep 2004 12:03:20 -0700") References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu> <4146394C.5090407@ee.byu.edu> <4146A3BB.4020208@enthought.com> <41469C9F.8040707@colorado.edu> <41474078.6090103@enthought.com> Message-ID: >>>>> "eric" == eric jones writes: Fernando> Perhaps we all would have wanted J. Hunter to hack on Chaco for Fernando> the last year. But at least now we have two viable systems, Fernando> one which works today and the other which is building a very Fernando> solid foundation for the future. I can only see this as a good Fernando> thing in the long run. When I was shopping for plotting libraries back in the day, I did give serious consideration to chaco. My needs were fairly specific: to find a high quality 2D lib I could embed in a GTK app, which Chaco did not support. It would have been a better use of community resources if I had extended Chaco to do what I needed, but at the time I was primarily considering the best use of *my resources*. It's a blessing and a partly a curse that python is so powerful, easy and fun to program that it can sometimes be easier to roll your own solution than extend someone else's. My goals at the time were limited and I wasn't thinking about developing a general purpose plotting library. Hindsight, as they say, is twenty-twenty. eric> This is exactly what I was trying to say, but perhaps not as eric> well. Having a solution sooner fixes many issues, and that eric> is very beneficial for the reasons you mention. However, eric> from a strictly person-hours required to reach the long term eric> goal of a utopian plotting toolkit, it would be more eric> efficient to have all hands working on the same project. eric> There are alwasy tradeoffs, and I think that the current eric> situation (two packages) is not bad at all. Yep, and as you mentioned in your other post, there are certainly possibilities for cross-pollination. My first priority vis-a-vis matplotlib is to add the two remaining essential plot types: contour and polar. After that, I plan to make some changes enhancing the core. Here are a few partially baked ideas that I'm ruminating over rendering: matplotlib still uses a variant of the gtk drawing model in the backend, which doesn't support arbitrary clipping paths, even-odd filling rules and the like. So I plan at some point to introduce a new drawing model and perhaps kiva would fit the bill. If matplotlib and chaco both used the same backend API then efforts on the backend of one package would be useful to the other. It would also make it easier to port frontend code from one package to another. Along these lines, I wonder how satisfied you are with the kiva API and implementation. I'm not sure this is the route I want to go, but it has come up in discussions with Perry and Paul on a couple of occasions and clearly has merit. envisage: I definitely have a need for a scientific application framework, and envisage and enable looked awesome; enable in particular addresses a lot of problems I keep encountering trying to support basic functionality across GUIs. I talked to David and Martin at scipy about the possibility of plugging matplotlib into envisage, and they were supportive of the concept but I gathered from them that the envisage API is still somewhat fluid. When this settles, if you guys are able to provide a 2D canvas/plotting API for envisage, I would be happy to do the work to port matplotlib to it. traits: I thought the new traits mode you presented at scipy looked great, and there has been interest for some time in using them in matplotlib. Since you've already ported these to wxpython and tk (right?), my guess is that I could add pygtk support, and the author of the matplotlib fltk backend might do the same for fltk, which would at least cover the spectrum of current matplotlib backends. Using traits in matplotlib might also help with matplotlib / envisage integration down the road. mathtext: As I mentioned at scipy, I would like to improve the mathtext layout engine. It's already decent, but there are definitely areas where it could be better. One good idea that came up at scipy was to use TeX where available, and I'm looking in to that. Almost all of the permutations off TeX that I've found thus far are GPL'd, which is unfortunate in my view because Knuth released TeX under an extremely permissive license -- basically the only restriction is that if you change the src, you must rename the file. In any case, whether I improve the matplotlib layout engine, expose tex, or both, chaco could probably reuse some of this code for mathematical expressions. Cheers! JDH From Fernando.Perez at colorado.edu Tue Sep 14 14:36:50 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 14 Sep 2004 12:36:50 -0600 Subject: [SciPy-dev] SciPy WishList and Priorities In-Reply-To: <414740BC.7000508@enthought.com> References: <414417A9.6090403@enthought.com> <4145FC58.1090703@colorado.edu> <41471096.9050108@enthought.com> <414740BC.7000508@enthought.com> Message-ID: <41473A42.4030703@colorado.edu> eric jones wrote: > Fernando, > > Please confirm when you get a chance that this is indeed working > correctly for you. It takes a loooong time to respond after clicking on 'Edit', but it works fine after that. I don't know if the slow response is a plone issue. At first I thought it had died, but it eventually came through. Once the edit page opened, saving my changes worked just fine. I added some comments about ipython, mainly to test the edit feature. Later I'll work more on that page. I'm also putting together the links list we discussed at the conference; I contacted Python.org and Skip Montanaro authorized using all their existing ScientificComputing pages (old but useful starter). This week I'm swamped, but hopefully next week I'll work a bit on this and put it up in the wiki. thanks! f From rkern at ucsd.edu Wed Sep 15 07:12:56 2004 From: rkern at ucsd.edu (Robert Kern) Date: Wed, 15 Sep 2004 04:12:56 -0700 Subject: [SciPy-dev] Re: [AstroPy] Python version of IDL Astron library In-Reply-To: References: <200409132204.i8DM4eV12702@laplace.astro.cornell.edu> <4146394C.5090407@ee.byu.edu> <4146A3BB.4020208@enthought.com> <41469C9F.8040707@colorado.edu> <41474078.6090103@enthought.com> Message-ID: <414823B8.6020302@ucsd.edu> John Hunter wrote: [snip] > rendering: matplotlib still uses a variant of the gtk drawing model > in the backend, which doesn't support arbitrary clipping paths, > even-odd filling rules and the like. So I plan at some point to > introduce a new drawing model and perhaps kiva would fit the bill. > If matplotlib and chaco both used the same backend API then efforts > on the backend of one package would be useful to the other. It > would also make it easier to port frontend code from one package to > another. Along these lines, I wonder how satisfied you are with the > kiva API and implementation. I'm not sure this is the route I want > to go, but it has come up in discussions with Perry and Paul on a > couple of occasions and clearly has merit. I think it works quite well. It needs some additions here and there. * I've implemented color gradients on the Quartz backend, but not all of the target backends support arbitrary functions for gradients. * Elliptical arcs should be added (I did these for reportlab years ago, so I guess I should do the same for Kiva). * Font-finding is, um, sub-optimal. > mathtext: As I mentioned at scipy, I would like to improve the > mathtext layout engine. It's already decent, but there are > definitely areas where it could be better. One good idea that came > up at scipy was to use TeX where available, and I'm looking in to > that. Almost all of the permutations off TeX that I've found thus > far are GPL'd, which is unfortunate in my view because Knuth > released TeX under an extremely permissive license -- basically the > only restriction is that if you change the src, you must rename the > file. I don't think anyone is suggesting that matplotlib distribute any part of TeX. Just use the programs if they are installed and manipulate the output. Suggest other programs like dvipng as appropriate. Use os.system and you don't have to worry about the GPL. Fall back to the current approach when the programs are unavailable. Kiva's Quartz backend can import PDF, so ideally, one can run pdftex on the snippet and draw the vector data directly onto the Kiva canvas with arbitrary transformation. Other backends will probably have to rely on a raster intermediate or figure out a way to import PDF. > In any case, whether I improve the matplotlib layout engine, > expose tex, or both, chaco could probably reuse some of this code > for mathematical expressions. > > Cheers! > JDH -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From alexey.goldin at gmail.com Thu Sep 16 13:28:23 2004 From: alexey.goldin at gmail.com (Alexey Goldin) Date: Thu, 16 Sep 2004 10:28:23 -0700 Subject: [SciPy-dev] Re: Python & Scientific computing links? In-Reply-To: References: Message-ID: I just found a great resource --- "social bookmarking tool" at http://del.icio.us. You can use it as a bookmark manager like bookmarks.yahoo.com, however this is much more then that. You can tag your bookmarks with one or more tag and browse other people bookmarks if they use the same tag. The database is accessible through RSS and some kind of RPC as well. Posting a bookmark via bookmarklets or Mozilla extension is trivial. For example, here is RSS feed for python related resources: http://del.icio.us/rss/hedgehog/python and here is one for scientific python: http://del.icio.us/rss/hedgehog/scipy (I think for now I am the only one using scipy tag for now) Using RSS feed is a nice way to discover resources other people found.... From prabhu at aero.iitm.ernet.in Sun Sep 19 06:41:05 2004 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Sun, 19 Sep 2004 16:11:05 +0530 Subject: [SciPy-dev] Performance Python with Weave article updated Message-ID: <16717.25153.598977.19108@monster.linux.in> Hi folks, Just to let you know that I've updated the article on weave compared with other options here: http://www.scipy.org/documentation/weave/weaveperformance.html The examples now work with weave from SciPy-0.3. I've also added a comparison with Psyco and Pyrex. The tarball has also been updated and includes all the sources along with a trivial setup.py to build the f2py module and the Pyrex module. cheers, prabhu From luthi at gi.alaska.edu Mon Sep 20 22:44:03 2004 From: luthi at gi.alaska.edu (=?iso-8859-1?q?Martin_L=FCthi?=) Date: 20 Sep 2004 18:44:03 -0800 Subject: [SciPy-dev] Re: Performance Python with Weave article updated In-Reply-To: <16717.25153.598977.19108@monster.linux.in> References: <16717.25153.598977.19108@monster.linux.in> Message-ID: Hi Prabhu Prabhu Ramachandran writes: > Just to let you know that I've updated the article on weave compared > with other options here: > > http://www.scipy.org/documentation/weave/weaveperformance.html Thanks for that article. Are you sure that the following line does what you mean? u[i,j] = ((u[i-1, j] + u[i+1, j])*dy^2 + Exponentiation in Python is with **, not with ^ Best, Martin From prabhu at aero.iitm.ernet.in Tue Sep 21 20:57:40 2004 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Wed, 22 Sep 2004 06:27:40 +0530 Subject: [SciPy-dev] Re: Performance Python with Weave article updated In-Reply-To: References: <16717.25153.598977.19108@monster.linux.in> Message-ID: <16720.52740.77392.560473@monster.linux.in> >>>>> "ML" == Martin L?thi writes: ML> Hi Prabhu Prabhu Ramachandran ML> writes: >> Just to let you know that I've updated the article on weave >> compared with other options here: >> >> http://www.scipy.org/documentation/weave/weaveperformance.html ML> Thanks for that article. Are you sure that the following line ML> does what you mean? ML> u[i,j] = ((u[i-1, j] + u[i+1, j])*dy^2 + ML> Exponentiation in Python is with **, not with ^ The first code sample was meant to be pseudo-code. It looks like Python, but is not working code. Only the subsequent code samples represent working Python code and there I use ** and not ^. Perhaps, I should change it anyway in the interest of consistency. Thanks for bringing it up. cheers, prabhu From mmetz at astro.uni-bonn.de Wed Sep 22 05:29:41 2004 From: mmetz at astro.uni-bonn.de (Manuel Metz) Date: Wed, 22 Sep 2004 11:29:41 +0200 Subject: [SciPy-dev] Weave - strange behaviour Message-ID: <41514605.4040705@astro.uni-bonn.de> Hi, I was really excited about weave. But I found a (to me) somewhat strange behaviour, which I do not understand. I used the following code to test weave: [...] class test(object): def wtest(self): a = numarray.array( [1,2,3] ) assert( type(a)==type( numarray.array([]) ) ) ccode2 = """ int val; using namespace std; for(int i=0; i>>>> file changed None 1 2 3 <<<<< Now if I only change the last line of the C-code from }""" to } """ so nothing more than just a line-break, I always (even after the C-code is already compiled) get the following output: >>>>> repairing catalog by removing key file changed None 1 2 3 <<<<< And it seams that EACH time I call the function the C-code gets compiled. This behavior also showed up depending on the first line of the C-code (line break or not; additionen spaces). Has anyone else seen this? Any idea what is going wrong? How can I prevent this strange behaviour (except try and error)? My system: - Debian linux (testing) - Package: python2.3-scipy Version: 0.3.0+266.4239-1 Manuel -- ------------------------------------- Manuel Metz Sternwarte der Universitaet Bonn Auf dem Huegel 71 (room 3.06) D - 53121 Bonn E-Mail: mmetz at astro.uni-bonn.de Phone: (+49) 228 / 73-3660 Fax: (+49) 228 / 73-3672 ------------------------------------- From charles.harris at sdl.usu.edu Wed Sep 22 11:57:02 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Wed, 22 Sep 2004 09:57:02 -0600 Subject: [SciPy-dev] Scipy for Numarray Message-ID: <1885D1238A4FFE40B113CEB26F87872B066844@cobra.usurf.usu.edu> Hi All, I've been thinking about getting started on a version of scipy for numarray. It's going to be a big job, so I have questions to put to the community. 1. How should it be set up in cvs? There is currently a scipy module to be checked out, should we have a numarray_scipy module at the same level? 2. Where should it install? To avoid name conflicts, I think it might be reasonable to install it, or its components, in the numarray packages directory. 3. There is stuff like Pycrust and ipython that is not directly related to scipy, should it be taken over also, or perhaps broken out into a scipy addons module. 4. There is a lot of infrastructure for the installation and testing of scipy. I am not very familiar with that, so how does it need to be modified to install a numarray version of scipy. 5. How can we keep both versions updated? Chuck From prabhu at aero.iitm.ernet.in Wed Sep 22 13:33:07 2004 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Wed, 22 Sep 2004 23:03:07 +0530 Subject: [SciPy-dev] Weave - strange behaviour In-Reply-To: <41514605.4040705@astro.uni-bonn.de> References: <41514605.4040705@astro.uni-bonn.de> Message-ID: <16721.46931.715578.277732@monster.linux.in> >>>>> "MM" == Manuel Metz writes: MM> Hi, I was really excited about weave. But I found a (to me) MM> somewhat strange behaviour, which I do not understand. I used MM> the following code to test weave: [...] MM> And it seams that EACH time I call the function the C-code MM> gets compiled. This behavior also showed up depending on the MM> first line of the C-code (line break or not; additionen MM> spaces). MM> Has anyone else seen this? Any idea what is going wrong? How MM> can I prevent this strange behaviour (except try and error)? I haven't tested your script but usually wipe out my cache and catalog if I run into any wierdness. Its usually stored in ~/.python23_compiled. Do an rm -rf in that directory and rerun, it should hopefully work. I think the problem is with the catalog. Perhaps the hash fails or is confused for some reason. I'm not sure. Weave is an amazing tool but it looks like it needs more community support. I don't think Eric has enough time on his hands to spend on it. I know that given the chance he'd love to work on it, but there are only 24 hours in a day. I personally don't know my way well enough with the code except for extending it a little here and there and adding support for VTK and SWIG2. I will try and add a few examples for these when I get the chance. However, I am already spread quite thin. Overall there are a few things that definitely need work. 1. Fixing the small bugs, like this one you mention. 2. Documentation. Both are painful. On 2. perhaps a simple wiki page with user examples would help. I'm sure Fernando has several. I can add a few myself. However, the basic docs (like the weave user guide) would need our weave guru, Eric's, attention. I also think that weave really shines when it comes to pure Python code-acceleration. The performance Python article clearly demonstrates its power, ease and simplicity (from the users standpoint). It would be cool if weave could actually be used as the engine to build any extension module from within Python (using f2py, pyrex, or with its own Numeric magic). That way you could embed C/C++/Fortran/Pyrex code into your Python code, depending on what you are comfortable with. I'm sure Eric has other cool ideas too. cheers, prabhu From Fernando.Perez at colorado.edu Wed Sep 22 13:39:53 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed, 22 Sep 2004 11:39:53 -0600 Subject: [SciPy-dev] Weave - strange behaviour In-Reply-To: <16721.46931.715578.277732@monster.linux.in> References: <41514605.4040705@astro.uni-bonn.de> <16721.46931.715578.277732@monster.linux.in> Message-ID: <4151B8E9.2000200@colorado.edu> Prabhu Ramachandran wrote: >>>>>>"MM" == Manuel Metz writes: > > > MM> Hi, I was really excited about weave. But I found a (to me) > MM> somewhat strange behaviour, which I do not understand. I used > MM> the following code to test weave: > > [...] > > MM> And it seams that EACH time I call the function the C-code > MM> gets compiled. This behavior also showed up depending on the > MM> first line of the C-code (line break or not; additionen > MM> spaces). > > MM> Has anyone else seen this? Any idea what is going wrong? How > MM> can I prevent this strange behaviour (except try and error)? > > I haven't tested your script but usually wipe out my cache and catalog > if I run into any wierdness. Its usually stored in > ~/.python23_compiled. Do an rm -rf in that directory and rerun, it > should hopefully work. I second that. I have fixed in the distant past some catalog-related bugs which were appearing with exactly the behaoviour you mention. But there may be more lying around. And that code is quite tricky, it took me a full day to track down a single shelve.close() missing call (or similar). But my fix is already incorporated, so unless you're running an ancient version the problem you are seeing is something else. Unfortunately right now I can't really spend any time on this, and my current codes are all behaving fine with weave. > On 2. perhaps a simple wiki page with user examples would help. I'm > sure Fernando has several. I can add a few myself. However, the > basic docs (like the weave user guide) would need our weave guru, > Eric's, attention. FWIW, here are my examples (attached). Please note that I wrote these a while back, and weave has changed some since then. I post them today only in case they prove useful to somebody, hopefully soon I'll have time to update them to current weave (I think some of them currently don't work). Best, f -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: weave_examples.py URL: From Fernando.Perez at colorado.edu Wed Sep 22 13:41:56 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed, 22 Sep 2004 11:41:56 -0600 Subject: [SciPy-dev] Scipy for Numarray In-Reply-To: <1885D1238A4FFE40B113CEB26F87872B066844@cobra.usurf.usu.edu> References: <1885D1238A4FFE40B113CEB26F87872B066844@cobra.usurf.usu.edu> Message-ID: <4151B964.1060803@colorado.edu> Charles Harris wrote: > 3. There is stuff like Pycrust and ipython that is not directly related > to scipy, should it be taken over also, or perhaps broken out into a > scipy addons module. IPython has no numeric/numarray dependencies, so for now I'd say just leave it where it is. It's already distributed from the main scipy.org site (http://ipython.scipy.org), and I package both sources and fedora RPMs. Others provide Debian and Fink packaging. You are going to have enough to worry about with scipy proper, so I'd suggest you leave the ipython problems to me :) Best, f From pearu at cens.ioc.ee Wed Sep 22 13:42:32 2004 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Wed, 22 Sep 2004 20:42:32 +0300 (EEST) Subject: [SciPy-dev] Scipy for Numarray In-Reply-To: <1885D1238A4FFE40B113CEB26F87872B066844@cobra.usurf.usu.edu> Message-ID: On Wed, 22 Sep 2004, Charles Harris wrote: > Hi All, > > I've been thinking about getting started on a version of scipy for > numarray. It's going to be a big job, so I have questions to put to the > community. > > 1. How should it be set up in cvs? There is currently a scipy module to > be checked out, should we have a numarray_scipy module at the same > level? > > 2. Where should it install? To avoid name conflicts, I think it might be > reasonable to install it, or its components, in the numarray packages > directory. > > 3. There is stuff like Pycrust and ipython that is not directly related > to scipy, should it be taken over also, or perhaps broken out into a > scipy addons module. > > 4. There is a lot of infrastructure for the installation and testing of > scipy. I am not very familiar with that, so how does it need to be > modified to install a numarray version of scipy. > > 5. How can we keep both versions updated? > Let me not give direct answers to your all specific questions but give comments on the issue that hopefully will answer the questions. First, there have been some off-list discussion about numarray integration with scipy among developers of both projects and at the moment the plan seems to be to provide scipy installation that has two versions of extension modules that are built against Numeric and numarray, respectively. When importing scipy then only one version of extension modules will be imported based on the state of some external flag, e.g. environment variable (imho, using env. variable is reasonable under unix system but how this sounds for Windows users? Alternative ideas?). Second, we would like to avoid creating numarray branch of scipy if possible. Otherwise, merging the Numeric and numarray branches in future might turn out to be quite a painful task. On the implementation of the plan: 1) An extension module, say foo, will be built into three files: _foo_na.so - foo built against numarray _foo_np.so - foo built against Numeric foo.py - Python module that imports one of the above modules This task should be doable on the top level of the scipy sources (in setup.py and scipy_distutils) without touching scipy subpackages at all. And f2py already supports numarray. 2) All scipy subpackages should explicitely import and use scipy_base functions instead of Numeric. Currently there are number of scipy subpackages that need to be fixed in this respect. The task should be trivial to do. 3) The most tricky part of getting numarray+scipy to work seems to be building scipy_base against numarray as scipy_base extension modules are very much Numeric specific. As I see it, the order of implementation should be 3), 2), and 1). Though, 2) is an independent step from others but without 3), the step 1) makes little sense. Pearu From charles.harris at sdl.usu.edu Wed Sep 22 16:01:16 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Wed, 22 Sep 2004 14:01:16 -0600 Subject: [SciPy-dev] Scipy for Numarray In-Reply-To: References: Message-ID: <4151DA0C.9010104@sdl.usu.edu> An HTML attachment was scrubbed... URL: From pearu at cens.ioc.ee Wed Sep 22 16:57:28 2004 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Wed, 22 Sep 2004 23:57:28 +0300 (EEST) Subject: [SciPy-dev] Scipy for Numarray In-Reply-To: <4151DA0C.9010104@sdl.usu.edu> Message-ID: On Wed, 22 Sep 2004, Charles Harris wrote: > Pearu Peterson wrote: > > First, there have been some off-list discussion about numarray integration > with scipy among developers of both projects and at the moment the plan > seems to be to provide scipy installation that has two versions of > extension modules that are built against Numeric and numarray, > respectively. When importing scipy then only one version of extension > modules will be imported based on the state of some external flag, > e.g. environment variable (imho, using env. variable is reasonable under > unix system but how this sounds for Windows users? Alternative ideas?). > > Chuck responded: > > I don't much like the idea of an external variable, since at this > point I use both Numeric and numarray, and don't want to mess with my > environment variables to switch between them. Perhaps we could put a > skeleton directory under numarray that would import the proper modules > if they are available, or even install the whole numarray version of > scipy there. This does mean programs have to be modified, but then > numarray itself is not completely backward compatible with Numeric. I > rather like doing the numarray.random_array sort of thing. I must agree that it makes sense to use both numarray and Numeric versions of scipy (or any other package) in the same Python applications whichever is more suitable for some specific task. However, I am not sure that numarray installation directory would be the proper place for scipy. Here is another idea: we could install numarray or Numeric versions of scipy under scipy installation directory so that scipy.na.linalg - scipy.linalg package built against numarray scipy.np.linalg - scipy.linalg package built against Numeric scipy.linalg - scipy.linalg package built against default target, be it Numeric or numarray. for example. Any thoughts? > On the implementation of the plan: > > 1) An extension module, say foo, will be built into three files: > _foo_na.so - foo built against numarray > _foo_np.so - foo built against Numeric > foo.py - Python module that imports one of the above modules > This task should be doable on the top level of the scipy sources > (in setup.py and scipy_distutils) without touching scipy subpackages > at all. And f2py already supports numarray. > > Chuck responded: > > We will have to change packages to conform to this notation. Not necessarily. This can be accomplished by hacking scipy_distutils alone. Now I think of introducing --targets flag to build_ext command so that scipy could be build using either python setup.py build_ext --targets=Numeric,numarray build python setup.py build_ext --targets=Numeric build python setup.py build_ext --targets=numarray build Btw, note that when the target is numarray then only -DNUMARRAY will be specified to compile an extension module. At least, this convention is used in f2py numarray support, the same convention could be used also in handwritten or swig generated extension modules. Pearu From jmiller at stsci.edu Wed Sep 22 17:16:12 2004 From: jmiller at stsci.edu (Todd Miller) Date: 22 Sep 2004 17:16:12 -0400 Subject: [SciPy-dev] Scipy for Numarray In-Reply-To: References: Message-ID: <1095887771.23859.310.camel@halloween.stsci.edu> On Wed, 2004-09-22 at 13:42, Pearu Peterson wrote: > On Wed, 22 Sep 2004, Charles Harris wrote: > > > Hi All, > > > > I've been thinking about getting started on a version of scipy for > > numarray. It's going to be a big job, so I have questions to put to the > > community. > > > > 1. How should it be set up in cvs? There is currently a scipy module to > > be checked out, should we have a numarray_scipy module at the same > > level? > > > > 2. Where should it install? To avoid name conflicts, I think it might be > > reasonable to install it, or its components, in the numarray packages > > directory. > > > > 3. There is stuff like Pycrust and ipython that is not directly related > > to scipy, should it be taken over also, or perhaps broken out into a > > scipy addons module. > > > > 4. There is a lot of infrastructure for the installation and testing of > > scipy. I am not very familiar with that, so how does it need to be > > modified to install a numarray version of scipy. > > > > 5. How can we keep both versions updated? > > > > Let me not give direct answers to your all specific questions but give > comments on the issue that hopefully will answer the questions. > > First, there have been some off-list discussion about numarray integration > with scipy among developers of both projects and at the moment the plan > seems to be to provide scipy installation that has two versions of > extension modules that are built against Numeric and numarray, > respectively. When importing scipy then only one version of extension > modules will be imported based on the state of some external flag, > e.g. environment variable (imho, using env. variable is reasonable under > unix system but how this sounds for Windows users? Alternative ideas?). For matplotlib we primarily use a setting in the .rc file but also support a command line switch of --Numeric or --numarray. Matplotlib also fails over to whichever array package is actually present. This selection is done through a module called "matplotlib.numerix" which provides the same basic facilities for either numarray or Numeric. > Second, we would like to avoid creating numarray branch of scipy if > possible. Otherwise, merging the Numeric and numarray branches in future > might turn out to be quite a painful task. > > On the implementation of the plan: > > 1) An extension module, say foo, will be built into three files: > _foo_na.so - foo built against numarray > _foo_np.so - foo built against Numeric > foo.py - Python module that imports one of the above modules > This task should be doable on the top level of the scipy sources > (in setup.py and scipy_distutils) without touching scipy subpackages > at all. And f2py already supports numarray. Top level support is a very cool idea. I just implemented this approach "manually" for matplotlib (only two extensions were needed) and so far it seems to work fine. > 2) All scipy subpackages should explicitely import and use > scipy_base functions instead of Numeric. Currently there are number of > scipy subpackages that need to be fixed in this respect. The task should > be trivial to do. > > 3) The most tricky part of getting numarray+scipy to work seems to > be building scipy_base against numarray as scipy_base extension modules > are very much Numeric specific. Looking at this today, I see two problems for numarray (there may be others) in scipy_base: 1. fastumathmodule.c uses the Numeric ufunc C-API which numarray doesn't support (yet anyway). IMHO, this breaks down into: a. Ordinary numarray ufuncs are probably good enough because anyone that really wants fastumath is not going to use numarray for other performance reasons anyway. b. There are some new features in fastumath that need to be supported. These latter features could be added directly to numarray. 2. Other scipy_base modules import Numeric and its addon modules directly, similar to your (2). Instead, they should import from a TBD numerix. Todd From charles.harris at sdl.usu.edu Wed Sep 22 17:27:56 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Wed, 22 Sep 2004 15:27:56 -0600 Subject: [SciPy-dev] Scipy for Numarray In-Reply-To: <1095887771.23859.310.camel@halloween.stsci.edu> References: <1095887771.23859.310.camel@halloween.stsci.edu> Message-ID: <4151EE5C.1000402@sdl.usu.edu> Todd Miller wrote: > >Looking at this today, I see two problems for numarray (there may be >others) in scipy_base: > >1. fastumathmodule.c uses the Numeric ufunc C-API which numarray doesn't >support (yet anyway). IMHO, this breaks down into: > > a. Ordinary numarray ufuncs are probably good enough because > anyone that really wants fastumath is not going to use numarray > for other performance reasons anyway. > > Hey, aren't we going to fix these performance issues? Just askin'. > > b. There are some new features in fastumath that need to be > supported. These latter features could be added directly to > numarray. > >2. Other scipy_base modules import Numeric and its addon modules >directly, similar to your (2). Instead, they should import from a TBD >numerix. > >Todd > >_______________________________________________ >Scipy-dev mailing list >Scipy-dev at scipy.net >http://www.scipy.net/mailman/listinfo/scipy-dev > > > From jmiller at stsci.edu Wed Sep 22 17:33:15 2004 From: jmiller at stsci.edu (Todd Miller) Date: 22 Sep 2004 17:33:15 -0400 Subject: [SciPy-dev] Scipy for Numarray In-Reply-To: <4151DA0C.9010104@sdl.usu.edu> References: <4151DA0C.9010104@sdl.usu.edu> Message-ID: <1095888794.23859.330.camel@halloween.stsci.edu> > > 3) The most tricky part of getting numarray+scipy to work seems to > > be building scipy_base against numarray as scipy_base extension modules > > are very much Numeric specific. > > > > > Yeah, I've started browsing these. I'm going to try to make the > special functions numarray compatible > in the near future (I want them), and that is why I am raising the > issue now. I started looking at special/cephes yesterday since it uses the Numeric ufunc C-API. I used the numarray code generation capabilities to create ufuncs for some/most of the cephes (cephes library proper) unary and binary functions. I've also started looking at adding n-ary ufuncs to numarray, but that's still a ways off. So, I have some preliminary cephes functions you can look over and try out if you want, and in any case, I'm looking at the special functions too. Todd From charles.harris at sdl.usu.edu Wed Sep 22 17:58:35 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Wed, 22 Sep 2004 15:58:35 -0600 Subject: [SciPy-dev] Scipy for Numarray In-Reply-To: <1095888794.23859.330.camel@halloween.stsci.edu> References: <4151DA0C.9010104@sdl.usu.edu> <1095888794.23859.330.camel@halloween.stsci.edu> Message-ID: <4151F58B.6020803@sdl.usu.edu> Todd Miller wrote: >>>3) The most tricky part of getting numarray+scipy to work seems to >>>be building scipy_base against numarray as scipy_base extension modules >>>are very much Numeric specific. >>> >>> >>> >>> >>Yeah, I've started browsing these. I'm going to try to make the >>special functions numarray compatible >>in the near future (I want them), and that is why I am raising the >>issue now. >> >> > >I started looking at special/cephes yesterday since it uses the Numeric >ufunc C-API. I used the numarray code generation capabilities to create >ufuncs for some/most of the cephes (cephes library proper) unary and >binary functions. I've also started looking at adding n-ary ufuncs to >numarray, but that's still a ways off. So, I have some preliminary >cephes functions you can look over and try out if you want, and in any >case, I'm looking at the special functions too. > >Todd > > Yes, I would be interested. I would also like some of the cdflib functions. There is some overlap between these two libraries, the gamma function for instance. From jmiller at stsci.edu Wed Sep 22 18:04:47 2004 From: jmiller at stsci.edu (Todd Miller) Date: 22 Sep 2004 18:04:47 -0400 Subject: [SciPy-dev] Scipy for Numarray In-Reply-To: <4151EE5C.1000402@sdl.usu.edu> References: <1095887771.23859.310.camel@halloween.stsci.edu> <4151EE5C.1000402@sdl.usu.edu> Message-ID: <1095890686.23859.365.camel@halloween.stsci.edu> On Wed, 2004-09-22 at 17:27, Charles Harris wrote: > Todd Miller wrote: > > > > > > >Looking at this today, I see two problems for numarray (there may be > >others) in scipy_base: > > > >1. fastumathmodule.c uses the Numeric ufunc C-API which numarray doesn't > >support (yet anyway). IMHO, this breaks down into: > > > > a. Ordinary numarray ufuncs are probably good enough because > > anyone that really wants fastumath is not going to use numarray > > for other performance reasons anyway. > > > > > Hey, aren't we going to fix these performance issues? Just askin'. My thinking is that we can do something like fastumath in numarray as part of the process of addresssing performance issues. Todd From chris at fisher.forestry.uga.edu Fri Sep 24 19:56:17 2004 From: chris at fisher.forestry.uga.edu (chris at fisher.forestry.uga.edu) Date: Fri, 24 Sep 2004 19:56:17 -0400 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? Message-ID: For the past couple of years, I have been maintaining a package called PyMC (http://pymc.sourceforge.net) which provides a general Markov chain Monte Carlo sampler (Metropolis-Hastings), linear and non-linear Kalman filtering algorithms and several reinforcement learning algorithms for AI applications. To date, it has been used mostly for ecological modeling applications, but is far more general than that. This package currently relies on SciPy, particularly for such tasks as plotting, and uses f2py extensively. I am wondering if there is interest in integrating such functionality into SciPy. If so, I would be happy to contribute, integrate and maintain the code under the auspices of the SciPy project. If not, I am equally content to continue maintaining PyMC separately. I encourage those interested to check out PyMC and let me know if it would be worthwhile to add any of this to SciPy. Cheers, Chris Fonnesbeck chris (at) fisher.forestry.uga.edu From charles.harris at sdl.usu.edu Fri Sep 24 16:09:22 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Fri, 24 Sep 2004 14:09:22 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: References: Message-ID: <41547EF2.8080104@sdl.usu.edu> chris at fisher.forestry.uga.edu wrote: >For the past couple of years, I have been maintaining a package called >PyMC (http://pymc.sourceforge.net) which provides a general Markov chain >Monte Carlo sampler (Metropolis-Hastings), linear and non-linear Kalman >filtering algorithms and several reinforcement learning algorithms for >AI applications. To date, it has been used mostly for ecological >modeling applications, but is far more general than that. This package >currently relies on SciPy, particularly for such tasks as plotting, and >uses f2py extensively. > >I am wondering if there is interest in integrating such functionality >into SciPy. If so, I would be happy to contribute, integrate and >maintain the code under the auspices of the SciPy project. If not, I am >equally content to continue maintaining PyMC separately. I encourage >those interested to check out PyMC and let me know if it would be >worthwhile to add any of this to SciPy. > >Cheers, >Chris Fonnesbeck > >chris (at) fisher.forestry.uga.edu > >_______________________________________________ >Scipy-dev mailing list >Scipy-dev at scipy.net >http://www.scipy.net/mailman/listinfo/scipy-dev > > > Chris, I'm definitely interested. The problem I always saw with the Montecarlo type samplers was that the core routine that I wanted to write in Python was always called in the innermost loop, so the sampling was slow. Are your routines pure python, or are they C code, and how do you get good performance? Chuck From chris at fisher.forestry.uga.edu Fri Sep 24 20:33:17 2004 From: chris at fisher.forestry.uga.edu (chris at fisher.forestry.uga.edu) Date: Fri, 24 Sep 2004 20:33:17 -0400 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <41547EF2.8080104@sdl.usu.edu> Message-ID: On 9/24/2004, "Charles Harris" wrote: >chris at fisher.forestry.uga.edu wrote: > >>For the past couple of years, I have been maintaining a package called >>PyMC (http://pymc.sourceforge.net) which provides a general Markov chain >>Monte Carlo sampler (Metropolis-Hastings), linear and non-linear Kalman >>filtering algorithms and several reinforcement learning algorithms for >>AI applications. To date, it has been used mostly for ecological >>modeling applications, but is far more general than that. This package >>currently relies on SciPy, particularly for such tasks as plotting, and >>uses f2py extensively. >> >>I am wondering if there is interest in integrating such functionality >>into SciPy. If so, I would be happy to contribute, integrate and >>maintain the code under the auspices of the SciPy project. If not, I am >>equally content to continue maintaining PyMC separately. I encourage >>those interested to check out PyMC and let me know if it would be >>worthwhile to add any of this to SciPy. >> >>Cheers, >>Chris Fonnesbeck >> >>chris (at) fisher.forestry.uga.edu >> >>_______________________________________________ >>Scipy-dev mailing list >>Scipy-dev at scipy.net >>http://www.scipy.net/mailman/listinfo/scipy-dev >> >> >> >Chris, > >I'm definitely interested. The problem I always saw with the Montecarlo >type samplers was that the core routine that I wanted to >write in Python was always called in the innermost loop, so the sampling >was slow. Are your routines pure python, or are they >C code, and how do you get good performance? > >Chuck > The performance is good, relative to extant packages such as the BUGS project. For the MCMC sampler, the bottleneck is in the repeated evaluation of statistical likelihoods; scipy's are too slow, so I use some f2py modules there; for the reinforcement learning stuff, the innermost loop is coded inline with weave. Chris From rkern at ucsd.edu Fri Sep 24 16:42:36 2004 From: rkern at ucsd.edu (Robert Kern) Date: Fri, 24 Sep 2004 13:42:36 -0700 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: References: Message-ID: <415486BC.3060305@ucsd.edu> chris at fisher.forestry.uga.edu wrote: > For the past couple of years, I have been maintaining a package called > PyMC (http://pymc.sourceforge.net) which provides a general Markov chain > Monte Carlo sampler (Metropolis-Hastings), linear and non-linear Kalman > filtering algorithms and several reinforcement learning algorithms for > AI applications. To date, it has been used mostly for ecological > modeling applications, but is far more general than that. This package > currently relies on SciPy, particularly for such tasks as plotting, and > uses f2py extensively. > > I am wondering if there is interest in integrating such functionality > into SciPy. If so, I would be happy to contribute, integrate and > maintain the code under the auspices of the SciPy project. If not, I am > equally content to continue maintaining PyMC separately. I encourage > those interested to check out PyMC and let me know if it would be > worthwhile to add any of this to SciPy. Fantastic! I am definitely interested in seeing MCMC and related algorithms integrated into SciPy. I didn't look too hard at at your code when you first announced it because I wanted something to go into SciPy even if I had to write it myself, and we're not incorporating LGPL code into SciPy. But if you're willing to relicense it under the BSD license, I don't have to redo all the hard work you've done. And that makes me happy. It would be worthwhile to go make sure the MCMC design is general enough to accommodate other sampling strategies (specifically, I'd like to implement some of the ideas of Radford M. Neal[1]). Gibbs sampling from models specified in a manner similar to BUGS would be cool, too. [1] http://www.cs.toronto.edu/~radford/fbm.software.html -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From chris at fisher.forestry.uga.edu Fri Sep 24 20:47:48 2004 From: chris at fisher.forestry.uga.edu (chris at fisher.forestry.uga.edu) Date: Fri, 24 Sep 2004 20:47:48 -0400 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415486BC.3060305@ucsd.edu> Message-ID: On 9/24/2004, "Robert Kern" wrote: > >It would be worthwhile to go make sure the MCMC design is general enough >to accommodate other sampling strategies (specifically, I'd like to >implement some of the ideas of Radford M. Neal[1]). Gibbs sampling from >models specified in a manner similar to BUGS would be cool, too. > The MCMC algorithm I implemented is about as general as you can get: a random-walk metropolis-hastings sampler. It avoids the conjugacy requirement of Gibss sampling, and allows the user to select from a number of proposal distributions. I invite you to look at the code; it is pretty well documented. Chris From oliphant at ee.byu.edu Sun Sep 26 00:22:34 2004 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sat, 25 Sep 2004 22:22:34 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: References: Message-ID: <4156440A.1040204@ee.byu.edu> chris at fisher.forestry.uga.edu wrote: >For the past couple of years, I have been maintaining a package called >PyMC (http://pymc.sourceforge.net) which provides a general Markov chain >Monte Carlo sampler (Metropolis-Hastings), linear and non-linear Kalman >filtering algorithms and several reinforcement learning algorithms for >AI applications. To date, it has been used mostly for ecological >modeling applications, but is far more general than that. This package >currently relies on SciPy, particularly for such tasks as plotting, and >uses f2py extensively. > >I am wondering if there is interest in integrating such functionality >into SciPy. If so, I would be happy to contribute, integrate and >maintain the code under the auspices of the SciPy project. If not, I am >equally content to continue maintaining PyMC separately. I encourage >those interested to check out PyMC and let me know if it would be >worthwhile to add any of this to SciPy. > > > Such additions would be fantastic. I share Robert's enthusiasm. I would love to see these in SciPy. -Travis From eric at enthought.com Sun Sep 26 03:16:13 2004 From: eric at enthought.com (eric jones) Date: Sun, 26 Sep 2004 02:16:13 -0500 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <4156440A.1040204@ee.byu.edu> References: <4156440A.1040204@ee.byu.edu> Message-ID: <41566CBD.70005@enthought.com> Where should these live? monte carlo and markov chain might fit in scipy.stats? kalman in scipy.signal? AI in... scipy.ai? Or should they all come in as new packages? Also, Chris, how are the tests and docs for these packages? eric Travis Oliphant wrote: > chris at fisher.forestry.uga.edu wrote: > >> For the past couple of years, I have been maintaining a package called >> PyMC (http://pymc.sourceforge.net) which provides a general Markov chain >> Monte Carlo sampler (Metropolis-Hastings), linear and non-linear Kalman >> filtering algorithms and several reinforcement learning algorithms for >> AI applications. To date, it has been used mostly for ecological >> modeling applications, but is far more general than that. This package >> currently relies on SciPy, particularly for such tasks as plotting, and >> uses f2py extensively. >> >> I am wondering if there is interest in integrating such functionality >> into SciPy. If so, I would be happy to contribute, integrate and >> maintain the code under the auspices of the SciPy project. If not, I am >> equally content to continue maintaining PyMC separately. I encourage >> those interested to check out PyMC and let me know if it would be >> worthwhile to add any of this to SciPy. >> >> >> > > Such additions would be fantastic. I share Robert's enthusiasm. > I would love to see these in SciPy. > > -Travis > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev From chris at fisher.forestry.uga.edu Sun Sep 26 15:10:15 2004 From: chris at fisher.forestry.uga.edu (chris at fisher.forestry.uga.edu) Date: Sun, 26 Sep 2004 15:10:15 -0400 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <41566CBD.70005@enthought.com> Message-ID: On 9/26/2004, "eric jones" wrote: >Where should these live? > >monte carlo and markov chain might fit in scipy.stats? >kalman in scipy.signal? >AI in... scipy.ai? That sounds good to me; the AI module in PyMC consists of reinforcement learning algorithms, but having a scipy.ai module would leave room for things like neural networks in the future. I would also be interested in helping to overhaul the stats.py module, so that this MCMC code could make better use of it. > >Or should they all come in as new packages? You can make an argument for either, but it may make sense to integrate where possible. > >Also, Chris, how are the tests and docs for these packages? > Each has a unit test using the unittest package. Try running: PyMC.MCMC.test() PyMC.KalmanFiltering.test() PyMC.ReinforcementLearning.test() The code is also amply documented. C. From charles.harris at sdl.usu.edu Mon Sep 27 14:02:21 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Mon, 27 Sep 2004 12:02:21 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <41566CBD.70005@enthought.com> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> Message-ID: <415855AD.4060701@sdl.usu.edu> eric jones wrote: > Where should these live? > monte carlo and markov chain might fit in scipy.stats? How about in monte_carlo or some such? I think there is too much stuff put in odd places. Why is zeros in optimize? Makes no sense, but there it is. I don't think now is the time to change all the directories around, but I hope we give some thought to the organization before it becomes unmanageable. The Dewey decimal classification was an achievement I am coming to appreciate. > kalman in scipy.signal? OK. > AI in... scipy.ai? Or should they all come in as new packages? > > Also, Chris, how are the tests and docs for these packages? > > eric > > Travis Oliphant wrote: > >> chris at fisher.forestry.uga.edu wrote: >> >>> For the past couple of years, I have been maintaining a package called >>> PyMC (http://pymc.sourceforge.net) which provides a general Markov >>> chain >>> Monte Carlo sampler (Metropolis-Hastings), linear and non-linear Kalman >>> filtering algorithms and several reinforcement learning algorithms for >>> AI applications. To date, it has been used mostly for ecological >>> modeling applications, but is far more general than that. This package >>> currently relies on SciPy, particularly for such tasks as plotting, and >>> uses f2py extensively. >>> >>> I am wondering if there is interest in integrating such functionality >>> into SciPy. If so, I would be happy to contribute, integrate and >>> maintain the code under the auspices of the SciPy project. If not, I am >>> equally content to continue maintaining PyMC separately. I encourage >>> those interested to check out PyMC and let me know if it would be >>> worthwhile to add any of this to SciPy. >>> >>> >>> >> >> Such additions would be fantastic. I share Robert's enthusiasm. >> I would love to see these in SciPy. >> >> -Travis >> >> _______________________________________________ >> Scipy-dev mailing list >> Scipy-dev at scipy.net >> http://www.scipy.net/mailman/listinfo/scipy-dev > > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From charles.harris at sdl.usu.edu Mon Sep 27 15:10:26 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Mon, 27 Sep 2004 13:10:26 -0600 Subject: [SciPy-dev] erf Message-ID: <415865A2.1000304@sdl.usu.edu> Which erf is scipy using? There is one in cephes/ndtr.c and another in cdflib/erf.f . The two don't give the same results for smallish x, and I suspect that erf.f is the more accurate because ndtr computes it by 1.0 - erfc(x) . From oliphant at ee.byu.edu Mon Sep 27 18:06:48 2004 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 27 Sep 2004 16:06:48 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415855AD.4060701@sdl.usu.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> Message-ID: <41588EF8.80407@ee.byu.edu> Charles Harris wrote: > eric jones wrote: > >> Where should these live? >> monte carlo and markov chain might fit in scipy.stats? > > > How about in monte_carlo or some such? I think there is too much stuff > put in odd places. Why is zeros in optimize? Makes no sense, but there > it is. The reason it is in optimize, is because the fsolve command for many-variable functions is a wrapper of something in minpack which formed the foundation for optimize. As you know there is a connection between optimization and finding zeros of functions. Later, one-variable root-finders were added to be near fsolve. We are not against reorganizations, but odd to you is not necessarily odd to someone else, and vice versa. So, let's just figure out were monte_carlo should go. I think it would go well under stats, or else a new AI subpackage. I agree that stats could use reorganization. Many routines were lifted from an old pstats.py file. While it has been significantly cleaned up, there are still problems. I hear various complaints occasionally about slowness in some of the distributions in stats. In order to improve things, these need to be better described. Their shouldn't be a lot of slow-down in most of the routines (aside from domain validity slowness). Some distributions don't have exactly defined cdf's or ppf's and these must be computed by SciPy using integration and zero-finding routines. This will be very slow. There are also several statistical-related routines available in special in all of their raw and speedy glory. -Travis O. From charles.harris at sdl.usu.edu Mon Sep 27 18:39:53 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Mon, 27 Sep 2004 16:39:53 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <41588EF8.80407@ee.byu.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41588EF8.80407@ee.byu.edu> Message-ID: <415896B9.4000402@sdl.usu.edu> Travis Oliphant wrote: > Charles Harris wrote: > >> eric jones wrote: >> >>> Where should these live? >>> monte carlo and markov chain might fit in scipy.stats? >> >> >> >> How about in monte_carlo or some such? I think there is too much stuff >> put in odd places. Why is zeros in optimize? Makes no sense, but there > > >> it is. > > > > The reason it is in optimize, is because the fsolve command for > many-variable functions is a wrapper of something in minpack which > formed the foundation for optimize. > > As you know there is a connection between optimization and finding > zeros of functions. Later, one-variable root-finders were added to > be near fsolve. > Yep, I know how and why it happened. I still think this would not be the first place a naive user would look for zero finders, but I can live with it. MCMC could also go in optimize, as it is often used to make estimates of most probable solutions, likewise for genetic algorithms. Some of these sorts of things have multiple uses. I am argueing that using more top level directories may be a simpler way of organizing things than trying to fit everything into a few broad categories. > We are not against reorganizations, but odd to you is not necessarily > odd to someone else, and vice versa. So, let's just figure out were > monte_carlo should go. I think it would go well under stats, or else > a new AI subpackage. > > > I agree that stats could use reorganization. Many routines were > lifted from an old pstats.py file. While it has been significantly > cleaned up, there are still problems. > I hear various complaints occasionally about slowness in some of the > distributions in stats. In order to improve things, these need to be > better described. Their shouldn't be a lot of slow-down in most of > the routines (aside from domain validity slowness). Some > distributions don't have exactly defined cdf's or ppf's and these must > be computed by SciPy using integration and zero-finding routines. > This will be very slow. > > There are also several statistical-related routines available in > special in all of their raw and speedy glory. > > -Travis O. > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From chris at fisher.forestry.uga.edu Mon Sep 27 22:53:53 2004 From: chris at fisher.forestry.uga.edu (chris at fisher.forestry.uga.edu) Date: Mon, 27 Sep 2004 22:53:53 -0400 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415855AD.4060701@sdl.usu.edu> Message-ID: In my view, stats makes sense, since Markov chain Monte Carlo is used mostly for statistical estimation. Perhaps a module called "bayes" would be appropriate, but it certainly would not be out of place in stats. Chris On 9/27/2004, "Charles Harris" wrote: >eric jones wrote: > >> Where should these live? >> monte carlo and markov chain might fit in scipy.stats? > >How about in monte_carlo or some such? I think there is too much stuff >put in odd places. Why is zeros in optimize? Makes no sense, but there >it is. I don't think now is the time to change all the directories >around, but >I hope we give some thought to the organization before it becomes >unmanageable. >The Dewey decimal classification was an achievement I am coming to >appreciate. > >> kalman in scipy.signal? > >OK. > >> AI in... scipy.ai? Or should they all come in as new packages? >> >> Also, Chris, how are the tests and docs for these packages? >> >> eric >> >> Travis Oliphant wrote: >> >>> chris at fisher.forestry.uga.edu wrote: >>> >>>> For the past couple of years, I have been maintaining a package called >>>> PyMC (http://pymc.sourceforge.net) which provides a general Markov >>>> chain >>>> Monte Carlo sampler (Metropolis-Hastings), linear and non-linear Kalman >>>> filtering algorithms and several reinforcement learning algorithms for >>>> AI applications. To date, it has been used mostly for ecological >>>> modeling applications, but is far more general than that. This package >>>> currently relies on SciPy, particularly for such tasks as plotting, and >>>> uses f2py extensively. >>>> >>>> I am wondering if there is interest in integrating such functionality >>>> into SciPy. If so, I would be happy to contribute, integrate and >>>> maintain the code under the auspices of the SciPy project. If not, I am >>>> equally content to continue maintaining PyMC separately. I encourage >>>> those interested to check out PyMC and let me know if it would be >>>> worthwhile to add any of this to SciPy. >>>> >>>> >>>> >>> >>> Such additions would be fantastic. I share Robert's enthusiasm. >>> I would love to see these in SciPy. >>> >>> -Travis >>> >>> _______________________________________________ >>> Scipy-dev mailing list >>> Scipy-dev at scipy.net >>> http://www.scipy.net/mailman/listinfo/scipy-dev >> >> >> >> _______________________________________________ >> Scipy-dev mailing list >> Scipy-dev at scipy.net >> http://www.scipy.net/mailman/listinfo/scipy-dev >> > >_______________________________________________ >Scipy-dev mailing list >Scipy-dev at scipy.net >http://www.scipy.net/mailman/listinfo/scipy-dev From chris at fisher.forestry.uga.edu Mon Sep 27 23:12:13 2004 From: chris at fisher.forestry.uga.edu (chris at fisher.forestry.uga.edu) Date: Mon, 27 Sep 2004 23:12:13 -0400 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <41588EF8.80407@ee.byu.edu> Message-ID: On 9/27/2004, "Travis Oliphant" wrote: >We are not against reorganizations, but odd to you is not necessarily >odd to someone else, and vice versa. So, let's just figure out were >monte_carlo should go. I think it would go well under stats, or else a >new AI subpackage. > MCMC isnt really artificial intelligence, it is bayesian stats, so stats would be best, I think. > >I agree that stats could use reorganization. Many routines were lifted >from an old pstats.py file. While it has been significantly cleaned up, >there are still problems. > I would be glad to take a run at re-doing stats, if only because I would be using it a *lot*. If so, I wouldnt mind getting a bit of background from Travis (and others) about the module, particularly with respectto what would need to be retained, and what the most desirable changes would be. >I hear various complaints occasionally about slowness in some of the >distributions in stats. In order to improve things, these need to be >better described. Their shouldn't be a lot of slow-down in most of the >routines (aside from domain validity slowness). Some distributions >don't have exactly defined cdf's or ppf's and these must be computed by >SciPy using integration and zero-finding routines. This will be very slow. The distributions that most folks would be using 95% of the time (normal,mv-normal,gamma,beta,binomial,exponential,poisson) are well defined and can be made pretty fast with FORTRAN extensions. Chris From chris at fisher.forestry.uga.edu Mon Sep 27 23:14:16 2004 From: chris at fisher.forestry.uga.edu (chris at fisher.forestry.uga.edu) Date: Mon, 27 Sep 2004 23:14:16 -0400 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <41588EF8.80407@ee.byu.edu> Message-ID: On 9/27/2004, "Travis Oliphant" wrote: >We are not against reorganizations, but odd to you is not necessarily >odd to someone else, and vice versa. So, let's just figure out were >monte_carlo should go. I think it would go well under stats, or else a >new AI subpackage. > MCMC isnt really artificial intelligence, it is bayesian stats, so stats would be best, I think. > >I agree that stats could use reorganization. Many routines were lifted >from an old pstats.py file. While it has been significantly cleaned up, >there are still problems. > I would be glad to take a run at re-doing stats, if only because I would be using it a *lot*. If so, I wouldnt mind getting a bit of background from Travis (and others) about the module, particularly with respectto what would need to be retained, and what the most desirable changes would be. >I hear various complaints occasionally about slowness in some of the >distributions in stats. In order to improve things, these need to be >better described. Their shouldn't be a lot of slow-down in most of the >routines (aside from domain validity slowness). Some distributions >don't have exactly defined cdf's or ppf's and these must be computed by >SciPy using integration and zero-finding routines. This will be very slow. The distributions that most folks would be using 95% of the time (normal,mv-normal,gamma,beta,binomial,exponential,poisson) are well defined and can be made pretty fast with FORTRAN extensions. Chris From pearu at scipy.org Mon Sep 27 19:43:01 2004 From: pearu at scipy.org (Pearu Peterson) Date: Mon, 27 Sep 2004 18:43:01 -0500 (CDT) Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415855AD.4060701@sdl.usu.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> Message-ID: On Mon, 27 Sep 2004, Charles Harris wrote: > eric jones wrote: > >> Where should these live? >> monte carlo and markov chain might fit in scipy.stats? > > How about in monte_carlo or some such? I think there is too much stuff > put in odd places. Why is zeros in optimize? Makes no sense, but there > it is. I don't think now is the time to change all the directories > around, but I hope we give some thought to the organization before it > becomes unmanageable. The Dewey decimal classification was an > achievement I am coming to appreciate. I agree that tools provided by Scipy are not organized well with respect to the mathematical subject that they deal with and so finding the needed tool for some specific task may not be always easy. And I agree that this issue should be tackled as early stage as possible in Scipy evolution, otherwise it will get more and more difficult to decide where to put contributions from the society and there is a danger of postponing such decisions to an unreachable future.. The current organization of Scipy packages is mostly based on underlying Fortran/C libraries and that is obviously not the best way to organize any high-level scientific tool. While Chuck mentioned Dewey decimal classification then there are other classification schemes available. For example, MSC (http://www.ams.org/msc/). A nice overview of MSC can be found in http://www.math.niu.edu/~rusin/known-math/index/index.html I don't know how well could these classification schemes be used for organizing Scipy packages, may be we should take a look how Matlab or Maple or Mathematica deal with organizing their tools. >From the maintenance point of view, IMHO, wrappers to external Fortran/C libraries should be refactored from scipy packages to some "lib" package. For example, there would be packages like scipy.lib.blas scipy.lib.lapack scipy.lib.fftpack scipy.lib.minpack scipy.lib.cephes scipy.lib.odepack scipy.lib.quadpack scipy.lib.fitpack scipy.lib.minpack scipy.lib.superlu scipy.lib.amos scipy.lib.cdflib etc and they will be used by higher level packages like scipy.linalg scipy.linalg.sparse scipy.stats scipy.optimize scipy.special etc. These higher level packages can be organized by whatever scheme will be chosen, the scheme may be changed or even replaced in future, but the core of scipy, scipy.lib, that contains most of the hard work in scipy, should stay as constant as possible. Please, send suggestions on how to better organize scipy high-level packages. Thanks, Pearu From charles.harris at sdl.usu.edu Mon Sep 27 19:29:03 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Mon, 27 Sep 2004 17:29:03 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> Message-ID: <4158A23F.6080207@sdl.usu.edu> Pearu Peterson wrote: > > > On Mon, 27 Sep 2004, Charles Harris wrote: > >> eric jones wrote: >> >>> Where should these live? >>> monte carlo and markov chain might fit in scipy.stats? >> >> >> How about in monte_carlo or some such? I think there is too much >> stuff put in odd places. Why is zeros in optimize? Makes no sense, >> but there it is. I don't think now is the time to change all the >> directories around, but I hope we give some thought to the >> organization before it becomes unmanageable. The Dewey decimal >> classification was an achievement I am coming to appreciate. > > > I agree that tools provided by Scipy are not organized well with > respect to the mathematical subject that they deal with and so finding > the needed tool for some specific task may not be always easy. And I > agree that this issue should be tackled as early stage as possible in > Scipy evolution, otherwise it will get more and more difficult to > decide where to put contributions from the society and there is a > danger of postponing such decisions to an unreachable future.. > > The current organization of Scipy packages is mostly based on underlying > Fortran/C libraries and that is obviously not the best way to organize > any high-level scientific tool. > > While Chuck mentioned Dewey decimal classification then there are other > classification schemes available. For example, MSC > (http://www.ams.org/msc/). A nice overview of MSC can be found in > http://www.math.niu.edu/~rusin/known-math/index/index.html > > I don't know how well could these classification schemes be used > for organizing Scipy packages, may be we should take a look how Matlab > or Maple or Mathematica deal with organizing their tools. > >> From the maintenance point of view, IMHO, wrappers to external Fortran/C > > libraries should be refactored from scipy packages to some "lib" > package. For example, there would be packages like > > scipy.lib.blas > scipy.lib.lapack > scipy.lib.fftpack > scipy.lib.minpack > scipy.lib.cephes > scipy.lib.odepack > scipy.lib.quadpack > scipy.lib.fitpack > scipy.lib.minpack > scipy.lib.superlu > scipy.lib.amos > scipy.lib.cdflib > etc > Hey, I like that idea a *lot*. > and they will be used by higher level packages like > > scipy.linalg > scipy.linalg.sparse > scipy.stats > scipy.optimize > scipy.special > etc. > > These higher level packages can be organized by whatever scheme will > be chosen, the scheme may be changed or even replaced in future, but the > core of scipy, scipy.lib, that contains most of the hard work in > scipy, should stay as constant as possible. > > Please, send suggestions on how to better organize scipy high-level > packages. > > Thanks, > Pearu > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From eric at enthought.com Tue Sep 28 02:30:23 2004 From: eric at enthought.com (eric jones) Date: Tue, 28 Sep 2004 01:30:23 -0500 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <4158A23F.6080207@sdl.usu.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <4158A23F.6080207@sdl.usu.edu> Message-ID: <415904FF.9070602@enthought.com> Charles Harris wrote: > Pearu Peterson wrote: > >> >> >> On Mon, 27 Sep 2004, Charles Harris wrote: >> >>> eric jones wrote: >>> >>>> Where should these live? >>>> monte carlo and markov chain might fit in scipy.stats? >>> >>> >>> >>> How about in monte_carlo or some such? I think there is too much >>> stuff put in odd places. Why is zeros in optimize? Makes no sense, >>> but there it is. I don't think now is the time to change all the >>> directories around, but I hope we give some thought to the >>> organization before it becomes unmanageable. The Dewey decimal >>> classification was an achievement I am coming to appreciate. >> >> >> >> I agree that tools provided by Scipy are not organized well with >> respect to the mathematical subject that they deal with and so >> finding the needed tool for some specific task may not be always >> easy. And I agree that this issue should be tackled as early stage as >> possible in Scipy evolution, otherwise it will get more and more >> difficult to decide where to put contributions from the society and >> there is a danger of postponing such decisions to an unreachable >> future.. >> >> The current organization of Scipy packages is mostly based on underlying >> Fortran/C libraries and that is obviously not the best way to organize >> any high-level scientific tool. >> >> While Chuck mentioned Dewey decimal classification then there are other >> classification schemes available. For example, MSC >> (http://www.ams.org/msc/). A nice overview of MSC can be found in >> http://www.math.niu.edu/~rusin/known-math/index/index.html >> >> I don't know how well could these classification schemes be used >> for organizing Scipy packages, may be we should take a look how >> Matlab or Maple or Mathematica deal with organizing their tools. >> >>> From the maintenance point of view, IMHO, wrappers to external >>> Fortran/C >> >> >> libraries should be refactored from scipy packages to some "lib" >> package. For example, there would be packages like >> >> scipy.lib.blas >> scipy.lib.lapack >> scipy.lib.fftpack >> scipy.lib.minpack >> scipy.lib.cephes >> scipy.lib.odepack >> scipy.lib.quadpack >> scipy.lib.fitpack >> scipy.lib.minpack >> scipy.lib.superlu >> scipy.lib.amos >> scipy.lib.cdflib >> etc >> > Hey, I like that idea a *lot*. I think I do too. It is mainly from a maintenance standpoint. On the down side, it separates some code out that seems like it should live together. But, it does remove some artificial constraints on function organization. If we decide to do it, we should move to svn before we try it though. That kind of re-org in CVS makes me cringe... eric From oliphant at ee.byu.edu Tue Sep 28 12:52:17 2004 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 28 Sep 2004 10:52:17 -0600 Subject: [SciPy-dev] Re: Slicot library and scipy In-Reply-To: <41597424.9010203@mecha.uni-stuttgart.de> References: <41597424.9010203@mecha.uni-stuttgart.de> Message-ID: <415996C1.9050703@ee.byu.edu> Nils Wagner wrote: > Dear Travis, > > As you encouraged me to learn f2py I would like to ask you, > if you can help me with an interface for continuous and discrete Lyapunov > equations as a starting point. Once I know how to manage it in general, > I would be glad to submit more slicot routines to scipy. It looks like your problems are related to making sure SB03MD has the libraries it needs to run. It looks like it needs a dummy SELECT function for DGEES (it actually never calls it but the interface requires it). You also need to be sure that SB03MY and SB03MX and MB01RD are compiled and linked along with SB03MD (it calls them). I'm attaching a modified .pyf file that you could use and a util.f file with the SELECT function defined. You need to compile like this (assuming you have a libslicot.a placed in ) f2py -L -lslicot -L -llapack -lf77blas -latlas -c slicot.pyf utils.f This is the better way to do it and will allow you to grow slicot.pyf until you've made available all of the routines. You can start the process of wrapping all of the slicot library by letting f2py construct the basic .pyf file by going to the slicot src directory and doing f2py -h slicot.pyf -m slicot *.f This will give you a basic, raw interface to all of the slicot routines. You will then want to edit the slicot.pyf file (like I did here for sb03md) to make the routine more useful --- hide some variables, show that some are input and some are output variables --- and so forth. You will find along the way that f2py will fail to parse some of the SLICOT fortran routines. Just don't include them in the list of routines to make an interface for. I needed to move IB03AD.f and IB03BD.f out of the way before running the command to get it to complete. The resulting slicot.pyf file is huge and contains a very raw interface to the entire slicot library (minus those two routines --- I'm submitting a bug report to Pearu now). The slicot.pyf file can be compiled with f2py as before. If you want to write a setup.py file you can (but that is another story). > > I am quite sure that many scipy users are interested in features > provided by > the slicot library. > > What do you think ? > SLICOT would be a nice addition to SciPy. But... I noticed that the SLICOT folks don't want people distributing SLICOT with their library, so it looks like only the interface could be distributed with SciPy and you would have to go and download the SLICOT tools from them, which is too bad unless some kind of agreement and separate license for SciPy could be obtained --- it looks like they mainly want credit, so in SciPy we could put in the help for the tools that if you use them you must credit them in a paper. At any rate, perhaps the interface could be distributed and then a check made at compile time (by default off) to see if the library is already there then the interface could be built. -Travis > I look forward to heraring from you. > > Cheers, > Nils > > > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: utils.f URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: slicot.pyf URL: From oliphant at ee.byu.edu Tue Sep 28 13:01:16 2004 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 28 Sep 2004 11:01:16 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: References: Message-ID: <415998DC.6060902@ee.byu.edu> chris at fisher.forestry.uga.edu wrote: >On 9/27/2004, "Travis Oliphant" wrote: > > > >>We are not against reorganizations, but odd to you is not necessarily >>odd to someone else, and vice versa. So, let's just figure out were >>monte_carlo should go. I think it would go well under stats, or else a >>new AI subpackage. >> >> >> > >MCMC isnt really artificial intelligence, it is bayesian stats, so stats >would be best, I think. > > > You have a like-minded friend in your thinking there. I actually don't like the name stats (except that it is short) and prefer probability or prob. >>I agree that stats could use reorganization. Many routines were lifted >> >> >>from an old pstats.py file. While it has been significantly cleaned up, > > >>there are still problems. >> >> >> > >I would be glad to take a run at re-doing stats, if only because I would >be using it a *lot*. If so, I wouldnt mind getting a bit of background >from Travis (and others) about the module, particularly with respectto >what would need to be retained, and what the most desirable changes >would be. > > > The distributions.py file received a lot of attention from me. The rest of it only a bit of attention. So, I'd have more to say about anything in distributions.py > > >>I hear various complaints occasionally about slowness in some of the >>distributions in stats. In order to improve things, these need to be >>better described. Their shouldn't be a lot of slow-down in most of the >>routines (aside from domain validity slowness). Some distributions >>don't have exactly defined cdf's or ppf's and these must be computed by >>SciPy using integration and zero-finding routines. This will be very slow. >> >> > >The distributions that most folks would be using 95% of the time >(normal,mv-normal,gamma,beta,binomial,exponential,poisson) are well >defined and can be made pretty fast with FORTRAN extensions. > > And these in particular should already be fast (i.e. based on FORTRAN and C extensions). If there is something specific in the interface that is slowing them down unacceptably then that needs to be addressed if at all possible. There will always be an overhead for making a library call robust. I don't mind having exposed "faster" methods if the overhead for checking arguments is unacceptable, but it will not be the front-line command. -Travis From oliphant at ee.byu.edu Tue Sep 28 13:09:24 2004 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 28 Sep 2004 11:09:24 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> Message-ID: <41599AC4.1030907@ee.byu.edu> Pearu Peterson wrote: > > > On Mon, 27 Sep 2004, Charles Harris wrote: > >> eric jones wrote: >> >>> Where should these live? >>> monte carlo and markov chain might fit in scipy.stats? >> >> >> How about in monte_carlo or some such? I think there is too much >> stuff put in odd places. Why is zeros in optimize? Makes no sense, >> but there it is. I don't think now is the time to change all the >> directories around, but I hope we give some thought to the >> organization before it becomes unmanageable. The Dewey decimal >> classification was an achievement I am coming to appreciate. > > > I agree that tools provided by Scipy are not organized well with > respect to the mathematical subject that they deal with and so finding > the needed tool for some specific task may not be always easy. And I > agree that this issue should be tackled as early stage as possible in > Scipy evolution, otherwise it will get more and more difficult to > decide where to put contributions from the society and there is a > danger of postponing such decisions to an unreachable future.. > > The current organization of Scipy packages is mostly based on underlying > Fortran/C libraries and that is obviously not the best way to organize > any high-level scientific tool. > > While Chuck mentioned Dewey decimal classification then there are other > classification schemes available. For example, MSC > (http://www.ams.org/msc/). A nice overview of MSC can be found in > http://www.math.niu.edu/~rusin/known-math/index/index.html > > I don't know how well could these classification schemes be used > for organizing Scipy packages, may be we should take a look how Matlab > or Maple or Mathematica deal with organizing their tools. > >> From the maintenance point of view, IMHO, wrappers to external Fortran/C > > libraries should be refactored from scipy packages to some "lib" > package. For example, there would be packages like > > scipy.lib.blas > scipy.lib.lapack > scipy.lib.fftpack > scipy.lib.minpack > scipy.lib.cephes > scipy.lib.odepack > scipy.lib.quadpack > scipy.lib.fitpack > scipy.lib.minpack > scipy.lib.superlu > scipy.lib.amos > scipy.lib.cdflib > etc > I like this idea too. Right now, we have just been putting the library interfaces in the same top-level domain as the packages, but I think it does make more sense to put them in a .lib sub-package which other top-level routines can then call. I like this naming discussion. We have not had enough of it in the past. I guarantee the names and divisions that Pearu, I and Eric have chosen have been more out of convenience and "getting something" working then any great thought. I agree we should do this sooner than later. And I think we should move to SVN too. I think it would be wise to use some sort of standard convention. Something like MSC is good, or the NIST classification GAMS. http://gams.nist.gov/serve.cgi Right now, we have been sort of using the MATLAB and MAPLE approach which is group stuff together and give it a package name. MATLAB has the notion of toolboxes. We could break things up like they do, but I kind of like the idea of using a more standard convention that is used by others as well. -Travis From Fernando.Perez at colorado.edu Tue Sep 28 13:49:55 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 28 Sep 2004 11:49:55 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <41599AC4.1030907@ee.byu.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> Message-ID: <4159A443.8080605@colorado.edu> Travis Oliphant schrieb: > but I kind of like the idea of using a more standard convention that is > used by others as well. +1 Cheers, f From perry at stsci.edu Tue Sep 28 13:52:02 2004 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 28 Sep 2004 13:52:02 -0400 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <4158A23F.6080207@sdl.usu.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <4158A23F.6080207@sdl.usu.edu> Message-ID: <1A1C914C-1177-11D9-8495-000A95B68E50@stsci.edu> On Sep 27, 2004, at 7:29 PM, Charles Harris wrote: > Pearu Peterson wrote: > >> >> >> On Mon, 27 Sep 2004, Charles Harris wrote: >> >>> eric jones wrote: >>> >>>> Where should these live? >>>> monte carlo and markov chain might fit in scipy.stats? >>> >>> >>> How about in monte_carlo or some such? I think there is too much >>> stuff put in odd places. Why is zeros in optimize? Makes no sense, >>> but there it is. I don't think now is the time to change all the >>> directories around, but I hope we give some thought to the >>> organization before it becomes unmanageable. The Dewey decimal >>> classification was an achievement I am coming to appreciate. >> >> >> I agree that tools provided by Scipy are not organized well with >> respect to the mathematical subject that they deal with and so >> finding the needed tool for some specific task may not be always >> easy. And I agree that this issue should be tackled as early stage as >> possible in Scipy evolution, otherwise it will get more and more >> difficult to decide where to put contributions from the society and >> there is a danger of postponing such decisions to an unreachable >> future.. >> >> The current organization of Scipy packages is mostly based on >> underlying >> Fortran/C libraries and that is obviously not the best way to organize >> any high-level scientific tool. >> >> While Chuck mentioned Dewey decimal classification then there are >> other >> classification schemes available. For example, MSC >> (http://www.ams.org/msc/). A nice overview of MSC can be found in >> http://www.math.niu.edu/~rusin/known-math/index/index.html >> >> I don't know how well could these classification schemes be used >> for organizing Scipy packages, may be we should take a look how >> Matlab or Maple or Mathematica deal with organizing their tools. >> >> I have a feeling that no matter what classification system is used, so long as it is purely hierarchical people are going to expect to find an item in another area. Some things naturally associate with more than one category. Picking a good system is important, but alone is unlikely to eliminate user frustration in finding things. I suspect that being able to find tools through keyword associations or some other means that allows classifying the tool in more than one way will also be necessary. Mind you, this doesn't need to be reflected in the package structure, just that there is a way of searching for the tool using alternate associations (through web forms, a script, or whatever). I think that the documentation system (embedded in doc strings or similar) should enable this sort of search. Perry From charles.harris at sdl.usu.edu Tue Sep 28 13:54:01 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Tue, 28 Sep 2004 11:54:01 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <41599AC4.1030907@ee.byu.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> Message-ID: <4159A539.5040601@sdl.usu.edu> Travis Oliphant wrote: > Pearu Peterson wrote: > >> >> >> On Mon, 27 Sep 2004, Charles Harris wrote: >> >>> eric jones wrote: >>> >>>> Where should these live? >>>> monte carlo and markov chain might fit in scipy.stats? >>> >>> >>> >>> How about in monte_carlo or some such? I think there is too much >>> stuff put in odd places. Why is zeros in optimize? Makes no sense, >>> but there it is. I don't think now is the time to change all the >>> directories around, but I hope we give some thought to the >>> organization before it becomes unmanageable. The Dewey decimal >>> classification was an achievement I am coming to appreciate. >> >> >> >> I agree that tools provided by Scipy are not organized well with >> respect to the mathematical subject that they deal with and so >> finding the needed tool for some specific task may not be always >> easy. And I agree that this issue should be tackled as early stage as >> possible in Scipy evolution, otherwise it will get more and more >> difficult to decide where to put contributions from the society and >> there is a danger of postponing such decisions to an unreachable >> future.. >> >> The current organization of Scipy packages is mostly based on underlying >> Fortran/C libraries and that is obviously not the best way to organize >> any high-level scientific tool. >> >> While Chuck mentioned Dewey decimal classification then there are other >> classification schemes available. For example, MSC >> (http://www.ams.org/msc/). A nice overview of MSC can be found in >> http://www.math.niu.edu/~rusin/known-math/index/index.html >> >> I don't know how well could these classification schemes be used >> for organizing Scipy packages, may be we should take a look how >> Matlab or Maple or Mathematica deal with organizing their tools. >> >>> From the maintenance point of view, IMHO, wrappers to external >>> Fortran/C >> >> >> libraries should be refactored from scipy packages to some "lib" >> package. For example, there would be packages like >> >> scipy.lib.blas >> scipy.lib.lapack >> scipy.lib.fftpack >> scipy.lib.minpack >> scipy.lib.cephes >> scipy.lib.odepack >> scipy.lib.quadpack >> scipy.lib.fitpack >> scipy.lib.minpack >> scipy.lib.superlu >> scipy.lib.amos >> scipy.lib.cdflib >> etc >> > > I like this idea too. > Right now, we have just been putting the library interfaces in the > same top-level domain as the packages, but I think it does make more > sense to put them in a .lib sub-package which other top-level routines > can then call. > > I like this naming discussion. We have not had enough of it in the > past. I guarantee the names and divisions that Pearu, I and Eric have > chosen have been more out of convenience and "getting something" > working then any great thought. I agree we should do this sooner than > later. And I think we should move to SVN too. > > I think it would be wise to use some sort of standard convention. > Something like MSC is good, or the NIST classification GAMS. > http://gams.nist.gov/serve.cgi > I looked at MSC and GAMS. From a purely mathematical view MSC is nice, but a bit too high level, though it would be nice if lattices and control were part of GAMS. I think GAMS would be the best fit. A few caveats: we might want to expand or modify the subclassifications, Data seems sort of a catchall, and where would such things as equivalence relations go (Data?). One virtue of such a classification scheme is that it highlights areas that need to be addressed. > Right now, we have been sort of using the MATLAB and MAPLE approach > which is group stuff together and give it a package name. MATLAB has > the notion of toolboxes. We could break things up like they do, but I > kind of like the idea of using a more standard convention that is used > by others as well. > > > -Travis > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From Fernando.Perez at colorado.edu Tue Sep 28 13:59:04 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 28 Sep 2004 11:59:04 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <1A1C914C-1177-11D9-8495-000A95B68E50@stsci.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <4158A23F.6080207@sdl.usu.edu> <1A1C914C-1177-11D9-8495-000A95B68E50@stsci.edu> Message-ID: <4159A668.6080001@colorado.edu> Perry Greenfield schrieb: > I have a feeling that no matter what classification system is used, so > long as it > is purely hierarchical people are going to expect to find an item in > another area. > Some things naturally associate with more than one category. Picking a > good system > is important, but alone is unlikely to eliminate user frustration in > finding things. This is true, and an inherent problem of _any_ hierarchical system. However, I think that as much as in the unix world symlinks help alleviate some of this in file management, we can use the fact that python names are lightweight bindings to our advantage. For things which can truly be expected to live in more than one place, judicious use of: scipy.foo.bar = scipy.otherfoo.bar can make life easy, if it's equally reasonable for users to expect bar to be under foo and under otherfoo. I don't think the library names should be mercilessly duplicated everywhere, but something like this can really help in a few places. Obviously, a good, searchable, cross-referenced documentation is a key part of the soultion to this problem, as you mention. Cheers, f From charles.harris at sdl.usu.edu Tue Sep 28 14:12:28 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Tue, 28 Sep 2004 12:12:28 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <1A1C914C-1177-11D9-8495-000A95B68E50@stsci.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <4158A23F.6080207@sdl.usu.edu> <1A1C914C-1177-11D9-8495-000A95B68E50@stsci.edu> Message-ID: <4159A98C.3040008@sdl.usu.edu> Perry Greenfield wrote: > > On Sep 27, 2004, at 7:29 PM, Charles Harris wrote: > >> Pearu Peterson wrote: >> >>> >>> >>> On Mon, 27 Sep 2004, Charles Harris wrote: >>> >>>> eric jones wrote: >>>> >>>>> Where should these live? >>>>> monte carlo and markov chain might fit in scipy.stats? >>>> >>>> >>>> >>>> How about in monte_carlo or some such? I think there is too much >>>> stuff put in odd places. Why is zeros in optimize? Makes no sense, >>>> but there it is. I don't think now is the time to change all the >>>> directories around, but I hope we give some thought to the >>>> organization before it becomes unmanageable. The Dewey decimal >>>> classification was an achievement I am coming to appreciate. >>> >>> >>> >>> I agree that tools provided by Scipy are not organized well with >>> respect to the mathematical subject that they deal with and so >>> finding the needed tool for some specific task may not be always >>> easy. And I agree that this issue should be tackled as early stage >>> as possible in Scipy evolution, otherwise it will get more and more >>> difficult to decide where to put contributions from the society and >>> there is a danger of postponing such decisions to an unreachable >>> future.. >>> >>> The current organization of Scipy packages is mostly based on >>> underlying >>> Fortran/C libraries and that is obviously not the best way to organize >>> any high-level scientific tool. >>> >>> While Chuck mentioned Dewey decimal classification then there are other >>> classification schemes available. For example, MSC >>> (http://www.ams.org/msc/). A nice overview of MSC can be found in >>> http://www.math.niu.edu/~rusin/known-math/index/index.html >>> >>> I don't know how well could these classification schemes be used >>> for organizing Scipy packages, may be we should take a look how >>> Matlab or Maple or Mathematica deal with organizing their tools. >>> >>> > I have a feeling that no matter what classification system is used, so > long as it > is purely hierarchical people are going to expect to find an item in > another area. > Some things naturally associate with more than one category. Picking a > good system > is important, but alone is unlikely to eliminate user frustration in > finding things. > I suspect that being able to find tools through keyword associations > or some other > means that allows classifying the tool in more than one way will also > be necessary. > Mind you, this doesn't need to be reflected in the package structure, > just that there > is a way of searching for the tool using alternate associations > (through web forms, > a script, or whatever). I think that the documentation system > (embedded in doc strings > or similar) should enable this sort of search. > There is some truth in this. There is also the difference between classifying by tool type, -- hammer, saw, etc. -- and classifying by expected use -- boat building, house building, etc. That said, I could see how GAMS would provide a nifty table of contents for the documentation. I also felt relief that I could see where several contributions I would like to add would fit in. I have felt hesitant to introduce new modules, and I feel the current group is too narrow. > Perry > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From oliphant at ee.byu.edu Tue Sep 28 14:12:48 2004 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 28 Sep 2004 12:12:48 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <4159A668.6080001@colorado.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <4158A23F.6080207@sdl.usu.edu> <1A1C914C-1177-11D9-8495-000A95B68E50@stsci.edu> <4159A668.6080001@colorado.edu> Message-ID: <4159A9A0.8070804@ee.byu.edu> > I don't think the library names should be mercilessly duplicated > everywhere, but something like this can really help in a few places. > Obviously, a good, searchable, cross-referenced documentation is a key > part of the soultion to this problem, as you mention. As far as searching goes. scipy.info started something like this a while back. The functionality was lost a bit when the delayed import mechanism was used (but it could be modified to work). The idea was that scipy.info() would start looking for in all the documentation it could find and then would print what it found. It still works on things that are in the namespace of scipy but not subpackages. try scipy.info("fft") for example It is rudimentary, but shows an idea that could be pursued. We should also think about a help-browser system that could pop up (in a different process I think), when the user enters a help command. This is what maple and matlab do effectively. -Travis From charles.harris at sdl.usu.edu Tue Sep 28 14:18:52 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Tue, 28 Sep 2004 12:18:52 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <4159A668.6080001@colorado.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <4158A23F.6080207@sdl.usu.edu> <1A1C914C-1177-11D9-8495-000A95B68E50@stsci.edu> <4159A668.6080001@colorado.edu> Message-ID: <4159AB0C.2000007@sdl.usu.edu> Fernando Perez wrote: > Perry Greenfield schrieb: > >> I have a feeling that no matter what classification system is used, >> so long as it >> is purely hierarchical people are going to expect to find an item in >> another area. >> Some things naturally associate with more than one category. Picking >> a good system >> is important, but alone is unlikely to eliminate user frustration in >> finding things. > > > This is true, and an inherent problem of _any_ hierarchical system. > However, I think that as much as in the unix world symlinks help > alleviate some of this in file management, we can use the fact that > python names are lightweight bindings to our advantage. For things > which can truly be expected to live in more than one place, judicious > use of: > I agree. Toplevel modules should be able to import tools from several of the lowlevel libraries in scipy.lib . I think any good factoring of a large program falls into this general "lozenge" shape of dependences. > scipy.foo.bar = scipy.otherfoo.bar > > can make life easy, if it's equally reasonable for users to expect bar > to be under foo and under otherfoo. > > I don't think the library names should be mercilessly duplicated > everywhere, but something like this can really help in a few places. > Obviously, a good, searchable, cross-referenced documentation is a key > part of the soultion to this problem, as you mention. > > Cheers, > > f > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From nwagner at mecha.uni-stuttgart.de Tue Sep 28 15:05:00 2004 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Tue, 28 Sep 2004 21:05:00 +0200 Subject: [SciPy-dev] Re: Slicot library and scipy In-Reply-To: <415996C1.9050703@ee.byu.edu> Message-ID: On Tue, 28 Sep 2004 10:52:17 -0600 Travis Oliphant wrote: >Nils Wagner wrote: > >>Dear Travis, >> >>As you encouraged me to learn f2py I would like to ask >>you, >>if you can help me with an interface for continuous and >>discrete Lyapunov >>equations as a starting point. Once I know how to manage >>it in general, >>I would be glad to submit more slicot routines to scipy. > > >It looks like your problems are related to making sure >SB03MD has the libraries it needs to run. It looks like >it needs a dummy SELECT function for DGEES (it actually >never calls it but the interface requires it). > >You also need to be sure that SB03MY and SB03MX and >MB01RD are compiled and linked along with SB03MD (it >calls them). >I'm attaching a modified .pyf file that you could use and >a util.f file with the SELECT function defined. >You need to compile like this (assuming you have a >libslicot.a placed in ) > >f2py -L -lslicot -L >-llapack -lf77blas -latlas -c slicot.pyf utils.f > >This is the better way to do it and will allow you to >grow slicot.pyf until you've made available all of the >routines. >You can start the process of wrapping all of the slicot >library by letting f2py construct the basic .pyf file by >going to the slicot src directory and doing > >f2py -h slicot.pyf -m slicot *.f > >This will give you a basic, raw interface to all of the >slicot routines. You will then want to edit the >slicot.pyf file (like I did here for sb03md) to make the >routine more useful --- hide some variables, show that >some are input and some are output variables --- and so >forth. > >You will find along the way that f2py will fail to parse >some of the SLICOT fortran routines. Just don't include >them in the list of routines to make an interface for. I >needed to move IB03AD.f and IB03BD.f out of the way >before running the command to get it to complete. The >resulting slicot.pyf file is huge and contains a very raw >interface to the entire slicot library (minus those two >routines --- I'm submitting a bug report to Pearu now). >The slicot.pyf file can be compiled with f2py as before. > If you want to write a setup.py file you can (but that >is another story). > > >> >>I am quite sure that many scipy users are interested in >>features >>provided by >>the slicot library. >> >>What do you think ? >> > >SLICOT would be a nice addition to SciPy. > >But... > >I noticed that the SLICOT folks don't want people >distributing SLICOT with their library, so it looks like >only the interface could be distributed with SciPy and >you would have to go and download the SLICOT tools from >them, which is too bad unless some kind of agreement and >separate license for SciPy could be obtained --- it looks >like they mainly want credit, so in SciPy we could put in >the help for the tools that if you use them you must >credit them in a paper. > >At any rate, perhaps the interface could be distributed >and then a check made at compile time (by default off) to >see if the library is already there then the interface >could be built. >-Travis > > > >>I look forward to heraring from you. >> >>Cheers, >> Nils Travis, Thank you very much for your detailed hints ! I will ask the developers of slicot for a possible integration of slicot in scipy and get back to you as soon as possible. Cheers, Nils >> >> >> > Dr.-Ing. Nils Wagner Universit?t Stuttgart Institut A f?r Mechanik Pfaffenwaldring 9 D-70550 Stuttgart Phone: +49 (0)711 685 6262 Fax: +49 (0)711 685 6282 E-mail: nwagner at mecha.uni-stuttgart.de From charles.harris at sdl.usu.edu Tue Sep 28 15:09:09 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Tue, 28 Sep 2004 13:09:09 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <4159A668.6080001@colorado.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <4158A23F.6080207@sdl.usu.edu> <1A1C914C-1177-11D9-8495-000A95B68E50@stsci.edu> <4159A668.6080001@colorado.edu> Message-ID: <4159B6D5.4090401@sdl.usu.edu> Fernando Perez wrote: > Perry Greenfield schrieb: > >> I have a feeling that no matter what classification system is used, >> so long as it >> is purely hierarchical people are going to expect to find an item in >> another area. >> Some things naturally associate with more than one category. Picking >> a good system >> is important, but alone is unlikely to eliminate user frustration in >> finding things. > > > This is true, and an inherent problem of _any_ hierarchical system. > However, I think that as much as in the unix world symlinks help > alleviate some of this in file management, we can use the fact that > python names are lightweight bindings to our advantage. For things > which can truly be expected to live in more than one place, judicious > use of: > > scipy.foo.bar = scipy.otherfoo.bar > > can make life easy, if it's equally reasonable for users to expect bar > to be under foo and under otherfoo. > > I don't think the library names should be mercilessly duplicated > everywhere, but something like this can really help in a few places. > Obviously, a good, searchable, cross-referenced documentation is a key > part of the soultion to this problem, as you mention. > To expand on this theme a bit more, some tools have multiple uses. Pure splines would fall into the interpolation category, splines in tension or b-splines used as a basis for least squares would fall under fitting. Likewise, triangulations would be used in both ODE and contour plotting, and perhaps call on data structures such as quad trees. Graphics might make use of kd-trees, etc., etc. > Cheers, > > f > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > From Fernando.Perez at colorado.edu Tue Sep 28 15:36:19 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 28 Sep 2004 13:36:19 -0600 Subject: [SciPy-dev] matrixmultiply!=dot when dotblas is present Message-ID: <4159BD33.5060604@colorado.edu> Hi all, I just reported a Numpy bug to the numpy list, but scipy can work around it trivially. The problem is the following (I'm copy/pasting my original msg): I found something today a bit unpleasant: if you install numeric without any BLAS support, 'matrixmultiply is dot==True', so they are fully interchangeable. However, to my surprise, if you build numeric with the blas optimizations, they are NOT identical. The reason is a bug in Numeric.py. After they define dot, they do this: #This is obsolete, don't use in new code matrixmultiply = dot and at the very end of the file, they do: # try to import blas optimized dot, innerproduct and vdot, if available try: from dotblas import dot, innerproduct, vdot except ImportError: pass Obviously this means that matrixmultiply is stuck with the _old_ definition of dot, and does not benefit from the blas optimizations. This is BAD, as for a 1024x1024 matrix the difference is staggering: planck[Numeric]> pylab In [1]: a=rand(1024,1024) In [2]: b=rand(1024,1024) In [3]: from IPython.genutils import timing In [4]: timing 1,dot,a,b ------> timing(1,dot,a,b) Out[4]: 0.55591500000000005 In [5]: timing 1,matrixmultiply,a,b ------> timing(1,matrixmultiply,a,b) Out[5]: 68.142640999999998 In [6]: _/__ Out[6]: 122.57744619231356 Pretty significant difference, eh? :) Scipy can work around this problem (which is still there in Numpy 23.4) by manually doing 'matrixmultiply=dot' AFTER dot has been imported from Numeric. This will guarantee that users of scipy.matrixmultiply() actually get the fast version. Cheers, f From pearu at scipy.org Tue Sep 28 16:35:46 2004 From: pearu at scipy.org (Pearu Peterson) Date: Tue, 28 Sep 2004 15:35:46 -0500 (CDT) Subject: [SciPy-dev] Re: Slicot library and scipy In-Reply-To: <415996C1.9050703@ee.byu.edu> References: <41597424.9010203@mecha.uni-stuttgart.de> <415996C1.9050703@ee.byu.edu> Message-ID: On Tue, 28 Sep 2004, Travis Oliphant wrote: > You will find along the way that f2py will fail to parse some of the SLICOT > fortran routines. Just don't include them in the list of routines to make an > interface for. I needed to move IB03AD.f and IB03BD.f out of the way before > running the command to get it to complete. The resulting slicot.pyf file is > huge and contains a very raw interface to the entire slicot library (minus > those two routines --- I'm submitting a bug report to Pearu now). Thanks for the bug report. The bug is fixed in f2py CVS. Pearu From pearu at scipy.org Wed Sep 29 08:18:36 2004 From: pearu at scipy.org (Pearu Peterson) Date: Wed, 29 Sep 2004 07:18:36 -0500 (CDT) Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <41599AC4.1030907@ee.byu.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> Message-ID: On Tue, 28 Sep 2004, Travis Oliphant wrote: >> From the maintenance point of view, IMHO, wrappers to external Fortran/C >> libraries should be refactored from scipy packages to some "lib" package. > > I like this idea too. Ok, it's official then. Any new wrappers to external libraries should be exposed via scipy.lib... > I like this naming discussion. We have not had enough of it in the past. I > guarantee the names and divisions that Pearu, I and Eric have chosen have > been more out of convenience and "getting something" working then any great > thought. I agree we should do this sooner than later. And I think we should > move to SVN too. Should we release Scipy 0.3.2 before the transition to SVN? I'd like to work more on Scipy in the coming months and it would be nice to have CVS->SVN change done. Pearu From charles.harris at sdl.usu.edu Wed Sep 29 12:16:26 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Wed, 29 Sep 2004 10:16:26 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> Message-ID: <415ADFDA.3000200@sdl.usu.edu> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PDF4.gif Type: image/gif Size: 130 bytes Desc: not available URL: From rkern at ucsd.edu Wed Sep 29 16:53:00 2004 From: rkern at ucsd.edu (Robert Kern) Date: Wed, 29 Sep 2004 13:53:00 -0700 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415ADFDA.3000200@sdl.usu.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415ADFDA.3000200@sdl.usu.edu> Message-ID: <415B20AC.4080103@ucsd.edu> Charles Harris wrote: [snip] > Just to continue the naming discussion, here are the GAM categories: A plaintext version is here: http://gams.nist.gov/Classes.plain I would suggest checking this file into the docs (renamed to GAMS-Classes.txt or something else more descriptive) and, for now, adding GAMS classification information to docstrings where appropriate. For example, scipy.special.kelvin would have at the bottom of its docstring GAMS Classifications: C10c. Kelvin functions And scipy.special would have GAMS Classifications: C. Elementary and special functions C10c. Kelvin Functions ... where ... is the union of all the GAMS classifications of all the other functions in scipy.special . Perhaps "C10. Bessel functions" should be in there, too, along with the other intermediate level classifications that apply. It's a judgement call. I'll just chip in my two cents now and agree with Perry and others that organizing the source tree according to GAMS is probably no better than an attempt to organize conscientiously. Search is a much better way to find what one is looking for (c.f. Google). Is it worth devising an inverted index search for scipy docstrings? Or should we just punt to CHM? I would note that the less I have to switch out of IPython, the happier I am. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From charles.harris at sdl.usu.edu Wed Sep 29 17:02:59 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Wed, 29 Sep 2004 15:02:59 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415B20AC.4080103@ucsd.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415ADFDA.3000200@sdl.usu.edu> <415B20AC.4080103@ucsd.edu> Message-ID: <415B2303.5010301@sdl.usu.edu> Robert Kern wrote: > Charles Harris wrote: > > [snip] > >> Just to continue the naming discussion, here are the GAM categories: > > > A plaintext version is here: > > http://gams.nist.gov/Classes.plain > > I would suggest checking this file into the docs (renamed to > GAMS-Classes.txt or something else more descriptive) and, for now, > adding GAMS classification information to docstrings where appropriate. > > For example, scipy.special.kelvin would have at the bottom of its > docstring > > GAMS Classifications: > C10c. Kelvin functions > > And scipy.special would have > > GAMS Classifications: > C. Elementary and special functions > C10c. Kelvin Functions > ... > > where ... is the union of all the GAMS classifications of all the > other functions in scipy.special . Perhaps "C10. Bessel functions" > should be in there, too, along with the other intermediate level > classifications that apply. It's a judgement call. > > I'll just chip in my two cents now and agree with Perry and others > that organizing the source tree according to GAMS is probably no > better than an attempt to organize conscientiously. Search is a much > better way to find what one is looking for (c.f. Google). > I agree that search and indexing are the best ways to find stuff, but I am mostly concerned as to where to commit stuff. Clustering, where does that go? Lattice methods, where do they go? How about useful data structures or combinatorics? So on and so forth. I think the upper level GAMS categories cover sufficient range that most things can be put into a directory without embarrassment. As to the detailed breakdown in the GAMS sub-classifications, I am not so sure. > Is it worth devising an inverted index search for scipy docstrings? Or > should we just punt to CHM? I would note that the less I have to > switch out of IPython, the happier I am. > From Fernando.Perez at colorado.edu Wed Sep 29 18:09:44 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed, 29 Sep 2004 16:09:44 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415B20AC.4080103@ucsd.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415ADFDA.3000200@sdl.usu.edu> <415B20AC.4080103@ucsd.edu> Message-ID: <415B32A8.6050604@colorado.edu> Robert Kern wrote: > I'll just chip in my two cents now and agree with Perry and others that > organizing the source tree according to GAMS is probably no better than > an attempt to organize conscientiously. Search is a much better way to > find what one is looking for (c.f. Google). I agree that for finding stuff, a good search engine beats the pants off _any_ hierarchical scheme. Google has proven that beyond the shadow of a doubt: it's easier to find a file on the internet than on your own hard disk! However, we still have to keep in mind that in usage, python code _is_ hierarchically strucured: scipy.linalg.blas..... So even if we have a phenomenal search system in place, we still need a _reasonable_ hierarchy for the code. It doesn't need to be perfect, and as I argued before a few well placed names can replicate useful things in key places. But for code commits and program writing, some hierarchy does remain a necessity. The argument for good search should, in my view, mean that we shouldn't spend an inordinate amount of effort in getting the hierarchical structure 'perfect', as that would be a waste of time. But something moderately reasonable is still needed, I think. Best, f From rkern at ucsd.edu Thu Sep 30 01:41:17 2004 From: rkern at ucsd.edu (Robert Kern) Date: Wed, 29 Sep 2004 22:41:17 -0700 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415B2303.5010301@sdl.usu.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415ADFDA.3000200@sdl.usu.edu> <415B20AC.4080103@ucsd.edu> <415B2303.5010301@sdl.usu.edu> Message-ID: <415B9C7D.4000008@ucsd.edu> Charles Harris wrote: [snip] > I agree that search and indexing are the best ways to find stuff, but I > am mostly concerned as to where to commit stuff. Clustering, where does > that go? scipy.cluster I would imagine. ;-) > Lattice methods, where do they go? How about useful data > structures or combinatorics? So on and so forth. I think the upper level > GAMS categories cover sufficient range that most things can be put into > a directory without embarrassment. As to the detailed breakdown in the > GAMS sub-classifications, I am not so sure. To make the discussion a bit more concrete, here is an example directory structure corresponding to the top-level GAMS classifications. The names are all my own, so feel free to pretend they are something more to your liking. scipy/ analysis/ numbertheory/ functions/ linalg/ interpolation/ rootfinding/ optimization/ calculus/ diffeq/ integraltransforms/ approximation/ probstat/ simulation/ datahandling/ symbolic/ geometry/ graphics/ service/ develop/ other/ Now that I see it, it is somewhat appealing. I would probably want to break up some of those into two or more top-level groups. I definitely don't want to see too many subpackages under each of the top-level groups ("Flat is better than nested."). Fernando, could you give an example or two where you would want to replicate a function across sub-packages? I'm wary of doing so as there is already the enormous amount of replication with respect to, at least, the base Numeric functions. Try scipy.special. in IPython. I realize what you're proposing doesn't even come close to that, but I'd like an example in any case. And since we are talking about re-organization, is there anything we can do about the problem I just mentioned? It wreaks havoc with not only tab-completion but also automatic documentation generation [1]. Is it practical to be careful about what we import into __init__.py? By which I mean not doing "from foo import *" in __init__.py where foo.py does "from scipy_base import *". On the other hand, explicitly listing all of the names in special is gonna be a major pain and fragile to boot. [1] http://www.scipy.org/documentation/apidocs/scipy/scipy.special.html -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Thu Sep 30 01:49:47 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed, 29 Sep 2004 23:49:47 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415B9C7D.4000008@ucsd.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415ADFDA.3000200@sdl.usu.edu> <415B20AC.4080103@ucsd.edu> <415B2303.5010301@sdl.usu.edu> <415B9C7D.4000008@ucsd.edu> Message-ID: <415B9E7B.5020204@colorado.edu> Robert Kern wrote: > Fernando, could you give an example or two where you would want to > replicate a function across sub-packages? I'm wary of doing so as there > is already the enormous amount of replication with respect to, at least, > the base Numeric functions. Try scipy.special. in IPython. I > realize what you're proposing doesn't even come close to that, but I'd > like an example in any case. I didn't really have anything specific in mind. I was actually thinking of things I _do_ have in my filesystem, for example: ~/ref is my directory with 'reference' stuff ~/research is my dir. with all research-related things. Where does research bibliography go? Well, I just put: ~/ref/research -> ~/research/ref with a symlink. I was just presenting a generic argument that in _specific_ cases where one might find a strong argument for something belonging in more than one place, then a python name to replicate it (given the low cost) might be a good solution. I was just trying to offer a way out of possible dilemmas on where to put something which _might_ need more than one home, not arguing for its use. I also think that such a measure should be used _sparingly_, and only after careful consideration. In practice, I think a reasonable (not necessarily perfect) hierarchy coupled with a good search system will solve the problem quite nicely. best, f From prabhu at aero.iitm.ernet.in Thu Sep 30 02:42:52 2004 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 30 Sep 2004 12:12:52 +0530 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415B32A8.6050604@colorado.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415ADFDA.3000200@sdl.usu.edu> <415B20AC.4080103@ucsd.edu> <415B32A8.6050604@colorado.edu> Message-ID: <16731.43756.429897.579014@monster.linux.in> >>>>> "FP" == Fernando Perez writes: FP> Robert Kern wrote: >> I'll just chip in my two cents now and agree with Perry and >> others that organizing the source tree according to GAMS is >> probably no better than an attempt to organize >> conscientiously. Search is a much better way to find what one >> is looking for (c.f. Google). FP> I agree that for finding stuff, a good search engine beats the FP> pants off _any_ hierarchical scheme. Google has proven that FP> beyond the shadow of a doubt: it's easier to find a file on FP> the internet than on your own hard disk! Some time in the distant past, I wrote a help browser for the VTK API, with a simple and/or based search capability. The code is part of MayaVi (inside ivtk.py) and really quite straightforward. It basically grabs all the class documentation from the 700 odd VTK class doc strings, and lets you search by class name or by any given string on the class docs. It then lets you view the class docs. Its definitely not perfect, but works. I'm sure it could be used on the scipy docs with some work. If you have MayaVi installed, run the attached script. The slow start up time is because its importing VTK and not because its generating the docs. :) The GUI is crappy but it can be improved. I'm sure the approach can be used for a web based search, inside IPython etc. So if you are interested you can use/modify that code. cheers, prabhu -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: vtk_doc URL: From nwagner at mecha.uni-stuttgart.de Thu Sep 30 02:49:38 2004 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Thu, 30 Sep 2004 08:49:38 +0200 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415B9C7D.4000008@ucsd.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415ADFDA.3000200@sdl.usu.edu> <415B20AC.4080103@ucsd.edu> <415B2303.5010301@sdl.usu.edu> <415B9C7D.4000008@ucsd.edu> Message-ID: <415BAC82.2030201@mecha.uni-stuttgart.de> Robert Kern wrote: > Charles Harris wrote: > > [snip] > >> I agree that search and indexing are the best ways to find stuff, but >> I am mostly concerned as to where to commit stuff. Clustering, where >> does that go? > > > scipy.cluster I would imagine. ;-) > >> Lattice methods, where do they go? How about useful data structures >> or combinatorics? So on and so forth. I think the upper level GAMS >> categories cover sufficient range that most things can be put into a >> directory without embarrassment. As to the detailed breakdown in the >> GAMS sub-classifications, I am not so sure. > > > To make the discussion a bit more concrete, here is an example > directory structure corresponding to the top-level GAMS > classifications. The names are all my own, so feel free to pretend > they are something more to your liking. > > scipy/ > analysis/ > numbertheory/ > functions/ > linalg/ > interpolation/ > rootfinding/ > optimization/ > calculus/ > diffeq/ > integraltransforms/ > approximation/ > probstat/ > simulation/ > datahandling/ > symbolic/ > geometry/ > graphics/ > service/ > develop/ > other/ How about an entry for Systems theory; control to reconcile with http://www.ams.org/msc/ I keep the slicot library in mind. Nils > > Now that I see it, it is somewhat appealing. I would probably want to > break up some of those into two or more top-level groups. I definitely > don't want to see too many subpackages under each of the top-level > groups ("Flat is better than nested."). > > Fernando, could you give an example or two where you would want to > replicate a function across sub-packages? I'm wary of doing so as > there is already the enormous amount of replication with respect to, > at least, the base Numeric functions. Try scipy.special. in > IPython. I realize what you're proposing doesn't even come close to > that, but I'd like an example in any case. > > And since we are talking about re-organization, is there anything we > can do about the problem I just mentioned? It wreaks havoc with not > only tab-completion but also automatic documentation generation [1]. > Is it practical to be careful about what we import into __init__.py? > By which I mean not doing "from foo import *" in __init__.py where > foo.py does "from scipy_base import *". On the other hand, explicitly > listing all of the names in special is gonna be a major pain and > fragile to boot. > > [1] http://www.scipy.org/documentation/apidocs/scipy/scipy.special.html > -- Dr.-Ing. Nils Wagner Institut A f?r Mechanik Universit?t Stuttgart Pfaffenwaldring 9 D-70550 Stuttgart Tel.: (+49) 0711 685 6262 Fax.: (+49) 0711 685 6282 E-mail: nwagner at mecha.uni-stuttgart.de From eric at enthought.com Thu Sep 30 03:40:59 2004 From: eric at enthought.com (eric jones) Date: Thu, 30 Sep 2004 02:40:59 -0500 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415B9C7D.4000008@ucsd.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415ADFDA.3000200@sdl.usu.edu> <415B20AC.4080103@ucsd.edu> <415B2303.5010301@sdl.usu.edu> <415B9C7D.4000008@ucsd.edu> Message-ID: <415BB88B.50408@enthought.com> Robert Kern wrote: > Charles Harris wrote: > > [snip] > >> I agree that search and indexing are the best ways to find stuff, but >> I am mostly concerned as to where to commit stuff. Clustering, where >> does that go? > > > scipy.cluster I would imagine. ;-) > >> Lattice methods, where do they go? How about useful data structures >> or combinatorics? So on and so forth. I think the upper level GAMS >> categories cover sufficient range that most things can be put into a >> directory without embarrassment. As to the detailed breakdown in the >> GAMS sub-classifications, I am not so sure. > > > To make the discussion a bit more concrete, here is an example > directory structure corresponding to the top-level GAMS > classifications. The names are all my own, so feel free to pretend > they are something more to your liking. > > > Now that I see it, it is somewhat appealing. I would probably want to > break up some of those into two or more top-level groups. I definitely > don't want to see too many subpackages under each of the top-level > groups ("Flat is better than nested."). > Here is where the current SciPy modules would likely get lumped in the GAMS hierarchy. scipy/ analysis/ numbertheory/ functions/ special linalg/ linalg, sparse interpolation/ interpolate rootfinding/ optimization/ optimize, ga calculus/ integrate diffeq/ integraltransforms/ fftpack approximation/ probstat/ stats simulation/ datahandling/ io symbolic/ geometry/ graphics/ xplt, gplt, plt service/ gui_thread develop/ other/ cow, cluster ??, signal ?? (Cluster and signal didn't fit anywhere obvious to me) The naming conventions are often quite similar. The SciPy names are generally shorter which is nice for typing. Where SciPy has multiple packages [(linalg, sparse), (optimize, ga), etc.], it is likely a good idea. Like you, I don't want to see a deep nesting in the package structure. Looking at this, I don't see any real reason to reorganize top level package names. Are any of them that bad or misleading? On the other hand, I do think we should reorganize the functions within them some to fix the places where they are organized based on "build" convenience instead of actual function. This will probably necessitate the addition of new top level groups and maybe the pruning of one of the current ones. I've made a Wiki page to keep suggestions that people have: http://www.scipy.org/wikis/featurerequests/PackageReorganization If you update the page, you might also post to python-dev so that people know to go check on the Wiki (that is so painful...). We can obviously also just discuss it here and then transfer to the Wiki later. [side note: this using a wiki and a mailing list for communication is also a little painful]. > Fernando, could you give an example or two where you would want to > replicate a function across sub-packages? I'm wary of doing so as > there is already the enormous amount of replication with respect to, > at least, the base Numeric functions. Try scipy.special. in > IPython. I realize what you're proposing doesn't even come close to > that, but I'd like an example in any case. I don't like the replication idea very well. I think things should live in one place. Otherwise people will wonder if two functions that are actually the same have different purposes, implementation, etc. > > And since we are talking about re-organization, is there anything we > can do about the problem I just mentioned? It wreaks havoc with not > only tab-completion but also automatic documentation generation [1]. > Is it practical to be careful about what we import into __init__.py? > By which I mean not doing "from foo import *" in __init__.py where > foo.py does "from scipy_base import *". On the other hand, explicitly > listing all of the names in special is gonna be a major pain and > fragile to boot. I used to love "from xxx import *" and swore it was the right way to handle Numeric, etc. since I few "array" and friends as builtin functions... I guess I've been hanging out with to many computer scientists lately. Or perhaps it is the few times where I have wondered "where is function xxx [which is broken] coming from?" and struggled through a large codebase to track it down. We've had a nasty bug or two where one import * unexpectedly clobbered some functions from a previous import *. In any case, it is a (seldom broken) policy to never use import * in our code bases at Enthought. It is probably a good idea to apply this same policy to SciPy. Doing so would partially solve the problem you discuss. The 2nd thought we have had on this is to put a "filter" tag within the doc-string of a module or package to specify a set of functions that tab-complete tools should ignore. You could say "filter: sin, cos, ..." to remove a set of names or "filter: Numeric-all" (or something like that) to exclude all the functions in a module/package from the tab-complete list for a module. IPython, PyCrust, Idle, and others could standardize on a format so that they all could benefit. > > [1] http://www.scipy.org/documentation/apidocs/scipy/scipy.special.html > eric From eric at enthought.com Thu Sep 30 03:48:54 2004 From: eric at enthought.com (eric jones) Date: Thu, 30 Sep 2004 02:48:54 -0500 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> Message-ID: <415BBA66.802@enthought.com> Hey Pearu, Pearu Peterson wrote: > > > On Tue, 28 Sep 2004, Travis Oliphant wrote: > >>> From the maintenance point of view, IMHO, wrappers to external >>> Fortran/C libraries should be refactored from scipy packages to some >>> "lib" package. >> >> >> I like this idea too. > > > Ok, it's official then. Any new wrappers to external libraries should > be exposed via scipy.lib... > >> I like this naming discussion. We have not had enough of it in the >> past. I guarantee the names and divisions that Pearu, I and Eric have >> chosen have been more out of convenience and "getting something" >> working then any great thought. I agree we should do this sooner >> than later. And I think we should move to SVN too. > > > Should we release Scipy 0.3.2 before the transition to SVN? Seems like a good idea to do a final release with all that you, Travis O., and others have added before we make any of the big changes we're all discussing. Do you have time to do the tag/build/release process? If so, what is the timeline for doing so? Of course, we should get some testing feedback on the main platforms with the current CVS before doing this. Besides that, does anyone have any specific (small) things to address before we do so? If so, what is the timeframe? > I'd like to work more on Scipy in the coming months and it would be > nice to have CVS->SVN change done. I think we can do the SVN move within a couple of days or so once the go-ahead is given. eric From prabhu at aero.iitm.ernet.in Thu Sep 30 04:18:17 2004 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Thu, 30 Sep 2004 13:48:17 +0530 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415BBA66.802@enthought.com> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415BBA66.802@enthought.com> Message-ID: <16731.49481.790080.535401@monster.linux.in> >>>>> "EJ" == eric jones writes: EJ> Of course, we should get some testing feedback on the main EJ> platforms with the current CVS before doing this. Besides EJ> that, does anyone have any specific (small) things to address EJ> before we do so? If so, what is the timeframe? I wanted to add a bunch of tests for weave, but I've decided to instead add a couple of working examples. I'll try and do that by this weekend if its not a problem. I won't affect the build or anything. If you want me to do that after the SVN move, thats ok too. But if you are pusing for a 0.3.2 release, I'll try and get this done by this weekend. Let me know if thats OK. cheers, prabhu From pearu at scipy.org Thu Sep 30 07:47:24 2004 From: pearu at scipy.org (Pearu Peterson) Date: Thu, 30 Sep 2004 06:47:24 -0500 (CDT) Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415BBA66.802@enthought.com> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <415BBA66.802@enthought.com> Message-ID: On Thu, 30 Sep 2004, eric jones wrote: >> Should we release Scipy 0.3.2 before the transition to SVN? > > Seems like a good idea to do a final release with all that you, Travis O., > and others have added before we make any of the big changes we're all > discussing. > > Do you have time to do the tag/build/release process? If so, what is the > timeline for doing so? > Of course, we should get some testing feedback on the main platforms with the > current CVS before doing this. Besides that, does anyone have any specific > (small) things to address before we do so? If so, what is the timeframe? I am currently running scipy build/install/test on different systems that I have access to, they are Debian, Gentoo, WinNT with cygwin,mingw32,msvc. It should take a day if there will be no major bugs to be fixed. [On 64bit gentoo there are currently 12 failures and 2 errors in scipy.test().] After that I could start the scipy release process. Once we have tagged the tree, we can start building the binaries. Travis O., do you have anything to commit that should go into 0.3.2? Prabhu wanted to add few examples to weave during the weekend. That is OK with me as it does not affect the build process. Now, if we could have SVN repository set up in scipy.org site then we could make the release by the end of this week. If setting up SVN repository takes more time then we can wait Prabhu patches and make the release once scipy.org SVN repository is up. Btw, I'll be away on Sunday. Pearu From nwagner at mecha.uni-stuttgart.de Thu Sep 30 07:35:16 2004 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Thu, 30 Sep 2004 13:35:16 +0200 Subject: [SciPy-dev] Status of iterative solvers Message-ID: <415BEF74.3090700@mecha.uni-stuttgart.de> Hi all, BTW, when are the iterative solvers up for general testing in scipy ? Nils From charles.harris at sdl.usu.edu Thu Sep 30 10:52:29 2004 From: charles.harris at sdl.usu.edu (Charles Harris) Date: Thu, 30 Sep 2004 08:52:29 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? Message-ID: From: scipy-dev-bounces at scipy.net on behalf of eric jones > >Robert Kern wrote: >> Charles Harris wrote: >> >> [snip] >>> >>> I agree that search and indexing are the best ways to find stuff, but >>> I am mostly concerned as to where to commit stuff. Clustering, where >>> does that go? >> >> >> scipy.cluster I would imagine. ;-) >> >>> Lattice methods, where do they go? How about useful data structures >>> or combinatorics? So on and so forth. I think the upper level GAMS >>> categories cover sufficient range that most things can be put into a >>> directory without embarrassment. As to the detailed breakdown in the >>> GAMS sub-classifications, I am not so sure. >> >> >> To make the discussion a bit more concrete, here is an example >> directory structure corresponding to the top-level GAMS >> classifications. The names are all my own, so feel free to pretend >> they are something more to your liking. >> >> >> Now that I see it, it is somewhat appealing. I would probably want to >> break up some of those into two or more top-level groups. I definitely >> don't want to see too many subpackages under each of the top-level >> groups ("Flat is better than nested."). >> >>Here is where the current SciPy modules would likely get lumped in the >>GAMS hierarchy. >scipy/ > analysis/ > numbertheory/ > functions/ special > linalg/ linalg, sparse > interpolation/ interpolate > rootfinding/ > optimization/ optimize, ga > calculus/ integrate > diffeq/ > integraltransforms/ fftpack > approximation/ > probstat/ stats > simulation/ > datahandling/ io > symbolic/ > geometry/ > graphics/ xplt, gplt, plt > service/ gui_thread > develop/ > other/ cow, cluster ??, signal ?? >(Cluster and signal didn't fit anywhere obvious to me) Gams has clustering under probstat. The union/find algorithm could also go under data where they have trees, etc. Lattice stuff goes into numbertheory. Service also contains the machine parameters, eps and such. Remez algorithm goes into approximation (L_inf). Weave into develop. Hmm, signal processing is missing somewhere. Markov chain into simulation. Grey codes, permutations into other, although GAMS has permutations under data. We could split diffeq into ode, pde, but it is probably ok as is. Control is under optimization (for optimal control) but could be brought to the top. The addition of rootfinding is good. Convolutions and such are under integraltransforms, Haar transforms would go there also. Filters are under probstats in time series analysis, so it might make sense to create a signal (time series?) directory, probstats seems to be an overloaded GAMS category that could use some upperlevel subdivision. >The naming conventions are often quite similar. The SciPy names are >generally shorter which is nice for typing. Where SciPy has multiple >packages [(linalg, sparse), (optimize, ga), etc.], it is likely a good >idea. Like you, I don't want to see a deep nesting in the package >structure. >Looking at this, I don't see any real reason to reorganize top level >package names. Are any of them that bad or misleading? On the other >hand, I do think we should reorganize the functions within them some to >fix the places where they are organized based on "build" convenience >instead of actual function. This will probably necessitate the addition >of new top level groups and maybe the pruning of one of the current >ones. I've made a Wiki page to keep suggestions that people have: > http://www.scipy.org/wikis/featurerequests/PackageReorganization >If you update the page, you might also post to python-dev so that people >know to go check on the Wiki (that is so painful...). We can obviously >also just discuss it here and then transfer to the Wiki later. [side >note: this using a wiki and a mailing list for communication is also a >little painful]. >> Fernando, could you give an example or two where you would want to >> replicate a function across sub-packages? I'm wary of doing so as >> there is already the enormous amount of replication with respect to, >> at least, the base Numeric functions. Try scipy.special. in >> IPython. I realize what you're proposing doesn't even come close to >> that, but I'd like an example in any case. >I don't like the replication idea very well. I think things should live >in one place. Otherwise people will wonder if two functions that are >actually the same have different purposes, implementation, etc. >> >> And since we are talking about re-organization, is there anything we >> can do about the problem I just mentioned? It wreaks havoc with not >> only tab-completion but also automatic documentation generation [1]. >> Is it practical to be careful about what we import into __init__.py? >> By which I mean not doing "from foo import *" in __init__.py where >> foo.py does "from scipy_base import *". On the other hand, explicitly >> listing all of the names in special is gonna be a major pain and >> fragile to boot. _______________________________________________ Scipy-dev mailing list Scipy-dev at scipy.net http://www.scipy.net/mailman/listinfo/scipy-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 5402 bytes Desc: not available URL: From nwagner at mecha.uni-stuttgart.de Thu Sep 30 11:07:43 2004 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Thu, 30 Sep 2004 17:07:43 +0200 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: References: Message-ID: <415C213F.1020906@mecha.uni-stuttgart.de> Charles Harris wrote: >From: scipy-dev-bounces at scipy.net on behalf of eric jones > > >>Robert Kern wrote: >> >> > > > >>>Charles Harris wrote: >>> >>>[snip] >>> >>> >>>>I agree that search and indexing are the best ways to find stuff, but >>>>I am mostly concerned as to where to commit stuff. Clustering, where >>>>does that go? >>>> >>>> >>>scipy.cluster I would imagine. ;-) >>> >>> >>> >>>>Lattice methods, where do they go? How about useful data structures >>>>or combinatorics? So on and so forth. I think the upper level GAMS >>>>categories cover sufficient range that most things can be put into a >>>>directory without embarrassment. As to the detailed breakdown in the >>>>GAMS sub-classifications, I am not so sure. >>>> >>>> >>>To make the discussion a bit more concrete, here is an example >>>directory structure corresponding to the top-level GAMS >>>classifications. The names are all my own, so feel free to pretend >>>they are something more to your liking. >>> >>> >>> >>> > > > >>>Now that I see it, it is somewhat appealing. I would probably want to >>>break up some of those into two or more top-level groups. I definitely >>>don't want to see too many subpackages under each of the top-level >>>groups ("Flat is better than nested."). >>> >>>Here is where the current SciPy modules would likely get lumped in the >>>GAMS hierarchy. >>> >>> > > > >>scipy/ >> analysis/ >> numbertheory/ >> functions/ special >> linalg/ linalg, sparse >> interpolation/ interpolate >> rootfinding/ >> optimization/ optimize, ga >> calculus/ integrate >> diffeq/ >> integraltransforms/ fftpack >> approximation/ >> probstat/ stats >> simulation/ >> datahandling/ io >> symbolic/ >> geometry/ >> graphics/ xplt, gplt, plt >> service/ gui_thread >> develop/ >> other/ cow, cluster ??, signal ?? >> >> > > > >>(Cluster and signal didn't fit anywhere obvious to me) >> >> > >Gams has clustering under probstat. The union/find algorithm could also go under data >where they have trees, etc. Lattice stuff goes into numbertheory. Service also contains >the machine parameters, eps and such. Remez algorithm goes into approximation (L_inf). Weave >into develop. Hmm, signal processing is missing somewhere. Markov chain into simulation. >Grey codes, permutations into other, although GAMS has permutations under data. > >We could split diffeq into ode, pde, but it is probably ok as is. > How about dde's (delay differential equations) ? http://fde.usaaa.ru/mirrors/www.cs.kuleuven.ac.be/~koen/delay/software.shtml >Control is under >optimization (for optimal control) but could be brought to the top. The addition of >rootfinding is good. Convolutions and such are under integraltransforms, Haar transforms >would go there also. Filters are under probstats in time series analysis, so it might make >sense to create a signal (time series?) directory, probstats seems to be an overloaded >GAMS category that could use some upperlevel subdivision. > > > > >>The naming conventions are often quite similar. The SciPy names are >>generally shorter which is nice for typing. Where SciPy has multiple >>packages [(linalg, sparse), (optimize, ga), etc.], it is likely a good >>idea. Like you, I don't want to see a deep nesting in the package >>structure. >> >> > > > >>Looking at this, I don't see any real reason to reorganize top level >>package names. Are any of them that bad or misleading? On the other >>hand, I do think we should reorganize the functions within them some to >>fix the places where they are organized based on "build" convenience >>instead of actual function. This will probably necessitate the addition >>of new top level groups and maybe the pruning of one of the current >>ones. I've made a Wiki page to keep suggestions that people have: >> >> > > > >> http://www.scipy.org/wikis/featurerequests/PackageReorganization >> >> > > > >>If you update the page, you might also post to python-dev so that people >>know to go check on the Wiki (that is so painful...). We can obviously >>also just discuss it here and then transfer to the Wiki later. [side >>note: this using a wiki and a mailing list for communication is also a >>little painful]. >> >> > > > >>>Fernando, could you give an example or two where you would want to >>>replicate a function across sub-packages? I'm wary of doing so as >>>there is already the enormous amount of replication with respect to, >>>at least, the base Numeric functions. Try scipy.special. in >>>IPython. I realize what you're proposing doesn't even come close to >>>that, but I'd like an example in any case. >>> >>> > > > >>I don't like the replication idea very well. I think things should live >>in one place. Otherwise people will wonder if two functions that are >>actually the same have different purposes, implementation, etc. >> >> > > > >>>And since we are talking about re-organization, is there anything we >>>can do about the problem I just mentioned? It wreaks havoc with not >>>only tab-completion but also automatic documentation generation [1]. >>>Is it practical to be careful about what we import into __init__.py? >>>By which I mean not doing "from foo import *" in __init__.py where >>>foo.py does "from scipy_base import *". On the other hand, explicitly >>>listing all of the names in special is gonna be a major pain and >>>fragile to boot. >>> >>> > > > > >_______________________________________________ >Scipy-dev mailing list >Scipy-dev at scipy.net >http://www.scipy.net/mailman/listinfo/scipy-dev > > > > >------------------------------------------------------------------------ > >_______________________________________________ >Scipy-dev mailing list >Scipy-dev at scipy.net >http://www.scipy.net/mailman/listinfo/scipy-dev > > -- Dr.-Ing. Nils Wagner Institut A f?r Mechanik Universit?t Stuttgart Pfaffenwaldring 9 D-70550 Stuttgart Tel.: (+49) 0711 685 6262 Fax.: (+49) 0711 685 6282 E-mail: nwagner at mecha.uni-stuttgart.de URL : http://www.mecha.uni-stuttgart.de From perry at stsci.edu Thu Sep 30 11:12:16 2004 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 30 Sep 2004 11:12:16 -0400 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415BB88B.50408@enthought.com> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415ADFDA.3000200@sdl.usu.edu> <415B20AC.4080103@ucsd.edu> <415B2303.5010301@sdl.usu.edu> <415B9C7D.4000008@ucsd.edu> <415BB88B.50408@enthought.com> Message-ID: <1D551B76-12F3-11D9-B931-000A95B68E50@stsci.edu> On Sep 30, 2004, at 3:40 AM, eric jones wrote: > I don't like the replication idea very well. I think things should > live in one place. Otherwise people will wonder if two functions that > are actually the same have different purposes, implementation, etc. > I agree with this sentiment. Use the search tool to find out where the one "true" item lives, but don't use aliasing within the package hierarchy. > I used to love "from xxx import *" and swore it was the right way to > handle Numeric, etc. since I few "array" and friends as builtin > functions... I guess I've been hanging out with to many computer > scientists lately. Or perhaps it is the few times where I have > wondered "where is function xxx [which is broken] coming from?" and > struggled through a large codebase to track it down. We've had a > nasty bug or two where one import * unexpectedly clobbered some > functions from a previous import *. In any case, it is a (seldom > broken) policy to never use import * in our code bases at Enthought. > It is probably a good idea to apply this same policy to SciPy. Doing > so would partially solve the problem you discuss. > This seems like a good idea as well. I don't see anything wrong with using something like: import numarray as na to make the code (and particularly expressions) more concise. It still prevents the above mentioned problems. > Perry From Fernando.Perez at colorado.edu Thu Sep 30 12:42:11 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 30 Sep 2004 10:42:11 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <1D551B76-12F3-11D9-B931-000A95B68E50@stsci.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415ADFDA.3000200@sdl.usu.edu> <415B20AC.4080103@ucsd.edu> <415B2303.5010301@sdl.usu.edu> <415B9C7D.4000008@ucsd.edu> <415BB88B.50408@enthought.com> <1D551B76-12F3-11D9-B931-000A95B68E50@stsci.edu> Message-ID: <415C3763.8030000@colorado.edu> Perry Greenfield schrieb: > On Sep 30, 2004, at 3:40 AM, eric jones wrote: > >>I don't like the replication idea very well. I think things should >>live in one place. Otherwise people will wonder if two functions that >>are actually the same have different purposes, implementation, etc. >> > > I agree with this sentiment. Use the search tool to find out where the > one "true" item lives, > but don't use aliasing within the package hierarchy. Fair enough :) > This seems like a good idea as well. I don't see anything wrong with > using > something like: > > import numarray as na > > to make the code (and particularly expressions) more concise. It still > prevents the above > mentioned problems. I've gotten into the habit of always writing: import Numeric as N everywhere, and it works quite well. It prevents the problems already mentioned, and it also allows the following kind of optimization: def foo(): dot = N.dot for i in longlist: dot(a,b)... since locals are much faster than globals, if you have some long loop which repeatedly accesses one of these functions, localizing it may pay off. If Numeric were imported as *, you'd need to change the local name, since: dot = dot wouldn't work. Cheers, f From eric at enthought.com Thu Sep 30 15:52:50 2004 From: eric at enthought.com (eric jones) Date: Thu, 30 Sep 2004 14:52:50 -0500 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <415BBA66.802@enthought.com> Message-ID: <415C6411.5030704@enthought.com> Pearu Peterson wrote: > > > On Thu, 30 Sep 2004, eric jones wrote: > >>> Should we release Scipy 0.3.2 before the transition to SVN? >> >> >> Seems like a good idea to do a final release with all that you, >> Travis O., and others have added before we make any of the big >> changes we're all discussing. >> >> Do you have time to do the tag/build/release process? If so, what is >> the timeline for doing so? Of course, we should get some testing >> feedback on the main platforms with the current CVS before doing >> this. Besides that, does anyone have any specific (small) things to >> address before we do so? If so, what is the timeframe? > > > I am currently running scipy build/install/test on different systems that > I have access to, they are Debian, Gentoo, WinNT with > cygwin,mingw32,msvc. > It should take a day if there will be no major bugs to be fixed. > [On 64bit gentoo there are currently 12 failures and 2 errors in > scipy.test().] > After that I could start the scipy release process. Once we have > tagged the tree, we can start building the binaries. > > Travis O., do you have anything to commit that should go into 0.3.2? > > Prabhu wanted to add few examples to weave during the weekend. That is > OK with me as it does not affect the build process. > > Now, if we could have SVN repository set up in scipy.org site then > we could make the release by the end of this week. If setting up SVN > repository takes more time then we can wait Prabhu patches and make > the release once scipy.org SVN repository is up. The SVN repository comes after the release, correct? I was thinking that we release using the CVS for one final time, and then switch the codebase to SVN after that. I'll defer to Joe on when the repository can be switched over. He has been out this week, so I'm guessing it is realistically next week. eric > > > Btw, I'll be away on Sunday. > > Pearu > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev From Fernando.Perez at colorado.edu Thu Sep 30 16:00:29 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 30 Sep 2004 14:00:29 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415BBA66.802@enthought.com> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415BBA66.802@enthought.com> Message-ID: <415C65DD.30204@colorado.edu> eric jones schrieb: > Of course, we should get some testing feedback on the main platforms > with the current CVS before doing this. Besides that, does anyone have > any specific (small) things to address before we do so? If so, what is > the timeframe? I have 2 things: 1. I'd like to have the implementation of Frobenius norms fixed. The current code, in scipy/linalg/basic.py does this: elif ord in ['fro','f']: X = scipy_base.mat(x) return sqrt(sum(diag(X.H * X))) This is a straightforward implementation of the formula: frob_norm(A) = sqrt(Trace(A*hermitian_conjugate(A))), [1] where A is a _matrix_, so that * is a true matrix product (not element-wise). Note, however, that this is equivalent to: frob_norm(A) = sqrt(sum_i(sum_j(|A_ij|**2))) [2] But [1] is an O(N**3) calculation, while [2] is O[N**2] for an NxN matrix, since [1] requires a matrix multiplication. The following will produce the right answer for complex matrices: sqrt(sum((a*conjugate(a)).flat)).real Here's a quick test of the difference, for a 1024x1024 complex matrix: In [27]: timing (1,scipy.linalg.norm,a) Out[27]: 7.7238250000000015 In [28]: def fnorm(a): ....: return sqrt(sum((a*conjugate(a)).flat)).real ....: In [29]: timing (1,fnorm,a) Out[29]: 0.12298099999999934 In [30]: scipy.linalg.norm(a)-fnorm(a) Out[30]: -2.2737367544323206e-13 Untested code valid for real/complex should probably be: if a.typecode()=='D': return sqrt(sum((a*conjugate(a)).flat)).real else: return sqrt(sum((a**2).flat)) 2. The matrixmultiply != bug I recently reported is _very_ serious for people working with large matrices. I also reported it in the numpy list, but nobody replied. Since scipy can trivially work around it with the one-liner I showed, I think this really should be done. Code for #1 is here (but test it further), and I posted the one-liner for #2 yesterday, so these two (IMHO important) fixes should take no time for a committer. Best, f From pearu at scipy.org Thu Sep 30 16:30:02 2004 From: pearu at scipy.org (Pearu Peterson) Date: Thu, 30 Sep 2004 15:30:02 -0500 (CDT) Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415C6411.5030704@enthought.com> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <415BBA66.802@enthought.com><415C6411.5030704@enthought.com> Message-ID: On Thu, 30 Sep 2004, eric jones wrote: > The SVN repository comes after the release, correct? I was thinking that we > release using the CVS for one final time, and then switch the codebase to SVN > after that. Right. I was just thinking that after the release commits to CVS would be minimal, but then again it doesn't really matter... > I'll defer to Joe on when the repository can be switched over. He has been > out this week, so I'm guessing it is realistically next week. Ok. Let's make the 0.3.2 release. And then switch over to SVN whenever it's server is ready. Pearu From Fernando.Perez at colorado.edu Thu Sep 30 16:09:22 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 30 Sep 2004 14:09:22 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <16731.49481.790080.535401@monster.linux.in> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415BBA66.802@enthought.com> <16731.49481.790080.535401@monster.linux.in> Message-ID: <415C67F2.80403@colorado.edu> Prabhu Ramachandran schrieb: >>>>>>"EJ" == eric jones writes: > > > EJ> Of course, we should get some testing feedback on the main > EJ> platforms with the current CVS before doing this. Besides > EJ> that, does anyone have any specific (small) things to address > EJ> before we do so? If so, what is the timeframe? > > I wanted to add a bunch of tests for weave, but I've decided to > instead add a couple of working examples. I'll try and do that by > this weekend if its not a problem. I won't affect the build or > anything. If you want me to do that after the SVN move, thats ok too. > But if you are pusing for a 0.3.2 release, I'll try and get this done > by this weekend. feel free to add any from here: http://amath.colorado.edu/faculty/fperez/python/weave_examples.html original source here: http://amath.colorado.edu/faculty/fperez/python/weave_examples.py I've been meaning to update these (some break with current weave/gcc) but haven't had the time for it. Best, f From oliphant at ee.byu.edu Thu Sep 30 16:24:30 2004 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 30 Sep 2004 14:24:30 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <415BBA66.802@enthought.com> Message-ID: <415C6B7E.2060407@ee.byu.edu> Pearu Peterson wrote: > > > On Thu, 30 Sep 2004, eric jones wrote: > >>> Should we release Scipy 0.3.2 before the transition to SVN? >> >> >> Seems like a good idea to do a final release with all that you, >> Travis O., and others have added before we make any of the big >> changes we're all discussing. >> >> Do you have time to do the tag/build/release process? If so, what is >> the timeline for doing so? Of course, we should get some testing >> feedback on the main platforms with the current CVS before doing >> this. Besides that, does anyone have any specific (small) things to >> address before we do so? If so, what is the timeframe? > > > I am currently running scipy build/install/test on different systems that > I have access to, they are Debian, Gentoo, WinNT with > cygwin,mingw32,msvc. > It should take a day if there will be no major bugs to be fixed. > [On 64bit gentoo there are currently 12 failures and 2 errors in > scipy.test().] > After that I could start the scipy release process. Once we have > tagged the tree, we can start building the binaries. > > Travis O., do you have anything to commit that should go into 0.3.2? > I wanted to get the iterative solvers up. I think this may take too long to get all of them working. But, I would like to get at least one or two working. So, perhaps we could wait for that. -Travis From oliphant at ee.byu.edu Thu Sep 30 16:27:22 2004 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 30 Sep 2004 14:27:22 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415C65DD.30204@colorado.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415BBA66.802@enthought.com> <415C65DD.30204@colorado.edu> Message-ID: <415C6C2A.8090209@ee.byu.edu> Fernando Perez wrote: > eric jones schrieb: > >> Of course, we should get some testing feedback on the main platforms >> with the current CVS before doing this. Besides that, does anyone >> have any specific (small) things to address before we do so? If so, >> what is the timeframe? > > > I have 2 things: > > 1. I'd like to have the implementation of Frobenius norms fixed. The > current code, in scipy/linalg/basic.py does this: > > > elif ord in ['fro','f']: > X = scipy_base.mat(x) > return sqrt(sum(diag(X.H * X))) good idea. > > 2. The matrixmultiply != bug I recently reported is _very_ serious for > people working with large matrices. I also reported it in the numpy > list, but nobody replied. Since scipy can trivially work around it > with the one-liner I showed, I think this really should be done. I think you will see that I released a new version of Numeric that fixed this problem. See 23.5 -Travis > > Code for #1 is here (but test it further), and I posted the one-liner > for #2 yesterday, so these two (IMHO important) fixes should take no > time for a committer. We could also fix it in SciPy too. From Fernando.Perez at colorado.edu Thu Sep 30 16:35:52 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 30 Sep 2004 14:35:52 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415C6C2A.8090209@ee.byu.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415BBA66.802@enthought.com> <415C65DD.30204@colorado.edu> <415C6C2A.8090209@ee.byu.edu> Message-ID: <415C6E28.8060408@colorado.edu> Travis Oliphant schrieb: >>2. The matrixmultiply != bug I recently reported is _very_ serious for >>people working with large matrices. I also reported it in the numpy >>list, but nobody replied. Since scipy can trivially work around it >>with the one-liner I showed, I think this really should be done. > > > I think you will see that I released a new version of Numeric that fixed > this problem. See 23.5 Great! But since you didn't mention anything on the list, and my telepathy link has been acting up lately, I missed it ;) >>Code for #1 is here (but test it further), and I posted the one-liner >>for #2 yesterday, so these two (IMHO important) fixes should take no >>time for a committer. > > > We could also fix it in SciPy too. It's just a one-liner, and it will help users of numeric <= 23.4, so why not. I'm glad to see these two go in, thanks a lot. I also had this old patch lying around, which I sent to the list about a year ago. It's far from critical, so feel free to drop it if you don't like it/disagree. I'll just paste my old email and reattach the patch (I hope it still applies). here's a simple patch for pilutil which adds a simple 'flatten' option to the image reading functions imread. Here are the resulting definition and docstrings: In [2]: scipy.fromimage? Definition: scipy.fromimage(im, flatten=0) Docstring: Takes a PIL image and returns a copy of the image in a Numeric container. If the image is RGB returns a 3-dimensional array: arr[:,:,n] is each channel Optional arguments: - flatten (0): if true, the image is flattened by calling convert('F') on the image object before extracting the numerical data. This flattens the color layers into a single grayscale layer. Note that the supplied image object is NOT modified. In [3]: scipy.imread? Definition: scipy.imread(name, flatten=0) Docstring: Read an image file from a filename. Optional arguments: - flatten (0): if true, the image is flattened by calling convert('F') on the resulting image object. This flattens the color layers into a single grayscale layer. This is useful in case you want to load straight from an image file (jpeg, tiff, ...) but you want a single grayscale array to do numerics on. The change is fully backwards compatible. I did it because we needed here to easily read images for testing some numerical transformations without dealing with the color channels. It was easier to patch scipy than to tell my coworkers to remember to convert every image first :) Regards, Fernando. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pilutil.diff URL: From oliphant at ee.byu.edu Thu Sep 30 16:52:51 2004 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 30 Sep 2004 14:52:51 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415C6E28.8060408@colorado.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415BBA66.802@enthought.com> <415C65DD.30204@colorado.edu> <415C6C2A.8090209@ee.byu.edu> <415C6E28.8060408@colorado.edu> Message-ID: <415C7223.1000009@ee.byu.edu> Fernando Perez wrote: > Travis Oliphant schrieb: > >>> 2. The matrixmultiply != bug I recently reported is _very_ serious >>> for people working with large matrices. I also reported it in the >>> numpy list, but nobody replied. Since scipy can trivially work >>> around it with the one-liner I showed, I think this really should be >>> done. >> >> >> >> I think you will see that I released a new version of Numeric that >> fixed this problem. See 23.5 > > > Great! But since you didn't mention anything on the list, and my > telepathy link has been acting up lately, I missed it ;) Sorry, I assumed you were monitoring releases of Numeric. I should have said something, or I should let you have my new telepathy upgrade... > > > I also had this old patch lying around, which I sent to the list about > a year ago. It's far from critical, so feel free to drop it if you > don't like it/disagree. I'll just paste my old email and reattach the > patch (I hope it still applies). > > here's a simple patch for pilutil which adds a simple 'flatten' option > to the > image reading functions imread. Here are the resulting definition and > docstrings: > > In [2]: scipy.fromimage? > Definition: scipy.fromimage(im, flatten=0) > Docstring: > Takes a PIL image and returns a copy of the image in a Numeric > container. > If the image is RGB returns a 3-dimensional array: arr[:,:,n] is > each > channel > > Optional arguments: > > - flatten (0): if true, the image is flattened by calling > convert('F') on > the image object before extracting the numerical data. This > flattens the > color layers into a single grayscale layer. Note that the > supplied image > object is NOT modified. > > > In [3]: scipy.imread? > Definition: scipy.imread(name, flatten=0) > Docstring: > Read an image file from a filename. > > Optional arguments: > > - flatten (0): if true, the image is flattened by calling > convert('F') on > the resulting image object. This flattens the color layers into > a single > grayscale layer. > > > This is useful in case you want to load straight from an image file > (jpeg, > tiff, ...) but you want a single grayscale array to do numerics on. The > change is fully backwards compatible. I did it because we needed here to > easily read images for testing some numerical transformations without > dealing > with the color channels. It was easier to patch scipy than to tell my > coworkers to remember to convert every image first :) > > Regards, > > Fernando. Seems reasonable. If there are no complaints. I'll add it. From prabhu at aero.iitm.ernet.in Thu Sep 30 17:14:45 2004 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Fri, 1 Oct 2004 02:44:45 +0530 Subject: Releasing 0.3.2 (was Re: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy?) In-Reply-To: References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <415BBA66.802@enthought.com> Message-ID: <16732.30533.572668.615973@monster.linux.in> >>>>> "PP" == Pearu Peterson writes: PP> Prabhu wanted to add few examples to weave during the PP> weekend. That is OK with me as it does not affect the build PP> process. Well, I'm done with my work. Just FYI, here is what I checked in: 1. scipy_core/weave/swig2_spec.py has been improved to transparently handle PyCObject based SWIG objects. This was a problem I just noticed because SWIG from CVS now defaults to using PyCObjects. There goes my sleep. 2. Added a few examples to scipy_core/weave/examples/. These include a. array3d.py: example to show how one can access a 3D numeric array with or without blitz. b. vtk_example.py: example illustrating how one can inline C++ to handle VTK objects. Only tested under Linux though. c. swig2_example.py, swig2_ext.h, swig2_ext.i: A simple example showing how SWIG2 wrapped objects can be handled with weave. The .h and .i files can be used to make a trivial SWIG2 module the commands for building which are mentioned in swig2_example.py. Just one point. swig2 support is not turned on by default, i.e. the following line in swig2_spec.py is commented:: #converters.default.insert(0, swig2_converter()) Folks who want to use swig2 support have to do this manually. I am not sure what to do about it and will let Eric decide. So, with that my job is done. You can go ahead and release 0.3.2 when you can. cheers, prabhu From prabhu at aero.iitm.ernet.in Thu Sep 30 17:29:03 2004 From: prabhu at aero.iitm.ernet.in (Prabhu Ramachandran) Date: Fri, 1 Oct 2004 02:59:03 +0530 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415C67F2.80403@colorado.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415BBA66.802@enthought.com> <16731.49481.790080.535401@monster.linux.in> <415C67F2.80403@colorado.edu> Message-ID: <16732.31391.526561.287892@monster.linux.in> >>>>> "FP" == Fernando Perez writes: FP> Prabhu Ramachandran schrieb: >> I wanted to add a bunch of tests for weave, but I've decided to >> instead add a couple of working examples. I'll try and do that >> by this weekend if its not a problem. I won't affect the build >> or anything. If you want me to do that after the SVN move, >> thats ok too. But if you are pusing for a 0.3.2 release, I'll >> try and get this done by this weekend. FP> feel free to add any from here: FP> http://amath.colorado.edu/faculty/fperez/python/weave_examples.html FP> original source here: FP> http://amath.colorado.edu/faculty/fperez/python/weave_examples.py FP> I've been meaning to update these (some break with current FP> weave/gcc) but haven't had the time for it. It would be great if these were part of the standard examples since they are well commented and illustrate a very good bit of functionality. Unfortunately, I really don't have the time to get it all working under the latest gcc, test it and check it in. I've lost too much sleep on this already. :( cheers, prabhu From rkern at ucsd.edu Thu Sep 30 21:01:20 2004 From: rkern at ucsd.edu (Robert Kern) Date: Thu, 30 Sep 2004 18:01:20 -0700 Subject: [SciPy-dev] Pre-release fixes Message-ID: <415CAC60.4060604@ucsd.edu> I've just checked in a fix for fastumath's logical functions. They used to give wrong answers on OSX and caused some tests to fail. I believe this to be an endianness issue, but I don't have another big-endian machine to test on. All tests still pass on Athlon/Linux. The changes made largely track what Numeric did, so I'm not sure if these were conscious deviations to speed up fastumath or just a failure to keep up with the bugfixes in Numeric. I have only implemented fastumath_unsigned.inc as yet. I will make similar changes in fastumath_nounsigned.inc as I accumulate tuits. Travis, could you check my changes to make sure I didn't revert differences you deliberately made? If they were deliberate, then we need to figure a way to revert to the correct, if slower, Numeric implementations on the Mac. Also, if anyone is using SciPy on a Sun or other big-endian machine, I would appreciate the output of scipy.test() with and without the patch. Thank you, everyone. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From eric at enthought.com Thu Sep 30 22:35:10 2004 From: eric at enthought.com (eric jones) Date: Thu, 30 Sep 2004 21:35:10 -0500 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415C65DD.30204@colorado.edu> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415BBA66.802@enthought.com> <415C65DD.30204@colorado.edu> Message-ID: <415CC25E.6080900@enthought.com> Fernando Perez wrote: > eric jones schrieb: > >> Of course, we should get some testing feedback on the main platforms >> with the current CVS before doing this. Besides that, does anyone >> have any specific (small) things to address before we do so? If so, >> what is the timeframe? > > > I have 2 things: > > > Code for #1 is here (but test it further), and I posted the one-liner > for #2 yesterday, so these two (IMHO important) fixes should take no > time for a committer. Hmmm. So here is the main thing I see that needs to be fixed from this email... why the heck aren't you a committer?? I thought you already had commit access. I will ask Joe to add you to the list (whether you want to or not :-) eric From oliphant at ee.byu.edu Thu Sep 30 23:41:08 2004 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 30 Sep 2004 21:41:08 -0600 Subject: [SciPy-dev] Pre-release fixes In-Reply-To: <415CAC60.4060604@ucsd.edu> References: <415CAC60.4060604@ucsd.edu> Message-ID: <415CD1D4.8010004@ee.byu.edu> Robert Kern wrote: > I've just checked in a fix for fastumath's logical functions. They > used to give wrong answers on OSX and caused some tests to fail. I > believe this to be an endianness issue, but I don't have another > big-endian machine to test on. All tests still pass on Athlon/Linux. > > The changes made largely track what Numeric did, so I'm not sure if > these were conscious deviations to speed up fastumath or just a > failure to keep up with the bugfixes in Numeric. > > I have only implemented fastumath_unsigned.inc as yet. I will make > similar changes in fastumath_nounsigned.inc as I accumulate tuits. > > Travis, could you check my changes to make sure I didn't revert > differences you deliberately made? If they were deliberate, then we > need to figure a way to revert to the correct, if slower, Numeric > implementations on the Mac. I'm not sure what the problem is here. But I think you undid deliberate changes that have implications for alter_numeric() and need to be discussed. I had intentially made the results of all logical operations be unsigned bytes. What was broken about this on the Mac? Could you explain your changes here, better? -Travis From Fernando.Perez at colorado.edu Thu Sep 30 23:54:27 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 30 Sep 2004 21:54:27 -0600 Subject: [SciPy-dev] MCMC, Kalman Filtering, AI for SciPy? In-Reply-To: <415CC25E.6080900@enthought.com> References: <4156440A.1040204@ee.byu.edu> <41566CBD.70005@enthought.com> <415855AD.4060701@sdl.usu.edu> <41599AC4.1030907@ee.byu.edu> <415BBA66.802@enthought.com> <415C65DD.30204@colorado.edu> <415CC25E.6080900@enthought.com> Message-ID: <415CD4F3.9060004@colorado.edu> eric jones wrote: > Hmmm. So here is the main thing I see that needs to be fixed from this > email... why the heck aren't you a committer?? I thought you already > had commit access. > > I will ask Joe to add you to the list (whether you want to or not :-) Well, I actually might be, I'm not sure. I have commit access (obviously) to ipython's directory in scipy's CVS. That may or may not give me access to the rest of scipy, I just don't know how things are set up. I was also a bit hesitant, but with your approval, I'll go ahead in the future (once things are confirmed working). Thanks (or not, I'm not sure it's a good thing for my already non-existent free time :) Best, f