From scott.sinclair.za at gmail.com Thu Dec 1 05:59:56 2011 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Thu, 1 Dec 2011 12:59:56 +0200 Subject: [SciPy-Dev] scipy docs edit rights In-Reply-To: <4ECE3D70.3050701@crans.org> References: <4ECE3D70.3050701@crans.org> Message-ID: On 24 November 2011 14:49, Pierre Haessig wrote: > I found a small glitch in the signal.freqz docstring the other day, so I > created an account (username: pierreh) > Could I get edit rights ? bump Seems this was missed by the folks with appropriate rights to add Pierre.. Cheers, Scott From pav at iki.fi Thu Dec 1 06:34:29 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 01 Dec 2011 12:34:29 +0100 Subject: [SciPy-Dev] scipy docs edit rights In-Reply-To: References: <4ECE3D70.3050701@crans.org> Message-ID: 01.12.2011 11:59, Scott Sinclair kirjoitti: > On 24 November 2011 14:49, Pierre Haessig wrote: >> I found a small glitch in the signal.freqz docstring the other day, so I >> created an account (username: pierreh) >> Could I get edit rights ? > > bump > > Seems this was missed by the folks with appropriate rights to add Pierre.. Added now. From pierre.haessig at crans.org Thu Dec 1 06:42:51 2011 From: pierre.haessig at crans.org (Pierre Haessig) Date: Thu, 01 Dec 2011 12:42:51 +0100 Subject: [SciPy-Dev] scipy docs edit rights In-Reply-To: References: <4ECE3D70.3050701@crans.org> Message-ID: <4ED7683B.4040602@crans.org> Le 01/12/2011 12:34, Pauli Virtanen a ?crit : >> bump >> > >> > Seems this was missed by the folks with appropriate rights to add Pierre.. > Added now. > Thanks ! -- Pierre From stefan at sun.ac.za Sat Dec 3 17:43:35 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 3 Dec 2011 14:43:35 -0800 Subject: [SciPy-Dev] scikits-image 0.4 release Message-ID: Announcement: scikits-image 0.4 =============================== We're happy to announce the 0.4 release of scikits-image, an image processing toolbox for SciPy. Please visit our examples gallery to see what we've been up to: http://scikits-image.org/docs/0.3/auto_examples/ Note that, in this release, we renamed the module from ``scikits.image`` to ``skimage``, to work around name space conflicts with other scikits (similarly, the machine learning scikit is now imported as ``sklearn``). A big shout-out also to everyone currently at SciPy India; have fun, and remember to join the scikits-image sprint! This release runs under all major operating systems where Python (>=2.6 or 3.x), NumPy and SciPy can be installed. For more information, visit our website http://scikits-image.org New Features ------------ - Module rename from ``scikits.image`` to ``skimage`` - Contour finding - Grey-level co-occurrence matrices - Skeletonization and medial axis transform - Convex hull images - New test data sets - GDAL I/O plugin ... as well as some bug fixes. Contributors to this release ---------------------------- * Andreas Mueller * Christopher Gohlke * Emmanuelle Gouillart * Neil Yager * Nelle Varoquaux * Riaan van den Dool * Stefan van der Walt * Thouis (Ray) Jones * Tony S Yu * Zachary Pincus From cimrman3 at ntc.zcu.cz Mon Dec 5 06:08:24 2011 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon, 05 Dec 2011 12:08:24 +0100 Subject: [SciPy-Dev] ANN: SfePy 2011.4 Message-ID: <4EDCA628.1030407@ntc.zcu.cz> I am pleased to announce release 2011.4 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method. The code is based on NumPy and SciPy packages. It is distributed under the new BSD license. Home page: http://sfepy.org Mailing lists, issue tracking: http://code.google.com/p/sfepy/ Git (source) repository: http://github.com/sfepy Documentation: http://docs.sfepy.org/doc Highlights of this release -------------------------- - cython used instead of swig to interface C code - many terms unified thanks to new optional material term argument type - updated Lagrangian formulation for large deformations - automatic generation of gallery of examples For more information on this release, see http://sfepy.googlecode.com/svn/web/releases/2011.4_RELEASE_NOTES.txt (full release notes, rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Vladim?r Luke?, Maty?? Nov?k From loluengo at gmail.com Mon Dec 5 10:16:32 2011 From: loluengo at gmail.com (Lorenzo Luengo) Date: Mon, 05 Dec 2011 12:16:32 -0300 Subject: [SciPy-Dev] scipy.io.matlab enhancement patch Message-ID: <4EDCE050.2060400@gmail.com> Last week I made a modification to scipy.io.matlab package, a slight one, but i think it's useful. I added a "whosmat" function, the only thing it does is listing variables saved inside a MAT file, without having to load any data, thus saving memory from loading a huge MAT file. Attached is a diff, i used ubuntu's version to make the enhancement, but i think it's trivial to port it to the development branch. Best regards. -------------- next part -------------- A non-text attachment was scrubbed... Name: scipy.diff Type: text/x-patch Size: 4966 bytes Desc: not available URL: From matt.terry at gmail.com Mon Dec 5 13:37:22 2011 From: matt.terry at gmail.com (Matt Terry) Date: Mon, 5 Dec 2011 10:37:22 -0800 Subject: [SciPy-Dev] discrete sine transforms In-Reply-To: References: Message-ID: On Wed, Nov 30, 2011 at 1:13 PM, Matt Terry wrote: > On Wed, Nov 30, 2011 at 11:36 AM, Warren Weckesser > wrote: >> >> >> On Wed, Nov 30, 2011 at 1:19 PM, Matt Terry wrote: >>> >>> Hi, >>> I noticed that scipy.fftpack has discrete cosine transforms, but not >>> sine transforms. ?It also looks like the dst's are in the fftpack >>> source, just not the scipy wrappers. ?Is there a special reason for >>> not wrapping the dst's, or do they just lack a champion? >> >> >> >> There's a ticket for that:? http://projects.scipy.org/scipy/ticket/1432 >> >> I don't see any reason for not making dst available.? I suspect it just >> hasn't been a top priority for anybody.? A pull request would be great, if >> anyone wants to implement this. >> >> Warren > > Sounds good. ?I should have something working and tested this weekend. > > -matt I have the wrappers written and mostly tested. I have a question for those that use fft's more than I. Often when people refer to "the DCT", they mean the type-2 DCT. Is there any such convention for the DST? My inclination is to make the default DST type to be the one with boundary conditions most like default DCT (type-2 DST). However, Matlab only implements the type-1 and for my particular use, the type-1 is most useful. Any thoughts on what the default type of scipy.fftpack.dst should be? -matt From ralf.gommers at googlemail.com Mon Dec 5 13:40:29 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 5 Dec 2011 19:40:29 +0100 Subject: [SciPy-Dev] discrete sine transforms In-Reply-To: References: Message-ID: On Mon, Dec 5, 2011 at 7:37 PM, Matt Terry wrote: > On Wed, Nov 30, 2011 at 1:13 PM, Matt Terry wrote: > > On Wed, Nov 30, 2011 at 11:36 AM, Warren Weckesser > > wrote: > >> > >> > >> On Wed, Nov 30, 2011 at 1:19 PM, Matt Terry > wrote: > >>> > >>> Hi, > >>> I noticed that scipy.fftpack has discrete cosine transforms, but not > >>> sine transforms. It also looks like the dst's are in the fftpack > >>> source, just not the scipy wrappers. Is there a special reason for > >>> not wrapping the dst's, or do they just lack a champion? > >> > >> > >> > >> There's a ticket for that: http://projects.scipy.org/scipy/ticket/1432 > >> > >> I don't see any reason for not making dst available. I suspect it just > >> hasn't been a top priority for anybody. A pull request would be great, > if > >> anyone wants to implement this. > >> > >> Warren > > > > Sounds good. I should have something working and tested this weekend. > > > > -matt > > I have the wrappers written and mostly tested. I have a question for > those that use fft's more than I. > > Often when people refer to "the DCT", they mean the type-2 DCT. Is > there any such convention for the DST? My inclination is to make the > default DST type to be the one with boundary conditions most like > default DCT (type-2 DST). However, Matlab only implements the type-1 > and for my particular use, the type-1 is most useful. Any thoughts on > what the default type of scipy.fftpack.dst should be? > > For consistency I'd think it should match the dct implementation: http://docs.scipy.org/doc/scipy/reference/generated/scipy.fftpack.dct.html. So type 2 as default. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From b.telenczuk at biologie.hu-berlin.de Mon Dec 5 13:52:33 2011 From: b.telenczuk at biologie.hu-berlin.de (Bartosz Telenczuk) Date: Mon, 5 Dec 2011 19:52:33 +0100 Subject: [SciPy-Dev] memory leak fftpack? (fixed) Message-ID: <953F3CCD-5E31-4ACD-9DA9-DDC00AE9785F@biologie.hu-berlin.de> Recently I have noticed that fftpack eats lots of memory when fft is used repeatedly on many signals of various lengths (other than powers of 2). In fact, it looks as if there was a memory leak somewhere in the library (see the example code which reproduces the problem). I guess this is a very common issue for people who do a lot of batch analysis. After many hours of debugging I found out that the problem is that fftpack caches the sine and cosine coefficients for future use. Although the approach can make the code much faster, sometimes the cache can fill up the memory very quickly. The caches can be cleaned up using _fftpack.destroy_*_cache functions, but they are not well documented. I guess the cache issue could be better documented and the functions to clean the cache exposed to the user. I might be also a good idea to provide a option to turn the cache off. It might save someone many hours of debugging. Cheers, Bartosz Code example: from scipy import fftpack import numpy as np import memprofile import os import subprocess def memusg(): pid = os.getpid() proc = subprocess.Popen('ps -p %d -o "rss="' % pid , shell=True ,stdout=subprocess.PIPE) return int(proc.stdout.read()) Nmax= 2116000 print " constant length" Nmin = Nmax for i in range(20): N = int(np.random.rand()*(Nmax-Nmin)/1000.)*1000+Nmin data = np.random.rand(N) data = fftpack.fft(data) print "Memory used: ",memusg(), "bytes" print "variable length" Nmin = 1400000 for i in range(20): N = int(np.random.rand()*(Nmax-Nmin)/1000.)*1000+Nmin data = np.random.rand(N) data = fftpack.fft(data) print "Memory used: ", memusg(), "bytes" The second loop takes 5 times more memory, although the maximum signal length is the same in both cases. Tested on scipy 0.9.0 on MacOSX 10.6.8. From matt.terry at gmail.com Mon Dec 5 14:08:20 2011 From: matt.terry at gmail.com (Matt Terry) Date: Mon, 5 Dec 2011 11:08:20 -0800 Subject: [SciPy-Dev] memory leak fftpack? (fixed) In-Reply-To: <953F3CCD-5E31-4ACD-9DA9-DDC00AE9785F@biologie.hu-berlin.de> References: <953F3CCD-5E31-4ACD-9DA9-DDC00AE9785F@biologie.hu-berlin.de> Message-ID: On Mon, Dec 5, 2011 at 10:52 AM, Bartosz Telenczuk wrote: > Recently I have noticed that fftpack eats lots of memory when fft is used repeatedly on many signals of various lengths (other than powers of 2). ?In fact, it looks as if there was a memory leak somewhere in the library (see the example code which reproduces the problem). I guess this is a very common issue for people who do a lot of batch analysis. > > After many hours of debugging I found out that the problem is that fftpack caches the sine and cosine coefficients for future use. Although the approach can make the code much faster, sometimes the cache can fill up the memory very quickly. The caches can be cleaned up using _fftpack.destroy_*_cache functions, but they are not well documented. > > I guess the cache issue could be better documented and the functions to clean the cache exposed to the user. I might be also a good idea to provide a option to turn the cache off. It might save someone many hours of debugging. > > Cheers, > > Bartosz > > Code example: > > from scipy import fftpack > import numpy as np > import memprofile > import os > import subprocess > > def memusg(): > ? pid = ?os.getpid() > ? proc = subprocess.Popen('ps ?-p %d -o "rss="' % pid , shell=True ,stdout=subprocess.PIPE) > ? return int(proc.stdout.read()) > > > Nmax= 2116000 > > print " constant length" > Nmin = Nmax > for i in range(20): > ? N = int(np.random.rand()*(Nmax-Nmin)/1000.)*1000+Nmin > ? data = np.random.rand(N) > ? data = fftpack.fft(data) > ? print "Memory used: ",memusg(), "bytes" > > > print "variable length" > Nmin = 1400000 > for i in range(20): > ? N = int(np.random.rand()*(Nmax-Nmin)/1000.)*1000+Nmin > ? data = np.random.rand(N) > ? data = fftpack.fft(data) > ? print "Memory used: ", memusg(), "bytes" > > The second loop takes 5 times more memory, although the maximum signal length is the same in both cases. Tested on scipy 0.9.0 on MacOSX 10.6.8. > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev http://docs.scipy.org/doc/scipy/reference/fftpack.html shows the destroy_*fft*_cache() functions (though lacking any documentation). Would it be sufficient to update the docs to mention that the fftpack functions keep a cache and point you to the appropriate destroy function to nuke it? -matt From b.telenczuk at biologie.hu-berlin.de Mon Dec 5 14:13:02 2011 From: b.telenczuk at biologie.hu-berlin.de (Bartosz Telenczuk) Date: Mon, 5 Dec 2011 20:13:02 +0100 Subject: [SciPy-Dev] memory leak fftpack? (fixed) In-Reply-To: References: <953F3CCD-5E31-4ACD-9DA9-DDC00AE9785F@biologie.hu-berlin.de> Message-ID: <64158C0D-0160-47C5-B5F1-06EC4FAF6F50@biologie.hu-berlin.de> Hi, > http://docs.scipy.org/doc/scipy/reference/fftpack.html shows the > destroy_*fft*_cache() functions (though lacking any documentation). > Would it be sufficient to update the docs to mention that the fftpack > functions keep a cache and point you to the appropriate destroy > function to nuke it? Yes, that would be very helpful, indeed. Perhaps, it might be a good idea to add a single function which cleans up all caches as well. It is worth alse mentioning that functions like signal.resample depend on fftpack and they suffer from the same problem. Bartek From matt.terry at gmail.com Mon Dec 5 23:55:28 2011 From: matt.terry at gmail.com (Matt Terry) Date: Mon, 5 Dec 2011 20:55:28 -0800 Subject: [SciPy-Dev] discrete sine transforms In-Reply-To: References: Message-ID: >> >>> Hi, >> >>> I noticed that scipy.fftpack has discrete cosine transforms, but not >> >>> sine transforms. ?It also looks like the dst's are in the fftpack >> >>> source, just not the scipy wrappers. ?Is there a special reason for >> >>> not wrapping the dst's, or do they just lack a champion? >> >> >> >> >> >> >> >> There's a ticket for that:? http://projects.scipy.org/scipy/ticket/1432 >> >> >> >> I don't see any reason for not making dst available.? I suspect it just >> >> hasn't been a top priority for anybody.? A pull request would be great, >> >> if >> >> anyone wants to implement this. >> >> >> >> Warren >> > >> > Sounds good. ?I should have something working and tested this weekend. >> > >> > -matt >> >> I have the wrappers written and mostly tested. ?I have a question for >> those that use fft's more than I. >> >> Often when people refer to "the DCT", they mean the type-2 DCT. ?Is >> there any such convention for the DST? ?My inclination is to make the >> default DST type to be the one with boundary conditions most like >> default DCT (type-2 DST). ?However, Matlab only implements the type-1 >> and for my particular use, the type-1 is most useful. ?Any thoughts on >> what the default type of scipy.fftpack.dst should be? >> > For consistency I'd think it should match the dct implementation: > http://docs.scipy.org/doc/scipy/reference/generated/scipy.fftpack.dct.html. > So type 2 as default. > > Ralf That is my leaning also. For what it is worth, Matlab's dst uses type-1, but Mathematica's FourierDST defaults to type-2. FFTW is silent on the matter. -matt From matthew.brett at gmail.com Tue Dec 6 01:02:53 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 5 Dec 2011 22:02:53 -0800 Subject: [SciPy-Dev] scipy.io.matlab enhancement patch In-Reply-To: <4EDCE050.2060400@gmail.com> References: <4EDCE050.2060400@gmail.com> Message-ID: Hi, On Mon, Dec 5, 2011 at 7:16 AM, Lorenzo Luengo wrote: > Last week I made a modification to scipy.io.matlab package, a slight one, > but i think it's useful. > > I added a "whosmat" function, the only thing it does is listing variables > saved inside a MAT file, without having to load any data, thus saving memory > from loading a huge MAT file. > > Attached is a diff, i used ubuntu's version to make the enhancement, but i > think it's trivial to port it to the development branch. Thanks - that does look useful. I wonder whether it would be worth doing what matlab 'who' does, and return more information about the contained variables? Would you consider making up a pull request for this? http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html Thanks, Matthew From illoul_lounes at yahoo.fr Tue Dec 6 08:37:42 2011 From: illoul_lounes at yahoo.fr (illoul lounes) Date: Tue, 6 Dec 2011 13:37:42 +0000 (GMT) Subject: [SciPy-Dev] cnemlib scikit In-Reply-To: <1323178181.36915.YahooMailNeo@web28614.mail.ukl.yahoo.com> References: <1323177438.75832.YahooMailNeo@web27803.mail.ukl.yahoo.com> <1323178181.36915.YahooMailNeo@web28614.mail.ukl.yahoo.com> Message-ID: <1323178662.95024.YahooMailNeo@web28601.mail.ukl.yahoo.com> Hi, We have developed a library (CNEMLIB), which propose an implementation of CNEM? in 2d and 3d [1],[2] The CNEM is a generalisation for non convex domain of the Natural Element Method[3]. The CNEM is a FEM like approach. The main advantages of the CNEM comparatively with the FEM, are the following: ????- nodes and quadrature points are identical,? all the unknowns (primal and dual unknowns introduced in Galerkin approaches) are attached to the nodes, ????- due to the usage of Natural Neighbour Shape functions, the CNEM doesn't depend on mesh. The main functionalities of CNEMLIB are: ????i) interpolation of scattered data spread on convex or non convex domains with : ????????- in 2d,? the Natural Neighbour interpolant (Sibson ) ????????- in 3d, the Natural Neighbour interpolant (Sibson or Laplace) or the linear finite element interpolant over the Delaunay tessellation. ????ii) a gradient matrix operator which allows to calculate nodal gradients for scattered data. The approach used is based on the stabilized nodal integration SCNI[4]. ????iii) a general assembling tools to construct assembled matrix associated with a weak formulation (heat problem, mechanic problem, hydrodynamic problem, general purpose problem) as such used with the Finite Element Method (FEM). The functionalities i) and ii) are written in c++ and wrapped for python(c++ extension). They are parallelised for 3d applications (shared memory using tbb). The assembling tools iii) are in pure python. The package is license under LGPL. For the 3d applications, we use the tetgen library(c++) writen by Hang Si under a MIT licence[5]. You can download the package (source files, setup file for py extension build, tutorials)? and the release version for different platforms here : http://plateformesn-m2p.ensam.eu/SphinxDoc/cnem/index.html If you think that the library can be useful for Scipy community, we should be very happy to add it in the scikit packages list ! Best regards, Lounes Illoul & Philippe Lorong. [1] L.Illoul,P.Lorong, On some aspects of the CNEM implementation in 3D in order to simulate high speed machining or shearing, Computers and Structures , Volume 89 Issue 11-12, June, 2011 [2] F.Chinesta, Natural Element Method for the Simulation of Structures and Processes, Wiley-ISTE; 1 edition (March 29, 2011) [3] N. Sukumar,? http://dilbert.engr.ucdavis.edu/~suku/nem/ [4] J.Chen, http://www.seas.ucla.edu/~jschen/2004%20SCNI%20for%20NEM.pdf [5] H.Si, http://tetgen.berlios.de/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From loluengo at gmail.com Tue Dec 6 08:57:05 2011 From: loluengo at gmail.com (Lorenzo Luengo) Date: Tue, 06 Dec 2011 10:57:05 -0300 Subject: [SciPy-Dev] scipy.io.matlab enhancement patch In-Reply-To: References: <4EDCE050.2060400@gmail.com> Message-ID: <4EDE1F31.8070607@gmail.com> I'm new to software development, I'll need to read about git. I was planning to extend this function to give more information, like shape, disk size, data type. Then may be I'll make the pull request. Thanks for pointing the route! On 06/12/11 03:02, Matthew Brett wrote: > Hi, > > On Mon, Dec 5, 2011 at 7:16 AM, Lorenzo Luengo wrote: >> Last week I made a modification to scipy.io.matlab package, a slight one, >> but i think it's useful. >> >> I added a "whosmat" function, the only thing it does is listing variables >> saved inside a MAT file, without having to load any data, thus saving memory >> from loading a huge MAT file. >> >> Attached is a diff, i used ubuntu's version to make the enhancement, but i >> think it's trivial to port it to the development branch. > Thanks - that does look useful. I wonder whether it would be worth > doing what matlab 'who' does, and return more information about the > contained variables? > > Would you consider making up a pull request for this? > > http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html > > Thanks, > > Matthew > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -- Lorenzo Luengo C. Ingeniero Civil Electr?nico Cel: 98270385 From stefan at sun.ac.za Thu Dec 8 19:09:56 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 8 Dec 2011 16:09:56 -0800 Subject: [SciPy-Dev] scikits-image 0.4 release In-Reply-To: References: Message-ID: 2011/12/3 St?fan van der Walt : > Announcement: scikits-image 0.4 > =============================== > > We're happy to announce the 0.4 release of scikits-image, an image processing > toolbox for SciPy. > > Please visit our examples gallery to see what we've been up to: > > ? http://scikits-image.org/docs/0.3/auto_examples/ Thanks to Michael Aye for pointing out the typo here. That should be http://scikits-image.org/docs/0.4/auto_examples/ (more examples!) Cheers St?fan From warren.weckesser at enthought.com Sat Dec 10 21:54:13 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sat, 10 Dec 2011 19:54:13 -0700 Subject: [SciPy-Dev] Wiki spammed, needs a revert Message-ID: The "Developer Zone" wiki page (http://www.scipy.org/Developer_Zone) has been spammed. Is there a simple way to revert the changes? Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Sun Dec 11 02:32:22 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 11 Dec 2011 08:32:22 +0100 Subject: [SciPy-Dev] Wiki spammed, needs a revert In-Reply-To: References: Message-ID: <20111211073222.GA4775@phare.normalesup.org> On Sat, Dec 10, 2011 at 07:54:13PM -0700, Warren Weckesser wrote: > The "Developer Zone" wiki page ([1]http://www.scipy.org/Developer_Zone) > has been spammed.? Is there a simple way to revert the changes? Click on 'Info', and in the corresponding page it is easy to find the revert button. I have done it. Gael From warren.weckesser at enthought.com Sun Dec 11 02:43:17 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sun, 11 Dec 2011 00:43:17 -0700 Subject: [SciPy-Dev] Wiki spammed, needs a revert In-Reply-To: <20111211073222.GA4775@phare.normalesup.org> References: <20111211073222.GA4775@phare.normalesup.org> Message-ID: On Sun, Dec 11, 2011 at 12:32 AM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Sat, Dec 10, 2011 at 07:54:13PM -0700, Warren Weckesser wrote: > > The "Developer Zone" wiki page ([1] > http://www.scipy.org/Developer_Zone) > > has been spammed. Is there a simple way to revert the changes? > > Click on 'Info', and in the corresponding page it is easy to find the > revert button. Easy for you, maybe. :) I don't see a revert button. In the menu on the left, in the pull-down "More actions", I see "Remove Spam", but I don't have the right permissions to use that. > I have done it. > Thanks! Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Sun Dec 11 03:25:44 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 11 Dec 2011 09:25:44 +0100 Subject: [SciPy-Dev] Wiki spammed, needs a revert In-Reply-To: References: <20111211073222.GA4775@phare.normalesup.org> Message-ID: <20111211082544.GA16146@phare.normalesup.org> On Sun, Dec 11, 2011 at 12:43:17AM -0700, Warren Weckesser wrote: > Click on 'Info', and in the corresponding page it is easy to find the > revert button. > Easy for you, maybe. :)? Sorry, I shouldn't have used this word. > I don't see a revert button.? In the menu on the left, in the > pull-down "More actions", I see "Remove Spam", but I don't have the > right permissions to use that. You need to click on 'Info', above that menu. G From ralf.gommers at googlemail.com Sun Dec 11 11:43:29 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 11 Dec 2011 17:43:29 +0100 Subject: [SciPy-Dev] memory leak fftpack? (fixed) In-Reply-To: <64158C0D-0160-47C5-B5F1-06EC4FAF6F50@biologie.hu-berlin.de> References: <953F3CCD-5E31-4ACD-9DA9-DDC00AE9785F@biologie.hu-berlin.de> <64158C0D-0160-47C5-B5F1-06EC4FAF6F50@biologie.hu-berlin.de> Message-ID: On Mon, Dec 5, 2011 at 8:13 PM, Bartosz Telenczuk < b.telenczuk at biologie.hu-berlin.de> wrote: > Hi, > > > http://docs.scipy.org/doc/scipy/reference/fftpack.html shows the > > destroy_*fft*_cache() functions (though lacking any documentation). > > Would it be sufficient to update the docs to mention that the fftpack > > functions keep a cache and point you to the appropriate destroy > > function to nuke it? > Matt, were you planning to do this while working on discrete sine transforms? If not, we should open a ticket for this. Ralf > > Yes, that would be very helpful, indeed. Perhaps, it might be a good idea > to add a single function which cleans up all caches as well. > > It is worth alse mentioning that functions like signal.resample depend on > fftpack and they suffer from the same problem. > > Bartek > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun Dec 11 11:48:58 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 11 Dec 2011 17:48:58 +0100 Subject: [SciPy-Dev] cnemlib scikit In-Reply-To: <1323178662.95024.YahooMailNeo@web28601.mail.ukl.yahoo.com> References: <1323177438.75832.YahooMailNeo@web27803.mail.ukl.yahoo.com> <1323178181.36915.YahooMailNeo@web28614.mail.ukl.yahoo.com> <1323178662.95024.YahooMailNeo@web28601.mail.ukl.yahoo.com> Message-ID: On Tue, Dec 6, 2011 at 2:37 PM, illoul lounes wrote: > > Hi, > > We have developed a library (CNEMLIB), which propose an implementation of > CNEM in 2d and 3d [1],[2] > The CNEM is a generalisation for non convex domain of the Natural Element > Method[3]. > > The CNEM is a FEM like approach. The main advantages of the CNEM > comparatively with the FEM, are the following: > - nodes and quadrature points are identical, all the unknowns > (primal and dual unknowns introduced in Galerkin approaches) are attached > to the nodes, > - due to the usage of Natural Neighbour Shape functions, the CNEM > doesn't depend on mesh. > > The main functionalities of CNEMLIB are: > i) interpolation of scattered data spread on convex or non convex > domains with : > - in 2d, the Natural Neighbour interpolant (Sibson ) > - in 3d, the Natural Neighbour interpolant (Sibson or Laplace) or > the linear finite element interpolant over the Delaunay tessellation. > > ii) a gradient matrix operator which allows to calculate nodal > gradients for scattered data. The approach used is based on the stabilized > nodal integration SCNI[4]. > iii) a general assembling tools to construct assembled matrix > associated with a weak formulation (heat problem, mechanic problem, > hydrodynamic problem, general purpose problem) as such used with the Finite > Element Method (FEM). > > The functionalities i) and ii) are written in c++ and wrapped for > python(c++ extension). They are parallelised for 3d applications (shared > memory using tbb). The assembling tools iii) are in pure python. > > The package is license under LGPL. For the 3d applications, we use the > tetgen library(c++) writen by Hang Si under a MIT licence[5]. > > You can download the package (source files, setup file for py extension > build, tutorials) and the release version for different platforms here : > http://plateformesn-m2p.ensam.eu/SphinxDoc/cnem/index.html > > If you think that the library can be useful for Scipy community, we should > be very happy to add it in the scikit packages list ! > > This looks pretty good. Please add it to this list: http://scipy.org/Topical_Software If you'd like to create a scikit from this, I don't see a problem with that. Either way I'm sure it'll be useful for some people in the SciPy community. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From trobinson at systemsbiology.org Mon Dec 12 15:17:59 2011 From: trobinson at systemsbiology.org (Thomas Robinson) Date: Mon, 12 Dec 2011 12:17:59 -0800 Subject: [SciPy-Dev] Proposed improvement: recursive integrator for arbitrary dimensionality using QUADPACK Message-ID: <4EE66177.3090302@systemsbiology.org> That's quite a mouthful. In simpler terms: I've been playing with scipy/integrate/quadpack.py lately, specifically using dblquad and tplquad to meet my needs for continuous mutual information (https://en.wikipedia.org/wiki/Mutual_information). After implementing this, I refactored my algorithm to perform a deterministic walk of multivariate mutual information (https://en.wikipedia.org/wiki/Interaction_information) for cases where the shared information increased. And so entered my dilemma: what I needed was a general-purpose, canned integrator that could handle an arbitrary number of terms during multiple integration using fundamentally the same algorithm (to minimize work). Here's what I came up with: def _infuncn(x,func,a,b,count,epsabs,epsrel,more_args): myargs = (x,) + more_args if count == 1: return quad(func,a,b,args=myargs,epsabs=epsabs,epsrel=epsrel)[0] else: return quad(_infuncn,a,b,(func,a,b,count-1,epsabs,epsrel,myargs),epsabs=epsabs,epsrel=epsrel) def nquad(func, a, b, args=(), epsabs=1.49e-8, epsrel=1.49e-8, count=2): ''' Recursive, N-depth integrator with number of variables given ''' # Preconditions if not isinstance( count, int ): raise TypeError('Variable "count" must be an integer value') if count < 2: raise ValueError('Variable "count" must be at least 2 to perform \ meaningful multivariate integration. Consider using quad(...) instead.') return quad(_infuncn,a,b,(func,a,b,count-1,epsabs,epsrel,args),epsabs=epsabs,epsrel=epsrel) Basically, what you get is this construct that recursively calls into QUADPACK with a series of simplifying assumptions (ie, fixed domain, equivalent epsilon values propagated down to all invocations of quad) and allowable calls from an arbitrarily-shaped function pointer. Invoking this function for any number of terms is as easy as abusing the lambda function and argument pointers in Python, like so: nquad(lambda *args: my_nspace_function(args), min_bound, max_bound, count=number_of_args) For example: # Define INF as float('inf') points = [[1,2,3],[1,1,2],[2,2,6]] gkde_n = gaussian_kde(points) # ... compute Shannon entropy, store as variable "entropy". mutual_info = lambda *args: gkde_n(args) * \ math.log((gkde_n(args) / (entropy_mult + \ MIN_DOUBLE)) + MIN_DOUBLE) # Compute MI(X1..Xn) (minfo_n, err_n) = nquad(mutual_info, -INF, INF, count=scale) I bring this up here for two reasons. The first is I'd find this to be greatly beneficial to the current Python interface into QUADPACK, insofar as it solves the general case instead of forcing programmers to rely on (for example) invocations directly into quad, dblquad, or tplquad. The second is my surprise that a function to compute the discrete and continuous Shannon-Weaver mutual information does not appear to be extant in SciPy or NumPy, which I found greatly distressing given their usefulness in discovering a select set of exceptionally useful properties about paired data sets that are not well expressed in monotonically-dependent tests for independence like Pearson's or Spearman's correlations. Hence my purpose bringing these issues up for general review. I am absolutely convinced that my prototypical code above may be greatly improved, and I would wholeheartedly support it insofar as I've been spinning my wheels over the problem of a robust, fast implementation. What I have is an implementation that reliably offers a good approximation, complains bitterly when integration fails to rapidly converge, and which makes relatively naive assumptions that could be greatly improved upon by developers more actively involved in the code than I am. Thanks for reading. I do hope someone out there finds this post valuable. - Tom Robinson From matt.terry at gmail.com Tue Dec 13 01:42:31 2011 From: matt.terry at gmail.com (Matt Terry) Date: Mon, 12 Dec 2011 22:42:31 -0800 Subject: [SciPy-Dev] memory leak fftpack? (fixed) In-Reply-To: References: <953F3CCD-5E31-4ACD-9DA9-DDC00AE9785F@biologie.hu-berlin.de> <64158C0D-0160-47C5-B5F1-06EC4FAF6F50@biologie.hu-berlin.de> Message-ID: i'm working on it. i have the dst/idst bindings working, and written the documentation (including documenting the destroy_cache functions). i'm finishing up a test suite comparable to what dct has. work is bearing down on me right now, but hopefully have a complete pull request in a week or so? i wasn't going to make a destroy_all_cache() function. i figure if you know enough to want to kill the cache, you can probably read the docs and see which cache you want killed. -matt On Sun, Dec 11, 2011 at 8:43 AM, Ralf Gommers wrote: > > > On Mon, Dec 5, 2011 at 8:13 PM, Bartosz Telenczuk > wrote: >> >> Hi, >> >> > http://docs.scipy.org/doc/scipy/reference/fftpack.html shows the >> > destroy_*fft*_cache() functions (though lacking any documentation). >> > Would it be sufficient to update the docs to mention that the fftpack >> > functions keep a cache and point you to the appropriate destroy >> > function to nuke it? > > > Matt, were you planning to do this while working on discrete sine > transforms? If not, we should open a ticket for this. > > Ralf > > >> >> >> Yes, that would be very helpful, indeed. Perhaps, it might be a good idea >> to add a single function which cleans up all caches as well. >> >> It is worth alse mentioning that functions like signal.resample depend on >> fftpack and they suffer from the same problem. >> >> Bartek >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From ralf.gommers at googlemail.com Tue Dec 13 01:56:53 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 13 Dec 2011 07:56:53 +0100 Subject: [SciPy-Dev] memory leak fftpack? (fixed) In-Reply-To: References: <953F3CCD-5E31-4ACD-9DA9-DDC00AE9785F@biologie.hu-berlin.de> <64158C0D-0160-47C5-B5F1-06EC4FAF6F50@biologie.hu-berlin.de> Message-ID: On Tue, Dec 13, 2011 at 7:42 AM, Matt Terry wrote: > i'm working on it. i have the dst/idst bindings working, and written > the documentation (including documenting the destroy_cache functions). > i'm finishing up a test suite comparable to what dct has. work is > bearing down on me right now, but hopefully have a complete pull > request in a week or so? > > Great. > i wasn't going to make a destroy_all_cache() function. i figure if > you know enough to want to kill the cache, you can probably read the > docs and see which cache you want killed. > I agree, documenting it is sufficient for now. Cheers, Ralf > > -matt > > On Sun, Dec 11, 2011 at 8:43 AM, Ralf Gommers > wrote: > > > > > > On Mon, Dec 5, 2011 at 8:13 PM, Bartosz Telenczuk > > wrote: > >> > >> Hi, > >> > >> > http://docs.scipy.org/doc/scipy/reference/fftpack.html shows the > >> > destroy_*fft*_cache() functions (though lacking any documentation). > >> > Would it be sufficient to update the docs to mention that the fftpack > >> > functions keep a cache and point you to the appropriate destroy > >> > function to nuke it? > > > > > > Matt, were you planning to do this while working on discrete sine > > transforms? If not, we should open a ticket for this. > > > > Ralf > > > > > >> > >> > >> Yes, that would be very helpful, indeed. Perhaps, it might be a good > idea > >> to add a single function which cleans up all caches as well. > >> > >> It is worth alse mentioning that functions like signal.resample depend > on > >> fftpack and they suffer from the same problem. > >> > >> Bartek > >> _______________________________________________ > >> SciPy-Dev mailing list > >> SciPy-Dev at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Dec 14 16:48:40 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 14 Dec 2011 22:48:40 +0100 Subject: [SciPy-Dev] [SciPy-User] test_arpack.test_symmetric_modes hangs, scipy 0.10 In-Reply-To: References: Message-ID: On Wed, Dec 14, 2011 at 10:29 PM, James Yoo wrote: > here's my compiler and python/numpy/scipy info... note we're also using > intel mkl (8.1.014) libs. > > Linux 2.6.18-238.3.1.el5 #1 SMP Tue Jan 25 18:05:40 EST 2011 > x86_64 x86_64 x86_64 GNU/Linux > > gcc -v > Using built-in specs. > Target: x86_64-redhat-linux > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man > --infodir=/usr/share/info --enable-shared --enable-threads=posix > --enable-checking=release --with-system-zlib --enable-__cxa_atexit > --disable-libunwind-exceptions --enable-libgcj-multifile > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada > --enable-java-awt=gtk --disable-dssi --disable-plugin > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic > --host=x86_64-redhat-linux > Thread model: posix > gcc version 4.1.2 20080704 (Red Hat 4.1.2-50) > > > python from python.org > numpy 1.6.1 and scipy 0.10 from sourceforge > > Thanks. Would anyone object to marking all single-precision Arpack tests as knownfail until someone with the motivation and skills to get this fixed comes along? Ralf > > > On Wed, Dec 14, 2011 at 3:21 PM, Ralf Gommers > wrote: > >> >> >> On Wed, Dec 14, 2011 at 5:20 PM, James Yoo wrote: >> >>> Hello, >>> >>> Test for scipy 0.10 hang at test_arpack.test_symmetric_modes >>> >>> Hmm, this is really turning into a never-ending story. We were thinking >> all test suite crashes/hangs were fixed at least. Can you provide details >> on your OS, compilers, where you got python, numpy, scipy? >> >> Related issues: >> http://projects.scipy.org/scipy/ticket/1515 >> http://projects.scipy.org/scipy/ticket/1523 >> http://projects.scipy.org/scipy/ticket/1472 >> >> Ralf >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Dec 17 10:55:46 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 17 Dec 2011 16:55:46 +0100 Subject: [SciPy-Dev] Proposed improvement: recursive integrator for arbitrary dimensionality using QUADPACK In-Reply-To: <4EE66177.3090302@systemsbiology.org> References: <4EE66177.3090302@systemsbiology.org> Message-ID: On Mon, Dec 12, 2011 at 9:17 PM, Thomas Robinson < trobinson at systemsbiology.org> wrote: > That's quite a mouthful. In simpler terms: I've been playing with > scipy/integrate/quadpack.py lately, specifically using dblquad and > tplquad to meet my needs for continuous mutual information > (https://en.wikipedia.org/wiki/Mutual_information). > > After implementing this, I refactored my algorithm to perform a > deterministic walk of multivariate mutual information > (https://en.wikipedia.org/wiki/Interaction_information) for cases where > the shared information increased. And so entered my dilemma: what I > needed was a general-purpose, canned integrator that could handle an > arbitrary number of terms during multiple integration using > fundamentally the same algorithm (to minimize work). > > > Here's what I came up with: > > def _infuncn(x,func,a,b,count,epsabs,epsrel,more_args): > myargs = (x,) + more_args > if count == 1: > return quad(func,a,b,args=myargs,epsabs=epsabs,epsrel=epsrel)[0] > else: > return > > quad(_infuncn,a,b,(func,a,b,count-1,epsabs,epsrel,myargs),epsabs=epsabs,epsrel=epsrel) > > def nquad(func, a, b, args=(), epsabs=1.49e-8, epsrel=1.49e-8, count=2): > ''' Recursive, N-depth integrator with number of variables given ''' > > # Preconditions > if not isinstance( count, int ): > raise TypeError('Variable "count" must be an integer value') > if count < 2: > raise ValueError('Variable "count" must be at least 2 to perform \ > meaningful multivariate integration. Consider using > quad(...) instead.') > > return > > quad(_infuncn,a,b,(func,a,b,count-1,epsabs,epsrel,args),epsabs=epsabs,epsrel=epsrel) > > An n-D integration routine could be a useful addition. An obvious problem I see with your code is that the integration limits are the same for all dimensions. This makes it unsuitable for solving the majority of problems I can think of that would need n-D integration. It would be straigtforward to at least allow different limits per integration step, and worth thinking about if it's useful to allow limits such as in dblquad/tplquad. > Basically, what you get is this construct that recursively calls into > QUADPACK with a series of simplifying assumptions (ie, fixed domain, > equivalent epsilon values propagated down to all invocations of quad) > and allowable calls from an arbitrarily-shaped function pointer. > Invoking this function for any number of terms is as easy as abusing the > lambda function and argument pointers in Python, like so: > > nquad(lambda *args: my_nspace_function(args), min_bound, max_bound, > count=number_of_args) > > > For example: > > # Define INF as float('inf') > > points = [[1,2,3],[1,1,2],[2,2,6]] > gkde_n = gaussian_kde(points) > > # ... compute Shannon entropy, store as variable "entropy". > > mutual_info = lambda *args: gkde_n(args) * \ > math.log((gkde_n(args) / (entropy_mult + \ > MIN_DOUBLE)) + MIN_DOUBLE) > > # Compute MI(X1..Xn) > (minfo_n, err_n) = nquad(mutual_info, -INF, INF, count=scale) > > > I bring this up here for two reasons. The first is I'd find this to be > greatly beneficial to the current Python interface into QUADPACK, > insofar as it solves the general case instead of forcing programmers to > rely on (for example) invocations directly into quad, dblquad, or > tplquad. The second is my surprise that a function to compute the > discrete and continuous Shannon-Weaver mutual information does not > appear to be extant in SciPy or NumPy, I had to look up what this is exactly, so I don't find it very surprising. It sounds a bit too specialized for scipy, scikit-learn is the first place I'd go look for it. Ralf > which I found greatly distressing > given their usefulness in discovering a select set of exceptionally > useful properties about paired data sets that are not well expressed in > monotonically-dependent tests for independence like Pearson's or > Spearman's correlations. > > Hence my purpose bringing these issues up for general review. I am > absolutely convinced that my prototypical code above may be greatly > improved, and I would wholeheartedly support it insofar as I've been > spinning my wheels over the problem of a robust, fast implementation. > What I have is an implementation that reliably offers a good > approximation, complains bitterly when integration fails to rapidly > converge, and which makes relatively naive assumptions that could be > greatly improved upon by developers more actively involved in the code > than I am. > > > Thanks for reading. I do hope someone out there finds this post valuable. > > - Tom Robinson > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Dec 17 11:55:54 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 17 Dec 2011 11:55:54 -0500 Subject: [SciPy-Dev] Proposed improvement: recursive integrator for arbitrary dimensionality using QUADPACK In-Reply-To: References: <4EE66177.3090302@systemsbiology.org> Message-ID: On Sat, Dec 17, 2011 at 10:55 AM, Ralf Gommers wrote: > > > On Mon, Dec 12, 2011 at 9:17 PM, Thomas Robinson > wrote: >> >> That's quite a mouthful. In simpler terms: I've been playing with >> scipy/integrate/quadpack.py lately, specifically using dblquad and >> tplquad to meet my needs for continuous mutual information >> (https://en.wikipedia.org/wiki/Mutual_information). >> >> After implementing this, I refactored my algorithm to perform a >> deterministic walk of multivariate mutual information >> (https://en.wikipedia.org/wiki/Interaction_information) for cases where >> the shared information increased. And so entered my dilemma: what I >> needed was a general-purpose, canned integrator that could handle an >> arbitrary number of terms during multiple integration using >> fundamentally the same algorithm (to minimize work). >> >> >> Here's what I came up with: >> >> def _infuncn(x,func,a,b,count,epsabs,epsrel,more_args): >> ? ? myargs = (x,) + more_args >> ? ? if count == 1: >> ? ? ? ? return quad(func,a,b,args=myargs,epsabs=epsabs,epsrel=epsrel)[0] >> ? ? else: >> ? ? ? ? return >> >> quad(_infuncn,a,b,(func,a,b,count-1,epsabs,epsrel,myargs),epsabs=epsabs,epsrel=epsrel) >> >> def nquad(func, a, b, args=(), epsabs=1.49e-8, epsrel=1.49e-8, count=2): >> ? ? ''' Recursive, N-depth integrator with number of variables given ''' >> >> ? ? # Preconditions >> ? ? if not isinstance( count, int ): >> ? ? ? ? raise TypeError('Variable "count" must be an integer value') >> ? ? if count < 2: >> ? ? ? ? raise ValueError('Variable "count" must be at least 2 to perform \ >> ? ? ? ? ? ? meaningful multivariate integration. Consider using >> quad(...) instead.') >> >> ? ? return >> >> quad(_infuncn,a,b,(func,a,b,count-1,epsabs,epsrel,args),epsabs=epsabs,epsrel=epsrel) >> > An n-D integration routine could be a useful addition. An obvious problem I > see with your code is that the integration limits are the same for all > dimensions. This makes it unsuitable for solving the majority of problems I > can think of that would need n-D integration. It would be straigtforward to > at least allow different limits per integration step, and worth thinking > about if it's useful to allow limits such as in dblquad/tplquad. > >> >> Basically, what you get is this construct that recursively calls into >> QUADPACK with a series of simplifying assumptions (ie, fixed domain, >> equivalent epsilon values propagated down to all invocations of quad) >> and allowable calls from an arbitrarily-shaped function pointer. >> Invoking this function for any number of terms is as easy as abusing the >> lambda function and argument pointers in Python, like so: >> >> nquad(lambda *args: my_nspace_function(args), min_bound, max_bound, >> count=number_of_args) >> >> >> For example: >> >> # Define INF as float('inf') >> >> points = [[1,2,3],[1,1,2],[2,2,6]] >> gkde_n = gaussian_kde(points) >> >> # ... compute Shannon entropy, store as variable "entropy". >> >> mutual_info = lambda *args: gkde_n(args) * \ >> ? ? ? ? ? ? ? ? ? ?math.log((gkde_n(args) / (entropy_mult + \ >> MIN_DOUBLE)) + MIN_DOUBLE) >> >> # Compute MI(X1..Xn) >> (minfo_n, err_n) = nquad(mutual_info, -INF, INF, count=scale) >> >> >> I bring this up here for two reasons. The first is I'd find this to be >> greatly beneficial to the current Python interface into QUADPACK, >> insofar as it solves the general case instead of forcing programmers to >> rely on (for example) invocations directly into quad, dblquad, or >> tplquad. The second is my surprise that a function to compute the >> discrete and continuous Shannon-Weaver mutual information does not >> appear to be extant in SciPy or NumPy, > > > I had to look up what this is exactly, so I don't find it very surprising. > It sounds a bit too specialized for scipy, scikit-learn is the first place > I'd go look for it. mutual information and other information criteria are getting pretty popular, for example http://stackoverflow.com/questions/8363085/continuous-mutual-information-in-python , scikit-learn uses it for cluster algorithms (AFAIK), statsmodels has some in the sandbox. It would be very useful to have good *tested* code somewhere available for it. However, my impression is that there are many application specific versions, and nobody seems to be much interested in maintaining it in scipy, for example, scipy.maxentropy didn't find a new maintainer, and nobody seems to complain that entropy for the stats.distributions have "quite a few" bugs. My impression is also that many developers (including myself) have fun writing code, but often don't go through the extra effort to make sure the code is actually *correct* and tested. Until the code is tested it might remain a very useful cookbook recipe. "Maintainers needed" Josef after spending a day looking for reference results to verify my code, with a sandbox that is growing faster than I'm able to move code out of it. > > Ralf > > >> >> which I found greatly distressing >> given their usefulness in discovering a select set of exceptionally >> useful properties about paired data sets that are not well expressed in >> monotonically-dependent tests for independence like Pearson's or >> Spearman's correlations. >> >> Hence my purpose bringing these issues up for general review. I am >> absolutely convinced that my prototypical code above may be greatly >> improved, and I would wholeheartedly support it insofar as I've been >> spinning my wheels over the problem of a robust, fast implementation. >> What I have is an implementation that reliably offers a good >> approximation, complains bitterly when integration fails to rapidly >> converge, and which makes relatively naive assumptions that could be >> greatly improved upon by developers more actively involved in the code >> than I am. >> >> >> Thanks for reading. I do hope someone out there finds this post valuable. >> >> - Tom Robinson >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From vanderplas at astro.washington.edu Mon Dec 19 07:25:02 2011 From: vanderplas at astro.washington.edu (Jacob VanderPlas) Date: Mon, 19 Dec 2011 04:25:02 -0800 Subject: [SciPy-Dev] Graph shortest path Message-ID: <4EEF2D1E.6030508@astro.washington.edu> Hello, A while ago in scikit-learn we implemented an efficient cython graph shortest-path search using both Dijkstra's algorithm and the Floyd-Warshall algorithm with Fibonacci heaps. Currently this is well-hidden in the scikit-learn utils: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/graph_shortest_path.pyx This is the kind of basic algorithm that I think belongs in scipy rather than scikit-learn. I know that some developers have been thinking about graph tools for scipy: are there ideas about whether/where the shortest path search would fit in scipy? Thanks Jake From warren.weckesser at enthought.com Mon Dec 19 11:03:29 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Mon, 19 Dec 2011 10:03:29 -0600 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: <4EEF2D1E.6030508@astro.washington.edu> References: <4EEF2D1E.6030508@astro.washington.edu> Message-ID: On Mon, Dec 19, 2011 at 6:25 AM, Jacob VanderPlas < vanderplas at astro.washington.edu> wrote: > Hello, > A while ago in scikit-learn we implemented an efficient cython graph > shortest-path search using both Dijkstra's algorithm and the > Floyd-Warshall algorithm with Fibonacci heaps. Currently this is > well-hidden in the scikit-learn utils: > > https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/graph_shortest_path.pyx > This is the kind of basic algorithm that I think belongs in scipy rather > than scikit-learn. I know that some developers have been thinking about > graph tools for scipy: are there ideas about whether/where the shortest > path search would fit in scipy? > Networkx (http://networkx.lanl.gov/) already provides an assortment of data structures and algorithms for graphs. It is an active, well-documented project. Personally, I'd rather not start adding code to scipy that duplicates it. If your code is better/faster/stronger than theirs, why not contribute it to networkx? Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Mon Dec 19 11:05:18 2011 From: vanderplas at astro.washington.edu (Jacob VanderPlas) Date: Mon, 19 Dec 2011 08:05:18 -0800 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> Message-ID: <4EEF60BE.1010703@astro.washington.edu> As far as I know, networkx uses pure python only. The cython Dijkstra algorithm in scikit-learn is quite a bit faster than anything in networkx Jake Warren Weckesser wrote: > > > On Mon, Dec 19, 2011 at 6:25 AM, Jacob VanderPlas > > wrote: > > Hello, > A while ago in scikit-learn we implemented an efficient cython graph > shortest-path search using both Dijkstra's algorithm and the > Floyd-Warshall algorithm with Fibonacci heaps. Currently this is > well-hidden in the scikit-learn utils: > https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/graph_shortest_path.pyx > This is the kind of basic algorithm that I think belongs in scipy > rather > than scikit-learn. I know that some developers have been thinking > about > graph tools for scipy: are there ideas about whether/where the > shortest > path search would fit in scipy? > > > > Networkx (http://networkx.lanl.gov/) already provides an assortment of > data structures and algorithms for graphs. It is an active, > well-documented project. Personally, I'd rather not start adding code > to scipy that duplicates it. If your code is better/faster/stronger > than theirs, why not contribute it to networkx? > > Warren > > ------------------------------------------------------------------------ > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From zachary.pincus at yale.edu Mon Dec 19 11:18:16 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 19 Dec 2011 11:18:16 -0500 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> Message-ID: <4BB9E7C7-550F-4DAB-986F-D216245649C7@yale.edu> Just for completeness in case someone comes across this googling, there's also a shortest-paths algorithm in scikits-image ('skimage'). That implementation (in cython, also using a heap underneath) is specific for computing shortest paths across a (weighted) lattice (that is, through an numpy array of weights), and so is less general but faster for that specific case. Zach On Dec 19, 2011, at 11:03 AM, Warren Weckesser wrote: > > > On Mon, Dec 19, 2011 at 6:25 AM, Jacob VanderPlas wrote: > Hello, > A while ago in scikit-learn we implemented an efficient cython graph > shortest-path search using both Dijkstra's algorithm and the > Floyd-Warshall algorithm with Fibonacci heaps. Currently this is > well-hidden in the scikit-learn utils: > https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/graph_shortest_path.pyx > This is the kind of basic algorithm that I think belongs in scipy rather > than scikit-learn. I know that some developers have been thinking about > graph tools for scipy: are there ideas about whether/where the shortest > path search would fit in scipy? > > > Networkx (http://networkx.lanl.gov/) already provides an assortment of data structures and algorithms for graphs. It is an active, well-documented project. Personally, I'd rather not start adding code to scipy that duplicates it. If your code is better/faster/stronger than theirs, why not contribute it to networkx? > > Warren > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From warren.weckesser at enthought.com Mon Dec 19 11:22:42 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Mon, 19 Dec 2011 10:22:42 -0600 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: <4EEF60BE.1010703@astro.washington.edu> References: <4EEF2D1E.6030508@astro.washington.edu> <4EEF60BE.1010703@astro.washington.edu> Message-ID: On Mon, Dec 19, 2011 at 10:05 AM, Jacob VanderPlas < vanderplas at astro.washington.edu> wrote: > As far as I know, networkx uses pure python only. The cython Dijkstra > algorithm in scikit-learn is quite a bit faster than anything in networkx > Jake > You're right, it looks like networkx is currrently pure python (well, python + numpy). I'll look into their plans (if any) for cythonizing their code. I see on their mailing list that back in August, Aric Hagberg commented on possibly using the latest version of cython for speeding up parts of networkx. Warren > > Warren Weckesser wrote: > > > > > > On Mon, Dec 19, 2011 at 6:25 AM, Jacob VanderPlas > > > > wrote: > > > > Hello, > > A while ago in scikit-learn we implemented an efficient cython graph > > shortest-path search using both Dijkstra's algorithm and the > > Floyd-Warshall algorithm with Fibonacci heaps. Currently this is > > well-hidden in the scikit-learn utils: > > > https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/graph_shortest_path.pyx > > This is the kind of basic algorithm that I think belongs in scipy > > rather > > than scikit-learn. I know that some developers have been thinking > > about > > graph tools for scipy: are there ideas about whether/where the > > shortest > > path search would fit in scipy? > > > > > > > > Networkx (http://networkx.lanl.gov/) already provides an assortment of > > data structures and algorithms for graphs. It is an active, > > well-documented project. Personally, I'd rather not start adding code > > to scipy that duplicates it. If your code is better/faster/stronger > > than theirs, why not contribute it to networkx? > > > > Warren > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Mon Dec 19 11:53:39 2011 From: vanderplas at astro.washington.edu (Jacob VanderPlas) Date: Mon, 19 Dec 2011 08:53:39 -0800 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> <4EEF60BE.1010703@astro.washington.edu> Message-ID: <4EEF6C13.10401@astro.washington.edu> Interesting - if networkx is going that direction, they could certainly borrow from scikit-learn in this case [ BSD license = :-) ]. In any case, I think the type of problem solved by sklearn.utils.graph_shortest_path is similar to that of scipy.sparse.cs_graph_components, so there is some precedent for this sort of graph functionality in scipy. Jake Warren Weckesser wrote: > > > On Mon, Dec 19, 2011 at 10:05 AM, Jacob VanderPlas > > wrote: > > As far as I know, networkx uses pure python only. The cython Dijkstra > algorithm in scikit-learn is quite a bit faster than anything in > networkx > Jake > > > > You're right, it looks like networkx is currrently pure python (well, > python + numpy). I'll look into their plans (if any) for cythonizing > their code. I see on their mailing list that back in August, Aric > Hagberg commented on possibly using the latest version of cython for > speeding up parts of networkx. > > Warren > > > > > Warren Weckesser wrote: > > > > > > On Mon, Dec 19, 2011 at 6:25 AM, Jacob VanderPlas > > > > >> wrote: > > > > Hello, > > A while ago in scikit-learn we implemented an efficient > cython graph > > shortest-path search using both Dijkstra's algorithm and the > > Floyd-Warshall algorithm with Fibonacci heaps. Currently > this is > > well-hidden in the scikit-learn utils: > > > https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/graph_shortest_path.pyx > > This is the kind of basic algorithm that I think belongs in > scipy > > rather > > than scikit-learn. I know that some developers have been > thinking > > about > > graph tools for scipy: are there ideas about whether/where the > > shortest > > path search would fit in scipy? > > > > > > > > Networkx (http://networkx.lanl.gov/) already provides an > assortment of > > data structures and algorithms for graphs. It is an active, > > well-documented project. Personally, I'd rather not start > adding code > > to scipy that duplicates it. If your code is > better/faster/stronger > > than theirs, why not contribute it to networkx? > > > > Warren > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > ------------------------------------------------------------------------ > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From charlesr.harris at gmail.com Mon Dec 19 14:04:06 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 19 Dec 2011 12:04:06 -0700 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> Message-ID: On Mon, Dec 19, 2011 at 9:03 AM, Warren Weckesser < warren.weckesser at enthought.com> wrote: > > > On Mon, Dec 19, 2011 at 6:25 AM, Jacob VanderPlas < > vanderplas at astro.washington.edu> wrote: > >> Hello, >> A while ago in scikit-learn we implemented an efficient cython graph >> shortest-path search using both Dijkstra's algorithm and the >> Floyd-Warshall algorithm with Fibonacci heaps. Currently this is >> well-hidden in the scikit-learn utils: >> >> https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/graph_shortest_path.pyx >> This is the kind of basic algorithm that I think belongs in scipy rather >> than scikit-learn. I know that some developers have been thinking about >> graph tools for scipy: are there ideas about whether/where the shortest >> path search would fit in scipy? >> > > > Networkx (http://networkx.lanl.gov/) already provides an assortment of > data structures and algorithms for graphs. It is an active, > well-documented project. Personally, I'd rather not start adding code to > scipy that duplicates it. If your code is better/faster/stronger than > theirs, why not contribute it to networkx? > > I think it is useful to have a small collection of fundamental algorithms and data structures in root packages like numpy/scipy that support a large computational infrastructure. The spacial index would be an example of such. Whether or not the proposed algorithm is in that class is something open to discussion, but it is sometimes the case that folks don't know they need something because it isn't easily available or is hidden away in some specialty. There may be other things, say linear assignment, that could be useful in developing algorithms. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Dec 19 16:29:20 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 19 Dec 2011 22:29:20 +0100 Subject: [SciPy-Dev] [SciPy-User] test_arpack.test_symmetric_modes hangs, scipy 0.10 In-Reply-To: References: Message-ID: On Wed, Dec 14, 2011 at 10:48 PM, Ralf Gommers wrote: > > Would anyone object to marking all single-precision Arpack tests as > knownfail until someone with the motivation and skills to get this fixed > comes along? > Anyone? This is clearly very broken, yet another bug report today (11 failures). I would even consider doing 0.10.1 release to clean this up. Ralf >> >> >> On Wed, Dec 14, 2011 at 3:21 PM, Ralf Gommers < >> ralf.gommers at googlemail.com> wrote: >> >>> >>> >>> On Wed, Dec 14, 2011 at 5:20 PM, James Yoo wrote: >>> >>>> Hello, >>>> >>>> Test for scipy 0.10 hang at test_arpack.test_symmetric_modes >>>> >>>> Hmm, this is really turning into a never-ending story. We were thinking >>> all test suite crashes/hangs were fixed at least. Can you provide details >>> on your OS, compilers, where you got python, numpy, scipy? >>> >>> Related issues: >>> http://projects.scipy.org/scipy/ticket/1515 >>> http://projects.scipy.org/scipy/ticket/1523 >>> http://projects.scipy.org/scipy/ticket/1472 >>> >>> Ralf >>> >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Mon Dec 19 19:10:47 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 20 Dec 2011 01:10:47 +0100 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> Message-ID: <20111220001047.GB2294@phare.normalesup.org> On Mon, Dec 19, 2011 at 10:03:29AM -0600, Warren Weckesser wrote: > Networkx ([3]http://networkx.lanl.gov/) already provides an assortment of > data structures and algorithms for graphs.? It is an active, > well-documented project.? Personally, I'd rather not start adding code to > scipy that duplicates it.?? If your code is better/faster/stronger than > theirs, why not contribute it to networkx? I think that there is some room for a lower-level and more efficient base structure from graphs in the scipy world, and that scipy is well suited for that. Simple graph algorithms are common to many applications. For instance a lot of the sparse linalg needs basic graph algorithms. It seems to that having sparse graph algorithms in scipy (and we already have some under the sparse package) would help the scipy world sharing code with regards to graph algorithms. For instance, the scikit-image, the scikit-learn, and pyamg all have shortest path algorithms. NetworkX is not well-suited for purely numerical application. First of all, it is pure-Python, which has a lot of overheads as many of the graph algorithms that cannot be implemented as array or matrix computation. Second, it is using a dict of dicts representation to store and manipulate graphs. The reason that they are using this representation is that it enables to store rich objects for the nodes, and to dynamically allocate and remove nodes. Such choices are great for the complex-systems types of problems that they are targeting. However, for number crunching applications, it is not great, as it has a significant memory overhead, and prevents calling easily compiled code. To put a few numbers on the computation cost, and the memory overhead, I did some small benchmarks on computing the shortest paths of a graph, something that we need in the scikit-learn. I compared the running time of the algorithm, but I also benched the time necessary to convert the data structures: In [1]: from sklearn.utils import graph_shortest_path as skl_graph In [2]: import networkx as nx * For medium sized graphs (negligeable memory cost): In [3]: G = nx.gnm_random_graph(100, 100, seed=0) In [4]: mat = nx.to_scipy_sparse_matrix(G).asformat('lil') In [5]: %timeit skl_graph.graph_shortest_path(mat, directed=False) 100 loops, best of 3: 3.61 ms per loop In [6]: %timeit nx.from_scipy_sparse_matrix(mat) 1000 loops, best of 3: 1.02 ms per loop In [7]: %timeit nx.shortest_paths.all_pairs_dijkstra_path_length(G) 10 loops, best of 3: 50.3 ms per loop * For biggish graphs (nx uses 800 Mb, scikit-learn around 10 to 20 Mb): In [8]: G = nx.gnm_random_graph(5000, 5000, seed=0) In [9]: mat = nx.to_scipy_sparse_matrix(G).asformat('lil') In [10]: %timeit skl_graph.graph_shortest_path(mat, directed=False) 1 loops, best of 3: 15.8 s per loop In [11]: %timeit nx.from_scipy_sparse_matrix(mat) 10 loops, best of 3: 59.6 ms per loop In [12]: %timeit nx.shortest_paths.all_pairs_dijkstra_path_length(G) 1 loops, best of 3: 197 s per loop With 10000 nodes, the above code does run on my laptop as it uses more than the available 2G of RAM. In addition, basic graph algorithms on numerical structures are needed for a lot of applications. People who need efficient implementation, like scikit-learn, scikit-image, or pyamg, will reimplement them. It seems profitable to have scipy as a common resource for sharing them. Scipy.sparse has all the necessary tools for that. Don't get me wrong, I love networkX and use it often because it has so many algorithms and is so versatile. I just think that a few graph algorithms are central and deserve a special treatment. Cheers, Gael From charlesr.harris at gmail.com Mon Dec 19 20:08:48 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 19 Dec 2011 18:08:48 -0700 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: <20111220001047.GB2294@phare.normalesup.org> References: <4EEF2D1E.6030508@astro.washington.edu> <20111220001047.GB2294@phare.normalesup.org> Message-ID: On Mon, Dec 19, 2011 at 5:10 PM, Gael Varoquaux < gael.varoquaux at normalesup.org> wrote: > On Mon, Dec 19, 2011 at 10:03:29AM -0600, Warren Weckesser wrote: > > Networkx ([3]http://networkx.lanl.gov/) already provides an > assortment of > > data structures and algorithms for graphs. It is an active, > > well-documented project. Personally, I'd rather not start adding > code to > > scipy that duplicates it. If your code is better/faster/stronger > than > > theirs, why not contribute it to networkx? > > I think that there is some room for a lower-level and more efficient base > structure from graphs in the scipy world, and that scipy is well suited > for that. Simple graph algorithms are common to many applications. For > instance a lot of the sparse linalg needs basic graph algorithms. It > seems to that having sparse graph algorithms in scipy (and we already > have some under the sparse package) would help the scipy world sharing > code with regards to graph algorithms. For instance, the scikit-image, > the scikit-learn, and pyamg all have shortest path algorithms. > > NetworkX is not well-suited for purely numerical application. First of > all, it is pure-Python, which has a lot of overheads as many of the graph > algorithms that cannot be implemented as array or matrix computation. > Second, it is using a dict of dicts representation to store and > manipulate graphs. The reason that they are using this representation is > that it enables to store rich objects for the nodes, and to dynamically > allocate and remove nodes. Such choices are great for the complex-systems > types of problems that they are targeting. However, for number crunching > applications, it is not great, as it has a significant memory overhead, > and prevents calling easily compiled code. > > To put a few numbers on the computation cost, and the memory overhead, I > did some small benchmarks on computing the shortest paths of a graph, > something that we need in the scikit-learn. I compared the running time > of the algorithm, but I also benched the time necessary to convert the > data structures: > > In [1]: from sklearn.utils import graph_shortest_path as skl_graph > > In [2]: import networkx as nx > > * For medium sized graphs (negligeable memory cost): > > In [3]: G = nx.gnm_random_graph(100, 100, seed=0) > > In [4]: mat = nx.to_scipy_sparse_matrix(G).asformat('lil') > > In [5]: %timeit skl_graph.graph_shortest_path(mat, directed=False) > 100 loops, best of 3: 3.61 ms per loop > > In [6]: %timeit nx.from_scipy_sparse_matrix(mat) > 1000 loops, best of 3: 1.02 ms per loop > > In [7]: %timeit nx.shortest_paths.all_pairs_dijkstra_path_length(G) > 10 loops, best of 3: 50.3 ms per loop > > * For biggish graphs (nx uses 800 Mb, scikit-learn around 10 to 20 Mb): > > In [8]: G = nx.gnm_random_graph(5000, 5000, seed=0) > > In [9]: mat = nx.to_scipy_sparse_matrix(G).asformat('lil') > > In [10]: %timeit skl_graph.graph_shortest_path(mat, directed=False) > 1 loops, best of 3: 15.8 s per loop > > In [11]: %timeit nx.from_scipy_sparse_matrix(mat) > 10 loops, best of 3: 59.6 ms per loop > > In [12]: %timeit nx.shortest_paths.all_pairs_dijkstra_path_length(G) > 1 loops, best of 3: 197 s per loop > > With 10000 nodes, the above code does run on my laptop as it uses more > than the available 2G of RAM. > > In addition, basic graph algorithms on numerical structures are needed > for a lot of applications. People who need efficient implementation, like > scikit-learn, scikit-image, or pyamg, will reimplement them. It seems > profitable to have scipy as a common resource for sharing them. > Scipy.sparse has all the necessary tools for that. > > Don't get me wrong, I love networkX and use it often because it has so > many algorithms and is so versatile. I just think that a few graph > algorithms are central and deserve a special treatment. > > Another graph algorithm I'd like to see in scipy is union-find. I've thought of adding it on occasion and keep a cython version around for my own use. One thing to be careful about is finding an interface that is widely useful without requiring a lot of special node structures and such. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aric.hagberg at gmail.com Mon Dec 19 22:38:07 2011 From: aric.hagberg at gmail.com (Aric Hagberg) Date: Mon, 19 Dec 2011 20:38:07 -0700 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> <20111220001047.GB2294@phare.normalesup.org> Message-ID: Warren posted over at the networkx-devel list and now I'm following up here too (see also http://groups.google.com/group/networkx-devel/browse_thread/thread/4bbd1bc3cafe3d54 ) On Mon, Dec 19, 2011 at 6:08 PM, Charles R Harris wrote: > > > On Mon, Dec 19, 2011 at 5:10 PM, Gael Varoquaux > wrote: >> >> On Mon, Dec 19, 2011 at 10:03:29AM -0600, Warren Weckesser wrote: >> > ? ?Networkx ([3]http://networkx.lanl.gov/) already provides an >> > assortment of >> > ? ?data structures and algorithms for graphs.? It is an active, >> > ? ?well-documented project.? Personally, I'd rather not start adding >> > code to >> > ? ?scipy that duplicates it.?? If your code is better/faster/stronger >> > than >> > ? ?theirs, why not contribute it to networkx? >> >> I think that there is some room for a lower-level and more efficient base >> structure from graphs in the scipy world, and that scipy is well suited >> for that. Simple graph algorithms are common to many applications. For >> instance a lot of the sparse linalg needs basic graph algorithms. It >> seems to that having sparse graph algorithms in scipy (and we already >> have some under the sparse package) would help the scipy world sharing >> code with regards to graph algorithms. For instance, the scikit-image, >> the scikit-learn, and pyamg all have shortest path algorithms. >> >> NetworkX is not well-suited for purely numerical application. First of >> all, it is pure-Python, which has a lot of overheads as many of the graph >> algorithms that cannot be implemented as array or matrix computation. >> Second, it is using a dict of dicts representation to store and >> manipulate graphs. The reason that they are using this representation is >> that it enables to store rich objects for the nodes, and to dynamically >> allocate and remove nodes. Such choices are great for the complex-systems >> types of problems that they are targeting. However, for number crunching >> applications, it is not great, as it has a significant memory overhead, >> and prevents calling easily compiled code. >> >> To put a few numbers on the computation cost, and the memory overhead, I >> did some small benchmarks on computing the shortest paths of a graph, >> something that we need in the scikit-learn. I compared the running time >> of the algorithm, but I also benched the time necessary to convert the >> data structures: Ga?l is right that the overhead of using Python dictionaries (and converting from SciPy sparse matrices and back) will be pretty expensive. For some graph algorithms that are easily expressed with adjacency matrices (scipy sparse matrices) it will be much better to implement them directly on that data structure. There are also big memory gains if you don't want to store arbitrary data on edges (instead either nothing or a single number). We have considered, and I think Dan Schult even prototyped, a NetworkX graph class based on SciPy sparse matrices. When we started NetworkX (2002) I don't think there even was a sparse matrix implementation in SciPy and we decided to use Python dictionaries. But there are some good benefits from using them and your use cases might be a good reason to do the engineering for that. Some parts are harder or just slower (like removing or adding nodes to a graph = removing or adding rows/columns to a scipy sparse matrix) but I think it can be done. >> Don't get me wrong, I love networkX and use it often because it has so >> many algorithms and is so versatile. I just think that a few graph >> algorithms are central and deserve a special treatment. There are some that I would definitely agree are useful in SciPy (or in a very SciPy sparse friendly NetworkX class?). For example I don't think there is a (reverse) Cuthill-McKee ordering algorithm in SciPy. I implemented one using NetworkX https://networkx.lanl.gov/trac/browser/networkx/networkx/utils/rcm.py but it would be great to have one that worked directly on a SciPy sparse matrix. > Another graph algorithm I'd like to see in scipy is union-find. I've thought > of adding it on occasion and keep a cython version around for my own use. I'd agree with that one too. We also use a nifty Python implementation that we borrowed from Eppstein and Carlson https://networkx.lanl.gov/trac/browser/networkx/networkx/utils/union_find.py If any other SciPy folks want to explore the possibilites of high-performance graph algorithms that work directly on sparse matrices I'm interested and would be willing to help with design and code. Aric From charlesr.harris at gmail.com Tue Dec 20 00:56:20 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 19 Dec 2011 22:56:20 -0700 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> <20111220001047.GB2294@phare.normalesup.org> Message-ID: On Mon, Dec 19, 2011 at 8:38 PM, Aric Hagberg wrote: > Warren posted over at the networkx-devel list and now I'm following up > here too > (see also > http://groups.google.com/group/networkx-devel/browse_thread/thread/4bbd1bc3cafe3d54 > ) > > On Mon, Dec 19, 2011 at 6:08 PM, Charles R Harris > wrote: > > > > > > On Mon, Dec 19, 2011 at 5:10 PM, Gael Varoquaux > > wrote: > >> > >> On Mon, Dec 19, 2011 at 10:03:29AM -0600, Warren Weckesser wrote: > >> > Networkx ([3]http://networkx.lanl.gov/) already provides an > >> > assortment of > >> > data structures and algorithms for graphs. It is an active, > >> > well-documented project. Personally, I'd rather not start adding > >> > code to > >> > scipy that duplicates it. If your code is better/faster/stronger > >> > than > >> > theirs, why not contribute it to networkx? > >> > >> I think that there is some room for a lower-level and more efficient > base > >> structure from graphs in the scipy world, and that scipy is well suited > >> for that. Simple graph algorithms are common to many applications. For > >> instance a lot of the sparse linalg needs basic graph algorithms. It > >> seems to that having sparse graph algorithms in scipy (and we already > >> have some under the sparse package) would help the scipy world sharing > >> code with regards to graph algorithms. For instance, the scikit-image, > >> the scikit-learn, and pyamg all have shortest path algorithms. > >> > >> NetworkX is not well-suited for purely numerical application. First of > >> all, it is pure-Python, which has a lot of overheads as many of the > graph > >> algorithms that cannot be implemented as array or matrix computation. > >> Second, it is using a dict of dicts representation to store and > >> manipulate graphs. The reason that they are using this representation is > >> that it enables to store rich objects for the nodes, and to dynamically > >> allocate and remove nodes. Such choices are great for the > complex-systems > >> types of problems that they are targeting. However, for number crunching > >> applications, it is not great, as it has a significant memory overhead, > >> and prevents calling easily compiled code. > >> > >> To put a few numbers on the computation cost, and the memory overhead, I > >> did some small benchmarks on computing the shortest paths of a graph, > >> something that we need in the scikit-learn. I compared the running time > >> of the algorithm, but I also benched the time necessary to convert the > >> data structures: > > Ga?l is right that the overhead of using Python dictionaries (and > converting from SciPy sparse matrices and back) will be pretty > expensive. For some graph algorithms that are easily expressed with > adjacency matrices (scipy sparse matrices) it will be much better to > implement them directly on that data structure. There are also big > memory gains if you don't want to store arbitrary data on edges > (instead either nothing or a single number). > > We have considered, and I think Dan Schult even prototyped, a NetworkX > graph class based on SciPy sparse matrices. When we started NetworkX > (2002) I don't think there even was a sparse matrix implementation in > SciPy and we decided to use Python dictionaries. But there are some > good benefits from using them and your use cases might be a good > reason to do the engineering for that. Some parts are harder or just > slower (like removing or adding nodes to a graph = removing or adding > rows/columns to a scipy sparse matrix) but I think it can be done. > > >> Don't get me wrong, I love networkX and use it often because it has so > >> many algorithms and is so versatile. I just think that a few graph > >> algorithms are central and deserve a special treatment. > > There are some that I would definitely agree are useful in SciPy (or > in a very SciPy sparse friendly NetworkX class?). For example I don't > think there is a (reverse) Cuthill-McKee ordering algorithm in SciPy. > I implemented one using NetworkX > https://networkx.lanl.gov/trac/browser/networkx/networkx/utils/rcm.py > but it would be great to have one that worked directly on a SciPy sparse > matrix. > > > Another graph algorithm I'd like to see in scipy is union-find. I've > thought > > of adding it on occasion and keep a cython version around for my own use. > > I'd agree with that one too. We also use a nifty Python > implementation that we borrowed from Eppstein and Carlson > > https://networkx.lanl.gov/trac/browser/networkx/networkx/utils/union_find.py > > I also keep the elements of each set in a linked circular list. Merging the lists when the sets are merged is easy and for my use cases being able to access all elements in a set given any representative element is convenient. I was also tempted to go the hashable object route but ended up just using integers since pretty much anything can be indexed and it is pretty efficient to just use an array instead of an dictionary. There might be a middle ground in there somewhere ;) If any other SciPy folks want to explore the possibilites of > high-performance graph algorithms that work directly on sparse > matrices I'm interested and would be willing to help with design and > code. > > That sounds like a useful thing to explore. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue Dec 20 01:53:02 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 20 Dec 2011 07:53:02 +0100 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> <20111220001047.GB2294@phare.normalesup.org> Message-ID: <20111220065302.GB12814@phare.normalesup.org> On Mon, Dec 19, 2011 at 06:08:48PM -0700, Charles R Harris wrote: > One thing to be careful about is finding an interface that is widely > useful without requiring a lot of special node structures and such. In scikit-learn, but also in pyamg and sfepy, the convention is to use a sparse matrix. Depending on the algorithm, different sparse matrices are more suited. We have found that it works well and does not introduce a new structure. Gael From gael.varoquaux at normalesup.org Tue Dec 20 02:06:15 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 20 Dec 2011 08:06:15 +0100 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> <20111220001047.GB2294@phare.normalesup.org> Message-ID: <20111220070615.GC12814@phare.normalesup.org> On Mon, Dec 19, 2011 at 08:38:07PM -0700, Aric Hagberg wrote: > There are some that I would definitely agree are useful in SciPy (or > in a very SciPy sparse friendly NetworkX class?). For example I don't > think there is a (reverse) Cuthill-McKee ordering algorithm in SciPy. > I implemented one using NetworkX > https://networkx.lanl.gov/trac/browser/networkx/networkx/utils/rcm.py > but it would be great to have one that worked directly on a SciPy sparse matrix. Yes, RCM is a good example. PyAMG had to implement one for linear-algebra purposes. > If any other SciPy folks want to explore the possibilites of > high-performance graph algorithms that work directly on sparse matrices > I'm interested and would be willing to help with design and code. Would you rather see these algorithms living in scipy or networkX? If scipy was to grow a small set of graph-based algorithms, would you use them in networkX? Cheers, Ga?l From pav at iki.fi Tue Dec 20 03:14:49 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 20 Dec 2011 09:14:49 +0100 Subject: [SciPy-Dev] [SciPy-User] test_arpack.test_symmetric_modes hangs, scipy 0.10 In-Reply-To: References: Message-ID: Hi, 19.12.2011 22:29, Ralf Gommers kirjoitti: [clip] > Anyone? > > This is clearly very broken, yet another bug report today (11 failures). > I would even consider doing 0.10.1 release to clean this up. I won't object. Given that hangs were seen not only on OSX raises some concerns that there is maybe something really wrong with the single-precision routines, and not only the tests. (The results from these tests however seem to indicate precision issues instead...) Pauli From avalle at famaf.unc.edu.ar Tue Dec 20 19:50:24 2011 From: avalle at famaf.unc.edu.ar (avalle at famaf.unc.edu.ar) Date: Tue, 20 Dec 2011 22:50:24 -0200 Subject: [SciPy-Dev] Broken toolchain: cannot link a simple C program Message-ID: <65841ffe90d1afe9017fd91fce1664f7.squirrel@webmail.famaf.unc.edu.ar> Hi, I am trying to compile numpy using fortran compiler. I am running Ubuntu 11.10 gnu: no Fortran 90 compiler found (Is that a problem?), and what means "cannot link a simple C program"? Thank you for your help lucia at takenoko:~/numpy-1.6.1$ python setup.py config --compiler=intelem build_clib --compiler=intelem build_ext --compiler=intelem install Running from numpy source directory.non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib Setting PTATLAS=ATLAS customize GnuFCompiler Could not locate executable g77 Found executable /usr/bin/f77 gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize IntelFCompiler Found executable /opt/intel/Compiler/11.1/038/bin/intel64/ifort customize LaheyFCompiler Could not locate executable lf95 customize PGroupFCompiler Could not locate executable pgf90 Could not locate executable pgf77 customize AbsoftFCompiler Could not locate executable f90 absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found customize NAGFCompiler Found executable /usr/bin/f95 customize VastFCompiler customize CompaqFCompiler Could not locate executable fort customize IntelItaniumFCompiler customize IntelEM64TFCompiler customize IntelEM64TFCompiler customize IntelEM64TFCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/usr/lib/atlas-base -lptf77blas -lptcblas -latlas -o _configtest ATLAS version 3.8.4 built by buildd on Sat Sep 10 23:12:12 UTC 2011: UNAME : Linux crested 2.6.24-29-server #1 SMP Wed Aug 10 15:58:57 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_HAMMER -DATL_CPUMHZ=1993 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 393216 F77 : gfortran, version GNU Fortran (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 F77FLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 SMCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 SKCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 -Wa,--noexecstack -fPIC -m64 success! removing: _configtest.c _configtest.o _configtest Setting PTATLAS=ATLAS FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack not found in /usr/lib/atlas-base numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] include_dirs = ['/usr/include/atlas'] running config running build_clib running build_src build_src building py_modules sources building library "npymath" sources Could not locate executable icc Could not locate executable ecc customize GnuFCompiler gnu: no Fortran 90 compiler found gnu: no Fortran 90 compiler found customize IntelFCompiler customize LaheyFCompiler customize PGroupFCompiler customize AbsoftFCompiler absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found customize NAGFCompiler customize VastFCompiler customize CompaqFCompiler customize IntelItaniumFCompiler customize IntelEM64TFCompiler customize IntelEM64TFCompiler customize IntelEM64TFCompiler using config C compiler: icc -m64 -fPIC compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/include/python2.7 -c' icc: _configtest.c sh: icc: not found sh: icc: not found failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 196, in setup_package() File "setup.py", line 189, in setup_package configuration=configuration ) File "/home/lucia/numpy-1.6.1/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/lucia/numpy-1.6.1/numpy/distutils/command/build_clib.py", line 62, in run self.run_command('build_src') File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/lucia/numpy-1.6.1/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/home/lucia/numpy-1.6.1/numpy/distutils/command/build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "/home/lucia/numpy-1.6.1/numpy/distutils/command/build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "/home/lucia/numpy-1.6.1/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 696, in get_mathlib_info raise RuntimeError("Broken toolchain: cannot link a simple C program") RuntimeError: Broken toolchain: cannot link a simple C program From aric.hagberg at gmail.com Tue Dec 20 19:59:26 2011 From: aric.hagberg at gmail.com (Aric Hagberg) Date: Tue, 20 Dec 2011 17:59:26 -0700 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: <20111220070615.GC12814@phare.normalesup.org> References: <4EEF2D1E.6030508@astro.washington.edu> <20111220001047.GB2294@phare.normalesup.org> <20111220070615.GC12814@phare.normalesup.org> Message-ID: On Tue, Dec 20, 2011 at 12:06 AM, Gael Varoquaux wrote: > On Mon, Dec 19, 2011 at 08:38:07PM -0700, Aric Hagberg wrote: >> There are some that I would definitely agree are useful in SciPy (or >> in a very SciPy sparse friendly NetworkX class?). ?For example I don't >> think there is a (reverse) Cuthill-McKee ordering algorithm in SciPy. >> I implemented one using NetworkX >> https://networkx.lanl.gov/trac/browser/networkx/networkx/utils/rcm.py >> but it would be great to have one that worked directly on a SciPy sparse matrix. > > Yes, RCM is a good example. PyAMG had to implement one for linear-algebra > purposes. > >> If any other SciPy folks want to explore the possibilites of >> high-performance graph algorithms that work directly on sparse matrices >> I'm interested and would be willing to help with design and code. > > Would you rather see these algorithms living in scipy or networkX? If > scipy was to grow a small set of graph-based algorithms, would you use > them in networkX? I would certainly consider using any algorithms developed in SciPy as part of NetworkX. We already use SciPy for some of the linear algebra operations (like computing PageRank). If we could develop a NetworkX graph class that simply and efficiently used SciPy sparse matrices for the data-store and implemented the API, then it might make sense for the algorithms to live in NetworkX. Then the existing NetworkX algorithms could be used too. You can already do that now but as you pointed out there are copies made into a less efficient storage structure, e.g. In [1]: import scipy.sparse In [2]: import networkx as nx In [3]: A=scipy.sparse.lil_matrix((5,5)) In [4]: A.setdiag([1,1,1,1],k=1) # path 0-1-2-3-4 In [5]: G=nx.Graph(A) # <- copy of lil matrix to networkx dictionaries In [6]: nx.connected_components(G) Out[6]: [[0, 1, 2, 3, 4]] Aric From scott.sinclair.za at gmail.com Wed Dec 21 01:28:29 2011 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Wed, 21 Dec 2011 08:28:29 +0200 Subject: [SciPy-Dev] Broken toolchain: cannot link a simple C program In-Reply-To: <65841ffe90d1afe9017fd91fce1664f7.squirrel@webmail.famaf.unc.edu.ar> References: <65841ffe90d1afe9017fd91fce1664f7.squirrel@webmail.famaf.unc.edu.ar> Message-ID: On 21 December 2011 02:50, wrote: > I am trying to compile numpy using fortran compiler. I am running Ubuntu > 11.10 > gnu: no Fortran 90 compiler found (Is that a problem?), and what means > "cannot link a simple C program"? > > Thank you for your help > > lucia at takenoko:~/numpy-1.6.1$ python setup.py config --compiler=intelem > build_clib --compiler=intelem build_ext --compiler=intelem install > Running from numpy source directory.non-existing path in > 'numpy/distutils': 'site.cfg' > F2PY Version 2 > blas_opt_info: > blas_mkl_info: > ?libraries mkl,vml,guide not found in /usr/local/lib > ?libraries mkl,vml,guide not found in /usr/lib > ?NOT AVAILABLE > > atlas_blas_threads_info: > Setting PTATLAS=ATLAS > ?libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib > Setting PTATLAS=ATLAS > customize GnuFCompiler > Could not locate executable g77 > Found executable /usr/bin/f77 > gnu: no Fortran 90 compiler found > gnu: no Fortran 90 compiler found > customize IntelFCompiler > Found executable /opt/intel/Compiler/11.1/038/bin/intel64/ifort > customize LaheyFCompiler > Could not locate executable lf95 > customize PGroupFCompiler > Could not locate executable pgf90 > Could not locate executable pgf77 > customize AbsoftFCompiler > Could not locate executable f90 > absoft: no Fortran 90 compiler found > absoft: no Fortran 90 compiler found > customize NAGFCompiler > Found executable /usr/bin/f95 > customize VastFCompiler > customize CompaqFCompiler > Could not locate executable fort > customize IntelItaniumFCompiler > customize IntelEM64TFCompiler > customize IntelEM64TFCompiler > customize IntelEM64TFCompiler using config > compiling '_configtest.c': > > /* This file is generated from numpy/distutils/system_info.py */ > void ATL_buildinfo(void); > int main(void) { > ?ATL_buildinfo(); > ?return 0; > } > > C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv > -O2 -Wall -Wstrict-prototypes -fPIC > > compile options: '-c' > gcc: _configtest.c > gcc -pthread _configtest.o -L/usr/lib/atlas-base -lptf77blas -lptcblas > -latlas -o _configtest > ATLAS version 3.8.4 built by buildd on Sat Sep 10 23:12:12 UTC 2011: > ? UNAME ? ?: Linux crested 2.6.24-29-server #1 SMP Wed Aug 10 15:58:57 > UTC 2011 x86_64 x86_64 x86_64 GNU/Linux > ? INSTFLG ?: -1 0 -a 1 > ? ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_HAMMER -DATL_CPUMHZ=1993 > -DATL_USE64BITS -DATL_GAS_x8664 > ? F2CDEFS ?: -DAdd_ -DF77_INTEGER=int -DStringSunStyle > ? CACHEEDGE: 393216 > ? F77 ? ? ?: gfortran, version GNU Fortran (Ubuntu/Linaro 4.6.1-9ubuntu2) > 4.6.1 > ? F77FLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 > -Wa,--noexecstack -fPIC -m64 > ? SMC ? ? ?: gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 > ? SMCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 > -Wa,--noexecstack -fPIC -m64 > ? SKC ? ? ?: gcc, version gcc (Ubuntu/Linaro 4.6.1-9ubuntu2) 4.6.1 > ? SKCFLAGS : -fomit-frame-pointer -mfpmath=387 -O2 -falign-loops=4 > -Wa,--noexecstack -fPIC -m64 > success! > removing: _configtest.c _configtest.o _configtest > Setting PTATLAS=ATLAS > ?FOUND: > ? ?libraries = ['ptf77blas', 'ptcblas', 'atlas'] > ? ?library_dirs = ['/usr/lib/atlas-base'] > ? ?language = c > ? ?define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] > ? ?include_dirs = ['/usr/include/atlas'] > > ?FOUND: > ? ?libraries = ['ptf77blas', 'ptcblas', 'atlas'] > ? ?library_dirs = ['/usr/lib/atlas-base'] > ? ?language = c > ? ?define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] > ? ?include_dirs = ['/usr/include/atlas'] > > lapack_opt_info: > lapack_mkl_info: > mkl_info: > ?libraries mkl,vml,guide not found in /usr/local/lib > ?libraries mkl,vml,guide not found in /usr/lib > ?NOT AVAILABLE > > ?NOT AVAILABLE > > atlas_threads_info: > Setting PTATLAS=ATLAS > ?libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib > ?libraries lapack_atlas not found in /usr/local/lib > ?libraries lapack not found in /usr/lib/atlas-base > numpy.distutils.system_info.atlas_threads_info > Setting PTATLAS=ATLAS > Setting PTATLAS=ATLAS > ?FOUND: > ? ?libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] > ? ?library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] > ? ?language = f77 > ? ?define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] > ? ?include_dirs = ['/usr/include/atlas'] > > ?FOUND: > ? ?libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] > ? ?library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] > ? ?language = f77 > ? ?define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] > ? ?include_dirs = ['/usr/include/atlas'] > > running config > running build_clib > running build_src > build_src > building py_modules sources > building library "npymath" sources > Could not locate executable icc > Could not locate executable ecc > customize GnuFCompiler > gnu: no Fortran 90 compiler found > gnu: no Fortran 90 compiler found > customize IntelFCompiler > customize LaheyFCompiler > customize PGroupFCompiler > customize AbsoftFCompiler > absoft: no Fortran 90 compiler found > absoft: no Fortran 90 compiler found > customize NAGFCompiler > customize VastFCompiler > customize CompaqFCompiler > customize IntelItaniumFCompiler > customize IntelEM64TFCompiler > customize IntelEM64TFCompiler > customize IntelEM64TFCompiler using config > C compiler: icc -m64 -fPIC > > compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray > -Inumpy/core/src/umath -Inumpy/core/include -I/usr/include/python2.7 -c' > icc: _configtest.c > sh: icc: not found > sh: icc: not found > failure. > removing: _configtest.c _configtest.o > Traceback (most recent call last): > ?File "setup.py", line 196, in > ? ?setup_package() > ?File "setup.py", line 189, in setup_package > ? ?configuration=configuration ) > ?File "/home/lucia/numpy-1.6.1/numpy/distutils/core.py", line 186, in setup > ? ?return old_setup(**new_attr) > ?File "/usr/lib/python2.7/distutils/core.py", line 152, in setup > ? ?dist.run_commands() > ?File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands > ? ?self.run_command(cmd) > ?File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command > ? ?cmd_obj.run() > ?File "/home/lucia/numpy-1.6.1/numpy/distutils/command/build_clib.py", > line 62, in run > ? ?self.run_command('build_src') > ?File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command > ? ?self.distribution.run_command(command) > ?File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command > ? ?cmd_obj.run() > ?File "/home/lucia/numpy-1.6.1/numpy/distutils/command/build_src.py", > line 152, in run > ? ?self.build_sources() > ?File "/home/lucia/numpy-1.6.1/numpy/distutils/command/build_src.py", > line 163, in build_sources > ? ?self.build_library_sources(*libname_info) > ?File "/home/lucia/numpy-1.6.1/numpy/distutils/command/build_src.py", > line 298, in build_library_sources > ? ?sources = self.generate_sources(sources, (lib_name, build_info)) > ?File "/home/lucia/numpy-1.6.1/numpy/distutils/command/build_src.py", > line 385, in generate_sources > ? ?source = func(extension, build_dir) > ?File "numpy/core/setup.py", line 696, in get_mathlib_info > ? ?raise RuntimeError("Broken toolchain: cannot link a simple C program") > RuntimeError: Broken toolchain: cannot link a simple C program Check that icc is installed and on your path. The line "sh: icc: not found" indicates that it isn't.. Cheers, Scott From ralf.gommers at googlemail.com Wed Dec 21 02:28:03 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 21 Dec 2011 08:28:03 +0100 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> <20111220001047.GB2294@phare.normalesup.org> <20111220070615.GC12814@phare.normalesup.org> Message-ID: On Wed, Dec 21, 2011 at 1:59 AM, Aric Hagberg wrote: > On Tue, Dec 20, 2011 at 12:06 AM, Gael Varoquaux > wrote: > > On Mon, Dec 19, 2011 at 08:38:07PM -0700, Aric Hagberg wrote: > >> There are some that I would definitely agree are useful in SciPy (or > >> in a very SciPy sparse friendly NetworkX class?). For example I don't > >> think there is a (reverse) Cuthill-McKee ordering algorithm in SciPy. > >> I implemented one using NetworkX > >> https://networkx.lanl.gov/trac/browser/networkx/networkx/utils/rcm.py > >> but it would be great to have one that worked directly on a SciPy > sparse matrix. > > > > Yes, RCM is a good example. PyAMG had to implement one for linear-algebra > > purposes. > > > >> If any other SciPy folks want to explore the possibilites of > >> high-performance graph algorithms that work directly on sparse matrices > >> I'm interested and would be willing to help with design and code. > > > > Would you rather see these algorithms living in scipy or networkX? If > > scipy was to grow a small set of graph-based algorithms, would you use > > them in networkX? > > I would certainly consider using any algorithms developed in SciPy as > part of NetworkX. We already use SciPy for some of the linear algebra > operations (like computing PageRank). > > If we could develop a NetworkX graph class that simply and efficiently > used SciPy sparse matrices for the data-store and implemented the API, > then it might make sense for the algorithms to live in NetworkX. This only makes sense if other libraries are prepared to add a dependency on NetworkX. Is that the case? Ralf > Then > the existing NetworkX algorithms could be used too. You can already > do that now but as you pointed out there are copies made into a less > efficient storage structure, e.g. > > In [1]: import scipy.sparse > > In [2]: import networkx as nx > > In [3]: A=scipy.sparse.lil_matrix((5,5)) > > In [4]: A.setdiag([1,1,1,1],k=1) # path 0-1-2-3-4 > > In [5]: G=nx.Graph(A) # <- copy of lil matrix to networkx dictionaries > > In [6]: nx.connected_components(G) > Out[6]: [[0, 1, 2, 3, 4]] > > > Aric > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Wed Dec 21 03:20:45 2011 From: vanderplas at astro.washington.edu (Jacob VanderPlas) Date: Wed, 21 Dec 2011 00:20:45 -0800 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> <20111220001047.GB2294@phare.normalesup.org> <20111220070615.GC12814@phare.normalesup.org> Message-ID: <4EF196DD.40104@astro.washington.edu> Ralf Gommers wrote: > > > On Wed, Dec 21, 2011 at 1:59 AM, Aric Hagberg > wrote: > > On Tue, Dec 20, 2011 at 12:06 AM, Gael Varoquaux > > wrote: > > On Mon, Dec 19, 2011 at 08:38:07PM -0700, Aric Hagberg wrote: > >> There are some that I would definitely agree are useful in > SciPy (or > >> in a very SciPy sparse friendly NetworkX class?). For example > I don't > >> think there is a (reverse) Cuthill-McKee ordering algorithm in > SciPy. > >> I implemented one using NetworkX > >> > https://networkx.lanl.gov/trac/browser/networkx/networkx/utils/rcm.py > >> but it would be great to have one that worked directly on a > SciPy sparse matrix. > > > > Yes, RCM is a good example. PyAMG had to implement one for > linear-algebra > > purposes. > > > >> If any other SciPy folks want to explore the possibilites of > >> high-performance graph algorithms that work directly on sparse > matrices > >> I'm interested and would be willing to help with design and code. > > > > Would you rather see these algorithms living in scipy or > networkX? If > > scipy was to grow a small set of graph-based algorithms, would > you use > > them in networkX? > > I would certainly consider using any algorithms developed in SciPy as > part of NetworkX. We already use SciPy for some of the linear algebra > operations (like computing PageRank). > > If we could develop a NetworkX graph class that simply and efficiently > used SciPy sparse matrices for the data-store and implemented the API, > then it might make sense for the algorithms to live in NetworkX. > > > This only makes sense if other libraries are prepared to add a > dependency on NetworkX. Is that the case? scikit-learn will likely not add dependency on networkx. One of our goals is to keep dependencies to a minimum: numpy and scipy for algorithms, adding matplotlib for examples. We'd love to see some core graph algorithms developed and maintained in scipy which can be used by scikit-learn, networkx, and other libraries. Jake > > Ralf > > > Then > the existing NetworkX algorithms could be used too. You can already > do that now but as you pointed out there are copies made into a less > efficient storage structure, e.g. > > In [1]: import scipy.sparse > > In [2]: import networkx as nx > > In [3]: A=scipy.sparse.lil_matrix((5,5)) > > In [4]: A.setdiag([1,1,1,1],k=1) # path 0-1-2-3-4 > > In [5]: G=nx.Graph(A) # <- copy of lil matrix to networkx > dictionaries > > In [6]: nx.connected_components(G) > Out[6]: [[0, 1, 2, 3, 4]] > > > Aric > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > ------------------------------------------------------------------------ > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From ralf.gommers at googlemail.com Thu Dec 22 12:19:27 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 22 Dec 2011 18:19:27 +0100 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: <4EF196DD.40104@astro.washington.edu> References: <4EEF2D1E.6030508@astro.washington.edu> <20111220001047.GB2294@phare.normalesup.org> <20111220070615.GC12814@phare.normalesup.org> <4EF196DD.40104@astro.washington.edu> Message-ID: On Wed, Dec 21, 2011 at 9:20 AM, Jacob VanderPlas < vanderplas at astro.washington.edu> wrote: > > > Ralf Gommers wrote: > > > > > > On Wed, Dec 21, 2011 at 1:59 AM, Aric Hagberg > > wrote: > > > > On Tue, Dec 20, 2011 at 12:06 AM, Gael Varoquaux > > > > wrote: > > > On Mon, Dec 19, 2011 at 08:38:07PM -0700, Aric Hagberg wrote: > > >> There are some that I would definitely agree are useful in > > SciPy (or > > >> in a very SciPy sparse friendly NetworkX class?). For example > > I don't > > >> think there is a (reverse) Cuthill-McKee ordering algorithm in > > SciPy. > > >> I implemented one using NetworkX > > >> > > > https://networkx.lanl.gov/trac/browser/networkx/networkx/utils/rcm.py > > >> but it would be great to have one that worked directly on a > > SciPy sparse matrix. > > > > > > Yes, RCM is a good example. PyAMG had to implement one for > > linear-algebra > > > purposes. > > > > > >> If any other SciPy folks want to explore the possibilites of > > >> high-performance graph algorithms that work directly on sparse > > matrices > > >> I'm interested and would be willing to help with design and code. > > > > > > Would you rather see these algorithms living in scipy or > > networkX? If > > > scipy was to grow a small set of graph-based algorithms, would > > you use > > > them in networkX? > > > > I would certainly consider using any algorithms developed in SciPy as > > part of NetworkX. We already use SciPy for some of the linear > algebra > > operations (like computing PageRank). > > > > If we could develop a NetworkX graph class that simply and > efficiently > > used SciPy sparse matrices for the data-store and implemented the > API, > > then it might make sense for the algorithms to live in NetworkX. > > > > > > This only makes sense if other libraries are prepared to add a > > dependency on NetworkX. Is that the case? > scikit-learn will likely not add dependency on networkx. One of our > goals is to keep dependencies to a minimum: numpy and scipy for > algorithms, adding matplotlib for examples. We'd love to see some core > graph algorithms developed and maintained in scipy which can be used by > scikit-learn, networkx, and other libraries. > > That's what I thought. I'm +1 for adding this to scipy. Preferably as scipy.sparse.graph or something like that, not a separate module. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanderplas at astro.washington.edu Fri Dec 23 11:40:17 2011 From: vanderplas at astro.washington.edu (Jacob VanderPlas) Date: Fri, 23 Dec 2011 08:40:17 -0800 Subject: [SciPy-Dev] Graph shortest path In-Reply-To: References: <4EEF2D1E.6030508@astro.washington.edu> <20111220001047.GB2294@phare.normalesup.org> <20111220070615.GC12814@phare.normalesup.org> <4EF196DD.40104@astro.washington.edu> Message-ID: <4EF4AEF1.2010103@astro.washington.edu> Hi, I've submitted an initial pull request with some modified versions of the graph utilities in scikit-learn/utils: https://github.com/scipy/scipy/pull/119 If anyone is interested, we should continue the discussion there. In particular, it would be nice to brainstorm the list of algorithms we think should be included. Thanks! Jake Ralf Gommers wrote: > > > On Wed, Dec 21, 2011 at 9:20 AM, Jacob VanderPlas > > wrote: > > > > Ralf Gommers wrote: > > > > > > On Wed, Dec 21, 2011 at 1:59 AM, Aric Hagberg > > > >> > wrote: > > > > On Tue, Dec 20, 2011 at 12:06 AM, Gael Varoquaux > > > > >> wrote: > > > On Mon, Dec 19, 2011 at 08:38:07PM -0700, Aric Hagberg wrote: > > >> There are some that I would definitely agree are useful in > > SciPy (or > > >> in a very SciPy sparse friendly NetworkX class?). For > example > > I don't > > >> think there is a (reverse) Cuthill-McKee ordering > algorithm in > > SciPy. > > >> I implemented one using NetworkX > > >> > > > https://networkx.lanl.gov/trac/browser/networkx/networkx/utils/rcm.py > > >> but it would be great to have one that worked directly on a > > SciPy sparse matrix. > > > > > > Yes, RCM is a good example. PyAMG had to implement one for > > linear-algebra > > > purposes. > > > > > >> If any other SciPy folks want to explore the possibilites of > > >> high-performance graph algorithms that work directly on > sparse > > matrices > > >> I'm interested and would be willing to help with design > and code. > > > > > > Would you rather see these algorithms living in scipy or > > networkX? If > > > scipy was to grow a small set of graph-based algorithms, would > > you use > > > them in networkX? > > > > I would certainly consider using any algorithms developed in > SciPy as > > part of NetworkX. We already use SciPy for some of the > linear algebra > > operations (like computing PageRank). > > > > If we could develop a NetworkX graph class that simply and > efficiently > > used SciPy sparse matrices for the data-store and > implemented the API, > > then it might make sense for the algorithms to live in NetworkX. > > > > > > This only makes sense if other libraries are prepared to add a > > dependency on NetworkX. Is that the case? > scikit-learn will likely not add dependency on networkx. One of our > goals is to keep dependencies to a minimum: numpy and scipy for > algorithms, adding matplotlib for examples. We'd love to see some > core > graph algorithms developed and maintained in scipy which can be > used by > scikit-learn, networkx, and other libraries. > > That's what I thought. I'm +1 for adding this to scipy. Preferably as > scipy.sparse.graph or something like that, not a separate module. > > Ralf > > > ------------------------------------------------------------------------ > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From ilanschnell at gmail.com Fri Dec 23 13:39:16 2011 From: ilanschnell at gmail.com (Ilan Schnell) Date: Fri, 23 Dec 2011 12:39:16 -0600 Subject: [SciPy-Dev] ANN: EPD 7.2 released Message-ID: Hello, I am pleased to announce the release of Enthought Python Distribution, EPD version 7.2, along with its "EPD Free" counterpart. The highlights of this release are: the addition of GDAL and updates to over 30 packages, including SciPy, matplotlib and IPython. The new IPython 0.12 includes the HTML notebook, which caused the Tornado web-server also to be added to EPD. To see which libraries are included in the free vs. full version, please see: http://www.enthought.com/products/epdlibraries.php The complete list of additions, updates and fixes is in the change log: http://www.enthought.com/products/changelog.php About EPD --------- The Enthought Python Distribution (EPD) is a "kitchen-sink-included" distribution of the Python programming language, including over 90 additional tools and libraries. The EPD bundle includes NumPy, SciPy, IPython, 2D and 3D visualization tools, and many other tools. EPD is currently available as a single-click installer for Windows XP, Vista and 7, MacOS (10.5 and 10.6), RedHat 3, 4, 5 and 6, as well as Solaris 10 (x86 and x86_64/amd64 on all platforms). All versions of EPD (32 and 64-bit) are free for academic use. An annual subscription including installation support is available for individual and commercial use. Additional support options, including customization, bug fixes and training classes are also available: http://www.enthought.com/products/epd_sublevels.php - Ilan From deshpande.jaidev at gmail.com Mon Dec 26 12:39:29 2011 From: deshpande.jaidev at gmail.com (Jaidev Deshpande) Date: Mon, 26 Dec 2011 23:09:29 +0530 Subject: [SciPy-Dev] Proposal for Scikit-Signal - a SciPy toolbox for signal processing Message-ID: Hi I gave a talk at SciPy India 2011 about a Python implementation of the Hilbert-Huang Transform that I was working on. The HHT is a method used as an alternative to Fourier and Wavelet analyses of nonlinear and nonstationary data. Following the talk Gael Varoquaux said that there's room for a separate scikit for signal processing. He also gave a lightning talk about bootstrapping a SciPy community project soon after. So with this list let us start working out what the project should be like. For noobs like me, Gael's talk was quite a useful guide. Here's the link to a gist he made about it - https://gist.github.com/1433151 Here's the link to my SciPy talk: http://urtalk.kpoint.in/kapsule/gcc-57b6c86b-2f12-4244-950c-a34360a2cc1f/view/search/tag%3Ascipy I personally am researching nonlinear and nonstationary signal processing, I'd love to know what others can bring to this project. Also, let's talk about the limitations of the current signal processing tools available in SciPy and other scikits. I think there's a lot of documentation to be worked out, and there is also a lack of physically meaningful examples in the documentation. Thanks PS: I'm ccing a few people who might already be on the scipy-dev list. Sorry for the inconvenience. From alan at ajackson.org Mon Dec 26 19:12:30 2011 From: alan at ajackson.org (alan at ajackson.org) Date: Mon, 26 Dec 2011 18:12:30 -0600 Subject: [SciPy-Dev] Proposal for Scikit-Signal - a SciPy toolbox for signal processing In-Reply-To: References: Message-ID: <20111226181230.25fb34b4@ajackson.org> >Hi > >I gave a talk at SciPy India 2011 about a Python implementation of the >Hilbert-Huang Transform that I was working on. The HHT is a method >used as an alternative to Fourier and Wavelet analyses of nonlinear >and nonstationary data. Following the talk Gael Varoquaux said that >there's room for a separate scikit for signal processing. He also gave >a lightning talk about bootstrapping a SciPy community project soon >after. > >So with this list let us start working out what the project should be like. > >For noobs like me, Gael's talk was quite a useful guide. Here's the >link to a gist he made about it - https://gist.github.com/1433151 > >Here's the link to my SciPy talk: >http://urtalk.kpoint.in/kapsule/gcc-57b6c86b-2f12-4244-950c-a34360a2cc1f/view/search/tag%3Ascipy > >I personally am researching nonlinear and nonstationary signal >processing, I'd love to know what others can bring to this project. >Also, let's talk about the limitations of the current signal >processing tools available in SciPy and other scikits. I think there's >a lot of documentation to be worked out, and there is also a lack of >physically meaningful examples in the documentation. > I've been playing with Empirical Mode Decomposition, and coded it up in numpy, and it is pretty neat, but I do believe that NASA has patented it, which would probably preclude distributing it in a scikit, without a lot of legal effort. From the Nasa website : "An example of successfully brokering NASA technology through a no-cost brokerage partnership was the exclusive license for the Hilbert-Huang Transform, composed of 10 U.S. patents and one domestic patent application, which was part of a lot auctioned by Ocean Tomo Federal Services LLC, in October 2008." Alan -- ----------------------------------------------------------------------- | Alan K. Jackson | To see a World in a Grain of Sand | | alan at ajackson.org | And a Heaven in a Wild Flower, | | www.ajackson.org | Hold Infinity in the palm of your hand | | Houston, Texas | And Eternity in an hour. - Blake | ----------------------------------------------------------------------- From deshpande.jaidev at gmail.com Tue Dec 27 12:55:59 2011 From: deshpande.jaidev at gmail.com (Jaidev Deshpande) Date: Tue, 27 Dec 2011 23:25:59 +0530 Subject: [SciPy-Dev] Proposal for Scikit-Signal - a SciPy toolbox for signal processing In-Reply-To: <20111226181230.25fb34b4@ajackson.org> References: <20111226181230.25fb34b4@ajackson.org> Message-ID: Hi Alan > I've been playing with Empirical Mode Decomposition, and coded it up in > numpy, and it is pretty neat, but I do believe that NASA has patented it, which > would probably preclude distributing it in a scikit, without a lot of legal > effort. EMD and HHT are nowhere in the scikit plan right now (unless others decide to put it up), I simply mentioned it because it's one of my interests. Let's start talking about a basic signal processing scikit first. I guess there will be enough room for adaptive methods like the HHT later. Regards From alexandre.gramfort at inria.fr Wed Dec 28 11:33:24 2011 From: alexandre.gramfort at inria.fr (Alexandre Gramfort) Date: Wed, 28 Dec 2011 17:33:24 +0100 Subject: [SciPy-Dev] Proposal for Scikit-Signal - a SciPy toolbox for signal processing In-Reply-To: References: <20111226181230.25fb34b4@ajackson.org> Message-ID: Hi all, I think a scikit-signal would be a nice project in the scipy ecosystem. My gut feeling is that there is already a lot of great pieces of code for signal processing in Python but it's too fragmented. Most of it is in scipy.signal but one may need some pieces in scipy.ndimage and external projects like for example for wavelets. I would be neat to have a main entry point for signal processing in Python. As demonstrated with scikit-learn a small and great project can emerge from scipy/numpy. The benefit is that the entry cost can be much lower for a developer compared to contributing directly to scipy and a small project can release more often and eventually back port new stuff from scipy core. As I already said to Jaydev, I think one should start by defining the scope of such a project and list/review existing codes to bootstrap the project. Best, Alex On Tue, Dec 27, 2011 at 6:55 PM, Jaidev Deshpande wrote: > Hi Alan > >> I've been playing with Empirical Mode Decomposition, and coded it up in >> numpy, and it is pretty neat, but I do believe that NASA has patented it, which >> would probably preclude distributing it in a scikit, without a lot of legal >> effort. > > EMD and HHT are nowhere in the scikit plan right now (unless others > decide to put it up), I simply mentioned it because it's one of my > interests. > > Let's start talking about a basic signal processing scikit first. I > guess there will be enough room for adaptive methods like the HHT > later. > > Regards > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From cournape at gmail.com Wed Dec 28 13:16:14 2011 From: cournape at gmail.com (David Cournapeau) Date: Wed, 28 Dec 2011 18:16:14 +0000 Subject: [SciPy-Dev] Proposal for Scikit-Signal - a SciPy toolbox for signal processing In-Reply-To: References: Message-ID: Hi Jaidev, On Mon, Dec 26, 2011 at 5:39 PM, Jaidev Deshpande wrote: > Hi > > I gave a talk at SciPy India 2011 about a Python implementation of the > Hilbert-Huang Transform that I was working on. The HHT is a method > used as an alternative to Fourier and Wavelet analyses of nonlinear > and nonstationary data. Following the talk Gael Varoquaux said that > there's room for a separate scikit for signal processing. He also gave > a lightning talk about bootstrapping a SciPy community project soon > after. > > So with this list let us start working out what the project should be like. > > For noobs like me, Gael's talk was quite a useful guide. Here's the > link to a gist he made about it - https://gist.github.com/1433151 > > Here's the link to my SciPy talk: > http://urtalk.kpoint.in/kapsule/gcc-57b6c86b-2f12-4244-950c-a34360a2cc1f/view/search/tag%3Ascipy > > I personally am researching nonlinear and nonstationary signal > processing, I'd love to know what others can bring to this project. > Also, let's talk about the limitations of the current signal > processing tools available in SciPy and other scikits. I think there's > a lot of documentation to be worked out, and there is also a lack of > physically meaningful examples in the documentation. I think it would be a good addition to the ecosystem. We are for example missing a lot of core algorithms even for linear signal processing, and scipy.signal itself would benefit from some refactoring. I myself started something (the talkbox scikit), but realistically, I won't have time to work on a full toolbox, so we can consolidate here. I would be willing to work on merging what I already have in what you have in mind: - Linear prediction coding with Levinson Durbin implementation - A start of periodogram function I could spend time to implement a few more things like MUSIC/PENCIL, and some basic matching pursuit algorithms. cheers, David From josef.pktd at gmail.com Wed Dec 28 13:46:43 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 28 Dec 2011 13:46:43 -0500 Subject: [SciPy-Dev] Proposal for Scikit-Signal - a SciPy toolbox for signal processing In-Reply-To: References: Message-ID: On Wed, Dec 28, 2011 at 1:16 PM, David Cournapeau wrote: > Hi Jaidev, > > On Mon, Dec 26, 2011 at 5:39 PM, Jaidev Deshpande > wrote: >> Hi >> >> I gave a talk at SciPy India 2011 about a Python implementation of the >> Hilbert-Huang Transform that I was working on. The HHT is a method >> used as an alternative to Fourier and Wavelet analyses of nonlinear >> and nonstationary data. Following the talk Gael Varoquaux said that >> there's room for a separate scikit for signal processing. He also gave >> a lightning talk about bootstrapping a SciPy community project soon >> after. >> >> So with this list let us start working out what the project should be like. >> >> For noobs like me, Gael's talk was quite a useful guide. Here's the >> link to a gist he made about it - https://gist.github.com/1433151 >> >> Here's the link to my SciPy talk: >> http://urtalk.kpoint.in/kapsule/gcc-57b6c86b-2f12-4244-950c-a34360a2cc1f/view/search/tag%3Ascipy >> >> I personally am researching nonlinear and nonstationary signal >> processing, I'd love to know what others can bring to this project. >> Also, let's talk about the limitations of the current signal >> processing tools available in SciPy and other scikits. I think there's >> a lot of documentation to be worked out, and there is also a lack of >> physically meaningful examples in the documentation. > > I think it would be a good addition to the ecosystem. We are for > example missing a lot of core algorithms even for linear signal > processing, and scipy.signal itself would benefit from some > refactoring. > > I myself started something (the talkbox scikit), but realistically, I > won't have time to work on a full toolbox, so we can consolidate here. > I would be willing to work on merging what I already have in what you > have in mind: > ?- Linear prediction coding with Levinson Durbin implementation > ?- A start of periodogram function > > ?I could spend time to implement a few more things like MUSIC/PENCIL, > and some basic matching pursuit algorithms. depending on your scope, nitime will also be interesting which has much more in terms of multi-dimensional signals, e.g. a multivariate Levinson Durbin and various cross-spectral functions. statsmodels has quite a bit of time series analysis now, but the focus and datasets are pretty different, although I benefited from reading the scipy.signal, talkbox and matplotlib codes for some basic tools. Josef > > cheers, > > David > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From alexandre.gramfort at inria.fr Wed Dec 28 13:53:20 2011 From: alexandre.gramfort at inria.fr (Alexandre Gramfort) Date: Wed, 28 Dec 2011 19:53:20 +0100 Subject: [SciPy-Dev] Proposal for Scikit-Signal - a SciPy toolbox for signal processing In-Reply-To: References: Message-ID: > depending on your scope, nitime will also be interesting which has > much more in terms of multi-dimensional signals, e.g. a multivariate > Levinson Durbin and various cross-spectral functions. I fully agree. nitime hides some great pieces of code like slepian tapers and various cross-sprectrum/coherence tools. I feel this code should be visible to a broader audience which would favor cross-fertilization between scientific domains and help nitime reach a critical mass of users/contributors. Alex From ndbecker2 at gmail.com Thu Dec 29 09:52:47 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 29 Dec 2011 09:52:47 -0500 Subject: [SciPy-Dev] Proposal for Scikit-Signal - a SciPy toolbox for signal processing References: Message-ID: David Cournapeau wrote: > Hi Jaidev, > > On Mon, Dec 26, 2011 at 5:39 PM, Jaidev Deshpande > wrote: >> Hi >> >> I gave a talk at SciPy India 2011 about a Python implementation of the >> Hilbert-Huang Transform that I was working on. The HHT is a method >> used as an alternative to Fourier and Wavelet analyses of nonlinear >> and nonstationary data. Following the talk Gael Varoquaux said that >> there's room for a separate scikit for signal processing. He also gave >> a lightning talk about bootstrapping a SciPy community project soon >> after. >> >> So with this list let us start working out what the project should be like. >> >> For noobs like me, Gael's talk was quite a useful guide. Here's the >> link to a gist he made about it - https://gist.github.com/1433151 >> >> Here's the link to my SciPy talk: >> http://urtalk.kpoint.in/kapsule/gcc-57b6c86b-2f12-4244-950c- a34360a2cc1f/view/search/tag%3Ascipy >> >> I personally am researching nonlinear and nonstationary signal >> processing, I'd love to know what others can bring to this project. >> Also, let's talk about the limitations of the current signal >> processing tools available in SciPy and other scikits. I think there's >> a lot of documentation to be worked out, and there is also a lack of >> physically meaningful examples in the documentation. > > I think it would be a good addition to the ecosystem. We are for > example missing a lot of core algorithms even for linear signal > processing, and scipy.signal itself would benefit from some > refactoring. > > I myself started something (the talkbox scikit), but realistically, I > won't have time to work on a full toolbox, so we can consolidate here. > I would be willing to work on merging what I already have in what you > have in mind: > - Linear prediction coding with Levinson Durbin implementation > - A start of periodogram function > > I could spend time to implement a few more things like MUSIC/PENCIL, > and some basic matching pursuit algorithms. > > cheers, > > David I have code for periodogram that you can use. I'm using my own fft wrapper around fftw (via pyublas), but you can easily modify this to use some other fft library. -------------- next part -------------- A non-text attachment was scrubbed... Name: spectral_est.py Type: text/x-python Size: 2705 bytes Desc: not available URL: From ralf.gommers at googlemail.com Fri Dec 30 13:54:45 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 30 Dec 2011 19:54:45 +0100 Subject: [SciPy-Dev] gaussian_kde changes - review request Message-ID: Hi all, At https://github.com/scipy/scipy/pull/123 I've submitted some changes to stats.gaussian_kde. Since it turned out to be relatively tricky to not break subclasses that can be found in various places (ML attachments, StackOverflow, etc.), I'm asking for a review here. Especially if you do have code that subclasses it, please try these changes. I've also written a tutorial that can be added to the docs, comments/additions welcome: https://gist.github.com/1534517 Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Dec 30 14:27:14 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 30 Dec 2011 19:27:14 +0000 Subject: [SciPy-Dev] gaussian_kde changes - review request In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 18:54, Ralf Gommers wrote: > Hi all, > > At https://github.com/scipy/scipy/pull/123 I've submitted some changes to > stats.gaussian_kde. Since it turned out to be relatively tricky to not break > subclasses that can be found in various places (ML attachments, > StackOverflow, etc.), I'm asking for a review here. Especially if you do > have code that subclasses it, please try these changes. > > I've also written a tutorial that can be added to the docs, > comments/additions welcome: https://gist.github.com/1534517 +1 for both. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From fperez.net at gmail.com Fri Dec 30 23:39:35 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 30 Dec 2011 20:39:35 -0800 Subject: [SciPy-Dev] ANN: EPD 7.2 released In-Reply-To: References: Message-ID: On Fri, Dec 23, 2011 at 10:39 AM, Ilan Schnell wrote: > I am pleased to announce the release of Enthought Python Distribution, EPD > version 7.2, along with its "EPD Free" counterpart. ?The highlights of this > release are: the addition of GDAL and updates to over 30 packages, including > SciPy, matplotlib and IPython. ?The new IPython 0.12 includes the HTML > notebook, which caused the Tornado web-server also to be added to EPD. Many thanks to Ilan and the rest of the team for this!!! A great Christmas present for us :) Happy New Year, f From ilanschnell at gmail.com Sat Dec 31 00:09:52 2011 From: ilanschnell at gmail.com (Ilan Schnell) Date: Fri, 30 Dec 2011 23:09:52 -0600 Subject: [SciPy-Dev] ANN: EPD 7.2 released In-Reply-To: References: Message-ID: It is my pleasure. Happy New Year - Ilan On Fri, Dec 30, 2011 at 10:39 PM, Fernando Perez wrote: > On Fri, Dec 23, 2011 at 10:39 AM, Ilan Schnell wrote: >> I am pleased to announce the release of Enthought Python Distribution, EPD >> version 7.2, along with its "EPD Free" counterpart. ?The highlights of this >> release are: the addition of GDAL and updates to over 30 packages, including >> SciPy, matplotlib and IPython. ?The new IPython 0.12 includes the HTML >> notebook, which caused the Tornado web-server also to be added to EPD. > > Many thanks to Ilan and the rest of the team for this!!! ?A great > Christmas present for us :) > > Happy New Year, > > f > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From ralf.gommers at googlemail.com Sat Dec 31 05:36:08 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 31 Dec 2011 11:36:08 +0100 Subject: [SciPy-Dev] gaussian_kde changes - review request In-Reply-To: References: Message-ID: On Fri, Dec 30, 2011 at 8:27 PM, Robert Kern wrote: > On Fri, Dec 30, 2011 at 18:54, Ralf Gommers > wrote: > > Hi all, > > > > At https://github.com/scipy/scipy/pull/123 I've submitted some changes > to > > stats.gaussian_kde. Since it turned out to be relatively tricky to not > break > > subclasses that can be found in various places (ML attachments, > > StackOverflow, etc.), I'm asking for a review here. Especially if you do > > have code that subclasses it, please try these changes. > > > > I've also written a tutorial that can be added to the docs, > > comments/additions welcome: https://gist.github.com/1534517 > > +1 for both. > As the original author, do you have a specific reference on which the implementation was based? The references I added didn't really address the multivariate case. Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Dec 31 05:44:46 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 31 Dec 2011 10:44:46 +0000 Subject: [SciPy-Dev] gaussian_kde changes - review request In-Reply-To: References: Message-ID: On Sat, Dec 31, 2011 at 10:36, Ralf Gommers wrote: > > On Fri, Dec 30, 2011 at 8:27 PM, Robert Kern wrote: >> >> On Fri, Dec 30, 2011 at 18:54, Ralf Gommers >> wrote: >> > Hi all, >> > >> > At https://github.com/scipy/scipy/pull/123 I've submitted some changes >> > to >> > stats.gaussian_kde. Since it turned out to be relatively tricky to not >> > break >> > subclasses that can be found in various places (ML attachments, >> > StackOverflow, etc.), I'm asking for a review here. Especially if you do >> > have code that subclasses it, please try these changes. >> > >> > I've also written a tutorial that can be added to the docs, >> > comments/additions welcome: https://gist.github.com/1534517 >> >> +1 for both. > > > As the original author, do you have a specific reference on which the > implementation was based? The references I added didn't really address the > multivariate case. I don't exactly know what I was looking at at the time, but here is a reference that reproduces the formulae. Go down to Section 3.6.2.1. http://fedc.wiwi.hu-berlin.de/xplore/ebooks/html/spm/spmhtmlnode18.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ralf.gommers at googlemail.com Sat Dec 31 05:50:11 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 31 Dec 2011 11:50:11 +0100 Subject: [SciPy-Dev] gaussian_kde changes - review request In-Reply-To: References: Message-ID: On Sat, Dec 31, 2011 at 11:44 AM, Robert Kern wrote: > On Sat, Dec 31, 2011 at 10:36, Ralf Gommers > wrote: > > > > On Fri, Dec 30, 2011 at 8:27 PM, Robert Kern > wrote: > >> > >> On Fri, Dec 30, 2011 at 18:54, Ralf Gommers < > ralf.gommers at googlemail.com> > >> wrote: > >> > Hi all, > >> > > >> > At https://github.com/scipy/scipy/pull/123 I've submitted some > changes > >> > to > >> > stats.gaussian_kde. Since it turned out to be relatively tricky to not > >> > break > >> > subclasses that can be found in various places (ML attachments, > >> > StackOverflow, etc.), I'm asking for a review here. Especially if you > do > >> > have code that subclasses it, please try these changes. > >> > > >> > I've also written a tutorial that can be added to the docs, > >> > comments/additions welcome: https://gist.github.com/1534517 > >> > >> +1 for both. > > > > > > As the original author, do you have a specific reference on which the > > implementation was based? The references I added didn't really address > the > > multivariate case. > > I don't exactly know what I was looking at at the time, but here is a > reference that reproduces the formulae. Go down to Section 3.6.2.1. > > http://fedc.wiwi.hu-berlin.de/xplore/ebooks/html/spm/spmhtmlnode18.html > > Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: