From arranger1044 at gmail.com Wed Oct 1 03:26:52 2014 From: arranger1044 at gmail.com (antonio vergari) Date: Wed, 1 Oct 2014 09:26:52 +0200 Subject: [SciPy-User] Fwd: scipy.cluster.spectral_clustering numerical issues In-Reply-To: References: Message-ID: Hi everyone, I'm trying to use the implementation of the spectral clustering algorithm found in the scipy.cluster package on a quite large data matrix (~16000 rows and 16 columns, binary data). I am also trying to give the algorithm an already computed affinity matrix representing a gaussian kernel (like in Andrew Ng's paper http://ai.stanford.edu/~ang/papers/nips01-spectral.pdf). I am not able to user the default eigensolver 'arpack' since I do not have enough memory and it starts constantly swapping (I have 8 Gb, but the matrix alone is almost 2Gb). Turning to the other alternative, 'lobpcg', I've found that errors arise. These are numerical errors spanning from a 'Not Xth minor semi definite' to the presence of Nans and 0s in the computation while I vary the value of the k clusters to find and the sigma value in the kernel (the variance). I am wondering if I lack some piece of theory and/or computing the similarity matrix in a wrong way. I do not appear to get these errors on smaller, randomly generate, binary matrices. Here is a printed matrix for sigma=3.0: [[ 1. 0.60653066 0.94595947 ..., 0.71653131 0.8007374 0.57375342] [ 0.60653066 1. 0.57375342 ..., 0.67780958 0.67780958 0.84648172] [ 0.94595947 0.57375342 1. ..., 0.75746513 0.75746513 0.60653066] ..., [ 0.71653131 0.67780958 0.75746513 ..., 1. 0.71653131 0.71653131] [ 0.8007374 0.67780958 0.75746513 ..., 0.71653131 1. 0.64118039] [ 0.57375342 0.84648172 0.60653066 ..., 0.71653131 0.64118039 1. ]] I am attaching also a piece of code and the data as a minimal working example. Thanks in advance -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: spec_mwe.zip Type: application/zip Size: 48174 bytes Desc: not available URL: From davidmenhur at gmail.com Wed Oct 1 05:18:37 2014 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Wed, 1 Oct 2014 11:18:37 +0200 Subject: [SciPy-User] Tests filling up RAM on Linux Message-ID: Hi, Running the scipy tests I get a sudden increase in memory usage, using more than the 16 GB or RAM I have, and thus freezing the computer for a few seconds twice, along with several test failures. It is reproducible every time. I have attached the output of the tests: $ nosetests scipy > test_stdout 2> test_stderr I have also monitored the input with vmstat: $ vmstat -n 1 > testing_vmstat The first freeze occurs between 130 and 160 s, and the second, shorter, occurs I believe at 350 s. Scipy was installed via pip on a Fedora 20 box and linked against ATLAS compiled from the official rpm. In [2]: scipy.__version__ Out[2]: '0.14.0' In [3]: scipy.show_config() atlas_threads_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas-sse3'] define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] language = f77 include_dirs = ['/usr/include'] blas_opt_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas-sse3'] define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] language = c include_dirs = ['/usr/include'] atlas_blas_threads_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas-sse3'] define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] language = c include_dirs = ['/usr/include'] openblas_info: NOT AVAILABLE lapack_opt_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas-sse3'] define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] language = f77 include_dirs = ['/usr/include'] openblas_lapack_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE mkl_info: NOT AVAILABLE $ python Python 2.7.5 (default, Sep 25 2014, 13:57:38) [GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Fedora has updated Python today [1], and I don't remember this happening before; but I am not sure. My laptop has the same configuration, but with Scipy 0.15.0.dev-704b554 and the tests don't use more than a 4% of the memory. /David. [1] http://koji.fedoraproject.org/koji/buildinfo?buildID=580875 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_stderr Type: application/octet-stream Size: 520673 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_stdout Type: application/octet-stream Size: 196958 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: plotter.py Type: text/x-python Size: 826 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testing_vmstat Type: application/octet-stream Size: 67694 bytes Desc: not available URL: From njs at pobox.com Wed Oct 1 10:52:47 2014 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 1 Oct 2014 15:52:47 +0100 Subject: [SciPy-User] Tests filling up RAM on Linux In-Reply-To: References: Message-ID: On 1 Oct 2014 10:32, "Da?id" wrote: > > Hi, > > Running the scipy tests I get a sudden increase in memory usage, using more than the 16 GB or RAM I have, and thus freezing the computer for a few seconds twice, along with several test failures. It is reproducible every time. I have attached the output of the tests: > > $ nosetests scipy > test_stdout 2> test_stderr Passing -v to nosetests will show you the name of each test as it executes, which may help you figure out which tests are causing the freezes. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From maria-rosaria.antonelli at curie.fr Wed Oct 1 10:56:46 2014 From: maria-rosaria.antonelli at curie.fr (Antonelli Maria Rosaria) Date: Wed, 1 Oct 2014 14:56:46 +0000 Subject: [SciPy-User] Virtual camera Message-ID: Hi, I would like to realise ? virtual camera in order to simulate the behaviour of an existing camera . Do you have any suggestion or idea around that ? Is there anything already available ? Where should I start with ? Best regards, Rosa -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jerome.Kieffer at esrf.fr Wed Oct 1 12:25:38 2014 From: Jerome.Kieffer at esrf.fr (Jerome Kieffer) Date: Wed, 1 Oct 2014 18:25:38 +0200 Subject: [SciPy-User] Virtual camera In-Reply-To: References: Message-ID: <20141001182538.3f68e1a7310f365b2881378f@esrf.fr> Hi Maria Rosaria, In OpenCV there is a "camera" object which can directly be linked to any webcam or more sophisticated firewire camera. This is very simplistic usage: https://github.com/kif/pycctv/blob/master/src/image.py Cheers, -- J?r?me Kieffer tel +33 476 882 445 From yw5aj at virginia.edu Wed Oct 1 13:36:18 2014 From: yw5aj at virginia.edu (Yuxiang Wang) Date: Wed, 1 Oct 2014 13:36:18 -0400 Subject: [SciPy-User] Does SciPy already supersed NumPy? In-Reply-To: References: Message-ID: I was reading through the link and it mentioned that the first form is preferred over the second one under most circumstances: # first form from scipy import stats stats.lomax(...) # second form from scipy.stats import distributions distributions.lomax(...) I honestly think the second form is far more frequently used in the examples given in the document, for example: http://docs.scipy.org/doc/scipy/reference/tutorial/special.html http://docs.scipy.org/doc/scipy/reference/tutorial/integrate.html http://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html http://docs.scipy.org/doc/scipy/reference/tutorial/fftpack.html Would you think it'd be better to update this link http://docs.scipy.org/doc/scipy/reference/api.html#guidelines-for-importing-functions-from-scipy ? -Shawn On Tue, Sep 30, 2014 at 11:18 AM, Robert Kern wrote: > On Mon, Sep 29, 2014 at 5:49 PM, Sergio Rojas wrote: >> In the not so long past we were suppose to do: >> >>>>> import numpy as np >>>>> anumpy = np.arange(15).reshape(3, 5) >>>>> anumpy >> array([[ 0, 1, 2, 3, 4], >> [ 5, 6, 7, 8, 9], >> [10, 11, 12, 13, 14]]) >>>>> anumpy.shape >> (3, 5) >>>>> anumpy.ndim >> 2 >>>>> np.array([6, 7, 8]) >> array([6, 7, 8]) >> >> Now, one can also do: >> >>>>> import scipy as sp >>>>> sp.__version__ >> '0.13.3' >>>>> ascipy = sp.arange(15).reshape(3, 5) >>>>> ascipy >> array([[ 0, 1, 2, 3, 4], >> [ 5, 6, 7, 8, 9], >> [10, 11, 12, 13, 14]]) >>>>> ascipy.shape >> (3, 5) >>>>> ascipy.ndim >> 2 >>>>> sp.array([6, 7, 8]) >> array([6, 7, 8]) >> >> Is it recomended to do so? >> >> Does this means that SciPy has overloaded NumPy functions or that SciPy >> superseding NumPy? > > No. See this section of the documentation: > > http://docs.scipy.org/doc/scipy/reference/api.html#guidelines-for-importing-functions-from-scipy > > -- > Robert Kern > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj at virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/ From andrea.gavana at gmail.com Wed Oct 1 15:39:07 2014 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Wed, 1 Oct 2014 21:39:07 +0200 Subject: [SciPy-User] AMPGO - Take 3 Message-ID: So, after receiving a bunch of requests for the AMPGO global optimization code and the associated benchmarks, I decided to put what I have on the public domain. I don't have that much time to work on it myself *but* I would still appreciate bug reports and feature requests. The AMPGO algorithm code and the 184 ND test functions - plus 20 1D test functions - live here: https://code.google.com/p/ampgo/ The algorithm itself can be tuned to use various local optimizers from the SciPy stack - and also from the OpenOpt stack, although the overall performances of the OpenOpt algorithms ranges between poor and abysmal - with the exception of the BOBYQA algorithm, and I'd say it's high time whoever is developing SciPy should include it as numerical optimization option. The AMPGO code as it stands is *not* suitable for inclusion into SciPy: the docstrings are almost non-existent, the return values are probably not compatible with the SciPy standards and in any case I am using SciPy 0.12.b1 still. Some time ago a kind soul provided me with the analytic definition of the gradient of the Tunnelling function inside AMPGO but I never managed to code it. This will give a great boost to AMPGO for problems with readily available gradient information. My main page of results and conclusions about global optimization algorithms available in Python is here: http://infinity77.net/global_optimization/ All the conclusions and results and ranking criteria are based on the *number of function evaluations*: everything else- like CPU time and internal algorithm performances, has been scrapped in the most brutal and merciless way. Most of us deal with real life problems where a single function evaluation (a simulation) may take hours to complete, so a few milliseconds gain in the internal algorithm processing feels like a joke. That,of course, has not stopped any of the "big" Global Optimization Projects (see COCONUT and funny friends), nor any of the reported benchmarks for "famous" solvers (BARON, I'm looking at you) from using ridiculous CPU-time metrics. Comments and suggestions are most welcome - especially if they help improving the overall performance of the AMPGO code (Tunnelling function gradient is a good start) or fixing bugs in the test functions - I'm sure there are a few lurking by, it's not that easy to stay focused while coding so much numerical optimization bruhaha. -- Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://www.infinity77.net # ------------------------------------------------------------- # def ask_mailing_list_support(email): if mention_platform_and_version() and include_sample_app(): send_message(email) else: install_malware() erase_hard_drives() # ------------------------------------------------------------- # -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergio_r at mail.com Thu Oct 2 18:35:28 2014 From: sergio_r at mail.com (Sergio Rojas) Date: Fri, 3 Oct 2014 00:35:28 +0200 Subject: [SciPy-User] On using scipy.ndimage.correlate to find common pattern in an image Message-ID: An HTML attachment was scrubbed... URL: From tsyu80 at gmail.com Fri Oct 3 00:38:16 2014 From: tsyu80 at gmail.com (Tony Yu) Date: Thu, 2 Oct 2014 23:38:16 -0500 Subject: [SciPy-User] On using scipy.ndimage.correlate to find common pattern in an image In-Reply-To: References: Message-ID: On Thu, Oct 2, 2014 at 5:35 PM, Sergio Rojas wrote: > > Hello, > > I'm trying to use the "correlate" function from the > "scipy.ndimage" module to find a common pattern in an image > (just in case, I am not trying to do character recognition per se, > just a warmout for it). > > The way I am doing so (see code attached below) > is via loading the image as an ndarray, > locate the pattern in the image and then applying the > "correlate" function in the hope that the returned ndarray > could show locations of the pattern in the image. > > Apparently the function did that. It is returning an ndarray > but I am having problems interpreting what is > returned by the function. > > Wondering if somebody could shed light on it? > Hi Sergio, There are a few pretty subtle points here that are making the result look really unexpected. - Using an RGB image can be confusing since each color channel may correlate slightly differently, and the correlation output is an RGB image (which means that the correlation for each channel is combined to produce an image that may not be intuitive). It'll be easier to look at a grayscale version of the image or a single color channel (this is done in the example below). - Unless you do a lot of image processing or low-level-ish coding, you probably don't deal with uint8 values that often. uint8 is great for storage but terrible for math. Well, that's assuming you ever want to subtract something. Or add/multiply values that produce a value > 255. The example below converts to float before doing any further computation. - Finally, a simple correlation can have surprising results. If the image patch you're looking for (the "target" image, in the code below) has values between 0 and 1 (black and white, respectively), then you'll get the maximum correlation for an all-white patch in the main image. It doesn't matter what the target image is: If you multiple all values in the target by 1 vs multiplying the target by a patch with values between 0 and 1, then the patch with all 1s will produce a larger result. A simple way to mitigate this effect is normalize your values so black is -1 and white is 1. With that in place, two black pixels give a large positive product, and two white pixels give a large positive product. On the other hand, a white and black pixel give a large negative product. So that's the basic idea. I've modified your script to take these ideas into account (see below). That said, if you want to play around with image processing, you should give scikit-image a try: http://scikit-image.org/. For this particular problem, you should play around with template matching, which is basically a normalize cross-correlation: http://scikit-image.org/docs/dev/auto_examples/plot_template.html Best, -Tony #~~~ Example code: import scipy import scipy.ndimage import scipy.misc import matplotlib.pyplot as plt import urllib from cStringIO import StringIO def get_image(): URL = 'http://i62.tinypic.com/jtryif.png' image_buffer = StringIO(urllib.urlopen(URL).read()) # Run correlation on a single color channel image = scipy.ndimage.imread(image_buffer)[:, :, 0] # Convert uint8 to floats between 0 and 1 (defacto standard float range) # since unsigned integers aren't great for operations that might go # negative. image = image.astype(float) / 255 # Rescale so that black is -1 and white is 1 image = image * 2 - 1 return image def plot_match(image, target, match_likelihood): fig, axes = plt.subplots(nrows=2, ncols=2) ax1, ax2, ax3, ax4 = axes.flatten() ax1.set_title('original') ax1.imshow(image, cmap=plt.cm.gray, interpolation='nearest') ax2.set_title('target patch') ax2.imshow(target, cmap=plt.cm.gray, interpolation='nearest') ax3.set_title('correlation') ax3.imshow(match_likelihood, cmap=plt.cm.gray, interpolation='nearest') ax4.set_title('match_likelihood==match_likelihood.max()') ax4.imshow(match_likelihood>=(0.8 * match_likelihood.max()), cmap=plt.cm.gray, interpolation='nearest') for ax in axes.flatten(): ax.set_axis_off() image = get_image() target = image[38:64, 233:298] match_likelihood = scipy.ndimage.correlate(image, target) plot_match(image, target, match_likelihood) plt.show() > > Thanks in advance, > > Sergio > > PS. In case the code does not look right, I made a link to > it at: https://gist.github.com/anonymous/40083f0795de5084a468 > > import scipy > import scipy.ndimage > import scipy.misc > import matplotlib.pyplot as plt > > # Read the image from web > # ----------------------- > import urllib > from cStringIO import StringIO > URL = 'http://i62.tinypic.com/jtryif.png' > thefile = StringIO(urllib.urlopen(URL).read()) > > img = scipy.ndimage.imread(thefile) > #thefile= 'scipy_text.png' > #img=scipy.ndimage.imread(thefile) > plt.imshow(img) #Needs to be in row,col order > plt.show() > #------ locate the patten to find in the image > patternTofind = img[38:64,233:298] > plt.imshow(patternTofind) > plt.show() > #------ apply correlate > found = scipy.ndimage.correlate(img,patternTofind) > plt.figure() > plt.subplot(321) > plt.title('img') > plt.imshow(img, cmap=plt.cm.gray, interpolation='nearest') > plt.axis('off') > # > plt.subplot(322) > plt.title('patternTofind') > plt.imshow(patternTofind, cmap=plt.cm.gray, interpolation='nearest') > plt.axis('off') > # > plt.subplot(312) > plt.imshow(found, cmap=plt.cm.gray, interpolation='nearest') > plt.title('found = scipy.ndimage.correlate(img,patternTofind)') > plt.axis('off') > # > plt.subplot(313) > plt.imshow(found==found.max(), cmap=plt.cm.gray, interpolation='nearest') > plt.title('found==found.max()') > plt.axis('off') > plt.show() > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergio_r at mail.com Sat Oct 4 15:22:33 2014 From: sergio_r at mail.com (Sergio Rojas) Date: Sat, 4 Oct 2014 21:22:33 +0200 Subject: [SciPy-User] On using scipy.ndimage.correlate to find common pattern in an image In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From bhmerchant at gmail.com Sun Oct 5 18:40:40 2014 From: bhmerchant at gmail.com (Brian Merchant) Date: Sun, 5 Oct 2014 15:40:40 -0700 Subject: [SciPy-User] Novice scipy user: having some efficiency issues with using scipy.integrate.odeint for my research project, and not sure how to proceed... Message-ID: Just to give you a bit regarding my background: I am a math undergraduate working on biology modelling problems. I have a strong background in Python, but it is only over the last few months that I have begun to dabble in scientific computation using scipy, numpy and most recently, Cython. This semester, I am taking my first proper course on numerical methods. Currently, in my research work, I am writing a program that solves a large number of non-linear ODEs. I use `scipy.integrate.odeint` as my workhorse, although I have dabbled with using `pyDSTool` as well, but found that the limited capacity I had in understanding advanced methods didn't allow me to use `pyDSTool` so that I had better performance than vanilla `scipy.integrate.odeint`. My supervisor recently pointed out a fairly major error in my previous implementation of the program: I was "breaking up" `scipy.integrate.odeint`'s flow. So if I had to solve a problem between times 0 and 10, I would make `odeint` work between `t in [0, 1]`, take the result at `t=1`, use it to update (non-linear) parameters for the next interval (`[1, 2]`) and then finally run the next interval using `t=1` results as the initial conditions. I thought it would be a good idea to do this in order to temporarily reduce the complexity of the problem by having some of the coefficients be constant over each sub-interval (something like a "quasi-equilibrium" condition?), and it also allowed me to straightforwardly implement non-DE rules to update parameters, if I needed to. However, problems with doing this include introducing errors, and causing loss of speed due to the breaking/restarting of `odeint`'s internals. So, I changed the implementation so that everything was done in "one-shot" (luckily the only non-DE rule I had so far was easily converted to a DE rule, although there is one coming up which I don't see any easy conversion for...[that is the subject of this question]( http://scicomp.stackexchange.com/questions/14765/scipy-integrate-odeint-how-can-odeint-access-a-parameter-set-that-is-evolving-i) ). However, I am finding that the "one-shot" method is *slower* than the "broken up" method! For a single cell (the number of ODEs increases linearly with increasing number of cells), solving for the interval between `[0, 1]` takes about 3 times longer than the "broken up method" (where I was breaking up the problem into intervals `[0, 0.5]` and `[0.5, 1]` for the same `rtol` and `atol`. Solving for the behaviour of a single cell over `[0, 10]` takes roughly 3 times longer than `[0, 1]` (hey! not bad!). Solving between `[0, 1000]` takes roughly 1500 times longer (weird :( ). Solving 2 cells (and solving between `[0,1]`) takes about 3 times longer than the single cell case, and solving 10 cells over the same time interval takes 60 times longer! So the times aren't growing linearly...whether I am increasing the length of the time interval, or increasing the number of cells. Alright, part of the problem can be explained by the fact that even though the number of ODEs grows linearly with the number of cells, the work done to calculate the Jacobian doesn't grow quite grow linearly -- still, in all cases, the "one shot" method is significantly slower than the "broken up" method (and the greater than linear effort required in calculating the Jacobian also affected the "broken up" method). Also, the Jacobian doesn't become harder to calculate when increasing the time interval...so shouldn't the time taken to solve the ODE grow linearly there at least? Alright, that's about all the certain information I have. My guess as to the number one culprit is that in the "one shot" method, the function used to calculate the Jacobian is doing a *lot* more work: recall, non-linear ODEs, so some internal iteration is required, and basically every iteration, all the coefficients are calculated again, while in the "broken up" method, I could choose which ones were held constant for a small time period, before recalculation. I thought reimplementing by making significant use of Cython (I used "annotation profiling" (`cython -a my_file.pyx`) to make sure I am cutting down on C lines wherever possible...so I am pretty sure I have not done a horribly ugly Python to C conversion. Still, even Cythonizing doesn't help with the fact that a lot more work is being done. I'm at a bit of a loss right now as to how to proceed - what questions should I begin asking myself in order to get a handle on things? -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Oct 5 23:08:22 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 5 Oct 2014 23:08:22 -0400 Subject: [SciPy-User] progress in scipy, special and stats Message-ID: Just a general thanks. Small things that make the life of a developer easier and improve numerical precision in corner cases. I just went over some statsmodels code that was written maybe 5 years ago with clumsy or missing 0*log(0) = 0 handling. scipy.special has been and is getting nice little (*) functions that handle some corner cases automatically and numerically in a more stable way. scipy.stats is one of the beneficiaries. Thanks to several maintainer/developers and contributors. ---- (*) little compared to the heavy stuff in scipy.special, but also useful (apropos heavy: 5 papers mention that there is no closed form solution for the pmf of a discrete distribution but we can calculate it using a recursion. The 6th article mentions it can be calculated with the modified Bessel function of the second kind. Why didn't they say so right away. I'd rather use scipy.special than figuring out how to do a numerically stable recursion.) Josef Watching from the sidelines. BTW: PIG or P-IG is not a good name for a discrete distribution if we care about google searchability. From jeffreback at gmail.com Tue Oct 7 08:30:52 2014 From: jeffreback at gmail.com (Jeff Reback) Date: Tue, 7 Oct 2014 08:30:52 -0400 Subject: [SciPy-User] ANN: Pandas 0.15.0 Release Candiate 1 Message-ID: Hi, I'm pleased to announce the availability of the first release candidate of Pandas 0.15.0. Please try this RC and report any issues here: Pandas Issues We will be releasing officially in 1-2 weeks or so. This is a major release from 0.14.1 and includes a number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. Highlights include: - Drop support for numpy < 1.7.0 - The Categorical type was integrated as a first-class pandas type - New scalar type Timedelta, and a new index type TimedeltaIndex - New DataFrame default display for df.info() to include memory usage - New datetimelike properties accessor .dt for Series - Split indexing documentation into Indexing and Selecting Data and MultiIndex / Advanced Indexing - Split out string methods documentation into Working with Text Data - read_csv will now by default ignore blank lines when parsing - API change in using Indexes in set operations - Internal refactoring of the Index class to no longer sub-class ndarray - dropping support for PyTables less than version 3.0.0, and numexpr less than version 2.1 Here are the full whatsnew and documentation links: v0.15.0 Whatsnew v0.15.0 Documentation Page Source tarballs, and windows builds are available here: Pandas v0.15.0rc1 Release A big thank you to everyone who contributed to this release! Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.clewley at gmail.com Wed Oct 8 11:26:28 2014 From: rob.clewley at gmail.com (Rob Clewley) Date: Wed, 8 Oct 2014 11:26:28 -0400 Subject: [SciPy-User] Novice scipy user: having some efficiency issues with using scipy.integrate.odeint for my research project, and not sure how to proceed... In-Reply-To: References: Message-ID: Hi Brian, I know we've talked about this before, but here are some more general suggestions about your problem. > > So, I changed the implementation so that everything was done in "one-shot" > (luckily the only non-DE rule I had so far was easily converted to a DE > rule, although there is one coming up which I don't see any easy conversion > for...[that is the subject of this > question](http://scicomp.stackexchange.com/questions/14765/scipy-integrate-odeint-how-can-odeint-access-a-parameter-set-that-is-evolving-i)). > Having looked at that, your problem is not so bad. You will need to solve a piecewise (a.k.a. "hybrid") system, but you shouldn't be discretizing it based on time but based on events -- i.e. when your particle crosses into a new discrete area you are calling p. You then have a discrete state update for your parameters and then you can restart your smooth integrator until the next transition. This is a well-studied problem in this form and is numerically soluble with a hybrid solver such as that provided by PyDSTool. Between positional transitions, your dynamical system is smooth and can be solved with a regular ODE solver. The Jacobian within each domain should be simple enough to pre-calculate (as a function of p or

). There are a few hybrid model examples with PyDSTool on the tutorial and I am willing to help a bit with the setup once you've given a new script for your problem a shot. Take a copy of a helpful example (e.g. the SLIP pogo stick dynamics) and adapt it to set up what you can. Put in some comments etc. of what you need to happen. PyDSTool will be able to convert the ODEs to C code automatically but not the transition rules, which will still happen in python. This will not be the *fastest* way to solve it but, more importantly, this way will give you an accurate solution in a form that you can understand and manipulate, and you can worry about optimizing speed later, IMO. -Rob From shaharb at stanford.edu Fri Oct 10 14:43:05 2014 From: shaharb at stanford.edu (Shahar Barak) Date: Fri, 10 Oct 2014 11:43:05 -0700 Subject: [SciPy-User] scipy.ndimage.filters.median_filter with NAN values Message-ID: I have a 3D array, in which some values are Nan (masked). I am using scipy.ndimage.filters.median_filter to smooth the data. Is there a way to have scipy.ndimage.filters.median_filter not include the NAN values in its median calculation? Thanks, Shahar -------------- next part -------------- An HTML attachment was scrubbed... URL: From parente at gmail.com Fri Oct 10 17:01:25 2014 From: parente at gmail.com (Peter Parente) Date: Fri, 10 Oct 2014 17:01:25 -0400 Subject: [SciPy-User] Numerical Recipes and scipy Message-ID: Hi all, I have a question about SciPy and its relation to the code from Numerical Recipes (http://www.nr.com/licenses/redistribute.html) I can see that both the SciPy FAQ and code comments today both contain references to the book. I noticed that in 2008 there was a discussion about removing SciPy code derived from the book to avoid infringing on the NR license. ( http://mail.scipy.org/pipermail/scipy-dev/2008-December/010446.html) I see that some code was identified and possibly replaced, but there is no distinct end to the activity. Does anyone have an understanding they can share about where SciPy stands today with respect to NR and its license terms? Thanks, Pete (I'm sorry if this is a dupe. I had a problem posting from my subscribed email address.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Oct 10 17:07:58 2014 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 10 Oct 2014 22:07:58 +0100 Subject: [SciPy-User] Numerical Recipes and scipy In-Reply-To: References: Message-ID: On Fri, Oct 10, 2014 at 10:01 PM, Peter Parente wrote: > Hi all, > > I have a question about SciPy and its relation to the code from Numerical > Recipes (http://www.nr.com/licenses/redistribute.html) > > I can see that both the SciPy FAQ and code comments today both contain > references to the book. I noticed that in 2008 there was a discussion about > removing SciPy code derived from the book to avoid infringing on the NR > license. > (http://mail.scipy.org/pipermail/scipy-dev/2008-December/010446.html) > > I see that some code was identified and possibly replaced, but there is no > distinct end to the activity. Does anyone have an understanding they can > share about where SciPy stands today with respect to NR and its license > terms? There is no NR code in scipy. -- Robert Kern From jzuhone at gmail.com Fri Oct 10 17:10:05 2014 From: jzuhone at gmail.com (John ZuHone) Date: Fri, 10 Oct 2014 17:10:05 -0400 Subject: [SciPy-User] Numerical Recipes and scipy In-Reply-To: References: Message-ID: <64D6EC85-C871-40BA-A2C1-80489DEC45CA@gmail.com> I for one am curious about the motivation behind the question. On Oct 10, 2014, at 5:01 PM, Peter Parente wrote: > Hi all, > > I have a question about SciPy and its relation to the code from Numerical Recipes (http://www.nr.com/licenses/redistribute.html) > > I can see that both the SciPy FAQ and code comments today both contain references to the book. I noticed that in 2008 there was a discussion about removing SciPy code derived from the book to avoid infringing on the NR license. (http://mail.scipy.org/pipermail/scipy-dev/2008-December/010446.html) > > I see that some code was identified and possibly replaced, but there is no distinct end to the activity. Does anyone have an understanding they can share about where SciPy stands today with respect to NR and its license terms? > > Thanks, > Pete > > (I'm sorry if this is a dupe. I had a problem posting from my subscribed email address.) > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From joseph.martinot-lagarde at m4x.org Fri Oct 10 17:52:41 2014 From: joseph.martinot-lagarde at m4x.org (Joseph Martinot-Lagarde) Date: Fri, 10 Oct 2014 23:52:41 +0200 Subject: [SciPy-User] scipy.ndimage.filters.median_filter with NAN values In-Reply-To: References: Message-ID: <54385529.7070102@m4x.org> If I remember well, astropy has such a function. Le 10/10/2014 20:43, Shahar Barak a ?crit : > I have a 3D array, in which some values are Nan (masked). > I am using scipy.ndimage.filters.median_filter to smooth the data. > Is there a way to have scipy.ndimage.filters.median_filter not include > the NAN values in its median calculation? > > Thanks, > > Shahar > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From parente at gmail.com Fri Oct 10 21:18:45 2014 From: parente at gmail.com (Peter Parente) Date: Fri, 10 Oct 2014 21:18:45 -0400 Subject: [SciPy-User] Numerical Recipes and scipy In-Reply-To: <64D6EC85-C871-40BA-A2C1-80489DEC45CA@gmail.com> References: <64D6EC85-C871-40BA-A2C1-80489DEC45CA@gmail.com> Message-ID: I'm considering using SciPy in a commercial application. The SciPy code makes reference to the Numerical Recipes book in various places ( https://github.com/scipy/scipy/search?utf8=%E2%9C%93&q=numerical+recipes). The NR license terms restrict the translation and distribution of derivative works, especially for commercial purposes ( http://www.nr.com/licenses/redistribute.html). I polled the list to see if anyone could answer if the references are purely for educational purposes or provenance of the SciPy implementations. On Fri, Oct 10, 2014 at 5:10 PM, John ZuHone wrote: > I for one am curious about the motivation behind the question. > > On Oct 10, 2014, at 5:01 PM, Peter Parente wrote: > > Hi all, > > I have a question about SciPy and its relation to the code from Numerical > Recipes (http://www.nr.com/licenses/redistribute.html) > > I can see that both the SciPy FAQ and code comments today both contain > references to the book. I noticed that in 2008 there was a discussion about > removing SciPy code derived from the book to avoid infringing on the NR > license. ( > http://mail.scipy.org/pipermail/scipy-dev/2008-December/010446.html) > > I see that some code was identified and possibly replaced, but there is no > distinct end to the activity. Does anyone have an understanding they can > share about where SciPy stands today with respect to NR and its license > terms? > > Thanks, > Pete > > (I'm sorry if this is a dupe. I had a problem posting from my subscribed > email address.) > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > -- Pete -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Fri Oct 10 21:37:20 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 10 Oct 2014 18:37:20 -0700 Subject: [SciPy-User] Numerical Recipes and scipy In-Reply-To: References: <64D6EC85-C871-40BA-A2C1-80489DEC45CA@gmail.com> Message-ID: Hi, On Fri, Oct 10, 2014 at 6:18 PM, Peter Parente wrote: > I'm considering using SciPy in a commercial application. The SciPy code > makes reference to the Numerical Recipes book in various places > (https://github.com/scipy/scipy/search?utf8=%E2%9C%93&q=numerical+recipes). > The NR license terms restrict the translation and distribution of derivative > works, especially for commercial purposes > (http://www.nr.com/licenses/redistribute.html). I polled the list to see if > anyone could answer if the references are purely for educational purposes or > provenance of the SciPy implementations. Yes, it would be very useful to have a wiki page going through the various mentions of NR with a one-line record of the check made for NR code. Who did these checks originally? Does anyone remember? Cheers, Matthew From josef.pktd at gmail.com Fri Oct 10 21:54:15 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 10 Oct 2014 21:54:15 -0400 Subject: [SciPy-User] Numerical Recipes and scipy In-Reply-To: References: <64D6EC85-C871-40BA-A2C1-80489DEC45CA@gmail.com> Message-ID: On Fri, Oct 10, 2014 at 9:37 PM, Matthew Brett wrote: > Hi, > > On Fri, Oct 10, 2014 at 6:18 PM, Peter Parente wrote: >> I'm considering using SciPy in a commercial application. The SciPy code >> makes reference to the Numerical Recipes book in various places >> (https://github.com/scipy/scipy/search?utf8=%E2%9C%93&q=numerical+recipes). >> The NR license terms restrict the translation and distribution of derivative >> works, especially for commercial purposes >> (http://www.nr.com/licenses/redistribute.html). I polled the list to see if >> anyone could answer if the references are purely for educational purposes or >> provenance of the SciPy implementations. > > Yes, it would be very useful to have a wiki page going through the > various mentions of NR with a one-line record of the check made for NR > code. > > Who did these checks originally? Does anyone remember? In the mailing list thread that is linked to in the original post we went over those parts of scipy where code comments referenced NR and we checked those. ("we" are the participants on that thread.) I guess, based on the github search, and based on that the references to NR that still show up are all only in the "Reference" sections, that they are only part of the documentation but have no direct link to the actual implementation. I remember that there were at least some comments about this for specific items, but I don't know or remember any details. Josef > > Cheers, > > Matthew > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From josef.pktd at gmail.com Fri Oct 10 22:33:31 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 10 Oct 2014 22:33:31 -0400 Subject: [SciPy-User] Numerical Recipes and scipy In-Reply-To: References: <64D6EC85-C871-40BA-A2C1-80489DEC45CA@gmail.com> Message-ID: On Fri, Oct 10, 2014 at 9:54 PM, wrote: > On Fri, Oct 10, 2014 at 9:37 PM, Matthew Brett wrote: >> Hi, >> >> On Fri, Oct 10, 2014 at 6:18 PM, Peter Parente wrote: >>> I'm considering using SciPy in a commercial application. The SciPy code >>> makes reference to the Numerical Recipes book in various places >>> (https://github.com/scipy/scipy/search?utf8=%E2%9C%93&q=numerical+recipes). >>> The NR license terms restrict the translation and distribution of derivative >>> works, especially for commercial purposes >>> (http://www.nr.com/licenses/redistribute.html). I polled the list to see if >>> anyone could answer if the references are purely for educational purposes or >>> provenance of the SciPy implementations. >> >> Yes, it would be very useful to have a wiki page going through the >> various mentions of NR with a one-line record of the check made for NR >> code. >> >> Who did these checks originally? Does anyone remember? > > In the mailing list thread that is linked to in the original post we > went over those parts of scipy where code comments referenced NR and > we checked those. ("we" are the participants on that thread.) > I guess, based on the github search, and based on that the references > to NR that still show up are all only in the "Reference" sections, > that they are only part of the documentation but have no direct link > to the actual implementation. > > I remember that there were at least some comments about this for > specific items, but I don't know or remember any details. PyMatrixID and Savitsky-Golay is more recent code with reference to NR. the random numer generator fortran file in interpolative has NR as only reference. Josef > > Josef > > >> >> Cheers, >> >> Matthew >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user From warren.weckesser at gmail.com Fri Oct 10 22:34:16 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Fri, 10 Oct 2014 22:34:16 -0400 Subject: [SciPy-User] scipy.ndimage.filters.median_filter with NAN values In-Reply-To: References: Message-ID: On Fri, Oct 10, 2014 at 2:43 PM, Shahar Barak wrote: > I have a 3D array, in which some values are Nan (masked). > I am using scipy.ndimage.filters.median_filter to smooth the data. > Is there a way to have scipy.ndimage.filters.median_filter not include the > NAN values in its median calculation? > > You could use scipy.ndimage.filters.generic_filter, with np.nanmedian as the 'function' argument, but it will be painfully slow compared to median_filter. E.g. In [97]: x = np.random.randn(50, 50, 50) In [98]: %timeit median_filter(x, size=3) 10 loops, best of 3: 47.9 ms per loop In [99]: %timeit generic_filter(x, np.nanmedian, size=3) 1 loops, best of 3: 2.21 s per loop That's about 46 times slower for an array with shape (50, 50, 50). Warren Thanks, > > Shahar > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sat Oct 11 05:36:58 2014 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 11 Oct 2014 12:36:58 +0300 Subject: [SciPy-User] Numerical Recipes and scipy In-Reply-To: References: Message-ID: 11.10.2014, 00:01, Peter Parente kirjoitti: [clip] > I see that some code was identified and possibly replaced, but there is no > distinct end to the activity. Does anyone have an understanding they can > share about where SciPy stands today with respect to NR and its license > terms? I am aware of no NR code in Scipy. References to the book in the documentation are indeed meant to be educational and do not signify that the algorithms are implemented using code from the book. The review in 2008 checked all parts of the code where NR was referenced, and did not appear to have found any NR code or NR code translated/obfuscated to Python. Code in one routine was found to have some similarities in structure and was replaced with a reimplementation. -- Pauli Virtanen From pav at iki.fi Sat Oct 11 05:58:54 2014 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 11 Oct 2014 12:58:54 +0300 Subject: [SciPy-User] Numerical Recipes and scipy In-Reply-To: References: <64D6EC85-C871-40BA-A2C1-80489DEC45CA@gmail.com> Message-ID: 11.10.2014, 05:33, josef.pktd at gmail.com kirjoitti: [clip] > PyMatrixID and Savitsky-Golay is more recent code with reference to NR. The comment on numerical recipes in savgol_coeffs() is about how to reproduce results mentioned in NR. The code itself is obviously not from NR. [clip] > the random numer generator fortran file in interpolative has NR as > only reference. The algorithm in Numerical Recipes is originally from D. Knuth, The Art of Computer Programming, vol 2. The code in PyMatrixID is neither Knuth's code nor NR's code, but a reimplementation. The algorithm itself consists of 8 lines of code, so they share unavoidable similarities. -- Pauli Virtanen From tanguanzhong at huawei.com Sat Oct 11 06:11:00 2014 From: tanguanzhong at huawei.com (Tanguanzhong) Date: Sat, 11 Oct 2014 10:11:00 +0000 Subject: [SciPy-User] Hilbert transform Message-ID: Dear all, I have made a test for Hilbert transform for PCM data. Suggest "pcmRawData" is PCM audio data and collect by pyaudio. After Hilbert transform, "pcmData" has 90 degree phase shift relative to "pcmRawData". Because phase shift for PCM audio is not sensitive to Human-being. So I guess I should not find the difference for "pcmData" and "pcmRawData". But on the contrary, the test result is the audio data of "pcmData" is not good to hear. There are some distortions during the scipy Hilbert transform. Could anyone can give some advice for this? Is any problem for apply Hilbert transform for audio by Scipy? This is some code example. from scipy import fftpack ... pcmData = fftpack.hilbert(pcmRawData) -------------- next part -------------- An HTML attachment was scrubbed... URL: From faridafandiyev at gmail.com Sat Oct 11 08:31:22 2014 From: faridafandiyev at gmail.com (Farid Afandiyev) Date: Sat, 11 Oct 2014 16:31:22 +0400 Subject: [SciPy-User] odeint - Excess work done on this call Message-ID: Hello! I'm newbie both in calculus and python/scipy so I apologize if this question is too dumb. I'm trying to model flow between two pressure vessels. Let's say we have two points and a link between them like this [Vc_1, P_1]=====A====[Vc_2, P_2] Vc_1, Vc_2 are constants volumes of the nodes and P_1, P_2 varying pressure of the nodes respectively. I end up writing differential questions below. Never mind physical meaning I just want to get math correct. dP_1/dt = dVa/dt * 1/(Vc_1 * B) dP_2/dt = dVa/dt * 1/(Vc_2 * B) Here B is compressibility. dVa/dt = A * K * sqrt(P_1 - P_2) dVa is amount of "flow" between nodes. K is some constant coefficient and A is link "weight". (P_1-P_2) can change sign so I've adjusted this in the software. Below is python program that I wrote to evaluate this. #!/usr/bin/env python import math from scipy.integrate import odeint from time import time import numpy B_compressibility = 0.0000033 #water compressibility K = 0.747871759938 #coefficient Vc_1 = 20 Vc_2 = 50 A = 0.01 P_1 = 4000 P_2 = 2000 def deriv(state, t): _P_1 = state[0] _P_2 = state[2] diff_P = _P_1 - _P_2 flow_direction = math.copysign(1, diff_P) dVa = flow_direction * A * K * math.sqrt(abs(diff_P)) dP_1 = -(dVa/Vc_1)/B_compressibility dP_2 = (dVa/Vc_2)/B_compressibility return [dP_1, -dVa, dP_2, dVa] if __name__ == '__main__': Va_1 = Vc_1 * P_1 * B_compressibility Va_2 = Vc_2 * P_2 * B_compressibility odeIterations = 10 timeperiod = numpy.linspace(0.0,1.0, odeIterations) initial_state = [P_1, Va_1, P_2, Va_2] t0 = time() state_array = odeint(deriv, initial_state, timeperiod) t1 = time() print 'runtime %fs' %(t1-t0) state = state_array[odeIterations-1] print state_array P_1 = state[0] Va_1 = state[1] P_2 = state[2] Va_2 = state[3] Below is output from program Excess work done on this call (perhaps wrong Dfun type). Run with full_output = 1 to get quantitative information. runtime 0.157000s [[ 4.00000000e+03 2.64000000e-01 2.00000000e+03 3.30000000e-01] [ 3.49242034e+03 2.30499743e-01 2.20303186e+03 3.63500257e-01] [ 3.09580400e+03 2.04323064e-01 2.36167840e+03 3.89676936e-01] [ 2.81015098e+03 1.85469965e-01 2.47593961e+03 4.08530035e-01] [ 2.63546127e+03 1.73940444e-01 2.54581549e+03 4.20059556e-01] [ 2.57173487e+03 1.69734501e-01 2.57130605e+03 4.24265499e-01] [ 2.57142857e+03 1.69714286e-01 2.57142857e+03 4.24285714e-01] [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] lsoda-- at current t (=r1), mxstep (=i1) steps taken on this call before reaching tout In above message, I1 = 500 In above message, R1 = 0.6110315150411E+00 Odeint gives correct results up to 7th line and then something goes seriously wrong. I have searched in google and it looks like I'm not the only one who struggles with scipy. Everybody suggests increasing mxstep but that doesn't solve my problem. In addition it slows down the method significantly. Somebody suggested reducing accuracy but I don't know how to do that. Also I don't need super accuracy from the odeint. Couple of digits after dot is more than enough for me. Any help is greatly appreciated! Kind regards, Farid. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexlib at eng.tau.ac.il Sat Oct 11 09:36:34 2014 From: alexlib at eng.tau.ac.il (Alex Liberzon) Date: Sat, 11 Oct 2014 16:36:34 +0300 Subject: [SciPy-User] odeint - Excess work done on this cal Message-ID: Apparently, the problem is with the time step, see the linked IPython notebook http://nbviewer.ipython.org/gist/alexlib/5220b92868c800fd1700 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Sat Oct 11 11:15:47 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 11 Oct 2014 15:15:47 +0000 (UTC) Subject: [SciPy-User] Hilbert transform References: Message-ID: <7289532434733193.090328sturla.molden-gmail.com@news.gmane.org> Tanguanzhong wrote: > I have made a test for Hilbert transform for PCM data. Suggest > "pcmRawData" is PCM audio data and collect by pyaudio. After Hilbert > transform, "pcmData" has 90 degree phase shift relative to "pcmRawData". The Hilbert transform should return a complex signal, at least that is how it is defined. Sturla From parente at gmail.com Sun Oct 12 19:31:46 2014 From: parente at gmail.com (Peter Parente) Date: Sun, 12 Oct 2014 19:31:46 -0400 Subject: [SciPy-User] Numerical Recipes and scipy In-Reply-To: References: Message-ID: Thanks for the insight, Pauli. To save others from wondering or asking my question again, would it be worth adding to the FAQ or wiki as Matthew suggested? On Sat, Oct 11, 2014 at 5:36 AM, Pauli Virtanen wrote: > 11.10.2014, 00:01, Peter Parente kirjoitti: > [clip] > > I see that some code was identified and possibly replaced, but there is > no > > distinct end to the activity. Does anyone have an understanding they can > > share about where SciPy stands today with respect to NR and its license > > terms? > > I am aware of no NR code in Scipy. > > References to the book in the documentation are indeed meant to be > educational and do not signify that the algorithms are implemented using > code from the book. > > The review in 2008 checked all parts of the code where NR was > referenced, and did not appear to have found any NR code or NR code > translated/obfuscated to Python. Code in one routine was found to have > some similarities in structure and was replaced with a reimplementation. > > -- > Pauli Virtanen > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Pete -------------- next part -------------- An HTML attachment was scrubbed... URL: From tanguanzhong at huawei.com Sun Oct 12 20:55:09 2014 From: tanguanzhong at huawei.com (Tanguanzhong) Date: Mon, 13 Oct 2014 00:55:09 +0000 Subject: [SciPy-User] =?gb2312?b?tPC4tDogIEhpbGJlcnQgdHJhbnNmb3Jt?= In-Reply-To: <7289532434733193.090328sturla.molden-gmail.com@news.gmane.org> References: <7289532434733193.090328sturla.molden-gmail.com@news.gmane.org> Message-ID: From the Scipy manual: It seems not complex signal but the real part of complex signal? scipy.fftpack.hilbert(x) Return Hilbert transform of a periodic sequence x. If x_j and y_j are Fourier coefficients of periodic functions x and y, respectively, then y_j = sqrt(-1)*sign(j) * x_j y_0 = 0 Parameters x : array_like The input array, should be periodic. _cache : dict, optional Dictionary that contains the kernel used to do a convolution with. Returns y : ndarray The transformed input. Notes If sum(x, axis=0) == 0 then hilbert(ihilbert(x)) == x. For even len(x), the Nyquist mode of x is taken zero. The sign of the returned transform does not have a factor -1 that is more often than not found in the definition of the Hilbert transform. BOBBY.TAN/tanguanzhong at huawei.com ???: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] ?? Sturla Molden ????: 2014?10?11? 23:16 ???: scipy-user at scipy.org ??: Re: [SciPy-User] Hilbert transform Tanguanzhong wrote: > I have made a test for Hilbert transform for PCM data. Suggest > "pcmRawData" is PCM audio data and collect by pyaudio. After Hilbert > transform, "pcmData" has 90 degree phase shift relative to "pcmRawData". The Hilbert transform should return a complex signal, at least that is how it is defined. Sturla _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From faridafandiyev at gmail.com Mon Oct 13 03:00:18 2014 From: faridafandiyev at gmail.com (Farid Afandiyev) Date: Mon, 13 Oct 2014 11:00:18 +0400 Subject: [SciPy-User] odeint - Excess work done on this cal In-Reply-To: References: Message-ID: Hi Alex! Looks like it doesn't solve the problem completely, because when at certain combination of initial conditions or if I add more equations problem suddenly appears again. I've been suggested that this is due to the P1-P2 term not being Lipschitz at P1=P2. I'm trying to comprehend that now. Cheers, Farid. On Sat, Oct 11, 2014 at 5:36 PM, Alex Liberzon wrote: > Apparently, the problem is with the time step, see the linked IPython > notebook > > http://nbviewer.ipython.org/gist/alexlib/5220b92868c800fd1700 > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Tue Oct 14 09:05:27 2014 From: vanforeest at gmail.com (nicky van foreest) Date: Tue, 14 Oct 2014 15:05:27 +0200 Subject: [SciPy-User] odeint - Excess work done on this call In-Reply-To: References: Message-ID: Hi Farid, Please see below. On 11 October 2014 14:31, Farid Afandiyev wrote: > Hello! > > I'm newbie both in calculus and python/scipy so I apologize if this > question is too dumb. > I'm trying to model flow between two pressure vessels. Let's say we have > two points and a link between them like this > [Vc_1, P_1]=====A====[Vc_2, P_2] > Vc_1, Vc_2 are constants volumes of the nodes and P_1, P_2 varying > pressure of the nodes respectively. > > I end up writing differential questions below. Never mind physical meaning > I just want to get math correct. > > dP_1/dt = dVa/dt * 1/(Vc_1 * B) > dP_2/dt = dVa/dt * 1/(Vc_2 * B) > > Here B is compressibility. > > dVa/dt = A * K * sqrt(P_1 - P_2) > > dVa is amount of "flow" between nodes. > > K is some constant coefficient and A is link "weight". (P_1-P_2) can > change sign so I've adjusted this in the software. > > Below is python program that I wrote to evaluate this. > > #!/usr/bin/env python > > import math > from scipy.integrate import odeint > from time import time > import numpy > > B_compressibility = 0.0000033 #water compressibility > K = 0.747871759938 #coefficient > > Vc_1 = 20 > Vc_2 = 50 > A = 0.01 > P_1 = 4000 > P_2 = 2000 > > def deriv(state, t): > _P_1 = state[0] > _P_2 = state[2] > diff_P = _P_1 - _P_2 > flow_direction = math.copysign(1, diff_P) > dVa = flow_direction * A * K * math.sqrt(abs(diff_P)) > dP_1 = -(dVa/Vc_1)/B_compressibility > dP_2 = (dVa/Vc_2)/B_compressibility > return [dP_1, -dVa, dP_2, dVa] > > if __name__ == '__main__': > Va_1 = Vc_1 * P_1 * B_compressibility > Va_2 = Vc_2 * P_2 * B_compressibility > odeIterations = 10 > timeperiod = numpy.linspace(0.0,1.0, odeIterations) > initial_state = [P_1, Va_1, P_2, Va_2] > t0 = time() > state_array = odeint(deriv, initial_state, timeperiod) > t1 = time() > print 'runtime %fs' %(t1-t0) > state = state_array[odeIterations-1] > print state_array > P_1 = state[0] > Va_1 = state[1] > P_2 = state[2] > Va_2 = state[3] > > Below is output from program > > Excess work done on this call (perhaps wrong Dfun type). > Run with full_output = 1 to get quantitative information. > runtime 0.157000s > [[ 4.00000000e+03 2.64000000e-01 2.00000000e+03 3.30000000e-01] > [ 3.49242034e+03 2.30499743e-01 2.20303186e+03 3.63500257e-01] > [ 3.09580400e+03 2.04323064e-01 2.36167840e+03 3.89676936e-01] > [ 2.81015098e+03 1.85469965e-01 2.47593961e+03 4.08530035e-01] > [ 2.63546127e+03 1.73940444e-01 2.54581549e+03 4.20059556e-01] > [ 2.57173487e+03 1.69734501e-01 2.57130605e+03 4.24265499e-01] > [ 2.57142857e+03 1.69714286e-01 2.57142857e+03 4.24285714e-01] > It appears that your P1 - P2 = 0 at this line. It might mean that the pressure in both vessels is the same at that moment in time, and then the flow should stop, right? So, what should the integrator tell you after this point in time? Nicky > [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] > [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] > [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] > lsoda-- at current t (=r1), mxstep (=i1) steps > taken on this call before reaching tout > In above message, I1 = 500 > In above message, R1 = 0.6110315150411E+00 > > > > Odeint gives correct results up to 7th line and then something goes > seriously wrong. I have searched in google and it looks like I'm not the > only one who struggles with scipy. Everybody suggests increasing mxstep but > that doesn't solve my problem. In addition it slows down the method > significantly. Somebody suggested reducing accuracy but I don't know how to > do that. Also I don't need super accuracy from the odeint. Couple of digits > after dot is more than enough for me. Any help is greatly appreciated! > > Kind regards, > > Farid. > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From faridafandiyev at gmail.com Tue Oct 14 11:08:39 2014 From: faridafandiyev at gmail.com (Farid Afandiyev) Date: Tue, 14 Oct 2014 19:08:39 +0400 Subject: [SciPy-User] odeint - Excess work done on this call In-Reply-To: References: Message-ID: Hi Nicky, Correct, flow will stop after pressures are equalized. I would like it to stop integration at that point, i.e. after pressure equilibrium is reached odeint should stop calling derivative function. Thanks, Farid. On Tue, Oct 14, 2014 at 5:05 PM, nicky van foreest wrote: > Hi Farid, > > Please see below. > > [[ 4.00000000e+03 2.64000000e-01 2.00000000e+03 3.30000000e-01] >> [ 3.49242034e+03 2.30499743e-01 2.20303186e+03 3.63500257e-01] >> [ 3.09580400e+03 2.04323064e-01 2.36167840e+03 3.89676936e-01] >> [ 2.81015098e+03 1.85469965e-01 2.47593961e+03 4.08530035e-01] >> [ 2.63546127e+03 1.73940444e-01 2.54581549e+03 4.20059556e-01] >> [ 2.57173487e+03 1.69734501e-01 2.57130605e+03 4.24265499e-01] >> [ 2.57142857e+03 1.69714286e-01 2.57142857e+03 4.24285714e-01] >> > > It appears that your P1 - P2 = 0 at this line. It might mean that the > pressure in both vessels is the same at that moment in time, and then the > flow should stop, right? So, what should the integrator tell you after > this point in time? > > Nicky > > >> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] >> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] >> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From brown.evand at gmail.com Tue Oct 14 12:46:39 2014 From: brown.evand at gmail.com (Evan Brown) Date: Tue, 14 Oct 2014 17:46:39 +0100 Subject: [SciPy-User] -KPIC flag compilation error Message-ID: Hi there I am trying to compile SciPy from source on Solaris 11. I have downloaded the latest version of the SciPi source code - scipy-0.14.0. I am running gcc 4.8.2 and gfortran and Python 2.6. When I run: python setup.py build in the SciPy source directory, the process runs for a while before exiting with the following error: --------------------- building 'rootfind' library compiling C sources C compiler: /usr/lib/python2.6/pycc -DNDEBUG -KPIC creating build/temp.solaris-2.11-i86pc-2.6/scipy/optimize/Zeros compile options: '-I/usr/lib/python2.6/vendor-packages/numpy/core/include -c' pycc: scipy/optimize/Zeros/brenth.c gcc: error: unrecognized command line option ?-KPIC? gcc: error: unrecognized command line option ?-KPIC? error: Command "/usr/lib/python2.6/pycc -DNDEBUG -KPIC -I/usr/lib/python2.6/vendor-packages/numpy/core/include -c scipy/optimize/Zeros/brenth.c -o build/temp.solaris-2.11-i86pc-2.6/scipy/optimize/Zeros/brenth.o" failed with exit status 1 ------------------------ My question is: how do I remove the apparently illegal -KPIC flag which is being passed to gcc? I can't find the location in the source or setup.py file where this flag is passed to the compiler. Any help would be much appreciated. Best wishes Evan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimparker96313 at gmail.com Tue Oct 14 22:15:01 2014 From: jimparker96313 at gmail.com (Jim Parker) Date: Tue, 14 Oct 2014 21:15:01 -0500 Subject: [SciPy-User] -KPIC flag compilation error In-Reply-To: References: Message-ID: Evan, It appears you are having a conflict between Solaris Sun C compiler and gcc. I'm guessing python was built with the native OS compiler for which -KPIC is a valid option. When building scipy, apparently the build is bringing in the same options that were used to build python. For reference see this https://www.prelude-ids.org/issues/288 Not the best reference, but I'm not a Solaris expert. Someone could improve on this by finding a better site. This was just a quick search result. The solution: a) rebuild python using gcc and link against that version b) pass CFLAGS=-fPIC at the command line > CFLAGS='-fPIC' python setup.py build for reference: http://stackoverflow.com/questions/6928110/how-may-i-override-the-compiler-gcc-flags-that-setup-py-uses-by-default Cheers, --Jim On Tue, Oct 14, 2014 at 11:46 AM, Evan Brown wrote: > Hi there > > I am trying to compile SciPy from source on Solaris 11. > > I have downloaded the latest version of the SciPi source code - > scipy-0.14.0. > > I am running gcc 4.8.2 and gfortran and Python 2.6. > > When I run: > > python setup.py build > > in the SciPy source directory, the process runs for a while before exiting > with the following error: > --------------------- > > building 'rootfind' library > > compiling C sources > > C compiler: /usr/lib/python2.6/pycc -DNDEBUG -KPIC > > > creating build/temp.solaris-2.11-i86pc-2.6/scipy/optimize/Zeros > > compile options: '-I/usr/lib/python2.6/vendor-packages/numpy/core/include > -c' > > pycc: scipy/optimize/Zeros/brenth.c > > gcc: error: unrecognized command line option ?-KPIC? > > gcc: error: unrecognized command line option ?-KPIC? > > error: Command "/usr/lib/python2.6/pycc -DNDEBUG -KPIC > -I/usr/lib/python2.6/vendor-packages/numpy/core/include -c > scipy/optimize/Zeros/brenth.c -o > build/temp.solaris-2.11-i86pc-2.6/scipy/optimize/Zeros/brenth.o" failed > with exit status 1 > > ------------------------ > > My question is: how do I remove the apparently illegal -KPIC flag which is > being passed to gcc? I can't find the location in the source or setup.py > file where this flag is passed to the compiler. > > > Any help would be much appreciated. > > > Best wishes > > > Evan > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Wed Oct 15 11:11:43 2014 From: vanforeest at gmail.com (nicky van foreest) Date: Wed, 15 Oct 2014 17:11:43 +0200 Subject: [SciPy-User] odeint - Excess work done on this call In-Reply-To: References: Message-ID: Hi, On 14 October 2014 17:08, Farid Afandiyev wrote: > Hi Nicky, > > Correct, flow will stop after pressures are equalized. I would like it to > stop integration at that point, i.e. after pressure equilibrium is reached > odeint should stop calling derivative function. > Ok. I don't know how to stop odeint at a point that you know that pressures are the same. BTW, I find it a bit strange that the pressures balance in finite time. My intuition tells me that the pressure difference should go down like an exponential function. Is all ok in your problem formulation, or I am just wrong? Nicky > > Thanks, > > Farid. > > On Tue, Oct 14, 2014 at 5:05 PM, nicky van foreest > wrote: > >> Hi Farid, >> >> Please see below. >> >> [[ 4.00000000e+03 2.64000000e-01 2.00000000e+03 3.30000000e-01] >>> [ 3.49242034e+03 2.30499743e-01 2.20303186e+03 3.63500257e-01] >>> [ 3.09580400e+03 2.04323064e-01 2.36167840e+03 3.89676936e-01] >>> [ 2.81015098e+03 1.85469965e-01 2.47593961e+03 4.08530035e-01] >>> [ 2.63546127e+03 1.73940444e-01 2.54581549e+03 4.20059556e-01] >>> [ 2.57173487e+03 1.69734501e-01 2.57130605e+03 4.24265499e-01] >>> [ 2.57142857e+03 1.69714286e-01 2.57142857e+03 4.24285714e-01] >>> >> >> It appears that your P1 - P2 = 0 at this line. It might mean that the >> pressure in both vessels is the same at that moment in time, and then the >> flow should stop, right? So, what should the integrator tell you after >> this point in time? >> >> Nicky >> >> >>> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] >>> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] >>> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] >>> >>> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Wed Oct 15 11:33:11 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 15 Oct 2014 11:33:11 -0400 Subject: [SciPy-User] odeint - Excess work done on this call In-Reply-To: References: Message-ID: On Wed, Oct 15, 2014 at 11:11 AM, nicky van foreest wrote: > Hi, > > On 14 October 2014 17:08, Farid Afandiyev > wrote: > >> Hi Nicky, >> >> Correct, flow will stop after pressures are equalized. I would like it to >> stop integration at that point, i.e. after pressure equilibrium is reached >> odeint should stop calling derivative function. >> > > Ok. I don't know how to stop odeint at a point that you know that > pressures are the same. BTW, I find it a bit strange that the pressures > balance in finite time. My intuition tells me that the pressure difference > should go down like an exponential function. Is all ok in your problem > formulation, or I am just wrong? > > That's what can happen when the differential equation doesn't satisfy a Lipschitz condition (http://en.wikipedia.org/wiki/Lipschitz_continuity). The classic example is dx/dt = x**(1/2). x(t) == 0 is a solution. Other solutions are parabolas with their vertices on the x axis (i.e. they reach 0 in finite time). Warren Nicky > >> >> Thanks, >> >> Farid. >> >> On Tue, Oct 14, 2014 at 5:05 PM, nicky van foreest >> wrote: >> >>> Hi Farid, >>> >>> Please see below. >>> >>> [[ 4.00000000e+03 2.64000000e-01 2.00000000e+03 3.30000000e-01] >>>> [ 3.49242034e+03 2.30499743e-01 2.20303186e+03 3.63500257e-01] >>>> [ 3.09580400e+03 2.04323064e-01 2.36167840e+03 3.89676936e-01] >>>> [ 2.81015098e+03 1.85469965e-01 2.47593961e+03 4.08530035e-01] >>>> [ 2.63546127e+03 1.73940444e-01 2.54581549e+03 4.20059556e-01] >>>> [ 2.57173487e+03 1.69734501e-01 2.57130605e+03 4.24265499e-01] >>>> [ 2.57142857e+03 1.69714286e-01 2.57142857e+03 4.24285714e-01] >>>> >>> >>> It appears that your P1 - P2 = 0 at this line. It might mean that the >>> pressure in both vessels is the same at that moment in time, and then the >>> flow should stop, right? So, what should the integrator tell you after >>> this point in time? >>> >>> Nicky >>> >>> >>>> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] >>>> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] >>>> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] >>>> >>>> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Wed Oct 15 20:25:16 2014 From: andyfaff at gmail.com (Andrew Nelson) Date: Thu, 16 Oct 2014 11:25:16 +1100 Subject: [SciPy-User] BFGS precision vs least_sq Message-ID: Dear scipy users, I am using BFGS as a final 'polishing' minimizer after using another minimization technique. I thought I would check how it performs against a known non-linear regression standard, http://www.itl.nist.gov/div898/strd/nls/data/thurber.shtml. I discovered that BFGS doesn't get to the absolute known minimum for this least squares problem, even if the starting point is almost the best solution. Rather, it finishes with a chi2 of 5648.6 against the lowest chi2 minimum of 5642.7 (using scipy.optimize.leastsq). The problem seems to be 'precision loss', as reported by fmin_bfgs. I have tried adjusting the gtol and epsilon options for fmin_bfgs, but did not do any better. Ok, so the two chi2 are only out by 0.1%, but I would like to know why BFGS can't get to the same minimum as leastsq. The following code illustrates the issue: import numpy as np from scipy.optimize import fmin_bfgs, leastsq a = np.asarray([80.574, 84.248, 87.264, 87.195, 89.076, 89.608, 89.868, 90.101, 92.405, 95.854, 100.696, 101.06, 401.672, 390.724, 567.534, 635.316, 733.054, 759.087, 894.206, 990.785, 1090.109, 1080.914, 1122.643, 1178.351, 1260.531, 1273.514, 1288.339, 1327.543, 1353.863, 1414.509, 1425.208, 1421.384, 1442.962, 1464.350, 1468.705, 1447.894, 1457.628]) b = np.asarray([-3.067, -2.981, -2.921, -2.912, -2.840, -2.797, -2.702, -2.699, -2.633, -2.481, -2.363, -2.322, -1.501, -1.460, -1.274, -1.212, -1.100, -1.046, -0.915, -0.714, -0.566, -0.545, -0.400, -0.309, -0.109, -0.103, 0.010, 0.119, 0.377, 0.790, 0.963, 1.006, 1.115, 1.572, 1.841, 2.047, 2.200]) def chi2(x, *args): vec = x[0] + x[1] * b + x[2] * b ** 2 + x[3] * b ** 3 vec /= 1 + x[4] * b + x[5] * b ** 2 + x[6] * b ** 3 return sum((vec - a)**2) startX = np.array([1.28861830e+03, 1.49582373e+03, 5.86557938e+02, 7.60498878e+01, 9.69658594e-01, 3.99904927e-01, 5.03732685e-02]) #epsilon = np.sqrt(np.finfo(np.float64).eps) * np.max(np.abs(startX), 0.1) x_BFGS = fmin_bfgs(chi2, startX) def res(x, *args): vec = x[0] + x[1] * b + x[2] * b ** 2 + x[3] * b ** 3 vec /= 1 + x[4] * b + x[5] * b ** 2 + x[6] * b ** 3 return (vec - a) x_leastsq = leastsq(res, startX, full_output=True)[0] print "Chi2 BFGS:", chi2(x_BFGS), "Chi2 leastsq", chi2(x_leastsq) I'm using the master branch of scipy with these tests. regards, Andrew. -- _____________________________________ Dr. Andrew Nelson _____________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Wed Oct 15 22:52:42 2014 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 15 Oct 2014 22:52:42 -0400 Subject: [SciPy-User] odeint - Excess work done on this call In-Reply-To: References: Message-ID: On Tue, Oct 14, 2014 at 11:08 AM, Farid Afandiyev wrote: > Hi Nicky, > > Correct, flow will stop after pressures are equalized. I would like it to > stop integration at that point, i.e. after pressure equilibrium is reached > odeint should stop calling derivative function. > > Farid, Your system of differential equations has an analytical solution. To derive it, subtract the equation for P_2'(t) from the equation for P_1'(t). You'll end of with an equation of the form d(P_1(t) + P_2(t))/dt = -c * sqrt(P_1(t) - P_2(t)) (assuming P_1(0) > P_2(0), and where c is a constant expression of the system parameters A, B, etc), or dx/dt = -c*sqrt(x) where x(t) = P_1(t) - P_2(t). That's a separable first order equation; the solution is x(t) = (-c*t/2 + sqrt(x0))**2, until x reaches 0 (i.e. until the vertex of the parabola). x0 is the initial condition x(0) = P_1(0) - P_2(0). If P_1(0) < P_2(0), you get x(t) = -(-c*t/2 + sqrt(-x0))**2 If you add Vc_1*B*P_1'(t) and Vc_2*B*P_2'(t), you find that Vc_1*B*P_1'(t) + Vc_2*B*P_2'(t) = 0 so Vc_1*P_1(t) + Vc_2*P_2(t) = constant = Vc_1*P_1(0) + Vc_2*P_2(0) >From that and the solution x(t), you can get P_1(t) and P_2(t). You can do the same type of calculation for the differential equations for the two Va components (if you need them). The attached script creates a plot comparing the analytical solution to the solution found using odeint. It generates the attached plot. Warren > Thanks, > > Farid. > > On Tue, Oct 14, 2014 at 5:05 PM, nicky van foreest > wrote: > >> Hi Farid, >> >> Please see below. >> >> [[ 4.00000000e+03 2.64000000e-01 2.00000000e+03 3.30000000e-01] >>> [ 3.49242034e+03 2.30499743e-01 2.20303186e+03 3.63500257e-01] >>> [ 3.09580400e+03 2.04323064e-01 2.36167840e+03 3.89676936e-01] >>> [ 2.81015098e+03 1.85469965e-01 2.47593961e+03 4.08530035e-01] >>> [ 2.63546127e+03 1.73940444e-01 2.54581549e+03 4.20059556e-01] >>> [ 2.57173487e+03 1.69734501e-01 2.57130605e+03 4.24265499e-01] >>> [ 2.57142857e+03 1.69714286e-01 2.57142857e+03 4.24285714e-01] >>> >> >> It appears that your P1 - P2 = 0 at this line. It might mean that the >> pressure in both vessels is the same at that moment in time, and then the >> flow should stop, right? So, what should the integrator tell you after >> this point in time? >> >> Nicky >> >> >>> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] >>> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] >>> [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] >>> >>> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pressure_odeint_question.py Type: text/x-python Size: 3158 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pressure.png Type: image/png Size: 73346 bytes Desc: not available URL: From newville at cars.uchicago.edu Thu Oct 16 07:59:44 2014 From: newville at cars.uchicago.edu (Matt Newville) Date: Thu, 16 Oct 2014 06:59:44 -0500 Subject: [SciPy-User] BFGS precision vs least_sq In-Reply-To: References: Message-ID: On Wed, Oct 15, 2014 at 7:25 PM, Andrew Nelson wrote: > Dear scipy users, > I am using BFGS as a final 'polishing' minimizer after using another > minimization technique. I thought I would check how it performs against a > known non-linear regression standard, > http://www.itl.nist.gov/div898/strd/nls/data/thurber.shtml. > > I discovered that BFGS doesn't get to the absolute known minimum for this > least squares problem, even if the starting point is almost the best > solution. Rather, it finishes with a chi2 of 5648.6 against the lowest > chi2 minimum of 5642.7 (using scipy.optimize.leastsq). The problem seems > to be 'precision loss', as reported by fmin_bfgs. I have tried adjusting > the gtol and epsilon options for fmin_bfgs, but did not do any better. > Ok, so the two chi2 are only out by 0.1%, but I would like to know why > BFGS can't get to the same minimum as leastsq. > > You mentioned the result for chi-square but not for the parameters themselves. Are those close to the certified values? Your observation is consistent with my experience with the NIST standard reference data. The tests are hard (and Thurber is listed as Higher level of difficulty), and leastsq() does much better than the scalar optimization methods from scipy both in terms of final results and number of function evaluations. I suspect that because leastsq() examines the full Jacboian (that is, has a residual array) instead of a single scalar value, it can better navigate the parameter space. Leastsq() mixes Gauss-Netwon and steepest descent, but I don't know why that would give an improvement in ultimate chi-square for the Thurber problem over Newton's method (which I understand BFGS to be). I also suspect that the NIST "certified results" are derived from analysis with MINPACK-1 (ie, leastsq). I don't know this for sure, but the tests are old enough (mid to late 1990s, perhaps older) that using MINPACK for the analysis would be reasonable. Furthermore, the results from leastsq are very, very good. For the Thurber data, the 37 data points have 5 to 7 digits for x, and 4 digits for y. The certified parameters and their uncertainties, and the residual sum of squares are given to 10 digits -- the Readme states that not all of these values are statistically significant, and that 3 digits are hard to get for some problems. A simple use of leastsq() for the Thurber data with no fussing with tolerances or scaling gives at least 4 correct digits for each parameter, at least 3 digits for the parameter uncertainties and 8 digits for the residual sum of squares, from either of the 2 supplied starting values. Similarly excellent results are found for most of the NIST tests with leastsq() -- it often does as well as NIST says to expect. The certified values had to come from somewhere... why not from a non-linear least-squares using MINPACK? That's just to say that comparing any method to leastsq on the NIST data may be a biased test in favor of leastsq. OTOH, BFGS fails to alter the initial values from either starting point for the Thurber data for me. So the fact that you're finding 3 digits for residual sum of squares is a huge improvement! Whether the Thurber test is a good test for the polishing step of a global optimizer may be another issue. For a global optimizer you probably *want* a scaler minimizer -- so that maximum likelihood can be used instead of least-squares, for example. That is, using leastsq() for the polishing stage may not be suitable. And if the idea of global optimization is to locate good starting points for local minimization, then it shouldn't matter too much what local optimizer you use. Not sure that really answers your question, but hope it helps.... --Matt Newville -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Thu Oct 16 17:52:33 2014 From: andyfaff at gmail.com (Andrew Nelson) Date: Fri, 17 Oct 2014 08:52:33 +1100 Subject: [SciPy-User] BFGS precision vs least_sq In-Reply-To: References: Message-ID: On 16 October 2014 22:59, Matt Newville wrote: > On Wed, Oct 15, 2014 at 7:25 PM, Andrew Nelson wrote: > >> Dear scipy users, >> I am using BFGS as a final 'polishing' minimizer after using another >> minimization technique. I thought I would check how it performs against a >> known non-linear regression standard, >> http://www.itl.nist.gov/div898/strd/nls/data/thurber.shtml. >> >> I discovered that BFGS doesn't get to the absolute known minimum for this >> least squares problem, even if the starting point is almost the best >> solution. Rather, it finishes with a chi2 of 5648.6 against the lowest >> chi2 minimum of 5642.7 (using scipy.optimize.leastsq). The problem seems >> to be 'precision loss', as reported by fmin_bfgs. I have tried adjusting >> the gtol and epsilon options for fmin_bfgs, but did not do any better. >> Ok, so the two chi2 are only out by 0.1%, but I would like to know why >> BFGS can't get to the same minimum as leastsq. >> >> > You mentioned the result for chi-square but not for the parameters > themselves. Are those close to the certified values? > > Your observation is consistent with my experience with the NIST standard > reference data. The tests are hard (and Thurber is listed as Higher level > of difficulty), and leastsq() does much better than the scalar optimization > methods from scipy both in terms of final results and number of function > evaluations. > > I suspect that because leastsq() examines the full Jacboian (that is, has > a residual array) instead of a single scalar value, it can better navigate > the parameter space. Leastsq() mixes Gauss-Netwon and steepest descent, > but I don't know why that would give an improvement in ultimate chi-square > for the Thurber problem over Newton's method (which I understand BFGS to > be). > > I also suspect that the NIST "certified results" are derived from analysis > with MINPACK-1 (ie, leastsq). I don't know this for sure, but the tests > are old enough (mid to late 1990s, perhaps older) that using MINPACK for > the analysis would be reasonable. Furthermore, the results from leastsq > are very, very good. For the Thurber data, the 37 data points have 5 to 7 > digits for x, and 4 digits for y. The certified parameters and their > uncertainties, and the residual sum of squares are given to 10 digits -- > the Readme states that not all of these values are statistically > significant, and that 3 digits are hard to get for some problems. A simple > use of leastsq() for the Thurber data with no fussing with tolerances or > scaling gives at least 4 correct digits for each parameter, at least 3 > digits for the parameter uncertainties and 8 digits for the residual sum of > squares, from either of the 2 supplied starting values. Similarly > excellent results are found for most of the NIST tests with leastsq() -- it > often does as well as NIST says to expect. The certified values had to > come from somewhere... why not from a non-linear least-squares using > MINPACK? That's just to say that comparing any method to leastsq on the > NIST data may be a biased test in favor of leastsq. > > OTOH, BFGS fails to alter the initial values from either starting point > for the Thurber data for me. So the fact that you're finding 3 digits for > residual sum of squares is a huge improvement! > > Whether the Thurber test is a good test for the polishing step of a global > optimizer may be another issue. For a global optimizer you probably *want* > a scaler minimizer -- so that maximum likelihood can be used instead of > least-squares, for example. That is, using leastsq() for the polishing > stage may not be suitable. And if the idea of global optimization is to > locate good starting points for local minimization, then it shouldn't > matter too much what local optimizer you use. > > > The starting parameters in the demo script I wrote are from a differential_evolution minimisation followed by a BFGS 'polish'. So in that respect I'm taking the endpoint of a BFGS run and trying to get it even lower by running BFGS again. I then wanted to see if leastsq got any lower. It's a bit unfair, the output vector from BFGS is very close to the certified values. I would've thought that both should be equally good at calculating the gradient/Jacobian (first derivative matrix), that's simply a vector of \frac{\partial \chi^2}{\partial x_i}. I understand that the way that they calculate the Hessians are different. The reason I am trying to get to the bottom of this is because I am working on the benchmark suite for (scalar) global optimizers. Each problem should have a way of judging if the global minimum has been reached. The easy way is to just set an absolute tolerance for differences from the known minimum energy e.g. 1e-4. However, that won't be fair in problems such as these - the BFGS is very close to the minimum energy (within 0.1%), it just can't polish to the absolute minimum like leastsq can (at least from this starting point). But is that a reason to mark it as a fail? So for some problems in the benchmarking perhaps there needs to be an rtol and atol. How does one set those rtol/atol? The polishing in differential_evolution is done by L-BFGS-S, it can't use leastsq for the reason you mentioned. But for the least squares problems the leastsq can be run independently. I went through the code for bfgs and got to line_search_wolfe (or something along those lines), to see where the precision warning message came from. There my code reading head gave out. I was wondering if there were any tolerances/epsilons I could set tighter to get that final drop of reduction in energy, but I didn't get any further. Perhaps the list can explain the situation in which the precision message is emitted? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebix at sebix.at Fri Oct 17 03:51:17 2014 From: sebix at sebix.at (Sebastian) Date: Fri, 17 Oct 2014 09:51:17 +0200 Subject: [SciPy-User] Graphics of Scipy-Stack Core Packages and dependencies Message-ID: <5440CA75.80903@sebix.at> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, For a presentation - giving an overview of the scientific stack - I'm looking for a graphics that shows the core packages (numpy, scipy, pandas, matplotlib, and possibly cython, pytables etc.) and how they are related. So: Numpy as basic package at the top and then pandas and scipy, matplotlib below of scipy. Does anyone know if such a graphics does already exist? If not, I will create one and appreciate your help for the details. I also think of adding details of third-pary requirements (the LAPACK, ATLAS etc.). best, Sebastian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) iQIcBAEBAgAGBQJUQMp1AAoJEBn0X+vcm0Y7HekQAIL9O2rNL1CVozEI4gWx+4CH PQTPGfM5YcDuYVSNng8GSBmcDAsHLs59Be6AS7beCpd8JiW7rQJtOzv4D4T9Ao9l hIwtEblF9lC3C86+D9BPJW9+9yDTocTeaoy+/Uubs9oskA+/D5xiRIgBgFjwoebX GLxlZ4ASygatPAA03SSwIfiB2Ch44PSd4vhQgLDAR2u7S7V1wPbXUllI1fVInJi1 rOs719IN/64+cB5GeCLJPhZZclSOawDIULI3YIfBLTSEiqjmr9vLYRlQu+5Iy8BB DjEAqF/ocXwKMak8fsl9Xpuz52ErPw3VWGj+984V6mYUAa/jQMDIReWrRdLHJZd4 idDP/78QNkcDID3kr1DoP2NqB0Y5kssed4zynsKbQ5IRKBMgazWKBRsort3HOeSZ bL32CCSKEa+gTUJw0tKskYvnSPRIuWwZ1vwkLbDpwyZUU/BUap6GIlK+WLpmi+Jt 61k+yxIO/y612yRZ3ruIee0ZMzv5NpdkvPQEeQ/rC36zHbOHDVU36OD9UrwL9jPz ErI85QRSSQ/EKbT17m2tN8afQ7WVG0H/9J6/EooFvucM3p3oorhtXjVVEZD73iwY oWur/rLOXR2+Ckm7NFc28zO9hkq4MorX1NMFyK/GIS3rABBNd4GXtElIfHqOiBUn EO2rJ3C+YWOZVlnd1qRb =JEZY -----END PGP SIGNATURE----- From newville at cars.uchicago.edu Fri Oct 17 16:13:47 2014 From: newville at cars.uchicago.edu (Matt Newville) Date: Fri, 17 Oct 2014 15:13:47 -0500 Subject: [SciPy-User] BFGS precision vs least_sq In-Reply-To: References: Message-ID: On Thu, Oct 16, 2014 at 4:52 PM, Andrew Nelson wrote: > > > On 16 October 2014 22:59, Matt Newville > wrote: > >> On Wed, Oct 15, 2014 at 7:25 PM, Andrew Nelson >> wrote: >> >>> Dear scipy users, >>> I am using BFGS as a final 'polishing' minimizer after using another >>> minimization technique. I thought I would check how it performs against a >>> known non-linear regression standard, >>> http://www.itl.nist.gov/div898/strd/nls/data/thurber.shtml. >>> >>> I discovered that BFGS doesn't get to the absolute known minimum for >>> this least squares problem, even if the starting point is almost the best >>> solution. Rather, it finishes with a chi2 of 5648.6 against the lowest >>> chi2 minimum of 5642.7 (using scipy.optimize.leastsq). The problem seems >>> to be 'precision loss', as reported by fmin_bfgs. I have tried adjusting >>> the gtol and epsilon options for fmin_bfgs, but did not do any better. >>> Ok, so the two chi2 are only out by 0.1%, but I would like to know why >>> BFGS can't get to the same minimum as leastsq. >>> >>> >> You mentioned the result for chi-square but not for the parameters >> themselves. Are those close to the certified values? >> >> Your observation is consistent with my experience with the NIST standard >> reference data. The tests are hard (and Thurber is listed as Higher level >> of difficulty), and leastsq() does much better than the scalar optimization >> methods from scipy both in terms of final results and number of function >> evaluations. >> >> I suspect that because leastsq() examines the full Jacboian (that is, has >> a residual array) instead of a single scalar value, it can better navigate >> the parameter space. Leastsq() mixes Gauss-Netwon and steepest descent, >> but I don't know why that would give an improvement in ultimate chi-square >> for the Thurber problem over Newton's method (which I understand BFGS to >> be). >> >> I also suspect that the NIST "certified results" are derived from >> analysis with MINPACK-1 (ie, leastsq). I don't know this for sure, but >> the tests are old enough (mid to late 1990s, perhaps older) that using >> MINPACK for the analysis would be reasonable. Furthermore, the results >> from leastsq are very, very good. For the Thurber data, the 37 data points >> have 5 to 7 digits for x, and 4 digits for y. The certified parameters and >> their uncertainties, and the residual sum of squares are given to 10 >> digits -- the Readme states that not all of these values are statistically >> significant, and that 3 digits are hard to get for some problems. A simple >> use of leastsq() for the Thurber data with no fussing with tolerances or >> scaling gives at least 4 correct digits for each parameter, at least 3 >> digits for the parameter uncertainties and 8 digits for the residual sum of >> squares, from either of the 2 supplied starting values. Similarly >> excellent results are found for most of the NIST tests with leastsq() -- it >> often does as well as NIST says to expect. The certified values had to >> come from somewhere... why not from a non-linear least-squares using >> MINPACK? That's just to say that comparing any method to leastsq on the >> NIST data may be a biased test in favor of leastsq. >> >> OTOH, BFGS fails to alter the initial values from either starting point >> for the Thurber data for me. So the fact that you're finding 3 digits for >> residual sum of squares is a huge improvement! >> >> Whether the Thurber test is a good test for the polishing step of a >> global optimizer may be another issue. For a global optimizer you probably >> *want* a scaler minimizer -- so that maximum likelihood can be used >> instead of least-squares, for example. That is, using leastsq() for the >> polishing stage may not be suitable. And if the idea of global >> optimization is to locate good starting points for local minimization, then >> it shouldn't matter too much what local optimizer you use. >> >> >> > The starting parameters in the demo script I wrote are from a > differential_evolution minimisation followed by a BFGS 'polish'. So in > that respect I'm taking the endpoint of a BFGS run and trying to get it > even lower by running BFGS again. I then wanted to see if leastsq got any > lower. It's a bit unfair, the output vector from BFGS is very close to the > certified values. > I would've thought that both should be equally good at calculating the > gradient/Jacobian (first derivative matrix), that's simply a vector of > \frac{\partial \chi^2}{\partial x_i}. I understand that the way that they > calculate the Hessians are different. > leastsq() uses all (in this case 37) observations of the residual array to make derivatives, not just one positive value to be minimized. That is, it is not looking at only chi-square, it is finding the parameters that gives the smallest sum of squares of the residual array... but it needs that residual array! leastsq() would not even try to use only one observation to find the best values of seven variables. A scalar minimization using chi-square for a multivariate/ multiple-observation problem is deliberately throwing away data, and cannot even use the sign of the residual array. It should not be surprising that a scalar minimizer does not perform as well on hard problems. The reason I am trying to get to the bottom of this is because I am working > on the benchmark suite for (scalar) global optimizers. Each problem should > have a way of judging if the global minimum has been reached. The easy way > is to just set an absolute tolerance for differences from the known minimum > energy e.g. 1e-4. However, that won't be fair in problems such as these - > the BFGS is very close to the minimum energy (within 0.1%), it just can't > polish to the absolute minimum like leastsq can (at least from this > starting point). But is that a reason to mark it as a fail? So for some > problems in the benchmarking perhaps there needs to be an rtol and atol. > How does one set those rtol/atol? > The polishing in differential_evolution is done by L-BFGS-S, it can't use > leastsq for the reason you mentioned. But for the least squares problems > the leastsq can be run independently. > I went through the code for bfgs and got to line_search_wolfe (or something > along those lines), to see where the precision warning message came from. > There my code reading head gave out. I was wondering if there were any > tolerances/epsilons I could set tighter to get that final drop of reduction > in energy, but I didn't get any further. Perhaps the list can explain the > situation in which the precision message is emitted? > Are you using fmin_bfgs() for fmin_l_bfgs_b() for the polishing? The former has a 'gtol' argument, the latter has 'factr' and 'pgtol' arguments. --Matt Newville -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebix at sebix.at Sat Oct 18 16:35:03 2014 From: sebix at sebix.at (Sebastian) Date: Sat, 18 Oct 2014 22:35:03 +0200 Subject: [SciPy-User] Graphics of Scipy-Stack Core Packages and dependencies In-Reply-To: <5440CA75.80903@sebix.at> References: <5440CA75.80903@sebix.at> Message-ID: <5442CEF7.8060301@sebix.at> I just created a diagram myself: https://sebix.at/tmp/scipystack.png (or .svg and .dia) It shows the most important dependencies of the most used packages. I left out some deps, like six, dateutil and so on, and some are optional but highly recommended. I want to especially ask you if dependencies on the bottom (the low-level codes) are correct? I found out that BLAS is only a function definition, LAPACK implements it in Fortran for all platforms (as opposed to MKL and ACML) and it is used through ATLAS (which provides the APIs) through f2py. And ARPACK is used via f2py in SciPy. It just came to my mind that MINPACK is missing (also used via f2py?) Are there any other important dependencies missing? Or any other important packages on the python-side? I think of creating two diagrams: one with more packages and one with only the most important, useful for slides or a quick presentation. best, On 2014-10-17 09:51, Sebastian wrote: > Hello, > > For a presentation - giving an overview of the scientific stack - I'm > looking for a graphics that shows the core packages (numpy, scipy, > pandas, matplotlib, and possibly cython, pytables etc.) and how they are > related. So: Numpy as basic package at the top and then pandas and > scipy, matplotlib below of scipy. Does anyone know if such a graphics > does already exist? > If not, I will create one and appreciate your help for the details. I > also think of adding details of third-pary requirements (the LAPACK, > ATLAS etc.). > > best, > Sebastian From jeffreback at gmail.com Sun Oct 19 17:34:43 2014 From: jeffreback at gmail.com (Jeff Reback) Date: Sun, 19 Oct 2014 17:34:43 -0400 Subject: [SciPy-User] ANN: Pandas 0.15.0 released Message-ID: Hello, We are proud to announce v0.15.0 of pandas, a major release from 0.14.1. This release includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. This was 4 months of work with 420 commits by 79 authors encompassing 236 issues. We recommend that all users upgrade to this version. *Highlights:* - Drop support for numpy < 1.7.0 - The Categorical type was integrated as a first-class pandas type - New scalar type Timedelta, and a new index type TimedeltaIndex - New DataFrame default display for df.info() to include memory usage - New datetimelike properties accessor .dt for Series - Split indexing documentation into Indexing and Selecting Data and MultiIndex / Advanced Indexing - Split out string methods documentation into Working with Text Data - read_csv will now by default ignore blank lines when parsing - API change in using Indexes in set operations - Internal refactoring of the Index class to no longer sub-class ndarray - dropping support for PyTables less than version 3.0.0, and numexpr less than version 2.1 See a full description of Whatsnew for v0.15.0 here: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html *What is it:* *pandas* is a Python package providing fast, flexible, and expressive data structures designed to make working with ?relational? or ?labeled? data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. Documentation: http://pandas.pydata.org/pandas-docs/stable/ Source tarballs, windows binaries are available on PyPI: https://pypi.python.org/pypi/pandas windows binaries are courtesy of Christoph Gohlke and are built on Numpy 1.8 macosx wheels are courtesy of Matthew Brett and are built on Numpy 1.7.1 Please report any issues here: https://github.com/pydata/pandas/issues Thanks The Pandas Development Team Contributors to the 0.15.0 release -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Oct 20 22:31:37 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 20 Oct 2014 22:31:37 -0400 Subject: [SciPy-User] what happened to http://scipy-central.org/ ? Message-ID: I'm getting a 502 ? Bad Gateway Josef From yw5aj at virginia.edu Thu Oct 23 11:36:46 2014 From: yw5aj at virginia.edu (Yuxiang Wang) Date: Thu, 23 Oct 2014 11:36:46 -0400 Subject: [SciPy-User] Doc on API Message-ID: Hi all, I was reading through the link (http://docs.scipy.org/doc/scipy/reference/api.html#guidelines-for-importing-functions-from-scipy) and it mentioned that the first form is preferred over the second one under most circumstances: # first form from scipy import stats stats.lomax(...) # second form from scipy.stats import distributions distributions.lomax(...) I honestly think the second form is far more frequently used in the examples given in the document, for example: http://docs.scipy.org/doc/scipy/reference/tutorial/special.html http://docs.scipy.org/doc/scipy/reference/tutorial/integrate.html http://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html http://docs.scipy.org/doc/scipy/reference/tutorial/fftpack.html Would you think it'd be better to update this link? -Shawn -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj at virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/ From jeremy at jeremysanders.net Thu Oct 23 16:28:07 2014 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Thu, 23 Oct 2014 22:28:07 +0200 Subject: [SciPy-User] ANN: Veusz 1.22 Message-ID: <544964D7.30107@jeremysanders.net> I'm pleased to announce the release of Veusz 1.22 - please see the notes attached. Jeremy -------------- next part -------------- Veusz 1.22 ---------- http://home.gna.org/veusz/ Veusz is a scientific plotting package. It is designed to produce publication-ready Postscript, PDF or SVG output. Graphs are built-up by combining plotting widgets. The user interface aims to be simple, consistent and powerful. Veusz provides GUI, Python module, command line, scripting, DBUS and SAMP interfaces to its plotting facilities. It also allows for manipulation and editing of datasets. Data can be captured from external sources such as Internet sockets or other programs. Changes in 1.22: * Add option to ignore white space following delimiters when importing CSVs * Allow x/yedge and x/ycent to be inverted to change data order when import 2D data * Rewrite of widget tree internals to fix miscellaneous bugs * Optionally use astropy for SAMP * Add setting to extend the python import path * Add option to select widgets with same name on page * Resize rectangular shapes keeping other corners fixed. Old symmetric resize behaviour is retained if shift is pressed. Bug fixes * Fix faulty caching of datasets expressions * Do not auto mirror axes if shared axis placed in grid and ignore other hidden axes for mirroring purposes * gridatedge 2D import option inverts values in rows * y values for 2D datasets in data edit dialog are inverted * Fix crash when constructing 2D dataset with constants * Fix crash when interpreting date values with unicode * Fix for crash when moving ellipses on broken axes * Fix crash when editing datetime data * Fix crash when renaming 2D datasets in dataset editor * If 2d dataset does not exist, don't crash when using colorbar * Ignore HDF5 data types which are not supported by h5py * Catch recursion limit errors properly * In sort and text sort plugins, clip length to minimum * Do not plot functions with more than 1D in function plotter * Catch error for shapes widgets with no coordinates * Handle zero length datasets in subtract minimum or mean * Clip range of zoom between 0.05 and 20x * Catch errors in CSV date format * Do not allow 2D datasets to be used for labels in label widget * Handle unicode data correctly in clipboard data * Fix data import dialog for fits files in unicode paths Features of package: Plotting features: * X-Y plots (with errorbars) * Line and function plots * Contour plots * Images (with colour mappings and colorbars) * Stepped plots (for histograms) * Bar graphs * Vector field plots * Box plots * Polar plots * Ternary plots * Plotting dates * Fitting functions to data * Stacked plots and arrays of plots * Nested plots * Plot keys * Plot labels * Shapes and arrows on plots * LaTeX-like formatting for text * Multiple axes * Axes with steps in axis scale (broken axes) * Axis scales using functional forms * Plotting functions of datasets Input and output: * EPS/PDF/PNG/SVG/EMF export * Dataset creation/manipulation * Embed Veusz within other programs * Text, HDF5, CSV, FITS, NPY/NPZ, QDP, binary and user-plugin importing * Data can be captured from external sources Extending: * Use as a Python module * User defined functions, constants and can import external Python functions * Plugin interface to allow user to write or load code to - import data using new formats - make new datasets, optionally linked to existing datasets - arbitrarily manipulate the document * Scripting interface * Control with DBUS and SAMP Other features: * Data picker * Interactive tutorial * Multithreaded rendering Requirements for source install: Python 2.x (2.6 or greater required) or 3.x (3.3 or greater required) http://www.python.org/ Qt >= 4.6 (free edition) http://www.trolltech.com/products/qt/ PyQt >= 4.5 (SIP is required to be installed first) http://www.riverbankcomputing.co.uk/software/pyqt/ http://www.riverbankcomputing.co.uk/software/sip/ numpy >= 1.0 http://numpy.scipy.org/ Optional requirements: h5py (optional for HDF5 support) http://www.h5py.org/ astropy >= 0.2 or PyFITS >= 1.1 (optional for FITS import) http://www.stsci.edu/resources/software_hardware/pyfits http://www.astropy.org/ pyemf >= 2.0.0 (optional for EMF export) http://pyemf.sourceforge.net/ PyMinuit >= 1.1.2 (optional improved fitting) http://code.google.com/p/pyminuit/ dbus-python, for dbus interface http://dbus.freedesktop.org/doc/dbus-python/ astropy (optional for VO table import) http://www.astropy.org/ SAMPy or astropy >= 0.4 (optional for SAMP support) http://pypi.python.org/pypi/sampy/ Veusz is Copyright (C) 2003-2014 Jeremy Sanders and contributors. It is licenced under the GPL (version 2 or greater). For documentation on using Veusz, see the "Documents" directory. The manual is in PDF, HTML and text format (generated from docbook). The examples are also useful documentation. Please also see and contribute to the Veusz wiki: https://github.com/jeremysanders/veusz/wiki Issues with the current version: * Due to a bug in the Qt XML processing, some MathML elements containing purely white space (e.g. thin space) will give an error. If you enjoy using Veusz, we would love to hear from you. Please join the mailing lists at https://gna.org/mail/?group=veusz to discuss new features or if you'd like to contribute code. The latest code can always be found in the Git repository at https://github.com/jeremysanders/veusz.git. From toddrjen at gmail.com Mon Oct 27 08:25:52 2014 From: toddrjen at gmail.com (Todd) Date: Mon, 27 Oct 2014 13:25:52 +0100 Subject: [SciPy-User] Documenting numpy vs. scipy fft Message-ID: As everyone here knows, both numpy and scipy have their own fft modules. I have looked at previous discussion on the mailing list, and it looks like scipy's is the recommended one. However, I can't find any mention of this in the numpy and scipy documentation. As far as I can see, neither numpy nor scipy even acknowledge in the documentation that there is an fft module in the other, not to mention provide any comparison or guidance to users on which they would want to use. If I didn't just miss it, might this be a good thing to have the differences between the two modules explained somewhere, and give users some hint as to which they would use (either in general or in specific situations)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Wed Oct 29 14:49:06 2014 From: toddrjen at gmail.com (Todd) Date: Wed, 29 Oct 2014 19:49:06 +0100 Subject: [SciPy-User] Documenting numpy vs. scipy fft In-Reply-To: References: Message-ID: On Oct 27, 2014 1:25 PM, "Todd" wrote: > > As everyone here knows, both numpy and scipy have their own fft modules. I have looked at previous discussion on the mailing list, and it looks like scipy's is the recommended one. > > However, I can't find any mention of this in the numpy and scipy documentation. As far as I can see, neither numpy nor scipy even acknowledge in the documentation that there is an fft module in the other, not to mention provide any comparison or guidance to users on which they would want to use. > > If I didn't just miss it, might this be a good thing to have the differences between the two modules explained somewhere, and give users some hint as to which they would use (either in general or in specific situations)? It seems they is as lot if discussion on how to move forward with overlap between numpy and scipy. However, in the short term there is still the issue of communicating to users how the modules relate to each other, how they differ, and when users should use one or the other. I see four possible solutions (besides simply deprecating one of the modules), although there may be more. These are not mutually exclusive, in fact they are probably complimentary: 1. Add documentation to the individual functions. This could be as simple as just a link to the corresponding function in the other project, although ideally it would have at least a sentence summarizing the major differences. 2. Add a section in one of the modules root documentation explaining in some detail the differences, with a link to that page in documentation for the other module. 3. A set of documentation on the scipy.org website explaining all the differences. This could even be automatically generated from 2. 4. A more general website comparing similar tools in the scientific python software stack. For this case, it could also cover other fft implementations like pyFFTW. However, it would also contain comparisons of other types of projects that seem superficially similar and where users just getting started might have trouble deciding what use, such as blaze vs. Pandas, the various plotting libraries, or the numerous physical units libraries. This could be either a conventional website, a wiki, or a blog with an index of links to the latest comparisons. -------------- next part -------------- An HTML attachment was scrubbed... URL: