From nxkryptor at gmail.com Sat Jul 2 05:02:51 2016 From: nxkryptor at gmail.com (nxkryptor nxkr) Date: Sat, 2 Jul 2016 12:02:51 +0300 Subject: [SciPy-User] How to optimize 2D data with leastsq in Python? Message-ID: I have data of the form given below: X Y1 Y2 Y3 Y4 Y5 Y61.42857 4.83 4.58 4.43 4.31 4.22 4.141.40845 3.87 3.63 3.49 3.38 3.3 3.231.38889 3.17 2.93 2.79 2.69 2.62 2.561.36986 2.65 2.41 2.27 2.17 2.11 2.051.35135 2.27 2.02 1.88 1.78 1.72 1.67 : : : : : : :1.01010 2.26 1.6 1.21 0.97 0.79 0.661.00000 2.01 1.44 1.1 0.88 0.73 0.62 I am trying to optimize a function of the form: [image: enter image description here] [image: enter image description here] Now my y variable has 6 columns, I can optimize one column (in MWE #3) which I have done in the MWE. How can I optimize the function to get the parameters A, n and B for all the 6 column values? In other words how can I get the value one value of A, n and B for all 6 values of y. *MWE* import numpy as npfrom scipy.signal import argrelextremaimport scipy.interpolatefrom scipy.optimize import leastsq with open('./exp_fit.dat', "r") as data: while True: line = data.readline() if not line.startswith('#'): break data_header = [i for i in line.strip().split('\t') if i] _data_ = np.genfromtxt(data, names = data_header, dtype = None, delimiter = '\t') _data_.dtype.names = [j.replace('_', ' ') for j in _data_.dtype.names] data = np.array(_data_.tolist()) x = _data_['X'] x_interp = np.linspace(x[-1], x[0], 100) y = np.zeros(shape = (len(x), 6)) y_interp = np.zeros(shape = (len(x_interp), 6))for i in range(6): y[:, i] = data[:, i + 1] tck = scipy.interpolate.splrep(x[::-1], y[::-1, i], s=0) y_interp[::-1, i] = scipy.interpolate.splev(x_interp, tck, der = 0) def residuals(p, y, x): A_lt, n_lt, B_lt, A_it, n_it, B_it, A_ht, n_ht, B_ht = p err = y - (1 / (1 / ((A_it * (15 ** (-n_it)) * np.exp(B_it * x / 1000)) + \ (A_lt * (15 ** (-n_lt)) * np.exp(B_lt * x / 1000))) + \ 1 / (A_ht * (15 ** (-n_ht)) * np.exp(B_ht * x / 1000)))) return err p1 = [2.789E-05, 1.47, 9.368E+03, 2.789E-05, 1.47, 9.368E+03, 2.789E-05, 1.47, 9.368E+03] plsq = leastsq(residuals, p1, args=(y_interp[:, 2], x_interp)) The question is also available on SO . nxkr -------------- next part -------------- An HTML attachment was scrubbed... URL: From leon.woo at db.com Sat Jul 2 08:02:15 2016 From: leon.woo at db.com (Leon Woo) Date: Sat, 2 Jul 2016 14:02:15 +0200 Subject: [SciPy-User] AUTO: Leon Woo is out of the office (returning 11/07/2016) Message-ID: I am out of the office until 11/07/2016. For standard requests within the scope of EMG PWM Berlin, please write to EMG PWM Berlin at DBEMEA. For non-standard requests, please cc Hien Pham-Thu. Note: This is an automated response to your message "SciPy-User Digest, Vol 155, Issue 1" sent on 02.07.2016 14:00:01. This is the only notification you will receive while this person is away.-- Informationen (einschlie?lich Pflichtangaben) zu einzelnen, innerhalb der EU t?tigen Gesellschaften und Zweigniederlassungen des Konzerns Deutsche Bank finden Sie unter http://www.deutsche-bank.de/de/content/pflichtangaben.htm. Diese E-Mail enth?lt vertrauliche und/ oder rechtlich gesch?tzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. Please refer to http://www.db.com/en/content/eu_disclosures.htm for information (including mandatory corporate particulars) on selected Deutsche Bank branches and group companies registered or incorporated in the European Union. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. -------------- next part -------------- An HTML attachment was scrubbed... URL: From newville at cars.uchicago.edu Sun Jul 3 19:10:49 2016 From: newville at cars.uchicago.edu (Matt Newville) Date: Sun, 3 Jul 2016 18:10:49 -0500 Subject: [SciPy-User] How to optimize 2D data with leastsq in Python? In-Reply-To: References: Message-ID: Nxkr, On Sat, Jul 2, 2016 at 4:02 AM, nxkryptor nxkr wrote: > I have data > of the > form given below: > > X Y1 Y2 Y3 Y4 Y5 Y61.42857 4.83 4.58 4.43 4.31 4.22 4.141.40845 3.87 3.63 3.49 3.38 3.3 3.231.38889 3.17 2.93 2.79 2.69 2.62 2.561.36986 2.65 2.41 2.27 2.17 2.11 2.051.35135 2.27 2.02 1.88 1.78 1.72 1.67 > : : : : : : :1.01010 2.26 1.6 1.21 0.97 0.79 0.661.00000 2.01 1.44 1.1 0.88 0.73 0.62 > > I am trying to optimize a function of the form: > > [image: enter image description here] [image: > enter image description here] > > Now my y variable has 6 columns, I can optimize one column (in MWE #3) > which I have done in the MWE. How can I optimize the function to get the > parameters A, n and B for all the 6 column values? In other words how can I > get the value one value of A, n and B for all 6 values of y. > > *MWE* > > import numpy as npfrom scipy.signal import argrelextremaimport scipy.interpolatefrom scipy.optimize import leastsq > > with open('./exp_fit.dat', "r") as data: > while True: > line = data.readline() > if not line.startswith('#'): > break > data_header = [i for i in line.strip().split('\t') if i] > _data_ = np.genfromtxt(data, names = data_header, dtype = None, delimiter = '\t') > _data_.dtype.names = [j.replace('_', ' ') for j in _data_.dtype.names] > data = np.array(_data_.tolist()) > > x = _data_['X'] > x_interp = np.linspace(x[-1], x[0], 100) > > y = np.zeros(shape = (len(x), 6)) > y_interp = np.zeros(shape = (len(x_interp), 6))for i in range(6): > y[:, i] = data[:, i + 1] > tck = scipy.interpolate.splrep(x[::-1], y[::-1, i], s=0) > y_interp[::-1, i] = scipy.interpolate.splev(x_interp, tck, der = 0) > > def residuals(p, y, x): > A_lt, n_lt, B_lt, A_it, n_it, B_it, A_ht, n_ht, B_ht = p > err = y - (1 / (1 / ((A_it * (15 ** (-n_it)) * np.exp(B_it * x / 1000)) + \ > (A_lt * (15 ** (-n_lt)) * np.exp(B_lt * x / 1000))) + \ > 1 / (A_ht * (15 ** (-n_ht)) * np.exp(B_ht * x / 1000)))) > return err > p1 = [2.789E-05, 1.47, 9.368E+03, 2.789E-05, 1.47, 9.368E+03, 2.789E-05, 1.47, 9.368E+03] > plsq = leastsq(residuals, p1, args=(y_interp[:, 2], x_interp)) > > The question is also available on SO . > > nxkr > > > I strongly recommend writing a function for your power-law/exponential function, and calling that 3 times. I assume your code does what you want, but it's far too messy to actually read. Also, it seems somewhat odd that you are interpolating your data onto a finer x-grid (but backwards??) rather than simply fitting the data you actually have. Maybe there's a good reason for that. Finally, I believe you can read in your data much more simply with data = np.loadtxt('./exp_fit.dat', skiprows=3) I'm afraid I do not actually understand what you are trying to do. You ask "How can I optimize the function to get the parameters A, n and B for all the 6 column values? In other words how can I get the value one value of A, n and B for all 6 values of y". Do you mean that you want to do 6 separate fits, one for each column? Or perhaps you mean that you want to fit 9 parameter values (3 each A, n, B for 3 of you exponential functions) to all of the data columns simultaneously? Or perhaps you mean something else entirely (for example that you mean one value for A, but a different value for n and B for each column). Sorry, but I can't tell. Finally, just so you're aware, fitting multiple damped exponential functions is often very difficult. --Matt Newville -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Jul 4 03:17:32 2016 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 4 Jul 2016 03:17:32 -0400 Subject: [SciPy-User] How to optimize 2D data with leastsq in Python? In-Reply-To: References: Message-ID: It looks to me that if P is a constant, then A_i and n_i are not separately identified. Josef On Sun, Jul 3, 2016 at 7:10 PM, Matt Newville wrote: > Nxkr, > > On Sat, Jul 2, 2016 at 4:02 AM, nxkryptor nxkr > wrote: > >> I have data >> of the >> form given below: >> >> X Y1 Y2 Y3 Y4 Y5 Y61.42857 4.83 4.58 4.43 4.31 4.22 4.141.40845 3.87 3.63 3.49 3.38 3.3 3.231.38889 3.17 2.93 2.79 2.69 2.62 2.561.36986 2.65 2.41 2.27 2.17 2.11 2.051.35135 2.27 2.02 1.88 1.78 1.72 1.67 >> : : : : : : :1.01010 2.26 1.6 1.21 0.97 0.79 0.661.00000 2.01 1.44 1.1 0.88 0.73 0.62 >> >> I am trying to optimize a function of the form: >> >> [image: enter image description here] >> [image: enter image description >> here] >> >> Now my y variable has 6 columns, I can optimize one column (in MWE #3) >> which I have done in the MWE. How can I optimize the function to get the >> parameters A, n and B for all the 6 column values? In other words how can I >> get the value one value of A, n and B for all 6 values of y. >> >> *MWE* >> >> import numpy as npfrom scipy.signal import argrelextremaimport scipy.interpolatefrom scipy.optimize import leastsq >> >> with open('./exp_fit.dat', "r") as data: >> while True: >> line = data.readline() >> if not line.startswith('#'): >> break >> data_header = [i for i in line.strip().split('\t') if i] >> _data_ = np.genfromtxt(data, names = data_header, dtype = None, delimiter = '\t') >> _data_.dtype.names = [j.replace('_', ' ') for j in _data_.dtype.names] >> data = np.array(_data_.tolist()) >> >> x = _data_['X'] >> x_interp = np.linspace(x[-1], x[0], 100) >> >> y = np.zeros(shape = (len(x), 6)) >> y_interp = np.zeros(shape = (len(x_interp), 6))for i in range(6): >> y[:, i] = data[:, i + 1] >> tck = scipy.interpolate.splrep(x[::-1], y[::-1, i], s=0) >> y_interp[::-1, i] = scipy.interpolate.splev(x_interp, tck, der = 0) >> >> def residuals(p, y, x): >> A_lt, n_lt, B_lt, A_it, n_it, B_it, A_ht, n_ht, B_ht = p >> err = y - (1 / (1 / ((A_it * (15 ** (-n_it)) * np.exp(B_it * x / 1000)) + \ >> (A_lt * (15 ** (-n_lt)) * np.exp(B_lt * x / 1000))) + \ >> 1 / (A_ht * (15 ** (-n_ht)) * np.exp(B_ht * x / 1000)))) >> return err >> p1 = [2.789E-05, 1.47, 9.368E+03, 2.789E-05, 1.47, 9.368E+03, 2.789E-05, 1.47, 9.368E+03] >> plsq = leastsq(residuals, p1, args=(y_interp[:, 2], x_interp)) >> >> The question is also available on SO . >> >> nxkr >> >> >> > I strongly recommend writing a function for your power-law/exponential > function, and calling that 3 times. I assume your code does what you > want, but it's far too messy to actually read. Also, it seems somewhat > odd that you are interpolating your data onto a finer x-grid (but > backwards??) rather than simply fitting the data you actually have. Maybe > there's a good reason for that. Finally, I believe you can read in your > data much more simply with > > data = np.loadtxt('./exp_fit.dat', skiprows=3) > > I'm afraid I do not actually understand what you are trying to do. You > ask "How can I optimize the function to get the parameters A, n and B for > all the 6 column values? In other words how can I get the value one value > of A, n and B for all 6 values of y". > > Do you mean that you want to do 6 separate fits, one for each column? > Or perhaps you mean that you want to fit 9 parameter values (3 each A, n, > B for 3 of you exponential functions) to all of the data columns > simultaneously? Or perhaps you mean something else entirely (for example > that you mean one value for A, but a different value for n and B for each > column). Sorry, but I can't tell. > > Finally, just so you're aware, fitting multiple damped exponential > functions is often very difficult. > > --Matt Newville > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mviljamaa at kapsi.fi Mon Jul 4 07:00:36 2016 From: mviljamaa at kapsi.fi (Matti Viljamaa) Date: Mon, 4 Jul 2016 14:00:36 +0300 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect Message-ID: I?m trying to use the scipy.signal.remez filter design function. What I?m doing is the following (with the help of http://www.ee.iitm.ac.in/~nitin/teaching/ee5480/firdesign.html ): import os import scipy.io.wavfile from scipy import signal from pylab import * os.chdir("/Users/mviljamaa/Music") sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) lpf = signal.remez(21, [0, 0.05, 0.1], [1.0, .0]) # Plot the magnitude response w, h = signal.freqz(lpf) plot(w/(2*pi), 20*log10(abs(h))) show() # Filtered data sout = signal.lfilter(lpf, 1, data) scipy.io.wavfile.write("equalized.wav", sr, sout) ? Now judging by the magnitude response of the filter lpf, it should be a lowpass of some sort (I?m not sure how to interpret the cutoff frequency yet). The output I get is substantially gained (up to the point where it sounds distorted) and I cannot hear the lowpass filter. What am I doing wrong? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mviljamaa at kapsi.fi Mon Jul 4 09:52:42 2016 From: mviljamaa at kapsi.fi (Matti Viljamaa) Date: Mon, 4 Jul 2016 16:52:42 +0300 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: Message-ID: > On 04 Jul 2016, at 14:00, Matti Viljamaa wrote: > > I?m trying to use the scipy.signal.remez filter design function. What I?m doing is the following (with the help of http://www.ee.iitm.ac.in/~nitin/teaching/ee5480/firdesign.html ): > > import os > import scipy.io.wavfile > from scipy import signal > from pylab import * > > os.chdir("/Users/mviljamaa/Music") > sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) > > lpf = signal.remez(21, [0, 0.05, 0.1], [1.0, .0]) > > # Plot the magnitude response > w, h = signal.freqz(lpf) > plot(w/(2*pi), 20*log10(abs(h))) > show() > > # Filtered data > sout = signal.lfilter(lpf, 1, data) > > scipy.io.wavfile.write("equalized.wav", sr, sout) > > ? > > Now judging by the magnitude response of the filter lpf, it should be a lowpass of some sort (I?m not sure how to interpret the cutoff frequency yet). > > The output I get is substantially gained (up to the point where it sounds distorted) and I cannot hear the lowpass filter. > > What am I doing wrong? Using a slightly modified code (from http://pastebin.com/LPEtXdzx ): import os import scipy.io.wavfile from scipy import signal from pylab import * import numpy as np os.chdir("/Users/mviljamaa/Music") sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) fs = 44100 bands = array([0,3500,4000,5500,6000,fs/2.0]) / fs desired = [1, 0, 0] lpf = signal.remez(513, bands, desired) from scipy.signal import freqz w, h = signal.freqz(lpf) plot(w/(2*pi), 20*log10(abs(h))) show() sout = signal.lfilter(lpf, 1, data) sout /= 1.05 * max(abs(sout)) scipy.io.wavfile.write("equalized.wav", sr, sout) ? This one is able to retain the gain, but I still don?t hear any lowpass filtering. -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Mon Jul 4 11:45:35 2016 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 4 Jul 2016 11:45:35 -0400 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: Message-ID: Matti, I don't have your input file, so I can't reproduce your result exactly. For a basic demonstration, it is probably easier to create a signal in the code instead of requiring an input wav file. Here's an example that does that. The input signal (called 'data') is the sum of two sine waves, one at 440 Hz and one at 3000 Hz. The cutoff frequency for the low-pass filter is 2000 Hz. (Actually the transition from the pass-band to the stop-band is from 1900 Hz to 2100 Hz.) You should be able to see in the plot and hear in the wav files that the high frequency component has been filtered out. Warren ----- from __future__ import division from scipy.signal import remez, freqz, lfilter from scipy.io import wavfile import numpy as np import matplotlib.pyplot as plt fs = 44100 # Design a low-pass filter using remez. cutoff = 2000.0 transition_width = 200 bands = np.array([0, cutoff - 0.5*transition_width, cutoff + 0.5*transition_width, fs/2.0]) / fs desired = [1, 0] lpf = remez(513, bands, desired) # Plot the frequency response of the filter. w, h = freqz(lpf) plt.figure(1) plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) plt.xlim(0, fs/2) plt.xlabel('Frequency (Hz)') plt.ylabel('Gain (dB)') plt.grid(True) # Create a test signal with two frequencies: one inside the pass-band, # are one far outside the pass-band that should be filtered out. T = 0.5 nsamples = int(T*fs) t = np.linspace(0, T, nsamples, endpoint=False) freq = 440 data = np.sin(2*np.pi*freq*t) data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) data /= 1.01*np.abs(data).max() # Filter the input using lfilter. (Alternatively, convolution could be used.) filtered_data = lfilter(lpf, 1, data) # Plot the input and output in the same figure. plt.figure(2) plt.plot(t, data) plt.plot(t, filtered_data) plt.show() # Save the test signal and the filtered signal as wav files. wavfile.write("data.wav", fs, data) wavfile.write("filtered_data.wav", fs, filtered_data) ----- On Mon, Jul 4, 2016 at 9:52 AM, Matti Viljamaa wrote: > > On 04 Jul 2016, at 14:00, Matti Viljamaa wrote: > > I?m trying to use the scipy.signal.remez filter design function. What I?m > doing is the following (with the help of > http://www.ee.iitm.ac.in/~nitin/teaching/ee5480/firdesign.html): > > import os > import scipy.io.wavfile > from scipy import signal > from pylab import * > > os.chdir("/Users/mviljamaa/Music") > sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) > > lpf = signal.remez(21, [0, 0.05, 0.1], [1.0, .0]) > > # Plot the magnitude response > w, h = signal.freqz(lpf) > plot(w/(2*pi), 20*log10(abs(h))) > show() > > # Filtered data > sout = signal.lfilter(lpf, 1, data) > > scipy.io.wavfile.write("equalized.wav", sr, sout) > > ? > > Now judging by the magnitude response of the filter lpf, it should be a > lowpass of some sort (I?m not sure how to interpret the cutoff frequency > yet). > > The output I get is substantially gained (up to the point where it sounds > distorted) and I cannot hear the lowpass filter. > > What am I doing wrong? > > > > Using a slightly modified code (from http://pastebin.com/LPEtXdzx): > > import os > import scipy.io.wavfile > from scipy import signal > from pylab import * > import numpy as np > > os.chdir("/Users/mviljamaa/Music") > sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) > > fs = 44100 > > bands = array([0,3500,4000,5500,6000,fs/2.0]) / fs > desired = [1, 0, 0] > lpf = signal.remez(513, bands, desired) > > from scipy.signal import freqz > w, h = signal.freqz(lpf) > plot(w/(2*pi), 20*log10(abs(h))) > show() > > sout = signal.lfilter(lpf, 1, data) > > sout /= 1.05 * max(abs(sout)) > > scipy.io.wavfile.write("equalized.wav", sr, sout) > > > ? > > This one is able to retain the gain, but I still don?t hear any lowpass > filtering. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Mon Jul 4 14:06:32 2016 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 04 Jul 2016 14:06:32 -0400 Subject: [SciPy-User] Online statistics (juliacon 2016) Message-ID: https://www.youtube.com/watch?v=3PpleRjaqeQ AFAIK, python ecosystem is missing online statistics. Myself, I've used wrappers for the boost::accumulators library to do some limited stats, but I'd love to see scipy offer more. From mviljamaa at kapsi.fi Wed Jul 6 07:53:41 2016 From: mviljamaa at kapsi.fi (Matti Viljamaa) Date: Wed, 6 Jul 2016 14:53:41 +0300 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: Message-ID: Are these the proper frequency domain plots+ w2, h2 = freqz(data) plt.figure(2) plt.plot(fs*w2/(2*np.pi), 20*np.log10(abs(h2))) plt.xlim(0, fs/2) plt.xlabel('Frequency (Hz)') plt.ylabel('Gain (dB)') plt.grid(True) w3, h3 = freqz(filtered_data) plt.figure(3) plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) plt.xlim(0, fs/2) plt.xlabel('Frequency (Hz)') plt.ylabel('Gain (dB)') plt.grid(True) plt. show() > On 04 Jul 2016, at 18:45, Warren Weckesser wrote: > > Matti, > > I don't have your input file, so I can't reproduce your result exactly. For a basic demonstration, it is probably easier to create a signal in the code instead of requiring an input wav file. Here's an example that does that. The input signal (called 'data') is the sum of two sine waves, one at 440 Hz and one at 3000 Hz. The cutoff frequency for the low-pass filter is 2000 Hz. (Actually the transition from the pass-band to the stop-band is from 1900 Hz to 2100 Hz.) You should be able to see in the plot and hear in the wav files that the high frequency component has been filtered out. > > Warren > > ----- > > from __future__ import division > > from scipy.signal import remez, freqz, lfilter > from scipy.io import wavfile > import numpy as np > import matplotlib.pyplot as plt > > > fs = 44100 > > # Design a low-pass filter using remez. > cutoff = 2000.0 > transition_width = 200 > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, fs/2.0]) / fs > desired = [1, 0] > lpf = remez(513, bands, desired) > > # Plot the frequency response of the filter. > w, h = freqz(lpf) > plt.figure(1) > plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) > plt.xlim(0, fs/2) > plt.xlabel('Frequency (Hz)') > plt.ylabel('Gain (dB)') > plt.grid(True) > > # Create a test signal with two frequencies: one inside the pass-band, > # are one far outside the pass-band that should be filtered out. > T = 0.5 > nsamples = int(T*fs) > t = np.linspace(0, T, nsamples, endpoint=False) > freq = 440 > data = np.sin(2*np.pi*freq*t) > data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) > data /= 1.01*np.abs(data).max() > > # Filter the input using lfilter. (Alternatively, convolution could be used.) > filtered_data = lfilter(lpf, 1, data) > > # Plot the input and output in the same figure. > plt.figure(2) > plt.plot(t, data) > plt.plot(t, filtered_data) > > plt.show() > > # Save the test signal and the filtered signal as wav files. > wavfile.write("data.wav", fs, data) > wavfile.write("filtered_data.wav", fs, filtered_data) > > > ----- > > > > > On Mon, Jul 4, 2016 at 9:52 AM, Matti Viljamaa > wrote: > >> On 04 Jul 2016, at 14:00, Matti Viljamaa > wrote: >> >> I?m trying to use the scipy.signal.remez filter design function. What I?m doing is the following (with the help of http://www.ee.iitm.ac.in/~nitin/teaching/ee5480/firdesign.html ): >> >> import os >> import scipy.io.wavfile >> from scipy import signal >> from pylab import * >> >> os.chdir("/Users/mviljamaa/Music") >> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >> >> lpf = signal.remez(21, [0, 0.05, 0.1], [1.0, .0]) >> >> # Plot the magnitude response >> w, h = signal.freqz(lpf) >> plot(w/(2*pi), 20*log10(abs(h))) >> show() >> >> # Filtered data >> sout = signal.lfilter(lpf, 1, data) >> >> scipy.io.wavfile.write("equalized.wav", sr, sout) >> >> ? >> >> Now judging by the magnitude response of the filter lpf, it should be a lowpass of some sort (I?m not sure how to interpret the cutoff frequency yet). >> >> The output I get is substantially gained (up to the point where it sounds distorted) and I cannot hear the lowpass filter. >> >> What am I doing wrong? > > > Using a slightly modified code (from http://pastebin.com/LPEtXdzx ): > > import os > import scipy.io.wavfile > from scipy import signal > from pylab import * > import numpy as np > > os.chdir("/Users/mviljamaa/Music") > sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) > > fs = 44100 > > bands = array([0,3500,4000,5500,6000,fs/2.0]) / fs > desired = [1, 0, 0] > lpf = signal.remez(513, bands, desired) > > from scipy.signal import freqz > w, h = signal.freqz(lpf) > plot(w/(2*pi), 20*log10(abs(h))) > show() > > sout = signal.lfilter(lpf, 1, data) > > sout /= 1.05 * max(abs(sout)) > > scipy.io.wavfile.write("equalized.wav", sr, sout) > > > ? > > This one is able to retain the gain, but I still don?t hear any lowpass filtering. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2016-07-06 at 14.53.08.png Type: image/png Size: 102569 bytes Desc: not available URL: From warren.weckesser at gmail.com Wed Jul 6 10:57:36 2016 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 6 Jul 2016 10:57:36 -0400 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: Message-ID: On Wed, Jul 6, 2016 at 7:53 AM, Matti Viljamaa wrote: > Are these the proper frequency domain plots+ > > w2, h2 = freqz(data) > > The arguments to 'freqz' are the filter coefficients, not the signal. See the example script in my previous email, where I have w, h = freqz(lpf) The attached plot is the frequency response that I get when I run that script. Warren > plt.figure(2) > plt.plot(fs*w2/(2*np.pi), 20*np.log10(abs(h2))) > plt.xlim(0, fs/2) > plt.xlabel('Frequency (Hz)') > plt.ylabel('Gain (dB)') > plt.grid(True) > > w3, h3 = freqz(filtered_data) > > plt.figure(3) > plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) > plt.xlim(0, fs/2) > plt.xlabel('Frequency (Hz)') > plt.ylabel('Gain (dB)') > plt.grid(True) > > plt. show() > > > On 04 Jul 2016, at 18:45, Warren Weckesser > wrote: > > Matti, > > I don't have your input file, so I can't reproduce your result exactly. > For a basic demonstration, it is probably easier to create a signal in the > code instead of requiring an input wav file. Here's an example that does > that. The input signal (called 'data') is the sum of two sine waves, one > at 440 Hz and one at 3000 Hz. The cutoff frequency for the low-pass filter > is 2000 Hz. (Actually the transition from the pass-band to the stop-band is > from 1900 Hz to 2100 Hz.) You should be able to see in the plot and hear > in the wav files that the high frequency component has been filtered out. > > Warren > > ----- > > from __future__ import division > > from scipy.signal import remez, freqz, lfilter > from scipy.io import wavfile > import numpy as np > import matplotlib.pyplot as plt > > > fs = 44100 > > # Design a low-pass filter using remez. > cutoff = 2000.0 > transition_width = 200 > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, fs/2.0]) / fs > desired = [1, 0] > lpf = remez(513, bands, desired) > > # Plot the frequency response of the filter. > w, h = freqz(lpf) > plt.figure(1) > plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) > plt.xlim(0, fs/2) > plt.xlabel('Frequency (Hz)') > plt.ylabel('Gain (dB)') > plt.grid(True) > > # Create a test signal with two frequencies: one inside the pass-band, > # are one far outside the pass-band that should be filtered out. > T = 0.5 > nsamples = int(T*fs) > t = np.linspace(0, T, nsamples, endpoint=False) > freq = 440 > data = np.sin(2*np.pi*freq*t) > data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) > data /= 1.01*np.abs(data).max() > > # Filter the input using lfilter. (Alternatively, convolution could be > used.) > filtered_data = lfilter(lpf, 1, data) > > # Plot the input and output in the same figure. > plt.figure(2) > plt.plot(t, data) > plt.plot(t, filtered_data) > > plt.show() > > # Save the test signal and the filtered signal as wav files. > wavfile.write("data.wav", fs, data) > wavfile.write("filtered_data.wav", fs, filtered_data) > > > ----- > > > > > On Mon, Jul 4, 2016 at 9:52 AM, Matti Viljamaa wrote: > >> >> On 04 Jul 2016, at 14:00, Matti Viljamaa wrote: >> >> I?m trying to use the scipy.signal.remez filter design function. What I?m >> doing is the following (with the help of >> http://www.ee.iitm.ac.in/~nitin/teaching/ee5480/firdesign.html): >> >> import os >> import scipy.io.wavfile >> from scipy import signal >> from pylab import * >> >> os.chdir("/Users/mviljamaa/Music") >> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >> >> lpf = signal.remez(21, [0, 0.05, 0.1], [1.0, .0]) >> >> # Plot the magnitude response >> w, h = signal.freqz(lpf) >> plot(w/(2*pi), 20*log10(abs(h))) >> show() >> >> # Filtered data >> sout = signal.lfilter(lpf, 1, data) >> >> scipy.io.wavfile.write("equalized.wav", sr, sout) >> >> ? >> >> Now judging by the magnitude response of the filter lpf, it should be a >> lowpass of some sort (I?m not sure how to interpret the cutoff frequency >> yet). >> >> The output I get is substantially gained (up to the point where it sounds >> distorted) and I cannot hear the lowpass filter. >> >> What am I doing wrong? >> >> >> >> Using a slightly modified code (from http://pastebin.com/LPEtXdzx): >> >> import os >> import scipy.io.wavfile >> from scipy import signal >> from pylab import * >> import numpy as np >> >> os.chdir("/Users/mviljamaa/Music") >> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >> >> fs = 44100 >> >> bands = array([0,3500,4000,5500,6000,fs/2.0]) / fs >> desired = [1, 0, 0] >> lpf = signal.remez(513, bands, desired) >> >> from scipy.signal import freqz >> w, h = signal.freqz(lpf) >> plot(w/(2*pi), 20*log10(abs(h))) >> show() >> >> sout = signal.lfilter(lpf, 1, data) >> >> sout /= 1.05 * max(abs(sout)) >> >> scipy.io.wavfile.write("equalized.wav", sr, sout) >> >> >> ? >> >> This one is able to retain the gain, but I still don?t hear any lowpass >> filtering. >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2016-07-06 at 14.53.08.png Type: image/png Size: 102569 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freqresp.png Type: image/png Size: 48423 bytes Desc: not available URL: From mviljamaa at kapsi.fi Thu Jul 7 06:08:48 2016 From: mviljamaa at kapsi.fi (Matti Viljamaa) Date: Thu, 7 Jul 2016 13:08:48 +0300 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: Message-ID: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> Any idea why this plot shows positive gain? Like +50dB around 500Hz? -Matti > On 06 Jul 2016, at 17:57, Warren Weckesser wrote: > > > > On Wed, Jul 6, 2016 at 7:53 AM, Matti Viljamaa > wrote: > Are these the proper frequency domain plots+ > > w2, h2 = freqz(data) > > > > The arguments to 'freqz' are the filter coefficients, not the signal. See the example script in my previous email, where I have > > w, h = freqz(lpf) > > The attached plot is the frequency response that I get when I run that script. > > Warren > > > > plt.figure(2) > plt.plot(fs*w2/(2*np.pi), 20*np.log10(abs(h2))) > plt.xlim(0, fs/2) > plt.xlabel('Frequency (Hz)') > plt.ylabel('Gain (dB)') > plt.grid(True) > > w3, h3 = freqz(filtered_data) > > plt.figure(3) > plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) > plt.xlim(0, fs/2) > plt.xlabel('Frequency (Hz)') > plt.ylabel('Gain (dB)') > plt.grid(True) > > plt. show() > > > >> On 04 Jul 2016, at 18:45, Warren Weckesser > wrote: >> >> Matti, >> >> I don't have your input file, so I can't reproduce your result exactly. For a basic demonstration, it is probably easier to create a signal in the code instead of requiring an input wav file. Here's an example that does that. The input signal (called 'data') is the sum of two sine waves, one at 440 Hz and one at 3000 Hz. The cutoff frequency for the low-pass filter is 2000 Hz. (Actually the transition from the pass-band to the stop-band is from 1900 Hz to 2100 Hz.) You should be able to see in the plot and hear in the wav files that the high frequency component has been filtered out. >> >> Warren >> >> ----- >> >> from __future__ import division >> >> from scipy.signal import remez, freqz, lfilter >> from scipy.io import wavfile >> import numpy as np >> import matplotlib.pyplot as plt >> >> >> fs = 44100 >> >> # Design a low-pass filter using remez. >> cutoff = 2000.0 >> transition_width = 200 >> bands = np.array([0, cutoff - 0.5*transition_width, >> cutoff + 0.5*transition_width, fs/2.0]) / fs >> desired = [1, 0] >> lpf = remez(513, bands, desired) >> >> # Plot the frequency response of the filter. >> w, h = freqz(lpf) >> plt.figure(1) >> plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) >> plt.xlim(0, fs/2) >> plt.xlabel('Frequency (Hz)') >> plt.ylabel('Gain (dB)') >> plt.grid(True) >> >> # Create a test signal with two frequencies: one inside the pass-band, >> # are one far outside the pass-band that should be filtered out. >> T = 0.5 >> nsamples = int(T*fs) >> t = np.linspace(0, T, nsamples, endpoint=False) >> freq = 440 >> data = np.sin(2*np.pi*freq*t) >> data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) >> data /= 1.01*np.abs(data).max() >> >> # Filter the input using lfilter. (Alternatively, convolution could be used.) >> filtered_data = lfilter(lpf, 1, data) >> >> # Plot the input and output in the same figure. >> plt.figure(2) >> plt.plot(t, data) >> plt.plot(t, filtered_data) >> >> plt.show() >> >> # Save the test signal and the filtered signal as wav files. >> wavfile.write("data.wav", fs, data) >> wavfile.write("filtered_data.wav", fs, filtered_data) >> >> >> ----- >> >> >> >> >> On Mon, Jul 4, 2016 at 9:52 AM, Matti Viljamaa > wrote: >> >>> On 04 Jul 2016, at 14:00, Matti Viljamaa > wrote: >>> >>> I?m trying to use the scipy.signal.remez filter design function. What I?m doing is the following (with the help of http://www.ee.iitm.ac.in/~nitin/teaching/ee5480/firdesign.html ): >>> >>> import os >>> import scipy.io.wavfile >>> from scipy import signal >>> from pylab import * >>> >>> os.chdir("/Users/mviljamaa/Music") >>> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >>> >>> lpf = signal.remez(21, [0, 0.05, 0.1], [1.0, .0]) >>> >>> # Plot the magnitude response >>> w, h = signal.freqz(lpf) >>> plot(w/(2*pi), 20*log10(abs(h))) >>> show() >>> >>> # Filtered data >>> sout = signal.lfilter(lpf, 1, data) >>> >>> scipy.io.wavfile.write("equalized.wav", sr, sout) >>> >>> ? >>> >>> Now judging by the magnitude response of the filter lpf, it should be a lowpass of some sort (I?m not sure how to interpret the cutoff frequency yet). >>> >>> The output I get is substantially gained (up to the point where it sounds distorted) and I cannot hear the lowpass filter. >>> >>> What am I doing wrong? >> >> >> Using a slightly modified code (from http://pastebin.com/LPEtXdzx ): >> >> import os >> import scipy.io.wavfile >> from scipy import signal >> from pylab import * >> import numpy as np >> >> os.chdir("/Users/mviljamaa/Music") >> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >> >> fs = 44100 >> >> bands = array([0,3500,4000,5500,6000,fs/2.0]) / fs >> desired = [1, 0, 0] >> lpf = signal.remez(513, bands, desired) >> >> from scipy.signal import freqz >> w, h = signal.freqz(lpf) >> plot(w/(2*pi), 20*log10(abs(h))) >> show() >> >> sout = signal.lfilter(lpf, 1, data) >> >> sout /= 1.05 * max(abs(sout)) >> >> scipy.io.wavfile.write("equalized.wav", sr, sout) >> >> >> ? >> >> This one is able to retain the gain, but I still don?t hear any lowpass filtering. >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From klonuo at gmail.com Fri Jul 8 08:45:15 2016 From: klonuo at gmail.com (klo uo) Date: Fri, 8 Jul 2016 14:45:15 +0200 Subject: [SciPy-User] Scikits portal broken Message-ID: Hi, http://scikits.appspot.com/scikits GAE code raises exception, apparently because pypi does allow only https protocol. Scikits portal is referenced in scipy documentation too: https://www.scipy.org/scikits.html#list-of-scikits -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Fri Jul 8 19:36:41 2016 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Sat, 9 Jul 2016 00:36:41 +0100 Subject: [SciPy-User] scipy 0.18.0 release candidate 2 Message-ID: On behalf of the scipy development team, I am pleased to announce the second release candidate for scipy 0.18.0. Please test it --- both the release itself on your machines and your code against this release --- and report results on the issue tracker or scipy-dev mailing list. Source tarballs and release notes are available from Github releases, https://github.com/scipy/scipy/releases/tag/v0.18.0rc2 OS X and Linux wheels are being built now, and I will upload them to PyPI when they are ready. I would like to flag two areas where additional scrutiny and testing would be most appreciated: * Precision loss in some corner cases was reported to occur on Fedora 23 and 24 with OpenBLAS, see https://github.com/scipy/scipy/issues/6286. * On Windows, a CRT incompatibility was reported when building with MSVC and MKL, see https://github.com/scipy/scipy/issues/6336. There are additional issues on s390x (https://github.com/scipy/scipy/issues/6338), so if you are using this platform, we appreciate your help resolving them. The (revised) release schedule is: 25 July: final release Thanks to everyone who contributed! Cheers, Evgeni From mviljamaa at kapsi.fi Sun Jul 10 12:00:23 2016 From: mviljamaa at kapsi.fi (Matti Viljamaa) Date: Sun, 10 Jul 2016 19:00:23 +0300 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> Message-ID: <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> Or why doing fs = 44100 # Design a low-pass filter using remez. cutoff = 2000.0 transition_width = 200 bands = np.array([0, cutoff - 0.5*transition_width, cutoff + 0.5*transition_width, fs/2.0]) / fs desired = [0, 1] # <------ lpf = remez(513, bands, desired) T = 0.5 nsamples = int(T*fs) t = np.linspace(0, T, nsamples, endpoint=False) freq = 440 data = np.sin(2*np.pi*freq*t) data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) data /= 1.01*np.abs(data).max() filtered_data = lfilter(lpf, 1, data) w3, h3 = freqz(filtered_data) plt.figure(3) plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) plt.xlim(0, fs/2) plt.xlabel('Frequency (Hz)') plt.ylabel('Gain (dB)') plt.grid(True) plt. show() gives which doesn?t look like a highpass? > On 07 Jul 2016, at 13:08, Matti Viljamaa wrote: > > Any idea why this plot shows positive gain? > Like +50dB around 500Hz? > > -Matti > >> On 06 Jul 2016, at 17:57, Warren Weckesser > wrote: >> >> >> >> On Wed, Jul 6, 2016 at 7:53 AM, Matti Viljamaa > wrote: >> Are these the proper frequency domain plots+ >> >> w2, h2 = freqz(data) >> >> >> >> The arguments to 'freqz' are the filter coefficients, not the signal. See the example script in my previous email, where I have >> >> w, h = freqz(lpf) >> >> The attached plot is the frequency response that I get when I run that script. >> >> Warren >> >> >> >> plt.figure(2) >> plt.plot(fs*w2/(2*np.pi), 20*np.log10(abs(h2))) >> plt.xlim(0, fs/2) >> plt.xlabel('Frequency (Hz)') >> plt.ylabel('Gain (dB)') >> plt.grid(True) >> >> w3, h3 = freqz(filtered_data) >> >> plt.figure(3) >> plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) >> plt.xlim(0, fs/2) >> plt.xlabel('Frequency (Hz)') >> plt.ylabel('Gain (dB)') >> plt.grid(True) >> >> plt. show() >> >> >> >>> On 04 Jul 2016, at 18:45, Warren Weckesser > wrote: >>> >>> Matti, >>> >>> I don't have your input file, so I can't reproduce your result exactly. For a basic demonstration, it is probably easier to create a signal in the code instead of requiring an input wav file. Here's an example that does that. The input signal (called 'data') is the sum of two sine waves, one at 440 Hz and one at 3000 Hz. The cutoff frequency for the low-pass filter is 2000 Hz. (Actually the transition from the pass-band to the stop-band is from 1900 Hz to 2100 Hz.) You should be able to see in the plot and hear in the wav files that the high frequency component has been filtered out. >>> >>> Warren >>> >>> ----- >>> >>> from __future__ import division >>> >>> from scipy.signal import remez, freqz, lfilter >>> from scipy.io import wavfile >>> import numpy as np >>> import matplotlib.pyplot as plt >>> >>> >>> fs = 44100 >>> >>> # Design a low-pass filter using remez. >>> cutoff = 2000.0 >>> transition_width = 200 >>> bands = np.array([0, cutoff - 0.5*transition_width, >>> cutoff + 0.5*transition_width, fs/2.0]) / fs >>> desired = [1, 0] >>> lpf = remez(513, bands, desired) >>> >>> # Plot the frequency response of the filter. >>> w, h = freqz(lpf) >>> plt.figure(1) >>> plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) >>> plt.xlim(0, fs/2) >>> plt.xlabel('Frequency (Hz)') >>> plt.ylabel('Gain (dB)') >>> plt.grid(True) >>> >>> # Create a test signal with two frequencies: one inside the pass-band, >>> # are one far outside the pass-band that should be filtered out. >>> T = 0.5 >>> nsamples = int(T*fs) >>> t = np.linspace(0, T, nsamples, endpoint=False) >>> freq = 440 >>> data = np.sin(2*np.pi*freq*t) >>> data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) >>> data /= 1.01*np.abs(data).max() >>> >>> # Filter the input using lfilter. (Alternatively, convolution could be used.) >>> filtered_data = lfilter(lpf, 1, data) >>> >>> # Plot the input and output in the same figure. >>> plt.figure(2) >>> plt.plot(t, data) >>> plt.plot(t, filtered_data) >>> >>> plt.show() >>> >>> # Save the test signal and the filtered signal as wav files. >>> wavfile.write("data.wav", fs, data) >>> wavfile.write("filtered_data.wav", fs, filtered_data) >>> >>> >>> ----- >>> >>> >>> >>> >>> On Mon, Jul 4, 2016 at 9:52 AM, Matti Viljamaa > wrote: >>> >>>> On 04 Jul 2016, at 14:00, Matti Viljamaa > wrote: >>>> >>>> I?m trying to use the scipy.signal.remez filter design function. What I?m doing is the following (with the help of http://www.ee.iitm.ac.in/~nitin/teaching/ee5480/firdesign.html ): >>>> >>>> import os >>>> import scipy.io.wavfile >>>> from scipy import signal >>>> from pylab import * >>>> >>>> os.chdir("/Users/mviljamaa/Music") >>>> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >>>> >>>> lpf = signal.remez(21, [0, 0.05, 0.1], [1.0, .0]) >>>> >>>> # Plot the magnitude response >>>> w, h = signal.freqz(lpf) >>>> plot(w/(2*pi), 20*log10(abs(h))) >>>> show() >>>> >>>> # Filtered data >>>> sout = signal.lfilter(lpf, 1, data) >>>> >>>> scipy.io.wavfile.write("equalized.wav", sr, sout) >>>> >>>> ? >>>> >>>> Now judging by the magnitude response of the filter lpf, it should be a lowpass of some sort (I?m not sure how to interpret the cutoff frequency yet). >>>> >>>> The output I get is substantially gained (up to the point where it sounds distorted) and I cannot hear the lowpass filter. >>>> >>>> What am I doing wrong? >>> >>> >>> Using a slightly modified code (from http://pastebin.com/LPEtXdzx ): >>> >>> import os >>> import scipy.io.wavfile >>> from scipy import signal >>> from pylab import * >>> import numpy as np >>> >>> os.chdir("/Users/mviljamaa/Music") >>> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >>> >>> fs = 44100 >>> >>> bands = array([0,3500,4000,5500,6000,fs/2.0]) / fs >>> desired = [1, 0, 0] >>> lpf = signal.remez(513, bands, desired) >>> >>> from scipy.signal import freqz >>> w, h = signal.freqz(lpf) >>> plot(w/(2*pi), 20*log10(abs(h))) >>> show() >>> >>> sout = signal.lfilter(lpf, 1, data) >>> >>> sout /= 1.05 * max(abs(sout)) >>> >>> scipy.io.wavfile.write("equalized.wav", sr, sout) >>> >>> >>> ? >>> >>> This one is able to retain the gain, but I still don?t hear any lowpass filtering. >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2016-07-10 at 18.58.09.png Type: image/png Size: 81049 bytes Desc: not available URL: From warren.weckesser at gmail.com Sun Jul 10 12:04:54 2016 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 10 Jul 2016 12:04:54 -0400 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> Message-ID: On Sun, Jul 10, 2016 at 12:00 PM, Matti Viljamaa wrote: > Or why doing > > fs = 44100 > > # Design a low-pass filter using remez. > cutoff = 2000.0 > transition_width = 200 > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, fs/2.0]) / fs > desired = [0, 1] # <------ > lpf = remez(513, bands, desired) > > T = 0.5 > nsamples = int(T*fs) > t = np.linspace(0, T, nsamples, endpoint=False) > freq = 440 > data = np.sin(2*np.pi*freq*t) > data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) > data /= 1.01*np.abs(data).max() > > filtered_data = lfilter(lpf, 1, data) > > w3, h3 = freqz(filtered_data) > > Matti, Did you see the email that I sent on July 6, and the plot that I attached to it? Here's what I said: The arguments to 'freqz' are the filter coefficients, not the signal. See the example script in my previous email, where I have w, h = freqz(lpf) Warren > plt.figure(3) > plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) > plt.xlim(0, fs/2) > plt.xlabel('Frequency (Hz)') > plt.ylabel('Gain (dB)') > plt.grid(True) > > plt. show() > > gives > > > which doesn?t look like a highpass? > > > On 07 Jul 2016, at 13:08, Matti Viljamaa wrote: > > Any idea why this plot shows positive gain? > Like +50dB around 500Hz? > > -Matti > > On 06 Jul 2016, at 17:57, Warren Weckesser > wrote: > > > > On Wed, Jul 6, 2016 at 7:53 AM, Matti Viljamaa wrote: > >> Are these the proper frequency domain plots+ >> >> w2, h2 = freqz(data) >> >> > > The arguments to 'freqz' are the filter coefficients, not the signal. See > the example script in my previous email, where I have > > w, h = freqz(lpf) > > The attached plot is the frequency response that I get when I run that > script. > > Warren > > > > >> plt.figure(2) >> plt.plot(fs*w2/(2*np.pi), 20*np.log10(abs(h2))) >> plt.xlim(0, fs/2) >> plt.xlabel('Frequency (Hz)') >> plt.ylabel('Gain (dB)') >> plt.grid(True) >> >> w3, h3 = freqz(filtered_data) >> >> plt.figure(3) >> plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) >> plt.xlim(0, fs/2) >> plt.xlabel('Frequency (Hz)') >> plt.ylabel('Gain (dB)') >> plt.grid(True) >> >> plt. show() >> >> >> >> On 04 Jul 2016, at 18:45, Warren Weckesser >> wrote: >> >> Matti, >> >> I don't have your input file, so I can't reproduce your result exactly. >> For a basic demonstration, it is probably easier to create a signal in the >> code instead of requiring an input wav file. Here's an example that does >> that. The input signal (called 'data') is the sum of two sine waves, one >> at 440 Hz and one at 3000 Hz. The cutoff frequency for the low-pass filter >> is 2000 Hz. (Actually the transition from the pass-band to the stop-band is >> from 1900 Hz to 2100 Hz.) You should be able to see in the plot and hear >> in the wav files that the high frequency component has been filtered out. >> >> Warren >> >> ----- >> >> from __future__ import division >> >> from scipy.signal import remez, freqz, lfilter >> from scipy.io import wavfile >> import numpy as np >> import matplotlib.pyplot as plt >> >> >> fs = 44100 >> >> # Design a low-pass filter using remez. >> cutoff = 2000.0 >> transition_width = 200 >> bands = np.array([0, cutoff - 0.5*transition_width, >> cutoff + 0.5*transition_width, fs/2.0]) / fs >> desired = [1, 0] >> lpf = remez(513, bands, desired) >> >> # Plot the frequency response of the filter. >> w, h = freqz(lpf) >> plt.figure(1) >> plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) >> plt.xlim(0, fs/2) >> plt.xlabel('Frequency (Hz)') >> plt.ylabel('Gain (dB)') >> plt.grid(True) >> >> # Create a test signal with two frequencies: one inside the pass-band, >> # are one far outside the pass-band that should be filtered out. >> T = 0.5 >> nsamples = int(T*fs) >> t = np.linspace(0, T, nsamples, endpoint=False) >> freq = 440 >> data = np.sin(2*np.pi*freq*t) >> data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) >> data /= 1.01*np.abs(data).max() >> >> # Filter the input using lfilter. (Alternatively, convolution could be >> used.) >> filtered_data = lfilter(lpf, 1, data) >> >> # Plot the input and output in the same figure. >> plt.figure(2) >> plt.plot(t, data) >> plt.plot(t, filtered_data) >> >> plt.show() >> >> # Save the test signal and the filtered signal as wav files. >> wavfile.write("data.wav", fs, data) >> wavfile.write("filtered_data.wav", fs, filtered_data) >> >> >> ----- >> >> >> >> >> On Mon, Jul 4, 2016 at 9:52 AM, Matti Viljamaa >> wrote: >> >>> >>> On 04 Jul 2016, at 14:00, Matti Viljamaa wrote: >>> >>> I?m trying to use the scipy.signal.remez filter design function. What >>> I?m doing is the following (with the help of >>> http://www.ee.iitm.ac.in/~nitin/teaching/ee5480/firdesign.html): >>> >>> import os >>> import scipy.io.wavfile >>> from scipy import signal >>> from pylab import * >>> >>> os.chdir("/Users/mviljamaa/Music") >>> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >>> >>> lpf = signal.remez(21, [0, 0.05, 0.1], [1.0, .0]) >>> >>> # Plot the magnitude response >>> w, h = signal.freqz(lpf) >>> plot(w/(2*pi), 20*log10(abs(h))) >>> show() >>> >>> # Filtered data >>> sout = signal.lfilter(lpf, 1, data) >>> >>> scipy.io.wavfile.write("equalized.wav", sr, sout) >>> >>> ? >>> >>> Now judging by the magnitude response of the filter lpf, it should be a >>> lowpass of some sort (I?m not sure how to interpret the cutoff frequency >>> yet). >>> >>> The output I get is substantially gained (up to the point where it >>> sounds distorted) and I cannot hear the lowpass filter. >>> >>> What am I doing wrong? >>> >>> >>> >>> Using a slightly modified code (from http://pastebin.com/LPEtXdzx): >>> >>> import os >>> import scipy.io.wavfile >>> from scipy import signal >>> from pylab import * >>> import numpy as np >>> >>> os.chdir("/Users/mviljamaa/Music") >>> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >>> >>> fs = 44100 >>> >>> bands = array([0,3500,4000,5500,6000,fs/2.0]) / fs >>> desired = [1, 0, 0] >>> lpf = signal.remez(513, bands, desired) >>> >>> from scipy.signal import freqz >>> w, h = signal.freqz(lpf) >>> plot(w/(2*pi), 20*log10(abs(h))) >>> show() >>> >>> sout = signal.lfilter(lpf, 1, data) >>> >>> sout /= 1.05 * max(abs(sout)) >>> >>> scipy.io.wavfile.write("equalized.wav", sr, sout) >>> >>> >>> ? >>> >>> This one is able to retain the gain, but I still don?t hear any lowpass >>> filtering. >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2016-07-10 at 18.58.09.png Type: image/png Size: 81049 bytes Desc: not available URL: From mviljamaa at kapsi.fi Sun Jul 10 12:08:49 2016 From: mviljamaa at kapsi.fi (Matti Viljamaa) Date: Sun, 10 Jul 2016 19:08:49 +0300 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> Message-ID: <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> > On 10 Jul 2016, at 19:04, Warren Weckesser wrote: > > > > On Sun, Jul 10, 2016 at 12:00 PM, Matti Viljamaa > wrote: > Or why doing > > fs = 44100 > > # Design a low-pass filter using remez. > cutoff = 2000.0 > transition_width = 200 > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, fs/2.0]) / fs > desired = [0, 1] # <------ > lpf = remez(513, bands, desired) > > T = 0.5 > nsamples = int(T*fs) > t = np.linspace(0, T, nsamples, endpoint=False) > freq = 440 > data = np.sin(2*np.pi*freq*t) > data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) > data /= 1.01*np.abs(data).max() > > filtered_data = lfilter(lpf, 1, data) > > w3, h3 = freqz(filtered_data) > > > > Matti, > > Did you see the email that I sent on July 6, and the plot that I attached to it? Here's what I said: > > The arguments to 'freqz' are the filter coefficients, not the signal. See the example script in my previous email, where I have > > w, h = freqz(lpf) > > Warren But what if I want to plot the effect of the lpf on the signal, rather than the filter magnitude response? -Matti > > plt.figure(3) > plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) > plt.xlim(0, fs/2) > plt.xlabel('Frequency (Hz)') > plt.ylabel('Gain (dB)') > plt.grid(True) > > plt. show() > > gives > > > > which doesn?t look like a highpass? > > >> On 07 Jul 2016, at 13:08, Matti Viljamaa > wrote: >> >> Any idea why this plot shows positive gain? >> Like +50dB around 500Hz? >> >> -Matti >> >>> On 06 Jul 2016, at 17:57, Warren Weckesser > wrote: >>> >>> >>> >>> On Wed, Jul 6, 2016 at 7:53 AM, Matti Viljamaa > wrote: >>> Are these the proper frequency domain plots+ >>> >>> w2, h2 = freqz(data) >>> >>> >>> >>> The arguments to 'freqz' are the filter coefficients, not the signal. See the example script in my previous email, where I have >>> >>> w, h = freqz(lpf) >>> >>> The attached plot is the frequency response that I get when I run that script. >>> >>> Warren >>> >>> >>> >>> plt.figure(2) >>> plt.plot(fs*w2/(2*np.pi), 20*np.log10(abs(h2))) >>> plt.xlim(0, fs/2) >>> plt.xlabel('Frequency (Hz)') >>> plt.ylabel('Gain (dB)') >>> plt.grid(True) >>> >>> w3, h3 = freqz(filtered_data) >>> >>> plt.figure(3) >>> plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) >>> plt.xlim(0, fs/2) >>> plt.xlabel('Frequency (Hz)') >>> plt.ylabel('Gain (dB)') >>> plt.grid(True) >>> >>> plt. show() >>> >>> >>> >>>> On 04 Jul 2016, at 18:45, Warren Weckesser > wrote: >>>> >>>> Matti, >>>> >>>> I don't have your input file, so I can't reproduce your result exactly. For a basic demonstration, it is probably easier to create a signal in the code instead of requiring an input wav file. Here's an example that does that. The input signal (called 'data') is the sum of two sine waves, one at 440 Hz and one at 3000 Hz. The cutoff frequency for the low-pass filter is 2000 Hz. (Actually the transition from the pass-band to the stop-band is from 1900 Hz to 2100 Hz.) You should be able to see in the plot and hear in the wav files that the high frequency component has been filtered out. >>>> >>>> Warren >>>> >>>> ----- >>>> >>>> from __future__ import division >>>> >>>> from scipy.signal import remez, freqz, lfilter >>>> from scipy.io import wavfile >>>> import numpy as np >>>> import matplotlib.pyplot as plt >>>> >>>> >>>> fs = 44100 >>>> >>>> # Design a low-pass filter using remez. >>>> cutoff = 2000.0 >>>> transition_width = 200 >>>> bands = np.array([0, cutoff - 0.5*transition_width, >>>> cutoff + 0.5*transition_width, fs/2.0]) / fs >>>> desired = [1, 0] >>>> lpf = remez(513, bands, desired) >>>> >>>> # Plot the frequency response of the filter. >>>> w, h = freqz(lpf) >>>> plt.figure(1) >>>> plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) >>>> plt.xlim(0, fs/2) >>>> plt.xlabel('Frequency (Hz)') >>>> plt.ylabel('Gain (dB)') >>>> plt.grid(True) >>>> >>>> # Create a test signal with two frequencies: one inside the pass-band, >>>> # are one far outside the pass-band that should be filtered out. >>>> T = 0.5 >>>> nsamples = int(T*fs) >>>> t = np.linspace(0, T, nsamples, endpoint=False) >>>> freq = 440 >>>> data = np.sin(2*np.pi*freq*t) >>>> data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) >>>> data /= 1.01*np.abs(data).max() >>>> >>>> # Filter the input using lfilter. (Alternatively, convolution could be used.) >>>> filtered_data = lfilter(lpf, 1, data) >>>> >>>> # Plot the input and output in the same figure. >>>> plt.figure(2) >>>> plt.plot(t, data) >>>> plt.plot(t, filtered_data) >>>> >>>> plt.show() >>>> >>>> # Save the test signal and the filtered signal as wav files. >>>> wavfile.write("data.wav", fs, data) >>>> wavfile.write("filtered_data.wav", fs, filtered_data) >>>> >>>> >>>> ----- >>>> >>>> >>>> >>>> >>>> On Mon, Jul 4, 2016 at 9:52 AM, Matti Viljamaa > wrote: >>>> >>>>> On 04 Jul 2016, at 14:00, Matti Viljamaa > wrote: >>>>> >>>>> I?m trying to use the scipy.signal.remez filter design function. What I?m doing is the following (with the help of http://www.ee.iitm.ac.in/~nitin/teaching/ee5480/firdesign.html ): >>>>> >>>>> import os >>>>> import scipy.io.wavfile >>>>> from scipy import signal >>>>> from pylab import * >>>>> >>>>> os.chdir("/Users/mviljamaa/Music") >>>>> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >>>>> >>>>> lpf = signal.remez(21, [0, 0.05, 0.1], [1.0, .0]) >>>>> >>>>> # Plot the magnitude response >>>>> w, h = signal.freqz(lpf) >>>>> plot(w/(2*pi), 20*log10(abs(h))) >>>>> show() >>>>> >>>>> # Filtered data >>>>> sout = signal.lfilter(lpf, 1, data) >>>>> >>>>> scipy.io.wavfile.write("equalized.wav", sr, sout) >>>>> >>>>> ? >>>>> >>>>> Now judging by the magnitude response of the filter lpf, it should be a lowpass of some sort (I?m not sure how to interpret the cutoff frequency yet). >>>>> >>>>> The output I get is substantially gained (up to the point where it sounds distorted) and I cannot hear the lowpass filter. >>>>> >>>>> What am I doing wrong? >>>> >>>> >>>> Using a slightly modified code (from http://pastebin.com/LPEtXdzx ): >>>> >>>> import os >>>> import scipy.io.wavfile >>>> from scipy import signal >>>> from pylab import * >>>> import numpy as np >>>> >>>> os.chdir("/Users/mviljamaa/Music") >>>> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >>>> >>>> fs = 44100 >>>> >>>> bands = array([0,3500,4000,5500,6000,fs/2.0]) / fs >>>> desired = [1, 0, 0] >>>> lpf = signal.remez(513, bands, desired) >>>> >>>> from scipy.signal import freqz >>>> w, h = signal.freqz(lpf) >>>> plot(w/(2*pi), 20*log10(abs(h))) >>>> show() >>>> >>>> sout = signal.lfilter(lpf, 1, data) >>>> >>>> sout /= 1.05 * max(abs(sout)) >>>> >>>> scipy.io.wavfile.write("equalized.wav", sr, sout) >>>> >>>> >>>> ? >>>> >>>> This one is able to retain the gain, but I still don?t hear any lowpass filtering. >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Sun Jul 10 12:57:56 2016 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 10 Jul 2016 12:57:56 -0400 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> Message-ID: On Sun, Jul 10, 2016 at 12:08 PM, Matti Viljamaa wrote: > > On 10 Jul 2016, at 19:04, Warren Weckesser > wrote: > > > > On Sun, Jul 10, 2016 at 12:00 PM, Matti Viljamaa > wrote: > >> Or why doing >> >> fs = 44100 >> >> # Design a low-pass filter using remez. >> cutoff = 2000.0 >> transition_width = 200 >> bands = np.array([0, cutoff - 0.5*transition_width, >> cutoff + 0.5*transition_width, fs/2.0]) / fs >> desired = [0, 1] # <------ >> lpf = remez(513, bands, desired) >> >> T = 0.5 >> nsamples = int(T*fs) >> t = np.linspace(0, T, nsamples, endpoint=False) >> freq = 440 >> data = np.sin(2*np.pi*freq*t) >> data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) >> data /= 1.01*np.abs(data).max() >> >> filtered_data = lfilter(lpf, 1, data) >> >> w3, h3 = freqz(filtered_data) >> >> > > Matti, > > Did you see the email that I sent on July 6, and the plot that I attached > to it? Here's what I said: > > The arguments to 'freqz' are the filter coefficients, not the signal. See > the example script in my previous email, where I have > > w, h = freqz(lpf) > > Warren > > > > But what if I want to plot the effect of the lpf on the signal, rather > than the filter magnitude response? > > For that, you want to plot the spectral content of the original and filtered data. You can use a Fourier transform and plot the magnitude of the Fourier coefficients against the frequencies, or you can use a function such as `scipy.signal.periodogram` or `scipy.signal.welch` which take care of the FFT details for you. Here's a new version of my script. It now creates a third figure containing plots of the periodogram of the original data and the filtered data. I used `scipy.signal.periodogram`, but I recommend experimenting with `welch` also. Warren ---------- from __future__ import division from scipy.signal import remez, freqz, lfilter, periodogram from scipy.io import wavfile import numpy as np import matplotlib.pyplot as plt fs = 44100 # Design a low-pass filter using remez. cutoff = 2000.0 transition_width = 200 bands = np.array([0, cutoff - 0.5*transition_width, cutoff + 0.5*transition_width, fs/2.0]) / fs desired = [1, 0] lpf = remez(513, bands, desired) # Plot the frequency response of the filter. w, h = freqz(lpf) plt.figure(1) plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) plt.xlim(0, fs/2) plt.xlabel('Frequency (Hz)') plt.ylabel('Gain (dB)') plt.grid(True) # Create a test signal with three frequencies: two inside the pass-band, # are one far outside the pass-band that should be filtered out. T = 0.5 nsamples = int(T*fs) t = np.linspace(0, T, nsamples, endpoint=False) freq = 440 data = np.sin(2*np.pi*freq*t) + 0.5*np.cos(2*np.pi*(2*freq)*t) data += 0.75*np.sin(2*np.pi*(cutoff + 5*transition_width)*t) data /= 1.01*np.abs(data).max() # Filter the input using lfilter. (Alternatively, convolution could be used.) filtered_data = lfilter(lpf, 1, data) # Plot the input and output in the same figure. plt.figure(2) plt.plot(t, data, 'b', label='original') plt.plot(t, filtered_data, 'g', label='filtered') plt.xlabel('Time (sec)') plt.ylim(-1.1, 1.1) plt.legend() plt.figure(3) freqs, data_spec = periodogram(data, fs=fs) freqs, filtered_data_spec = periodogram(filtered_data, fs=fs) plt.subplot(2, 1, 1) plt.plot(freqs, data_spec, 'b', label='original') plt.xlim(0, 3*cutoff) plt.axvline(cutoff, color='k', alpha=0.15) plt.legend() plt.subplot(2, 1, 2) plt.plot(freqs, filtered_data_spec, 'g', label='filtered') plt.xlim(0, 3*cutoff) plt.axvline(cutoff, color='k', alpha=0.15) plt.legend() plt.xlabel('Frequency (Hz)') plt.show() # Save the test signal and the filtered signal as wav files. wavfile.write("data.wav", fs, data) wavfile.write("filtered_data.wav", fs, filtered_data) ---------- > -Matti > > > >> plt.figure(3) >> plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) >> plt.xlim(0, fs/2) >> plt.xlabel('Frequency (Hz)') >> plt.ylabel('Gain (dB)') >> plt.grid(True) >> >> plt. show() >> >> gives >> >> >> >> which doesn?t look like a highpass? >> >> >> On 07 Jul 2016, at 13:08, Matti Viljamaa wrote: >> >> Any idea why this plot shows positive gain? >> Like +50dB around 500Hz? >> >> -Matti >> >> On 06 Jul 2016, at 17:57, Warren Weckesser >> wrote: >> >> >> >> On Wed, Jul 6, 2016 at 7:53 AM, Matti Viljamaa >> wrote: >> >>> Are these the proper frequency domain plots+ >>> >>> w2, h2 = freqz(data) >>> >>> >> >> The arguments to 'freqz' are the filter coefficients, not the signal. >> See the example script in my previous email, where I have >> >> w, h = freqz(lpf) >> >> The attached plot is the frequency response that I get when I run that >> script. >> >> Warren >> >> >> >> >>> plt.figure(2) >>> plt.plot(fs*w2/(2*np.pi), 20*np.log10(abs(h2))) >>> plt.xlim(0, fs/2) >>> plt.xlabel('Frequency (Hz)') >>> plt.ylabel('Gain (dB)') >>> plt.grid(True) >>> >>> w3, h3 = freqz(filtered_data) >>> >>> plt.figure(3) >>> plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) >>> plt.xlim(0, fs/2) >>> plt.xlabel('Frequency (Hz)') >>> plt.ylabel('Gain (dB)') >>> plt.grid(True) >>> >>> plt. show() >>> >>> >>> >>> On 04 Jul 2016, at 18:45, Warren Weckesser >>> wrote: >>> >>> Matti, >>> >>> I don't have your input file, so I can't reproduce your result exactly. >>> For a basic demonstration, it is probably easier to create a signal in the >>> code instead of requiring an input wav file. Here's an example that does >>> that. The input signal (called 'data') is the sum of two sine waves, one >>> at 440 Hz and one at 3000 Hz. The cutoff frequency for the low-pass filter >>> is 2000 Hz. (Actually the transition from the pass-band to the stop-band is >>> from 1900 Hz to 2100 Hz.) You should be able to see in the plot and hear >>> in the wav files that the high frequency component has been filtered out. >>> >>> Warren >>> >>> ----- >>> >>> from __future__ import division >>> >>> from scipy.signal import remez, freqz, lfilter >>> from scipy.io import wavfile >>> import numpy as np >>> import matplotlib.pyplot as plt >>> >>> >>> fs = 44100 >>> >>> # Design a low-pass filter using remez. >>> cutoff = 2000.0 >>> transition_width = 200 >>> bands = np.array([0, cutoff - 0.5*transition_width, >>> cutoff + 0.5*transition_width, fs/2.0]) / fs >>> desired = [1, 0] >>> lpf = remez(513, bands, desired) >>> >>> # Plot the frequency response of the filter. >>> w, h = freqz(lpf) >>> plt.figure(1) >>> plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) >>> plt.xlim(0, fs/2) >>> plt.xlabel('Frequency (Hz)') >>> plt.ylabel('Gain (dB)') >>> plt.grid(True) >>> >>> # Create a test signal with two frequencies: one inside the pass-band, >>> # are one far outside the pass-band that should be filtered out. >>> T = 0.5 >>> nsamples = int(T*fs) >>> t = np.linspace(0, T, nsamples, endpoint=False) >>> freq = 440 >>> data = np.sin(2*np.pi*freq*t) >>> data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) >>> data /= 1.01*np.abs(data).max() >>> >>> # Filter the input using lfilter. (Alternatively, convolution could be >>> used.) >>> filtered_data = lfilter(lpf, 1, data) >>> >>> # Plot the input and output in the same figure. >>> plt.figure(2) >>> plt.plot(t, data) >>> plt.plot(t, filtered_data) >>> >>> plt.show() >>> >>> # Save the test signal and the filtered signal as wav files. >>> wavfile.write("data.wav", fs, data) >>> wavfile.write("filtered_data.wav", fs, filtered_data) >>> >>> >>> ----- >>> >>> >>> >>> >>> On Mon, Jul 4, 2016 at 9:52 AM, Matti Viljamaa >>> wrote: >>> >>>> >>>> On 04 Jul 2016, at 14:00, Matti Viljamaa wrote: >>>> >>>> I?m trying to use the scipy.signal.remez filter design function. What >>>> I?m doing is the following (with the help of >>>> http://www.ee.iitm.ac.in/~nitin/teaching/ee5480/firdesign.html): >>>> >>>> import os >>>> import scipy.io.wavfile >>>> from scipy import signal >>>> from pylab import * >>>> >>>> os.chdir("/Users/mviljamaa/Music") >>>> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >>>> >>>> lpf = signal.remez(21, [0, 0.05, 0.1], [1.0, .0]) >>>> >>>> # Plot the magnitude response >>>> w, h = signal.freqz(lpf) >>>> plot(w/(2*pi), 20*log10(abs(h))) >>>> show() >>>> >>>> # Filtered data >>>> sout = signal.lfilter(lpf, 1, data) >>>> >>>> scipy.io.wavfile.write("equalized.wav", sr, sout) >>>> >>>> ? >>>> >>>> Now judging by the magnitude response of the filter lpf, it should be a >>>> lowpass of some sort (I?m not sure how to interpret the cutoff frequency >>>> yet). >>>> >>>> The output I get is substantially gained (up to the point where it >>>> sounds distorted) and I cannot hear the lowpass filter. >>>> >>>> What am I doing wrong? >>>> >>>> >>>> >>>> Using a slightly modified code (from http://pastebin.com/LPEtXdzx): >>>> >>>> import os >>>> import scipy.io.wavfile >>>> from scipy import signal >>>> from pylab import * >>>> import numpy as np >>>> >>>> os.chdir("/Users/mviljamaa/Music") >>>> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >>>> >>>> fs = 44100 >>>> >>>> bands = array([0,3500,4000,5500,6000,fs/2.0]) / fs >>>> desired = [1, 0, 0] >>>> lpf = signal.remez(513, bands, desired) >>>> >>>> from scipy.signal import freqz >>>> w, h = signal.freqz(lpf) >>>> plot(w/(2*pi), 20*log10(abs(h))) >>>> show() >>>> >>>> sout = signal.lfilter(lpf, 1, data) >>>> >>>> sout /= 1.05 * max(abs(sout)) >>>> >>>> scipy.io.wavfile.write("equalized.wav", sr, sout) >>>> >>>> >>>> ? >>>> >>>> This one is able to retain the gain, but I still don?t hear any lowpass >>>> filtering. >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: figure_3.png Type: image/png Size: 28659 bytes Desc: not available URL: From warren.weckesser at gmail.com Sun Jul 10 13:24:37 2016 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 10 Jul 2016 13:24:37 -0400 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> Message-ID: On Sun, Jul 10, 2016 at 12:57 PM, Warren Weckesser < warren.weckesser at gmail.com> wrote: > > > On Sun, Jul 10, 2016 at 12:08 PM, Matti Viljamaa > wrote: > >> >> On 10 Jul 2016, at 19:04, Warren Weckesser >> wrote: >> >> >> >> On Sun, Jul 10, 2016 at 12:00 PM, Matti Viljamaa >> wrote: >> >>> Or why doing >>> >>> fs = 44100 >>> >>> # Design a low-pass filter using remez. >>> cutoff = 2000.0 >>> transition_width = 200 >>> bands = np.array([0, cutoff - 0.5*transition_width, >>> cutoff + 0.5*transition_width, fs/2.0]) / fs >>> desired = [0, 1] # <------ >>> lpf = remez(513, bands, desired) >>> >>> T = 0.5 >>> nsamples = int(T*fs) >>> t = np.linspace(0, T, nsamples, endpoint=False) >>> freq = 440 >>> data = np.sin(2*np.pi*freq*t) >>> data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) >>> data /= 1.01*np.abs(data).max() >>> >>> filtered_data = lfilter(lpf, 1, data) >>> >>> w3, h3 = freqz(filtered_data) >>> >>> >> >> Matti, >> >> Did you see the email that I sent on July 6, and the plot that I attached >> to it? Here's what I said: >> >> The arguments to 'freqz' are the filter coefficients, not the signal. >> See the example script in my previous email, where I have >> >> w, h = freqz(lpf) >> >> Warren >> >> >> >> But what if I want to plot the effect of the lpf on the signal, rather >> than the filter magnitude response? >> >> > > For that, you want to plot the spectral content of the original and > filtered data. You can use a Fourier transform and plot the magnitude of > the Fourier coefficients against the frequencies, or you can use a function > such as `scipy.signal.periodogram` or `scipy.signal.welch` which take care > of the FFT details for you. > > Also, matplotlib has its own tools for this. See, for example, the `psd` function demonstrated in this example: http://matplotlib.org/examples/pylab_examples/psd_demo.html Warren > Here's a new version of my script. It now creates a third figure > containing plots of the periodogram of the original data and the filtered > data. I used `scipy.signal.periodogram`, but I recommend experimenting > with `welch` also. > > Warren > > ---------- > > from __future__ import division > > from scipy.signal import remez, freqz, lfilter, periodogram > from scipy.io import wavfile > import numpy as np > import matplotlib.pyplot as plt > > > fs = 44100 > > # Design a low-pass filter using remez. > cutoff = 2000.0 > transition_width = 200 > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, fs/2.0]) / fs > desired = [1, 0] > lpf = remez(513, bands, desired) > > # Plot the frequency response of the filter. > w, h = freqz(lpf) > plt.figure(1) > plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) > plt.xlim(0, fs/2) > plt.xlabel('Frequency (Hz)') > plt.ylabel('Gain (dB)') > plt.grid(True) > > # Create a test signal with three frequencies: two inside the pass-band, > # are one far outside the pass-band that should be filtered out. > T = 0.5 > nsamples = int(T*fs) > t = np.linspace(0, T, nsamples, endpoint=False) > freq = 440 > data = np.sin(2*np.pi*freq*t) + 0.5*np.cos(2*np.pi*(2*freq)*t) > data += 0.75*np.sin(2*np.pi*(cutoff + 5*transition_width)*t) > data /= 1.01*np.abs(data).max() > > # Filter the input using lfilter. (Alternatively, convolution could be > used.) > filtered_data = lfilter(lpf, 1, data) > > # Plot the input and output in the same figure. > plt.figure(2) > plt.plot(t, data, 'b', label='original') > plt.plot(t, filtered_data, 'g', label='filtered') > plt.xlabel('Time (sec)') > plt.ylim(-1.1, 1.1) > plt.legend() > > plt.figure(3) > freqs, data_spec = periodogram(data, fs=fs) > freqs, filtered_data_spec = periodogram(filtered_data, fs=fs) > plt.subplot(2, 1, 1) > plt.plot(freqs, data_spec, 'b', label='original') > plt.xlim(0, 3*cutoff) > plt.axvline(cutoff, color='k', alpha=0.15) > plt.legend() > plt.subplot(2, 1, 2) > plt.plot(freqs, filtered_data_spec, 'g', label='filtered') > plt.xlim(0, 3*cutoff) > plt.axvline(cutoff, color='k', alpha=0.15) > plt.legend() > plt.xlabel('Frequency (Hz)') > > plt.show() > > # Save the test signal and the filtered signal as wav files. > wavfile.write("data.wav", fs, data) > wavfile.write("filtered_data.wav", fs, filtered_data) > > > ---------- > > > >> -Matti >> >> >> >>> plt.figure(3) >>> plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) >>> plt.xlim(0, fs/2) >>> plt.xlabel('Frequency (Hz)') >>> plt.ylabel('Gain (dB)') >>> plt.grid(True) >>> >>> plt. show() >>> >>> gives >>> >>> >>> >>> which doesn?t look like a highpass? >>> >>> >>> On 07 Jul 2016, at 13:08, Matti Viljamaa wrote: >>> >>> Any idea why this plot shows positive gain? >>> Like +50dB around 500Hz? >>> >>> -Matti >>> >>> On 06 Jul 2016, at 17:57, Warren Weckesser >>> wrote: >>> >>> >>> >>> On Wed, Jul 6, 2016 at 7:53 AM, Matti Viljamaa >>> wrote: >>> >>>> Are these the proper frequency domain plots+ >>>> >>>> w2, h2 = freqz(data) >>>> >>>> >>> >>> The arguments to 'freqz' are the filter coefficients, not the signal. >>> See the example script in my previous email, where I have >>> >>> w, h = freqz(lpf) >>> >>> The attached plot is the frequency response that I get when I run that >>> script. >>> >>> Warren >>> >>> >>> >>> >>>> plt.figure(2) >>>> plt.plot(fs*w2/(2*np.pi), 20*np.log10(abs(h2))) >>>> plt.xlim(0, fs/2) >>>> plt.xlabel('Frequency (Hz)') >>>> plt.ylabel('Gain (dB)') >>>> plt.grid(True) >>>> >>>> w3, h3 = freqz(filtered_data) >>>> >>>> plt.figure(3) >>>> plt.plot(fs*w3/(2*np.pi), 20*np.log10(abs(h3))) >>>> plt.xlim(0, fs/2) >>>> plt.xlabel('Frequency (Hz)') >>>> plt.ylabel('Gain (dB)') >>>> plt.grid(True) >>>> >>>> plt. show() >>>> >>>> >>>> >>>> On 04 Jul 2016, at 18:45, Warren Weckesser >>>> wrote: >>>> >>>> Matti, >>>> >>>> I don't have your input file, so I can't reproduce your result >>>> exactly. For a basic demonstration, it is probably easier to create a >>>> signal in the code instead of requiring an input wav file. Here's an >>>> example that does that. The input signal (called 'data') is the sum of two >>>> sine waves, one at 440 Hz and one at 3000 Hz. The cutoff frequency for the >>>> low-pass filter is 2000 Hz. (Actually the transition from the pass-band to >>>> the stop-band is from 1900 Hz to 2100 Hz.) You should be able to see in >>>> the plot and hear in the wav files that the high frequency component has >>>> been filtered out. >>>> >>>> Warren >>>> >>>> ----- >>>> >>>> from __future__ import division >>>> >>>> from scipy.signal import remez, freqz, lfilter >>>> from scipy.io import wavfile >>>> import numpy as np >>>> import matplotlib.pyplot as plt >>>> >>>> >>>> fs = 44100 >>>> >>>> # Design a low-pass filter using remez. >>>> cutoff = 2000.0 >>>> transition_width = 200 >>>> bands = np.array([0, cutoff - 0.5*transition_width, >>>> cutoff + 0.5*transition_width, fs/2.0]) / fs >>>> desired = [1, 0] >>>> lpf = remez(513, bands, desired) >>>> >>>> # Plot the frequency response of the filter. >>>> w, h = freqz(lpf) >>>> plt.figure(1) >>>> plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) >>>> plt.xlim(0, fs/2) >>>> plt.xlabel('Frequency (Hz)') >>>> plt.ylabel('Gain (dB)') >>>> plt.grid(True) >>>> >>>> # Create a test signal with two frequencies: one inside the pass-band, >>>> # are one far outside the pass-band that should be filtered out. >>>> T = 0.5 >>>> nsamples = int(T*fs) >>>> t = np.linspace(0, T, nsamples, endpoint=False) >>>> freq = 440 >>>> data = np.sin(2*np.pi*freq*t) >>>> data += np.sin(2*np.pi*(cutoff + 5*transition_width)*t) >>>> data /= 1.01*np.abs(data).max() >>>> >>>> # Filter the input using lfilter. (Alternatively, convolution could be >>>> used.) >>>> filtered_data = lfilter(lpf, 1, data) >>>> >>>> # Plot the input and output in the same figure. >>>> plt.figure(2) >>>> plt.plot(t, data) >>>> plt.plot(t, filtered_data) >>>> >>>> plt.show() >>>> >>>> # Save the test signal and the filtered signal as wav files. >>>> wavfile.write("data.wav", fs, data) >>>> wavfile.write("filtered_data.wav", fs, filtered_data) >>>> >>>> >>>> ----- >>>> >>>> >>>> >>>> >>>> On Mon, Jul 4, 2016 at 9:52 AM, Matti Viljamaa >>>> wrote: >>>> >>>>> >>>>> On 04 Jul 2016, at 14:00, Matti Viljamaa wrote: >>>>> >>>>> I?m trying to use the scipy.signal.remez filter design function. What >>>>> I?m doing is the following (with the help of >>>>> http://www.ee.iitm.ac.in/~nitin/teaching/ee5480/firdesign.html): >>>>> >>>>> import os >>>>> import scipy.io.wavfile >>>>> from scipy import signal >>>>> from pylab import * >>>>> >>>>> os.chdir("/Users/mviljamaa/Music") >>>>> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >>>>> >>>>> lpf = signal.remez(21, [0, 0.05, 0.1], [1.0, .0]) >>>>> >>>>> # Plot the magnitude response >>>>> w, h = signal.freqz(lpf) >>>>> plot(w/(2*pi), 20*log10(abs(h))) >>>>> show() >>>>> >>>>> # Filtered data >>>>> sout = signal.lfilter(lpf, 1, data) >>>>> >>>>> scipy.io.wavfile.write("equalized.wav", sr, sout) >>>>> >>>>> ? >>>>> >>>>> Now judging by the magnitude response of the filter lpf, it should be >>>>> a lowpass of some sort (I?m not sure how to interpret the cutoff frequency >>>>> yet). >>>>> >>>>> The output I get is substantially gained (up to the point where it >>>>> sounds distorted) and I cannot hear the lowpass filter. >>>>> >>>>> What am I doing wrong? >>>>> >>>>> >>>>> >>>>> Using a slightly modified code (from http://pastebin.com/LPEtXdzx): >>>>> >>>>> import os >>>>> import scipy.io.wavfile >>>>> from scipy import signal >>>>> from pylab import * >>>>> import numpy as np >>>>> >>>>> os.chdir("/Users/mviljamaa/Music") >>>>> sr, data = scipy.io.wavfile.read(open(?sound.wav", 'r')) >>>>> >>>>> fs = 44100 >>>>> >>>>> bands = array([0,3500,4000,5500,6000,fs/2.0]) / fs >>>>> desired = [1, 0, 0] >>>>> lpf = signal.remez(513, bands, desired) >>>>> >>>>> from scipy.signal import freqz >>>>> w, h = signal.freqz(lpf) >>>>> plot(w/(2*pi), 20*log10(abs(h))) >>>>> show() >>>>> >>>>> sout = signal.lfilter(lpf, 1, data) >>>>> >>>>> sout /= 1.05 * max(abs(sout)) >>>>> >>>>> scipy.io.wavfile.write("equalized.wav", sr, sout) >>>>> >>>>> >>>>> ? >>>>> >>>>> This one is able to retain the gain, but I still don?t hear any >>>>> lowpass filtering. >>>>> >>>>> _______________________________________________ >>>>> SciPy-User mailing list >>>>> SciPy-User at scipy.org >>>>> https://mail.scipy.org/mailman/listinfo/scipy-user >>>>> >>>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> https://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >>> >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mviljamaa at kapsi.fi Mon Jul 11 07:03:35 2016 From: mviljamaa at kapsi.fi (Matti Viljamaa) Date: Mon, 11 Jul 2016 14:03:35 +0300 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> Message-ID: > On 10 Jul 2016, at 19:57, Warren Weckesser wrote: > > > > On Sun, Jul 10, 2016 at 12:08 PM, Matti Viljamaa > wrote: > > But what if I want to plot the effect of the lpf on the signal, rather than the filter magnitude response? > > > > For that, you want to plot the spectral content of the original and filtered data. You can use a Fourier transform and plot the magnitude of the Fourier coefficients against the frequencies, or you can use a function such as `scipy.signal.periodogram` or `scipy.signal.welch` which take care of the FFT details for you. > > Here's a new version of my script. It now creates a third figure containing plots of the periodogram of the original data and the filtered data. I used `scipy.signal.periodogram`, but I recommend experimenting with `welch` also. > > Warren > > ---------- > > from __future__ import division > > from scipy.signal import remez, freqz, lfilter, periodogram > from scipy.io import wavfile > import numpy as np > import matplotlib.pyplot as plt > > > fs = 44100 > > # Design a low-pass filter using remez. > cutoff = 2000.0 > transition_width = 200 > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, fs/2.0]) / fs > desired = [1, 0] > lpf = remez(513, bands, desired) > > # Plot the frequency response of the filter. > w, h = freqz(lpf) > plt.figure(1) > plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) > plt.xlim(0, fs/2) > plt.xlabel('Frequency (Hz)') > plt.ylabel('Gain (dB)') > plt.grid(True) > > # Create a test signal with three frequencies: two inside the pass-band, > # are one far outside the pass-band that should be filtered out. > T = 0.5 > nsamples = int(T*fs) > t = np.linspace(0, T, nsamples, endpoint=False) > freq = 440 > data = np.sin(2*np.pi*freq*t) + 0.5*np.cos(2*np.pi*(2*freq)*t) > data += 0.75*np.sin(2*np.pi*(cutoff + 5*transition_width)*t) > data /= 1.01*np.abs(data).max() > > # Filter the input using lfilter. (Alternatively, convolution could be used.) > filtered_data = lfilter(lpf, 1, data) > > # Plot the input and output in the same figure. > plt.figure(2) > plt.plot(t, data, 'b', label='original') > plt.plot(t, filtered_data, 'g', label='filtered') > plt.xlabel('Time (sec)') > plt.ylim(-1.1, 1.1) > plt.legend() > > plt.figure(3) > freqs, data_spec = periodogram(data, fs=fs) > freqs, filtered_data_spec = periodogram(filtered_data, fs=fs) > plt.subplot(2, 1, 1) > plt.plot(freqs, data_spec, 'b', label='original') > plt.xlim(0, 3*cutoff) > plt.axvline(cutoff, color='k', alpha=0.15) > plt.legend() > plt.subplot(2, 1, 2) > plt.plot(freqs, filtered_data_spec, 'g', label='filtered') > plt.xlim(0, 3*cutoff) > plt.axvline(cutoff, color='k', alpha=0.15) > plt.legend() > plt.xlabel('Frequency (Hz)') > > plt.show() > > # Save the test signal and the filtered signal as wav files. > wavfile.write("data.wav", fs, data) > wavfile.write("filtered_data.wav", fs, filtered_data) > > > ????? Thanks for this. However, I think there?s some other problem. Modifying the above remez to the following: fs = 44100 # Design a low-pass filter using remez. cutoff = 2000.0 transition_width = 200 bands = np.array([0, cutoff - 0.5*transition_width, cutoff + 0.5*transition_width, cutoff + 0.5*transition_width + 1200.0, cutoff + 0.5*transition_width + 2400.0,fs/2.0]) / fs desired = [0, 1, 0] lpf = remez(513, bands, desired) gives the following plots: Why do these parameters lead to that +150dB peak, the peak in the ?filtered" frequency plot and the strange full bandwidth burst in the last picture? -Matti -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2016-07-11 at 14.01.40.png Type: image/png Size: 284336 bytes Desc: not available URL: From warren.weckesser at gmail.com Mon Jul 11 09:30:17 2016 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 11 Jul 2016 09:30:17 -0400 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> Message-ID: On Mon, Jul 11, 2016 at 7:03 AM, Matti Viljamaa wrote: > > On 10 Jul 2016, at 19:57, Warren Weckesser > wrote: > > > > On Sun, Jul 10, 2016 at 12:08 PM, Matti Viljamaa > wrote: > >> >> But what if I want to plot the effect of the lpf on the signal, rather >> than the filter magnitude response? >> >> > > For that, you want to plot the spectral content of the original and > filtered data. You can use a Fourier transform and plot the magnitude of > the Fourier coefficients against the frequencies, or you can use a function > such as `scipy.signal.periodogram` or `scipy.signal.welch` which take care > of the FFT details for you. > > Here's a new version of my script. It now creates a third figure > containing plots of the periodogram of the original data and the filtered > data. I used `scipy.signal.periodogram`, but I recommend experimenting > with `welch` also. > > Warren > > ---------- > > from __future__ import division > > from scipy.signal import remez, freqz, lfilter, periodogram > from scipy.io import wavfile > import numpy as np > import matplotlib.pyplot as plt > > > fs = 44100 > > # Design a low-pass filter using remez. > cutoff = 2000.0 > transition_width = 200 > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, fs/2.0]) / fs > desired = [1, 0] > lpf = remez(513, bands, desired) > > # Plot the frequency response of the filter. > w, h = freqz(lpf) > plt.figure(1) > plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) > plt.xlim(0, fs/2) > plt.xlabel('Frequency (Hz)') > plt.ylabel('Gain (dB)') > plt.grid(True) > > # Create a test signal with three frequencies: two inside the pass-band, > # are one far outside the pass-band that should be filtered out. > T = 0.5 > nsamples = int(T*fs) > t = np.linspace(0, T, nsamples, endpoint=False) > freq = 440 > data = np.sin(2*np.pi*freq*t) + 0.5*np.cos(2*np.pi*(2*freq)*t) > data += 0.75*np.sin(2*np.pi*(cutoff + 5*transition_width)*t) > data /= 1.01*np.abs(data).max() > > # Filter the input using lfilter. (Alternatively, convolution could be > used.) > filtered_data = lfilter(lpf, 1, data) > > # Plot the input and output in the same figure. > plt.figure(2) > plt.plot(t, data, 'b', label='original') > plt.plot(t, filtered_data, 'g', label='filtered') > plt.xlabel('Time (sec)') > plt.ylim(-1.1, 1.1) > plt.legend() > > plt.figure(3) > freqs, data_spec = periodogram(data, fs=fs) > freqs, filtered_data_spec = periodogram(filtered_data, fs=fs) > plt.subplot(2, 1, 1) > plt.plot(freqs, data_spec, 'b', label='original') > plt.xlim(0, 3*cutoff) > plt.axvline(cutoff, color='k', alpha=0.15) > plt.legend() > plt.subplot(2, 1, 2) > plt.plot(freqs, filtered_data_spec, 'g', label='filtered') > plt.xlim(0, 3*cutoff) > plt.axvline(cutoff, color='k', alpha=0.15) > plt.legend() > plt.xlabel('Frequency (Hz)') > > plt.show() > > # Save the test signal and the filtered signal as wav files. > wavfile.write("data.wav", fs, data) > wavfile.write("filtered_data.wav", fs, filtered_data) > > > ????? > > > Thanks for this. However, I think there?s some other problem. > > Modifying the above remez to the following: > > fs = 44100 > > # Design a low-pass filter using remez. > cutoff = 2000.0 > transition_width = 200 > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, cutoff + > 0.5*transition_width + 1200.0, cutoff + 0.5*transition_width + > 2400.0,fs/2.0]) / fs > desired = [0, 1, 0] > You specified three intervals in `bands`: [0, 1900], [2100, 3300] and [4500, 22050] with gains 0, 1, and 0, respectively (i.e. you are designing a bandpass filter). The remez algorithm designs a filter that is equiripple in the specified bands. Outside those bands, however, the filter gain is unspecified; typically these intervals are described as the "don't care" intervals. The remez algorithm makes no promises about the gain in these intervals, and it can--as you have found--result in unacceptable behavior. The big peak in the filter gain occurs in the "don't care" interval [3300, 4500]. To fix it, you can try reducing the length of that "don't care" interval, or increasing the length of the filter (if that is an option). It might require several iterations of band adjustments to achieve a desirable result. You can also try the `weight` argument--perhaps give the stop bands a lower weight than the pass band. Some experimentation and iteration will be involved. Warren lpf = remez(513, bands, desired) > > gives the following plots: > > > Why do these parameters lead to that +150dB peak, the peak in the > ?filtered" frequency plot and the strange full bandwidth burst in the last > picture? > > -Matti > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2016-07-11 at 14.01.40.png Type: image/png Size: 284336 bytes Desc: not available URL: From mviljamaa at kapsi.fi Mon Jul 11 10:13:55 2016 From: mviljamaa at kapsi.fi (Matti Viljamaa) Date: Mon, 11 Jul 2016 17:13:55 +0300 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> Message-ID: <6F02E6C5-101F-4D7F-B8F2-56DDE663EFA5@kapsi.fi> > On 11 Jul 2016, at 16:30, Warren Weckesser wrote: > > > > On Mon, Jul 11, 2016 at 7:03 AM, Matti Viljamaa > wrote: > >> On 10 Jul 2016, at 19:57, Warren Weckesser > wrote: >> >> >> >> On Sun, Jul 10, 2016 at 12:08 PM, Matti Viljamaa > wrote: >> >> But what if I want to plot the effect of the lpf on the signal, rather than the filter magnitude response? >> >> >> >> For that, you want to plot the spectral content of the original and filtered data. You can use a Fourier transform and plot the magnitude of the Fourier coefficients against the frequencies, or you can use a function such as `scipy.signal.periodogram` or `scipy.signal.welch` which take care of the FFT details for you. >> >> Here's a new version of my script. It now creates a third figure containing plots of the periodogram of the original data and the filtered data. I used `scipy.signal.periodogram`, but I recommend experimenting with `welch` also. >> >> Warren >> >> ---------- >> >> from __future__ import division >> >> from scipy.signal import remez, freqz, lfilter, periodogram >> from scipy.io import wavfile >> import numpy as np >> import matplotlib.pyplot as plt >> >> >> fs = 44100 >> >> # Design a low-pass filter using remez. >> cutoff = 2000.0 >> transition_width = 200 >> bands = np.array([0, cutoff - 0.5*transition_width, >> cutoff + 0.5*transition_width, fs/2.0]) / fs >> desired = [1, 0] >> lpf = remez(513, bands, desired) >> >> # Plot the frequency response of the filter. >> w, h = freqz(lpf) >> plt.figure(1) >> plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) >> plt.xlim(0, fs/2) >> plt.xlabel('Frequency (Hz)') >> plt.ylabel('Gain (dB)') >> plt.grid(True) >> >> # Create a test signal with three frequencies: two inside the pass-band, >> # are one far outside the pass-band that should be filtered out. >> T = 0.5 >> nsamples = int(T*fs) >> t = np.linspace(0, T, nsamples, endpoint=False) >> freq = 440 >> data = np.sin(2*np.pi*freq*t) + 0.5*np.cos(2*np.pi*(2*freq)*t) >> data += 0.75*np.sin(2*np.pi*(cutoff + 5*transition_width)*t) >> data /= 1.01*np.abs(data).max() >> >> # Filter the input using lfilter. (Alternatively, convolution could be used.) >> filtered_data = lfilter(lpf, 1, data) >> >> # Plot the input and output in the same figure. >> plt.figure(2) >> plt.plot(t, data, 'b', label='original') >> plt.plot(t, filtered_data, 'g', label='filtered') >> plt.xlabel('Time (sec)') >> plt.ylim(-1.1, 1.1) >> plt.legend() >> >> plt.figure(3) >> freqs, data_spec = periodogram(data, fs=fs) >> freqs, filtered_data_spec = periodogram(filtered_data, fs=fs) >> plt.subplot(2, 1, 1) >> plt.plot(freqs, data_spec, 'b', label='original') >> plt.xlim(0, 3*cutoff) >> plt.axvline(cutoff, color='k', alpha=0.15) >> plt.legend() >> plt.subplot(2, 1, 2) >> plt.plot(freqs, filtered_data_spec, 'g', label='filtered') >> plt.xlim(0, 3*cutoff) >> plt.axvline(cutoff, color='k', alpha=0.15) >> plt.legend() >> plt.xlabel('Frequency (Hz)') >> >> plt.show() >> >> # Save the test signal and the filtered signal as wav files. >> wavfile.write("data.wav", fs, data) >> wavfile.write("filtered_data.wav", fs, filtered_data) >> >> >> ????? > > Thanks for this. However, I think there?s some other problem. > > Modifying the above remez to the following: > > fs = 44100 > > # Design a low-pass filter using remez. > cutoff = 2000.0 > transition_width = 200 > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, cutoff + 0.5*transition_width + 1200.0, cutoff + 0.5*transition_width + 2400.0,fs/2.0]) / fs > desired = [0, 1, 0] > > > You specified three intervals in `bands`: [0, 1900], [2100, 3300] and [4500, 22050] with gains 0, 1, and 0, respectively (i.e. you are designing a bandpass filter). The remez algorithm designs a filter that is equiripple in the specified bands. Outside those bands, however, the filter gain is unspecified; typically these intervals are described as the "don't care" intervals. The remez algorithm makes no promises about the gain in these intervals, and it can--as you have found--result in unacceptable behavior. The big peak in the filter gain occurs in the "don't care" interval [3300, 4500]. > > To fix it, you can try reducing the length of that "don't care" interval, or increasing the length of the filter (if that is an option). It might require several iterations of band adjustments to achieve a desirable result. You can also try the `weight` argument--perhaps give the stop bands a lower weight than the pass band. Some experimentation and iteration will be involved. > > Warren So doing e.g. bands = np.array([0, cutoff - 0.5*transition_width, cutoff + 0.5*transition_width, cutoff + 0.5*transition_width + 2400.0, cutoff + 0.5*transition_width + 2500.0,fs/2.0]) / fs i.e. [ 0., 1900.] [2100., 4500.] [4600., 22050.] the filter is shaped properly. However, do you know why [ 0., 1900.] [2100., 4500.] [4500., 22050.] fails? I.e. when the transition band is 0Hz? -Matti > > lpf = remez(513, bands, desired) > > gives the following plots: > > > > Why do these parameters lead to that +150dB peak, the peak in the ?filtered" frequency plot and the strange full bandwidth burst in the last picture? > > -Matti > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From mviljamaa at kapsi.fi Mon Jul 11 10:23:52 2016 From: mviljamaa at kapsi.fi (Matti Viljamaa) Date: Mon, 11 Jul 2016 17:23:52 +0300 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: <6F02E6C5-101F-4D7F-B8F2-56DDE663EFA5@kapsi.fi> References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> <6F02E6C5-101F-4D7F-B8F2-56DDE663EFA5@kapsi.fi> Message-ID: What about the following: is this (the almost -15dB cut in the transition from <2000Hz to >2000Hz) a ?feature" of the filter? plotted with: bands = np.array([0, cutoff - 0.5*transition_width, cutoff + 0.5*transition_width, cutoff + 0.5*transition_width + 2400.0, cutoff + 0.5*transition_width + 2500.0,fs/2.0]) / fs desired = [0.5, 1, 0.3] -Matti -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2016-07-11 at 17.22.02.png Type: image/png Size: 58599 bytes Desc: not available URL: From warren.weckesser at gmail.com Mon Jul 11 11:50:57 2016 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 11 Jul 2016 11:50:57 -0400 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: <6F02E6C5-101F-4D7F-B8F2-56DDE663EFA5@kapsi.fi> References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> <6F02E6C5-101F-4D7F-B8F2-56DDE663EFA5@kapsi.fi> Message-ID: On Mon, Jul 11, 2016 at 10:13 AM, Matti Viljamaa wrote: > On 11 Jul 2016, at 16:30, Warren Weckesser > wrote: > > > > On Mon, Jul 11, 2016 at 7:03 AM, Matti Viljamaa > wrote: > >> >> On 10 Jul 2016, at 19:57, Warren Weckesser >> wrote: >> >> >> >> On Sun, Jul 10, 2016 at 12:08 PM, Matti Viljamaa >> wrote: >> >>> >>> But what if I want to plot the effect of the lpf on the signal, rather >>> than the filter magnitude response? >>> >>> >> >> For that, you want to plot the spectral content of the original and >> filtered data. You can use a Fourier transform and plot the magnitude of >> the Fourier coefficients against the frequencies, or you can use a function >> such as `scipy.signal.periodogram` or `scipy.signal.welch` which take care >> of the FFT details for you. >> >> Here's a new version of my script. It now creates a third figure >> containing plots of the periodogram of the original data and the filtered >> data. I used `scipy.signal.periodogram`, but I recommend experimenting >> with `welch` also. >> >> Warren >> >> ---------- >> >> from __future__ import division >> >> from scipy.signal import remez, freqz, lfilter, periodogram >> from scipy.io import wavfile >> import numpy as np >> import matplotlib.pyplot as plt >> >> >> fs = 44100 >> >> # Design a low-pass filter using remez. >> cutoff = 2000.0 >> transition_width = 200 >> bands = np.array([0, cutoff - 0.5*transition_width, >> cutoff + 0.5*transition_width, fs/2.0]) / fs >> desired = [1, 0] >> lpf = remez(513, bands, desired) >> >> # Plot the frequency response of the filter. >> w, h = freqz(lpf) >> plt.figure(1) >> plt.plot(fs*w/(2*np.pi), 20*np.log10(abs(h))) >> plt.xlim(0, fs/2) >> plt.xlabel('Frequency (Hz)') >> plt.ylabel('Gain (dB)') >> plt.grid(True) >> >> # Create a test signal with three frequencies: two inside the pass-band, >> # are one far outside the pass-band that should be filtered out. >> T = 0.5 >> nsamples = int(T*fs) >> t = np.linspace(0, T, nsamples, endpoint=False) >> freq = 440 >> data = np.sin(2*np.pi*freq*t) + 0.5*np.cos(2*np.pi*(2*freq)*t) >> data += 0.75*np.sin(2*np.pi*(cutoff + 5*transition_width)*t) >> data /= 1.01*np.abs(data).max() >> >> # Filter the input using lfilter. (Alternatively, convolution could be >> used.) >> filtered_data = lfilter(lpf, 1, data) >> >> # Plot the input and output in the same figure. >> plt.figure(2) >> plt.plot(t, data, 'b', label='original') >> plt.plot(t, filtered_data, 'g', label='filtered') >> plt.xlabel('Time (sec)') >> plt.ylim(-1.1, 1.1) >> plt.legend() >> >> plt.figure(3) >> freqs, data_spec = periodogram(data, fs=fs) >> freqs, filtered_data_spec = periodogram(filtered_data, fs=fs) >> plt.subplot(2, 1, 1) >> plt.plot(freqs, data_spec, 'b', label='original') >> plt.xlim(0, 3*cutoff) >> plt.axvline(cutoff, color='k', alpha=0.15) >> plt.legend() >> plt.subplot(2, 1, 2) >> plt.plot(freqs, filtered_data_spec, 'g', label='filtered') >> plt.xlim(0, 3*cutoff) >> plt.axvline(cutoff, color='k', alpha=0.15) >> plt.legend() >> plt.xlabel('Frequency (Hz)') >> >> plt.show() >> >> # Save the test signal and the filtered signal as wav files. >> wavfile.write("data.wav", fs, data) >> wavfile.write("filtered_data.wav", fs, filtered_data) >> >> >> ????? >> >> >> Thanks for this. However, I think there?s some other problem. >> >> Modifying the above remez to the following: >> >> fs = 44100 >> >> # Design a low-pass filter using remez. >> cutoff = 2000.0 >> transition_width = 200 >> bands = np.array([0, cutoff - 0.5*transition_width, >> cutoff + 0.5*transition_width, cutoff + >> 0.5*transition_width + 1200.0, cutoff + 0.5*transition_width + >> 2400.0,fs/2.0]) / fs >> desired = [0, 1, 0] >> > > > You specified three intervals in `bands`: [0, 1900], [2100, 3300] and > [4500, 22050] with gains 0, 1, and 0, respectively (i.e. you are designing > a bandpass filter). The remez algorithm designs a filter that is > equiripple in the specified bands. Outside those bands, however, the > filter gain is unspecified; typically these intervals are described as the > "don't care" intervals. The remez algorithm makes no promises about the > gain in these intervals, and it can--as you have found--result in > unacceptable behavior. The big peak in the filter gain occurs in the > "don't care" interval [3300, 4500]. > > To fix it, you can try reducing the length of that "don't care" interval, > or increasing the length of the filter (if that is an option). It might > require several iterations of band adjustments to achieve a desirable > result. You can also try the `weight` argument--perhaps give the stop > bands a lower weight than the pass band. Some experimentation and > iteration will be involved. > > Warren > > > So doing e.g. > > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, cutoff + > 0.5*transition_width + 2400.0, cutoff + 0.5*transition_width + > 2500.0,fs/2.0]) / fs > > i.e. > > [ 0., 1900.] [2100., 4500.] [4600., 22050.] > > the filter is shaped properly. > > However, > > do you know why > > [ 0., 1900.] [2100., 4500.] [4500., 22050.] > > fails? I.e. when the transition band is 0Hz? > > You are asking for an ideal transition at 4500 Hz from perfect pass band to perfect stop band. The remez algorithm doesn't converge. You'd have to dig into the details of the algorithm for a good answer to "why", so I'll just say that the problem is too hard--possibly even impossible--for the remez algorithm. Warren > -Matti > > > lpf = remez(513, bands, desired) >> >> gives the following plots: >> >> >> >> Why do these parameters lead to that +150dB peak, the peak in the >> ?filtered" frequency plot and the strange full bandwidth burst in the last >> picture? >> >> -Matti >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-user >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Mon Jul 11 11:56:57 2016 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 11 Jul 2016 11:56:57 -0400 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> <6F02E6C5-101F-4D7F-B8F2-56DDE663EFA5@kapsi.fi> Message-ID: On Mon, Jul 11, 2016 at 10:23 AM, Matti Viljamaa wrote: > What about the following: > > is this (the almost -15dB cut in the transition from <2000Hz to >2000Hz) a > ?feature" of the filter? > > > plotted with: > > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, cutoff + > 0.5*transition_width + 2400.0, cutoff + 0.5*transition_width + > 2500.0,fs/2.0]) / fs > desired = [0.5, 1, 0.3] > > That dip in the gain looks like it is in the "don't care" interval [cutoff - 0.5*transition_width, cutoff + 0.5*transition_width] (i.e. [1900, 2100]). The remez algorithm doesn't specify the behavior in the "don't care" intervals. If it turns out that you get something desirable, then sure, it's a feature, but if you tweak the parameters (i.e. the band edges or filter length), the behavior in the "don't care" intervals could be completely different. Warren -Matti > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2016-07-11 at 17.22.02.png Type: image/png Size: 58599 bytes Desc: not available URL: From clancyr at gmail.com Mon Jul 11 12:08:06 2016 From: clancyr at gmail.com (Clancy Rowley) Date: Mon, 11 Jul 2016 12:08:06 -0400 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> <6F02E6C5-101F-4D7F-B8F2-56DDE663EFA5@kapsi.fi> Message-ID: <8E0D6BAE-EE2A-4DE7-9F3B-725D04C959CE@gmail.com> Just a suggestion: for low-pass and band-pass filters, you might have better luck with Butterworth filters: from scipy.signal import butter, freqz fs = 44100 nyquist = 0.5/fs passband = np.array([2000.,4500.]) * nyquist order = 4 num, den = butter(order, passband, btype='bandpass') w, h = freqz(num, den) -clancy > On Jul 11, 2016, at 10:23 AM, Matti Viljamaa wrote: > > What about the following: > > is this (the almost -15dB cut in the transition from <2000Hz to >2000Hz) a ?feature" of the filter? > > > > plotted with: > > bands = np.array([0, cutoff - 0.5*transition_width, > cutoff + 0.5*transition_width, cutoff + 0.5*transition_width + 2400.0, cutoff + 0.5*transition_width + 2500.0,fs/2.0]) / fs > desired = [0.5, 1, 0.3] > > -Matti > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user From mviljamaa at kapsi.fi Tue Jul 12 10:42:49 2016 From: mviljamaa at kapsi.fi (Matti Viljamaa) Date: Tue, 12 Jul 2016 17:42:49 +0300 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> <6F02E6C5-101F-4D7F-B8F2-56DDE663EFA5@kapsi.fi> Message-ID: Are there general ways for ensuring that the remez produces a filter that?s ?correct?, i.e. no huge undesired boosts or other such ?glitches?? -Matti From ralf.gommers at gmail.com Wed Jul 13 18:29:11 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 14 Jul 2016 00:29:11 +0200 Subject: [SciPy-User] Scikits portal broken In-Reply-To: References: Message-ID: On Fri, Jul 8, 2016 at 2:45 PM, klo uo wrote: > Hi, > > http://scikits.appspot.com/scikits GAE code raises exception, apparently > because pypi does allow only https protocol. > Don't see anything named GAE on that page. I checked a few http:// PyPI links for randomly selected scikits, those all work fine. Can you be more specific about what the problem is? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Jul 13 18:31:01 2016 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 13 Jul 2016 23:31:01 +0100 Subject: [SciPy-User] Scikits portal broken In-Reply-To: References: Message-ID: On Wed, Jul 13, 2016 at 11:29 PM, Ralf Gommers wrote: > > On Fri, Jul 8, 2016 at 2:45 PM, klo uo wrote: >> >> Hi, >> >> http://scikits.appspot.com/scikits GAE code raises exception, apparently because pypi does allow only https protocol. > > Don't see anything named GAE on that page. I checked a few http:// PyPI links for randomly selected scikits, those all work fine. Can you be more specific about what the problem is? I pinged St?fan about it, and he fixed it. Sorry, I should have posted a note here when he did. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jul 13 18:38:49 2016 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 14 Jul 2016 00:38:49 +0200 Subject: [SciPy-User] Scikits portal broken In-Reply-To: References: Message-ID: On Thu, Jul 14, 2016 at 12:31 AM, Robert Kern wrote: > On Wed, Jul 13, 2016 at 11:29 PM, Ralf Gommers > wrote: > > > > On Fri, Jul 8, 2016 at 2:45 PM, klo uo wrote: > >> > >> Hi, > >> > >> http://scikits.appspot.com/scikits GAE code raises exception, > apparently because pypi does allow only https protocol. > > > > Don't see anything named GAE on that page. I checked a few http:// PyPI > links for randomly selected scikits, those all work fine. Can you be more > specific about what the problem is? > > I pinged St?fan about it, and he fixed it. Sorry, I should have posted a > note here when he did. > > No worries, thanks for setting the wheels in motion there. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Wed Jul 13 23:35:07 2016 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 13 Jul 2016 23:35:07 -0400 Subject: [SciPy-User] scipy.signal.remez producing odd filter effects / not producing desired effect In-Reply-To: References: <990240E9-E1D3-42D4-9E37-31C62E06FD09@kapsi.fi> <1FB69009-3F73-428D-B678-80B01E9A6FB7@kapsi.fi> <4A99E1A3-0253-4479-A8F9-B6B309CA9AE4@kapsi.fi> <6F02E6C5-101F-4D7F-B8F2-56DDE663EFA5@kapsi.fi> Message-ID: On Tue, Jul 12, 2016 at 10:42 AM, Matti Viljamaa wrote: > Are there general ways for ensuring that the remez produces a filter > that?s ?correct?, i.e. no huge undesired boosts or other such ?glitches?? > > Not that I know of. I always check the filter response with freqz. Warren > -Matti > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From winash12 at gmail.com Wed Jul 20 02:43:16 2016 From: winash12 at gmail.com (ashwinD12 .) Date: Wed, 20 Jul 2016 12:13:16 +0530 Subject: [SciPy-User] Memory error with scipy.interpolate Message-ID: Hello, I get a memory error when I interpolate a data set of 567 points as shown in this stack trace Traceback (most recent call last): File "contour.py", line 34, in rbf = scipy.interpolate.Rbf(x, y, data, function='linear') File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 200, in __init__ r = self._call_norm(self.xi, self.xi) File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 231, in _call_norm return self.norm(x1, x2) File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 118, in _euclidean_norm return np.sqrt(((x1 - x2)**2).sum(axis=0)) MemoryError With numpy earlier I had gotten the same error and I had moved past it by this option - lon,lat = np.meshgrid(lon, lat,sparse=True,copy=False) This is the minimum code snippet that should reproduce the problem grbs = pygrib.open('20000') grb = grbs.select(name='Geopotential',level=500)[0] data = grb.values hgt = [x/10 for x in data] lat,lon = grb.latlons() lon,lat = np.meshgrid(lon, lat,sparse=True,copy=False) m=Basemap(projection='mill',lat_ts=10,llcrnrlon=lon.min(), \ urcrnrlon=lon.max(),llcrnrlat=lat.min(),urcrnrlat=lat.max(), \ resolution=None) x,y = m(lon,lat) rbf = scipy.interpolate.Rbf(x, y, data, function='linear') zi = rbf(x, y) Regards, Ashwin. -------------- next part -------------- An HTML attachment was scrubbed... URL: From winash12 at gmail.com Wed Jul 20 11:33:44 2016 From: winash12 at gmail.com (ashwinD12 .) Date: Wed, 20 Jul 2016 21:03:44 +0530 Subject: [SciPy-User] Memory error with scipy.interpolate In-Reply-To: References: Message-ID: >From this old thread - https://mail.scipy.org/pipermail/scipy-user/2009-April/020607.html it appears that Scipy.Rbf has a limitation with number of points. If so what is the alternative to doing an interpolation so I can do a contour plot ? On Wed, Jul 20, 2016 at 12:13 PM, ashwinD12 . wrote: > Hello, > I get a memory error when I interpolate a data set of 567 points > as shown in this stack trace > > Traceback (most recent call last): > File "contour.py", line 34, in > rbf = scipy.interpolate.Rbf(x, y, data, function='linear') > File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 200, in __init__ > r = self._call_norm(self.xi, self.xi) > File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 231, in _call_norm > return self.norm(x1, x2) > File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 118, in _euclidean_norm > return np.sqrt(((x1 - x2)**2).sum(axis=0)) > MemoryError > > With numpy earlier I had gotten the same error and I had moved past it > > by this option - > lon,lat = np.meshgrid(lon, lat,sparse=True,copy=False) > > This is the minimum code snippet that should reproduce the problem > grbs = pygrib.open('20000') > > > grb = grbs.select(name='Geopotential',level=500)[0] > data = grb.values > > hgt = [x/10 for x in data] > lat,lon = grb.latlons() > > lon,lat = np.meshgrid(lon, lat,sparse=True,copy=False) > > m=Basemap(projection='mill',lat_ts=10,llcrnrlon=lon.min(), \ > urcrnrlon=lon.max(),llcrnrlat=lat.min(),urcrnrlat=lat.max(), \ > resolution=None) > > > x,y = m(lon,lat) > > rbf = scipy.interpolate.Rbf(x, y, data, function='linear') > zi = rbf(x, y) > > Regards, > > Ashwin. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Wed Jul 20 12:05:22 2016 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Wed, 20 Jul 2016 18:05:22 +0200 Subject: [SciPy-User] Memory error with scipy.interpolate In-Reply-To: References: Message-ID: I can run rbf on 567 points and don't even notice the memory usage. Can you turn your example into a reproducible example? Also, in your case, "zi" is going to be the same as "data" because you are evaluating in the same place you are interpolating from. /David. From evgeny.burovskiy at gmail.com Mon Jul 25 13:54:00 2016 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Mon, 25 Jul 2016 18:54:00 +0100 Subject: [SciPy-User] ANN: scipy 0.18.0 release Message-ID: On behalf of the Scipy development team I am pleased to announce the availability of Scipy 0.18.0. This release contains several great new features and a large number of bug fixes and various improvements, as detailed in the release notes below. 99 people contributed to this release over the course of six months. Thanks to everyone who contributed! This release requires Python 2.7 or 3.4-3.5 and NumPy 1.7.2 or greater. Source tarballs and release notes can be found at https://github.com/scipy/scipy/releases/tag/v0.18.0. OS X and Linux wheels are available from PyPI. For security-conscious, the wheels themselves are signed with my GPG key. Additionally, you can checksum the wheels and verify the checksums with those listed in the README file at https://github.com/scipy/scipy/releases/tag/v0.18.0. Cheers, Evgeni -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 ========================== SciPy 0.18.0 Release Notes ========================== .. contents:: SciPy 0.18.0 is the culmination of 6 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.19.x branch, and on adding new features on the master branch. This release requires Python 2.7 or 3.4-3.5 and NumPy 1.7.1 or greater. Highlights of this release include: - - A new ODE solver for two-point boundary value problems, `scipy.optimize.solve_bvp`. - - A new class, `CubicSpline`, for cubic spline interpolation of data. - - N-dimensional tensor product polynomials, `scipy.interpolate.NdPPoly`. - - Spherical Voronoi diagrams, `scipy.spatial.SphericalVoronoi`. - - Support for discrete-time linear systems, `scipy.signal.dlti`. New features ============ `scipy.integrate` improvements - ------------------------------ A solver of two-point boundary value problems for ODE systems has been implemented in `scipy.integrate.solve_bvp`. The solver allows for non-separated boundary conditions, unknown parameters and certain singular terms. It finds a C1 continious solution using a fourth-order collocation algorithm. `scipy.interpolate` improvements - -------------------------------- Cubic spline interpolation is now available via `scipy.interpolate.CubicSpline`. This class represents a piecewise cubic polynomial passing through given points and C2 continuous. It is represented in the standard polynomial basis on each segment. A representation of n-dimensional tensor product piecewise polynomials is available as the `scipy.interpolate.NdPPoly` class. Univariate piecewise polynomial classes, `PPoly` and `Bpoly`, can now be evaluated on periodic domains. Use ``extrapolate="periodic"`` keyword argument for this. `scipy.fftpack` improvements - ---------------------------- `scipy.fftpack.next_fast_len` function computes the next "regular" number for FFTPACK. Padding the input to this length can give significant performance increase for `scipy.fftpack.fft`. `scipy.signal` improvements - --------------------------- Resampling using polyphase filtering has been implemented in the function `scipy.signal.resample_poly`. This method upsamples a signal, applies a zero-phase low-pass FIR filter, and downsamples using `scipy.signal.upfirdn` (which is also new in 0.18.0). This method can be faster than FFT-based filtering provided by `scipy.signal.resample` for some signals. `scipy.signal.firls`, which constructs FIR filters using least-squares error minimization, was added. `scipy.signal.sosfiltfilt`, which does forward-backward filtering like `scipy.signal.filtfilt` but for second-order sections, was added. Discrete-time linear systems ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `scipy.signal.dlti` provides an implementation of discrete-time linear systems. Accordingly, the `StateSpace`, `TransferFunction` and `ZerosPolesGain` classes have learned a the new keyword, `dt`, which can be used to create discrete-time instances of the corresponding system representation. `scipy.sparse` improvements - --------------------------- The functions `sum`, `max`, `mean`, `min`, `transpose`, and `reshape` in `scipy.sparse` have had their signatures augmented with additional arguments and functionality so as to improve compatibility with analogously defined functions in `numpy`. Sparse matrices now have a `count_nonzero` method, which counts the number of nonzero elements in the matrix. Unlike `getnnz()` and ``nnz`` propety, which return the number of stored entries (the length of the data attribute), this method counts the actual number of non-zero entries in data. `scipy.optimize` improvements - ----------------------------- The implementation of Nelder-Mead minimization, `scipy.minimize(..., method="Nelder-Mead")`, obtained a new keyword, `initial_simplex`, which can be used to specify the initial simplex for the optimization process. Initial step size selection in CG and BFGS minimizers has been improved. We expect that this change will improve numeric stability of optimization in some cases. See pull request gh-5536 for details. Handling of infinite bounds in SLSQP optimization has been improved. We expect that this change will improve numeric stability of optimization in the some cases. See pull request gh-6024 for details. A large suite of global optimization benchmarks has been added to ``scipy/benchmarks/go_benchmark_functions``. See pull request gh-4191 for details. Nelder-Mead and Powell minimization will now only set defaults for maximum iterations or function evaluations if neither limit is set by the caller. In some cases with a slow converging function and only 1 limit set, the minimization may continue for longer than with previous versions and so is more likely to reach convergence. See issue gh-5966. `scipy.stats` improvements - -------------------------- Trapezoidal distribution has been implemented as `scipy.stats.trapz`. Skew normal distribution has been implemented as `scipy.stats.skewnorm`. Burr type XII distribution has been implemented as `scipy.stats.burr12`. Three- and four-parameter kappa distributions have been implemented as `scipy.stats.kappa3` and `scipy.stats.kappa4`, respectively. New `scipy.stats.iqr` function computes the interquartile region of a distribution. Random matrices ~~~~~~~~~~~~~~~ `scipy.stats.special_ortho_group` and `scipy.stats.ortho_group` provide generators of random matrices in the SO(N) and O(N) groups, respectively. They generate matrices in the Haar distribution, the only uniform distribution on these group manifolds. `scipy.stats.random_correlation` provides a generator for random correlation matrices, given specified eigenvalues. `scipy.linalg` improvements - --------------------------- `scipy.linalg.svd` gained a new keyword argument, ``lapack_driver``. Available drivers are ``gesdd`` (default) and ``gesvd``. `scipy.linalg.lapack.ilaver` returns the version of the LAPACK library SciPy links to. `scipy.spatial` improvements - ---------------------------- Boolean distances, `scipy.spatial.pdist`, have been sped up. Improvements vary by the function and the input size. In many cases, one can expect a speed-up of x2--x10. New class `scipy.spatial.SphericalVoronoi` constructs Voronoi diagrams on the surface of a sphere. See pull request gh-5232 for details. `scipy.cluster` improvements - ---------------------------- A new clustering algorithm, the nearest neighbor chain algorithm, has been implemented for `scipy.cluster.hierarchy.linkage`. As a result, one can expect a significant algorithmic improvement (:math:`O(N^2)` instead of :math:`O(N^3)`) for several linkage methods. `scipy.special` improvements - ---------------------------- The new function `scipy.special.loggamma` computes the principal branch of the logarithm of the Gamma function. For real input, ``loggamma`` is compatible with `scipy.special.gammaln`. For complex input, it has more consistent behavior in the complex plane and should be preferred over ``gammaln``. Vectorized forms of spherical Bessel functions have been implemented as `scipy.special.spherical_jn`, `scipy.special.spherical_kn`, `scipy.special.spherical_in` and `scipy.special.spherical_yn`. They are recommended for use over ``sph_*`` functions, which are now deprecated. Several special functions have been extended to the complex domain and/or have seen domain/stability improvements. This includes `spence`, `digamma`, `log1p` and several others. Deprecated features =================== The cross-class properties of `lti` systems have been deprecated. The following properties/setters will raise a `DeprecationWarning`: Name - (accessing/setting raises warning) - (setting raises warning) * StateSpace - (`num`, `den`, `gain`) - (`zeros`, `poles`) * TransferFunction (`A`, `B`, `C`, `D`, `gain`) - (`zeros`, `poles`) * ZerosPolesGain (`A`, `B`, `C`, `D`, `num`, `den`) - () Spherical Bessel functions, ``sph_in``, ``sph_jn``, ``sph_kn``, ``sph_yn``, ``sph_jnyn`` and ``sph_inkn`` have been deprecated in favor of `scipy.special.spherical_jn` and ``spherical_kn``, ``spherical_yn``, ``spherical_in``. The following functions in `scipy.constants` are deprecated: ``C2K``, ``K2C``, ``C2F``, ``F2C``, ``F2K`` and ``K2F``. They are superceded by a new function `scipy.constants.convert_temperature` that can perform all those conversions plus to/from the Rankine temperature scale. Backwards incompatible changes ============================== `scipy.optimize` - ---------------- The convergence criterion for ``optimize.bisect``, ``optimize.brentq``, ``optimize.brenth``, and ``optimize.ridder`` now works the same as ``numpy.allclose``. `scipy.ndimage` - --------------- The offset in ``ndimage.iterpolation.affine_transform`` is now consistently added after the matrix is applied, independent of if the matrix is specified using a one-dimensional or a two-dimensional array. `scipy.stats` - ------------- ``stats.ks_2samp`` used to return nonsensical values if the input was not real or contained nans. It now raises an exception for such inputs. Several deprecated methods of `scipy.stats` distributions have been removed: ``est_loc_scale``, ``vecfunc``, ``veccdf`` and ``vec_generic_moment``. Deprecated functions ``nanmean``, ``nanstd`` and ``nanmedian`` have been removed from `scipy.stats`. These functions were deprecated in scipy 0.15.0 in favor of their `numpy` equivalents. A bug in the ``rvs()`` method of the distributions in `scipy.stats` has been fixed. When arguments to ``rvs()`` were given that were shaped for broadcasting, in many cases the returned random samples were not random. A simple example of the problem is ``stats.norm.rvs(loc=np.zeros(10))``. Because of the bug, that call would return 10 identical values. The bug only affected code that relied on the broadcasting of the shape, location and scale parameters. The ``rvs()`` method also accepted some arguments that it should not have. There is a potential for backwards incompatibility in cases where ``rvs()`` accepted arguments that are not, in fact, compatible with broadcasting. An example is stats.gamma.rvs([2, 5, 10, 15], size=(2,2)) The shape of the first argument is not compatible with the requested size, but the function still returned an array with shape (2, 2). In scipy 0.18, that call generates a ``ValueError``. `scipy.io` - ---------- `scipy.io.netcdf` masking now gives precedence to the ``_FillValue`` attribute over the ``missing_value`` attribute, if both are given. Also, data are only treated as missing if they match one of these attributes exactly: values that differ by roundoff from ``_FillValue`` or ``missing_value`` are no longer treated as missing values. `scipy.interpolate` - ------------------- `scipy.interpolate.PiecewisePolynomial` class has been removed. It has been deprecated in scipy 0.14.0, and `scipy.interpolate.BPoly.from_derivatives` serves as a drop-in replacement. Other changes ============= Scipy now uses ``setuptools`` for its builds instead of plain distutils. This fixes usage of ``install_requires='scipy'`` in the ``setup.py`` files of projects that depend on Scipy (see Numpy issue gh-6551 for details). It potentially affects the way that build/install methods for Scipy itself behave though. Please report any unexpected behavior on the Scipy issue tracker. PR `#6240 `__ changes the interpretation of the `maxfun` option in `L-BFGS-B` based routines in the `scipy.optimize` module. An `L-BFGS-B` search consists of multiple iterations, with each iteration consisting of one or more function evaluations. Whereas the old search strategy terminated immediately upon reaching `maxfun` function evaluations, the new strategy allows the current iteration to finish despite reaching `maxfun`. The bundled copy of Qhull in the `scipy.spatial` subpackage has been upgraded to version 2015.2. The bundled copy of ARPACK in the `scipy.sparse.linalg` subpackage has been upgraded to arpack-ng 3.3.0. The bundled copy of SuperLU in the `scipy.sparse` subpackage has been upgraded to version 5.1.1. Authors ======= * @endolith * @yanxun827 + * @kleskjr + * @MYheavyGo + * @solarjoe + * Gregory Allen + * Gilles Aouizerate + * Tom Augspurger + * Henrik Bengtsson + * Felix Berkenkamp * Per Brodtkorb * Lars Buitinck * Daniel Bunting + * Evgeni Burovski * CJ Carey * Tim Cera * Grey Christoforo + * Robert Cimrman * Philip DeBoer + * Yves Delley + * D?vid Bodn?r + * Ion Elberdin + * Gabriele Farina + * Yu Feng * Andrew Fowlie + * Joseph Fox-Rabinovitz * Simon Gibbons + * Neil Girdhar + * Kolja Glogowski + * Christoph Gohlke * Ralf Gommers * Todd Goodall + * Johnnie Gray + * Alex Griffing * Olivier Grisel * Thomas Haslwanter + * Michael Hirsch + * Derek Homeier * Golnaz Irannejad + * Marek Jacob + * InSuk Joung + * Tetsuo Koyama + * Eugene Krokhalev + * Eric Larson * Denis Laxalde * Antony Lee * Jerry Li + * Henry Lin + * Nelson Liu + * Lo?c Est?ve * Lei Ma + * Osvaldo Martin + * Stefano Martina + * Nikolay Mayorov * Matthieu Melot + * Sturla Molden * Eric Moore * Alistair Muldal + * Maniteja Nandana * Tavi Nathanson + * Andrew Nelson * Joel Nothman * Behzad Nouri * Nikolai Nowaczyk + * Juan Nunez-Iglesias + * Ted Pudlik * Eric Quintero * Yoav Ram * Jonas Rauber + * Tyler Reddy + * Juha Remes * Garrett Reynolds + * Ariel Rokem + * Fabian Rost + * Bill Sacks + * Jona Sassenhagen + * Kari Schoonbee + * Marcello Seri + * Sourav Singh + * Martin Spacek + * S?ren Fuglede J?rgensen + * Bhavika Tekwani + * Martin Thoma + * Sam Tygier + * Meet Udeshi + * Utkarsh Upadhyay * Bram Vandekerckhove + * Sebasti?n Vanrell + * Ze Vinicius + * Pauli Virtanen * Stefan van der Walt * Warren Weckesser * Jakub Wilk + * Josh Wilson * Phillip J. Wolfram + * Nathan Woods * Haochen Wu * G Young + A total of 99 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 0.18.0 - ------------------------ - - `#1484 `__: SVD using *GESVD lapack drivers (Trac #957) - - `#1547 `__: Inconsistent use of offset in ndimage.interpolation.affine_transform()... - - `#1609 `__: special.hyp0f1 returns nan (Trac #1082) - - `#1656 `__: fmin_slsqp enhancement (Trac #1129) - - `#2069 `__: stats broadcasting in rvs (Trac #1544) - - `#2165 `__: sph_jn returns false results for some orders/values (Trac #1640) - - `#2255 `__: Incorrect order of translation and rotation in affine_transform... - - `#2332 `__: hyp0f1 args and return values are unnumpyic (Trac #1813) - - `#2534 `__: The sparse .sum() method with uint8 dtype does not act like the... - - `#3113 `__: Implement ufuncs for CSPHJY, SPHJ, SPHY, CSPHIK, SPHI, SPHIK... - - `#3568 `__: SciPy 0.13.3 - CentOS5 - Errors in test_arpack - - `#3581 `__: optimize: stepsize in fmin_bfgs is "bad" - - `#4476 `__: scipy.sparse non-native endian bug - - `#4484 `__: ftol in optimize.fmin fails to work - - `#4510 `__: sparsetools.cxx call_thunk can segfault due to out of bounds... - - `#5051 `__: ftol and xtol for _minimize_neldermead are absolute instead of... - - `#5097 `__: proposal: spherical Voronoi diagrams - - `#5123 `__: Call to `scipy.sparse.coo_matrix` fails when passed Cython typed... - - `#5220 `__: scipy.cluster.hierarchy.{ward,median,centroid} does not work... - - `#5379 `__: Add a build step at the end of .travis.yml that uploads working... - - `#5440 `__: scipy.optimize.basinhopping: accept_test returning numpy.bool_... - - `#5452 `__: Error in scipy.integrate.nquad when using variable integration... - - `#5520 `__: Cannot inherit csr_matrix properly - - `#5533 `__: Kendall tau implementation uses Python mergesort - - `#5553 `__: stats.tiecorrect overflows - - `#5589 `__: Add the Type XII Burr distribution to stats. - - `#5612 `__: sparse.linalg factorizations slow for small k due to default... - - `#5626 `__: io.netcdf masking should use masked_equal rather than masked_value - - `#5637 `__: Simple cubic spline interpolation? - - `#5683 `__: BUG: Akima1DInterpolator may return nans given multidimensional... - - `#5686 `__: scipy.stats.ttest_ind_from_stats does not accept arrays - - `#5702 `__: scipy.ndimage.interpolation.affine_transform lacks documentation... - - `#5718 `__: Wrong computation of weighted minkowski distance in cdist - - `#5745 `__: move to setuptools for next release - - `#5752 `__: DOC: solve_discrete_lyapunov equation puts transpose in wrong... - - `#5760 `__: signal.ss2tf doesn't handle zero-order state-space models - - `#5764 `__: Hypergeometric function hyp0f1 behaves incorrectly for complex... - - `#5814 `__: stats NaN Policy Error message inconsistent with code - - `#5833 `__: docstring of stats.binom_test() needs an update - - `#5853 `__: Error in scipy.linalg.expm for complex matrix with shape (1,1) - - `#5856 `__: Specify Nelder-Mead initial simplex - - `#5865 `__: scipy.linalg.expm fails for certain numpy matrices - - `#5915 `__: optimize.basinhopping - variable referenced before assignment. - - `#5916 `__: LSQUnivariateSpline fitting failed with knots generated from... - - `#5927 `__: unicode vs. string comparison in scipy.stats.binned_statistic_dd - - `#5936 `__: faster implementation of ks_2samp - - `#5948 `__: csc matrix .mean returns single element matrix rather than scalar - - `#5959 `__: BUG: optimize test error for root when using lgmres - - `#5972 `__: Test failures for sparse sum tests on 32-bit Python - - `#5976 `__: Unexpected exception in scipy.sparse.bmat while using 0 x 0 matrix - - `#6008 `__: scipy.special.kl_div not available in 0.14.1 - - `#6011 `__: The von-Mises entropy is broken - - `#6016 `__: python crashes for linalg.interpolative.svd with certain large... - - `#6017 `__: Wilcoxon signed-rank test with zero_method="pratt" or "zsplit"... - - `#6028 `__: stats.distributions does not have trapezoidal distribution - - `#6035 `__: Wrong link in f_oneway - - `#6056 `__: BUG: signal.decimate should only accept discrete LTI objects - - `#6093 `__: Precision error on Linux 32 bit with openblas - - `#6101 `__: Barycentric transforms test error on Python3, 32-bit Linux - - `#6105 `__: scipy.misc.face docstring is incorrect - - `#6113 `__: scipy.linalg.logm fails for a trivial matrix - - `#6128 `__: Error in dot method of sparse COO array, when used with numpy... - - `#6132 `__: Failures with latest MKL - - `#6136 `__: Failures on `master` with MKL - - `#6162 `__: fmin_l_bfgs_b returns inconsistent results (fmin ? f(xmin)) and... - - `#6165 `__: optimize.minimize infinite loop with Newton-CG - - `#6167 `__: incorrect distribution fitting for data containing boundary values. - - `#6194 `__: lstsq() and others detect numpy.complex256 as real - - `#6216 `__: ENH: improve accuracy of ppf cdf roundtrip for bradford - - `#6217 `__: BUG: weibull_min.logpdf return nan for c=1 and x=0 - - `#6218 `__: Is there a method to cap shortest path search distances? - - `#6222 `__: PchipInterpolator no longer handles a 2-element array - - `#6226 `__: ENH: improve accuracy for logistic.ppf and logistic.isf - - `#6227 `__: ENH: improve accuracy for rayleigh.logpdf and rayleigh.logsf... - - `#6228 `__: ENH: improve accuracy of ppf cdf roundtrip for gumbel_l - - `#6235 `__: BUG: alpha.pdf and alpha.logpdf returns nan for x=0 - - `#6245 `__: ENH: improve accuracy for ppf-cdf and sf-isf roundtrips for invgamma - - `#6263 `__: BUG: stats: Inconsistency in the multivariate_normal docstring - - `#6292 `__: Python 3 unorderable type errors in test_sparsetools.TestInt32Overflow - - `#6316 `__: TestCloughTocher2DInterpolator.test_dense crashes python3.5.2rc1_64bit... - - `#6318 `__: Scipy interp1d 'nearest' not working for high values on x-axis Pull requests for 0.18.0 - ------------------------ - - `#3226 `__: DOC: Change `nb` and `na` to conventional m and n - - `#3867 `__: allow cKDTree.query taking a list input in k. - - `#4191 `__: ENH: Benchmarking global optimizers - - `#4356 `__: ENH: add PPoly.solve(y) for solving ``p(x) == y`` - - `#4370 `__: DOC separate boolean distance functions for clarity - - `#4678 `__: BUG: sparse: ensure index dtype is large enough to pass all parameters... - - `#4881 `__: scipy.signal: Add the class dlti for linear discrete-time systems.... - - `#4901 `__: MAINT: add benchmark and improve docstring for signal.lfilter - - `#5043 `__: ENH: sparse: add count_nonzero method - - `#5136 `__: Attribute kurtosistest() to Anscombe & Glynn (1983) - - `#5186 `__: ENH: Port upfirdn - - `#5232 `__: ENH: adding spherical Voronoi diagram algorithm to scipy.spatial - - `#5279 `__: ENH: Bessel filters with different normalizations, high order - - `#5384 `__: BUG: Closes #5027 distance function always casts bool to double - - `#5392 `__: ENH: Add zero_phase kwarg to signal.decimate - - `#5394 `__: MAINT: sparse: non-canonical test cleanup and fixes - - `#5424 `__: DOC: add Scipy developers guide - - `#5442 `__: STY: PEP8 amendments - - `#5472 `__: Online QR in LGMRES - - `#5526 `__: BUG: stats: Fix broadcasting in the rvs() method of the distributions. - - `#5530 `__: MAINT: sparse: set `format` attr explicitly - - `#5536 `__: optimize: fix up cg/bfgs initial step sizes - - `#5548 `__: PERF: improves performance in stats.kendalltau - - `#5549 `__: ENH: Nearest-neighbor chain algorithm for hierarchical clustering - - `#5554 `__: MAINT/BUG: closes overflow bug in stats.tiecorrect - - `#5557 `__: BUG: modify optimize.bisect to achieve desired tolerance - - `#5581 `__: DOC: Tutorial for least_squares - - `#5606 `__: ENH: differential_evolution - moving core loop of solve method... - - `#5609 `__: [MRG] test against numpy dev - - `#5611 `__: use setuptools for bdist_egg distributions - - `#5615 `__: MAINT: linalg: tighten _decomp_update + special: remove unused... - - `#5622 `__: Add SO(N) rotation matrix generator - - `#5623 `__: ENH: special: Add vectorized spherical Bessel functions. - - `#5627 `__: Response to issue #5160, implements the skew normal distribution... - - `#5628 `__: DOC: Align the description and operation - - `#5632 `__: DOC: special: Expanded docs for Airy, elliptic, Bessel functions. - - `#5633 `__: MAINT: linalg: unchecked malloc in _decomp_update - - `#5634 `__: MAINT: optimize: tighten _group_columns - - `#5640 `__: Fixes for io.netcdf masking - - `#5645 `__: MAINT: size 0 vector handling in cKDTree range queries - - `#5649 `__: MAINT: update license text - - `#5650 `__: DOC: Clarify Exponent Order in ltisys.py - - `#5651 `__: DOC: Clarify Documentation for scipy.special.gammaln - - `#5652 `__: DOC: Fixed scipy.special.betaln Doc - - `#5653 `__: [MRG] ENH: CubicSpline interpolator - - `#5654 `__: ENH: Burr12 distribution to stats module - - `#5659 `__: DOC: Define BEFORE/AFTER in runtests.py -h for bench-compare - - `#5660 `__: MAINT: remove functions deprecated before 0.16.0 - - `#5662 `__: ENH: Circular statistic optimization - - `#5663 `__: MAINT: remove uses of np.testing.rand - - `#5665 `__: MAINT: spatial: remove matching distance implementation - - `#5667 `__: Change some HTTP links to HTTPS - - `#5669 `__: DOC: zpk2sos can't do analog, array_like, etc. - - `#5670 `__: Update conf.py - - `#5672 `__: MAINT: move a sample distribution to a subclass of rv_discrete - - `#5678 `__: MAINT: stats: remove est_loc_scale method - - `#5679 `__: MAINT: DRY up generic computations for discrete distributions - - `#5680 `__: MAINT: stop shadowing builtins in stats.distributions - - `#5681 `__: forward port ENH: Re-enable broadcasting of fill_value - - `#5684 `__: BUG: Fix Akima1DInterpolator returning nans - - `#5690 `__: BUG: fix stats.ttest_ind_from_stats to handle arrays. - - `#5691 `__: BUG: fix generator in io._loadarff to comply with PEP 0479 - - `#5693 `__: ENH: use math.factorial for exact factorials - - `#5695 `__: DOC: dx might be a float, not only an integer - - `#5699 `__: MAINT: io: micro-optimize Matlab reading code for size - - `#5701 `__: Implement OptimizeResult.__dir__ - - `#5703 `__: ENH: stats: make R? printing optional in probplot - - `#5704 `__: MAINT: typo ouf->out - - `#5705 `__: BUG: fix typo in query_pairs - - `#5707 `__: DOC:Add some explanation for ftol xtol in scipy.optimize.fmin - - `#5708 `__: DOC: optimize: PEP8 minimize docstring - - `#5709 `__: MAINT: optimize Cython code for speed and size - - `#5713 `__: [DOC] Fix broken link to reference - - `#5717 `__: DOC: curve_fit raises RuntimeError on failure. - - `#5724 `__: forward port gh-5720 - - `#5728 `__: STY: remove a blank line - - `#5729 `__: ENH: spatial: speed up boolean distances - - `#5732 `__: MAINT: differential_evolution changes to default keywords break... - - `#5733 `__: TST: differential_evolution - population initiation tests - - `#5736 `__: Complex number support in log1p, expm1, and xlog1py - - `#5741 `__: MAINT: sparse: clean up extraction functions - - `#5742 `__: DOC: signal: Explain fftbins in get_window - - `#5748 `__: ENH: Add O(N) random matrix generator - - `#5749 `__: ENH: Add polyphase resampling - - `#5756 `__: RFC: Bump the minimum numpy version, drop older python versions - - `#5761 `__: DOC: Some improvements to least squares docstrings - - `#5762 `__: MAINT: spatial: distance refactoring - - `#5768 `__: DOC: Fix io.loadmat docstring for mdict param - - `#5770 `__: BUG: Accept anything np.dtype can handle for a dtype in sparse.random - - `#5772 `__: Update sparse.csgraph.laplacian docstring - - `#5777 `__: BUG: fix special.hyp0f1 to work correctly for complex inputs. - - `#5780 `__: DOC: Update PIL error install URL - - `#5781 `__: DOC: Fix documentation on solve_discrete_lyapunov - - `#5782 `__: DOC: cKDTree and KDTree now reference each other - - `#5783 `__: DOC: Clarify finish behaviour in scipy.optimize.brute - - `#5784 `__: MAINT: Change default tolerances of least_squares to 1e-8 - - `#5787 `__: BUG: Allow Processing of Zero Order State Space Models in signal.ss2tf - - `#5788 `__: DOC, BUG: Clarify and Enforce Input Types to 'Data' Objects - - `#5789 `__: ENH: sparse: speedup LIL matrix slicing (was #3338) - - `#5791 `__: DOC: README: remove coveralls.io - - `#5792 `__: MAINT: remove uses of deprecated np.random.random_integers - - `#5794 `__: fix affine_transform (fixes #1547 and #5702) - - `#5795 `__: DOC: Removed uniform method from kmeans2 doc - - `#5797 `__: DOC: Clarify the computation of weighted minkowski - - `#5798 `__: BUG: Ensure scipy's _asfarray returns ndarray - - `#5799 `__: TST: Mpmath testing patch - - `#5801 `__: allow reading of certain IDL 8.0 .sav files - - `#5803 `__: DOC: fix module name in error message - - `#5804 `__: DOC: special: Expanded docs for special functions. - - `#5805 `__: DOC: Fix order of returns in _spectral_helper - - `#5806 `__: ENH: sparse: vectorized coo_matrix.diagonal - - `#5808 `__: ENH: Added iqr function to compute IQR metric in scipy/stats/stats.py - - `#5810 `__: MAINT/BENCH: sparse: Benchmark cleanup and additions - - `#5811 `__: DOC: sparse.linalg: shape, not size - - `#5813 `__: Update sparse ARPACK functions min `ncv` value - - `#5815 `__: BUG: Error message contained wrong values - - `#5816 `__: remove dead code from stats tests - - `#5820 `__: "in"->"a" in order_filter docstring - - `#5821 `__: DOC: README: INSTALL.txt was renamed in 2014 - - `#5825 `__: DOC: typo in the docstring of least_squares - - `#5826 `__: MAINT: sparse: increase test coverage - - `#5827 `__: NdPPoly rebase - - `#5828 `__: Improve numerical stability of hyp0f1 for large orders - - `#5829 `__: ENH: sparse: Add copy parameter to all .toXXX() methods in sparse... - - `#5830 `__: DOC: rework INSTALL.rst.txt - - `#5831 `__: Adds plotting options to voronoi_plot_2d - - `#5834 `__: Update stats.binom_test() docstring - - `#5836 `__: ENH, TST: Allow SIMO tf's for tf2ss - - `#5837 `__: DOC: Image examples - - `#5838 `__: ENH: sparse: add eliminate_zeros() to coo_matrix - - `#5839 `__: BUG: Fixed name of NumpyVersion.__repr__ - - `#5845 `__: MAINT: Fixed typos in documentation - - `#5847 `__: Fix bugs in sparsetools - - `#5848 `__: BUG: sparse.linalg: add locks to ensure ARPACK threadsafety - - `#5849 `__: ENH: sparse.linalg: upgrade to superlu 5.1.1 - - `#5851 `__: ENH: expose lapack's ilaver to python to allow lapack verion... - - `#5852 `__: MAINT: runtests.py: ensure Ctrl-C interrupts the build - - `#5854 `__: DOC: Minor update to documentation - - `#5855 `__: Pr 5640 - - `#5859 `__: ENH: Add random correlation matrix generator - - `#5862 `__: BUG: Allow expm for complex matrix with shape (1, 1) - - `#5863 `__: FIX: Fix test - - `#5864 `__: DOC: add a little note about the Normal survival function (Q-function) - - `#5867 `__: Fix for #5865 - - `#5869 `__: extend normal distribution cdf to complex domain - - `#5872 `__: DOC: Note that morlet and cwt don't work together - - `#5875 `__: DOC: interp2d class description - - `#5876 `__: MAINT: spatial: remove a stray print statement - - `#5878 `__: MAINT: Fixed noisy UserWarnings in ndimage tests. Fixes #5877 - - `#5879 `__: MAINT: sparse.linalg/superlu: add explicit casts to resolve compiler... - - `#5880 `__: MAINT: signal: import gcd from math and not fractions when on... - - `#5887 `__: Neldermead initial simplex - - `#5894 `__: BUG: _CustomLinearOperator unpickalable in python3.5 - - `#5895 `__: DOC: special: slightly improve the multigammaln docstring - - `#5900 `__: Remove duplicate assignment. - - `#5901 `__: Update bundled ARPACK - - `#5904 `__: ENH: Make convolve and correlate order-agnostic - - `#5905 `__: ENH: sparse.linalg: further LGMRES cleanups - - `#5906 `__: Enhancements and cleanup in scipy.integrate (attempt #2) - - `#5907 `__: ENH: Change sparse `.sum` and `.mean` dtype casting to match... - - `#5909 `__: changes for convolution symmetry - - `#5913 `__: MAINT: basinhopping remove instance test closes #5440 - - `#5919 `__: MAINT: uninitialised var if basinhopping niter=0. closes #5915 - - `#5920 `__: BLD: Fix missing lsame.c error for MKL - - `#5921 `__: DOC: interpolate: add example showing how to work around issue... - - `#5926 `__: MAINT: spatial: upgrade to Qhull 2015.2 - - `#5928 `__: MAINT: sparse: optimize DIA sum/diagonal, csgraph.laplacian - - `#5929 `__: Update info/URL for octave-maintainers discussion - - `#5930 `__: TST: special: silence DeprecationWarnings from sph_yn - - `#5931 `__: ENH: implement the principle branch of the logarithm of Gamma. - - `#5934 `__: Typo: "mush" => "must" - - `#5935 `__: BUG:string comparison stats._binned_statistic closes #5927 - - `#5938 `__: Cythonize stats.ks_2samp for a ~33% gain in speed. - - `#5939 `__: DOC: fix optimize.fmin convergence docstring - - `#5941 `__: Fix minor typo in squareform docstring - - `#5942 `__: Update linregress stderr description. - - `#5943 `__: ENH: Improve numerical accuracy of lognorm - - `#5944 `__: Merge vonmises into stats pyx - - `#5945 `__: MAINT: interpolate: Tweak declaration to avoid cython warning... - - `#5946 `__: MAINT: sparse: clean up format conversion methods - - `#5949 `__: BUG: fix sparse .mean to return a scalar instead of a matrix - - `#5955 `__: MAINT: Replace calls to `hanning` with `hann` - - `#5956 `__: DOC: Missing periods interfering with parsing - - `#5958 `__: MAINT: add a test for lognorm.sf underflow - - `#5961 `__: MAINT _centered(): rename size to shape - - `#5962 `__: ENH: constants: Add multi-scale temperature conversion function - - `#5965 `__: ENH: special: faster way for calculating comb() for exact=True - - `#5975 `__: ENH: Improve FIR path of signal.decimate - - `#5977 `__: MAINT/BUG: sparse: remove overzealous bmat checks - - `#5978 `__: minimize_neldermead() stop at user requested maxiter or maxfev - - `#5983 `__: ENH: make sparse `sum` cast dtypes like NumPy `sum` for 32-bit... - - `#5985 `__: BUG, API: Add `jac` parameter to curve_fit - - `#5989 `__: ENH: Add firls least-squares fitting - - `#5990 `__: BUG: read tries to handle 20-bit WAV files but shouldn't - - `#5991 `__: DOC: Cleanup wav read/write docs and add tables for common types - - `#5994 `__: ENH: Add gesvd method for svd - - `#5996 `__: MAINT: Wave cleanup - - `#5997 `__: TST: Break up upfirdn tests & compare to lfilter - - `#6001 `__: Filter design docs - - `#6002 `__: COMPAT: Expand compatibility fromnumeric.py - - `#6007 `__: ENH: Skip conversion of TF to TF in freqresp - - `#6009 `__: DOC: fix incorrect versionadded for entr, rel_entr, kl_div - - `#6013 `__: Fixed the entropy calculation of the von Mises distribution. - - `#6014 `__: MAINT: make gamma, rgamma use loggamma for complex arguments - - `#6020 `__: WIP: ENH: add exact=True factorial for vectors - - `#6022 `__: Added 'lanczos' to the image interpolation function list. - - `#6024 `__: BUG: optimize: do not use dummy constraints in SLSQP when no... - - `#6025 `__: ENH: Boundary value problem solver for ODE systems - - `#6029 `__: MAINT: Future imports for optimize._lsq - - `#6030 `__: ENH: stats.trap - adding trapezoidal distribution closes #6028 - - `#6031 `__: MAINT: Some improvements to optimize._numdiff - - `#6032 `__: MAINT: Add special/_comb.c to .gitignore - - `#6033 `__: BUG: check the requested approximation rank in interpolative.svd - - `#6034 `__: DOC: Doc for mannwhitneyu in stats.py corrected - - `#6040 `__: FIX: Edit the wrong link in f_oneway - - `#6044 `__: BUG: (ordqz) always increase parameter lwork by 1. - - `#6047 `__: ENH: extend special.spence to complex arguments. - - `#6049 `__: DOC: Add documentation of PR #5640 to the 0.18.0 release notes - - `#6050 `__: MAINT: small cleanups related to loggamma - - `#6070 `__: Add asarray to explicitly cast list to numpy array in wilcoxon... - - `#6071 `__: DOC: antialiasing filter and link decimate resample, etc. - - `#6075 `__: MAINT: reimplement special.digamma for complex arguments - - `#6080 `__: avoid multiple computation in kstest - - `#6081 `__: Clarified pearson correlation return value - - `#6085 `__: ENH: allow long indices of sparse matrix with umfpack in spsolve() - - `#6086 `__: fix description for associated Laguerre polynomials - - `#6087 `__: Corrected docstring of splrep. - - `#6094 `__: ENH: special: change zeta signature to zeta(x, q=1) - - `#6095 `__: BUG: fix integer overflow in special.spence - - `#6106 `__: Fixed Issue #6105 - - `#6116 `__: BUG: matrix logarithm edge case - - `#6119 `__: TST: DeprecationWarnings in stats on python 3.5 closes #5885 - - `#6120 `__: MAINT: sparse: clean up sputils.isintlike - - `#6122 `__: DOC: optimize: linprog docs should say minimize instead of maximize - - `#6123 `__: DOC: optimize: document the `fun` field in `scipy.optimize.OptimizeResult` - - `#6124 `__: Move FFT zero-padding calculation from signaltools to fftpack - - `#6125 `__: MAINT: improve special.gammainc in the ``a ~ x`` regime. - - `#6130 `__: BUG: sparse: Fix COO dot with zero columns - - `#6138 `__: ENH: stats: Improve behavior of genextreme.sf and genextreme.isf - - `#6146 `__: MAINT: simplify the expit implementation - - `#6151 `__: MAINT: special: make generate_ufuncs.py output deterministic - - `#6152 `__: TST: special: better test for gammainc at large arguments - - `#6153 `__: ENH: Make next_fast_len public and faster - - `#6154 `__: fix typo "mush"-->"must" - - `#6155 `__: DOC: Fix some incorrect RST definition lists - - `#6160 `__: make logsumexp error out on a masked array - - `#6161 `__: added missing bracket to rosen documentation - - `#6163 `__: ENH: Added "kappa4" and "kappa3" distributions. - - `#6164 `__: DOC: Minor clean-up in integrate._bvp - - `#6169 `__: Fix mpf_assert_allclose to handle iterable results, such as maps - - `#6170 `__: Fix pchip_interpolate convenience function - - `#6172 `__: Corrected misplaced bracket in doc string - - `#6175 `__: ENH: sparse.csgraph: Pass indices to shortest_path - - `#6178 `__: TST: increase test coverage of sf and isf of a generalized extreme... - - `#6179 `__: TST: avoid a deprecation warning from numpy - - `#6181 `__: ENH: Boundary conditions for CubicSpline - - `#6182 `__: DOC: Add examples/graphs to max_len_seq - - `#6183 `__: BLD: update Bento build config files for recent changes. - - `#6184 `__: BUG: fix issue in io/wavfile for float96 input. - - `#6186 `__: ENH: Periodic extrapolation for PPoly and BPoly - - `#6192 `__: MRG: Add circle-CI - - `#6193 `__: ENH: sparse: avoid setitem densification - - `#6196 `__: Fixed missing sqrt in docstring of Mahalanobis distance in cdist,... - - `#6206 `__: MAINT: Minor changes in solve_bvp - - `#6207 `__: BUG: linalg: for BLAS, downcast complex256 to complex128, not... - - `#6209 `__: BUG: io.matlab: avoid buffer overflows in read_element_into - - `#6210 `__: BLD: use setuptools when building. - - `#6214 `__: BUG: sparse.linalg: fix bug in LGMRES breakdown handling - - `#6215 `__: MAINT: special: make loggamma use zdiv - - `#6220 `__: DOC: Add parameter - - `#6221 `__: ENH: Improve Newton solver for solve_bvp - - `#6223 `__: pchip should work for length-2 arrays - - `#6224 `__: signal.lti: deprecate cross-class properties/setters - - `#6229 `__: BUG: optimize: avoid an infinite loop in Newton-CG - - `#6230 `__: Add example for application of gaussian filter - - `#6236 `__: MAINT: gumbel_l accuracy - - `#6237 `__: MAINT: rayleigh accuracy - - `#6238 `__: MAINT: logistic accuracy - - `#6239 `__: MAINT: bradford distribution accuracy - - `#6240 `__: MAINT: avoid bad fmin in l-bfgs-b due to maxfun interruption - - `#6241 `__: MAINT: weibull_min accuracy - - `#6246 `__: ENH: Add _support_mask to distributions - - `#6247 `__: fixed a print error for an example of ode - - `#6249 `__: MAINT: change x-axis label for stats.probplot to "theoretical... - - `#6250 `__: DOC: fix typos - - `#6251 `__: MAINT: constants: filter out test noise from deprecated conversions - - `#6252 `__: MAINT: io/arff: remove unused variable - - `#6253 `__: Add examples to scipy.ndimage.filters - - `#6254 `__: MAINT: special: fix some build warnings - - `#6258 `__: MAINT: inverse gamma distribution accuracy - - `#6260 `__: MAINT: signal.decimate - Use discrete-time objects - - `#6262 `__: BUG: odr: fix string formatting - - `#6267 `__: TST: fix some test issues in interpolate and stats. - - `#6269 `__: TST: fix some warnings in the test suite - - `#6274 `__: ENH: Add sosfiltfilt - - `#6276 `__: DOC: update release notes for 0.18.0 - - `#6277 `__: MAINT: update the author name mapping - - `#6282 `__: DOC: Correcting references for scipy.stats.normaltest - - `#6283 `__: DOC: some more additions to 0.18.0 release notes. - - `#6284 `__: Add `.. versionadded::` directive to `loggamma`. - - `#6285 `__: BUG: stats: Inconsistency in the multivariate_normal docstring... - - `#6290 `__: Add author list, gh-lists to 0.18.0 release notes - - `#6293 `__: TST: special: relax a test's precision - - `#6295 `__: BUG: sparse: stop comparing None and int in bsr_matrix constructor - - `#6313 `__: MAINT: Fix for python 3.5 travis-ci build problem. - - `#6327 `__: TST: signal: use assert_allclose for testing near-equality in... - - `#6330 `__: BUG: spatial/qhull: allocate qhT via malloc to ensure CRT likes... - - `#6332 `__: TST: fix stats.iqr test to not emit warnings, and fix line lengths. - - `#6334 `__: MAINT: special: fix a test for hyp0f1 - - `#6347 `__: TST: spatial.qhull: skip a test on 32-bit platforms - - `#6350 `__: BUG: optimize/slsqp: don't overwrite an array out of bounds - - `#6351 `__: BUG: #6318 Interp1d 'nearest' integer x-axis overflow issue fixed - - `#6355 `__: Backports for 0.18.0 Checksums ========= MD5 ~~~ e9ddb97336421e110a3f20e1b1f6c057 scipy-0.18.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl f203cdea575c4e5d9c11f88480e609c4 scipy-0.18.0-cp27-cp27m-manylinux1_i686.whl ccca6aeb87717ca383a15e1d3cd3b3e9 scipy-0.18.0-cp27-cp27m-manylinux1_x86_64.whl 3426dd5731fcdb9f56ee30623def9ac0 scipy-0.18.0-cp27-cp27mu-manylinux1_i686.whl 96f1e6ac2ed3a845802050840929681a scipy-0.18.0-cp27-cp27mu-manylinux1_x86_64.whl aec151bd64fee0651dc804f8b57a1948 scipy-0.18.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 6e7bcd119bfcd3504f5d432424779fd2 scipy-0.18.0-cp34-cp34m-manylinux1_i686.whl d8b88298b048884225708494bc51f249 scipy-0.18.0-cp34-cp34m-manylinux1_x86_64.whl 3886c82aa705b44662964d5d1de179b8 scipy-0.18.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 1bdc3664de8456bbc45620afbfc3d2cf scipy-0.18.0-cp35-cp35m-manylinux1_i686.whl 8f28a30aba8dd7fa4c1704088347684e scipy-0.18.0-cp35-cp35m-manylinux1_x86_64.whl d70e7f533622ab705bc016dac328d93e scipy-0.18.0.tar.gz 59bceff108f58b0e72dfac6fb719476e scipy-0.18.0.tar.xz 9ec1363dde2f2c16e833d3cd09f0dd13 scipy-0.18.0.zip SHA256 ~~~~~~ c1357a8d7da900f3eb72c817a554fa82f71a123d232c1d28c4ef5624694ec937 scipy-0.18.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl f1ce34715420670253aef746ca9ac0de6af0db975bc2f7145697b227ed43a411 scipy-0.18.0-cp27-cp27m-manylinux1_i686.whl b10ccbb451d781f5a31bce940150c626260369c31dc7c6cc609c012aae5b8b77 scipy-0.18.0-cp27-cp27m-manylinux1_x86_64.whl 919c035bfd63a8ecd9076e46488108db0847a13ff7bb3e2eb52561af68ffb798 scipy-0.18.0-cp27-cp27mu-manylinux1_i686.whl 7b77d7721a2017fe6fe2369edf6631c1cb4d2665f7a0e0562ed3796e8d8007d4 scipy-0.18.0-cp27-cp27mu-manylinux1_x86_64.whl 61441deb6c9ff093c60b48362a2104fa22212bb8baa09b62643f71df3f1fd361 scipy-0.18.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 41252b1083e657be1beb391c8508fc70298cc3c47214d7b874e5b50e2676229e scipy-0.18.0-cp34-cp34m-manylinux1_i686.whl 1709f0696508047742a80c5f3c7699c7bd28f006cbbe4d2b001ed6ee9fc862a8 scipy-0.18.0-cp34-cp34m-manylinux1_x86_64.whl 5ed943f23f7553c34aec6444a3f929b9eccb0f330b7b4416a3132dd0c4c1068f scipy-0.18.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl d71c4ac8019b5820acd881d1178461e4d900e8dcfe920f3b63faf9381db24a1c scipy-0.18.0-cp35-cp35m-manylinux1_i686.whl ab9db1de7140eaf602d4ba190bbbe07ca608d5d298203afbfeceb170a3f03ef4 scipy-0.18.0-cp35-cp35m-manylinux1_x86_64.whl f01784fb1c2bc246d4211f2482ecf4369db5abaecb9d5afb9d94f6c59663286a scipy-0.18.0.tar.gz fc2d01887276839b1c709e661ba318d8db69a5c398217c79b97d6460a93c2d41 scipy-0.18.0.tar.xz a4f9fb8cddbe681e2d5465a8dcf715ef454b16fe8c9eddf92bffb10bb50ac75e scipy-0.18.0.zip -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEcBAEBAgAGBQJXljjOAAoJEIp0pQ0zQcu+748IAL8pSPGDmi0PQOASCM00DUgK 2tq/3gnI5RIqpMDshQmcuI0/+6iqn0dGYVMJVIWYYBCVSm5D13xtz/3jTC9XGB6E hsnVdSBI/RiZHoEpB7VguzyvOjZIFngkFk5XUpGbr8uaqYcfdQhzKfLnzq7ZHcuB hUoYcku7gdQY5LZDlFTKZYjEBJM9odHa4uYIwezic09Rvr5X1PVjwIhahSXZekpo zH37Zwi/bMNJxjUWtLonIdOB1xFi3rSFgEkycGNYWaiwtxy8dJGrkstiLf9T6HCC ZV0lIVJcgUDG8VuRg8D7dQAoNSpDkYuR3HA17OvPfaKXETb6BDPcFFtqSxwvOF8= =GM4X -----END PGP SIGNATURE----- From pmhobson at gmail.com Tue Jul 26 14:39:55 2016 From: pmhobson at gmail.com (Paul Hobson) Date: Tue, 26 Jul 2016 11:39:55 -0700 Subject: [SciPy-User] Help with the truncnorm distribution Message-ID: Hey everyone, I'm having a little trouble fixing the shape parameters for the truncnorm distribution when fitting it to some data. For example, if I start with this: import numpy from scipy import stats numpy.random.seed(0) lower, upper = -0.5, 2.5 loc, scale = 1, 4 # shape params a = (lower - loc) / scale # -0.375 b = (upper - loc) / scale # +0.375 stn = stats.truncnorm(a, b, loc, scale) data = stn.rvs(size=37000) print(data.min(), data.max()) # prints -0.499800757031 2.49990005171 But then when I do this: stats.truncnorm.fit(data, a, b) # fix a & b? I get this in return: (-0.20306416258741233, # not a 1.0244302986808909, # not b -0.0035940308456371621, 2.443791525672637) How can I fix the a & b parameters in the call to stats.truncnorm.fit? Regards, -Paul (context: I'm adding truncnorm to paramnormal https://github.com/phobson/paramnormal/pull/36) -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Tue Jul 26 14:55:59 2016 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Tue, 26 Jul 2016 19:55:59 +0100 Subject: [SciPy-User] Help with the truncnorm distribution In-Reply-To: References: Message-ID: On Tue, Jul 26, 2016 at 7:39 PM, Paul Hobson wrote: > Hey everyone, > > I'm having a little trouble fixing the shape parameters for the truncnorm > distribution when fitting it to some data. For example, if I start with > this: > > import numpy > from scipy import stats > > numpy.random.seed(0) > > lower, upper = -0.5, 2.5 > loc, scale = 1, 4 > > # shape params > a = (lower - loc) / scale # -0.375 > b = (upper - loc) / scale # +0.375 > > stn = stats.truncnorm(a, b, loc, scale) > data = stn.rvs(size=37000) > print(data.min(), data.max()) > # prints -0.499800757031 2.49990005171 > > But then when I do this: > > stats.truncnorm.fit(data, a, b) # fix a & b? > > I get this in return: > > (-0.20306416258741233, # not a > 1.0244302986808909, # not b > -0.0035940308456371621, > 2.443791525672637) > > > > How can I fix the a & b parameters in the call to stats.truncnorm.fit? > > Regards, > -Paul > > (context: I'm adding truncnorm to paramnormal > https://github.com/phobson/paramnormal/pull/36) > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-user > truncnorm.fit(data, fa=a, fb=b) ? Indeed, In [1]: from scipy.stats import truncnorm In [2]: a, b = 1, 2 In [3]: import numpy as np In [4]: data = truncnorm.rvs(a, b, size=100) In [5]: truncnorm.fit(data, fa=a, fb=b) /home/br/virtualenvs/scipy_py27/local/lib/python2.7/site-packages/scipy/stats/_continuous_distns.py:4417: RuntimeWarning: divide by zero encountered in log self._logdelta = log(self._delta) Out[5]: (1, 2, 0.011142481322181465, 0.99278381920540337) Depending on how old scipy versions you want to support, you might need to use f0 and f1 instead. From pmhobson at gmail.com Tue Jul 26 15:01:08 2016 From: pmhobson at gmail.com (Paul Hobson) Date: Tue, 26 Jul 2016 12:01:08 -0700 Subject: [SciPy-User] Help with the truncnorm distribution In-Reply-To: References: Message-ID: > On Tue, Jul 26, 2016 at 7:39 PM, Paul Hobson wrote: > > Hey everyone, > > > > I'm having a little trouble fixing the shape parameters for the truncnorm > > distribution when fitting it to some data. > [snip] > > > > How can I fix the a & b parameters in the call to stats.truncnorm.fit? > > > > Regards, > > -Paul > On Tue, Jul 26, 2016 at 11:55 AM, Evgeni Burovski < evgeny.burovskiy at gmail.com> wrote: > > truncnorm.fit(data, fa=a, fb=b) ? > > > Indeed, > > In [1]: from scipy.stats import truncnorm > > In [2]: a, b = 1, 2 > > In [3]: import numpy as np > > In [4]: data = truncnorm.rvs(a, b, size=100) > > In [5]: truncnorm.fit(data, fa=a, fb=b) > > /home/br/virtualenvs/scipy_py27/local/lib/python2.7/site-packages/scipy/stats/_continuous_distns.py:4417: > RuntimeWarning: divide by zero encountered in log > self._logdelta = log(self._delta) > Out[5]: (1, 2, 0.011142481322181465, 0.99278381920540337) > > > Depending on how old scipy versions you want to support, you might > need to use f0 and f1 instead. > Ahhh -- thanks very much for such a quick reply! This answers my question entirely. -p -------------- next part -------------- An HTML attachment was scrubbed... URL: From fausto_barbuto at yahoo.ca Wed Jul 27 10:15:48 2016 From: fausto_barbuto at yahoo.ca (Fausto Arinos de A. Barbuto) Date: Wed, 27 Jul 2016 14:15:48 +0000 (UTC) Subject: [SciPy-User] Is there a way to build Scipy on Cygwin? References: <1567117578.5071076.1469628948747.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1567117578.5071076.1469628948747.JavaMail.yahoo@mail.yahoo.com> Hi, I've been trying to build and install Scipy 0.16.1? on Cygwin from the sourcebut I haven't been successful. The process goes apparently well (with lots of compilation warnings as usual) until a fatal compilation error is hit (see at the bottom of this message). Is there a workaround for this or a detailed set of instructions that I mightfollow so as to have Scipy installed on a Cygwin environment?? Scipy installspretty well on both? Windows and Linux,? thus Cygwin shouldn't be a problem Iguess. Thanks! Fausto gcc: build/src.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/dst.c /usr/bin/gfortran -Wall -g -Wall -g -shared build/temp.cygwin-2.5.2-x86_64-2.7/build/src.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/_fftpackmodule.o build/temp.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/zfft.o build/temp.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/drfft.o build/temp.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/zrfft.o build/temp.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/zfftnd.o build/temp.cygwin-2.5.2-x86_64-2.7/build/src.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/dct.o build/temp.cygwin-2.5.2-x86_64-2.7/build/src.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/dst.o build/temp.cygwin-2.5.2-x86_64-2.7/build/src.cygwin-2.5.2-x86_64-2.7/fortranobject.o -L/usr/lib/gcc/x86_64-pc-cygwin/5.4.0 -L/usr/lib/python2.7/config -L/usr/lib -Lbuild/temp.cygwin-2.5.2-x86_64-2.7 -ldfftpack -lfftpack -lpython2.7 -lgfortran -o build/lib.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/_fftpack.dll build/temp.cygwin-2.5.2-x86_64-2.7/libfftpack.a(cosqb.o): In function `cosqb1_': /cygdrive/c/Users/Fausto/Downloads/scipy-0.16.1/scipy/fftpack/src/fftpack/cosqb.f:16: multiple definition of `cosqb1_' /usr/lib/libdfftpack.a(cosqb1.o):cosqb1.f:(.text+0x0): first defined here build/temp.cygwin-2.5.2-x86_64-2.7/libfftpack.a(cosqf.o): In function `cosqf1_': /cygdrive/c/Users/Fausto/Downloads/scipy-0.16.1/scipy/fftpack/src/fftpack/cosqf.f:14: multiple definition of `cosqf1_' /usr/lib/libdfftpack.a(cosqf1.o):cosqf1.f:(.text+0x0): first defined here collect2: error: ld returned 1 exit status build/temp.cygwin-2.5.2-x86_64-2.7/libfftpack.a(cosqb.o): In function `cosqb1_': /cygdrive/c/Users/Fausto/Downloads/scipy-0.16.1/scipy/fftpack/src/fftpack/cosqb.f:16: multiple definition of `cosqb1_' /usr/lib/libdfftpack.a(cosqb1.o):cosqb1.f:(.text+0x0): first defined here build/temp.cygwin-2.5.2-x86_64-2.7/libfftpack.a(cosqf.o): In function `cosqf1_': /cygdrive/c/Users/Fausto/Downloads/scipy-0.16.1/scipy/fftpack/src/fftpack/cosqf.f:14: multiple definition of `cosqf1_' /usr/lib/libdfftpack.a(cosqf1.o):cosqf1.f:(.text+0x0): first defined here collect2: error: ld returned 1 exit status error: Command "/usr/bin/gfortran -Wall -g -Wall -g -shared build/temp.cygwin-2.5.2-x86_64-2.7/build/src.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/_fftpackmodule.o build/temp.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/zfft.o build/temp.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/drfft.o build/temp.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/zrfft.o build/temp.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/zfftnd.o build/temp.cygwin-2.5.2-x86_64-2.7/build/src.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/dct.o build/temp.cygwin-2.5.2-x86_64-2.7/build/src.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/src/dst.o build/temp.cygwin-2.5.2-x86_64-2.7/build/src.cygwin-2.5.2-x86_64-2.7/fortranobject.o -L/usr/lib/gcc/x86_64-pc-cygwin/5.4.0 -L/usr/lib/python2.7/config -L/usr/lib -Lbuild/temp.cygwin-2.5.2-x86_64-2.7 -ldfftpack -lfftpack -lpython2.7 -lgfortran -o build/lib.cygwin-2.5.2-x86_64-2.7/scipy/fftpack/_fftpack.dll" failed with exit status 1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From winash12 at gmail.com Sat Jul 30 03:29:59 2016 From: winash12 at gmail.com (ashwinD12 .) Date: Sat, 30 Jul 2016 12:59:59 +0530 Subject: [SciPy-User] Memory error with scipy.interpolate In-Reply-To: References: Message-ID: Hello David, Thanks for your response. I am not sure whether I was clear in the example I gave you. I gave gridded data. The data array is a 2D array and has shape 561 x 401. Over on github I was told by Pauli Virtanen my data array is too big for Rbf, Since I have grid data I was asked to use a method corresponding to that. I tried using scipy.interpolate.griddata and I am just really confused how to proceed further. I have a 2-dimensional array of heights data and I want to make a contour plot with x axis longitude and y axis longitude. I chose the interpolation method griddata. Following is my code - Originally hgt is array of shape 561 x 401 and lat and lon have the same dimensions. grb = grbs.select(name='Geopotential',level=500)[0] data = grb.values hgt = [x/10 for x in data] lat,lon = grb.latlons() hgt = np.array(hgt) lona,lata = np.linspace(lon.min(),lon.max(),100),np.linspace(lat.min(),lat.max(),100) hgt1 = np.linspace(hgt.min(),hgt.max(),100) hgt1 = np.resize(hgt,(100,100)) longr,latgr = np.meshgrid(lona, lata) print(x.shape,y.shape,hgt.shape) gd = scipy.interpolate.griddata((lon,lat),hgt,(longr,latgr),method='linear') When I do this I get this error message - Traceback (most recent call last): File "contour.py", line 44, in rbf = scipy.interpolate.griddata((lon,lat),hgt,(longr,latgr),method='linear') File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/ndgriddata.py", line 217, in griddata rescale=rescale) File "scipy/interpolate/interpnd.pyx", line 243, in scipy.interpolate.interpnd.LinearNDInterpolator.__init__ (scipy/interpolate/interpnd.c:4934) File "scipy/interpolate/interpnd.pyx", line 78, in scipy.interpolate.interpnd.NDInterpolatorBase.__init__ (scipy/interpolate/interpnd.c:2386) File "scipy/interpolate/interpnd.pyx", line 191, in scipy.interpolate.interpnd._check_init_shape (scipy/interpolate/interpnd.c:4593) ValueError: invalid shape for input data points On Wed, Jul 20, 2016 at 9:03 PM, ashwinD12 . wrote: > From this old thread - > https://mail.scipy.org/pipermail/scipy-user/2009-April/020607.html > > it appears that Scipy.Rbf has a limitation with number of points. If so > what is the alternative to doing an interpolation so I can do a contour > plot ? > > > On Wed, Jul 20, 2016 at 12:13 PM, ashwinD12 . wrote: > >> Hello, >> I get a memory error when I interpolate a data set of 567 points >> as shown in this stack trace >> >> Traceback (most recent call last): >> File "contour.py", line 34, in >> rbf = scipy.interpolate.Rbf(x, y, data, function='linear') >> File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 200, in __init__ >> r = self._call_norm(self.xi, self.xi) >> File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 231, in _call_norm >> return self.norm(x1, x2) >> File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 118, in _euclidean_norm >> return np.sqrt(((x1 - x2)**2).sum(axis=0)) >> MemoryError >> >> With numpy earlier I had gotten the same error and I had moved past it >> >> by this option - >> lon,lat = np.meshgrid(lon, lat,sparse=True,copy=False) >> >> This is the minimum code snippet that should reproduce the problem >> grbs = pygrib.open('20000') >> >> >> grb = grbs.select(name='Geopotential',level=500)[0] >> data = grb.values >> >> hgt = [x/10 for x in data] >> lat,lon = grb.latlons() >> >> lon,lat = np.meshgrid(lon, lat,sparse=True,copy=False) >> >> m=Basemap(projection='mill',lat_ts=10,llcrnrlon=lon.min(), \ >> urcrnrlon=lon.max(),llcrnrlat=lat.min(),urcrnrlat=lat.max(), \ >> resolution=None) >> >> >> x,y = m(lon,lat) >> >> rbf = scipy.interpolate.Rbf(x, y, data, function='linear') >> zi = rbf(x, y) >> >> Regards, >> >> Ashwin. >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From winash12 at gmail.com Sat Jul 30 03:54:48 2016 From: winash12 at gmail.com (ashwinD12 .) Date: Sat, 30 Jul 2016 13:24:48 +0530 Subject: [SciPy-User] Memory error with scipy.interpolate In-Reply-To: References: Message-ID: I was able to get this to work in the following way but I would really appreciate a code review of my approach.When I used linear I got a qHull error. Then I swtiched to nearest and the code seems to run. But the details of the why and what would be deeply appreciated. grb = grbs.select(name='Geopotential',level=500)[0] data = grb.values hgt = [x/10 for x in data] lat,lon = grb.latlons() hgt = np.array(hgt) lona,lata = np.linspace(lon.min(),lon.max(),100),np.linspace(lat.min(),lat.max(),100) hgt1 = np.linspace(hgt.min(),hgt.max(),100) hgt1 = np.resize(hgt,(100,100)) longr,latgr = np.meshgrid(lona, lata) x,y = m(longr,latgr) zi = scipy.interpolate.griddata((lona,lata),hgt1,(longr,latgr),method='nearest') On Sat, Jul 30, 2016 at 12:59 PM, ashwinD12 . wrote: > Hello David, > Thanks for your response. I am not sure whether I was > clear in the example I gave you. I gave gridded data. > > The data array is a 2D array and has shape 561 x 401. Over on github I was > told by Pauli Virtanen my data array is too big for Rbf, Since I have grid > data I was asked to use a method corresponding to that. > > I tried using scipy.interpolate.griddata and I am just really confused how > to proceed further. I have a 2-dimensional array of heights data and I want > to make a contour plot with x axis longitude and y axis longitude. I chose > the interpolation method griddata. Following is my code - > > Originally hgt is array of shape 561 x 401 and lat and lon have the same > dimensions. > > grb = grbs.select(name='Geopotential',level=500)[0] > data = grb.values > > hgt = [x/10 for x in data] > > lat,lon = grb.latlons() > > hgt = np.array(hgt) > > > lona,lata = > np.linspace(lon.min(),lon.max(),100),np.linspace(lat.min(),lat.max(),100) > hgt1 = np.linspace(hgt.min(),hgt.max(),100) > > hgt1 = np.resize(hgt,(100,100)) > > longr,latgr = np.meshgrid(lona, lata) > > > print(x.shape,y.shape,hgt.shape) > > gd = > scipy.interpolate.griddata((lon,lat),hgt,(longr,latgr),method='linear') > > > When I do this I get this error message - > > Traceback (most recent call last): > File "contour.py", line 44, in > rbf = > scipy.interpolate.griddata((lon,lat),hgt,(longr,latgr),method='linear') > File > "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/ndgriddata.py", > line 217, in griddata > rescale=rescale) > File "scipy/interpolate/interpnd.pyx", line 243, in > scipy.interpolate.interpnd.LinearNDInterpolator.__init__ > (scipy/interpolate/interpnd.c:4934) > File "scipy/interpolate/interpnd.pyx", line 78, in > scipy.interpolate.interpnd.NDInterpolatorBase.__init__ > (scipy/interpolate/interpnd.c:2386) > File "scipy/interpolate/interpnd.pyx", line 191, in > scipy.interpolate.interpnd._check_init_shape > (scipy/interpolate/interpnd.c:4593) > ValueError: invalid shape for input data points > > > On Wed, Jul 20, 2016 at 9:03 PM, ashwinD12 . wrote: > >> From this old thread - >> https://mail.scipy.org/pipermail/scipy-user/2009-April/020607.html >> >> it appears that Scipy.Rbf has a limitation with number of points. If so >> what is the alternative to doing an interpolation so I can do a contour >> plot ? >> >> >> On Wed, Jul 20, 2016 at 12:13 PM, ashwinD12 . wrote: >> >>> Hello, >>> I get a memory error when I interpolate a data set of 567 >>> points as shown in this stack trace >>> >>> Traceback (most recent call last): >>> File "contour.py", line 34, in >>> rbf = scipy.interpolate.Rbf(x, y, data, function='linear') >>> File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 200, in __init__ >>> r = self._call_norm(self.xi, self.xi) >>> File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 231, in _call_norm >>> return self.norm(x1, x2) >>> File "/usr/local/lib/python3.4/dist-packages/scipy/interpolate/rbf.py", line 118, in _euclidean_norm >>> return np.sqrt(((x1 - x2)**2).sum(axis=0)) >>> MemoryError >>> >>> With numpy earlier I had gotten the same error and I had moved past it >>> >>> by this option - >>> lon,lat = np.meshgrid(lon, lat,sparse=True,copy=False) >>> >>> This is the minimum code snippet that should reproduce the problem >>> grbs = pygrib.open('20000') >>> >>> >>> grb = grbs.select(name='Geopotential',level=500)[0] >>> data = grb.values >>> >>> hgt = [x/10 for x in data] >>> lat,lon = grb.latlons() >>> >>> lon,lat = np.meshgrid(lon, lat,sparse=True,copy=False) >>> >>> m=Basemap(projection='mill',lat_ts=10,llcrnrlon=lon.min(), \ >>> urcrnrlon=lon.max(),llcrnrlat=lat.min(),urcrnrlat=lat.max(), \ >>> resolution=None) >>> >>> >>> x,y = m(lon,lat) >>> >>> rbf = scipy.interpolate.Rbf(x, y, data, function='linear') >>> zi = rbf(x, y) >>> >>> Regards, >>> >>> Ashwin. >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: