From phillip.m.feldman at gmail.com Sun Apr 1 23:12:28 2018 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Sun, 1 Apr 2018 20:12:28 -0700 Subject: [SciPy-Dev] bad formula for chi-squared probability density function Message-ID: In the following page, the formula for the chi-squared density is incorrect: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chi2.html (See the Wikipedia page at https://en.wikipedia.org/wiki/Chi-squared_distribution#Probability_density_function for the correct expression). Phillip -------------- next part -------------- An HTML attachment was scrubbed... URL: From jvmirca at gmail.com Sun Apr 1 23:53:24 2018 From: jvmirca at gmail.com (=?UTF-8?B?WsOpIFZpbsOtY2l1cw==?=) Date: Mon, 2 Apr 2018 11:53:24 +0800 Subject: [SciPy-Dev] bad formula for chi-squared probability density function In-Reply-To: References: Message-ID: It looks correct to me other than the $\gamma$ which should have been $\Gamma$. On Mon, Apr 2, 2018 at 11:12 AM, Phillip Feldman < phillip.m.feldman at gmail.com> wrote: > In the following page, the formula for the chi-squared density is > incorrect: > > https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chi2.html > (See the Wikipedia page at https://en.wikipedia.org/wiki/ > Chi-squared_distribution#Probability_density_function for the correct > expression). > > Phillip > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -- webpage: http://mirca.github.io github: @mirca twitter: @mircaze -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Sun Apr 1 23:54:47 2018 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Sun, 1 Apr 2018 23:54:47 -0400 Subject: [SciPy-Dev] bad formula for chi-squared probability density function In-Reply-To: References: Message-ID: On Sun, Apr 1, 2018 at 11:12 PM, Phillip Feldman < phillip.m.feldman at gmail.com> wrote: > In the following page, the formula for the chi-squared density is > incorrect: > > https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chi2.html > (See the Wikipedia page at https://en.wikipedia.org/wiki/ > Chi-squared_distribution#Probability_density_function for the correct > expression). > > Thanks, Phillip, that looks like a typesetting problem. There is an extraneous opening parenthesis before the 2 in the denominator, and that lower-case gamma should be upper-case--it is the Gamma function. Warren Phillip > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jvmirca at gmail.com Sun Apr 1 23:56:29 2018 From: jvmirca at gmail.com (=?UTF-8?B?WsOpIFZpbsOtY2l1cw==?=) Date: Mon, 2 Apr 2018 11:56:29 +0800 Subject: [SciPy-Dev] bad formula for chi-squared probability density function In-Reply-To: References: Message-ID: Looks like the issue with $\gamma$ ---> $\Gamma$ is spread all over due to the rewrite from pseudo-code to TeX. See Evgeni Burovski's comment here https://github.com/scipy/scipy/pull/8563 On Mon, Apr 2, 2018 at 11:53 AM, Z? Vin?cius wrote: > It looks correct to me other than the $\gamma$ which should have been > $\Gamma$. > > On Mon, Apr 2, 2018 at 11:12 AM, Phillip Feldman < > phillip.m.feldman at gmail.com> wrote: > >> In the following page, the formula for the chi-squared density is >> incorrect: >> >> https://docs.scipy.org/doc/scipy/reference/generated/scipy. >> stats.chi2.html >> (See the Wikipedia page at https://en.wikipedia.org/wiki/ >> Chi-squared_distribution#Probability_density_function for the correct >> expression). >> >> Phillip >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> > > > -- > webpage: http://mirca.github.io > github: @mirca > twitter: @mircaze > -- webpage: http://mirca.github.io github: @mirca twitter: @mircaze -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Mon Apr 2 14:50:36 2018 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Mon, 2 Apr 2018 14:50:36 -0400 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> Message-ID: On Fri, Mar 30, 2018 at 8:17 PM, Ralf Gommers wrote: > > > On Fri, Mar 30, 2018 at 12:03 PM, Eric Larson > wrote: > >> Top-level module for them alone sounds overkill, and I'm not sure if >>> discoverability alone is enough. >>> >> >> Fine by me. And if we follow the idea that these should be added >> sparingly, we can maintain discoverability without it growing out of >> hand by populating the See Also sections of each function. >> > > I agree with this, the 2 images and 1 ECG signal (to be added) that we > have doesn't justify a top-level module. We don't want to grow more than > the absolute minimum of datasets. The package is already very large, which > is problematic in certain cases. E.g. numpy + scipy still fits in the AWS > Lambda limit of 50 MB, but there's not much margin. > > Note: this is a reply to the thread, and not specifically to Ralf's comments (but those are included). After reading all the replies, the first question that comes to mind is: should SciPy have *any* datasets? I think this question has already been answered: we have had functions that return images in scipy.misc for a long time, and I don't recall anyone ever suggesting that these be removed. (Well, there was lena(), but I don't think anyone had a problem with adding a replacement image.) And the pull request for the ECG dataset has been merged (added to scipy.misc), so there is current support among the developers for providing datasets. So the remaining questions are: (1) Where do the datasets reside? (2) What are the criteria for adding a new datasets? Here's my 2?: (1) Where do the datasets reside? My preference is to keep all the datasets in the top-level module scipy.datasets. Robert preferred this module for discoverability, and I agree. By having all the datasets in one place, anyone can easily see what is available. Teachers and others developing educational material know where to find source material for examples. Developers, too, can easily look for examples to use in our docstrings or tutorials. (By the way, adding examples to the docstrings of all functions is an ongoing effort: https://github.com/scipy/scipy/issues/7168.) Also, there are many well-known datasets that could be used as examples for multiple scipy packages. For a concrete example, a dataset that I could see adding to scipy is the Hald cement dataset. SciPy should eventually have an implementation of the PCA decomposition, and it could conceivably live in scipy.linalg. It would be reasonable to use the Hald data in the docstrings of the new PCA function(s) (cf. https://www.mathworks.com/help/stats/pca.html). At the same time, the Hald data could enrich the docstrings of some functions in scipy.stats. Similarly, Fisher's iris dataset provides a well-known example that could be used in docstrings in both scipy.cluster and scipy.stats. (2) What are the criteria for adding a new datasets? So far, the only compelling reason I can see to even have datasets is to have interesting examples in the docstrings (or at least in our tutorials). For example, the docstring for scipy.ndimage.gaussian_filter and several other transformations in ndimage use the image returned by scipy.misc.ascent(): https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.gaussian_filter.html I could see the benefit of having well-known datasets such as Fisher's iris data, the Hald cement data, and some version of a sunspot activity time series, to be used in the docstrings in scipy.stats, scipy.signal, scipy.cluster, scipy.linalg, and elsewhere. St?fan expressed regret about including datasets in sciki-image. The main issue seems to be "bloat". Scikit-image is an image processing library, so the datasets used there are likely all images, and there is a minimum size for a sample image to be useful as an example. For scipy, we already have two images, and I don't think we'll need more. The newly added ECG dataset is 116K (which is less than the existing image datasets: "ascent.dat" is 515K and "face.dat" is 1.5M). The potential datasets that I mentioned above (Hald, iris, sunspots) are all very small. If we are conservative about what we include, and focus on datasets chosen specifically to demonstrate scipy functionality, we should be able to avoid dataset bloat. This leads to my proposal for the criteria for adding a dataset: (a) Not too big. The size of a dataset should not exceed $MAX (but I don't have a good suggestion for what $MAX should be at the moment). (b) The dataset should be well-known, where "well-known" means that the dataset is one that is already widely used as an example and many people will know it by name (e.g. the iris dataset), or the dataset is a sample of a common signal type or format (e.g an ECG signal, or an image such as misc.ascent). (c) We actually *use* the dataset in one of *our* docstrings or tutorials. I don't think our datasets package should become a repository of interesting scientific data with no connection to the scipy code. Its purpose should be to enrich our documentation. (Note that by this criterion, the recently added ECG signal would not qualify!) To summarize: I'm in favor scipy.datasets, a conservatively curated subpackage containing well-known datasets. Warren P.S. I should add that I'm not in favor putting code in scipy that fetches data from the web. That type of data retrieval could be useful, but it seems more appropriate for a package that is independent of scipy. > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Tue Apr 3 04:06:31 2018 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Tue, 3 Apr 2018 10:06:31 +0200 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> Message-ID: On 31 March 2018 at 02:17, Ralf Gommers wrote: > > > On Fri, Mar 30, 2018 at 12:03 PM, Eric Larson > wrote: > >> Top-level module for them alone sounds overkill, and I'm not sure if >>> discoverability alone is enough. >>> >> >> Fine by me. And if we follow the idea that these should be added >> sparingly, we can maintain discoverability without it growing out of >> hand by populating the See Also sections of each function. >> > > I agree with this, the 2 images and 1 ECG signal (to be added) that we > have doesn't justify a top-level module. We don't want to grow more than > the absolute minimum of datasets. The package is already very large, which > is problematic in certain cases. E.g. numpy + scipy still fits in the AWS > Lambda limit of 50 MB, but there's not much margin. > The biggest subpackage is sparse, and there most of the space is taken by _ sparsetools.cpython-35m-x86_64-linux-gnu.so According to size -A -d, the biggest sections are debug. The same goes for the second biggest, special. Can it run without those sections? On preliminary checks, it seems that stripping .debug_info and .debug_loc trim down the size from 38 to 3.7 MB, and the test suite still passes. If we really need to trim down the size for installing in things like Lambda, could we have a scipy-lite for production environments, that is the same as scipy but without unnecessary debug? I imagine tracebacks would not be as informative, but that shouldn't matter for production environments. My first thought was to remove docstrings, comments, tests, and data, but maybe they don't amount to so much for the trouble. On the topic at hand, I would agree to having a few, small datasets to showcase functionality. I think a few kilobytes can go a long way to show and benchmark. As far as I can see, a top level module is free: it wouldn't add any maintenance burden, and would make them easier to find. /David. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 4 10:54:41 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Apr 2018 08:54:41 -0600 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> Message-ID: On Mon, Apr 2, 2018 at 12:50 PM, Warren Weckesser < warren.weckesser at gmail.com> wrote: > > > On Fri, Mar 30, 2018 at 8:17 PM, Ralf Gommers > wrote: > >> >> >> On Fri, Mar 30, 2018 at 12:03 PM, Eric Larson >> wrote: >> >>> Top-level module for them alone sounds overkill, and I'm not sure if >>>> discoverability alone is enough. >>>> >>> >>> Fine by me. And if we follow the idea that these should be added >>> sparingly, we can maintain discoverability without it growing out of >>> hand by populating the See Also sections of each function. >>> >> >> I agree with this, the 2 images and 1 ECG signal (to be added) that we >> have doesn't justify a top-level module. We don't want to grow more than >> the absolute minimum of datasets. The package is already very large, which >> is problematic in certain cases. E.g. numpy + scipy still fits in the AWS >> Lambda limit of 50 MB, but there's not much margin. >> >> > > Note: this is a reply to the thread, and not specifically to Ralf's > comments (but those are included). > > After reading all the replies, the first question that comes to mind is: > should SciPy have *any* datasets? > > I think this question has already been answered: we have had functions > that return images in scipy.misc for a long time, and I don't recall anyone > ever suggesting that these be removed. (Well, there was lena(), but I > don't think anyone had a problem with adding a replacement image.) And the > pull request for the ECG dataset has been merged (added to scipy.misc), so > there is current support among the developers for providing datasets. > > So the remaining questions are: > (1) Where do the datasets reside? > (2) What are the criteria for adding a new datasets? > > Here's my 2?: > > (1) Where do the datasets reside? > > My preference is to keep all the datasets in the top-level module > scipy.datasets. Robert preferred this module for discoverability, and I > agree. By having all the datasets in one place, anyone can easily see what > is available. Teachers and others developing educational material know > where to find source material for examples. Developers, too, can easily > look for examples to use in our docstrings or tutorials. (By the way, > adding examples to the docstrings of all functions is an ongoing effort: > https://github.com/scipy/scipy/issues/7168.) > > Also, there are many well-known datasets that could be used as examples > for multiple scipy packages. For a concrete example, a dataset that I > could see adding to scipy is the Hald cement dataset. SciPy should > eventually have an implementation of the PCA decomposition, and it could > conceivably live in scipy.linalg. It would be reasonable to use the Hald > data in the docstrings of the new PCA function(s) (cf. > https://www.mathworks.com/help/stats/pca.html). At the same time, the > Hald data could enrich the docstrings of some functions in scipy.stats. > > Similarly, Fisher's iris dataset provides a well-known example that could > be used in docstrings in both scipy.cluster and scipy.stats. > > > (2) What are the criteria for adding a new datasets? > > So far, the only compelling reason I can see to even have datasets is to > have interesting examples in the docstrings (or at least in our > tutorials). For example, the docstring for scipy.ndimage.gaussian_filter > and several other transformations in ndimage use the image returned by > scipy.misc.ascent(): > > https://docs.scipy.org/doc/scipy/reference/generated/ > scipy.ndimage.gaussian_filter.html > > I could see the benefit of having well-known datasets such as Fisher's > iris data, the Hald cement data, and some version of a sunspot activity > time series, to be used in the docstrings in scipy.stats, scipy.signal, > scipy.cluster, scipy.linalg, and elsewhere. > > St?fan expressed regret about including datasets in sciki-image. The main > issue seems to be "bloat". Scikit-image is an image processing library, so > the datasets used there are likely all images, and there is a minimum size > for a sample image to be useful as an example. For scipy, we already have > two images, and I don't think we'll need more. The newly added ECG dataset > is 116K (which is less than the existing image datasets: "ascent.dat" is > 515K and "face.dat" is 1.5M). The potential datasets that I mentioned > above (Hald, iris, sunspots) are all very small. If we are conservative > about what we include, and focus on datasets chosen specifically to > demonstrate scipy functionality, we should be able to avoid dataset bloat. > > This leads to my proposal for the criteria for adding a dataset: > > (a) Not too big. The size of a dataset should not exceed $MAX (but I > don't have a good suggestion for what $MAX should be at the moment). > (b) The dataset should be well-known, where "well-known" means that the > dataset is one that is already widely used as an example and many people > will know it by name (e.g. the iris dataset), or the dataset is a sample of > a common signal type or format (e.g an ECG signal, or an image such as > misc.ascent). > (c) We actually *use* the dataset in one of *our* docstrings or > tutorials. I don't think our datasets package should become a repository > of interesting scientific data with no connection to the scipy code. Its > purpose should be to enrich our documentation. (Note that by this > criterion, the recently added ECG signal would not qualify!) > > To summarize: I'm in favor scipy.datasets, a conservatively curated > subpackage containing well-known datasets. > > There are also some standard functions used for testing optimization. I wonder if it would be reasonable to make those public? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikofski at berkeley.edu Thu Apr 5 00:07:14 2018 From: mikofski at berkeley.edu (Mark Alexander Mikofski) Date: Thu, 05 Apr 2018 04:07:14 +0000 Subject: [SciPy-Dev] =?utf-8?q?TST=3A_add_benchmark_for_newton=2C_secant?= =?utf-8?q?=2C_halley_=C2=B7_Issue_=238587_=C2=B7_scipy/scipy?= Message-ID: Hi all, I've added some airspeed velocity benchmarks for the Newton methods in scipy.optimize.zeros. You can compare them to the c-language zeros methods like brentq. The 1st test function "f1", a parabola, is similar to the c-language "f2", also a parabola. This is the PR: https://github.com/scipy/scipy/pull/8587 Thank, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From haberland at ucla.edu Fri Apr 6 15:08:20 2018 From: haberland at ucla.edu (Matt Haberland) Date: Fri, 6 Apr 2018 12:08:20 -0700 Subject: [SciPy-Dev] New subpackage: scipy.data Message-ID: I wanted to extend Charles' comment that in addition to the test problems there are optimization *benchmarks*. For instance, in scipy/benchmarks/ benchmarks/linprog_benchmark_files there are ~90 benchmark problems (all NETLIB LP benchmarks as .npz files) totaling ~12MB. The current linprog benchmark only uses two of them by default. Sounds like these should be moved if space is such a concern. On Wed, Apr 4, 2018 at 7:54 AM, Charles R Harris wrote: > > > On Mon, Apr 2, 2018 at 12:50 PM, Warren Weckesser < > warren.weckesser at gmail.com> wrote: > >> >> >> On Fri, Mar 30, 2018 at 8:17 PM, Ralf Gommers >> wrote: >> >>> >>> >>> On Fri, Mar 30, 2018 at 12:03 PM, Eric Larson >>> wrote: >>> >>>> Top-level module for them alone sounds overkill, and I'm not sure if >>>>> discoverability alone is enough. >>>>> >>>> >>>> Fine by me. And if we follow the idea that these should be added >>>> sparingly, we can maintain discoverability without it growing out of >>>> hand by populating the See Also sections of each function. >>>> >>> >>> I agree with this, the 2 images and 1 ECG signal (to be added) that we >>> have doesn't justify a top-level module. We don't want to grow more than >>> the absolute minimum of datasets. The package is already very large, which >>> is problematic in certain cases. E.g. numpy + scipy still fits in the AWS >>> Lambda limit of 50 MB, but there's not much margin. >>> >>> >> >> Note: this is a reply to the thread, and not specifically to Ralf's >> comments (but those are included). >> >> After reading all the replies, the first question that comes to mind is: >> should SciPy have *any* datasets? >> >> I think this question has already been answered: we have had functions >> that return images in scipy.misc for a long time, and I don't recall anyone >> ever suggesting that these be removed. (Well, there was lena(), but I >> don't think anyone had a problem with adding a replacement image.) And the >> pull request for the ECG dataset has been merged (added to scipy.misc), so >> there is current support among the developers for providing datasets. >> >> So the remaining questions are: >> (1) Where do the datasets reside? >> (2) What are the criteria for adding a new datasets? >> >> Here's my 2?: >> >> (1) Where do the datasets reside? >> >> My preference is to keep all the datasets in the top-level module >> scipy.datasets. Robert preferred this module for discoverability, and I >> agree. By having all the datasets in one place, anyone can easily see what >> is available. Teachers and others developing educational material know >> where to find source material for examples. Developers, too, can easily >> look for examples to use in our docstrings or tutorials. (By the way, >> adding examples to the docstrings of all functions is an ongoing effort: >> https://github.com/scipy/scipy/issues/7168.) >> >> Also, there are many well-known datasets that could be used as examples >> for multiple scipy packages. For a concrete example, a dataset that I >> could see adding to scipy is the Hald cement dataset. SciPy should >> eventually have an implementation of the PCA decomposition, and it could >> conceivably live in scipy.linalg. It would be reasonable to use the Hald >> data in the docstrings of the new PCA function(s) (cf. >> https://www.mathworks.com/help/stats/pca.html). At the same time, the >> Hald data could enrich the docstrings of some functions in scipy.stats. >> >> Similarly, Fisher's iris dataset provides a well-known example that could >> be used in docstrings in both scipy.cluster and scipy.stats. >> >> >> (2) What are the criteria for adding a new datasets? >> >> So far, the only compelling reason I can see to even have datasets is to >> have interesting examples in the docstrings (or at least in our >> tutorials). For example, the docstring for scipy.ndimage.gaussian_filter >> and several other transformations in ndimage use the image returned by >> scipy.misc.ascent(): >> >> https://docs.scipy.org/doc/scipy/reference/generated/scipy. >> ndimage.gaussian_filter.html >> >> I could see the benefit of having well-known datasets such as Fisher's >> iris data, the Hald cement data, and some version of a sunspot activity >> time series, to be used in the docstrings in scipy.stats, scipy.signal, >> scipy.cluster, scipy.linalg, and elsewhere. >> >> St?fan expressed regret about including datasets in sciki-image. The >> main issue seems to be "bloat". Scikit-image is an image processing >> library, so the datasets used there are likely all images, and there is a >> minimum size for a sample image to be useful as an example. For scipy, we >> already have two images, and I don't think we'll need more. The newly >> added ECG dataset is 116K (which is less than the existing image datasets: >> "ascent.dat" is 515K and "face.dat" is 1.5M). The potential datasets that >> I mentioned above (Hald, iris, sunspots) are all very small. If we are >> conservative about what we include, and focus on datasets chosen >> specifically to demonstrate scipy functionality, we should be able to avoid >> dataset bloat. >> >> This leads to my proposal for the criteria for adding a dataset: >> >> (a) Not too big. The size of a dataset should not exceed $MAX (but I >> don't have a good suggestion for what $MAX should be at the moment). >> (b) The dataset should be well-known, where "well-known" means that the >> dataset is one that is already widely used as an example and many people >> will know it by name (e.g. the iris dataset), or the dataset is a sample of >> a common signal type or format (e.g an ECG signal, or an image such as >> misc.ascent). >> (c) We actually *use* the dataset in one of *our* docstrings or >> tutorials. I don't think our datasets package should become a repository >> of interesting scientific data with no connection to the scipy code. Its >> purpose should be to enrich our documentation. (Note that by this >> criterion, the recently added ECG signal would not qualify!) >> >> To summarize: I'm in favor scipy.datasets, a conservatively curated >> subpackage containing well-known datasets. >> >> > There are also some standard functions used for testing optimization. I > wonder if it would be reasonable to make those public? > > Chuck > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -- Matt Haberland Assistant Adjunct Professor in the Program in Computing Department of Mathematics 6617A Math Sciences Building, UCLA -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Apr 6 16:46:57 2018 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 06 Apr 2018 22:46:57 +0200 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: Message-ID: <640cabb49de7554b0ac56c6e1b2b35d4b57801b7.camel@iki.fi> pe, 2018-04-06 kello 12:08 -0700, Matt Haberland kirjoitti: > I wanted to extend Charles' comment that in addition to the test > problems > there are optimization *benchmarks*. For instance, in > scipy/benchmarks/ > benchmarks/linprog_benchmark_files there are ~90 benchmark problems > (all > NETLIB LP benchmarks as .npz files) totaling ~12MB. The current > linprog > benchmark only uses two of them by default. Sounds like these should > be > moved if space is such a concern. Note that "pip install scipy" does not install those files, so that it is off less concern for deployment on resource-limited machines. Pauli > > On Wed, Apr 4, 2018 at 7:54 AM, Charles R Harris il.com> > wrote: > > > > > > > On Mon, Apr 2, 2018 at 12:50 PM, Warren Weckesser < > > warren.weckesser at gmail.com> wrote: > > > > > > > > > > > On Fri, Mar 30, 2018 at 8:17 PM, Ralf Gommers > > .com> > > > wrote: > > > > > > > > > > > > > > > On Fri, Mar 30, 2018 at 12:03 PM, Eric Larson > > > ail.com> > > > > wrote: > > > > > > > > > Top-level module for them alone sounds overkill, and I'm not > > > > > sure if > > > > > > discoverability alone is enough. > > > > > > > > > > > > > > > > Fine by me. And if we follow the idea that these should be > > > > > added > > > > > sparingly, we can maintain discoverability without it growing > > > > > out of > > > > > hand by populating the See Also sections of each function. > > > > > > > > > > > > > I agree with this, the 2 images and 1 ECG signal (to be added) > > > > that we > > > > have doesn't justify a top-level module. We don't want to grow > > > > more than > > > > the absolute minimum of datasets. The package is already very > > > > large, which > > > > is problematic in certain cases. E.g. numpy + scipy still fits > > > > in the AWS > > > > Lambda limit of 50 MB, but there's not much margin. > > > > > > > > > > > > > > Note: this is a reply to the thread, and not specifically to > > > Ralf's > > > comments (but those are included). > > > > > > After reading all the replies, the first question that comes to > > > mind is: > > > should SciPy have *any* datasets? > > > > > > I think this question has already been answered: we have had > > > functions > > > that return images in scipy.misc for a long time, and I don't > > > recall anyone > > > ever suggesting that these be removed. (Well, there was lena(), > > > but I > > > don't think anyone had a problem with adding a replacement > > > image.) And the > > > pull request for the ECG dataset has been merged (added to > > > scipy.misc), so > > > there is current support among the developers for providing > > > datasets. > > > > > > So the remaining questions are: > > > (1) Where do the datasets reside? > > > (2) What are the criteria for adding a new datasets? > > > > > > Here's my 2?: > > > > > > (1) Where do the datasets reside? > > > > > > My preference is to keep all the datasets in the top-level module > > > scipy.datasets. Robert preferred this module for discoverability, > > > and I > > > agree. By having all the datasets in one place, anyone can > > > easily see what > > > is available. Teachers and others developing educational > > > material know > > > where to find source material for examples. Developers, too, can > > > easily > > > look for examples to use in our docstrings or tutorials. (By the > > > way, > > > adding examples to the docstrings of all functions is an ongoing > > > effort: > > > https://github.com/scipy/scipy/issues/7168.) > > > > > > Also, there are many well-known datasets that could be used as > > > examples > > > for multiple scipy packages. For a concrete example, a dataset > > > that I > > > could see adding to scipy is the Hald cement dataset. SciPy > > > should > > > eventually have an implementation of the PCA decomposition, and > > > it could > > > conceivably live in scipy.linalg. It would be reasonable to use > > > the Hald > > > data in the docstrings of the new PCA function(s) (cf. > > > https://www.mathworks.com/help/stats/pca.html). At the same > > > time, the > > > Hald data could enrich the docstrings of some functions in > > > scipy.stats. > > > > > > Similarly, Fisher's iris dataset provides a well-known example > > > that could > > > be used in docstrings in both scipy.cluster and scipy.stats. > > > > > > > > > (2) What are the criteria for adding a new datasets? > > > > > > So far, the only compelling reason I can see to even have > > > datasets is to > > > have interesting examples in the docstrings (or at least in our > > > tutorials). For example, the docstring for > > > scipy.ndimage.gaussian_filter > > > and several other transformations in ndimage use the image > > > returned by > > > scipy.misc.ascent(): > > > > > > https://docs.scipy.org/doc/scipy/reference/generated/scipy. > > > ndimage.gaussian_filter.html > > > > > > I could see the benefit of having well-known datasets such as > > > Fisher's > > > iris data, the Hald cement data, and some version of a sunspot > > > activity > > > time series, to be used in the docstrings in scipy.stats, > > > scipy.signal, > > > scipy.cluster, scipy.linalg, and elsewhere. > > > > > > St?fan expressed regret about including datasets in sciki- > > > image. The > > > main issue seems to be "bloat". Scikit-image is an image > > > processing > > > library, so the datasets used there are likely all images, and > > > there is a > > > minimum size for a sample image to be useful as an example. For > > > scipy, we > > > already have two images, and I don't think we'll need more. The > > > newly > > > added ECG dataset is 116K (which is less than the existing image > > > datasets: > > > "ascent.dat" is 515K and "face.dat" is 1.5M). The potential > > > datasets that > > > I mentioned above (Hald, iris, sunspots) are all very small. If > > > we are > > > conservative about what we include, and focus on datasets chosen > > > specifically to demonstrate scipy functionality, we should be > > > able to avoid > > > dataset bloat. > > > > > > This leads to my proposal for the criteria for adding a dataset: > > > > > > (a) Not too big. The size of a dataset should not exceed $MAX > > > (but I > > > don't have a good suggestion for what $MAX should be at the > > > moment). > > > (b) The dataset should be well-known, where "well-known" means > > > that the > > > dataset is one that is already widely used as an example and many > > > people > > > will know it by name (e.g. the iris dataset), or the dataset is a > > > sample of > > > a common signal type or format (e.g an ECG signal, or an image > > > such as > > > misc.ascent). > > > (c) We actually *use* the dataset in one of *our* docstrings or > > > tutorials. I don't think our datasets package should become a > > > repository > > > of interesting scientific data with no connection to the scipy > > > code. Its > > > purpose should be to enrich our documentation. (Note that by > > > this > > > criterion, the recently added ECG signal would not qualify!) > > > > > > To summarize: I'm in favor scipy.datasets, a conservatively > > > curated > > > subpackage containing well-known datasets. > > > > > > > > > > There are also some standard functions used for testing > > optimization. I > > wonder if it would be reasonable to make those public? > > > > Chuck > > > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From virginia.m.frey at gmail.com Sat Apr 7 18:14:47 2018 From: virginia.m.frey at gmail.com (Virginia Frey) Date: Sun, 8 Apr 2018 08:14:47 +1000 Subject: [SciPy-Dev] Dev suggestion: expand signal.slepian functions Message-ID: Hi everyone, In my work I do a lot with Slepian functions, and I find scipy's current capability to make those functions very limited, so I propose to expand it. I'll start with a little motivation for why I think Slepians are great and why it's worth to expand scipy's Slepian capabilities, and then I propose the changes I would make and the new code I would add. *1. Motivation* Slepians (or, using the official term: "discrete, prolate, spheroidal sequences (DPSS)") are the provably optimal solution to the so-called spectral concentration problem [1], so they have a lot of application in frequency analysis [2] and recently my group and I started working on them in the context of ambient noise analysis for quantum computers [3, 4]. Slepians are the eigensolutions of the so-called Toelpitz matrix and there are therefor many of them. The different eigenvectors are called "orders" of the Slepians. In frequency analysis and in my work, it is very common to use multiple orders because (given the right bandwidth parameter) a set of Slepian orders forms a maximally spectrally concentrated "rectangle" in frequency space. Based on those Slepian orders, Thompson [2] developed the "multitaper" spectral reconstruction technique, in which multiple orders are applied as window functions and the optimal guess for the frequency spectrum is obtained by combining individual order estimates. *2. What's the current implementation* The current implementation of Slepians in scipy's signal module only offers the zeroth order, so it is not possible to use for e.g. the multitaper spectral analysis. In addition to that, because of the way the eigensystem is solved, it breaks with a MemoryError when using high numbers of points (i.e. a few thousand). The request for high numbers of points is not unreasonable: people do take long time-traces and the FFT makes it possible to compute Fourier transforms on those. *3. How could it be expanded?* I suggest to re-implement the existing function in order to: a) calculate a variable number of Slepian orders b) allow for higher point numbers by using a Newtonian solver for the eigenvalues I have working code for this, which is based on the implementation suggested in [5] but is in pure Python. It consists of three functions: the main Slepian function plus two little helper functions which would not be exposed to the user. Most of the code is linted and pep8'd but I haven't got unit tests yet. Once we have the full set of Slepians available, from there we (or I) could also implement the multitaper spectral analysis technique, but let's do one step at a time. On a side node, not sure if this is a good argument for expanding the library, but both a Slepian function and a multitaper function exist in Matlab as well. *4. What's next?* So, what do you guys think of my proposal? If you like it, please advise me on how to proceed as this is the first time I'm planning to contribute. From what I took of the web page, I'll start by downloading the repo - and then should I make a new branch? I'm also happy to send some code through via email for your review first. Thanks, and I'm looking forward to hearing back from you, Virginia (from sunny Sydney) ---- References: [1] https://en.wikipedia.org/wiki/Spectral_concentration_problem [2] Thompson, "Spectrum estimation and harmonic analysis". Proc. IEEE 70(9), 1982 DOI: 10.1109/PROC.1982.12433 [3] V. Frey et al. "Application of optimal band-limited control protocols to quantum noise sensing", Nature Comms 8, 2189, 2017. DOI: 10.1038/s41467-017-02298-2 [4] L.M. Norris et al. "Optimally band-limited spectroscopy of control noise using a qubit sensor", arXiv preprint at https://arxiv.org/abs/1803.05538 [5] W.H. Press "Numerical Recipes in C++: The Art of Scientific Computing" Cambrige University Press, 3rd Edition -------------- next part -------------- An HTML attachment was scrubbed... URL: From blairuk at gmail.com Sun Apr 8 00:21:47 2018 From: blairuk at gmail.com (Blair Azzopardi) Date: Sun, 8 Apr 2018 05:21:47 +0100 Subject: [SciPy-Dev] [scipy/scipy] ENH: Add PDF, CDF and parameter estimation for Stable Distributions (#7374) Message-ID: > On 8 April 2018 at 01:49, Andrea Karlova wrote: > >> On 28 November 2017 at 18:33, Blair Azzopardi wrote: >> >>> On Tue, 28 Nov 2017, 16:01 Andrea Karlova, wrote: >>> >>>> On 28 November 2017 at 13:17, Blair Azzopardi wrote: >>>> >>>>> ---------- Forwarded message --------- >>>>> From: An >>>>> Date: Tue, 28 Nov 2017, 10:00 >>>>> Subject: Re: [scipy/scipy] ENH: Add PDF, CDF and parameter estimation for Stable Distributions (#7374) >>>>> To: scipy/scipy >>>>> >>>>> >>>>> Blair, is there a way to have a chat via email? >>>>> >>>>> On 24 November 2017 at 20:38, Blair Azzopardi < notifications at github.com> >>>>> wrote: >>>>> >>>>> Hi @an81 . Thank you for the 550+ page book. >>>>> Please can you be a bit more specific? Some sample code goes a long way >>>>> too. Also can you perhaps test the existing code and highlight where the >>>>> Gibbs effect might be more prominent? eg low alpha etc; perhaps this can be >>>>> just documented with a recommendation that users use quad in these cases >>>>> (already in code). This is until better implementation is available. >>>>> >>>>> ? >>>>> You are receiving this because you were mentioned. >>>>> Reply to this email directly, view it on GitHub >>>>> , or mute >>>>> the thread >>>>> >>>> >>>> Hi Andrea >>>> >>>> I hope you're well and yes no problem talking chatting via email. Actually makes sense. >>>> >>>> Also, I hope you don't mind me messaging you via an email address found in a previous comment. >>>> >>>> I took a look at that book you linked to but unfortunately I don't have enough time to process it currently. I will read it in time mind you. I've found a shorter paper that might offer similar suggestions to yours: >>>> >>>> http://prac.im.pwr.edu.pl/~hugo/publ/SFB2005-008_Borak_ Haerdle_Weron.pdf >>>> >>>> Although I'm not 100% sure as it doesn't mention mejer g functions. What are your thoughts? >>>> >>>> Kind regards >>>> Blair >>> >>> Hi Blair, >>> >>> thats great. >>> Thx for your email. >>> >>> I ll give you access to my Dropbox folder where I have materials relevant to stable laws. >>> For working with stable laws I found really useful to understand >>> and actively switch between different parametrizations of the characteristic exponent. >>> Zolotarev and others would have polenty of parametrization and each of them is helpful >>> for different task. >>> >>> I guess I can share with you my python code on the github, >>> you can contribute to it if you would feel like so >>> and then we can just plug it into scipy lib. >>> >>> Is it ok to use this gmail account for sending you invitation to Dropbox? >>> >>> Thx, >>> Kind regards, >>> >>> Andrea >> >> Hi Andrea >> >> Yes, please do share with this email address. >> >> Is it possible you could commit your changes to the existing PR I've already set up? I believe this is possible by forking my repo (script fork) and committing to that. The PR includes parameter estimation and some general framework changes around this distribution too. Also I feel there are still use cases where it's useful keep fft method. I've added you a collaborator on my fork. >> >> https://github.com/bsdz/scipy >> >> Kind regards >> Blair > > Hi Blair, > > hope you are well. > I was looking into your code on stable laws in scipy. > I ll be running a group of 4 people at hackaton which is organized by AHL in 2 weeks time > and I am planning to revise and add more code into stable laws implementation. > > I was wondering if you can give me some quick update on what is done so far > and what are urgent issues at this point according to your opinion? > I have some ideas what I would like add, also the documentation page needs to be written, > but just a quick check on what methods you implemented for calculating of > pdfs, cdfs, and parameters estimates and how it went? > > Thanks a lot. > Kind regards, > Andrea > Hi Andrea Thanks for your email. I'm cc-ing this email along with out previous correspondence to the scipy-dev mailing list (as bottom post) . Please direct all your future messages here. It's good to hear you plan to improve my PR (coincidentally at my old employer Man, although at the time they weren't interested in Stable laws). I did implement your suggestions to use Zolotarev's formulations. All the code is under https://github.com/scipy/scipy/pull/7374 as you are aware. You could perhaps look at improving documentation, adding more / improving tests and any optimisations you can think of. Either I can re-add you as a collaborator to my repo (I removed you when I hadn't heard back from you previously) or you can email me a patch and I'll integrate it into PR 7374. Thanks Blair -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikofski at berkeley.edu Sun Apr 8 04:48:32 2018 From: mikofski at berkeley.edu (Mark Alexander Mikofski) Date: Sun, 08 Apr 2018 08:48:32 +0000 Subject: [SciPy-Dev] =?utf-8?q?ENH=3A_WIP=3A_Cython_optimize_=C2=B7_Issue?= =?utf-8?q?_=238431_=C2=B7_scipy/scipy?= Message-ID: I've started work on a cython_optimize.zeros package to address the need for fast looping of scalar zero methods over arrays, see issues 8354 and 7242. https://github.com/scipy/scipy/pull/8431 This PR is still a work in progress. I wanted to get feedback on the API(s) that would be most useful. I don't think there should be too many versions but it should be something widely acceptible. Sorry if my proposals are a bit premature or insufficiently developed, but I wanted to start the discussion. Here are some ideas to start with: Option 1: ctypedef double (*callback_type_tup)(double, tuple) - takes a double and a Python tuple with extra arguments, this is the only interface implemented so far, and only for bisect and a super-lite version of newton, it is modeled after the existing Python C-API methods in zeros.c for bisect and other solvers, see description in cython_optimize/__init__.py Option 2: ctypedef double (*callback_type_struct)(double, void*) - takes a double and a structure that contains a structure with the extra arguments, uses a wrapper to extract the extra arguments and call the function Option 3: ctypedef double (*callback_type)(double, void*) - takes a double and a structure that contains the extra arguments, this implementation does not have a wrapper function to extract the args and call the function Option 4: ctypedef double (*callback_type_array)(double, void*) - takes a double and a structure that contains an array with the extra arguments, this method uses a wrapper to extract the extra arguments and call the function I guess I'm leaning toward either option 2 or 3, but perhaps there are other API(s) that are more desirable? Also there are other questions that this PR raises like: is it okay to split up Newton, Halley, and secant? And is it okay to *not* return a `RootsResult` object from the Ridder, Brent, and bisect methods, or should the API(s) match their Python equivalents? Should there be a numpy cython API for super fast looping with native numpy arrays (or other array type like what?) Also note there is a parallel effort in PR 8357 which uses numpy but only with scipy.optimize.zeros.newton. IMHO both of these efforts are desirable and do not compete against each other. Thanks for your feedback! -------------- next part -------------- An HTML attachment was scrubbed... URL: From 3ukip0s02 at sneakemail.com Sun Apr 8 09:29:01 2018 From: 3ukip0s02 at sneakemail.com (3ukip0s02 at sneakemail.com) Date: Sun, 8 Apr 2018 09:29:01 -0400 Subject: [SciPy-Dev] Dev suggestion: expand signal.slepian functions (Virginia Frey) Message-ID: <26906-1523194162-242017@sneakemail.com> On Sun, 8 Apr 2018 08:14:47 +1000, Virginia Frey wrote: > The current implementation of Slepians in scipy's signal module only > offers the zeroth order, so it is not possible to use for e.g. the > multitaper spectral analysis. In addition to that, because of the way > the eigensystem is solved, it breaks with a MemoryError when using high > numbers of points (i.e. a few thousand). The request for high numbers of > points is not unreasonable: people do take long time-traces and the FFT > makes it possible to compute Fourier transforms on those. > In SciPy 1.1.0, windows.slepian will be deprecated and replaced by windows.dpss : https://scipy.github.io/devdocs/generated/scipy.signal.windows.dpss.html https://github.com/scipy/scipy/pull/7802 https://github.com/scipy/scipy/issues/4354 Does that do what you want? I have working code for this, which is based on the implementation > suggested in [5] but is in pure Python. Aren't there copyright/license issues with using code from Numerical Recipes? > Once we have the full set of Slepians available, from there we (or I) > could also implement the multitaper spectral analysis technique > That would be useful! > So, what do you guys think of my proposal? If you like it, please advise > me on how to proceed as this is the first time I'm planning to > contribute. From what I took of the web page, I'll start by downloading > the repo - and then should I make a new branch? I'm also happy to send > some code through via email for your review first. Can you read through https://github.com/scipy/scipy/pull/7802 and comment if you see a problem with it, or it doesn't do all that you would want it to do? Also we were a little confused about how best to normalize the amplitude of the functions. In your work, do you use the L2 norm? -------------- next part -------------- An HTML attachment was scrubbed... URL: From blairuk at gmail.com Sun Apr 8 13:05:07 2018 From: blairuk at gmail.com (Blair Azzopardi) Date: Sun, 8 Apr 2018 18:05:07 +0100 Subject: [SciPy-Dev] [scipy/scipy] ENH: Add PDF, CDF and parameter estimation for Stable Distributions (#7374) In-Reply-To: References: Message-ID: > On 8 April 2018 at 12:49, Andrea Karlova wrote: >> On 8 April 2018 at 05:21, Blair Azzopardi wrote: >>> On 8 April 2018 at 01:49, Andrea Karlova wrote: >>>> On 28 November 2017 at 18:33, Blair Azzopardi wrote: >>>>> On Tue, 28 Nov 2017, 16:01 Andrea Karlova, wrote: >>>>>> On 28 November 2017 at 13:17, Blair Azzopardi wrote: >>>>>>> ---------- Forwarded message --------- >>>>>>> From: An >>>>>>> Date: Tue, 28 Nov 2017, 10:00 >>>>>>> Subject: Re: [scipy/scipy] ENH: Add PDF, CDF and parameter estimation for Stable Distributions (#7374) >>>>>>> To: scipy/scipy >>>>>>> >>>>>>> Blair, is there a way to have a chat via email? >>>>>>> >>>>>>> On 24 November 2017 at 20:38, Blair Azzopardi < notifications at github.com> >>>>>>> wrote: >>>>>>> >>>>>>> Hi @an81 . Thank you for the 550+ page book. >>>>>>> Please can you be a bit more specific? Some sample code goes a long way >>>>>>> too. Also can you perhaps test the existing code and highlight where the >>>>>>> Gibbs effect might be more prominent? eg low alpha etc; perhaps this can be >>>>>>> just documented with a recommendation that users use quad in these cases >>>>>>> (already in code). This is until better implementation is available. >>>>>>> ? >>>>>>> You are receiving this because you were mentioned. >>>>>>> Reply to this email directly, view it on GitHub >>>>>>> , or mute >>>>>>> the thread >>>>>>> < https://github.com/notifications/unsubscribe-auth/AIfE8VNjHdqANvYfUG8Gg6feKb5np_kLks5s5ylhgaJpZM4NQBiP > >>>>>> >>>>>> Hi Andrea >>>>>> >>>>>> I hope you're well and yes no problem talking chatting via email. Actually makes sense. >>>>>> >>>>>> Also, I hope you don't mind me messaging you via an email address found in a previous comment. >>>>>> >>>>>> I took a look at that book you linked to but unfortunately I don't have enough time to process it currently. I will read it in time mind you. I've found a shorter paper that might offer similar suggestions to yours: >>>>>> >>>>>> http://prac.im.pwr.edu.pl/~hugo/publ/SFB2005-008_Borak_Haerdle_Weron.pdf >>>>>> >>>>>> Although I'm not 100% sure as it doesn't mention mejer g functions. What are your thoughts? >>>>>> >>>>>> Kind regards >>>>>> Blair >>>>> >>>>> Hi Blair, >>>>> >>>>> thats great. >>>>> Thx for your email. >>>>> >>>>> I ll give you access to my Dropbox folder where I have materials relevant to stable laws. >>>>> For working with stable laws I found really useful to understand >>>>> and actively switch between different parametrizations of the characteristic exponent. >>>>> Zolotarev and others would have polenty of parametrization and each of them is helpful >>>>> for different task. >>>>> >>>>> I guess I can share with you my python code on the github, >>>>> you can contribute to it if you would feel like so >>>>> and then we can just plug it into scipy lib. >>>>> >>>>> Is it ok to use this gmail account for sending you invitation to Dropbox? >>>>> >>>>> Thx, >>>>> Kind regards, >>>>> >>>>> Andrea >>>> >>>> Hi Andrea >>>> >>>> Yes, please do share with this email address. >>>> >>>> Is it possible you could commit your changes to the existing PR I've already set up? I believe this is possible by forking my repo (script fork) and committing to that. The PR includes parameter estimation and some general framework changes around this distribution too. Also I feel there are still use cases where it's useful keep fft method. I've added you a collaborator on my fork. >>>> >>>> https://github.com/bsdz/scipy >>>> >>>> Kind regards >>>> Blair >>> >>> Hi Blair, >>> >>> hope you are well. >>> I was looking into your code on stable laws in scipy. >>> I ll be running a group of 4 people at hackaton which is organized by AHL in 2 weeks time >>> and I am planning to revise and add more code into stable laws implementation. >>> >>> I was wondering if you can give me some quick update on what is done so far >>> and what are urgent issues at this point according to your opinion? >>> I have some ideas what I would like add, also the documentation page needs to be written, >>> but just a quick check on what methods you implemented for calculating of >>> pdfs, cdfs, and parameters estimates and how it went? >>> >>> Thanks a lot. >>> Kind regards, >>> Andrea >>> >> >> Hi Andrea >> >> Thanks for your email. I'm cc-ing this email along with out previous correspondence to the scipy-dev mailing list (as bottom post) . Please direct all your future messages here. >> >> It's good to hear you plan to improve my PR (coincidentally at my old employer Man, although at the time they weren't interested in Stable laws). >> >> I did implement your suggestions to use Zolotarev's formulations. All the code is under https://github.com/scipy/scipy/pull/7374 as you are aware. >> >> You could perhaps look at improving documentation, adding more / improving tests and any optimisations you can think of. >> >> Either I can re-add you as a collaborator to my repo (I removed you when I hadn't heard back from you previously) or you can email me a patch and I'll integrate it into PR 7374. >> >> Thanks >> Blair > > Hi Blair, > > yes, if you could add me to your repository, > so I can look into what is done and start to test your code, > so that I can look into some patches, that would be great. Hi Andrea Actually, just to be cautious. Could you fork my scipy repo (it has all the changes there) and then create a PR into that repo? I can then merge it and it should push upstream into my main PR. > How does your code performs in neighbourhood of alpha = 1? > How does it perform when the asymetry parameter beta gets close to its boundary values? Please run the code and check. My testing took samples from Nolan's public domain executables and tested against those values. Although I tested against a range of parameters I might not have tested those ranges specifically. It's all in the unit tests in the PR, see test class TestLevyStable. Extending the testing here would be helpful. > Do you have somewhere in Latex notes on your fft implementation, > with some theoretical investigation of the performance of the method? This is all in the code documentation along with references. > I am planning to write a research paper about the implementation in the Scipy, which I ll put on my ArXiv, > and to which I ll refer in the documentation, so that if people are interested in the performance, they can look into paper. > So I ll be listing the co-authors of this piece of code there. > If there are other people who worked on the stable laws implementation in Scipy, it would be nice if I can have the list of them. > Also if you have some note on fft, I can join it into my research note and add you as a coauthor of the note, > depends upon you. So far, I'm the sole author of this PR. Again, all papers and references are in the PR. Please please do look at the code. > > Kind regards, > Andrea Just some additional points, you need to subscribe to the scipy-dev mailing list for your messages to get through. Also, the mailing list uses bottom posting for email messages; that means, all replies are bottom of original message (not the top.) Look forward to your contribution - thanks! Blair -------------- next part -------------- An HTML attachment was scrubbed... URL: From virginia.m.frey at gmail.com Tue Apr 10 06:47:33 2018 From: virginia.m.frey at gmail.com (Virginia Frey) Date: Tue, 10 Apr 2018 20:47:33 +1000 Subject: [SciPy-Dev] Dev suggestion: expand signal.slepian functions (Virginia Frey) In-Reply-To: <26906-1523194162-242017@sneakemail.com> References: <26906-1523194162-242017@sneakemail.com> Message-ID: Hey, Thanks, I didn't see that. This is actually exactly what I wanted, so that's great! In my work I normalise the DPSS by their area because I need them all to have the same integrated spectral weight. I'll have another look at the multitaper technique and maybe I'll come across a different form of modulation there, in which case I'd let you know. Thanks again, Virginia On 08/04/18 23:29, 3ukip0s02 at sneakemail.com wrote: > On Sun, 8 Apr 2018 08:14:47 +1000, Virginia Frey wrote: > > The current implementation of Slepians in scipy's signal module only > offers the zeroth order, so it is not possible to use for e.g. the > multitaper spectral analysis. In addition to that, because of the way > the eigensystem is solved, it breaks with a MemoryError when using > high > numbers of points (i.e. a few thousand). The request for high > numbers of > points is not unreasonable: people do take long time-traces and > the FFT > makes it possible to compute Fourier transforms on those. > > > In SciPy 1.1.0, windows.slepian will be deprecated and replaced by > windows.dpss : > > https://scipy.github.io/devdocs/generated/scipy.signal.windows.dpss.html > > https://github.com/scipy/scipy/pull/7802 > > https://github.com/scipy/scipy/issues/4354 > > > Does that do what you want? > > I have working code for this, which is based on the implementation > suggested in [5] but is in pure Python. > > > Aren't there copyright/license issues with using code from Numerical > Recipes? > > Once we have the full set of Slepians available, from there we (or I) > could also implement the multitaper spectral analysis technique > > > That would be useful! > > So, what do you guys think of my proposal? If you like it, please > advise > me on how to proceed as this is the first time I'm planning to > contribute. From what I took of the web page, I'll start by > downloading > the repo - and then should I make a new branch? I'm also happy to send > some code through via email for your review first. > > > Can you read through https://github.com/scipy/scipy/pull/7802 > ?and comment if you see a > problem with it, or it doesn't do all that you would want it to do? > > Also we were a little confused about how best to normalize the > amplitude of the functions.? In your work, do you use the L2 norm? > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From blairuk at gmail.com Tue Apr 10 11:34:21 2018 From: blairuk at gmail.com (Blair Azzopardi) Date: Tue, 10 Apr 2018 16:34:21 +0100 Subject: [SciPy-Dev] [scipy/scipy] ENH: Add PDF, CDF and parameter estimation for Stable Distributions (#7374) In-Reply-To: References: Message-ID: On 10 April 2018 at 12:34, Andrea Karlova wrote: > On 8 April 2018 at 18:05, Blair Azzopardi wrote: > >> > On 8 April 2018 at 12:49, Andrea Karlova >> wrote: >> >> On 8 April 2018 at 05:21, Blair Azzopardi wrote: >> >>> On 8 April 2018 at 01:49, Andrea Karlova >> wrote: >> >>>> On 28 November 2017 at 18:33, Blair Azzopardi >> wrote: >> >>>>> On Tue, 28 Nov 2017, 16:01 Andrea Karlova, < >> andrea.karlova at gmail.com> wrote: >> >>>>>> On 28 November 2017 at 13:17, Blair Azzopardi >> wrote: >> >>>>>>> ---------- Forwarded message --------- >> >>>>>>> From: An >> >>>>>>> Date: Tue, 28 Nov 2017, 10:00 >> >>>>>>> Subject: Re: [scipy/scipy] ENH: Add PDF, CDF and parameter >> estimation for Stable Distributions (#7374) >> >>>>>>> To: scipy/scipy >> >>>>>>> >> >>>>>>> Blair, is there a way to have a chat via email? >> >>>>>>> >> >>>>>>> On 24 November 2017 at 20:38, Blair Azzopardi < >> notifications at github.com> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>> Hi @an81 . Thank you for the 550+ page >> book. >> >>>>>>> Please can you be a bit more specific? Some sample code goes a >> long way >> >>>>>>> too. Also can you perhaps test the existing code and highlight >> where the >> >>>>>>> Gibbs effect might be more prominent? eg low alpha etc; perhaps >> this can be >> >>>>>>> just documented with a recommendation that users use quad in >> these cases >> >>>>>>> (already in code). This is until better implementation is >> available. >> >>>>>>> ? >> >>>>>>> You are receiving this because you were mentioned. >> >>>>>>> Reply to this email directly, view it on GitHub >> >>>>>>> , >> or mute >> >>>>>>> the thread >> >>>>>>> > dqANvYfUG8Gg6feKb5np_kLks5s5ylhgaJpZM4NQBiP> >> >>>>>> >> >>>>>> Hi Andrea >> >>>>>> >> >>>>>> I hope you're well and yes no problem talking chatting via email. >> Actually makes sense. >> >>>>>> >> >>>>>> Also, I hope you don't mind me messaging you via an email address >> found in a previous comment. >> >>>>>> >> >>>>>> I took a look at that book you linked to but unfortunately I don't >> have enough time to process it currently. I will read it in time mind you. >> I've found a shorter paper that might offer similar suggestions to yours: >> >>>>>> >> >>>>>> http://prac.im.pwr.edu.pl/~hugo/publ/SFB2005-008_Borak_Haerd >> le_Weron.pdf >> >>>>>> >> >>>>>> Although I'm not 100% sure as it doesn't mention mejer g >> functions. What are your thoughts? >> >>>>>> >> >>>>>> Kind regards >> >>>>>> Blair >> >>>>> >> >>>>> Hi Blair, >> >>>>> >> >>>>> thats great. >> >>>>> Thx for your email. >> >>>>> >> >>>>> I ll give you access to my Dropbox folder where I have materials >> relevant to stable laws. >> >>>>> For working with stable laws I found really useful to understand >> >>>>> and actively switch between different parametrizations of the >> characteristic exponent. >> >>>>> Zolotarev and others would have polenty of parametrization and each >> of them is helpful >> >>>>> for different task. >> >>>>> >> >>>>> I guess I can share with you my python code on the github, >> >>>>> you can contribute to it if you would feel like so >> >>>>> and then we can just plug it into scipy lib. >> >>>>> >> >>>>> Is it ok to use this gmail account for sending you invitation to >> Dropbox? >> >>>>> >> >>>>> Thx, >> >>>>> Kind regards, >> >>>>> >> >>>>> Andrea >> >>>> >> >>>> Hi Andrea >> >>>> >> >>>> Yes, please do share with this email address. >> >>>> >> >>>> Is it possible you could commit your changes to the existing PR I've >> already set up? I believe this is possible by forking my repo (script fork) >> and committing to that. The PR includes parameter estimation and some >> general framework changes around this distribution too. Also I feel there >> are still use cases where it's useful keep fft method. I've added you a >> collaborator on my fork. >> >>>> >> >>>> https://github.com/bsdz/scipy >> >>>> >> >>>> Kind regards >> >>>> Blair >> >>> >> >>> Hi Blair, >> >>> >> >>> hope you are well. >> >>> I was looking into your code on stable laws in scipy. >> >>> I ll be running a group of 4 people at hackaton which is organized >> by AHL in 2 weeks time >> >>> and I am planning to revise and add more code into stable laws >> implementation. >> >>> >> >>> I was wondering if you can give me some quick update on what is done >> so far >> >>> and what are urgent issues at this point according to your opinion? >> >>> I have some ideas what I would like add, also the documentation page >> needs to be written, >> >>> but just a quick check on what methods you implemented for >> calculating of >> >>> pdfs, cdfs, and parameters estimates and how it went? >> >>> >> >>> Thanks a lot. >> >>> Kind regards, >> >>> Andrea >> >>> >> >> >> >> Hi Andrea >> >> >> >> Thanks for your email. I'm cc-ing this email along with out previous >> correspondence to the scipy-dev mailing list (as bottom post) . Please >> direct all your future messages here. >> >> >> >> It's good to hear you plan to improve my PR (coincidentally at my old >> employer Man, although at the time they weren't interested in Stable laws). >> >> >> >> I did implement your suggestions to use Zolotarev's formulations. All >> the code is under https://github.com/scipy/scipy/pull/7374 as you are >> aware. >> >> >> >> You could perhaps look at improving documentation, adding more / >> improving tests and any optimisations you can think of. >> >> >> >> Either I can re-add you as a collaborator to my repo (I removed you >> when I hadn't heard back from you previously) or you can email me a patch >> and I'll integrate it into PR 7374. >> >> >> >> Thanks >> >> Blair >> > >> > Hi Blair, >> > >> > yes, if you could add me to your repository, >> > so I can look into what is done and start to test your code, >> > so that I can look into some patches, that would be great. >> >> Hi Andrea >> >> Actually, just to be cautious. Could you fork my scipy repo (it has all >> the changes there) and then create a PR into that repo? I can then merge it >> and it should push upstream into my main PR. >> > > Sure, no worries. > > >> >> > How does your code performs in neighbourhood of alpha = 1? >> > How does it perform when the asymetry parameter beta gets close to its >> boundary values? >> >> Please run the code and check. My testing took samples from Nolan's >> public domain executables and tested against those values. Although I >> tested against a range of parameters I might not have tested those ranges >> specifically. It's all in the unit tests in the PR, see test class >> TestLevyStable. Extending the testing here would be helpful. >> > > Ok. > > >> >> > Do you have somewhere in Latex notes on your fft implementation, >> > with some theoretical investigation of the performance of the method? >> >> This is all in the code documentation along with references. >> > > Ok, so the answer to my question is no and there is only available info in > the comments in the code, right? > It's in the code, please look at function _pdf_from_cf_with_fft(), it has a docstring that refers to "[MS]". The full reference is then in the main docstring along with page numbers. https://github.com/scipy/scipy/pull/7374/files#diff-6e4287784ff3e7a5c6d89d442bea3fbbR3683-R3694 > >> > I am planning to write a research paper about the implementation in the >> Scipy, which I ll put on my ArXiv, >> > and to which I ll refer in the documentation, so that if people are >> interested in the performance, they can look into paper. >> > So I ll be listing the co-authors of this piece of code there. >> > If there are other people who worked on the stable laws implementation >> in Scipy, it would be nice if I can have the list of them. >> > Also if you have some note on fft, I can join it into my research note >> and add you as a coauthor of the note, >> > depends upon you. >> >> So far, I'm the sole author of this PR. Again, all papers and references >> are in the PR. Please please do look at the code. >> > Ok, done. Thanks. > > >> >> > >> > Kind regards, >> > Andrea >> >> Just some additional points, you need to subscribe to the scipy-dev >> mailing list for your messages to get through. Also, the mailing list uses >> bottom posting for email messages; that means, all replies are bottom of >> original message (not the top.) >> >> > Sure. > Btw, it seems your messages aren't getting through to scip-dev mailing list. You can subscribe here: https://mail.python.org/mailman/listinfo/scipy-dev > > >> Look forward to your contribution - thanks! >> > > Me too :) > >> >> Blair >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Apr 10 15:39:02 2018 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 10 Apr 2018 21:39:02 +0200 Subject: [SciPy-Dev] SciPy 1.1.0 release schedule Message-ID: Hi, 2018-03-25 17:45 -0700, Ralf Gommers wrote: > It's 5 months after the 1.0 release, so it's time to start planning > for > 1.1. There's still quite a few open issues and PRs marked for 1.1.0 > (see > https://github.com/scipy/scipy/milestone/34), but not many that seem > blocking or really difficult to resolve. So I'd like to propose the > following schedule: > > April 11: branch 1.1.x > April 13: rc1 > April 27: rc2 (if needed) > May 4: final release Just a friendly heads up/reminder: staying roughly with the schedule proposed earlier, I'll branch 1.1.x on this Thursday, and work towards getting 1.1.0rc1 out during the weekend. If there are features almost ready to be merged, there's a few days before the branch point. After that, it's mainly bugfixes that'll get to 1.1.0. There are several feature PRs marked in the milestone and elsewhere that in principle might use help with reviewing/status triaging --- I've been working toward getting low-hanging ones in, but there are several which I'm not sure about. Nothing seems blocking, though. If there are blocking problems that need to be addressed currently in master before 1.1.0, please remind about them (I'm not aware of such issues though). Best, Pauli -- From mikofski at berkeley.edu Tue Apr 10 16:12:37 2018 From: mikofski at berkeley.edu (Mark Alexander Mikofski) Date: Tue, 10 Apr 2018 20:12:37 +0000 Subject: [SciPy-Dev] SciPy 1.1.0 release schedule In-Reply-To: References: Message-ID: Can PR #8357 please be considered for v1.1.0? It has been reviewed by several maintainers and contributors, it closes two issues, and has support of several users. Also it is tested, benchmarked, and documented. I believe I have addressed all comments from all reviewers, but please let me know if anyone has any more comments. Thanks so much! On Tue, Apr 10, 2018, 1:05 PM Pauli Virtanen wrote: > Hi, > > 2018-03-25 17:45 -0700, Ralf Gommers wrote: > > It's 5 months after the 1.0 release, so it's time to start planning > > for > > 1.1. There's still quite a few open issues and PRs marked for 1.1.0 > > (see > > https://github.com/scipy/scipy/milestone/34), but not many that seem > > blocking or really difficult to resolve. So I'd like to propose the > > following schedule: > > > > April 11: branch 1.1.x > > April 13: rc1 > > April 27: rc2 (if needed) > > May 4: final release > > Just a friendly heads up/reminder: staying roughly with the schedule > proposed earlier, I'll branch 1.1.x on this Thursday, and work towards > getting 1.1.0rc1 out during the weekend. > > If there are features almost ready to be merged, there's a few days > before the branch point. After that, it's mainly bugfixes that'll get > to 1.1.0. > > There are several feature PRs marked in the milestone and elsewhere > that in principle might use help with reviewing/status triaging --- > I've been working toward getting low-hanging ones in, but there are > several which I'm not sure about. Nothing seems blocking, though. > > If there are blocking problems that need to be addressed currently in > master before 1.1.0, please remind about them (I'm not aware of such > issues though). > > Best, > Pauli > > -- > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Apr 10 15:02:21 2018 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 10 Apr 2018 21:02:21 +0200 Subject: [SciPy-Dev] SciPy 1.1.0 release schedule In-Reply-To: References: Message-ID: <25564720afdc9f408862f670492b03d03e094696.camel@iki.fi> Hi, su, 2018-03-25 kello 17:45 -0700, Ralf Gommers kirjoitti: > It's 5 months after the 1.0 release, so it's time to start planning > for > 1.1. There's still quite a few open issues and PRs marked for 1.1.0 > (see > https://github.com/scipy/scipy/milestone/34), but not many that seem > blocking or really difficult to resolve. So I'd like to propose the > following schedule: > > April 11: branch 1.1.x > April 13: rc1 > April 27: rc2 (if needed) > May 4: final release Just a friendly heads up/reminder: staying roughly with the schedule proposed earlier, I'll branch 1.1.x on this Thursday, and work towards getting 1.1.0rc1 out during the weekend. If there are features almost ready to be merged, there's a few days before the branch point. After that, it's mainly bugfixes that'll get to 1.1.0. There are several feature PRs marked in the milestone and elsewhere that in principle might use help with reviewing/status triaging --- I've been working toward getting low-hanging ones in, but there are several which I'm not sure about. Nothing seems blocking, though. If there are blocking problems that need to be addressed currently in master before 1.1.0, please remind about them (I'm not aware of such issues though). Best, Pauli From pav at iki.fi Tue Apr 10 15:04:13 2018 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 10 Apr 2018 21:04:13 +0200 Subject: [SciPy-Dev] SciPy 1.1.0 release schedule In-Reply-To: References: Message-ID: <91b19d6380a8b8a68fdc8c2cda8d599a27b1d7f8.camel@iki.fi> Hi, su, 2018-03-25 kello 17:45 -0700, Ralf Gommers kirjoitti: > It's 5 months after the 1.0 release, so it's time to start planning > for > 1.1. There's still quite a few open issues and PRs marked for 1.1.0 > (see > https://github.com/scipy/scipy/milestone/34), but not many that seem > blocking or really difficult to resolve. So I'd like to propose the > following schedule: > > April 11: branch 1.1.x > April 13: rc1 > April 27: rc2 (if needed) > May 4: final release Just a friendly heads up/reminder: staying roughly with the schedule proposed earlier, I'll branch 1.1.x on this Thursday, and work towards getting 1.1.0rc1 out during the weekend. If there are features almost ready to be merged, there's a few days before the branch point. After that, it's mainly bugfixes that'll get to 1.1.0. There are several feature PRs marked in the milestone and elsewhere that in principle might use help with reviewing/status triaging --- I've been working toward getting low-hanging ones in, but there are several which I'm not sure about. Nothing seems blocking, though. If there are blocking problems that need to be addressed currently in master before 1.1.0, please remind about them (I'm not aware of such issues though). Best, Pauli From blairuk at gmail.com Tue Apr 10 17:15:33 2018 From: blairuk at gmail.com (Blair Azzopardi) Date: Tue, 10 Apr 2018 22:15:33 +0100 Subject: [SciPy-Dev] SciPy 1.1.0 release schedule In-Reply-To: <91b19d6380a8b8a68fdc8c2cda8d599a27b1d7f8.camel@iki.fi> References: <91b19d6380a8b8a68fdc8c2cda8d599a27b1d7f8.camel@iki.fi> Message-ID: On 10 April 2018 at 20:04, Pauli Virtanen wrote: > Hi, > > su, 2018-03-25 kello 17:45 -0700, Ralf Gommers kirjoitti: > > It's 5 months after the 1.0 release, so it's time to start planning > > for > > 1.1. There's still quite a few open issues and PRs marked for 1.1.0 > > (see > > https://github.com/scipy/scipy/milestone/34), but not many that seem > > blocking or really difficult to resolve. So I'd like to propose the > > following schedule: > > > > April 11: branch 1.1.x > > April 13: rc1 > > April 27: rc2 (if needed) > > May 4: final release > > Just a friendly heads up/reminder: staying roughly with the schedule > proposed earlier, I'll branch 1.1.x on this Thursday, and work towards > getting 1.1.0rc1 out during the weekend. > > If there are features almost ready to be merged, there's a few days > before the branch point. After that, it's mainly bugfixes that'll get > to 1.1.0. > > There are several feature PRs marked in the milestone and elsewhere > that in principle might use help with reviewing/status triaging --- > I've been working toward getting low-hanging ones in, but there are > several which I'm not sure about. Nothing seems blocking, though. > > If there are blocking problems that need to be addressed currently in > master before 1.1.0, please remind about them (I'm not aware of such > issues though). > > Best, > Pauli > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > Please can you consider merging PR #7374. It's already in the 1.1 milestone and I'd say it's pretty mature. The code offers 3 different ways of calculating stable pdf/cdfs all of which match samples provided by Nolan's public domain executable. It also offers parameter estimation using quantiles. Any further optimisations can be added in a later release. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sylvain.Gubian at pmi.com Tue Apr 10 17:37:35 2018 From: Sylvain.Gubian at pmi.com (Gubian, Sylvain) Date: Tue, 10 Apr 2018 21:37:35 +0000 Subject: [SciPy-Dev] SciPy 1.1.0 release schedule Message-ID: <1dc8f2d9444f480cbccc9882c6eaf43f@PMICHLAUEXM264.PMINTL.NET> Dear All, #PR8203 is planned for 1.1.0 release. Few minors comments to address but very close to be merged according reviewers. I'll do my best to address them ASAP. Sylvain. -----Original Message----- From: SciPy-Dev [mailto:scipy-dev-bounces+sylvain.gubian=pmi.com at python.org] On Behalf Of scipy-dev-request at python.org Sent: mardi 10 avril 2018 23:16 To: scipy-dev at python.org Subject: SciPy-Dev Digest, Vol 174, Issue 10 Send SciPy-Dev mailing list submissions to scipy-dev at python.org To subscribe or unsubscribe via the World Wide Web, visit https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fscipy-dev&data=02%7C01%7Csylvain.gubian%40pmi.com%7C38a531959d654daaa84908d59f28752b%7C8b86a65e3c3a44068ac319a6b5cc52bc%7C0%7C0%7C636589918472558141&sdata=qMVAKjIj6VyXGq%2BYbluHPuQnIBX8NNuExIZnJEcgfcU%3D&reserved=0 or, via email, send a message with subject or body 'help' to scipy-dev-request at python.org You can reach the person managing the list at scipy-dev-owner at python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of SciPy-Dev digest..." Today's Topics: 1. SciPy 1.1.0 release schedule (Pauli Virtanen) 2. Re: SciPy 1.1.0 release schedule (Mark Alexander Mikofski) 3. Re: SciPy 1.1.0 release schedule (Pauli Virtanen) 4. Re: SciPy 1.1.0 release schedule (Pauli Virtanen) 5. Re: SciPy 1.1.0 release schedule (Blair Azzopardi) ---------------------------------------------------------------------- Message: 1 Date: Tue, 10 Apr 2018 21:39:02 +0200 From: Pauli Virtanen To: scipy-dev Subject: [SciPy-Dev] SciPy 1.1.0 release schedule Message-ID: Content-Type: text/plain; charset="UTF-8" Hi, 2018-03-25 17:45 -0700, Ralf Gommers wrote: > It's 5 months after the 1.0 release, so it's time to start planning > for 1.1. There's still quite a few open issues and PRs marked for > 1.1.0 (see > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit > hub.com%2Fscipy%2Fscipy%2Fmilestone%2F34&data=02%7C01%7Csylvain.gubian > %40pmi.com%7C38a531959d654daaa84908d59f28752b%7C8b86a65e3c3a44068ac319a6b5cc52bc%7C0%7C0%7C636589918472558141&sdata=gGcS%2BZH%2B%2FhauPHDbB4bHx%2B9K0ZtMMeBUgvEe7vO63wA%3D&reserved=0), but not many that seem blocking or really difficult to resolve. So I'd like to propose the following schedule: > > April 11: branch 1.1.x > April 13: rc1 > April 27: rc2 (if needed) > May 4: final release Just a friendly heads up/reminder: staying roughly with the schedule proposed earlier, I'll branch 1.1.x on this Thursday, and work towards getting 1.1.0rc1 out during the weekend. If there are features almost ready to be merged, there's a few days before the branch point. After that, it's mainly bugfixes that'll get to 1.1.0. There are several feature PRs marked in the milestone and elsewhere that in principle might use help with reviewing/status triaging --- I've been working toward getting low-hanging ones in, but there are several which I'm not sure about. Nothing seems blocking, though. If there are blocking problems that need to be addressed currently in master before 1.1.0, please remind about them (I'm not aware of such issues though). Best, Pauli -- ------------------------------ Message: 2 Date: Tue, 10 Apr 2018 20:12:37 +0000 From: Mark Alexander Mikofski To: SciPy Developers List Subject: Re: [SciPy-Dev] SciPy 1.1.0 release schedule Message-ID: Content-Type: text/plain; charset="utf-8" Can PR #8357 please be considered for v1.1.0? It has been reviewed by several maintainers and contributors, it closes two issues, and has support of several users. Also it is tested, benchmarked, and documented. I believe I have addressed all comments from all reviewers, but please let me know if anyone has any more comments. Thanks so much! On Tue, Apr 10, 2018, 1:05 PM Pauli Virtanen wrote: > Hi, > > 2018-03-25 17:45 -0700, Ralf Gommers wrote: > > It's 5 months after the 1.0 release, so it's time to start planning > > for 1.1. There's still quite a few open issues and PRs marked for > > 1.1.0 (see > > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fg > > ithub.com%2Fscipy%2Fscipy%2Fmilestone%2F34&data=02%7C01%7Csylvain.gu > > bian%40pmi.com%7C38a531959d654daaa84908d59f28752b%7C8b86a65e3c3a44068ac319a6b5cc52bc%7C0%7C0%7C636589918472558141&sdata=gGcS%2BZH%2B%2FhauPHDbB4bHx%2B9K0ZtMMeBUgvEe7vO63wA%3D&reserved=0), but not many that seem blocking or really difficult to resolve. So I'd like to propose the following schedule: > > > > April 11: branch 1.1.x > > April 13: rc1 > > April 27: rc2 (if needed) > > May 4: final release > > Just a friendly heads up/reminder: staying roughly with the schedule > proposed earlier, I'll branch 1.1.x on this Thursday, and work towards > getting 1.1.0rc1 out during the weekend. > > If there are features almost ready to be merged, there's a few days > before the branch point. After that, it's mainly bugfixes that'll get > to 1.1.0. > > There are several feature PRs marked in the milestone and elsewhere > that in principle might use help with reviewing/status triaging --- > I've been working toward getting low-hanging ones in, but there are > several which I'm not sure about. Nothing seems blocking, though. > > If there are blocking problems that need to be addressed currently in > master before 1.1.0, please remind about them (I'm not aware of such > issues though). > > Best, > Pauli > > -- > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmai > l.python.org%2Fmailman%2Flistinfo%2Fscipy-dev&data=02%7C01%7Csylvain.g > ubian%40pmi.com%7C38a531959d654daaa84908d59f28752b%7C8b86a65e3c3a44068 > ac319a6b5cc52bc%7C0%7C0%7C636589918472558141&sdata=qMVAKjIj6VyXGq%2BYb > luHPuQnIBX8NNuExIZnJEcgfcU%3D&reserved=0 > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Tue, 10 Apr 2018 21:02:21 +0200 From: Pauli Virtanen To: scipy-dev at python.org Subject: Re: [SciPy-Dev] SciPy 1.1.0 release schedule Message-ID: <25564720afdc9f408862f670492b03d03e094696.camel at iki.fi> Content-Type: text/plain; charset="UTF-8" Hi, su, 2018-03-25 kello 17:45 -0700, Ralf Gommers kirjoitti: > It's 5 months after the 1.0 release, so it's time to start planning > for 1.1. There's still quite a few open issues and PRs marked for > 1.1.0 (see > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit > hub.com%2Fscipy%2Fscipy%2Fmilestone%2F34&data=02%7C01%7Csylvain.gubian > %40pmi.com%7C38a531959d654daaa84908d59f28752b%7C8b86a65e3c3a44068ac319a6b5cc52bc%7C0%7C0%7C636589918472558141&sdata=gGcS%2BZH%2B%2FhauPHDbB4bHx%2B9K0ZtMMeBUgvEe7vO63wA%3D&reserved=0), but not many that seem blocking or really difficult to resolve. So I'd like to propose the following schedule: > > April 11: branch 1.1.x > April 13: rc1 > April 27: rc2 (if needed) > May 4: final release Just a friendly heads up/reminder: staying roughly with the schedule proposed earlier, I'll branch 1.1.x on this Thursday, and work towards getting 1.1.0rc1 out during the weekend. If there are features almost ready to be merged, there's a few days before the branch point. After that, it's mainly bugfixes that'll get to 1.1.0. There are several feature PRs marked in the milestone and elsewhere that in principle might use help with reviewing/status triaging --- I've been working toward getting low-hanging ones in, but there are several which I'm not sure about. Nothing seems blocking, though. If there are blocking problems that need to be addressed currently in master before 1.1.0, please remind about them (I'm not aware of such issues though). Best, Pauli ------------------------------ Message: 4 Date: Tue, 10 Apr 2018 21:04:13 +0200 From: Pauli Virtanen To: scipy-dev at python.org Subject: Re: [SciPy-Dev] SciPy 1.1.0 release schedule Message-ID: <91b19d6380a8b8a68fdc8c2cda8d599a27b1d7f8.camel at iki.fi> Content-Type: text/plain; charset="UTF-8" Hi, su, 2018-03-25 kello 17:45 -0700, Ralf Gommers kirjoitti: > It's 5 months after the 1.0 release, so it's time to start planning > for 1.1. There's still quite a few open issues and PRs marked for > 1.1.0 (see > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit > hub.com%2Fscipy%2Fscipy%2Fmilestone%2F34&data=02%7C01%7Csylvain.gubian > %40pmi.com%7C38a531959d654daaa84908d59f28752b%7C8b86a65e3c3a44068ac319a6b5cc52bc%7C0%7C0%7C636589918472558141&sdata=gGcS%2BZH%2B%2FhauPHDbB4bHx%2B9K0ZtMMeBUgvEe7vO63wA%3D&reserved=0), but not many that seem blocking or really difficult to resolve. So I'd like to propose the following schedule: > > April 11: branch 1.1.x > April 13: rc1 > April 27: rc2 (if needed) > May 4: final release Just a friendly heads up/reminder: staying roughly with the schedule proposed earlier, I'll branch 1.1.x on this Thursday, and work towards getting 1.1.0rc1 out during the weekend. If there are features almost ready to be merged, there's a few days before the branch point. After that, it's mainly bugfixes that'll get to 1.1.0. There are several feature PRs marked in the milestone and elsewhere that in principle might use help with reviewing/status triaging --- I've been working toward getting low-hanging ones in, but there are several which I'm not sure about. Nothing seems blocking, though. If there are blocking problems that need to be addressed currently in master before 1.1.0, please remind about them (I'm not aware of such issues though). Best, Pauli ------------------------------ Message: 5 Date: Tue, 10 Apr 2018 22:15:33 +0100 From: Blair Azzopardi To: SciPy Developers List Subject: Re: [SciPy-Dev] SciPy 1.1.0 release schedule Message-ID: Content-Type: text/plain; charset="utf-8" On 10 April 2018 at 20:04, Pauli Virtanen wrote: > Hi, > > su, 2018-03-25 kello 17:45 -0700, Ralf Gommers kirjoitti: > > It's 5 months after the 1.0 release, so it's time to start planning > > for 1.1. There's still quite a few open issues and PRs marked for > > 1.1.0 (see > > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fg > > ithub.com%2Fscipy%2Fscipy%2Fmilestone%2F34&data=02%7C01%7Csylvain.gu > > bian%40pmi.com%7C38a531959d654daaa84908d59f28752b%7C8b86a65e3c3a44068ac319a6b5cc52bc%7C0%7C0%7C636589918472558141&sdata=gGcS%2BZH%2B%2FhauPHDbB4bHx%2B9K0ZtMMeBUgvEe7vO63wA%3D&reserved=0), but not many that seem blocking or really difficult to resolve. So I'd like to propose the following schedule: > > > > April 11: branch 1.1.x > > April 13: rc1 > > April 27: rc2 (if needed) > > May 4: final release > > Just a friendly heads up/reminder: staying roughly with the schedule > proposed earlier, I'll branch 1.1.x on this Thursday, and work towards > getting 1.1.0rc1 out during the weekend. > > If there are features almost ready to be merged, there's a few days > before the branch point. After that, it's mainly bugfixes that'll get > to 1.1.0. > > There are several feature PRs marked in the milestone and elsewhere > that in principle might use help with reviewing/status triaging --- > I've been working toward getting low-hanging ones in, but there are > several which I'm not sure about. Nothing seems blocking, though. > > If there are blocking problems that need to be addressed currently in > master before 1.1.0, please remind about them (I'm not aware of such > issues though). > > Best, > Pauli > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmai > l.python.org%2Fmailman%2Flistinfo%2Fscipy-dev&data=02%7C01%7Csylvain.g > ubian%40pmi.com%7C38a531959d654daaa84908d59f28752b%7C8b86a65e3c3a44068 > ac319a6b5cc52bc%7C0%7C0%7C636589918472558141&sdata=qMVAKjIj6VyXGq%2BYb > luHPuQnIBX8NNuExIZnJEcgfcU%3D&reserved=0 > Please can you consider merging PR #7374. It's already in the 1.1 milestone and I'd say it's pretty mature. The code offers 3 different ways of calculating stable pdf/cdfs all of which match samples provided by Nolan's public domain executable. It also offers parameter estimation using quantiles. Any further optimisations can be added in a later release. -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Subject: Digest Footer _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fscipy-dev&data=02%7C01%7Csylvain.gubian%40pmi.com%7C38a531959d654daaa84908d59f28752b%7C8b86a65e3c3a44068ac319a6b5cc52bc%7C0%7C0%7C636589918472558141&sdata=qMVAKjIj6VyXGq%2BYbluHPuQnIBX8NNuExIZnJEcgfcU%3D&reserved=0 ------------------------------ End of SciPy-Dev Digest, Vol 174, Issue 10 ****************************************** From warren.weckesser at gmail.com Wed Apr 11 02:12:39 2018 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Wed, 11 Apr 2018 02:12:39 -0400 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> Message-ID: On Mon, Apr 2, 2018 at 2:50 PM, Warren Weckesser wrote: > > > On Fri, Mar 30, 2018 at 8:17 PM, Ralf Gommers > wrote: > >> >> >> On Fri, Mar 30, 2018 at 12:03 PM, Eric Larson >> wrote: >> >>> Top-level module for them alone sounds overkill, and I'm not sure if >>>> discoverability alone is enough. >>>> >>> >>> Fine by me. And if we follow the idea that these should be added >>> sparingly, we can maintain discoverability without it growing out of >>> hand by populating the See Also sections of each function. >>> >> >> I agree with this, the 2 images and 1 ECG signal (to be added) that we >> have doesn't justify a top-level module. We don't want to grow more than >> the absolute minimum of datasets. The package is already very large, which >> is problematic in certain cases. E.g. numpy + scipy still fits in the AWS >> Lambda limit of 50 MB, but there's not much margin. >> >> > > Note: this is a reply to the thread, and not specifically to Ralf's > comments (but those are included). > > After reading all the replies, the first question that comes to mind is: > should SciPy have *any* datasets? > > I think this question has already been answered: we have had functions > that return images in scipy.misc for a long time, and I don't recall anyone > ever suggesting that these be removed. (Well, there was lena(), but I > don't think anyone had a problem with adding a replacement image.) And the > pull request for the ECG dataset has been merged (added to scipy.misc), so > there is current support among the developers for providing datasets. > > So the remaining questions are: > (1) Where do the datasets reside? > (2) What are the criteria for adding a new datasets? > > Here's my 2?: > > (1) Where do the datasets reside? > > My preference is to keep all the datasets in the top-level module > scipy.datasets. Robert preferred this module for discoverability, and I > agree. By having all the datasets in one place, anyone can easily see what > is available. Teachers and others developing educational material know > where to find source material for examples. Developers, too, can easily > look for examples to use in our docstrings or tutorials. (By the way, > adding examples to the docstrings of all functions is an ongoing effort: > https://github.com/scipy/scipy/issues/7168.) > > Also, there are many well-known datasets that could be used as examples > for multiple scipy packages. For a concrete example, a dataset that I > could see adding to scipy is the Hald cement dataset. SciPy should > eventually have an implementation of the PCA decomposition, and it could > conceivably live in scipy.linalg. It would be reasonable to use the Hald > data in the docstrings of the new PCA function(s) (cf. > https://www.mathworks.com/help/stats/pca.html). At the same time, the > Hald data could enrich the docstrings of some functions in scipy.stats. > > Similarly, Fisher's iris dataset provides a well-known example that could > be used in docstrings in both scipy.cluster and scipy.stats. > > > (2) What are the criteria for adding a new datasets? > > So far, the only compelling reason I can see to even have datasets is to > have interesting examples in the docstrings (or at least in our > tutorials). For example, the docstring for scipy.ndimage.gaussian_filter > and several other transformations in ndimage use the image returned by > scipy.misc.ascent(): > > https://docs.scipy.org/doc/scipy/reference/generated/ > scipy.ndimage.gaussian_filter.html > > I could see the benefit of having well-known datasets such as Fisher's > iris data, the Hald cement data, and some version of a sunspot activity > time series, to be used in the docstrings in scipy.stats, scipy.signal, > scipy.cluster, scipy.linalg, and elsewhere. > > St?fan expressed regret about including datasets in sciki-image. The main > issue seems to be "bloat". Scikit-image is an image processing library, so > the datasets used there are likely all images, and there is a minimum size > for a sample image to be useful as an example. For scipy, we already have > two images, and I don't think we'll need more. The newly added ECG dataset > is 116K (which is less than the existing image datasets: "ascent.dat" is > 515K and "face.dat" is 1.5M). The potential datasets that I mentioned > above (Hald, iris, sunspots) are all very small. If we are conservative > about what we include, and focus on datasets chosen specifically to > demonstrate scipy functionality, we should be able to avoid dataset bloat. > > This leads to my proposal for the criteria for adding a dataset: > > (a) Not too big. The size of a dataset should not exceed $MAX (but I > don't have a good suggestion for what $MAX should be at the moment). > (b) The dataset should be well-known, where "well-known" means that the > dataset is one that is already widely used as an example and many people > will know it by name (e.g. the iris dataset), or the dataset is a sample of > a common signal type or format (e.g an ECG signal, or an image such as > misc.ascent). > (c) We actually *use* the dataset in one of *our* docstrings or > tutorials. I don't think our datasets package should become a repository > of interesting scientific data with no connection to the scipy code. Its > purpose should be to enrich our documentation. (Note that by this > criterion, the recently added ECG signal would not qualify!) > > To summarize: I'm in favor scipy.datasets, a conservatively curated > subpackage containing well-known datasets. > > In case we can reach agreement on the addition of the 'datasets' subpackage, I've create a pull request that implements it: https://github.com/scipy/scipy/pull/8707 The changes are setup up to be included in 1.1. That would be nice; otherwise we have to deprecate the new misc.electrocardiogram after just one release, when the data files are moved to their new home in 1.2. Warren > > Warren > > > P.S. I should add that I'm not in favor putting code in scipy that fetches > data from the web. That type of data retrieval could be useful, but it > seems more appropriate for a package that is independent of scipy. > > > > > >> Ralf >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Apr 11 04:27:32 2018 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 11 Apr 2018 10:27:32 +0200 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> Message-ID: <1523435252.21592.26.camel@iki.fi> ke, 2018-04-11 kello 02:12 -0400, Warren Weckesser kirjoitti: [clip] > In case we can reach agreement on the addition of the 'datasets' > subpackage, I've create a pull request that implements it: > https://github.com/scipy/scipy/pull/8707 > The changes are setup up to be included in 1.1. That would be nice; > otherwise we have to deprecate the new misc.electrocardiogram after > just > one release, when the data files are moved to their new home in 1.2. It's not going to be in 1.1, as we don't have agreement to include it yet, and it does not appear to be urgent. That electrocardiogram would need to move is probably not a big issue in practice given that it's a new addition. Pauli From pav at iki.fi Sun Apr 15 14:24:11 2018 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 15 Apr 2018 20:24:11 +0200 Subject: [SciPy-Dev] SciPy 1.1.0rc1 release candidate Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi all, SciPy 1.1.0rc1 release candidate release is now available. Please try it out also against your own codes and report issues you find. Sources and binary wheels can be found at https://pypi.python.org/pypi/scipy and . To install with pip: pip install --pre --upgrade scipy Thanks to everyone who contributed to this release! Pauli =============================== SciPy 1.1.0 Release Notes (rc1) =============================== .. note:: Scipy 1.1.0 is not released yet! .. contents:: SciPy 1.1.0 is the culmination of 7 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Before upgrading, we recommend that users check that their own code does not use deprecated SciPy functionality (to do so, run your code with ``python -Wd`` and check for ``DeprecationWarning`` s). Our development attention will now shift to bug-fix releases on the 1.1.x branch, and on adding new features on the master branch. This release requires Python 2.7 or 3.4+ and NumPy 1.8.2 or greater. This release has improved but not necessarily 100% compatibility with the `PyPy `__ Python implementation. For running on PyPy, PyPy 6.0+ and Numpy 1.15.0+ are required. New features ============ `scipy.integrate` improvements - - ------------------------------ The argument ``tfirst`` has been added to the function `scipy.integrate.odeint`. This allows odeint to use the same user functions as `scipy.integrate.solve_ivp` and `scipy.integrate.ode` without the need for wrapping them in a function that swaps the first two arguments. Error messages from ``quad()`` are now clearer. `scipy.linalg` improvements - - --------------------------- The function `scipy.linalg.ldl` has been added for factorization of indefinite symmetric/hermitian matrices into triangular and block diagonal matrices. Python wrappers for LAPACK ``sygst``, ``hegst`` added in `scipy.linalg.lapack`. Added `scipy.linalg.null_space`, `scipy.linalg.cdf2rdf`, `scipy.linalg.rsf2csf`. `scipy.misc` improvements - - ------------------------- An electrocardiogram has been added as an example dataset for a one-dimensional signal. It can be accessed through `scipy.misc.electrocardiogram`. `scipy.ndimage` improvements - - ---------------------------- The routines `scipy.ndimage.binary_opening`, and `scipy.ndimage.binary_closing` now support masks and different border values. `scipy.optimize` improvements - - ----------------------------- The method ``trust-constr`` has been added to `scipy.optimize.minimize`. The method switches between two implementations depending on the problem definition. For equality constrained problems it is an implementation of a trust-region sequential quadratic programming solver and, when inequality constraints are imposed, it switches to a trust-region interior point method. Both methods are appropriate for large scale problems. Quasi-Newton options BFGS and SR1 were implemented and can be used to approximate second order derivatives for this new method. Also, finite-differences can be used to approximate either first-order or second-order derivatives. Random-to-Best/1/bin and Random-to-Best/1/exp mutation strategies were added to `scipy.optimize.differential_evolution` as ``randtobest1bin`` and ``randtobest1exp``, respectively. Note: These names were already in use but implemented a different mutation strategy. See `Backwards incompatible changes <#backwards-incompatible-changes>`__, below. The ``init`` keyword for the `scipy.optimize.differential_evolution` function can now accept an array. This array allows the user to specify the entire population. Add an ``adaptive`` option to Nelder-Mead to use step parameters adapted to the dimensionality of the problem. Minor improvements in `scipy.optimize.basinhopping`. `scipy.signal` improvements - - --------------------------- Three new functions for peak finding in one-dimensional arrays were added. `scipy.signal.find_peaks` searches for peaks (local maxima) based on simple value comparison of neighbouring samples and returns those peaks whose properties match optionally specified conditions for their height, prominence, width, threshold and distance to each other. `scipy.signal.peak_prominences` and `scipy.signal.peak_widths` can directly calculate the prominences or widths of known peaks. Added ZPK versions of frequency transformations: `scipy.signal.bilinear_zpk`, `scipy.signal.lp2bp_zpk`, `scipy.signal.lp2bs_zpk`, `scipy.signal.lp2hp_zpk`, `scipy.signal.lp2lp_zpk`. Added `scipy.signal.windows.dpss`, `scipy.signal.windows.general_cosine` and `scipy.signal.windows.general_hamming`. `scipy.sparse` improvements - - --------------------------- An in-place ``resize`` method has been added to all sparse matrix formats, which was only available for `scipy.sparse.dok_matrix` in previous releases. `scipy.special` improvements - - ---------------------------- Added Owen?s T function as `scipy.special.owens_t`. Accuracy improvements in ``chndtr``, ``digamma``, ``gammaincinv``, ``lambertw``, ``zetac``. `scipy.stats` improvements - - -------------------------- The Moyal distribution has been added as `scipy.stats.moyal`. Added the normal inverse Gaussian distribution as `scipy.stats.norminvgauss`. Deprecated features =================== The iterative linear equation solvers in `scipy.sparse.linalg` had a sub-optimal way of how absolute tolerance is considered. The default behavior will be changed in a future Scipy release to a more standard and less surprising one. To silence deprecation warnings, set the ``atol=`` parameter explicitly. `scipy.signal.windows.slepian` is deprecated, replaced by `scipy.signal.windows.dpss`. The window functions in `scipy.signal` are now available in `scipy.signal.windows`. They will remain also available in the old location in the `scipy.signal` namespace in future Scipy versions. However, importing them from `scipy.signal.windows` is preferred, and new window functions will be added only there. Indexing sparse matrices with floating-point numbers instead of integers is deprecated. The function `scipy.stats.itemfreq` is deprecated. Backwards incompatible changes ============================== Previously, `scipy.linalg.orth` used a singular value cutoff value appropriate for double precision numbers also for single-precision input. The cutoff value is now tunable, and the default has been changed to depend on the input data precision. In previous versions of Scipy, the ``randtobest1bin`` and ``randtobest1exp`` mutation strategies in `scipy.optimize.differential_evolution` were actually implemented using the Current-to-Best/1/bin and Current-to-Best/1/exp strategies, respectively. These strategies were renamed to ``currenttobest1bin`` and ``currenttobest1exp`` and the implementations of ``randtobest1bin`` and ``randtobest1exp`` strategies were corrected. Functions in the ndimage module now always return their output array. Before this most functions only returned the output array if it had been allocated by the function, and would return ``None`` if it had been provided by the user. Distance metrics in `scipy.spatial.distance` now require non-negative weights. `scipy.special.loggamma` returns now real-valued result when the input is real-valued. Other changes ============= When building on Linux with GNU compilers, the ``.so`` Python extension files now hide all symbols except those required by Python, which can avoid problems when embedding the Python interpreter. Authors ======= * Saurabh Agarwal + * Diogo Aguiam + * Joseph Albert + * Gerrit Ansmann + * Astrofysicus + * Jean-Fran?ois B + * Vahan Babayan + * Alessandro Pietro Bardelli * Christoph Baumgarten + * Felix Berkenkamp * Lilian Besson + * Aditya Bharti + * Matthew Brett * Evgeni Burovski * CJ Carey * Martin ?. Christensen + * Robert Cimrman * Vicky Close + * Peter Cock + * Philip DeBoer * Jaime Fernandez del Rio * Dieter Werthm?ller + * Tom Donoghue + * Matt Dzugan + * Lars G + * Jacques Gaudin + * Andriy Gelman + * Sean Gillies + * Dezmond Goff * Christoph Gohlke * Ralf Gommers * Uri Goren + * Deepak Kumar Gouda + * Douglas Lessa Graciosa + * Matt Haberland * David Hagen * Charles Harris * Jordan Heemskerk + * Danny Hermes + * Stephan Hoyer + * Theodore Hu + * Jean-Fran?ois B. + * Mads Jensen + * Jon Haitz Legarreta Gorro?o + * Ben Jude + * Noel Kippers + * Julius Bier Kirkegaard + * Maria Knorps + * Mikkel Kristensen + * Eric Larson * Kasper Primdal Lauritzen + * Denis Laxalde * KangWon Lee + * Jan Lehky + * Jackie Leng + * P.L. Lim + * Nikolay Mayorov * Mihai Capot? + * Max Mikhaylov + * Mark Mikofski + * Jarrod Millman * Raden Muhammad + * Paul Nation * Andrew Nelson * Nico Schl?mer * Joel Nothman * Kyle Oman + * Egor Panfilov + * Nick Papior * Anubhav Patel + * Oleksandr Pavlyk * Ilhan Polat * Robert Pollak + * Anant Prakash + * Aman Pratik * Sean Quinn + * Giftlin Rajaiah + * Tyler Reddy * Joscha Reimer * Antonio H Ribeiro + * Antonio Horta Ribeiro * Benjamin Rose + * Fabian Rost * Divakar Roy + * Scott Sievert * Leo Singer * Sourav Singh * Martino Sorbaro + * Eric Stansifer + * Martin Thoma * Phil Tooley + * Piotr Uchwat + * Paul van Mulbregt * Pauli Virtanen * Stefan van der Walt * Warren Weckesser * Florian Weimer + * Eric Wieser * Josh Wilson * Ted Ying + * Evgeny Zhurko * Z? Vin?cius * @awakenting + * @endolith * @FormerPhysicist + * @gaulinmp + * @hugovk * @ksemb + * @kshitij12345 + * @luzpaz + * @NKrvavica + * @rafalalgo + * @samyak0210 + * @soluwalana + * @sudheerachary + * @Tokixix + * @tttthomasssss + * @vkk800 + * @xoviat * @ziejcow + A total of 122 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.1.0 - - ----------------------- * `#979 `__: Allow Hermitian matrices in lobpcg (Trac #452) * `#2694 `__: Solution of iterative solvers can be less accurate than tolerance... * `#3164 `__: RectBivariateSpline usage inconsistent with other interpolation... * `#4161 `__: Missing ITMAX optional argument in scipy.optimize.nnls * `#4354 `__: signal.slepian should use definition of digital window * `#4866 `__: Shouldn't scipy.linalg.sqrtm raise an error if matrix is singular? * `#4953 `__: The dirichlet distribution unnecessarily requires strictly positive... * `#5336 `__: sqrtm on a diagonal matrix can warn "Matrix is singular and may... * `#5922 `__: Suboptimal convergence of Halley's method? * `#6036 `__: Incorrect edge case in scipy.stats.triang.pdf * `#6202 `__: Enhancement: Add LDLt factorization to scipy * `#6589 `__: sparse.random with custom rvs callable does pass on arg to subclass * `#6654 `__: Spearman's rank correlation coefficient slow with nan values... * `#6794 `__: Remove NumarrayType struct with numarray type names from ndimage * `#7136 `__: The dirichlet distribution unnecessarily rejects probabilities... * `#7169 `__: Will it be possible to add LDL' factorization for Hermitian indefinite... * `#7291 `__: fsolve docs should say it doesn't handle over- or under-determined... * `#7453 `__: binary_opening/binary_closing missing arguments * `#7500 `__: linalg.solve test failure on OS X with Accelerate * `#7555 `__: Integratig a function with singularities using the quad routine * `#7624 `__: allow setting both absolute and relative tolerance of sparse... * `#7724 `__: odeint documentation refers to t0 instead of t * `#7746 `__: False CDF values for skew normal distribution * `#7750 `__: mstats.winsorize documentation needs clarification * `#7787 `__: Documentation error in spherical Bessel, Neumann, modified spherical... * `#7836 `__: Scipy mmwrite incorrectly writes the zeros for skew-symmetric,... * `#7839 `__: sqrtm is unable to compute square root of zero matrix * `#7847 `__: solve is very slow since #6775 * `#7888 `__: Scipy 1.0.0b1 prints spurious DVODE/ZVODE/lsoda messages * `#7909 `__: bessel kv function in 0 is nan * `#7915 `__: LinearOperator's __init__ runs two times when instantiating the... * `#7958 `__: integrate.quad could use better error messages when given bad... * `#7968 `__: integrate.quad handles decreasing limits (b`__: ENH: matching return dtype for loggamma/gammaln * `#7991 `__: `lfilter` segfaults for integer inputs * `#8076 `__: "make dist" for the docs doesn't complete cleanly * `#8080 `__: Use JSON in `special/_generate_pyx.py`? * `#8127 `__: scipy.special.psi(x) very slow for some values of x * `#8145 `__: BUG: ndimage geometric_transform and zoom using deprecated NumPy... * `#8158 `__: BUG: romb print output requires correction * `#8181 `__: loadmat() raises TypeError instead of FileNotFound when reading... * `#8228 `__: bug for log1p on csr_matrix * `#8235 `__: scipy.stats multinomial pmf return nan * `#8271 `__: scipy.io.mmwrite raises type error for uint16 * `#8288 `__: Should tests be written for scipy.sparse.linalg.isolve.minres... * `#8298 `__: Broken links on scipy API web page * `#8329 `__: `_gels` fails for fat A matrix * `#8346 `__: Avoidable overflow in scipy.special.binom(n, k) * `#8371 `__: BUG: special: zetac(x) returns 0 for x < -30.8148 * `#8382 `__: collections.OrderedDict in test_mio.py * `#8492 `__: Missing documentation for `brute_force` parameter in scipy.ndimage.morphology * `#8532 `__: leastsq needlessly appends extra dimension for scalar problems * `#8544 `__: [feature request] Convert complex diagonal form to real block... * `#8561 `__: [Bug?] Example of Bland's Rule for optimize.linprog (simplex)... * `#8562 `__: CI: Appveyor builds fail because it can't import ConvexHull from... * `#8576 `__: BUG: optimize: `show_options(solver='minimize', method='Newton-CG')`... * `#8603 `__: test_roots_gegenbauer/chebyt/chebyc failures on manylinux * `#8604 `__: Test failures in scipy.sparse test_inplace_dense * `#8616 `__: special: ellpj.c code can be cleaned up a bit * `#8625 `__: scipy 1.0.1 no longer allows overwriting variables in netcdf... * `#8629 `__: gcrotmk.test_atol failure with MKL * `#8632 `__: Sigma clipping on data with the same value * `#8646 `__: scipy.special.sinpi test failures in test_zero_sign on old MSVC * `#8663 `__: linprog with method=interior-point produced incorrect answer... * `#8694 `__: linalg:TestSolve.test_all_type_size_routine_combinations fails... * `#8703 `__: Q: Does runtests.py --refguide-check need env (or other) variables... Pull requests for 1.1.0 - - ----------------------- * `#6590 `__: BUG: sparse: fix custom rvs callable argument in sparse.random * `#7004 `__: ENH: scipy.linalg.eigsh cannot get all eigenvalues * `#7120 `__: ENH: implemented Owen's T function * `#7483 `__: ENH: Addition/multiplication operators for StateSpace systems * `#7566 `__: Informative exception when passing a sparse matrix * `#7592 `__: Adaptive Nelder-Mead * `#7729 `__: WIP: ENH: optimize: large-scale constrained optimization algorithms... * `#7802 `__: MRG: Add dpss window function * `#7803 `__: DOC: Add examples to spatial.distance * `#7821 `__: Add Returns section to the docstring * `#7833 `__: ENH: Performance improvements in scipy.linalg.special_matrices * `#7864 `__: MAINT: sparse: Simplify sputils.isintlike * `#7865 `__: ENH: Improved speed of copy into L, U matrices * `#7871 `__: ENH: sparse: Add 64-bit integer to sparsetools * `#7879 `__: ENH: re-enabled old sv lapack routine as defaults * `#7889 `__: DOC: Show probability density functions as math * `#7900 `__: API: Soft deprecate signal.* windows * `#7910 `__: ENH: allow `sqrtm` to compute the root of some singular matrices * `#7911 `__: MAINT: Avoid unnecessary array copies in xdist * `#7913 `__: DOC: Clarifies the meaning of `initial` of scipy.integrate.cumtrapz() * `#7916 `__: BUG: sparse.linalg: fix wrong use of __new__ in LinearOperator * `#7921 `__: BENCH: split spatial benchmark imports * `#7927 `__: ENH: added sygst/hegst routines to lapack * `#7934 `__: MAINT: add `io/_test_fortranmodule` to `.gitignore` * `#7936 `__: DOC: Fixed typo in scipy.special.roots_jacobi documentation * `#7937 `__: MAINT: special: Mark a test that fails on i686 as a known failure. * `#7941 `__: ENH: LDLt decomposition for indefinite symmetric/hermitian matrices * `#7945 `__: ENH: Implement reshape method on sparse matrices * `#7947 `__: DOC: update docs on releasing and installing/upgrading * `#7954 `__: Basin-hopping changes * `#7964 `__: BUG: test_falker not robust against numerical fuss in eigenvalues * `#7967 `__: QUADPACK Errors - human friendly errors to replace 'Invalid Input' * `#7975 `__: Make sure integrate.quad doesn't double-count singular points * `#7978 `__: TST: ensure negative weights are not allowed in distance metrics * `#7980 `__: MAINT: Truncate the warning msg about ill-conditioning * `#7981 `__: BUG: special: fix hyp2f1 behavior in certain circumstances * `#7983 `__: ENH: special: Add a real dispatch to `loggamma` * `#7989 `__: BUG: special: make `kv` return `inf` at a zero real argument * `#7990 `__: TST: special: test ufuncs in special at `nan` inputs * `#7994 `__: DOC: special: fix typo in spherical Bessel function documentation * `#7995 `__: ENH: linalg: add null_space for computing null spaces via svd * `#7999 `__: BUG: optimize: Protect _minpack calls with a lock. * `#8003 `__: MAINT: consolidate c99 compatibility * `#8004 `__: TST: special: get all `cython_special` tests running again * `#8006 `__: MAINT: Consolidate an additional _c99compat.h * `#8011 `__: Add new example of integrate.quad * `#8015 `__: DOC: special: remove `jn` from the refguide (again) * `#8018 `__: BUG - Issue with uint datatypes for array in get_index_dtype * `#8021 `__: DOC: spatial: Simplify Delaunay plotting * `#8024 `__: Documentation fix * `#8027 `__: BUG: io.matlab: fix saving unicode matrix names on py2 * `#8028 `__: BUG: special: some fixes for `lambertw` * `#8030 `__: MAINT: Bump Cython version * `#8034 `__: BUG: sparse.linalg: fix corner-case bug in expm * `#8035 `__: MAINT: special: remove complex division hack * `#8038 `__: ENH: Cythonize pyx files if pxd dependencies change * `#8042 `__: TST: stats: reduce required precision in test_fligner * `#8043 `__: TST: Use diff. values for decimal keyword for single and doubles * `#8044 `__: TST: accuracy of tests made different for singles and doubles * `#8049 `__: Unhelpful error message when calling scipy.sparse.save_npz on... * `#8052 `__: TST: spatial: add a regression test for gh-8051 * `#8059 `__: BUG: special: fix ufunc results for `nan` arguments * `#8066 `__: MAINT: special: reimplement inverses of incomplete gamma functions * `#8072 `__: Example for scipy.fftpack.ifft, https://github.com/scipy/scipy/issues/7168 * `#8073 `__: Example for ifftn, https://github.com/scipy/scipy/issues/7168 * `#8078 `__: Link to CoC in contributing.rst doc * `#8085 `__: BLD: Fix npy_isnan of integer variables in cephes * `#8088 `__: DOC: note version for which new attributes have been added to... * `#8090 `__: BUG: special: add nan check to `_legacy_cast_check` functions * `#8091 `__: Doxy Typos + trivial comment typos (2nd attempt) * `#8096 `__: TST: special: simplify `Arg` * `#8101 `__: MAINT: special: run `_generate_pyx.py` when `add_newdocs.py`... * `#8104 `__: Input checking for scipy.sparse.linalg.inverse() * `#8105 `__: DOC: special: Update the 'euler' docstring. * `#8109 `__: MAINT: fixing code comments and hyp2f1 docstring: see issues... * `#8112 `__: More trivial typos * `#8113 `__: MAINT: special: generate test data npz files in setup.py and... * `#8116 `__: DOC: add build instructions * `#8120 `__: DOC: Clean up README * `#8121 `__: DOC: Add missing colons in docstrings * `#8123 `__: BLD: update Bento build config files for recent C99 changes. * `#8124 `__: Change to avoid use of `fmod` in scipy.signal.chebwin * `#8126 `__: Added examples for mode arg in geometric_transform * `#8128 `__: relax relative tolerance parameter in TestMinumumPhase.test_hilbert * `#8129 `__: ENH: special: use rational approximation for `digamma` on `[1,... * `#8137 `__: DOC Correct matrix width * `#8141 `__: MAINT: optimize: remove unused `__main__` code in L-BSGS-B * `#8147 `__: BLD: update Bento build for removal of .npz scipy.special test... * `#8148 `__: Alias hanning as an explanatory function of hann * `#8149 `__: MAINT: special: small fixes for `digamma` * `#8159 `__: Update version classifiers * `#8164 `__: BUG: riccati solvers don't catch ill-conditioned problems sufficiently... * `#8168 `__: DOC: release note for sparse resize methods * `#8170 `__: BUG: correctly pad netCDF files with null bytes * `#8171 `__: ENH added normal inverse gaussian distribution to scipy.stats * `#8175 `__: DOC: Add example to scipy.ndimage.zoom * `#8177 `__: MAINT: diffev small speedup in ensure constraint * `#8178 `__: FIX: linalg._qz String formatter syntax error * `#8179 `__: TST: Added pdist to asv spatial benchmark suite * `#8180 `__: TST: ensure constraint test improved * `#8183 `__: 0d conj correlate * `#8186 `__: BUG: special: fix derivative of `spherical_jn(1, 0)` * `#8194 `__: Fix warning message * `#8196 `__: BUG: correctly handle inputs with nan's and ties in spearmanr * `#8198 `__: MAINT: stats.triang edge case fixes #6036 * `#8200 `__: DOC: Completed "Examples" sections of all linalg funcs * `#8201 `__: MAINT: stats.trapz edge cases * `#8204 `__: ENH: sparse.linalg/lobpcg: change .T to .T.conj() to support... * `#8206 `__: MAINT: missed triang edge case. * `#8214 `__: BUG: Fix memory corruption in linalg._decomp_update C extension * `#8222 `__: DOC: recommend scipy.integrate.solve_ivp * `#8223 `__: ENH: added Moyal distribution to scipy.stats * `#8232 `__: BUG: sparse: Use deduped data for numpy ufuncs * `#8236 `__: Fix #8235 * `#8253 `__: BUG: optimize: fix bug related with function call calculation... * `#8264 `__: ENH: Extend peak finding capabilities in scipy.signal * `#8273 `__: BUG fixed printing of convergence message in minimize_scalar... * `#8276 `__: DOC: Add notes to explain constrains on overwrite_<> * `#8279 `__: CI: fixing doctests * `#8282 `__: MAINT: weightedtau, change search for nan * `#8287 `__: Improving documentation of solve_ivp and the underlying solvers * `#8291 `__: DOC: fix non-ascii characters in docstrings which broke the doc... * `#8292 `__: CI: use numpy 1.13 for refguide check build * `#8296 `__: Fixed bug reported in issue #8181 * `#8297 `__: DOC: Examples for linalg/decomp eigvals function * `#8300 `__: MAINT: Housekeeping for minimizing the linalg compiler warnings * `#8301 `__: DOC: make public API documentation cross-link to refguide. * `#8302 `__: make sure _onenorm_matrix_power_nnm actually returns a float * `#8313 `__: Change copyright to outdated 2008-2016 to 2008-year * `#8315 `__: TST: Add tests for `scipy.sparse.linalg.isolve.minres` * `#8318 `__: ENH: odeint: Add the argument 'tfirst' to odeint. * `#8328 `__: ENH: optimize: ``trust-constr`` optimization algorithms [GSoC... * `#8330 `__: ENH: add a maxiter argument to NNLS * `#8331 `__: DOC: tweak the Moyal distribution docstring * `#8333 `__: FIX: Rewrapped ?gels and ?gels_lwork routines * `#8336 `__: MAINT: integrate: handle b < a in quad * `#8337 `__: BUG: special: Ensure zetac(1) returns inf. * `#8347 `__: BUG: Fix overflow in special.binom. Issue #8346 * `#8356 `__: DOC: Corrected Documentation Issue #7750 winsorize function * `#8358 `__: ENH: stats: Use explicit MLE formulas in lognorm.fit and expon.fit * `#8374 `__: BUG: gh7854, maxiter for l-bfgs-b closes #7854 * `#8379 `__: CI: enable gcov coverage on travis * `#8383 `__: Removed collections.OrderedDict import ignore. * `#8384 `__: TravisCI: tool pep8 is now pycodestyle * `#8387 `__: MAINT: special: remove unused specfun code for Struve functions * `#8393 `__: DOC: Replace old type names in ndimage tutorial. * `#8400 `__: Fix tolerance specification in sparse.linalg iterative solvers * `#8402 `__: MAINT: Some small cleanups in ndimage. * `#8403 `__: FIX: Make scipy.optimize.zeros run under PyPy * `#8407 `__: BUG: sparse.linalg: fix termination bugs for cg, cgs * `#8409 `__: MAINT: special: add a `.pxd` file for Cephes functions * `#8412 `__: MAINT: special: remove `cephes/protos.h` * `#8421 `__: Setting "unknown" message in OptimizeResult when calling MINPACK. * `#8423 `__: FIX: Handle unsigned integers in mmio * `#8426 `__: DOC: correct FAQ entry on Apache license compatibility. Closes... * `#8433 `__: MAINT: add `.pytest_cache` to the `.gitignore` * `#8436 `__: MAINT: scipy.sparse: less copies at transpose method * `#8437 `__: BUG: correct behavior for skew-symmetric matrices in io.mmwrite * `#8440 `__: DOC:Add examples to integrate.quadpack docstrings * `#8441 `__: BUG: sparse.linalg/gmres: deal with exact breakdown in gmres * `#8442 `__: MAINT: special: clean up Cephes header files * `#8448 `__: TST: Generalize doctest stopwords .axis( .plot( * `#8457 `__: MAINT: special: use JSON for function signatures in `_generate_pyx.py` * `#8461 `__: MAINT: Simplify return value of ndimage functions. * `#8464 `__: MAINT: Trivial typos * `#8474 `__: BUG: spatial: make qhull.pyx more pypy-friendly * `#8476 `__: TST: _lib: disable refcounting tests on PyPy * `#8479 `__: BUG: io/matlab: fix issues in matlab i/o on pypy * `#8481 `__: DOC: Example for signal.cmplx_sort * `#8482 `__: TST: integrate: use integers instead of PyCapsules to store pointers * `#8483 `__: ENH: io/netcdf: make mmap=False the default on PyPy * `#8484 `__: BUG: io/matlab: work around issue in to_writeable on PyPy * `#8488 `__: MAINT: special: add const/static specifiers where possible * `#8489 `__: BUG: ENH: use common halley's method instead of parabolic variant * `#8491 `__: DOC: fix typos * `#8496 `__: ENH: special: make Chebyshev nodes symmetric * `#8501 `__: BUG: stats: Split the integral used to compute skewnorm.cdf. * `#8502 `__: WIP: Port CircleCI to v2 * `#8507 `__: DOC: Add missing description to `brute_force` parameter. * `#8509 `__: BENCH: forgot to add nelder-mead to list of methods * `#8512 `__: MAINT: Move spline interpolation code to spline.c * `#8513 `__: TST: special: mark a slow test as xslow * `#8514 `__: CircleCI: Share data between jobs * `#8515 `__: ENH: special: improve accuracy of `zetac` for negative arguments * `#8520 `__: TST: Decrease the array sizes for two linalg tests * `#8522 `__: TST: special: restrict range of `test_besselk`/`test_besselk_int` * `#8527 `__: Documentation - example added for voronoi_plot_2d * `#8528 `__: DOC: Better, shared docstrings in ndimage * `#8533 `__: BUG: Fix PEP8 errors introduced in #8528. * `#8534 `__: ENH: Expose additional window functions * `#8538 `__: MAINT: Fix a couple mistakes in .pyf files. * `#8540 `__: ENH: interpolate: allow string aliases in make_interp_spline... * `#8541 `__: ENH: Cythonize peak_prominences * `#8542 `__: Remove numerical arguments from convolve2d / correlate2d * `#8546 `__: ENH: New arguments, documentation, and tests for ndimage.binary_opening * `#8547 `__: Giving both size and input now raises UserWarning (#7334) * `#8549 `__: DOC: stats: invweibull is also known as Frechet or type II extreme... * `#8550 `__: add cdf2rdf function * `#8551 `__: ENH: Port of most of the dd_real part of the qd high-precision... * `#8553 `__: Note in docs to address issue #3164. * `#8554 `__: ENH: stats: Use explicit MLE formulas in uniform.fit() * `#8555 `__: MAINT: adjust benchmark config * `#8557 `__: [DOC]: fix Nakagami density docstring * `#8559 `__: DOC: Fix docstring of diric(x, n) * `#8563 `__: [DOC]: fix gamma density docstring * `#8564 `__: BLD: change default Python version for doc build from 2.7 to... * `#8568 `__: BUG: Fixes Bland's Rule for pivot row/leaving variable, closes... * `#8572 `__: ENH: Add previous/next to interp1d * `#8578 `__: Example for linalg.eig() * `#8580 `__: DOC: update link to asv docs * `#8584 `__: filter_design: switch to explicit arguments, keeping None as... * `#8586 `__: DOC: stats: Add parentheses that were missing in the exponnorm... * `#8587 `__: TST: add benchmark for newton, secant, halley * `#8588 `__: DOC: special: Remove heaviside from "functions not in special"... * `#8591 `__: DOC: cdf2rdf Added version info and "See also" * `#8594 `__: ENH: Cythonize peak_widths * `#8595 `__: MAINT/ENH/BUG/TST: cdf2rdf: Address review comments made after... * `#8597 `__: DOC: add versionadded 1.1.0 for new keywords in ndimage.morphology * `#8605 `__: MAINT: special: improve implementations of `sinpi` and `cospi` * `#8607 `__: MAINT: add 2D benchmarks for convolve * `#8608 `__: FIX: Fix int check * `#8613 `__: fix typo in doc of signal.peak_widths * `#8615 `__: TST: fix failing linalg.qz float32 test by decreasing precision. * `#8617 `__: MAINT: clean up code in ellpj.c * `#8618 `__: add fsolve docs it doesn't handle over- or under-determined problems * `#8620 `__: DOC: add note on dtype attribute of aslinearoperator() argument * `#8627 `__: ENH: Add example 1D signal (ECG) to scipy.misc * `#8630 `__: ENH: Remove unnecessary copying in stats.percentileofscore * `#8631 `__: BLD: fix pdf doc build. closes gh-8076 * `#8633 `__: BUG: fix regression in `io.netcdf_file` with append mode. * `#8635 `__: MAINT: remove spurious warning from (z)vode and lsoda. Closes... * `#8636 `__: BUG: sparse.linalg/gcrotmk: avoid rounding error in termination... * `#8637 `__: For pdf build * `#8639 `__: CI: build pdf documentation on circleci * `#8640 `__: TST: fix special test that was importing `np.testing.utils` (deprecated) * `#8641 `__: BUG: optimize: fixed sparse redundancy removal bug * `#8645 `__: BUG: modified sigmaclip to avoid clipping of constant input in... * `#8647 `__: TST: sparse: skip test_inplace_dense for numpy<1.13 * `#8657 `__: Latex reduce left margins * `#8659 `__: TST: special: skip sign-of-zero test on 32-bit win32 with old... * `#8661 `__: Fix dblquad and tplquad not accepting float boundaries * `#8666 `__: DOC: fixes #8532 * `#8667 `__: BUG: optimize: fixed issue #8663 * `#8668 `__: Fix example in docstring of netcdf_file * `#8671 `__: DOC: Replace deprecated matplotlib kwarg * `#8673 `__: BUG: special: Use a stricter tolerance for the chndtr calculation. * `#8674 `__: ENH: In the Dirichlet distribution allow x_i to be 0 if alpha_i... * `#8676 `__: BUG: optimize: partial fix to linprog fails to detect infeasibility... * `#8685 `__: DOC: Add interp1d-next/previous example to tutorial * `#8687 `__: TST: netcdf: explicit mmap=True in test * `#8688 `__: BUG: signal, stats: use Python sum() instead of np.sum for summing... * `#8689 `__: TST: bump tolerances in tests * `#8690 `__: DEP: deprecate stats.itemfreq * `#8691 `__: BLD: special: fix build vs. dd_real.h package * `#8695 `__: DOC: Improve examples in signal.find_peaks with ECG signal * `#8697 `__: BUG: Fix `setup.py build install egg_info`, which did not previously... * `#8704 `__: TST: linalg: drop large size from solve() test * `#8705 `__: DOC: Describe signal.find_peaks and related functions behavior... * `#8706 `__: DOC: Specify encoding of rst file, remove an ambiguity in an... * `#8710 `__: MAINT: fix an import cycle sparse -> special -> integrate ->... * `#8711 `__: ENH: remove an avoidable overflow in scipy.stats.norminvgauss.pdf() * `#8716 `__: BUG: interpolate: allow list inputs for make_interp_spline(...,... * `#8720 `__: np.testing import that is compatible with numpy 1.15 * `#8724 `__: CI: don't use pyproject.toml in the CI builds Checksums ========= MD5 ~~~ c87600cca9adb73dd29dfeec900177fe scipy-1.1.0rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 97dde23e2f86fed4479906e2aa3e35f3 scipy-1.1.0rc1-cp27-cp27m-manylinux1_i686.whl eab2aac5065061c4c85b74b9d881dc59 scipy-1.1.0rc1-cp27-cp27m-manylinux1_x86_64.whl 2469f532bcd4ee16d3938532b0f8dfbb scipy-1.1.0rc1-cp27-cp27mu-manylinux1_i686.whl 1fb4e67a7b951087a45fb17b898db79a scipy-1.1.0rc1-cp27-cp27mu-manylinux1_x86_64.whl 1f4853e82c3ada95443296e4f5372839 scipy-1.1.0rc1-cp27-none-win32.whl e03b5f7c415acd264025ab872cfc298d scipy-1.1.0rc1-cp27-none-win_amd64.whl 5c78e958d66a5e62299362b65eeeffc8 scipy-1.1.0rc1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl d80db66d4e4732c300925bbdb3c85a01 scipy-1.1.0rc1-cp34-cp34m-manylinux1_i686.whl 7594a8f1673b617e6e787ada5eace297 scipy-1.1.0rc1-cp34-cp34m-manylinux1_x86_64.whl 1fa7a8c7801619fee41f383ecc13c42d scipy-1.1.0rc1-cp34-none-win32.whl 8a8fdf0e9b041095b57b25f22d45da58 scipy-1.1.0rc1-cp34-none-win_amd64.whl 1ebd4eab6a381de5bc6b6c2822e8b9ed scipy-1.1.0rc1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 712825cd048a4c1a33904fa2871ad2a5 scipy-1.1.0rc1-cp35-cp35m-manylinux1_i686.whl 357a4c2f3c2b74e8dd419511c51b3a65 scipy-1.1.0rc1-cp35-cp35m-manylinux1_x86_64.whl 39813d581e0775a1cdb234a94c84b648 scipy-1.1.0rc1-cp35-none-win32.whl 8d57dbdd67f77a15b92889de39b9aa65 scipy-1.1.0rc1-cp35-none-win_amd64.whl 2ac9ab606f16045f2a02147ce5ca1011 scipy-1.1.0rc1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 8d65f4b94943e9f388f545f1f7baf8e4 scipy-1.1.0rc1-cp36-cp36m-manylinux1_i686.whl 2f70f7ac43a181a4d809e38e77fa7518 scipy-1.1.0rc1-cp36-cp36m-manylinux1_x86_64.whl b9255992eb8f3a765c68a8f8aee6cc5d scipy-1.1.0rc1-cp36-none-win32.whl 4fccc0e481f9c295fc4cb17470f7556c scipy-1.1.0rc1-cp36-none-win_amd64.whl 108f3d178ada75e4ac8cd272320f6df8 scipy-1.1.0rc1.tar.gz 1a5ea56b5be7e2ba4dbd27c58d4d33ef scipy-1.1.0rc1.tar.xz 75e9321f74104c69bb9b880406261fd5 scipy-1.1.0rc1.zip SHA256 ~~~~~~ 7dbeaa7528ea4fc3c534be96a25f8944bf96b1d5c45a9ffd0b8768495d9eda9e scipy-1.1.0rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl d0b69078bf316945ed2e307c6a88e5e6f01e7dcae9d1c358fc02234ecf29438c scipy-1.1.0rc1-cp27-cp27m-manylinux1_i686.whl 7315f9b72b799703120120390770305a2875f2001a2b3f6ba9c64c6208dce20d scipy-1.1.0rc1-cp27-cp27m-manylinux1_x86_64.whl a3f2fac3c40f43e7a405d366e7b06c35b1464141cde58b85cecd1bf88b64fe5b scipy-1.1.0rc1-cp27-cp27mu-manylinux1_i686.whl 05c44da1f1ea36363177b3c92f617868fb4b317fac0e25e1e21f943c32f2fe33 scipy-1.1.0rc1-cp27-cp27mu-manylinux1_x86_64.whl 58c6b4e20d9027fc43ba395c94e9b18a0bf82f36f9bffe7fd4b9a0c5947212c7 scipy-1.1.0rc1-cp27-none-win32.whl 9c2fd23728e7e529be2ea05b74e3e5ad48bc88f557ac0c28303e1dd917d2f6af scipy-1.1.0rc1-cp27-none-win_amd64.whl 93b9590b2d8b32c547319270afaabd765b3e76a33b839d690da50319e870060e scipy-1.1.0rc1-cp34-cpwhl 9ce2799e92aa5e59383f8375cacdaa4be9d0f213d671f4427d07123f9d68e9d9 scipy-1.1.0rc1-cp35-none-win32.whl 123b42d14f350534996c6f928f606710859803cc34aeaf116f86d17a62feaa5c scipy-1.1.0rc1-cp35-none-win_amd64.whl 620b3f0af8dc943dc6e103954d002fc322749b8c19be8747ad49c796086ad498 scipy-1.1.0rc1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl a3d8f06e3b735b9ef9cda8fe6538e6d04d6352f260c5eb1e8c39119fb0408249 scipy-1.1.0rc1-cp36-cp36m-manylinux1_i686.whl f62862ec99bdb1eea257d5424505a20533e5392d54c1fcc8ad2a75fef9107e49 scipy-1.1.0rc1-cp36-cp36m-manylinux1_x86_64.whl c2ae4beaa12b5db1121e7c4dbed6fdd78ec8b5628b09991a37d93ef21c02e90f scipy-1.1.0rc1-cp36-none-win32.whl e4123e79e68b52fa582b8f5eb73a610aa06eee500016f651c01768cd2fc3e249 scipy-1.1.0rc1-cp36-none-win_amd64.whl e3370ca5814e519cc8f3017a3a75cb7e5070848a4448df04f1da661b3fadb360 scipy-1.1.0rc1.tar.gz 66c42c3160627e4a6632bc5d23892e1aabb27daae9cc0091234290c5a339fbff scipy-1.1.0rc1.tar.xz d7862d0e1374b2aedfff29fa33b16284259aa3e642cf30f98686f07fe2c38eae scipy-1.1.0rc1.zip -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEiNDtwDFNJaodBiJZHOiglY3YpVcFAlrTmMsACgkQHOiglY3Y pVeiyRAAtPG67a1dJWmTmKiM6J9qt4w5oabJ0bqQEsDmIZsPs2OqlwiWs4X4OHge GPBw5XOseBrg4Vo67+nh9otsFskG2JYvrrQFo0RpZovBexGjlJEA4aqgzvw6QGdY 20pIa7PeemGZ2aF8In/PMcViqntQ90QnZ9fUKhGf8nkd3ns0aVZ20pryTUDXUfDE pXuIrNhoMiLUWhgng45fkK+jb5+AroQPWjQ1ioelq/CXGsvJtxIc7hUiJB1pcy48 c4ymVKjCULhXMczJmRBiRqwXxoy7h9Yq7YEuP4oqM+AcmCMgVPD0320xQVk2NmCZ kzvJaEGOFpLAoyOBTHlfHNwAnD1Y7E2GCPHnzFWgStu+Frf0W7Ma1E39PaRg8dyR NNznIqT8JNxjl57oDKES0UgRiZ8EGOREFFkZ/J4jcts3jzlpP/uhJ/VmxTcfKsWe 4PvZBR1ouuvc7QCc7T1PZbdh6BFKziod7To6/QYHgVCHQ5x3Qkj8o2YFDqB7NCBd 4yc3ljZDLGrzjjNaovGn54BNPocniMJama2FfyADbP0EifT93eiAFTJ3LsyCWNK0 CQBAl1NG4BmuYlrTZmXqIdvav5CuJPQ2COE2d0RCBfrLMhmmlp7c8Ixvq0ySAXwA rNe/iedBidu8sjomtDxqvFwSWhizC0dd/hJk+gIoC3XyblNFOvw= =vWCt -----END PGP SIGNATURE----- From charlesr.harris at gmail.com Sun Apr 15 15:53:46 2018 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 15 Apr 2018 13:53:46 -0600 Subject: [SciPy-Dev] SciPy 1.1.0rc1 release candidate In-Reply-To: References: Message-ID: On Sun, Apr 15, 2018 at 12:24 PM, Pauli Virtanen wrote: > > Hi all, > > SciPy 1.1.0rc1 release candidate release is now available. Please try > it out also against your own codes and report issues you find. > > Sources and binary wheels can be found at > https://pypi.python.org/pypi/scipy > and > . > To install with pip: > > pip install --pre --upgrade scipy > > Thanks to everyone who contributed to this release! > > Pauli > Thanks for getting this out despite pip's best efforts to delay it ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at drhagen.com Wed Apr 18 07:05:30 2018 From: david at drhagen.com (David Hagen) Date: Wed, 18 Apr 2018 07:05:30 -0400 Subject: [SciPy-Dev] [SciPy-User] SciPy 1.1.0rc1 release candidate In-Reply-To: References: Message-ID: Is it too late to update the release notes? The changes to scipy.sparse are a little different from what's listed. The reshape method only worked on LIL matrices, and in-place reshaping did not work on any matrices. Both operations are now implemented for all matrices. Handling of shapes has been made consistent with numpy.matrix throughout the sparse module (shape can be a tuple or splatted, negative number acts as placeholder, padding and unpadding dimensions of size 1 to ensure length-2 shape). -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Apr 18 14:42:00 2018 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 18 Apr 2018 20:42:00 +0200 Subject: [SciPy-Dev] [SciPy-User] SciPy 1.1.0rc1 release candidate In-Reply-To: References: Message-ID: <9e669f8174a3da8e70ad178744edf04ea9580244.camel@iki.fi> Hi, ke, 2018-04-18 kello 07:05 -0400, David Hagen kirjoitti: > Is it too late to update the release notes? The changes to > scipy.sparse > are a little different from what's listed. Sure, it's release candidate so there is still time to update things. Could you make the changes you want here: https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.1.0 Best, Pauli > > The reshape method only worked on LIL matrices, and in-place > reshaping did > not work on any matrices. Both operations are now implemented for all > matrices. Handling of shapes has been made consistent with > numpy.matrix > throughout the sparse module (shape can be a tuple or splatted, > negative > number acts as placeholder, padding and unpadding dimensions of size > 1 to > ensure length-2 shape). > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From mikofski at berkeley.edu Thu Apr 19 13:17:00 2018 From: mikofski at berkeley.edu (Mark Alexander Mikofski) Date: Thu, 19 Apr 2018 10:17:00 -0700 Subject: [SciPy-Dev] SciPy in conda-forge env on Windows Message-ID: Hi, Sorry if this question is redundant. I checked the "install" page in the SciPy docs, and couldn't find any mention of conda-forge. Can SciPy from pip (scipy-1.0.1-cp36-none-win_amd64.whl) be used with the numpy-feedstock from conda-forge? When I run the tests for scipy, the fail and hang after "scipy\linalg\tests\test_basic.py". The reason I thought this might work is because they both use a variant of OpenBLAS, but perhaps they are not compatible? I can see that NumPy openblas.dll is in miniconda/envs//Library/bin (31,053KB) where as the SciPy openblas.dll is in site-packages/scipy/extra-dll (41,762KB) and the name is vendorized with a long alphanumeric string. Unfortunately if I try to use conda to install SciPy, conda uses the default "anaconda" channel, because win_amd64 conda-forge scipy-feedstock has been disabled, and when I run the tests using this mix of conda-forge:numpy and anaconda:scipy, I get the same failure at "scipy\linalg\tests\test_basic.py", which I would expect because the anaconda:scipy uses Intel-MKL not OpenBLAS, and I think these libraries need to be compatible for scipy to work, right? The test that fails is "TestDet.test_random()" scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%] this is the pytest traceback: --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) in () ----> 1 test_det.test_random() ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\scipy\linalg\tests\test_basic.py in test_random(self) 907 d1 = det(a) 908 d2 = basic_det(a) --> 909 assert_almost_equal(d1, d2) 910 911 def test_random_complex(self): ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\numpy\testing\nose_tools\utils.py in assert_almost_equal(actual, desired, decimal, err_msg, verbose) 579 pass 580 if abs(desired - actual) >= 1.5 * 10.0**(-decimal): --> 581 raise AssertionError(_build_err_msg()) 582 583 AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.001303440758814572 DESIRED: -0.09307217461347188 Then the next test hangs for several minute, and I have to kill the process. scipy/linalg/tests/test_basic.py::TestDet::test_random_complex It also hangs when I call it directly from an interpreter. If I use just pip to install both numpy (numpy-1.14.2-cp36-none-win_amd64.whl) and scipy ( scipy-1.0.1-cp36-none-win_amd64.whl) then all of the tests pass. And now I can see the vendorized version of openBLAS for numpy in site-packags/numpy/.libs (41762KB) matches the openBLAS in scipy/extra-dll, the vendor alpha-numeric string is also the same: " BNVRK7633HSX7YVO2TADGR4A5KEKXJAW" Maybe this is very basic, and I shouldn't be asking. But I don't understand how to use conda-forge then with SciPy and NumPy on Windows. It seems like users should only use all pip or only the default "anaconda" channel. Or if using conda-forge on window, then only use NumPy without SciPy, but that also means other packages like statsmodels are out too. Also maybe this is a question for conda-forge and not scipy-dev. Again, so sorry if my question is out of place. I'm also sorry if this is received poorly. I am so happy and very grateful for all of the volunteer work that has gone into making SciPy and NumPy work on Windows. I remember the old days when I had to refer people to cgohlke's website. If there is anything I can do to help, I can try. I will take a look at reactivating the conda-forge build for win_amd64, I believe that would solve the problem. Also I will bring up the issue of dependency clashes with conda-forge as well. Thanks, Mark -- Mark Mikofski, PhD (2005) *Fiat Lux* -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Apr 19 16:23:33 2018 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 19 Apr 2018 20:23:33 +0000 Subject: [SciPy-Dev] SciPy in conda-forge env on Windows In-Reply-To: References: Message-ID: Mixing wheels and conda packages can cause some problems over time because of pip and conda getting confused about what's installed. But if you made a conda env, and then did a pip install, and that's all you've done, then that should work fine AFAIK. And the scipy wheels should be self contained and work correctly in any environment, including a conda env where numpy is using mkl or a different version of openblas. I don't know what's going wrong in your case, but I can at least say that as far as I know, it *should* work and you're hitting a genuine bug. On Thu, Apr 19, 2018, 10:17 Mark Alexander Mikofski wrote: > Hi, > > Sorry if this question is redundant. I checked the "install" page in the > SciPy docs, and couldn't find any mention of conda-forge. > > Can SciPy from pip (scipy-1.0.1-cp36-none-win_amd64.whl) be used with the > numpy-feedstock from conda-forge? When I run the tests for scipy, the fail > and hang after "scipy\linalg\tests\test_basic.py". > > The reason I thought this might work is because they both use a variant of > OpenBLAS, but perhaps they are not compatible? I can see that NumPy > openblas.dll is in miniconda/envs//Library/bin (31,053KB) where as > the SciPy openblas.dll is in site-packages/scipy/extra-dll (41,762KB) and > the name is vendorized with a long alphanumeric string. > > Unfortunately if I try to use conda to install SciPy, conda uses the > default "anaconda" channel, because win_amd64 conda-forge scipy-feedstock > has been disabled, and when I run the tests using this mix of > conda-forge:numpy and anaconda:scipy, I get the same failure at "scipy\linalg\tests\test_basic.py", > which I would expect because the anaconda:scipy uses Intel-MKL not > OpenBLAS, and I think these libraries need to be compatible for scipy to > work, right? > > The test that fails is "TestDet.test_random()" > > scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%] > > this is the pytest traceback: > > --------------------------------------------------------------------------- > AssertionError Traceback (most recent call last) > in () > ----> 1 test_det.test_random() > > ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\scipy\linalg\tests\test_basic.py > in test_random(self) > 907 d1 = det(a) > 908 d2 = basic_det(a) > --> 909 assert_almost_equal(d1, d2) > 910 > 911 def test_random_complex(self): > > ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\numpy\testing\nose_tools\utils.py > in assert_almost_equal(actual, desired, decimal, err_msg, verbose) > 579 pass > 580 if abs(desired - actual) >= 1.5 * 10.0**(-decimal): > --> 581 raise AssertionError(_build_err_msg()) > 582 > 583 > > AssertionError: > Arrays are not almost equal to 7 decimals > ACTUAL: 0.001303440758814572 > DESIRED: -0.09307217461347188 > > Then the next test hangs for several minute, and I have to kill the > process. > > scipy/linalg/tests/test_basic.py::TestDet::test_random_complex > > It also hangs when I call it directly from an interpreter. > > If I use just pip to install both numpy > (numpy-1.14.2-cp36-none-win_amd64.whl) and scipy ( > scipy-1.0.1-cp36-none-win_amd64.whl) then all of the tests pass. And now > I can see the vendorized version of openBLAS for numpy in > site-packags/numpy/.libs (41762KB) matches the openBLAS in scipy/extra-dll, > the vendor alpha-numeric string is also the same: " > BNVRK7633HSX7YVO2TADGR4A5KEKXJAW" > > Maybe this is very basic, and I shouldn't be asking. But I don't > understand how to use conda-forge then with SciPy and NumPy on Windows. It > seems like users should only use all pip or only the default "anaconda" > channel. Or if using conda-forge on window, then only use NumPy without > SciPy, but that also means other packages like statsmodels are out too. > Also maybe this is a question for conda-forge and not scipy-dev. Again, so > sorry if my question is out of place. > > I'm also sorry if this is received poorly. I am so happy and very grateful > for all of the volunteer work that has gone into making SciPy and NumPy > work on Windows. I remember the old days when I had to refer people to > cgohlke's website. If there is anything I can do to help, I can try. I will > take a look at reactivating the conda-forge build for win_amd64, I believe > that would solve the problem. Also I will bring up the issue of dependency > clashes with conda-forge as well. > > Thanks, > Mark > > -- > Mark Mikofski, PhD (2005) > *Fiat Lux* > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikofski at berkeley.edu Thu Apr 19 20:14:19 2018 From: mikofski at berkeley.edu (Mark Alexander Mikofski) Date: Thu, 19 Apr 2018 17:14:19 -0700 Subject: [SciPy-Dev] SciPy in conda-forge env on Windows In-Reply-To: References: Message-ID: Thanks for your response. I tested it again, and I still hit the same issue. Here's my procedure: (base)> conda create -n forge (base)> conda activate forge (forge)> conda config --env --add channels conda-forge (forge)> conda install python=3 pip setuptools wheel (forge)> conda install nose pytest (forge)> conda install numpy (forge)>python >>> import numpy >>> numpy.test() Ran 6413 tests in 84.363s OK (KNOWNFAIL=19, SKIP=16) >>> exit() (forge)> pip install scipy (forge)> python >>> import scipy >>> scipy.test(verbose=3, tests=['scipy.linalg.tests.test_basic']) scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%] scipy/linalg/tests/test_basic.py::TestDet::test_random_complex I have to kill the process here --- Note, the same thing happens if I install scipy from conda, which installs scipy from anaconda channel on win_amd64 b/c scipy-feedstock is currently disabled on conda-forge --- But, everything passes fine if I just install both numpy and scipy from pip. And everything passes fine if I install scipy first, which then installs numpy from the default "anaconda" channel --- I posted an issue on the numpy-feedstock, and a comment on the conda-forge google group: * https://github.com/conda-forge/numpy-feedstock/issues/82 * https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/conda-forge/PPLWn4U1ZHA/frHthukWCgAJ --- I'm really sorry if this is a red-herring or wild goose chase. I didn't mean to cause any trouble. I think following Chris Barker's advice, I should have installed scipy first, and then I wouldn't have been in this mess. Thanks! On Thu, Apr 19, 2018 at 1:23 PM, Nathaniel Smith wrote: > Mixing wheels and conda packages can cause some problems over time because > of pip and conda getting confused about what's installed. But if you made a > conda env, and then did a pip install, and that's all you've done, then > that should work fine AFAIK. > > And the scipy wheels should be self contained and work correctly in any > environment, including a conda env where numpy is using mkl or a different > version of openblas. > > I don't know what's going wrong in your case, but I can at least say that > as far as I know, it *should* work and you're hitting a genuine bug. > > On Thu, Apr 19, 2018, 10:17 Mark Alexander Mikofski > wrote: > >> Hi, >> >> Sorry if this question is redundant. I checked the "install" page in the >> SciPy docs, and couldn't find any mention of conda-forge. >> >> Can SciPy from pip (scipy-1.0.1-cp36-none-win_amd64.whl) be used with >> the numpy-feedstock from conda-forge? When I run the tests for scipy, the >> fail and hang after "scipy\linalg\tests\test_basic.py". >> >> The reason I thought this might work is because they both use a variant >> of OpenBLAS, but perhaps they are not compatible? I can see that NumPy >> openblas.dll is in miniconda/envs//Library/bin (31,053KB) where >> as the SciPy openblas.dll is in site-packages/scipy/extra-dll (41,762KB) >> and the name is vendorized with a long alphanumeric string. >> >> Unfortunately if I try to use conda to install SciPy, conda uses the >> default "anaconda" channel, because win_amd64 conda-forge scipy-feedstock >> has been disabled, and when I run the tests using this mix of >> conda-forge:numpy and anaconda:scipy, I get the same failure at >> "scipy\linalg\tests\test_basic.py", which I would expect because the >> anaconda:scipy uses Intel-MKL not OpenBLAS, and I think these libraries >> need to be compatible for scipy to work, right? >> >> The test that fails is "TestDet.test_random()" >> >> scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%] >> >> this is the pytest traceback: >> >> ------------------------------------------------------------ >> --------------- >> AssertionError Traceback (most recent call >> last) >> in () >> ----> 1 test_det.test_random() >> >> ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\ >> site-packages\scipy\linalg\tests\test_basic.py in test_random(self) >> 907 d1 = det(a) >> 908 d2 = basic_det(a) >> --> 909 assert_almost_equal(d1, d2) >> 910 >> 911 def test_random_complex(self): >> >> ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\ >> site-packages\numpy\testing\nose_tools\utils.py in >> assert_almost_equal(actual, desired, decimal, err_msg, verbose) >> 579 pass >> 580 if abs(desired - actual) >= 1.5 * 10.0**(-decimal): >> --> 581 raise AssertionError(_build_err_msg()) >> 582 >> 583 >> >> AssertionError: >> Arrays are not almost equal to 7 decimals >> ACTUAL: 0.001303440758814572 >> DESIRED: -0.09307217461347188 >> >> Then the next test hangs for several minute, and I have to kill the >> process. >> >> scipy/linalg/tests/test_basic.py::TestDet::test_random_complex >> >> It also hangs when I call it directly from an interpreter. >> >> If I use just pip to install both numpy (numpy-1.14.2-cp36-none-win_amd64.whl) >> and scipy (scipy-1.0.1-cp36-none-win_amd64.whl) then all of the tests >> pass. And now I can see the vendorized version of openBLAS for numpy in >> site-packags/numpy/.libs (41762KB) matches the openBLAS in scipy/extra-dll, >> the vendor alpha-numeric string is also the same: " >> BNVRK7633HSX7YVO2TADGR4A5KEKXJAW" >> >> Maybe this is very basic, and I shouldn't be asking. But I don't >> understand how to use conda-forge then with SciPy and NumPy on Windows. It >> seems like users should only use all pip or only the default "anaconda" >> channel. Or if using conda-forge on window, then only use NumPy without >> SciPy, but that also means other packages like statsmodels are out too. >> Also maybe this is a question for conda-forge and not scipy-dev. Again, so >> sorry if my question is out of place. >> >> I'm also sorry if this is received poorly. I am so happy and very >> grateful for all of the volunteer work that has gone into making SciPy and >> NumPy work on Windows. I remember the old days when I had to refer people >> to cgohlke's website. If there is anything I can do to help, I can try. I >> will take a look at reactivating the conda-forge build for win_amd64, I >> believe that would solve the problem. Also I will bring up the issue of >> dependency clashes with conda-forge as well. >> >> Thanks, >> Mark >> >> -- >> Mark Mikofski, PhD (2005) >> *Fiat Lux* >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -- Mark Mikofski, PhD (2005) *Fiat Lux* -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Apr 20 01:17:43 2018 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 19 Apr 2018 22:17:43 -0700 Subject: [SciPy-Dev] SciPy in conda-forge env on Windows In-Reply-To: References: Message-ID: I'm suspicious about the really complex pattern of success/failure, including things that seem like they shouldn't matter, and the fact that the test is called "test_random". It does have a setUp method that attempts to seed the RNG, but maybe that's not working somehow, and it's actually a non-deterministic test failure? (To be clear: if it fails at all then that's probably a bug, but it might be a bug that's happening in all these configurations but only sometimes results in a test failure.) You should file a bug at https://github.com/scipy/scipy/issues -n On Thu, Apr 19, 2018 at 5:14 PM, Mark Alexander Mikofski wrote: > Thanks for your response. I tested it again, and I still hit the same issue. > > Here's my procedure: > > (base)> conda create -n forge > (base)> conda activate forge > (forge)> conda config --env --add channels conda-forge > (forge)> conda install python=3 pip setuptools wheel > (forge)> conda install nose pytest > (forge)> conda install numpy > (forge)>python >>>> import numpy >>>> numpy.test() > > Ran 6413 tests in 84.363s > > OK (KNOWNFAIL=19, SKIP=16) > > >>>> exit() > > (forge)> pip install scipy > (forge)> python >>>> import scipy >>>> scipy.test(verbose=3, tests=['scipy.linalg.tests.test_basic']) > > scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%] > scipy/linalg/tests/test_basic.py::TestDet::test_random_complex > > I have to kill the process here > > --- > > Note, the same thing happens if I install scipy from conda, which installs > scipy from anaconda channel on win_amd64 b/c scipy-feedstock is currently > disabled on conda-forge > > --- > > But, everything passes fine if I just install both numpy and scipy from pip. > > And everything passes fine if I install scipy first, which then installs > numpy from the default "anaconda" channel > > --- > > I posted an issue on the numpy-feedstock, and a comment on the conda-forge > google group: > > * https://github.com/conda-forge/numpy-feedstock/issues/82 > > * > https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/conda-forge/PPLWn4U1ZHA/frHthukWCgAJ > > --- > > I'm really sorry if this is a red-herring or wild goose chase. I didn't mean > to cause any trouble. I think following Chris Barker's advice, I should have > installed scipy first, and then I wouldn't have been in this mess. > > Thanks! > > > On Thu, Apr 19, 2018 at 1:23 PM, Nathaniel Smith wrote: >> >> Mixing wheels and conda packages can cause some problems over time because >> of pip and conda getting confused about what's installed. But if you made a >> conda env, and then did a pip install, and that's all you've done, then that >> should work fine AFAIK. >> >> And the scipy wheels should be self contained and work correctly in any >> environment, including a conda env where numpy is using mkl or a different >> version of openblas. >> >> I don't know what's going wrong in your case, but I can at least say that >> as far as I know, it *should* work and you're hitting a genuine bug. >> >> On Thu, Apr 19, 2018, 10:17 Mark Alexander Mikofski >> wrote: >>> >>> Hi, >>> >>> Sorry if this question is redundant. I checked the "install" page in the >>> SciPy docs, and couldn't find any mention of conda-forge. >>> >>> Can SciPy from pip (scipy-1.0.1-cp36-none-win_amd64.whl) be used with the >>> numpy-feedstock from conda-forge? When I run the tests for scipy, the fail >>> and hang after "scipy\linalg\tests\test_basic.py". >>> >>> The reason I thought this might work is because they both use a variant >>> of OpenBLAS, but perhaps they are not compatible? I can see that NumPy >>> openblas.dll is in miniconda/envs//Library/bin (31,053KB) where as >>> the SciPy openblas.dll is in site-packages/scipy/extra-dll (41,762KB) and >>> the name is vendorized with a long alphanumeric string. >>> >>> Unfortunately if I try to use conda to install SciPy, conda uses the >>> default "anaconda" channel, because win_amd64 conda-forge scipy-feedstock >>> has been disabled, and when I run the tests using this mix of >>> conda-forge:numpy and anaconda:scipy, I get the same failure at >>> "scipy\linalg\tests\test_basic.py", which I would expect because the >>> anaconda:scipy uses Intel-MKL not OpenBLAS, and I think these libraries need >>> to be compatible for scipy to work, right? >>> >>> The test that fails is "TestDet.test_random()" >>> >>> scipy/linalg/tests/test_basic.py::TestDet::test_random FAILED [ 54%] >>> >>> this is the pytest traceback: >>> >>> >>> --------------------------------------------------------------------------- >>> AssertionError Traceback (most recent call >>> last) >>> in () >>> ----> 1 test_det.test_random() >>> >>> >>> ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\scipy\linalg\tests\test_basic.py >>> in test_random(self) >>> 907 d1 = det(a) >>> 908 d2 = basic_det(a) >>> --> 909 assert_almost_equal(d1, d2) >>> 910 >>> 911 def test_random_complex(self): >>> >>> >>> ~\AppData\Local\Continuum\miniconda3\envs\forge\lib\site-packages\numpy\testing\nose_tools\utils.py >>> in assert_almost_equal(actual, desired, decimal, err_msg, verbose) >>> 579 pass >>> 580 if abs(desired - actual) >= 1.5 * 10.0**(-decimal): >>> --> 581 raise AssertionError(_build_err_msg()) >>> 582 >>> 583 >>> >>> AssertionError: >>> Arrays are not almost equal to 7 decimals >>> ACTUAL: 0.001303440758814572 >>> DESIRED: -0.09307217461347188 >>> >>> Then the next test hangs for several minute, and I have to kill the >>> process. >>> >>> scipy/linalg/tests/test_basic.py::TestDet::test_random_complex >>> >>> It also hangs when I call it directly from an interpreter. >>> >>> If I use just pip to install both numpy >>> (numpy-1.14.2-cp36-none-win_amd64.whl) and scipy >>> (scipy-1.0.1-cp36-none-win_amd64.whl) then all of the tests pass. And now I >>> can see the vendorized version of openBLAS for numpy in >>> site-packags/numpy/.libs (41762KB) matches the openBLAS in scipy/extra-dll, >>> the vendor alpha-numeric string is also the same: >>> "BNVRK7633HSX7YVO2TADGR4A5KEKXJAW" >>> >>> Maybe this is very basic, and I shouldn't be asking. But I don't >>> understand how to use conda-forge then with SciPy and NumPy on Windows. It >>> seems like users should only use all pip or only the default "anaconda" >>> channel. Or if using conda-forge on window, then only use NumPy without >>> SciPy, but that also means other packages like statsmodels are out too. Also >>> maybe this is a question for conda-forge and not scipy-dev. Again, so sorry >>> if my question is out of place. >>> >>> I'm also sorry if this is received poorly. I am so happy and very >>> grateful for all of the volunteer work that has gone into making SciPy and >>> NumPy work on Windows. I remember the old days when I had to refer people to >>> cgohlke's website. If there is anything I can do to help, I can try. I will >>> take a look at reactivating the conda-forge build for win_amd64, I believe >>> that would solve the problem. Also I will bring up the issue of dependency >>> clashes with conda-forge as well. >>> >>> Thanks, >>> Mark >>> >>> -- >>> Mark Mikofski, PhD (2005) >>> Fiat Lux >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > > > > -- > Mark Mikofski, PhD (2005) > Fiat Lux > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Nathaniel J. Smith -- https://vorpus.org From matthew.brett at gmail.com Fri Apr 20 08:23:35 2018 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 20 Apr 2018 13:23:35 +0100 Subject: [SciPy-Dev] Help with Hackathon, tomorrow Message-ID: Hi, I'm sorry this is such late notice, but can I ask for some help? I unwisely volunteered to help with a hackathon in London tomorrow: https://www.ahl.com/hackathon I have every reason to think there will be many excellent programmers coming, and some of them will want to work on Scipy. It could be very useful for engaging people in working on Scipy development. Do any of you have suggestions for issues we could work on? I'd love to hear any suggestions. Thanks a lot, Matthew From larson.eric.d at gmail.com Fri Apr 20 09:22:44 2018 From: larson.eric.d at gmail.com (Eric Larson) Date: Fri, 20 Apr 2018 09:22:44 -0400 Subject: [SciPy-Dev] Help with Hackathon, tomorrow In-Reply-To: References: Message-ID: > > Do any of you have suggestions for issues we could work on? I'd love > to hear any suggestions. > One option would be to suggest that each interested participant open the SciPy issues and PRs list, and then sub-select based on the `label` corresponding to a submodule they might be interested in. That way people will gravitate toward domains where they have knowledge and feel comfortable, and it cuts down on the number of issues they need to look at (because we have so many). And of course there is also the "good first issue" label, which can be useful even if it's inconsistently applied. As for any specific issues, I can't think of any offhand. Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at gmail.com Fri Apr 20 20:12:09 2018 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Fri, 20 Apr 2018 20:12:09 -0400 Subject: [SciPy-Dev] Help with Hackathon, tomorrow In-Reply-To: References: Message-ID: On Fri, Apr 20, 2018 at 9:22 AM, Eric Larson wrote: > Do any of you have suggestions for issues we could work on? I'd love >> to hear any suggestions. >> > > One option would be to suggest that each interested participant open the > SciPy issues and PRs list, and then sub-select based on the `label` > corresponding to a submodule they might be interested in. That way people > will gravitate toward domains where they have knowledge and feel > comfortable, and it cuts down on the number of issues they need to look at > (because we have so many). > > And of course there is also the "good first issue" label, which can be > useful even if it's inconsistently applied. > > One of those issues is https://github.com/scipy/scipy/issues/7168. There are hundreds of function in the library that do not have an "Examples" section. It would be great to bring that number down. Adding some examples could be a good way to get folks started with the SciPy development work flow (building the library, building the docs, and making pull requests). If you pursue this, be sure that everyone is aware of PEP 8 (https://www.python.org/dev/peps/pep-0008/) and knows where to find the docstring standard ( https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard). Have fun! Warren > As for any specific issues, I can't think of any offhand. > > Eric > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Apr 21 02:40:44 2018 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 21 Apr 2018 07:40:44 +0100 Subject: [SciPy-Dev] Help with Hackathon, tomorrow In-Reply-To: References: Message-ID: Hi, On Sat, Apr 21, 2018 at 1:12 AM, Warren Weckesser wrote: > > > On Fri, Apr 20, 2018 at 9:22 AM, Eric Larson > wrote: >>> >>> Do any of you have suggestions for issues we could work on? I'd love >>> to hear any suggestions. >> >> >> One option would be to suggest that each interested participant open the >> SciPy issues and PRs list, and then sub-select based on the `label` >> corresponding to a submodule they might be interested in. That way people >> will gravitate toward domains where they have knowledge and feel >> comfortable, and it cuts down on the number of issues they need to look at >> (because we have so many). >> >> And of course there is also the "good first issue" label, which can be >> useful even if it's inconsistently applied. >> > > > One of those issues is https://github.com/scipy/scipy/issues/7168. There > are hundreds of function in the library that do not have an "Examples" > section. It would be great to bring that number down. Adding some examples > could be a good way to get folks started with the SciPy development work > flow (building the library, building the docs, and making pull requests). > If you pursue this, be sure that everyone is aware of PEP 8 > (https://www.python.org/dev/peps/pep-0008/) and knows where to find the > docstring standard > (https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard). Excellent suggestion, thanks. I'll offer that as a fall-back, and maybe as a first hurdle, before trying something with more meat. Cheers, Matthew From toddrjen at gmail.com Sat Apr 21 16:12:40 2018 From: toddrjen at gmail.com (Todd) Date: Sat, 21 Apr 2018 16:12:40 -0400 Subject: [SciPy-Dev] Help with Hackathon, tomorrow In-Reply-To: References: Message-ID: On Apr 21, 2018 02:41, "Matthew Brett" wrote: Hi, On Sat, Apr 21, 2018 at 1:12 AM, Warren Weckesser wrote: > > > On Fri, Apr 20, 2018 at 9:22 AM, Eric Larson > wrote: >>> >>> Do any of you have suggestions for issues we could work on? I'd love >>> to hear any suggestions. >> >> >> One option would be to suggest that each interested participant open the >> SciPy issues and PRs list, and then sub-select based on the `label` >> corresponding to a submodule they might be interested in. That way people >> will gravitate toward domains where they have knowledge and feel >> comfortable, and it cuts down on the number of issues they need to look at >> (because we have so many). >> >> And of course there is also the "good first issue" label, which can be >> useful even if it's inconsistently applied. >> > > > One of those issues is https://github.com/scipy/scipy/issues/7168. There > are hundreds of function in the library that do not have an "Examples" > section. It would be great to bring that number down. Adding some examples > could be a good way to get folks started with the SciPy development work > flow (building the library, building the docs, and making pull requests). > If you pursue this, be sure that everyone is aware of PEP 8 > (https://www.python.org/dev/peps/pep-0008/) and knows where to find the > docstring standard > (https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard ). Excellent suggestion, thanks. I'll offer that as a fall-back, and maybe as a first hurdle, before trying something with more meat. Cheers, Matthew A pet peeve of mine, but I know there has been some work on getting the scipy.signal.fftconvolve function to broadcast across dimensions [1], instead of only working with vectors. However, it has never been completed. This is a particularly important case because it is a necessary prerequisite for an implementation of the overlap-add convolution algorithm [2], one of the more critical basic signal processing algorithms that scipy still lacks. It \provides a massive performance boost when you have two large (both >= ~500 samples) but different-sized vectors. So although getting the overlap-add method implemented is probably too much, just allowing applying the 1D fftconvolve across dimensions might be within in reason. The advantage for the hackathon is that it provides the groundwork for a big improvement, but shouldn't require any serious signal-processing knowledge, since the actual algorithm is already implemented. This just requires implement the dimension-handling logic. [1] https://github.com/scipy/scipy/issues/3525 [2] https://en.wikipedia.org/wiki/Overlap-add_method -------------- next part -------------- An HTML attachment was scrubbed... URL: From corinhoad at gmail.com Sat Apr 21 19:42:31 2018 From: corinhoad at gmail.com (Corin Hoad) Date: Sun, 22 Apr 2018 00:42:31 +0100 Subject: [SciPy-Dev] scipy.stats.ks_2samp with weighted data Message-ID: Hello developers, I recently needed an implementation of the Kolmogorov-Smirnov 2 sample test which required the incorporation of a weight associated with each element of the data. This lead me to this stackexchange answer https://stats.stackexchange.com/questions/193439/two-sample-kolmogorov-smirnov-test-with-weights where a procedure for a weighted 2-sample KS test is taken from Numerical Methods of Statistics by Monohan. My current implementation of this can be found here: https://github.com/brunel-physics/tact/blob/2b0ee2a28a30f014b103319118b64be52070f001/tact/metrics.py#L198 Would there by any interest in incorporating this functionality into scipy? Yours, Corin Hoad -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Apr 21 20:30:37 2018 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 21 Apr 2018 20:30:37 -0400 Subject: [SciPy-Dev] scipy.stats.ks_2samp with weighted data In-Reply-To: References: Message-ID: On Sat, Apr 21, 2018 at 7:42 PM, Corin Hoad wrote: > Hello developers, > > I recently needed an implementation of the Kolmogorov-Smirnov 2 sample > test which required the incorporation of a weight associated with each > element of the data. This lead me to this stackexchange answer > https://stats.stackexchange.com/questions/193439/two- > sample-kolmogorov-smirnov-test-with-weights where a procedure for a > weighted 2-sample KS test is taken from Numerical Methods of Statistics by > Monohan. > > My current implementation of this can be found here: > > https://github.com/brunel-physics/tact/blob/2b0ee2a28a30f014b103319118b64b > e52070f001/tact/metrics.py#L198 > > Would there by any interest in incorporating this functionality into scipy? > I have potentially two problems: What's the definition or interpretation of the weights? Is there distribution of the test statistic correct? My guess is that it would change when weighting is introduced. I didn't find much in a brief Google search. Whether the distribution/p-value is correct could also be checked with the rejection probabilities in a simulation. Josef > > Yours, > > Corin Hoad > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillip.m.feldman at gmail.com Sat Apr 21 23:15:48 2018 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Sat, 21 Apr 2018 20:15:48 -0700 Subject: [SciPy-Dev] scipy.stats.ks_2samp with weighted data In-Reply-To: References: Message-ID: Just out of curiosity: What is the significance of the weights? If you are trying to represent the fact that distributional differences are more important in some regime than in another, e.g., you care more about the tails, then using weights is probably not the right approach. On Sat, Apr 21, 2018 at 4:42 PM, Corin Hoad wrote: > Hello developers, > > I recently needed an implementation of the Kolmogorov-Smirnov 2 sample > test which required the incorporation of a weight associated with each > element of the data. This lead me to this stackexchange answer > https://stats.stackexchange.com/questions/193439/two- > sample-kolmogorov-smirnov-test-with-weights where a procedure for a > weighted 2-sample KS test is taken from Numerical Methods of Statistics by > Monohan. > > My current implementation of this can be found here: > > https://github.com/brunel-physics/tact/blob/2b0ee2a28a30f014b103319118b64b > e52070f001/tact/metrics.py#L198 > > Would there by any interest in incorporating this functionality into scipy? > > Yours, > > Corin Hoad > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sun Apr 22 09:29:51 2018 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 22 Apr 2018 09:29:51 -0400 Subject: [SciPy-Dev] scipy.stats.ks_2samp with weighted data In-Reply-To: References: Message-ID: On Sat, Apr 21, 2018 at 11:15 PM, Phillip Feldman < phillip.m.feldman at gmail.com> wrote: > Just out of curiosity: What is the significance of the weights? If you > are trying to represent the fact that distributional differences are more > important in some regime than in another, e.g., you care more about the > tails, then using weights is probably not the right approach. > I don't remember for sup tests like KS, but for integral tests like Anderson-Darling there are variations of the test that use different weights to emphasize different regions of the distribution, e.g. Cramer-Von Mises uses different weights than AD https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test I briefly skimmed parts of the Monahan chapter and that is specific for importance sampling weights. In that case choosing the weights should just compensate for the unequal sampling of points. So maybe in that case the distribution of the KS (or AD) test statistic might not change much. In either case, I think the distribution of the test statistic depends on the meaning or interpretation of the weights. Josef > > On Sat, Apr 21, 2018 at 4:42 PM, Corin Hoad wrote: > >> Hello developers, >> >> I recently needed an implementation of the Kolmogorov-Smirnov 2 sample >> test which required the incorporation of a weight associated with each >> element of the data. This lead me to this stackexchange answer >> https://stats.stackexchange.com/questions/193439/two-sample- >> kolmogorov-smirnov-test-with-weights where a procedure for a weighted >> 2-sample KS test is taken from Numerical Methods of Statistics by Monohan. >> >> My current implementation of this can be found here: >> >> https://github.com/brunel-physics/tact/blob/2b0ee2a28a30f014 >> b103319118b64be52070f001/tact/metrics.py#L198 >> >> Would there by any interest in incorporating this functionality into >> scipy? >> >> Yours, >> >> Corin Hoad >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From corinhoad at gmail.com Sun Apr 22 10:39:04 2018 From: corinhoad at gmail.com (Corin Hoad) Date: Sun, 22 Apr 2018 15:39:04 +0100 Subject: [SciPy-Dev] scipy.stats.ks_2samp with weighted data In-Reply-To: References: Message-ID: > > What's the definition or interpretation of the weights? Just out of curiosity: What is the significance of the weights? If you are > trying to represent the fact that distributional differences are more > important in some regime than in another, e.g., you care more about the > tails, then using weights is probably not the right approach > I briefly skimmed parts of the Monahan chapter and that is specific for > importance sampling weights. In that case choosing the weights should just > compensate for the unequal sampling of points. So maybe in that case the > distribution of the KS (or AD) test statistic might not change much. > The weights are sample weights, indicating the relative frequency of observations. My specific use case is in particle physics where studies often involve simulated data. Certain particle physics processes may have a large amount of simulated data available, but in reality we expect them to be very rare so sample weights are used to compensate. Corin On 22 April 2018 at 14:29, wrote: > > > On Sat, Apr 21, 2018 at 11:15 PM, Phillip Feldman < > phillip.m.feldman at gmail.com> wrote: > >> Just out of curiosity: What is the significance of the weights? If you >> are trying to represent the fact that distributional differences are more >> important in some regime than in another, e.g., you care more about the >> tails, then using weights is probably not the right approach. >> > > I don't remember for sup tests like KS, but for integral tests like > Anderson-Darling there are variations of the test that use different > weights to emphasize different regions of the distribution, e.g. Cramer-Von > Mises uses different weights than AD > https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test > > > I briefly skimmed parts of the Monahan chapter and that is specific for > importance sampling weights. In that case choosing the weights should just > compensate for the unequal sampling of points. So maybe in that case the > distribution of the KS (or AD) test statistic might not change much. > > In either case, I think the distribution of the test statistic depends on > the meaning or interpretation of the weights. > > Josef > > > >> >> On Sat, Apr 21, 2018 at 4:42 PM, Corin Hoad wrote: >> >>> Hello developers, >>> >>> I recently needed an implementation of the Kolmogorov-Smirnov 2 sample >>> test which required the incorporation of a weight associated with each >>> element of the data. This lead me to this stackexchange answer >>> https://stats.stackexchange.com/questions/193439/two-sample- >>> kolmogorov-smirnov-test-with-weights where a procedure for a weighted >>> 2-sample KS test is taken from Numerical Methods of Statistics by Monohan. >>> >>> My current implementation of this can be found here: >>> >>> https://github.com/brunel-physics/tact/blob/2b0ee2a28a30f014 >>> b103319118b64be52070f001/tact/metrics.py#L198 >>> >>> Would there by any interest in incorporating this functionality into >>> scipy? >>> >>> Yours, >>> >>> Corin Hoad >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >>> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Mon Apr 23 06:20:32 2018 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Mon, 23 Apr 2018 12:20:32 +0200 Subject: [SciPy-Dev] linalg.solve keyword deprecation Message-ID: Hi there, Some time ago, we have included different matrix structure types such as symmetric, positive definite and hermitian together with the generic matrices to solve AX=B problems in linalg.solve. Previously, there were two keywords in the signature namely "debug" and "sym_pos". These keywords, in my opinion, are obsolete and actually clutter the function signature because the dummy keyword "debug" has no functionality and "sym_pos" is covered by "assume_a" keyword as a special case. Hence I've proposed to deprecate these in https://github.com/scipy/scipy/pull/8715/files . Pauli also chimed in and mentioned that this might not be a good idea since this is a central function and the benefits might not be worth the effort and backwards compatibility problems. With Python2 dying, I think the backwards compatibility part won't be such important problem anymore and the benefit is that we don't need to have such strange signature. Thus, I'd like to ask around what others think about this since this indeed "needs-decision". ilhan -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Mon Apr 23 09:23:57 2018 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 23 Apr 2018 15:23:57 +0200 Subject: [SciPy-Dev] linalg.solve keyword deprecation In-Reply-To: References: Message-ID: <1524489837.2359.16.camel@iki.fi> ma, 2018-04-23 kello 12:20 +0200, Ilhan Polat kirjoitti: [clip: solve(sym_pos=, debug=)] > Hence I've proposed to deprecate these in > https://github.com/scipy/scipy/pull/8715/files . Pauli also chimed in > and > mentioned that this might not be a good idea since this is a central > function and the benefits might not be worth the effort and backwards > compatibility problems. With Python2 dying, I think the backwards > compatibility part won't be such important problem anymore and the > benefit > is that we don't need to have such strange signature. This I think is relevant: http://blog.khinsen.net/posts/2017/11/22/stability-in-the-scipy-ecosystem-a-summary-of-the-discussion/ My own position is that any breakage should have good reasons behind it, and cosmetic reasons (function signature with some duplicated functionality) usually are not strong enough. Re: Python3 --- I think that Python 3 also breaks things should not be a permission to break more things. Pauli From ilhanpolat at gmail.com Mon Apr 23 09:46:12 2018 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Mon, 23 Apr 2018 15:46:12 +0200 Subject: [SciPy-Dev] linalg.solve keyword deprecation In-Reply-To: <1524489837.2359.16.camel@iki.fi> References: <1524489837.2359.16.camel@iki.fi> Message-ID: Yes, I have engaged with Konrad before on this issue on twitter and GitHub. However, I have to say I disagree with most of his take on backwards compatibility issues. I don't want to revamp that discussion once again but I don't buy this "old code" argument. Old code also has to be maintained continuously. I don't mean to break things in every release but then again any "ENH" PR can be rejected with the same argument such that we can only fix bugs and nothing else. Personally, this was the reason that pushed me away from matlab with strange syntax and unusable UX choices that has been around since 30 years (8.3 filenames, nargin, nargout frenzy etc.). But even mathworks break code pretty often in every toolbox, see some of the exclamation marks, say in, https://ww2.mathworks.cn/help/control/release-notes.html These keywords won't be removed, possibly, until v.1.5.x only warning will be emitted hence there is ample time to adapt. To be clear on the Python3 detail, it was about the keyword order restrictions. On Mon, Apr 23, 2018 at 3:23 PM, Pauli Virtanen wrote: > ma, 2018-04-23 kello 12:20 +0200, Ilhan Polat kirjoitti: > [clip: solve(sym_pos=, debug=)] > > Hence I've proposed to deprecate these in > > https://github.com/scipy/scipy/pull/8715/files . Pauli also chimed in > > and > > mentioned that this might not be a good idea since this is a central > > function and the benefits might not be worth the effort and backwards > > compatibility problems. With Python2 dying, I think the backwards > > compatibility part won't be such important problem anymore and the > > benefit > > is that we don't need to have such strange signature. > > This I think is relevant: > > http://blog.khinsen.net/posts/2017/11/22/stability-in-the- > scipy-ecosystem-a-summary-of-the-discussion/ > > My own position is that any breakage should have good reasons behind > it, and cosmetic reasons (function signature with some duplicated > functionality) usually are not strong enough. > > Re: Python3 --- I think that Python 3 also breaks things should not be > a permission to break more things. > > Pauli > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Apr 23 09:51:38 2018 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 23 Apr 2018 09:51:38 -0400 Subject: [SciPy-Dev] linalg.solve keyword deprecation In-Reply-To: <1524489837.2359.16.camel@iki.fi> References: <1524489837.2359.16.camel@iki.fi> Message-ID: On Mon, Apr 23, 2018 at 9:23 AM, Pauli Virtanen wrote: > ma, 2018-04-23 kello 12:20 +0200, Ilhan Polat kirjoitti: > [clip: solve(sym_pos=, debug=)] > > Hence I've proposed to deprecate these in > > https://github.com/scipy/scipy/pull/8715/files . Pauli also chimed in > > and > > mentioned that this might not be a good idea since this is a central > > function and the benefits might not be worth the effort and backwards > > compatibility problems. With Python2 dying, I think the backwards > > compatibility part won't be such important problem anymore and the > > benefit > > is that we don't need to have such strange signature. > > This I think is relevant: > > http://blog.khinsen.net/posts/2017/11/22/stability-in-the- > scipy-ecosystem-a-summary-of-the-discussion/ > > My own position is that any breakage should have good reasons behind > it, and cosmetic reasons (function signature with some duplicated > functionality) usually are not strong enough. > > Re: Python3 --- I think that Python 3 also breaks things should not be > a permission to break more things. > I think even "cosmetic" (confusing signatures) should be cleaned up in the long run, otherwise they never go away unless there is a big break release. But deprecation periods should be long, e.g. > 3 years. (I had to deprecate three functions in statsmodels because I had misspelled the function names when writing them.) Josef > > Pauli > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Mon Apr 23 12:02:08 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Mon, 23 Apr 2018 12:02:08 -0400 Subject: [SciPy-Dev] Allow non-canonical form in COO? Message-ID: Hello SciPy-Dev, I submitted a pull request to pydata/sparse (I?m one of its maintainers) about always keeping a COO array in canonical form. We (Matthew Rocklin, CJ Carey and I) were having a discussion there about whether it makes sense to allow sparse COO arrays to be in non-canonical order. In short, canonical form is: - Coordinates are sorted in lexographical order, i.e., the same order they would appear in in a C-contiguous array. - There are no duplicate coordinates allowed. - There are no stored coordinates with a zero value. With this in mind, what are the SciPy developers? opinions on this? Does allowing COO arrays to be in non-canonical form make sense? Feedback welcome on the PR or the mailing list. Best regards, Hameer Abbasi Sent from Astro for Mac -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Apr 24 00:26:37 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 23 Apr 2018 21:26:37 -0700 Subject: [SciPy-Dev] GSoC'18 accepted student announced Message-ID: Hi all, Google announced the results of the GSoC application process. SciPy received 1 slot - I'm happy to congratulate Aditya Bharti as our accepted student for this year! And a big thanks to Nikolay Mayorov and Eric Larson for stepping up to mentor this year! Looking forward to an interesting project! Aditya's project: Official summary: https://summerofcode.withgoogle.com/projects/#6542621229449216 Full proposal (latest draft, final pdf is not public): https://docs.google.com/document/d/1UriyLABwgjUcYfBofSr4hqAnLzblJ71pkzm7i26O8ws/edit Co-mentors: Nikolay Mayorov, Eric Larson (I'm listed as mentor as well, but that's mainly for admin reasons). The "community bonding" period has just started, let's all make sure to welcome Aditya and help him where needed, so he can hit the ground running from May 14th onwards! Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Tue Apr 24 01:11:49 2018 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Mon, 23 Apr 2018 22:11:49 -0700 Subject: [SciPy-Dev] GSoC'18 accepted student announced In-Reply-To: References: Message-ID: <162f610a608.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Hi Aditya, Congratulations and welcome! This looks like a fun and useful project, and I'm looking forward to seeing its development. Best regards, St?fan On April 23, 2018 21:27:09 Ralf Gommers wrote: Hi all, Google announced the results of the GSoC application process. SciPy received 1 slot - I'm happy to congratulate Aditya Bharti as our accepted student for this year! And a big thanks to Nikolay Mayorov and Eric Larson for stepping up to mentor this year! Looking forward to an interesting project! -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge.guelton at telecom-bretagne.eu Tue Apr 24 03:09:31 2018 From: serge.guelton at telecom-bretagne.eu (Serge Guelton) Date: Tue, 24 Apr 2018 09:09:31 +0200 Subject: [SciPy-Dev] Pythran 0.8.5 Message-ID: <20180424070931.GA19986@lakota> (sorry for the double posting if any) It is my pleasure to announce a new version of the Pythran compiler. Pythran is an Ahead of Time compiler for a subset of the Python language, with a focus on scientific kernels. It can turn code like the one below: #pythran export weights(uint8[:,:]) import numpy as np def weights(input_data, threshold=0.3): n_seq, length = input_data.shape weights = np.zeros(n_seq, dtype=np.float32) for i in range(n_seq): vector = input_data[i, None, :] count_matches = np.sum(vector == input_data, axis=1) over_threshold = count_matches > (threshold * length) total = np.sum(over_threshold) weights[i] = 1 / total return weights into a native module that runs roughly 10 times faster than when interpreted by cpython. It's available on PyPi, Conda and GitHub under BSD license. For the curious reader, the Changelog is reproduced below. It's a megablast! ---- 2018-04-23 Serge Guelton - numpy.fft support (thanks to Jean Laroche) - Faster generalized expression - Faster numpy.transpose, numpy.argmax, numpy reduction - Sphinx-compatible generated docstring (thanks to Pierre Augier) - Python output through -P (thanks to Pierre Augier) - Many bugfixes and numpy improvements (thanks to Yann Diorecet and Jean Laroche) From adibhar97 at gmail.com Tue Apr 24 06:47:22 2018 From: adibhar97 at gmail.com (Aditya Bharti) Date: Tue, 24 Apr 2018 16:17:22 +0530 Subject: [SciPy-Dev] GSoC'18 accepted student announced In-Reply-To: <162f610a608.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> References: <162f610a608.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: Hi all, Thank you for all your help so far! Your feedback was invaluable and went a long way towards the strength of my application. Let me introduce myself formally. My name is Aditya Bharti and I am currently an undergraduate researcher in Computer Vision and Machine Learning at the International Institute of Information Technology, Hyderabad, India. I've been using numpy and scipy for over 3 years, and python in general for longer. I'd also like to know more about the community in general. What sort of areas are the members from? From my interviews and the proposal feedback, I found the community here to be extremely diverse; people are working on all sorts of things ranging from LIDAR to linear dynamical systems. I look forward to working with scipy over the summer! Regards, Aditya On 24 April 2018 at 10:41, Stefan van der Walt wrote: > Hi Aditya, > > Congratulations and welcome! This looks like a fun and useful project, and > I'm looking forward to seeing its development. > > Best regards, > St?fan > > On April 23, 2018 21:27:09 Ralf Gommers wrote: > >> Hi all, >> >> Google announced the results of the GSoC application process. SciPy >> received 1 slot - I'm happy to congratulate Aditya Bharti as our accepted >> student for this year! And a big thanks to Nikolay Mayorov and Eric Larson >> for stepping up to mentor this year! Looking forward to an interesting >> project! >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adibhar97 at gmail.com Tue Apr 24 06:50:07 2018 From: adibhar97 at gmail.com (Aditya Bharti) Date: Tue, 24 Apr 2018 16:20:07 +0530 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions Message-ID: Hi all, This will serve as the discussion thread for this year's GSoC project. In the spirit of hitting the ground running: 1. Development environment has been set up. No worries there. 2. I had initially created a sample implementation of the rotation API. However, due to community feedback and mentor feedback the design was overhauled and I thought it best to start the implementation from scratch. I do need some help: 1. Since this project will include documentation, I'm trying to build the docs. However, from the Makefile at scipy/doc it appears that python3.6 is required, whereas only python3.5 is available for my platform (Ubuntu 16.04 LTS). I generally try to avoid adding third-party repos unless absolutely required, so I am asking for help here. Is installing python3.6 the only solution, or will changing the PYVERS variable in the Makefile work? 2. Regarding the GSOC blog that needs to be set up, what is the preferred method/site/url for scipy? And what exactly should a blog post contain? I'm a bit unclear on those details. Regards, Aditya -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Tue Apr 24 06:56:07 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Tue, 24 Apr 2018 12:56:07 +0200 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: References: Message-ID: Hi Aditya, About Python 3.6, you can set up a user (non-local) installation with Anaconda , it's what I use to set up multiple different interpreters, one for each project. You won't need to modify or touch your system Python installation. If you want something more minimal, you can also go for Miniconda . The latest version of conda doesn't require you to edit the path. Feel free to e-mail me personally if you need any more help. Hameer Abbasi On Tue, Apr 24, 2018 at 12:50 PM, Aditya Bharti wrote: > Hi all, > This will serve as the discussion thread for this year's GSoC project. > > In the spirit of hitting the ground running: > > 1. Development environment has been set up. No worries there. > 2. I had initially created a sample implementation of the rotation > API. However, due to community feedback and mentor feedback the design > was overhauled and I thought it best to start the implementation from > scratch. > > I do need some help: > > 1. Since this project will include documentation, I'm trying to build > the docs. However, from the Makefile at scipy/doc it appears that > python3.6 is required, whereas only python3.5 is available for my platform > (Ubuntu 16.04 LTS). I generally try to avoid adding third-party repos > unless absolutely required, so I am asking for help here. Is installing > python3.6 the only solution, or will changing the PYVERS variable in > the Makefile work? > 2. Regarding the GSOC blog that needs to be set up, what is the > preferred method/site/url for scipy? And what exactly should a blog > post contain? I'm a bit unclear on those details. > > Regards, > Aditya > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From larson.eric.d at gmail.com Tue Apr 24 11:40:24 2018 From: larson.eric.d at gmail.com (Eric Larson) Date: Tue, 24 Apr 2018 11:40:24 -0400 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: References: Message-ID: > > > 1. Since this project will include documentation, I'm trying to build > the docs. However, from the Makefile at scipy/doc it appears that > python3.6 is required, whereas only python3.5 is available for my platform > (Ubuntu 16.04 LTS). I generally try to avoid adding third-party repos > unless absolutely required, so I am asking for help here. Is installing > python3.6 the only solution, or will changing the PYVERS variable in > the Makefile work? > > I suspect 3.5 should work once you get it to use your Python binaries. I doubt we have anything that is Python-3.6 specific (though there very well be some Python 3-isms that will not work on 2.7). > > 1. Regarding the GSOC blog that needs to be set up, what is the > preferred method/site/url for scipy? And what exactly should a blog > post contain? I'm a bit unclear on those details. > > You can choose any blogging platform you want. Usually people use it to give approximately weekly updates about progress, difficulties, successes, cool tricks, etc. See how it's mentioned here (and elsewhere in the student guide): https://google.github.io/gsocguides/student/time-management-for-students Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From einstein.edison at gmail.com Tue Apr 24 11:50:53 2018 From: einstein.edison at gmail.com (Hameer Abbasi) Date: Tue, 24 Apr 2018 08:50:53 -0700 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: References: Message-ID: Hi Aditya, About Blogging, I would suggest a GitHub fork with GitHub issues relating to what you are doing is an excellent way to keep track of what you?ve done and what you have yet to do. Best regards, Hameer Abbasi Sent from Astro for Mac On Apr 24, 2018 at 12:50, Aditya Bharti wrote: Hi all, This will serve as the discussion thread for this year's GSoC project. In the spirit of hitting the ground running: 1. Development environment has been set up. No worries there. 2. I had initially created a sample implementation of the rotation API. However, due to community feedback and mentor feedback the design was overhauled and I thought it best to start the implementation from scratch. I do need some help: 1. Since this project will include documentation, I'm trying to build the docs. However, from the Makefile at scipy/doc it appears that python3.6 is required, whereas only python3.5 is available for my platform (Ubuntu 16.04 LTS). I generally try to avoid adding third-party repos unless absolutely required, so I am asking for help here. Is installing python3.6 the only solution, or will changing the PYVERS variable in the Makefile work? 2. Regarding the GSOC blog that needs to be set up, what is the preferred method/site/url for scipy? And what exactly should a blog post contain? I'm a bit unclear on those details. Regards, Aditya _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Apr 24 12:21:04 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 24 Apr 2018 09:21:04 -0700 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: References: Message-ID: On Tue, Apr 24, 2018 at 8:40 AM, Eric Larson wrote: > >> 1. Since this project will include documentation, I'm trying to build >> the docs. However, from the Makefile at scipy/doc it appears that >> python3.6 is required, whereas only python3.5 is available for my platform >> (Ubuntu 16.04 LTS). I generally try to avoid adding third-party repos >> unless absolutely required, so I am asking for help here. Is installing >> python3.6 the only solution, or will changing the PYVERS variable in >> the Makefile work? >> >> I suspect 3.5 should work once you get it to use your Python binaries. I > doubt we have anything that is Python-3.6 specific (though there very well > be some Python 3-isms that will not work on 2.7). > There's a PYVER=3.6 at the top of the Makefile. Simply doing `make html PYVER=3.5` should build the docs with your 3.5 install (if not, change that one line in the Makefile to say 3.5). Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From adibhar97 at gmail.com Tue Apr 24 14:36:02 2018 From: adibhar97 at gmail.com (Aditya Bharti) Date: Wed, 25 Apr 2018 00:06:02 +0530 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: References: Message-ID: Regarding building the documentation, I was able to figure out that I needed to install sphinx separately. The HACKING.rst.txt file only mentions Numpy Cython and pytest as dependencies. I was able to get the make command running with `make html PYVER=3.5`, but I'm getting the follwing error: > Could not import extension numpydoc (exception: cannot import name > 'Directive') > I'm sure I have the extension installed, since it's listed in the sphinx directory. Any idea what the problem is? On 24 April 2018 at 21:51, Ralf Gommers wrote: > > > On Tue, Apr 24, 2018 at 8:40 AM, Eric Larson > wrote: > >> >>> 1. Since this project will include documentation, I'm trying to >>> build the docs. However, from the Makefile at scipy/doc it appears >>> that python3.6 is required, whereas only python3.5 is available for my >>> platform (Ubuntu 16.04 LTS). I generally try to avoid adding third-party >>> repos unless absolutely required, so I am asking for help here. Is >>> installing python3.6 the only solution, or will changing the PYVERS >>> variable in the Makefile work? >>> >>> I suspect 3.5 should work once you get it to use your Python binaries. I >> doubt we have anything that is Python-3.6 specific (though there very well >> be some Python 3-isms that will not work on 2.7). >> > > There's a PYVER=3.6 at the top of the Makefile. Simply doing `make html > PYVER=3.5` should build the docs with your 3.5 install (if not, change that > one line in the Makefile to say 3.5). > > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Apr 24 14:47:18 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 24 Apr 2018 11:47:18 -0700 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: References: Message-ID: On Tue, Apr 24, 2018 at 11:36 AM, Aditya Bharti wrote: > Regarding building the documentation, I was able to figure out that I > needed to install sphinx separately. The HACKING.rst.txt file only mentions > Numpy Cython and pytest as dependencies. > I was able to get the make command running with `make html PYVER=3.5`, but > I'm getting the follwing error: > > >> Could not import extension numpydoc (exception: cannot import name >> 'Directive') >> > > I'm sure I have the extension installed, since it's listed in the sphinx > directory. Any idea what the problem is? > There seems to be something wrong with your install of either Sphinx or numpydoc. If it's missing completely, the traceback says: Could not import extension numpydoc (exception: No module named 'numpydoc') rather than complaining about "Directive". Possibly a mixup with versions? Also note that you're saying that you see numpydoc in the sphinx directory - did you mean in site-packages instead? That's where it should be: In [1]: import numpydoc In [2]: numpydoc.__file__ Out[2]: '/home/rgommers/anaconda3/lib/python3.6/site-packages/numpydoc/__init__.py' Ralf > On 24 April 2018 at 21:51, Ralf Gommers wrote: > >> >> >> On Tue, Apr 24, 2018 at 8:40 AM, Eric Larson >> wrote: >> >>> >>>> 1. Since this project will include documentation, I'm trying to >>>> build the docs. However, from the Makefile at scipy/doc it appears >>>> that python3.6 is required, whereas only python3.5 is available for my >>>> platform (Ubuntu 16.04 LTS). I generally try to avoid adding third-party >>>> repos unless absolutely required, so I am asking for help here. Is >>>> installing python3.6 the only solution, or will changing the PYVERS >>>> variable in the Makefile work? >>>> >>>> I suspect 3.5 should work once you get it to use your Python binaries. >>> I doubt we have anything that is Python-3.6 specific (though there very >>> well be some Python 3-isms that will not work on 2.7). >>> >> >> There's a PYVER=3.6 at the top of the Makefile. Simply doing `make html >> PYVER=3.5` should build the docs with your 3.5 install (if not, change that >> one line in the Makefile to say 3.5). >> >> Ralf >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adibhar97 at gmail.com Tue Apr 24 15:28:15 2018 From: adibhar97 at gmail.com (Aditya Bharti) Date: Wed, 25 Apr 2018 00:58:15 +0530 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: References: Message-ID: > There seems to be something wrong with your install of either Sphinx or > numpydoc. If it's missing completely, the traceback says: > > Could not import extension numpydoc (exception: No module named > 'numpydoc') > > rather than complaining about "Directive". Possibly a mixup with versions? > Also note that you're saying that you see numpydoc in the sphinx directory > - did you mean in site-packages instead? That's where it should be: > You're right, I meant in the site-packages directory. For me it's here: '/home/aditya/scipy-dev/lib/python3.5/site-packages/numpydoc/__init__.py'. /home/aditya/scipy-dev is my virtualenv for scipy. It uses python3.5. I tried making the docs outside the virtual environment, I'm getting the same error. For reference, I'm using: - python 3.5.2 - Sphinx version 1.7.3 - numpydoc version 0.8.0 - scipy version 1.0.0 / version 1.2.0.dev0+3988d44 (virtualenv) -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagru at mailbox.org Tue Apr 24 18:10:33 2018 From: lagru at mailbox.org (Lars G.) Date: Wed, 25 Apr 2018 00:10:33 +0200 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: References: Message-ID: <19947416-eb05-b038-ae36-ed470fbe6127@mailbox.org> On 24.04.2018 20:36, Aditya Bharti wrote: > Regarding building the documentation, I was able to figure out that I > needed to install sphinx separately. The HACKING.rst.txt file only > mentions Numpy Cython and pytest as dependencies. > I was able to get the make command running with `make html PYVER=3.5`, > but I'm getting the follwing error: > ? > > Could not import extension numpydoc (exception: cannot import name > 'Directive') Hello Aditya, I had the same problem with my setup. I'm still a little bit confused but I think the error was due to an old version of numpydoc trying use an import that didn't work with new versions of sphinx. I think got the old version with the git-submodules included in the doc-dirctory (can be downloaded with `git submodule update`). Anyway I got it to work by modifying the faulty import itself. Try to find the file numpydoc/numpydoc.py and try to find the line: from sphinx.util.compat import Directive If you can find that line you may be in luck because replacing that line with: from docutils.parsers.rst import Directive fixed it for me. I hope this helps. By the way, perhaps it would be useful to extend the doc/README.txt with a more thorough explanation of the build process and mention that the submodules can be downloaded with `git submodule update`. I only found out because I looked at guide for NumPy https://github.com/numpy/numpy/blob/master/doc/source/docs/howto_build_docs.rst while trying to figure out the build process myself. Best regards, Lars From warren.weckesser at gmail.com Tue Apr 24 18:42:00 2018 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Tue, 24 Apr 2018 18:42:00 -0400 Subject: [SciPy-Dev] GSoC'18 accepted student announced In-Reply-To: References: Message-ID: On Tue, Apr 24, 2018 at 12:26 AM, Ralf Gommers wrote: > Hi all, > > Google announced the results of the GSoC application process. SciPy > received 1 slot - I'm happy to congratulate Aditya Bharti as our accepted > student for this year! And a big thanks to Nikolay Mayorov and Eric Larson > for stepping up to mentor this year! Looking forward to an interesting > project! > > Aditya's project: > Official summary: https://summerofcode.withgoogle.com/projects/# > 6542621229449216 > Full proposal (latest draft, final pdf is not public): > https://docs.google.com/document/d/1UriyLABwgjUcYfBofSr4hqAnLzblJ > 71pkzm7i26O8ws/edit > Co-mentors: Nikolay Mayorov, Eric Larson (I'm listed as mentor as well, > but that's mainly for admin reasons). > > The "community bonding" period has just started, let's all make sure to > welcome Aditya and help him where needed, so he can hit the ground running > from May 14th onwards! > > Congratulations, Aditya! Warren > Cheers, > Ralf > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.denaxas at gmail.com Wed Apr 25 04:20:47 2018 From: s.denaxas at gmail.com (Spiros Denaxas) Date: Wed, 25 Apr 2018 09:20:47 +0100 Subject: [SciPy-Dev] GSoC'18 accepted student announced In-Reply-To: References: Message-ID: On Tue, Apr 24, 2018 at 11:42 PM, Warren Weckesser < warren.weckesser at gmail.com> wrote: > > > On Tue, Apr 24, 2018 at 12:26 AM, Ralf Gommers > wrote: > >> Hi all, >> >> Google announced the results of the GSoC application process. SciPy >> received 1 slot - I'm happy to congratulate Aditya Bharti as our accepted >> student for this year! And a big thanks to Nikolay Mayorov and Eric Larson >> for stepping up to mentor this year! Looking forward to an interesting >> project! >> >> Aditya's project: >> Official summary: https://summerofcode.withgoogl >> e.com/projects/#6542621229449216 >> Full proposal (latest draft, final pdf is not public): >> https://docs.google.com/document/d/1UriyLABwgjUcYfBofSr4hqAn >> LzblJ71pkzm7i26O8ws/edit >> Co-mentors: Nikolay Mayorov, Eric Larson (I'm listed as mentor as well, >> but that's mainly for admin reasons). >> >> The "community bonding" period has just started, let's all make sure to >> welcome Aditya and help him where needed, so he can hit the ground running >> from May 14th onwards! >> >> > Congratulations Aditya..! Spiros -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Apr 26 00:39:06 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 25 Apr 2018 21:39:06 -0700 Subject: [SciPy-Dev] Allow non-canonical form in COO? In-Reply-To: References: Message-ID: On Mon, Apr 23, 2018 at 9:02 AM, Hameer Abbasi wrote: > Hello SciPy-Dev, > > I submitted a pull request to > pydata/sparse (I?m one of its > maintainers) about always keeping a COO array in canonical form. > > We (Matthew Rocklin, CJ Carey and I) were having a discussion there about > whether it makes sense to allow sparse COO arrays to be in non-canonical > order. In short, canonical form is: > > - Coordinates are sorted in lexographical order, i.e., the same order > they would appear in in a C-contiguous array. > - There are no duplicate coordinates allowed. > - There are no stored coordinates with a zero value. > > With this in mind, what are the SciPy developers? opinions on this? Does > allowing COO arrays to be in non-canonical form make sense? Feedback > welcome on the PR or the mailing list. > > Your PR as it is now makes sense to me. Thanks for keeping this list in the loop on important decisions about sparse arrays. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Apr 26 01:01:18 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 25 Apr 2018 22:01:18 -0700 Subject: [SciPy-Dev] GSoC'18 accepted student announced In-Reply-To: References: <162f610a608.27ae.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: On Tue, Apr 24, 2018 at 3:47 AM, Aditya Bharti wrote: > Hi all, > Thank you for all your help so far! Your feedback was invaluable and went > a long way towards the strength of my application. > > Let me introduce myself formally. My name is Aditya Bharti and I am > currently an undergraduate researcher in Computer Vision and Machine > Learning at the International Institute of Information Technology, > Hyderabad, India. I've been using numpy and scipy for over 3 years, and > python in general for longer. > > I'd also like to know more about the community in general. What sort of > areas are the members from? From my interviews and the proposal feedback, I > found the community here to be extremely diverse; people are working on all > sorts of things ranging from LIDAR to linear dynamical systems. I look > forward to working with scipy over the summer! > Probably every scientific discipline is represented here - the community is pretty large:) Ralf > > Regards, > > Aditya > > On 24 April 2018 at 10:41, Stefan van der Walt > wrote: > >> Hi Aditya, >> >> Congratulations and welcome! This looks like a fun and useful project, >> and I'm looking forward to seeing its development. >> >> Best regards, >> St?fan >> >> On April 23, 2018 21:27:09 Ralf Gommers wrote: >> >>> Hi all, >>> >>> Google announced the results of the GSoC application process. SciPy >>> received 1 slot - I'm happy to congratulate Aditya Bharti as our accepted >>> student for this year! And a big thanks to Nikolay Mayorov and Eric Larson >>> for stepping up to mentor this year! Looking forward to an interesting >>> project! >>> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Apr 26 01:23:31 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 25 Apr 2018 22:23:31 -0700 Subject: [SciPy-Dev] linalg.solve keyword deprecation In-Reply-To: References: <1524489837.2359.16.camel@iki.fi> Message-ID: On Mon, Apr 23, 2018 at 6:51 AM, wrote: > > > On Mon, Apr 23, 2018 at 9:23 AM, Pauli Virtanen wrote: > >> ma, 2018-04-23 kello 12:20 +0200, Ilhan Polat kirjoitti: >> [clip: solve(sym_pos=, debug=)] >> > Hence I've proposed to deprecate these in >> > https://github.com/scipy/scipy/pull/8715/files . Pauli also chimed in >> > and >> > mentioned that this might not be a good idea since this is a central >> > function and the benefits might not be worth the effort and backwards >> > compatibility problems. With Python2 dying, I think the backwards >> > compatibility part won't be such important problem anymore and the >> > benefit >> > is that we don't need to have such strange signature. >> >> This I think is relevant: >> >> http://blog.khinsen.net/posts/2017/11/22/stability-in-the-sc >> ipy-ecosystem-a-summary-of-the-discussion/ > > I think that that story suffers from a very narrow viewpoint and a lot of exaggeration, but it's still a good example of why we need solid reasons to deprecate code. > >> >> My own position is that any breakage should have good reasons behind >> it, and cosmetic reasons (function signature with some duplicated >> functionality) usually are not strong enough. >> >> Re: Python3 --- I think that Python 3 also breaks things should not be >> a permission to break more things. >> > +1 > > > I think even "cosmetic" (confusing signatures) should be cleaned up in the > long run, > This is useful in principle, but the gain has to balance the cost. In a case like this, a doc-only deprecation would be fine. It could then be cleaned up at some point if there's a SciPy 2.0 in some years (with deprecation in code now). otherwise they never go away unless there is a big break release. > But deprecation periods should be long, e.g. > 3 years. > Deprecations in code (so users see a deprecation warning) have a very specific meaning: the deprecated feature gets removed after 2 releases. Ralf > (I had to deprecate three functions in statsmodels because I had > misspelled the function names when writing them.) > > Josef > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From allanhaldane at gmail.com Sat Apr 28 14:00:57 2018 From: allanhaldane at gmail.com (Allan Haldane) Date: Sat, 28 Apr 2018 14:00:57 -0400 Subject: [SciPy-Dev] NumPy 1.14.3 released Message-ID: Hi All, I am pleased to announce the release of NumPy 1.14.3. This is a bugfix release for a few bugs reported following the 1.14.2 release: * np.lib.recfunctions.fromrecords accepts a list-of-lists, until 1.15 * In python2, float types use the new print style when printing to a file * style arg in "legacy" print mode now works for 0d arrays The Python versions supported in this release are 2.7 and 3.4 - 3.6. The Python 3.6 wheels available from PIP are built with Python 3.6.2 and should be compatible with all previous versions of Python 3.6. The source releases were cythonized with Cython 0.28.2. Contributors ============ A total of 6 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Allan Haldane * Charles Harris * Jonathan March + * Malcolm Smith + * Matti Picus * Pauli Virtanen Pull requests merged ==================== A total of 8 pull requests were merged for this release. * `#10862 `__: BUG: floating types should override tp_print (1.14 backport) * `#10905 `__: BUG: for 1.14 back-compat, accept list-of-lists in fromrecords * `#10947 `__: BUG: 'style' arg to array2string broken in legacy mode (1.14... * `#10959 `__: BUG: test, fix for missing flags['WRITEBACKIFCOPY'] key * `#10960 `__: BUG: Add missing underscore to prototype in check_embedded_lapack * `#10961 `__: BUG: Fix encoding regression in ma/bench.py (Issue #10868) * `#10962 `__: BUG: core: fix NPY_TITLE_KEY macro on pypy * `#10974 `__: BUG: test, fix PyArray_DiscardWritebackIfCopy... Cheers, Allan Haldane From adibhar97 at gmail.com Sun Apr 29 00:10:24 2018 From: adibhar97 at gmail.com (Aditya Bharti) Date: Sun, 29 Apr 2018 09:40:24 +0530 Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions In-Reply-To: <19947416-eb05-b038-ae36-ed470fbe6127@mailbox.org> References: <19947416-eb05-b038-ae36-ed470fbe6127@mailbox.org> Message-ID: So I finally got the docs built. Couple of points which I think might be helpful to people building docs in the future: - You need to have sphinx installed in your working environment (native installation, virtualenv, conda, whatever). I did not see sphinx listed as a dependency online. Maybe it should be included in the HACKING.rst.txt file? - You do *NOT* need to have numpydoc installed on your system. Irrespective of where it is installed, the build process only uses the numpydoc package present under the sphinxext directory. I wasted a lot of hours trying to find which version of numpydoc was being used as I had it installed both in my virtualenv, and native installation; neither of those had the faulty import line. - After using `git submodule update` to get the latest sphinxext directory and numpydoc installation, the faulty import line in file doc/sphinxext/numpydoc/numpydoc.py needs to be changed. From > from sphinx.util.compat import Directive > to > from docutils.parsers.rst import Directive > Thank you Lars for pointing that out. - Finally, and this was my fault entirely, I had my virtual environment running python version 3.5.2, but as python3 or python. Calling python3.5 (as the Makefile was doing when I set PYVER=3.5) called the python executable in /usr/bin/python3.5, which couldn't find the sphinx module itself (as I didn't install it along with my native installation). Setting PYVER=3, or PYTHON=/path/to/python/in/virtualenv fixed this. On 25 April 2018 at 03:40, Lars G. wrote: > On 24.04.2018 20:36, Aditya Bharti wrote: > > Regarding building the documentation, I was able to figure out that I > > needed to install sphinx separately. The HACKING.rst.txt file only > > mentions Numpy Cython and pytest as dependencies. > > I was able to get the make command running with `make html PYVER=3.5`, > > but I'm getting the follwing error: > > > > > > Could not import extension numpydoc (exception: cannot import name > > 'Directive') > > Hello Aditya, > > I had the same problem with my setup. I'm still a little bit confused > but I think the error was due to an old version of numpydoc trying use > an import that didn't work with new versions of sphinx. > I think got the old version with the git-submodules included in the > doc-dirctory (can be downloaded with `git submodule update`). > > Anyway I got it to work by modifying the faulty import itself. Try to > find the file numpydoc/numpydoc.py and try to find the line: > > from sphinx.util.compat import Directive > > If you can find that line you may be in luck because replacing that line > with: > > from docutils.parsers.rst import Directive > > fixed it for me. I hope this helps. > > By the way, perhaps it would be useful to extend the doc/README.txt with > a more thorough explanation of the build process and mention that the > submodules can be downloaded with `git submodule update`. I only found > out because I looked at guide for NumPy > https://github.com/numpy/numpy/blob/master/doc/source/ > docs/howto_build_docs.rst > while trying to figure out the build process myself. > > Best regards, > Lars > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Apr 29 01:45:50 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 28 Apr 2018 22:45:50 -0700 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> Message-ID: On Mon, Apr 2, 2018 at 11:50 AM, Warren Weckesser < warren.weckesser at gmail.com> wrote: > > > On Fri, Mar 30, 2018 at 8:17 PM, Ralf Gommers > wrote: > >> >> >> On Fri, Mar 30, 2018 at 12:03 PM, Eric Larson >> wrote: >> >>> Top-level module for them alone sounds overkill, and I'm not sure if >>>> discoverability alone is enough. >>>> >>> >>> Fine by me. And if we follow the idea that these should be added >>> sparingly, we can maintain discoverability without it growing out of >>> hand by populating the See Also sections of each function. >>> >> >> I agree with this, the 2 images and 1 ECG signal (to be added) that we >> have doesn't justify a top-level module. We don't want to grow more than >> the absolute minimum of datasets. The package is already very large, which >> is problematic in certain cases. E.g. numpy + scipy still fits in the AWS >> Lambda limit of 50 MB, but there's not much margin. >> >> > > Note: this is a reply to the thread, and not specifically to Ralf's > comments (but those are included). > > After reading all the replies, the first question that comes to mind is: > should SciPy have *any* datasets? > > I think this question has already been answered: we have had functions > that return images in scipy.misc for a long time, and I don't recall anyone > ever suggesting that these be removed. (Well, there was lena(), but I > don't think anyone had a problem with adding a replacement image.) And the > pull request for the ECG dataset has been merged (added to scipy.misc), so > there is current support among the developers for providing datasets. > > So the remaining questions are: > (1) Where do the datasets reside? > (2) What are the criteria for adding a new datasets? > > Here's my 2?: > > (1) Where do the datasets reside? > > My preference is to keep all the datasets in the top-level module > scipy.datasets. Robert preferred this module for discoverability, and I > agree. By having all the datasets in one place, anyone can easily see what > is available. Teachers and others developing educational material know > where to find source material for examples. Developers, too, can easily > look for examples to use in our docstrings or tutorials. (By the way, > adding examples to the docstrings of all functions is an ongoing effort: > https://github.com/scipy/scipy/issues/7168.) > > Also, there are many well-known datasets that could be used as examples > for multiple scipy packages. For a concrete example, a dataset that I > could see adding to scipy is the Hald cement dataset. SciPy should > eventually have an implementation of the PCA decomposition, and it could > conceivably live in scipy.linalg. It would be reasonable to use the Hald > data in the docstrings of the new PCA function(s) (cf. > https://www.mathworks.com/help/stats/pca.html). At the same time, the > Hald data could enrich the docstrings of some functions in scipy.stats. > > Similarly, Fisher's iris dataset provides a well-known example that could > be used in docstrings in both scipy.cluster and scipy.stats. > > > (2) What are the criteria for adding a new datasets? > > So far, the only compelling reason I can see to even have datasets is to > have interesting examples in the docstrings (or at least in our > tutorials). For example, the docstring for scipy.ndimage.gaussian_filter > and several other transformations in ndimage use the image returned by > scipy.misc.ascent(): > > https://docs.scipy.org/doc/scipy/reference/generated/ > scipy.ndimage.gaussian_filter.html > > I could see the benefit of having well-known datasets such as Fisher's > iris data, the Hald cement data, and some version of a sunspot activity > time series, to be used in the docstrings in scipy.stats, scipy.signal, > scipy.cluster, scipy.linalg, and elsewhere. > > St?fan expressed regret about including datasets in sciki-image. The main > issue seems to be "bloat". Scikit-image is an image processing library, so > the datasets used there are likely all images, and there is a minimum size > for a sample image to be useful as an example. For scipy, we already have > two images, and I don't think we'll need more. The newly added ECG dataset > is 116K (which is less than the existing image datasets: "ascent.dat" is > 515K and "face.dat" is 1.5M). The potential datasets that I mentioned > above (Hald, iris, sunspots) are all very small. If we are conservative > about what we include, and focus on datasets chosen specifically to > demonstrate scipy functionality, we should be able to avoid dataset bloat. > > This leads to my proposal for the criteria for adding a dataset: > > (a) Not too big. The size of a dataset should not exceed $MAX (but I > don't have a good suggestion for what $MAX should be at the moment). > (b) The dataset should be well-known, where "well-known" means that the > dataset is one that is already widely used as an example and many people > will know it by name (e.g. the iris dataset), or the dataset is a sample of > a common signal type or format (e.g an ECG signal, or an image such as > misc.ascent). > (c) We actually *use* the dataset in one of *our* docstrings or > tutorials. I don't think our datasets package should become a repository > of interesting scientific data with no connection to the scipy code. Its > purpose should be to enrich our documentation. (Note that by this > criterion, the recently added ECG signal would not qualify!) > I'd add the criterion that we should *only* use any dataset in the docs. Hence there are zero internal imports, and the whole datasets submodule can then very simply be stripped for space-constrained usage scenarios. (in those cases a separate package would help even) > > To summarize: I'm in favor scipy.datasets, a conservatively curated > subpackage containing well-known datasets. > This rationale and the implementation in https://github.com/scipy/scipy/pull/8707 is fairly convincing. I'll change my vote to +0.5 for scipy.datasets > > > Warren > > > P.S. I should add that I'm not in favor putting code in scipy that fetches > data from the web. That type of data retrieval could be useful, but it > seems more appropriate for a package that is independent of scipy. > +1 Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Apr 29 01:58:44 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 28 Apr 2018 22:58:44 -0700 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> Message-ID: On Tue, Apr 3, 2018 at 1:06 AM, Da?id wrote: > > > On 31 March 2018 at 02:17, Ralf Gommers wrote: > >> >> >> On Fri, Mar 30, 2018 at 12:03 PM, Eric Larson >> wrote: >> >>> Top-level module for them alone sounds overkill, and I'm not sure if >>>> discoverability alone is enough. >>>> >>> >>> Fine by me. And if we follow the idea that these should be added >>> sparingly, we can maintain discoverability without it growing out of >>> hand by populating the See Also sections of each function. >>> >> >> I agree with this, the 2 images and 1 ECG signal (to be added) that we >> have doesn't justify a top-level module. We don't want to grow more than >> the absolute minimum of datasets. The package is already very large, which >> is problematic in certain cases. E.g. numpy + scipy still fits in the AWS >> Lambda limit of 50 MB, but there's not much margin. >> > > The biggest subpackage is sparse, and there most of the space is taken by _ > sparsetools.cpython-35m-x86_64-linux-gnu.so According to size -A -d, the > biggest sections are debug. The same goes for the second biggest, special. > Can it run without those sections? On preliminary checks, it seems that > stripping .debug_info and .debug_loc trim down the size from 38 to 3.7 MB, > and the test suite still passes. > Should work. That's a lot more gain than I'd realized. Given that we hardly ever get useful gdb tracebacks, it may be worth considering doing that for releases. > > If we really need to trim down the size for installing in things like > Lambda, could we have a scipy-lite for production environments, that is the > same as scipy but without unnecessary debug? I imagine tracebacks would not > be as informative, but that shouldn't matter for production environments. > My first thought was to remove docstrings, comments, tests, and data, but > maybe they don't amount to so much for the trouble. > Recipes for such things are floating around, and it makes sense to do that. I'd rather not maintain an official scipy-lite package though, rather just make choices within scipy that enable third parties to do that. Ralf > > > On the topic at hand, I would agree to having a few, small datasets to > showcase functionality. I think a few kilobytes can go a long way to show > and benchmark. As far as I can see, a top level module is free: it wouldn't > add any maintenance burden, and would make them easier to find. > > /David. > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Apr 29 02:21:55 2018 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 29 Apr 2018 06:21:55 +0000 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> Message-ID: On Sat, Apr 28, 2018 at 10:46 PM Ralf Gommers wrote: > > On Mon, Apr 2, 2018 at 11:50 AM, Warren Weckesser < warren.weckesser at gmail.com> wrote: >> (c) We actually *use* the dataset in one of *our* docstrings or tutorials. I don't think our datasets package should become a repository of interesting scientific data with no connection to the scipy code. Its purpose should be to enrich our documentation. (Note that by this criterion, the recently added ECG signal would not qualify!) > > I'd add the criterion that we should *only* use any dataset in the docs. Hence there are zero internal imports, and the whole datasets submodule can then very simply be stripped for space-constrained usage scenarios. (in those cases a separate package would help even) I believe that one of the motivations for adding the ECG dataset was to make some of the scipy.signal unit tests more realistic. Is that something you'd like to forbid? On the one hand, if you're strapped for space, you probably want to remove the test suites as well. On the other hand, you do want to be able to test your stripped installation! -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Apr 29 02:41:39 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 28 Apr 2018 23:41:39 -0700 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> Message-ID: On Sat, Apr 28, 2018 at 11:21 PM, Robert Kern wrote: > On Sat, Apr 28, 2018 at 10:46 PM Ralf Gommers > wrote: > > > > On Mon, Apr 2, 2018 at 11:50 AM, Warren Weckesser < > warren.weckesser at gmail.com> wrote: > > >> (c) We actually *use* the dataset in one of *our* docstrings or > tutorials. I don't think our datasets package should become a repository > of interesting scientific data with no connection to the scipy code. Its > purpose should be to enrich our documentation. (Note that by this > criterion, the recently added ECG signal would not qualify!) > > > > I'd add the criterion that we should *only* use any dataset in the docs. > Hence there are zero internal imports, and the whole datasets submodule can > then very simply be stripped for space-constrained usage scenarios. (in > those cases a separate package would help even) > > I believe that one of the motivations for adding the ECG dataset was to > make some of the scipy.signal unit tests more realistic. Is that something > you'd like to forbid? On the one hand, if you're strapped for space, you > probably want to remove the test suites as well. On the other hand, you do > want to be able to test your stripped installation! > Hmm, tough question. Ideally I'd like to say yes, however we do need test data in some cases. In practice I think one would want to strip the test suite anyway; scipy/special/tests/data/*.npz is over 1 MB already. So let's say that importing from within tests is okay. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagru at mailbox.org Sun Apr 29 04:13:56 2018 From: lagru at mailbox.org (Lars G.) Date: Sun, 29 Apr 2018 10:13:56 +0200 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> Message-ID: <91f4e42e-be08-0dba-ae09-7ea9c480f09c@mailbox.org> On 29.04.2018 08:21, Robert Kern wrote: > I believe that one of the motivations for adding the ECG dataset was to > make some of the scipy.signal unit tests more realistic. Is that > something you'd like to forbid? On the one hand, if you're strapped for > space, you probably want to remove the test suites as well. On the other > hand, you do want to be able to test your stripped installation! Right now, the ECG dataset is not used in unit tests. Do you perhaps mean the benchmark suite? That could be another use case for datasets within SciPy (e.g #8769) Best regards, Lars From helmrp at yahoo.com Sun Apr 29 08:15:37 2018 From: helmrp at yahoo.com (The Helmbolds) Date: Sun, 29 Apr 2018 12:15:37 +0000 (UTC) Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 174, Issue 31 In-Reply-To: References: Message-ID: <1603656540.2113745.1525004137247@mail.yahoo.com> How much of the DATASETS issues could be handled simply by references in the documentation to where users can find those datasets that are generally considered both "standard" and potentially useful, without "physically" incorporating those datasets into SciPy? E.g, could the ECG dataset be handled that way? "You won't find the right answers if you don't ask the right questions!" (Robert Helmbold, 2013) On ?Saturday?, ?April? ?28?, ?2018? ?11?:?42?:?46? ?PM? ?MST, scipy-dev-request at python.org wrote: Send SciPy-Dev mailing list submissions to ??? scipy-dev at python.org To subscribe or unsubscribe via the World Wide Web, visit ??? https://mail.python.org/mailman/listinfo/scipy-dev or, via email, send a message with subject or body 'help' to ??? scipy-dev-request at python.org You can reach the person managing the list at ??? scipy-dev-owner at python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of SciPy-Dev digest..." Today's Topics: ? 1. Re: New subpackage: scipy.data (Ralf Gommers) ? 2. Re: New subpackage: scipy.data (Robert Kern) ? 3. Re: New subpackage: scipy.data (Ralf Gommers) ---------------------------------------------------------------------- Message: 1 Date: Sat, 28 Apr 2018 22:58:44 -0700 From: Ralf Gommers To: SciPy Developers List Subject: Re: [SciPy-Dev] New subpackage: scipy.data Message-ID: ??? Content-Type: text/plain; charset="utf-8" On Tue, Apr 3, 2018 at 1:06 AM, Da?id wrote: > > > On 31 March 2018 at 02:17, Ralf Gommers wrote: > >> >> >> On Fri, Mar 30, 2018 at 12:03 PM, Eric Larson >> wrote: >> >>> Top-level module for them alone sounds overkill, and I'm not sure if >>>> discoverability alone is enough. >>>> >>> >>> Fine by me. And if we follow the idea that these should be added >>> sparingly, we can maintain discoverability without it growing out of >>> hand by populating the See Also sections of each function. >>> >> >> I agree with this, the 2 images and 1 ECG signal (to be added) that we >> have doesn't justify a top-level module. We don't want to grow more than >> the absolute minimum of datasets. The package is already very large, which >> is problematic in certain cases. E.g. numpy + scipy still fits in the AWS >> Lambda limit of 50 MB, but there's not much margin. >> > > The biggest subpackage is sparse, and there most of the space is taken by _ > sparsetools.cpython-35m-x86_64-linux-gnu.so According to size -A -d, the > biggest sections are debug. The same goes for the second biggest, special. > Can it run without those sections? On preliminary checks, it seems that > stripping .debug_info and .debug_loc trim down the size from 38 to 3.7 MB, > and the test suite still passes. > Should work. That's a lot more gain than I'd realized. Given that we hardly ever get useful gdb tracebacks, it may be worth considering doing that for releases. > > If we really need to trim down the size for installing in things like > Lambda, could we have a scipy-lite for production environments, that is the > same as scipy but without unnecessary debug? I imagine tracebacks would not > be as informative, but that shouldn't matter for production environments. > My first thought was to remove docstrings, comments, tests, and data, but > maybe they don't amount to so much for the trouble. > Recipes for such things are floating around, and it makes sense to do that. I'd rather not maintain an official scipy-lite package though, rather just make choices within scipy that enable third parties to do that. Ralf > > > On the topic at hand, I would agree to having a few, small datasets to > showcase functionality. I think a few kilobytes can go a long way to show > and benchmark. As far as I can see, a top level module is free: it wouldn't > add any maintenance burden, and would make them easier to find. > > /David. > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 2 Date: Sun, 29 Apr 2018 06:21:55 +0000 From: Robert Kern To: SciPy Developers List Subject: Re: [SciPy-Dev] New subpackage: scipy.data Message-ID: ??? Content-Type: text/plain; charset="utf-8" On Sat, Apr 28, 2018 at 10:46 PM Ralf Gommers wrote: > > On Mon, Apr 2, 2018 at 11:50 AM, Warren Weckesser < warren.weckesser at gmail.com> wrote: >> (c) We actually *use* the dataset in one of *our* docstrings or tutorials.? I don't think our datasets package should become a repository of interesting scientific data with no connection to the scipy code.? Its purpose should be to enrich our documentation.? (Note that by this criterion, the recently added ECG signal would not qualify!) > > I'd add the criterion that we should *only* use any dataset in the docs. Hence there are zero internal imports, and the whole datasets submodule can then very simply be stripped for space-constrained usage scenarios. (in those cases a separate package would help even) I believe that one of the motivations for adding the ECG dataset was to make some of the scipy.signal unit tests more realistic. Is that something you'd like to forbid? On the one hand, if you're strapped for space, you probably want to remove the test suites as well. On the other hand, you do want to be able to test your stripped installation! -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Sat, 28 Apr 2018 23:41:39 -0700 From: Ralf Gommers To: SciPy Developers List Subject: Re: [SciPy-Dev] New subpackage: scipy.data Message-ID: ??? Content-Type: text/plain; charset="utf-8" On Sat, Apr 28, 2018 at 11:21 PM, Robert Kern wrote: > On Sat, Apr 28, 2018 at 10:46 PM Ralf Gommers > wrote: > > > > On Mon, Apr 2, 2018 at 11:50 AM, Warren Weckesser < > warren.weckesser at gmail.com> wrote: > > >> (c) We actually *use* the dataset in one of *our* docstrings or > tutorials.? I don't think our datasets package should become a repository > of interesting scientific data with no connection to the scipy code.? Its > purpose should be to enrich our documentation.? (Note that by this > criterion, the recently added ECG signal would not qualify!) > > > > I'd add the criterion that we should *only* use any dataset in the docs. > Hence there are zero internal imports, and the whole datasets submodule can > then very simply be stripped for space-constrained usage scenarios. (in > those cases a separate package would help even) > > I believe that one of the motivations for adding the ECG dataset was to > make some of the scipy.signal unit tests more realistic. Is that something > you'd like to forbid? On the one hand, if you're strapped for space, you > probably want to remove the test suites as well. On the other hand, you do > want to be able to test your stripped installation! > Hmm, tough question. Ideally I'd like to say yes, however we do need test data in some cases. In practice I think one would want to strip the test suite anyway; scipy/special/tests/data/*.npz is over 1 MB already. So let's say that importing from within tests is okay. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Subject: Digest Footer _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev ------------------------------ End of SciPy-Dev Digest, Vol 174, Issue 31 ****************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun Apr 29 11:51:45 2018 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 29 Apr 2018 17:51:45 +0200 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> Message-ID: <458af4f104cf620ffc76b6b4abd1e32431df2aea.camel@iki.fi> la, 2018-04-28 kello 22:58 -0700, Ralf Gommers kirjoitti: [clip] > > The biggest subpackage is sparse, and there most of the space is > > taken by _ > > sparsetools.cpython-35m-x86_64-linux-gnu.so According to size -A > > -d, the > > biggest sections are debug. The same goes for the second biggest, > > special. > > Can it run without those sections? On preliminary checks, it seems > > that > > stripping .debug_info and .debug_loc trim down the size from 38 to > > 3.7 MB, > > and the test suite still passes. > > > > Should work. That's a lot more gain than I'd realized. Given that we > hardly > ever get useful gdb tracebacks, it may be worth considering doing > that for > releases. I thought we already enabled stripping debug symbols for 1.1.0rc1, https://github.com/MacPython/scipy-wheels/pull/26 but this indeed doesn't seem to be the case. There's still time to fix it for 1.1.0 if we figure it out soon. Pauli From pav at iki.fi Sun Apr 29 13:13:20 2018 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 29 Apr 2018 19:13:20 +0200 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: <458af4f104cf620ffc76b6b4abd1e32431df2aea.camel@iki.fi> References: <1522425978.9176.106.camel@iki.fi> <458af4f104cf620ffc76b6b4abd1e32431df2aea.camel@iki.fi> Message-ID: <77f6f78a36a7cc732750c19f0e317cf9378370c0.camel@iki.fi> su, 2018-04-29 kello 17:51 +0200, Pauli Virtanen kirjoitti: [clip] > I thought we already enabled stripping debug symbols for 1.1.0rc1, > https://github.com/MacPython/scipy-wheels/pull/26 > but this indeed doesn't seem to be the case. > > There's still time to fix it for 1.1.0 if we figure it out soon. Ok, I was looking at wrong files. This is indeed already fixed in 1.1.0rc1, so no further action needed: $ wget https://files.pythonhosted.org/packages/d3/06/1517f6b3fbdbefaa7ccf31628d3e19530d058aea4abeb6685dcb3dce1733/scipy-1.1.0rc1-cp35-cp35m-manylinux1_x86_64.whl $ unzip scipy-1.1.0rc1-cp35-cp35m-manylinux1_x86_64.whl $ du -csh scipy/sparse/_sparsetools*.so 3,3M scipy/sparse/_sparsetools.cpython-35m-x86_64-linux-gnu.so $ size -A -d scipy/sparse/_sparsetools*.so scipy/sparse/_sparsetools.cpython-35m-x86_64-linux-gnu.so : section size addr .note.gnu.build-id 36 400 .gnu.hash 36 440 .dynsym 1464 480 .dynstr 1203 1944 .gnu.version 122 3148 .gnu.version_r 144 3272 .rela.dyn 4200 3416 .rela.plt 864 7616 .init 14 8480 .plt 592 8496 .text 3037580 9088 .fini 9 3046668 .rodata 5132 3046688 .eh_frame_hdr 21956 3051820 .eh_frame 261124 3073776 .gcc_except_table 52170 3334900 .init_array 8 5484544 .fini_array 8 5484552 .jcr 8 5484560 .data.rel.ro 8 5484568 .dynamic 512 5484576 .got 168 5485088 .got.plt 312 5485256 .data 2520 5485568 .bss 24 5488096 .comment 137 0 Total 3390351 Pauli From ralf.gommers at gmail.com Sun Apr 29 14:21:52 2018 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 29 Apr 2018 11:21:52 -0700 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: <77f6f78a36a7cc732750c19f0e317cf9378370c0.camel@iki.fi> References: <1522425978.9176.106.camel@iki.fi> <458af4f104cf620ffc76b6b4abd1e32431df2aea.camel@iki.fi> <77f6f78a36a7cc732750c19f0e317cf9378370c0.camel@iki.fi> Message-ID: On Sun, Apr 29, 2018 at 10:13 AM, Pauli Virtanen wrote: > su, 2018-04-29 kello 17:51 +0200, Pauli Virtanen kirjoitti: > [clip] > > I thought we already enabled stripping debug symbols for 1.1.0rc1, > > https://github.com/MacPython/scipy-wheels/pull/26 > > but this indeed doesn't seem to be the case. > > > > There's still time to fix it for 1.1.0 if we figure it out soon. > > Ok, I was looking at wrong files. This is indeed already fixed in > 1.1.0rc1, so no further action needed: > Interesting, I had managed to miss or forget about that. Looks like it's only partially done though; a quick check on the wheel linked below shows that the unzipped wheel is still 113 MB and further gains are possible. E.g. $ ls -l cython_special.cpython-35m-x86_64-linux-gnu.so ... 10694424 Apr 15 15:04 cython_special.cpython-35m-x86_64-linux-gnu.so $ strip cython_special.cpython-35m-x86_64-linux-gnu.so $ ls -l cython_special.cpython-35m-x86_64-linux-gnu.so ... 3065680 Apr 29 11:16 cython_special.cpython-35m-x86_64-linux-gnu.so $ cd ../linalg $ ls -l _flapack.cpython-35m-x86_64-linux-gnu.so ... 3445392 Apr 15 15:04 _flapack.cpython-35m-x86_64-linux-gnu.so $ strip _flapack.cpython-35m-x86_64-linux-gnu.so $ ls -l _flapack.cpython-35m-x86_64-linux-gnu.so ... 1333960 Apr 29 11:17 _flapack.cpython-35m-x86_64-linux-gnu.so I'll open an issue for that. Ralf > $ wget https://files.pythonhosted.org/packages/d3/06/ > 1517f6b3fbdbefaa7ccf31628d3e19530d058aea4abeb6685dcb3dce1733 > /scipy-1.1.0rc1-cp35-cp35m-manylinux1_x86_64.whl > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sun Apr 29 15:25:17 2018 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 29 Apr 2018 21:25:17 +0200 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> <458af4f104cf620ffc76b6b4abd1e32431df2aea.camel@iki.fi> <77f6f78a36a7cc732750c19f0e317cf9378370c0.camel@iki.fi> Message-ID: su, 2018-04-29 kello 11:21 -0700, Ralf Gommers kirjoitti: > $ ls -l cython_special.cpython-35m-x86_64-linux-gnu.so > ... 10694424 Apr 15 15:04 cython_special.cpython-35m-x86_64-linux- > gnu.so > $ strip cython_special.cpython-35m-x86_64-linux-gnu.so > $ ls -l cython_special.cpython-35m-x86_64-linux-gnu.so > ... 3065680 Apr 29 11:16 cython_special.cpython-35m-x86_64-linux- > gnu.so > $ cd ../linalg > $ ls -l _flapack.cpython-35m-x86_64-linux-gnu.so > ... 3445392 Apr 15 15:04 _flapack.cpython-35m-x86_64-linux-gnu.so > $ strip _flapack.cpython-35m-x86_64-linux-gnu.so > $ ls -l _flapack.cpython-35m-x86_64-linux-gnu.so > ... 1333960 Apr 29 11:17 _flapack.cpython-35m-x86_64-linux-gnu.so > > I'll open an issue for that. Right, the link commands appear different: /opt/rh/devtoolset-2/root/usr/bin/gfortran -Wall -g -Wall -g -shared build/temp.linux-x86_64-3.6/scipy/special/cython_special.o build/temp.linux-x86_64-3.6/scipy/special/sf_error.o build/temp.linux-x86_64-3.6/build/src.linux-x86_64-3.6/scipy/special/_logit.o build/temp.linux-x86_64-3.6/scipy/special/amos_wrappers.o build/temp.linux-x86_64-3.6/scipy/special/cdf_wrappers.o build/temp.linux-x86_64-3.6/scipy/special/specfun_wrappers.o -L/usr/local/lib -L/opt/_internal/cpython-3.6.4/lib/python3.6/site-packages/numpy/core/lib -Lbuild/temp.linux-x86_64-3.6 -lopenblas -lopenblas -lsc_amos -lsc_c_misc -lsc_cephes -lsc_mach -lsc_cdf -lsc_specfun -lnpymath -lm -lgfortran -o build/lib.linux-x86_64-3.6/scipy/special/cython_special.cpython-36m-x86_64-linux-gnu.so -Wl,--version-script=build/temp.linux-x86_64-3.6/link-version-scipy.special.cython_special.map vs. gcc -pthread -shared -Wl,-strip-all -L/usr/local/include build/temp.linux-x86_64-3.6/scipy/special/_comb.o -Lbuild/temp.linux-x86_64-3.6 -o build/lib.linux-x86_64-3.6/scipy/special/_comb.cpython-36m-x86_64-linux-gnu.so -Wl,--version-script=build/temp.linux-x86_64-3.6/link-version-scipy.special._comb.map I guess this is some setuptools issue, with fortran and c/c++ handled differently. FFLAGS is supposed to be set in config.sh, but maybe they are not used for linking. Maybe the strip flags should be added to LDFLAGS too. Pauli From mikofski at berkeley.edu Sun Apr 29 18:44:30 2018 From: mikofski at berkeley.edu (Mark Alexander Mikofski) Date: Sun, 29 Apr 2018 22:44:30 +0000 Subject: [SciPy-Dev] Vectorized Newton method PR 8357 Message-ID: Hi, Does anyone have any comments on PR 8357 for a vectorized Newton method? https://github.com/scipy/scipy/pull/8357 I believe it is nearly ready to be merged, but there haven't been any new comments on it in about 3 weeks. I know the SciPy maintainers are volunteering their free time, which I greatly appreciate. So I understand that it may take a long time, but I was just wondering if there was any news on it from any of the maintainers. I have spent about 3 months on it addressing many reviewers' suggestions, and I think it is in very good shape. It should improve performance for vectorized functions considerably while having almost no impact on current users' scalar functions. Several users have indicated that PR 8357 will improve their projects, and this PR also closes two outstanding issues. There are already docs, tests, and benchmarks, but if there are still more concerns I am happy to address them. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Apr 29 23:10:15 2018 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 30 Apr 2018 03:10:15 +0000 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: <91f4e42e-be08-0dba-ae09-7ea9c480f09c@mailbox.org> References: <1522425978.9176.106.camel@iki.fi> <91f4e42e-be08-0dba-ae09-7ea9c480f09c@mailbox.org> Message-ID: On Sun, Apr 29, 2018 at 1:14 AM Lars G. wrote: > > On 29.04.2018 08:21, Robert Kern wrote: > > I believe that one of the motivations for adding the ECG dataset was to > > make some of the scipy.signal unit tests more realistic. Is that > > something you'd like to forbid? On the one hand, if you're strapped for > > space, you probably want to remove the test suites as well. On the other > > hand, you do want to be able to test your stripped installation! > > Right now, the ECG dataset is not used in unit tests. Do you perhaps > mean the benchmark suite? Yes, I misremembered the comment about making the benchmarks more realistic as making the tests more realistic. Nonetheless, my point remains: a fairly reasonable motivation for have datasets are the unit tests, as statsmodels does. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Sun Apr 29 23:21:09 2018 From: andyfaff at gmail.com (Andrew Nelson) Date: Mon, 30 Apr 2018 13:21:09 +1000 Subject: [SciPy-Dev] New subpackage: scipy.data In-Reply-To: References: <1522425978.9176.106.camel@iki.fi> <91f4e42e-be08-0dba-ae09-7ea9c480f09c@mailbox.org> Message-ID: There's lots of possible datasets that one can add. For example, it might be worthwhile providing the NIST Statistical Reference Datasets, such as those for non-linear regression. On 30 April 2018 at 13:10, Robert Kern wrote: > On Sun, Apr 29, 2018 at 1:14 AM Lars G. wrote: > > > > On 29.04.2018 08:21, Robert Kern wrote: > > > I believe that one of the motivations for adding the ECG dataset was to > > > make some of the scipy.signal unit tests more realistic. Is that > > > something you'd like to forbid? On the one hand, if you're strapped for > > > space, you probably want to remove the test suites as well. On the > other > > > hand, you do want to be able to test your stripped installation! > > > > Right now, the ECG dataset is not used in unit tests. Do you perhaps > > mean the benchmark suite? > > Yes, I misremembered the comment about making the benchmarks more > realistic as making the tests more realistic. Nonetheless, my point > remains: a fairly reasonable motivation for have datasets are the unit > tests, as statsmodels does. > > -- > Robert Kern > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -- _____________________________________ Dr. Andrew Nelson _____________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: