From charlesr.harris at gmail.com Sun Jan 1 19:04:08 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 1 Jan 2017 17:04:08 -0700 Subject: [SciPy-Dev] NumPy 1.12.0rc2 release. Message-ID: Hi All, I'm pleased to announce the NumPy 1.12.0rc2 New Year's release. This release supports Python 2.7 and 3.4-3.6. Wheels for all supported Python versions may be downloaded from PiPY , the tarball and zip files may be downloaded from Github . The release notes and files hashes may also be found at Github . NumPy 1.12.0rc 2 is the result of 413 pull requests submitted by 139 contributors and comprises a large number of fixes and improvements. Among the many improvements it is difficult to pick out just a few as standing above the others, but the following may be of particular interest or indicate areas likely to have future consequences. * Order of operations in ``np.einsum`` can now be optimized for large speed improvements. * New ``signature`` argument to ``np.vectorize`` for vectorizing with core dimensions. * The ``keepdims`` argument was added to many functions. * New context manager for testing warnings * Support for BLIS in numpy.distutils * Much improved support for PyPy (not yet finished) Enjoy, Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jan 4 18:10:08 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 5 Jan 2017 12:10:08 +1300 Subject: [SciPy-Dev] new foreign function interface merged Message-ID: Hi all, I think this deserves some advertising: Pauli wrote a new unified foreign function interface, which was just merged [1]. It's exposed as scipy.LowLevelCallable [2] and will be part of the 0.19.0 release. [1] https://github.com/scipy/scipy/pull/6509 [2] http://scipy.github.io/devdocs/ccallback.html Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Jan 5 23:58:49 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 6 Jan 2017 17:58:49 +1300 Subject: [SciPy-Dev] adding more window functions to scipy.signal Message-ID: Hi all, There are 3 PRs open for adding new window functions to scipy.signal: - Kaiser-Bessel, https://github.com/scipy/scipy/pull/4747 - Planck, https://github.com/scipy/scipy/pull/6012 (also interest in adding Planck-Bessel there) - Lanczos, https://github.com/scipy/scipy/pull/6837 So far Kaiser-Bessel and Lanczos got a positive reception, Planck didn't because it looks too specific - only uses found are in gravitational wave research. One of the annoyances with the window functions is that they really clutter the signal namespace as we keep on adding more. We have two options I think: 1. Keep on adding new window functions with the current style (provided that they're general enough, i.e. used in more than one field). 2. Only add them behind a unified interface, like signal.get_window. Could also be a new interface, not everyone is a fan of get_window (see https://github.com/scipy/scipy/pull/6837). Opinions? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jan 6 00:07:44 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 5 Jan 2017 21:07:44 -0800 Subject: [SciPy-Dev] adding more window functions to scipy.signal In-Reply-To: References: Message-ID: On Thu, Jan 5, 2017 at 8:58 PM, Ralf Gommers wrote: > Hi all, > > There are 3 PRs open for adding new window functions to scipy.signal: > - Kaiser-Bessel, https://github.com/scipy/scipy/pull/4747 > - Planck, https://github.com/scipy/scipy/pull/6012 (also interest in adding > Planck-Bessel there) > - Lanczos, https://github.com/scipy/scipy/pull/6837 > > So far Kaiser-Bessel and Lanczos got a positive reception, Planck didn't > because it looks too specific - only uses found are in gravitational wave > research. > > One of the annoyances with the window functions is that they really clutter > the signal namespace as we keep on adding more. We have two options I think: > > 1. Keep on adding new window functions with the current style (provided that > they're general enough, i.e. used in more than one field). > 2. Only add them behind a unified interface, like signal.get_window. Could > also be a new interface, not everyone is a fan of get_window (see > https://github.com/scipy/scipy/pull/6837). Not sure what you're pointing to with that link, but isn't get_window basically reinventing the concept of the python namespace? Maybe declare scipy.signal.windows (which already exists!) to be the public namespace where you find your windows? -n -- Nathaniel J. Smith -- https://vorpus.org From ralf.gommers at gmail.com Fri Jan 6 00:27:01 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 6 Jan 2017 18:27:01 +1300 Subject: [SciPy-Dev] adding more window functions to scipy.signal In-Reply-To: References: Message-ID: On Fri, Jan 6, 2017 at 6:07 PM, Nathaniel Smith wrote: > On Thu, Jan 5, 2017 at 8:58 PM, Ralf Gommers > wrote: > > Hi all, > > > > There are 3 PRs open for adding new window functions to scipy.signal: > > - Kaiser-Bessel, https://github.com/scipy/scipy/pull/4747 > > - Planck, https://github.com/scipy/scipy/pull/6012 (also interest in > adding > > Planck-Bessel there) > > - Lanczos, https://github.com/scipy/scipy/pull/6837 > > > > So far Kaiser-Bessel and Lanczos got a positive reception, Planck didn't > > because it looks too specific - only uses found are in gravitational wave > > research. > > > > One of the annoyances with the window functions is that they really > clutter > > the signal namespace as we keep on adding more. We have two options I > think: > > > > 1. Keep on adding new window functions with the current style (provided > that > > they're general enough, i.e. used in more than one field). > > 2. Only add them behind a unified interface, like signal.get_window. > Could > > also be a new interface, not everyone is a fan of get_window (see > > https://github.com/scipy/scipy/pull/6837). > > Not sure what you're pointing to with that link, Sorry, copy-paste error - should be https://github.com/scipy/scipy/pull/6895 > but isn't get_window > basically reinventing the concept of the python namespace? Maybe > declare scipy.signal.windows (which already exists!) to be the public > namespace where you find your windows? > Not really. For example, from gh-6895, this (which uses get_window): > signal.firwin(80, 0.5, window=('kaiser', 8)) is equivalent to: signal.firwin(80, 0.5, window=functools.partial(signal.kaiser, beta=8)) Despite the arguments in gh-6895, I'm quite sure that most users prefer not to have to know about functools.partial. Agreed that get_window is a bit ugly though ... Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jan 6 00:42:31 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 5 Jan 2017 21:42:31 -0800 Subject: [SciPy-Dev] adding more window functions to scipy.signal In-Reply-To: References: Message-ID: On Thu, Jan 5, 2017 at 9:27 PM, Ralf Gommers wrote: > > > On Fri, Jan 6, 2017 at 6:07 PM, Nathaniel Smith wrote: >> >> On Thu, Jan 5, 2017 at 8:58 PM, Ralf Gommers >> wrote: >> > Hi all, >> > >> > There are 3 PRs open for adding new window functions to scipy.signal: >> > - Kaiser-Bessel, https://github.com/scipy/scipy/pull/4747 >> > - Planck, https://github.com/scipy/scipy/pull/6012 (also interest in >> > adding >> > Planck-Bessel there) >> > - Lanczos, https://github.com/scipy/scipy/pull/6837 >> > >> > So far Kaiser-Bessel and Lanczos got a positive reception, Planck didn't >> > because it looks too specific - only uses found are in gravitational >> > wave >> > research. >> > >> > One of the annoyances with the window functions is that they really >> > clutter >> > the signal namespace as we keep on adding more. We have two options I >> > think: >> > >> > 1. Keep on adding new window functions with the current style (provided >> > that >> > they're general enough, i.e. used in more than one field). >> > 2. Only add them behind a unified interface, like signal.get_window. >> > Could >> > also be a new interface, not everyone is a fan of get_window (see >> > https://github.com/scipy/scipy/pull/6837). >> >> Not sure what you're pointing to with that link, > > > Sorry, copy-paste error - should be https://github.com/scipy/scipy/pull/6895 > >> >> but isn't get_window >> basically reinventing the concept of the python namespace? Maybe >> declare scipy.signal.windows (which already exists!) to be the public >> namespace where you find your windows? > > >> Not really. For example, from gh-6895, this (which uses get_window): > > > signal.firwin(80, 0.5, window=('kaiser', 8)) > > is equivalent to: > > signal.firwin(80, 0.5, window=functools.partial(signal.kaiser, beta=8)) > > Despite the arguments in gh-6895, I'm quite sure that most users prefer not > to have to know about functools.partial. ...maybe? It's certainly a thing to learn, but so is the current idiosyncratic mix of strings and functions and tuples, and at least partial is a reusable tool? Not-really-serious suggestion: # Add this line to scipy/signal/__init__.py, then users don't have to learn functools.partial from functools import partial as make_window Maybe-serious suggestion: If the problem is that we need a way to refer to abstractions like "a hamming window (with some to-be-determined number of taps)" or "a kaiser window with beta=8 (and some to-be-determined number of taps)", maybe instead of using these ad hoc strings/tuples/etc. we should make these objects. scipy.signals.windows.Hamming(), scipy.signals.windows.Kaiser(beta=8), ...? -n -- Nathaniel J. Smith -- https://vorpus.org From gael.varoquaux at normalesup.org Fri Jan 6 01:01:04 2017 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 6 Jan 2017 07:01:04 +0100 Subject: [SciPy-Dev] adding more window functions to scipy.signal In-Reply-To: References: Message-ID: <20170106060104.GW250062@phare.normalesup.org> On Thu, Jan 05, 2017 at 09:07:44PM -0800, Nathaniel Smith wrote: > Not sure what you're pointing to with that link, but isn't get_window > basically reinventing the concept of the python namespace? Maybe > declare scipy.signal.windows (which already exists!) to be the public > namespace where you find your windows? That were my thoughts exactly. get_windows brings interesting functionality, but the discoverability isn't great. Namespaces bring in tab completion. They are one honking great idea! Ga?l From helmrp at yahoo.com Fri Jan 6 12:07:28 2017 From: helmrp at yahoo.com (The Helmbolds) Date: Fri, 6 Jan 2017 17:07:28 +0000 (UTC) Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 159, Issue 3 In-Reply-To: References: Message-ID: <178423966.1267315.1483722448566@mail.yahoo.com> What if I want to try three or four windows to the same data stream?Can't I specify a number of windows to be tried one right after the other?Perhaps specified as a tuple of window identifiers??"You won't find the right answers if you don't ask the right questions!" (Robert Helmbold, 2013) From: "scipy-dev-request at scipy.org" To: scipy-dev at scipy.org Sent: Thursday, January 5, 2017 11:01 PM Subject: SciPy-Dev Digest, Vol 159, Issue 3 Send SciPy-Dev mailing list submissions to ??? scipy-dev at scipy.org To subscribe or unsubscribe via the World Wide Web, visit ??? https://mail.scipy.org/mailman/listinfo/scipy-dev or, via email, send a message with subject or body 'help' to ??? scipy-dev-request at scipy.org You can reach the person managing the list at ??? scipy-dev-owner at scipy.org When replying, please edit your Subject line so it is more specific than "Re: Contents of SciPy-Dev digest..." Today's Topics: ? 1. adding more window functions to scipy.signal (Ralf Gommers) ? 2. Re: adding more window functions to scipy.signal (Nathaniel Smith) ? 3. Re: adding more window functions to scipy.signal (Ralf Gommers) ? 4. Re: adding more window functions to scipy.signal (Nathaniel Smith) ? 5. Re: adding more window functions to scipy.signal (Gael Varoquaux) ---------------------------------------------------------------------- Message: 1 Date: Fri, 6 Jan 2017 17:58:49 +1300 From: Ralf Gommers To: SciPy Developers List Subject: [SciPy-Dev] adding more window functions to scipy.signal Message-ID: ??? Content-Type: text/plain; charset="utf-8" Hi all, There are 3 PRs open for adding new window functions to scipy.signal: - Kaiser-Bessel, https://github.com/scipy/scipy/pull/4747 - Planck, https://github.com/scipy/scipy/pull/6012 (also interest in adding Planck-Bessel there) - Lanczos, https://github.com/scipy/scipy/pull/6837 So far Kaiser-Bessel and Lanczos got a positive reception, Planck didn't because it looks too specific - only uses found are in gravitational wave research. One of the annoyances with the window functions is that they really clutter the signal namespace as we keep on adding more. We have two options I think: 1. Keep on adding new window functions with the current style (provided that they're general enough, i.e. used in more than one field). 2. Only add them behind a unified interface, like signal.get_window. Could also be a new interface, not everyone is a fan of get_window (see https://github.com/scipy/scipy/pull/6837). Opinions? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 2 Date: Thu, 5 Jan 2017 21:07:44 -0800 From: Nathaniel Smith To: SciPy Developers List Subject: Re: [SciPy-Dev] adding more window functions to scipy.signal Message-ID: ??? Content-Type: text/plain; charset=UTF-8 On Thu, Jan 5, 2017 at 8:58 PM, Ralf Gommers wrote: > Hi all, > > There are 3 PRs open for adding new window functions to scipy.signal: > - Kaiser-Bessel, https://github.com/scipy/scipy/pull/4747 > - Planck, https://github.com/scipy/scipy/pull/6012 (also interest in adding > Planck-Bessel there) > - Lanczos, https://github.com/scipy/scipy/pull/6837 > > So far Kaiser-Bessel and Lanczos got a positive reception, Planck didn't > because it looks too specific - only uses found are in gravitational wave > research. > > One of the annoyances with the window functions is that they really clutter > the signal namespace as we keep on adding more. We have two options I think: > > 1. Keep on adding new window functions with the current style (provided that > they're general enough, i.e. used in more than one field). > 2. Only add them behind a unified interface, like signal.get_window. Could > also be a new interface, not everyone is a fan of get_window (see > https://github.com/scipy/scipy/pull/6837). Not sure what you're pointing to with that link, but isn't get_window basically reinventing the concept of the python namespace? Maybe declare scipy.signal.windows (which already exists!) to be the public namespace where you find your windows? -n -- Nathaniel J. Smith -- https://vorpus.org ------------------------------ Message: 3 Date: Fri, 6 Jan 2017 18:27:01 +1300 From: Ralf Gommers To: SciPy Developers List Subject: Re: [SciPy-Dev] adding more window functions to scipy.signal Message-ID: ??? Content-Type: text/plain; charset="utf-8" On Fri, Jan 6, 2017 at 6:07 PM, Nathaniel Smith wrote: > On Thu, Jan 5, 2017 at 8:58 PM, Ralf Gommers > wrote: > > Hi all, > > > > There are 3 PRs open for adding new window functions to scipy.signal: > > - Kaiser-Bessel, https://github.com/scipy/scipy/pull/4747 > > - Planck, https://github.com/scipy/scipy/pull/6012 (also interest in > adding > > Planck-Bessel there) > > - Lanczos, https://github.com/scipy/scipy/pull/6837 > > > > So far Kaiser-Bessel and Lanczos got a positive reception, Planck didn't > > because it looks too specific - only uses found are in gravitational wave > > research. > > > > One of the annoyances with the window functions is that they really > clutter > > the signal namespace as we keep on adding more. We have two options I > think: > > > > 1. Keep on adding new window functions with the current style (provided > that > > they're general enough, i.e. used in more than one field). > > 2. Only add them behind a unified interface, like signal.get_window. > Could > > also be a new interface, not everyone is a fan of get_window (see > > https://github.com/scipy/scipy/pull/6837). > > Not sure what you're pointing to with that link, Sorry, copy-paste error - should be https://github.com/scipy/scipy/pull/6895 > but isn't get_window > basically reinventing the concept of the python namespace? Maybe > declare scipy.signal.windows (which already exists!) to be the public > namespace where you find your windows? > Not really. For example, from gh-6895, this (which uses get_window): > ? ? signal.firwin(80, 0.5, window=('kaiser', 8)) is equivalent to: ? signal.firwin(80, 0.5, window=functools.partial(signal.kaiser, beta=8)) Despite the arguments in gh-6895, I'm quite sure that most users prefer not to have to know about functools.partial. Agreed that get_window is a bit ugly though ... Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 4 Date: Thu, 5 Jan 2017 21:42:31 -0800 From: Nathaniel Smith To: SciPy Developers List Subject: Re: [SciPy-Dev] adding more window functions to scipy.signal Message-ID: ??? Content-Type: text/plain; charset=UTF-8 On Thu, Jan 5, 2017 at 9:27 PM, Ralf Gommers wrote: > > > On Fri, Jan 6, 2017 at 6:07 PM, Nathaniel Smith wrote: >> >> On Thu, Jan 5, 2017 at 8:58 PM, Ralf Gommers >> wrote: >> > Hi all, >> > >> > There are 3 PRs open for adding new window functions to scipy.signal: >> > - Kaiser-Bessel, https://github.com/scipy/scipy/pull/4747 >> > - Planck, https://github.com/scipy/scipy/pull/6012 (also interest in >> > adding >> > Planck-Bessel there) >> > - Lanczos, https://github.com/scipy/scipy/pull/6837 >> > >> > So far Kaiser-Bessel and Lanczos got a positive reception, Planck didn't >> > because it looks too specific - only uses found are in gravitational >> > wave >> > research. >> > >> > One of the annoyances with the window functions is that they really >> > clutter >> > the signal namespace as we keep on adding more. We have two options I >> > think: >> > >> > 1. Keep on adding new window functions with the current style (provided >> > that >> > they're general enough, i.e. used in more than one field). >> > 2. Only add them behind a unified interface, like signal.get_window. >> > Could >> > also be a new interface, not everyone is a fan of get_window (see >> > https://github.com/scipy/scipy/pull/6837). >> >> Not sure what you're pointing to with that link, > > > Sorry, copy-paste error - should be https://github.com/scipy/scipy/pull/6895 > >> >> but isn't get_window >> basically reinventing the concept of the python namespace? Maybe >> declare scipy.signal.windows (which already exists!) to be the public >> namespace where you find your windows? > > >> Not really. For example, from gh-6895, this (which uses get_window): > > >? ? signal.firwin(80, 0.5, window=('kaiser', 8)) > > is equivalent to: > >? ? signal.firwin(80, 0.5, window=functools.partial(signal.kaiser, beta=8)) > > Despite the arguments in gh-6895, I'm quite sure that most users prefer not > to have to know about functools.partial. ...maybe? It's certainly a thing to learn, but so is the current idiosyncratic mix of strings and functions and tuples, and at least partial is a reusable tool? Not-really-serious suggestion: # Add this line to scipy/signal/__init__.py, then users don't have to learn functools.partial from functools import partial as make_window Maybe-serious suggestion: If the problem is that we need a way to refer to abstractions like "a hamming window (with some to-be-determined number of taps)" or "a kaiser window with beta=8 (and some to-be-determined number of taps)", maybe instead of using these ad hoc strings/tuples/etc. we should make these objects. scipy.signals.windows.Hamming(), scipy.signals.windows.Kaiser(beta=8), ...? -n -- Nathaniel J. Smith -- https://vorpus.org ------------------------------ Message: 5 Date: Fri, 6 Jan 2017 07:01:04 +0100 From: Gael Varoquaux To: SciPy Developers List Subject: Re: [SciPy-Dev] adding more window functions to scipy.signal Message-ID: <20170106060104.GW250062 at phare.normalesup.org> Content-Type: text/plain; charset=iso-8859-1 On Thu, Jan 05, 2017 at 09:07:44PM -0800, Nathaniel Smith wrote: > Not sure what you're pointing to with that link, but isn't get_window > basically reinventing the concept of the python namespace? Maybe > declare scipy.signal.windows (which already exists!) to be the public > namespace where you find your windows? That were my thoughts exactly. get_windows brings interesting functionality, but the discoverability isn't great. Namespaces bring in tab completion. They are one honking great idea! Ga?l ------------------------------ Subject: Digest Footer _______________________________________________ SciPy-Dev mailing list SciPy-Dev at scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev ------------------------------ End of SciPy-Dev Digest, Vol 159, Issue 3 ***************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 6 20:04:14 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Jan 2017 18:04:14 -0700 Subject: [SciPy-Dev] adding more window functions to scipy.signal In-Reply-To: References: Message-ID: On Thu, Jan 5, 2017 at 10:27 PM, Ralf Gommers wrote: > > > On Fri, Jan 6, 2017 at 6:07 PM, Nathaniel Smith wrote: > >> On Thu, Jan 5, 2017 at 8:58 PM, Ralf Gommers >> wrote: >> > Hi all, >> > >> > There are 3 PRs open for adding new window functions to scipy.signal: >> > - Kaiser-Bessel, https://github.com/scipy/scipy/pull/4747 >> > - Planck, https://github.com/scipy/scipy/pull/6012 (also interest in >> adding >> > Planck-Bessel there) >> > - Lanczos, https://github.com/scipy/scipy/pull/6837 >> > >> > So far Kaiser-Bessel and Lanczos got a positive reception, Planck didn't >> > because it looks too specific - only uses found are in gravitational >> wave >> > research. >> > >> > One of the annoyances with the window functions is that they really >> clutter >> > the signal namespace as we keep on adding more. We have two options I >> think: >> > >> > 1. Keep on adding new window functions with the current style (provided >> that >> > they're general enough, i.e. used in more than one field). >> > 2. Only add them behind a unified interface, like signal.get_window. >> Could >> > also be a new interface, not everyone is a fan of get_window (see >> > https://github.com/scipy/scipy/pull/6837). >> >> Not sure what you're pointing to with that link, > > > Sorry, copy-paste error - should be https://github.com/scipy/ > scipy/pull/6895 > > >> but isn't get_window >> basically reinventing the concept of the python namespace? Maybe >> declare scipy.signal.windows (which already exists!) to be the public >> namespace where you find your windows? >> > > Not really. For example, from gh-6895, this (which uses get_window): >> > > signal.firwin(80, 0.5, window=('kaiser', 8)) > > is equivalent to: > > signal.firwin(80, 0.5, window=functools.partial(signal.kaiser, beta=8)) > > Despite the arguments in gh-6895, I'm quite sure that most users prefer > not to have to know about functools.partial. > > Agreed that get_window is a bit ugly though ... > Once upon a time (the 80's called) I solved the problem by providing Kaiser-Bessell windows for a select set of side lobes, basically negative powers of ten. Maybe there is a better way to parameterize the windows functions that isn't specific to the type. In practice, one is usually interested in half width, side lobe, and fall off and the different windows achieve different tradeoffs. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 6 20:21:34 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 6 Jan 2017 18:21:34 -0700 Subject: [SciPy-Dev] adding more window functions to scipy.signal In-Reply-To: References: Message-ID: IIRC, I liked this reference , or another by the same author. It is old, but has a nice table. Perhaps the windows functions should return an object that also contains the computed figures of merit. Chuck On Fri, Jan 6, 2017 at 6:04 PM, Charles R Harris wrote: > > > On Thu, Jan 5, 2017 at 10:27 PM, Ralf Gommers > wrote: > >> >> >> On Fri, Jan 6, 2017 at 6:07 PM, Nathaniel Smith wrote: >> >>> On Thu, Jan 5, 2017 at 8:58 PM, Ralf Gommers >>> wrote: >>> > Hi all, >>> > >>> > There are 3 PRs open for adding new window functions to scipy.signal: >>> > - Kaiser-Bessel, https://github.com/scipy/scipy/pull/4747 >>> > - Planck, https://github.com/scipy/scipy/pull/6012 (also interest in >>> adding >>> > Planck-Bessel there) >>> > - Lanczos, https://github.com/scipy/scipy/pull/6837 >>> > >>> > So far Kaiser-Bessel and Lanczos got a positive reception, Planck >>> didn't >>> > because it looks too specific - only uses found are in gravitational >>> wave >>> > research. >>> > >>> > One of the annoyances with the window functions is that they really >>> clutter >>> > the signal namespace as we keep on adding more. We have two options I >>> think: >>> > >>> > 1. Keep on adding new window functions with the current style >>> (provided that >>> > they're general enough, i.e. used in more than one field). >>> > 2. Only add them behind a unified interface, like signal.get_window. >>> Could >>> > also be a new interface, not everyone is a fan of get_window (see >>> > https://github.com/scipy/scipy/pull/6837). >>> >>> Not sure what you're pointing to with that link, >> >> >> Sorry, copy-paste error - should be https://github.com/scipy/scipy >> /pull/6895 >> >> >>> but isn't get_window >>> basically reinventing the concept of the python namespace? Maybe >>> declare scipy.signal.windows (which already exists!) to be the public >>> namespace where you find your windows? >>> >> >> Not really. For example, from gh-6895, this (which uses get_window): >>> >> >> signal.firwin(80, 0.5, window=('kaiser', 8)) >> >> is equivalent to: >> >> signal.firwin(80, 0.5, window=functools.partial(signal.kaiser, >> beta=8)) >> >> Despite the arguments in gh-6895, I'm quite sure that most users prefer >> not to have to know about functools.partial. >> >> Agreed that get_window is a bit ugly though ... >> > > Once upon a time (the 80's called) I solved the problem by providing > Kaiser-Bessell windows for a select set of side lobes, basically negative > powers of ten. Maybe there is a better way to parameterize the windows > functions that isn't specific to the type. In practice, one is usually > interested in half width, side lobe, and fall off and the different windows > achieve different tradeoffs. > > Chuck > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Jan 7 03:10:26 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 7 Jan 2017 21:10:26 +1300 Subject: [SciPy-Dev] Arff-handling? In-Reply-To: References: Message-ID: On Thu, Dec 22, 2016 at 11:12 AM, Moy Git wrote: > Hi folks. > > My team is writing code to slice and dice arff files: add/delete/update > columns, save arffs, join two arffs on a key, append rows, etc. If there's > interest in adding this functionality to SciPy's arff module please let me > know and I'll send a PR. > > (Brief description: I'll extend arffread's MetaData class and add Arff and > Data classes. Arff will have a convert_to_numpy method, and arffread's > loadarff function will just create an Arff and call its convert_to_numpy. > This will maintain backward-compatibility and also let us add new stuff to > Arff.) > Hi Moy, given the lack of interest and that (AFAIK) no SciPy devs are users of arff files, it may be better to keep the functionality in scipy.io simple. There's a dedicated package for arff files called liac-arff [1, 2] that looks pretty good and may be a better place for the more advanced functionality you are proposing. [1] https://github.com/renatopp/liac-arff [2] https://pythonhosted.org/liac-arff/ Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Jan 7 03:41:26 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 7 Jan 2017 00:41:26 -0800 Subject: [SciPy-Dev] Arff-handling? In-Reply-To: References: Message-ID: On Sat, Jan 7, 2017 at 12:10 AM, Ralf Gommers wrote: > > > On Thu, Dec 22, 2016 at 11:12 AM, Moy Git wrote: >> >> Hi folks. >> >> My team is writing code to slice and dice arff files: add/delete/update >> columns, save arffs, join two arffs on a key, append rows, etc. If there's >> interest in adding this functionality to SciPy's arff module please let me >> know and I'll send a PR. >> >> (Brief description: I'll extend arffread's MetaData class and add Arff and >> Data classes. Arff will have a convert_to_numpy method, and arffread's >> loadarff function will just create an Arff and call its convert_to_numpy. >> This will maintain backward-compatibility and also let us add new stuff to >> Arff.) > > > Hi Moy, given the lack of interest and that (AFAIK) no SciPy devs are users > of arff files, it may be better to keep the functionality in scipy.io > simple. There's a dedicated package for arff files called liac-arff [1, 2] > that looks pretty good and may be a better place for the more advanced > functionality you are proposing. > > [1] https://github.com/renatopp/liac-arff > [2] https://pythonhosted.org/liac-arff/ It might make sense to add a note to the scipy.io docs pointing to this as an option for those who need something fancier, though? -n -- Nathaniel J. Smith -- https://vorpus.org From ralf.gommers at gmail.com Sat Jan 7 03:53:11 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 7 Jan 2017 21:53:11 +1300 Subject: [SciPy-Dev] Arff-handling? In-Reply-To: References: Message-ID: On Sat, Jan 7, 2017 at 9:41 PM, Nathaniel Smith wrote: > On Sat, Jan 7, 2017 at 12:10 AM, Ralf Gommers > wrote: > > > > > > On Thu, Dec 22, 2016 at 11:12 AM, Moy Git wrote: > >> > >> Hi folks. > >> > >> My team is writing code to slice and dice arff files: add/delete/update > >> columns, save arffs, join two arffs on a key, append rows, etc. If > there's > >> interest in adding this functionality to SciPy's arff module please let > me > >> know and I'll send a PR. > >> > >> (Brief description: I'll extend arffread's MetaData class and add Arff > and > >> Data classes. Arff will have a convert_to_numpy method, and arffread's > >> loadarff function will just create an Arff and call its > convert_to_numpy. > >> This will maintain backward-compatibility and also let us add new stuff > to > >> Arff.) > > > > > > Hi Moy, given the lack of interest and that (AFAIK) no SciPy devs are > users > > of arff files, it may be better to keep the functionality in scipy.io > > simple. There's a dedicated package for arff files called liac-arff [1, > 2] > > that looks pretty good and may be a better place for the more advanced > > functionality you are proposing. > > > > [1] https://github.com/renatopp/liac-arff > > [2] https://pythonhosted.org/liac-arff/ > > It might make sense to add a note to the scipy.io docs pointing to > this as an option for those who need something fancier, though? > Good idea, will do. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From wavexx at thregr.org Sat Jan 7 06:07:16 2017 From: wavexx at thregr.org (Yuri D'Elia) Date: Sat, 07 Jan 2017 12:07:16 +0100 Subject: [SciPy-Dev] adding more window functions to scipy.signal References: Message-ID: <87zij3p1qz.fsf@wavexx.thregr.org> On Fri, Jan 06 2017, Ralf Gommers wrote: > Sorry, copy-paste error - should be https://github.com/scipy/scipy/pull/6895 > > but isn't get_window > basically reinventing the concept of the python namespace? Maybe > declare scipy.signal.windows (which already exists!) to be the public > namespace where you find your windows? > > Not really. For example, from gh-6895, this (which uses get_window): > > signal.firwin(80, 0.5, window=('kaiser', 8)) > > is equivalent to: > > signal.firwin(80, 0.5, window=functools.partial(signal.kaiser, beta=8)) > > Despite the arguments in gh-6895, I'm quite sure that most users prefer not to > have to know about functools.partial. > > Agreed that get_window is a bit ugly though ... [thanks for mentioning this thread in PR#6012] As I also wrote in the planck PR, I'd move all window functions to the windows namespace as well. This is much cleaner. Providing aliases for backward-compatibility is trivial. Name clashes go away. I think the benefits are pretty clear. We can rewrite get_window so that it simply uses ft.partial anyway, with the added exception that, for strings as first arguments, we resolve the function in the signal.windows namespace to keep the code concise. I don't think that knowing 'partial' is bad per se, but using it directly in this case would get pretty verbose very often. For experimentation, I did like the window=tuple "hack". From moygit1729 at gmail.com Tue Jan 10 21:40:53 2017 From: moygit1729 at gmail.com (Moy Git) Date: Tue, 10 Jan 2017 21:40:53 -0500 Subject: [SciPy-Dev] Arff-handling? In-Reply-To: References: Message-ID: Okay, thanks, guys. On Sat, Jan 7, 2017 at 3:53 AM, Ralf Gommers wrote: > > > On Sat, Jan 7, 2017 at 9:41 PM, Nathaniel Smith wrote: > >> On Sat, Jan 7, 2017 at 12:10 AM, Ralf Gommers >> wrote: >> > >> > >> > On Thu, Dec 22, 2016 at 11:12 AM, Moy Git wrote: >> >> >> >> Hi folks. >> >> >> >> My team is writing code to slice and dice arff files: add/delete/update >> >> columns, save arffs, join two arffs on a key, append rows, etc. If >> there's >> >> interest in adding this functionality to SciPy's arff module please >> let me >> >> know and I'll send a PR. >> >> >> >> (Brief description: I'll extend arffread's MetaData class and add Arff >> and >> >> Data classes. Arff will have a convert_to_numpy method, and arffread's >> >> loadarff function will just create an Arff and call its >> convert_to_numpy. >> >> This will maintain backward-compatibility and also let us add new >> stuff to >> >> Arff.) >> > >> > >> > Hi Moy, given the lack of interest and that (AFAIK) no SciPy devs are >> users >> > of arff files, it may be better to keep the functionality in scipy.io >> > simple. There's a dedicated package for arff files called liac-arff [1, >> 2] >> > that looks pretty good and may be a better place for the more advanced >> > functionality you are proposing. >> > >> > [1] https://github.com/renatopp/liac-arff >> > [2] https://pythonhosted.org/liac-arff/ >> >> It might make sense to add a note to the scipy.io docs pointing to >> this as an option for those who need something fancier, though? >> > > Good idea, will do. > > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jan 11 04:13:13 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 11 Jan 2017 22:13:13 +1300 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Tue, Sep 13, 2016 at 9:03 PM, Ralf Gommers wrote: > > > On Sat, Sep 10, 2016 at 10:25 AM, Ralf Gommers > wrote: > >> >> >> On Wed, Sep 7, 2016 at 1:37 AM, Eric Larson >> wrote: >> >>> >>> My feeling is that having a clear leader in place is important, so I'm >>>> also leaning away from the numpy model towards one where >>>> responsibilities are more explicitly assigned. Exactly how to best make >>>> that assignment is still unclear to me. >>>> >>> >>> +1 for BD(FL) / leader-style from me, too. I like Matthew's suggestion >>> of the top 5 active folks discuss to see which of them are actually >>> interested in taking on that role. >>> >> >> Thanks for the feedback everyone. Looks like everyone likes this >> suggestion so far, so we'll give that a try. >> > > Hi all, I'm happy to report that we've worked this out. Pauli was our > preferred candidate, and he has agreed to take up the role of BDFL! > Hi all, it has taken a while longer than I had planned, but here is a PR with the draft governance doc: https://github.com/scipy/scipy/pull/6953. Review and feedback very welcome. If there's something really major about the structure of it, this list is probably a better place to discuss then on the PR. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis at laxalde.org Wed Jan 11 04:38:30 2017 From: denis at laxalde.org (Denis Laxalde) Date: Wed, 11 Jan 2017 10:38:30 +0100 Subject: [SciPy-Dev] API to control iteration in differential evolution #6923 Message-ID: Hi, Recently there have been discussions about possible extensions of scipy.optimize.differential_evolution() : * https://github.com/scipy/scipy/issues/6878 is about having the callback function receive the function value * https://github.com/scipy/scipy/issues/6879 is about customizing of population initialization The differential_evolution function being already quite complex, we thought it would be better to expose a "lower level" API to control the iteration in the algorithm. This would be done by making the DifferentialEvolutionSolver class (in scipy/optimize/_differentialevolution.py) public and cleaning it a bit. I submitted a pull request for this : https://github.com/scipy/scipy/pull/6923 One noticeable thing is that it also makes the class a context manager to encapsulate the population initialization (in enter step) and final polishing (in exit step). Ultimately, usage of this class (renamed as DifferentialEvolution) would be: with DifferentialEvolution(func, bounds) as solver: # iterate until maxiter/maxfev is reached or the algorithm converged for step in solver: if mycallback(step.x, step.fun): break result = solver.result I might look a bit fancy, but I think it makes sense to regard a computation as a kind of context. Feedback welcome! Cheers, Denis. From ralf.gommers at gmail.com Wed Jan 11 04:56:53 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 11 Jan 2017 22:56:53 +1300 Subject: [SciPy-Dev] adding more window functions to scipy.signal In-Reply-To: <87zij3p1qz.fsf@wavexx.thregr.org> References: <87zij3p1qz.fsf@wavexx.thregr.org> Message-ID: On Sun, Jan 8, 2017 at 12:07 AM, Yuri D'Elia wrote: > On Fri, Jan 06 2017, Ralf Gommers wrote: > > Sorry, copy-paste error - should be https://github.com/scipy/ > scipy/pull/6895 > > > > but isn't get_window > > basically reinventing the concept of the python namespace? Maybe > > declare scipy.signal.windows (which already exists!) to be the public > > namespace where you find your windows? > > > > Not really. For example, from gh-6895, this (which uses get_window): > > > > signal.firwin(80, 0.5, window=('kaiser', 8)) > > > > is equivalent to: > > > > signal.firwin(80, 0.5, window=functools.partial(signal.kaiser, beta=8)) > > > > Despite the arguments in gh-6895, I'm quite sure that most users prefer > not to > > have to know about functools.partial. > > > > Agreed that get_window is a bit ugly though ... > > [thanks for mentioning this thread in PR#6012] > > As I also wrote in the planck PR, I'd move all window functions to the > windows namespace as well. This is much cleaner. Providing aliases for > backward-compatibility is trivial. Name clashes go away. I think the > benefits are pretty clear. > Sounds like adding signal.windows to the public API is the consensus - let's go with that. I'll have a look at preparing a PR for that. Then we can add all the new windows in the open PRs just there. Cheers, Ralf > We can rewrite get_window so that it simply uses ft.partial anyway, with > the added exception that, for strings as first arguments, we resolve the > function in the signal.windows namespace to keep the code concise. > > I don't think that knowing 'partial' is bad per se, but using it > directly in this case would get pretty verbose very often. For > experimentation, I did like the window=tuple "hack". > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniele at grinta.net Thu Jan 12 07:39:56 2017 From: daniele at grinta.net (Daniele Nicolodi) Date: Thu, 12 Jan 2017 05:39:56 -0700 Subject: [SciPy-Dev] Include Allan variance estimators in SciPy Message-ID: <654beb7c-12ab-7af7-30a8-97949b2dfff9@grinta.net> Hello, the Allan family of statistical estimators [0] (Allan variance, modified Allan variance, total variance) are extensively used in some research fields, like frequency metrology and related fields. While the algorithm to compute the Allan variance is quite straight forward, efficiently coding the modified Allan variance, or the estimator based on phase rather than on frequency samples, are a bit more tricky. I have implementations of those (in cython) that I would like to contribute to SciPy, most probably as an `scipy.stats.allan` module. Does people deem those interesting? Should I prepare a pull request to discuss the implementation? Thanks. Cheers, Daniele [0] https://en.wikipedia.org/wiki/Allan_variance From matthew.brett at gmail.com Fri Jan 13 12:22:58 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 13 Jan 2017 09:22:58 -0800 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: Hi, On Wed, Jan 11, 2017 at 1:13 AM, Ralf Gommers wrote: > > > On Tue, Sep 13, 2016 at 9:03 PM, Ralf Gommers > wrote: >> >> >> >> On Sat, Sep 10, 2016 at 10:25 AM, Ralf Gommers >> wrote: >>> >>> >>> >>> On Wed, Sep 7, 2016 at 1:37 AM, Eric Larson >>> wrote: >>>> >>>> >>>>> My feeling is that having a clear leader in place is important, so I'm >>>>> also leaning away from the numpy model towards one where >>>>> responsibilities are more explicitly assigned. Exactly how to best >>>>> make >>>>> that assignment is still unclear to me. >>>> >>>> >>>> +1 for BD(FL) / leader-style from me, too. I like Matthew's suggestion >>>> of the top 5 active folks discuss to see which of them are actually >>>> interested in taking on that role. >>> >>> >>> Thanks for the feedback everyone. Looks like everyone likes this >>> suggestion so far, so we'll give that a try. >> >> >> Hi all, I'm happy to report that we've worked this out. Pauli was our >> preferred candidate, and he has agreed to take up the role of BDFL! > > > Hi all, it has taken a while longer than I had planned, but here is a PR > with the draft governance doc: https://github.com/scipy/scipy/pull/6953. > > Review and feedback very welcome. If there's something really major about > the structure of it, this list is probably a better place to discuss then on > the PR. The main question I'd like to raise, is whether we really want a BDFL, as opposed to an elected projected leader. I think a BDFL makes sense where there's one person who started the project, wrote most of the code (at least at some point in the project's history), has been in charge since the beginning, and is still very active. I think that does correspond to the situation for Linus / Linux; Guido / Python; and Fernando / IPython. I don't think we have anyone matching that description in Scipy. On the other hand, I do think it's important to have a project leader, with final authority on the direction of the project, and who takes responsibility for the health of the project. So, I propose, rather than have a BDFL, we have a system for choosing a leader, say every 4 years, where we may or may not have limits on the number of consecutive terms. We can use that 4 year cycle to make sure we're reconsidering the direction of the project regularly, and thinking about where we could improve, and where we might be messing up. It provides a natural way to give people a rest from the job, if they want one. If one leader steps back, and sees another leader doing something better than they did, they can learn from that when they next have a leadership term. Of course that requires some formalization, but I think it's a considerably better system than the BDFL, for our case. Cheers, Matthew From evgeny.burovskiy at gmail.com Fri Jan 13 18:24:47 2017 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Sat, 14 Jan 2017 02:24:47 +0300 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Fri, Jan 13, 2017 at 8:22 PM, Matthew Brett wrote: > Hi, > > On Wed, Jan 11, 2017 at 1:13 AM, Ralf Gommers wrote: >> >> >> On Tue, Sep 13, 2016 at 9:03 PM, Ralf Gommers >> wrote: >>> >>> >>> >>> On Sat, Sep 10, 2016 at 10:25 AM, Ralf Gommers >>> wrote: >>>> >>>> >>>> >>>> On Wed, Sep 7, 2016 at 1:37 AM, Eric Larson >>>> wrote: >>>>> >>>>> >>>>>> My feeling is that having a clear leader in place is important, so I'm >>>>>> also leaning away from the numpy model towards one where >>>>>> responsibilities are more explicitly assigned. Exactly how to best >>>>>> make >>>>>> that assignment is still unclear to me. >>>>> >>>>> >>>>> +1 for BD(FL) / leader-style from me, too. I like Matthew's suggestion >>>>> of the top 5 active folks discuss to see which of them are actually >>>>> interested in taking on that role. >>>> >>>> >>>> Thanks for the feedback everyone. Looks like everyone likes this >>>> suggestion so far, so we'll give that a try. >>> >>> >>> Hi all, I'm happy to report that we've worked this out. Pauli was our >>> preferred candidate, and he has agreed to take up the role of BDFL! > The main question I'd like to raise, is whether we really want a BDFL, > as opposed to an elected projected leader. > > I think a BDFL makes sense where there's one person who started the > project, wrote most of the code (at least at some point in the > project's history), has been in charge since the beginning, and is > still very active. I think that does correspond to the situation for > Linus / Linux; Guido / Python; and Fernando / IPython. I don't > think we have anyone matching that description in Scipy. > > On the other hand, I do think it's important to have a project leader, > with final authority on the direction of the project, and who takes > responsibility for the health of the project. > > So, I propose, rather than have a BDFL, we have a system for choosing > a leader, Which is exactly how it worked this time --- using the system of your suggestion, https://mail.scipy.org/pipermail/scipy-dev/2016-September/021476.html > say every 4 years, where we may or may not have limits on > the number of consecutive terms. While I agree that this model is better in the vast majority of situations, it feels to be a bit of over-engineering for scipy. > We can use that 4 year cycle to > make sure we're reconsidering the direction of the project regularly, > and thinking about where we could improve, and where we might be > messing up. It provides a natural way to give people a rest from the > job, if they want one. If one leader steps back, and sees another > leader doing something better than they did, they can learn from that > when they next have a leadership term. I agree it's worth it to periodically sit down and think about the direction of the project. I'm not sure though there is benefit in tying this up to a machinery of fixed-time leadership terms, holding formal elections and so on. > Of course that requires some formalization, but I think it's a > considerably better system than the BDFL, for our case. It seems to me that the effort needed to formalize it is not worth the benefit, specifically in our case. My 2 cents, Evgeni From matthew.brett at gmail.com Fri Jan 13 18:32:00 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 13 Jan 2017 15:32:00 -0800 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: Hi, On Fri, Jan 13, 2017 at 3:24 PM, Evgeni Burovski wrote: > On Fri, Jan 13, 2017 at 8:22 PM, Matthew Brett wrote: >> Hi, >> >> On Wed, Jan 11, 2017 at 1:13 AM, Ralf Gommers wrote: >>> >>> >>> On Tue, Sep 13, 2016 at 9:03 PM, Ralf Gommers >>> wrote: >>>> >>>> >>>> >>>> On Sat, Sep 10, 2016 at 10:25 AM, Ralf Gommers >>>> wrote: >>>>> >>>>> >>>>> >>>>> On Wed, Sep 7, 2016 at 1:37 AM, Eric Larson >>>>> wrote: >>>>>> >>>>>> >>>>>>> My feeling is that having a clear leader in place is important, so I'm >>>>>>> also leaning away from the numpy model towards one where >>>>>>> responsibilities are more explicitly assigned. Exactly how to best >>>>>>> make >>>>>>> that assignment is still unclear to me. >>>>>> >>>>>> >>>>>> +1 for BD(FL) / leader-style from me, too. I like Matthew's suggestion >>>>>> of the top 5 active folks discuss to see which of them are actually >>>>>> interested in taking on that role. >>>>> >>>>> >>>>> Thanks for the feedback everyone. Looks like everyone likes this >>>>> suggestion so far, so we'll give that a try. >>>> >>>> >>>> Hi all, I'm happy to report that we've worked this out. Pauli was our >>>> preferred candidate, and he has agreed to take up the role of BDFL! > > > >> The main question I'd like to raise, is whether we really want a BDFL, >> as opposed to an elected projected leader. >> >> I think a BDFL makes sense where there's one person who started the >> project, wrote most of the code (at least at some point in the >> project's history), has been in charge since the beginning, and is >> still very active. I think that does correspond to the situation for >> Linus / Linux; Guido / Python; and Fernando / IPython. I don't >> think we have anyone matching that description in Scipy. >> >> On the other hand, I do think it's important to have a project leader, >> with final authority on the direction of the project, and who takes >> responsibility for the health of the project. >> >> So, I propose, rather than have a BDFL, we have a system for choosing >> a leader, > > Which is exactly how it worked this time --- using the system of your > suggestion, https://mail.scipy.org/pipermail/scipy-dev/2016-September/021476.html > > >> say every 4 years, where we may or may not have limits on >> the number of consecutive terms. > > While I agree that this model is better in the vast majority of > situations, it feels to be a bit of over-engineering for scipy. > >> We can use that 4 year cycle to >> make sure we're reconsidering the direction of the project regularly, >> and thinking about where we could improve, and where we might be >> messing up. It provides a natural way to give people a rest from the >> job, if they want one. If one leader steps back, and sees another >> leader doing something better than they did, they can learn from that >> when they next have a leadership term. > > I agree it's worth it to periodically sit down and think about the > direction of the project. > I'm not sure though there is benefit in tying this up to a machinery > of fixed-time leadership terms, holding formal elections and so on. > > >> Of course that requires some formalization, but I think it's a >> considerably better system than the BDFL, for our case. > > It seems to me that the effort needed to formalize it is not worth the > benefit, specifically in our case. Well - as a broader community, I think we'll have to do this anyway. For example, I know that Stefan vdW wants to set up this model for scikit-image. I am sure he'd be happy to help draft it, I know I would. Maybe we could do that in relation to this PR, making sure that we set some reasonable time limit for getting it done, say 3 weeks. Cheers, Matthew From josef.pktd at gmail.com Fri Jan 13 19:13:10 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 13 Jan 2017 19:13:10 -0500 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Fri, Jan 13, 2017 at 6:32 PM, Matthew Brett wrote: > Hi, > > On Fri, Jan 13, 2017 at 3:24 PM, Evgeni Burovski > wrote: > > On Fri, Jan 13, 2017 at 8:22 PM, Matthew Brett > wrote: > >> Hi, > >> > >> On Wed, Jan 11, 2017 at 1:13 AM, Ralf Gommers > wrote: > >>> > >>> > >>> On Tue, Sep 13, 2016 at 9:03 PM, Ralf Gommers > >>> wrote: > >>>> > >>>> > >>>> > >>>> On Sat, Sep 10, 2016 at 10:25 AM, Ralf Gommers < > ralf.gommers at gmail.com> > >>>> wrote: > >>>>> > >>>>> > >>>>> > >>>>> On Wed, Sep 7, 2016 at 1:37 AM, Eric Larson > > >>>>> wrote: > >>>>>> > >>>>>> > >>>>>>> My feeling is that having a clear leader in place is important, so > I'm > >>>>>>> also leaning away from the numpy model towards one where > >>>>>>> responsibilities are more explicitly assigned. Exactly how to best > >>>>>>> make > >>>>>>> that assignment is still unclear to me. > >>>>>> > >>>>>> > >>>>>> +1 for BD(FL) / leader-style from me, too. I like Matthew's > suggestion > >>>>>> of the top 5 active folks discuss to see which of them are actually > >>>>>> interested in taking on that role. > >>>>> > >>>>> > >>>>> Thanks for the feedback everyone. Looks like everyone likes this > >>>>> suggestion so far, so we'll give that a try. > >>>> > >>>> > >>>> Hi all, I'm happy to report that we've worked this out. Pauli was our > >>>> preferred candidate, and he has agreed to take up the role of BDFL! > > > > > > > >> The main question I'd like to raise, is whether we really want a BDFL, > >> as opposed to an elected projected leader. > >> > >> I think a BDFL makes sense where there's one person who started the > >> project, wrote most of the code (at least at some point in the > >> project's history), has been in charge since the beginning, and is > >> still very active. I think that does correspond to the situation for > >> Linus / Linux; Guido / Python; and Fernando / IPython. I don't > >> think we have anyone matching that description in Scipy. > >> > >> On the other hand, I do think it's important to have a project leader, > >> with final authority on the direction of the project, and who takes > >> responsibility for the health of the project. > >> > >> So, I propose, rather than have a BDFL, we have a system for choosing > >> a leader, > > > > Which is exactly how it worked this time --- using the system of your > > suggestion, https://mail.scipy.org/pipermail/scipy-dev/2016- > September/021476.html > > > > > >> say every 4 years, where we may or may not have limits on > >> the number of consecutive terms. > > > > While I agree that this model is better in the vast majority of > > situations, it feels to be a bit of over-engineering for scipy. > > > >> We can use that 4 year cycle to > >> make sure we're reconsidering the direction of the project regularly, > >> and thinking about where we could improve, and where we might be > >> messing up. It provides a natural way to give people a rest from the > >> job, if they want one. If one leader steps back, and sees another > >> leader doing something better than they did, they can learn from that > >> when they next have a leadership term. > > > > I agree it's worth it to periodically sit down and think about the > > direction of the project. > > I'm not sure though there is benefit in tying this up to a machinery > > of fixed-time leadership terms, holding formal elections and so on. > > > > > >> Of course that requires some formalization, but I think it's a > >> considerably better system than the BDFL, for our case. > > > > It seems to me that the effort needed to formalize it is not worth the > > benefit, specifically in our case. > > Well - as a broader community, I think we'll have to do this anyway. > For example, I know that Stefan vdW wants to set up this model for > scikit-image. I am sure he'd be happy to help draft it, I know I > would. Maybe we could do that in relation to this PR, making sure > that we set some reasonable time limit for getting it done, say 3 > weeks. > I like the stability and continuity that a known BDFL offers and signals, and wouldn't want a formalized voting system where somebody might try to game the system and get the majority of "electoral votes". Either scheduled elections are redundant because of a consensus or they create additional stress if you need to get x%. Josef "Make Scipy great again" > > Cheers, > > Matthew > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jan 13 19:15:34 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 13 Jan 2017 16:15:34 -0800 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Fri, Jan 13, 2017 at 3:32 PM, Matthew Brett wrote: > Hi, > > On Fri, Jan 13, 2017 at 3:24 PM, Evgeni Burovski > wrote: >> On Fri, Jan 13, 2017 at 8:22 PM, Matthew Brett wrote: [...] >>> Of course that requires some formalization, but I think it's a >>> considerably better system than the BDFL, for our case. >> >> It seems to me that the effort needed to formalize it is not worth the >> benefit, specifically in our case. > > Well - as a broader community, I think we'll have to do this anyway. > For example, I know that Stefan vdW wants to set up this model for > scikit-image. I am sure he'd be happy to help draft it, I know I > would. Maybe we could do that in relation to this PR, making sure > that we set some reasonable time limit for getting it done, say 3 > weeks. It's still the case that this is a novel social organization you invented that AFAICT has never been tested by any F/OSS project, and directly goes against the F/OSS community's hard-won cultural knowledge about what kinds of organizations work well (see e.g. [1]). Now- these are not necessarily bad things! Our community is legitimately different than a "traditional" group of F/OSS developers in a variety of ways, and less encultured to the "traditional" way of doing things. And social experimentation is great -- how else can we find better ways to live? While there's a lot of wisdom and experience in Karl Fogel's book, it's surely not the final word. But... we should also be realistic that when someone shows up saying "hey I've worked out a better method of social organization based on first principles and thinking really hard, it'll 100% definitely be awesome", then historically it *usually* doesn't quite work out so nicely as promised. And it's often difficult to effectively do this kind of experimentation at the same time as doing the actual work of like, developing software. "Choose boring technology" [2] applies to social technology too. If scikit-image is set on doing this, maybe the pragmatic thing to do is wait and see how it works out for them? I've seen zero appetite from anyone else on this list for elections and such. -n [1] http://producingoss.com/en/producingoss.html#social-infrastructure [2] http://mcfunley.com/choose-boring-technology -- Nathaniel J. Smith -- https://vorpus.org From matthew.brett at gmail.com Fri Jan 13 20:01:12 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 13 Jan 2017 17:01:12 -0800 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: Hi, On Fri, Jan 13, 2017 at 4:15 PM, Nathaniel Smith wrote: > On Fri, Jan 13, 2017 at 3:32 PM, Matthew Brett wrote: >> Hi, >> >> On Fri, Jan 13, 2017 at 3:24 PM, Evgeni Burovski >> wrote: >>> On Fri, Jan 13, 2017 at 8:22 PM, Matthew Brett wrote: > [...] >>>> Of course that requires some formalization, but I think it's a >>>> considerably better system than the BDFL, for our case. >>> >>> It seems to me that the effort needed to formalize it is not worth the >>> benefit, specifically in our case. >> >> Well - as a broader community, I think we'll have to do this anyway. >> For example, I know that Stefan vdW wants to set up this model for >> scikit-image. I am sure he'd be happy to help draft it, I know I >> would. Maybe we could do that in relation to this PR, making sure >> that we set some reasonable time limit for getting it done, say 3 >> weeks. > > It's still the case that this is a novel social organization you > invented that AFAICT has never been tested by any F/OSS project, and > directly goes against the F/OSS community's hard-won cultural > knowledge about what kinds of organizations work well (see e.g. [1]). > Now- these are not necessarily bad things! Our community is > legitimately different than a "traditional" group of F/OSS developers > in a variety of ways, and less encultured to the "traditional" way of > doing things. And social experimentation is great -- how else can we > find better ways to live? While there's a lot of wisdom and experience > in Karl Fogel's book, it's surely not the final word. > > But... we should also be realistic that when someone shows up saying > "hey I've worked out a better method of social organization based on > first principles and thinking really hard, it'll 100% definitely be > awesome", then historically it *usually* doesn't quite work out so > nicely as promised. And it's often difficult to effectively do this > kind of experimentation at the same time as doing the actual work of > like, developing software. "Choose boring technology" [2] applies to > social technology too. * I agree with boring technology, but I doubt you're really arguing that choosing a leader at regular intervals is novel in open source or elsewhere. Debian is an obvious example [1]; * I don't know if an 'election' is the right method of choosing someone, that's really up for debate. Obviously an election is a very standard way of doing that; * I don't personally know of a BDFL system working well in the absence of the criteria I put above, but it would certainly be useful to have a look at a few examples. Can you suggest a few to consider? > If scikit-image is set on doing this, maybe the pragmatic thing to do > is wait and see how it works out for them? I've seen zero appetite > from anyone else on this list for elections and such. I think your idea here is the BDFL is low risk and choosing a leader is high risk, but it seems to me that both have risks, and that the best way of assessing the relative risks is to consider and refine a couple of concrete proposals, with discussion of prior experience, where applicable. For the appetite thing, you are probably referring to the nervous atmosphere that surrounds any discussion of governance, which is presumably due to the strong reactions against any such discussion in the past. I'm sure you'd agree that that 'get it over with as quickly as possible' is not the best way to come to a good solution. Having said that, if Ralf and / or Pauli do not have much interest in this topic, discussion will quickly become futile and counterproductive, and we will have to stop quickly to avoid making a mess. Cheers, Matthew [1] https://www.debian.org/devel/leader From josef.pktd at gmail.com Fri Jan 13 21:36:45 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 13 Jan 2017 21:36:45 -0500 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Fri, Jan 13, 2017 at 8:01 PM, Matthew Brett wrote: > Hi, > > On Fri, Jan 13, 2017 at 4:15 PM, Nathaniel Smith wrote: > > On Fri, Jan 13, 2017 at 3:32 PM, Matthew Brett > wrote: > >> Hi, > >> > >> On Fri, Jan 13, 2017 at 3:24 PM, Evgeni Burovski > >> wrote: > >>> On Fri, Jan 13, 2017 at 8:22 PM, Matthew Brett < > matthew.brett at gmail.com> wrote: > > [...] > >>>> Of course that requires some formalization, but I think it's a > >>>> considerably better system than the BDFL, for our case. > >>> > >>> It seems to me that the effort needed to formalize it is not worth the > >>> benefit, specifically in our case. > >> > >> Well - as a broader community, I think we'll have to do this anyway. > >> For example, I know that Stefan vdW wants to set up this model for > >> scikit-image. I am sure he'd be happy to help draft it, I know I > >> would. Maybe we could do that in relation to this PR, making sure > >> that we set some reasonable time limit for getting it done, say 3 > >> weeks. > > > > It's still the case that this is a novel social organization you > > invented that AFAICT has never been tested by any F/OSS project, and > > directly goes against the F/OSS community's hard-won cultural > > knowledge about what kinds of organizations work well (see e.g. [1]). > > Now- these are not necessarily bad things! Our community is > > legitimately different than a "traditional" group of F/OSS developers > > in a variety of ways, and less encultured to the "traditional" way of > > doing things. And social experimentation is great -- how else can we > > find better ways to live? While there's a lot of wisdom and experience > > in Karl Fogel's book, it's surely not the final word. > > > > But... we should also be realistic that when someone shows up saying > > "hey I've worked out a better method of social organization based on > > first principles and thinking really hard, it'll 100% definitely be > > awesome", then historically it *usually* doesn't quite work out so > > nicely as promised. And it's often difficult to effectively do this > > kind of experimentation at the same time as doing the actual work of > > like, developing software. "Choose boring technology" [2] applies to > > social technology too. > > * I agree with boring technology, but I doubt you're really arguing > that choosing a leader at regular intervals is novel in open source or > elsewhere. Debian is an obvious example [1]; > * I don't know if an 'election' is the right method of choosing > someone, that's really up for debate. Obviously an election is a very > standard way of doing that; > * I don't personally know of a BDFL system working well in the absence > of the criteria I put above, but it would certainly be useful to have > a look at a few examples. Can you suggest a few to consider? > > > If scikit-image is set on doing this, maybe the pragmatic thing to do > > is wait and see how it works out for them? I've seen zero appetite > > from anyone else on this list for elections and such. > > I think your idea here is the BDFL is low risk and choosing a leader > is high risk, but it seems to me that both have risks, and that the > best way of assessing the relative risks is to consider and refine a > couple of concrete proposals, with discussion of prior experience, > where applicable. > > For the appetite thing, you are probably referring to the nervous > atmosphere that surrounds any discussion of governance, which is > presumably due to the strong reactions against any such discussion in > the past. I'm sure you'd agree that that 'get it over with as > quickly as possible' is not the best way to come to a good solution. > Having said that, if Ralf and / or Pauli do not have much interest in > this topic, discussion will quickly become futile and > counterproductive, and we will have to stop quickly to avoid making a > mess. > > Cheers, > > Matthew > > [1] https://www.debian.org/devel/leader One big difference between Debian and scipy is the scale. There are 21 members on the scipy steering council and just a few of them that cover a larger part of scipy. The BDFL is a combination of technical council and a bit of a president/Project Leader (internal and external representation) in Debian terms, AFAIU. aside: It looks like the Steering Council is also the constitutional assembly that could switch from Benevolent Dictatorship to Democracy. "Update policy documents such as this one." At least this induced me to really read Ralf's PR, instead of just superficially skimming it. Overall, and given the composition and recruiting of scipy contributors and maintainers, I don't see much difference if the technical council is a BDFL or a BDF5YR (5 year renewable). Does Pauli feel forced to stick with scipy because he is BDFL? Does Evgeni (for example) put in more work than he already does because he wants to be elected the next BDF5YR? Do we get 25 new institutional contributors and council members so they can take over the BDF5YR role? or Does live go on as usual? Josef > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jan 13 22:38:00 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 13 Jan 2017 19:38:00 -0800 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Fri, Jan 13, 2017 at 5:01 PM, Matthew Brett wrote: > Hi, > > On Fri, Jan 13, 2017 at 4:15 PM, Nathaniel Smith wrote: >> On Fri, Jan 13, 2017 at 3:32 PM, Matthew Brett wrote: >>> Hi, >>> >>> On Fri, Jan 13, 2017 at 3:24 PM, Evgeni Burovski >>> wrote: >>>> On Fri, Jan 13, 2017 at 8:22 PM, Matthew Brett wrote: >> [...] >>>>> Of course that requires some formalization, but I think it's a >>>>> considerably better system than the BDFL, for our case. >>>> >>>> It seems to me that the effort needed to formalize it is not worth the >>>> benefit, specifically in our case. >>> >>> Well - as a broader community, I think we'll have to do this anyway. >>> For example, I know that Stefan vdW wants to set up this model for >>> scikit-image. I am sure he'd be happy to help draft it, I know I >>> would. Maybe we could do that in relation to this PR, making sure >>> that we set some reasonable time limit for getting it done, say 3 >>> weeks. >> >> It's still the case that this is a novel social organization you >> invented that AFAICT has never been tested by any F/OSS project, and >> directly goes against the F/OSS community's hard-won cultural >> knowledge about what kinds of organizations work well (see e.g. [1]). >> Now- these are not necessarily bad things! Our community is >> legitimately different than a "traditional" group of F/OSS developers >> in a variety of ways, and less encultured to the "traditional" way of >> doing things. And social experimentation is great -- how else can we >> find better ways to live? While there's a lot of wisdom and experience >> in Karl Fogel's book, it's surely not the final word. >> >> But... we should also be realistic that when someone shows up saying >> "hey I've worked out a better method of social organization based on >> first principles and thinking really hard, it'll 100% definitely be >> awesome", then historically it *usually* doesn't quite work out so >> nicely as promised. And it's often difficult to effectively do this >> kind of experimentation at the same time as doing the actual work of >> like, developing software. "Choose boring technology" [2] applies to >> social technology too. > > * I agree with boring technology, but I doubt you're really arguing > that choosing a leader at regular intervals is novel in open source or > elsewhere. I'm arguing exactly that. (In open source, obviously; scipy is not a nation-state.) > Debian is an obvious example [1]; But the Debian Project Leader is *nothing at all* like a BDFL. In fact their powers are extraordinarily limited; mostly it's just "convince people to do stuff by talking to them" (i.e. "exercising leadership") and "serve as a project figurehead". Which is what your links says! They explicitly *cannot* make decisions about the technical direction of the project; in the Debian system that power is delegated in a complicated way to individual maintainers, mailing list consensus, the CTTE, and GRs. If I seem frustrated in discussing these topics with you, then this is why :-(. As is probably obvious to everyone, I actually love geeking out about this kind of thing! But when you make such misleading and hand-wavy arguments it feels lazy, like you're more interested in vague in-principle discussions than in actually trying to put together a real system that can be implemented and help the project move on and accomplish its real goals. I know that's not your intention and I'm sorry if that sounds harsh. But at this point I'm having trouble seeing how your comments are helping move things forward in any kind of practical way. > * I don't know if an 'election' is the right method of choosing > someone, that's really up for debate. Obviously an election is a very > standard way of doing that; > * I don't personally know of a BDFL system working well in the absence > of the criteria I put above, but it would certainly be useful to have > a look at a few examples. Can you suggest a few to consider? > >> If scikit-image is set on doing this, maybe the pragmatic thing to do >> is wait and see how it works out for them? I've seen zero appetite >> from anyone else on this list for elections and such. > > I think your idea here is the BDFL is low risk and choosing a leader > is high risk, but it seems to me that both have risks, and that the > best way of assessing the relative risks is to consider and refine a > couple of concrete proposals, with discussion of prior experience, > where applicable. > > For the appetite thing, you are probably referring to the nervous > atmosphere that surrounds any discussion of governance, which is > presumably due to the strong reactions against any such discussion in > the past. I'm sure you'd agree that that 'get it over with as > quickly as possible' is not the best way to come to a good solution. > Having said that, if Ralf and / or Pauli do not have much interest in > this topic, discussion will quickly become futile and > counterproductive, and we will have to stop quickly to avoid making a > mess. I'm more referring about the part where scipy *has* a governance document now that seems perfectly workable. It's not identical to the one I would have written, but so what, there are lots of workable models and this looks like one of them. I'm not seeing folks jumping in eager to redo that process for unclear benefits. -n -- Nathaniel J. Smith -- https://vorpus.org From matthew.brett at gmail.com Fri Jan 13 22:57:52 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 13 Jan 2017 19:57:52 -0800 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Fri, Jan 13, 2017 at 7:38 PM, Nathaniel Smith wrote: > On Fri, Jan 13, 2017 at 5:01 PM, Matthew Brett wrote: >> Hi, >> >> On Fri, Jan 13, 2017 at 4:15 PM, Nathaniel Smith wrote: >>> On Fri, Jan 13, 2017 at 3:32 PM, Matthew Brett wrote: >>>> Hi, >>>> >>>> On Fri, Jan 13, 2017 at 3:24 PM, Evgeni Burovski >>>> wrote: >>>>> On Fri, Jan 13, 2017 at 8:22 PM, Matthew Brett wrote: >>> [...] >>>>>> Of course that requires some formalization, but I think it's a >>>>>> considerably better system than the BDFL, for our case. >>>>> >>>>> It seems to me that the effort needed to formalize it is not worth the >>>>> benefit, specifically in our case. >>>> >>>> Well - as a broader community, I think we'll have to do this anyway. >>>> For example, I know that Stefan vdW wants to set up this model for >>>> scikit-image. I am sure he'd be happy to help draft it, I know I >>>> would. Maybe we could do that in relation to this PR, making sure >>>> that we set some reasonable time limit for getting it done, say 3 >>>> weeks. >>> >>> It's still the case that this is a novel social organization you >>> invented that AFAICT has never been tested by any F/OSS project, and >>> directly goes against the F/OSS community's hard-won cultural >>> knowledge about what kinds of organizations work well (see e.g. [1]). >>> Now- these are not necessarily bad things! Our community is >>> legitimately different than a "traditional" group of F/OSS developers >>> in a variety of ways, and less encultured to the "traditional" way of >>> doing things. And social experimentation is great -- how else can we >>> find better ways to live? While there's a lot of wisdom and experience >>> in Karl Fogel's book, it's surely not the final word. >>> >>> But... we should also be realistic that when someone shows up saying >>> "hey I've worked out a better method of social organization based on >>> first principles and thinking really hard, it'll 100% definitely be >>> awesome", then historically it *usually* doesn't quite work out so >>> nicely as promised. And it's often difficult to effectively do this >>> kind of experimentation at the same time as doing the actual work of >>> like, developing software. "Choose boring technology" [2] applies to >>> social technology too. >> >> * I agree with boring technology, but I doubt you're really arguing >> that choosing a leader at regular intervals is novel in open source or >> elsewhere. > > I'm arguing exactly that. (In open source, obviously; scipy is not a > nation-state.) or an arm of local government or a school or a ... >> Debian is an obvious example [1]; > > But the Debian Project Leader is *nothing at all* like a BDFL. In fact > their powers are extraordinarily limited; mostly it's just "convince > people to do stuff by talking to them" (i.e. "exercising leadership") > and "serve as a project figurehead". Which is what your links says! > They explicitly *cannot* make decisions about the technical direction > of the project; in the Debian system that power is delegated in a > complicated way to individual maintainers, mailing list consensus, the > CTTE, and GRs. Look - please - calm down. We can have serious calm discussion about this. Sure, Debian uses it's leader in a different way, as could we, I don't think we have to bring out the shotguns here. > If I seem frustrated in discussing these topics with you, then this is > why :-(. As is probably obvious to everyone, I actually love geeking > out about this kind of thing! But when you make such misleading and > hand-wavy arguments it feels lazy, like you're more interested in > vague in-principle discussions than in actually trying to put together > a real system that can be implemented and help the project move on and > accomplish its real goals. So - this is really very frustrating. I just proposed writing up a document, and comparing to the current one, in a short and reasonable period. I told you that Stefan, who's credentials as a project leader can't reasonably be challenged, is also thinking hard about this. It's terribly tiring to have to justify my good faith every time we have this discussion. I know that's not your intention and I'm > sorry if that sounds harsh. But at this point I'm having trouble > seeing how your comments are helping move things forward in any kind > of practical way. I don't know about harsh, but it certainly sounds impatient and patronizing. Incidentally, I had hoped you'd provide a couple of examples of BDFL projects where the BDFL was not the founder / major author. Maybe the discussion could get better if we covered stuff like that. >> * I don't know if an 'election' is the right method of choosing >> someone, that's really up for debate. Obviously an election is a very >> standard way of doing that; >> * I don't personally know of a BDFL system working well in the absence >> of the criteria I put above, but it would certainly be useful to have >> a look at a few examples. Can you suggest a few to consider? >> >>> If scikit-image is set on doing this, maybe the pragmatic thing to do >>> is wait and see how it works out for them? I've seen zero appetite >>> from anyone else on this list for elections and such. >> >> I think your idea here is the BDFL is low risk and choosing a leader >> is high risk, but it seems to me that both have risks, and that the >> best way of assessing the relative risks is to consider and refine a >> couple of concrete proposals, with discussion of prior experience, >> where applicable. >> >> For the appetite thing, you are probably referring to the nervous >> atmosphere that surrounds any discussion of governance, which is >> presumably due to the strong reactions against any such discussion in >> the past. I'm sure you'd agree that that 'get it over with as >> quickly as possible' is not the best way to come to a good solution. >> Having said that, if Ralf and / or Pauli do not have much interest in >> this topic, discussion will quickly become futile and >> counterproductive, and we will have to stop quickly to avoid making a >> mess. > > I'm more referring about the part where scipy *has* a governance > document now that seems perfectly workable. It's not identical to the > one I would have written, but so what, there are lots of workable > models and this looks like one of them. I'm not seeing folks jumping > in eager to redo that process for unclear benefits. I had hoped to cover that in my previous email. In practice, if Ralf and Pauli do not want to discuss this, this discussion is pointless, and we should stop this right now. However, contrary to your apparent assumption, I did not start this discussion to annoy, confuse or impress, I started it in the hope of finding the best possible model for Scipy governance, Best, Matthew From josef.pktd at gmail.com Fri Jan 13 23:47:50 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 13 Jan 2017 23:47:50 -0500 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Fri, Jan 13, 2017 at 10:57 PM, Matthew Brett wrote: > On Fri, Jan 13, 2017 at 7:38 PM, Nathaniel Smith wrote: > > On Fri, Jan 13, 2017 at 5:01 PM, Matthew Brett > wrote: > >> Hi, > >> > >> On Fri, Jan 13, 2017 at 4:15 PM, Nathaniel Smith wrote: > >>> On Fri, Jan 13, 2017 at 3:32 PM, Matthew Brett < > matthew.brett at gmail.com> wrote: > >>>> Hi, > >>>> > >>>> On Fri, Jan 13, 2017 at 3:24 PM, Evgeni Burovski > >>>> wrote: > >>>>> On Fri, Jan 13, 2017 at 8:22 PM, Matthew Brett < > matthew.brett at gmail.com> wrote: > >>> [...] > >>>>>> Of course that requires some formalization, but I think it's a > >>>>>> considerably better system than the BDFL, for our case. > >>>>> > >>>>> It seems to me that the effort needed to formalize it is not worth > the > >>>>> benefit, specifically in our case. > >>>> > >>>> Well - as a broader community, I think we'll have to do this anyway. > >>>> For example, I know that Stefan vdW wants to set up this model for > >>>> scikit-image. I am sure he'd be happy to help draft it, I know I > >>>> would. Maybe we could do that in relation to this PR, making sure > >>>> that we set some reasonable time limit for getting it done, say 3 > >>>> weeks. > >>> > >>> It's still the case that this is a novel social organization you > >>> invented that AFAICT has never been tested by any F/OSS project, and > >>> directly goes against the F/OSS community's hard-won cultural > >>> knowledge about what kinds of organizations work well (see e.g. [1]). > >>> Now- these are not necessarily bad things! Our community is > >>> legitimately different than a "traditional" group of F/OSS developers > >>> in a variety of ways, and less encultured to the "traditional" way of > >>> doing things. And social experimentation is great -- how else can we > >>> find better ways to live? While there's a lot of wisdom and experience > >>> in Karl Fogel's book, it's surely not the final word. > >>> > >>> But... we should also be realistic that when someone shows up saying > >>> "hey I've worked out a better method of social organization based on > >>> first principles and thinking really hard, it'll 100% definitely be > >>> awesome", then historically it *usually* doesn't quite work out so > >>> nicely as promised. And it's often difficult to effectively do this > >>> kind of experimentation at the same time as doing the actual work of > >>> like, developing software. "Choose boring technology" [2] applies to > >>> social technology too. > >> > >> * I agree with boring technology, but I doubt you're really arguing > >> that choosing a leader at regular intervals is novel in open source or > >> elsewhere. > > > > I'm arguing exactly that. (In open source, obviously; scipy is not a > > nation-state.) > > or an arm of local government or a school or a ... > > >> Debian is an obvious example [1]; > > > > But the Debian Project Leader is *nothing at all* like a BDFL. In fact > > their powers are extraordinarily limited; mostly it's just "convince > > people to do stuff by talking to them" (i.e. "exercising leadership") > > and "serve as a project figurehead". Which is what your links says! > > They explicitly *cannot* make decisions about the technical direction > > of the project; in the Debian system that power is delegated in a > > complicated way to individual maintainers, mailing list consensus, the > > CTTE, and GRs. > > Look - please - calm down. We can have serious calm discussion about > this. Sure, Debian uses it's leader in a different way, as could we, > I don't think we have to bring out the shotguns here. > > > If I seem frustrated in discussing these topics with you, then this is > > why :-(. As is probably obvious to everyone, I actually love geeking > > out about this kind of thing! But when you make such misleading and > > hand-wavy arguments it feels lazy, like you're more interested in > > vague in-principle discussions than in actually trying to put together > > a real system that can be implemented and help the project move on and > > accomplish its real goals. > > So - this is really very frustrating. I just proposed writing up a > document, and comparing to the current one, in a short and reasonable > period. I told you that Stefan, who's credentials as a project > leader can't reasonably be challenged, is also thinking hard about > this. It's terribly tiring to have to justify my good faith every > time we have this discussion. > > I know that's not your intention and I'm > > sorry if that sounds harsh. But at this point I'm having trouble > > seeing how your comments are helping move things forward in any kind > > of practical way. > > I don't know about harsh, but it certainly sounds impatient and > patronizing. > > Incidentally, I had hoped you'd provide a couple of examples of BDFL > projects where the BDFL was not the founder / major author. Maybe the > discussion could get better if we covered stuff like that. > The example is scipy. Pauli has been the implied BDFL for years with Ralf as "Assistant BDFL". Didn't matplotlib and pandas not also have transition to new de-facto BDFLs? I once mentioned in a related mailing list discussion that we should just codify the status quo which has worked and is working pretty well. Two of the longest scipy github discussions I have been involved with https://github.com/scipy/scipy/pull/448 compromise implemented https://github.com/scipy/scipy/issues/3129 rejected, full solution about 2 years later (In scipy stats I'm glad to leave all controversial decisions to Evgeni and Ralf, outside those related to statistical theory.) > > >> * I don't know if an 'election' is the right method of choosing > >> someone, that's really up for debate. Obviously an election is a very > >> standard way of doing that; > >> * I don't personally know of a BDFL system working well in the absence > >> of the criteria I put above, but it would certainly be useful to have > >> a look at a few examples. Can you suggest a few to consider? > >> > >>> If scikit-image is set on doing this, maybe the pragmatic thing to do > >>> is wait and see how it works out for them? I've seen zero appetite > >>> from anyone else on this list for elections and such. > >> > >> I think your idea here is the BDFL is low risk and choosing a leader > >> is high risk, but it seems to me that both have risks, and that the > >> best way of assessing the relative risks is to consider and refine a > >> couple of concrete proposals, with discussion of prior experience, > >> where applicable. > >> > >> For the appetite thing, you are probably referring to the nervous > >> atmosphere that surrounds any discussion of governance, which is > >> presumably due to the strong reactions against any such discussion in > >> the past. I'm sure you'd agree that that 'get it over with as > >> quickly as possible' is not the best way to come to a good solution. > >> Having said that, if Ralf and / or Pauli do not have much interest in > >> this topic, discussion will quickly become futile and > >> counterproductive, and we will have to stop quickly to avoid making a > >> mess. > > > > I'm more referring about the part where scipy *has* a governance > > document now that seems perfectly workable. It's not identical to the > > one I would have written, but so what, there are lots of workable > > models and this looks like one of them. I'm not seeing folks jumping > > in eager to redo that process for unclear benefits. > > I had hoped to cover that in my previous email. In practice, if Ralf > and Pauli do not want to discuss this, this discussion is pointless, > and we should stop this right now. However, contrary to your > apparent assumption, I did not start this discussion to annoy, confuse > or impress, I started it in the hope of finding the best possible > model for Scipy governance, > You are waiting for a pronouncement from our BDFLs? Josef > > Best, > > Matthew > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Jan 14 03:34:04 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 14 Jan 2017 00:34:04 -0800 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: Hi, On Fri, Jan 13, 2017 at 8:47 PM, wrote: > > > On Fri, Jan 13, 2017 at 10:57 PM, Matthew Brett > wrote: >> >> On Fri, Jan 13, 2017 at 7:38 PM, Nathaniel Smith wrote: >> > On Fri, Jan 13, 2017 at 5:01 PM, Matthew Brett >> > wrote: >> >> Hi, >> >> >> >> On Fri, Jan 13, 2017 at 4:15 PM, Nathaniel Smith wrote: >> >>> On Fri, Jan 13, 2017 at 3:32 PM, Matthew Brett >> >>> wrote: >> >>>> Hi, >> >>>> >> >>>> On Fri, Jan 13, 2017 at 3:24 PM, Evgeni Burovski >> >>>> wrote: >> >>>>> On Fri, Jan 13, 2017 at 8:22 PM, Matthew Brett >> >>>>> wrote: >> >>> [...] >> >>>>>> Of course that requires some formalization, but I think it's a >> >>>>>> considerably better system than the BDFL, for our case. >> >>>>> >> >>>>> It seems to me that the effort needed to formalize it is not worth >> >>>>> the >> >>>>> benefit, specifically in our case. >> >>>> >> >>>> Well - as a broader community, I think we'll have to do this anyway. >> >>>> For example, I know that Stefan vdW wants to set up this model for >> >>>> scikit-image. I am sure he'd be happy to help draft it, I know I >> >>>> would. Maybe we could do that in relation to this PR, making sure >> >>>> that we set some reasonable time limit for getting it done, say 3 >> >>>> weeks. >> >>> >> >>> It's still the case that this is a novel social organization you >> >>> invented that AFAICT has never been tested by any F/OSS project, and >> >>> directly goes against the F/OSS community's hard-won cultural >> >>> knowledge about what kinds of organizations work well (see e.g. [1]). >> >>> Now- these are not necessarily bad things! Our community is >> >>> legitimately different than a "traditional" group of F/OSS developers >> >>> in a variety of ways, and less encultured to the "traditional" way of >> >>> doing things. And social experimentation is great -- how else can we >> >>> find better ways to live? While there's a lot of wisdom and experience >> >>> in Karl Fogel's book, it's surely not the final word. >> >>> >> >>> But... we should also be realistic that when someone shows up saying >> >>> "hey I've worked out a better method of social organization based on >> >>> first principles and thinking really hard, it'll 100% definitely be >> >>> awesome", then historically it *usually* doesn't quite work out so >> >>> nicely as promised. And it's often difficult to effectively do this >> >>> kind of experimentation at the same time as doing the actual work of >> >>> like, developing software. "Choose boring technology" [2] applies to >> >>> social technology too. >> >> >> >> * I agree with boring technology, but I doubt you're really arguing >> >> that choosing a leader at regular intervals is novel in open source or >> >> elsewhere. >> > >> > I'm arguing exactly that. (In open source, obviously; scipy is not a >> > nation-state.) >> >> or an arm of local government or a school or a ... >> >> >> Debian is an obvious example [1]; >> > >> > But the Debian Project Leader is *nothing at all* like a BDFL. In fact >> > their powers are extraordinarily limited; mostly it's just "convince >> > people to do stuff by talking to them" (i.e. "exercising leadership") >> > and "serve as a project figurehead". Which is what your links says! >> > They explicitly *cannot* make decisions about the technical direction >> > of the project; in the Debian system that power is delegated in a >> > complicated way to individual maintainers, mailing list consensus, the >> > CTTE, and GRs. >> >> Look - please - calm down. We can have serious calm discussion about >> this. Sure, Debian uses it's leader in a different way, as could we, >> I don't think we have to bring out the shotguns here. >> >> > If I seem frustrated in discussing these topics with you, then this is >> > why :-(. As is probably obvious to everyone, I actually love geeking >> > out about this kind of thing! But when you make such misleading and >> > hand-wavy arguments it feels lazy, like you're more interested in >> > vague in-principle discussions than in actually trying to put together >> > a real system that can be implemented and help the project move on and >> > accomplish its real goals. >> >> So - this is really very frustrating. I just proposed writing up a >> document, and comparing to the current one, in a short and reasonable >> period. I told you that Stefan, who's credentials as a project >> leader can't reasonably be challenged, is also thinking hard about >> this. It's terribly tiring to have to justify my good faith every >> time we have this discussion. >> >> I know that's not your intention and I'm >> > sorry if that sounds harsh. But at this point I'm having trouble >> > seeing how your comments are helping move things forward in any kind >> > of practical way. >> >> I don't know about harsh, but it certainly sounds impatient and >> patronizing. >> >> Incidentally, I had hoped you'd provide a couple of examples of BDFL >> projects where the BDFL was not the founder / major author. Maybe the >> discussion could get better if we covered stuff like that. > > > The example is scipy. Pauli has been the implied BDFL for years with Ralf as > "Assistant BDFL". I don't think we knew who was in charge of Scipy until the candidates discussed that a few months ago. Remember that the thing we're discussing is the FL in BDFL. I get the impression we Scipiers mostly agree that having a named leader is a good idea. > Didn't matplotlib and pandas not also have transition to > new de-facto BDFLs? Here are the Pandas governance docs : https://github.com/pandas-dev/pandas-governance . Of course Pandas is a young project, but it also has a founder and main author, making the BDFL a more obvious model. Matplotlib did just adopt a similar doc, where the nominated BDFL is not the founder, but they've only had their governance doc for six months or so, and so we haven't got any data yet on the FL part: https://github.com/matplotlib/governance/blob/master/governance.md > I once mentioned in a related mailing list discussion that we should just > codify the status quo which has worked and is working pretty well. Yes, right - it is working well - but the point of a governance document is to keep it working well for a long time. The value of the 4 year (or whatever) cycle is to have a chance to review whether things are still working well, and make changes if they are not. > Two of the longest scipy github discussions I have been involved with > https://github.com/scipy/scipy/pull/448 compromise implemented > https://github.com/scipy/scipy/issues/3129 rejected, full solution about 2 > years later > > > (In scipy stats I'm glad to leave all controversial decisions to Evgeni and > Ralf, outside those related to statistical theory.) > > >> >> >> >> * I don't know if an 'election' is the right method of choosing >> >> someone, that's really up for debate. Obviously an election is a very >> >> standard way of doing that; >> >> * I don't personally know of a BDFL system working well in the absence >> >> of the criteria I put above, but it would certainly be useful to have >> >> a look at a few examples. Can you suggest a few to consider? >> >> >> >>> If scikit-image is set on doing this, maybe the pragmatic thing to do >> >>> is wait and see how it works out for them? I've seen zero appetite >> >>> from anyone else on this list for elections and such. >> >> >> >> I think your idea here is the BDFL is low risk and choosing a leader >> >> is high risk, but it seems to me that both have risks, and that the >> >> best way of assessing the relative risks is to consider and refine a >> >> couple of concrete proposals, with discussion of prior experience, >> >> where applicable. >> >> >> >> For the appetite thing, you are probably referring to the nervous >> >> atmosphere that surrounds any discussion of governance, which is >> >> presumably due to the strong reactions against any such discussion in >> >> the past. I'm sure you'd agree that that 'get it over with as >> >> quickly as possible' is not the best way to come to a good solution. >> >> Having said that, if Ralf and / or Pauli do not have much interest in >> >> this topic, discussion will quickly become futile and >> >> counterproductive, and we will have to stop quickly to avoid making a >> >> mess. >> > >> > I'm more referring about the part where scipy *has* a governance >> > document now that seems perfectly workable. It's not identical to the >> > one I would have written, but so what, there are lots of workable >> > models and this looks like one of them. I'm not seeing folks jumping >> > in eager to redo that process for unclear benefits. >> >> I had hoped to cover that in my previous email. In practice, if Ralf >> and Pauli do not want to discuss this, this discussion is pointless, >> and we should stop this right now. However, contrary to your >> apparent assumption, I did not start this discussion to annoy, confuse >> or impress, I started it in the hope of finding the best possible >> model for Scipy governance, > > > You are waiting for a pronouncement from our BDFLs? >From our de-facto leaders, yes. When the conversation veers off the content and onto the motivations or personal qualities of the participants, the only way of fixing that, is for someone with authority to intervene, calm the discussion down, and make space for the discussion to continue without rancor. If that doesn't happen, then the discussion is useless because there's no space to work out the ideas. Cheers, Matthew From evgeny.burovskiy at gmail.com Sat Jan 14 08:10:05 2017 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Sat, 14 Jan 2017 16:10:05 +0300 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: > Does Evgeni (for example) put in more work than he already does because he > wants to be elected the next BDF5YR? FTR, I don't :-). I'm perfectly happy with the current arrangement. From evgeny.burovskiy at gmail.com Sat Jan 14 08:49:07 2017 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Sat, 14 Jan 2017 16:49:07 +0300 Subject: [SciPy-Dev] API to control iteration in differential evolution #6923 In-Reply-To: References: Message-ID: On Wed, Jan 11, 2017 at 12:38 PM, Denis Laxalde wrote: > Hi, > > Recently there have been discussions about possible extensions of > scipy.optimize.differential_evolution() : > > * https://github.com/scipy/scipy/issues/6878 is about having the > callback function receive the function value > > * https://github.com/scipy/scipy/issues/6879 is about customizing of > population initialization > > The differential_evolution function being already quite complex, we > thought it would be better to expose a "lower level" API to control the > iteration in the algorithm. This would be done by making the > DifferentialEvolutionSolver class (in > scipy/optimize/_differentialevolution.py) public and cleaning it a bit. > > I submitted a pull request for this : > https://github.com/scipy/scipy/pull/6923 > One noticeable thing is that it also makes the class a context manager > to encapsulate the population initialization (in enter step) and final > polishing (in exit step). Ultimately, usage of this class (renamed as > DifferentialEvolution) would be: > > with DifferentialEvolution(func, bounds) as solver: > # iterate until maxiter/maxfev is reached or the algorithm converged > for step in solver: > if mycallback(step.x, step.fun): > break > result = solver.result > > I might look a bit fancy, but I think it makes sense to regard a > computation as a kind of context. Feedback welcome! FWIW, +1 from me. (I'm not a heavy user however) Exposing the iteration API seems reasonable and better than relying on callbacks. Small details, feel free to take or leave: 1. What is `step` in `for step in solver`? If it's a current "state of the minimizer", then can its contents be retrieved from the solver object itself (step vs `solver.result` above)? Or is it just a step number and the solver holds all the information at each step via `solver.result?` This needs to be specified and documented. 2. Since the solver class is iterable, just iterating it without the context manager should be equivalent. Something like this (pseudocode) solver = DifferentialEvolutionSolver(...) while not_converged: result = next(solver) should be equivalent to your example above (modulo callback). 3. My mental model would be that the solver being iterable and the context manager are both a syntax sugar for a pair of methods, roughly `Solver.make_a_single_step_then_pause()` and `Solver.do_full_minimization_until_convergence()` which repeatedly calls the first one. (This might be influenced by a DIY MC code I had a while ago, https://github.com/ev-br/mcba/blob/master/mcba/abstract_walker.py#L80, not sure how well it maps onto this case, so feel free to disregard) 4. Having the solver class iterable, with or without the context manager, forces the iteration to happen at the python level. This is likely not a problem with the current code, where the whole class is in python anyway. However if some future refactoring moves parts of the implementation to a compiled code, the overhead can get significant. This seems to argue for a black-box `do_a_full_simulation` method. (Again, this might be my MCMC background talking, where a single step is cheap and the python generator overhead is non-trivial.) Cheers, Evgeni From josef.pktd at gmail.com Sat Jan 14 09:31:10 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 14 Jan 2017 09:31:10 -0500 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Sat, Jan 14, 2017 at 3:34 AM, Matthew Brett wrote: > Hi, > > On Fri, Jan 13, 2017 at 8:47 PM, wrote: > > > > > > On Fri, Jan 13, 2017 at 10:57 PM, Matthew Brett > > > wrote: > >> > >> On Fri, Jan 13, 2017 at 7:38 PM, Nathaniel Smith wrote: > >> > On Fri, Jan 13, 2017 at 5:01 PM, Matthew Brett < > matthew.brett at gmail.com> > >> > wrote: > >> >> Hi, > >> >> > >> >> On Fri, Jan 13, 2017 at 4:15 PM, Nathaniel Smith > wrote: > >> >>> On Fri, Jan 13, 2017 at 3:32 PM, Matthew Brett > >> >>> wrote: > >> >>>> Hi, > >> >>>> > >> >>>> On Fri, Jan 13, 2017 at 3:24 PM, Evgeni Burovski > >> >>>> wrote: > >> >>>>> On Fri, Jan 13, 2017 at 8:22 PM, Matthew Brett > >> >>>>> wrote: > >> >>> [...] > >> >>>>>> Of course that requires some formalization, but I think it's a > >> >>>>>> considerably better system than the BDFL, for our case. > >> >>>>> > >> >>>>> It seems to me that the effort needed to formalize it is not worth > >> >>>>> the > >> >>>>> benefit, specifically in our case. > >> >>>> > >> >>>> Well - as a broader community, I think we'll have to do this > anyway. > >> >>>> For example, I know that Stefan vdW wants to set up this model for > >> >>>> scikit-image. I am sure he'd be happy to help draft it, I know I > >> >>>> would. Maybe we could do that in relation to this PR, making sure > >> >>>> that we set some reasonable time limit for getting it done, say 3 > >> >>>> weeks. > >> >>> > >> >>> It's still the case that this is a novel social organization you > >> >>> invented that AFAICT has never been tested by any F/OSS project, and > >> >>> directly goes against the F/OSS community's hard-won cultural > >> >>> knowledge about what kinds of organizations work well (see e.g. > [1]). > >> >>> Now- these are not necessarily bad things! Our community is > >> >>> legitimately different than a "traditional" group of F/OSS > developers > >> >>> in a variety of ways, and less encultured to the "traditional" way > of > >> >>> doing things. And social experimentation is great -- how else can we > >> >>> find better ways to live? While there's a lot of wisdom and > experience > >> >>> in Karl Fogel's book, it's surely not the final word. > >> >>> > >> >>> But... we should also be realistic that when someone shows up saying > >> >>> "hey I've worked out a better method of social organization based on > >> >>> first principles and thinking really hard, it'll 100% definitely be > >> >>> awesome", then historically it *usually* doesn't quite work out so > >> >>> nicely as promised. And it's often difficult to effectively do this > >> >>> kind of experimentation at the same time as doing the actual work of > >> >>> like, developing software. "Choose boring technology" [2] applies to > >> >>> social technology too. > >> >> > >> >> * I agree with boring technology, but I doubt you're really arguing > >> >> that choosing a leader at regular intervals is novel in open source > or > >> >> elsewhere. > >> > > >> > I'm arguing exactly that. (In open source, obviously; scipy is not a > >> > nation-state.) > >> > >> or an arm of local government or a school or a ... > >> > >> >> Debian is an obvious example [1]; > >> > > >> > But the Debian Project Leader is *nothing at all* like a BDFL. In fact > >> > their powers are extraordinarily limited; mostly it's just "convince > >> > people to do stuff by talking to them" (i.e. "exercising leadership") > >> > and "serve as a project figurehead". Which is what your links says! > >> > They explicitly *cannot* make decisions about the technical direction > >> > of the project; in the Debian system that power is delegated in a > >> > complicated way to individual maintainers, mailing list consensus, the > >> > CTTE, and GRs. > >> > >> Look - please - calm down. We can have serious calm discussion about > >> this. Sure, Debian uses it's leader in a different way, as could we, > >> I don't think we have to bring out the shotguns here. > >> > >> > If I seem frustrated in discussing these topics with you, then this is > >> > why :-(. As is probably obvious to everyone, I actually love geeking > >> > out about this kind of thing! But when you make such misleading and > >> > hand-wavy arguments it feels lazy, like you're more interested in > >> > vague in-principle discussions than in actually trying to put together > >> > a real system that can be implemented and help the project move on and > >> > accomplish its real goals. > >> > >> So - this is really very frustrating. I just proposed writing up a > >> document, and comparing to the current one, in a short and reasonable > >> period. I told you that Stefan, who's credentials as a project > >> leader can't reasonably be challenged, is also thinking hard about > >> this. It's terribly tiring to have to justify my good faith every > >> time we have this discussion. > >> > >> I know that's not your intention and I'm > >> > sorry if that sounds harsh. But at this point I'm having trouble > >> > seeing how your comments are helping move things forward in any kind > >> > of practical way. > >> > >> I don't know about harsh, but it certainly sounds impatient and > >> patronizing. > >> > >> Incidentally, I had hoped you'd provide a couple of examples of BDFL > >> projects where the BDFL was not the founder / major author. Maybe the > >> discussion could get better if we covered stuff like that. > > > > > > The example is scipy. Pauli has been the implied BDFL for years with > Ralf as > > "Assistant BDFL". > > I don't think we knew who was in charge of Scipy until the candidates > discussed that a few months ago. > It wasn't spelled out, but it was pretty clear how decisions were worked out. Actually, closer to the terms of Debian, Ralf was more the project lead in terms of pushing the "vision" and internal and external communication. Ralf also had the decision power of the release manager. Pauli has been the technical committee for final technical decisions. (Besides being also just regular contributors.) > > Remember that the thing we're discussing is the FL in BDFL. I get > the impression we Scipiers mostly agree that having a named leader is > a good idea. > I consider FL to mean For Unlimited Time. He or she can always step down and go on to other things and appoint a successor or let the steering council choose a successor. For Life is a long time in open source development, and Guido is the only one I know of that lasted for a long time. The point of a BDFL is also that we have more trust in a person than in the political system. So we choose a BDFL instead of having to write a constitution that has to survive and make the system function for hundreds of years and for a few hundred million members. Josef > > > Didn't matplotlib and pandas not also have transition to > > new de-facto BDFLs? > > Here are the Pandas governance docs : > https://github.com/pandas-dev/pandas-governance . Of course Pandas is > a young project, but it also has a founder and main author, making the > BDFL a more obvious model. > > Matplotlib did just adopt a similar doc, where the nominated BDFL is > not the founder, but they've only had their governance doc for six > months or so, and so we haven't got any data yet on the FL part: > > https://github.com/matplotlib/governance/blob/master/governance.md > > > I once mentioned in a related mailing list discussion that we should just > > codify the status quo which has worked and is working pretty well. > > Yes, right - it is working well - but the point of a governance > document is to keep it working well for a long time. The value of > the 4 year (or whatever) cycle is to have a chance to review whether > things are still working well, and make changes if they are not. > > > Two of the longest scipy github discussions I have been involved with > > https://github.com/scipy/scipy/pull/448 compromise implemented > > https://github.com/scipy/scipy/issues/3129 rejected, full solution > about 2 > > years later > > > > > > (In scipy stats I'm glad to leave all controversial decisions to Evgeni > and > > Ralf, outside those related to statistical theory.) > > > > > >> > >> > >> >> * I don't know if an 'election' is the right method of choosing > >> >> someone, that's really up for debate. Obviously an election is a > very > >> >> standard way of doing that; > >> >> * I don't personally know of a BDFL system working well in the > absence > >> >> of the criteria I put above, but it would certainly be useful to have > >> >> a look at a few examples. Can you suggest a few to consider? > >> >> > >> >>> If scikit-image is set on doing this, maybe the pragmatic thing to > do > >> >>> is wait and see how it works out for them? I've seen zero appetite > >> >>> from anyone else on this list for elections and such. > >> >> > >> >> I think your idea here is the BDFL is low risk and choosing a leader > >> >> is high risk, but it seems to me that both have risks, and that the > >> >> best way of assessing the relative risks is to consider and refine a > >> >> couple of concrete proposals, with discussion of prior experience, > >> >> where applicable. > >> >> > >> >> For the appetite thing, you are probably referring to the nervous > >> >> atmosphere that surrounds any discussion of governance, which is > >> >> presumably due to the strong reactions against any such discussion in > >> >> the past. I'm sure you'd agree that that 'get it over with as > >> >> quickly as possible' is not the best way to come to a good solution. > >> >> Having said that, if Ralf and / or Pauli do not have much interest in > >> >> this topic, discussion will quickly become futile and > >> >> counterproductive, and we will have to stop quickly to avoid making a > >> >> mess. > >> > > >> > I'm more referring about the part where scipy *has* a governance > >> > document now that seems perfectly workable. It's not identical to the > >> > one I would have written, but so what, there are lots of workable > >> > models and this looks like one of them. I'm not seeing folks jumping > >> > in eager to redo that process for unclear benefits. > >> > >> I had hoped to cover that in my previous email. In practice, if Ralf > >> and Pauli do not want to discuss this, this discussion is pointless, > >> and we should stop this right now. However, contrary to your > >> apparent assumption, I did not start this discussion to annoy, confuse > >> or impress, I started it in the hope of finding the best possible > >> model for Scipy governance, > > > > > > You are waiting for a pronouncement from our BDFLs? > > From our de-facto leaders, yes. When the conversation veers off the > content and onto the motivations or personal qualities of the > participants, the only way of fixing that, is for someone with > authority to intervene, calm the discussion down, and make space for > the discussion to continue without rancor. If that doesn't happen, > then the discussion is useless because there's no space to work out > the ideas. > > Cheers, > > Matthew > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jstevenson131 at gmail.com Sat Jan 14 09:42:09 2017 From: jstevenson131 at gmail.com (Jacob Stevenson) Date: Sat, 14 Jan 2017 14:42:09 +0000 Subject: [SciPy-Dev] API to control iteration in differential evolution #6923 In-Reply-To: References: Message-ID: I also give a big +1 to exposing the individual steps in the algorithm. However I echo Evgeni in that treating it as an iterable `for step in solver:` is a bit unintuitive and I prefer the more straightforward `solver. make_a_single_step_then_pause()`. For what it's worth the underlying solver in basinhopping uses this mechanism, although it's not exposed publicly. for i in range(niter): new_global_min = bh.one_cycle() If we expose this functionality in differential_evolution we could do the same in basinhopping using whatever common syntax is agreed upon. On Sat, 14 Jan 2017 at 13:49 Evgeni Burovski wrote: > On Wed, Jan 11, 2017 at 12:38 PM, Denis Laxalde wrote: > > Hi, > > > > Recently there have been discussions about possible extensions of > > scipy.optimize.differential_evolution() : > > > > * https://github.com/scipy/scipy/issues/6878 is about having the > > callback function receive the function value > > > > * https://github.com/scipy/scipy/issues/6879 is about customizing of > > population initialization > > > > The differential_evolution function being already quite complex, we > > thought it would be better to expose a "lower level" API to control the > > iteration in the algorithm. This would be done by making the > > DifferentialEvolutionSolver class (in > > scipy/optimize/_differentialevolution.py) public and cleaning it a bit. > > > > I submitted a pull request for this : > > https://github.com/scipy/scipy/pull/6923 > > One noticeable thing is that it also makes the class a context manager > > to encapsulate the population initialization (in enter step) and final > > polishing (in exit step). Ultimately, usage of this class (renamed as > > DifferentialEvolution) would be: > > > > with DifferentialEvolution(func, bounds) as solver: > > # iterate until maxiter/maxfev is reached or the algorithm converged > > for step in solver: > > if mycallback(step.x, step.fun): > > break > > result = solver.result > > > > I might look a bit fancy, but I think it makes sense to regard a > > computation as a kind of context. Feedback welcome! > > > FWIW, +1 from me. (I'm not a heavy user however) > > Exposing the iteration API seems reasonable and better than relying on > callbacks. > > Small details, feel free to take or leave: > > 1. What is `step` in `for step in solver`? If it's a current "state of > the minimizer", then can its contents be retrieved from the solver > object itself (step vs `solver.result` above)? Or is it just a step > number and the solver holds all the information at each step via > `solver.result?` This needs to be specified and documented. > > 2. Since the solver class is iterable, just iterating it without the > context manager should be equivalent. Something like this (pseudocode) > > solver = DifferentialEvolutionSolver(...) > while not_converged: > result = next(solver) > > should be equivalent to your example above (modulo callback). > > 3. My mental model would be that the solver being iterable and the > context manager are both a syntax sugar for a pair of methods, roughly > `Solver.make_a_single_step_then_pause()` and > `Solver.do_full_minimization_until_convergence()` which repeatedly > calls the first one. > (This might be influenced by a DIY MC code I had a while ago, > https://github.com/ev-br/mcba/blob/master/mcba/abstract_walker.py#L80, > not sure how well it maps onto this case, so feel free to disregard) > > 4. Having the solver class iterable, with or without the context > manager, forces the iteration to happen at the python level. > This is likely not a problem with the current code, where the whole > class is in python anyway. However if some future refactoring moves > parts of the implementation to a compiled code, the overhead can get > significant. This seems to argue for a black-box > `do_a_full_simulation` method. > (Again, this might be my MCMC background talking, where a single step > is cheap and the python generator overhead is non-trivial.) > > > Cheers, > > Evgeni > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Sat Jan 14 11:17:52 2017 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 14 Jan 2017 16:17:52 +0000 (UTC) Subject: [SciPy-Dev] SciPy governance model References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: Fri, 13 Jan 2017 09:22:58 -0800, Matthew Brett kirjoitti: [clip] > So, I propose, rather than have a BDFL, we have a system for choosing a > leader, say every 4 years, where we may or may not have limits on the > number of consecutive terms. I have no problem in principle with a model with a less permanent final decision authority role, or with more fallbacks for cases of serious conflict. However, I don't immediately know how to write down the details of such a model, and I think amendments would be best communicated via change suggestions in the PR. On the other hand, I must say I don't immediately see the value in making this role regularly rotating. This is because: (i) the person can step down at any time, and acting in good faith, will also listen to serious calls to do so, (ii) regular elections can generate unnecessary work and politics -- looking at the past 10 years of scipy, there has been little of this, and I believe this is for the best, (iii) this is more of a role for fallback decision making and less for a director/CEO. For those who did not read the linked document, the purpose of this particular role, despite the purposefully silly title, is mainly to enable the project to choose a direction in cases where consensus cannot be found. In the optimal case, this never happens, and there is no need to do anything. In particular, making decisions on the direction and vision for the project is in the hands of those on the steering council. Best regards, Pauli From denis at laxalde.org Sat Jan 14 13:03:43 2017 From: denis at laxalde.org (Denis Laxalde) Date: Sat, 14 Jan 2017 19:03:43 +0100 Subject: [SciPy-Dev] API to control iteration in differential evolution #6923 In-Reply-To: References: Message-ID: <6b000cae-8295-e690-39d3-054450d5cef1@laxalde.org> Evgeni Burovski a ?crit : > Small details, feel free to take or leave: > > 1. What is `step` in `for step in solver`? If it's a current "state of > the minimizer", then can its contents be retrieved from the solver > object itself (step vs `solver.result` above)? Or is it just a step > number and the solver holds all the information at each step via > `solver.result?` This needs to be specified and documented. This `step` is an `OptimizeStep`. It's a new class introduced here to wrap any useful information produced by the solver during one iteration. In this case, it holds the current `x` and function value along with iteration number. The idea is to return an extensible object instead of, say, a tuple that may cause bw compat issues in the future. I think this is documented, feel free to comment further if things are not clear enough. > 2. Since the solver class is iterable, just iterating it without the > context manager should be equivalent. Something like this (pseudocode) > > solver = DifferentialEvolutionSolver(...) > while not_converged: > result = next(solver) > > should be equivalent to your example above (modulo callback). It's already possible as: while not solver.converged(): step = next(solver) (only the `converged()` is new from this PR.) Also, the context manager interface is actually orthogonal to the iteration API. The context manager abstracts out the solver *initialization* (population initialization and energies calculation) and *termination* (possible polishing and build of the `OptimizeResult`). In the PR, there have been discussions to expose these steps as public methods to make the use of the context manager optional; I'm fine with this if it makes things clearer (though the use case is not clear to me). > > 3. My mental model would be that the solver being iterable and the > context manager are both a syntax sugar for a pair of methods, roughly > `Solver.make_a_single_step_then_pause()` and > `Solver.do_full_minimization_until_convergence()` which repeatedly > calls the first one. > (This might be influenced by a DIY MC code I had a while ago, > https://github.com/ev-br/mcba/blob/master/mcba/abstract_walker.py#L80, > not sure how well it maps onto this case, so feel free to disregard) This `make_a_single_step_then_pause` method is just called `__next__` (this is not new). I personally find `step = next(solver)` explicit enough not to require a more specific method, but that's arguable. The `Solver.do_full_minimization_until_convergence()` is basically the (convenience) function `differential_evolution`. There used to be a `solve()` method on the solver class but I dropped it as I did not see any use case where one would want this over just using the convenience function (also arguable). > 4. Having the solver class iterable, with or without the context > manager, forces the iteration to happen at the python level. > This is likely not a problem with the current code, where the whole > class is in python anyway. However if some future refactoring moves > parts of the implementation to a compiled code, the overhead can get > significant. This seems to argue for a black-box > `do_a_full_simulation` method. > (Again, this might be my MCMC background talking, where a single step > is cheap and the python generator overhead is non-trivial.) Can't really comment on this point. The "problem" already exists in the current implementation AFAICT. Maybe this means we'll ultimately have to move the whole class to a compiled code implementation? Thank you for your feedback. From matthew.brett at gmail.com Sat Jan 14 13:03:43 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 14 Jan 2017 10:03:43 -0800 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: Hi, Thanks for this reply - it's very helpful. On Sat, Jan 14, 2017 at 8:17 AM, Pauli Virtanen wrote: > Fri, 13 Jan 2017 09:22:58 -0800, Matthew Brett kirjoitti: > [clip] >> So, I propose, rather than have a BDFL, we have a system for choosing a >> leader, say every 4 years, where we may or may not have limits on the >> number of consecutive terms. > > I have no problem in principle with a model with a less permanent final > decision authority role, or with more fallbacks for cases of serious > conflict. However, I don't immediately know how to write down the details > of such a model, and I think amendments would be best communicated via > change suggestions in the PR. > > On the other hand, I must say I don't immediately see the value in making > this role regularly rotating. This is because: (i) the person can step > down at any time, and acting in good faith, will also listen to serious > calls to do so, (ii) regular elections can generate unnecessary work and > politics -- looking at the past 10 years of scipy, there has been little > of this, and I believe this is for the best, (iii) this is more of a role > for fallback decision making and less for a director/CEO. Yes, right, we should certainly try to avoid adding a political process where it isn't needed. What do you think about the idea of having regular state-of-scipy reviews to make sure we're conscious about keeping on track, assessing risks, improving process? Cheers, Matthew From denis at laxalde.org Sat Jan 14 13:08:31 2017 From: denis at laxalde.org (Denis Laxalde) Date: Sat, 14 Jan 2017 19:08:31 +0100 Subject: [SciPy-Dev] API to control iteration in differential evolution #6923 In-Reply-To: References: Message-ID: <1920903c-ce4e-03c0-2940-904890c7cf25@laxalde.org> Jacob Stevenson a ?crit : > I also give a big +1 to exposing the individual steps in the algorithm. > However I echo Evgeni in that treating it as an iterable `for step in > solver:` is a bit unintuitive and I prefer the more straightforward > `solver.make_a_single_step_then_pause()`. For what it's worth the > underlying solver in basinhopping uses this mechanism, although it's not > exposed publicly. > > for i in range(niter): > new_global_min = bh.one_cycle() See my answer to Evgeni's point; it's implemented as a `__next__` method. > If we expose this functionality in differential_evolution we could do > the same in basinhopping using whatever common syntax is agreed upon That'd be nice, indeed. From pav at iki.fi Sat Jan 14 14:15:49 2017 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 14 Jan 2017 19:15:49 +0000 (UTC) Subject: [SciPy-Dev] SciPy governance model References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: Hi, Sat, 14 Jan 2017 10:03:43 -0800, Matthew Brett kirjoitti: [clip] > What do you think about the idea of having regular state-of-scipy > reviews to make sure we're conscious about keeping on track, assessing > risks, improving process? For the technical aspect, this sounds something like the Scipy roadmap (https://github.com/scipy/scipy/blob/master/doc/ROADMAP.rst.txt), and the discussion leading to it, in conferences and online in public. Something like regular prompts for discussion of technical and organisation roadmap could be useful. At minimum, this could be simply a (bi-?)yearly post on the mailing list, to remind to update the roadmap and to summarize / bring up / discuss any relevant organisation updates / issues in the preceding period. It could also be something more involved (suggestions welcome), but probably useful to keep in mind this is drawing from a limited time pool. Best, Pauli From andyfaff at gmail.com Sat Jan 14 15:20:11 2017 From: andyfaff at gmail.com (Andrew Nelson) Date: Sun, 15 Jan 2017 07:20:11 +1100 Subject: [SciPy-Dev] API to control iteration in differential evolution #6923 In-Reply-To: <6b000cae-8295-e690-39d3-054450d5cef1@laxalde.org> References: <6b000cae-8295-e690-39d3-054450d5cef1@laxalde.org> Message-ID: On 15 January 2017 at 05:03, Denis Laxalde wrote: > Evgeni Burovski a ?crit : > >> Small details, feel free to take or leave: >> >> 1. What is `step` in `for step in solver`? If it's a current "state of >> the minimizer", then can its contents be retrieved from the solver >> object itself (step vs `solver.result` above)? Or is it just a step >> number and the solver holds all the information at each step via >> `solver.result?` This needs to be specified and documented. >> > > This `step` is an `OptimizeStep`. It's a new class introduced here to > wrap any useful information produced by the solver during one iteration. > In this case, it holds the current `x` and function value along with > iteration number. The idea is to return an extensible object instead of, > say, a tuple that may cause bw compat issues in the future. I think this > is documented, feel free to comment further if things are not clear enough. Best `x` and best energy are already available as `solver.x` and `solver.population_energy[0]`, but I think this kind of object being returned from `__next__` is a good idea. Does it have to be an `OptimizeStep`, or can we return an preliminary `OptimizeResult`? 2. Since the solver class is iterable, just iterating it without the >> context manager should be equivalent. Something like this (pseudocode) >> >> solver = DifferentialEvolutionSolver(...) >> while not_converged: >> result = next(solver) >> > > Also, the context manager interface is actually orthogonal to the > iteration API. The context manager abstracts out the solver > *initialization* (population initialization and energies calculation) > and *termination* (possible polishing and build of the > `OptimizeResult`). > Whilst the context manager is orthogonal to the iteration API, Evgeni's example would not be possible with the PR is it currently exists - the only way of initialising the solver (population initialization and energies calculation) would be to go through the context manager. Denis suggested that a way around this would be to have a separate initialize_population() method. It's a good idea to have this method (for reasons I outline later). However, if you create the object outside a context manager I don't think that one should have to call an additional initialize_population() method to be able to iterate the object. There are a few reasons for this: (a) it's an extra line to type (b) iterating the object without going through the context manager is not equivalent (c) the solver object possesses public population and population_energies attributes, these attributes need to be present in a newly created object, not be created at a later stage (d) if the population and population_energies attributes are present I'd argue it makes more sense for them to contain relevant values (e) given a solver object how does one know if initialize_population has already been called (which would be a common occurrence when used in an interactive interpreter)? (f) you can't call next(solver) if the population hasn't been initialised. You shouldn't need to know the state of the solver object to be able to iterate it, it should be iterable straightaway. Currently the initial energy calculations are done in `__init__` and one uses the object as: # stepwise solver = DifferentialEvolutionSolver(...) while not solver.converged: next(solver) -or- # all at once solver = DifferentialEvolutionSolver(...) result = solver.solve() The proposed PR: with solver as DifferentialEvolutionSolver(...): while not solver.converged(): next(solver) result = solver.result -or possibly- solver = DifferentialEvolutionSolver(...) # extra line to type, solver.initialize_population() while not solver.converged: next(solver) Perhaps the following course could be followed: (1) make the __next__ method return an OptimizeStep object (or OptimizeResult). (2) add a polish() method that takes the current best solution and polishes it. Having a separate polish method is useful as you can polish the object at any step, getting functionality similar to optimize.brute. (3) Add __enter__() and __exit__() methods to allow the object to be used as a context manager. The __exit__() method can use the public polish() method. I'd like to keep the existing solve() method, it can call __exit__() at the end. (4) population initialization and the initial energy calculation is moved to a public method, initialize_population(). This method is either called from __init__ (the status quo), or it is called on the first call to __next__(), making the first call to __next__ take twice as long. My preference is to keep the status quo because one can create the solver and immediately polish the best population member (c.f. optimize.brute), as well as the points raised above. The rationale for making it public is that once you've done one optimization you can re-initialise it with a different method ('latinhypercube' or 'random') and solve again or you could imagine re-initialising the population with a given population array (which has been requested previously) and start iterating again. A. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Jan 15 18:43:41 2017 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 15 Jan 2017 16:43:41 -0700 Subject: [SciPy-Dev] NumPy 1.12.0 release Message-ID: Hi All, I'm pleased to announce the NumPy 1.12.0 release. This release supports Python 2.7 and 3.4-3.6. Wheels for all supported Python versions may be downloaded from PiPY , the tarball and zip files may be downloaded from Github . The release notes and files hashes may also be found at Github . NumPy 1.12.0rc 2 is the result of 418 pull requests submitted by 139 contributors and comprises a large number of fixes and improvements. Among the many improvements it is difficult to pick out just a few as standing above the others, but the following may be of particular interest or indicate areas likely to have future consequences. * Order of operations in ``np.einsum`` can now be optimized for large speed improvements. * New ``signature`` argument to ``np.vectorize`` for vectorizing with core dimensions. * The ``keepdims`` argument was added to many functions. * New context manager for testing warnings * Support for BLIS in numpy.distutils * Much improved support for PyPy (not yet finished) Enjoy, Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Jan 16 04:01:26 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 16 Jan 2017 22:01:26 +1300 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Sun, Jan 15, 2017 at 8:15 AM, Pauli Virtanen wrote: Interesting discussion so far! > > Sat, 14 Jan 2017 10:03:43 -0800, Matthew Brett kirjoitti: > [clip] > > What do you think about the idea of having regular state-of-scipy > > reviews to make sure we're conscious about keeping on track, assessing > > risks, improving process? > > For the technical aspect, this sounds something like the Scipy roadmap > (https://github.com/scipy/scipy/blob/master/doc/ROADMAP.rst.txt), and the > discussion leading to it, in conferences and online in public. > > Something like regular prompts for discussion of technical and > organisation roadmap could be useful. At minimum, this could be simply a > (bi-?)yearly post on the mailing list, to remind to update the roadmap > and to summarize / bring up / discuss any relevant organisation updates / > issues in the preceding period. > I quite like this idea. Documents like a roadmap can easily go out of date if they're not actively maintained. Having a critical look at it once or twice a year will be helpful. Also +1 for adding some organizational items to it (I'm thinking CoC, FSA, etc. should have been on there). The list of people on the steering committee also needs to be updated with this kind of frequency. How about doing this around 1 January and 1 July every year? I'm not sensing a lot of enthusiasm for the fixed-term/election idea, so I'd gently suggest to not go down that path but instead look at the regular review above as an opportunity to bring up concerns regarding any aspect of our organisational structure (we can put in wording like that, PR welcome I'd say). Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Jan 16 04:32:31 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 16 Jan 2017 22:32:31 +1300 Subject: [SciPy-Dev] branching 0.19.x soon? Message-ID: Hi all, We had planned the 0.19.0 release for the end of January. We won't make that anymore, but branching this month seems doable (nothing particularly large or blocking in https://github.com/scipy/scipy/milestone/30). @Evgeni, did you plan / want to be the release manager for this release again, or do you want someone else to take that role? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Jan 16 06:05:08 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 17 Jan 2017 00:05:08 +1300 Subject: [SciPy-Dev] Include Allan variance estimators in SciPy In-Reply-To: <654beb7c-12ab-7af7-30a8-97949b2dfff9@grinta.net> References: <654beb7c-12ab-7af7-30a8-97949b2dfff9@grinta.net> Message-ID: On Fri, Jan 13, 2017 at 1:39 AM, Daniele Nicolodi wrote: > Hello, > > the Allan family of statistical estimators [0] (Allan variance, modified > Allan variance, total variance) are extensively used in some research > fields, like frequency metrology and related fields. > > While the algorithm to compute the Allan variance is quite straight > forward, efficiently coding the modified Allan variance, or the > estimator based on phase rather than on frequency samples, are a bit > more tricky. > > I have implementations of those (in cython) that I would like to > contribute to SciPy, most probably as an `scipy.stats.allan` module. > Does people deem those interesting? Should I prepare a pull request to > discuss the implementation? > There was a reasonable amount of interest last time we discussed it [1] and it seems applicable in more than one field, so I'd say yes. A separate module is overkill though - a single function ``stats.allan`` with perhaps a keyword to select between the different variants of the algorithm might make sense? Cheers, Ralf [1] https://mail.python.org/pipermail/scipy-user/2013-March/034254.html > Thanks. Cheers, > Daniele > > [0] https://en.wikipedia.org/wiki/Allan_variance > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carsonc at gmail.com Mon Jan 16 09:23:01 2017 From: carsonc at gmail.com (Cantwell Carson) Date: Mon, 16 Jan 2017 09:23:01 -0500 Subject: [SciPy-Dev] Include Allan variance estimators in SciPy In-Reply-To: References: <654beb7c-12ab-7af7-30a8-97949b2dfff9@grinta.net> Message-ID: On Jan 16, 2017 6:05 AM, "Ralf Gommers" wrote: On Fri, Jan 13, 2017 at 1:39 AM, Daniele Nicolodi wrote: > Hello, > > the Allan family of statistical estimators [0] (Allan variance, modified > Allan variance, total variance) are extensively used in some research > fields, like frequency metrology and related fields. > > While the algorithm to compute the Allan variance is quite straight > forward, efficiently coding the modified Allan variance, or the > estimator based on phase rather than on frequency samples, are a bit > more tricky. > > I have implementations of those (in cython) that I would like to > contribute to SciPy, most probably as an `scipy.stats.allan` module. > Does people deem those interesting? Should I prepare a pull request to > discuss the implementation? > There was a reasonable amount of interest last time we discussed it [1] and it seems applicable in more than one field, so I'd say yes. A separate module is overkill though - a single function ``stats.allan`` with perhaps a keyword to select between the different variants of the algorithm might make sense? Cheers, Ralf [1] https://mail.python.org/pipermail/scipy-user/2013-March/034254.html > Thanks. Cheers, > Daniele > > [0] https://en.wikipedia.org/wiki/Allan_variance > _______________________________________________ SciPy-Dev mailing list SciPy-Dev at scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev Dear all, I just wanted to make a plug here for the allantools package by Anders Wallin (cc'ed), and to which I am a contributor. It was indispensable to me and, though not complete (what project ever is), it is nonetheless very comprehensive with respect to variance estimators and their uncertainty. Take a look and see if there might be something worth incorporating. Yours, Cantwell G. Carson -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Jan 16 10:07:12 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 16 Jan 2017 10:07:12 -0500 Subject: [SciPy-Dev] Include Allan variance estimators in SciPy In-Reply-To: References: <654beb7c-12ab-7af7-30a8-97949b2dfff9@grinta.net> Message-ID: On Mon, Jan 16, 2017 at 9:23 AM, Cantwell Carson wrote: > > > On Jan 16, 2017 6:05 AM, "Ralf Gommers" wrote: > > > > On Fri, Jan 13, 2017 at 1:39 AM, Daniele Nicolodi > wrote: > >> Hello, >> >> the Allan family of statistical estimators [0] (Allan variance, modified >> Allan variance, total variance) are extensively used in some research >> fields, like frequency metrology and related fields. >> >> While the algorithm to compute the Allan variance is quite straight >> forward, efficiently coding the modified Allan variance, or the >> estimator based on phase rather than on frequency samples, are a bit >> more tricky. >> >> I have implementations of those (in cython) that I would like to >> contribute to SciPy, most probably as an `scipy.stats.allan` module. >> Does people deem those interesting? Should I prepare a pull request to >> discuss the implementation? >> > > There was a reasonable amount of interest last time we discussed it [1] > and it seems applicable in more than one field, so I'd say yes. > > A separate module is overkill though - a single function ``stats.allan`` > with perhaps a keyword to select between the different variants of the > algorithm might make sense? > > Cheers, > Ralf > > > [1] https://mail.python.org/pipermail/scipy-user/2013-March/034254.html > > >> Thanks. Cheers, >> Daniele >> >> [0] https://en.wikipedia.org/wiki/Allan_variance >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > > > Dear all, > > I just wanted to make a plug here for the allantools package by Anders > Wallin (cc'ed), and to which I am a contributor. It was indispensable to me > and, though not complete (what project ever is), it is nonetheless very > comprehensive with respect to variance estimators and their uncertainty. > > Take a look and see if there might be something worth incorporating. > allentools is GPLv3+, so any code would need to be relicensed to be compatible with BSD. Josef > > Yours, > Cantwell G. Carson > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.linke88 at gmail.com Mon Jan 16 10:23:05 2017 From: max.linke88 at gmail.com (Max Linke) Date: Mon, 16 Jan 2017 16:23:05 +0100 Subject: [SciPy-Dev] GSOC 2017: NumFOCUS will be an umbrella organization Message-ID: <40f85dfb-0635-a042-79b7-039d3dd347a9@gmail.com> Hi Organizations can start submitting applications for Google Summer of Code 2017 on January 19 (and the deadline is February 9) https://developers.google.com/open-source/gsoc/timeline?hl=en NumFOCUS will be applying again this year. If you want to work with us please let me know and if you apply as an organization yourself or under a different umbrella organization please tell me as well. If you participate with us it would be great if you start to add possible projects to the ideas page on github soon. We some general information for mentors on github. https://github.com/numfocus/gsoc/blob/master/CONTRIBUTING-mentors.md We also have a template for ideas that might help. It lists the things Google likes to see. https://github.com/numfocus/gsoc/blob/master/2017/ideas-list-skeleton.md In case you participated in earlier years with NumFOCUS there are some small changes this year. Raniere won't be the admin this year. Instead I'm going to be the admin. We are also planning to include two explicit rules when a student should be failed, they have to communicate regularly and commit code into your development branch at the end of the summer. best, Max From perimosocordiae at gmail.com Mon Jan 16 12:35:30 2017 From: perimosocordiae at gmail.com (CJ Carey) Date: Mon, 16 Jan 2017 11:35:30 -0600 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Mon, Jan 16, 2017 at 3:01 AM, Ralf Gommers wrote: > How about doing this around 1 January and 1 July every year? > Those dates fall during times when many people are traveling or otherwise not responsive. What about phase-shifting 3 months to April 1st / October 1st? -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Jan 16 12:39:51 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 16 Jan 2017 09:39:51 -0800 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: Hi, On Mon, Jan 16, 2017 at 1:01 AM, Ralf Gommers wrote: > > > On Sun, Jan 15, 2017 at 8:15 AM, Pauli Virtanen wrote: > > Interesting discussion so far! > >> >> >> Sat, 14 Jan 2017 10:03:43 -0800, Matthew Brett kirjoitti: >> [clip] >> > What do you think about the idea of having regular state-of-scipy >> > reviews to make sure we're conscious about keeping on track, assessing >> > risks, improving process? >> >> For the technical aspect, this sounds something like the Scipy roadmap >> (https://github.com/scipy/scipy/blob/master/doc/ROADMAP.rst.txt), and the >> discussion leading to it, in conferences and online in public. >> >> Something like regular prompts for discussion of technical and >> organisation roadmap could be useful. At minimum, this could be simply a >> (bi-?)yearly post on the mailing list, to remind to update the roadmap >> and to summarize / bring up / discuss any relevant organisation updates / >> issues in the preceding period. > > > I quite like this idea. Documents like a roadmap can easily go out of date > if they're not actively maintained. Having a critical look at it once or > twice a year will be helpful. Also +1 for adding some organizational items > to it (I'm thinking CoC, FSA, etc. should have been on there). > > The list of people on the steering committee also needs to be updated with > this kind of frequency. > > How about doing this around 1 January and 1 July every year? > > I'm not sensing a lot of enthusiasm for the fixed-term/election idea, I'm afraid the previous discussion on this thread has made it very unlikely that you would see any enthusiasm, even if, in another world, it was a good idea. That's the tragedy of mailing list discussions, as satirized here http://www.smbc-comics.com/?id=2939 and evident from the recent discussion on the Jupyter governance document. If anyone gets angry or dismissive about an idea, and that isn't addressed, that's a very effective way of inducing consent and shutting down the discussion. Cheers, Matthew From josef.pktd at gmail.com Mon Jan 16 14:19:13 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 16 Jan 2017 14:19:13 -0500 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Mon, Jan 16, 2017 at 12:39 PM, Matthew Brett wrote: > Hi, > > On Mon, Jan 16, 2017 at 1:01 AM, Ralf Gommers > wrote: > > > > > > On Sun, Jan 15, 2017 at 8:15 AM, Pauli Virtanen wrote: > > > > Interesting discussion so far! > > > >> > >> > >> Sat, 14 Jan 2017 10:03:43 -0800, Matthew Brett kirjoitti: > >> [clip] > >> > What do you think about the idea of having regular state-of-scipy > >> > reviews to make sure we're conscious about keeping on track, assessing > >> > risks, improving process? > >> > >> For the technical aspect, this sounds something like the Scipy roadmap > >> (https://github.com/scipy/scipy/blob/master/doc/ROADMAP.rst.txt), and > the > >> discussion leading to it, in conferences and online in public. > >> > >> Something like regular prompts for discussion of technical and > >> organisation roadmap could be useful. At minimum, this could be simply a > >> (bi-?)yearly post on the mailing list, to remind to update the roadmap > >> and to summarize / bring up / discuss any relevant organisation updates > / > >> issues in the preceding period. > > > > > > I quite like this idea. Documents like a roadmap can easily go out of > date > > if they're not actively maintained. Having a critical look at it once or > > twice a year will be helpful. Also +1 for adding some organizational > items > > to it (I'm thinking CoC, FSA, etc. should have been on there). > > > > The list of people on the steering committee also needs to be updated > with > > this kind of frequency. > > > > How about doing this around 1 January and 1 July every year? > > > > I'm not sensing a lot of enthusiasm for the fixed-term/election idea, > > I'm afraid the previous discussion on this thread has made it very > unlikely that you would see any enthusiasm, even if, in another world, > it was a good idea. That's the tragedy of mailing list discussions, > as satirized here > > http://www.smbc-comics.com/?id=2939 > > and evident from the recent discussion on the Jupyter governance > document. If anyone gets angry or dismissive about an idea, and that > isn't addressed, that's a very effective way of inducing consent and > shutting down the discussion. > Matthew, I think that's mis-characterizing this a bit. We had several governance discussion over the last years, and my impression was that the vast majority of those commenting where in favor of a more laid back approach instead of a fully specified constitution and associated discussion. (I think what's missing is a chair wo/man of the steering council and heads/lieutenants for each sup-package, and a specification of the rights of the release manager and maybe some more.) IMO, given the current contributors and the way contributors are recruited and integrated, I don't see much difference between unlimited time and fixed time with renewal. So, we can as well stick with what we know. Josef > > Cheers, > > Matthew > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Mon Jan 16 14:56:23 2017 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Mon, 16 Jan 2017 22:56:23 +0300 Subject: [SciPy-Dev] branching 0.19.x soon? In-Reply-To: References: Message-ID: > We had planned the 0.19.0 release for the end of January. We won't make that > anymore, but branching this month seems doable (nothing particularly large > or blocking in https://github.com/scipy/scipy/milestone/30). Thanks for bringing this up Ralf! The dblquad regression would be good to fix. (Ah, as noted on GH already) Can't think of other blockers off the top of my head. Some more time for merging open PRs would be nice to have, but optional. > @Evgeni, did you plan / want to be the release manager for this release > again, or do you want someone else to take that role? If someone wants it, great. Volunteers? Otherwise I'll happily do it. Cheers, Evgeni From pav at iki.fi Mon Jan 16 15:09:41 2017 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 16 Jan 2017 20:09:41 +0000 (UTC) Subject: [SciPy-Dev] SciPy governance model References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: Mon, 16 Jan 2017 11:35:30 -0600, CJ Carey kirjoitti: > On Mon, Jan 16, 2017 at 3:01 AM, Ralf Gommers > wrote: > >> How about doing this around 1 January and 1 July every year? >> >> > Those dates fall during times when many people are traveling or > otherwise not responsive. What about phase-shifting 3 months to April > 1st / October 1st? Not that important, but now that we are optimizing this, I would push it a bit forward to eg. Apr 20 and Oct 20 to avoid beginning of academic semesters. Pauli From ralf.gommers at gmail.com Tue Jan 17 03:54:21 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 17 Jan 2017 21:54:21 +1300 Subject: [SciPy-Dev] branching 0.19.x soon? In-Reply-To: References: Message-ID: On Tue, Jan 17, 2017 at 8:56 AM, Evgeni Burovski wrote: > > We had planned the 0.19.0 release for the end of January. We won't make > that > > anymore, but branching this month seems doable (nothing particularly > large > > or blocking in https://github.com/scipy/scipy/milestone/30). > > Thanks for bringing this up Ralf! > > The dblquad regression would be good to fix. > (Ah, as noted on GH already) > Can't think of other blockers off the top of my head. Some more time > for merging open PRs would be nice to have, but optional. > It seems like we're always hovering around 130-140 PRs, no matter how much time we take. But probably good to give it say two weeks, so everyone has a chance to go and review/merge the PRs they would really like to get in? > > > @Evgeni, did you plan / want to be the release manager for this release > > again, or do you want someone else to take that role? > > If someone wants it, great. Volunteers? > Otherwise I'll happily do it. > "want" is a big word:) I'm also happy to take a turn again, either for this release or a following one. But if someone new to the role volunteers, that'd be quite nice. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Jan 17 13:02:42 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 17 Jan 2017 10:02:42 -0800 Subject: [SciPy-Dev] [Numpy-discussion] NumPy 1.12.0 release In-Reply-To: References: Message-ID: Hi, On Tue, Jan 17, 2017 at 5:56 AM, Neal Becker wrote: > Charles R Harris wrote: > >> Hi All, >> >> I'm pleased to announce the NumPy 1.12.0 release. This release supports >> Python 2.7 and 3.4-3.6. Wheels for all supported Python versions may be >> downloaded from PiPY >> , the tarball >> and zip files may be downloaded from Github >> . The release notes >> and files hashes may also be found at Github >> . >> >> NumPy 1.12.0rc 2 is the result of 418 pull requests submitted by 139 >> contributors and comprises a large number of fixes and improvements. Among >> the many improvements it is difficult to pick out just a few as standing >> above the others, but the following may be of particular interest or >> indicate areas likely to have future consequences. >> >> * Order of operations in ``np.einsum`` can now be optimized for large >> speed improvements. >> * New ``signature`` argument to ``np.vectorize`` for vectorizing with core >> dimensions. >> * The ``keepdims`` argument was added to many functions. >> * New context manager for testing warnings >> * Support for BLIS in numpy.distutils >> * Much improved support for PyPy (not yet finished) >> >> Enjoy, >> >> Chuck > > I've installed via pip3 on linux x86_64, which gives me a wheel. My > question is, am I loosing significant performance choosing this pre-built > binary vs. compiling myself? For example, my processor might have some more > features than the base version used to build wheels. I guess you are thinking about using this built wheel on some other machine? You'd have to be lucky for that to work; the wheel depends on the symbols it found at build time, which may not exist in the same places on your other machine. If it does work, the speed will primarily depend on your BLAS library. The pypi wheels should be pretty fast; they are built with OpenBLAS, which is at or near top of range for speed, across a range of platforms. Cheers, Matthew From ndbecker2 at gmail.com Tue Jan 17 18:47:46 2017 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 17 Jan 2017 18:47:46 -0500 Subject: [SciPy-Dev] [Numpy-discussion] NumPy 1.12.0 release References: Message-ID: Matthew Brett wrote: > Hi, > > On Tue, Jan 17, 2017 at 5:56 AM, Neal Becker wrote: >> Charles R Harris wrote: >> >>> Hi All, >>> >>> I'm pleased to announce the NumPy 1.12.0 release. This release supports >>> Python 2.7 and 3.4-3.6. Wheels for all supported Python versions may be >>> downloaded from PiPY >>> , the >>> tarball and zip files may be downloaded from Github >>> . The release notes >>> and files hashes may also be found at Github >>> . >>> >>> NumPy 1.12.0rc 2 is the result of 418 pull requests submitted by 139 >>> contributors and comprises a large number of fixes and improvements. >>> Among >>> the many improvements it is difficult to pick out just a few as >>> standing above the others, but the following may be of particular >>> interest or indicate areas likely to have future consequences. >>> >>> * Order of operations in ``np.einsum`` can now be optimized for large >>> speed improvements. >>> * New ``signature`` argument to ``np.vectorize`` for vectorizing with >>> core dimensions. >>> * The ``keepdims`` argument was added to many functions. >>> * New context manager for testing warnings >>> * Support for BLIS in numpy.distutils >>> * Much improved support for PyPy (not yet finished) >>> >>> Enjoy, >>> >>> Chuck >> >> I've installed via pip3 on linux x86_64, which gives me a wheel. My >> question is, am I loosing significant performance choosing this pre-built >> binary vs. compiling myself? For example, my processor might have some >> more features than the base version used to build wheels. > > I guess you are thinking about using this built wheel on some other > machine? You'd have to be lucky for that to work; the wheel depends > on the symbols it found at build time, which may not exist in the same > places on your other machine. > > If it does work, the speed will primarily depend on your BLAS library. > > The pypi wheels should be pretty fast; they are built with OpenBLAS, > which is at or near top of range for speed, across a range of > platforms. > > Cheers, > > Matthew I installed using pip3 install, and it installed a wheel package. I did not build it - aren't wheels already compiled packages? So isn't it built for the common denominator architecture, not necessarily as fast as one I built myself on my own machine? My question is, on x86_64, is this potential difference large enough to bother with not using precompiled wheel packages? From matthew.brett at gmail.com Tue Jan 17 19:14:14 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 17 Jan 2017 16:14:14 -0800 Subject: [SciPy-Dev] [Numpy-discussion] NumPy 1.12.0 release In-Reply-To: References: Message-ID: On Tue, Jan 17, 2017 at 3:47 PM, Neal Becker wrote: > Matthew Brett wrote: > >> Hi, >> >> On Tue, Jan 17, 2017 at 5:56 AM, Neal Becker wrote: >>> Charles R Harris wrote: >>> >>>> Hi All, >>>> >>>> I'm pleased to announce the NumPy 1.12.0 release. This release supports >>>> Python 2.7 and 3.4-3.6. Wheels for all supported Python versions may be >>>> downloaded from PiPY >>>> , the >>>> tarball and zip files may be downloaded from Github >>>> . The release notes >>>> and files hashes may also be found at Github >>>> . >>>> >>>> NumPy 1.12.0rc 2 is the result of 418 pull requests submitted by 139 >>>> contributors and comprises a large number of fixes and improvements. >>>> Among >>>> the many improvements it is difficult to pick out just a few as >>>> standing above the others, but the following may be of particular >>>> interest or indicate areas likely to have future consequences. >>>> >>>> * Order of operations in ``np.einsum`` can now be optimized for large >>>> speed improvements. >>>> * New ``signature`` argument to ``np.vectorize`` for vectorizing with >>>> core dimensions. >>>> * The ``keepdims`` argument was added to many functions. >>>> * New context manager for testing warnings >>>> * Support for BLIS in numpy.distutils >>>> * Much improved support for PyPy (not yet finished) >>>> >>>> Enjoy, >>>> >>>> Chuck >>> >>> I've installed via pip3 on linux x86_64, which gives me a wheel. My >>> question is, am I loosing significant performance choosing this pre-built >>> binary vs. compiling myself? For example, my processor might have some >>> more features than the base version used to build wheels. >> >> I guess you are thinking about using this built wheel on some other >> machine? You'd have to be lucky for that to work; the wheel depends >> on the symbols it found at build time, which may not exist in the same >> places on your other machine. >> >> If it does work, the speed will primarily depend on your BLAS library. >> >> The pypi wheels should be pretty fast; they are built with OpenBLAS, >> which is at or near top of range for speed, across a range of >> platforms. >> >> Cheers, >> >> Matthew > > I installed using pip3 install, and it installed a wheel package. I did not > build it - aren't wheels already compiled packages? So isn't it built for > the common denominator architecture, not necessarily as fast as one I built > myself on my own machine? My question is, on x86_64, is this potential > difference large enough to bother with not using precompiled wheel packages? Ah - my guess is that you'd be hard pressed to make a numpy that is as fast as the precompiled wheel. The OpenBLAS library included in numpy selects the routines for your CPU at run-time, so they will generally be fast on your CPU. You might be able to get equivalent or even better performance with a ATLAS BLAS library recompiled on your exact machine, but that's quite a serious investment of time to get working, and you'd have to benchmark to find if you were really doing any better. Cheers, Matthew From njs at pobox.com Tue Jan 17 19:20:12 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 17 Jan 2017 16:20:12 -0800 Subject: [SciPy-Dev] [Numpy-discussion] NumPy 1.12.0 release In-Reply-To: References: Message-ID: On Tue, Jan 17, 2017 at 3:47 PM, Neal Becker wrote: > Matthew Brett wrote: > >> Hi, >> >> On Tue, Jan 17, 2017 at 5:56 AM, Neal Becker wrote: >>> Charles R Harris wrote: >>> >>>> Hi All, >>>> >>>> I'm pleased to announce the NumPy 1.12.0 release. This release supports >>>> Python 2.7 and 3.4-3.6. Wheels for all supported Python versions may be >>>> downloaded from PiPY >>>> , the >>>> tarball and zip files may be downloaded from Github >>>> . The release notes >>>> and files hashes may also be found at Github >>>> . >>>> >>>> NumPy 1.12.0rc 2 is the result of 418 pull requests submitted by 139 >>>> contributors and comprises a large number of fixes and improvements. >>>> Among >>>> the many improvements it is difficult to pick out just a few as >>>> standing above the others, but the following may be of particular >>>> interest or indicate areas likely to have future consequences. >>>> >>>> * Order of operations in ``np.einsum`` can now be optimized for large >>>> speed improvements. >>>> * New ``signature`` argument to ``np.vectorize`` for vectorizing with >>>> core dimensions. >>>> * The ``keepdims`` argument was added to many functions. >>>> * New context manager for testing warnings >>>> * Support for BLIS in numpy.distutils >>>> * Much improved support for PyPy (not yet finished) >>>> >>>> Enjoy, >>>> >>>> Chuck >>> >>> I've installed via pip3 on linux x86_64, which gives me a wheel. My >>> question is, am I loosing significant performance choosing this pre-built >>> binary vs. compiling myself? For example, my processor might have some >>> more features than the base version used to build wheels. >> >> I guess you are thinking about using this built wheel on some other >> machine? You'd have to be lucky for that to work; the wheel depends >> on the symbols it found at build time, which may not exist in the same >> places on your other machine. >> >> If it does work, the speed will primarily depend on your BLAS library. >> >> The pypi wheels should be pretty fast; they are built with OpenBLAS, >> which is at or near top of range for speed, across a range of >> platforms. >> >> Cheers, >> >> Matthew > > I installed using pip3 install, and it installed a wheel package. I did not > build it - aren't wheels already compiled packages? So isn't it built for > the common denominator architecture, not necessarily as fast as one I built > myself on my own machine? My question is, on x86_64, is this potential > difference large enough to bother with not using precompiled wheel packages? Ultimately, it's going to depend on all sorts of things, including most importantly your actual code. Like most speed questions, the only real way to know is to try it and measure the difference. The wheels do ship with a fast BLAS (OpenBLAS configured to automatically adapt to your CPU at runtime), so the performance will at least be reasonable. Possible improvements would include using a different and somehow better BLAS (MKL might be faster in some cases), tweaking your compiler options to take advantage of whatever SIMD ISAs your particular CPU supports (numpy's build system doesn't do this automatically but in principle you could do it by hand -- were you bothering before? does it even make a difference in practice? I dunno), and using a new compiler (the linux wheels use a somewhat ancient version of gcc for Reasons; newer compilers are better at optimizing -- how much does it matter? again I dunno). Basically: if you want to experiment and report back then I think we'd all be interested to hear; OTOH if you aren't feeling particularly curious/ambitious then I wouldn't worry about it :-). -n -- Nathaniel J. Smith -- https://vorpus.org From ralf.gommers at gmail.com Wed Jan 18 03:36:50 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 18 Jan 2017 21:36:50 +1300 Subject: [SciPy-Dev] GSoC'17 participation? Message-ID: Hi all, It's that time of year again, we should think about GSoC participation. For SciPy participating in previous years has definitely been worth it. Here is the ideas page from last year: https://github.com/scipy/scipy/wiki/GSoC-2016-project-ideas (not a whole lot to reuse). New ideas very welcome (ideally with mentor attached ...). Who is interested and available to (co-)mentor this year? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.linke88 at gmail.com Wed Jan 18 04:22:32 2017 From: max.linke88 at gmail.com (Max Linke) Date: Wed, 18 Jan 2017 10:22:32 +0100 Subject: [SciPy-Dev] GSoC'17 participation? In-Reply-To: References: Message-ID: <99e92935-7486-eeba-849e-a45b12ddccbd@gmail.com> Hi Do you plan to apply under the PSF umbrella again? best Max On 01/18/2017 09:36 AM, Ralf Gommers wrote: > Hi all, > > It's that time of year again, we should think about GSoC participation. > For SciPy participating in previous years has definitely been worth it. > > Here is the ideas page from last year: > https://github.com/scipy/scipy/wiki/GSoC-2016-project-ideas (not a whole > lot to reuse). > > New ideas very welcome (ideally with mentor attached ...). > > Who is interested and available to (co-)mentor this year? > > Cheers, > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > From ralf.gommers at gmail.com Wed Jan 18 04:40:11 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 18 Jan 2017 22:40:11 +1300 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Tue, Jan 17, 2017 at 8:19 AM, wrote: > > > On Mon, Jan 16, 2017 at 12:39 PM, Matthew Brett > wrote: > >> Hi, >> >> On Mon, Jan 16, 2017 at 1:01 AM, Ralf Gommers >> wrote: >> > >> > >> > On Sun, Jan 15, 2017 at 8:15 AM, Pauli Virtanen wrote: >> > >> > Interesting discussion so far! >> > >> >> >> >> >> >> Sat, 14 Jan 2017 10:03:43 -0800, Matthew Brett kirjoitti: >> >> [clip] >> >> > What do you think about the idea of having regular state-of-scipy >> >> > reviews to make sure we're conscious about keeping on track, >> assessing >> >> > risks, improving process? >> >> >> >> For the technical aspect, this sounds something like the Scipy roadmap >> >> (https://github.com/scipy/scipy/blob/master/doc/ROADMAP.rst.txt), and >> the >> >> discussion leading to it, in conferences and online in public. >> >> >> >> Something like regular prompts for discussion of technical and >> >> organisation roadmap could be useful. At minimum, this could be simply >> a >> >> (bi-?)yearly post on the mailing list, to remind to update the roadmap >> >> and to summarize / bring up / discuss any relevant organisation >> updates / >> >> issues in the preceding period. >> > >> > >> > I quite like this idea. Documents like a roadmap can easily go out of >> date >> > if they're not actively maintained. Having a critical look at it once or >> > twice a year will be helpful. Also +1 for adding some organizational >> items >> > to it (I'm thinking CoC, FSA, etc. should have been on there). >> > >> > The list of people on the steering committee also needs to be updated >> with >> > this kind of frequency. >> > >> > How about doing this around 1 January and 1 July every year? >> > >> > I'm not sensing a lot of enthusiasm for the fixed-term/election idea, >> >> I'm afraid the previous discussion on this thread has made it very >> unlikely that you would see any enthusiasm, even if, in another world, >> it was a good idea. That's the tragedy of mailing list discussions, >> as satirized here >> >> http://www.smbc-comics.com/?id=2939 >> >> and evident from the recent discussion on the Jupyter governance >> document. If anyone gets angry or dismissive about an idea, and that >> isn't addressed, that's a very effective way of inducing consent and >> shutting down the discussion. >> > > > Matthew, > I think that's mis-characterizing this a bit. > I think so too, but just in case: if anyone feels uncomfortable to contribute in this discussion on-list for whatever reason, then I'd much appreciate to hear his/her opinion offline (both on the governance topic and on what could have made contributing in public easier). > We had several governance discussion over the last years, and my > impression was that the vast majority of those commenting where in favor of > a more laid back approach instead of a fully specified constitution and > associated discussion. > I think Pauli hit the nail on the head with this: "(i) the person can step down at any time, and acting in good faith, will also listen to serious calls to do so, (ii) regular elections can generate unnecessary work and politics -- looking at the past 10 years of scipy, there has been little of this, and I believe this is for the best, (iii) this is more of a role for fallback decision making and less for a director/CEO." (i) and (iii) would be good to add to the document itself I think. > (I think what's missing is a chair wo/man of the steering council > This could be helpful, just to be able to give that person the responsibility to ping everyone for bi-yearly updates, ensure the list of people stays up to date, contact people that have stopped being active about removal from the council, etc. and heads/lieutenants for each sup-package, > We used to have this, in a MAINTAINERS document. The added value wasn't high imho and it went out of date, so we removed it. > and a specification of the rights of the release manager > That's probably useful to add, will do so. > and maybe some more.) > Please add comments on the PR if more comes to mind. Ralf > IMO, given the current contributors and the way contributors are recruited > and integrated, I don't see much difference between unlimited time and > fixed time with renewal. So, we can as well stick with what we know. > > Josef > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jan 18 04:41:29 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 18 Jan 2017 22:41:29 +1300 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Tue, Jan 17, 2017 at 9:09 AM, Pauli Virtanen wrote: > Mon, 16 Jan 2017 11:35:30 -0600, CJ Carey kirjoitti: > > > On Mon, Jan 16, 2017 at 3:01 AM, Ralf Gommers > > wrote: > > > >> How about doing this around 1 January and 1 July every year? > >> > >> > > Those dates fall during times when many people are traveling or > > otherwise not responsive. What about phase-shifting 3 months to April > > 1st / October 1st? > > Not that important, but now that we are optimizing this, I would push it > a bit forward to eg. Apr 20 and Oct 20 to avoid beginning of academic > semesters. > Sounds good, will add to the PR. We can micro-optimize there if needed:) Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jan 18 04:43:28 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 18 Jan 2017 22:43:28 +1300 Subject: [SciPy-Dev] GSoC'17 participation? In-Reply-To: <99e92935-7486-eeba-849e-a45b12ddccbd@gmail.com> References: <99e92935-7486-eeba-849e-a45b12ddccbd@gmail.com> Message-ID: On Wed, Jan 18, 2017 at 10:22 PM, Max Linke wrote: > Hi > > Do you plan to apply under the PSF umbrella again? > Yes. We've always gotten as many slots as we needed from the PSF, and doing so would leave more slots for other NumFOCUS projects. Ralf > best Max > > > On 01/18/2017 09:36 AM, Ralf Gommers wrote: > >> Hi all, >> >> It's that time of year again, we should think about GSoC participation. >> For SciPy participating in previous years has definitely been worth it. >> >> Here is the ideas page from last year: >> https://github.com/scipy/scipy/wiki/GSoC-2016-project-ideas (not a whole >> lot to reuse). >> >> New ideas very welcome (ideally with mentor attached ...). >> >> Who is interested and available to (co-)mentor this year? >> >> Cheers, >> Ralf >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-dev >> >> _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From irvin.probst at ensta-bretagne.fr Wed Jan 18 05:23:54 2017 From: irvin.probst at ensta-bretagne.fr (Irvin Probst) Date: Wed, 18 Jan 2017 11:23:54 +0100 Subject: [SciPy-Dev] GSoC'17 participation? In-Reply-To: References: Message-ID: <587F423A.6070300@ensta-bretagne.fr> Hi, in january 2016 there were a discussion here about cleaning the ode/odeint interface, see https://mail.scipy.org/pipermail/scipy-dev/2016-January/thread.html#21176 but I can't be a mentor for that (lack of knowledge of Numpy/Scipy internals and no time, sorry). Regards. On 18/01/2017 09:36, Ralf Gommers wrote: > Hi all, > > It's that time of year again, we should think about GSoC > participation. For SciPy participating in previous years has > definitely been worth it. > > Here is the ideas page from last year: > https://github.com/scipy/scipy/wiki/GSoC-2016-project-ideas (not a > whole lot to reuse). > > New ideas very welcome (ideally with mentor attached ...). > > Who is interested and available to (co-)mentor this year? > > Cheers, > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Wed Jan 18 07:02:01 2017 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 18 Jan 2017 07:02:01 -0500 Subject: [SciPy-Dev] [Numpy-discussion] NumPy 1.12.0 release References: Message-ID: Matthew Brett wrote: > On Tue, Jan 17, 2017 at 3:47 PM, Neal Becker wrote: >> Matthew Brett wrote: >> >>> Hi, >>> >>> On Tue, Jan 17, 2017 at 5:56 AM, Neal Becker >>> wrote: >>>> Charles R Harris wrote: >>>> >>>>> Hi All, >>>>> >>>>> I'm pleased to announce the NumPy 1.12.0 release. This release >>>>> supports Python 2.7 and 3.4-3.6. Wheels for all supported Python >>>>> versions may be downloaded from PiPY >>>>> , the >>>>> tarball and zip files may be downloaded from Github >>>>> . The release >>>>> notes and files hashes may also be found at Github >>>>> . >>>>> >>>>> NumPy 1.12.0rc 2 is the result of 418 pull requests submitted by 139 >>>>> contributors and comprises a large number of fixes and improvements. >>>>> Among >>>>> the many improvements it is difficult to pick out just a few as >>>>> standing above the others, but the following may be of particular >>>>> interest or indicate areas likely to have future consequences. >>>>> >>>>> * Order of operations in ``np.einsum`` can now be optimized for large >>>>> speed improvements. >>>>> * New ``signature`` argument to ``np.vectorize`` for vectorizing with >>>>> core dimensions. >>>>> * The ``keepdims`` argument was added to many functions. >>>>> * New context manager for testing warnings >>>>> * Support for BLIS in numpy.distutils >>>>> * Much improved support for PyPy (not yet finished) >>>>> >>>>> Enjoy, >>>>> >>>>> Chuck >>>> >>>> I've installed via pip3 on linux x86_64, which gives me a wheel. My >>>> question is, am I loosing significant performance choosing this >>>> pre-built >>>> binary vs. compiling myself? For example, my processor might have some >>>> more features than the base version used to build wheels. >>> >>> I guess you are thinking about using this built wheel on some other >>> machine? You'd have to be lucky for that to work; the wheel depends >>> on the symbols it found at build time, which may not exist in the same >>> places on your other machine. >>> >>> If it does work, the speed will primarily depend on your BLAS library. >>> >>> The pypi wheels should be pretty fast; they are built with OpenBLAS, >>> which is at or near top of range for speed, across a range of >>> platforms. >>> >>> Cheers, >>> >>> Matthew >> >> I installed using pip3 install, and it installed a wheel package. I did >> not >> build it - aren't wheels already compiled packages? So isn't it built >> for the common denominator architecture, not necessarily as fast as one I >> built >> myself on my own machine? My question is, on x86_64, is this potential >> difference large enough to bother with not using precompiled wheel >> packages? > > Ah - my guess is that you'd be hard pressed to make a numpy that is as > fast as the precompiled wheel. The OpenBLAS library included in > numpy selects the routines for your CPU at run-time, so they will > generally be fast on your CPU. You might be able to get equivalent > or even better performance with a ATLAS BLAS library recompiled on > your exact machine, but that's quite a serious investment of time to > get working, and you'd have to benchmark to find if you were really > doing any better. > > Cheers, > > Matthew OK, so at least for BLAS things should be pretty well optimized. From benny.malengier at gmail.com Wed Jan 18 07:59:59 2017 From: benny.malengier at gmail.com (Benny Malengier) Date: Wed, 18 Jan 2017 13:59:59 +0100 Subject: [SciPy-Dev] GSoC'17 participation? In-Reply-To: <587F423A.6070300@ensta-bretagne.fr> References: <587F423A.6070300@ensta-bretagne.fr> Message-ID: Irvin, work on ode is happening here https://github.com/scipy/scipy/pull/6326 2017-01-18 11:23 GMT+01:00 Irvin Probst : > Hi, > in january 2016 there were a discussion here about cleaning the ode/odeint > interface, see https://mail.scipy.org/pipermail/scipy-dev/2016- > January/thread.html#21176 but I can't be a mentor for that (lack of > knowledge of Numpy/Scipy internals and no time, sorry). > Regards. > > > On 18/01/2017 09:36, Ralf Gommers wrote: > > Hi all, > > It's that time of year again, we should think about GSoC participation. > For SciPy participating in previous years has definitely been worth it. > > Here is the ideas page from last year: > > https://github.com/scipy/scipy/wiki/GSoC-2016-project-ideas (not a whole > lot to reuse). > > New ideas very welcome (ideally with mentor attached ...). > > Who is interested and available to (co-)mentor this year? > > Cheers, > Ralf > > > > _______________________________________________ > SciPy-Dev mailing listSciPy-Dev at scipy.orghttps://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Jan 18 08:41:44 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 18 Jan 2017 08:41:44 -0500 Subject: [SciPy-Dev] SciPy governance model In-Reply-To: References: <1473151205.1482512.716887281.63889567@webmail.messagingengine.com> Message-ID: On Wed, Jan 18, 2017 at 4:40 AM, Ralf Gommers wrote: > > > On Tue, Jan 17, 2017 at 8:19 AM, wrote: > >> >> >> On Mon, Jan 16, 2017 at 12:39 PM, Matthew Brett >> wrote: >> >>> Hi, >>> >>> On Mon, Jan 16, 2017 at 1:01 AM, Ralf Gommers >>> wrote: >>> > >>> > >>> > On Sun, Jan 15, 2017 at 8:15 AM, Pauli Virtanen wrote: >>> > >>> > Interesting discussion so far! >>> > >>> >> >>> >> >>> >> Sat, 14 Jan 2017 10:03:43 -0800, Matthew Brett kirjoitti: >>> >> [clip] >>> >> > What do you think about the idea of having regular state-of-scipy >>> >> > reviews to make sure we're conscious about keeping on track, >>> assessing >>> >> > risks, improving process? >>> >> >>> >> For the technical aspect, this sounds something like the Scipy roadmap >>> >> (https://github.com/scipy/scipy/blob/master/doc/ROADMAP.rst.txt), >>> and the >>> >> discussion leading to it, in conferences and online in public. >>> >> >>> >> Something like regular prompts for discussion of technical and >>> >> organisation roadmap could be useful. At minimum, this could be >>> simply a >>> >> (bi-?)yearly post on the mailing list, to remind to update the roadmap >>> >> and to summarize / bring up / discuss any relevant organisation >>> updates / >>> >> issues in the preceding period. >>> > >>> > >>> > I quite like this idea. Documents like a roadmap can easily go out of >>> date >>> > if they're not actively maintained. Having a critical look at it once >>> or >>> > twice a year will be helpful. Also +1 for adding some organizational >>> items >>> > to it (I'm thinking CoC, FSA, etc. should have been on there). >>> > >>> > The list of people on the steering committee also needs to be updated >>> with >>> > this kind of frequency. >>> > >>> > How about doing this around 1 January and 1 July every year? >>> > >>> > I'm not sensing a lot of enthusiasm for the fixed-term/election idea, >>> >>> I'm afraid the previous discussion on this thread has made it very >>> unlikely that you would see any enthusiasm, even if, in another world, >>> it was a good idea. That's the tragedy of mailing list discussions, >>> as satirized here >>> >>> http://www.smbc-comics.com/?id=2939 >>> >>> and evident from the recent discussion on the Jupyter governance >>> document. If anyone gets angry or dismissive about an idea, and that >>> isn't addressed, that's a very effective way of inducing consent and >>> shutting down the discussion. >>> >> >> >> Matthew, >> I think that's mis-characterizing this a bit. >> > > I think so too, but just in case: if anyone feels uncomfortable to > contribute in this discussion on-list for whatever reason, then I'd much > appreciate to hear his/her opinion offline (both on the governance topic > and on what could have made contributing in public easier). > > >> We had several governance discussion over the last years, and my >> impression was that the vast majority of those commenting where in favor of >> a more laid back approach instead of a fully specified constitution and >> associated discussion. >> > > I think Pauli hit the nail on the head with this: > > "(i) the person can step > down at any time, and acting in good faith, will also listen to serious > calls to do so, (ii) regular elections can generate unnecessary work and > politics -- looking at the past 10 years of scipy, there has been little > of this, and I believe this is for the best, (iii) this is more of a role > for fallback decision making and less for a director/CEO." > > (i) and (iii) would be good to add to the document itself I think. > > >> (I think what's missing is a chair wo/man of the steering council >> > > This could be helpful, just to be able to give that person the > responsibility to ping everyone for bi-yearly updates, ensure the list of > people stays up to date, contact people that have stopped being active > about removal from the council, etc. > > and heads/lieutenants for each sup-package, >> > > We used to have this, in a MAINTAINERS document. The added value wasn't > high imho and it went out of date, so we removed it. > I don't think it needs to be in the governance document. It illustrates the size difference in the organization compared to Debian for example. scipy has experts (developers familiar with code and topic) in some areas and generic maintainers (especially for topics without expert), but who is available fluctuates given the small numbers of maintainers relative to the size of scipy. . > > >> and a specification of the rights of the release manager >> > > That's probably useful to add, will do so. > > >> and maybe some more.) >> > > Please add comments on the PR if more comes to mind. > I think additional roles can be added in future, if the growth of scipy makes it large enough to require that those roles are formally defined. In general, I would prefer to have some impeachment clauses instead of just pointing at forking, but given that Pauli is the BDFL, I don't think it's necessary and we don't have to come up with a system for replacing the BDFL, chairman of the council and the council itself. Josef > > Ralf > > >> IMO, given the current contributors and the way contributors are >> recruited and integrated, I don't see much difference between unlimited >> time and fixed time with renewal. So, we can as well stick with what we >> know. >> >> Josef >> >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Wed Jan 18 14:32:12 2017 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Wed, 18 Jan 2017 22:32:12 +0300 Subject: [SciPy-Dev] GSoC'17 participation? In-Reply-To: References: Message-ID: On Wed, Jan 18, 2017 at 11:36 AM, Ralf Gommers wrote: > Hi all, > > It's that time of year again, we should think about GSoC participation. For > SciPy participating in previous years has definitely been worth it. > > Here is the ideas page from last year: > https://github.com/scipy/scipy/wiki/GSoC-2016-project-ideas (not a whole lot > to reuse). > > New ideas very welcome (ideally with mentor attached ...). > > Who is interested and available to (co-)mentor this year? Thanks for starting it Ralf! I might have some bandwidth this summer to co-mentor. A few random ideas: 1. scipy.diff is still a nice one IMO. The focus can be on moving `approx_derivative` to be public facing. https://github.com/scipy/scipy/issues/6026 2. B-splines (again!). https://github.com/scipy/scipy/issues/6730 and https://github.com/scipy/scipy/issues/6710 list possible subprojects of ranging difficulty. This would require a student to be able to read literature though. Alternatively, there's rational interpolation, https://github.com/scipy/scipy/issues/6929 and Pauli's PR for barycentric interpolation. 3. hypergeometric functions would be great, but this might be too difficult. Josh, Nikolay, Ted --- you guys looked at this at some point; any comments? 4. Testing: A relatively easy task could be to enable a move away from nose to pytest, for both scipy and numpy. Ideally as a result the test suites can be run with either of those with all the bells and whistles, with fast/slow/xslow tests, skipifs and knownfailures. Cheers, Evgeni From josh.craig.wilson at gmail.com Wed Jan 18 15:18:21 2017 From: josh.craig.wilson at gmail.com (Joshua Wilson) Date: Wed, 18 Jan 2017 14:18:21 -0600 Subject: [SciPy-Dev] GSoC'17 participation? In-Reply-To: References: Message-ID: > hypergeometric functions would be great, but this might be too difficult I concur with too difficult. I don't think I would want to set a student a task that I'm not confident I could do myself. That being said, I could possibly help mentor a special functions project; there are plenty of more straightforward implementations to be done. Parabolic cylinder functions might be a good project; I started working on those a while ago but it's not looking like I'll have time to finish anytime soon. There's a sequence of papers that give a complete implementation. Since this would be from scratch the person would only have to know (or be able to pick up) Cython; the harder part would be finding someone comfortable with the math involved. On Wed, Jan 18, 2017 at 1:32 PM, Evgeni Burovski wrote: > On Wed, Jan 18, 2017 at 11:36 AM, Ralf Gommers wrote: >> Hi all, >> >> It's that time of year again, we should think about GSoC participation. For >> SciPy participating in previous years has definitely been worth it. >> >> Here is the ideas page from last year: >> https://github.com/scipy/scipy/wiki/GSoC-2016-project-ideas (not a whole lot >> to reuse). >> >> New ideas very welcome (ideally with mentor attached ...). >> >> Who is interested and available to (co-)mentor this year? > > Thanks for starting it Ralf! > > I might have some bandwidth this summer to co-mentor. > > A few random ideas: > > 1. scipy.diff is still a nice one IMO. The focus can be on moving > `approx_derivative` to be public facing. > https://github.com/scipy/scipy/issues/6026 > > 2. B-splines (again!). https://github.com/scipy/scipy/issues/6730 and > https://github.com/scipy/scipy/issues/6710 list possible subprojects > of ranging difficulty. This would require a student to be able to read > literature though. > Alternatively, there's rational interpolation, > https://github.com/scipy/scipy/issues/6929 and Pauli's PR for > barycentric interpolation. > > 3. hypergeometric functions would be great, but this might be too > difficult. Josh, Nikolay, Ted --- you guys looked at this at some > point; any comments? > > 4. Testing: A relatively easy task could be to enable a move away from > nose to pytest, for both scipy and numpy. Ideally as a result the test > suites can be run with either of those with all the bells and > whistles, with fast/slow/xslow tests, skipifs and knownfailures. > > Cheers, > > Evgeni > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev From evgeny.burovskiy at gmail.com Thu Jan 19 07:29:58 2017 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Thu, 19 Jan 2017 15:29:58 +0300 Subject: [SciPy-Dev] GSoC'17 participation? In-Reply-To: References: Message-ID: > That being said, I could possibly help mentor a special functions > project; there are plenty of more straightforward implementations to > be done. Parabolic cylinder functions might be a good project; I > started working on those a while ago but it's not looking like I'll > have time to finish anytime soon. There's a sequence of papers that > give a complete implementation. Since this would be from scratch the > person would only have to know (or be able to pick up) Cython; the > harder part would be finding someone comfortable with the math > involved. Would you be able to add this to the wiki? I'll do some advertising in my university, and it'd be helpful to have something to point students to. From evgeny.burovskiy at gmail.com Thu Jan 19 07:59:56 2017 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Thu, 19 Jan 2017 15:59:56 +0300 Subject: [SciPy-Dev] API to control iteration in differential evolution #6923 In-Reply-To: References: <6b000cae-8295-e690-39d3-054450d5cef1@laxalde.org> Message-ID: On Sat, Jan 14, 2017 at 11:20 PM, Andrew Nelson wrote: > On 15 January 2017 at 05:03, Denis Laxalde wrote: >> >> Evgeni Burovski a ?crit : >>> >>> Small details, feel free to take or leave: >>> >>> 1. What is `step` in `for step in solver`? If it's a current "state of >>> the minimizer", then can its contents be retrieved from the solver >>> object itself (step vs `solver.result` above)? Or is it just a step >>> number and the solver holds all the information at each step via >>> `solver.result?` This needs to be specified and documented. >> >> >> This `step` is an `OptimizeStep`. It's a new class introduced here to >> wrap any useful information produced by the solver during one iteration. >> In this case, it holds the current `x` and function value along with >> iteration number. The idea is to return an extensible object instead of, >> say, a tuple that may cause bw compat issues in the future. I think this >> is documented, feel free to comment further if things are not clear >> enough. > > > Best `x` and best energy are already available as `solver.x` and > `solver.population_energy[0]`, but I think this kind of object being > returned from `__next__` is a good idea. Does it have to be an > `OptimizeStep`, or can we return an preliminary `OptimizeResult`? > >>> 2. Since the solver class is iterable, just iterating it without the >>> context manager should be equivalent. Something like this (pseudocode) >>> >>> solver = DifferentialEvolutionSolver(...) >>> while not_converged: >>> result = next(solver) > > >> >> Also, the context manager interface is actually orthogonal to the >> iteration API. The context manager abstracts out the solver >> *initialization* (population initialization and energies calculation) >> and *termination* (possible polishing and build of the >> `OptimizeResult`). > > > > Whilst the context manager is orthogonal to the iteration API, Evgeni's > example would not be possible with the PR is it currently exists - the only > way of initialising the solver (population initialization and energies > calculation) would be to go through the context manager. Denis suggested > that a way around this would be to have a separate initialize_population() > method. It's a good idea to have this method (for reasons I outline later). > However, if you create the object outside a context manager I don't think > that one should have to call an additional initialize_population() method to > be able to iterate the object. There are a few reasons for this: > (a) it's an extra line to type > (b) iterating the object without going through the context manager is not > equivalent > (c) the solver object possesses public population and population_energies > attributes, these attributes need to be present in a newly created object, > not be created at a later stage > (d) if the population and population_energies attributes are present I'd > argue it makes more sense for them to contain relevant values > (e) given a solver object how does one know if initialize_population has > already been called (which would be a common occurrence when used in an > interactive interpreter)? > (f) you can't call next(solver) if the population hasn't been initialised. > You shouldn't need to know the state of the solver object to be able to > iterate it, it should be iterable straightaway. > > Currently the initial energy calculations are done in `__init__` and one > uses the object as: > > # stepwise > solver = DifferentialEvolutionSolver(...) > while not solver.converged: > next(solver) > > -or- > > # all at once > solver = DifferentialEvolutionSolver(...) > result = solver.solve() > > The proposed PR: > > with solver as DifferentialEvolutionSolver(...): > while not solver.converged(): > next(solver) > result = solver.result > > -or possibly- > > solver = DifferentialEvolutionSolver(...) > # extra line to type, > solver.initialize_population() > while not solver.converged: > next(solver) > > > Perhaps the following course could be followed: > > (1) make the __next__ method return an OptimizeStep object (or > OptimizeResult). > (2) add a polish() method that takes the current best solution and polishes > it. Having a separate polish method is useful as you can polish the object > at any step, getting functionality similar to optimize.brute. > (3) Add __enter__() and __exit__() methods to allow the object to be used as > a context manager. The __exit__() method can use the public polish() method. > I'd like to keep the existing solve() method, it can call __exit__() at the > end. > (4) population initialization and the initial energy calculation is moved to > a public method, initialize_population(). This method is either called from > __init__ (the status quo), or it is called on the first call to __next__(), > making the first call to __next__ take twice as long. My preference is to > keep the status quo because one can create the solver and immediately polish > the best population member (c.f. optimize.brute), as well as the points > raised above. The rationale for making it public is that once you've done > one optimization you can re-initialise it with a different method > ('latinhypercube' or 'random') and solve again or you could imagine > re-initialising the population with a given population array (which has been > requested previously) and start iterating again. It seems to me we are all saying pretty much the same thing: 1. If an instance of DifferentialEvolutionSolver is used as a context manager, __init__ is called before __enter__ anyway, so all initialization can happen in __init__, like it does in master: In [33]: class C(object): def __init__(self): print('init') def __enter__(self): print('enter') def __exit__(self, *args, **kwds): print('exit', args, kwds) ....: In [34]: with C() as c: ....: pass ....: init enter ('exit', (None, None, None), {}) This way it seems there's no need for either having a special `init_population` method or doing something special at first call to __next__ 2. Since __init__ is always called, various initialization strategies can be handled as a keyword arg in __init__(..., init_method="latinhypercube"), possibly overloaded for a callable or a population from a previous run? Or maybe the latter (reusing a population) is better handled by an alternative constructor. But this is certainly separate from both the context manager or iteration. 3. I agree with Denis that `make_a_step_then_stop` is spelled __next__ in python :-). 4. `.solve` method is indeed the standalone `differential_evolution` function. 5. It seems natural to use __exit__ for polishing, if desired. 6. OptimizeStep could be a (current version of) OptimizeResult. In fact, I'd suggest it *is* an OptimizeResult. Am I missing something? Evgeni From tpudlik at gmail.com Thu Jan 19 11:18:51 2017 From: tpudlik at gmail.com (Ted Pudlik) Date: Thu, 19 Jan 2017 16:18:51 +0000 Subject: [SciPy-Dev] GSoC'17 participation? In-Reply-To: References: Message-ID: I agree that it would be very challenging. It would also be more of a research project in applied maths than a straightforward implementation exercise. For the confluent hypergeometric function specifically (on which I've worked unsuccessfully), the main difficulty is that no known algorithm (https://arxiv.org/abs/1407.7786) is reliable throughout parameter space, and in fact finding any combination of algorithms that works well is rather hard. I have more specific results tucked away somewhere, including interesting plots (one attached as an example---number of correct digits computed by the optimally-truncated asymptotic series as a function of the parameters). I'll try to clean them up and post somewhere for posterity's sake, assuming I won't be able to finish this work myself.[image: 3.png] On Wed, Jan 18, 2017 at 11:32 AM Evgeni Burovski wrote: > 3. hypergeometric functions would be great, but this might be too > difficult. Josh, Nikolay, Ted --- you guys looked at this at some > point; any comments? > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 3.png Type: image/png Size: 82262 bytes Desc: not available URL: From alebarde at gmail.com Thu Jan 19 15:40:46 2017 From: alebarde at gmail.com (alebarde at gmail.com) Date: Thu, 19 Jan 2017 21:40:46 +0100 Subject: [SciPy-Dev] ENH: added args and kwargs to Xdist Message-ID: Hi all, the current implementations of pdist and cdist takes as keyword arguments some extra parameters for a few specific metrics. At the same time, they do not allow for others args/kwargs that may be needed by a new user-defined metric. Plus, in 0.19 it is possible to call: pdist([[1,2,3]], metric='euclidean', V=[1,2,3], w=[1,2,3]) pdist([[1,2,3]], metric='seuclidean', VI=[1,2,3]) IMHO this is quite strange behaviour since I would have expected an error. In particular, in the second case, the error would also probably help debugging the code since it may be a copy-and-paste typo (seuclidean takes V as kwarg, while mahalanobis requires VI) I have opened a PR (https://github.com/scipy/scipy/pull/6974) that introduces support for *args and **kwargs that gets passed on to the selected metric. The new functions should not break any existing code with the exception of the previous example. However, if needed, I can try to catch those cases, and for now just raise a deprecate warning. what do you think? Alessandro -- -------------------------------------------------------------------------- NOTICE: Dlgs 196/2003 this e-mail and any attachments thereto may contain confidential information and are intended for the sole use of the recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any dissemination or copying of this message is strictly prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you. -------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.craig.wilson at gmail.com Thu Jan 19 18:29:49 2017 From: josh.craig.wilson at gmail.com (Joshua Wilson) Date: Thu, 19 Jan 2017 17:29:49 -0600 Subject: [SciPy-Dev] GSoC'17 participation? In-Reply-To: References: Message-ID: > Would you be able to add this to the wiki? Here it is: https://github.com/scipy/scipy/wiki/GSoC-2016-project-ideas#improve-the-parabolic-cylinder-functions On Thu, Jan 19, 2017 at 10:18 AM, Ted Pudlik wrote: > I agree that it would be very challenging. It would also be more of a > research project in applied maths than a straightforward implementation > exercise. For the confluent hypergeometric function specifically (on which > I've worked unsuccessfully), the main difficulty is that no known algorithm > (https://arxiv.org/abs/1407.7786) is reliable throughout parameter space, > and in fact finding any combination of algorithms that works well is rather > hard. > > I have more specific results tucked away somewhere, including interesting > plots (one attached as an example---number of correct digits computed by > the optimally-truncated asymptotic series as a function of the > parameters). I'll try to clean them up and post somewhere for posterity's > sake, assuming I won't be able to finish this work myself.[image: 3.png] > > On Wed, Jan 18, 2017 at 11:32 AM Evgeni Burovski < > evgeny.burovskiy at gmail.com> wrote: > >> 3. hypergeometric functions would be great, but this might be too >> difficult. Josh, Nikolay, Ted --- you guys looked at this at some >> point; any comments? >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 3.png Type: image/png Size: 82262 bytes Desc: not available URL: From josh.craig.wilson at gmail.com Thu Jan 19 18:47:23 2017 From: josh.craig.wilson at gmail.com (Joshua Wilson) Date: Thu, 19 Jan 2017 17:47:23 -0600 Subject: [SciPy-Dev] GSoC'17 participation? In-Reply-To: References: Message-ID: P.S. If anyone wants to co-mentor that would be welcome. On Thu, Jan 19, 2017 at 5:29 PM, Joshua Wilson wrote: > > Would you be able to add this to the wiki? > > Here it is: > > https://github.com/scipy/scipy/wiki/GSoC-2016-project- > ideas#improve-the-parabolic-cylinder-functions > > On Thu, Jan 19, 2017 at 10:18 AM, Ted Pudlik wrote: > >> I agree that it would be very challenging. It would also be more of a >> research project in applied maths than a straightforward implementation >> exercise. For the confluent hypergeometric function specifically (on which >> I've worked unsuccessfully), the main difficulty is that no known algorithm >> (https://arxiv.org/abs/1407.7786) is reliable throughout parameter >> space, and in fact finding any combination of algorithms that works well is >> rather hard. >> >> I have more specific results tucked away somewhere, including interesting >> plots (one attached as an example---number of correct digits computed by >> the optimally-truncated asymptotic series as a function of the >> parameters). I'll try to clean them up and post somewhere for posterity's >> sake, assuming I won't be able to finish this work myself.[image: 3.png] >> >> On Wed, Jan 18, 2017 at 11:32 AM Evgeni Burovski < >> evgeny.burovskiy at gmail.com> wrote: >> >>> 3. hypergeometric functions would be great, but this might be too >>> difficult. Josh, Nikolay, Ted --- you guys looked at this at some >>> point; any comments? >>> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 3.png Type: image/png Size: 82262 bytes Desc: not available URL: From alebarde at gmail.com Fri Jan 20 05:21:26 2017 From: alebarde at gmail.com (alebarde at gmail.com) Date: Fri, 20 Jan 2017 11:21:26 +0100 Subject: [SciPy-Dev] [Numpy]: making np.gradient support unevenly spaced data Message-ID: Hi all, current version of numpy gradient supports only data with uniform spacing. I have implemented a proposed enhancement for the np.gradient function that allows to compute the gradient on non uniform grids. (PR: https://github.com/numpy/numpy/pull/8446) Since it may be of interest also for the scipy user/developers and it may be useful also for scipy.diff I am posting here too. The proposed implementation has a behaviour/signature is similar to that of Matlab/Octave. As argument it can take: 1. A single scalar to specify a sample distance for all dimensions. 2. N scalars to specify a constant sample distance for each dimension. i.e. `dx`, `dy`, `dz`, ... 3. N arrays to specify the coordinates of the values along each dimension of F. The length of the array must match the size of the corresponding dimension 4. Any combination of N scalars/arrays with the meaning of 2. and 3. e.g., you can do the following: >>> f = np.array([[1, 2, 6], [3, 4, 5]], dtype=np.float) >>> dx = 2. >>> y = [1., 1.5, 3.5] >>> np.gradient(f, dx, y) [array([[ 1. , 1. , -0.5], [ 1. , 1. , -0.5]]), array([[ 2. , 2. , 2. ], [ 2. , 1.7, 0.5]])] It should not break any existing code since as of 1.12 only scalars or list of scalars are allowed. A possible alternative API could be pass arrays of sampling steps instead of the coordinates. On the one hand, this would have the advantage of having "differences" both in the scalar case and in the array case. On the other hand, if you are dealing with non uniformly-spaced data (e.g, data is mapped on a grid or it is a time-series), in most cases you already have the coordinates/timestamps. Therefore, in the case of difference as argument, you would almost always have a call np.diff before np.gradient. In the end, I would rather prefer the coordinates option since IMHO it is more handy, I don't think that would be too much "surprising" and it is what Matlab already does. Also, it could not easily lead to "silly" mistakes since the length have to match the size of the corresponding dimension. What do you think? Thanks Alessandro -- -------------------------------------------------------------------------- NOTICE: Dlgs 196/2003 this e-mail and any attachments thereto may contain confidential information and are intended for the sole use of the recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any dissemination or copying of this message is strictly prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you. -------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Jan 20 08:44:02 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 20 Jan 2017 08:44:02 -0500 Subject: [SciPy-Dev] [Numpy]: making np.gradient support unevenly spaced data In-Reply-To: References: Message-ID: On Fri, Jan 20, 2017 at 5:21 AM, alebarde at gmail.com wrote: > Hi all, > > current version of numpy gradient supports only data with uniform spacing. > I have implemented a proposed enhancement for the np.gradient function that > allows to compute the gradient on non uniform grids. (PR: > https://github.com/numpy/numpy/pull/8446) > Since it may be of interest also for the scipy user/developers and it may > be useful also for scipy.diff > > I am posting here too. > > The proposed implementation has a behaviour/signature is similar to that > of Matlab/Octave. As argument it can take: > 1. A single scalar to specify a sample distance for all dimensions. > 2. N scalars to specify a constant sample distance for each dimension. > i.e. `dx`, `dy`, `dz`, ... > 3. N arrays to specify the coordinates of the values along each > dimension of F. The length of the array must match the size of > the corresponding dimension > 4. Any combination of N scalars/arrays with the meaning of 2. and 3. > > e.g., you can do the following: > > >>> f = np.array([[1, 2, 6], [3, 4, 5]], dtype=np.float) > >>> dx = 2. > >>> y = [1., 1.5, 3.5] > >>> np.gradient(f, dx, y) > [array([[ 1. , 1. , -0.5], [ 1. , 1. , -0.5]]), > array([[ 2. , 2. , 2. ], [ 2. , 1.7, 0.5]])] > > It should not break any existing code since as of 1.12 only scalars or > list of scalars are allowed. > A possible alternative API could be pass arrays of sampling steps instead > of the coordinates. > On the one hand, this would have the advantage of having "differences" > both in the scalar case and in the array case. > On the other hand, if you are dealing with non uniformly-spaced data (e.g, > data is mapped on a grid or it is a time-series), in most cases you already > have the coordinates/timestamps. Therefore, in the case of difference as > argument, you would almost always have a call np.diff before np.gradient. > > In the end, I would rather prefer the coordinates option since IMHO it is > more handy, I don't think that would be too much "surprising" and it is > what Matlab already does. Also, it could not easily lead to "silly" > mistakes since the length have to match the size of the corresponding > dimension. > > What do you think? > In general I also like specifying x better than dx. I looked at np.gradient two days ago to see if it does what I need. But it was missing the unequal spacing, and it is missing the second derivative. more general question: I found the dimension story with dx, dy, dz, ... confusing. Isn't this just applying gradient with respect to each axis separately? (aside: If I read your link in the PR http://websrv.cs.umt.edu/isis/index.php/Finite_differencing:_Introduction correctly, then I might have just implemented a rescaled version of second (kth) order differencing with unequal spaced x using sparse linear algebra. I didn't see anything premade in numpy or scipy for it.) Josef > > Thanks > > Alessandro > > -- > -------------------------------------------------------------------------- > NOTICE: Dlgs 196/2003 this e-mail and any attachments thereto may contain > confidential information and are intended for the sole use of the > recipient(s) named above. If you are not the intended recipient of this > message you are hereby notified that any dissemination or copying of this > message is strictly prohibited. If you have received this e-mail in error, > please notify the sender either by telephone or by e-mail and delete the > material from any computer. Thank you. > -------------------------------------------------------------------------- > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alebarde at gmail.com Fri Jan 20 09:09:03 2017 From: alebarde at gmail.com (alebarde at gmail.com) Date: Fri, 20 Jan 2017 15:09:03 +0100 Subject: [SciPy-Dev] [Numpy]: making np.gradient support unevenly spaced data In-Reply-To: References: Message-ID: 2017-01-20 14:44 GMT+01:00 : > > > On Fri, Jan 20, 2017 at 5:21 AM, alebarde at gmail.com > wrote: > >> Hi all, >> >> current version of numpy gradient supports only data with uniform >> spacing. I have implemented a proposed enhancement for the np.gradient >> function that allows to compute the gradient on non uniform grids. (PR: >> https://github.com/numpy/numpy/pull/8446) >> Since it may be of interest also for the scipy user/developers and it may >> be useful also for scipy.diff >> >> I am posting here too. >> >> The proposed implementation has a behaviour/signature is similar to that >> of Matlab/Octave. As argument it can take: >> 1. A single scalar to specify a sample distance for all dimensions. >> 2. N scalars to specify a constant sample distance for each dimension. >> i.e. `dx`, `dy`, `dz`, ... >> 3. N arrays to specify the coordinates of the values along each >> dimension of F. The length of the array must match the size of >> the corresponding dimension >> 4. Any combination of N scalars/arrays with the meaning of 2. and 3. >> >> e.g., you can do the following: >> >> >>> f = np.array([[1, 2, 6], [3, 4, 5]], dtype=np.float) >> >>> dx = 2. >> >>> y = [1., 1.5, 3.5] >> >>> np.gradient(f, dx, y) >> [array([[ 1. , 1. , -0.5], [ 1. , 1. , -0.5]]), >> array([[ 2. , 2. , 2. ], [ 2. , 1.7, 0.5]])] >> >> It should not break any existing code since as of 1.12 only scalars or >> list of scalars are allowed. >> A possible alternative API could be pass arrays of sampling steps instead >> of the coordinates. >> On the one hand, this would have the advantage of having "differences" >> both in the scalar case and in the array case. >> On the other hand, if you are dealing with non uniformly-spaced data >> (e.g, data is mapped on a grid or it is a time-series), in most cases you >> already have the coordinates/timestamps. Therefore, in the case of >> difference as argument, you would almost always have a call np.diff before >> np.gradient. >> >> In the end, I would rather prefer the coordinates option since IMHO it is >> more handy, I don't think that would be too much "surprising" and it is >> what Matlab already does. Also, it could not easily lead to "silly" >> mistakes since the length have to match the size of the corresponding >> dimension. >> >> What do you think? >> > > In general I also like specifying x better than dx. > > I looked at np.gradient two days ago to see if it does what I need. But it > was missing the unequal spacing, and it is missing the second derivative. > Yes, and I believe it is quite a useful feature. that's why I have implemented it. On the other hand I don't think that `gradient` should also be able to compute the second or kth derivative. In that case a more general function called `differentiate` would probably be better. > > more general question: I found the dimension story with dx, dy, dz, ... > confusing. Isn't this just applying gradient with respect to each axis > separately? > Yes it is just a way to pass the sampling step in each direction (and actually they are not keywords). I have kept the original documentation on this but it is probably better to change into h1,h2,h3. > > > (aside: If I read your link in the PR http://websrv.cs.umt.edu/isis/ > index.php/Finite_differencing:_Introduction correctly, then I might have > just implemented a rescaled version of second (kth) order differencing with > unequal spaced x using sparse linear algebra. I didn't see anything premade > in numpy or scipy for it.) > what do you mean with "rescaled version"? and you meant second order or second derivative? in the first case I think that would be equivalent to what my PR does. In the latter, I confirm that there is currently no implementation of a second derivative. > > Josef > > > >> >> Thanks >> >> Alessandro >> >> -- >> ------------------------------------------------------------ >> -------------- >> NOTICE: Dlgs 196/2003 this e-mail and any attachments thereto may >> contain confidential information and are intended for the sole use of the >> recipient(s) named above. If you are not the intended recipient of this >> message you are hereby notified that any dissemination or copying of this >> message is strictly prohibited. If you have received this e-mail in error, >> please notify the sender either by telephone or by e-mail and delete the >> material from any computer. Thank you. >> ------------------------------------------------------------ >> -------------- >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -- -------------------------------------------------------------------------- NOTICE: Dlgs 196/2003 this e-mail and any attachments thereto may contain confidential information and are intended for the sole use of the recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any dissemination or copying of this message is strictly prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you. -------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Jan 20 10:20:07 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 20 Jan 2017 10:20:07 -0500 Subject: [SciPy-Dev] [Numpy]: making np.gradient support unevenly spaced data In-Reply-To: References: Message-ID: On Fri, Jan 20, 2017 at 9:09 AM, alebarde at gmail.com wrote: > 2017-01-20 14:44 GMT+01:00 : > >> >> >> On Fri, Jan 20, 2017 at 5:21 AM, alebarde at gmail.com >> wrote: >> >>> Hi all, >>> >>> current version of numpy gradient supports only data with uniform >>> spacing. I have implemented a proposed enhancement for the np.gradient >>> function that allows to compute the gradient on non uniform grids. (PR: >>> https://github.com/numpy/numpy/pull/8446) >>> Since it may be of interest also for the scipy user/developers and it >>> may be useful also for scipy.diff >>> >>> I am posting here too. >>> >>> The proposed implementation has a behaviour/signature is similar to that >>> of Matlab/Octave. As argument it can take: >>> 1. A single scalar to specify a sample distance for all dimensions. >>> 2. N scalars to specify a constant sample distance for each dimension. >>> i.e. `dx`, `dy`, `dz`, ... >>> 3. N arrays to specify the coordinates of the values along each >>> dimension of F. The length of the array must match the size of >>> the corresponding dimension >>> 4. Any combination of N scalars/arrays with the meaning of 2. and 3. >>> >>> e.g., you can do the following: >>> >>> >>> f = np.array([[1, 2, 6], [3, 4, 5]], dtype=np.float) >>> >>> dx = 2. >>> >>> y = [1., 1.5, 3.5] >>> >>> np.gradient(f, dx, y) >>> [array([[ 1. , 1. , -0.5], [ 1. , 1. , -0.5]]), >>> array([[ 2. , 2. , 2. ], [ 2. , 1.7, 0.5]])] >>> >>> It should not break any existing code since as of 1.12 only scalars or >>> list of scalars are allowed. >>> A possible alternative API could be pass arrays of sampling steps >>> instead of the coordinates. >>> On the one hand, this would have the advantage of having "differences" >>> both in the scalar case and in the array case. >>> On the other hand, if you are dealing with non uniformly-spaced data >>> (e.g, data is mapped on a grid or it is a time-series), in most cases you >>> already have the coordinates/timestamps. Therefore, in the case of >>> difference as argument, you would almost always have a call np.diff before >>> np.gradient. >>> >>> In the end, I would rather prefer the coordinates option since IMHO it >>> is more handy, I don't think that would be too much "surprising" and it is >>> what Matlab already does. Also, it could not easily lead to "silly" >>> mistakes since the length have to match the size of the corresponding >>> dimension. >>> >>> What do you think? >>> >> >> In general I also like specifying x better than dx. >> >> I looked at np.gradient two days ago to see if it does what I need. But >> it was missing the unequal spacing, and it is missing the second derivative. >> > > Yes, and I believe it is quite a useful feature. that's why I have > implemented it. On the other hand I don't think that `gradient` should also > be able to compute the second or kth derivative. In that case a more > general function called `differentiate` would probably be better. > > >> >> more general question: I found the dimension story with dx, dy, dz, ... >> confusing. Isn't this just applying gradient with respect to each axis >> separately? >> > > Yes it is just a way to pass the sampling step in each direction (and > actually they are not keywords). I have kept the original documentation on > this but it is probably better to change into h1,h2,h3. > > >> >> >> (aside: If I read your link in the PR http://websrv.cs.umt.edu/isis/ >> index.php/Finite_differencing:_Introduction correctly, then I might have >> just implemented a rescaled version of second (kth) order differencing with >> unequal spaced x using sparse linear algebra. I didn't see anything premade >> in numpy or scipy for it.) >> > > what do you mean with "rescaled version"? and you meant second order or > second derivative? in the first case I think that would be equivalent to > what my PR does. In the latter, I confirm that there is currently no > implementation of a second derivative. > I think it's k-th derivative. Rescaled means that I don't actually compute the derivative, I want to get a nonparametric estimate of the residual which assumes, I think, that the underlying function for the non-noise version has approximately constant k-th derivative (e.g. k-th order polynomial, or (k+1)-th). The sum of squared window coefficients is normalized to 1. I didn't try to understand the math, but the kernels or windows in the equal spaced case are >>> _diff_kernel(0) array([ 1.]) >>> _diff_kernel(1) array([ 0.70710678, -0.70710678]) >>> _diff_kernel(2) array([ 0.40824829, -0.81649658, 0.40824829]) >>> _diff_kernel(2) / _diff_kernel(2)[1] array([-0.5, 1. , -0.5]) >>> _diff_kernel(3) array([ 0.2236068 , -0.67082039, 0.67082039, -0.2236068 ]) >>> _diff_kernel(4) array([ 0.11952286, -0.47809144, 0.71713717, -0.47809144, 0.11952286]) >>> (_diff_kernel(4)**2).sum() 0.99999999999999989 Josef > > >> >> Josef >> >> >> >>> >>> Thanks >>> >>> Alessandro >>> >>> -- >>> ------------------------------------------------------------ >>> -------------- >>> NOTICE: Dlgs 196/2003 this e-mail and any attachments thereto may >>> contain confidential information and are intended for the sole use of the >>> recipient(s) named above. If you are not the intended recipient of this >>> message you are hereby notified that any dissemination or copying of this >>> message is strictly prohibited. If you have received this e-mail in error, >>> please notify the sender either by telephone or by e-mail and delete the >>> material from any computer. Thank you. >>> ------------------------------------------------------------ >>> -------------- >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> https://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> https://mail.scipy.org/mailman/listinfo/scipy-dev >> >> > > > -- > -------------------------------------------------------------------------- > NOTICE: Dlgs 196/2003 this e-mail and any attachments thereto may contain > confidential information and are intended for the sole use of the > recipient(s) named above. If you are not the intended recipient of this > message you are hereby notified that any dissemination or copying of this > message is strictly prohibited. If you have received this e-mail in error, > please notify the sender either by telephone or by e-mail and delete the > material from any computer. Thank you. > -------------------------------------------------------------------------- > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis at laxalde.org Sat Jan 21 04:41:53 2017 From: denis at laxalde.org (Denis Laxalde) Date: Sat, 21 Jan 2017 10:41:53 +0100 Subject: [SciPy-Dev] API to control iteration in differential evolution #6923 In-Reply-To: References: <6b000cae-8295-e690-39d3-054450d5cef1@laxalde.org> Message-ID: <7b660f99-a616-50c5-720c-976a58e21eed@laxalde.org> Evgeni Burovski a ?crit : > It seems to me we are all saying pretty much the same thing: More or less ;) I agree with most of your points, some more comments below. > 2. Since __init__ is always called, various initialization strategies > can be handled as a keyword arg in __init__(..., > init_method="latinhypercube"), possibly overloaded for a callable or a > population from a previous run? Or maybe the latter (reusing a > population) is better handled by an alternative constructor. But this > is certainly separate from both the context manager or iteration. We can certainly do that. As mentioned in the PR thread, it seems to me that __init__ method is not the proper place to run (possibly long) calculations such as population initialization. This is just a matter of separation of concerns. > 6. OptimizeStep could be a (current version of) OptimizeResult. In > fact, I'd suggest it *is* an OptimizeResult. Not sure I agree here. It seems to me that there's some value in distinguishing a step from a final result and using a different type looks appropriate. If you get an OptimizeResult at every iteration of the loop, it would be incomplete; missing in particular the "status" and "message" attributes, unless we set them to, e.g., status=None, message="in progress". This could bring confusion to the user who wouldn't know if its result is actually a result or just a step unless they introspect its content. To continue discussion on technical aspects, I would suggest moving back to the PR thread. From alebarde at gmail.com Mon Jan 23 13:56:11 2017 From: alebarde at gmail.com (alebarde at gmail.com) Date: Mon, 23 Jan 2017 19:56:11 +0100 Subject: [SciPy-Dev] [Numpy]: making np.gradient support unevenly spaced data In-Reply-To: References: Message-ID: 2017-01-20 16:20 GMT+01:00 : > > > On Fri, Jan 20, 2017 at 9:09 AM, alebarde at gmail.com > wrote: > >> 2017-01-20 14:44 GMT+01:00 : >> >>> >>> >>> On Fri, Jan 20, 2017 at 5:21 AM, alebarde at gmail.com >>> wrote: >>> >>>> Hi all, >>>> >>>> current version of numpy gradient supports only data with uniform >>>> spacing. I have implemented a proposed enhancement for the np.gradient >>>> function that allows to compute the gradient on non uniform grids. (PR: >>>> https://github.com/numpy/numpy/pull/8446) >>>> Since it may be of interest also for the scipy user/developers and it >>>> may be useful also for scipy.diff >>>> >>>> I am posting here too. >>>> >>>> The proposed implementation has a behaviour/signature is similar to >>>> that of Matlab/Octave. As argument it can take: >>>> 1. A single scalar to specify a sample distance for all dimensions. >>>> 2. N scalars to specify a constant sample distance for each dimension. >>>> i.e. `dx`, `dy`, `dz`, ... >>>> 3. N arrays to specify the coordinates of the values along each >>>> dimension of F. The length of the array must match the size of >>>> the corresponding dimension >>>> 4. Any combination of N scalars/arrays with the meaning of 2. and 3. >>>> >>>> e.g., you can do the following: >>>> >>>> >>> f = np.array([[1, 2, 6], [3, 4, 5]], dtype=np.float) >>>> >>> dx = 2. >>>> >>> y = [1., 1.5, 3.5] >>>> >>> np.gradient(f, dx, y) >>>> [array([[ 1. , 1. , -0.5], [ 1. , 1. , -0.5]]), >>>> array([[ 2. , 2. , 2. ], [ 2. , 1.7, 0.5]])] >>>> >>>> It should not break any existing code since as of 1.12 only scalars or >>>> list of scalars are allowed. >>>> A possible alternative API could be pass arrays of sampling steps >>>> instead of the coordinates. >>>> On the one hand, this would have the advantage of having "differences" >>>> both in the scalar case and in the array case. >>>> On the other hand, if you are dealing with non uniformly-spaced data >>>> (e.g, data is mapped on a grid or it is a time-series), in most cases you >>>> already have the coordinates/timestamps. Therefore, in the case of >>>> difference as argument, you would almost always have a call np.diff before >>>> np.gradient. >>>> >>>> In the end, I would rather prefer the coordinates option since IMHO it >>>> is more handy, I don't think that would be too much "surprising" and it is >>>> what Matlab already does. Also, it could not easily lead to "silly" >>>> mistakes since the length have to match the size of the corresponding >>>> dimension. >>>> >>>> What do you think? >>>> >>> >>> In general I also like specifying x better than dx. >>> >>> I looked at np.gradient two days ago to see if it does what I need. But >>> it was missing the unequal spacing, and it is missing the second derivative. >>> >> >> Yes, and I believe it is quite a useful feature. that's why I have >> implemented it. On the other hand I don't think that `gradient` should also >> be able to compute the second or kth derivative. In that case a more >> general function called `differentiate` would probably be better. >> >> >>> >>> more general question: I found the dimension story with dx, dy, dz, ... >>> confusing. Isn't this just applying gradient with respect to each axis >>> separately? >>> >> >> Yes it is just a way to pass the sampling step in each direction (and >> actually they are not keywords). I have kept the original documentation on >> this but it is probably better to change into h1,h2,h3. >> >> >>> >>> >>> (aside: If I read your link in the PR http://websrv.cs.umt.edu/isis/ >>> index.php/Finite_differencing:_Introduction correctly, then I might >>> have just implemented a rescaled version of second (kth) order differencing >>> with unequal spaced x using sparse linear algebra. I didn't see anything >>> premade in numpy or scipy for it.) >>> >> >> what do you mean with "rescaled version"? and you meant second order or >> second derivative? in the first case I think that would be equivalent to >> what my PR does. In the latter, I confirm that there is currently no >> implementation of a second derivative. >> > > I think it's k-th derivative. Rescaled means that I don't actually compute > the derivative, I want to get a nonparametric estimate of the residual > which assumes, I think, that the underlying function for the non-noise > version has approximately constant k-th derivative (e.g. k-th order > polynomial, or (k+1)-th). > The sum of squared window coefficients is normalized to 1. > I didn't try to understand the math, but the kernels or windows in the > equal spaced case are > > >>> _diff_kernel(0) > array([ 1.]) > >>> _diff_kernel(1) > array([ 0.70710678, -0.70710678]) > >>> _diff_kernel(2) > array([ 0.40824829, -0.81649658, 0.40824829]) > >>> _diff_kernel(2) / _diff_kernel(2)[1] > array([-0.5, 1. , -0.5]) > > >>> _diff_kernel(3) > array([ 0.2236068 , -0.67082039, 0.67082039, -0.2236068 ]) > >>> _diff_kernel(4) > array([ 0.11952286, -0.47809144, 0.71713717, -0.47809144, 0.11952286]) > >>> (_diff_kernel(4)**2).sum() > 0.99999999999999989 > > Yes, if I am not wrong you are getting the coefficient of the 1st order k-th derivative of the forward/backard approximation ( https://en.wikipedia.org/wiki/Finite_difference_coefficient#Forward_and_backward_finite_difference ) Fornberg ( http://www.ams.org/journals/mcom/1988-51-184/S0025-5718-1988-0935077-0/S0025-5718-1988-0935077-0.pdf) proposed an algorithm to get the coefficients for a general N-order K-th derivative but I don't know how efficient it is. Also, I am not sure if it applies to your case but mind that when going to higher order strange effects happens unless "nodes" are properly chosen see e.g., figures last page of ( https://www.rsmas.miami.edu/users/miskandarani/Courses/MSC321/lectfiniteDifference.pdf) -- -------------------------------------------------------------------------- NOTICE: Dlgs 196/2003 this e-mail and any attachments thereto may contain confidential information and are intended for the sole use of the recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any dissemination or copying of this message is strictly prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you. -------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Jan 23 14:54:59 2017 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 23 Jan 2017 14:54:59 -0500 Subject: [SciPy-Dev] [Numpy]: making np.gradient support unevenly spaced data In-Reply-To: References: Message-ID: On Mon, Jan 23, 2017 at 1:56 PM, alebarde at gmail.com wrote: > > > 2017-01-20 16:20 GMT+01:00 : > >> >> >> On Fri, Jan 20, 2017 at 9:09 AM, alebarde at gmail.com >> wrote: >> >>> 2017-01-20 14:44 GMT+01:00 : >>> >>>> >>>> >>>> On Fri, Jan 20, 2017 at 5:21 AM, alebarde at gmail.com >>> > wrote: >>>> >>>>> Hi all, >>>>> >>>>> current version of numpy gradient supports only data with uniform >>>>> spacing. I have implemented a proposed enhancement for the np.gradient >>>>> function that allows to compute the gradient on non uniform grids. (PR: >>>>> https://github.com/numpy/numpy/pull/8446) >>>>> Since it may be of interest also for the scipy user/developers and it >>>>> may be useful also for scipy.diff >>>>> >>>>> I am posting here too. >>>>> >>>>> The proposed implementation has a behaviour/signature is similar to >>>>> that of Matlab/Octave. As argument it can take: >>>>> 1. A single scalar to specify a sample distance for all dimensions. >>>>> 2. N scalars to specify a constant sample distance for each dimension. >>>>> i.e. `dx`, `dy`, `dz`, ... >>>>> 3. N arrays to specify the coordinates of the values along each >>>>> dimension of F. The length of the array must match the size of >>>>> the corresponding dimension >>>>> 4. Any combination of N scalars/arrays with the meaning of 2. and 3. >>>>> >>>>> e.g., you can do the following: >>>>> >>>>> >>> f = np.array([[1, 2, 6], [3, 4, 5]], dtype=np.float) >>>>> >>> dx = 2. >>>>> >>> y = [1., 1.5, 3.5] >>>>> >>> np.gradient(f, dx, y) >>>>> [array([[ 1. , 1. , -0.5], [ 1. , 1. , -0.5]]), >>>>> array([[ 2. , 2. , 2. ], [ 2. , 1.7, 0.5]])] >>>>> >>>>> It should not break any existing code since as of 1.12 only scalars or >>>>> list of scalars are allowed. >>>>> A possible alternative API could be pass arrays of sampling steps >>>>> instead of the coordinates. >>>>> On the one hand, this would have the advantage of having "differences" >>>>> both in the scalar case and in the array case. >>>>> On the other hand, if you are dealing with non uniformly-spaced data >>>>> (e.g, data is mapped on a grid or it is a time-series), in most cases you >>>>> already have the coordinates/timestamps. Therefore, in the case of >>>>> difference as argument, you would almost always have a call np.diff before >>>>> np.gradient. >>>>> >>>>> In the end, I would rather prefer the coordinates option since IMHO it >>>>> is more handy, I don't think that would be too much "surprising" and it is >>>>> what Matlab already does. Also, it could not easily lead to "silly" >>>>> mistakes since the length have to match the size of the corresponding >>>>> dimension. >>>>> >>>>> What do you think? >>>>> >>>> >>>> In general I also like specifying x better than dx. >>>> >>>> I looked at np.gradient two days ago to see if it does what I need. But >>>> it was missing the unequal spacing, and it is missing the second derivative. >>>> >>> >>> Yes, and I believe it is quite a useful feature. that's why I have >>> implemented it. On the other hand I don't think that `gradient` should also >>> be able to compute the second or kth derivative. In that case a more >>> general function called `differentiate` would probably be better. >>> >>> >>>> >>>> more general question: I found the dimension story with dx, dy, dz, ... >>>> confusing. Isn't this just applying gradient with respect to each axis >>>> separately? >>>> >>> >>> Yes it is just a way to pass the sampling step in each direction (and >>> actually they are not keywords). I have kept the original documentation on >>> this but it is probably better to change into h1,h2,h3. >>> >>> >>>> >>>> >>>> (aside: If I read your link in the PR http://websrv.cs.umt.edu/isis/ >>>> index.php/Finite_differencing:_Introduction correctly, then I might >>>> have just implemented a rescaled version of second (kth) order differencing >>>> with unequal spaced x using sparse linear algebra. I didn't see anything >>>> premade in numpy or scipy for it.) >>>> >>> >>> what do you mean with "rescaled version"? and you meant second order or >>> second derivative? in the first case I think that would be equivalent to >>> what my PR does. In the latter, I confirm that there is currently no >>> implementation of a second derivative. >>> >> >> I think it's k-th derivative. Rescaled means that I don't actually >> compute the derivative, I want to get a nonparametric estimate of the >> residual which assumes, I think, that the underlying function for the >> non-noise version has approximately constant k-th derivative (e.g. k-th >> order polynomial, or (k+1)-th). >> The sum of squared window coefficients is normalized to 1. >> I didn't try to understand the math, but the kernels or windows in the >> equal spaced case are >> >> >>> _diff_kernel(0) >> array([ 1.]) >> >>> _diff_kernel(1) >> array([ 0.70710678, -0.70710678]) >> >>> _diff_kernel(2) >> array([ 0.40824829, -0.81649658, 0.40824829]) >> >>> _diff_kernel(2) / _diff_kernel(2)[1] >> array([-0.5, 1. , -0.5]) >> >> >>> _diff_kernel(3) >> array([ 0.2236068 , -0.67082039, 0.67082039, -0.2236068 ]) >> >>> _diff_kernel(4) >> array([ 0.11952286, -0.47809144, 0.71713717, -0.47809144, 0.11952286]) >> >>> (_diff_kernel(4)**2).sum() >> 0.99999999999999989 >> >> Yes, if I am not wrong you are getting the coefficient of the 1st order > k-th derivative of the forward/backard approximation ( > https://en.wikipedia.org/wiki/Finite_difference_coefficient#Forward_and_ > backward_finite_difference) > Fornberg (http://www.ams.org/journals/mcom/1988-51-184/S0025-5718- > 1988-0935077-0/S0025-5718-1988-0935077-0.pdf) proposed an algorithm to > get the coefficients for a general N-order K-th derivative but I don't know > how efficient it is. > Thanks for the links. My underlying story ended up being close to savitzky-golay e.g. central differences with increasing accuracy >>> signal.savgol_coeffs(3, 2, deriv=1) array([ 5.00000000e-01, -4.53646149e-17, -5.00000000e-01]) >>> signal.savgol_coeffs(5, 4, deriv=1) array([ -8.33333333e-02, 6.66666667e-01, 1.31151738e-15, -6.66666667e-01, 8.33333333e-02]) In the simple differencing appoach window length is tied to the number of derivatives/differencing. But with local polynomials like savitzky-golay the window length (and order of approximation) can be separated from the derivative order. This works with equal spaced points, but I gave up for now going beyond simple differencing for unequal spaced points, because I don't see a fast way to create the windows for general local polynomial regression. (In case anyone is interested: https://github.com/statsmodels/statsmodels/pull/3380 where it took me a while to figure out the relationship to savitzky-golay) > Also, I am not sure if it applies to your case but mind that when going to > higher order strange effects happens unless "nodes" are properly chosen see > e.g., figures last page of (https://www.rsmas.miami.edu/ > users/miskandarani/Courses/MSC321/lectfiniteDifference.pdf) > In my experience this is more of a problem for interpolation than for lower order smoothing, but it's not ruled out. It should be even less of a problem with local, usually low order polynomials as in savitzky-golay. Josef > -- > -------------------------------------------------------------------------- > NOTICE: Dlgs 196/2003 this e-mail and any attachments thereto may contain > confidential information and are intended for the sole use of the > recipient(s) named above. If you are not the intended recipient of this > message you are hereby notified that any dissemination or copying of this > message is strictly prohibited. If you have received this e-mail in error, > please notify the sender either by telephone or by e-mail and delete the > material from any computer. Thank you. > -------------------------------------------------------------------------- > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: