From andyfaff at gmail.com  Mon Jan  1 03:31:44 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Mon, 1 Jan 2018 19:31:44 +1100
Subject: [SciPy-Dev] TravisCI issues
Message-ID: <CAAbtOZebHmKS+vGmm3xB5K22tcy9QyqY5LEWRzmPpZeHE_7umA@mail.gmail.com>

Happy New Year everyone.

Whilst submitting a PR today I noticed that some of the travis builds are
failing with the error:

"oci runtime error: exec failed: container_linux.go:265: starting container
process caused "could not create session key: disk quota exceeded"

I'm hoping it's a temporary issue.

A.
-- 
_____________________________________
Dr. Andrew Nelson


_____________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180101/39275d7f/attachment.html>

From matthew.brett at gmail.com  Mon Jan  1 07:05:34 2018
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 1 Jan 2018 12:05:34 +0000
Subject: [SciPy-Dev] TravisCI issues
In-Reply-To: <CAAbtOZebHmKS+vGmm3xB5K22tcy9QyqY5LEWRzmPpZeHE_7umA@mail.gmail.com>
References: <CAAbtOZebHmKS+vGmm3xB5K22tcy9QyqY5LEWRzmPpZeHE_7umA@mail.gmail.com>
Message-ID: <CAH6Pt5o3sYogcGRtskS+3qD63exzvD1u1qgyNu+uu8ediWWD6A@mail.gmail.com>

Hi,

On Mon, Jan 1, 2018 at 8:31 AM, Andrew Nelson <andyfaff at gmail.com> wrote:
> Happy New Year everyone.
>
> Whilst submitting a PR today I noticed that some of the travis builds are
> failing with the error:
>
> "oci runtime error: exec failed: container_linux.go:265: starting container
> process caused "could not create session key: disk quota exceeded"
>
> I'm hoping it's a temporary issue.

I'm getting the same error on another repo:

https://travis-ci.org/matthew-brett/nb2plots/jobs/323694309

I've sent an email to their support address,

Cheers,

Matthew

From ilhanpolat at gmail.com  Mon Jan  1 11:23:54 2018
From: ilhanpolat at gmail.com (Ilhan Polat)
Date: Mon, 1 Jan 2018 17:23:54 +0100
Subject: [SciPy-Dev] TravisCI issues
In-Reply-To: <CAH6Pt5o3sYogcGRtskS+3qD63exzvD1u1qgyNu+uu8ediWWD6A@mail.gmail.com>
References: <CAAbtOZebHmKS+vGmm3xB5K22tcy9QyqY5LEWRzmPpZeHE_7umA@mail.gmail.com>
 <CAH6Pt5o3sYogcGRtskS+3qD63exzvD1u1qgyNu+uu8ediWWD6A@mail.gmail.com>
Message-ID: <CAEBuzr_bnviVPxdaQM3TOBt0qwg9rruqLLB3jVES4Z4NVwms8Q@mail.gmail.com>

I have manually started one-by-one so far all passed. So it is probably a
TravisCI issue that is being resolved.

On Mon, Jan 1, 2018 at 1:05 PM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> Hi,
>
> On Mon, Jan 1, 2018 at 8:31 AM, Andrew Nelson <andyfaff at gmail.com> wrote:
> > Happy New Year everyone.
> >
> > Whilst submitting a PR today I noticed that some of the travis builds are
> > failing with the error:
> >
> > "oci runtime error: exec failed: container_linux.go:265: starting
> container
> > process caused "could not create session key: disk quota exceeded"
> >
> > I'm hoping it's a temporary issue.
>
> I'm getting the same error on another repo:
>
> https://travis-ci.org/matthew-brett/nb2plots/jobs/323694309
>
> I've sent an email to their support address,
>
> Cheers,
>
> Matthew
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180101/08c0c969/attachment.html>

From stefanv at berkeley.edu  Thu Jan  4 16:14:13 2018
From: stefanv at berkeley.edu (Stefan van der Walt)
Date: Thu, 04 Jan 2018 13:14:13 -0800
Subject: [SciPy-Dev] Position at BIDS (UC Berkeley) to work on NumPy
Message-ID: <1515100453.2960132.1224583776.2F500AEF@webmail.messagingengine.com>

Hi everyone,

The Berkeley Institute for Data Science (BIDS) is hiring scientific Python Developers to contribute to NumPy.  You can read more about the new positions here:

https://bids.berkeley.edu/news/bids-receives-sloan-foundation-grant-contribute-numpy-development

If you enjoy collaborative work as well as the technical challenges posed by numerical computing, this is an excellent opportunity to play a fundamental role in the development of one of the most impactful libraries in the entire Python ecosystem.

Best regards
St?fan

Job link: https://jobsprod.is.berkeley.edu/psc/jobsprod/EMPLOYEE/HRMS/c/HRS_HRAM.HRS_CE.GBL?Page=HRS_CE_JOB_DTL&Action=A&JobOpeningId=24142&SiteId=1&PostingSeq=1

From charlesr.harris at gmail.com  Thu Jan  4 18:52:30 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 4 Jan 2018 16:52:30 -0700
Subject: [SciPy-Dev] Position at BIDS (UC Berkeley) to work on NumPy
In-Reply-To: <1515100453.2960132.1224583776.2F500AEF@webmail.messagingengine.com>
References: <1515100453.2960132.1224583776.2F500AEF@webmail.messagingengine.com>
Message-ID: <CAB6mnx+Fn8vnruhVEuMppRSTzabVRc8+cSpgXqMbC9-Ap2PV=A@mail.gmail.com>

Should send this to the NumPy list also.

On Thu, Jan 4, 2018 at 2:14 PM, Stefan van der Walt <stefanv at berkeley.edu>
wrote:

> Hi everyone,
>
> The Berkeley Institute for Data Science (BIDS) is hiring scientific Python
> Developers to contribute to NumPy.  You can read more about the new
> positions here:
>
> https://bids.berkeley.edu/news/bids-receives-sloan-
> foundation-grant-contribute-numpy-development
>
> If you enjoy collaborative work as well as the technical challenges posed
> by numerical computing, this is an excellent opportunity to play a
> fundamental role in the development of one of the most impactful libraries
> in the entire Python ecosystem.
>
> Best regards
> St?fan
>
> Job link: https://jobsprod.is.berkeley.edu/psc/jobsprod/EMPLOYEE/
> HRMS/c/HRS_HRAM.HRS_CE.GBL?Page=HRS_CE_JOB_DTL&Action=A&
> JobOpeningId=24142&SiteId=1&PostingSeq=1
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180104/fc9a05af/attachment.html>

From charlesr.harris at gmail.com  Sat Jan  6 20:00:20 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 6 Jan 2018 18:00:20 -0700
Subject: [SciPy-Dev] NumPy 1.14.0 release
Message-ID: <CAB6mnxJDJp4R2dbtqdbccYFB=22nwhv93-R9DqTRc4a77LAgpQ@mail.gmail.com>

Hi All,

On behalf of the NumPy team, I am pleased to announce NumPy 1.14.0. Numpy
1.14.0 is the result of seven months of work and contains a large number of
bug fixes and new features, along with several changes with potential
compatibility issues. The major change that users will notice are the
stylistic changes in the way numpy arrays and scalars are printed, a change
that will affect doctests. See the release notes for details on how to
preserve the old style printing when needed.

A major decision affecting future development concerns the schedule for
dropping Python 2.7 support in the runup to 2020. The decision has been
made to support 2.7 for all releases made in 2018, with the last release
being designated a long term release with support for bug fixes extending
through the end of 2019. Starting from January, 2019 support for 2.7 will
be dropped in all new releases. More details can be found in the relevant
NEP
<https://github.com/numpy/numpy/blob/master/doc/neps/dropping-python2.7-proposal.rst>
.

This release supports Python 2.7 and 3.4 - 3.6. Wheels for the release are
available on PyPI. Source tarballs, zipfiles, release notes, and the
changelog are available on github
<https://github.com/numpy/numpy/releases/tag/v1.14.0>.


*Highlights*

   - The ``np.einsum`` function uses BLAS when possible

   - ``genfromtxt``, ``loadtxt``, ``fromregex`` and ``savetxt`` can now
   handle files with arbitrary Python supported encoding.

   - Major improvements to printing of NumPy arrays and scalars.


*New functions*

   - ``parametrize``: decorator added to numpy.testing


   - ``chebinterpolate``: Interpolate function at Chebyshev points.


   - ``format_float_positional`` and ``format_float_scientific`` : format
   floating-point scalars unambiguously with control of rounding and padding.


   - ``PyArray_ResolveWritebackIfCopy`` and ``PyArray_SetWritebackIfCopyBase``,
   new C-API functions useful in achieving PyPy compatibity.


*Contributors*

A total of 100 people contributed to this release.  People with a "+" by
their
names contributed a patch for the first time.

* Alexey Brodkin +
* Allan Haldane
* Andras Deak +
* Andrew Lawson +
* Anna Chiara +
* Antoine Pitrou
* Bernhard M. Wiedemann +
* Bob Eldering +
* Brandon Carter
* CJ Carey
* Charles Harris
* Chris Lamb
* Christoph Boeddeker +
* Christoph Gohlke
* Daniel Hrisca +
* Daniel Smith
* Danny Hermes
* David Freese
* David Hagen
* David Linke +
* David Schaefer +
* Dillon Niederhut +
* Egor Panfilov +
* Emilien Kofman
* Eric Wieser
* Erik Bray +
* Erik Quaeghebeur +
* Garry Polley +
* Gunjan +
* Han Shen +
* Henke Adolfsson +
* Hidehiro NAGAOKA +
* Hemil Desai +
* Hong Xu +
* Iryna Shcherbina +
* Jaime Fernandez
* James Bourbeau +
* Jamie Townsend +
* Jarrod Millman
* Jean Helie +
* Jeroen Demeyer +
* John Goetz +
* John Kirkham
* John Zwinck
* Jonathan Helmus
* Joseph Fox-Rabinovitz
* Joseph Paul Cohen +
* Joshua Leahy +
* Julian Taylor
* J?rg D?pfert +
* Keno Goertz +
* Kevin Sheppard +
* Kexuan Sun +
* Konrad Kapp +
* Kristofor Maynard +
* Licht Takeuchi +
* Lo?c Est?ve
* Lukas Mericle +
* Marten van Kerkwijk
* Matheus Portela +
* Matthew Brett
* Matti Picus
* Michael Lamparski +
* Michael Odintsov +
* Michael Schnaitter +
* Michael Seifert
* Mike Nolta
* Nathaniel J. Smith
* Nelle Varoquaux +
* Nicholas Del Grosso +
* Nico Schl?mer +
* Oleg Zabluda +
* Oleksandr Pavlyk
* Pauli Virtanen
* Pim de Haan +
* Ralf Gommers
* Robert T. McGibbon +
* Roland Kaufmann
* Sebastian Berg
* Serhiy Storchaka +
* Shitian Ni +
* Spencer Hill +
* Srinivas Reddy Thatiparthy +
* Stefan Winkler +
* Stephan Hoyer
* Steven Maude +
* SuperBo +
* Thomas K?ppe +
* Toon Verstraelen
* Vedant Misra +
* Warren Weckesser
* Wirawan Purwanto +
* Yang Li +
* Ziyan Zhou +
* chaoyu3 +
* orbit-stabilizer +
* solarjoe
* wufangjie +
* xoviat +
* ?lie Gouzien +

Cheers,

Charles Harris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180106/6c68a36d/attachment.html>

From andyfaff at gmail.com  Mon Jan  8 19:35:22 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Tue, 9 Jan 2018 11:35:22 +1100
Subject: [SciPy-Dev] General discussion on parallelisation
Message-ID: <CAAbtOZd=pGftqOEabw9qkQyV1KszHF-q3x5tE6GKVCa9yoEBPw@mail.gmail.com>

As part of https://github.com/scipy/scipy/pull/8259 I'm proposing that a
`workers` keyword is added to optimize.differential_evolution to
parallelise some computation.

The proposal is that:

1. the workers keyword accepts either an integer or an object with a
map-like method.
2. If an integer is supplied then the parallelisation is taken care of by
scipy (more on that later), with -1 signifying that all processors are to
be used.
3. If an object with a map-like method is supplied, e.g.
`multiprocessing.Pool.map`, `mpi4py.futures.MPIPoolExecutor.map`, etc, then
the parallelisation is taken care of by that object. This allows the user
to specify the parallelisation configuration for their problem.
4. If workers=1, then computation will be done by the builtin `map`
function.

Now we come to the under the hood part. I've written something called
PoolWrapper (
https://github.com/andyfaff/scipy/blob/b14bb513c0ffb9807a67663d39b9ab399375d37d/scipy/_lib/_util.py#L343)
which wraps `multiprocessing.Pool` to achieve the behaviour outlined above.
It can be used as a context manager, or the user of the object can decide
when to close the resources opened by PoolWrapper.

I've looked at using joblib instead of PoolWrapper and it seems useful but
it doesn't have a couple of bits of functionality that are needed for this
specific problem:

viz:
5. joblib.Parallel doesn't have a map method (desirable to allow 3) so a
small wrapper would have to be created anyway.
6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the
Parallel object is `__call__`ed. This leads to significant overhead. One
can use the Parallel object with a context manager, which allows reuse of
the Pool, but I don't think that's do-able in the context of using the
DifferentialEvolutionSolver (DES) object as an iterator:

>>> solver = DifferentialEvolutionSolver(func, bounds)
>>> # use DES object as an iterator
>>> for it in solver:
...        res = next(solver)
>>> print(res)
>>> # use the DES.solve method
>>> res = solver.solve()

Whilst the DES object is not currently public (it's called by the
differential_evolution function) it would be nice to expose it in the
future, and people will want to use both approaches. Unfortunately with the
first approach if we used joblib.Parallel we'd have to use
Parallel.__call__ in DES.next() which has the overhead of
creating/destroying Pools. For efficient use of resources the Pool should
persist for the lifetime of the DES object.

I also looked into `concurrent.futures.ProcessPoolExecutor`, but it's not
available for Python 2.7.

The purpose of this email is to elicit feedback for developing
parallelisation strategy for scipy - what does the public interface look
like, what does scipy do under the hood?

Under the hood I think a mixture of PoolWrapper and joblib.Parallel could
be used (with scipy vendoring joblib).

A.
--
_____________________________________
Dr. Andrew Nelson


_____________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180109/2b0c51de/attachment.html>

From larson.eric.d at gmail.com  Mon Jan  8 21:48:42 2018
From: larson.eric.d at gmail.com (Eric Larson)
Date: Mon, 8 Jan 2018 21:48:42 -0500
Subject: [SciPy-Dev] General discussion on parallelisation
In-Reply-To: <CAAbtOZd=pGftqOEabw9qkQyV1KszHF-q3x5tE6GKVCa9yoEBPw@mail.gmail.com>
References: <CAAbtOZd=pGftqOEabw9qkQyV1KszHF-q3x5tE6GKVCa9yoEBPw@mail.gmail.com>
Message-ID: <CAGu2niX736CpSQ+_zaxZOJVVcxaeh4rvNDu6wTAxWZsfgiG40A@mail.gmail.com>

>
> 6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the
> Parallel object is `__call__`ed. This leads to significant overhead. One
> can use the Parallel object with a context manager, which allows reuse of
> the Pool, but I don't think that's do-able in the context of using the
> DifferentialEvolutionSolver (DES) object as an iterator:
>

If you can use `__iter__` instead of `__next__` in your
DifferentialEvolutionSolver class I think you can avoid this problem with
something along the lines of:

class DifferentialEvolutionSolver(object):
    ...
    def __iter__(self):
        with Parallel(...) ...:
            for it in ...:
                yield it

?
As for the map problem I don't know, but it's probably worth asking the
`joblib` people if there is a suitable solution or workaround.

Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180108/131eaff4/attachment.html>

From andyfaff at gmail.com  Mon Jan  8 23:22:17 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Tue, 9 Jan 2018 15:22:17 +1100
Subject: [SciPy-Dev] General discussion on parallelisation
In-Reply-To: <CAGu2niX736CpSQ+_zaxZOJVVcxaeh4rvNDu6wTAxWZsfgiG40A@mail.gmail.com>
References: <CAAbtOZd=pGftqOEabw9qkQyV1KszHF-q3x5tE6GKVCa9yoEBPw@mail.gmail.com>
 <CAGu2niX736CpSQ+_zaxZOJVVcxaeh4rvNDu6wTAxWZsfgiG40A@mail.gmail.com>
Message-ID: <CAAbtOZeL6AgW3d5-kLGA8pYLWEn+S4AGYX95oHun-GXZViX0rA@mail.gmail.com>

>
> On 9 January 2018 at 13:48, Eric Larson <larson.eric.d at gmail.com> wrote:
> > If you can use `__iter__` instead of `__next__` in your
> DifferentialEvolutionSolver class I think you can avoid this problem with
> something along the lines of:
> >
> > class DifferentialEvolutionSolver(object):
> >     ...
> >     def __iter__(self):
> >         with Parallel(...) ...:
> >             for it in ...:
> >                 yield it
>

That seems Pythonic, I'll look into that in more detail. One hurdle might
be when the two approaches are mixed, starting off with a generator, uses
the solver method, etc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180109/674579e0/attachment.html>

From njs at pobox.com  Tue Jan  9 03:24:17 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 9 Jan 2018 00:24:17 -0800
Subject: [SciPy-Dev] RFC: comments to BLAS committee from numpy/scipy devs
Message-ID: <CAPJVwBk_fO1K+CFFOAF05Hgpd6YbKfnOfXYpX2q0+gPPsd-kZg@mail.gmail.com>

Hi all,

As mentioned earlier [1][2], there's work underway to revise and
update the BLAS standard -- e.g. we might get support for strided
arrays and lose xerbla! There's a draft at [3]. They're interested in
feedback from users, so I've written up a first draft of comments
about what we would like as NumPy/SciPy developers. This is very much
a first attempt -- I know we have lots of people who are more expert
on BLAS than me on these lists :-). Please let me know what you think.

-n

[1] https://mail.python.org/pipermail/numpy-discussion/2017-November/077420.html
[2] https://mail.python.org/pipermail/scipy-dev/2017-November/022267.html
[3] https://docs.google.com/document/d/1DY4ImZT1coqri2382GusXgBTTTVdBDvtD5I14QHp9OE/edit

-----

# Comments from NumPy / SciPy developers on "A Proposal for a
Next-Generation BLAS"

These are comments on [A Proposal for a Next-Generation
BLAS](https://docs.google.com/document/d/1DY4ImZT1coqri2382GusXgBTTTVdBDvtD5I14QHp9OE/edit#)
(version as of 2017-12-13), from the perspective of the developers of
the NumPy and SciPy libraries. We hope this feedback is useful, and
welcome further discussion.

## Who are we?

NumPy and SciPy are the two foundational libraries of the Python
numerical ecosystem, and one of their duties is to wrap BLAS and
expose it for the use of other Python libraries. (NumPy primarily
provides a GEMM wrapper, while SciPy exposes more specialized
operations.) It's unclear how many users we have exactly, but we
certainly ship multiple million copies of BLAS every month, and
provide one of the most popular numerical toolkits for both novice and
expert users.

Looking at the original BLAS and LAPACK interfaces, it often seems
that their imagined user is something like a classic supercomputer
consumer, who writes code directly in Fortran or C against the BLAS
API, and where the person writing the code and running the code are
the same. NumPy/SciPy are coming from a very different perspective:
our users generally know nothing about the details of the underlying
BLAS; they just want to describe their problem in some high-level way,
and the library is responsible for making it happen as efficiently as
possible, and is often integrated into some larger system (e.g. a
real-time analytics platform embedded in a web server).

When it comes to our BLAS usage, we mostly use only a small subset of
the routines. However, as "consumer software" used by a wide variety
of users with differing degress of technical expertise, we're expected
to Just Work on a wide variety of systems, and with as many different
vendor BLAS libraries as possible. On the other hand, the fact that
we're working with Python means we don't tend to worry about small
inefficiencies that will be lost in the noise in any case, and are
willing to sacrifice some performance to get more reliable operation
across our diverse userbase.

## Comments on specific aspects of the proposal

### Data Layout

We are **strongly in favor** of the proposal to support arbitrary
strided data layouts. Ideally, this would support strides *specified
in bytes* (allowing for unaligned data layouts), and allow for truly
arbitrary strides, including *zero or negative* values. However, we
think it's fine if some of the weirder cases suffer a performance
penalty.

Rationale: NumPy ? and thus, most of the scientific Python ecosystem ?
only has one way of representing an array: the `numpy.ndarray` type,
which is an arbitrary dimensional tensor with arbitrary strides. It is
common to encounter matrices with non-trivial strides. For example::

    # Make a 3-dimensional tensor, 10 x 9 x 8
    t = np.zeros((10, 9, 8))
    # Considering this as a stack of eight 10x9 matrices, extract the first:
    mat = t[:, :, 0]

Now `mat` has non-trivial strides on both axes. (If running this in a
Python interpreter, you can see this by looking at the value of
`mat.strides`.) Another case where interesting strides arise is when
performing ["broadcasting"](https://docs.scipy.org/doc/numpy-1.13.0/user/basics.broadcasting.html),
which is the name for NumPy's rules for stretching arrays to make
their shapes match. For example, in an expression like::

    np.array([1, 2, 3]) + 1

the scalar `1` is "broadcast" to create a vector `[1, 1, 1]`. This is
accomplished without allocating memory, by creating a vector with
settings length = 3, strides = 0 ? so all the elements share a single
location in memory. Similarly, by using negative strides we can
reverse an array without allocating memory::

    a = np.array([1, 2, 3])
    a_flipped = a[::-1]

Now `a_flipped` has the value `[3, 2, 1]`, while sharing storage with
the array `a = [1, 2, 3]`. Misaligned data is also possible (e.g. an
array of 8-byte doubles with a 9-byte stride), though it arises more
rarely. (An example of when it might occurs is in an on-disk data
format that alternates between storing a double value and then a
single byte value, which is then memory-mapped.)

While this array representation is very flexible and useful, it makes
interfacing with BLAS a challenge: how do you perform a GEMM operation
on two arrays with arbitrary strides? Currently, NumPy attempts to
detect a number of special cases: if the strides in both arrays imply
a column-major layout, then call BLAS directly; if one of them has
strides corresponding to a row-major layout, then set the
corresponding `transA`/`transB` argument, etc. ? and if all else
fails, either copy the data into a contiguous buffer, or else fall
back on a naive triple-nested-loop GEMM implementation. (There's also
a check where if we can determine through examining the arrays' data
pointers and strides that they're actually transposes of each other,
then we instead dispatch to SYRK.)

So for us, native stride support in the BLAS has two major advantages:

- It allows a wider variety of operations to be transparently handled
using high-speed kernels.

- It reduces the complexity of the NumPy/BLAS interface code.

Any kind of stride support produces both of these advantages to some
extent. The ideal would be if BLAS supports strides specified in bytes
(not elements), and allowing truly arbitrary values, because in this
case, we can simply pass through our arrays directly to the library
without having to do any fiddly checking at all. Note that for these
purposes, it's OK if "weird" arrays pay a speed penalty: if the BLAS
didn't support them, we'd just have to fall back on some even worse
strategy for handling them ourselves. And this approach would mean
that if some BLAS vendors do at some point decide to optimize these
cases, then our users will immediately see the advantages.

### Error-reporting

We are **strongly in favor** of the draft's proposal to replace XERBLA
with proper error codes. XERBLA is fine for one-off programs in
Fortran, but awful for wrapper libraries like NumPy/SciPy.

### Handling of exceptional values

A recurring issue in low-level linear-algebra libraries is that they
may crash, freeze, or produce unexpected results when receiving
non-finite inputs. Of course as a wrapper library, NumPy/SciPy can't
control what kind of strange data our users might pass in, and we have
to produce sensible results regardless, so we sometimes accumulate
hacks like having to validate all incoming data for exceptional values

We support the proposal's recommendation of "NaN interpretation (4)".

We also note that contrary to the note under "Interpretation (1)"; in
the R statistical environment NaN does *not* represent missing data. R
maintains a clear distinction between "NA" (missing) values and "NaN"
(invalid) values. This is important for various statistical procedures
such as imputation, where you might want to estimate the true value of
an unmeasured variable, but you don't want to estimate the true value
of 0/0. For efficiency, they do perform some calculations on NA values
by storing them as NaNs with a specific payload; in the R context,
therefore, it's particularly valuable if operations **correctly
propagate NaN payloads**. NumPy does not yet provide first-class
missing value support, but we plan to add it within the next 1-2
years, and when we do it seems likely that we'll have similar needs to
R.

### BLAS G2

NumPy has a rich type system, including 16-, 32-, and 64-bit floats
(plus extended precision on some platforms), a similar set of complex
values, and is further extensible with new types. We have a similarly
rich dispatch system for operations that attempts to find the best
matching optimized-loop for an arbitrary set of input types. We're
thus quite interested in the proposed mixed-type operations, and if
implemented we'll probably start transparently taking advantage of
them (i.e., in cases where we're currently upcasting to perform a
requested operation, we may switch to using the native precision
operation without requiring any changes in user code).

Two comments, though:

1. The various tricks to make names less redundant (e.g., specifying
just one type if they're all the same) are convenient for those
calling these routines by hand, but rather inconvenient for those of
us who will be generating a table mapping different type combinations
to different functions. When designing a compression scheme for names,
please remember that the scheme will have to be unambiguously
implemented in code by multiple projects. Another option to consider:
ask vendors to implement the full "uncompressed" names, and then
provide a shim library mapping short "convenience" names to their full
form.

2. Regarding the suggestion that the naming scheme will allow for a
wide variety of differently typed routines, but that only some subset
will be implemented by any particular vendor: we are worried about
this subset business. For a library like NumPy that wants to wrap
"all" the provided routines, while supporting "all" the different BLAS
libraries ... how will this work? **How do we know which routines any
particular library provides?** What if new routines are added to the
list later?

### Reproducible BLAS

This is interesting, but we don't have any particularly useful comments.

### Batch BLAS

NumPy/SciPy actually provide batched operations in many cases,
currently implemented using a `for` loop around individual calls to
BLAS/LAPACK. However, our interface is relatively restricted compared
to some of the proposals considered here: generally all we need is the
ability to perform an operation on two "stacks" of size-homogenous
matrices, with the offset between matrices determined by an arbitrary
(potentially zero) stride.

### Fixed-point BLAS

This is interesting, but we don't have any particularly useful comments.


-- 
Nathaniel J. Smith -- https://vorpus.org

From ilhanpolat at gmail.com  Tue Jan  9 06:40:02 2018
From: ilhanpolat at gmail.com (Ilhan Polat)
Date: Tue, 9 Jan 2018 12:40:02 +0100
Subject: [SciPy-Dev] RFC: comments to BLAS committee from numpy/scipy
 devs
In-Reply-To: <CAPJVwBk_fO1K+CFFOAF05Hgpd6YbKfnOfXYpX2q0+gPPsd-kZg@mail.gmail.com>
References: <CAPJVwBk_fO1K+CFFOAF05Hgpd6YbKfnOfXYpX2q0+gPPsd-kZg@mail.gmail.com>
Message-ID: <CAEBuzr8+RtABKtYVPwzoaUs5GCFd+9RsGq0djy1JE8QGexK+CQ@mail.gmail.com>

I couldn't find an item to place this but I think ilaenv and also calling
the function twice (one with lwork=-1 and reading the optimal block size
and the call the function again properly with lwork=<result>) in LAPACK
needs to be gotten rid of.

That's a major annoyance during the wrapping of LAPACK routines for SciPy.

I don't know if this is realistic but the values ilaenv needed can be
computed once (or again if hardware is changed) at the install and can be
read off by the routines.



On Jan 9, 2018 09:25, "Nathaniel Smith" <njs at pobox.com> wrote:

> Hi all,
>
> As mentioned earlier [1][2], there's work underway to revise and
> update the BLAS standard -- e.g. we might get support for strided
> arrays and lose xerbla! There's a draft at [3]. They're interested in
> feedback from users, so I've written up a first draft of comments
> about what we would like as NumPy/SciPy developers. This is very much
> a first attempt -- I know we have lots of people who are more expert
> on BLAS than me on these lists :-). Please let me know what you think.
>
> -n
>
> [1] https://mail.python.org/pipermail/numpy-discussion/
> 2017-November/077420.html
> [2] https://mail.python.org/pipermail/scipy-dev/2017-November/022267.html
> [3] https://docs.google.com/document/d/1DY4ImZT1coqri2382GusXgBTTTVdB
> DvtD5I14QHp9OE/edit
>
> -----
>
> # Comments from NumPy / SciPy developers on "A Proposal for a
> Next-Generation BLAS"
>
> These are comments on [A Proposal for a Next-Generation
> BLAS](https://docs.google.com/document/d/1DY4ImZT1coqri2382GusXgBTTTVdB
> DvtD5I14QHp9OE/edit#)
> (version as of 2017-12-13), from the perspective of the developers of
> the NumPy and SciPy libraries. We hope this feedback is useful, and
> welcome further discussion.
>
> ## Who are we?
>
> NumPy and SciPy are the two foundational libraries of the Python
> numerical ecosystem, and one of their duties is to wrap BLAS and
> expose it for the use of other Python libraries. (NumPy primarily
> provides a GEMM wrapper, while SciPy exposes more specialized
> operations.) It's unclear how many users we have exactly, but we
> certainly ship multiple million copies of BLAS every month, and
> provide one of the most popular numerical toolkits for both novice and
> expert users.
>
> Looking at the original BLAS and LAPACK interfaces, it often seems
> that their imagined user is something like a classic supercomputer
> consumer, who writes code directly in Fortran or C against the BLAS
> API, and where the person writing the code and running the code are
> the same. NumPy/SciPy are coming from a very different perspective:
> our users generally know nothing about the details of the underlying
> BLAS; they just want to describe their problem in some high-level way,
> and the library is responsible for making it happen as efficiently as
> possible, and is often integrated into some larger system (e.g. a
> real-time analytics platform embedded in a web server).
>
> When it comes to our BLAS usage, we mostly use only a small subset of
> the routines. However, as "consumer software" used by a wide variety
> of users with differing degress of technical expertise, we're expected
> to Just Work on a wide variety of systems, and with as many different
> vendor BLAS libraries as possible. On the other hand, the fact that
> we're working with Python means we don't tend to worry about small
> inefficiencies that will be lost in the noise in any case, and are
> willing to sacrifice some performance to get more reliable operation
> across our diverse userbase.
>
> ## Comments on specific aspects of the proposal
>
> ### Data Layout
>
> We are **strongly in favor** of the proposal to support arbitrary
> strided data layouts. Ideally, this would support strides *specified
> in bytes* (allowing for unaligned data layouts), and allow for truly
> arbitrary strides, including *zero or negative* values. However, we
> think it's fine if some of the weirder cases suffer a performance
> penalty.
>
> Rationale: NumPy ? and thus, most of the scientific Python ecosystem ?
> only has one way of representing an array: the `numpy.ndarray` type,
> which is an arbitrary dimensional tensor with arbitrary strides. It is
> common to encounter matrices with non-trivial strides. For example::
>
>     # Make a 3-dimensional tensor, 10 x 9 x 8
>     t = np.zeros((10, 9, 8))
>     # Considering this as a stack of eight 10x9 matrices, extract the
> first:
>     mat = t[:, :, 0]
>
> Now `mat` has non-trivial strides on both axes. (If running this in a
> Python interpreter, you can see this by looking at the value of
> `mat.strides`.) Another case where interesting strides arise is when
> performing ["broadcasting"](https://docs.scipy.org/doc/numpy-1.13.0/
> user/basics.broadcasting.html),
> which is the name for NumPy's rules for stretching arrays to make
> their shapes match. For example, in an expression like::
>
>     np.array([1, 2, 3]) + 1
>
> the scalar `1` is "broadcast" to create a vector `[1, 1, 1]`. This is
> accomplished without allocating memory, by creating a vector with
> settings length = 3, strides = 0 ? so all the elements share a single
> location in memory. Similarly, by using negative strides we can
> reverse an array without allocating memory::
>
>     a = np.array([1, 2, 3])
>     a_flipped = a[::-1]
>
> Now `a_flipped` has the value `[3, 2, 1]`, while sharing storage with
> the array `a = [1, 2, 3]`. Misaligned data is also possible (e.g. an
> array of 8-byte doubles with a 9-byte stride), though it arises more
> rarely. (An example of when it might occurs is in an on-disk data
> format that alternates between storing a double value and then a
> single byte value, which is then memory-mapped.)
>
> While this array representation is very flexible and useful, it makes
> interfacing with BLAS a challenge: how do you perform a GEMM operation
> on two arrays with arbitrary strides? Currently, NumPy attempts to
> detect a number of special cases: if the strides in both arrays imply
> a column-major layout, then call BLAS directly; if one of them has
> strides corresponding to a row-major layout, then set the
> corresponding `transA`/`transB` argument, etc. ? and if all else
> fails, either copy the data into a contiguous buffer, or else fall
> back on a naive triple-nested-loop GEMM implementation. (There's also
> a check where if we can determine through examining the arrays' data
> pointers and strides that they're actually transposes of each other,
> then we instead dispatch to SYRK.)
>
> So for us, native stride support in the BLAS has two major advantages:
>
> - It allows a wider variety of operations to be transparently handled
> using high-speed kernels.
>
> - It reduces the complexity of the NumPy/BLAS interface code.
>
> Any kind of stride support produces both of these advantages to some
> extent. The ideal would be if BLAS supports strides specified in bytes
> (not elements), and allowing truly arbitrary values, because in this
> case, we can simply pass through our arrays directly to the library
> without having to do any fiddly checking at all. Note that for these
> purposes, it's OK if "weird" arrays pay a speed penalty: if the BLAS
> didn't support them, we'd just have to fall back on some even worse
> strategy for handling them ourselves. And this approach would mean
> that if some BLAS vendors do at some point decide to optimize these
> cases, then our users will immediately see the advantages.
>
> ### Error-reporting
>
> We are **strongly in favor** of the draft's proposal to replace XERBLA
> with proper error codes. XERBLA is fine for one-off programs in
> Fortran, but awful for wrapper libraries like NumPy/SciPy.
>
> ### Handling of exceptional values
>
> A recurring issue in low-level linear-algebra libraries is that they
> may crash, freeze, or produce unexpected results when receiving
> non-finite inputs. Of course as a wrapper library, NumPy/SciPy can't
> control what kind of strange data our users might pass in, and we have
> to produce sensible results regardless, so we sometimes accumulate
> hacks like having to validate all incoming data for exceptional values
>
> We support the proposal's recommendation of "NaN interpretation (4)".
>
> We also note that contrary to the note under "Interpretation (1)"; in
> the R statistical environment NaN does *not* represent missing data. R
> maintains a clear distinction between "NA" (missing) values and "NaN"
> (invalid) values. This is important for various statistical procedures
> such as imputation, where you might want to estimate the true value of
> an unmeasured variable, but you don't want to estimate the true value
> of 0/0. For efficiency, they do perform some calculations on NA values
> by storing them as NaNs with a specific payload; in the R context,
> therefore, it's particularly valuable if operations **correctly
> propagate NaN payloads**. NumPy does not yet provide first-class
> missing value support, but we plan to add it within the next 1-2
> years, and when we do it seems likely that we'll have similar needs to
> R.
>
> ### BLAS G2
>
> NumPy has a rich type system, including 16-, 32-, and 64-bit floats
> (plus extended precision on some platforms), a similar set of complex
> values, and is further extensible with new types. We have a similarly
> rich dispatch system for operations that attempts to find the best
> matching optimized-loop for an arbitrary set of input types. We're
> thus quite interested in the proposed mixed-type operations, and if
> implemented we'll probably start transparently taking advantage of
> them (i.e., in cases where we're currently upcasting to perform a
> requested operation, we may switch to using the native precision
> operation without requiring any changes in user code).
>
> Two comments, though:
>
> 1. The various tricks to make names less redundant (e.g., specifying
> just one type if they're all the same) are convenient for those
> calling these routines by hand, but rather inconvenient for those of
> us who will be generating a table mapping different type combinations
> to different functions. When designing a compression scheme for names,
> please remember that the scheme will have to be unambiguously
> implemented in code by multiple projects. Another option to consider:
> ask vendors to implement the full "uncompressed" names, and then
> provide a shim library mapping short "convenience" names to their full
> form.
>
> 2. Regarding the suggestion that the naming scheme will allow for a
> wide variety of differently typed routines, but that only some subset
> will be implemented by any particular vendor: we are worried about
> this subset business. For a library like NumPy that wants to wrap
> "all" the provided routines, while supporting "all" the different BLAS
> libraries ... how will this work? **How do we know which routines any
> particular library provides?** What if new routines are added to the
> list later?
>
> ### Reproducible BLAS
>
> This is interesting, but we don't have any particularly useful comments.
>
> ### Batch BLAS
>
> NumPy/SciPy actually provide batched operations in many cases,
> currently implemented using a `for` loop around individual calls to
> BLAS/LAPACK. However, our interface is relatively restricted compared
> to some of the proposals considered here: generally all we need is the
> ability to perform an operation on two "stacks" of size-homogenous
> matrices, with the offset between matrices determined by an arbitrary
> (potentially zero) stride.
>
> ### Fixed-point BLAS
>
> This is interesting, but we don't have any particularly useful comments.
>
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180109/0e48d2a0/attachment-0001.html>

From gael.varoquaux at normalesup.org  Tue Jan  9 08:34:48 2018
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 9 Jan 2018 14:34:48 +0100
Subject: [SciPy-Dev] General discussion on parallelisation
In-Reply-To: <CAAbtOZd=pGftqOEabw9qkQyV1KszHF-q3x5tE6GKVCa9yoEBPw@mail.gmail.com>
References: <CAAbtOZd=pGftqOEabw9qkQyV1KszHF-q3x5tE6GKVCa9yoEBPw@mail.gmail.com>
Message-ID: <20180109133448.GA832752@phare.normalesup.org>

> 5. joblib.Parallel doesn't have a map method (desirable to allow 3) so a small

joblib has a custom backend framework that can be used for such purpose
(if I understnad you well):
https://pythonhosted.org/joblib/parallel.html#custom-backend-api-experimental

There are currently a Yarn and a dask.distributed backend that are
getting better and better.

> 6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the
> Parallel object is `__call__`ed. This leads to significant overhead. One can
> use the Parallel object with a context manager, which allows reuse of the Pool,
> but I don't think that's do-able in the context of using the
> DifferentialEvolutionSolver (DES) object as an iterator:

This is evolving. However, the reason behind this is that Pool get
corrupted and lead to deadlock. Olivier Grisel and Thomas Moreau are
working on fixing this in the Python standard library (first PR merged
recently)!

One of the vision of joblib is to provide very light mid-layer that can
connect to multiprocessing and threading (though we are considering
switching to concurrent.futures) as well as other backends. Hopefully
this common language makes it easier to do things like embedding dask in
numerical algorithms without a hard dependencies (yes we are working with
the dask team on this).

Ga?l

From tyler.je.reddy at gmail.com  Tue Jan  9 15:53:10 2018
From: tyler.je.reddy at gmail.com (Tyler Reddy)
Date: Tue, 9 Jan 2018 13:53:10 -0700
Subject: [SciPy-Dev] RFC: comments to BLAS committee from numpy/scipy
 devs
In-Reply-To: <CAEBuzr8+RtABKtYVPwzoaUs5GCFd+9RsGq0djy1JE8QGexK+CQ@mail.gmail.com>
References: <CAPJVwBk_fO1K+CFFOAF05Hgpd6YbKfnOfXYpX2q0+gPPsd-kZg@mail.gmail.com>
 <CAEBuzr8+RtABKtYVPwzoaUs5GCFd+9RsGq0djy1JE8QGexK+CQ@mail.gmail.com>
Message-ID: <CAHPuU_YR2evcLxDcjYXYYEA9DL3MtRhn2KxZvFGSMA4DmHO5kw@mail.gmail.com>

One common issue in computational geometry is the need to operate rapidly
on arrays with "heterogeneous shapes."

So, an array that has rows with different numbers of columns -- shape (1,3)
for the first polygon and shape (1, 12) for the second polygon and so on.

This seems like a particularly nasty scenario when the loss of
"homogeneity" in shape precludes traditional vectorization -- I think numpy
effectively converts these to dtype=object, etc. I don't
think is necessarily a BLAS issue since wrapping comp. geo. libraries does
happen in a subset of cases to handle this, but if there's overlap in
utility you could pass it along I suppose.



On 9 January 2018 at 04:40, Ilhan Polat <ilhanpolat at gmail.com> wrote:

> I couldn't find an item to place this but I think ilaenv and also calling
> the function twice (one with lwork=-1 and reading the optimal block size
> and the call the function again properly with lwork=<result>) in LAPACK
> needs to be gotten rid of.
>
> That's a major annoyance during the wrapping of LAPACK routines for SciPy.
>
> I don't know if this is realistic but the values ilaenv needed can be
> computed once (or again if hardware is changed) at the install and can be
> read off by the routines.
>
>
>
> On Jan 9, 2018 09:25, "Nathaniel Smith" <njs at pobox.com> wrote:
>
>> Hi all,
>>
>> As mentioned earlier [1][2], there's work underway to revise and
>> update the BLAS standard -- e.g. we might get support for strided
>> arrays and lose xerbla! There's a draft at [3]. They're interested in
>> feedback from users, so I've written up a first draft of comments
>> about what we would like as NumPy/SciPy developers. This is very much
>> a first attempt -- I know we have lots of people who are more expert
>> on BLAS than me on these lists :-). Please let me know what you think.
>>
>> -n
>>
>> [1] https://mail.python.org/pipermail/numpy-discussion/2017-
>> November/077420.html
>> [2] https://mail.python.org/pipermail/scipy-dev/2017-November/022267.html
>> [3] https://docs.google.com/document/d/1DY4ImZT1coqri2382GusXgBT
>> TTVdBDvtD5I14QHp9OE/edit
>>
>> -----
>>
>> # Comments from NumPy / SciPy developers on "A Proposal for a
>> Next-Generation BLAS"
>>
>> These are comments on [A Proposal for a Next-Generation
>> BLAS](https://docs.google.com/document/d/1DY4ImZT1coqri2382G
>> usXgBTTTVdBDvtD5I14QHp9OE/edit#)
>> (version as of 2017-12-13), from the perspective of the developers of
>> the NumPy and SciPy libraries. We hope this feedback is useful, and
>> welcome further discussion.
>>
>> ## Who are we?
>>
>> NumPy and SciPy are the two foundational libraries of the Python
>> numerical ecosystem, and one of their duties is to wrap BLAS and
>> expose it for the use of other Python libraries. (NumPy primarily
>> provides a GEMM wrapper, while SciPy exposes more specialized
>> operations.) It's unclear how many users we have exactly, but we
>> certainly ship multiple million copies of BLAS every month, and
>> provide one of the most popular numerical toolkits for both novice and
>> expert users.
>>
>> Looking at the original BLAS and LAPACK interfaces, it often seems
>> that their imagined user is something like a classic supercomputer
>> consumer, who writes code directly in Fortran or C against the BLAS
>> API, and where the person writing the code and running the code are
>> the same. NumPy/SciPy are coming from a very different perspective:
>> our users generally know nothing about the details of the underlying
>> BLAS; they just want to describe their problem in some high-level way,
>> and the library is responsible for making it happen as efficiently as
>> possible, and is often integrated into some larger system (e.g. a
>> real-time analytics platform embedded in a web server).
>>
>> When it comes to our BLAS usage, we mostly use only a small subset of
>> the routines. However, as "consumer software" used by a wide variety
>> of users with differing degress of technical expertise, we're expected
>> to Just Work on a wide variety of systems, and with as many different
>> vendor BLAS libraries as possible. On the other hand, the fact that
>> we're working with Python means we don't tend to worry about small
>> inefficiencies that will be lost in the noise in any case, and are
>> willing to sacrifice some performance to get more reliable operation
>> across our diverse userbase.
>>
>> ## Comments on specific aspects of the proposal
>>
>> ### Data Layout
>>
>> We are **strongly in favor** of the proposal to support arbitrary
>> strided data layouts. Ideally, this would support strides *specified
>> in bytes* (allowing for unaligned data layouts), and allow for truly
>> arbitrary strides, including *zero or negative* values. However, we
>> think it's fine if some of the weirder cases suffer a performance
>> penalty.
>>
>> Rationale: NumPy ? and thus, most of the scientific Python ecosystem ?
>> only has one way of representing an array: the `numpy.ndarray` type,
>> which is an arbitrary dimensional tensor with arbitrary strides. It is
>> common to encounter matrices with non-trivial strides. For example::
>>
>>     # Make a 3-dimensional tensor, 10 x 9 x 8
>>     t = np.zeros((10, 9, 8))
>>     # Considering this as a stack of eight 10x9 matrices, extract the
>> first:
>>     mat = t[:, :, 0]
>>
>> Now `mat` has non-trivial strides on both axes. (If running this in a
>> Python interpreter, you can see this by looking at the value of
>> `mat.strides`.) Another case where interesting strides arise is when
>> performing ["broadcasting"](https://docs.scipy.org/doc/numpy-1.13.0/use
>> r/basics.broadcasting.html),
>> which is the name for NumPy's rules for stretching arrays to make
>> their shapes match. For example, in an expression like::
>>
>>     np.array([1, 2, 3]) + 1
>>
>> the scalar `1` is "broadcast" to create a vector `[1, 1, 1]`. This is
>> accomplished without allocating memory, by creating a vector with
>> settings length = 3, strides = 0 ? so all the elements share a single
>> location in memory. Similarly, by using negative strides we can
>> reverse an array without allocating memory::
>>
>>     a = np.array([1, 2, 3])
>>     a_flipped = a[::-1]
>>
>> Now `a_flipped` has the value `[3, 2, 1]`, while sharing storage with
>> the array `a = [1, 2, 3]`. Misaligned data is also possible (e.g. an
>> array of 8-byte doubles with a 9-byte stride), though it arises more
>> rarely. (An example of when it might occurs is in an on-disk data
>> format that alternates between storing a double value and then a
>> single byte value, which is then memory-mapped.)
>>
>> While this array representation is very flexible and useful, it makes
>> interfacing with BLAS a challenge: how do you perform a GEMM operation
>> on two arrays with arbitrary strides? Currently, NumPy attempts to
>> detect a number of special cases: if the strides in both arrays imply
>> a column-major layout, then call BLAS directly; if one of them has
>> strides corresponding to a row-major layout, then set the
>> corresponding `transA`/`transB` argument, etc. ? and if all else
>> fails, either copy the data into a contiguous buffer, or else fall
>> back on a naive triple-nested-loop GEMM implementation. (There's also
>> a check where if we can determine through examining the arrays' data
>> pointers and strides that they're actually transposes of each other,
>> then we instead dispatch to SYRK.)
>>
>> So for us, native stride support in the BLAS has two major advantages:
>>
>> - It allows a wider variety of operations to be transparently handled
>> using high-speed kernels.
>>
>> - It reduces the complexity of the NumPy/BLAS interface code.
>>
>> Any kind of stride support produces both of these advantages to some
>> extent. The ideal would be if BLAS supports strides specified in bytes
>> (not elements), and allowing truly arbitrary values, because in this
>> case, we can simply pass through our arrays directly to the library
>> without having to do any fiddly checking at all. Note that for these
>> purposes, it's OK if "weird" arrays pay a speed penalty: if the BLAS
>> didn't support them, we'd just have to fall back on some even worse
>> strategy for handling them ourselves. And this approach would mean
>> that if some BLAS vendors do at some point decide to optimize these
>> cases, then our users will immediately see the advantages.
>>
>> ### Error-reporting
>>
>> We are **strongly in favor** of the draft's proposal to replace XERBLA
>> with proper error codes. XERBLA is fine for one-off programs in
>> Fortran, but awful for wrapper libraries like NumPy/SciPy.
>>
>> ### Handling of exceptional values
>>
>> A recurring issue in low-level linear-algebra libraries is that they
>> may crash, freeze, or produce unexpected results when receiving
>> non-finite inputs. Of course as a wrapper library, NumPy/SciPy can't
>> control what kind of strange data our users might pass in, and we have
>> to produce sensible results regardless, so we sometimes accumulate
>> hacks like having to validate all incoming data for exceptional values
>>
>> We support the proposal's recommendation of "NaN interpretation (4)".
>>
>> We also note that contrary to the note under "Interpretation (1)"; in
>> the R statistical environment NaN does *not* represent missing data. R
>> maintains a clear distinction between "NA" (missing) values and "NaN"
>> (invalid) values. This is important for various statistical procedures
>> such as imputation, where you might want to estimate the true value of
>> an unmeasured variable, but you don't want to estimate the true value
>> of 0/0. For efficiency, they do perform some calculations on NA values
>> by storing them as NaNs with a specific payload; in the R context,
>> therefore, it's particularly valuable if operations **correctly
>> propagate NaN payloads**. NumPy does not yet provide first-class
>> missing value support, but we plan to add it within the next 1-2
>> years, and when we do it seems likely that we'll have similar needs to
>> R.
>>
>> ### BLAS G2
>>
>> NumPy has a rich type system, including 16-, 32-, and 64-bit floats
>> (plus extended precision on some platforms), a similar set of complex
>> values, and is further extensible with new types. We have a similarly
>> rich dispatch system for operations that attempts to find the best
>> matching optimized-loop for an arbitrary set of input types. We're
>> thus quite interested in the proposed mixed-type operations, and if
>> implemented we'll probably start transparently taking advantage of
>> them (i.e., in cases where we're currently upcasting to perform a
>> requested operation, we may switch to using the native precision
>> operation without requiring any changes in user code).
>>
>> Two comments, though:
>>
>> 1. The various tricks to make names less redundant (e.g., specifying
>> just one type if they're all the same) are convenient for those
>> calling these routines by hand, but rather inconvenient for those of
>> us who will be generating a table mapping different type combinations
>> to different functions. When designing a compression scheme for names,
>> please remember that the scheme will have to be unambiguously
>> implemented in code by multiple projects. Another option to consider:
>> ask vendors to implement the full "uncompressed" names, and then
>> provide a shim library mapping short "convenience" names to their full
>> form.
>>
>> 2. Regarding the suggestion that the naming scheme will allow for a
>> wide variety of differently typed routines, but that only some subset
>> will be implemented by any particular vendor: we are worried about
>> this subset business. For a library like NumPy that wants to wrap
>> "all" the provided routines, while supporting "all" the different BLAS
>> libraries ... how will this work? **How do we know which routines any
>> particular library provides?** What if new routines are added to the
>> list later?
>>
>> ### Reproducible BLAS
>>
>> This is interesting, but we don't have any particularly useful comments.
>>
>> ### Batch BLAS
>>
>> NumPy/SciPy actually provide batched operations in many cases,
>> currently implemented using a `for` loop around individual calls to
>> BLAS/LAPACK. However, our interface is relatively restricted compared
>> to some of the proposals considered here: generally all we need is the
>> ability to perform an operation on two "stacks" of size-homogenous
>> matrices, with the offset between matrices determined by an arbitrary
>> (potentially zero) stride.
>>
>> ### Fixed-point BLAS
>>
>> This is interesting, but we don't have any particularly useful comments.
>>
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180109/4d2d3e09/attachment-0001.html>

From andyfaff at gmail.com  Tue Jan  9 19:01:05 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Wed, 10 Jan 2018 11:01:05 +1100
Subject: [SciPy-Dev] General discussion on parallelisation
In-Reply-To: <CAGu2niX736CpSQ+_zaxZOJVVcxaeh4rvNDu6wTAxWZsfgiG40A@mail.gmail.com>
References: <CAAbtOZd=pGftqOEabw9qkQyV1KszHF-q3x5tE6GKVCa9yoEBPw@mail.gmail.com>
 <CAGu2niX736CpSQ+_zaxZOJVVcxaeh4rvNDu6wTAxWZsfgiG40A@mail.gmail.com>
Message-ID: <CAAbtOZcQY8ej6AgPH3sXr2BAzUdhpeBqyJgDDmBc5Yt=JL6XZQ@mail.gmail.com>

On 9 January 2018 at 13:48, Eric Larson <larson.eric.d at gmail.com> wrote:
>>
>> If you can use `__iter__` instead of `__next__` in your
DifferentialEvolutionSolver class I think you can avoid this problem with
something along the lines of:
>
> class DifferentialEvolutionSolver(object):
>     ...
>     def __iter__(self):
>         with Parallel(...) ...:
>             for it in ...:
>                 yield it
>
> As for the map problem I don't know, but it's probably worth asking the
`joblib` people if there is a suitable solution or workaround.

In the scheme of things I'd prefer to retain the use of `__next__`, see
https://github.com/scipy/scipy/pull/6923 for the direction of where I think
things are headed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180110/755a8a29/attachment.html>

From njs at pobox.com  Tue Jan  9 19:34:04 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 9 Jan 2018 16:34:04 -0800
Subject: [SciPy-Dev] RFC: comments to BLAS committee from numpy/scipy
 devs
In-Reply-To: <CAEBuzr8+RtABKtYVPwzoaUs5GCFd+9RsGq0djy1JE8QGexK+CQ@mail.gmail.com>
References: <CAPJVwBk_fO1K+CFFOAF05Hgpd6YbKfnOfXYpX2q0+gPPsd-kZg@mail.gmail.com>
 <CAEBuzr8+RtABKtYVPwzoaUs5GCFd+9RsGq0djy1JE8QGexK+CQ@mail.gmail.com>
Message-ID: <CAPJVwBk+USGTHzKNxeOfg-fxw9V0sEDyh+Dv_aqwUF6EhFc2xg@mail.gmail.com>

On Tue, Jan 9, 2018 at 3:40 AM, Ilhan Polat <ilhanpolat at gmail.com> wrote:
> I couldn't find an item to place this but I think ilaenv and also calling
> the function twice (one with lwork=-1 and reading the optimal block size and
> the call the function again properly with lwork=<result>) in LAPACK needs to
> be gotten rid of.
>
> That's a major annoyance during the wrapping of LAPACK routines for SciPy.
>
> I don't know if this is realistic but the values ilaenv needed can be
> computed once (or again if hardware is changed) at the install and can be
> read off by the routines.

Unfortunately I think this effort is just to revise BLAS, not LAPACK.
Maybe you should try starting a conversation with the LAPACK
developers though ? I don't know much about how they work but maybe
they'd be interested in feedback.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Tue Jan  9 19:37:18 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 9 Jan 2018 16:37:18 -0800
Subject: [SciPy-Dev] RFC: comments to BLAS committee from numpy/scipy
 devs
In-Reply-To: <CAHPuU_YR2evcLxDcjYXYYEA9DL3MtRhn2KxZvFGSMA4DmHO5kw@mail.gmail.com>
References: <CAPJVwBk_fO1K+CFFOAF05Hgpd6YbKfnOfXYpX2q0+gPPsd-kZg@mail.gmail.com>
 <CAEBuzr8+RtABKtYVPwzoaUs5GCFd+9RsGq0djy1JE8QGexK+CQ@mail.gmail.com>
 <CAHPuU_YR2evcLxDcjYXYYEA9DL3MtRhn2KxZvFGSMA4DmHO5kw@mail.gmail.com>
Message-ID: <CAPJVwBmDnsB5Ge38xRQi8MXB4ZQP_REDBRZ6FEDuX9d+gyOYTA@mail.gmail.com>

On Tue, Jan 9, 2018 at 12:53 PM, Tyler Reddy <tyler.je.reddy at gmail.com> wrote:
> One common issue in computational geometry is the need to operate rapidly on
> arrays with "heterogeneous shapes."
>
> So, an array that has rows with different numbers of columns -- shape (1,3)
> for the first polygon and shape (1, 12) for the second polygon and so on.
>
> This seems like a particularly nasty scenario when the loss of "homogeneity"
> in shape precludes traditional vectorization -- I think numpy effectively
> converts these to dtype=object, etc. I don't
> think is necessarily a BLAS issue since wrapping comp. geo. libraries does
> happen in a subset of cases to handle this, but if there's overlap in
> utility you could pass it along I suppose.

You might be interested in this discussion of "Batch BLAS":
https://docs.google.com/document/d/1DY4ImZT1coqri2382GusXgBTTTVdBDvtD5I14QHp9OE/edit#heading=h.pvsif1mxvaqq

I didn't get into it in the draft response, because it didn't seem
like something where NumPy/SciPy have any useful experience to offer,
but it sounds like there are people worrying about this case.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From andyfaff at gmail.com  Wed Jan 10 00:08:42 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Wed, 10 Jan 2018 16:08:42 +1100
Subject: [SciPy-Dev] master branch failing on travis + appveyor at the moment
Message-ID: <CAAbtOZfQo_C+_BLccAq18NvEXYXi6CWvHSyUJ=iYDpTi=n2W9Q@mail.gmail.com>

https://travis-ci.org/scipy/scipy.svg?branch=master

The doctests look to be the culprit. From what I can tell there are
offending spaces causing the issue. I don't know why it's reared it's head
now, probably some underlying package has received an update.

A.

-- 
_____________________________________
Dr. Andrew Nelson


_____________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180110/427f8f91/attachment.html>

From larson.eric.d at gmail.com  Wed Jan 10 01:06:34 2018
From: larson.eric.d at gmail.com (Eric Larson)
Date: Wed, 10 Jan 2018 01:06:34 -0500
Subject: [SciPy-Dev] master branch failing on travis + appveyor at the
 moment
In-Reply-To: <CAAbtOZfQo_C+_BLccAq18NvEXYXi6CWvHSyUJ=iYDpTi=n2W9Q@mail.gmail.com>
References: <CAAbtOZfQo_C+_BLccAq18NvEXYXi6CWvHSyUJ=iYDpTi=n2W9Q@mail.gmail.com>
Message-ID: <CAGu2niWprK9CMbUe-1YpK3APNWUwrpBFf=kNmTxQZdGRxeG_4w@mail.gmail.com>

NumPy 1.14 changed array printing behavior:

https://github.com/numpy/numpy/blob/master/doc/release/1.14.0-notes.rst#many-changes-to-array-printing-disableable-with-the-new-legacy-printing-mode

Eric


On Wed, Jan 10, 2018 at 12:08 AM, Andrew Nelson <andyfaff at gmail.com> wrote:

> https://travis-ci.org/scipy/scipy.svg?branch=master
>
> The doctests look to be the culprit. From what I can tell there are
> offending spaces causing the issue. I don't know why it's reared it's head
> now, probably some underlying package has received an update.
>
> A.
>
> --
> _____________________________________
> Dr. Andrew Nelson
>
>
> _____________________________________
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180110/a39851ee/attachment-0001.html>

From andyfaff at gmail.com  Wed Jan 10 02:42:30 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Wed, 10 Jan 2018 18:42:30 +1100
Subject: [SciPy-Dev] master branch failing on travis + appveyor at the
 moment
In-Reply-To: <CAGu2niWprK9CMbUe-1YpK3APNWUwrpBFf=kNmTxQZdGRxeG_4w@mail.gmail.com>
References: <CAAbtOZfQo_C+_BLccAq18NvEXYXi6CWvHSyUJ=iYDpTi=n2W9Q@mail.gmail.com>
 <CAGu2niWprK9CMbUe-1YpK3APNWUwrpBFf=kNmTxQZdGRxeG_4w@mail.gmail.com>
Message-ID: <CAAbtOZdQ9mD0T-JBxkd75Xn=rz1JDCp-YKv1dqbucw09Xpb8Tg@mail.gmail.com>

I guess the right thing to do going forward is change all the doctests
then. I'll have a bash at that tonight.

On 10 Jan 2018 5:07 pm, "Eric Larson" <larson.eric.d at gmail.com> wrote:

> NumPy 1.14 changed array printing behavior:
>
> https://github.com/numpy/numpy/blob/master/doc/release/
> 1.14.0-notes.rst#many-changes-to-array-printing-disableable-
> with-the-new-legacy-printing-mode
>
> Eric
>
>
> On Wed, Jan 10, 2018 at 12:08 AM, Andrew Nelson <andyfaff at gmail.com>
> wrote:
>
>> https://travis-ci.org/scipy/scipy.svg?branch=master
>>
>> The doctests look to be the culprit. From what I can tell there are
>> offending spaces causing the issue. I don't know why it's reared it's head
>> now, probably some underlying package has received an update.
>>
>> A.
>>
>> --
>> _____________________________________
>> Dr. Andrew Nelson
>>
>>
>> _____________________________________
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180110/3780c545/attachment.html>

From evgeny.burovskiy at gmail.com  Wed Jan 10 02:58:36 2018
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Wed, 10 Jan 2018 10:58:36 +0300
Subject: [SciPy-Dev] master branch failing on travis + appveyor at the
 moment
In-Reply-To: <CAAbtOZdQ9mD0T-JBxkd75Xn=rz1JDCp-YKv1dqbucw09Xpb8Tg@mail.gmail.com>
References: <CAAbtOZfQo_C+_BLccAq18NvEXYXi6CWvHSyUJ=iYDpTi=n2W9Q@mail.gmail.com>
 <CAGu2niWprK9CMbUe-1YpK3APNWUwrpBFf=kNmTxQZdGRxeG_4w@mail.gmail.com>
 <CAAbtOZdQ9mD0T-JBxkd75Xn=rz1JDCp-YKv1dqbucw09Xpb8Tg@mail.gmail.com>
Message-ID: <CAMRo0it-LEFBvJU8+cANXKX2piC80ZJ7GcofozZPbnv+giD49Q@mail.gmail.com>

On Jan 10, 2018 10:42 AM, "Andrew Nelson" <andyfaff at gmail.com> wrote:

I guess the right thing to do going forward is change all the doctests
then. I'll have a bash at that tonight.


Might want to take a look at why refguide-check fails: it is supposed to
understand the array syntax and compare arrays numerically.

Also this should have been caught a while ago by the CI against the numpy
master branch. There might be something to fix in the CI setup if that
failed to trigger.


On 10 Jan 2018 5:07 pm, "Eric Larson" <larson.eric.d at gmail.com> wrote:

> NumPy 1.14 changed array printing behavior:
>
> https://github.com/numpy/numpy/blob/master/doc/release/1.14.
> 0-notes.rst#many-changes-to-array-printing-disableable-wit
> h-the-new-legacy-printing-mode
>
> Eric
>
>
> On Wed, Jan 10, 2018 at 12:08 AM, Andrew Nelson <andyfaff at gmail.com>
> wrote:
>
>> https://travis-ci.org/scipy/scipy.svg?branch=master
>>
>> The doctests look to be the culprit. From what I can tell there are
>> offending spaces causing the issue. I don't know why it's reared it's head
>> now, probably some underlying package has received an update.
>>
>> A.
>>
>> --
>> _____________________________________
>> Dr. Andrew Nelson
>>
>>
>> _____________________________________
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev at python.org
https://mail.python.org/mailman/listinfo/scipy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180110/a1341ab6/attachment.html>

From ralf.gommers at gmail.com  Wed Jan 10 03:25:51 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Wed, 10 Jan 2018 21:25:51 +1300
Subject: [SciPy-Dev] master branch failing on travis + appveyor at the
 moment
In-Reply-To: <CAMRo0it-LEFBvJU8+cANXKX2piC80ZJ7GcofozZPbnv+giD49Q@mail.gmail.com>
References: <CAAbtOZfQo_C+_BLccAq18NvEXYXi6CWvHSyUJ=iYDpTi=n2W9Q@mail.gmail.com>
 <CAGu2niWprK9CMbUe-1YpK3APNWUwrpBFf=kNmTxQZdGRxeG_4w@mail.gmail.com>
 <CAAbtOZdQ9mD0T-JBxkd75Xn=rz1JDCp-YKv1dqbucw09Xpb8Tg@mail.gmail.com>
 <CAMRo0it-LEFBvJU8+cANXKX2piC80ZJ7GcofozZPbnv+giD49Q@mail.gmail.com>
Message-ID: <CABL7CQg6g2PjUJWdSfyjOz_zbQ5bW3cDjAh46HtM7pmFC=gCyA@mail.gmail.com>

On Wed, Jan 10, 2018 at 8:58 PM, Evgeni Burovski <evgeny.burovskiy at gmail.com
> wrote:

>
>
> On Jan 10, 2018 10:42 AM, "Andrew Nelson" <andyfaff at gmail.com> wrote:
>
> I guess the right thing to do going forward is change all the doctests
> then. I'll have a bash at that tonight.
>
>
> Might want to take a look at why refguide-check fails: it is supposed to
> understand the array syntax and compare arrays numerically.
>

Agreed, shouldn't have failed. Changing the doctests as well may still be
good for doc purposes (so it matches the latest numpy), assuming it's not
too much work.


> Also this should have been caught a while ago by the CI against the numpy
> master branch. There might be something to fix in the CI setup if that
> failed to trigger.
>

The refguide check is only run once, and that's against the last released
numpy. It should only be run once, but we could consider using numpy master
for it.

Ralf


>
> On 10 Jan 2018 5:07 pm, "Eric Larson" <larson.eric.d at gmail.com> wrote:
>
>> NumPy 1.14 changed array printing behavior:
>>
>> https://github.com/numpy/numpy/blob/master/doc/release/1.14.
>> 0-notes.rst#many-changes-to-array-printing-disableable-with-
>> the-new-legacy-printing-mode
>>
>> Eric
>>
>>
>> On Wed, Jan 10, 2018 at 12:08 AM, Andrew Nelson <andyfaff at gmail.com>
>> wrote:
>>
>>> https://travis-ci.org/scipy/scipy.svg?branch=master
>>>
>>> The doctests look to be the culprit. From what I can tell there are
>>> offending spaces causing the issue. I don't know why it's reared it's head
>>> now, probably some underlying package has received an update.
>>>
>>> A.
>>>
>>> --
>>> _____________________________________
>>> Dr. Andrew Nelson
>>>
>>>
>>> _____________________________________
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180110/8f16c2c7/attachment-0001.html>

From andyfaff at gmail.com  Wed Jan 10 03:31:23 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Wed, 10 Jan 2018 19:31:23 +1100
Subject: [SciPy-Dev] master branch failing on travis + appveyor at the
 moment
In-Reply-To: <CABL7CQg6g2PjUJWdSfyjOz_zbQ5bW3cDjAh46HtM7pmFC=gCyA@mail.gmail.com>
References: <CAAbtOZfQo_C+_BLccAq18NvEXYXi6CWvHSyUJ=iYDpTi=n2W9Q@mail.gmail.com>
 <CAGu2niWprK9CMbUe-1YpK3APNWUwrpBFf=kNmTxQZdGRxeG_4w@mail.gmail.com>
 <CAAbtOZdQ9mD0T-JBxkd75Xn=rz1JDCp-YKv1dqbucw09Xpb8Tg@mail.gmail.com>
 <CAMRo0it-LEFBvJU8+cANXKX2piC80ZJ7GcofozZPbnv+giD49Q@mail.gmail.com>
 <CABL7CQg6g2PjUJWdSfyjOz_zbQ5bW3cDjAh46HtM7pmFC=gCyA@mail.gmail.com>
Message-ID: <CAAbtOZc=p-u6zJLajrdA-Y2Lb10vst4Krq6rjy1myFozf2Hk+A@mail.gmail.com>

It's easy for me to change the doctests, but I'm not familiar with
refguide_check.py. I'm going to leave that to someone else.

A.

On 10 January 2018 at 19:25, Ralf Gommers <ralf.gommers at gmail.com> wrote:

>
>
> On Wed, Jan 10, 2018 at 8:58 PM, Evgeni Burovski <
> evgeny.burovskiy at gmail.com> wrote:
>
>>
>>
>> On Jan 10, 2018 10:42 AM, "Andrew Nelson" <andyfaff at gmail.com> wrote:
>>
>> I guess the right thing to do going forward is change all the doctests
>> then. I'll have a bash at that tonight.
>>
>>
>> Might want to take a look at why refguide-check fails: it is supposed to
>> understand the array syntax and compare arrays numerically.
>>
>
> Agreed, shouldn't have failed. Changing the doctests as well may still be
> good for doc purposes (so it matches the latest numpy), assuming it's not
> too much work.
>
>
>> Also this should have been caught a while ago by the CI against the numpy
>> master branch. There might be something to fix in the CI setup if that
>> failed to trigger.
>>
>
> The refguide check is only run once, and that's against the last released
> numpy. It should only be run once, but we could consider using numpy master
> for it.
>
> Ralf
>
>
>>
>> On 10 Jan 2018 5:07 pm, "Eric Larson" <larson.eric.d at gmail.com> wrote:
>>
>>> NumPy 1.14 changed array printing behavior:
>>>
>>> https://github.com/numpy/numpy/blob/master/doc/release/1.14.
>>> 0-notes.rst#many-changes-to-array-printing-disableable-with-
>>> the-new-legacy-printing-mode
>>>
>>> Eric
>>>
>>>
>>> On Wed, Jan 10, 2018 at 12:08 AM, Andrew Nelson <andyfaff at gmail.com>
>>> wrote:
>>>
>>>> https://travis-ci.org/scipy/scipy.svg?branch=master
>>>>
>>>> The doctests look to be the culprit. From what I can tell there are
>>>> offending spaces causing the issue. I don't know why it's reared it's head
>>>> now, probably some underlying package has received an update.
>>>>
>>>> A.
>>>>
>>>> --
>>>> _____________________________________
>>>> Dr. Andrew Nelson
>>>>
>>>>
>>>> _____________________________________
>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>


-- 
_____________________________________
Dr. Andrew Nelson


_____________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180110/92daf258/attachment.html>

From ralf.gommers at gmail.com  Wed Jan 10 04:42:31 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Wed, 10 Jan 2018 22:42:31 +1300
Subject: [SciPy-Dev] GSoC'18 participation?
Message-ID: <CABL7CQjL4x2gWLSO_pMnOA-sGyN8=bKM9EinM18e6kqJQVGRsw@mail.gmail.com>

Hi all,

The GSoC schedule is a bit earlier than normal this year. The PSF is asking
for ideas pages to be up and in decent shape by Jan 19th. So we'll need to
come up with some content quick if we want to participate.

Who is interested in mentoring this year?

I'm happy to do the admin again, but probably won't have time to mentor.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180110/7965adc8/attachment.html>

From jomsdev at gmail.com  Wed Jan 10 07:02:33 2018
From: jomsdev at gmail.com (Jordi Montes)
Date: Wed, 10 Jan 2018 13:02:33 +0100
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <CABL7CQjL4x2gWLSO_pMnOA-sGyN8=bKM9EinM18e6kqJQVGRsw@mail.gmail.com>
References: <CABL7CQjL4x2gWLSO_pMnOA-sGyN8=bKM9EinM18e6kqJQVGRsw@mail.gmail.com>
Message-ID: <CAJfd0CMiX2ygfdSFDbGX-9w462xPFGya5xhUqg+xTByfj_q-JA@mail.gmail.com>

Hi all,

Last year I started implementing some methods for Randomized Numerical
Linear Algebra (RNLA) in scipy.
By now it is only the CountMin Sketch (clarkson_woodruff_transformation)
for reducing the dimensionality of a vector space to an embedded space.

I think that it would be interesting to add to scipy other methods for
subspace embedding (like the Johnson-Lindenstrauss) and build some
algorithms on top of it for things like least squeres or low rank
approximation.

Would some other people be interesting in this?


PS: I have a project called RandNLA <https://github.com/jomsdev/randNLA>
where I implemented some of the methods of RNLA. The idea is to implement
only the most important methods of RNLA in scipy and have this other
library for experimenting with new methods and APIs. That will let us not
overloading scipy with features if people are not interested in them and
focus on the ones that really brings value to the community.

Thanks,

Jordi.

On 10 January 2018 at 10:42, Ralf Gommers <ralf.gommers at gmail.com> wrote:

> Hi all,
>
> The GSoC schedule is a bit earlier than normal this year. The PSF is
> asking for ideas pages to be up and in decent shape by Jan 19th. So we'll
> need to come up with some content quick if we want to participate.
>
> Who is interested in mentoring this year?
>
> I'm happy to do the admin again, but probably won't have time to mentor.
>
> Cheers,
> Ralf
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180110/ca67fcf8/attachment-0001.html>

From ralf.gommers at gmail.com  Wed Jan 10 13:24:01 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Thu, 11 Jan 2018 07:24:01 +1300
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <CAJfd0CMiX2ygfdSFDbGX-9w462xPFGya5xhUqg+xTByfj_q-JA@mail.gmail.com>
References: <CABL7CQjL4x2gWLSO_pMnOA-sGyN8=bKM9EinM18e6kqJQVGRsw@mail.gmail.com>
 <CAJfd0CMiX2ygfdSFDbGX-9w462xPFGya5xhUqg+xTByfj_q-JA@mail.gmail.com>
Message-ID: <CABL7CQj=O+njBVH37gLknFbVNvoDdgEwdBxVYdrvhayaei=UoQ@mail.gmail.com>

On Thu, Jan 11, 2018 at 1:02 AM, Jordi Montes <jomsdev at gmail.com> wrote:

> Hi all,
>
> Last year I started implementing some methods for Randomized Numerical
> Linear Algebra (RNLA) in scipy.
> By now it is only the CountMin Sketch (clarkson_woodruff_transformation)
> for reducing the dimensionality of a vector space to an embedded space.
>
> I think that it would be interesting to add to scipy other methods for
> subspace embedding (like the Johnson-Lindenstrauss) and build some
> algorithms on top of it for things like least squeres or low rank
> approximation.
>
> Would some other people be interesting in this?
>
>
> PS: I have a project called RandNLA <https://github.com/jomsdev/randNLA>
> where I implemented some of the methods of RNLA. The idea is to implement
> only the most important methods of RNLA in scipy and have this other
> library for experimenting with new methods and APIs. That will let us not
> overloading scipy with features if people are not interested in them and
> focus on the ones that really brings value to the community.
>

That sounds like a good approach. Worth a discussion in a separate thread
or as a draft enhancement proposal / GSoC idea to detail out which would be
those most important methods.

Would you be interested in mentoring then?

Ralf


> Thanks,
>
> Jordi.
>
> On 10 January 2018 at 10:42, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>> Hi all,
>>
>> The GSoC schedule is a bit earlier than normal this year. The PSF is
>> asking for ideas pages to be up and in decent shape by Jan 19th. So we'll
>> need to come up with some content quick if we want to participate.
>>
>> Who is interested in mentoring this year?
>>
>> I'm happy to do the admin again, but probably won't have time to mentor.
>>
>> Cheers,
>> Ralf
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180111/5ab0a95c/attachment.html>

From jomsdev at gmail.com  Wed Jan 10 14:16:20 2018
From: jomsdev at gmail.com (Jordi Montes)
Date: Wed, 10 Jan 2018 20:16:20 +0100
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <CABL7CQj=O+njBVH37gLknFbVNvoDdgEwdBxVYdrvhayaei=UoQ@mail.gmail.com>
References: <CABL7CQjL4x2gWLSO_pMnOA-sGyN8=bKM9EinM18e6kqJQVGRsw@mail.gmail.com>
 <CAJfd0CMiX2ygfdSFDbGX-9w462xPFGya5xhUqg+xTByfj_q-JA@mail.gmail.com>
 <CABL7CQj=O+njBVH37gLknFbVNvoDdgEwdBxVYdrvhayaei=UoQ@mail.gmail.com>
Message-ID: <CAJfd0CPiakNHnXtGUwtzs2c-Hzrj9OaSthYG8Ne0DzrBqeFu9A@mail.gmail.com>

Hello Ralf,

I have been working on these methods before. I do not have any problem
writing down a draft developing the idea and starting a discussion about
the scope after.

However, I had in mind to be the student and not the mentor. This is my
last year as Master student so it is my last chance for participating in a
GSoC.  I am sure that we can find good mentors for this initiative who are
more expert in the field than myself.

Jordi.

On 10 January 2018 at 19:24, Ralf Gommers <ralf.gommers at gmail.com> wrote:

>
>
> On Thu, Jan 11, 2018 at 1:02 AM, Jordi Montes <jomsdev at gmail.com> wrote:
>
>> Hi all,
>>
>> Last year I started implementing some methods for Randomized Numerical
>> Linear Algebra (RNLA) in scipy.
>> By now it is only the CountMin Sketch (clarkson_woodruff_transformation)
>> for reducing the dimensionality of a vector space to an embedded space.
>>
>> I think that it would be interesting to add to scipy other methods for
>> subspace embedding (like the Johnson-Lindenstrauss) and build some
>> algorithms on top of it for things like least squeres or low rank
>> approximation.
>>
>> Would some other people be interesting in this?
>>
>>
>> PS: I have a project called RandNLA <https://github.com/jomsdev/randNLA>
>> where I implemented some of the methods of RNLA. The idea is to implement
>> only the most important methods of RNLA in scipy and have this other
>> library for experimenting with new methods and APIs. That will let us not
>> overloading scipy with features if people are not interested in them and
>> focus on the ones that really brings value to the community.
>>
>
> That sounds like a good approach. Worth a discussion in a separate thread
> or as a draft enhancement proposal / GSoC idea to detail out which would be
> those most important methods.
>
> Would you be interested in mentoring then?
>
> Ralf
>
>
>> Thanks,
>>
>> Jordi.
>>
>> On 10 January 2018 at 10:42, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> The GSoC schedule is a bit earlier than normal this year. The PSF is
>>> asking for ideas pages to be up and in decent shape by Jan 19th. So we'll
>>> need to come up with some content quick if we want to participate.
>>>
>>> Who is interested in mentoring this year?
>>>
>>> I'm happy to do the admin again, but probably won't have time to mentor.
>>>
>>> Cheers,
>>> Ralf
>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180110/18b5b30d/attachment.html>

From ralf.gommers at gmail.com  Wed Jan 10 14:34:11 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Thu, 11 Jan 2018 08:34:11 +1300
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <CAJfd0CPiakNHnXtGUwtzs2c-Hzrj9OaSthYG8Ne0DzrBqeFu9A@mail.gmail.com>
References: <CABL7CQjL4x2gWLSO_pMnOA-sGyN8=bKM9EinM18e6kqJQVGRsw@mail.gmail.com>
 <CAJfd0CMiX2ygfdSFDbGX-9w462xPFGya5xhUqg+xTByfj_q-JA@mail.gmail.com>
 <CABL7CQj=O+njBVH37gLknFbVNvoDdgEwdBxVYdrvhayaei=UoQ@mail.gmail.com>
 <CAJfd0CPiakNHnXtGUwtzs2c-Hzrj9OaSthYG8Ne0DzrBqeFu9A@mail.gmail.com>
Message-ID: <CABL7CQgZtEXk7kxk9VXFORRv_r7KWha3iZqhPsp+H_HxK=v6KA@mail.gmail.com>

On Thu, Jan 11, 2018 at 8:16 AM, Jordi Montes <jomsdev at gmail.com> wrote:

> Hello Ralf,
>
> I have been working on these methods before. I do not have any problem
> writing down a draft developing the idea and starting a discussion about
> the scope after.
>
> However, I had in mind to be the student and not the mentor.
>

Okay, that's great!

This is my last year as Master student so it is my last chance for
> participating in a GSoC.  I am sure that we can find good mentors for this
> initiative who are more expert in the field than myself.
>

If we can pair one domain expert with one of the core developers, that
would be a good mentoring config.

Ralf



>
> Jordi.
>
> On 10 January 2018 at 19:24, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>>
>>
>> On Thu, Jan 11, 2018 at 1:02 AM, Jordi Montes <jomsdev at gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> Last year I started implementing some methods for Randomized Numerical
>>> Linear Algebra (RNLA) in scipy.
>>> By now it is only the CountMin Sketch (clarkson_woodruff_transformation)
>>> for reducing the dimensionality of a vector space to an embedded space.
>>>
>>> I think that it would be interesting to add to scipy other methods for
>>> subspace embedding (like the Johnson-Lindenstrauss) and build some
>>> algorithms on top of it for things like least squeres or low rank
>>> approximation.
>>>
>>> Would some other people be interesting in this?
>>>
>>>
>>> PS: I have a project called RandNLA <https://github.com/jomsdev/randNLA>
>>> where I implemented some of the methods of RNLA. The idea is to implement
>>> only the most important methods of RNLA in scipy and have this other
>>> library for experimenting with new methods and APIs. That will let us not
>>> overloading scipy with features if people are not interested in them and
>>> focus on the ones that really brings value to the community.
>>>
>>
>> That sounds like a good approach. Worth a discussion in a separate thread
>> or as a draft enhancement proposal / GSoC idea to detail out which would be
>> those most important methods.
>>
>> Would you be interested in mentoring then?
>>
>> Ralf
>>
>>
>>> Thanks,
>>>
>>> Jordi.
>>>
>>> On 10 January 2018 at 10:42, Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> The GSoC schedule is a bit earlier than normal this year. The PSF is
>>>> asking for ideas pages to be up and in decent shape by Jan 19th. So we'll
>>>> need to come up with some content quick if we want to participate.
>>>>
>>>> Who is interested in mentoring this year?
>>>>
>>>> I'm happy to do the admin again, but probably won't have time to mentor.
>>>>
>>>> Cheers,
>>>> Ralf
>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180111/fd95759b/attachment-0001.html>

From ralf.gommers at gmail.com  Sat Jan 13 03:30:12 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 13 Jan 2018 21:30:12 +1300
Subject: [SciPy-Dev] Restructuring flapack.src.pyf file
In-Reply-To: <CAEBuzr-hQKR=atVGzww7_n4ZzL-6m-igpjzNjo+i5EXPYZ4h8w@mail.gmail.com>
References: <CAEBuzr9RKzGzjvY77gM5iKkhK=hifio+UKMBPyzUvwDf9xYoxA@mail.gmail.com>
 <3sm3kp.p1jh54.rthztg-qmf@smtp.gmail.com>
 <CAEBuzr-hQKR=atVGzww7_n4ZzL-6m-igpjzNjo+i5EXPYZ4h8w@mail.gmail.com>
Message-ID: <CABL7CQikKiuug=DWY_7WXTMeWOTyicU2DtKKJaubUQf-5pmMyQ@mail.gmail.com>

On Wed, Dec 27, 2017 at 2:40 AM, Ilhan Polat <ilhanpolat at gmail.com> wrote:

> The situation, as usual, turns out a bit more involved
>
> In the discussion, https://github.com/scipy/scipy/issues/6502 Tiziano
> writes
>
> "Hi, I am the author of the original wrappers for those LAPACK routines.
> If I remember correctly, the reason for hiding the eigenvalues in an
> interval option was simply that the original implementation of
> scipy.linalg.eigh did not even allow to limit the number of
> eigenvalues/eigenvectors returned. I needed only the possiblity to define
> the indices of the eigenvalues and not the range, and the result is the
> option eigvals to scipy.linalg.eigh. I think that your wrapper is too
> generic. You should not allow setting from Python the values of things like
> lwork: a wrong value there and the user can generate a segfault from
> Python, which is a no-no."
>
> and I can see more LAPACK routines that are wrapped along this line.
> Having 8 more "xxxx_full" and "xxxx_full_lwork" routines doesn't seem to be
> elegant though I don't have any alternative yet. Maybe it is OK to have
> them for a few versions but I guess it would be nice if we have a parallel
> running deprecation plans for this issue. It looks like _np.deprecate works
> pretty OK. And we have four deprecations already in lapack.py e.g.,
>
> cgegv = _np.deprecate(cgegv, old_name='cgegv', message=_dep_message)
>
> This will print a deprecation warning when it is called or docstring is
> printed. "DeprecationWarning: `dgegv` is deprecated! The `*gegv` family of
> routines has been deprecated in LAPACK 3.6.0 in favor of the `*ggev` family
> of routines.The corresponding wrappers will be removed from SciPy in a
> future release." So I would like to deprecate the routines abouve with a
> message
>
> "The *XXXXX family of routines will have a different signature in the
> future versions of  SciPy. The new versions are now available as
> *XXXXX_full and will eventually replace the original *XXXXX family"
>
> Would you dis/agree?
>

Just wondering about the tradeoff between deprecating + having a new
function with arguments in the same order as LAPACK, vs. just appending the
keywords to the existing function. The latter case would need a thin Python
function in lapack.py, but unless that hurts performance significantly I'm
not sure that's much of an issue.

Ralf




>
>
>
> On Mon, Dec 25, 2017 at 11:58 PM, <pav at iki.fi> wrote:
>
>> Hi,
>>
>> Ilhan Polat kirjoitti tiistai 26. joulukuuta 2017:
>> [clip]
>> > I've done a similar thing by freeing out all ins and outs. It seems to
>> me
>> > that the signature change is inevitable but the question is how to
>> handle
>> > this?
>>
>> New function sounds best, maximally unimaginative new name is probably
>> most appropriate.
>>
>> Not sure how well np.deprecate etc python level wrappers work on f2py.
>>
>>
>> > 2) It looks like flapack.src.pyf file is getting exceedingly long and
>> > unstructured. Should we break it apart just like LAPACK documentation?
>>
>> If you think that helps it significantly and is worthwhile use of your
>> time then why not. (I suspect it's not particularly worthwhile but ymmv.)
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180113/f9e256c1/attachment.html>

From ilhanpolat at gmail.com  Sat Jan 13 05:16:30 2018
From: ilhanpolat at gmail.com (Ilhan Polat)
Date: Sat, 13 Jan 2018 11:16:30 +0100
Subject: [SciPy-Dev] Restructuring flapack.src.pyf file
In-Reply-To: <CABL7CQikKiuug=DWY_7WXTMeWOTyicU2DtKKJaubUQf-5pmMyQ@mail.gmail.com>
References: <CAEBuzr9RKzGzjvY77gM5iKkhK=hifio+UKMBPyzUvwDf9xYoxA@mail.gmail.com>
 <3sm3kp.p1jh54.rthztg-qmf@smtp.gmail.com>
 <CAEBuzr-hQKR=atVGzww7_n4ZzL-6m-igpjzNjo+i5EXPYZ4h8w@mail.gmail.com>
 <CABL7CQikKiuug=DWY_7WXTMeWOTyicU2DtKKJaubUQf-5pmMyQ@mail.gmail.com>
Message-ID: <CAEBuzr8+yCOtkEuvos3hLKUWa8RKd=S80ZQcgQeinpXqq_K1jA@mail.gmail.com>

Most of them would indeed just gain new keywords xxgvd, xxgv, and so on.
These would only get lwork and liwork keywords. No deprecations needed.

?(sy/he)evr on the other hand not only gets these extra keywords but also
would gain two more output arguments which is actually the reason for this
thread. So fortunately the amount of deprecation is "a few"~
ssyevr/dsyevr/cheevr/zheevr so far.

These are used in the regular case of scipy.linalg.eigh and was causing
problems due to unoptimized lwork values. Also they were not completely
exposed.hence the new ouput args.

I would like to start a separate discussion about the deprecation of
keywords "eigvals" and "turbo" via a decorator later once this is done.

 Because now we are able to give "eig_range" as an input if need be as a
new functionality. And "eigvals" does not correspond to giving an index
range say, "eig_index". But that will have to wait until this surgery on
the LAPACK side.



On Jan 13, 2018 09:31, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:

>
>
> On Wed, Dec 27, 2017 at 2:40 AM, Ilhan Polat <ilhanpolat at gmail.com> wrote:
>
>> The situation, as usual, turns out a bit more involved
>>
>> In the discussion, https://github.com/scipy/scipy/issues/6502 Tiziano
>> writes
>>
>> "Hi, I am the author of the original wrappers for those LAPACK routines.
>> If I remember correctly, the reason for hiding the eigenvalues in an
>> interval option was simply that the original implementation of
>> scipy.linalg.eigh did not even allow to limit the number of
>> eigenvalues/eigenvectors returned. I needed only the possiblity to define
>> the indices of the eigenvalues and not the range, and the result is the
>> option eigvals to scipy.linalg.eigh. I think that your wrapper is too
>> generic. You should not allow setting from Python the values of things like
>> lwork: a wrong value there and the user can generate a segfault from
>> Python, which is a no-no."
>>
>> and I can see more LAPACK routines that are wrapped along this line.
>> Having 8 more "xxxx_full" and "xxxx_full_lwork" routines doesn't seem to be
>> elegant though I don't have any alternative yet. Maybe it is OK to have
>> them for a few versions but I guess it would be nice if we have a parallel
>> running deprecation plans for this issue. It looks like _np.deprecate works
>> pretty OK. And we have four deprecations already in lapack.py e.g.,
>>
>> cgegv = _np.deprecate(cgegv, old_name='cgegv', message=_dep_message)
>>
>> This will print a deprecation warning when it is called or docstring is
>> printed. "DeprecationWarning: `dgegv` is deprecated! The `*gegv` family of
>> routines has been deprecated in LAPACK 3.6.0 in favor of the `*ggev` family
>> of routines.The corresponding wrappers will be removed from SciPy in a
>> future release." So I would like to deprecate the routines abouve with a
>> message
>>
>> "The *XXXXX family of routines will have a different signature in the
>> future versions of  SciPy. The new versions are now available as
>> *XXXXX_full and will eventually replace the original *XXXXX family"
>>
>> Would you dis/agree?
>>
>
> Just wondering about the tradeoff between deprecating + having a new
> function with arguments in the same order as LAPACK, vs. just appending the
> keywords to the existing function. The latter case would need a thin Python
> function in lapack.py, but unless that hurts performance significantly I'm
> not sure that's much of an issue.
>
> Ralf
>
>
>
>
>>
>>
>>
>> On Mon, Dec 25, 2017 at 11:58 PM, <pav at iki.fi> wrote:
>>
>>> Hi,
>>>
>>> Ilhan Polat kirjoitti tiistai 26. joulukuuta 2017:
>>> [clip]
>>> > I've done a similar thing by freeing out all ins and outs. It seems to
>>> me
>>> > that the signature change is inevitable but the question is how to
>>> handle
>>> > this?
>>>
>>> New function sounds best, maximally unimaginative new name is probably
>>> most appropriate.
>>>
>>> Not sure how well np.deprecate etc python level wrappers work on f2py.
>>>
>>>
>>> > 2) It looks like flapack.src.pyf file is getting exceedingly long and
>>> > unstructured. Should we break it apart just like LAPACK documentation?
>>>
>>> If you think that helps it significantly and is worthwhile use of your
>>> time then why not. (I suspect it's not particularly worthwhile but ymmv.)
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180113/690cbf14/attachment-0001.html>

From ralf.gommers at gmail.com  Fri Jan 19 04:15:53 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 19 Jan 2018 22:15:53 +1300
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <CAJfd0CMiX2ygfdSFDbGX-9w462xPFGya5xhUqg+xTByfj_q-JA@mail.gmail.com>
References: <CABL7CQjL4x2gWLSO_pMnOA-sGyN8=bKM9EinM18e6kqJQVGRsw@mail.gmail.com>
 <CAJfd0CMiX2ygfdSFDbGX-9w462xPFGya5xhUqg+xTByfj_q-JA@mail.gmail.com>
Message-ID: <CABL7CQifzz9M+yp_sF_+Xc_TU4pbw_5g7cqMU7h1FLkUzOKApQ@mail.gmail.com>

Hi all,

I have created https://github.com/scipy/scipy/wiki/GSoC-2018-project-ideas
and added Jordi's idea to it. If anyone would be interested in co-mentoring
this topic, please add your name. Also if you have other ideas and are
interested to mentor those if a good student comes along, please add!

I'll send the link through to the PSF organisers so we get on the list.
Clearly we're having a bit of an issue with bandwidth of the usual suspects
for mentoring (and PR reviewing) right now, so while we will keep the door
open to participate we're going to have to be quite selective with students.

Cheers,
Ralf



On Thu, Jan 11, 2018 at 1:02 AM, Jordi Montes <jomsdev at gmail.com> wrote:

> Hi all,
>
> Last year I started implementing some methods for Randomized Numerical
> Linear Algebra (RNLA) in scipy.
> By now it is only the CountMin Sketch (clarkson_woodruff_transformation)
> for reducing the dimensionality of a vector space to an embedded space.
>
> I think that it would be interesting to add to scipy other methods for
> subspace embedding (like the Johnson-Lindenstrauss) and build some
> algorithms on top of it for things like least squeres or low rank
> approximation.
>
> Would some other people be interesting in this?
>
>
> PS: I have a project called RandNLA <https://github.com/jomsdev/randNLA>
> where I implemented some of the methods of RNLA. The idea is to implement
> only the most important methods of RNLA in scipy and have this other
> library for experimenting with new methods and APIs. That will let us not
> overloading scipy with features if people are not interested in them and
> focus on the ones that really brings value to the community.
>
> Thanks,
>
> Jordi.
>
> On 10 January 2018 at 10:42, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>> Hi all,
>>
>> The GSoC schedule is a bit earlier than normal this year. The PSF is
>> asking for ideas pages to be up and in decent shape by Jan 19th. So we'll
>> need to come up with some content quick if we want to participate.
>>
>> Who is interested in mentoring this year?
>>
>> I'm happy to do the admin again, but probably won't have time to mentor.
>>
>> Cheers,
>> Ralf
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180119/1b95e145/attachment.html>

From nikolay.mayorov at zoho.com  Fri Jan 19 10:11:31 2018
From: nikolay.mayorov at zoho.com (Nikolay Mayorov)
Date: Fri, 19 Jan 2018 20:11:31 +0500
Subject: [SciPy-Dev] GSoC'18 participation?
Message-ID: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>



Hi!

I have this idea, which I'm well familiar with. The module would be called like scipy.spatial.rotation and be devoted to the rotation formalism in 3 dimensions. 

The main objects are Euler angles (and their variations), direction cosine matrices, quaternions and rotation vectors. We can go with an abstraction class Rotation (using DCMs or Qs internally), but we should be able to create that from any representation and see it in any representation. In spirit of scipy/numpy we use vectorized/bulk approaches (i.e. many rotations in single Rotation class).

Rotation should support 2 operations: compose 2 consecutive rotations and rotate/project a 3d vector.??Of course all procedures must be 100% robust and there are some fine points, especially in conversions between representations.
Also we can add some algorithms, like: quaternion interpolation (SLERP), least-squares vector matching by a rotation (Whabba's problem), more advanced and less known algotithms for rotation interpolation, and I will try to come up with something more.

Overall it seems reasonably straightforward , but with enough challenges in design and implementation

As currently described, it might be not enough volume for the GSoC, but we can develop it farther.?

Also I'm not sure if its applicability is broad enough to include it into scipy. I believe similar functionality is available in Aerospace toolbox in Matlab. I want to hear some opinions on that.

Nikolay


---- On Wed, 10 Jan 2018 17:02:33 +0500 jomsdev at gmail.com wrote ----

Hi all,

Last year I started implementing some methods for Randomized Numerical Linear Algebra (RNLA) in scipy. 
By now it is only the CountMin Sketch (clarkson_woodruff_transformation) for reducing the dimensionality of a vector space to an embedded space. 

I think that it would be interesting to add to scipy other methods for subspace embedding (like the Johnson-Lindenstrauss) and build some algorithms on top of it for things like least squeres or low rank approximation.

Would some other people be interesting in this?


PS: I have a project called RandNLA where I implemented some of the methods of RNLA. The idea is to implement only the most important methods of RNLA in scipy and have this other library for experimenting with new methods and APIs. That will let us not overloading scipy with features if people are not interested in them and focus on the ones that really brings value to the community.

Thanks,

Jordi.

On 10 January 2018 at 10:42, Ralf Gommers <ralf.gommers at gmail.com> wrote:
Hi all,

The GSoC schedule is a bit earlier than normal this year. The PSF is asking for ideas pages to be up and in decent shape by Jan 19th. So we'll need to come up with some content quick if we want to participate.

Who is interested in mentoring this year?

I'm happy to do the admin again, but probably won't have time to mentor.

Cheers,
Ralf


_______________________________________________
SciPy-Dev mailing list
SciPy-Dev at python.org
https://mail.python.org/mailman/listinfo/scipy-dev


_______________________________________________ 
SciPy-Dev mailing list 
SciPy-Dev at python.org 
https://mail.python.org/mailman/listinfo/scipy-dev 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180119/8007b386/attachment.html>

From larson.eric.d at gmail.com  Fri Jan 19 10:25:17 2018
From: larson.eric.d at gmail.com (Eric Larson)
Date: Fri, 19 Jan 2018 10:25:17 -0500
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>
References: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>
Message-ID: <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>

I have personally run into the need for such transformations in two
separate domains (3D visualization, neuroscience / electrophysiology) and I
know it's used in multiple other places, too. So I think it would be
sufficiently general. I'd look forward to having it in SciPy!

I'd be happy to be a secondary mentor on this if you (or someone else)
wants to be primary.

Best,
Eric


On Fri, Jan 19, 2018 at 10:11 AM, Nikolay Mayorov <nikolay.mayorov at zoho.com>
wrote:

> Hi!
>
> I have this idea, which I'm well familiar with. The module would be called
> like scipy.spatial.rotation and be devoted to the rotation formalism in 3
> dimensions.
>
> The main objects are Euler angles (and their variations), direction cosine
> matrices, quaternions and rotation vectors. We can go with an abstraction
> class Rotation (using DCMs or Qs internally), but we should be able to
> create that from any representation and see it in any representation. In
> spirit of scipy/numpy we use vectorized/bulk approaches (i.e. many
> rotations in single Rotation class).
>
> Rotation should support 2 operations: compose 2 consecutive rotations and
> rotate/project a 3d vector.  Of course all procedures must be 100% robust
> and there are some fine points, especially in conversions between
> representations.
> Also we can add some algorithms, like: quaternion interpolation (SLERP),
> least-squares vector matching by a rotation (Whabba's problem), more
> advanced and less known algotithms for rotation interpolation, and I will
> try to come up with something more.
>
> Overall it seems reasonably straightforward , but with enough challenges
> in design and implementation
>
> As currently described, it might be not enough volume for the GSoC, but we
> can develop it farther.
>
> Also I'm not sure if its applicability is broad enough to include it into
> scipy. I believe similar functionality is available in Aerospace toolbox in
> Matlab. I want to hear some opinions on that.
>
> Nikolay
>
>
> ---- On Wed, 10 Jan 2018 17:02:33 +0500 * jomsdev at gmail.com
> <jomsdev at gmail.com> * wrote ----
>
> Hi all,
>
> Last year I started implementing some methods for Randomized Numerical
> Linear Algebra (RNLA) in scipy.
> By now it is only the CountMin Sketch (clarkson_woodruff_transformation)
> for reducing the dimensionality of a vector space to an embedded space.
>
> I think that it would be interesting to add to scipy other methods for
> subspace embedding (like the Johnson-Lindenstrauss) and build some
> algorithms on top of it for things like least squeres or low rank
> approximation.
>
> Would some other people be interesting in this?
>
>
> PS: I have a project called RandNLA <https://github.com/jomsdev/randNLA>
> where I implemented some of the methods of RNLA. The idea is to implement
> only the most important methods of RNLA in scipy and have this other
> library for experimenting with new methods and APIs. That will let us not
> overloading scipy with features if people are not interested in them and
> focus on the ones that really brings value to the community.
>
> Thanks,
>
> Jordi.
>
> On 10 January 2018 at 10:42, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
> Hi all,
>
> The GSoC schedule is a bit earlier than normal this year. The PSF is
> asking for ideas pages to be up and in decent shape by Jan 19th. So we'll
> need to come up with some content quick if we want to participate.
>
> Who is interested in mentoring this year?
>
> I'm happy to do the admin again, but probably won't have time to mentor.
>
> Cheers,
> Ralf
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180119/b0aaeaa5/attachment-0001.html>

From nikolay.mayorov at zoho.com  Fri Jan 19 12:00:46 2018
From: nikolay.mayorov at zoho.com (Nikolay Mayorov)
Date: Fri, 19 Jan 2018 22:00:46 +0500
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>
References: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>
 <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>
Message-ID: <1610f5d92cc.116a5cf7615287.4034716040473351782@zoho.com>

Hi, Eric!



Thank you for the feedback. You mentioned 3D visualization in which, as far as I know, 4D homogeneous coordinates are commonly used. But we shouldn't go in this direction and consider 3D vectors and rotations only, right?



Great that you are interested in mentoring this project as well. I will add the idea in a basic form to the wiki. Generally I think we should leave some space for a student to investigate, instead of specifying absolutely everything. Do you agree?



Best,

Nikolay 





---- On Fri, 19 Jan 2018 20:25:17 +0500 Eric Larson &lt;larson.eric.d at gmail.com&gt; wrote ----




I have personally run into the need for such transformations in two separate domains (3D visualization, neuroscience / electrophysiology) and I know it's used in multiple other places, too. So I think it would be sufficiently general. I'd look forward to having it in SciPy!



I'd be happy to be a secondary mentor on this if you (or someone else) wants to be primary.



Best,

Eric







On Fri, Jan 19, 2018 at 10:11 AM, Nikolay Mayorov &lt;nikolay.mayorov at zoho.com&gt; wrote:







_______________________________________________

SciPy-Dev mailing list 

SciPy-Dev at python.org 

https://mail.python.org/mailman/listinfo/scipy-dev 




Hi!



I have this idea, which I'm well familiar with. The module would be called like scipy.spatial.rotation and be devoted to the rotation formalism in 3 dimensions. 



The main objects are Euler angles (and their variations), direction cosine matrices, quaternions and rotation vectors. We can go with an abstraction class Rotation (using DCMs or Qs internally), but we should be able to create that from any representation and see it in any representation. In spirit of scipy/numpy we use vectorized/bulk approaches (i.e. many rotations in single Rotation class).



Rotation should support 2 operations: compose 2 consecutive rotations and rotate/project a 3d vector.  Of course all procedures must be 100% robust and there are some fine points, especially in conversions between representations.



Also we can add some algorithms, like: quaternion interpolation (SLERP), least-squares vector matching by a rotation (Whabba's problem), more advanced and less known algotithms for rotation interpolation, and I will try to come up with something more.



Overall it seems reasonably straightforward , but with enough challenges in design and implementation



As currently described, it might be not enough volume for the GSoC, but we can develop it farther. 



Also I'm not sure if its applicability is broad enough to include it into scipy. I believe similar functionality is available in Aerospace toolbox in Matlab. I want to hear some opinions on that.



Nikolay





---- On Wed, 10 Jan 2018 17:02:33 +0500 jomsdev at gmail.com  wrote ----





Hi all,


Last year I started implementing some methods for Randomized Numerical Linear Algebra (RNLA) in scipy. 


By now it is only the CountMin Sketch (clarkson_woodruff_transformation) for reducing the dimensionality of a vector space to an embedded space. 





I think that it would be interesting to add to scipy other methods for subspace embedding (like the Johnson-Lindenstrauss) and build some algorithms on top of it for things like least squeres or low rank approximation.



Would some other people be interesting in this?





PS: I have a project called RandNLA where I implemented some of the methods of RNLA. The idea is to implement only the most important methods of RNLA in scipy and have this other library for experimenting with new methods and APIs. That will let us not overloading scipy with features if people are not interested in them and focus on the ones that really brings value to the community.



Thanks,



Jordi.




On 10 January 2018 at 10:42, Ralf Gommers &lt;ralf.gommers at gmail.com&gt; wrote:

Hi all,


The GSoC schedule is a bit earlier than normal this year. The PSF is asking for ideas pages to be up and in decent shape by Jan 19th. So we'll need to come up with some content quick if we want to participate.




Who is interested in mentoring this year?




I'm happy to do the admin again, but probably won't have time to mentor.




Cheers,


Ralf






_______________________________________________

 SciPy-Dev mailing list

 SciPy-Dev at python.org

 https://mail.python.org/mailman/listinfo/scipy-dev

 





_______________________________________________

SciPy-Dev mailing list 

SciPy-Dev at python.org 

https://mail.python.org/mailman/listinfo/scipy-dev 



















_______________________________________________

 SciPy-Dev mailing list

 SciPy-Dev at python.org

 https://mail.python.org/mailman/listinfo/scipy-dev

 





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180119/c43f9287/attachment.html>

From larson.eric.d at gmail.com  Fri Jan 19 12:18:27 2018
From: larson.eric.d at gmail.com (Eric Larson)
Date: Fri, 19 Jan 2018 12:18:27 -0500
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <1610f5d92cc.116a5cf7615287.4034716040473351782@zoho.com>
References: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>
 <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>
 <1610f5d92cc.116a5cf7615287.4034716040473351782@zoho.com>
Message-ID: <CAGu2niVcrfVbm4X9n1g7P4dK2NKKjzvSQmp9VBOfxjHrL+oRQg@mail.gmail.com>

>
> Thank you for the feedback. You mentioned 3D visualization in which, as
> far as I know, 4D homogeneous coordinates are commonly used. But we
> shouldn't go in this direction and consider 3D vectors and rotations only,
> right?
>

Yeah at least at first (for GSoC or first implementation) this makes sense
to me. In viz I ran into needing quaternions specifically. In neuroscience
I've needed that and also e.g. computing the angle between two rotation
matrices (or quaternions), among other things.

Great that you are interested in mentoring this project as well. I will add
> the idea in a basic form to the wiki. Generally I think we should leave
> some space for a student to investigate, instead of specifying absolutely
> everything. Do you agree?
>

Yes that sounds reasonable to me.

If anyone thinks this isn't within the scope of SciPy, speak now or forever
hold your peace :)

Cheers,
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180119/8d0a4542/attachment.html>

From mark.campanelli at gmail.com  Fri Jan 19 14:58:03 2018
From: mark.campanelli at gmail.com (Mark Campanelli)
Date: Fri, 19 Jan 2018 19:58:03 +0000
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <CAGu2niVcrfVbm4X9n1g7P4dK2NKKjzvSQmp9VBOfxjHrL+oRQg@mail.gmail.com>
References: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>
 <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>
 <1610f5d92cc.116a5cf7615287.4034716040473351782@zoho.com>
 <CAGu2niVcrfVbm4X9n1g7P4dK2NKKjzvSQmp9VBOfxjHrL+oRQg@mail.gmail.com>
Message-ID: <CAES-E-5e3p_dtTWL0QbbWuP2nfBJgbQPxsKox8QCv03MAr0+kQ@mail.gmail.com>

Anyone familiar with Geometric Algebra? Might be worth consideration during
preliminary discussions. For example, thinks looks potentially relevant:
https://dl.acm.org/citation.cfm?id=1610323

On Fri, Jan 19, 2018 at 10:19 Eric Larson <larson.eric.d at gmail.com> wrote:

> Thank you for the feedback. You mentioned 3D visualization in which, as
>> far as I know, 4D homogeneous coordinates are commonly used. But we
>> shouldn't go in this direction and consider 3D vectors and rotations only,
>> right?
>>
>
> Yeah at least at first (for GSoC or first implementation) this makes sense
> to me. In viz I ran into needing quaternions specifically. In neuroscience
> I've needed that and also e.g. computing the angle between two rotation
> matrices (or quaternions), among other things.
>
> Great that you are interested in mentoring this project as well. I will
>> add the idea in a basic form to the wiki. Generally I think we should leave
>> some space for a student to investigate, instead of specifying absolutely
>> everything. Do you agree?
>>
>
> Yes that sounds reasonable to me.
>
> If anyone thinks this isn't within the scope of SciPy, speak now or
> forever hold your peace :)
>
> Cheers,
> Eric
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180119/658aaa44/attachment-0001.html>

From cimrman3 at ntc.zcu.cz  Fri Jan 19 16:11:42 2018
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Fri, 19 Jan 2018 22:11:42 +0100
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <CAES-E-5e3p_dtTWL0QbbWuP2nfBJgbQPxsKox8QCv03MAr0+kQ@mail.gmail.com>
References: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>
 <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>
 <1610f5d92cc.116a5cf7615287.4034716040473351782@zoho.com>
 <CAGu2niVcrfVbm4X9n1g7P4dK2NKKjzvSQmp9VBOfxjHrL+oRQg@mail.gmail.com>
 <CAES-E-5e3p_dtTWL0QbbWuP2nfBJgbQPxsKox8QCv03MAr0+kQ@mail.gmail.com>
Message-ID: <c8352ba8-a5a8-fea5-06a8-7b8931d138dc@ntc.zcu.cz>

On 01/19/2018 08:58 PM, Mark Campanelli wrote:
> Anyone familiar with Geometric Algebra? Might be worth consideration during
> preliminary discussions. For example, thinks looks potentially relevant:
> https://dl.acm.org/citation.cfm?id=1610323

FYI: a freely accessible resource on geometric algebra: 
http://www.jaapsuter.com/geometric-algebra/

And ,of course, there is a Python package for that: https://clifford.readthedocs.io

r.

> On Fri, Jan 19, 2018 at 10:19 Eric Larson <larson.eric.d at gmail.com> wrote:
> 
>> Thank you for the feedback. You mentioned 3D visualization in which, as
>>> far as I know, 4D homogeneous coordinates are commonly used. But we
>>> shouldn't go in this direction and consider 3D vectors and rotations only,
>>> right?
>>>
>>
>> Yeah at least at first (for GSoC or first implementation) this makes sense
>> to me. In viz I ran into needing quaternions specifically. In neuroscience
>> I've needed that and also e.g. computing the angle between two rotation
>> matrices (or quaternions), among other things.
>>
>> Great that you are interested in mentoring this project as well. I will
>>> add the idea in a basic form to the wiki. Generally I think we should leave
>>> some space for a student to investigate, instead of specifying absolutely
>>> everything. Do you agree?
>>>
>>
>> Yes that sounds reasonable to me.
>>
>> If anyone thinks this isn't within the scope of SciPy, speak now or
>> forever hold your peace :)
>>
>> Cheers,
>> Eric
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> 
> 
> 
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
> 


From oss at multiwave.ch  Fri Jan 19 16:28:36 2018
From: oss at multiwave.ch (oss)
Date: Fri, 19 Jan 2018 22:28:36 +0100
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>
References: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>
 <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>
Message-ID: <C0FFA963-9481-4E38-AC59-C74369D3A7DA@multiwave.ch>

Hello,

Maybe this could come in handy regarding transforms matrices quaternions
 etc.

https://www.lfd.uci.edu/~gohlke/code/transformations.py.html <https://www.lfd.uci.edu/~gohlke/code/transformations.py.html>

Best

Tryfon


> On Jan 19, 2018, at 4:25 PM, Eric Larson <larson.eric.d at gmail.com> wrote:
> 
> I have personally run into the need for such transformations in two separate domains (3D visualization, neuroscience / electrophysiology) and I know it's used in multiple other places, too. So I think it would be sufficiently general. I'd look forward to having it in SciPy!
> 
> I'd be happy to be a secondary mentor on this if you (or someone else) wants to be primary.
> 
> Best,
> Eric
> 
> 
> On Fri, Jan 19, 2018 at 10:11 AM, Nikolay Mayorov <nikolay.mayorov at zoho.com <mailto:nikolay.mayorov at zoho.com>> wrote:
> Hi!
> 
> I have this idea, which I'm well familiar with. The module would be called like scipy.spatial.rotation and be devoted to the rotation formalism in 3 dimensions. 
> 
> The main objects are Euler angles (and their variations), direction cosine matrices, quaternions and rotation vectors. We can go with an abstraction class Rotation (using DCMs or Qs internally), but we should be able to create that from any representation and see it in any representation. In spirit of scipy/numpy we use vectorized/bulk approaches (i.e. many rotations in single Rotation class).
> 
> Rotation should support 2 operations: compose 2 consecutive rotations and rotate/project a 3d vector.  Of course all procedures must be 100% robust and there are some fine points, especially in conversions between representations.
> Also we can add some algorithms, like: quaternion interpolation (SLERP), least-squares vector matching by a rotation (Whabba's problem), more advanced and less known algotithms for rotation interpolation, and I will try to come up with something more.
> 
> Overall it seems reasonably straightforward , but with enough challenges in design and implementation
> 
> As currently described, it might be not enough volume for the GSoC, but we can develop it farther. 
> 
> Also I'm not sure if its applicability is broad enough to include it into scipy. I believe similar functionality is available in Aerospace toolbox in Matlab. I want to hear some opinions on that.
> 
> Nikolay
> 
> 
> ---- On Wed, 10 Jan 2018 17:02:33 +0500 jomsdev at gmail.com <mailto:jomsdev at gmail.com> wrote ----
> 
> Hi all,
> 
> Last year I started implementing some methods for Randomized Numerical Linear Algebra (RNLA) in scipy. 
> By now it is only the CountMin Sketch (clarkson_woodruff_transformation) for reducing the dimensionality of a vector space to an embedded space. 
> 
> I think that it would be interesting to add to scipy other methods for subspace embedding (like the Johnson-Lindenstrauss) and build some algorithms on top of it for things like least squeres or low rank approximation.
> 
> Would some other people be interesting in this?
> 
> 
> PS: I have a project called RandNLA <https://github.com/jomsdev/randNLA> where I implemented some of the methods of RNLA. The idea is to implement only the most important methods of RNLA in scipy and have this other library for experimenting with new methods and APIs. That will let us not overloading scipy with features if people are not interested in them and focus on the ones that really brings value to the community.
> 
> Thanks,
> 
> Jordi.
> 
> On 10 January 2018 at 10:42, Ralf Gommers <ralf.gommers at gmail.com <mailto:ralf.gommers at gmail.com>> wrote:
> Hi all,
> 
> The GSoC schedule is a bit earlier than normal this year. The PSF is asking for ideas pages to be up and in decent shape by Jan 19th. So we'll need to come up with some content quick if we want to participate.
> 
> Who is interested in mentoring this year?
> 
> I'm happy to do the admin again, but probably won't have time to mentor.
> 
> Cheers,
> Ralf
> 
> 
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org <mailto:SciPy-Dev at python.org>
> https://mail.python.org/mailman/listinfo/scipy-dev <https://mail.python.org/mailman/listinfo/scipy-dev>
> 
> 
> _______________________________________________ 
> SciPy-Dev mailing list 
> SciPy-Dev at python.org <mailto:SciPy-Dev at python.org> 
> https://mail.python.org/mailman/listinfo/scipy-dev <https://mail.python.org/mailman/listinfo/scipy-dev> 
> 
> 
> 
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org <mailto:SciPy-Dev at python.org>
> https://mail.python.org/mailman/listinfo/scipy-dev <https://mail.python.org/mailman/listinfo/scipy-dev>
> 
> 
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180119/ed99321a/attachment.html>

From mark.campanelli at gmail.com  Fri Jan 19 18:17:01 2018
From: mark.campanelli at gmail.com (Mark Campanelli)
Date: Fri, 19 Jan 2018 23:17:01 +0000
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <c8352ba8-a5a8-fea5-06a8-7b8931d138dc@ntc.zcu.cz>
References: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>
 <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>
 <1610f5d92cc.116a5cf7615287.4034716040473351782@zoho.com>
 <CAGu2niVcrfVbm4X9n1g7P4dK2NKKjzvSQmp9VBOfxjHrL+oRQg@mail.gmail.com>
 <CAES-E-5e3p_dtTWL0QbbWuP2nfBJgbQPxsKox8QCv03MAr0+kQ@mail.gmail.com>
 <c8352ba8-a5a8-fea5-06a8-7b8931d138dc@ntc.zcu.cz>
Message-ID: <CAES-E-5YmUeGiFt7tvyk2tRVUjg2YZ9JT7fDJhKcWHZ0DYavjA@mail.gmail.com>

Thanks Robert! GA seems to have advanced nicely since I last dabbled with
it. I would hope any ?core? approach based on GA could remain accessible to
those with a more traditional background. I recall this bridge seemed very
worth crossing over.

On Fri, Jan 19, 2018 at 14:12 Robert Cimrman <cimrman3 at ntc.zcu.cz> wrote:

> On 01/19/2018 08:58 PM, Mark Campanelli wrote:
> > Anyone familiar with Geometric Algebra? Might be worth consideration
> during
> > preliminary discussions. For example, thinks looks potentially relevant:
> > https://dl.acm.org/citation.cfm?id=1610323
>
> FYI: a freely accessible resource on geometric algebra:
> http://www.jaapsuter.com/geometric-algebra/
>
> And ,of course, there is a Python package for that:
> https://clifford.readthedocs.io
>
> r.
>
> > On Fri, Jan 19, 2018 at 10:19 Eric Larson <larson.eric.d at gmail.com>
> wrote:
> >
> >> Thank you for the feedback. You mentioned 3D visualization in which, as
> >>> far as I know, 4D homogeneous coordinates are commonly used. But we
> >>> shouldn't go in this direction and consider 3D vectors and rotations
> only,
> >>> right?
> >>>
> >>
> >> Yeah at least at first (for GSoC or first implementation) this makes
> sense
> >> to me. In viz I ran into needing quaternions specifically. In
> neuroscience
> >> I've needed that and also e.g. computing the angle between two rotation
> >> matrices (or quaternions), among other things.
> >>
> >> Great that you are interested in mentoring this project as well. I will
> >>> add the idea in a basic form to the wiki. Generally I think we should
> leave
> >>> some space for a student to investigate, instead of specifying
> absolutely
> >>> everything. Do you agree?
> >>>
> >>
> >> Yes that sounds reasonable to me.
> >>
> >> If anyone thinks this isn't within the scope of SciPy, speak now or
> >> forever hold your peace :)
> >>
> >> Cheers,
> >> Eric
> >>
> >> _______________________________________________
> >> SciPy-Dev mailing list
> >> SciPy-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/scipy-dev
> >>
> >
> >
> >
> > _______________________________________________
> > SciPy-Dev mailing list
> > SciPy-Dev at python.org
> > https://mail.python.org/mailman/listinfo/scipy-dev
> >
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180119/643549d1/attachment-0001.html>

From ralf.gommers at gmail.com  Sat Jan 20 00:09:11 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 20 Jan 2018 18:09:11 +1300
Subject: [SciPy-Dev] Restructuring flapack.src.pyf file
In-Reply-To: <CAEBuzr8+yCOtkEuvos3hLKUWa8RKd=S80ZQcgQeinpXqq_K1jA@mail.gmail.com>
References: <CAEBuzr9RKzGzjvY77gM5iKkhK=hifio+UKMBPyzUvwDf9xYoxA@mail.gmail.com>
 <3sm3kp.p1jh54.rthztg-qmf@smtp.gmail.com>
 <CAEBuzr-hQKR=atVGzww7_n4ZzL-6m-igpjzNjo+i5EXPYZ4h8w@mail.gmail.com>
 <CABL7CQikKiuug=DWY_7WXTMeWOTyicU2DtKKJaubUQf-5pmMyQ@mail.gmail.com>
 <CAEBuzr8+yCOtkEuvos3hLKUWa8RKd=S80ZQcgQeinpXqq_K1jA@mail.gmail.com>
Message-ID: <CABL7CQgE8DeNRW4HNO7+_uJFR7cTJ2bAfpYomviPf=3wH9sD1A@mail.gmail.com>

On Sat, Jan 13, 2018 at 11:16 PM, Ilhan Polat <ilhanpolat at gmail.com> wrote:

> Most of them would indeed just gain new keywords xxgvd, xxgv, and so on.
> These would only get lwork and liwork keywords. No deprecations needed.
>
> ?(sy/he)evr on the other hand not only gets these extra keywords but also
> would gain two more output arguments which is actually the reason for this
> thread. So fortunately the amount of deprecation is "a few"~
> ssyevr/dsyevr/cheevr/zheevr so far.
>

Okay, that sounds a lot more sensible / less worrying. Maybe just open a PR
and we'll look at the detailed changes there?

Cheers,
Ralf


These are used in the regular case of scipy.linalg.eigh and was causing
> problems due to unoptimized lwork values. Also they were not completely
> exposed.hence the new ouput args.
>
> I would like to start a separate discussion about the deprecation of
> keywords "eigvals" and "turbo" via a decorator later once this is done.
>
>  Because now we are able to give "eig_range" as an input if need be as a
> new functionality. And "eigvals" does not correspond to giving an index
> range say, "eig_index". But that will have to wait until this surgery on
> the LAPACK side.
>
>
>
> On Jan 13, 2018 09:31, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:
>
>>
>>
>> On Wed, Dec 27, 2017 at 2:40 AM, Ilhan Polat <ilhanpolat at gmail.com>
>> wrote:
>>
>>> The situation, as usual, turns out a bit more involved
>>>
>>> In the discussion, https://github.com/scipy/scipy/issues/6502 Tiziano
>>> writes
>>>
>>> "Hi, I am the author of the original wrappers for those LAPACK routines.
>>> If I remember correctly, the reason for hiding the eigenvalues in an
>>> interval option was simply that the original implementation of
>>> scipy.linalg.eigh did not even allow to limit the number of
>>> eigenvalues/eigenvectors returned. I needed only the possiblity to define
>>> the indices of the eigenvalues and not the range, and the result is the
>>> option eigvals to scipy.linalg.eigh. I think that your wrapper is too
>>> generic. You should not allow setting from Python the values of things like
>>> lwork: a wrong value there and the user can generate a segfault from
>>> Python, which is a no-no."
>>>
>>> and I can see more LAPACK routines that are wrapped along this line.
>>> Having 8 more "xxxx_full" and "xxxx_full_lwork" routines doesn't seem to be
>>> elegant though I don't have any alternative yet. Maybe it is OK to have
>>> them for a few versions but I guess it would be nice if we have a parallel
>>> running deprecation plans for this issue. It looks like _np.deprecate works
>>> pretty OK. And we have four deprecations already in lapack.py e.g.,
>>>
>>> cgegv = _np.deprecate(cgegv, old_name='cgegv', message=_dep_message)
>>>
>>> This will print a deprecation warning when it is called or docstring is
>>> printed. "DeprecationWarning: `dgegv` is deprecated! The `*gegv` family of
>>> routines has been deprecated in LAPACK 3.6.0 in favor of the `*ggev` family
>>> of routines.The corresponding wrappers will be removed from SciPy in a
>>> future release." So I would like to deprecate the routines abouve with a
>>> message
>>>
>>> "The *XXXXX family of routines will have a different signature in the
>>> future versions of  SciPy. The new versions are now available as
>>> *XXXXX_full and will eventually replace the original *XXXXX family"
>>>
>>> Would you dis/agree?
>>>
>>
>> Just wondering about the tradeoff between deprecating + having a new
>> function with arguments in the same order as LAPACK, vs. just appending the
>> keywords to the existing function. The latter case would need a thin Python
>> function in lapack.py, but unless that hurts performance significantly I'm
>> not sure that's much of an issue.
>>
>> Ralf
>>
>>
>>
>>
>>>
>>>
>>>
>>> On Mon, Dec 25, 2017 at 11:58 PM, <pav at iki.fi> wrote:
>>>
>>>> Hi,
>>>>
>>>> Ilhan Polat kirjoitti tiistai 26. joulukuuta 2017:
>>>> [clip]
>>>> > I've done a similar thing by freeing out all ins and outs. It seems
>>>> to me
>>>> > that the signature change is inevitable but the question is how to
>>>> handle
>>>> > this?
>>>>
>>>> New function sounds best, maximally unimaginative new name is probably
>>>> most appropriate.
>>>>
>>>> Not sure how well np.deprecate etc python level wrappers work on f2py.
>>>>
>>>>
>>>> > 2) It looks like flapack.src.pyf file is getting exceedingly long and
>>>> > unstructured. Should we break it apart just like LAPACK documentation?
>>>>
>>>> If you think that helps it significantly and is worthwhile use of your
>>>> time then why not. (I suspect it's not particularly worthwhile but ymmv.)
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180120/bbf60f55/attachment.html>

From b.rosa at unistra.fr  Sat Jan 20 03:12:12 2018
From: b.rosa at unistra.fr (Benoit Rosa)
Date: Sat, 20 Jan 2018 09:12:12 +0100
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <C0FFA963-9481-4E38-AC59-C74369D3A7DA@multiwave.ch>
References: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>
 <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>
 <C0FFA963-9481-4E38-AC59-C74369D3A7DA@multiwave.ch>
Message-ID: <b0d689db-b23d-ae2d-25de-da3e915020ca@unistra.fr>

Hi,

I have used that library quite a few times, and it is rather slow. 
Adding a transformation (or, for starters, rotation) module to scipy 
would be, in my opinion, a nice addition.

Speaking about adding a few algorithms, one interesting add could be a 
function to uniformly sample the rotation space. It is a core 
functionality needed in a number of cases, and not that straightforward 
to perform properly (again, depending on the chosen formalism for 
representing the rotation).

Best
Benoit


On 19/01/2018 22:28, oss wrote:
> Hello,
>
> Maybe this could come in handy regarding transforms matrices quaternions
> ?etc.
>
> https://www.lfd.uci.edu/~gohlke/code/transformations.py.html 
> <https://www.lfd.uci.edu/%7Egohlke/code/transformations.py.html>
>
> Best
>
> Tryfon
>
>
>> On Jan 19, 2018, at 4:25 PM, Eric Larson <larson.eric.d at gmail.com 
>> <mailto:larson.eric.d at gmail.com>> wrote:
>>
>> I have personally run into the need for such transformations in two 
>> separate domains (3D visualization, neuroscience / electrophysiology) 
>> and I know it's used in multiple other places, too. So I think it 
>> would be sufficiently general. I'd look forward to having it in SciPy!
>>
>> I'd be happy to be a secondary mentor on this if you (or someone 
>> else) wants to be primary.
>>
>> Best,
>> Eric
>>
>>
>> On Fri, Jan 19, 2018 at 10:11 AM, Nikolay Mayorov 
>> <nikolay.mayorov at zoho.com <mailto:nikolay.mayorov at zoho.com>> wrote:
>>
>>     Hi!
>>
>>     I have this idea, which I'm well familiar with. The module would
>>     be called like scipy.spatial.rotation and be devoted to the
>>     rotation formalism in 3 dimensions.
>>
>>     The main objects are Euler angles (and their variations),
>>     direction cosine matrices, quaternions and rotation vectors. We
>>     can go with an abstraction class Rotation (using DCMs or Qs
>>     internally), but we should be able to create that from any
>>     representation and see it in any representation. In spirit of
>>     scipy/numpy we use vectorized/bulk approaches (i.e. many
>>     rotations in single Rotation class).
>>
>>     Rotation should support 2 operations: compose 2 consecutive
>>     rotations and rotate/project a 3d vector.??Of course all
>>     procedures must be 100% robust and there are some fine points,
>>     especially in conversions between representations.Also we can add
>>     some algorithms, like: quaternion interpolation (SLERP),
>>     least-squares vector matching by a rotation (Whabba's problem),
>>     more advanced and less known algotithms for rotation
>>     interpolation, and I will try to come up with something more.
>>
>>     Overall it seems reasonably straightforward , but with enough
>>     challenges in design and implementation
>>
>>     As currently described, it might be not enough volume for the
>>     GSoC, but we can develop it farther.
>>
>>     Also I'm not sure if its applicability is broad enough to include
>>     it into scipy. I believe similar functionality is available in
>>     Aerospace toolbox in Matlab. I want to hear some opinions on that.
>>
>>     Nikolay
>>
>>
>>     ---- On Wed, 10 Jan 2018 17:02:33 +0500 *jomsdev at gmail.com
>>     <mailto:jomsdev at gmail.com> * wrote ----
>>
>>         Hi all,
>>
>>         Last year I started implementing some methods for Randomized
>>         Numerical Linear Algebra (RNLA) in scipy.
>>         By now it is only the CountMin Sketch
>>         (clarkson_woodruff_transformation) for reducing the
>>         dimensionality of a vector space to an embedded space.
>>
>>         I think that it would be interesting to add to scipy other
>>         methods for subspace embedding (like the
>>         Johnson-Lindenstrauss) and build some algorithms on top of it
>>         for things like least squeres or low rank approximation.
>>
>>         Would some other people be interesting in this?
>>
>>
>>         PS: I have a project called RandNLA
>>         <https://github.com/jomsdev/randNLA> where I implemented some
>>         of the methods of RNLA. The idea is to implement only the
>>         most important methods of RNLA in scipy and have this other
>>         library for experimenting with new methods and APIs. That
>>         will let us not overloading scipy with features if people are
>>         not interested in them and focus on the ones that really
>>         brings value to the community.
>>
>>         Thanks,
>>
>>         Jordi.
>>
>>         On 10 January 2018 at 10:42, Ralf Gommers
>>         <ralf.gommers at gmail.com <mailto:ralf.gommers at gmail.com>> wrote:
>>
>>             Hi all,
>>
>>             The GSoC schedule is a bit earlier than normal this year.
>>             The PSF is asking for ideas pages to be up and in decent
>>             shape by Jan 19th. So we'll need to come up with some
>>             content quick if we want to participate.
>>
>>             Who is interested in mentoring this year?
>>
>>             I'm happy to do the admin again, but probably won't have
>>             time to mentor.
>>
>>             Cheers,
>>             Ralf
>>
>>
>>             _______________________________________________
>>             SciPy-Dev mailing list
>>             SciPy-Dev at python.org <mailto:SciPy-Dev at python.org>
>>             https://mail.python.org/mailman/listinfo/scipy-dev
>>             <https://mail.python.org/mailman/listinfo/scipy-dev>
>>
>>
>>         _______________________________________________
>>         SciPy-Dev mailing list
>>         SciPy-Dev at python.org <mailto:SciPy-Dev at python.org>
>>         https://mail.python.org/mailman/listinfo/scipy-dev
>>         <https://mail.python.org/mailman/listinfo/scipy-dev>
>>
>>
>>
>>
>>     _______________________________________________
>>     SciPy-Dev mailing list
>>     SciPy-Dev at python.org <mailto:SciPy-Dev at python.org>
>>     https://mail.python.org/mailman/listinfo/scipy-dev
>>     <https://mail.python.org/mailman/listinfo/scipy-dev>
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org <mailto:SciPy-Dev at python.org>
>> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180120/eb894864/attachment-0001.html>

From nikolay.mayorov at zoho.com  Sat Jan 20 05:09:15 2018
From: nikolay.mayorov at zoho.com (Nikolay Mayorov)
Date: Sat, 20 Jan 2018 15:09:15 +0500
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <b0d689db-b23d-ae2d-25de-da3e915020ca@unistra.fr>
References: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>
 <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>
 <C0FFA963-9481-4E38-AC59-C74369D3A7DA@multiwave.ch>
 <b0d689db-b23d-ae2d-25de-da3e915020ca@unistra.fr>
Message-ID: <161130b3084.cf9acf3121661.3858523581452506771@zoho.com>

Thanks for the suggestion, Benoit! I think the easiest way to achieve that is sample a quaternion uniformly on a unit 4D sphere. 



By the way I updated the ideas wiki and I will include your suggestion as well. Also as was suggested we could use a more general name scipy.spatial.transform.



Nikolay





Hi,

 

 I have used that library quite a few times, and it is rather slow. Adding a transformation (or, for starters, rotation) module to scipy would be, in my opinion, a nice addition. 

 

 Speaking about adding a few algorithms, one interesting add could be a function to uniformly sample the rotation space. It is a core functionality needed in a number of cases, and not that straightforward to perform properly (again, depending on the chosen formalism for representing the rotation). 

 

 Best

 Benoit

 

 

 

On 19/01/2018 22:28, oss wrote:

Hello, 



Maybe this could come in handy regarding transforms matrices quaternions

 etc.



https://www.lfd.uci.edu/~gohlke/code/transformations.py.html



Best



Tryfon





On Jan 19, 2018, at 4:25 PM, Eric Larson &lt;larson.eric.d at gmail.com&gt; wrote:



I have personally run into the need for such transformations in two separate domains (3D visualization, neuroscience / electrophysiology) and I know it's used in multiple other places, too. So I think it would be sufficiently general. I'd look forward to having it in SciPy! 



I'd be happy to be a secondary mentor on this if you (or someone else) wants to be primary.



Best, 

Eric







On Fri, Jan 19, 2018 at 10:11 AM, Nikolay Mayorov &lt;nikolay.mayorov at zoho.com&gt; wrote:

 

Hi!

 

 I have this idea, which I'm well familiar with. The module would be called like scipy.spatial.rotation and be devoted to the rotation formalism in 3 dimensions. 

 

 The main objects are Euler angles (and their variations), direction cosine matrices, quaternions and rotation vectors. We can go with an abstraction class Rotation (using DCMs or Qs internally), but we should be able to create that from any representation and see it in any representation. In spirit of scipy/numpy we use vectorized/bulk approaches (i.e. many rotations in single Rotation class).

 

 Rotation should support 2 operations: compose 2 consecutive rotations and rotate/project a 3d vector.  Of course all procedures must be 100% robust and there are some fine points, especially in conversions between representations.Also we can add some algorithms, like: quaternion interpolation (SLERP), least-squares vector matching by a rotation (Whabba's problem), more advanced and less known algotithms for rotation interpolation, and I will try to come up with something more. 



Overall it seems reasonably straightforward , but with enough challenges in design and implementation



As currently described, it might be not enough volume for the GSoC, but we can develop it farther. 



Also I'm not sure if its applicability is broad enough to include it into scipy. I believe similar functionality is available in Aerospace toolbox in Matlab. I want to hear some opinions on that.

 



Nikolay





---- On Wed, 10 Jan 2018 17:02:33 +0500 jomsdev at gmail.com  wrote ----





Hi all,

 


Last year I started implementing some methods for Randomized Numerical Linear Algebra (RNLA) in scipy.

 


By now it is only the CountMin Sketch (clarkson_woodruff_transformation) for reducing the dimensionality of a vector space to an embedded space.

 





I think that it would be interesting to add to scipy other methods for subspace embedding (like the Johnson-Lindenstrauss) and build some algorithms on top of it for things like least squeres or low rank approximation.



Would some other people be interesting in this?





PS: I have a project called RandNLA where I implemented some of the methods of RNLA. The idea is to implement only the most important methods of RNLA in scipy and have this other library for experimenting with new methods and APIs. That will let us not overloading scipy with features if people are not interested in them and focus on the ones that really brings value to the community.



Thanks,



Jordi.




On 10 January 2018 at 10:42, Ralf Gommers &lt;ralf.gommers at gmail.com&gt; wrote:

 

Hi all,

 


The GSoC schedule is a bit earlier than normal this year. The PSF is asking for ideas pages to be up and in decent shape by Jan 19th. So we'll need to come up with some content quick if we want to participate.

 

 


Who is interested in mentoring this year?

 

 


I'm happy to do the admin again, but probably won't have time to mentor.

 

 


Cheers,

 


Ralf

 

 




_______________________________________________

 SciPy-Dev mailing list

 SciPy-Dev at python.org

 https://mail.python.org/mailman/listinfo/scipy-dev

 





_______________________________________________

 SciPy-Dev mailing list 

 SciPy-Dev at python.org 

 https://mail.python.org/mailman/listinfo/scipy-dev 















_______________________________________________

 SciPy-Dev mailing list

 SciPy-Dev at python.org

 https://mail.python.org/mailman/listinfo/scipy-dev

 





_______________________________________________

 SciPy-Dev mailing list

 SciPy-Dev at python.org

 https://mail.python.org/mailman/listinfo/scipy-dev










_______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev 


_______________________________________________

SciPy-Dev mailing list 

SciPy-Dev at python.org 

https://mail.python.org/mailman/listinfo/scipy-dev 






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180120/d9bf3d39/attachment.html>

From serge.guelton at telecom-bretagne.eu  Sat Jan 20 05:10:34 2018
From: serge.guelton at telecom-bretagne.eu (Serge Guelton)
Date: Sat, 20 Jan 2018 11:10:34 +0100
Subject: [SciPy-Dev] Using Pythran to compile some of the scipy internals
Message-ID: <20180120101034.GA3707@lakota>

Hi Scipy-dev,

= Context

I've started the experiment of compiling some of the scipy kernels with
the Pythran[0] compiler.

The approach was basically to pick some easy .pyx file, turn them back
to high-level Python + Numpy files then processing them with Pythran.
This results in easier to maintain, Python-compatible files that run at
the same speed as their Cython alternative up to 10 times faster
depending on the kernels and whether SIMD instructions are activated or
not (I did not play with parallelization in that experiment).

This required a few improvement to Pythran, but not that much, see

    https://github.com/serge-sans-paille/pythran/pull/770

for the associated PR on pythran, which includes the following scipy
kernels:

    hausdorff.py
    max_len_seq_inner.py
    solve_toeplitz.py
    spectral.py

= Why bother?

According to that first round of experiments, using Pythran can bring
extra speed improvements while remaining at a higher level than Cython.
Both points looks like an improvement to me.

= Cost?

This adds an extra dependency on Pythran, which uses C++ as backend.
This increases the failure surface. Although alive since 2012 and being
tested a lot [1] on Linux (but scarcely on Windows), its is obviously
less mature than cython

= Alternatives

There is an experimental Pythran mode in cython[2] that uses Pythran as
a backend for numpy operations. Unfortunately it is still at early
stages and cannot translate calls to ``np.roll`` or ``np.sum(a[i,:] - b[j, :])`` while Pythran supports it.

Instead of translating Cython files, I could also focus on some
pure-python functions. I tested Pythran on the rosenbrock function and I
get good speedup (from 1.5x to 4x depending on vectorization being
enabled or not) there too.

So yeah, that's a rather long introduction to probe the interest here
around that idea :-)

Best,
Serge

PS: I started https://github.com/scipy/scipy/pull/8306 meanwhile, but
let's discuss here first :-)



[0] https://github.com/serge-sans-paille/pythran
[1] https://travis-ci.org/serge-sans-paille/pythran/builds/330052310
[2] http://cython.readthedocs.io/en/latest/src/userguide/numpy_pythran.html

From cgohlke at uci.edu  Sat Jan 20 10:34:35 2018
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Sat, 20 Jan 2018 07:34:35 -0800
Subject: [SciPy-Dev] GSoC'18 participation?
In-Reply-To: <b0d689db-b23d-ae2d-25de-da3e915020ca@unistra.fr>
References: <1610ef99097.100af0851154280.4991198338195760990@zoho.com>
 <CAGu2niX4iykkDKOdc9QxUwTkJcxw1+ZOZH0LqcQzgp+tZ+sPww@mail.gmail.com>
 <C0FFA963-9481-4E38-AC59-C74369D3A7DA@multiwave.ch>
 <b0d689db-b23d-ae2d-25de-da3e915020ca@unistra.fr>
Message-ID: <aa56d8f3-8c2b-22ce-ef9f-e8f365198480@uci.edu>



On 1/20/2018 12:12 AM, Benoit Rosa wrote:
> Hi,
> 
> I have used that library quite a few times, and it is rather slow. 

That's because using small numpy arrays (4x4 or 1x4) to represent 
homogeneous transformation matrices, vectors, points, and quaternions is 
inherently inefficient, even when using the C API. As usual if that 
overhead matters, vectorize the algorithm or implement it in 
Cython/C/C++/FORTRAN/etc. There's a speedier C extension module 
<https://www.lfd.uci.edu/~gohlke/code/transformations.c.html> but it 
still has to go through the numpy C API.

The transformations.py module has been mostly superseded by the 
Transforms3d package <https://pypi.python.org/pypi/transforms3d>.

IMO quaternions should be a numpy dtype living in numpy or in a small 
package, not in scipy. See <https://github.com/moble/quaternion>

Christoph


> Adding a transformation (or, for starters, rotation) module to scipy 
> would be, in my opinion, a nice addition.
> 
> Speaking about adding a few algorithms, one interesting add could be a 
> function to uniformly sample the rotation space. It is a core 
> functionality needed in a number of cases, and not that straightforward 
> to perform properly (again, depending on the chosen formalism for 
> representing the rotation).
> 
> Best
> Benoit
> 
> 
> On 19/01/2018 22:28, oss wrote:
>> Hello,
>>
>> Maybe this could come in handy regarding transforms matrices quaternions
>> ?etc.
>>
>> https://www.lfd.uci.edu/~gohlke/code/transformations.py.html 
>> <https://www.lfd.uci.edu/%7Egohlke/code/transformations.py.html>
>>
>> Best
>>
>> Tryfon
>>
>>
>>> On Jan 19, 2018, at 4:25 PM, Eric Larson <larson.eric.d at gmail.com 
>>> <mailto:larson.eric.d at gmail.com>> wrote:
>>>
>>> I have personally run into the need for such transformations in two 
>>> separate domains (3D visualization, neuroscience / electrophysiology) 
>>> and I know it's used in multiple other places, too. So I think it 
>>> would be sufficiently general. I'd look forward to having it in SciPy!
>>>
>>> I'd be happy to be a secondary mentor on this if you (or someone 
>>> else) wants to be primary.
>>>
>>> Best,
>>> Eric
>>>
>>>
>>> On Fri, Jan 19, 2018 at 10:11 AM, Nikolay Mayorov 
>>> <nikolay.mayorov at zoho.com <mailto:nikolay.mayorov at zoho.com>> wrote:
>>>
>>>     Hi!
>>>
>>>     I have this idea, which I'm well familiar with. The module would
>>>     be called like scipy.spatial.rotation and be devoted to the
>>>     rotation formalism in 3 dimensions.
>>>
>>>     The main objects are Euler angles (and their variations),
>>>     direction cosine matrices, quaternions and rotation vectors. We
>>>     can go with an abstraction class Rotation (using DCMs or Qs
>>>     internally), but we should be able to create that from any
>>>     representation and see it in any representation. In spirit of
>>>     scipy/numpy we use vectorized/bulk approaches (i.e. many
>>>     rotations in single Rotation class).
>>>
>>>     Rotation should support 2 operations: compose 2 consecutive
>>>     rotations and rotate/project a 3d vector.??Of course all
>>>     procedures must be 100% robust and there are some fine points,
>>>     especially in conversions between representations.Also we can add
>>>     some algorithms, like: quaternion interpolation (SLERP),
>>>     least-squares vector matching by a rotation (Whabba's problem),
>>>     more advanced and less known algotithms for rotation
>>>     interpolation, and I will try to come up with something more.
>>>
>>>     Overall it seems reasonably straightforward , but with enough
>>>     challenges in design and implementation
>>>
>>>     As currently described, it might be not enough volume for the
>>>     GSoC, but we can develop it farther.
>>>
>>>     Also I'm not sure if its applicability is broad enough to include
>>>     it into scipy. I believe similar functionality is available in
>>>     Aerospace toolbox in Matlab. I want to hear some opinions on that.
>>>
>>>     Nikolay
>>>
>>>
>>>     ---- On Wed, 10 Jan 2018 17:02:33 +0500 *jomsdev at gmail.com
>>>     <mailto:jomsdev at gmail.com> * wrote ----
>>>
>>>         Hi all,
>>>
>>>         Last year I started implementing some methods for Randomized
>>>         Numerical Linear Algebra (RNLA) in scipy.
>>>         By now it is only the CountMin Sketch
>>>         (clarkson_woodruff_transformation) for reducing the
>>>         dimensionality of a vector space to an embedded space.
>>>
>>>         I think that it would be interesting to add to scipy other
>>>         methods for subspace embedding (like the
>>>         Johnson-Lindenstrauss) and build some algorithms on top of it
>>>         for things like least squeres or low rank approximation.
>>>
>>>         Would some other people be interesting in this?
>>>
>>>
>>>         PS: I have a project called RandNLA
>>>         <https://github.com/jomsdev/randNLA> where I implemented some
>>>         of the methods of RNLA. The idea is to implement only the
>>>         most important methods of RNLA in scipy and have this other
>>>         library for experimenting with new methods and APIs. That
>>>         will let us not overloading scipy with features if people are
>>>         not interested in them and focus on the ones that really
>>>         brings value to the community.
>>>
>>>         Thanks,
>>>
>>>         Jordi.
>>>
>>>         On 10 January 2018 at 10:42, Ralf Gommers
>>>         <ralf.gommers at gmail.com <mailto:ralf.gommers at gmail.com>> wrote:
>>>
>>>             Hi all,
>>>
>>>             The GSoC schedule is a bit earlier than normal this year.
>>>             The PSF is asking for ideas pages to be up and in decent
>>>             shape by Jan 19th. So we'll need to come up with some
>>>             content quick if we want to participate.
>>>
>>>             Who is interested in mentoring this year?
>>>
>>>             I'm happy to do the admin again, but probably won't have
>>>             time to mentor.
>>>
>>>             Cheers,
>>>             Ralf
>>>
>>>
>>>             _______________________________________________
>>>             SciPy-Dev mailing list
>>>             SciPy-Dev at python.org <mailto:SciPy-Dev at python.org>
>>>             https://mail.python.org/mailman/listinfo/scipy-dev
>>>             <https://mail.python.org/mailman/listinfo/scipy-dev>
>>>
>>>
>>>         _______________________________________________
>>>         SciPy-Dev mailing list
>>>         SciPy-Dev at python.org <mailto:SciPy-Dev at python.org>
>>>         https://mail.python.org/mailman/listinfo/scipy-dev
>>>         <https://mail.python.org/mailman/listinfo/scipy-dev>
>>>
>>>
>>>
>>>
>>>     _______________________________________________
>>>     SciPy-Dev mailing list
>>>     SciPy-Dev at python.org <mailto:SciPy-Dev at python.org>
>>>     https://mail.python.org/mailman/listinfo/scipy-dev
>>>     <https://mail.python.org/mailman/listinfo/scipy-dev>
>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org <mailto:SciPy-Dev at python.org>
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
> 
> 
> 
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
> 

From ilhanpolat at gmail.com  Sat Jan 20 15:42:44 2018
From: ilhanpolat at gmail.com (Ilhan Polat)
Date: Sat, 20 Jan 2018 21:42:44 +0100
Subject: [SciPy-Dev] Restructuring flapack.src.pyf file
In-Reply-To: <CABL7CQgE8DeNRW4HNO7+_uJFR7cTJ2bAfpYomviPf=3wH9sD1A@mail.gmail.com>
References: <CAEBuzr9RKzGzjvY77gM5iKkhK=hifio+UKMBPyzUvwDf9xYoxA@mail.gmail.com>
 <3sm3kp.p1jh54.rthztg-qmf@smtp.gmail.com>
 <CAEBuzr-hQKR=atVGzww7_n4ZzL-6m-igpjzNjo+i5EXPYZ4h8w@mail.gmail.com>
 <CABL7CQikKiuug=DWY_7WXTMeWOTyicU2DtKKJaubUQf-5pmMyQ@mail.gmail.com>
 <CAEBuzr8+yCOtkEuvos3hLKUWa8RKd=S80ZQcgQeinpXqq_K1jA@mail.gmail.com>
 <CABL7CQgE8DeNRW4HNO7+_uJFR7cTJ2bAfpYomviPf=3wH9sD1A@mail.gmail.com>
Message-ID: <CAEBuzr-=HOeBPAJee6nq+pHqX=mtDCQ24yxkXQzGAm97AYhggQ@mail.gmail.com>

I'm secretly working on it :) Hopefully this week I can submit something.

On Jan 20, 2018 06:10, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:

>
>
> On Sat, Jan 13, 2018 at 11:16 PM, Ilhan Polat <ilhanpolat at gmail.com>
> wrote:
>
>> Most of them would indeed just gain new keywords xxgvd, xxgv, and so on.
>> These would only get lwork and liwork keywords. No deprecations needed.
>>
>> ?(sy/he)evr on the other hand not only gets these extra keywords but also
>> would gain two more output arguments which is actually the reason for this
>> thread. So fortunately the amount of deprecation is "a few"~
>> ssyevr/dsyevr/cheevr/zheevr so far.
>>
>
> Okay, that sounds a lot more sensible / less worrying. Maybe just open a
> PR and we'll look at the detailed changes there?
>
> Cheers,
> Ralf
>
>
> These are used in the regular case of scipy.linalg.eigh and was causing
>> problems due to unoptimized lwork values. Also they were not completely
>> exposed.hence the new ouput args.
>>
>> I would like to start a separate discussion about the deprecation of
>> keywords "eigvals" and "turbo" via a decorator later once this is done.
>>
>>  Because now we are able to give "eig_range" as an input if need be as a
>> new functionality. And "eigvals" does not correspond to giving an index
>> range say, "eig_index". But that will have to wait until this surgery on
>> the LAPACK side.
>>
>>
>>
>> On Jan 13, 2018 09:31, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:
>>
>>>
>>>
>>> On Wed, Dec 27, 2017 at 2:40 AM, Ilhan Polat <ilhanpolat at gmail.com>
>>> wrote:
>>>
>>>> The situation, as usual, turns out a bit more involved
>>>>
>>>> In the discussion, https://github.com/scipy/scipy/issues/6502 Tiziano
>>>> writes
>>>>
>>>> "Hi, I am the author of the original wrappers for those LAPACK
>>>> routines. If I remember correctly, the reason for hiding the eigenvalues in
>>>> an interval option was simply that the original implementation of
>>>> scipy.linalg.eigh did not even allow to limit the number of
>>>> eigenvalues/eigenvectors returned. I needed only the possiblity to define
>>>> the indices of the eigenvalues and not the range, and the result is the
>>>> option eigvals to scipy.linalg.eigh. I think that your wrapper is too
>>>> generic. You should not allow setting from Python the values of things like
>>>> lwork: a wrong value there and the user can generate a segfault from
>>>> Python, which is a no-no."
>>>>
>>>> and I can see more LAPACK routines that are wrapped along this line.
>>>> Having 8 more "xxxx_full" and "xxxx_full_lwork" routines doesn't seem to be
>>>> elegant though I don't have any alternative yet. Maybe it is OK to have
>>>> them for a few versions but I guess it would be nice if we have a parallel
>>>> running deprecation plans for this issue. It looks like _np.deprecate works
>>>> pretty OK. And we have four deprecations already in lapack.py e.g.,
>>>>
>>>> cgegv = _np.deprecate(cgegv, old_name='cgegv', message=_dep_message)
>>>>
>>>> This will print a deprecation warning when it is called or docstring is
>>>> printed. "DeprecationWarning: `dgegv` is deprecated! The `*gegv` family of
>>>> routines has been deprecated in LAPACK 3.6.0 in favor of the `*ggev` family
>>>> of routines.The corresponding wrappers will be removed from SciPy in a
>>>> future release." So I would like to deprecate the routines abouve with a
>>>> message
>>>>
>>>> "The *XXXXX family of routines will have a different signature in the
>>>> future versions of  SciPy. The new versions are now available as
>>>> *XXXXX_full and will eventually replace the original *XXXXX family"
>>>>
>>>> Would you dis/agree?
>>>>
>>>
>>> Just wondering about the tradeoff between deprecating + having a new
>>> function with arguments in the same order as LAPACK, vs. just appending the
>>> keywords to the existing function. The latter case would need a thin Python
>>> function in lapack.py, but unless that hurts performance significantly I'm
>>> not sure that's much of an issue.
>>>
>>> Ralf
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>> On Mon, Dec 25, 2017 at 11:58 PM, <pav at iki.fi> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Ilhan Polat kirjoitti tiistai 26. joulukuuta 2017:
>>>>> [clip]
>>>>> > I've done a similar thing by freeing out all ins and outs. It seems
>>>>> to me
>>>>> > that the signature change is inevitable but the question is how to
>>>>> handle
>>>>> > this?
>>>>>
>>>>> New function sounds best, maximally unimaginative new name is probably
>>>>> most appropriate.
>>>>>
>>>>> Not sure how well np.deprecate etc python level wrappers work on f2py.
>>>>>
>>>>>
>>>>> > 2) It looks like flapack.src.pyf file is getting exceedingly long and
>>>>> > unstructured. Should we break it apart just like LAPACK
>>>>> documentation?
>>>>>
>>>>> If you think that helps it significantly and is worthwhile use of your
>>>>> time then why not. (I suspect it's not particularly worthwhile but ymmv.)
>>>>> _______________________________________________
>>>>> SciPy-Dev mailing list
>>>>> SciPy-Dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180120/83f2b756/attachment.html>

From ralf.gommers at gmail.com  Sat Jan 20 16:03:25 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 21 Jan 2018 10:03:25 +1300
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
Message-ID: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>

Hi all,

TL;DR, let's write the long journal paper on SciPy that we've wanted for a
while, let's form a small committee to coordinate, and get it out the door
in 2-3 months.


Motivation
---------------
(credits for most of this text: Evgeni)

Many scipy contributors' day jobs are in academia. Bibliometry -- papers in
refereed journals and citations of papers by other papers -- is one of the
main
performance indicators in most academic establishments. Since we do not
generate papers, scipy contributions are all but invisible for the purposes
of a
contributor's annual report. Of course, details vary wildly; in many cases a
contributor manages to balance their time, or to argue common sense with
their
superiors, or get an approval for scipy work, or just ignores the issue
altogether -- but sooner or later there is a form to be filled or boxes to
be
checked, and scipy contributions simply do not fit in. A peer-reviewed
journal paper on scipy will help contributors get the academic credit they
deserve.

We can write *the* paper for SciPy 1.0, with overall project structure,
goals, etc., and for specific features/modules a focus on say the last 3
years.


History
----------
For SciPy 1.0 we had three targets on the publicity/credits front: an
interesting release announcement, interesting blogs/stories (NumFOCUS blog,
Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in
the end, the rest was successful.

[1] is a previous announcement on this list about writing (a) paper(s) on
SciPy. We wanted both "short papers" to cover one or two releases (target
journal JOSS) and a full paper as the authoritative reference for SciPy.

We had an earlier attempt for a "short paper", it's mostly written but has
stalled (see [2]). We ran out of steam on that one. To avoid that this time
around, it would be good to have a clear public plan, target dates, and a
small committee rather than one person to drive things forward.


Proposal
------------
Here's a proposal for all aspects of this exercise that I can think about
now. Some parts stolen from the AstroPy paper [3] (because their process
worked quite well).

Form a small coordination committee of 3-5 people that set up the paper
structure, move things along when parts stall, propose/take decisions as
needed, invite co-authors, and organise paper submission/rework.

Paper writing to be done by whoever volunteers for a section, not just the
coordination committee. First outline/structure to be done by committee,
which then asks for review of structure and volunteers for section writing.

Scope: a 6-10 page paper, covering history, package scope and structure,
community/organisational aspects, key features and recent enhancements per
module, and roadmap.

Authorship: anyone who made a substantial contribution in the history of
the project. Here "substantial" is interpreted as anything beyond a
one-line doc fix. Rationale: better to be too inclusive than exclusive.
Sign-up via a web form, we send the link to that form to all email
addresses in the commit history till v1.0.

Author order (details tbd by committee):
1. The SciPy Developers
2. Maintainers, paper writers, other key contributors - in order of
contribution level
3. All other authors - alphabetically ordered

Submission target: mid-April, to either PeerJ Computer Science or Journal
of Open Research Software (tbd by committee).

Comments? Volunteers for committee?


References
----------------
[1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html
[2] https://github.com/scipy/scipy-articles/pull/4
[3] https://github.com/astropy/astropy-v2.0-paper

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180121/523826fe/attachment-0001.html>

From ralf.gommers at gmail.com  Sat Jan 20 16:03:06 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 21 Jan 2018 10:03:06 +1300
Subject: [SciPy-Dev] maintainership Sunday morning thoughts
Message-ID: <CABL7CQisSwcahONuBz-+LCpsW5EDy13rXJXZ4zOZLx8k8t1pQA@mail.gmail.com>

Hi all,

I wanted to share this excellent talk, "Rebuilding the cathedral", from
Nadia Eghbal about how open source software gets maintained:
https://www.youtube.com/watch?v=VS6IpvTWwkQ

If you're a maintainer that experiences feeling guilty about not answering
questions or reviewing PRs on Github quickly enough (I certainly do
sometimes), or are a contributor that wonders why your well crafted PR
doesn't get reviewed or merged, watching this talk may explain a few things.

In terms of "rewards" for maintainership of SciPy, we could do better. One
obvious thing is a paper that contributors can be co-authors on - this is
one thing that we planned but didn't manage to do in the rush to get SciPy
1.0 polished and out the door. I'm seeing a bit of free time coming up, and
fixing that omission is on my todo list for that time (really this time) -
second email to follow shortly. More substantial rewards ($$) is a bit of a
chicken-and-egg problem - it requires investing more time than anyone can
put in at the moment to apply for funding. It'll be interesting to see how
the dynamics of NumPy development change with the two grants for the next
two years...

Finally, a thank you to the people who've jumped in to fill the post-1.0
code review gap we seem to be experiencing - mainly Ilhan, Andrew and Tyler.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180121/44c5e45b/attachment.html>

From charlesr.harris at gmail.com  Sat Jan 20 19:29:09 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 20 Jan 2018 17:29:09 -0700
Subject: [SciPy-Dev] maintainership Sunday morning thoughts
In-Reply-To: <CABL7CQisSwcahONuBz-+LCpsW5EDy13rXJXZ4zOZLx8k8t1pQA@mail.gmail.com>
References: <CABL7CQisSwcahONuBz-+LCpsW5EDy13rXJXZ4zOZLx8k8t1pQA@mail.gmail.com>
Message-ID: <CAB6mnx+RjMnsRzZZL8TZMGip+QKH2sf-_UcRsCtwLikaJE2diQ@mail.gmail.com>

On Sat, Jan 20, 2018 at 2:03 PM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:

> Hi all,
>
> I wanted to share this excellent talk, "Rebuilding the cathedral", from
> Nadia Eghbal about how open source software gets maintained:
> https://www.youtube.com/watch?v=VS6IpvTWwkQ
>

Thanks for the link, good talk.

Considering the exponential growth in the number of users, I'm surprised
that NumPy/SciPy aren't completely swamped in bug reports. We're falling
behind, sure, but not as badly as might be expected.


> If you're a maintainer that experiences feeling guilty about not answering
> questions or reviewing PRs on Github quickly enough (I certainly do
> sometimes), or are a contributor that wonders why your well crafted PR
> doesn't get reviewed or merged, watching this talk may explain a few things.
>
> In terms of "rewards" for maintainership of SciPy, we could do better. One
> obvious thing is a paper that contributors can be co-authors on - this is
> one thing that we planned but didn't manage to do in the rush to get SciPy
> 1.0 polished and out the door. I'm seeing a bit of free time coming up, and
> fixing that omission is on my todo list for that time (really this time) -
> second email to follow shortly. More substantial rewards ($$) is a bit of a
> chicken-and-egg problem - it requires investing more time than anyone can
> put in at the moment to apply for funding. It'll be interesting to see how
> the dynamics of NumPy development change with the two grants for the next
> two years...
>
> Finally, a thank you to the people who've jumped in to fill the post-1.0
> code review gap we seem to be experiencing - mainly Ilhan, Andrew and Tyler.
>
>
Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180120/9515e6ba/attachment.html>

From josef.pktd at gmail.com  Sat Jan 20 19:51:46 2018
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 20 Jan 2018 19:51:46 -0500
Subject: [SciPy-Dev] maintainership Sunday morning thoughts
In-Reply-To: <CAB6mnx+RjMnsRzZZL8TZMGip+QKH2sf-_UcRsCtwLikaJE2diQ@mail.gmail.com>
References: <CABL7CQisSwcahONuBz-+LCpsW5EDy13rXJXZ4zOZLx8k8t1pQA@mail.gmail.com>
 <CAB6mnx+RjMnsRzZZL8TZMGip+QKH2sf-_UcRsCtwLikaJE2diQ@mail.gmail.com>
Message-ID: <CAMMTP+AkpZPaOsirpTVpYFErxbrS-Urrd=gc1tu21nmVT3=Z9g@mail.gmail.com>

On Sat, Jan 20, 2018 at 7:29 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Sat, Jan 20, 2018 at 2:03 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>
>> Hi all,
>>
>> I wanted to share this excellent talk, "Rebuilding the cathedral", from
>> Nadia Eghbal about how open source software gets maintained:
>> https://www.youtube.com/watch?v=VS6IpvTWwkQ
>>
>
> Thanks for the link, good talk.
>
> Considering the exponential growth in the number of users, I'm surprised
> that NumPy/SciPy aren't completely swamped in bug reports. We're falling
> behind, sure, but not as badly as might be expected.
>

I think it's because there is no "swamp of bugs".
I have the impression in keeping roughly track of scipy issues and PRs that
PR review and unit testing works very well so that only occasional bugs
escape and bug the users.

Josef


>
>
>> If you're a maintainer that experiences feeling guilty about not
>> answering questions or reviewing PRs on Github quickly enough (I certainly
>> do sometimes), or are a contributor that wonders why your well crafted PR
>> doesn't get reviewed or merged, watching this talk may explain a few things.
>>
>> In terms of "rewards" for maintainership of SciPy, we could do better.
>> One obvious thing is a paper that contributors can be co-authors on - this
>> is one thing that we planned but didn't manage to do in the rush to get
>> SciPy 1.0 polished and out the door. I'm seeing a bit of free time coming
>> up, and fixing that omission is on my todo list for that time (really this
>> time) - second email to follow shortly. More substantial rewards ($$) is a
>> bit of a chicken-and-egg problem - it requires investing more time than
>> anyone can put in at the moment to apply for funding. It'll be interesting
>> to see how the dynamics of NumPy development change with the two grants for
>> the next two years...
>>
>> Finally, a thank you to the people who've jumped in to fill the post-1.0
>> code review gap we seem to be experiencing - mainly Ilhan, Andrew and Tyler.
>>
>>
> Chuck
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180120/3dccba9b/attachment.html>

From ralf.gommers at gmail.com  Sat Jan 20 20:02:53 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 21 Jan 2018 14:02:53 +1300
Subject: [SciPy-Dev] maintainership Sunday morning thoughts
In-Reply-To: <CAMMTP+AkpZPaOsirpTVpYFErxbrS-Urrd=gc1tu21nmVT3=Z9g@mail.gmail.com>
References: <CABL7CQisSwcahONuBz-+LCpsW5EDy13rXJXZ4zOZLx8k8t1pQA@mail.gmail.com>
 <CAB6mnx+RjMnsRzZZL8TZMGip+QKH2sf-_UcRsCtwLikaJE2diQ@mail.gmail.com>
 <CAMMTP+AkpZPaOsirpTVpYFErxbrS-Urrd=gc1tu21nmVT3=Z9g@mail.gmail.com>
Message-ID: <CABL7CQgamgMSbR5f30gH1H=04ikFDoAYbYe9cByS6wut6RW8Gw@mail.gmail.com>

On Sun, Jan 21, 2018 at 1:51 PM, <josef.pktd at gmail.com> wrote:

>
>
> On Sat, Jan 20, 2018 at 7:29 PM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Sat, Jan 20, 2018 at 2:03 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I wanted to share this excellent talk, "Rebuilding the cathedral", from
>>> Nadia Eghbal about how open source software gets maintained:
>>> https://www.youtube.com/watch?v=VS6IpvTWwkQ
>>>
>>
>> Thanks for the link, good talk.
>>
>> Considering the exponential growth in the number of users, I'm surprised
>> that NumPy/SciPy aren't completely swamped in bug reports. We're falling
>> behind, sure, but not as badly as might be expected.
>>
>
> I think it's because there is no "swamp of bugs".
> I have the impression in keeping roughly track of scipy issues and PRs
> that PR review and unit testing works very well so that only occasional
> bugs escape and bug the users.
>

Agreed, there's no swamp of bugs. Maybe a swamp of rough edges....

What we do struggle with is lack of progress on big-ticket items, e.g.
  - sparse arrays instead of (or in addition to) sparse matrices
  - replacing all the spline implementations (in interpolate, signal,
ndimage) with a single clean design
  - making ndimage consistent for the "is it a pixel or is it a data point"
question
Basically the stuff one can't do piece by piece but requires someone to sit
down for a couple of months. That requires luck (a person with the right
skills and motivation and time coming along) or funding.

Ralf



> Josef
>
>
>>
>>
>>> If you're a maintainer that experiences feeling guilty about not
>>> answering questions or reviewing PRs on Github quickly enough (I certainly
>>> do sometimes), or are a contributor that wonders why your well crafted PR
>>> doesn't get reviewed or merged, watching this talk may explain a few things.
>>>
>>> In terms of "rewards" for maintainership of SciPy, we could do better.
>>> One obvious thing is a paper that contributors can be co-authors on - this
>>> is one thing that we planned but didn't manage to do in the rush to get
>>> SciPy 1.0 polished and out the door. I'm seeing a bit of free time coming
>>> up, and fixing that omission is on my todo list for that time (really this
>>> time) - second email to follow shortly. More substantial rewards ($$) is a
>>> bit of a chicken-and-egg problem - it requires investing more time than
>>> anyone can put in at the moment to apply for funding. It'll be interesting
>>> to see how the dynamics of NumPy development change with the two grants for
>>> the next two years...
>>>
>>> Finally, a thank you to the people who've jumped in to fill the post-1.0
>>> code review gap we seem to be experiencing - mainly Ilhan, Andrew and Tyler.
>>>
>>>
>> Chuck
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180121/b14333d7/attachment-0001.html>

From josef.pktd at gmail.com  Sat Jan 20 20:20:16 2018
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 20 Jan 2018 20:20:16 -0500
Subject: [SciPy-Dev] maintainership Sunday morning thoughts
In-Reply-To: <CABL7CQgamgMSbR5f30gH1H=04ikFDoAYbYe9cByS6wut6RW8Gw@mail.gmail.com>
References: <CABL7CQisSwcahONuBz-+LCpsW5EDy13rXJXZ4zOZLx8k8t1pQA@mail.gmail.com>
 <CAB6mnx+RjMnsRzZZL8TZMGip+QKH2sf-_UcRsCtwLikaJE2diQ@mail.gmail.com>
 <CAMMTP+AkpZPaOsirpTVpYFErxbrS-Urrd=gc1tu21nmVT3=Z9g@mail.gmail.com>
 <CABL7CQgamgMSbR5f30gH1H=04ikFDoAYbYe9cByS6wut6RW8Gw@mail.gmail.com>
Message-ID: <CAMMTP+DQcXaO7HWQr9fgAEY8O+KyfuzX77RWK2xUbOSter2bXg@mail.gmail.com>

On Sat, Jan 20, 2018 at 8:02 PM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:

>
>
> On Sun, Jan 21, 2018 at 1:51 PM, <josef.pktd at gmail.com> wrote:
>
>>
>>
>> On Sat, Jan 20, 2018 at 7:29 PM, Charles R Harris <
>> charlesr.harris at gmail.com> wrote:
>>
>>>
>>>
>>> On Sat, Jan 20, 2018 at 2:03 PM, Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I wanted to share this excellent talk, "Rebuilding the cathedral", from
>>>> Nadia Eghbal about how open source software gets maintained:
>>>> https://www.youtube.com/watch?v=VS6IpvTWwkQ
>>>>
>>>
>>> Thanks for the link, good talk.
>>>
>>> Considering the exponential growth in the number of users, I'm surprised
>>> that NumPy/SciPy aren't completely swamped in bug reports. We're falling
>>> behind, sure, but not as badly as might be expected.
>>>
>>
>> I think it's because there is no "swamp of bugs".
>> I have the impression in keeping roughly track of scipy issues and PRs
>> that PR review and unit testing works very well so that only occasional
>> bugs escape and bug the users.
>>
>
> Agreed, there's no swamp of bugs. Maybe a swamp of rough edges....
>
> What we do struggle with is lack of progress on big-ticket items, e.g.
>   - sparse arrays instead of (or in addition to) sparse matrices
>   - replacing all the spline implementations (in interpolate, signal,
> ndimage) with a single clean design
>   - making ndimage consistent for the "is it a pixel or is it a data
> point" question
>

and at least differential equations, AFAIK as bystander


> Basically the stuff one can't do piece by piece but requires someone to
> sit down for a couple of months. That requires luck (a person with the
> right skills and motivation and time coming along) or funding.
>

I agree with all the requirements. However, it also needs some difficult
design work by experts or experienced developers.
E.g. How can general interpolating splines, smoothing splines, splines on a
one dimensional grid and splines on a nd dimensional grid be combined into
a single clean design without loosing ... (sounds between tough and
impossible to me)

Josef


>
>
> Ralf
>
>
>
>> Josef
>>
>>
>>>
>>>
>>>> If you're a maintainer that experiences feeling guilty about not
>>>> answering questions or reviewing PRs on Github quickly enough (I certainly
>>>> do sometimes), or are a contributor that wonders why your well crafted PR
>>>> doesn't get reviewed or merged, watching this talk may explain a few things.
>>>>
>>>> In terms of "rewards" for maintainership of SciPy, we could do better.
>>>> One obvious thing is a paper that contributors can be co-authors on - this
>>>> is one thing that we planned but didn't manage to do in the rush to get
>>>> SciPy 1.0 polished and out the door. I'm seeing a bit of free time coming
>>>> up, and fixing that omission is on my todo list for that time (really this
>>>> time) - second email to follow shortly. More substantial rewards ($$) is a
>>>> bit of a chicken-and-egg problem - it requires investing more time than
>>>> anyone can put in at the moment to apply for funding. It'll be interesting
>>>> to see how the dynamics of NumPy development change with the two grants for
>>>> the next two years...
>>>>
>>>> Finally, a thank you to the people who've jumped in to fill the
>>>> post-1.0 code review gap we seem to be experiencing - mainly Ilhan, Andrew
>>>> and Tyler.
>>>>
>>>>
>>> Chuck
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180120/e19a03e1/attachment.html>

From ralf.gommers at gmail.com  Sat Jan 20 20:34:59 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 21 Jan 2018 14:34:59 +1300
Subject: [SciPy-Dev] Using Pythran to compile some of the scipy internals
In-Reply-To: <20180120101034.GA3707@lakota>
References: <20180120101034.GA3707@lakota>
Message-ID: <CABL7CQgr8AWjoQD7TV+tcb9a5wcfnrf+ttCb-hGhZKWha0WUXQ@mail.gmail.com>

On Sat, Jan 20, 2018 at 11:10 PM, Serge Guelton <serge.guelton at telecom-
bretagne.eu> wrote:

> Hi Scipy-dev,
>
> = Context
>
> I've started the experiment of compiling some of the scipy kernels with
> the Pythran[0] compiler.
>
> The approach was basically to pick some easy .pyx file, turn them back
> to high-level Python + Numpy files then processing them with Pythran.
> This results in easier to maintain, Python-compatible files that run at
> the same speed as their Cython alternative up to 10 times faster
> depending on the kernels and whether SIMD instructions are activated or
> not (I did not play with parallelization in that experiment).
>
> This required a few improvement to Pythran, but not that much, see
>
>     https://github.com/serge-sans-paille/pythran/pull/770
>
> for the associated PR on pythran, which includes the following scipy
> kernels:
>
>     hausdorff.py
>     max_len_seq_inner.py
>     solve_toeplitz.py
>     spectral.py
>
> = Why bother?
>
> According to that first round of experiments, using Pythran can bring
> extra speed improvements while remaining at a higher level than Cython.
> Both points looks like an improvement to me.
>

Another potential benefit is to decrease the size of the binary
distribution of SciPy. Cython extensions are quite expensive in this
respect. Do you have an idea of how Pythran compares? Both in case of only
float64 inputs, and with templated inputs? A good example of the latter is
scipy.ndimage.label, which is a straightforward function that ends up being
a 700kb .so


> = Cost?
>
> This adds an extra dependency on Pythran, which uses C++ as backend.
>

Only a build-time dependency. That's not my major worry from the
maintenance point of view.


> This increases the failure surface. Although alive since 2012 and being
> tested a lot [1] on Linux (but scarcely on Windows), its is obviously
> less mature than cython
>

This is a worry. We need Windows, Linux, macOS (officially supported, also
the 32-bit flavors) as well as less commonly used Unix/BSD-like platforms.

I guess Windows needs separate testing, both with gcc and MSVC. But for all
other platforms, can you say something about portability based on the kind
of C++ Pythran generates?


> = Alternatives
>
> There is an experimental Pythran mode in cython[2] that uses Pythran as
> a backend for numpy operations. Unfortunately it is still at early
> stages and cannot translate calls to ``np.roll`` or ``np.sum(a[i,:] - b[j,
> :])`` while Pythran supports it.
>
> Instead of translating Cython files, I could also focus on some
> pure-python functions. I tested Pythran on the rosenbrock function and I
> get good speedup (from 1.5x to 4x depending on vectorization being
> enabled or not) there too.
>
> So yeah, that's a rather long introduction to probe the interest here
> around that idea :-)
>

Interest in general, but there's a long way to go - our requirements are
pretty demanding. I have seen some comparisons between Cython, Pythran and
Numba in terms of performance and ease of use, but never a comprehensive
comparison from the point of view of library authors. I know Travis O. has
an interest in seeing Numba being adopted more widely, which will also need
such a comparison. It should cover at least:

- portability
- performance
- maturity
- maintenance status (active devs, how quick do bugs get fixed after a
release with an issue)
- ease of use (@jit vs. Pythran comments vs. translate to .pyx syntax)
- size of generated binaries
- templating support for multiple dtypes
- debugging and optimization experience/tools

Is anyone aware of such a comparison? Or interested in putting it together?

Cheers,
Ralf



> Best,
> Serge
>
> PS: I started https://github.com/scipy/scipy/pull/8306 meanwhile, but
> let's discuss here first :-)
>
>
>
> [0] https://github.com/serge-sans-paille/pythran
> [1] https://travis-ci.org/serge-sans-paille/pythran/builds/330052310
> [2] http://cython.readthedocs.io/en/latest/src/userguide/numpy_p
> ythran.html
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180121/ffb2c3b9/attachment-0001.html>

From tyler.je.reddy at gmail.com  Sat Jan 20 22:08:21 2018
From: tyler.je.reddy at gmail.com (Tyler Reddy)
Date: Sat, 20 Jan 2018 20:08:21 -0700
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
Message-ID: <CAHPuU_b_Hk3tqf7AbjKb8PjTqZFsJzXqUfXPghwA7WNbEmWSzQ@mail.gmail.com>

Sounds good -- I'd be up for committee work. Definitely +1 on a paper for
time justification, etc.

On 20 January 2018 at 14:03, Ralf Gommers <ralf.gommers at gmail.com> wrote:

> Hi all,
>
> TL;DR, let's write the long journal paper on SciPy that we've wanted for a
> while, let's form a small committee to coordinate, and get it out the door
> in 2-3 months.
>
>
> Motivation
> ---------------
> (credits for most of this text: Evgeni)
>
> Many scipy contributors' day jobs are in academia. Bibliometry -- papers in
> refereed journals and citations of papers by other papers -- is one of the
> main
> performance indicators in most academic establishments. Since we do not
> generate papers, scipy contributions are all but invisible for the
> purposes of a
> contributor's annual report. Of course, details vary wildly; in many cases
> a
> contributor manages to balance their time, or to argue common sense with
> their
> superiors, or get an approval for scipy work, or just ignores the issue
> altogether -- but sooner or later there is a form to be filled or boxes to
> be
> checked, and scipy contributions simply do not fit in. A peer-reviewed
> journal paper on scipy will help contributors get the academic credit they
> deserve.
>
> We can write *the* paper for SciPy 1.0, with overall project structure,
> goals, etc., and for specific features/modules a focus on say the last 3
> years.
>
>
> History
> ----------
> For SciPy 1.0 we had three targets on the publicity/credits front: an
> interesting release announcement, interesting blogs/stories (NumFOCUS blog,
> Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in
> the end, the rest was successful.
>
> [1] is a previous announcement on this list about writing (a) paper(s) on
> SciPy. We wanted both "short papers" to cover one or two releases (target
> journal JOSS) and a full paper as the authoritative reference for SciPy.
>
> We had an earlier attempt for a "short paper", it's mostly written but has
> stalled (see [2]). We ran out of steam on that one. To avoid that this time
> around, it would be good to have a clear public plan, target dates, and a
> small committee rather than one person to drive things forward.
>
>
> Proposal
> ------------
> Here's a proposal for all aspects of this exercise that I can think about
> now. Some parts stolen from the AstroPy paper [3] (because their process
> worked quite well).
>
> Form a small coordination committee of 3-5 people that set up the paper
> structure, move things along when parts stall, propose/take decisions as
> needed, invite co-authors, and organise paper submission/rework.
>
> Paper writing to be done by whoever volunteers for a section, not just the
> coordination committee. First outline/structure to be done by committee,
> which then asks for review of structure and volunteers for section writing.
>
> Scope: a 6-10 page paper, covering history, package scope and structure,
> community/organisational aspects, key features and recent enhancements per
> module, and roadmap.
>
> Authorship: anyone who made a substantial contribution in the history of
> the project. Here "substantial" is interpreted as anything beyond a
> one-line doc fix. Rationale: better to be too inclusive than exclusive.
> Sign-up via a web form, we send the link to that form to all email
> addresses in the commit history till v1.0.
>
> Author order (details tbd by committee):
> 1. The SciPy Developers
> 2. Maintainers, paper writers, other key contributors - in order of
> contribution level
> 3. All other authors - alphabetically ordered
>
> Submission target: mid-April, to either PeerJ Computer Science or Journal
> of Open Research Software (tbd by committee).
>
> Comments? Volunteers for committee?
>
>
> References
> ----------------
> [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html
> [2] https://github.com/scipy/scipy-articles/pull/4
> [3] https://github.com/astropy/astropy-v2.0-paper
>
> Cheers,
> Ralf
>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180120/09c02274/attachment.html>

From warren.weckesser at gmail.com  Sun Jan 21 12:36:16 2018
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sun, 21 Jan 2018 12:36:16 -0500
Subject: [SciPy-Dev] Adding an argument to 'odeint' to control the order of
 the first two arguments of the user's function.
Message-ID: <CAGzF1uc83v+6c8pBabXzXa-Kup2YUA=1aUjRfbbguwV-45OQ=g@mail.gmail.com>

In 'scipy.integrate', the 'ode' class and the function 'solve_ivp' assume
that first two arguments of the functions provided by the user are 't'
("time") followed by 'y' (the current state).  The older solver, 'odeint',
expects those arguments to be in the opposite order.  This is a nuisance
for anyone who wants to use the same functions with the different solvers.

To fix this, I've created a pull request in which the argument 'tfirst' is
added to 'odeint': https://github.com/scipy/scipy/pull/8318. When 'tfirst'
is True, 'odeint' uses the same convention as 'solve_ivp' and the 'ode'
class.  The change is implemented in the C code to avoid the overhead that
would result from using a function wrapper with reversed arguments.

Comments on the pull request are welcome.

Warren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180121/cf5aa3f2/attachment.html>

From andyfaff at gmail.com  Sun Jan 21 15:26:53 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Mon, 22 Jan 2018 07:26:53 +1100
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <CAHPuU_b_Hk3tqf7AbjKb8PjTqZFsJzXqUfXPghwA7WNbEmWSzQ@mail.gmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <CAHPuU_b_Hk3tqf7AbjKb8PjTqZFsJzXqUfXPghwA7WNbEmWSzQ@mail.gmail.com>
Message-ID: <CAAbtOZeh_CaM_LRAHm43Jyu-=3r98_v1f2dbS0majTcXEE0Pcg@mail.gmail.com>

I'm obviously up for being on the paper :-), but I'd prefer to contribute
by writing small sections/editing of a paper rather than coordinating.

On 21 January 2018 at 14:08, Tyler Reddy <tyler.je.reddy at gmail.com> wrote:

> Sounds good -- I'd be up for committee work. Definitely +1 on a paper for
> time justification, etc.
>
> On 20 January 2018 at 14:03, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>> Hi all,
>>
>> TL;DR, let's write the long journal paper on SciPy that we've wanted for
>> a while, let's form a small committee to coordinate, and get it out the
>> door in 2-3 months.
>>
>>
>> Motivation
>> ---------------
>> (credits for most of this text: Evgeni)
>>
>> Many scipy contributors' day jobs are in academia. Bibliometry -- papers
>> in
>> refereed journals and citations of papers by other papers -- is one of
>> the main
>> performance indicators in most academic establishments. Since we do not
>> generate papers, scipy contributions are all but invisible for the
>> purposes of a
>> contributor's annual report. Of course, details vary wildly; in many
>> cases a
>> contributor manages to balance their time, or to argue common sense with
>> their
>> superiors, or get an approval for scipy work, or just ignores the issue
>> altogether -- but sooner or later there is a form to be filled or boxes
>> to be
>> checked, and scipy contributions simply do not fit in. A peer-reviewed
>> journal paper on scipy will help contributors get the academic credit they
>> deserve.
>>
>> We can write *the* paper for SciPy 1.0, with overall project structure,
>> goals, etc., and for specific features/modules a focus on say the last 3
>> years.
>>
>>
>> History
>> ----------
>> For SciPy 1.0 we had three targets on the publicity/credits front: an
>> interesting release announcement, interesting blogs/stories (NumFOCUS blog,
>> Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in
>> the end, the rest was successful.
>>
>> [1] is a previous announcement on this list about writing (a) paper(s) on
>> SciPy. We wanted both "short papers" to cover one or two releases (target
>> journal JOSS) and a full paper as the authoritative reference for SciPy.
>>
>> We had an earlier attempt for a "short paper", it's mostly written but
>> has stalled (see [2]). We ran out of steam on that one. To avoid that this
>> time around, it would be good to have a clear public plan, target dates,
>> and a small committee rather than one person to drive things forward.
>>
>>
>> Proposal
>> ------------
>> Here's a proposal for all aspects of this exercise that I can think about
>> now. Some parts stolen from the AstroPy paper [3] (because their process
>> worked quite well).
>>
>> Form a small coordination committee of 3-5 people that set up the paper
>> structure, move things along when parts stall, propose/take decisions as
>> needed, invite co-authors, and organise paper submission/rework.
>>
>> Paper writing to be done by whoever volunteers for a section, not just
>> the coordination committee. First outline/structure to be done by
>> committee, which then asks for review of structure and volunteers for
>> section writing.
>>
>> Scope: a 6-10 page paper, covering history, package scope and structure,
>> community/organisational aspects, key features and recent enhancements per
>> module, and roadmap.
>>
>> Authorship: anyone who made a substantial contribution in the history of
>> the project. Here "substantial" is interpreted as anything beyond a
>> one-line doc fix. Rationale: better to be too inclusive than exclusive.
>> Sign-up via a web form, we send the link to that form to all email
>> addresses in the commit history till v1.0.
>>
>> Author order (details tbd by committee):
>> 1. The SciPy Developers
>> 2. Maintainers, paper writers, other key contributors - in order of
>> contribution level
>> 3. All other authors - alphabetically ordered
>>
>> Submission target: mid-April, to either PeerJ Computer Science or Journal
>> of Open Research Software (tbd by committee).
>>
>> Comments? Volunteers for committee?
>>
>>
>> References
>> ----------------
>> [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html
>> [2] https://github.com/scipy/scipy-articles/pull/4
>> [3] https://github.com/astropy/astropy-v2.0-paper
>>
>> Cheers,
>> Ralf
>>
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>


-- 
_____________________________________
Dr. Andrew Nelson


_____________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180122/f9376370/attachment.html>

From ralf.gommers at gmail.com  Mon Jan 22 01:14:22 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 22 Jan 2018 19:14:22 +1300
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <CAAbtOZeh_CaM_LRAHm43Jyu-=3r98_v1f2dbS0majTcXEE0Pcg@mail.gmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <CAHPuU_b_Hk3tqf7AbjKb8PjTqZFsJzXqUfXPghwA7WNbEmWSzQ@mail.gmail.com>
 <CAAbtOZeh_CaM_LRAHm43Jyu-=3r98_v1f2dbS0majTcXEE0Pcg@mail.gmail.com>
Message-ID: <CABL7CQht4jLtkeYbTouaT5A6F0RBcygr+HWPbHYmxF42-AQVRA@mail.gmail.com>

On Mon, Jan 22, 2018 at 9:26 AM, Andrew Nelson <andyfaff at gmail.com> wrote:

> I'm obviously up for being on the paper :-), but I'd prefer to contribute
> by writing small sections/editing of a paper rather than coordinating.
>

No worries, help of any kind is welcome, and no obligations of course.


> On 21 January 2018 at 14:08, Tyler Reddy <tyler.je.reddy at gmail.com> wrote:
>
>> Sounds good -- I'd be up for committee work. Definitely +1 on a paper for
>> time justification, etc.
>>
>
Thanks Tyler!

Ralf



>> On 20 January 2018 at 14:03, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> TL;DR, let's write the long journal paper on SciPy that we've wanted for
>>> a while, let's form a small committee to coordinate, and get it out the
>>> door in 2-3 months.
>>>
>>>
>>> Motivation
>>> ---------------
>>> (credits for most of this text: Evgeni)
>>>
>>> Many scipy contributors' day jobs are in academia. Bibliometry -- papers
>>> in
>>> refereed journals and citations of papers by other papers -- is one of
>>> the main
>>> performance indicators in most academic establishments. Since we do not
>>> generate papers, scipy contributions are all but invisible for the
>>> purposes of a
>>> contributor's annual report. Of course, details vary wildly; in many
>>> cases a
>>> contributor manages to balance their time, or to argue common sense with
>>> their
>>> superiors, or get an approval for scipy work, or just ignores the issue
>>> altogether -- but sooner or later there is a form to be filled or boxes
>>> to be
>>> checked, and scipy contributions simply do not fit in. A peer-reviewed
>>> journal paper on scipy will help contributors get the academic credit they
>>> deserve.
>>>
>>> We can write *the* paper for SciPy 1.0, with overall project structure,
>>> goals, etc., and for specific features/modules a focus on say the last 3
>>> years.
>>>
>>>
>>> History
>>> ----------
>>> For SciPy 1.0 we had three targets on the publicity/credits front: an
>>> interesting release announcement, interesting blogs/stories (NumFOCUS blog,
>>> Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in
>>> the end, the rest was successful.
>>>
>>> [1] is a previous announcement on this list about writing (a) paper(s)
>>> on SciPy. We wanted both "short papers" to cover one or two releases
>>> (target journal JOSS) and a full paper as the authoritative reference for
>>> SciPy.
>>>
>>> We had an earlier attempt for a "short paper", it's mostly written but
>>> has stalled (see [2]). We ran out of steam on that one. To avoid that this
>>> time around, it would be good to have a clear public plan, target dates,
>>> and a small committee rather than one person to drive things forward.
>>>
>>>
>>> Proposal
>>> ------------
>>> Here's a proposal for all aspects of this exercise that I can think
>>> about now. Some parts stolen from the AstroPy paper [3] (because their
>>> process worked quite well).
>>>
>>> Form a small coordination committee of 3-5 people that set up the paper
>>> structure, move things along when parts stall, propose/take decisions as
>>> needed, invite co-authors, and organise paper submission/rework.
>>>
>>> Paper writing to be done by whoever volunteers for a section, not just
>>> the coordination committee. First outline/structure to be done by
>>> committee, which then asks for review of structure and volunteers for
>>> section writing.
>>>
>>> Scope: a 6-10 page paper, covering history, package scope and structure,
>>> community/organisational aspects, key features and recent enhancements per
>>> module, and roadmap.
>>>
>>> Authorship: anyone who made a substantial contribution in the history of
>>> the project. Here "substantial" is interpreted as anything beyond a
>>> one-line doc fix. Rationale: better to be too inclusive than exclusive.
>>> Sign-up via a web form, we send the link to that form to all email
>>> addresses in the commit history till v1.0.
>>>
>>> Author order (details tbd by committee):
>>> 1. The SciPy Developers
>>> 2. Maintainers, paper writers, other key contributors - in order of
>>> contribution level
>>> 3. All other authors - alphabetically ordered
>>>
>>> Submission target: mid-April, to either PeerJ Computer Science or
>>> Journal of Open Research Software (tbd by committee).
>>>
>>> Comments? Volunteers for committee?
>>>
>>>
>>> References
>>> ----------------
>>> [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html
>>> [2] https://github.com/scipy/scipy-articles/pull/4
>>> [3] https://github.com/astropy/astropy-v2.0-paper
>>>
>>> Cheers,
>>> Ralf
>>>
>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
>
> --
> _____________________________________
> Dr. Andrew Nelson
>
>
> _____________________________________
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180122/440107f9/attachment-0001.html>

From antonior92 at gmail.com  Mon Jan 22 06:09:30 2018
From: antonior92 at gmail.com (Antonio Ribeiro)
Date: Mon, 22 Jan 2018 09:09:30 -0200
Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 171, Issue 19
In-Reply-To: <mailman.10.1516482208.2751.scipy-dev@python.org>
References: <mailman.10.1516482208.2751.scipy-dev@python.org>
Message-ID: <CAFStupGs7iSqSU6hi=mf2KDLgXQr_9-U9PFDaorAA3-K2B5z4w@mail.gmail.com>

Hi all,

Just to understand it better. I have some contributions before v1.0:
https://github.com/scipy/scipy/pull/6404/commits/caa58db0a0695ec1cc0636ed85903449fef5d57a
https://github.com/scipy/scipy/pull/6919/commits/e0e7e55502fb6fa0142b2f173f8de6b73b254594

Should I have received a web form?

If it is the case, I should be happy to help writing about
"scipy.optimize", if the committee decide to include some section about it.

All the best,
Ant?nio

2018-01-20 19:03 GMT-02:00 <scipy-dev-request at python.org>:
>
> Date: Sun, 21 Jan 2018 10:03:25 +1300
> From: Ralf Gommers <ralf.gommers at gmail.com>
> To: SciPy Developers List <scipy-dev at python.org>
> Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
> Message-ID:
>         <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuK
> bzKpos3WQ at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi all,
>
> TL;DR, let's write the long journal paper on SciPy that we've wanted for a
> while, let's form a small committee to coordinate, and get it out the door
> in 2-3 months.
>
>
> Motivation
> ---------------
> (credits for most of this text: Evgeni)
>
> Many scipy contributors' day jobs are in academia. Bibliometry -- papers in
> refereed journals and citations of papers by other papers -- is one of the
> main
> performance indicators in most academic establishments. Since we do not
> generate papers, scipy contributions are all but invisible for the purposes
> of a
> contributor's annual report. Of course, details vary wildly; in many cases
> a
> contributor manages to balance their time, or to argue common sense with
> their
> superiors, or get an approval for scipy work, or just ignores the issue
> altogether -- but sooner or later there is a form to be filled or boxes to
> be
> checked, and scipy contributions simply do not fit in. A peer-reviewed
> journal paper on scipy will help contributors get the academic credit they
> deserve.
>
> We can write *the* paper for SciPy 1.0, with overall project structure,
> goals, etc., and for specific features/modules a focus on say the last 3
> years.
>
>
> History
> ----------
> For SciPy 1.0 we had three targets on the publicity/credits front: an
> interesting release announcement, interesting blogs/stories (NumFOCUS blog,
> Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in
> the end, the rest was successful.
>
> [1] is a previous announcement on this list about writing (a) paper(s) on
> SciPy. We wanted both "short papers" to cover one or two releases (target
> journal JOSS) and a full paper as the authoritative reference for SciPy.
>
> We had an earlier attempt for a "short paper", it's mostly written but has
> stalled (see [2]). We ran out of steam on that one. To avoid that this time
> around, it would be good to have a clear public plan, target dates, and a
> small committee rather than one person to drive things forward.
>
>
> Proposal
> ------------
> Here's a proposal for all aspects of this exercise that I can think about
> now. Some parts stolen from the AstroPy paper [3] (because their process
> worked quite well).
>
> Form a small coordination committee of 3-5 people that set up the paper
> structure, move things along when parts stall, propose/take decisions as
> needed, invite co-authors, and organise paper submission/rework.
>
> Paper writing to be done by whoever volunteers for a section, not just the
> coordination committee. First outline/structure to be done by committee,
> which then asks for review of structure and volunteers for section writing.
>
> Scope: a 6-10 page paper, covering history, package scope and structure,
> community/organisational aspects, key features and recent enhancements per
> module, and roadmap.
>
> Authorship: anyone who made a substantial contribution in the history of
> the project. Here "substantial" is interpreted as anything beyond a
> one-line doc fix. Rationale: better to be too inclusive than exclusive.
> Sign-up via a web form, we send the link to that form to all email
> addresses in the commit history till v1.0.
>
> Author order (details tbd by committee):
> 1. The SciPy Developers
> 2. Maintainers, paper writers, other key contributors - in order of
> contribution level
> 3. All other authors - alphabetically ordered
>
> Submission target: mid-April, to either PeerJ Computer Science or Journal
> of Open Research Software (tbd by committee).
>
> Comments? Volunteers for committee?
>
>
> References
> ----------------
> [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html
> [2] https://github.com/scipy/scipy-articles/pull/4
> [3] https://github.com/astropy/astropy-v2.0-paper
>
> Cheers,
> Ralf
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/scipy-dev/
> attachments/20180121/523826fe/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
> ------------------------------
>
> End of SciPy-Dev Digest, Vol 171, Issue 19
> ******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180122/c9c1f3d5/attachment.html>

From s.denaxas at gmail.com  Mon Jan 22 07:33:21 2018
From: s.denaxas at gmail.com (Spiros Denaxas)
Date: Mon, 22 Jan 2018 12:33:21 +0000
Subject: [SciPy-Dev] New feature - discrete Frechet distance for
 scipy.spatial.distance
Message-ID: <CAO6Zhu0Rx9Z2AncQwkKNzETasjiFdaaM+u389QHcYtRYPUmMww@mail.gmail.com>

Dear SciPy devs,

I recently decided to clean up and package a Python implementation [1] for
calculating the discrete Frechet distance [2] between two polygonal curves
that I had lying around.

I was thinking this could potentially be a useful addition to
scipy.spatial.distance. More than happy to do a pull request if you think
it will be useful.

Spiros

[1] https://github.com/spiros/discrete_frechet
[2] http://www.kr.tuwien.ac.at/staff/eiter/et-archive/cdtr9464.pdf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180122/9b9c0122/attachment.html>

From serge.guelton at telecom-bretagne.eu  Mon Jan 22 09:28:40 2018
From: serge.guelton at telecom-bretagne.eu (Serge Guelton)
Date: Mon, 22 Jan 2018 15:28:40 +0100
Subject: [SciPy-Dev] Using Pythran to compile some of the scipy internals
In-Reply-To: <CABL7CQgr8AWjoQD7TV+tcb9a5wcfnrf+ttCb-hGhZKWha0WUXQ@mail.gmail.com>
References: <20180120101034.GA3707@lakota>
 <CABL7CQgr8AWjoQD7TV+tcb9a5wcfnrf+ttCb-hGhZKWha0WUXQ@mail.gmail.com>
Message-ID: <20180122142840.GA22566@lakota>

> Another potential benefit is to decrease the size of the binary distribution of
> SciPy. Cython extensions are quite expensive in this respect. Do you have an
> idea of how Pythran compares? Both in case of only float64 inputs, and with
> templated inputs? A good example of the latter is scipy.ndimage.label, which is
> a straightforward function that ends up being a 700kb .so

Thanks for pointing out this aspect. There's still room for improvement
here, but with current pythran's master, and same compilation flags, the
cython implementation of `max_len_seq_inner` takes ~700kb while
pythran's version takes ~200kb.

I'm going to investigate that a bet more though, and probably post the
results in this thread later on.

>     This adds an extra dependency on Pythran, which uses C++ as backend.
> 
> 
> Only a build-time dependency. That's not my major worry from the maintenance
> point of view.

And a dependency to libstc++ too.

>     This increases the failure surface. Although alive since 2012 and being
>     tested a lot [1] on Linux (but scarcely on Windows), its is obviously
>     less mature than cython
> 
> 
> This is a worry. We need Windows, Linux, macOS (officially supported, also the
> 32-bit flavors) as well as less commonly used Unix/BSD-like platforms.
> 
> I guess Windows needs separate testing, both with gcc and MSVC. But for all
> other platforms, can you say something about portability based on the kind of
> C++ Pythran generates?

Portability on OSX never required extra work, it's just less tested than
the Linux version. There was some C++11 support issue with MSVC compiler
back when I tried, but that was some 2 years ago. I'll try again.


>     = Alternatives
> 
>     There is an experimental Pythran mode in cython[2] that uses Pythran as
>     a backend for numpy operations. Unfortunately it is still at early
>     stages and cannot translate calls to ``np.roll`` or ``np.sum(a[i,:] - b[j,
>     :])`` while Pythran supports it.
> 
>     Instead of translating Cython files, I could also focus on some
>     pure-python functions. I tested Pythran on the rosenbrock function and I
>     get good speedup (from 1.5x to 4x depending on vectorization being
>     enabled or not) there too.
> 
>     So yeah, that's a rather long introduction to probe the interest here
>     around that idea :-)
> 
> 
> Interest in general, but there's a long way to go - our requirements are pretty
> demanding. I have seen some comparisons between Cython, Pythran and Numba in
> terms of performance and ease of use, but never a comprehensive comparison from
> the point of view of library authors. I know Travis O. has an interest in
> seeing Numba being adopted more widely, which will also need such a comparison.
> It should cover at least:
> 
> - portability
> - performance
> - maturity
> - maintenance status (active devs, how quick do bugs get fixed after a release
> with an issue)
> - ease of use (@jit vs. Pythran comments vs. translate to .pyx syntax)
> - size of generated binaries
> - templating support for multiple dtypes
> - debugging and optimization experience/tools

Independently of the potential inclusion of Pythran in scipy, those are
very valuable points. I'm 100% biased but would say that community and
debugging are two weak points of Pythran, esp. when compared to Cython.

++
Serge

From tyler.je.reddy at gmail.com  Mon Jan 22 10:14:48 2018
From: tyler.je.reddy at gmail.com (Tyler Reddy)
Date: Mon, 22 Jan 2018 08:14:48 -0700
Subject: [SciPy-Dev] New feature - discrete Frechet distance for
 scipy.spatial.distance
In-Reply-To: <CAO6Zhu0Rx9Z2AncQwkKNzETasjiFdaaM+u389QHcYtRYPUmMww@mail.gmail.com>
References: <CAO6Zhu0Rx9Z2AncQwkKNzETasjiFdaaM+u389QHcYtRYPUmMww@mail.gmail.com>
Message-ID: <CAHPuU_Y7fm1va8bomtcHxn6Qsx6DiDqkeHijTQQ2zSTEASG_Gw@mail.gmail.com>

There are a few Python implementations of discrete Frechet around (i.e., in
MDA:
https://www.mdanalysis.org/docs/documentation_pages/analysis/psa.html#MDAnalysis.analysis.psa.discrete_frechet
). Most of them use the quadratic time complexity dynamic programming
approach, and it seems you have used that as well. I think there's probably
interest in incorporating discrete Frechet in scipy since we already have
directed_hausdorff, which is thematically similar.

One thing that would be really impressive is if we could get a subquadratic
implementation of discrete Frechet ( i.e.,
http://epubs.siam.org/doi/abs/10.1137/130920526 ), but these algorithms
seem to be extraordinarily complicated -- indeed the authors of the paper I
cited there are not aware of any working implementations in the wild & when
I asked one I think they suggested it wasn't worth implementing because for
most input data sizes the much simpler algorithm will still win. Perhaps
this sort of thing could be summarized in the docstring.

There are quite a few different flavors of Frechet distance (continuous,
homotopic, weak, etc.). From a design standpoint it might makes sense to
have some kind of keyword to allow expansion for those in the future.

On 22 January 2018 at 05:33, Spiros Denaxas <s.denaxas at gmail.com> wrote:

> Dear SciPy devs,
>
> I recently decided to clean up and package a Python implementation [1] for
> calculating the discrete Frechet distance [2] between two polygonal curves
> that I had lying around.
>
> I was thinking this could potentially be a useful addition to
> scipy.spatial.distance. More than happy to do a pull request if you think
> it will be useful.
>
> Spiros
>
> [1] https://github.com/spiros/discrete_frechet
> [2] http://www.kr.tuwien.ac.at/staff/eiter/et-archive/cdtr9464.pdf
>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180122/29bb634a/attachment.html>

From davidmenhur at gmail.com  Mon Jan 22 10:34:32 2018
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Mon, 22 Jan 2018 16:34:32 +0100
Subject: [SciPy-Dev] Using Pythran to compile some of the scipy internals
In-Reply-To: <20180120101034.GA3707@lakota>
References: <20180120101034.GA3707@lakota>
Message-ID: <CAJhcF=2t_3zHeE_vXsZgSX7QRmNYdA-ippOFsgq4bQt=k+8k2g@mail.gmail.com>

On 20 January 2018 at 11:10, Serge Guelton <serge.guelton at telecom-
bretagne.eu> wrote:

> = Cost?
>
> This adds an extra dependency on Pythran, which uses C++ as backend.
> This increases the failure surface. Although alive since 2012 and being
> tested a lot [1] on Linux (but scarcely on Windows), its is obviously
> less mature than cython
>

I think this is a blocker, but it seems it is on its way to be fixed:

"Pythran supports Python *2.7* and also has a beta Python *3* support."

scipy has a lot of Cython code, much of it fairly well tested, which can be
very useful for testing Pythran itself.

A question, does Pythran eliminate the need for storing intermediate
results? Like (a*a).sum(axis=1) for a large a, will it store the full
a-squared? (If it is, I'd be very sorry I didn't run into it earlier, and
very happy I finally did).


/David.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180122/972000be/attachment.html>

From ralf.gommers at gmail.com  Mon Jan 22 11:00:34 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 23 Jan 2018 05:00:34 +1300
Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 171, Issue 19
In-Reply-To: <CAFStupGs7iSqSU6hi=mf2KDLgXQr_9-U9PFDaorAA3-K2B5z4w@mail.gmail.com>
References: <mailman.10.1516482208.2751.scipy-dev@python.org>
 <CAFStupGs7iSqSU6hi=mf2KDLgXQr_9-U9PFDaorAA3-K2B5z4w@mail.gmail.com>
Message-ID: <CABL7CQgku0z=yrCYp47zPft5U+GZS0+R3hC4pxnVgq=uGo0nuA@mail.gmail.com>

On Tue, Jan 23, 2018 at 12:09 AM, Antonio Ribeiro <antonior92 at gmail.com>
wrote:

> Hi all,
>
> Just to understand it better. I have some contributions before v1.0:
> https://github.com/scipy/scipy/pull/6404/commits/
> caa58db0a0695ec1cc0636ed85903449fef5d57a
> https://github.com/scipy/scipy/pull/6919/commits/
> e0e7e55502fb6fa0142b2f173f8de6b73b254594
>
> Should I have received a web form?
>

Not yet because we haven't created it yet. But yes, the plan is that you
will receive a link to a form on the email address in the commit history.


> If it is the case, I should be happy to help writing about
> "scipy.optimize", if the committee decide to include some section about it.
>

Thanks!

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180123/8260e858/attachment.html>

From serge.guelton at telecom-bretagne.eu  Mon Jan 22 17:34:57 2018
From: serge.guelton at telecom-bretagne.eu (Serge Guelton)
Date: Mon, 22 Jan 2018 23:34:57 +0100
Subject: [SciPy-Dev] Using Pythran to compile some of the scipy internals
In-Reply-To: <CAJhcF=2t_3zHeE_vXsZgSX7QRmNYdA-ippOFsgq4bQt=k+8k2g@mail.gmail.com>
References: <20180120101034.GA3707@lakota>
 <CAJhcF=2t_3zHeE_vXsZgSX7QRmNYdA-ippOFsgq4bQt=k+8k2g@mail.gmail.com>
Message-ID: <20180122223457.GA14580@lakota>

On Mon, Jan 22, 2018 at 04:34:32PM +0100, Da?id wrote:
> 
> On 20 January 2018 at 11:10, Serge Guelton <serge.guelton at telecom-bretagne.eu>
> wrote:
> 
>     = Cost?
> 
>     This adds an extra dependency on Pythran, which uses C++ as backend.
>     This increases the failure surface. Although alive since 2012 and being
>     tested a lot [1] on Linux (but scarcely on Windows), its is obviously
>     less mature than cython
> 
> 
> I think this is a blocker, but it seems it is on its way to be fixed:
> 
> "Pythran supports Python 2.7 and also has a beta Python 3 support."
> 
> scipy has a lot of Cython code, much of it fairly well tested, which can be
> very useful for testing Pythran itself.

Yes, I started integrating cython code from scipy into pythran testbed,
after converting them back to Python. It did found some bugs in Pythran,
so that's great!

> A question, does Pythran eliminate the need for storing intermediate results?

Yep, it turns code like

    a = b + 1
    c = a * 2

into

    c = (b + 1) * 2

It of course also handles interaction with control flow to
forward-substitute only if it does not triggers recomputation.

> Like (a*a).sum(axis=1) for a large a, will it store the full a-squared? (If it
> is, I'd be very sorry I didn't run into it earlier, and very happy I finally
> did).

Yes, it does perfom lazy evaluation.

From rlucente at pipeline.com  Mon Jan 22 18:25:39 2018
From: rlucente at pipeline.com (Robert Lucente)
Date: Mon, 22 Jan 2018 18:25:39 -0500 (GMT-05:00)
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
Message-ID: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>

>What we do struggle with is lack of progress on big-ticket items

Could we get some of these "big-ticket" items to be graduate student projects as part of an independent study or ...?

From andyfaff at gmail.com  Mon Jan 22 23:15:21 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Tue, 23 Jan 2018 15:15:21 +1100
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
Message-ID: <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>

On 23 January 2018 at 10:25, Robert Lucente <rlucente at pipeline.com> wrote:

> >What we do struggle with is lack of progress on big-ticket items
>
> Could we get some of these "big-ticket" items to be graduate student
> projects as part of an independent study or ...?
>

The difficult in progressing big ticket items looms large, the difficulty
here cannot be underestimated. Big ticket items require several iterations
which tests the patience of the proposer and reviewer, it's a large amount
of effort on both ends. It requires the reviewer is familiar with the
subject material (not a given), and that the proposer is familiar with the
scipy workflow. Giving it to a grad student isn't necessarily going to
solve either of those issues.

I know on my part that I've been reluctant to review some PR's because I
don't feel I have enough knowledge, and worry I would commit something that
isn't good enough. But if there's no one else to do it, then what should
happen?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180123/579fe16b/attachment.html>

From s.denaxas at gmail.com  Tue Jan 23 04:38:31 2018
From: s.denaxas at gmail.com (Spiros Denaxas)
Date: Tue, 23 Jan 2018 09:38:31 +0000
Subject: [SciPy-Dev] New feature - discrete Frechet distance for
 scipy.spatial.distance
In-Reply-To: <CAHPuU_Y7fm1va8bomtcHxn6Qsx6DiDqkeHijTQQ2zSTEASG_Gw@mail.gmail.com>
References: <CAO6Zhu0Rx9Z2AncQwkKNzETasjiFdaaM+u389QHcYtRYPUmMww@mail.gmail.com>
 <CAHPuU_Y7fm1va8bomtcHxn6Qsx6DiDqkeHijTQQ2zSTEASG_Gw@mail.gmail.com>
Message-ID: <CAO6Zhu02dhNqK71taxs07RVagxCXoUya1H75W_Aa7VJHC1ny8w@mail.gmail.com>

Hi Tyler

Thank you for the feedback. I agree that a subquadratic implementation
would be really cool but I havent seen one either and just by glancing at
the paper realize that this would take a substantial amount of effort.

What is the canonical way of specying the flavour ? the two options I can
think of are: a) scipy.spatial.distance.continuous_frechet or b)
scipy.spatial.distance.frechet(P,Q, type="continuous")

best
Spiros


On Mon, Jan 22, 2018 at 3:14 PM, Tyler Reddy <tyler.je.reddy at gmail.com>
wrote:

> There are a few Python implementations of discrete Frechet around (i.e.,
> in MDA: https://www.mdanalysis.org/docs/documentation_pages/
> analysis/psa.html#MDAnalysis.analysis.psa.discrete_frechet ). Most of
> them use the quadratic time complexity dynamic programming approach, and it
> seems you have used that as well. I think there's probably interest in
> incorporating discrete Frechet in scipy since we already have
> directed_hausdorff, which is thematically similar.
>
> One thing that would be really impressive is if we could get a
> subquadratic implementation of discrete Frechet ( i.e.,
> http://epubs.siam.org/doi/abs/10.1137/130920526 ), but these algorithms
> seem to be extraordinarily complicated -- indeed the authors of the paper I
> cited there are not aware of any working implementations in the wild & when
> I asked one I think they suggested it wasn't worth implementing because for
> most input data sizes the much simpler algorithm will still win. Perhaps
> this sort of thing could be summarized in the docstring.
>
> There are quite a few different flavors of Frechet distance (continuous,
> homotopic, weak, etc.). From a design standpoint it might makes sense to
> have some kind of keyword to allow expansion for those in the future.
>
> On 22 January 2018 at 05:33, Spiros Denaxas <s.denaxas at gmail.com> wrote:
>
>> Dear SciPy devs,
>>
>> I recently decided to clean up and package a Python implementation [1]
>> for calculating the discrete Frechet distance [2] between two polygonal
>> curves that I had lying around.
>>
>> I was thinking this could potentially be a useful addition to
>> scipy.spatial.distance. More than happy to do a pull request if you think
>> it will be useful.
>>
>> Spiros
>>
>> [1] https://github.com/spiros/discrete_frechet
>> [2] http://www.kr.tuwien.ac.at/staff/eiter/et-archive/cdtr9464.pdf
>>
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180123/aea5e070/attachment.html>

From rlucente at pipeline.com  Tue Jan 23 04:57:01 2018
From: rlucente at pipeline.com (Robert Lucente - Pipeline.Com)
Date: Tue, 23 Jan 2018 04:57:01 -0500
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
Message-ID: <14f101d39430$84b127e0$8e1377a0$@pipeline.com>

>good enough

I am a newbie and have just been lurking.

 

Perhaps we could put some bounds around unacceptable, good enough and perfection? I realize that it is hard to put words to this. Perhaps an example to kick off a conversation?

 

Perhaps we could release features w/ tags like alpha, beta, GA and part of official release?

 

Perhaps people could volunteer to just help someone w/ the SciPy workflow?

 

Suggesting some ideas to stimulate conversation.

 

From: Andrew Nelson [mailto:andyfaff at gmail.com] 
Sent: Monday, January 22, 2018 11:15 PM
To: Robert Lucente <rlucente at pipeline.com>; SciPy Developers List <scipy-dev at python.org>
Subject: Re: [SciPy-Dev] What we do struggle with is lack of progress on big-ticket items

 

On 23 January 2018 at 10:25, Robert Lucente <rlucente at pipeline.com <mailto:rlucente at pipeline.com> > wrote:

>What we do struggle with is lack of progress on big-ticket items

Could we get some of these "big-ticket" items to be graduate student projects as part of an independent study or ...?

 

The difficult in progressing big ticket items looms large, the difficulty here cannot be underestimated. Big ticket items require several iterations which tests the patience of the proposer and reviewer, it's a large amount of effort on both ends. It requires the reviewer is familiar with the subject material (not a given), and that the proposer is familiar with the scipy workflow. Giving it to a grad student isn't necessarily going to solve either of those issues.

 

I know on my part that I've been reluctant to review some PR's because I don't feel I have enough knowledge, and worry I would commit something that isn't good enough. But if there's no one else to do it, then what should happen?

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180123/7ff8fdc5/attachment-0001.html>

From tyler.je.reddy at gmail.com  Tue Jan 23 10:27:35 2018
From: tyler.je.reddy at gmail.com (Tyler Reddy)
Date: Tue, 23 Jan 2018 08:27:35 -0700
Subject: [SciPy-Dev] New feature - discrete Frechet distance for
 scipy.spatial.distance
In-Reply-To: <CAO6Zhu02dhNqK71taxs07RVagxCXoUya1H75W_Aa7VJHC1ny8w@mail.gmail.com>
References: <CAO6Zhu0Rx9Z2AncQwkKNzETasjiFdaaM+u389QHcYtRYPUmMww@mail.gmail.com>
 <CAHPuU_Y7fm1va8bomtcHxn6Qsx6DiDqkeHijTQQ2zSTEASG_Gw@mail.gmail.com>
 <CAO6Zhu02dhNqK71taxs07RVagxCXoUya1H75W_Aa7VJHC1ny8w@mail.gmail.com>
Message-ID: <CAHPuU_aOhWdnojLqKKq9ma8WiH4o4=CY=-a0Td=Vu7VBGAxApQ@mail.gmail.com>

I guess both call signatures have tradeoffs. What I did with
directed_hausdorff looks more like "a," so you can probably start with that
for now. I'd encourage you to start drafting a (work in progress) pull
request -- you can progressively chip away at docstring, unit tests,
cythonization (if this is helpful or you haven't already), maybe a concise
asv performance benchmark to accompany it, etc.

On 23 January 2018 at 02:38, Spiros Denaxas <s.denaxas at gmail.com> wrote:

> Hi Tyler
>
> Thank you for the feedback. I agree that a subquadratic implementation
> would be really cool but I havent seen one either and just by glancing at
> the paper realize that this would take a substantial amount of effort.
>
> What is the canonical way of specying the flavour ? the two options I can
> think of are: a) scipy.spatial.distance.continuous_frechet or b)
> scipy.spatial.distance.frechet(P,Q, type="continuous")
>
> best
> Spiros
>
>
> On Mon, Jan 22, 2018 at 3:14 PM, Tyler Reddy <tyler.je.reddy at gmail.com>
> wrote:
>
>> There are a few Python implementations of discrete Frechet around (i.e.,
>> in MDA: https://www.mdanalysis.org/docs/documentation_pages/analysis
>> /psa.html#MDAnalysis.analysis.psa.discrete_frechet ). Most of them use
>> the quadratic time complexity dynamic programming approach, and it seems
>> you have used that as well. I think there's probably interest in
>> incorporating discrete Frechet in scipy since we already have
>> directed_hausdorff, which is thematically similar.
>>
>> One thing that would be really impressive is if we could get a
>> subquadratic implementation of discrete Frechet ( i.e.,
>> http://epubs.siam.org/doi/abs/10.1137/130920526 ), but these algorithms
>> seem to be extraordinarily complicated -- indeed the authors of the paper I
>> cited there are not aware of any working implementations in the wild & when
>> I asked one I think they suggested it wasn't worth implementing because for
>> most input data sizes the much simpler algorithm will still win. Perhaps
>> this sort of thing could be summarized in the docstring.
>>
>> There are quite a few different flavors of Frechet distance (continuous,
>> homotopic, weak, etc.). From a design standpoint it might makes sense to
>> have some kind of keyword to allow expansion for those in the future.
>>
>> On 22 January 2018 at 05:33, Spiros Denaxas <s.denaxas at gmail.com> wrote:
>>
>>> Dear SciPy devs,
>>>
>>> I recently decided to clean up and package a Python implementation [1]
>>> for calculating the discrete Frechet distance [2] between two polygonal
>>> curves that I had lying around.
>>>
>>> I was thinking this could potentially be a useful addition to
>>> scipy.spatial.distance. More than happy to do a pull request if you think
>>> it will be useful.
>>>
>>> Spiros
>>>
>>> [1] https://github.com/spiros/discrete_frechet
>>> [2] http://www.kr.tuwien.ac.at/staff/eiter/et-archive/cdtr9464.pdf
>>>
>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180123/00cffc8c/attachment.html>

From s.denaxas at gmail.com  Tue Jan 23 11:41:51 2018
From: s.denaxas at gmail.com (Spiros Denaxas)
Date: Tue, 23 Jan 2018 16:41:51 +0000
Subject: [SciPy-Dev] New feature - discrete Frechet distance for
 scipy.spatial.distance
In-Reply-To: <CAHPuU_aOhWdnojLqKKq9ma8WiH4o4=CY=-a0Td=Vu7VBGAxApQ@mail.gmail.com>
References: <CAO6Zhu0Rx9Z2AncQwkKNzETasjiFdaaM+u389QHcYtRYPUmMww@mail.gmail.com>
 <CAHPuU_Y7fm1va8bomtcHxn6Qsx6DiDqkeHijTQQ2zSTEASG_Gw@mail.gmail.com>
 <CAO6Zhu02dhNqK71taxs07RVagxCXoUya1H75W_Aa7VJHC1ny8w@mail.gmail.com>
 <CAHPuU_aOhWdnojLqKKq9ma8WiH4o4=CY=-a0Td=Vu7VBGAxApQ@mail.gmail.com>
Message-ID: <CAO6Zhu1KepKjiB5qLU+mX23bYFU=950U2fHwpCyGgK+nY-4mYQ@mail.gmail.com>

Thanks - will do.
Spiros

On Tue, Jan 23, 2018 at 3:27 PM, Tyler Reddy <tyler.je.reddy at gmail.com>
wrote:

> I guess both call signatures have tradeoffs. What I did with
> directed_hausdorff looks more like "a," so you can probably start with that
> for now. I'd encourage you to start drafting a (work in progress) pull
> request -- you can progressively chip away at docstring, unit tests,
> cythonization (if this is helpful or you haven't already), maybe a concise
> asv performance benchmark to accompany it, etc.
>
> On 23 January 2018 at 02:38, Spiros Denaxas <s.denaxas at gmail.com> wrote:
>
>> Hi Tyler
>>
>> Thank you for the feedback. I agree that a subquadratic implementation
>> would be really cool but I havent seen one either and just by glancing at
>> the paper realize that this would take a substantial amount of effort.
>>
>> What is the canonical way of specying the flavour ? the two options I can
>> think of are: a) scipy.spatial.distance.continuous_frechet or b)
>> scipy.spatial.distance.frechet(P,Q, type="continuous")
>>
>> best
>> Spiros
>>
>>
>> On Mon, Jan 22, 2018 at 3:14 PM, Tyler Reddy <tyler.je.reddy at gmail.com>
>> wrote:
>>
>>> There are a few Python implementations of discrete Frechet around (i.e.,
>>> in MDA: https://www.mdanalysis.org/docs/documentation_pages/analysis
>>> /psa.html#MDAnalysis.analysis.psa.discrete_frechet ). Most of them use
>>> the quadratic time complexity dynamic programming approach, and it seems
>>> you have used that as well. I think there's probably interest in
>>> incorporating discrete Frechet in scipy since we already have
>>> directed_hausdorff, which is thematically similar.
>>>
>>> One thing that would be really impressive is if we could get a
>>> subquadratic implementation of discrete Frechet ( i.e.,
>>> http://epubs.siam.org/doi/abs/10.1137/130920526 ), but these algorithms
>>> seem to be extraordinarily complicated -- indeed the authors of the paper I
>>> cited there are not aware of any working implementations in the wild & when
>>> I asked one I think they suggested it wasn't worth implementing because for
>>> most input data sizes the much simpler algorithm will still win. Perhaps
>>> this sort of thing could be summarized in the docstring.
>>>
>>> There are quite a few different flavors of Frechet distance (continuous,
>>> homotopic, weak, etc.). From a design standpoint it might makes sense to
>>> have some kind of keyword to allow expansion for those in the future.
>>>
>>> On 22 January 2018 at 05:33, Spiros Denaxas <s.denaxas at gmail.com> wrote:
>>>
>>>> Dear SciPy devs,
>>>>
>>>> I recently decided to clean up and package a Python implementation [1]
>>>> for calculating the discrete Frechet distance [2] between two polygonal
>>>> curves that I had lying around.
>>>>
>>>> I was thinking this could potentially be a useful addition to
>>>> scipy.spatial.distance. More than happy to do a pull request if you think
>>>> it will be useful.
>>>>
>>>> Spiros
>>>>
>>>> [1] https://github.com/spiros/discrete_frechet
>>>> [2] http://www.kr.tuwien.ac.at/staff/eiter/et-archive/cdtr9464.pdf
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180123/95ee3b27/attachment-0001.html>

From stefanv at berkeley.edu  Tue Jan 23 14:20:38 2018
From: stefanv at berkeley.edu (Stefan van der Walt)
Date: Tue, 23 Jan 2018 11:20:38 -0800
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
 <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
Message-ID: <20180123192038.mgenugikitd2x4wy@fastmail.com>

Hi Robert

On Tue, 23 Jan 2018 04:57:01 -0500, Robert Lucente - Pipeline.Com wrote:
> I am a newbie and have just been lurking.

Thank you for raising your thoughts here; it's so helpful to hear
perspectives from outside the established developer community.

> Perhaps we could put some bounds around unacceptable, good enough and
> perfection? I realize that it is hard to put words to this. Perhaps an
> example to kick off a conversation?

It's a tricky balance.  My feeling is that a person should lower the
barrier to contribution as much as possible: avoid unnecessary technical
challenges like git rebasing, push to PRs to guide new contributors, and
provide good documentation.

That said, I think it's better to educate than to adjust standards.
Experience has taught that introducing code of insufficient quality into
the code base inevitably leads to headaches later on (we all have a lot
of hypothetical time to fix things up in the future, right?).

> Perhaps people could volunteer to just help someone w/ the SciPy
> workflow?

This is an excellent suggestion, and one we've also been considering for
scikit-image.  The idea of having existing developers acting as mentors
is often what happens informally, but it may be helpful to establish
more obvious ways for that to take place.  Do you have any thoughts on
how this could look?

Best regards
St?fan

From charlesr.harris at gmail.com  Tue Jan 23 14:56:50 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 23 Jan 2018 12:56:50 -0700
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <20180123192038.mgenugikitd2x4wy@fastmail.com>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
 <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
 <20180123192038.mgenugikitd2x4wy@fastmail.com>
Message-ID: <CAB6mnxLQqcdnwGi8nURH=q3nYd--+XJsZMq3R8yisj=6cJDokA@mail.gmail.com>

On Tue, Jan 23, 2018 at 12:20 PM, Stefan van der Walt <stefanv at berkeley.edu>
wrote:

> Hi Robert
>
> On Tue, 23 Jan 2018 04:57:01 -0500, Robert Lucente - Pipeline.Com wrote:
> > I am a newbie and have just been lurking.
>
> Thank you for raising your thoughts here; it's so helpful to hear
> perspectives from outside the established developer community.
>
> > Perhaps we could put some bounds around unacceptable, good enough and
> > perfection? I realize that it is hard to put words to this. Perhaps an
> > example to kick off a conversation?
>
> It's a tricky balance.  My feeling is that a person should lower the
> barrier to contribution as much as possible: avoid unnecessary technical
> challenges like git rebasing, push to PRs to guide new contributors, and
> provide good documentation.
>
> That said, I think it's better to educate than to adjust standards.
> Experience has taught that introducing code of insufficient quality into
> the code base inevitably leads to headaches later on (we all have a lot
> of hypothetical time to fix things up in the future, right?).
>
> > Perhaps people could volunteer to just help someone w/ the SciPy
> > workflow?
>
> This is an excellent suggestion, and one we've also been considering for
> scikit-image.  The idea of having existing developers acting as mentors
> is often what happens informally, but it may be helpful to establish
> more obvious ways for that to take place.  Do you have any thoughts on
> how this could look?
>

I know there have been several presentations on the workflow, Jamie Frio
did a couple. It might useful to see if anyone has made video's of a
tutorial like that so we could put it up on youtube or some other place.
Maybe NumFocus could have a youtube channel with such resources.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180123/d5e64529/attachment.html>

From perimosocordiae at gmail.com  Tue Jan 23 15:01:38 2018
From: perimosocordiae at gmail.com (CJ Carey)
Date: Tue, 23 Jan 2018 15:01:38 -0500
Subject: [SciPy-Dev] New feature - discrete Frechet distance for
 scipy.spatial.distance
In-Reply-To: <CAO6Zhu1KepKjiB5qLU+mX23bYFU=950U2fHwpCyGgK+nY-4mYQ@mail.gmail.com>
References: <CAO6Zhu0Rx9Z2AncQwkKNzETasjiFdaaM+u389QHcYtRYPUmMww@mail.gmail.com>
 <CAHPuU_Y7fm1va8bomtcHxn6Qsx6DiDqkeHijTQQ2zSTEASG_Gw@mail.gmail.com>
 <CAO6Zhu02dhNqK71taxs07RVagxCXoUya1H75W_Aa7VJHC1ny8w@mail.gmail.com>
 <CAHPuU_aOhWdnojLqKKq9ma8WiH4o4=CY=-a0Td=Vu7VBGAxApQ@mail.gmail.com>
 <CAO6Zhu1KepKjiB5qLU+mX23bYFU=950U2fHwpCyGgK+nY-4mYQ@mail.gmail.com>
Message-ID: <CAEfGn+zGX1i4oRJqAcdTfA0eH69ONvOu2Dj0H9v=CNW6MMHAGA@mail.gmail.com>

Frechet distance is pretty similar to Dynamic Time Warping, which has also
been requested for inclusion in scipy:
https://github.com/scipy/scipy/issues/7638

AFAIK, there's no current effort to add DTW, but I suspect it would make
sense to have both it and Frechet variants in a similar namespace.

On Tue, Jan 23, 2018 at 11:41 AM, Spiros Denaxas <s.denaxas at gmail.com>
wrote:

> Thanks - will do.
> Spiros
>
> On Tue, Jan 23, 2018 at 3:27 PM, Tyler Reddy <tyler.je.reddy at gmail.com>
> wrote:
>
>> I guess both call signatures have tradeoffs. What I did with
>> directed_hausdorff looks more like "a," so you can probably start with that
>> for now. I'd encourage you to start drafting a (work in progress) pull
>> request -- you can progressively chip away at docstring, unit tests,
>> cythonization (if this is helpful or you haven't already), maybe a concise
>> asv performance benchmark to accompany it, etc.
>>
>> On 23 January 2018 at 02:38, Spiros Denaxas <s.denaxas at gmail.com> wrote:
>>
>>> Hi Tyler
>>>
>>> Thank you for the feedback. I agree that a subquadratic implementation
>>> would be really cool but I havent seen one either and just by glancing at
>>> the paper realize that this would take a substantial amount of effort.
>>>
>>> What is the canonical way of specying the flavour ? the two options I
>>> can think of are: a) scipy.spatial.distance.continuous_frechet or b)
>>> scipy.spatial.distance.frechet(P,Q, type="continuous")
>>>
>>> best
>>> Spiros
>>>
>>>
>>> On Mon, Jan 22, 2018 at 3:14 PM, Tyler Reddy <tyler.je.reddy at gmail.com>
>>> wrote:
>>>
>>>> There are a few Python implementations of discrete Frechet around
>>>> (i.e., in MDA: https://www.mdanalysis.org/doc
>>>> s/documentation_pages/analysis/psa.html#MDAnalysis.analysis.
>>>> psa.discrete_frechet ). Most of them use the quadratic time complexity
>>>> dynamic programming approach, and it seems you have used that as well. I
>>>> think there's probably interest in incorporating discrete Frechet in scipy
>>>> since we already have directed_hausdorff, which is thematically similar.
>>>>
>>>> One thing that would be really impressive is if we could get a
>>>> subquadratic implementation of discrete Frechet ( i.e.,
>>>> http://epubs.siam.org/doi/abs/10.1137/130920526 ), but these
>>>> algorithms seem to be extraordinarily complicated -- indeed the authors of
>>>> the paper I cited there are not aware of any working implementations in the
>>>> wild & when I asked one I think they suggested it wasn't worth implementing
>>>> because for most input data sizes the much simpler algorithm will still
>>>> win. Perhaps this sort of thing could be summarized in the docstring.
>>>>
>>>> There are quite a few different flavors of Frechet distance
>>>> (continuous, homotopic, weak, etc.). From a design standpoint it might
>>>> makes sense to have some kind of keyword to allow expansion for those in
>>>> the future.
>>>>
>>>> On 22 January 2018 at 05:33, Spiros Denaxas <s.denaxas at gmail.com>
>>>> wrote:
>>>>
>>>>> Dear SciPy devs,
>>>>>
>>>>> I recently decided to clean up and package a Python implementation [1]
>>>>> for calculating the discrete Frechet distance [2] between two polygonal
>>>>> curves that I had lying around.
>>>>>
>>>>> I was thinking this could potentially be a useful addition to
>>>>> scipy.spatial.distance. More than happy to do a pull request if you think
>>>>> it will be useful.
>>>>>
>>>>> Spiros
>>>>>
>>>>> [1] https://github.com/spiros/discrete_frechet
>>>>> [2] http://www.kr.tuwien.ac.at/staff/eiter/et-archive/cdtr9464.pdf
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-Dev mailing list
>>>>> SciPy-Dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>
>>>>>
>>>>
>>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180123/236c429e/attachment-0001.html>

From tyler.je.reddy at gmail.com  Tue Jan 23 15:05:53 2018
From: tyler.je.reddy at gmail.com (Tyler Reddy)
Date: Tue, 23 Jan 2018 13:05:53 -0700
Subject: [SciPy-Dev] New feature - discrete Frechet distance for
 scipy.spatial.distance
In-Reply-To: <CAEfGn+zGX1i4oRJqAcdTfA0eH69ONvOu2Dj0H9v=CNW6MMHAGA@mail.gmail.com>
References: <CAO6Zhu0Rx9Z2AncQwkKNzETasjiFdaaM+u389QHcYtRYPUmMww@mail.gmail.com>
 <CAHPuU_Y7fm1va8bomtcHxn6Qsx6DiDqkeHijTQQ2zSTEASG_Gw@mail.gmail.com>
 <CAO6Zhu02dhNqK71taxs07RVagxCXoUya1H75W_Aa7VJHC1ny8w@mail.gmail.com>
 <CAHPuU_aOhWdnojLqKKq9ma8WiH4o4=CY=-a0Td=Vu7VBGAxApQ@mail.gmail.com>
 <CAO6Zhu1KepKjiB5qLU+mX23bYFU=950U2fHwpCyGgK+nY-4mYQ@mail.gmail.com>
 <CAEfGn+zGX1i4oRJqAcdTfA0eH69ONvOu2Dj0H9v=CNW6MMHAGA@mail.gmail.com>
Message-ID: <CAHPuU_ZM5qG7wZqKs+HkzrOMm-9iorEnGn6ikz5awzbdViK-+g@mail.gmail.com>

Oh yeah, I forgot about that proposal!

Here's what the first author of the subquadratic Frechet paper had to say
when I asked him about implementations (hopefully they don't mind me
sharing!):

"I am not aware of such a code, and I am not sure whether it's worth the
effort for a variety of reasons. n^2 algorithm is so simple that it will
perform significantly better even for polygonal chains with tens of
thousands of vertices.

In practice, dynamic time warping does a better job in measuring
similarity  and outside CG community  (eg. computer vision, compuational
biology, speech recognition) DTW is often used instead of Frechet.

I have a paper in the ACM-GIS conference this year on approximation
algorithms for DTW, with an implementation.
The approx algorithm performs better than the quadratic DP algorithm when
the number of vertices is a few thousand or more.

-Pankaj"

On 23 January 2018 at 13:01, CJ Carey <perimosocordiae at gmail.com> wrote:

> Frechet distance is pretty similar to Dynamic Time Warping, which has also
> been requested for inclusion in scipy: https://github.com/
> scipy/scipy/issues/7638
>
> AFAIK, there's no current effort to add DTW, but I suspect it would make
> sense to have both it and Frechet variants in a similar namespace.
>
> On Tue, Jan 23, 2018 at 11:41 AM, Spiros Denaxas <s.denaxas at gmail.com>
> wrote:
>
>> Thanks - will do.
>> Spiros
>>
>> On Tue, Jan 23, 2018 at 3:27 PM, Tyler Reddy <tyler.je.reddy at gmail.com>
>> wrote:
>>
>>> I guess both call signatures have tradeoffs. What I did with
>>> directed_hausdorff looks more like "a," so you can probably start with that
>>> for now. I'd encourage you to start drafting a (work in progress) pull
>>> request -- you can progressively chip away at docstring, unit tests,
>>> cythonization (if this is helpful or you haven't already), maybe a concise
>>> asv performance benchmark to accompany it, etc.
>>>
>>> On 23 January 2018 at 02:38, Spiros Denaxas <s.denaxas at gmail.com> wrote:
>>>
>>>> Hi Tyler
>>>>
>>>> Thank you for the feedback. I agree that a subquadratic implementation
>>>> would be really cool but I havent seen one either and just by glancing at
>>>> the paper realize that this would take a substantial amount of effort.
>>>>
>>>> What is the canonical way of specying the flavour ? the two options I
>>>> can think of are: a) scipy.spatial.distance.continuous_frechet or b)
>>>> scipy.spatial.distance.frechet(P,Q, type="continuous")
>>>>
>>>> best
>>>> Spiros
>>>>
>>>>
>>>> On Mon, Jan 22, 2018 at 3:14 PM, Tyler Reddy <tyler.je.reddy at gmail.com>
>>>> wrote:
>>>>
>>>>> There are a few Python implementations of discrete Frechet around
>>>>> (i.e., in MDA: https://www.mdanalysis.org/doc
>>>>> s/documentation_pages/analysis/psa.html#MDAnalysis.analysis.
>>>>> psa.discrete_frechet ). Most of them use the quadratic time
>>>>> complexity dynamic programming approach, and it seems you have used that as
>>>>> well. I think there's probably interest in incorporating discrete Frechet
>>>>> in scipy since we already have directed_hausdorff, which is thematically
>>>>> similar.
>>>>>
>>>>> One thing that would be really impressive is if we could get a
>>>>> subquadratic implementation of discrete Frechet ( i.e.,
>>>>> http://epubs.siam.org/doi/abs/10.1137/130920526 ), but these
>>>>> algorithms seem to be extraordinarily complicated -- indeed the authors of
>>>>> the paper I cited there are not aware of any working implementations in the
>>>>> wild & when I asked one I think they suggested it wasn't worth implementing
>>>>> because for most input data sizes the much simpler algorithm will still
>>>>> win. Perhaps this sort of thing could be summarized in the docstring.
>>>>>
>>>>> There are quite a few different flavors of Frechet distance
>>>>> (continuous, homotopic, weak, etc.). From a design standpoint it might
>>>>> makes sense to have some kind of keyword to allow expansion for those in
>>>>> the future.
>>>>>
>>>>> On 22 January 2018 at 05:33, Spiros Denaxas <s.denaxas at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Dear SciPy devs,
>>>>>>
>>>>>> I recently decided to clean up and package a Python implementation
>>>>>> [1] for calculating the discrete Frechet distance [2] between two polygonal
>>>>>> curves that I had lying around.
>>>>>>
>>>>>> I was thinking this could potentially be a useful addition to
>>>>>> scipy.spatial.distance. More than happy to do a pull request if you think
>>>>>> it will be useful.
>>>>>>
>>>>>> Spiros
>>>>>>
>>>>>> [1] https://github.com/spiros/discrete_frechet
>>>>>> [2] http://www.kr.tuwien.ac.at/staff/eiter/et-archive/cdtr9464.pdf
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> SciPy-Dev mailing list
>>>>>> SciPy-Dev at python.org
>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180123/3003405b/attachment.html>

From stefanv at berkeley.edu  Tue Jan 23 15:22:18 2018
From: stefanv at berkeley.edu (Stefan van der Walt)
Date: Tue, 23 Jan 2018 12:22:18 -0800
Subject: [SciPy-Dev] New feature - discrete Frechet distance for
 scipy.spatial.distance
In-Reply-To: <CAEfGn+zGX1i4oRJqAcdTfA0eH69ONvOu2Dj0H9v=CNW6MMHAGA@mail.gmail.com>
References: <CAO6Zhu0Rx9Z2AncQwkKNzETasjiFdaaM+u389QHcYtRYPUmMww@mail.gmail.com>
 <CAHPuU_Y7fm1va8bomtcHxn6Qsx6DiDqkeHijTQQ2zSTEASG_Gw@mail.gmail.com>
 <CAO6Zhu02dhNqK71taxs07RVagxCXoUya1H75W_Aa7VJHC1ny8w@mail.gmail.com>
 <CAHPuU_aOhWdnojLqKKq9ma8WiH4o4=CY=-a0Td=Vu7VBGAxApQ@mail.gmail.com>
 <CAO6Zhu1KepKjiB5qLU+mX23bYFU=950U2fHwpCyGgK+nY-4mYQ@mail.gmail.com>
 <CAEfGn+zGX1i4oRJqAcdTfA0eH69ONvOu2Dj0H9v=CNW6MMHAGA@mail.gmail.com>
Message-ID: <20180123202218.v5htepo3655vyqzm@fastmail.com>

On Tue, 23 Jan 2018 15:01:38 -0500, CJ Carey wrote:
> Frechet distance is pretty similar to Dynamic Time Warping, which has also
> been requested for inclusion in scipy:
> https://github.com/scipy/scipy/issues/7638
> 
> AFAIK, there's no current effort to add DTW, but I suspect it would make
> sense to have both it and Frechet variants in a similar namespace.

We've wanted to implement DTW in scikit-image as well for a long time.
This discussion has lots of interesting pieces of information:

https://github.com/scipy/scipy/issues/7638

We even have an old PR that implements the simplest cases [0] but, as the R
package author mentions in the discussion above, it really is a family
of algorithms that requires significant effort to get right.

A cherry on the cake would be an implementation that supports variable
endpoints.

Best regards
St?fan

[0] https://github.com/scikit-image/scikit-image/pull/518


From haberland at ucla.edu  Tue Jan 23 17:49:48 2018
From: haberland at ucla.edu (Matt Haberland)
Date: Tue, 23 Jan 2018 14:49:48 -0800
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <CAB6mnxLQqcdnwGi8nURH=q3nYd--+XJsZMq3R8yisj=6cJDokA@mail.gmail.com>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
 <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
 <20180123192038.mgenugikitd2x4wy@fastmail.com>
 <CAB6mnxLQqcdnwGi8nURH=q3nYd--+XJsZMq3R8yisj=6cJDokA@mail.gmail.com>
Message-ID: <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>

Funny, I was just writing about this to suggest videos, too.

I looked on YouTube but didn't find anything useful when searching for
numpy/scipy development environment/workflow.

Besides the day-to-day workflow, I also had a hard time getting my
development environment set up initially. A video on that would be really
helpful, too.

I'd be happy to make some videos with a screen recording and voice-over. I
want to start from scratch on Mac, as my environment seems to have broken
over the past few months, and I'd like to try again on Windows, which I
never got working. If someone with real experience on one or both of those
platforms has some time to answer questions that inevitably arise, I'd be
willing to work on this immediately.

Matt


On Tue, Jan 23, 2018 at 11:56 AM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

>
>
> On Tue, Jan 23, 2018 at 12:20 PM, Stefan van der Walt <
> stefanv at berkeley.edu> wrote:
>
>> Hi Robert
>>
>> On Tue, 23 Jan 2018 04:57:01 -0500, Robert Lucente - Pipeline.Com wrote:
>> > I am a newbie and have just been lurking.
>>
>> Thank you for raising your thoughts here; it's so helpful to hear
>> perspectives from outside the established developer community.
>>
>> > Perhaps we could put some bounds around unacceptable, good enough and
>> > perfection? I realize that it is hard to put words to this. Perhaps an
>> > example to kick off a conversation?
>>
>> It's a tricky balance.  My feeling is that a person should lower the
>> barrier to contribution as much as possible: avoid unnecessary technical
>> challenges like git rebasing, push to PRs to guide new contributors, and
>> provide good documentation.
>>
>> That said, I think it's better to educate than to adjust standards.
>> Experience has taught that introducing code of insufficient quality into
>> the code base inevitably leads to headaches later on (we all have a lot
>> of hypothetical time to fix things up in the future, right?).
>>
>> > Perhaps people could volunteer to just help someone w/ the SciPy
>> > workflow?
>>
>> This is an excellent suggestion, and one we've also been considering for
>> scikit-image.  The idea of having existing developers acting as mentors
>> is often what happens informally, but it may be helpful to establish
>> more obvious ways for that to take place.  Do you have any thoughts on
>> how this could look?
>>
>
> I know there have been several presentations on the workflow, Jamie Frio
> did a couple. It might useful to see if anyone has made video's of a
> tutorial like that so we could put it up on youtube or some other place.
> Maybe NumFocus could have a youtube channel with such resources.
>
> Chuck
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>


-- 
Matt Haberland
Assistant Adjunct Professor in the Program in Computing
Department of Mathematics
6617A Math Sciences Building, UCLA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180123/be2260a3/attachment-0001.html>

From andyfaff at gmail.com  Tue Jan 23 18:05:44 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Wed, 24 Jan 2018 10:05:44 +1100
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
 <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
 <20180123192038.mgenugikitd2x4wy@fastmail.com>
 <CAB6mnxLQqcdnwGi8nURH=q3nYd--+XJsZMq3R8yisj=6cJDokA@mail.gmail.com>
 <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
Message-ID: <CAAbtOZecpiY-RA7k_5B4fBzOaE9c599PXjH5WW3HDT-WhVEX9g@mail.gmail.com>

I do all my work on OSX.

This is a highlevel view of how I did my setup:

0) Make sure I have the command line tools for Xcode. Means installing
Xcode, then install those tools.
1) Install git from source on terminal. (might be difficult for those
unacquainted with POSIX, in that use of the terminal is required, etc)
2) Install miniconda
3) Create a conda Python36 environment and activate it (unless I have to I
don't touch 27 anymore)
4) conda install numpy, matplotlib, jupyter, cython, nose, pytest, etc into
that environment
5) create a local copy of my scipy fork on github.
6) Install pycharm as a IDE.
7) Install gfortran, I think I downloaded an installer from somewhere, but
I can't remember where. (I may have even built it myself, not sure).
8) test scipy: python runtests.py
9) install scipy: pip install .

Dev workflow:
1) Update master branch of local repo from scipy/master
2) Create a feature branch
3) Make changes in Pycharm, remember to add tests for the issue. Sometimes
I fiddle around with code in jupyter notebooks to see if what I've written
would work.
4) python runtests.py to see if the changes worked.
5) git Commit and push to github fork.
6) Create PR in scipy from github fork.

A lot of that might be scriptable. There are some that use brew, etc, but
I've always been a bit wary of using that kind of package manager. Usually
if you build it yourself (e.g. git) it works just fine.

When doing PR's I rely on Appveyor to tell me if something works on Windows
(just too complicated otherwise).
I'm happy to advise on any of those steps.
A.


On 24 January 2018 at 09:49, Matt Haberland <haberland at ucla.edu> wrote:

> Funny, I was just writing about this to suggest videos, too.
>
> I looked on YouTube but didn't find anything useful when searching for
> numpy/scipy development environment/workflow.
>
> Besides the day-to-day workflow, I also had a hard time getting my
> development environment set up initially. A video on that would be really
> helpful, too.
>
> I'd be happy to make some videos with a screen recording and voice-over. I
> want to start from scratch on Mac, as my environment seems to have broken
> over the past few months, and I'd like to try again on Windows, which I
> never got working. If someone with real experience on one or both of those
> platforms has some time to answer questions that inevitably arise, I'd be
> willing to work on this immediately.
>
> Matt
>
>
> On Tue, Jan 23, 2018 at 11:56 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Tue, Jan 23, 2018 at 12:20 PM, Stefan van der Walt <
>> stefanv at berkeley.edu> wrote:
>>
>>> Hi Robert
>>>
>>> On Tue, 23 Jan 2018 04:57:01 -0500, Robert Lucente - Pipeline.Com wrote:
>>> > I am a newbie and have just been lurking.
>>>
>>> Thank you for raising your thoughts here; it's so helpful to hear
>>> perspectives from outside the established developer community.
>>>
>>> > Perhaps we could put some bounds around unacceptable, good enough and
>>> > perfection? I realize that it is hard to put words to this. Perhaps an
>>> > example to kick off a conversation?
>>>
>>> It's a tricky balance.  My feeling is that a person should lower the
>>> barrier to contribution as much as possible: avoid unnecessary technical
>>> challenges like git rebasing, push to PRs to guide new contributors, and
>>> provide good documentation.
>>>
>>> That said, I think it's better to educate than to adjust standards.
>>> Experience has taught that introducing code of insufficient quality into
>>> the code base inevitably leads to headaches later on (we all have a lot
>>> of hypothetical time to fix things up in the future, right?).
>>>
>>> > Perhaps people could volunteer to just help someone w/ the SciPy
>>> > workflow?
>>>
>>> This is an excellent suggestion, and one we've also been considering for
>>> scikit-image.  The idea of having existing developers acting as mentors
>>> is often what happens informally, but it may be helpful to establish
>>> more obvious ways for that to take place.  Do you have any thoughts on
>>> how this could look?
>>>
>>
>> I know there have been several presentations on the workflow, Jamie Frio
>> did a couple. It might useful to see if anyone has made video's of a
>> tutorial like that so we could put it up on youtube or some other place.
>> Maybe NumFocus could have a youtube channel with such resources.
>>
>> Chuck
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
>
> --
> Matt Haberland
> Assistant Adjunct Professor in the Program in Computing
> Department of Mathematics
> 6617A Math Sciences Building, UCLA
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>


-- 
_____________________________________
Dr. Andrew Nelson


_____________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180124/6a932bb0/attachment.html>

From stefanv at berkeley.edu  Tue Jan 23 18:29:35 2018
From: stefanv at berkeley.edu (Stefan van der Walt)
Date: Tue, 23 Jan 2018 15:29:35 -0800
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
 <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
 <20180123192038.mgenugikitd2x4wy@fastmail.com>
 <CAB6mnxLQqcdnwGi8nURH=q3nYd--+XJsZMq3R8yisj=6cJDokA@mail.gmail.com>
 <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
Message-ID: <20180123232935.mr5sg4r2e63t4pil@fastmail.com>

On Tue, 23 Jan 2018 14:49:48 -0800, Matt Haberland wrote:
> Funny, I was just writing about this to suggest videos, too.
> 
> I looked on YouTube but didn't find anything useful when searching for
> numpy/scipy development environment/workflow.

Here is the scikit-image contributor guide:

http://scikit-image.org/docs/dev/contribute.html

It can do with some love, but it's probably quite similar to what we use
in SciPy.  Perhaps it's worth collectively updating such a set of notes
first, that we can then turn into a series of small video snippets?

> Besides the day-to-day workflow, I also had a hard time getting my
> development environment set up initially. A video on that would be really
> helpful, too.

For that, I suggest we make one opinionated choice of environment for
illustrative purposes.  We can probably come up with a set of 5 or 10
things that a workable editing environment needs to have (spaces not
tabs, PEP8 checker, etc.).

St?fan

From rlucente at pipeline.com  Tue Jan 23 17:52:31 2018
From: rlucente at pipeline.com (Robert Lucente - Pipeline.Com)
Date: Tue, 23 Jan 2018 17:52:31 -0500
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
Message-ID: <176a01d3949c$da9ba9c0$8fd2fd40$@pipeline.com>

> Perhaps we could put some bounds around unacceptable, good enough and 
> perfection? I realize that it is hard to put words to this. Perhaps an 
> example to kick off a conversation?

It's a tricky balance.  My feeling is that a person should lower the barrier
to contribution as much as possible: avoid unnecessary technical challenges
like git rebasing, push to PRs to guide new contributors, and provide good
documentation.

That said, I think it's better to educate than to adjust standards.
RL: Cool. The above corresponds to my personal bias. However, I was not sure
what the appropriate tribal norms were.

Experience has taught that introducing code of insufficient quality into the
code base inevitably leads to headaches later on (we all have a lot of
hypothetical time to fix things up in the future, right?).
RL: Sorry for the repetition - Cool. The above corresponds to my personal
bias. However, I was not sure what the appropriate tribal norms were.

> Perhaps people could volunteer to just help someone w/ the SciPy 
> workflow?

This is an excellent suggestion, and one we've also been considering for
scikit-image.  The idea of having existing developers acting as mentors is
often what happens informally, but it may be helpful to establish more
obvious ways for that to take place.  Do you have any thoughts on how this
could look?
RL: The following steps might seem obvious but let me state them just in
case

1) After establishing that person x has the background and code for topic y;
someone volunteers to help them w/ the SciPy workflow.

2) The 2 individuals agree to the level of help. It will vary from pair to
pair. Nothing formal is required. However, there should be an informal email
or some other written doco stating the level of help. It is just amazing
what happens when have to write something down.

I know there have been several presentations on the workflow, Jamie Frio did
a couple. 
It might useful to see if anyone has made video's of a tutorial like that so
we could put it up 
on youtube or some other place. Maybe NumFocus could have a youtube channel
with such resources.

RL: When I go to http://scikit-learn.org/stable/documentation.html, I see 2
sections

1) Development: Information on how to contribute. This also contains useful
information for advanced users, for example how to build their own
estimators.

2) FAQ: Frequently asked questions about the project and contributing.

Q: Would you guys consider have a section titled something like
"Contributing"?

This way, if someone is interested in contributing, they only have to look
in 1 place. In this 1 place we could add presentations and YouTube videos.

I will gladly help w/ this if there is interest.



From ilhanpolat at gmail.com  Tue Jan 23 19:50:46 2018
From: ilhanpolat at gmail.com (Ilhan Polat)
Date: Wed, 24 Jan 2018 01:50:46 +0100
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
 <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
 <20180123192038.mgenugikitd2x4wy@fastmail.com>
 <CAB6mnxLQqcdnwGi8nURH=q3nYd--+XJsZMq3R8yisj=6cJDokA@mail.gmail.com>
 <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
Message-ID: <CAEBuzr-2KG_a=v94rZVM95a5Y-e4TVv2CJ-G7DVRQ_8y428L7g@mail.gmail.com>

> I'd be happy to make some videos with a screen recording and voice-over. I
> want to start from scratch on Mac, as my environment seems to have broken
> over the past few months, and I'd like to try again on Windows, which I
> never got working. If someone with real experience on one or both of those
> platforms has some time to answer questions that inevitably arise, I'd be
> willing to work on this immediately.
>
>
Matt, please have a look at this. All feedback is welcome

https://scipy.github.io/devdocs/building/windows.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180124/f3237781/attachment.html>

From ilhanpolat at gmail.com  Tue Jan 23 21:29:42 2018
From: ilhanpolat at gmail.com (Ilhan Polat)
Date: Wed, 24 Jan 2018 03:29:42 +0100
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <CAEBuzr9LsG8X2Nhamkg514VenM2LmDKes9KSyw7rhuMeGkqQ2g@mail.gmail.com>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
 <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
 <20180123192038.mgenugikitd2x4wy@fastmail.com>
 <CAB6mnxLQqcdnwGi8nURH=q3nYd--+XJsZMq3R8yisj=6cJDokA@mail.gmail.com>
 <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
 <CAEBuzr-2KG_a=v94rZVM95a5Y-e4TVv2CJ-G7DVRQ_8y428L7g@mail.gmail.com>
 <CADuxUizzozQn6L019u+PtzzTUwj3uZwmoYzpfxCmgcRa5sf2BQ@mail.gmail.com>
 <CAEBuzr9LsG8X2Nhamkg514VenM2LmDKes9KSyw7rhuMeGkqQ2g@mail.gmail.com>
Message-ID: <CAEBuzr9ZHay6UrEPzMD4eH=SxmmpnJn_f3xZ-64sGGFGHurSaA@mail.gmail.com>

Some big enhancement tickets have some LAPACK wrapping codes in it and just
to increase the review base I've started writing this wiki

https://github.com/scipy/scipy/wiki/WIP:-A-brief-How-to-for-wrapping-LAPACK-routines

As we are all aware f2py is a strange mixture of code and voodoo, I wanted
to make it as verbose as possible. What I would really appreciate is the
help for all the pointer business that it uses internally. For this
document it doesn't matter much since I'll just gloss over what to do but
it is indeed a major mystery to me what it actually tries to do.





On Wed, Jan 24, 2018 at 2:55 AM, Ilhan Polat <ilhanpolat at gmail.com> wrote:

> Yes indeed, we have tested ourselves but the more the buggier I guess :)
>
> On Wed, Jan 24, 2018 at 2:09 AM, Matt Haberland <haberland at ucla.edu>
> wrote:
>
>> I remember that came out - it's relatively new, right?
>> I'll try it shortly, provided that I have enough space on my tiny laptop
>> ssd.
>>
>> On Tue, Jan 23, 2018 at 4:50 PM, Ilhan Polat <ilhanpolat at gmail.com>
>> wrote:
>>
>>>
>>> I'd be happy to make some videos with a screen recording and voice-over.
>>>> I want to start from scratch on Mac, as my environment seems to have broken
>>>> over the past few months, and I'd like to try again on Windows, which I
>>>> never got working. If someone with real experience on one or both of those
>>>> platforms has some time to answer questions that inevitably arise, I'd be
>>>> willing to work on this immediately.
>>>>
>>>>
>>> Matt, please have a look at this. All feedback is welcome
>>>
>>> https://scipy.github.io/devdocs/building/windows.html
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>>
>>
>> --
>> Matt Haberland
>> Assistant Adjunct Professor in the Program in Computing
>> Department of Mathematics
>> 6617A Math Sciences Building, UCLA
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180124/d6d4f98e/attachment.html>

From lagru at mailbox.org  Wed Jan 24 06:09:51 2018
From: lagru at mailbox.org (Lars G.)
Date: Wed, 24 Jan 2018 12:09:51 +0100
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
 <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
 <20180123192038.mgenugikitd2x4wy@fastmail.com>
 <CAB6mnxLQqcdnwGi8nURH=q3nYd--+XJsZMq3R8yisj=6cJDokA@mail.gmail.com>
 <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
Message-ID: <83e510a0-3d85-f5e1-f865-eeab8d70368d@mailbox.org>

Hello, to add to this:

On 23.01.2018 23:49, Matt Haberland wrote:

> Besides the day-to-day workflow, I also had a hard time getting my
> development environment set up initially. A video on that would be
> really helpful, too.
I'd find additional hints on how to execute & debug changes to a local
development version of SciPy really useful. I'm still struggling with
this.?
I think to try out & debug changes to Python code one needs to compile &
install the full package which is not really feasible for every change.?
My current hack is to overwrite selected files in the site-packages
folder (where SciPy-Dev is installed) with the modified files through a
simple script. That way I don't need to rebuild the binaries.
I'm sure there is a better approach and it may be useful to cover these
steps (how to debug or execute selected unit tests in PyCharm or any
other IDE) as well.

At least on Linux I found the contribution guide quite helpful for the
initial setup. However I think some tips on how to install from source
into a conda or virtualenv environment would improve this further. As it
is, coming from Windows, some backround knowledge about Linux is
definitly required.

Greetings, Lars


From andyfaff at gmail.com  Wed Jan 24 06:37:24 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Wed, 24 Jan 2018 22:37:24 +1100
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <83e510a0-3d85-f5e1-f865-eeab8d70368d@mailbox.org>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
 <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
 <20180123192038.mgenugikitd2x4wy@fastmail.com>
 <CAB6mnxLQqcdnwGi8nURH=q3nYd--+XJsZMq3R8yisj=6cJDokA@mail.gmail.com>
 <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
 <83e510a0-3d85-f5e1-f865-eeab8d70368d@mailbox.org>
Message-ID: <CAAbtOZeL7nxmKH=A4_SbQHjONhFuk9YzvgCC9KBNMQGphDiUWw@mail.gmail.com>

On 24 January 2018 at 22:09, Lars G. <lagru at mailbox.org> wrote:

>
> I'd find additional hints on how to execute & debug changes to a local
> development version of SciPy really useful. I'm still struggling with
> this.
> I think to try out & debug changes to Python code one needs to compile &
> install the full package which is not really feasible for every change.
>

`python runtests.py` is the easiest way to do this. If you make small
changes it doesn't take long to build, and installation is into a virtual
environment so you don't need to worry about it interfering with your
production environment. You can use `python runtests.py -s optimize` to
just run the unit tests for the optimize module, and you can also select
one specific test to run, thereby saving a lot of test time.
So if there's an issue I start off by writing a unit test that would
exercise the problem, then alter code such that it then passes. A single
build/test cycle only takes one or two minutes.


> My current hack is to overwrite selected files in the site-packages
> folder (where SciPy-Dev is installed) with the modified files through a
> simple script. That way I don't need to rebuild the binaries.
>

Once you start altering compiled code, then yes, it does take longer. But
only a single file would need to be compiled and that module relinked.
Sometimes I copy a whole class/function into a jupyter notebook and mess
around with it in there until it does what is needed, then I make the
relevant changes to the file in the scipy codebase. Once it works locally
then you can commit and push to your fork on github.


> I'm sure there is a better approach and it may be useful to cover these
> steps (how to debug or execute selected unit tests in PyCharm or any
> other IDE) as well.
>
> At least on Linux I found the contribution guide quite helpful for the
> initial setup. However I think some tips on how to install from source
> into a conda or virtualenv environment would improve this further. As it
> is, coming from Windows, some backround knowledge about Linux is
> definitly required.
>
> Greetings, Lars
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>



-- 
_____________________________________
Dr. Andrew Nelson


_____________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180124/633f2506/attachment-0001.html>

From ilhanpolat at gmail.com  Wed Jan 24 07:40:21 2018
From: ilhanpolat at gmail.com (Ilhan Polat)
Date: Wed, 24 Jan 2018 13:40:21 +0100
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <CAAbtOZeL7nxmKH=A4_SbQHjONhFuk9YzvgCC9KBNMQGphDiUWw@mail.gmail.com>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
 <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
 <20180123192038.mgenugikitd2x4wy@fastmail.com>
 <CAB6mnxLQqcdnwGi8nURH=q3nYd--+XJsZMq3R8yisj=6cJDokA@mail.gmail.com>
 <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
 <83e510a0-3d85-f5e1-f865-eeab8d70368d@mailbox.org>
 <CAAbtOZeL7nxmKH=A4_SbQHjONhFuk9YzvgCC9KBNMQGphDiUWw@mail.gmail.com>
Message-ID: <CAEBuzr8P6oYZxKyfgYYHWQP94UgbfTeaOqhcR1n4+QmDNCS1_A@mail.gmail.com>

Sometimes it is even easier to use the -t switch

python runtests.py -t <filename>

only tests that file. Also if I remember correctly pytest supports
different verbosity levels so you can add -v -v -v for more output about
the performed tests.




On Wed, Jan 24, 2018 at 12:37 PM, Andrew Nelson <andyfaff at gmail.com> wrote:

>
>
> On 24 January 2018 at 22:09, Lars G. <lagru at mailbox.org> wrote:
>
>>
>> I'd find additional hints on how to execute & debug changes to a local
>> development version of SciPy really useful. I'm still struggling with
>> this.
>> I think to try out & debug changes to Python code one needs to compile &
>> install the full package which is not really feasible for every change.
>>
>
> `python runtests.py` is the easiest way to do this. If you make small
> changes it doesn't take long to build, and installation is into a virtual
> environment so you don't need to worry about it interfering with your
> production environment. You can use `python runtests.py -s optimize` to
> just run the unit tests for the optimize module, and you can also select
> one specific test to run, thereby saving a lot of test time.
> So if there's an issue I start off by writing a unit test that would
> exercise the problem, then alter code such that it then passes. A single
> build/test cycle only takes one or two minutes.
>
>
>> My current hack is to overwrite selected files in the site-packages
>> folder (where SciPy-Dev is installed) with the modified files through a
>> simple script. That way I don't need to rebuild the binaries.
>>
>
> Once you start altering compiled code, then yes, it does take longer. But
> only a single file would need to be compiled and that module relinked.
> Sometimes I copy a whole class/function into a jupyter notebook and mess
> around with it in there until it does what is needed, then I make the
> relevant changes to the file in the scipy codebase. Once it works locally
> then you can commit and push to your fork on github.
>
>
>> I'm sure there is a better approach and it may be useful to cover these
>> steps (how to debug or execute selected unit tests in PyCharm or any
>> other IDE) as well.
>>
>> At least on Linux I found the contribution guide quite helpful for the
>> initial setup. However I think some tips on how to install from source
>> into a conda or virtualenv environment would improve this further. As it
>> is, coming from Windows, some backround knowledge about Linux is
>> definitly required.
>>
>> Greetings, Lars
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
>
>
> --
> _____________________________________
> Dr. Andrew Nelson
>
>
> _____________________________________
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180124/69695e2d/attachment.html>

From rlucente at pipeline.com  Wed Jan 24 08:27:47 2018
From: rlucente at pipeline.com (Robert Lucente - Pipeline.Com)
Date: Wed, 24 Jan 2018 08:27:47 -0500
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
Message-ID: <184c01d39517$202f17e0$608d47a0$@pipeline.com>

Would creating Amazon Machine Images (AMI) be helpful? This way, not
everyone would have to setup their own environment?

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html

We could probably get Amazon to sponsor this type of thing?

I am pretty sure that the mailing list knows someone that works at Amazon.
If not, we could try to use

https://www.amazon.com/p/feature/evmc8u96fp595js

https://www.amazon.com/p/feature/evmc8u96fp595js


From sievert.scott at gmail.com  Wed Jan 24 10:23:01 2018
From: sievert.scott at gmail.com (Scott Sievert)
Date: Wed, 24 Jan 2018 07:23:01 -0800
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
Message-ID: <CAPZ=bb0jkaOX9Vcm=d-mFhb6uPzwJDRRXACydaB-CYHOvxRCDw@mail.gmail.com>

I think creating an AMI would be useful. I?ve run into build issues on my
OSX machine and turned to Amazon to build SciPy and run the tests.

I do know someone at Amazon who might be in a place to sponsor this.

I?d guess that this inexpensive (upper bound: $1/mo if our AMI is 10GB
large, which it won?t be).

Scott

On January 24, 2018 at 7:27:47 AM, Robert Lucente - Pipeline.Com (
rlucente at pipeline.com) wrote:

> Would creating Amazon Machine Images (AMI) be helpful? This way, not
> everyone would have to setup their own environment?
>
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html
>
> We could probably get Amazon to sponsor this type of thing?
>
> I am pretty sure that the mailing list knows someone that works at Amazon.
> If not, we could try to use
>
> https://www.amazon.com/p/feature/evmc8u96fp595js
>
> https://www.amazon.com/p/feature/evmc8u96fp595js
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180124/95566a8e/attachment.html>

From stefanv at berkeley.edu  Wed Jan 24 14:12:48 2018
From: stefanv at berkeley.edu (Stefan van der Walt)
Date: Wed, 24 Jan 2018 11:12:48 -0800
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
In-Reply-To: <CAPZ=bb0jkaOX9Vcm=d-mFhb6uPzwJDRRXACydaB-CYHOvxRCDw@mail.gmail.com>
References: <CAPZ=bb0jkaOX9Vcm=d-mFhb6uPzwJDRRXACydaB-CYHOvxRCDw@mail.gmail.com>
Message-ID: <20180124191248.425kypfnvc5h63tf@fastmail.com>

On Wed, 24 Jan 2018 07:23:01 -0800, Scott Sievert wrote:
>I think creating an AMI would be useful. I?ve run into build issues on my
>OSX machine and turned to Amazon to build SciPy and run the tests.

This sounds like a fairly cumbersome way to develop.  We already have
Travis CI to do the testing in case you really can't compile, and
there's always Docker Linux images if you're in a fix.

If it were up to me, we'd rather spend energy on updating the developer
documentation so that each person can setup their own functioning
environment.

St?fan

From njs at pobox.com  Wed Jan 24 15:49:39 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 24 Jan 2018 12:49:39 -0800
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
In-Reply-To: <20180124191248.425kypfnvc5h63tf@fastmail.com>
References: <CAPZ=bb0jkaOX9Vcm=d-mFhb6uPzwJDRRXACydaB-CYHOvxRCDw@mail.gmail.com>
 <20180124191248.425kypfnvc5h63tf@fastmail.com>
Message-ID: <CAPJVwB=nU=VjihfPSmPgHd40aJ1YCevPQyn_=9Kpfp2wCo=GiQ@mail.gmail.com>

Yeah, I've tried using AMIs for development, and local VMs are a *way*
better experience. Much less fiddly copying of files back and forth, better
latency, etc.

It might make sense to have instructions for getting a docker-based Linux
development environment spun up quickly on Windows and MacOS ? these are
really popular nowadays so there's quite a bit of infrastructure for making
it easy.

On Jan 24, 2018 11:13 AM, "Stefan van der Walt" <stefanv at berkeley.edu>
wrote:

> On Wed, 24 Jan 2018 07:23:01 -0800, Scott Sievert wrote:
>
>> I think creating an AMI would be useful. I?ve run into build issues on my
>> OSX machine and turned to Amazon to build SciPy and run the tests.
>>
>
> This sounds like a fairly cumbersome way to develop.  We already have
> Travis CI to do the testing in case you really can't compile, and
> there's always Docker Linux images if you're in a fix.
>
> If it were up to me, we'd rather spend energy on updating the developer
> documentation so that each person can setup their own functioning
> environment.
>
> St?fan
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180124/308dfac1/attachment.html>

From sievert.scott at gmail.com  Wed Jan 24 16:34:19 2018
From: sievert.scott at gmail.com (Scott Sievert)
Date: Wed, 24 Jan 2018 13:34:19 -0800
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
In-Reply-To: <CAPJVwB=nU=VjihfPSmPgHd40aJ1YCevPQyn_=9Kpfp2wCo=GiQ@mail.gmail.com>
References: <CAPJVwB=nU=VjihfPSmPgHd40aJ1YCevPQyn_=9Kpfp2wCo=GiQ@mail.gmail.com>
Message-ID: <CAPZ=bb1rxMjnqjktehfq38Pd=kjiOzKZUkXRa6t977EAvpfDsw@mail.gmail.com>

? to docker. I?d far prefer that if it has clear documentation. It?s far
simpler.

Scott

On January 24, 2018 at 2:49:39 PM, Nathaniel Smith (njs at pobox.com) wrote:

> Yeah, I've tried using AMIs for development, and local VMs are a *way*
> better experience. Much less fiddly copying of files back and forth, better
> latency, etc.
>
> It might make sense to have instructions for getting a docker-based Linux
> development environment spun up quickly on Windows and MacOS ? these are
> really popular nowadays so there's quite a bit of infrastructure for making
> it easy.
>
> On Jan 24, 2018 11:13 AM, "Stefan van der Walt" <stefanv at berkeley.edu>
> wrote:
>
>> On Wed, 24 Jan 2018 07:23:01 -0800, Scott Sievert wrote:
>>
>>> I think creating an AMI would be useful. I?ve run into build issues on my
>>> OSX machine and turned to Amazon to build SciPy and run the tests.
>>>
>>
>> This sounds like a fairly cumbersome way to develop.  We already have
>> Travis CI to do the testing in case you really can't compile, and
>> there's always Docker Linux images if you're in a fix.
>>
>> If it were up to me, we'd rather spend energy on updating the developer
>> documentation so that each person can setup their own functioning
>> environment.
>>
>> St?fan
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180124/b573512a/attachment.html>

From jomsdev at gmail.com  Wed Jan 24 16:36:44 2018
From: jomsdev at gmail.com (Jordi Montes)
Date: Wed, 24 Jan 2018 22:36:44 +0100
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
In-Reply-To: <CAPZ=bb1rxMjnqjktehfq38Pd=kjiOzKZUkXRa6t977EAvpfDsw@mail.gmail.com>
References: <CAPJVwB=nU=VjihfPSmPgHd40aJ1YCevPQyn_=9Kpfp2wCo=GiQ@mail.gmail.com>
 <CAPZ=bb1rxMjnqjktehfq38Pd=kjiOzKZUkXRa6t977EAvpfDsw@mail.gmail.com>
Message-ID: <CAJfd0CNMwBqU43fOZqUde3Ho6sZFW_-4ZSixs=V84z_W=asE1w@mail.gmail.com>

I have been contributing to other open source projects and docker is really
extended. On top of that it is easy for newbies so it is not a barrier to
get new people neither.

On 24 January 2018 at 22:34, Scott Sievert <sievert.scott at gmail.com> wrote:

> ? to docker. I?d far prefer that if it has clear documentation. It?s far
> simpler.
>
> Scott
>
> On January 24, 2018 at 2:49:39 PM, Nathaniel Smith (njs at pobox.com) wrote:
>
>> Yeah, I've tried using AMIs for development, and local VMs are a *way*
>> better experience. Much less fiddly copying of files back and forth, better
>> latency, etc.
>>
>> It might make sense to have instructions for getting a docker-based Linux
>> development environment spun up quickly on Windows and MacOS ? these are
>> really popular nowadays so there's quite a bit of infrastructure for making
>> it easy.
>>
>> On Jan 24, 2018 11:13 AM, "Stefan van der Walt" <stefanv at berkeley.edu>
>> wrote:
>>
>>> On Wed, 24 Jan 2018 07:23:01 -0800, Scott Sievert wrote:
>>>
>>>> I think creating an AMI would be useful. I?ve run into build issues on
>>>> my
>>>> OSX machine and turned to Amazon to build SciPy and run the tests.
>>>>
>>>
>>> This sounds like a fairly cumbersome way to develop.  We already have
>>> Travis CI to do the testing in case you really can't compile, and
>>> there's always Docker Linux images if you're in a fix.
>>>
>>> If it were up to me, we'd rather spend energy on updating the developer
>>> documentation so that each person can setup their own functioning
>>> environment.
>>>
>>> St?fan
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180124/5a21b94b/attachment-0001.html>

From andyfaff at gmail.com  Wed Jan 24 16:40:32 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Thu, 25 Jan 2018 08:40:32 +1100
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
In-Reply-To: <CAPZ=bb1rxMjnqjktehfq38Pd=kjiOzKZUkXRa6t977EAvpfDsw@mail.gmail.com>
References: <CAPJVwB=nU=VjihfPSmPgHd40aJ1YCevPQyn_=9Kpfp2wCo=GiQ@mail.gmail.com>
 <CAPZ=bb1rxMjnqjktehfq38Pd=kjiOzKZUkXRa6t977EAvpfDsw@mail.gmail.com>
Message-ID: <CAAbtOZcW71BEkkiEG60euOpig70uq6=Oqiaj-B7f7oBnnMpUnQ@mail.gmail.com>

+1 for docker instructions,  no idea at the moment.
Perhaps it's my warped view, but I regard that as a bigger learning
challenge than understanding git workflow, or writing a good commit
message.

On 25 Jan 2018 8:35 am, "Scott Sievert" <sievert.scott at gmail.com> wrote:

> ? to docker. I?d far prefer that if it has clear documentation. It?s far
> simpler.
>
> Scott
>
> On January 24, 2018 at 2:49:39 PM, Nathaniel Smith (njs at pobox.com) wrote:
>
>> Yeah, I've tried using AMIs for development, and local VMs are a *way*
>> better experience. Much less fiddly copying of files back and forth, better
>> latency, etc.
>>
>> It might make sense to have instructions for getting a docker-based Linux
>> development environment spun up quickly on Windows and MacOS ? these are
>> really popular nowadays so there's quite a bit of infrastructure for making
>> it easy.
>>
>> On Jan 24, 2018 11:13 AM, "Stefan van der Walt" <stefanv at berkeley.edu>
>> wrote:
>>
>>> On Wed, 24 Jan 2018 07:23:01 -0800, Scott Sievert wrote:
>>>
>>>> I think creating an AMI would be useful. I?ve run into build issues on
>>>> my
>>>> OSX machine and turned to Amazon to build SciPy and run the tests.
>>>>
>>>
>>> This sounds like a fairly cumbersome way to develop.  We already have
>>> Travis CI to do the testing in case you really can't compile, and
>>> there's always Docker Linux images if you're in a fix.
>>>
>>> If it were up to me, we'd rather spend energy on updating the developer
>>> documentation so that each person can setup their own functioning
>>> environment.
>>>
>>> St?fan
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180125/32646b65/attachment.html>

From evgeny.burovskiy at gmail.com  Thu Jan 25 15:29:52 2018
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Thu, 25 Jan 2018 23:29:52 +0300
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <CABL7CQht4jLtkeYbTouaT5A6F0RBcygr+HWPbHYmxF42-AQVRA@mail.gmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <CAHPuU_b_Hk3tqf7AbjKb8PjTqZFsJzXqUfXPghwA7WNbEmWSzQ@mail.gmail.com>
 <CAAbtOZeh_CaM_LRAHm43Jyu-=3r98_v1f2dbS0majTcXEE0Pcg@mail.gmail.com>
 <CABL7CQht4jLtkeYbTouaT5A6F0RBcygr+HWPbHYmxF42-AQVRA@mail.gmail.com>
Message-ID: <CAMRo0isZkLuCpwfDBvWLecsR5_1JBTSbQpWT+fF209kR8mZbDg@mail.gmail.com>

On Mon, Jan 22, 2018 at 9:14 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
> On Mon, Jan 22, 2018 at 9:26 AM, Andrew Nelson <andyfaff at gmail.com> wrote:
>>
>> I'm obviously up for being on the paper :-), but I'd prefer to contribute
>> by writing small sections/editing of a paper rather than coordinating.
>
>
> No worries, help of any kind is welcome, and no obligations of course.
>
>>
>> On 21 January 2018 at 14:08, Tyler Reddy <tyler.je.reddy at gmail.com> wrote:
>>>
>>> Sounds good -- I'd be up for committee work. Definitely +1 on a paper for
>>> time justification, etc.
>
>
> Thanks Tyler!
>
> Ralf
>
>
>>>
>>> On 20 January 2018 at 14:03, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> TL;DR, let's write the long journal paper on SciPy that we've wanted for
>>>> a while, let's form a small committee to coordinate, and get it out the door
>>>> in 2-3 months.
>>>>
>>>>
>>>> Motivation
>>>> ---------------
>>>> (credits for most of this text: Evgeni)
>>>>
>>>> Many scipy contributors' day jobs are in academia. Bibliometry -- papers
>>>> in
>>>> refereed journals and citations of papers by other papers -- is one of
>>>> the main
>>>> performance indicators in most academic establishments. Since we do not
>>>> generate papers, scipy contributions are all but invisible for the
>>>> purposes of a
>>>> contributor's annual report. Of course, details vary wildly; in many
>>>> cases a
>>>> contributor manages to balance their time, or to argue common sense with
>>>> their
>>>> superiors, or get an approval for scipy work, or just ignores the issue
>>>> altogether -- but sooner or later there is a form to be filled or boxes
>>>> to be
>>>> checked, and scipy contributions simply do not fit in. A peer-reviewed
>>>> journal paper on scipy will help contributors get the academic credit they
>>>> deserve.
>>>>
>>>> We can write *the* paper for SciPy 1.0, with overall project structure,
>>>> goals, etc., and for specific features/modules a focus on say the last 3
>>>> years.
>>>>
>>>>
>>>> History
>>>> ----------
>>>> For SciPy 1.0 we had three targets on the publicity/credits front: an
>>>> interesting release announcement, interesting blogs/stories (NumFOCUS blog,
>>>> Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in
>>>> the end, the rest was successful.
>>>>
>>>> [1] is a previous announcement on this list about writing (a) paper(s)
>>>> on SciPy. We wanted both "short papers" to cover one or two releases (target
>>>> journal JOSS) and a full paper as the authoritative reference for SciPy.
>>>>
>>>> We had an earlier attempt for a "short paper", it's mostly written but
>>>> has stalled (see [2]). We ran out of steam on that one. To avoid that this
>>>> time around, it would be good to have a clear public plan, target dates, and
>>>> a small committee rather than one person to drive things forward.
>>>>
>>>>
>>>> Proposal
>>>> ------------
>>>> Here's a proposal for all aspects of this exercise that I can think
>>>> about now. Some parts stolen from the AstroPy paper [3] (because their
>>>> process worked quite well).
>>>>
>>>> Form a small coordination committee of 3-5 people that set up the paper
>>>> structure, move things along when parts stall, propose/take decisions as
>>>> needed, invite co-authors, and organise paper submission/rework.
>>>>
>>>> Paper writing to be done by whoever volunteers for a section, not just
>>>> the coordination committee. First outline/structure to be done by committee,
>>>> which then asks for review of structure and volunteers for section writing.
>>>>
>>>> Scope: a 6-10 page paper, covering history, package scope and structure,
>>>> community/organisational aspects, key features and recent enhancements per
>>>> module, and roadmap.
>>>>
>>>> Authorship: anyone who made a substantial contribution in the history of
>>>> the project. Here "substantial" is interpreted as anything beyond a one-line
>>>> doc fix. Rationale: better to be too inclusive than exclusive. Sign-up via a
>>>> web form, we send the link to that form to all email addresses in the commit
>>>> history till v1.0.
>>>>
>>>> Author order (details tbd by committee):
>>>> 1. The SciPy Developers
>>>> 2. Maintainers, paper writers, other key contributors - in order of
>>>> contribution level
>>>> 3. All other authors - alphabetically ordered
>>>>
>>>> Submission target: mid-April, to either PeerJ Computer Science or
>>>> Journal of Open Research Software (tbd by committee).
>>>>
>>>> Comments? Volunteers for committee?
>>>>
>>>>
>>>> References
>>>> ----------------
>>>> [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html
>>>> [2] https://github.com/scipy/scipy-articles/pull/4
>>>> [3] https://github.com/astropy/astropy-v2.0-paper
>>>>
>>>> Cheers,
>>>> Ralf
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>
>>
>>
>> --
>> _____________________________________
>> Dr. Andrew Nelson
>>
>>


Thanks Ralf for pushing this forward!
Count me in --- even though I have to publicly admit that I fell flat
on my face on the previous attempt.

Cheers,

Evgeni

From josh.craig.wilson at gmail.com  Thu Jan 25 17:55:21 2018
From: josh.craig.wilson at gmail.com (Joshua Wilson)
Date: Thu, 25 Jan 2018 16:55:21 -0600
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <CAMRo0isZkLuCpwfDBvWLecsR5_1JBTSbQpWT+fF209kR8mZbDg@mail.gmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <CAHPuU_b_Hk3tqf7AbjKb8PjTqZFsJzXqUfXPghwA7WNbEmWSzQ@mail.gmail.com>
 <CAAbtOZeh_CaM_LRAHm43Jyu-=3r98_v1f2dbS0majTcXEE0Pcg@mail.gmail.com>
 <CABL7CQht4jLtkeYbTouaT5A6F0RBcygr+HWPbHYmxF42-AQVRA@mail.gmail.com>
 <CAMRo0isZkLuCpwfDBvWLecsR5_1JBTSbQpWT+fF209kR8mZbDg@mail.gmail.com>
Message-ID: <CAKFGQGytWx-P0XrgQhzPGSOivDO8J0k5SekNdtix6gkvTNUHXA@mail.gmail.com>

> even though I have to publicly admit that I fell flat on my face on the previous attempt.

That doesn't seem fair to yourself; as I recall it the plan was to
have a smaller paper that was more inclusive of all contributors, and
it seemed that non-core people were modest about being included so it
just didn't shake out.

Regardless, I can help with committee stuff.

- Josh

On Thu, Jan 25, 2018 at 2:29 PM, Evgeni Burovski
<evgeny.burovskiy at gmail.com> wrote:
> On Mon, Jan 22, 2018 at 9:14 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>
>>
>> On Mon, Jan 22, 2018 at 9:26 AM, Andrew Nelson <andyfaff at gmail.com> wrote:
>>>
>>> I'm obviously up for being on the paper :-), but I'd prefer to contribute
>>> by writing small sections/editing of a paper rather than coordinating.
>>
>>
>> No worries, help of any kind is welcome, and no obligations of course.
>>
>>>
>>> On 21 January 2018 at 14:08, Tyler Reddy <tyler.je.reddy at gmail.com> wrote:
>>>>
>>>> Sounds good -- I'd be up for committee work. Definitely +1 on a paper for
>>>> time justification, etc.
>>
>>
>> Thanks Tyler!
>>
>> Ralf
>>
>>
>>>>
>>>> On 20 January 2018 at 14:03, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> TL;DR, let's write the long journal paper on SciPy that we've wanted for
>>>>> a while, let's form a small committee to coordinate, and get it out the door
>>>>> in 2-3 months.
>>>>>
>>>>>
>>>>> Motivation
>>>>> ---------------
>>>>> (credits for most of this text: Evgeni)
>>>>>
>>>>> Many scipy contributors' day jobs are in academia. Bibliometry -- papers
>>>>> in
>>>>> refereed journals and citations of papers by other papers -- is one of
>>>>> the main
>>>>> performance indicators in most academic establishments. Since we do not
>>>>> generate papers, scipy contributions are all but invisible for the
>>>>> purposes of a
>>>>> contributor's annual report. Of course, details vary wildly; in many
>>>>> cases a
>>>>> contributor manages to balance their time, or to argue common sense with
>>>>> their
>>>>> superiors, or get an approval for scipy work, or just ignores the issue
>>>>> altogether -- but sooner or later there is a form to be filled or boxes
>>>>> to be
>>>>> checked, and scipy contributions simply do not fit in. A peer-reviewed
>>>>> journal paper on scipy will help contributors get the academic credit they
>>>>> deserve.
>>>>>
>>>>> We can write *the* paper for SciPy 1.0, with overall project structure,
>>>>> goals, etc., and for specific features/modules a focus on say the last 3
>>>>> years.
>>>>>
>>>>>
>>>>> History
>>>>> ----------
>>>>> For SciPy 1.0 we had three targets on the publicity/credits front: an
>>>>> interesting release announcement, interesting blogs/stories (NumFOCUS blog,
>>>>> Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in
>>>>> the end, the rest was successful.
>>>>>
>>>>> [1] is a previous announcement on this list about writing (a) paper(s)
>>>>> on SciPy. We wanted both "short papers" to cover one or two releases (target
>>>>> journal JOSS) and a full paper as the authoritative reference for SciPy.
>>>>>
>>>>> We had an earlier attempt for a "short paper", it's mostly written but
>>>>> has stalled (see [2]). We ran out of steam on that one. To avoid that this
>>>>> time around, it would be good to have a clear public plan, target dates, and
>>>>> a small committee rather than one person to drive things forward.
>>>>>
>>>>>
>>>>> Proposal
>>>>> ------------
>>>>> Here's a proposal for all aspects of this exercise that I can think
>>>>> about now. Some parts stolen from the AstroPy paper [3] (because their
>>>>> process worked quite well).
>>>>>
>>>>> Form a small coordination committee of 3-5 people that set up the paper
>>>>> structure, move things along when parts stall, propose/take decisions as
>>>>> needed, invite co-authors, and organise paper submission/rework.
>>>>>
>>>>> Paper writing to be done by whoever volunteers for a section, not just
>>>>> the coordination committee. First outline/structure to be done by committee,
>>>>> which then asks for review of structure and volunteers for section writing.
>>>>>
>>>>> Scope: a 6-10 page paper, covering history, package scope and structure,
>>>>> community/organisational aspects, key features and recent enhancements per
>>>>> module, and roadmap.
>>>>>
>>>>> Authorship: anyone who made a substantial contribution in the history of
>>>>> the project. Here "substantial" is interpreted as anything beyond a one-line
>>>>> doc fix. Rationale: better to be too inclusive than exclusive. Sign-up via a
>>>>> web form, we send the link to that form to all email addresses in the commit
>>>>> history till v1.0.
>>>>>
>>>>> Author order (details tbd by committee):
>>>>> 1. The SciPy Developers
>>>>> 2. Maintainers, paper writers, other key contributors - in order of
>>>>> contribution level
>>>>> 3. All other authors - alphabetically ordered
>>>>>
>>>>> Submission target: mid-April, to either PeerJ Computer Science or
>>>>> Journal of Open Research Software (tbd by committee).
>>>>>
>>>>> Comments? Volunteers for committee?
>>>>>
>>>>>
>>>>> References
>>>>> ----------------
>>>>> [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html
>>>>> [2] https://github.com/scipy/scipy-articles/pull/4
>>>>> [3] https://github.com/astropy/astropy-v2.0-paper
>>>>>
>>>>> Cheers,
>>>>> Ralf
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-Dev mailing list
>>>>> SciPy-Dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>
>>>
>>>
>>> --
>>> _____________________________________
>>> Dr. Andrew Nelson
>>>
>>>
>
>
> Thanks Ralf for pushing this forward!
> Count me in --- even though I have to publicly admit that I fell flat
> on my face on the previous attempt.
>
> Cheers,
>
> Evgeni
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev

From stefanv at berkeley.edu  Fri Jan 26 02:44:29 2018
From: stefanv at berkeley.edu (Stefan van der Walt)
Date: Thu, 25 Jan 2018 23:44:29 -0800
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
Message-ID: <20180126074429.3csbkpsg7hbme22g@fastmail.com>

On Sun, 21 Jan 2018 10:03:25 +1300, Ralf Gommers wrote:
>TL;DR, let's write the long journal paper on SciPy that we've wanted for a
>while, let's form a small committee to coordinate, and get it out the door
>in 2-3 months.

I'd love to see this realized.  Count me in for planning the overall
paper structure, organizing, and writing (perhaps the section on
ndimage, in collaboration with Jaime, Juan, and others?).

FWIW, when we wrote the scikit-image paper we borrowed the build tools
from the SciPy Conference Proceedings, which allowed us to write in ReST
(nice-looking GitHub diffs), and finally converting to LaTeX for the
journal.  At work, we've also been using Overleaf (online LaTeX editor)
with good success (the Git commit log may not be as pretty, but it's
very practical if you're willing to skip individual commit reviews).

Best regards,
St?fan

From s.denaxas at gmail.com  Fri Jan 26 04:52:15 2018
From: s.denaxas at gmail.com (Spiros Denaxas)
Date: Fri, 26 Jan 2018 09:52:15 +0000
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <20180126074429.3csbkpsg7hbme22g@fastmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <20180126074429.3csbkpsg7hbme22g@fastmail.com>
Message-ID: <CAO6Zhu12SLcvNE0MFgmx6_PrCUwZ25OZ0CoABH_N0UL23s-FaQ@mail.gmail.com>

I am (very) new here but happy to lend a helping hand with writing and
coordination.
Using something like Overleaf or Sharelatex is a great idea.

best,
Spiros


On Fri, Jan 26, 2018 at 7:44 AM, Stefan van der Walt <stefanv at berkeley.edu>
wrote:

> On Sun, 21 Jan 2018 10:03:25 +1300, Ralf Gommers wrote:
>
>> TL;DR, let's write the long journal paper on SciPy that we've wanted for a
>> while, let's form a small committee to coordinate, and get it out the door
>> in 2-3 months.
>>
>
> I'd love to see this realized.  Count me in for planning the overall
> paper structure, organizing, and writing (perhaps the section on
> ndimage, in collaboration with Jaime, Juan, and others?).
>
> FWIW, when we wrote the scikit-image paper we borrowed the build tools
> from the SciPy Conference Proceedings, which allowed us to write in ReST
> (nice-looking GitHub diffs), and finally converting to LaTeX for the
> journal.  At work, we've also been using Overleaf (online LaTeX editor)
> with good success (the Git commit log may not be as pretty, but it's
> very practical if you're willing to skip individual commit reviews).
>
> Best regards,
> St?fan
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180126/181c7462/attachment.html>

From josef.pktd at gmail.com  Fri Jan 26 11:44:22 2018
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 26 Jan 2018 11:44:22 -0500
Subject: [SciPy-Dev] https://pypi.python.org/pypi/scipy.stats
Message-ID: <CAMMTP+ASBWGd7sNsQRaqUsvdd0G9hTUdU2eRTA6KBLNXo1autg@mail.gmail.com>

https://pypi.python.org/pypi/scipy.stats


Does somebody know how to prevent this?


Josef
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180126/1374da95/attachment.html>

From charlesr.harris at gmail.com  Fri Jan 26 12:06:54 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 26 Jan 2018 12:06:54 -0500
Subject: [SciPy-Dev] https://pypi.python.org/pypi/scipy.stats
In-Reply-To: <CAMMTP+ASBWGd7sNsQRaqUsvdd0G9hTUdU2eRTA6KBLNXo1autg@mail.gmail.com>
References: <CAMMTP+ASBWGd7sNsQRaqUsvdd0G9hTUdU2eRTA6KBLNXo1autg@mail.gmail.com>
Message-ID: <CAB6mnxL=ENwLBDyLg=tTsyiJYAUuibbAk2jh01UofmhSpYRMag@mail.gmail.com>

On Fri, Jan 26, 2018 at 11:44 AM, <josef.pktd at gmail.com> wrote:

> https://pypi.python.org/pypi/scipy.stats
>
>
> Does somebody know how to prevent this?
>

Prevent what? The small, 596B size, or its appearance at all.

Chuck

>
>
> Josef
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180126/df7f841a/attachment.html>

From josef.pktd at gmail.com  Fri Jan 26 12:10:17 2018
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 26 Jan 2018 12:10:17 -0500
Subject: [SciPy-Dev] https://pypi.python.org/pypi/scipy.stats
In-Reply-To: <CAB6mnxL=ENwLBDyLg=tTsyiJYAUuibbAk2jh01UofmhSpYRMag@mail.gmail.com>
References: <CAMMTP+ASBWGd7sNsQRaqUsvdd0G9hTUdU2eRTA6KBLNXo1autg@mail.gmail.com>
 <CAB6mnxL=ENwLBDyLg=tTsyiJYAUuibbAk2jh01UofmhSpYRMag@mail.gmail.com>
Message-ID: <CAMMTP+CDHJwm3n-k4HupaMXfxhZHvTYB1OT5UPRvCNdAHkEk+g@mail.gmail.com>

On Fri, Jan 26, 2018 at 12:06 PM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

>
>
> On Fri, Jan 26, 2018 at 11:44 AM, <josef.pktd at gmail.com> wrote:
>
>> https://pypi.python.org/pypi/scipy.stats
>>
>>
>> Does somebody know how to prevent this?
>>
>
> Prevent what? The small, 596B size, or its appearance at all.
>

Its appearance.

name conflict and "brandname" protection

Josef



>
> Chuck
>
>>
>>
>> Josef
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180126/787a2eb7/attachment.html>

From pav at iki.fi  Fri Jan 26 12:09:35 2018
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 26 Jan 2018 18:09:35 +0100
Subject: [SciPy-Dev] https://pypi.python.org/pypi/scipy.stats
In-Reply-To: <CAMMTP+ASBWGd7sNsQRaqUsvdd0G9hTUdU2eRTA6KBLNXo1autg@mail.gmail.com>
References: <CAMMTP+ASBWGd7sNsQRaqUsvdd0G9hTUdU2eRTA6KBLNXo1autg@mail.gmail.com>
Message-ID: <ddf5ab29-bddb-a050-7026-7f55496af90b@iki.fi>

josef.pktd at gmail.com kirjoitti 26.01.2018 klo 17:44:
> https://pypi.python.org/pypi/scipy.stats
> 
> Does somebody know how to prevent this?

I think these issues are handled manually by the index maintainers.
https://www.python.org/dev/peps/pep-0541/

According to that PEP, complaints should be filed here:
https://sourceforge.net/p/pypi/support-requests/

I think in this case it's clear the package is name squatting.

	Pauli

From pav at iki.fi  Fri Jan 26 14:41:29 2018
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 26 Jan 2018 20:41:29 +0100
Subject: [SciPy-Dev] https://pypi.python.org/pypi/scipy.stats
In-Reply-To: <CAMMTP+CDHJwm3n-k4HupaMXfxhZHvTYB1OT5UPRvCNdAHkEk+g@mail.gmail.com>
References: <CAMMTP+ASBWGd7sNsQRaqUsvdd0G9hTUdU2eRTA6KBLNXo1autg@mail.gmail.com>
 <CAB6mnxL=ENwLBDyLg=tTsyiJYAUuibbAk2jh01UofmhSpYRMag@mail.gmail.com>
 <CAMMTP+CDHJwm3n-k4HupaMXfxhZHvTYB1OT5UPRvCNdAHkEk+g@mail.gmail.com>
Message-ID: <1516995689.2576.4.camel@iki.fi>

pe, 2018-01-26 kello 12:10 -0500, josef.pktd at gmail.com kirjoitti:
> Its appearance.
> 
> name conflict and "brandname" protection

According to PEP 541 https://www.python.org/dev/peps/pep-0541/ this
probably is name squatting and package is to be removed (reported on
the support tracker pointed out in that PEP).

It's also not the only package that specific user is trying to name
squat: https://pypi.org/user/maxxy/ There's little doubt the reason for
registering such packages is malicious.

	Pauli

From lagru at mailbox.org  Fri Jan 26 14:49:31 2018
From: lagru at mailbox.org (Lars G.)
Date: Fri, 26 Jan 2018 20:49:31 +0100
Subject: [SciPy-Dev] ENH: Extend peak finding capabilities in scipy.signal
 (#8264)
Message-ID: <d3fac7a9-f1cf-bedc-cb55-967e94ba9ea8@mailbox.org>

Dear SciPy devs,

I'd like to highlight my current pull request which extends SciPy's peak
finding capabilities in the signal module.
https://github.com/scipy/scipy/pull/8264
I've tried to address all feedback that was given (until now) in said
PR. However there are still some areas that would profit from a code
review and an additional pair of eyes.

Furthermore there are still some outstanding issues. It may be best to
discuss them here:

- There are still "Changes requested" which I have addressed or are
outdated. I don't know how to go about removing this flag.

- I think the Travis CI build fail is unrelated to my PR (something in
the stats module). Perhaps someone with more knowledge should confirm this.

- How to handle if invalid peaks are passed to the functions
peak_prominences and peak_widths? Is it okay to describe this as
undefined behavior in the documentation?

- argrelmax doesn't catch peaks that are wider than one sample. Decide
how to deal with this. I have implemented an alternative here which may
even be faster. But I plan to make a performance comparison to confirm
this. I have a feeling this should be addressed (if wanted) in a new
pull request.

- Should the new functionality be covered in the tutorial for
scipy.signal? I feel like that would be the best place to show more
complex examples which are out of place in the docstrings itself. If so,
I think a new PR would be the best place.

- The decision whether find_peaks should have a postfix like
"find_peaks_cwt" is still pending. If so which one?

Again, thank you all in advance for taking the time to review this PR
and any feedback given.

Best regards,
Lars


From larson.eric.d at gmail.com  Fri Jan 26 14:59:32 2018
From: larson.eric.d at gmail.com (Eric Larson)
Date: Fri, 26 Jan 2018 14:59:32 -0500
Subject: [SciPy-Dev] ENH: Extend peak finding capabilities in
 scipy.signal (#8264)
In-Reply-To: <d3fac7a9-f1cf-bedc-cb55-967e94ba9ea8@mailbox.org>
References: <d3fac7a9-f1cf-bedc-cb55-967e94ba9ea8@mailbox.org>
Message-ID: <CAGu2niWO_sR=oFxHUWzsGus5QW9Eh=X-Bd_G3cYatUUuYpm2xA@mail.gmail.com>

>
> - There are still "Changes requested" which I have addressed or are
> outdated. I don't know how to go about removing this flag.
>

You can ping previous reviewers to take another look. They can then remove
the tag.

- I think the Travis CI build fail is unrelated to my PR (something in
> the stats module). Perhaps someone with more knowledge should confirm this.
>

This unfortunately happens from time to time, especially if it's on the
NumPy dev/master build.

- How to handle if invalid peaks are passed to the functions
> peak_prominences and peak_widths? Is it okay to describe this as
> undefined behavior in the documentation?
>

Yes I think so. Best to raise a sensible error if possible (e.g., argument
shapes don't make sense) but I don't think it's worth spending too much
time protecting against all failure modes.

- argrelmax doesn't catch peaks that are wider than one sample. Decide
> how to deal with this. I have implemented an alternative here which may
> even be faster. But I plan to make a performance comparison to confirm
> this. I have a feeling this should be addressed (if wanted) in a new
> pull request.
>

Another PR is fine, but we should merge that one before this find_peaks PR
to avoid creating backward compatibility problems.

- Should the new functionality be covered in the tutorial for
> scipy.signal? I feel like that would be the best place to show more
> complex examples which are out of place in the docstrings itself. If so,
> I think a new PR would be the best place.
>

I think it makes sense to have this as part of the `find_peaks` PR. Having
good / sufficient documentation of functions is worth holding off on
merging. It is a good test for API sanity, and also makes it easier for
others to review at a high level.

- The decision whether find_peaks should have a postfix like
> "find_peaks_cwt" is still pending. If so which one?
>

There is existing discussion of this in the PR, so if people have thoughts
on this, it's worth looking there to see what ground has been covered first.

Again, thank you all in advance for taking the time to review this PR
> and any feedback given.
>

Thanks for working on this functionality!

Cheers,
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180126/4ff8a0e5/attachment-0001.html>

From lagru at mailbox.org  Fri Jan 26 15:19:00 2018
From: lagru at mailbox.org (Lars G.)
Date: Fri, 26 Jan 2018 21:19:00 +0100
Subject: [SciPy-Dev] ENH: Extend peak finding capabilities in
 scipy.signal (#8264)
In-Reply-To: <CAGu2niWO_sR=oFxHUWzsGus5QW9Eh=X-Bd_G3cYatUUuYpm2xA@mail.gmail.com>
References: <d3fac7a9-f1cf-bedc-cb55-967e94ba9ea8@mailbox.org>
 <CAGu2niWO_sR=oFxHUWzsGus5QW9Eh=X-Bd_G3cYatUUuYpm2xA@mail.gmail.com>
Message-ID: <f1f87bcf-289d-4652-6174-2da1020afffb@mailbox.org>

To extend this:

>> - argrelmax doesn't catch peaks that are wider than one sample. Decide
>> how to deal with this. I have implemented an alternative here which may
>> even be faster. But I plan to make a performance comparison to confirm
>> this. I have a feeling this should be addressed (if wanted) in a new
>> pull request.

I forgot to include the link for the proposed alternative implementation:
https://gitlab.com/lagru/peakfinder-demo/blob/master/peak_finder.py#L430

>> - The decision whether find_peaks should have a postfix like
>> "find_peaks_cwt" is still pending. If so which one?
> 
> There is existing discussion of this in the PR, so if people have
> thoughts on this, it's worth looking there to see what ground has been
> covered first.

The new function find_peaks does essentially two things:

1. find all local maxima
2. return a subset of these maxima based on certain criteria (height,
prominence, ...)

The first step (currently uses argrelmax) could be solved in many
different ways (e.g. my alternate solution or even with wavelet
transformation). Therefore I'm not a big fan of adding a prefix based on
the current implementation detail.

Lars

From robert.kern at gmail.com  Fri Jan 26 16:49:47 2018
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 27 Jan 2018 06:49:47 +0900
Subject: [SciPy-Dev] https://pypi.python.org/pypi/scipy.stats
In-Reply-To: <1516995689.2576.4.camel@iki.fi>
References: <CAMMTP+ASBWGd7sNsQRaqUsvdd0G9hTUdU2eRTA6KBLNXo1autg@mail.gmail.com>
 <CAB6mnxL=ENwLBDyLg=tTsyiJYAUuibbAk2jh01UofmhSpYRMag@mail.gmail.com>
 <CAMMTP+CDHJwm3n-k4HupaMXfxhZHvTYB1OT5UPRvCNdAHkEk+g@mail.gmail.com>
 <1516995689.2576.4.camel@iki.fi>
Message-ID: <CAF6FJitp5ap5B5j_MCm0iDNqhSvee5i+Vz39E=Z-i5m91BtCug@mail.gmail.com>

On Sat, Jan 27, 2018 at 4:41 AM, Pauli Virtanen <pav at iki.fi> wrote:
>
> pe, 2018-01-26 kello 12:10 -0500, josef.pktd at gmail.com kirjoitti:
> > Its appearance.
> >
> > name conflict and "brandname" protection
>
> According to PEP 541 https://www.python.org/dev/peps/pep-0541/ this
> probably is name squatting and package is to be removed (reported on
> the support tracker pointed out in that PEP).

And it's gone. Thanks Pauli!

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180127/a395759e/attachment.html>

From stefanv at berkeley.edu  Fri Jan 26 17:05:10 2018
From: stefanv at berkeley.edu (Stefan van der Walt)
Date: Fri, 26 Jan 2018 14:05:10 -0800
Subject: [SciPy-Dev] ENH: Extend peak finding capabilities in
 scipy.signal (#8264)
In-Reply-To: <f1f87bcf-289d-4652-6174-2da1020afffb@mailbox.org>
References: <d3fac7a9-f1cf-bedc-cb55-967e94ba9ea8@mailbox.org>
 <CAGu2niWO_sR=oFxHUWzsGus5QW9Eh=X-Bd_G3cYatUUuYpm2xA@mail.gmail.com>
 <f1f87bcf-289d-4652-6174-2da1020afffb@mailbox.org>
Message-ID: <20180126220510.nuhqetao5sqiqlua@fastmail.com>

On Fri, 26 Jan 2018 21:19:00 +0100, Lars G. wrote:
>>> - argrelmax doesn't catch peaks that are wider than one sample. Decide
>>> how to deal with this. I have implemented an alternative here which may
>>> even be faster. But I plan to make a performance comparison to confirm
>>> this. I have a feeling this should be addressed (if wanted) in a new
>>> pull request.
>
>I forgot to include the link for the proposed alternative implementation:
>https://gitlab.com/lagru/peakfinder-demo/blob/master/peak_finder.py#L430

For comparison, this is what we use in skimage:

http://scikit-image.org/docs/dev/api/skimage.feature.html#skimage.feature.peak_local_max

St?fan

From lagru at mailbox.org  Sat Jan 27 12:05:26 2018
From: lagru at mailbox.org (Lars G.)
Date: Sat, 27 Jan 2018 18:05:26 +0100
Subject: [SciPy-Dev] ENH: Extend peak finding capabilities in
 scipy.signal (#8264)
In-Reply-To: <CAGu2niWO_sR=oFxHUWzsGus5QW9Eh=X-Bd_G3cYatUUuYpm2xA@mail.gmail.com>
References: <d3fac7a9-f1cf-bedc-cb55-967e94ba9ea8@mailbox.org>
 <CAGu2niWO_sR=oFxHUWzsGus5QW9Eh=X-Bd_G3cYatUUuYpm2xA@mail.gmail.com>
Message-ID: <7dcdb2e4-2123-e354-2f5d-2b1106ef5fc2@mailbox.org>

Dear SciPy devs,

>> - argrelmax doesn't catch peaks that are wider than one sample. Decide
>> how to deal with this. I have implemented an alternative here which may
>> even be faster. But I plan to make a performance comparison to confirm
>> this. I have a feeling this should be addressed (if wanted) in a new
>> pull request.

On 26.01.2018 20:59, Eric Larson wrote:
> Another PR is fine, but we should merge that one before this find_peaks
> PR to avoid creating backward compatibility problems.?

I have prepared a performance evaluation for you to review here:
http://nbviewer.jupyter.org/urls/gitlab.com/snippets/1695752/raw
You can download the notebook here:
https://gitlab.com/snippets/1695752

I hope my approach & evaluation is not to naive. Please let me know
whether you think this approach would be a viable alternative to use in
find_peaks. In that case I'd start a new PR to add this functionality
before merging the one we are currently discussing.

Best regards,
Lars

From ralf.gommers at gmail.com  Sun Jan 28 17:24:03 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 29 Jan 2018 11:24:03 +1300
Subject: [SciPy-Dev] ENH: Extend peak finding capabilities in
 scipy.signal (#8264)
In-Reply-To: <7dcdb2e4-2123-e354-2f5d-2b1106ef5fc2@mailbox.org>
References: <d3fac7a9-f1cf-bedc-cb55-967e94ba9ea8@mailbox.org>
 <CAGu2niWO_sR=oFxHUWzsGus5QW9Eh=X-Bd_G3cYatUUuYpm2xA@mail.gmail.com>
 <7dcdb2e4-2123-e354-2f5d-2b1106ef5fc2@mailbox.org>
Message-ID: <CABL7CQigziXgSrEhznp6UHQRQ9D_T3K6rEhz2ywD4J+6fr=Biw@mail.gmail.com>

On Sun, Jan 28, 2018 at 6:05 AM, Lars G. <lagru at mailbox.org> wrote:

> Dear SciPy devs,
>
> >> - argrelmax doesn't catch peaks that are wider than one sample. Decide
> >> how to deal with this. I have implemented an alternative here which may
> >> even be faster. But I plan to make a performance comparison to confirm
> >> this. I have a feeling this should be addressed (if wanted) in a new
> >> pull request.
>
> On 26.01.2018 20:59, Eric Larson wrote:
> > Another PR is fine, but we should merge that one before this find_peaks
> > PR to avoid creating backward compatibility problems.
>
> I have prepared a performance evaluation for you to review here:
> http://nbviewer.jupyter.org/urls/gitlab.com/snippets/1695752/raw


Thanks, such easy to read comparisons are very helpful to evaluate proposed
new functionality.

Regarding the flat peaks issue, I'd have a preference for only a single
index being returned. Rationale:
- whether a peak is considered flat or not can be determined by small
amount of noise (either numerical or real)
- behavior of the function for different inputs should be consistent,
changing the number of returned indices will be annoying for users
Regarding the execution time: that should be only a secondary concern here.

Ralf


> You can download the notebook here:
> https://gitlab.com/snippets/1695752
>
> I hope my approach & evaluation is not to naive. Please let me know

whether you think this approach would be a viable alternative to use in
> find_peaks. In that case I'd start a new PR to add this functionality
> before merging the one we are currently discussing.
>
> Best regards,
> Lars
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180129/18c49c04/attachment.html>

From ralf.gommers at gmail.com  Sun Jan 28 17:36:41 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 29 Jan 2018 11:36:41 +1300
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <20180126074429.3csbkpsg7hbme22g@fastmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <20180126074429.3csbkpsg7hbme22g@fastmail.com>
Message-ID: <CABL7CQi5g===ntdphLV9BybWSACv_T2d4Ot0WOmKBpdCRL_+XA@mail.gmail.com>

On Fri, Jan 26, 2018 at 8:44 PM, Stefan van der Walt <stefanv at berkeley.edu>
wrote:

> On Sun, 21 Jan 2018 10:03:25 +1300, Ralf Gommers wrote:
>
>> TL;DR, let's write the long journal paper on SciPy that we've wanted for a
>> while, let's form a small committee to coordinate, and get it out the door
>> in 2-3 months.
>>
>
> I'd love to see this realized.  Count me in for planning the overall
> paper structure, organizing, and writing (perhaps the section on
> ndimage, in collaboration with Jaime, Juan, and others?).
>

Thanks Stefan, and thanks everyone else who volunteered.

Looks like for coordinating we have the following team: Tyler, Josh,
Evgeni, Matt, Stefan, Ralf. Let's start a bit of planning and setting up
the structure of and build tools for the paper - I'll send out a separate
email to you.


> FWIW, when we wrote the scikit-image paper we borrowed the build tools
> from the SciPy Conference Proceedings, which allowed us to write in ReST
> (nice-looking GitHub diffs), and finally converting to LaTeX for the
> journal.  At work, we've also been using Overleaf (online LaTeX editor)
> with good success (the Git commit log may not be as pretty, but it's
> very practical if you're willing to skip individual commit reviews).
>

 Good ideas. Either way, we should borrow something rather than start from
scratch. Let's decide which one in the coordination committee.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180129/a329265a/attachment.html>

From ralf.gommers at gmail.com  Sun Jan 28 17:38:08 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 29 Jan 2018 11:38:08 +1300
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <CAO6Zhu12SLcvNE0MFgmx6_PrCUwZ25OZ0CoABH_N0UL23s-FaQ@mail.gmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <20180126074429.3csbkpsg7hbme22g@fastmail.com>
 <CAO6Zhu12SLcvNE0MFgmx6_PrCUwZ25OZ0CoABH_N0UL23s-FaQ@mail.gmail.com>
Message-ID: <CABL7CQgYam7Qqib6xQE5r9uC844+95JACvPn7T367dT8utVwdA@mail.gmail.com>

On Fri, Jan 26, 2018 at 10:52 PM, Spiros Denaxas <s.denaxas at gmail.com>
wrote:

> I am (very) new here but happy to lend a helping hand with writing and
> coordination.
> Using something like Overleaf or Sharelatex is a great idea.
>

Hi Spiros, thank you for the offer. I cannot find any contribution from you
in the commit history, so I'd like to leave you out of the coordination
committee. Please ping me offline in case I missed your previous
contributions.

Cheers,
Ralf


> best,
> Spiros
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180129/ed35cd1c/attachment.html>

From ralf.gommers at gmail.com  Sun Jan 28 18:28:24 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 29 Jan 2018 12:28:24 +1300
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
In-Reply-To: <CAAbtOZcW71BEkkiEG60euOpig70uq6=Oqiaj-B7f7oBnnMpUnQ@mail.gmail.com>
References: <CAPJVwB=nU=VjihfPSmPgHd40aJ1YCevPQyn_=9Kpfp2wCo=GiQ@mail.gmail.com>
 <CAPZ=bb1rxMjnqjktehfq38Pd=kjiOzKZUkXRa6t977EAvpfDsw@mail.gmail.com>
 <CAAbtOZcW71BEkkiEG60euOpig70uq6=Oqiaj-B7f7oBnnMpUnQ@mail.gmail.com>
Message-ID: <CABL7CQgZ-aZFYuRKdoP1-hF5E0qOfmr5yFvKLPmf=9aV=ha8SA@mail.gmail.com>

On Thu, Jan 25, 2018 at 10:40 AM, Andrew Nelson <andyfaff at gmail.com> wrote:

> +1 for docker instructions,  no idea at the moment.
>

+1 good idea. I've opened https://github.com/scipy/scipy/issues/8340 for
this.

The other open issue is for clarifying the build/install docs on Windows:
https://github.com/scipy/scipy/issues/7140

Ralf



> Perhaps it's my warped view, but I regard that as a bigger learning
> challenge than understanding git workflow, or writing a good commit
> message.
>
> On 25 Jan 2018 8:35 am, "Scott Sievert" <sievert.scott at gmail.com> wrote:
>
>> ? to docker. I?d far prefer that if it has clear documentation. It?s far
>> simpler.
>>
>> Scott
>>
>> On January 24, 2018 at 2:49:39 PM, Nathaniel Smith (njs at pobox.com) wrote:
>>
>>> Yeah, I've tried using AMIs for development, and local VMs are a *way*
>>> better experience. Much less fiddly copying of files back and forth, better
>>> latency, etc.
>>>
>>> It might make sense to have instructions for getting a docker-based
>>> Linux development environment spun up quickly on Windows and MacOS ? these
>>> are really popular nowadays so there's quite a bit of infrastructure for
>>> making it easy.
>>>
>>> On Jan 24, 2018 11:13 AM, "Stefan van der Walt" <stefanv at berkeley.edu>
>>> wrote:
>>>
>>>> On Wed, 24 Jan 2018 07:23:01 -0800, Scott Sievert wrote:
>>>>
>>>>> I think creating an AMI would be useful. I?ve run into build issues on
>>>>> my
>>>>> OSX machine and turned to Amazon to build SciPy and run the tests.
>>>>>
>>>>
>>>> This sounds like a fairly cumbersome way to develop.  We already have
>>>> Travis CI to do the testing in case you really can't compile, and
>>>> there's always Docker Linux images if you're in a fix.
>>>>
>>>> If it were up to me, we'd rather spend energy on updating the developer
>>>> documentation so that each person can setup their own functioning
>>>> environment.
>>>>
>>>> St?fan
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180129/00ae3323/attachment-0001.html>

From lagru at mailbox.org  Sun Jan 28 20:06:59 2018
From: lagru at mailbox.org (Lars G.)
Date: Mon, 29 Jan 2018 02:06:59 +0100
Subject: [SciPy-Dev] ENH: Extend peak finding capabilities in
 scipy.signal (#8264)
In-Reply-To: <CABL7CQigziXgSrEhznp6UHQRQ9D_T3K6rEhz2ywD4J+6fr=Biw@mail.gmail.com>
References: <d3fac7a9-f1cf-bedc-cb55-967e94ba9ea8@mailbox.org>
 <CAGu2niWO_sR=oFxHUWzsGus5QW9Eh=X-Bd_G3cYatUUuYpm2xA@mail.gmail.com>
 <7dcdb2e4-2123-e354-2f5d-2b1106ef5fc2@mailbox.org>
 <CABL7CQigziXgSrEhznp6UHQRQ9D_T3K6rEhz2ywD4J+6fr=Biw@mail.gmail.com>
Message-ID: <3b0c7867-ee1c-5752-d3d2-5f74b60c00f9@mailbox.org>

On 28.01.2018 23:24, Ralf Gommers wrote:
> 
> On Sun, Jan 28, 2018 at 6:05 AM, Lars G. <lagru at mailbox.org> wrote:
>> I have prepared a performance evaluation for you to review here:
>> http://nbviewer.jupyter.org/urls/gitlab.com/snippets/1695752/raw
> 
> Thanks, such easy to read comparisons are very helpful to evaluate
> proposed new functionality.?
> 
> Regarding the flat peaks issue, I'd have a preference for only a single
> index being returned. Rationale:?
> - whether a peak is considered flat or not can be determined by small
> amount of noise (either numerical or real)
> - behavior of the function for different inputs should be consistent,
> changing the number of returned indices will be annoying for users
> Regarding the execution time: that should be only a secondary concern here.

Okay, it seems we are in agreement here. :)

I also played around with Cython and managed to come up with a solution
that is both
- faster than SciPy's argrelextrema and my old pure Python suggestion
- and can detect flat peaks by default.

I have updated the linked notebook with the new Cython function for
those who are interested.

I'll use this as the basis of a new PR when I find the time.

Best regards,
Lars


From rlucente at pipeline.com  Sun Jan 28 20:11:25 2018
From: rlucente at pipeline.com (Robert Lucente - Pipeline.Com)
Date: Sun, 28 Jan 2018 20:11:25 -0500
Subject: [SciPy-Dev] What we do struggle with is lack of progress on (Ralf
 Gommers)
Message-ID: <000501d3989e$163df410$42b9dc30$@pipeline.com>

> The other open issue is for clarifying the build/install docs on Windows:
https://github.com/scipy/scipy/issues/7140

I would like to volunteer w/the install docs on windows.

I am new to Python and its associated scientific stack. However, I do have
some background

1) Check out my blog post "Study Algorithms by Playing Pick 2: Find the Sum
of Any 2 Integers from a List that are Equal to a Specific Value"

http://rlucente.blogspot.com/2017/03/learn-algorithms-by-playing-pick-2-find
.html

https://www.youtube.com/watch?v=8Lb9WRMzl5Y

2) If you Google "The Bit Plumber", I will show up on 1st page

Please let me know what you guys would like the next step to be



From ralf.gommers at gmail.com  Sun Jan 28 20:23:22 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 29 Jan 2018 14:23:22 +1300
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 (Ralf Gommers)
In-Reply-To: <000501d3989e$163df410$42b9dc30$@pipeline.com>
References: <000501d3989e$163df410$42b9dc30$@pipeline.com>
Message-ID: <CABL7CQh_kY-2zZ1ZNO7m=F1RimPdcoLLQw1YUomYKfK6stLurA@mail.gmail.com>

Hi Robert, note that your email client is misconfigured - your replies are
breaking the threading. Can you please look into that?



On Mon, Jan 29, 2018 at 2:11 PM, Robert Lucente - Pipeline.Com <
rlucente at pipeline.com> wrote:

> > The other open issue is for clarifying the build/install docs on Windows:
> https://github.com/scipy/scipy/issues/7140
>
> I would like to volunteer w/the install docs on windows.
>

Great!


>
> I am new to Python and its associated scientific stack. However, I do have
> some background
>
> 1) Check out my blog post "Study Algorithms by Playing Pick 2: Find the Sum
> of Any 2 Integers from a List that are Equal to a Specific Value"
>
> http://rlucente.blogspot.com/2017/03/learn-algorithms-by-
> playing-pick-2-find
> .html
>
> https://www.youtube.com/watch?v=8Lb9WRMzl5Y
>
> 2) If you Google "The Bit Plumber", I will show up on 1st page
>
> Please let me know what you guys would like the next step to be
>

There were some improvements and simplifications after I wrote this todo
list: https://github.com/scipy/scipy/issues/7140#issuecomment-316036368.
However that's still about right for the current todo list I'd say.

The current instructions live at
https://github.com/scipy/scipy/blob/master/doc/source/building/windows.rst.
Checking those for correctness, and then editing them with the updates as
suggested in the todo list would be good.



>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180129/c397b3c5/attachment.html>

From s.denaxas at gmail.com  Mon Jan 29 03:26:50 2018
From: s.denaxas at gmail.com (Spiros Denaxas)
Date: Mon, 29 Jan 2018 08:26:50 +0000
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <CABL7CQgYam7Qqib6xQE5r9uC844+95JACvPn7T367dT8utVwdA@mail.gmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <20180126074429.3csbkpsg7hbme22g@fastmail.com>
 <CAO6Zhu12SLcvNE0MFgmx6_PrCUwZ25OZ0CoABH_N0UL23s-FaQ@mail.gmail.com>
 <CABL7CQgYam7Qqib6xQE5r9uC844+95JACvPn7T367dT8utVwdA@mail.gmail.com>
Message-ID: <CAO6Zhu2=jKAsj8x64wtoo1tAk0rNUw0o0w4U1ar6TZ2s_sZ5NA@mail.gmail.com>

Absolutely fine!

best
Spiros

On Sun, Jan 28, 2018 at 10:38 PM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:

>
>
> On Fri, Jan 26, 2018 at 10:52 PM, Spiros Denaxas <s.denaxas at gmail.com>
> wrote:
>
>> I am (very) new here but happy to lend a helping hand with writing and
>> coordination.
>> Using something like Overleaf or Sharelatex is a great idea.
>>
>
> Hi Spiros, thank you for the offer. I cannot find any contribution from
> you in the commit history, so I'd like to leave you out of the coordination
> committee. Please ping me offline in case I missed your previous
> contributions.
>
> Cheers,
> Ralf
>
>
>> best,
>> Spiros
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180129/6b2d659d/attachment.html>

From serge.guelton at telecom-bretagne.eu  Mon Jan 29 06:34:02 2018
From: serge.guelton at telecom-bretagne.eu (serge guelton)
Date: Mon, 29 Jan 2018 12:34:02 +0100
Subject: [SciPy-Dev] Using Pythran to compile some of the scipy internals
In-Reply-To: <20180122142840.GA22566@lakota>
References: <20180120101034.GA3707@lakota>
 <CABL7CQgr8AWjoQD7TV+tcb9a5wcfnrf+ttCb-hGhZKWha0WUXQ@mail.gmail.com>
 <20180122142840.GA22566@lakota>
Message-ID: <20180129113402.GA2065@lakota>

On Mon, Jan 22, 2018 at 03:28:40PM +0100, Serge Guelton wrote:
> > Another potential benefit is to decrease the size of the binary distribution of
> > SciPy. Cython extensions are quite expensive in this respect. Do you have an
> > idea of how Pythran compares? Both in case of only float64 inputs, and with
> > templated inputs? A good example of the latter is scipy.ndimage.label, which is
> > a straightforward function that ends up being a 700kb .so
> 
> Thanks for pointing out this aspect. There's still room for improvement
> here, but with current pythran's master, and same compilation flags, the
> cython implementation of `max_len_seq_inner` takes ~700kb while
> pythran's version takes ~200kb.
> 
> I'm going to investigate that a bet more though, and probably post the
> results in this thread later on.

Hey scip-dev, finally the result of some dev + thoughts on
Pythran-generated code:

    http://serge-sans-paille.github.io/pythran-stories/shrinking-pythran-generated-binaries.html

I'll move on to Windows support now :-)

From rlucente at pipeline.com  Mon Jan 29 08:41:18 2018
From: rlucente at pipeline.com (Robert Lucente)
Date: Mon, 29 Jan 2018 08:41:18 -0500 (GMT-05:00)
Subject: [SciPy-Dev] email client is misconfigured - replies are breaking
 the threading
Message-ID: <983049641.1364.1517233279055@wamui-hamster.atl.sa.earthlink.net>

>Date: Mon, 29 Jan 2018 14:23:22 +1300
>From: Ralf Gommers <ralf.gommers at gmail.com>
>Subject: Re: [SciPy-Dev] What we do struggle with is lack of progress
>	on (Ralf Gommers)
>
>Hi Robert, note that your email client is misconfigured - your replies are
>breaking the threading. Can you please look into that?

I am not sure what "breaking the threading" means.

In https://www.scipy.org/scipylib/mailing-lists.html, I found

Use bottom-posting.

When replying to a message, please use bottom-posting. This means quoting the content you reply to and inserting your replies below each quote. For more details, see posting style.

Before, I was using Outlook from Microsoft Office Professional 2013. I have now switched to using EarthLink's Web Mail to reply to these emails. It seems to produce the appropriate number of ">".

Is this better?

If not, please make some suggestions. I just got my System 76 Bonobo running Ubuntu. Perhaps I need to use a Unix based email client?


From deak.andris at gmail.com  Mon Jan 29 09:00:55 2018
From: deak.andris at gmail.com (Andras Deak)
Date: Mon, 29 Jan 2018 15:00:55 +0100
Subject: [SciPy-Dev] email client is misconfigured - replies are
 breaking the threading
In-Reply-To: <983049641.1364.1517233279055@wamui-hamster.atl.sa.earthlink.net>
References: <983049641.1364.1517233279055@wamui-hamster.atl.sa.earthlink.net>
Message-ID: <CAMEWA4PPHN6TAK+hmvF_F5n=VwYJ1ghC225rS0qiEW0360=gTw@mail.gmail.com>

On Mon, Jan 29, 2018 at 2:41 PM, Robert Lucente <rlucente at pipeline.com> wrote:
>>Date: Mon, 29 Jan 2018 14:23:22 +1300
>>From: Ralf Gommers <ralf.gommers at gmail.com>
>>Subject: Re: [SciPy-Dev] What we do struggle with is lack of progress
>>       on (Ralf Gommers)
>>
>>Hi Robert, note that your email client is misconfigured - your replies are
>>breaking the threading. Can you please look into that?
>
> I am not sure what "breaking the threading" means.
>
> In https://www.scipy.org/scipylib/mailing-lists.html, I found
>
> Use bottom-posting.
>
> When replying to a message, please use bottom-posting. This means quoting the content you reply to and inserting your replies below each quote. For more details, see posting style.
>
> Before, I was using Outlook from Microsoft Office Professional 2013. I have now switched to using EarthLink's Web Mail to reply to these emails. It seems to produce the appropriate number of ">".
>
> Is this better?
>
> If not, please make some suggestions. I just got my System 76 Bonobo running Ubuntu. Perhaps I need to use a Unix based email client?
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev

Looking at both gmail's view and the online archives at
https://mail.python.org/pipermail/scipy-dev/2018-January/thread.html
it seems that the subject of your emails that broke their respective
threads were edited. If you're posting to a thread, you should
probably preserve the automatic reply subject.
For example, there was a thread
"[SciPy-Dev] What we do struggle with is lack of progress on big-ticket items"
then a new thread
"[SciPy-Dev] What we do struggle with is lack of progress on"
then
"[SciPy-Dev] What we do struggle with is lack of progress on (Ralf Gommers) "

As you see, every new thread at the archive corresponds to a change in
e-mail subject.
Regards,

Andr?s

From ilhanpolat at gmail.com  Mon Jan 29 09:01:49 2018
From: ilhanpolat at gmail.com (Ilhan Polat)
Date: Mon, 29 Jan 2018 15:01:49 +0100
Subject: [SciPy-Dev] email client is misconfigured - replies are
 breaking the threading
In-Reply-To: <983049641.1364.1517233279055@wamui-hamster.atl.sa.earthlink.net>
References: <983049641.1364.1517233279055@wamui-hamster.atl.sa.earthlink.net>
Message-ID: <CAEBuzr-8PmYHPQYZt2ho83tcLHhbVsKMei=BLQZrzgdKFGhHAQ@mail.gmail.com>

I think you are starting a new thread each time instead of replying a
thread. Subscribe your email to the mailing list and receive the mails in
your email account. Then you can reply to the latest email in the thread
and it should work fine.

Breaking thread means your email starts a new thread as I quickly marked
with my horrible drawing skills

[image: Inline image 1]

On Mon, Jan 29, 2018 at 2:41 PM, Robert Lucente <rlucente at pipeline.com>
wrote:

> >Date: Mon, 29 Jan 2018 14:23:22 +1300
> >From: Ralf Gommers <ralf.gommers at gmail.com>
> >Subject: Re: [SciPy-Dev] What we do struggle with is lack of progress
> >       on (Ralf Gommers)
> >
> >Hi Robert, note that your email client is misconfigured - your replies are
> >breaking the threading. Can you please look into that?
>
> I am not sure what "breaking the threading" means.
>
> In https://www.scipy.org/scipylib/mailing-lists.html, I found
>
> Use bottom-posting.
>
> When replying to a message, please use bottom-posting. This means quoting
> the content you reply to and inserting your replies below each quote. For
> more details, see posting style.
>
> Before, I was using Outlook from Microsoft Office Professional 2013. I
> have now switched to using EarthLink's Web Mail to reply to these emails.
> It seems to produce the appropriate number of ">".
>
> Is this better?
>
> If not, please make some suggestions. I just got my System 76 Bonobo
> running Ubuntu. Perhaps I need to use a Unix based email client?
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180129/7233bf26/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 99848 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180129/7233bf26/attachment-0001.png>

From rlucente at pipeline.com  Mon Jan 29 10:01:24 2018
From: rlucente at pipeline.com (Robert Lucente)
Date: Mon, 29 Jan 2018 10:01:24 -0500 (GMT-05:00)
Subject: [SciPy-Dev] email client is misconfigured - replies are
 breaking the threading
Message-ID: <953711872.3194.1517238084474@wamui-cinderella.atl.sa.earthlink.net>

>When replying, please edit your Subject line so it is more specific
>than "Re: Contents of SciPy-Dev digest..."

It seems that I am screwing up changing the email "Subject". 

I received the email w/ the subject "Re: SciPy-Dev Digest, Vol 171, Issue 42"

I have changed the email subject to "Re: [SciPy-Dev] email client is misconfigured - replies are breaking the threading".

>Message: 2
>Date: Mon, 29 Jan 2018 15:00:55 +0100
>From: Andras Deak <deak.andris at gmail.com>
>To: SciPy Developers List <scipy-dev at python.org>
>Subject: Re: [SciPy-Dev] email client is misconfigured - replies are
>	breaking the threading

...

>Looking at both gmail's view and the online archives at
>https://mail.python.org/pipermail/scipy-dev/2018-January/thread.html
>it seems that the subject of your emails that broke their respective
>threads were edited. If you're posting to a thread, you should
>probably preserve the automatic reply subject.
>For example, there was a thread
>"[SciPy-Dev] What we do struggle with is lack of progress on big-ticket items"
>then a new thread
>"[SciPy-Dev] What we do struggle with is lack of progress on"
>then
>"[SciPy-Dev] What we do struggle with is lack of progress on (Ralf Gommers) "
>
>As you see, every new thread at the archive corresponds to a change in
>e-mail subject.

Do I have the correct email subject? I am not sure whether or not to include "Re: " in the subject.

From deak.andris at gmail.com  Mon Jan 29 13:33:15 2018
From: deak.andris at gmail.com (Andras Deak)
Date: Mon, 29 Jan 2018 19:33:15 +0100
Subject: [SciPy-Dev] email client is misconfigured - replies are
 breaking the threading
In-Reply-To: <953711872.3194.1517238084474@wamui-cinderella.atl.sa.earthlink.net>
References: <953711872.3194.1517238084474@wamui-cinderella.atl.sa.earthlink.net>
Message-ID: <CAMEWA4MjLYhyVg-Worr0SuBY+S1WsU3QPe3cn1WD4YyEeUPZxA@mail.gmail.com>

On Mon, Jan 29, 2018 at 4:01 PM, Robert Lucente <rlucente at pipeline.com> wrote:
>>When replying, please edit your Subject line so it is more specific
>>than "Re: Contents of SciPy-Dev digest..."
>
> It seems that I am screwing up changing the email "Subject".
>
> I received the email w/ the subject "Re: SciPy-Dev Digest, Vol 171, Issue 42"
>
> I have changed the email subject to "Re: [SciPy-Dev] email client is misconfigured - replies are breaking the threading".
>
>>Message: 2
>>Date: Mon, 29 Jan 2018 15:00:55 +0100
>>From: Andras Deak <deak.andris at gmail.com>
>>To: SciPy Developers List <scipy-dev at python.org>
>>Subject: Re: [SciPy-Dev] email client is misconfigured - replies are
>>       breaking the threading
>
> ...
>
>>Looking at both gmail's view and the online archives at
>>https://mail.python.org/pipermail/scipy-dev/2018-January/thread.html
>>it seems that the subject of your emails that broke their respective
>>threads were edited. If you're posting to a thread, you should
>>probably preserve the automatic reply subject.
>>For example, there was a thread
>>"[SciPy-Dev] What we do struggle with is lack of progress on big-ticket items"
>>then a new thread
>>"[SciPy-Dev] What we do struggle with is lack of progress on"
>>then
>>"[SciPy-Dev] What we do struggle with is lack of progress on (Ralf Gommers) "
>>
>>As you see, every new thread at the archive corresponds to a change in
>>e-mail subject.
>
> Do I have the correct email subject? I am not sure whether or not to include "Re: " in the subject.
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev

Ah, I didn't realize you had a digest subscription! Yes, leaving the
"Re: " is fine, the mailing list engine expects that (your last e-mail
kept the thread nicely). But other than that the subject should be the
same. When in doubt you can check the threaded view of the online
archive.

Andr?s

From robert.kern at gmail.com  Mon Jan 29 13:39:59 2018
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 30 Jan 2018 03:39:59 +0900
Subject: [SciPy-Dev] email client is misconfigured - replies are
 breaking the threading
In-Reply-To: <953711872.3194.1517238084474@wamui-cinderella.atl.sa.earthlink.net>
References: <953711872.3194.1517238084474@wamui-cinderella.atl.sa.earthlink.net>
Message-ID: <CAF6FJisD8aU_kvLLtoEua_UGYKsbMyaXeAhMv_kO3uOTHeQsow@mail.gmail.com>

On Tue, Jan 30, 2018 at 12:01 AM, Robert Lucente <rlucente at pipeline.com>
wrote:
>
> >When replying, please edit your Subject line so it is more specific
> >than "Re: Contents of SciPy-Dev digest..."
>
> It seems that I am screwing up changing the email "Subject".
>
> I received the email w/ the subject "Re: SciPy-Dev Digest, Vol 171, Issue
42"
>
> I have changed the email subject to "Re: [SciPy-Dev] email client is
misconfigured - replies are breaking the threading".

To be honest, if you are going to actively participate in the mailing, it
would be best for everyone if you just subscribe to the mailing list
normally and not use the digest. The digest is good for read-only use and
maybe if you only occasionally reply. If you want to participate in an
extended discussion, however, it would really help everyone (yourself
included!) if you don't use the digest.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180130/68b3d6df/attachment.html>

From rlucente at pipeline.com  Mon Jan 29 14:36:28 2018
From: rlucente at pipeline.com (Robert Lucente)
Date: Mon, 29 Jan 2018 14:36:28 -0500 (GMT-05:00)
Subject: [SciPy-Dev] email client is misconfigured - replies are
 breaking the threading
Message-ID: <712242179.8971.1517254588928@wamui-cinderella.atl.sa.earthlink.net>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180129/98ae9fe0/attachment.html>

From robert.kern at gmail.com  Mon Jan 29 14:42:34 2018
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 30 Jan 2018 04:42:34 +0900
Subject: [SciPy-Dev] email client is misconfigured - replies are
 breaking the threading
In-Reply-To: <712242179.8971.1517254588928@wamui-cinderella.atl.sa.earthlink.net>
References: <712242179.8971.1517254588928@wamui-cinderella.atl.sa.earthlink.net>
Message-ID: <CAF6FJiu6_jejtqcf1LaoN5i57riozBi7B1_Z5_zrjmKdipA7-Q@mail.gmail.com>

On Tue, Jan 30, 2018 at 4:36 AM, Robert Lucente <rlucente at pipeline.com>
wrote:

> >subscribe to the mailing list normally and not use the digest
> Okay
>
> I went to
> https://mail.scipy.org/mailman/options/scipy-dev
>
> and set "Set Digest Mode" to Off.

Thank you! I appreciate it! I hope that the adjustment in your email
workflow goes smoothly for you.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180130/4f5effd1/attachment.html>

From lagru at mailbox.org  Tue Jan 30 08:19:16 2018
From: lagru at mailbox.org (Lars G.)
Date: Tue, 30 Jan 2018 14:19:16 +0100
Subject: [SciPy-Dev] ENH: Extend peak finding capabilities in
 scipy.signal (#8264)
In-Reply-To: <CAGu2niWO_sR=oFxHUWzsGus5QW9Eh=X-Bd_G3cYatUUuYpm2xA@mail.gmail.com>
References: <d3fac7a9-f1cf-bedc-cb55-967e94ba9ea8@mailbox.org>
 <CAGu2niWO_sR=oFxHUWzsGus5QW9Eh=X-Bd_G3cYatUUuYpm2xA@mail.gmail.com>
Message-ID: <6d6d414a-bc6c-3752-0ec3-5630498c9442@mailbox.org>

On 26.01.2018 20:59, Eric Larson wrote:
>> - argrelmax doesn't catch peaks that are wider than one sample. Decide
>> how to deal with this. I have implemented an alternative here which may
>> even be faster. But I plan to make a performance comparison to confirm
>> this. I have a feeling this should be addressed (if wanted) in a new
>> pull request.
> 
> Another PR is fine, but we should merge that one before this find_peaks
> PR to avoid creating backward compatibility problems.?

Actually I have changed my mind about submitting the alternative in
another PR. My current solution
http://nbviewer.jupyter.org/urls/gitlab.com/snippets/1695752/raw#Proposed-solution-2:-Cython
compares to `argrelmax` as follows:

- It is measurably faster than `argrelmax` for all
  supported cases I could think of.
- It can find flat maxima / peaks which is my main
  gripe with `argrelmax`.
- It uses Cython with all its implications. I have a
  feeling that Python is preferred over creating
  another file to compile but I think usage of
  Cython is well-founded here. Dealing with flat peaks
  was really costly in my pure-Python experiments.
- It only supports 1D arrays (which honestly is the
  main use case for the signal module).

Because this function will be wrapped by find_peaks (or however choose
to name it) it doesn't really need to be exposed to the user
`find_peaks(x)` would be equivalent to `_maxima1D(x)`.

If nobody has any objections I'll include this as part of my original PR.

Regards, Lars


From larson.eric.d at gmail.com  Tue Jan 30 09:57:50 2018
From: larson.eric.d at gmail.com (Eric Larson)
Date: Tue, 30 Jan 2018 09:57:50 -0500
Subject: [SciPy-Dev] ENH: Extend peak finding capabilities in
 scipy.signal (#8264)
In-Reply-To: <6d6d414a-bc6c-3752-0ec3-5630498c9442@mailbox.org>
References: <d3fac7a9-f1cf-bedc-cb55-967e94ba9ea8@mailbox.org>
 <CAGu2niWO_sR=oFxHUWzsGus5QW9Eh=X-Bd_G3cYatUUuYpm2xA@mail.gmail.com>
 <6d6d414a-bc6c-3752-0ec3-5630498c9442@mailbox.org>
Message-ID: <CAGu2niW066Cf6AtTLfjYSUAnJyAdpkqCZ28RsteG88kGK1MB_g@mail.gmail.com>

>
> Actually I have changed my mind about submitting the alternative in
> another PR.


Separate PRs can reduce review burden. In this case, though, it seems like
the code is simple enough that it could be included. We just need to make
sure we sufficiently thoroughly test the compiled function for possible
failure modes.

- It is ...
>

All of this sounds fine to me. The compiled code is worth it for speed
gains, assuming its complexity (and thus maintenance burden) is not too
high. Looking at it, it looks justified.

Assuming nobody objects to this path, feel free to implement and ping me
(larsoner) on GitHub for a review when it's ready.

Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180130/e578a2fc/attachment.html>

From larson.eric.d at gmail.com  Tue Jan 30 11:43:22 2018
From: larson.eric.d at gmail.com (Eric Larson)
Date: Tue, 30 Jan 2018 11:43:22 -0500
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <CABL7CQi5g===ntdphLV9BybWSACv_T2d4Ot0WOmKBpdCRL_+XA@mail.gmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <20180126074429.3csbkpsg7hbme22g@fastmail.com>
 <CABL7CQi5g===ntdphLV9BybWSACv_T2d4Ot0WOmKBpdCRL_+XA@mail.gmail.com>
Message-ID: <CAGu2niWyHjU-v2QxnfeHOk56JOsXuhX1mAne_9hMXo5KiuSdWg@mail.gmail.com>

>
> Looks like for coordinating we have the following team: Tyler, Josh,
> Evgeni, Matt, Stefan, Ralf. Let's start a bit of planning and setting up
> the structure of and build tools for the paper - I'll send out a separate
> email to you.
>

Sorry for my delayed response -- I'm also happy to help coordinate and
write, as necessary. If you need an extra hand let me know.

Best,
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180130/6368aed4/attachment.html>

From serge.guelton at telecom-bretagne.eu  Tue Jan 30 14:57:25 2018
From: serge.guelton at telecom-bretagne.eu (Serge Guelton)
Date: Tue, 30 Jan 2018 20:57:25 +0100
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <CAGu2niWyHjU-v2QxnfeHOk56JOsXuhX1mAne_9hMXo5KiuSdWg@mail.gmail.com>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <20180126074429.3csbkpsg7hbme22g@fastmail.com>
 <CABL7CQi5g===ntdphLV9BybWSACv_T2d4Ot0WOmKBpdCRL_+XA@mail.gmail.com>
 <CAGu2niWyHjU-v2QxnfeHOk56JOsXuhX1mAne_9hMXo5KiuSdWg@mail.gmail.com>
Message-ID: <20180130195725.GC20306@lakota>

On Tue, Jan 30, 2018 at 11:43:22AM -0500, Eric Larson wrote:
>     Looks like for coordinating we have the following team: Tyler, Josh,
>     Evgeni, Matt, Stefan, Ralf. Let's start a bit of planning and setting up
>     the structure of and build tools for the paper - I'll send out a separate
>     email to you.
> 
> 
> Sorry for my delayed response -- I'm also happy to help coordinate and write,
> as necessary. If you need an extra hand let me know.

Don't know if that's relevant, but if there's a section on native
extension, and although I'm a newbie here, I'll be happy to participate.

From stefanv at berkeley.edu  Tue Jan 30 15:08:53 2018
From: stefanv at berkeley.edu (Stefan van der Walt)
Date: Tue, 30 Jan 2018 12:08:53 -0800
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <20180130195725.GC20306@lakota>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <20180126074429.3csbkpsg7hbme22g@fastmail.com>
 <CABL7CQi5g===ntdphLV9BybWSACv_T2d4Ot0WOmKBpdCRL_+XA@mail.gmail.com>
 <CAGu2niWyHjU-v2QxnfeHOk56JOsXuhX1mAne_9hMXo5KiuSdWg@mail.gmail.com>
 <20180130195725.GC20306@lakota>
Message-ID: <20180130200853.p57s7iahoiv7nb5l@fastmail.com>

On Tue, 30 Jan 2018 20:57:25 +0100, Serge Guelton wrote:
> On Tue, Jan 30, 2018 at 11:43:22AM -0500, Eric Larson wrote:
> >     Looks like for coordinating we have the following team: Tyler, Josh,
> >     Evgeni, Matt, Stefan, Ralf. Let's start a bit of planning and setting up
> >     the structure of and build tools for the paper - I'll send out a separate
> >     email to you.
> > 
> > 
> > Sorry for my delayed response -- I'm also happy to help coordinate and write,
> > as necessary. If you need an extra hand let me know.
> 
> Don't know if that's relevant, but if there's a section on native
> extension, and although I'm a newbie here, I'll be happy to
> participate.

Extensions, and here I'm thinking interoperability with Cython, Numba,
and C via LowLevelCallable, should certainly be considered for
inclusion.  But we'll need to do an outline of the paper first before
knowing to what depth to drill down.

St?fan

From ilhanpolat at gmail.com  Tue Jan 30 15:10:48 2018
From: ilhanpolat at gmail.com (Ilhan Polat)
Date: Tue, 30 Jan 2018 21:10:48 +0100
Subject: [SciPy-Dev] SciPy 1.0 paper writing proposal
In-Reply-To: <20180130195725.GC20306@lakota>
References: <CABL7CQi9Wa-JLyvqx4g15Kkj3pV0EuRijkNWV=wuKbzKpos3WQ@mail.gmail.com>
 <20180126074429.3csbkpsg7hbme22g@fastmail.com>
 <CABL7CQi5g===ntdphLV9BybWSACv_T2d4Ot0WOmKBpdCRL_+XA@mail.gmail.com>
 <CAGu2niWyHjU-v2QxnfeHOk56JOsXuhX1mAne_9hMXo5KiuSdWg@mail.gmail.com>
 <20180130195725.GC20306@lakota>
Message-ID: <CAEBuzr8Mya4d0CTcNQKMKupde2vbuZbCF9mMHziggqGakw6U2g@mail.gmail.com>

Same as Eric also if you need some TeX stuff, I have a very broad and
useless TeX experience.

On Tue, Jan 30, 2018 at 8:57 PM, Serge Guelton <
serge.guelton at telecom-bretagne.eu> wrote:

> On Tue, Jan 30, 2018 at 11:43:22AM -0500, Eric Larson wrote:
> >     Looks like for coordinating we have the following team: Tyler, Josh,
> >     Evgeni, Matt, Stefan, Ralf. Let's start a bit of planning and
> setting up
> >     the structure of and build tools for the paper - I'll send out a
> separate
> >     email to you.
> >
> >
> > Sorry for my delayed response -- I'm also happy to help coordinate and
> write,
> > as necessary. If you need an extra hand let me know.
>
> Don't know if that's relevant, but if there's a section on native
> extension, and although I'm a newbie here, I'll be happy to participate.
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180130/e262fada/attachment.html>

From haberland at ucla.edu  Tue Jan 30 18:26:47 2018
From: haberland at ucla.edu (Matt Haberland)
Date: Tue, 30 Jan 2018 15:26:47 -0800
Subject: [SciPy-Dev] What we do struggle with is lack of progress on
 big-ticket items
In-Reply-To: <CAEBuzr-2KG_a=v94rZVM95a5Y-e4TVv2CJ-G7DVRQ_8y428L7g@mail.gmail.com>
References: <1075549019.8362.1516663539259@wamui-cinderella.atl.sa.earthlink.net>
 <CAAbtOZfHbKh13m=cmZjHAvRE=q98Zut5=umtr_nqOb6iArss9g@mail.gmail.com>
 <14f101d39430$84b127e0$8e1377a0$@pipeline.com>
 <20180123192038.mgenugikitd2x4wy@fastmail.com>
 <CAB6mnxLQqcdnwGi8nURH=q3nYd--+XJsZMq3R8yisj=6cJDokA@mail.gmail.com>
 <CADuxUiyZaiAAfz_gW8zJv8PMfPfQkfztgQ50-as-_BXGyfdkJw@mail.gmail.com>
 <CAEBuzr-2KG_a=v94rZVM95a5Y-e4TVv2CJ-G7DVRQ_8y428L7g@mail.gmail.com>
Message-ID: <CADuxUixKVbxwyCu4EabQLQDcGooeD-0WFn2-Qt=BPjV0hccn4g@mail.gmail.com>

Thanks, Ilhan. I tried those instructions and left feedback at:
https://github.com/scipy/scipy/issues/7140

In summary, there were some snags (which I noted) but eventually scipy
reported building successfully. Definitely easier than some old Cygwin
instructions i tried about a year ago. The tests crashed, however,
reporting a segfault. : (

Matt

On Tue, Jan 23, 2018 at 4:50 PM, Ilhan Polat <ilhanpolat at gmail.com> wrote:

>
> I'd be happy to make some videos with a screen recording and voice-over. I
>> want to start from scratch on Mac, as my environment seems to have broken
>> over the past few months, and I'd like to try again on Windows, which I
>> never got working. If someone with real experience on one or both of those
>> platforms has some time to answer questions that inevitably arise, I'd be
>> willing to work on this immediately.
>>
>>
> Matt, please have a look at this. All feedback is welcome
>
> https://scipy.github.io/devdocs/building/windows.html
>
>
>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>


-- 
Matt Haberland
Assistant Adjunct Professor in the Program in Computing
Department of Mathematics
6617A Math Sciences Building, UCLA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180130/dc1b6cf3/attachment.html>

From serge.guelton at telecom-bretagne.eu  Wed Jan 31 06:42:23 2018
From: serge.guelton at telecom-bretagne.eu (serge guelton)
Date: Wed, 31 Jan 2018 12:42:23 +0100
Subject: [SciPy-Dev] Using Pythran to compile some of the scipy internals
In-Reply-To: <20180129113402.GA2065@lakota>
References: <20180120101034.GA3707@lakota>
 <CABL7CQgr8AWjoQD7TV+tcb9a5wcfnrf+ttCb-hGhZKWha0WUXQ@mail.gmail.com>
 <20180122142840.GA22566@lakota> <20180129113402.GA2065@lakota>
Message-ID: <20180131114223.GA20197@lakota>

On Mon, Jan 29, 2018 at 12:34:02PM +0100, serge guelton wrote:
> On Mon, Jan 22, 2018 at 03:28:40PM +0100, Serge Guelton wrote:
> > > Another potential benefit is to decrease the size of the binary distribution of
> > > SciPy. Cython extensions are quite expensive in this respect. Do you have an
> > > idea of how Pythran compares? Both in case of only float64 inputs, and with
> > > templated inputs? A good example of the latter is scipy.ndimage.label, which is
> > > a straightforward function that ends up being a 700kb .so
> > 
> > Thanks for pointing out this aspect. There's still room for improvement
> > here, but with current pythran's master, and same compilation flags, the
> > cython implementation of `max_len_seq_inner` takes ~700kb while
> > pythran's version takes ~200kb.
> > 
> > I'm going to investigate that a bet more though, and probably post the
> > results in this thread later on.
> 
> Hey scip-dev, finally the result of some dev + thoughts on
> Pythran-generated code:
> 
>     http://serge-sans-paille.github.io/pythran-stories/shrinking-pythran-generated-binaries.html
> 
> I'll move on to Windows support now :-)

Just to make sure: which toolchain is used to build scipy on windows:
mingw or msvc? What's the default build environment? WinPython or the
official Python binaries from Python.org?

From olivier.grisel at ensta.org  Wed Jan 31 09:33:29 2018
From: olivier.grisel at ensta.org (Olivier Grisel)
Date: Wed, 31 Jan 2018 15:33:29 +0100
Subject: [SciPy-Dev] Using Pythran to compile some of the scipy internals
In-Reply-To: <20180131114223.GA20197@lakota>
References: <20180120101034.GA3707@lakota>
 <CABL7CQgr8AWjoQD7TV+tcb9a5wcfnrf+ttCb-hGhZKWha0WUXQ@mail.gmail.com>
 <20180122142840.GA22566@lakota> <20180129113402.GA2065@lakota>
 <20180131114223.GA20197@lakota>
Message-ID: <CAFvE7K7qQ0KJOodWKX3ZUVtussgZfPQwRLimMNCFZ6VYB+EC8A@mail.gmail.com>

msvc is the only supported toolchain to generate wheels that are compatible
with the official binary distribution of Python from python.org and
therefore the only compiler that can generate wheels suitable for
distribution PyPI.

For for the Fortran bits the story is more complicated but I guess that
should not have any impact on the use pythran.
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180131/9633ecf3/attachment.html>

From serge.guelton at telecom-bretagne.eu  Wed Jan 31 11:49:25 2018
From: serge.guelton at telecom-bretagne.eu (Serge Guelton)
Date: Wed, 31 Jan 2018 17:49:25 +0100
Subject: [SciPy-Dev] Using Pythran to compile some of the scipy internals
In-Reply-To: <CAFvE7K7qQ0KJOodWKX3ZUVtussgZfPQwRLimMNCFZ6VYB+EC8A@mail.gmail.com>
References: <20180120101034.GA3707@lakota>
 <CABL7CQgr8AWjoQD7TV+tcb9a5wcfnrf+ttCb-hGhZKWha0WUXQ@mail.gmail.com>
 <20180122142840.GA22566@lakota> <20180129113402.GA2065@lakota>
 <20180131114223.GA20197@lakota>
 <CAFvE7K7qQ0KJOodWKX3ZUVtussgZfPQwRLimMNCFZ6VYB+EC8A@mail.gmail.com>
Message-ID: <20180131164925.GA28870@lakota>

On Wed, Jan 31, 2018 at 03:33:29PM +0100, Olivier Grisel wrote:
> msvc is the only supported toolchain to generate wheels that are compatible
> with the official binary distribution of Python from python.org and therefore
> the only compiler that can generate wheels suitable for distribution PyPI.

OK. I have no trouble using the VS2017 toolchain fro python3, I hope I'm not bound to VS2008 (say for Python 2.7)?

> For for the Fortran bits the story is more complicated but I guess that should
> not have any impact on the use pythran.

In spite of the name similarity, you're correct!

From olivier.grisel at ensta.org  Wed Jan 31 11:55:27 2018
From: olivier.grisel at ensta.org (Olivier Grisel)
Date: Wed, 31 Jan 2018 17:55:27 +0100
Subject: [SciPy-Dev] Using Pythran to compile some of the scipy internals
In-Reply-To: <20180131164925.GA28870@lakota>
References: <20180120101034.GA3707@lakota>
 <CABL7CQgr8AWjoQD7TV+tcb9a5wcfnrf+ttCb-hGhZKWha0WUXQ@mail.gmail.com>
 <20180122142840.GA22566@lakota> <20180129113402.GA2065@lakota>
 <20180131114223.GA20197@lakota>
 <CAFvE7K7qQ0KJOodWKX3ZUVtussgZfPQwRLimMNCFZ6VYB+EC8A@mail.gmail.com>
 <20180131164925.GA28870@lakota>
Message-ID: <CAFvE7K6xzHVFmsHcyBFf-Rpwv92QWhh+APGPY4EnPUTPeuuphA@mail.gmail.com>

2018-01-31 17:49 GMT+01:00 Serge Guelton <serge.guelton at telecom-bretagne.eu>
:

> On Wed, Jan 31, 2018 at 03:33:29PM +0100, Olivier Grisel wrote:
> > msvc is the only supported toolchain to generate wheels that are
> compatible
> > with the official binary distribution of Python from python.org and
> therefore
> > the only compiler that can generate wheels suitable for distribution
> PyPI.
>
> OK. I have no trouble using the VS2017 toolchain fro python3, I hope I'm
> not bound to VS2008 (say for Python 2.7)?


Unfortunately yes.  But I guess Python 2.7 support for new feature releases
of scipy will be phased out in 2019 or so, so maybe it's not that a big
problem to not provide pythran-based acceleration for Python 2. Beter not
waste too much time on Python 2 ;)

-- 
Olivier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180131/c4d1f489/attachment.html>