From christoph.baumgarten at gmail.com  Sat Jun  2 03:33:21 2018
From: christoph.baumgarten at gmail.com (Christoph Baumgarten)
Date: Sat, 2 Jun 2018 09:33:21 +0200
Subject: [SciPy-Dev] Help with PR 8681: import
Message-ID: <CABXY2qDPidvkhtgjsGm+9BTLJqWtObaBShJr067nH6qX91vLeA@mail.gmail.com>

Hi,

I would like to ask for help with PR 8681.

I want to use the function rvs_ratio_unif from stats/stats.py to generate
random variates in stats/_continuous_distributions.py

In_continuous_distributions.py, I use "from . import stats" which does not
work with 2.7 and 3.4, I assume it is a circular reference from the error
message below:


___________ ERROR collecting scipy/signal/tests/test_array_tools.py
____________
ImportError while importing test module
'/home/travis/virtualenv/python3.4.6/lib/python3.4/site-packages/scipy/signal/tests/test_array_tools.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/travis/virtualenv/python3.4.6/lib/python3.4/site-packages/scipy/signal/__init__.py:329:
in <module>
    from ._peak_finding import *
/home/travis/virtualenv/python3.4.6/lib/python3.4/site-packages/scipy/signal/_peak_finding.py:11:
in <module>
    from scipy.stats import scoreatpercentile
/home/travis/virtualenv/python3.4.6/lib/python3.4/site-packages/scipy/stats/__init__.py:348:
in <module>
    from .stats import *
/home/travis/virtualenv/python3.4.6/lib/python3.4/site-packages/scipy/stats/stats.py:175:
in <module>
    from . import distributions
/home/travis/virtualenv/python3.4.6/lib/python3.4/site-packages/scipy/stats/distributions.py:13:
in <module>
    from . import _continuous_distns
/home/travis/virtualenv/python3.4.6/lib/python3.4/site-packages/scipy/stats/_continuous_distns.py:18:
in <module>
    from . import stats
E   ImportError: cannot import name 'stats'

Thanks

Christoph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180602/5302e664/attachment.html>

From robert.kern at gmail.com  Sat Jun  2 03:57:19 2018
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 2 Jun 2018 00:57:19 -0700
Subject: [SciPy-Dev] Help with PR 8681: import
In-Reply-To: <CABXY2qDPidvkhtgjsGm+9BTLJqWtObaBShJr067nH6qX91vLeA@mail.gmail.com>
References: <CABXY2qDPidvkhtgjsGm+9BTLJqWtObaBShJr067nH6qX91vLeA@mail.gmail.com>
Message-ID: <CAF6FJivgH5SfiT9fjmFnFAKfFsyp-MZrh3RKgS65TeGU-CqyEA@mail.gmail.com>

On Sat, Jun 2, 2018 at 12:34 AM Christoph Baumgarten <
christoph.baumgarten at gmail.com> wrote:
>
> Hi,
>
> I would like to ask for help with PR 8681.
>
> I want to use the function rvs_ratio_unif from stats/stats.py to generate
random variates in stats/_continuous_distributions.py
>
> In_continuous_distributions.py, I use "from . import stats" which does
not work with 2.7 and 3.4, I assume it is a circular reference from the
error message below:

Yes. You can't use `scipy.stats.stats` in `scipy.stats._continuous_distns`.
At least, not as a top-level import. You might locally import
`rvs_ratio_unif` inside the method where it is used, with appropriate
comments. Otherwise, you may find another place to define that function
where both modules can import it.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180602/9275aa75/attachment.html>

From mikofski at berkeley.edu  Sat Jun  2 12:57:01 2018
From: mikofski at berkeley.edu (Mark Alexander Mikofski)
Date: Sat, 2 Jun 2018 09:57:01 -0700
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
Message-ID: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>

Hi All,

I am having issues using

    python runtest.py

from scipy master branch since commit
14de485c053cc275682bc7de0316fcca72b341f8
https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8

there are two tracebacks, I think this may have something to do with
changes to LAPACK and dropping Apple Accelerate support, but not sure, and
don't know how to fix it. Should I brew install openblas? (I'm going to try
that) Use anaconda and create a conda env? (maybe this next?) Currently I'm
using python 3.6 from homebrew in a virtualenv with wheels from PyPI, after
I create the virtualenv using the homebrew python-3 interpreter, I do pip
install requirements.txt (see attached).

1st traceback:

ImportError while importing test module
'/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
scipy/cluster/__init__.py:27: in <module>
    from . import vq, hierarchy
scipy/cluster/vq.py:75: in <module>
    from scipy.spatial.distance import cdist
scipy/spatial/__init__.py:97: in <module>
    from ._spherical_voronoi import SphericalVoronoi
scipy/spatial/_spherical_voronoi.py:19: in <module>
    from scipy.spatial.distance import pdist
scipy/spatial/distance.py:123: in <module>
    from ..linalg import norm
scipy/linalg/__init__.py:207: in <module>
    from ._decomp_update import *
_decomp_update.pyx:1: in init scipy.linalg._decomp_update
    ???
E   ImportError:
dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/
cython_lapack.cpython-36m-darwin.so, 2): Symbol not found: _cbbcsd_
E     Referenced from:
/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/
cython_lapack.cpython-36m-darwin.so
E     Expected in: flat namespace
E    in
/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/
cython_lapack.cpython-36m-darwin.so

2nd traceback:

ImportError while importing test module
'/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/integrate/tests/test_banded_ode_solvers.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
scipy/integrate/__init__.py:93: in <module>
    from ._bvp import solve_bvp
scipy/integrate/_bvp.py:10: in <module>
    from scipy.sparse.linalg import splu
scipy/sparse/linalg/__init__.py:114: in <module>
    from .isolve import *
scipy/sparse/linalg/isolve/__init__.py:8: in <module>
    from .lgmres import lgmres
scipy/sparse/linalg/isolve/lgmres.py:13: in <module>
    from ._gcrotmk import _fgmres
scipy/sparse/linalg/isolve/_gcrotmk.py:10: in <module>
    from scipy.linalg import (get_blas_funcs, qr, solve, svd, qr_insert,
lstsq)
E   ImportError: cannot import name 'qr_insert'

I apologize for not trying to research and trouble shoot this more myself,
but I've a tight window while my kids are at gymnastics to go running, and
you all know the drill, just thought I'd ask to see if anyone had ever run
across this issue before.

Thanks for your help! Sorry for the intrusion!

All the best,
Mark

-- 
Mark Mikofski, PhD (2005)
*Fiat Lux*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180602/5cef119e/attachment.html>
-------------- next part --------------
appnope==0.1.0
asv==0.2.1
attrs==17.4.0
coverage==4.5.1
Cython==0.28.1
decorator==4.2.1
ipython==6.2.1
ipython-genutils==0.2.0
jedi==0.11.1
more-itertools==4.1.0
nose==1.3.7
numpy==1.14.3
parso==0.1.1
pexpect==4.4.0
pickleshare==0.7.4
pluggy==0.6.0
prompt-toolkit==1.0.15
ptyprocess==0.5.2
py==1.5.3
Pygments==2.2.0
pytest==3.5.0
simplegeneric==0.8.1
six==1.11.0
traitlets==4.3.2
wcwidth==0.1.7

From ralf.gommers at gmail.com  Sat Jun  2 13:05:18 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 2 Jun 2018 10:05:18 -0700
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
Message-ID: <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>

On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski <
mikofski at berkeley.edu> wrote:

> Hi All,
>
> I am having issues using
>
>     python runtest.py
>
> from scipy master branch since commit 14de485c053cc275682bc7de0316fc
> ca72b341f8
> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fc
> ca72b341f8
>
> there are two tracebacks, I think this may have something to do with
> changes to LAPACK and dropping Apple Accelerate support, but not sure, and
> don't know how to fix it. Should I brew install openblas? (I'm going to try
> that)
>

Yes that is the recommendation (if you're not using Anaconda).



> Use anaconda and create a conda env? (maybe this next?)
>

If you're already using Anaconda, and the numpy from the defaults channel,
SciPy should automatically link against MKL.



> Currently I'm using python 3.6 from homebrew in a virtualenv with wheels
> from PyPI, after I create the virtualenv using the homebrew python-3
> interpreter, I do pip install requirements.txt (see attached).
>
> 1st traceback:
>
> ImportError while importing test module '/Users/markmikofski/Projects/
> scipy/build/testenv/lib/python3.6/site-packages/scipy/
> cluster/tests/test_hierarchy.py'.
> Hint: make sure your test modules/packages have valid Python names.
> Traceback:
> scipy/cluster/__init__.py:27: in <module>
>     from . import vq, hierarchy
> scipy/cluster/vq.py:75: in <module>
>     from scipy.spatial.distance import cdist
> scipy/spatial/__init__.py:97: in <module>
>     from ._spherical_voronoi import SphericalVoronoi
> scipy/spatial/_spherical_voronoi.py:19: in <module>
>     from scipy.spatial.distance import pdist
> scipy/spatial/distance.py:123: in <module>
>     from ..linalg import norm
> scipy/linalg/__init__.py:207: in <module>
>     from ._decomp_update import *
> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>     ???
> E   ImportError: dlopen(/Users/markmikofski/Projects/scipy/build/testenv/
> lib/python3.6/site-packages/scipy/linalg/cython_lapack.
> cpython-36m-darwin.so, 2): Symbol not found: _cbbcsd_
> E     Referenced from: /Users/markmikofski/Projects/
> scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/
> cython_lapack.cpython-36m-darwin.so
> E     Expected in: flat namespace
> E    in /Users/markmikofski/Projects/scipy/build/testenv/lib/
> python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>

This is with an already built SciPy right, without rebuilding? That would
explain the uninformative traceback.

Cheers,
Ralf



>
> 2nd traceback:
>
> ImportError while importing test module '/Users/markmikofski/Projects/
> scipy/build/testenv/lib/python3.6/site-packages/scipy/
> integrate/tests/test_banded_ode_solvers.py'.
> Hint: make sure your test modules/packages have valid Python names.
> Traceback:
> scipy/integrate/__init__.py:93: in <module>
>     from ._bvp import solve_bvp
> scipy/integrate/_bvp.py:10: in <module>
>     from scipy.sparse.linalg import splu
> scipy/sparse/linalg/__init__.py:114: in <module>
>     from .isolve import *
> scipy/sparse/linalg/isolve/__init__.py:8: in <module>
>     from .lgmres import lgmres
> scipy/sparse/linalg/isolve/lgmres.py:13: in <module>
>     from ._gcrotmk import _fgmres
> scipy/sparse/linalg/isolve/_gcrotmk.py:10: in <module>
>     from scipy.linalg import (get_blas_funcs, qr, solve, svd, qr_insert,
> lstsq)
> E   ImportError: cannot import name 'qr_insert'
>
> I apologize for not trying to research and trouble shoot this more myself,
> but I've a tight window while my kids are at gymnastics to go running, and
> you all know the drill, just thought I'd ask to see if anyone had ever run
> across this issue before.
>
> Thanks for your help! Sorry for the intrusion!
>
> All the best,
> Mark
>
> --
> Mark Mikofski, PhD (2005)
> *Fiat Lux*
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180602/ac4e4340/attachment-0001.html>

From adibhar97 at gmail.com  Sun Jun  3 16:22:51 2018
From: adibhar97 at gmail.com (Aditya Bharti)
Date: Mon, 4 Jun 2018 01:52:51 +0530
Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3
 dimensions
In-Reply-To: <CANRzPHUynj4LSmpUeBMzY-my1EmGKUp944mdy_tyqOLGY5Fmrw@mail.gmail.com>
References: <163213d9b99.e31ddfbf140246.3867213885941953730@zoho.com>
 <CANRzPHVN47mDJqc4DuUq70X9sAiRqBRWOSHszgtL3QyM1O6ihA@mail.gmail.com>
 <CANRzPHV0BswFBevgZ2fsf2vKm_J_LQEET=qtDZ1WO6H7dMDNyQ@mail.gmail.com>
 <CAB2ViTbimDdS0jZ3Qp71rorkhfALBhHQtTztv1OD11dY7V1ADg@mail.gmail.com>
 <CANRzPHUynj4LSmpUeBMzY-my1EmGKUp944mdy_tyqOLGY5Fmrw@mail.gmail.com>
Message-ID: <CANRzPHXWbx=YN4LDtetKj-P3kaT-mO9MbPwUnX1xHaPNi9vqbQ@mail.gmail.com>

Hi all,
This week was mainly spent in implementing the `as_euler` function, which
is responsible for returning an Euler angle representation for the
underlying rotation. The implementation took some time (we're still ironing
out the gritty details) for two major reasons:

   - There are a few conventions for Euler angles and rotations in general.
   Figuring out which convention this paper
   <https://arc.aiaa.org/doi/abs/10.2514/1.16622> followed and then
   extending the algorithm to extrinsic and intrinsic rotations was time
   consuming.
   - The infamous *Gimbal Lock*. While converting from Euler angles this
   presents no problems, converting to euler angles presents issues. In this
   case it is not possible to uniquely determine all the angles and algorithms
   become numerically unstable, which leads to small tweaks (which should work
   in theory) breaking our implementation by changing the sign of small
   quantities close to zero.

This function is the last representation related function of the class,
after which we move on to compositions, applications, and inversions of
general Rotation objects. You can go through the code
<https://github.com/adbugger/scipy/tree/rotation/scipy/spatial/transform>
or check out my blog <https://blogs.python-gsoc.org/aditya-bharti> to
follow along. Stay tuned for more updates.

Thanks,
Aditya

On Wed, 30 May 2018 at 02:51, Aditya Bharti <adibhar97 at gmail.com> wrote:

> Hi all,
> Continuing the work so far, the following have been implemented this week:
>
>    - `from_rotvec`, `as_rotvec`: Representing Euler angles as rotation
>    vectors, with appropriate Taylor series expansions for small angles
>    - `from_euler`: Initialization from Euler angles, along with a string
>    based axis sequence specification. Refer to docs for more details.
>
> As always, the project lives here
> <https://github.com/adbugger/scipy/tree/rotation/scipy/spatial/transform>,
> and my personal experiences can be found on the blog
> <https://blogs.python-gsoc.org/aditya-bharti>.
>
> Thanks,
> Aditya
>
> On Sun, 20 May 2018 at 13:28, Phillip Feldman <phillip.m.feldman at gmail.com>
> wrote:
>
>> When you say "discrete cosine matrix", I think that you mean "direction
>> cosine matrix" (see
>> https://en.wikiversity.org/wiki/PlanetPhysics/Direction_Cosine_Matrix).
>>
>> Phillip
>>
>> On Sat, May 19, 2018 at 12:23 PM, Aditya Bharti <adibhar97 at gmail.com>
>> wrote:
>>
>>> Hi all,
>>> So this concludes week 1 of GSoC 2018. I must say it was a great
>>> learning experience and I invite you all to check out my account of this
>>> week on the blog <https://blogs.python-gsoc.org/aditya-bharti/>. This
>>> email is more of a technical update.
>>>
>>>    - So, the main `Rotation` class will live under a new sub module
>>>    `scipy.spatial.transform`.
>>>    - Conversion between quaternions and discrete cosine matrices was
>>>    implemented.
>>>    - The rotation class now supports `from_quaternion`, `from_dcm`,
>>>    `as_quaternion` and `as_dcm`, with support for multiple rotations in one
>>>    call.
>>>
>>> The project currently lives in my own fork of scipy here
>>> <https://github.com/adbugger/scipy/tree/rotation/scipy/spatial/transform>.
>>> Stay tuned for more updates!
>>>
>>> Best,
>>> Aditya
>>>
>>> On Wed, 2 May 2018 at 21:03, Aditya Bharti <adibhar97 at gmail.com> wrote:
>>>
>>>> Hi Nikolay,
>>>>
>>>> I've used Wordpress only once before, so I don't know much about it.
>>>> From my limited experience, it is extremely customizable. You can customize
>>>> every thing from the look and feel to SEO characteristics. There are
>>>> apparently a lot of wordpress plugins for these kind of tasks. For this
>>>> particular blog, PSF had already setup an account for me with a site on it.
>>>> All I had to do was click on the 'New' button and open up the new post
>>>> page. There's a field for a header and body text, with options for adding
>>>> audio, video and hyperlinks.
>>>>
>>>> As regards to the post itself, sure I'll expand it with a brief
>>>> overview, motivation and an example. Note that the example will only show
>>>> sample usage, not any internals. I plan to borrow heavily from my proposal
>>>> for this purpose, I hope that's ok.
>>>>
>>>> Regards,
>>>> Aditya
>>>>
>>>> On 2 May 2018 at 19:54, Nikolay Mayorov <nikolay.mayorov at zoho.com>
>>>> wrote:
>>>>
>>>>> Hi, Aditya!
>>>>>
>>>>> Glad that you set up the blog and good job on setting up the
>>>>> documentation build as well.
>>>>>
>>>>> Curious, what is this blogging platform like? How do you create posts
>>>>> in it?
>>>>>
>>>>> As for your first post: while not strictly necessary I think it would
>>>>> be nice to see a more thorough introductory post with a brief overview,
>>>>> motivation and/or an example. Do you want to work on it?
>>>>>
>>>>> Best,
>>>>> Nikolay
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-Dev mailing list
>>>>> SciPy-Dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>
>>>>>
>>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180604/c612648e/attachment.html>

From phillip.m.feldman at gmail.com  Mon Jun  4 00:14:28 2018
From: phillip.m.feldman at gmail.com (Phillip Feldman)
Date: Sun, 3 Jun 2018 21:14:28 -0700
Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3
 dimensions
In-Reply-To: <CANRzPHUynj4LSmpUeBMzY-my1EmGKUp944mdy_tyqOLGY5Fmrw@mail.gmail.com>
References: <163213d9b99.e31ddfbf140246.3867213885941953730@zoho.com>
 <CANRzPHVN47mDJqc4DuUq70X9sAiRqBRWOSHszgtL3QyM1O6ihA@mail.gmail.com>
 <CANRzPHV0BswFBevgZ2fsf2vKm_J_LQEET=qtDZ1WO6H7dMDNyQ@mail.gmail.com>
 <CAB2ViTbimDdS0jZ3Qp71rorkhfALBhHQtTztv1OD11dY7V1ADg@mail.gmail.com>
 <CANRzPHUynj4LSmpUeBMzY-my1EmGKUp944mdy_tyqOLGY5Fmrw@mail.gmail.com>
Message-ID: <CAB2ViTYKO285BtwFTjOR4brjQJE_3SJ0d4jSkWeaUp-DJc8ing@mail.gmail.com>

"Refer to docs for more details."  Will there be user documentation beyond
the doc strings in the various methods?

On Tue, May 29, 2018 at 2:21 PM, Aditya Bharti <adibhar97 at gmail.com> wrote:

> Hi all,
> Continuing the work so far, the following have been implemented this week:
>
>    - `from_rotvec`, `as_rotvec`: Representing Euler angles as rotation
>    vectors, with appropriate Taylor series expansions for small angles
>    - `from_euler`: Initialization from Euler angles, along with a string
>    based axis sequence specification. Refer to docs for more details.
>
> As always, the project lives here
> <https://github.com/adbugger/scipy/tree/rotation/scipy/spatial/transform>,
> and my personal experiences can be found on the blog
> <https://blogs.python-gsoc.org/aditya-bharti>.
>
> Thanks,
> Aditya
>
> On Sun, 20 May 2018 at 13:28, Phillip Feldman <phillip.m.feldman at gmail.com>
> wrote:
>
>> When you say "discrete cosine matrix", I think that you mean "direction
>> cosine matrix" (see https://en.wikiversity.org/
>> wiki/PlanetPhysics/Direction_Cosine_Matrix).
>>
>> Phillip
>>
>> On Sat, May 19, 2018 at 12:23 PM, Aditya Bharti <adibhar97 at gmail.com>
>> wrote:
>>
>>> Hi all,
>>> So this concludes week 1 of GSoC 2018. I must say it was a great
>>> learning experience and I invite you all to check out my account of this
>>> week on the blog <https://blogs.python-gsoc.org/aditya-bharti/>. This
>>> email is more of a technical update.
>>>
>>>    - So, the main `Rotation` class will live under a new sub module
>>>    `scipy.spatial.transform`.
>>>    - Conversion between quaternions and discrete cosine matrices was
>>>    implemented.
>>>    - The rotation class now supports `from_quaternion`, `from_dcm`,
>>>    `as_quaternion` and `as_dcm`, with support for multiple rotations in one
>>>    call.
>>>
>>> The project currently lives in my own fork of scipy here
>>> <https://github.com/adbugger/scipy/tree/rotation/scipy/spatial/transform>.
>>> Stay tuned for more updates!
>>>
>>> Best,
>>> Aditya
>>>
>>> On Wed, 2 May 2018 at 21:03, Aditya Bharti <adibhar97 at gmail.com> wrote:
>>>
>>>> Hi Nikolay,
>>>>
>>>> I've used Wordpress only once before, so I don't know much about it.
>>>> From my limited experience, it is extremely customizable. You can customize
>>>> every thing from the look and feel to SEO characteristics. There are
>>>> apparently a lot of wordpress plugins for these kind of tasks. For this
>>>> particular blog, PSF had already setup an account for me with a site on it.
>>>> All I had to do was click on the 'New' button and open up the new post
>>>> page. There's a field for a header and body text, with options for adding
>>>> audio, video and hyperlinks.
>>>>
>>>> As regards to the post itself, sure I'll expand it with a brief
>>>> overview, motivation and an example. Note that the example will only show
>>>> sample usage, not any internals. I plan to borrow heavily from my proposal
>>>> for this purpose, I hope that's ok.
>>>>
>>>> Regards,
>>>> Aditya
>>>>
>>>> On 2 May 2018 at 19:54, Nikolay Mayorov <nikolay.mayorov at zoho.com>
>>>> wrote:
>>>>
>>>>> Hi, Aditya!
>>>>>
>>>>> Glad that you set up the blog and good job on setting up the
>>>>> documentation build as well.
>>>>>
>>>>> Curious, what is this blogging platform like? How do you create posts
>>>>> in it?
>>>>>
>>>>> As for your first post: while not strictly necessary I think it would
>>>>> be nice to see a more thorough introductory post with a brief overview,
>>>>> motivation and/or an example. Do you want to work on it?
>>>>>
>>>>> Best,
>>>>> Nikolay
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-Dev mailing list
>>>>> SciPy-Dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>
>>>>>
>>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180603/c814fd94/attachment-0001.html>

From adibhar97 at gmail.com  Mon Jun  4 16:50:19 2018
From: adibhar97 at gmail.com (Aditya Bharti)
Date: Tue, 5 Jun 2018 02:20:19 +0530
Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3
 dimensions
In-Reply-To: <CAB2ViTYKO285BtwFTjOR4brjQJE_3SJ0d4jSkWeaUp-DJc8ing@mail.gmail.com>
References: <163213d9b99.e31ddfbf140246.3867213885941953730@zoho.com>
 <CANRzPHVN47mDJqc4DuUq70X9sAiRqBRWOSHszgtL3QyM1O6ihA@mail.gmail.com>
 <CANRzPHV0BswFBevgZ2fsf2vKm_J_LQEET=qtDZ1WO6H7dMDNyQ@mail.gmail.com>
 <CAB2ViTbimDdS0jZ3Qp71rorkhfALBhHQtTztv1OD11dY7V1ADg@mail.gmail.com>
 <CANRzPHUynj4LSmpUeBMzY-my1EmGKUp944mdy_tyqOLGY5Fmrw@mail.gmail.com>
 <CAB2ViTYKO285BtwFTjOR4brjQJE_3SJ0d4jSkWeaUp-DJc8ing@mail.gmail.com>
Message-ID: <CANRzPHXqGtT6d0VkDDYqqz9NQbKXV36T1jk9vCCf_oEZhqb5JA@mail.gmail.com>

We do plan to create a tutorial with examples highlighting the major
functionality of the module and some common use cases, but its exact nature
and extent will depend upon how much time we have left after completing the
implementation (including docstrings and tests).

Best,
Aditya

On Mon, 4 Jun 2018 at 09:45, Phillip Feldman <phillip.m.feldman at gmail.com>
wrote:

> "Refer to docs for more details."  Will there be user documentation
> beyond the doc strings in the various methods?
>
> On Tue, May 29, 2018 at 2:21 PM, Aditya Bharti <adibhar97 at gmail.com>
> wrote:
>
>> Hi all,
>> Continuing the work so far, the following have been implemented this week:
>>
>>    - `from_rotvec`, `as_rotvec`: Representing Euler angles as rotation
>>    vectors, with appropriate Taylor series expansions for small angles
>>    - `from_euler`: Initialization from Euler angles, along with a string
>>    based axis sequence specification. Refer to docs for more details.
>>
>> As always, the project lives here
>> <https://github.com/adbugger/scipy/tree/rotation/scipy/spatial/transform>,
>> and my personal experiences can be found on the blog
>> <https://blogs.python-gsoc.org/aditya-bharti>.
>>
>> Thanks,
>> Aditya
>>
>> On Sun, 20 May 2018 at 13:28, Phillip Feldman <
>> phillip.m.feldman at gmail.com> wrote:
>>
>>> When you say "discrete cosine matrix", I think that you mean "direction
>>> cosine matrix" (see
>>> https://en.wikiversity.org/wiki/PlanetPhysics/Direction_Cosine_Matrix).
>>>
>>> Phillip
>>>
>>> On Sat, May 19, 2018 at 12:23 PM, Aditya Bharti <adibhar97 at gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>> So this concludes week 1 of GSoC 2018. I must say it was a great
>>>> learning experience and I invite you all to check out my account of this
>>>> week on the blog <https://blogs.python-gsoc.org/aditya-bharti/>. This
>>>> email is more of a technical update.
>>>>
>>>>    - So, the main `Rotation` class will live under a new sub module
>>>>    `scipy.spatial.transform`.
>>>>    - Conversion between quaternions and discrete cosine matrices was
>>>>    implemented.
>>>>    - The rotation class now supports `from_quaternion`, `from_dcm`,
>>>>    `as_quaternion` and `as_dcm`, with support for multiple rotations in one
>>>>    call.
>>>>
>>>> The project currently lives in my own fork of scipy here
>>>> <https://github.com/adbugger/scipy/tree/rotation/scipy/spatial/transform>.
>>>> Stay tuned for more updates!
>>>>
>>>> Best,
>>>> Aditya
>>>>
>>>> On Wed, 2 May 2018 at 21:03, Aditya Bharti <adibhar97 at gmail.com> wrote:
>>>>
>>>>> Hi Nikolay,
>>>>>
>>>>> I've used Wordpress only once before, so I don't know much about it.
>>>>> From my limited experience, it is extremely customizable. You can customize
>>>>> every thing from the look and feel to SEO characteristics. There are
>>>>> apparently a lot of wordpress plugins for these kind of tasks. For this
>>>>> particular blog, PSF had already setup an account for me with a site on it.
>>>>> All I had to do was click on the 'New' button and open up the new post
>>>>> page. There's a field for a header and body text, with options for adding
>>>>> audio, video and hyperlinks.
>>>>>
>>>>> As regards to the post itself, sure I'll expand it with a brief
>>>>> overview, motivation and an example. Note that the example will only show
>>>>> sample usage, not any internals. I plan to borrow heavily from my proposal
>>>>> for this purpose, I hope that's ok.
>>>>>
>>>>> Regards,
>>>>> Aditya
>>>>>
>>>>> On 2 May 2018 at 19:54, Nikolay Mayorov <nikolay.mayorov at zoho.com>
>>>>> wrote:
>>>>>
>>>>>> Hi, Aditya!
>>>>>>
>>>>>> Glad that you set up the blog and good job on setting up the
>>>>>> documentation build as well.
>>>>>>
>>>>>> Curious, what is this blogging platform like? How do you create posts
>>>>>> in it?
>>>>>>
>>>>>> As for your first post: while not strictly necessary I think it would
>>>>>> be nice to see a more thorough introductory post with a brief overview,
>>>>>> motivation and/or an example. Do you want to work on it?
>>>>>>
>>>>>> Best,
>>>>>> Nikolay
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> SciPy-Dev mailing list
>>>>>> SciPy-Dev at python.org
>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>>
>>>>>>
>>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180605/fc928cef/attachment.html>

From mikofski at berkeley.edu  Tue Jun  5 13:51:35 2018
From: mikofski at berkeley.edu (Mark Alexander Mikofski)
Date: Tue, 5 Jun 2018 10:51:35 -0700
Subject: [SciPy-Dev] Should a RuntimeWarning be raised for
 optimize.zeros.newton if the initial guess is zero?
Message-ID: <CAEqRcW02_U8VyX3UrAV__8y+EusHq3fu5tEq6y+32Pvp7rkV+w@mail.gmail.com>

Hi All,

I would like to solicit discussion about an issue that was raised in #8357
by one of the reviewers regarding the case when the initial guess is zero,
but the derivative is also zero - currently scipy.optmize.newton raises a
RuntimeWarning and there is also a test to check for this condition,
"scipy/optimize/tests/test_zeros.py::TestBasic::test_deriv_zero_warning"

I have created an issue #8904 for this case, but I don't know what the
SciPy community opinion is on this issue. Is the current condition the
desired outcome? Or should an exception be made for initial guess that are
already zero, even if the derivative would be zero yielding an infinite
newton step?

I'm also concerned that this issue #8904 is blocking #8357, which IMHO is
not related to this philosophical question.

Thanks for your consideration!

-- 
Mark Mikofski, PhD (2005)
*Fiat Lux*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180605/57d6fcaf/attachment.html>

From matthew.brett at gmail.com  Tue Jun  5 14:14:32 2018
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 5 Jun 2018 19:14:32 +0100
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
Message-ID: <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>

Hi,

On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
> On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
> <mikofski at berkeley.edu> wrote:
>>
>> Hi All,
>>
>> I am having issues using
>>
>>     python runtest.py
>>
>> from scipy master branch since commit
>> 14de485c053cc275682bc7de0316fcca72b341f8
>>
>> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8
>>
>> there are two tracebacks, I think this may have something to do with
>> changes to LAPACK and dropping Apple Accelerate support, but not sure, and
>> don't know how to fix it. Should I brew install openblas? (I'm going to try
>> that)
>
>
> Yes that is the recommendation (if you're not using Anaconda).
>
>
>>
>> Use anaconda and create a conda env? (maybe this next?)
>
>
> If you're already using Anaconda, and the numpy from the defaults channel,
> SciPy should automatically link against MKL.
>
>
>>
>> Currently I'm using python 3.6 from homebrew in a virtualenv with wheels
>> from PyPI, after I create the virtualenv using the homebrew python-3
>> interpreter, I do pip install requirements.txt (see attached).
>>
>> 1st traceback:
>>
>> ImportError while importing test module
>> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
>> Hint: make sure your test modules/packages have valid Python names.
>> Traceback:
>> scipy/cluster/__init__.py:27: in <module>
>>     from . import vq, hierarchy
>> scipy/cluster/vq.py:75: in <module>
>>     from scipy.spatial.distance import cdist
>> scipy/spatial/__init__.py:97: in <module>
>>     from ._spherical_voronoi import SphericalVoronoi
>> scipy/spatial/_spherical_voronoi.py:19: in <module>
>>     from scipy.spatial.distance import pdist
>> scipy/spatial/distance.py:123: in <module>
>>     from ..linalg import norm
>> scipy/linalg/__init__.py:207: in <module>
>>     from ._decomp_update import *
>> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>>     ???
>> E   ImportError:
>> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so,
>> 2): Symbol not found: _cbbcsd_
>> E     Referenced from:
>> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>> E     Expected in: flat namespace
>> E    in
>> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so

I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
macOS machines, using our usual wheel-building procedure (+OpenBLAS):

* Python 3.4: https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
* Python 3.5: https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605

It might also happen on Python 2.7, but that is segfaulting too often
to get a full traceback:

https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298603

Cheers,

Matthew

From ralf.gommers at gmail.com  Tue Jun  5 14:30:48 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 5 Jun 2018 11:30:48 -0700
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
 <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
Message-ID: <CABL7CQggEE3TLVABn9UZt2MZkndF+K6+A-Ajaph17mdPmShhrg@mail.gmail.com>

On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> Hi,
>
> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >
> >
> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
> > <mikofski at berkeley.edu> wrote:
> >>
> >> Hi All,
> >>
> >> I am having issues using
> >>
> >>     python runtest.py
> >>
> >> from scipy master branch since commit
> >> 14de485c053cc275682bc7de0316fcca72b341f8
> >>
> >> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fc
> ca72b341f8
> >>
> >> there are two tracebacks, I think this may have something to do with
> >> changes to LAPACK and dropping Apple Accelerate support, but not sure,
> and
> >> don't know how to fix it. Should I brew install openblas? (I'm going to
> try
> >> that)
> >
> >
> > Yes that is the recommendation (if you're not using Anaconda).
> >
> >
> >>
> >> Use anaconda and create a conda env? (maybe this next?)
> >
> >
> > If you're already using Anaconda, and the numpy from the defaults
> channel,
> > SciPy should automatically link against MKL.
> >
> >
> >>
> >> Currently I'm using python 3.6 from homebrew in a virtualenv with wheels
> >> from PyPI, after I create the virtualenv using the homebrew python-3
> >> interpreter, I do pip install requirements.txt (see attached).
> >>
> >> 1st traceback:
> >>
> >> ImportError while importing test module
> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/
> python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
> >> Hint: make sure your test modules/packages have valid Python names.
> >> Traceback:
> >> scipy/cluster/__init__.py:27: in <module>
> >>     from . import vq, hierarchy
> >> scipy/cluster/vq.py:75: in <module>
> >>     from scipy.spatial.distance import cdist
> >> scipy/spatial/__init__.py:97: in <module>
> >>     from ._spherical_voronoi import SphericalVoronoi
> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
> >>     from scipy.spatial.distance import pdist
> >> scipy/spatial/distance.py:123: in <module>
> >>     from ..linalg import norm
> >> scipy/linalg/__init__.py:207: in <module>
> >>     from ._decomp_update import *
> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
> >>     ???
> >> E   ImportError:
> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/
> lib/python3.6/site-packages/scipy/linalg/cython_lapack.
> cpython-36m-darwin.so,
> >> 2): Symbol not found: _cbbcsd_
> >> E     Referenced from:
> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/
> python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
> >> E     Expected in: flat namespace
> >> E    in
> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/
> python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>
> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
>
> * Python 3.4: https://travis-ci.org/matthew-brett/scipy-wheels/jobs/
> 388298604
> * Python 3.5: https://travis-ci.org/matthew-brett/scipy-wheels/jobs/
> 388298605


You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either need a
site.cfg or install it to one of the regular locations):

openblas_info:

  libraries  not found in
['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
'/usr/local/lib', '/usr/lib']

  NOT AVAILABLE



atlas_blas_threads_info:

Setting PTATLAS=ATLAS

  libraries ptf77blas,ptcblas,atlas not found in
['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
'/usr/local/lib', '/usr/lib']

  NOT AVAILABLE



atlas_blas_info:

  libraries f77blas,cblas,atlas not found in
['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
'/usr/local/lib', '/usr/lib']

  NOT AVAILABLE



  FOUND:

    libraries = ['blas']

    library_dirs = ['/usr/lib']

    define_macros = [('NO_ATLAS_INFO', 1)]

    language = f77


Ralf


>
>
> It might also happen on Python 2.7, but that is segfaulting too often
> to get a full traceback:
>
> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298603
>
> Cheers,
>
> Matthew
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180605/245a1a2a/attachment-0001.html>

From mikofski at berkeley.edu  Tue Jun  5 14:46:31 2018
From: mikofski at berkeley.edu (Mark Alexander Mikofski)
Date: Tue, 5 Jun 2018 11:46:31 -0700
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CABL7CQggEE3TLVABn9UZt2MZkndF+K6+A-Ajaph17mdPmShhrg@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
 <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
 <CABL7CQggEE3TLVABn9UZt2MZkndF+K6+A-Ajaph17mdPmShhrg@mail.gmail.com>
Message-ID: <CAEqRcW1sK8d8cM=MZprPgAMjR5bSK=U5+MXkGnOykCWrpd=HHQ@mail.gmail.com>

That's what I did. After brew installing openblas on my local machine, I
saved https://github.com/scipy/scipy/blob/master/site.cfg.example as
site.cfg, uncommented the [openblas] section, lines 129-133 as of 444a95b0,
and changed them to the specific path that homebrew told me were the
library and include directories (/usr/local/opt/openblas/lib and
/usr/local/opt/openblas/include
in my specific case) .

On Tue, Jun 5, 2018 at 11:30 AM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:

>
>
> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>
>> Hi,
>>
>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>> >
>> >
>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
>> > <mikofski at berkeley.edu> wrote:
>> >>
>> >> Hi All,
>> >>
>> >> I am having issues using
>> >>
>> >>     python runtest.py
>> >>
>> >> from scipy master branch since commit
>> >> 14de485c053cc275682bc7de0316fcca72b341f8
>> >>
>> >> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de
>> 0316fcca72b341f8
>> >>
>> >> there are two tracebacks, I think this may have something to do with
>> >> changes to LAPACK and dropping Apple Accelerate support, but not sure,
>> and
>> >> don't know how to fix it. Should I brew install openblas? (I'm going
>> to try
>> >> that)
>> >
>> >
>> > Yes that is the recommendation (if you're not using Anaconda).
>> >
>> >
>> >>
>> >> Use anaconda and create a conda env? (maybe this next?)
>> >
>> >
>> > If you're already using Anaconda, and the numpy from the defaults
>> channel,
>> > SciPy should automatically link against MKL.
>> >
>> >
>> >>
>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
>> wheels
>> >> from PyPI, after I create the virtualenv using the homebrew python-3
>> >> interpreter, I do pip install requirements.txt (see attached).
>> >>
>> >> 1st traceback:
>> >>
>> >> ImportError while importing test module
>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python
>> 3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
>> >> Hint: make sure your test modules/packages have valid Python names.
>> >> Traceback:
>> >> scipy/cluster/__init__.py:27: in <module>
>> >>     from . import vq, hierarchy
>> >> scipy/cluster/vq.py:75: in <module>
>> >>     from scipy.spatial.distance import cdist
>> >> scipy/spatial/__init__.py:97: in <module>
>> >>     from ._spherical_voronoi import SphericalVoronoi
>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
>> >>     from scipy.spatial.distance import pdist
>> >> scipy/spatial/distance.py:123: in <module>
>> >>     from ..linalg import norm
>> >> scipy/linalg/__init__.py:207: in <module>
>> >>     from ._decomp_update import *
>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>> >>     ???
>> >> E   ImportError:
>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/
>> python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so,
>> >> 2): Symbol not found: _cbbcsd_
>> >> E     Referenced from:
>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3
>> .6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>> >> E     Expected in: flat namespace
>> >> E    in
>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3
>> .6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>
>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
>> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
>>
>> * Python 3.4: https://travis-ci.org/matthew-
>> brett/scipy-wheels/jobs/388298604
>> * Python 3.5: https://travis-ci.org/matthew-
>> brett/scipy-wheels/jobs/388298605
>
>
> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either need a
> site.cfg or install it to one of the regular locations):
>
> openblas_info:
>
>   libraries  not found in ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib', '/usr/local/lib', '/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_blas_threads_info:
>
> Setting PTATLAS=ATLAS
>
>   libraries ptf77blas,ptcblas,atlas not found in ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib', '/usr/local/lib', '/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_blas_info:
>
>   libraries f77blas,cblas,atlas not found in ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib', '/usr/local/lib', '/usr/lib']
>
>   NOT AVAILABLE
>
>
>
>   FOUND:
>
>     libraries = ['blas']
>
>     library_dirs = ['/usr/lib']
>
>     define_macros = [('NO_ATLAS_INFO', 1)]
>
>     language = f77
>
>
> Ralf
>
>
>>
>>
>> It might also happen on Python 2.7, but that is segfaulting too often
>> to get a full traceback:
>>
>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298603
>>
>> Cheers,
>>
>> Matthew
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>


-- 
Mark Mikofski, PhD (2005)
*Fiat Lux*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180605/24e1cb03/attachment.html>

From pav at iki.fi  Tue Jun  5 15:05:12 2018
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 05 Jun 2018 21:05:12 +0200
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
 <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
Message-ID: <bd7e893dd8dfa31afbe8ec44ecd1eed693ac022b.camel@iki.fi>

ti, 2018-06-05 kello 19:14 +0100, Matthew Brett kirjoitti:
[clip]
> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
> 
> * Python 3.4: https://travis-ci.org/matthew-brett/scipy-wheels/jobs/3
> 88298604
> * Python 3.5: https://travis-ci.org/matthew-brett/scipy-wheels/jobs/3
> 88298605

The build logs show that it's picking up /usr/lib/libblas.* and
/usr/lib/liblapack.* and using those. Apparently, they are missing
symbols.

Are these some standard component on OSX? Part of Accelerate maybe?
If yes, they could be blacklisted in the top-level setup.py.

-- 
Pauli Virtanen

From matthew.brett at gmail.com  Tue Jun  5 15:09:12 2018
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 5 Jun 2018 20:09:12 +0100
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CABL7CQggEE3TLVABn9UZt2MZkndF+K6+A-Ajaph17mdPmShhrg@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
 <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
 <CABL7CQggEE3TLVABn9UZt2MZkndF+K6+A-Ajaph17mdPmShhrg@mail.gmail.com>
Message-ID: <CAH6Pt5p9t7YZmx+=-kjDKbaqc08hxaYq65xLYs6Nm2L+=FhVkw@mail.gmail.com>

Hi,

On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>> >
>> >
>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
>> > <mikofski at berkeley.edu> wrote:
>> >>
>> >> Hi All,
>> >>
>> >> I am having issues using
>> >>
>> >>     python runtest.py
>> >>
>> >> from scipy master branch since commit
>> >> 14de485c053cc275682bc7de0316fcca72b341f8
>> >>
>> >>
>> >> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8
>> >>
>> >> there are two tracebacks, I think this may have something to do with
>> >> changes to LAPACK and dropping Apple Accelerate support, but not sure,
>> >> and
>> >> don't know how to fix it. Should I brew install openblas? (I'm going to
>> >> try
>> >> that)
>> >
>> >
>> > Yes that is the recommendation (if you're not using Anaconda).
>> >
>> >
>> >>
>> >> Use anaconda and create a conda env? (maybe this next?)
>> >
>> >
>> > If you're already using Anaconda, and the numpy from the defaults
>> > channel,
>> > SciPy should automatically link against MKL.
>> >
>> >
>> >>
>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
>> >> wheels
>> >> from PyPI, after I create the virtualenv using the homebrew python-3
>> >> interpreter, I do pip install requirements.txt (see attached).
>> >>
>> >> 1st traceback:
>> >>
>> >> ImportError while importing test module
>> >>
>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
>> >> Hint: make sure your test modules/packages have valid Python names.
>> >> Traceback:
>> >> scipy/cluster/__init__.py:27: in <module>
>> >>     from . import vq, hierarchy
>> >> scipy/cluster/vq.py:75: in <module>
>> >>     from scipy.spatial.distance import cdist
>> >> scipy/spatial/__init__.py:97: in <module>
>> >>     from ._spherical_voronoi import SphericalVoronoi
>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
>> >>     from scipy.spatial.distance import pdist
>> >> scipy/spatial/distance.py:123: in <module>
>> >>     from ..linalg import norm
>> >> scipy/linalg/__init__.py:207: in <module>
>> >>     from ._decomp_update import *
>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>> >>     ???
>> >> E   ImportError:
>> >>
>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so,
>> >> 2): Symbol not found: _cbbcsd_
>> >> E     Referenced from:
>> >>
>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>> >> E     Expected in: flat namespace
>> >> E    in
>> >>
>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>
>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
>> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
>>
>> * Python 3.4:
>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
>> * Python 3.5:
>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
>
>
> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either need a
> site.cfg or install it to one of the regular locations):
>
> openblas_info:
>
>   libraries  not found in
> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> '/usr/local/lib', '/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_blas_threads_info:
>
> Setting PTATLAS=ATLAS
>
>   libraries ptf77blas,ptcblas,atlas not found in
> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> '/usr/local/lib', '/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_blas_info:
>
>   libraries f77blas,cblas,atlas not found in
> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> '/usr/local/lib', '/usr/lib']
>
>   NOT AVAILABLE
>
>
>
>   FOUND:
>
>     libraries = ['blas']
>
>     library_dirs = ['/usr/lib']
>
>     define_macros = [('NO_ATLAS_INFO', 1)]
>
>     language = f77

Hmm - you're right - that's very odd.  OpenBLAS is being unpacked into
/usr/local/lib, where it should be detected.  On Python 3.6, using the
same machinery, it is detected - see:

https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606

Cheers,

Matthew

From ralf.gommers at gmail.com  Tue Jun  5 15:13:06 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 5 Jun 2018 12:13:06 -0700
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <bd7e893dd8dfa31afbe8ec44ecd1eed693ac022b.camel@iki.fi>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
 <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
 <bd7e893dd8dfa31afbe8ec44ecd1eed693ac022b.camel@iki.fi>
Message-ID: <CABL7CQgU7W3atF-+vtMuyDvyRx1gXUX5eycniDhrKXx0ZWSkUA@mail.gmail.com>

On Tue, Jun 5, 2018 at 12:05 PM, Pauli Virtanen <pav at iki.fi> wrote:

> ti, 2018-06-05 kello 19:14 +0100, Matthew Brett kirjoitti:
> [clip]
> > I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
> > macOS machines, using our usual wheel-building procedure (+OpenBLAS):
> >
> > * Python 3.4: https://travis-ci.org/matthew-brett/scipy-wheels/jobs/3
> > 88298604
> > * Python 3.5: https://travis-ci.org/matthew-brett/scipy-wheels/jobs/3
> > 88298605
>
> The build logs show that it's picking up /usr/lib/libblas.* and
> /usr/lib/liblapack.* and using those. Apparently, they are missing
> symbols.
>
> Are these some standard component on OSX? Part of Accelerate maybe?

If yes, they could be blacklisted in the top-level setup.py.
>

Good idea to blacklist. Indeed part of Accelerate:

rgommers at machine:/usr/lib$ otool -L libblas.dylib
libblas.dylib:

/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
(compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1252.50.4)
rgommers at machine:/usr/lib$ otool -L liblapack.dylib
liblapack.dylib:

/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
(compatibility version 1.0.0, current version 1.0.0)

/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
(compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1252.50.4)

Ralf



>
> --
> Pauli Virtanen
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180605/1125c8c6/attachment.html>

From matthew.brett at gmail.com  Tue Jun  5 16:17:34 2018
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 5 Jun 2018 21:17:34 +0100
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CAH6Pt5p9t7YZmx+=-kjDKbaqc08hxaYq65xLYs6Nm2L+=FhVkw@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
 <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
 <CABL7CQggEE3TLVABn9UZt2MZkndF+K6+A-Ajaph17mdPmShhrg@mail.gmail.com>
 <CAH6Pt5p9t7YZmx+=-kjDKbaqc08hxaYq65xLYs6Nm2L+=FhVkw@mail.gmail.com>
Message-ID: <CAH6Pt5pwxFXoYYDmVJa7ZQDu5nuavFdtLwjuTBfhGonXnqBRYQ@mail.gmail.com>

On Tue, Jun 5, 2018 at 8:09 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>
>>
>> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett <matthew.brett at gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>> >
>>> >
>>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
>>> > <mikofski at berkeley.edu> wrote:
>>> >>
>>> >> Hi All,
>>> >>
>>> >> I am having issues using
>>> >>
>>> >>     python runtest.py
>>> >>
>>> >> from scipy master branch since commit
>>> >> 14de485c053cc275682bc7de0316fcca72b341f8
>>> >>
>>> >>
>>> >> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8
>>> >>
>>> >> there are two tracebacks, I think this may have something to do with
>>> >> changes to LAPACK and dropping Apple Accelerate support, but not sure,
>>> >> and
>>> >> don't know how to fix it. Should I brew install openblas? (I'm going to
>>> >> try
>>> >> that)
>>> >
>>> >
>>> > Yes that is the recommendation (if you're not using Anaconda).
>>> >
>>> >
>>> >>
>>> >> Use anaconda and create a conda env? (maybe this next?)
>>> >
>>> >
>>> > If you're already using Anaconda, and the numpy from the defaults
>>> > channel,
>>> > SciPy should automatically link against MKL.
>>> >
>>> >
>>> >>
>>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
>>> >> wheels
>>> >> from PyPI, after I create the virtualenv using the homebrew python-3
>>> >> interpreter, I do pip install requirements.txt (see attached).
>>> >>
>>> >> 1st traceback:
>>> >>
>>> >> ImportError while importing test module
>>> >>
>>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
>>> >> Hint: make sure your test modules/packages have valid Python names.
>>> >> Traceback:
>>> >> scipy/cluster/__init__.py:27: in <module>
>>> >>     from . import vq, hierarchy
>>> >> scipy/cluster/vq.py:75: in <module>
>>> >>     from scipy.spatial.distance import cdist
>>> >> scipy/spatial/__init__.py:97: in <module>
>>> >>     from ._spherical_voronoi import SphericalVoronoi
>>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
>>> >>     from scipy.spatial.distance import pdist
>>> >> scipy/spatial/distance.py:123: in <module>
>>> >>     from ..linalg import norm
>>> >> scipy/linalg/__init__.py:207: in <module>
>>> >>     from ._decomp_update import *
>>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>>> >>     ???
>>> >> E   ImportError:
>>> >>
>>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so,
>>> >> 2): Symbol not found: _cbbcsd_
>>> >> E     Referenced from:
>>> >>
>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>> >> E     Expected in: flat namespace
>>> >> E    in
>>> >>
>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>>
>>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
>>> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
>>>
>>> * Python 3.4:
>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
>>> * Python 3.5:
>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
>>
>>
>> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either need a
>> site.cfg or install it to one of the regular locations):
>>
>> openblas_info:
>>
>>   libraries  not found in
>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> '/usr/local/lib', '/usr/lib']
>>
>>   NOT AVAILABLE
>>
>>
>>
>> atlas_blas_threads_info:
>>
>> Setting PTATLAS=ATLAS
>>
>>   libraries ptf77blas,ptcblas,atlas not found in
>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> '/usr/local/lib', '/usr/lib']
>>
>>   NOT AVAILABLE
>>
>>
>>
>> atlas_blas_info:
>>
>>   libraries f77blas,cblas,atlas not found in
>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> '/usr/local/lib', '/usr/lib']
>>
>>   NOT AVAILABLE
>>
>>
>>
>>   FOUND:
>>
>>     libraries = ['blas']
>>
>>     library_dirs = ['/usr/lib']
>>
>>     define_macros = [('NO_ATLAS_INFO', 1)]
>>
>>     language = f77
>
> Hmm - you're right - that's very odd.  OpenBLAS is being unpacked into
> /usr/local/lib, where it should be detected.  On Python 3.6, using the
> same machinery, it is detected - see:
>
> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606

It looks like it's numpy distutils.  numpy 1.8.2 doesn't appear to find
OpenBLAS in /usr/local/lib, even though it claims it looked.  Numpy
1.9.3 does find it, which is why the build is OK for Python 3.6.
Writing an explicit site.cfg:

    cat << EOF > $HOME/site.cfg
[openblas]
libraries = openblas
library_dirs = /usr/local/lib
include_dirs = /usr/local/include
runtime_library_dirs = /usr/local/lib
EOF

fixes that locally.  Testing on Travis-CI now:

https://travis-ci.org/matthew-brett/scipy-wheels/builds/388461944

Cheers,

Matthew

From matthew.brett at gmail.com  Tue Jun  5 19:27:00 2018
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 6 Jun 2018 00:27:00 +0100
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CAH6Pt5pwxFXoYYDmVJa7ZQDu5nuavFdtLwjuTBfhGonXnqBRYQ@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
 <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
 <CABL7CQggEE3TLVABn9UZt2MZkndF+K6+A-Ajaph17mdPmShhrg@mail.gmail.com>
 <CAH6Pt5p9t7YZmx+=-kjDKbaqc08hxaYq65xLYs6Nm2L+=FhVkw@mail.gmail.com>
 <CAH6Pt5pwxFXoYYDmVJa7ZQDu5nuavFdtLwjuTBfhGonXnqBRYQ@mail.gmail.com>
Message-ID: <CAH6Pt5qqYp_a79Z=Op5VtMMQeipy4ZTYHw3v4Gs3_YcSEqf==A@mail.gmail.com>

On Tue, Jun 5, 2018 at 9:17 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> On Tue, Jun 5, 2018 at 8:09 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Hi,
>>
>> On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>
>>>
>>> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett <matthew.brett at gmail.com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com>
>>>> wrote:
>>>> >
>>>> >
>>>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
>>>> > <mikofski at berkeley.edu> wrote:
>>>> >>
>>>> >> Hi All,
>>>> >>
>>>> >> I am having issues using
>>>> >>
>>>> >>     python runtest.py
>>>> >>
>>>> >> from scipy master branch since commit
>>>> >> 14de485c053cc275682bc7de0316fcca72b341f8
>>>> >>
>>>> >>
>>>> >> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8
>>>> >>
>>>> >> there are two tracebacks, I think this may have something to do with
>>>> >> changes to LAPACK and dropping Apple Accelerate support, but not sure,
>>>> >> and
>>>> >> don't know how to fix it. Should I brew install openblas? (I'm going to
>>>> >> try
>>>> >> that)
>>>> >
>>>> >
>>>> > Yes that is the recommendation (if you're not using Anaconda).
>>>> >
>>>> >
>>>> >>
>>>> >> Use anaconda and create a conda env? (maybe this next?)
>>>> >
>>>> >
>>>> > If you're already using Anaconda, and the numpy from the defaults
>>>> > channel,
>>>> > SciPy should automatically link against MKL.
>>>> >
>>>> >
>>>> >>
>>>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
>>>> >> wheels
>>>> >> from PyPI, after I create the virtualenv using the homebrew python-3
>>>> >> interpreter, I do pip install requirements.txt (see attached).
>>>> >>
>>>> >> 1st traceback:
>>>> >>
>>>> >> ImportError while importing test module
>>>> >>
>>>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
>>>> >> Hint: make sure your test modules/packages have valid Python names.
>>>> >> Traceback:
>>>> >> scipy/cluster/__init__.py:27: in <module>
>>>> >>     from . import vq, hierarchy
>>>> >> scipy/cluster/vq.py:75: in <module>
>>>> >>     from scipy.spatial.distance import cdist
>>>> >> scipy/spatial/__init__.py:97: in <module>
>>>> >>     from ._spherical_voronoi import SphericalVoronoi
>>>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
>>>> >>     from scipy.spatial.distance import pdist
>>>> >> scipy/spatial/distance.py:123: in <module>
>>>> >>     from ..linalg import norm
>>>> >> scipy/linalg/__init__.py:207: in <module>
>>>> >>     from ._decomp_update import *
>>>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>>>> >>     ???
>>>> >> E   ImportError:
>>>> >>
>>>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so,
>>>> >> 2): Symbol not found: _cbbcsd_
>>>> >> E     Referenced from:
>>>> >>
>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>>> >> E     Expected in: flat namespace
>>>> >> E    in
>>>> >>
>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>>>
>>>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
>>>> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
>>>>
>>>> * Python 3.4:
>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
>>>> * Python 3.5:
>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
>>>
>>>
>>> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either need a
>>> site.cfg or install it to one of the regular locations):
>>>
>>> openblas_info:
>>>
>>>   libraries  not found in
>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>> '/usr/local/lib', '/usr/lib']
>>>
>>>   NOT AVAILABLE
>>>
>>>
>>>
>>> atlas_blas_threads_info:
>>>
>>> Setting PTATLAS=ATLAS
>>>
>>>   libraries ptf77blas,ptcblas,atlas not found in
>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>> '/usr/local/lib', '/usr/lib']
>>>
>>>   NOT AVAILABLE
>>>
>>>
>>>
>>> atlas_blas_info:
>>>
>>>   libraries f77blas,cblas,atlas not found in
>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>> '/usr/local/lib', '/usr/lib']
>>>
>>>   NOT AVAILABLE
>>>
>>>
>>>
>>>   FOUND:
>>>
>>>     libraries = ['blas']
>>>
>>>     library_dirs = ['/usr/lib']
>>>
>>>     define_macros = [('NO_ATLAS_INFO', 1)]
>>>
>>>     language = f77
>>
>> Hmm - you're right - that's very odd.  OpenBLAS is being unpacked into
>> /usr/local/lib, where it should be detected.  On Python 3.6, using the
>> same machinery, it is detected - see:
>>
>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606
>
> It looks like it's numpy distutils.  numpy 1.8.2 doesn't appear to find
> OpenBLAS in /usr/local/lib, even though it claims it looked.  Numpy
> 1.9.3 does find it, which is why the build is OK for Python 3.6.
> Writing an explicit site.cfg:
>
>     cat << EOF > $HOME/site.cfg
> [openblas]
> libraries = openblas
> library_dirs = /usr/local/lib
> include_dirs = /usr/local/include
> runtime_library_dirs = /usr/local/lib
> EOF
>
> fixes that locally.  Testing on Travis-CI now:
>
> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388461944

Oops, no, Python 3.6 was using numpy 1.11.3, but this is now broken
with the site.cfg, giving

ld: unknown option: -rpath=/usr/local/lib

https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388461965#L8489

I can replicate this locally.  Deleting site.cfg fixes the error.

The other builds continue to be broken with the same kind of errors;
they pick up OpenBLAS, but also the /usr/lib stuff.  So, you're right,
I think it needs blacklisting.  I hope that would make it possible to
build with numpy 1.9.3 at least, making sure to avoid having a
site.cfg.

Cheers,

Matthew

From matthew.brett at gmail.com  Wed Jun  6 06:55:09 2018
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 6 Jun 2018 11:55:09 +0100
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CAH6Pt5qqYp_a79Z=Op5VtMMQeipy4ZTYHw3v4Gs3_YcSEqf==A@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
 <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
 <CABL7CQggEE3TLVABn9UZt2MZkndF+K6+A-Ajaph17mdPmShhrg@mail.gmail.com>
 <CAH6Pt5p9t7YZmx+=-kjDKbaqc08hxaYq65xLYs6Nm2L+=FhVkw@mail.gmail.com>
 <CAH6Pt5pwxFXoYYDmVJa7ZQDu5nuavFdtLwjuTBfhGonXnqBRYQ@mail.gmail.com>
 <CAH6Pt5qqYp_a79Z=Op5VtMMQeipy4ZTYHw3v4Gs3_YcSEqf==A@mail.gmail.com>
Message-ID: <CAH6Pt5oKFTk2mZTZF8cJG+xp1zD3muLCg=nfk27bfpXqWvd7nA@mail.gmail.com>

On Wed, Jun 6, 2018 at 12:27 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> On Tue, Jun 5, 2018 at 9:17 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> On Tue, Jun 5, 2018 at 8:09 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>>> Hi,
>>>
>>> On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>>
>>>>
>>>> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett <matthew.brett at gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com>
>>>>> wrote:
>>>>> >
>>>>> >
>>>>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
>>>>> > <mikofski at berkeley.edu> wrote:
>>>>> >>
>>>>> >> Hi All,
>>>>> >>
>>>>> >> I am having issues using
>>>>> >>
>>>>> >>     python runtest.py
>>>>> >>
>>>>> >> from scipy master branch since commit
>>>>> >> 14de485c053cc275682bc7de0316fcca72b341f8
>>>>> >>
>>>>> >>
>>>>> >> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8
>>>>> >>
>>>>> >> there are two tracebacks, I think this may have something to do with
>>>>> >> changes to LAPACK and dropping Apple Accelerate support, but not sure,
>>>>> >> and
>>>>> >> don't know how to fix it. Should I brew install openblas? (I'm going to
>>>>> >> try
>>>>> >> that)
>>>>> >
>>>>> >
>>>>> > Yes that is the recommendation (if you're not using Anaconda).
>>>>> >
>>>>> >
>>>>> >>
>>>>> >> Use anaconda and create a conda env? (maybe this next?)
>>>>> >
>>>>> >
>>>>> > If you're already using Anaconda, and the numpy from the defaults
>>>>> > channel,
>>>>> > SciPy should automatically link against MKL.
>>>>> >
>>>>> >
>>>>> >>
>>>>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
>>>>> >> wheels
>>>>> >> from PyPI, after I create the virtualenv using the homebrew python-3
>>>>> >> interpreter, I do pip install requirements.txt (see attached).
>>>>> >>
>>>>> >> 1st traceback:
>>>>> >>
>>>>> >> ImportError while importing test module
>>>>> >>
>>>>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
>>>>> >> Hint: make sure your test modules/packages have valid Python names.
>>>>> >> Traceback:
>>>>> >> scipy/cluster/__init__.py:27: in <module>
>>>>> >>     from . import vq, hierarchy
>>>>> >> scipy/cluster/vq.py:75: in <module>
>>>>> >>     from scipy.spatial.distance import cdist
>>>>> >> scipy/spatial/__init__.py:97: in <module>
>>>>> >>     from ._spherical_voronoi import SphericalVoronoi
>>>>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
>>>>> >>     from scipy.spatial.distance import pdist
>>>>> >> scipy/spatial/distance.py:123: in <module>
>>>>> >>     from ..linalg import norm
>>>>> >> scipy/linalg/__init__.py:207: in <module>
>>>>> >>     from ._decomp_update import *
>>>>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>>>>> >>     ???
>>>>> >> E   ImportError:
>>>>> >>
>>>>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so,
>>>>> >> 2): Symbol not found: _cbbcsd_
>>>>> >> E     Referenced from:
>>>>> >>
>>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>>>> >> E     Expected in: flat namespace
>>>>> >> E    in
>>>>> >>
>>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>>>>
>>>>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
>>>>> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
>>>>>
>>>>> * Python 3.4:
>>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
>>>>> * Python 3.5:
>>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
>>>>
>>>>
>>>> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either need a
>>>> site.cfg or install it to one of the regular locations):
>>>>
>>>> openblas_info:
>>>>
>>>>   libraries  not found in
>>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>>> '/usr/local/lib', '/usr/lib']
>>>>
>>>>   NOT AVAILABLE
>>>>
>>>>
>>>>
>>>> atlas_blas_threads_info:
>>>>
>>>> Setting PTATLAS=ATLAS
>>>>
>>>>   libraries ptf77blas,ptcblas,atlas not found in
>>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>>> '/usr/local/lib', '/usr/lib']
>>>>
>>>>   NOT AVAILABLE
>>>>
>>>>
>>>>
>>>> atlas_blas_info:
>>>>
>>>>   libraries f77blas,cblas,atlas not found in
>>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>>> '/usr/local/lib', '/usr/lib']
>>>>
>>>>   NOT AVAILABLE
>>>>
>>>>
>>>>
>>>>   FOUND:
>>>>
>>>>     libraries = ['blas']
>>>>
>>>>     library_dirs = ['/usr/lib']
>>>>
>>>>     define_macros = [('NO_ATLAS_INFO', 1)]
>>>>
>>>>     language = f77
>>>
>>> Hmm - you're right - that's very odd.  OpenBLAS is being unpacked into
>>> /usr/local/lib, where it should be detected.  On Python 3.6, using the
>>> same machinery, it is detected - see:
>>>
>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606
>>
>> It looks like it's numpy distutils.  numpy 1.8.2 doesn't appear to find
>> OpenBLAS in /usr/local/lib, even though it claims it looked.  Numpy
>> 1.9.3 does find it, which is why the build is OK for Python 3.6.
>> Writing an explicit site.cfg:
>>
>>     cat << EOF > $HOME/site.cfg
>> [openblas]
>> libraries = openblas
>> library_dirs = /usr/local/lib
>> include_dirs = /usr/local/include
>> runtime_library_dirs = /usr/local/lib
>> EOF
>>
>> fixes that locally.  Testing on Travis-CI now:
>>
>> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388461944
>
> Oops, no, Python 3.6 was using numpy 1.11.3, but this is now broken
> with the site.cfg, giving
>
> ld: unknown option: -rpath=/usr/local/lib
>
> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388461965#L8489
>
> I can replicate this locally.  Deleting site.cfg fixes the error.
>
> The other builds continue to be broken with the same kind of errors;
> they pick up OpenBLAS, but also the /usr/lib stuff.  So, you're right,
> I think it needs blacklisting.  I hope that would make it possible to
> build with numpy 1.9.3 at least, making sure to avoid having a
> site.cfg.

All macOS builds work correctly if I force numpy 1.11.3.

https://travis-ci.org/matthew-brett/scipy-wheels/builds/388525456

Any ideas why numpy 1.11.3 is not picking up the /usr/lib stuff?

Building wheels against 1.11.3 would be a fairly significant leap in
numpy requirement, so it would be better if we could build with numpy
1.9.3, at least.

Cheers,

Matthew

From ralf.gommers at gmail.com  Wed Jun  6 12:53:01 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Wed, 6 Jun 2018 09:53:01 -0700
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CAH6Pt5oKFTk2mZTZF8cJG+xp1zD3muLCg=nfk27bfpXqWvd7nA@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
 <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
 <CABL7CQggEE3TLVABn9UZt2MZkndF+K6+A-Ajaph17mdPmShhrg@mail.gmail.com>
 <CAH6Pt5p9t7YZmx+=-kjDKbaqc08hxaYq65xLYs6Nm2L+=FhVkw@mail.gmail.com>
 <CAH6Pt5pwxFXoYYDmVJa7ZQDu5nuavFdtLwjuTBfhGonXnqBRYQ@mail.gmail.com>
 <CAH6Pt5qqYp_a79Z=Op5VtMMQeipy4ZTYHw3v4Gs3_YcSEqf==A@mail.gmail.com>
 <CAH6Pt5oKFTk2mZTZF8cJG+xp1zD3muLCg=nfk27bfpXqWvd7nA@mail.gmail.com>
Message-ID: <CABL7CQiAK8aOdcJrHWnG87UegDM9oMKQUtwzfYbDAiyHszOSmQ@mail.gmail.com>

On Wed, Jun 6, 2018 at 3:55 AM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> On Wed, Jun 6, 2018 at 12:27 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> > On Tue, Jun 5, 2018 at 9:17 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> >> On Tue, Jun 5, 2018 at 8:09 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> >>> Hi,
> >>>
> >>> On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >>>>
> >>>>
> >>>> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett <
> matthew.brett at gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com
> >
> >>>>> wrote:
> >>>>> >
> >>>>> >
> >>>>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
> >>>>> > <mikofski at berkeley.edu> wrote:
> >>>>> >>
> >>>>> >> Hi All,
> >>>>> >>
> >>>>> >> I am having issues using
> >>>>> >>
> >>>>> >>     python runtest.py
> >>>>> >>
> >>>>> >> from scipy master branch since commit
> >>>>> >> 14de485c053cc275682bc7de0316fcca72b341f8
> >>>>> >>
> >>>>> >>
> >>>>> >> https://github.com/scipy/scipy/tree/
> 14de485c053cc275682bc7de0316fcca72b341f8
> >>>>> >>
> >>>>> >> there are two tracebacks, I think this may have something to do
> with
> >>>>> >> changes to LAPACK and dropping Apple Accelerate support, but not
> sure,
> >>>>> >> and
> >>>>> >> don't know how to fix it. Should I brew install openblas? (I'm
> going to
> >>>>> >> try
> >>>>> >> that)
> >>>>> >
> >>>>> >
> >>>>> > Yes that is the recommendation (if you're not using Anaconda).
> >>>>> >
> >>>>> >
> >>>>> >>
> >>>>> >> Use anaconda and create a conda env? (maybe this next?)
> >>>>> >
> >>>>> >
> >>>>> > If you're already using Anaconda, and the numpy from the defaults
> >>>>> > channel,
> >>>>> > SciPy should automatically link against MKL.
> >>>>> >
> >>>>> >
> >>>>> >>
> >>>>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
> >>>>> >> wheels
> >>>>> >> from PyPI, after I create the virtualenv using the homebrew
> python-3
> >>>>> >> interpreter, I do pip install requirements.txt (see attached).
> >>>>> >>
> >>>>> >> 1st traceback:
> >>>>> >>
> >>>>> >> ImportError while importing test module
> >>>>> >>
> >>>>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/
> python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
> >>>>> >> Hint: make sure your test modules/packages have valid Python
> names.
> >>>>> >> Traceback:
> >>>>> >> scipy/cluster/__init__.py:27: in <module>
> >>>>> >>     from . import vq, hierarchy
> >>>>> >> scipy/cluster/vq.py:75: in <module>
> >>>>> >>     from scipy.spatial.distance import cdist
> >>>>> >> scipy/spatial/__init__.py:97: in <module>
> >>>>> >>     from ._spherical_voronoi import SphericalVoronoi
> >>>>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
> >>>>> >>     from scipy.spatial.distance import pdist
> >>>>> >> scipy/spatial/distance.py:123: in <module>
> >>>>> >>     from ..linalg import norm
> >>>>> >> scipy/linalg/__init__.py:207: in <module>
> >>>>> >>     from ._decomp_update import *
> >>>>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
> >>>>> >>     ???
> >>>>> >> E   ImportError:
> >>>>> >>
> >>>>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/
> lib/python3.6/site-packages/scipy/linalg/cython_lapack.
> cpython-36m-darwin.so,
> >>>>> >> 2): Symbol not found: _cbbcsd_
> >>>>> >> E     Referenced from:
> >>>>> >>
> >>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/
> python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
> >>>>> >> E     Expected in: flat namespace
> >>>>> >> E    in
> >>>>> >>
> >>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/
> python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
> >>>>>
> >>>>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
> >>>>> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
> >>>>>
> >>>>> * Python 3.4:
> >>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
> >>>>> * Python 3.5:
> >>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
> >>>>
> >>>>
> >>>> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either
> need a
> >>>> site.cfg or install it to one of the regular locations):
> >>>>
> >>>> openblas_info:
> >>>>
> >>>>   libraries  not found in
> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> >>>> '/usr/local/lib', '/usr/lib']
> >>>>
> >>>>   NOT AVAILABLE
> >>>>
> >>>>
> >>>>
> >>>> atlas_blas_threads_info:
> >>>>
> >>>> Setting PTATLAS=ATLAS
> >>>>
> >>>>   libraries ptf77blas,ptcblas,atlas not found in
> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> >>>> '/usr/local/lib', '/usr/lib']
> >>>>
> >>>>   NOT AVAILABLE
> >>>>
> >>>>
> >>>>
> >>>> atlas_blas_info:
> >>>>
> >>>>   libraries f77blas,cblas,atlas not found in
> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> >>>> '/usr/local/lib', '/usr/lib']
> >>>>
> >>>>   NOT AVAILABLE
> >>>>
> >>>>
> >>>>
> >>>>   FOUND:
> >>>>
> >>>>     libraries = ['blas']
> >>>>
> >>>>     library_dirs = ['/usr/lib']
> >>>>
> >>>>     define_macros = [('NO_ATLAS_INFO', 1)]
> >>>>
> >>>>     language = f77
> >>>
> >>> Hmm - you're right - that's very odd.  OpenBLAS is being unpacked into
> >>> /usr/local/lib, where it should be detected.  On Python 3.6, using the
> >>> same machinery, it is detected - see:
> >>>
> >>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606
> >>
> >> It looks like it's numpy distutils.  numpy 1.8.2 doesn't appear to find
> >> OpenBLAS in /usr/local/lib, even though it claims it looked.  Numpy
> >> 1.9.3 does find it, which is why the build is OK for Python 3.6.
> >> Writing an explicit site.cfg:
> >>
> >>     cat << EOF > $HOME/site.cfg
> >> [openblas]
> >> libraries = openblas
> >> library_dirs = /usr/local/lib
> >> include_dirs = /usr/local/include
> >> runtime_library_dirs = /usr/local/lib
> >> EOF
> >>
> >> fixes that locally.  Testing on Travis-CI now:
> >>
> >> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388461944
> >
> > Oops, no, Python 3.6 was using numpy 1.11.3, but this is now broken
> > with the site.cfg, giving
> >
> > ld: unknown option: -rpath=/usr/local/lib
> >
> > https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388461965#L8489
> >
> > I can replicate this locally.  Deleting site.cfg fixes the error.
> >
> > The other builds continue to be broken with the same kind of errors;
> > they pick up OpenBLAS, but also the /usr/lib stuff.  So, you're right,
> > I think it needs blacklisting.  I hope that would make it possible to
> > build with numpy 1.9.3 at least, making sure to avoid having a
> > site.cfg.
>
> All macOS builds work correctly if I force numpy 1.11.3.
>
> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388525456
>
> Any ideas why numpy 1.11.3 is not picking up the /usr/lib stuff?
>

There were a lot of changes to system_info related to rpath and
default_lib_dirs handling, plus some reordering of the check for MKL,
addition of BLIS, etc. Could be any of those, bisecting is probably easiest
here.


>
> Building wheels against 1.11.3 would be a fairly significant leap in
> numpy requirement, so it would be better if we could build with numpy
> 1.9.3, at least.
>

A jump to 1.11.3 would still support 4 numpy versions (1.11-1.14), and 1.15
will arrive before scipy 1.2.0. So not a big problem to make 1.11.3 the
minimum required version.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180606/e13450c4/attachment.html>

From charlesr.harris at gmail.com  Wed Jun  6 14:06:48 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 6 Jun 2018 12:06:48 -0600
Subject: [SciPy-Dev] NumPy 1.14.4 released.
Message-ID: <CAB6mnx+yQD+HMWJFMjxio0ycBKHDar7Q0pRkxQPLaUWTQT+yFw@mail.gmail.com>

Hi All,

On behalf of the NumPy team, I am pleased to announce the release of NumPy
1.14.4.
This is a bugfix release for bugs reported following the 1.14.3 release. The
most significant fixes are:

* fixes for compiler instruction reordering that resulted in NaN's not being
  properly propagated in `np.max` and `np.min`,

* fixes for bus faults on SPARC and older ARM due to incorrect alignment
  checks.

There are also improvements to printing of long doubles on PPC platforms.
All
is not yet perfect on that platform, the whitespace padding is still
incorrect
and is to be fixed in numpy 1.15, consequently NumPy still fails some
printing-related (and other) unit tests on ppc systems. However, the printed
values are now correct.

Note that NumPy will error on import if it detects incorrect float32 `dot`
results. This problem has been seen on the Mac when working in the Anaconda
enviroment and is due to a subtle interaction between MKL and PyQt5.  It is
not
strictly a NumPy problem, but it is best that users be aware of it.  See the
gh-8577 NumPy issue for more information.

The Python versions supported in this release are 2.7 and 3.4 - 3.6. Wheels
for all supported versions  are available from PIP
<https://pypi.org/project/numpy/> and source releases are available
on github <https://github.com/numpy/numpy/releases/tag/v1.14.4>. The source
releases were cythonized with Cython 0.28.2 and should
be compatible with the upcoming Python 3.7.

Contributors
============

A total of 7 people contributed to this release.  People with a "+" by their
names contributed a patch for the first time.

* Allan Haldane
* Charles Harris
* Marten van Kerkwijk
* Matti Picus
* Pauli Virtanen
* Ryan Soklaski +
* Sebastian Berg

Pull requests merged
====================

A total of 11 pull requests were merged for this release.

* #11104: BUG: str of DOUBLE_DOUBLE format wrong on ppc64
* #11170: TST: linalg: add regression test for gh-8577
* #11174: MAINT: add sanity-checks to be run at import time
* #11181: BUG: void dtype setup checked offset not actual pointer for
alignment
* #11194: BUG: Python2 doubles don't print correctly in interactive shell.
* #11198: BUG: optimizing compilers can reorder call to npy_get_floatstatus
* #11199: BUG: reduce using SSE only warns if inside SSE loop
* #11203: BUG: Bytes delimiter/comments in genfromtxt should be decoded


Cheers,

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180606/c33f1807/attachment-0001.html>

From matthew.brett at gmail.com  Wed Jun  6 17:28:41 2018
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 6 Jun 2018 22:28:41 +0100
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CABL7CQiAK8aOdcJrHWnG87UegDM9oMKQUtwzfYbDAiyHszOSmQ@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
 <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
 <CABL7CQggEE3TLVABn9UZt2MZkndF+K6+A-Ajaph17mdPmShhrg@mail.gmail.com>
 <CAH6Pt5p9t7YZmx+=-kjDKbaqc08hxaYq65xLYs6Nm2L+=FhVkw@mail.gmail.com>
 <CAH6Pt5pwxFXoYYDmVJa7ZQDu5nuavFdtLwjuTBfhGonXnqBRYQ@mail.gmail.com>
 <CAH6Pt5qqYp_a79Z=Op5VtMMQeipy4ZTYHw3v4Gs3_YcSEqf==A@mail.gmail.com>
 <CAH6Pt5oKFTk2mZTZF8cJG+xp1zD3muLCg=nfk27bfpXqWvd7nA@mail.gmail.com>
 <CABL7CQiAK8aOdcJrHWnG87UegDM9oMKQUtwzfYbDAiyHszOSmQ@mail.gmail.com>
Message-ID: <CAH6Pt5p=fHboETch-APB4LbLki1m4bX6HhMhF90Y3B3MGXqM6Q@mail.gmail.com>

On Wed, Jun 6, 2018 at 5:53 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
> On Wed, Jun 6, 2018 at 3:55 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>> On Wed, Jun 6, 2018 at 12:27 AM, Matthew Brett <matthew.brett at gmail.com>
>> wrote:
>> > On Tue, Jun 5, 2018 at 9:17 PM, Matthew Brett <matthew.brett at gmail.com>
>> > wrote:
>> >> On Tue, Jun 5, 2018 at 8:09 PM, Matthew Brett <matthew.brett at gmail.com>
>> >> wrote:
>> >>> Hi,
>> >>>
>> >>> On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> >>> wrote:
>> >>>>
>> >>>>
>> >>>> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett
>> >>>> <matthew.brett at gmail.com>
>> >>>> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers
>> >>>>> <ralf.gommers at gmail.com>
>> >>>>> wrote:
>> >>>>> >
>> >>>>> >
>> >>>>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
>> >>>>> > <mikofski at berkeley.edu> wrote:
>> >>>>> >>
>> >>>>> >> Hi All,
>> >>>>> >>
>> >>>>> >> I am having issues using
>> >>>>> >>
>> >>>>> >>     python runtest.py
>> >>>>> >>
>> >>>>> >> from scipy master branch since commit
>> >>>>> >> 14de485c053cc275682bc7de0316fcca72b341f8
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8
>> >>>>> >>
>> >>>>> >> there are two tracebacks, I think this may have something to do
>> >>>>> >> with
>> >>>>> >> changes to LAPACK and dropping Apple Accelerate support, but not
>> >>>>> >> sure,
>> >>>>> >> and
>> >>>>> >> don't know how to fix it. Should I brew install openblas? (I'm
>> >>>>> >> going to
>> >>>>> >> try
>> >>>>> >> that)
>> >>>>> >
>> >>>>> >
>> >>>>> > Yes that is the recommendation (if you're not using Anaconda).
>> >>>>> >
>> >>>>> >
>> >>>>> >>
>> >>>>> >> Use anaconda and create a conda env? (maybe this next?)
>> >>>>> >
>> >>>>> >
>> >>>>> > If you're already using Anaconda, and the numpy from the defaults
>> >>>>> > channel,
>> >>>>> > SciPy should automatically link against MKL.
>> >>>>> >
>> >>>>> >
>> >>>>> >>
>> >>>>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
>> >>>>> >> wheels
>> >>>>> >> from PyPI, after I create the virtualenv using the homebrew
>> >>>>> >> python-3
>> >>>>> >> interpreter, I do pip install requirements.txt (see attached).
>> >>>>> >>
>> >>>>> >> 1st traceback:
>> >>>>> >>
>> >>>>> >> ImportError while importing test module
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
>> >>>>> >> Hint: make sure your test modules/packages have valid Python
>> >>>>> >> names.
>> >>>>> >> Traceback:
>> >>>>> >> scipy/cluster/__init__.py:27: in <module>
>> >>>>> >>     from . import vq, hierarchy
>> >>>>> >> scipy/cluster/vq.py:75: in <module>
>> >>>>> >>     from scipy.spatial.distance import cdist
>> >>>>> >> scipy/spatial/__init__.py:97: in <module>
>> >>>>> >>     from ._spherical_voronoi import SphericalVoronoi
>> >>>>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
>> >>>>> >>     from scipy.spatial.distance import pdist
>> >>>>> >> scipy/spatial/distance.py:123: in <module>
>> >>>>> >>     from ..linalg import norm
>> >>>>> >> scipy/linalg/__init__.py:207: in <module>
>> >>>>> >>     from ._decomp_update import *
>> >>>>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>> >>>>> >>     ???
>> >>>>> >> E   ImportError:
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so,
>> >>>>> >> 2): Symbol not found: _cbbcsd_
>> >>>>> >> E     Referenced from:
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>> >>>>> >> E     Expected in: flat namespace
>> >>>>> >> E    in
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>> >>>>>
>> >>>>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
>> >>>>> macOS machines, using our usual wheel-building procedure
>> >>>>> (+OpenBLAS):
>> >>>>>
>> >>>>> * Python 3.4:
>> >>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
>> >>>>> * Python 3.5:
>> >>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
>> >>>>
>> >>>>
>> >>>> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either
>> >>>> need a
>> >>>> site.cfg or install it to one of the regular locations):
>> >>>>
>> >>>> openblas_info:
>> >>>>
>> >>>>   libraries  not found in
>> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> >>>> '/usr/local/lib', '/usr/lib']
>> >>>>
>> >>>>   NOT AVAILABLE
>> >>>>
>> >>>>
>> >>>>
>> >>>> atlas_blas_threads_info:
>> >>>>
>> >>>> Setting PTATLAS=ATLAS
>> >>>>
>> >>>>   libraries ptf77blas,ptcblas,atlas not found in
>> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> >>>> '/usr/local/lib', '/usr/lib']
>> >>>>
>> >>>>   NOT AVAILABLE
>> >>>>
>> >>>>
>> >>>>
>> >>>> atlas_blas_info:
>> >>>>
>> >>>>   libraries f77blas,cblas,atlas not found in
>> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>> >>>> '/usr/local/lib', '/usr/lib']
>> >>>>
>> >>>>   NOT AVAILABLE
>> >>>>
>> >>>>
>> >>>>
>> >>>>   FOUND:
>> >>>>
>> >>>>     libraries = ['blas']
>> >>>>
>> >>>>     library_dirs = ['/usr/lib']
>> >>>>
>> >>>>     define_macros = [('NO_ATLAS_INFO', 1)]
>> >>>>
>> >>>>     language = f77
>> >>>
>> >>> Hmm - you're right - that's very odd.  OpenBLAS is being unpacked into
>> >>> /usr/local/lib, where it should be detected.  On Python 3.6, using the
>> >>> same machinery, it is detected - see:
>> >>>
>> >>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606
>> >>
>> >> It looks like it's numpy distutils.  numpy 1.8.2 doesn't appear to find
>> >> OpenBLAS in /usr/local/lib, even though it claims it looked.  Numpy
>> >> 1.9.3 does find it, which is why the build is OK for Python 3.6.
>> >> Writing an explicit site.cfg:
>> >>
>> >>     cat << EOF > $HOME/site.cfg
>> >> [openblas]
>> >> libraries = openblas
>> >> library_dirs = /usr/local/lib
>> >> include_dirs = /usr/local/include
>> >> runtime_library_dirs = /usr/local/lib
>> >> EOF
>> >>
>> >> fixes that locally.  Testing on Travis-CI now:
>> >>
>> >> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388461944
>> >
>> > Oops, no, Python 3.6 was using numpy 1.11.3, but this is now broken
>> > with the site.cfg, giving
>> >
>> > ld: unknown option: -rpath=/usr/local/lib
>> >
>> > https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388461965#L8489
>> >
>> > I can replicate this locally.  Deleting site.cfg fixes the error.
>> >
>> > The other builds continue to be broken with the same kind of errors;
>> > they pick up OpenBLAS, but also the /usr/lib stuff.  So, you're right,
>> > I think it needs blacklisting.  I hope that would make it possible to
>> > build with numpy 1.9.3 at least, making sure to avoid having a
>> > site.cfg.
>>
>> All macOS builds work correctly if I force numpy 1.11.3.
>>
>> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388525456
>>
>> Any ideas why numpy 1.11.3 is not picking up the /usr/lib stuff?
>
>
> There were a lot of changes to system_info related to rpath and
> default_lib_dirs handling, plus some reordering of the check for MKL,
> addition of BLIS, etc. Could be any of those, bisecting is probably easiest
> here.
>
>>
>>
>> Building wheels against 1.11.3 would be a fairly significant leap in
>> numpy requirement, so it would be better if we could build with numpy
>> 1.9.3, at least.
>
>
> A jump to 1.11.3 would still support 4 numpy versions (1.11-1.14), and 1.15
> will arrive before scipy 1.2.0. So not a big problem to make 1.11.3 the
> minimum required version.

1.11.3 is only 1 year 6 months old, it would be a shame to force that
if we could avoid it...

Cheers,

Matthew

From robert.open at rehammar.se  Thu Jun  7 01:24:43 2018
From: robert.open at rehammar.se (Robert Rehammar)
Date: Thu, 7 Jun 2018 07:24:43 +0200
Subject: [SciPy-Dev] numpy.random.* call convention
Message-ID: <50f8e83b-ee1e-7d25-06e4-8bbdeeab08ad@rehammar.se>

Dear num-py developers,

For the methods in numpy.random, the distributions typically have the 
signature (a, b, ...[, size]). a, b, ... are parameters to the 
distributions and size is optional that can be used to control how may 
samples to draw and the shape of the returned structure. The parameters 
can also be array_like to draw from different (parametrized) distributions.

However, it is not what I can see, possible to have the parameters 
array_like and at the same time use size!=None. It would be desirable to 
be able to have that. Some different conventions could be used, e.g. the 
returned structure res could be

res.shape == a.shape + size.shape

or the distributions could require that the first dimensions of size 
should be the same as a.shape. (I guess a.shape == b.shape = ...)

This all would allow me to generate more random samples in one go. Any 
thoughts on that? Adding this feature would not break compatibility 
since it is just allowing additional parameter combinations to the 
distributions. Legacy code will function the same.

Best,

Robert


From robert.kern at gmail.com  Thu Jun  7 01:44:38 2018
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 6 Jun 2018 22:44:38 -0700
Subject: [SciPy-Dev] numpy.random.* call convention
In-Reply-To: <50f8e83b-ee1e-7d25-06e4-8bbdeeab08ad@rehammar.se>
References: <50f8e83b-ee1e-7d25-06e4-8bbdeeab08ad@rehammar.se>
Message-ID: <CAF6FJit5TZ7UH77JzsePzOaGz1ym5zsfuWAiGJEQ3E+9tD9iUQ@mail.gmail.com>

On Wed, Jun 6, 2018 at 10:26 PM Robert Rehammar <robert.open at rehammar.se>
wrote:
>
> Dear num-py developers,
>
> For the methods in numpy.random, the distributions typically have the
> signature (a, b, ...[, size]). a, b, ... are parameters to the
> distributions and size is optional that can be used to control how may
> samples to draw and the shape of the returned structure. The parameters
> can also be array_like to draw from different (parametrized)
distributions.
>
> However, it is not what I can see, possible to have the parameters
> array_like and at the same time use size!=None.

Sure, you can! Let `shape` be `a.shape` (or the broadcasted shape of all of
those parameters. *Prepend* your desired number (or shape) of draws to this
`shape` to get the `size` that you need to specify. So if I have two
different `scale` parameters for a normal distribution, and I want 12 draws
from each in a (3,4) shape (for whatever forsaken reason):

[~]
|4> np.random.normal(0.0, [1.0, 2.0], size=(3, 4, 2))
array([[[ 1.72551057,  2.33545059],
        [-1.45966289,  4.81820745],
        [-0.13912257,  1.79127867],
        [ 0.27693464,  1.45313416]],

       [[ 1.41031607,  3.18113465],
        [ 1.64033152,  1.47355763],
        [ 1.18554024, -1.11605743],
        [ 0.73556545,  2.44352574]],

       [[-0.42889339,  3.88389374],
        [-0.24146162,  0.54163374],
        [ 0.53821574,  0.07862412],
        [ 0.7418073 , -2.35439217]]])

It's *not* a particularly intuitive API, obviously, but it should let you
do everything that you want to do.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180606/65cd5665/attachment.html>

From robert.open at rehammar.se  Thu Jun  7 01:48:58 2018
From: robert.open at rehammar.se (Robert Rehammar)
Date: Thu, 7 Jun 2018 07:48:58 +0200
Subject: [SciPy-Dev] numpy.random.* call convention
In-Reply-To: <CAF6FJit5TZ7UH77JzsePzOaGz1ym5zsfuWAiGJEQ3E+9tD9iUQ@mail.gmail.com>
References: <50f8e83b-ee1e-7d25-06e4-8bbdeeab08ad@rehammar.se>
 <CAF6FJit5TZ7UH77JzsePzOaGz1ym5zsfuWAiGJEQ3E+9tD9iUQ@mail.gmail.com>
Message-ID: <366af7da-4d43-2621-b9b9-1fbd60dcc9d6@rehammar.se>

Ah, thanks Robert, that solved it. It was actually my second suggestion 
how to solve it, but I missed the prepend-bit. Maybe that could be added 
to the documentation. I appended it :)

Best,

Robert


On 06/07/2018 07:44 AM, Robert Kern wrote:
> On Wed, Jun 6, 2018 at 10:26 PM Robert Rehammar 
> <robert.open at rehammar.se <mailto:robert.open at rehammar.se>> wrote:
> >
> > Dear num-py developers,
> >
> > For the methods in numpy.random, the distributions typically have the
> > signature (a, b, ...[, size]). a, b, ... are parameters to the
> > distributions and size is optional that can be used to control how may
> > samples to draw and the shape of the returned structure. The parameters
> > can also be array_like to draw from different (parametrized) 
> distributions.
> >
> > However, it is not what I can see, possible to have the parameters
> > array_like and at the same time use size!=None.
>
> Sure, you can! Let `shape` be `a.shape` (or the broadcasted shape of 
> all of those parameters. *Prepend* your desired number (or shape) of 
> draws to this `shape` to get the `size` that you need to specify. So 
> if I have two different `scale` parameters for a normal distribution, 
> and I want 12 draws from each in a (3,4) shape (for whatever forsaken 
> reason):
>
> [~]
> |4> np.random.normal(0.0, [1.0, 2.0], size=(3, 4, 2))
> array([[[ 1.72551057,? 2.33545059],
> ? ? ? ? [-1.45966289,? 4.81820745],
> ? ? ? ? [-0.13912257,? 1.79127867],
> ? ? ? ? [ 0.27693464,? 1.45313416]],
>
> ? ? ? ?[[ 1.41031607,? 3.18113465],
> ? ? ? ? [ 1.64033152,? 1.47355763],
> ? ? ? ? [ 1.18554024, -1.11605743],
> ? ? ? ? [ 0.73556545,? 2.44352574]],
>
> ? ? ? ?[[-0.42889339,? 3.88389374],
> ? ? ? ? [-0.24146162,? 0.54163374],
> ? ? ? ? [ 0.53821574,? 0.07862412],
> ? ? ? ? [ 0.7418073 , -2.35439217]]])
>
> It's *not* a particularly intuitive API, obviously, but it should let 
> you do everything that you want to do.
>
> --
> Robert Kern
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180607/7aa7c1a2/attachment-0001.html>

From pav at iki.fi  Thu Jun  7 06:03:36 2018
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 7 Jun 2018 12:03:36 +0200
Subject: [SciPy-Dev] runtest issue mac os with master
 14de485c053cc275682bc7de0316fcca72b341f8
In-Reply-To: <CAH6Pt5p=fHboETch-APB4LbLki1m4bX6HhMhF90Y3B3MGXqM6Q@mail.gmail.com>
References: <CAEqRcW1ki=Zj_M09E0g_R34YNgfobCkOOoLEFWGW5tAJU3B12w@mail.gmail.com>
 <CABL7CQjgk+tY4frhk2sKBwKpXAJuQGoz+yU2mbNzCBWU1WMqAQ@mail.gmail.com>
 <CAH6Pt5oxhEP8RkRHCc92yOkYGFgH+PX_DUxE1c8Zzmz22pYFnQ@mail.gmail.com>
 <CABL7CQggEE3TLVABn9UZt2MZkndF+K6+A-Ajaph17mdPmShhrg@mail.gmail.com>
 <CAH6Pt5p9t7YZmx+=-kjDKbaqc08hxaYq65xLYs6Nm2L+=FhVkw@mail.gmail.com>
 <CAH6Pt5pwxFXoYYDmVJa7ZQDu5nuavFdtLwjuTBfhGonXnqBRYQ@mail.gmail.com>
 <CAH6Pt5qqYp_a79Z=Op5VtMMQeipy4ZTYHw3v4Gs3_YcSEqf==A@mail.gmail.com>
 <CAH6Pt5oKFTk2mZTZF8cJG+xp1zD3muLCg=nfk27bfpXqWvd7nA@mail.gmail.com>
 <CABL7CQiAK8aOdcJrHWnG87UegDM9oMKQUtwzfYbDAiyHszOSmQ@mail.gmail.com>
 <CAH6Pt5p=fHboETch-APB4LbLki1m4bX6HhMhF90Y3B3MGXqM6Q@mail.gmail.com>
Message-ID: <CAEVY7kTGuS01n_KSi++sq2QWMVbA09zvAQz8d0mLsx4ZTRmdDQ@mail.gmail.com>

You can do like in the windows build, and replace distutils/ folder in
numpy with one from a newer version. See appveyor.yml

ke 6. kes?kuuta 2018 23.30 Matthew Brett <matthew.brett at gmail.com>
kirjoitti:

> On Wed, Jun 6, 2018 at 5:53 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >
> >
> > On Wed, Jun 6, 2018 at 3:55 AM, Matthew Brett <matthew.brett at gmail.com>
> > wrote:
> >>
> >> On Wed, Jun 6, 2018 at 12:27 AM, Matthew Brett <matthew.brett at gmail.com
> >
> >> wrote:
> >> > On Tue, Jun 5, 2018 at 9:17 PM, Matthew Brett <
> matthew.brett at gmail.com>
> >> > wrote:
> >> >> On Tue, Jun 5, 2018 at 8:09 PM, Matthew Brett <
> matthew.brett at gmail.com>
> >> >> wrote:
> >> >>> Hi,
> >> >>>
> >> >>> On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <
> ralf.gommers at gmail.com>
> >> >>> wrote:
> >> >>>>
> >> >>>>
> >> >>>> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett
> >> >>>> <matthew.brett at gmail.com>
> >> >>>> wrote:
> >> >>>>>
> >> >>>>> Hi,
> >> >>>>>
> >> >>>>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers
> >> >>>>> <ralf.gommers at gmail.com>
> >> >>>>> wrote:
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
> >> >>>>> > <mikofski at berkeley.edu> wrote:
> >> >>>>> >>
> >> >>>>> >> Hi All,
> >> >>>>> >>
> >> >>>>> >> I am having issues using
> >> >>>>> >>
> >> >>>>> >>     python runtest.py
> >> >>>>> >>
> >> >>>>> >> from scipy master branch since commit
> >> >>>>> >> 14de485c053cc275682bc7de0316fcca72b341f8
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>
> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8
> >> >>>>> >>
> >> >>>>> >> there are two tracebacks, I think this may have something to do
> >> >>>>> >> with
> >> >>>>> >> changes to LAPACK and dropping Apple Accelerate support, but
> not
> >> >>>>> >> sure,
> >> >>>>> >> and
> >> >>>>> >> don't know how to fix it. Should I brew install openblas? (I'm
> >> >>>>> >> going to
> >> >>>>> >> try
> >> >>>>> >> that)
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > Yes that is the recommendation (if you're not using Anaconda).
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >>
> >> >>>>> >> Use anaconda and create a conda env? (maybe this next?)
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > If you're already using Anaconda, and the numpy from the
> defaults
> >> >>>>> > channel,
> >> >>>>> > SciPy should automatically link against MKL.
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >>
> >> >>>>> >> Currently I'm using python 3.6 from homebrew in a virtualenv
> with
> >> >>>>> >> wheels
> >> >>>>> >> from PyPI, after I create the virtualenv using the homebrew
> >> >>>>> >> python-3
> >> >>>>> >> interpreter, I do pip install requirements.txt (see attached).
> >> >>>>> >>
> >> >>>>> >> 1st traceback:
> >> >>>>> >>
> >> >>>>> >> ImportError while importing test module
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>
> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
> >> >>>>> >> Hint: make sure your test modules/packages have valid Python
> >> >>>>> >> names.
> >> >>>>> >> Traceback:
> >> >>>>> >> scipy/cluster/__init__.py:27: in <module>
> >> >>>>> >>     from . import vq, hierarchy
> >> >>>>> >> scipy/cluster/vq.py:75: in <module>
> >> >>>>> >>     from scipy.spatial.distance import cdist
> >> >>>>> >> scipy/spatial/__init__.py:97: in <module>
> >> >>>>> >>     from ._spherical_voronoi import SphericalVoronoi
> >> >>>>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
> >> >>>>> >>     from scipy.spatial.distance import pdist
> >> >>>>> >> scipy/spatial/distance.py:123: in <module>
> >> >>>>> >>     from ..linalg import norm
> >> >>>>> >> scipy/linalg/__init__.py:207: in <module>
> >> >>>>> >>     from ._decomp_update import *
> >> >>>>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
> >> >>>>> >>     ???
> >> >>>>> >> E   ImportError:
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>
> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/
> cython_lapack.cpython-36m-darwin.so,
> >> >>>>> >> 2): Symbol not found: _cbbcsd_
> >> >>>>> >> E     Referenced from:
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>
> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/
> cython_lapack.cpython-36m-darwin.so
> >> >>>>> >> E     Expected in: flat namespace
> >> >>>>> >> E    in
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>
> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/
> cython_lapack.cpython-36m-darwin.so
> >> >>>>>
> >> >>>>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
> >> >>>>> macOS machines, using our usual wheel-building procedure
> >> >>>>> (+OpenBLAS):
> >> >>>>>
> >> >>>>> * Python 3.4:
> >> >>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
> >> >>>>> * Python 3.5:
> >> >>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
> >> >>>>
> >> >>>>
> >> >>>> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either
> >> >>>> need a
> >> >>>> site.cfg or install it to one of the regular locations):
> >> >>>>
> >> >>>> openblas_info:
> >> >>>>
> >> >>>>   libraries  not found in
> >> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> >> >>>> '/usr/local/lib', '/usr/lib']
> >> >>>>
> >> >>>>   NOT AVAILABLE
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> atlas_blas_threads_info:
> >> >>>>
> >> >>>> Setting PTATLAS=ATLAS
> >> >>>>
> >> >>>>   libraries ptf77blas,ptcblas,atlas not found in
> >> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> >> >>>> '/usr/local/lib', '/usr/lib']
> >> >>>>
> >> >>>>   NOT AVAILABLE
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> atlas_blas_info:
> >> >>>>
> >> >>>>   libraries f77blas,cblas,atlas not found in
> >> >>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
> >> >>>> '/usr/local/lib', '/usr/lib']
> >> >>>>
> >> >>>>   NOT AVAILABLE
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>   FOUND:
> >> >>>>
> >> >>>>     libraries = ['blas']
> >> >>>>
> >> >>>>     library_dirs = ['/usr/lib']
> >> >>>>
> >> >>>>     define_macros = [('NO_ATLAS_INFO', 1)]
> >> >>>>
> >> >>>>     language = f77
> >> >>>
> >> >>> Hmm - you're right - that's very odd.  OpenBLAS is being unpacked
> into
> >> >>> /usr/local/lib, where it should be detected.  On Python 3.6, using
> the
> >> >>> same machinery, it is detected - see:
> >> >>>
> >> >>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606
> >> >>
> >> >> It looks like it's numpy distutils.  numpy 1.8.2 doesn't appear to
> find
> >> >> OpenBLAS in /usr/local/lib, even though it claims it looked.  Numpy
> >> >> 1.9.3 does find it, which is why the build is OK for Python 3.6.
> >> >> Writing an explicit site.cfg:
> >> >>
> >> >>     cat << EOF > $HOME/site.cfg
> >> >> [openblas]
> >> >> libraries = openblas
> >> >> library_dirs = /usr/local/lib
> >> >> include_dirs = /usr/local/include
> >> >> runtime_library_dirs = /usr/local/lib
> >> >> EOF
> >> >>
> >> >> fixes that locally.  Testing on Travis-CI now:
> >> >>
> >> >> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388461944
> >> >
> >> > Oops, no, Python 3.6 was using numpy 1.11.3, but this is now broken
> >> > with the site.cfg, giving
> >> >
> >> > ld: unknown option: -rpath=/usr/local/lib
> >> >
> >> > https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388461965#L8489
> >> >
> >> > I can replicate this locally.  Deleting site.cfg fixes the error.
> >> >
> >> > The other builds continue to be broken with the same kind of errors;
> >> > they pick up OpenBLAS, but also the /usr/lib stuff.  So, you're right,
> >> > I think it needs blacklisting.  I hope that would make it possible to
> >> > build with numpy 1.9.3 at least, making sure to avoid having a
> >> > site.cfg.
> >>
> >> All macOS builds work correctly if I force numpy 1.11.3.
> >>
> >> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388525456
> >>
> >> Any ideas why numpy 1.11.3 is not picking up the /usr/lib stuff?
> >
> >
> > There were a lot of changes to system_info related to rpath and
> > default_lib_dirs handling, plus some reordering of the check for MKL,
> > addition of BLIS, etc. Could be any of those, bisecting is probably
> easiest
> > here.
> >
> >>
> >>
> >> Building wheels against 1.11.3 would be a fairly significant leap in
> >> numpy requirement, so it would be better if we could build with numpy
> >> 1.9.3, at least.
> >
> >
> > A jump to 1.11.3 would still support 4 numpy versions (1.11-1.14), and
> 1.15
> > will arrive before scipy 1.2.0. So not a big problem to make 1.11.3 the
> > minimum required version.
>
> 1.11.3 is only 1 year 6 months old, it would be a shame to force that
> if we could avoid it...
>
> Cheers,
>
> Matthew
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180607/39db20bf/attachment-0001.html>

From ralf.gommers at gmail.com  Sat Jun  9 16:12:27 2018
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 9 Jun 2018 13:12:27 -0700
Subject: [SciPy-Dev] Interest in improvements to the cKDTree codebase?
In-Reply-To: <CAH1y-JWppmXpuGoE05w5akV-hObjQ9gR+hs3iKg6Cjd4KkQf+Q@mail.gmail.com>
References: <CAH1y-JWppmXpuGoE05w5akV-hObjQ9gR+hs3iKg6Cjd4KkQf+Q@mail.gmail.com>
Message-ID: <CABL7CQgwUUfphHM-65mdjgUMBQRabUOznOGPbwgVecnbUU1OeA@mail.gmail.com>

Hi Jesse,


On Tue, May 22, 2018 at 10:56 AM, Jesse Livezey <jesse.livezey at gmail.com>
wrote:

> Hi everyone,
>
> I'm using cKDTrees for a project and noticed two potential ways to improve
> the code and have written one additional count method that might be useful
> to others.
>
> I have written code and some tests for all three items and can contribute
> if there is interest.
>
> 1) Allowing an array of rs in cKDTree.query_ball_point(). I started a PR
> here <https://github.com/scipy/scipy/pull/8818>. In principle, this
> should speed up multiple queries with different rs, but see 2.
>
> 2) I noticed that for the cases when cKDTree.query_ball_point() returns
> large numbers of points (>100), it is faster to loop over queries in
> python. This is true both with and without my PR. This is largely because
> single point queries do not sort the return indices, but multi-point
> queries DO sort them (see details here
> <https://github.com/scipy/scipy/issues/8838>). Removing the sorted()
> leads to considerable speedups, but is not backwards compatible. However,
> the sorted behavior is not in the method description and is not even
> internally consistent, so maybe it could just be removed or made optional?
>

I agree with Sturla's reply on the issue; adding a keyword that defaults to
the current behavior and allows turning off sorting is probably the way to
go.


> 3) I have written a cKDTree.count_ball_point() method that behaves like
> query_ball_point() but just returns the number of points rather than their
> indices. This is much faster than calling len(query_ball_point()).
>

How much faster is it after adding the keyword for (2)?

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180609/90a18d9a/attachment.html>

From matti.picus at gmail.com  Tue Jun 12 20:16:09 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 12 Jun 2018 17:16:09 -0700
Subject: [SciPy-Dev] NumPy 1.14.5 released
Message-ID: <ab1f2bd0-01dd-75c3-f585-4a46503e7768@gmail.com>

Hi All,

I am pleased to announce the release of NumPy 14.4.5.

This is a bugfix release for bugs reported following the 1.14.4 release. The
most significant fixes are:

* fixes for compilation errors on alpine and NetBSD

The Python versions supported in this release are 2.7 and 3.4 - 3.6. The 
Python
3.6 wheels available from PIP are built with Python 3.6.2 and should be
compatible with all previous versions of Python 3.6. The source releases 
were
cythonized with Cython 0.28.2 and should work for the upcoming Python 3.7.

Contributors
============

A total of 1 person contributed to this release.? People with a "+" by their
names contributed a patch for the first time.

* Charles Harris

Pull requests merged
====================

A total of 2 pull requests were merged for this release.

* `#11274 <https://github.com/numpy/numpy/pull/11274>`__: BUG: Correct 
use of NPY_UNUSED.
* `#11294 <https://github.com/numpy/numpy/pull/11294>`__: BUG: Remove 
extra trailing parentheses.

Cheers,
Matti

From charlesr.harris at gmail.com  Tue Jun 12 20:28:41 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 12 Jun 2018 18:28:41 -0600
Subject: [SciPy-Dev] NumPy 1.14.5 released
In-Reply-To: <ab1f2bd0-01dd-75c3-f585-4a46503e7768@gmail.com>
References: <ab1f2bd0-01dd-75c3-f585-4a46503e7768@gmail.com>
Message-ID: <CAB6mnxLyDhxQqFFV3twmbZEKowH1NEua40imrhU=19ectFi2gg@mail.gmail.com>

On Tue, Jun 12, 2018 at 6:16 PM, Matti Picus <matti.picus at gmail.com> wrote:

> Hi All,
>
> I am pleased to announce the release of NumPy 14.4.5.
>
> This is a bugfix release for bugs reported following the 1.14.4 release.
> The
> most significant fixes are:
>
> * fixes for compilation errors on alpine and NetBSD
>
> The Python versions supported in this release are 2.7 and 3.4 - 3.6. The
> Python
> 3.6 wheels available from PIP are built with Python 3.6.2 and should be
> compatible with all previous versions of Python 3.6. The source releases
> were
> cythonized with Cython 0.28.2 and should work for the upcoming Python 3.7.
>
> Contributors
> ============
>
> A total of 1 person contributed to this release.  People with a "+" by
> their
> names contributed a patch for the first time.
>
> * Charles Harris
>
> Pull requests merged
> ====================
>
> A total of 2 pull requests were merged for this release.
>
> * `#11274 <https://github.com/numpy/numpy/pull/11274>`__: BUG: Correct
> use of NPY_UNUSED.
> * `#11294 <https://github.com/numpy/numpy/pull/11294>`__: BUG: Remove
> extra trailing parentheses.
>
>
Congratulations :) How was it?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180612/57687fd7/attachment.html>

From maurizio.cipollina at gmail.com  Fri Jun 15 13:51:34 2018
From: maurizio.cipollina at gmail.com (Maurizio Cipollina)
Date: Fri, 15 Jun 2018 19:51:34 +0200
Subject: [SciPy-Dev] Stochastic calculus package
In-Reply-To: <CAMMTP+BevuDHcDa5xw2Lj3niaZMYo_x=WYRUaW-Grj8eRK9xJw@mail.gmail.com>
References: <CAKfLqW08MOO+zQOMpxWO4jrxTHN-6BMDU_Qb=d0SUYp0iNi2Eg@mail.gmail.com>
 <CABL7CQg8_GeR1F6qeu65HJhx0xCN1FXav1avk4Zrp_YacDv21w@mail.gmail.com>
 <CAKfLqW2YDCg=+WSbAhzDNK9_Hz-2RC1UokZxtYXz2A-pq6TG4g@mail.gmail.com>
 <CAMMTP+BevuDHcDa5xw2Lj3niaZMYo_x=WYRUaW-Grj8eRK9xJw@mail.gmail.com>
Message-ID: <CAKfLqW0rBsznR8kou4t15YG3qQn4D8VMN+PbN=QJ6SYBC33GOQ@mail.gmail.com>

Hi everybody,

following up on this conversation of some time ago, a pre-release of the
package, renamed SdePy, is now available on pip (
https://pypi.org/project/sdepy  ) and as a project on github (
https://github.com/sdepy/sdepy ), version 1.0.0rc3. A short quickstart
guide (https://sdepy.readthedocs.io/en/latest/intro.html#id2 ) is the
easiest path to see what it does and what it does not. Any feedback,
comments and suggestions are very welcome!

Since last September, the package has evolved quite a bit and had its fair
share of refactoring, going through several 0.x unpublished versions before
stabilizing in its current form (marked as beta). Here are a few takeaways
I got from this thread, which I acted upon:

-  Integrating user defined SDEs turned out to be a main point of interest:
this ?odeint? like functionality has been considerably streamlined and
  simplified - formerly, preset processes were mainly in focus, and getting
a new process up and running took a deep dive in a not-so-welcoming API,
now upgraded as well (the former ?integrator? has been superseded by a
?paths_generator? parent class, and cooperating ?integrator? and ?SDE?
classes).

-  The ?kfunc? beast was admittedly a hard sell: all undue dependencies of
the core modules on its machinery have been eliminated, it plays now the
role of an optional decorator for the shortcuts of frequently used stuff
(it?s fully functional though and, in my experience, quite handy).

-  The question ?Do you REALLY need a ?process? class? clearly stood out in
your comments: after some experimentation, the main reason for answering in
the yes turned out, in my view, to be its seamless interoperability with
the rest of the package, hardly achievable without a dedicated design. Now
process instances can indeed be fed to integrators, both as time-dependent
and/or path-dependent SDE parameters, and as stochasticity sources: this
opened countless hacking opportunities when doing computations, and
spoiling them seemed a pity.

In case any of you is willing to take a look, feel free to get in touch for
any problems/bugs/questions you might have

Cheers
Maurizio

On Mon, 11 Sep 2017 at 19:39, <josef.pktd at gmail.com> wrote:

>
>
> On Mon, Sep 11, 2017 at 12:35 PM, Maurizio Cipollina <
> maurizio.cipollina at gmail.com> wrote:
>
>> Hi Ralf,
>>
>>
>> thanks for going through this, here are some follow up comments on your
>> points:
>>
>>
>>
>> On the two packages you found:
>>
>>
>>    - Sdeint: provides several SDE integration algorithms but no specific
>>    processes; does not cover processes with jumps; offers no explicit
>>    provisions for handling solutions in a number of paths, which is what you
>>    need since the integration output is random variables, not numbers.
>>    - SDE-higham is not a package or tool, but rather a collection of
>>    recipes for a few textbook examples (no jumps either), in the form that an
>>    interactive session might take (set globals, do calculations, plot and
>>    print results).
>>
>>
>>
>> About your questions/concerns:
>>
>>
>>    1. ndarray subclassing: the integrator does not depend on the process
>>    class, and process realizations may be easily tweaked to return an ndarray
>>    instead of a process instance; however, a main point of the package is not
>>    just to produce SDE solutions, but to allow the user to handle them
>>    (almost) effortlessly once obtained: so one might possibly hide the process
>>    class inside the testing suite, which depends on it, but hardly get rid of
>>    it altogether. Indeed, I subclassed ndarray as safely as I could figure
>>    out, but I will surely go through the scipy-dev archives and find out what
>>    advised you against doing so.
>>    2. odeint and scipy.integrate: my view is that odeint churns out
>>    numbers dependant on numbers, while SDE integration churns out random
>>    variables dependant on random variables (the stochasticity sources), so
>>    there is a shift of paradigm and the two schemes do not fit smoothly one
>>    within the other. The step by step integration of SDEs makes sense, and
>>    gets useful, if you can easily control the inputs (eg. correlations among
>>    stochasticity sources fed into different SDEs), generate the output in a
>>    decent number of paths, and easily extract statistics of expressions
>>    dependent on it (hence the flow ?get a number of paths packaged as a
>>    process instance? -> ?do numpy algebra with it? -> ?estimate probability
>>    distributions etc. using the process class methods? -> ?scale it up with
>>    the montecarlo class in case the needed paths do not fit into memory?).
>>    3. Montecarlo simulations is a big topic indeed, that goes far beyond
>>    this package; what is called the montecarlo class is in fact a tool to
>>    cumulate statistical information extracted from a number of realizations of
>>    a random variable, and is focused on making easy to interpret the output of
>>    SDE integration.
>>    4. Black-Scholes assists one more tests in a rather extensive testing
>>    suite, and might be painlessly dropped.
>>
>>
>>
>> This said, I see your general feeling is that this whole subject is too
>> specialized to fit into scipy at this stage. I suggested otherwise, in my
>> proposal, because stochastic calculus is regarded as one of the pillars of
>> modern probability theory, and covering it into scipy might be relevant to
>> lots of mathematicians and physicists, both practitioners and students,
>> that might otherwise hesitate to adopt, and build dependencies upon, a
>> standalone package. In case you reconsider let me know, I?d be happy to
>> help (now or in the future?).
>>
>>
>>
>> If you have further questions, let me know as well.
>>
>> Cheers
>>
>> Maurizio
>>
>
>
> To partially follow up, similar to Ralph's view.
>
> I think it would be good to have this as a separate package, or at least
> the code on github.
> With distributions like conda it is now much easier to install additional
> packages, once they are established enough and are included in
> distributions or easily pip installable.
>
> The main reason that I think scipy is currently not the appropriate
> package for it is that the design and review would require a lot of work. I
> guess that there are not many maintainers and reviewers that are familiar
> with this area. I also know only of applications in finance and it is not
> clear whether or which parts would be of more general interest.
>
> In my opinion, some core tools for SDE and jump diffusions would fit in
> scipy. But the application support will not or might not fit well in a
> general purpose scientific numerical library. This would be similar to
> other areas where scipy provides the core computational tools, but
> applications are supported by other packages.
>
> For example, the process class might be similar in magnitude to scipy
> stats distributions or signal statespace classes, but I have no idea what
> design for it would fit in scipy.
> Some of the time aggregation properties sound similar to the corresponding
> pandas functions, and it is not clear whether users would want to use a
> separate class or just use pandas instead.
>
> Similarly, the kfunc class sounds like something that doesn't have a
> similar pattern in scipy.
>
> Design changes can be made more easily in a standalone package, while all
> refactorings of classes in scipy are difficult issues because of the
> backwards compatibility policy which requires that the design should be
> largely settled before a merge.
>
> Josef
>
> https://groups.google.com/d/msg/pystatsmodels/xsttiEiyqAg/maR6n4EeAgAJ
> (stackoverflow question has been deleted "Is there a library out there to
> calibrate an Ornstein-Uhlenbeck model?")
>
>
> https://github.com/statsmodels/statsmodels/blob/master/statsmodels/sandbox/tsa/diffusion.py
> and diffusion2.py
>
>
>
>
>>
>>
>>
>> On 9 September 2017 at 03:31, Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>>
>>> Hi Maurizio,
>>>
>>>
>>> On Fri, Sep 8, 2017 at 7:55 PM, Maurizio Cipollina <
>>> maurizio.cipollina at gmail.com> wrote:
>>>
>>>> Hi everybody, and first of all thanks for the great job you have been
>>>> doing, on which I relied *A LOT* over time.
>>>>
>>>> Stochastic calculus and Stochastic Differential Equations are a
>>>> significant branch of mathematics that to my understanding is
>>>> underrepresented in scipy, as well as in the open source python stack at
>>>> large. I may well be missing something, but as I can see one can find out
>>>> there many recipies or collections of recipies, and a number of
>>>> packages/projects skewed towards finance, but no structured framework for
>>>> generating and handling stochastic processes and numerically solving SDEs.
>>>>
>>>
>>> I've never used these, but a quick search turns up two packages:
>>>   https://pypi.python.org/pypi/sdeint
>>>   https://github.com/alu042/SDE-higham
>>> Not very mature probably, but would be good to compare your code with
>>> those.
>>>
>>>
>>>> This is a pity, since scipy/numpy provide efficen implementations of
>>>> all the basic ingredients needed to do the trick. Out of professional need,
>>>> and prompted by this gap, I have developed a general purpose package of
>>>> which I own the copyright, and which I would be happy to release under the
>>>> BSD license as part of scipy, in case there is a consensus among the scipy
>>>> community that it makes sense to do so (the package has no dependencies
>>>> beyond standard library, numpy and scipy, and might probably fit best as a
>>>> subpackage of scipy.stats).
>>>>
>>>> Before setting up a PR, I would be happy to hear your thoughts: I have
>>>> pasted below the package docstring, that should give an overview of its
>>>> scope and limitations. Please note that some of the realized processes
>>>> stray into mathematical finance territory, somehow inevitably since finance
>>>> is one major field of application of stochastic calculus, but the focus is
>>>> the latter, not the former, at least in my intentions.
>>>>
>>>
>>> First thought: this looks like a useful addition to the scientific
>>> Python ecosystem, but is probably too specialized for SciPy (at least at
>>> this stage).
>>>
>>> Based on your docstring below, here are some questions/concerns:
>>> 1. We definitely don't want a new ndarray subclass (there's a pretty
>>> strong consensus at this point that subclassing ndarray is usually not a
>>> good solution), so your `process` doesn't sound attractive.
>>> 2. The `integrator` class sounds a lot like odeint, so maybe this would
>>> fit better with scipy.integrate?
>>> 3. Monte-Carlo simulation is a big and fairly technical topic. There are
>>> whole packages (e.g. PyMC3, emcee) dedicated to this. It may not fit well
>>> with SciPy.
>>> 4. Things specific to finance like Black-Scholes and put/call options
>>> are not a good fit.
>>> 5. Maintainers for this added code. The ODE integrators in
>>> scipy.integrate already suffer from lack of a dedicated maintainer, SDEs
>>> are more specialized so are likely to not be very well-maintained by other
>>> devs than you.
>>>
>>> Without seeing your code it's hard to say for sure, but it looks to me
>>> like it would be better to release your code as a standalone package.
>>>
>>> Cheers,
>>> Ralf
>>>
>>>
>>>
>>>> Thanks in advance for taking your time to go through this, and for your
>>>> comments and suggestions.
>>>> Maurizio
>>>>
>>>> ???
>>>> ===========================================================
>>>> Stochastic - Monte Carlo simulation of stochastic processes
>>>> ===========================================================
>>>>
>>>> This package provides:
>>>>
>>>> 1.  A `process` class, a subclass of `numpy.ndarray` representing
>>>>     a sequence of values in time, realized in one or several paths.
>>>>     Algebraic manipulations and ufunc computations are supported for
>>>>     instances that share the same timeline and comply with numpy
>>>>     broadcasting rules. Interpolation along the timeline is supported
>>>>     via callability of `process` instances. Process-specific
>>>> functionalities,
>>>>     such as averaging and indexing along time or across paths, are
>>>> delegated
>>>>     to process-specific methods, attributes and properties (no
>>>> overriding
>>>>     of `numpy.ndarray` operations).
>>>>
>>>> 2.  The `source` class and its subclasses, stochasticity sources
>>>> providing
>>>>     numerical realizations of the differentials commonly found
>>>>     in stochastic differential equations (SDEs), with or without
>>>>     memory of formerly invoked realizations.
>>>>
>>>> 3.  The `integrator` class, a general framework for numerical SDE
>>>>     integration, intended for subclassing, to assist the definition of
>>>> step by
>>>>     step integration algorithms and the computation of numerical
>>>> realizations of
>>>>     stochastic processes.
>>>>
>>>> 4.  Several objects providing realizations of specific stochastic
>>>> processes
>>>>     along a given timeline and across a requested number of paths.
>>>>     Realizations are obtained via explicit formulae, in case the
>>>> relevant SDE
>>>>     has a closed form solution, or otherwise as `integrator` subclasses
>>>> performing
>>>>     Euler-Maruyama numerical integration.
>>>>
>>>> 5.  A `montecarlo` class, as an aid to cumulate the results of several
>>>>     Monte Carlo simulations of a given stochastic variable, and to
>>>> extract
>>>>     summary estimates for its probability distribution function and
>>>>     statistics, thus supporting simulations across a number of paths
>>>> that exceeds
>>>>     the maximum allowed by available memory.
>>>>
>>>> 6.  Several analytical results relating to the supported stochastic
>>>>     processes, as a general reference and for testing purpouses.
>>>>
>>>> For all sources and realizations, process values can take any shape,
>>>> scalar or multidimensional. Correlated multivariate stochasticity
>>>> sources are
>>>> supported. Time-varying process parameters (correlations, intensity of
>>>> Poisson
>>>> processes, volatilities etc.) are allowed whenever applicable.
>>>> `process` instances act as valid stochasticity source realizations (as
>>>> does
>>>> any callable object complying with a `source` protocol), and may be
>>>> passed as a source specification when computing the realization of a
>>>> given
>>>> process.
>>>>
>>>> Computations are fully vectorized across paths, providing an efficient
>>>> infrastructure for simulating a large number of process realizations.
>>>> Less so, for large number of time steps: integrating 1000 time steps
>>>> across 100000 paths takes seconds, one million time steps across
>>>> 100 paths takes minutes.
>>>>
>>>> General infrastructure
>>>> ======================
>>>> .. autosummary::
>>>>    :toctree: generated/
>>>>
>>>>    process     -- Array representation of a process (a subclass of
>>>>                   -- numpy.ndarray).
>>>>    kfunc       -- Base class for functions with managed keyword
>>>> arguments.
>>>>    source      -- Base class for stochasticity sources.
>>>>    true_source -- Base class for stochasticity sources with memory.
>>>>    integrator  -- General framework for numerical SDE integration.
>>>>    montecarlo  -- Summmary statistics of Monte Carlo simulations.
>>>>
>>>>
>>>> Stochasticity sources
>>>> =====================
>>>> .. autosummary::
>>>>    :toctree: generated/
>>>>
>>>>    wiener              -- dW, a source of standard Wiener process
>>>> (Brownian
>>>>                           -- motion) increments.
>>>>    symmetric_wiener    -- as above, with symmetric paths (averages
>>>> exactly 0).
>>>>    true_wiener         -- dW, a source of standard Wiener process
>>>> (Brownian
>>>>                           -- motion) increments, with memory.
>>>>    poisson             -- dN, a source of Poisson process increments.
>>>>    compound_poisson    -- dJ, a source of compound Poisson process
>>>> increments.
>>>>                           -- (jumps distributed in time as a Poisson
>>>> process,
>>>>                           -- and in size as a given `scipy.stats`
>>>> distribution).
>>>>    compound_poisson_rv -- Preset jump size distributions for compound
>>>> Poisson
>>>>                           -- process increments.
>>>>
>>>>
>>>> Stochastic process realizations
>>>> ===============================
>>>> .. autosummary::
>>>>    :toctree: generated/
>>>>
>>>>    const_wiener_process          -- Wiener process (Brownian motion),
>>>> with
>>>>                                     -- time-independent parameters.
>>>>    const_lognorm_process         -- Lognormal process, with
>>>> time-independent
>>>>                                     -- parameters.
>>>>    wiener_process                -- Wiener process (Brownian motion).
>>>>    lognorm_process               -- Lognormal process.
>>>>    ornstein_uhlenbeck_process    -- Ornstein-Uhlenbeck process
>>>> (mean-reverting
>>>>                                     -- Brownian motion).
>>>>    hull_white_process            -- F-factors Hull-White process (sum
>>>> of F
>>>>                                     -- correlated mean-reverting
>>>> Brownian
>>>>                                     -- motions).
>>>>    hull_white_1factor_process    -- 1-factor Hull-White process (F=1
>>>> Hull-White
>>>>                                     -- process with F-index collapsed
>>>> to a
>>>>                                     -- scalar).
>>>>    cox_ingersoll_ross_process    -- Cox-Ingersoll-Ross mean-reverting
>>>> process.
>>>>    full_heston_process           -- Heston stochastic volatility process
>>>>                                     -- (returns both process and
>>>> volatility).
>>>>    heston_process                -- Heston stochastic volatility process
>>>>                                     -- (stores and returns process
>>>> only).
>>>>    jump_diffusion_process        -- Jump-diffusion process (lognormal
>>>> process
>>>>                                     -- with compound Poisson jumps).
>>>>    merton_jump_diffusion_process -- Merton jump-diffusion process
>>>>                                     -- (jump-diffusion process with
>>>> normal
>>>>                                     -- jump size distribution).
>>>>    kou_jump_diffusion_process    -- Kou jump-diffusion process
>>>> (jump-diffusion
>>>>                                     -- process with double exponential
>>>>                                     -- jump size distribution).
>>>>
>>>> Shortcuts::
>>>>
>>>>    lognorm -- lognorm_process
>>>>    oruh    -- ornstein_uhlenbeck_process
>>>>    hwp     -- hull_white_process
>>>>    hw1f    -- hull_white_1factor_process
>>>>    cir     -- cox_ingersoll_ross_process
>>>>    heston  -- heston_process
>>>>    mjd     -- merton_jump_diffusion_process
>>>>    kou     -- kou_jump_diffusion_process
>>>>
>>>>
>>>> Analytical results
>>>> ==================
>>>> Exact statistics for the realized stochastic processes
>>>> are listed below, limited to the case of constant process parameters
>>>> and with
>>>> some further limitations to the parameters' domains.
>>>> Function arguments are consistent with those of the corresponding
>>>> processes.
>>>> Suffixes `pdf`, `cdf` and `chf` stand respectively for probability
>>>> distribution
>>>> function, cumulative probability distribution function, and
>>>> characteristic
>>>> function. Black-Scholes formulae for the valuation of call and put
>>>> options have been
>>>> included (prefixed with `bs` below).
>>>>
>>>> .. autosummary::
>>>>    :toctree: generated/
>>>>
>>>>    wiener_mean
>>>>    wiener_var
>>>>    wiener_std
>>>>    wiener_pdf
>>>>    wiener_cdf
>>>>    wiener_chf
>>>>
>>>>    lognorm_mean
>>>>    lognorm_var
>>>>    lognorm_std
>>>>    lognorm_pdf
>>>>    lognorm_cdf
>>>>    lognorm_log_chf
>>>>
>>>>    oruh_mean
>>>>    oruh_var
>>>>    oruh_std
>>>>    oruh_pdf
>>>>    oruh_cdf
>>>>
>>>>    hw2f_mean
>>>>    hw2f_var
>>>>    hw2f_std
>>>>    hw2f_pdf
>>>>    hw2f_cdf
>>>>
>>>>    cir_mean
>>>>    cir_var
>>>>    cir_std
>>>>    cir_pdf
>>>>
>>>>    heston_log_mean
>>>>    heston_log_var
>>>>    heston_log_std
>>>>    heston_log_pdf
>>>>    heston_log_chf
>>>>
>>>>    mjd_log_pdf
>>>>    mjd_log_chf
>>>>
>>>>    kou_mean
>>>>    kou_log_pdf
>>>>    kou_log_chf
>>>>
>>>>    bsd1d2
>>>>    bscall
>>>>    bscall_delta
>>>>    bsput
>>>>    bsput_delta
>>>>
>>>> Notes
>>>> =====
>>>> To the benefit of interactive and notebook sessions, and as an aid to
>>>> fluently
>>>> freeze or vary the plurality of parameters that define each stochastic
>>>> process,
>>>> all sources, process realizations and analytical results are objects
>>>> with managed keywords (subclasses of `kfunc`): if the corresponding
>>>> classes
>>>> are instantiated with both variables (non keyword arguments) and
>>>> parameters
>>>> (keyword arguments), they behave as functions returning the
>>>> computations'
>>>> result; if called with parameters only, return an instance that stores
>>>> the set
>>>> parameters, and exposes the same calling pattern (call with variables
>>>> and optionally with parameters to get results, call with parameters only
>>>> to get a new instance storing modified parameters).
>>>> Parameters that are not specified fall back to the class defaults, if
>>>> calling
>>>> the class, or to the instance's stored values, if calling an instance.
>>>> """
>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180615/fe7a450f/attachment-0001.html>

From josef.pktd at gmail.com  Fri Jun 15 14:37:59 2018
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 15 Jun 2018 14:37:59 -0400
Subject: [SciPy-Dev] Stochastic calculus package
In-Reply-To: <CAKfLqW0rBsznR8kou4t15YG3qQn4D8VMN+PbN=QJ6SYBC33GOQ@mail.gmail.com>
References: <CAKfLqW08MOO+zQOMpxWO4jrxTHN-6BMDU_Qb=d0SUYp0iNi2Eg@mail.gmail.com>
 <CABL7CQg8_GeR1F6qeu65HJhx0xCN1FXav1avk4Zrp_YacDv21w@mail.gmail.com>
 <CAKfLqW2YDCg=+WSbAhzDNK9_Hz-2RC1UokZxtYXz2A-pq6TG4g@mail.gmail.com>
 <CAMMTP+BevuDHcDa5xw2Lj3niaZMYo_x=WYRUaW-Grj8eRK9xJw@mail.gmail.com>
 <CAKfLqW0rBsznR8kou4t15YG3qQn4D8VMN+PbN=QJ6SYBC33GOQ@mail.gmail.com>
Message-ID: <CAMMTP+CjQnbgL+oCm1OnOudoO5JMBnORQhUS2u-PDhaY2=w2fg@mail.gmail.com>

Sounds good,

One thing that immediately jumps into my eyes are the lower case class names.
I think using pep-8 capital/camel case would make it look more modern.
Only old fashioned packages like numpy and scipy still have the legacy
spelling (when developers thought python has to look like matlab :)

Josef



On Fri, Jun 15, 2018 at 1:51 PM, Maurizio Cipollina
<maurizio.cipollina at gmail.com> wrote:
> Hi everybody,
>
> following up on this conversation of some time ago, a pre-release of the
> package, renamed SdePy, is now available on pip
> (https://pypi.org/project/sdepy  ) and as a project on github
> (https://github.com/sdepy/sdepy ), version 1.0.0rc3. A short quickstart
> guide (https://sdepy.readthedocs.io/en/latest/intro.html#id2 ) is the
> easiest path to see what it does and what it does not. Any feedback,
> comments and suggestions are very welcome!
>
> Since last September, the package has evolved quite a bit and had its fair
> share of refactoring, going through several 0.x unpublished versions before
> stabilizing in its current form (marked as beta). Here are a few takeaways I
> got from this thread, which I acted upon:
>
> -  Integrating user defined SDEs turned out to be a main point of interest:
> this ?odeint? like functionality has been considerably streamlined and
> simplified - formerly, preset processes were mainly in focus, and getting a
> new process up and running took a deep dive in a not-so-welcoming API, now
> upgraded as well (the former ?integrator? has been superseded by a
> ?paths_generator? parent class, and cooperating ?integrator? and ?SDE?
> classes).
>
> -  The ?kfunc? beast was admittedly a hard sell: all undue dependencies of
> the core modules on its machinery have been eliminated, it plays now the
> role of an optional decorator for the shortcuts of frequently used stuff
> (it?s fully functional though and, in my experience, quite handy).
>
> -  The question ?Do you REALLY need a ?process? class? clearly stood out in
> your comments: after some experimentation, the main reason for answering in
> the yes turned out, in my view, to be its seamless interoperability with the
> rest of the package, hardly achievable without a dedicated design. Now
> process instances can indeed be fed to integrators, both as time-dependent
> and/or path-dependent SDE parameters, and as stochasticity sources: this
> opened countless hacking opportunities when doing computations, and spoiling
> them seemed a pity.
>
> In case any of you is willing to take a look, feel free to get in touch for
> any problems/bugs/questions you might have
>
> Cheers
> Maurizio
>
> On Mon, 11 Sep 2017 at 19:39, <josef.pktd at gmail.com> wrote:
>>
>>
>>
>> On Mon, Sep 11, 2017 at 12:35 PM, Maurizio Cipollina
>> <maurizio.cipollina at gmail.com> wrote:
>>>
>>> Hi Ralf,
>>>
>>>
>>> thanks for going through this, here are some follow up comments on your
>>> points:
>>>
>>>
>>>
>>> On the two packages you found:
>>>
>>> Sdeint: provides several SDE integration algorithms but no specific
>>> processes; does not cover processes with jumps; offers no explicit
>>> provisions for handling solutions in a number of paths, which is what you
>>> need since the integration output is random variables, not numbers.
>>> SDE-higham is not a package or tool, but rather a collection of recipes
>>> for a few textbook examples (no jumps either), in the form that an
>>> interactive session might take (set globals, do calculations, plot and print
>>> results).
>>>
>>>
>>>
>>> About your questions/concerns:
>>>
>>> ndarray subclassing: the integrator does not depend on the process class,
>>> and process realizations may be easily tweaked to return an ndarray instead
>>> of a process instance; however, a main point of the package is not just to
>>> produce SDE solutions, but to allow the user to handle them (almost)
>>> effortlessly once obtained: so one might possibly hide the process class
>>> inside the testing suite, which depends on it, but hardly get rid of it
>>> altogether. Indeed, I subclassed ndarray as safely as I could figure out,
>>> but I will surely go through the scipy-dev archives and find out what
>>> advised you against doing so.
>>> odeint and scipy.integrate: my view is that odeint churns out numbers
>>> dependant on numbers, while SDE integration churns out random variables
>>> dependant on random variables (the stochasticity sources), so there is a
>>> shift of paradigm and the two schemes do not fit smoothly one within the
>>> other. The step by step integration of SDEs makes sense, and gets useful, if
>>> you can easily control the inputs (eg. correlations among stochasticity
>>> sources fed into different SDEs), generate the output in a decent number of
>>> paths, and easily extract statistics of expressions dependent on it (hence
>>> the flow ?get a number of paths packaged as a process instance? -> ?do numpy
>>> algebra with it? -> ?estimate probability distributions etc. using the
>>> process class methods? -> ?scale it up with the montecarlo class in case the
>>> needed paths do not fit into memory?).
>>> Montecarlo simulations is a big topic indeed, that goes far beyond this
>>> package; what is called the montecarlo class is in fact a tool to cumulate
>>> statistical information extracted from a number of realizations of a random
>>> variable, and is focused on making easy to interpret the output of SDE
>>> integration.
>>> Black-Scholes assists one more tests in a rather extensive testing suite,
>>> and might be painlessly dropped.
>>>
>>>
>>>
>>> This said, I see your general feeling is that this whole subject is too
>>> specialized to fit into scipy at this stage. I suggested otherwise, in my
>>> proposal, because stochastic calculus is regarded as one of the pillars of
>>> modern probability theory, and covering it into scipy might be relevant to
>>> lots of mathematicians and physicists, both practitioners and students, that
>>> might otherwise hesitate to adopt, and build dependencies upon, a standalone
>>> package. In case you reconsider let me know, I?d be happy to help (now or in
>>> the future?).
>>>
>>>
>>>
>>> If you have further questions, let me know as well.
>>>
>>> Cheers
>>>
>>> Maurizio
>>
>>
>>
>> To partially follow up, similar to Ralph's view.
>>
>> I think it would be good to have this as a separate package, or at least
>> the code on github.
>> With distributions like conda it is now much easier to install additional
>> packages, once they are established enough and are included in distributions
>> or easily pip installable.
>>
>> The main reason that I think scipy is currently not the appropriate
>> package for it is that the design and review would require a lot of work. I
>> guess that there are not many maintainers and reviewers that are familiar
>> with this area. I also know only of applications in finance and it is not
>> clear whether or which parts would be of more general interest.
>>
>> In my opinion, some core tools for SDE and jump diffusions would fit in
>> scipy. But the application support will not or might not fit well in a
>> general purpose scientific numerical library. This would be similar to other
>> areas where scipy provides the core computational tools, but applications
>> are supported by other packages.
>>
>> For example, the process class might be similar in magnitude to scipy
>> stats distributions or signal statespace classes, but I have no idea what
>> design for it would fit in scipy.
>> Some of the time aggregation properties sound similar to the corresponding
>> pandas functions, and it is not clear whether users would want to use a
>> separate class or just use pandas instead.
>>
>> Similarly, the kfunc class sounds like something that doesn't have a
>> similar pattern in scipy.
>>
>> Design changes can be made more easily in a standalone package, while all
>> refactorings of classes in scipy are difficult issues because of the
>> backwards compatibility policy which requires that the design should be
>> largely settled before a merge.
>>
>> Josef
>>
>> https://groups.google.com/d/msg/pystatsmodels/xsttiEiyqAg/maR6n4EeAgAJ
>> (stackoverflow question has been deleted "Is there a library out there to
>> calibrate an Ornstein-Uhlenbeck model?")
>>
>>
>> https://github.com/statsmodels/statsmodels/blob/master/statsmodels/sandbox/tsa/diffusion.py
>> and diffusion2.py
>>
>>
>>
>>>
>>>
>>>
>>>
>>> On 9 September 2017 at 03:31, Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>>>
>>>> Hi Maurizio,
>>>>
>>>>
>>>> On Fri, Sep 8, 2017 at 7:55 PM, Maurizio Cipollina
>>>> <maurizio.cipollina at gmail.com> wrote:
>>>>>
>>>>> Hi everybody, and first of all thanks for the great job you have been
>>>>> doing, on which I relied *A LOT* over time.
>>>>>
>>>>> Stochastic calculus and Stochastic Differential Equations are a
>>>>> significant branch of mathematics that to my understanding is
>>>>> underrepresented in scipy, as well as in the open source python stack at
>>>>> large. I may well be missing something, but as I can see one can find out
>>>>> there many recipies or collections of recipies, and a number of
>>>>> packages/projects skewed towards finance, but no structured framework for
>>>>> generating and handling stochastic processes and numerically solving SDEs.
>>>>
>>>>
>>>> I've never used these, but a quick search turns up two packages:
>>>>   https://pypi.python.org/pypi/sdeint
>>>>   https://github.com/alu042/SDE-higham
>>>> Not very mature probably, but would be good to compare your code with
>>>> those.
>>>>
>>>>>
>>>>> This is a pity, since scipy/numpy provide efficen implementations of
>>>>> all the basic ingredients needed to do the trick. Out of professional need,
>>>>> and prompted by this gap, I have developed a general purpose package of
>>>>> which I own the copyright, and which I would be happy to release under the
>>>>> BSD license as part of scipy, in case there is a consensus among the scipy
>>>>> community that it makes sense to do so (the package has no dependencies
>>>>> beyond standard library, numpy and scipy, and might probably fit best as a
>>>>> subpackage of scipy.stats).
>>>>>
>>>>> Before setting up a PR, I would be happy to hear your thoughts: I have
>>>>> pasted below the package docstring, that should give an overview of its
>>>>> scope and limitations. Please note that some of the realized processes stray
>>>>> into mathematical finance territory, somehow inevitably since finance is one
>>>>> major field of application of stochastic calculus, but the focus is the
>>>>> latter, not the former, at least in my intentions.
>>>>
>>>>
>>>> First thought: this looks like a useful addition to the scientific
>>>> Python ecosystem, but is probably too specialized for SciPy (at least at
>>>> this stage).
>>>>
>>>> Based on your docstring below, here are some questions/concerns:
>>>> 1. We definitely don't want a new ndarray subclass (there's a pretty
>>>> strong consensus at this point that subclassing ndarray is usually not a
>>>> good solution), so your `process` doesn't sound attractive.
>>>> 2. The `integrator` class sounds a lot like odeint, so maybe this would
>>>> fit better with scipy.integrate?
>>>> 3. Monte-Carlo simulation is a big and fairly technical topic. There are
>>>> whole packages (e.g. PyMC3, emcee) dedicated to this. It may not fit well
>>>> with SciPy.
>>>> 4. Things specific to finance like Black-Scholes and put/call options
>>>> are not a good fit.
>>>> 5. Maintainers for this added code. The ODE integrators in
>>>> scipy.integrate already suffer from lack of a dedicated maintainer, SDEs are
>>>> more specialized so are likely to not be very well-maintained by other devs
>>>> than you.
>>>>
>>>> Without seeing your code it's hard to say for sure, but it looks to me
>>>> like it would be better to release your code as a standalone package.
>>>>
>>>> Cheers,
>>>> Ralf
>>>>
>>>>
>>>>>
>>>>> Thanks in advance for taking your time to go through this, and for your
>>>>> comments and suggestions.
>>>>> Maurizio
>>>>>
>>>>> ???
>>>>> ===========================================================
>>>>> Stochastic - Monte Carlo simulation of stochastic processes
>>>>> ===========================================================
>>>>>
>>>>> This package provides:
>>>>>
>>>>> 1.  A `process` class, a subclass of `numpy.ndarray` representing
>>>>>     a sequence of values in time, realized in one or several paths.
>>>>>     Algebraic manipulations and ufunc computations are supported for
>>>>>     instances that share the same timeline and comply with numpy
>>>>>     broadcasting rules. Interpolation along the timeline is supported
>>>>>     via callability of `process` instances. Process-specific
>>>>> functionalities,
>>>>>     such as averaging and indexing along time or across paths, are
>>>>> delegated
>>>>>     to process-specific methods, attributes and properties (no
>>>>> overriding
>>>>>     of `numpy.ndarray` operations).
>>>>>
>>>>> 2.  The `source` class and its subclasses, stochasticity sources
>>>>> providing
>>>>>     numerical realizations of the differentials commonly found
>>>>>     in stochastic differential equations (SDEs), with or without
>>>>>     memory of formerly invoked realizations.
>>>>>
>>>>> 3.  The `integrator` class, a general framework for numerical SDE
>>>>>     integration, intended for subclassing, to assist the definition of
>>>>> step by
>>>>>     step integration algorithms and the computation of numerical
>>>>> realizations of
>>>>>     stochastic processes.
>>>>>
>>>>> 4.  Several objects providing realizations of specific stochastic
>>>>> processes
>>>>>     along a given timeline and across a requested number of paths.
>>>>>     Realizations are obtained via explicit formulae, in case the
>>>>> relevant SDE
>>>>>     has a closed form solution, or otherwise as `integrator` subclasses
>>>>> performing
>>>>>     Euler-Maruyama numerical integration.
>>>>>
>>>>> 5.  A `montecarlo` class, as an aid to cumulate the results of several
>>>>>     Monte Carlo simulations of a given stochastic variable, and to
>>>>> extract
>>>>>     summary estimates for its probability distribution function and
>>>>>     statistics, thus supporting simulations across a number of paths
>>>>> that exceeds
>>>>>     the maximum allowed by available memory.
>>>>>
>>>>> 6.  Several analytical results relating to the supported stochastic
>>>>>     processes, as a general reference and for testing purpouses.
>>>>>
>>>>> For all sources and realizations, process values can take any shape,
>>>>> scalar or multidimensional. Correlated multivariate stochasticity
>>>>> sources are
>>>>> supported. Time-varying process parameters (correlations, intensity of
>>>>> Poisson
>>>>> processes, volatilities etc.) are allowed whenever applicable.
>>>>> `process` instances act as valid stochasticity source realizations (as
>>>>> does
>>>>> any callable object complying with a `source` protocol), and may be
>>>>> passed as a source specification when computing the realization of a
>>>>> given
>>>>> process.
>>>>>
>>>>> Computations are fully vectorized across paths, providing an efficient
>>>>> infrastructure for simulating a large number of process realizations.
>>>>> Less so, for large number of time steps: integrating 1000 time steps
>>>>> across 100000 paths takes seconds, one million time steps across
>>>>> 100 paths takes minutes.
>>>>>
>>>>> General infrastructure
>>>>> ======================
>>>>> .. autosummary::
>>>>>    :toctree: generated/
>>>>>
>>>>>    process     -- Array representation of a process (a subclass of
>>>>>                   -- numpy.ndarray).
>>>>>    kfunc       -- Base class for functions with managed keyword
>>>>> arguments.
>>>>>    source      -- Base class for stochasticity sources.
>>>>>    true_source -- Base class for stochasticity sources with memory.
>>>>>    integrator  -- General framework for numerical SDE integration.
>>>>>    montecarlo  -- Summmary statistics of Monte Carlo simulations.
>>>>>
>>>>>
>>>>> Stochasticity sources
>>>>> =====================
>>>>> .. autosummary::
>>>>>    :toctree: generated/
>>>>>
>>>>>    wiener              -- dW, a source of standard Wiener process
>>>>> (Brownian
>>>>>                           -- motion) increments.
>>>>>    symmetric_wiener    -- as above, with symmetric paths (averages
>>>>> exactly 0).
>>>>>    true_wiener         -- dW, a source of standard Wiener process
>>>>> (Brownian
>>>>>                           -- motion) increments, with memory.
>>>>>    poisson             -- dN, a source of Poisson process increments.
>>>>>    compound_poisson    -- dJ, a source of compound Poisson process
>>>>> increments.
>>>>>                           -- (jumps distributed in time as a Poisson
>>>>> process,
>>>>>                           -- and in size as a given `scipy.stats`
>>>>> distribution).
>>>>>    compound_poisson_rv -- Preset jump size distributions for compound
>>>>> Poisson
>>>>>                           -- process increments.
>>>>>
>>>>>
>>>>> Stochastic process realizations
>>>>> ===============================
>>>>> .. autosummary::
>>>>>    :toctree: generated/
>>>>>
>>>>>    const_wiener_process          -- Wiener process (Brownian motion),
>>>>> with
>>>>>                                     -- time-independent parameters.
>>>>>    const_lognorm_process         -- Lognormal process, with
>>>>> time-independent
>>>>>                                     -- parameters.
>>>>>    wiener_process                -- Wiener process (Brownian motion).
>>>>>    lognorm_process               -- Lognormal process.
>>>>>    ornstein_uhlenbeck_process    -- Ornstein-Uhlenbeck process
>>>>> (mean-reverting
>>>>>                                     -- Brownian motion).
>>>>>    hull_white_process            -- F-factors Hull-White process (sum
>>>>> of F
>>>>>                                     -- correlated mean-reverting
>>>>> Brownian
>>>>>                                     -- motions).
>>>>>    hull_white_1factor_process    -- 1-factor Hull-White process (F=1
>>>>> Hull-White
>>>>>                                     -- process with F-index collapsed
>>>>> to a
>>>>>                                     -- scalar).
>>>>>    cox_ingersoll_ross_process    -- Cox-Ingersoll-Ross mean-reverting
>>>>> process.
>>>>>    full_heston_process           -- Heston stochastic volatility
>>>>> process
>>>>>                                     -- (returns both process and
>>>>> volatility).
>>>>>    heston_process                -- Heston stochastic volatility
>>>>> process
>>>>>                                     -- (stores and returns process
>>>>> only).
>>>>>    jump_diffusion_process        -- Jump-diffusion process (lognormal
>>>>> process
>>>>>                                     -- with compound Poisson jumps).
>>>>>    merton_jump_diffusion_process -- Merton jump-diffusion process
>>>>>                                     -- (jump-diffusion process with
>>>>> normal
>>>>>                                     -- jump size distribution).
>>>>>    kou_jump_diffusion_process    -- Kou jump-diffusion process
>>>>> (jump-diffusion
>>>>>                                     -- process with double exponential
>>>>>                                     -- jump size distribution).
>>>>>
>>>>> Shortcuts::
>>>>>
>>>>>    lognorm -- lognorm_process
>>>>>    oruh    -- ornstein_uhlenbeck_process
>>>>>    hwp     -- hull_white_process
>>>>>    hw1f    -- hull_white_1factor_process
>>>>>    cir     -- cox_ingersoll_ross_process
>>>>>    heston  -- heston_process
>>>>>    mjd     -- merton_jump_diffusion_process
>>>>>    kou     -- kou_jump_diffusion_process
>>>>>
>>>>>
>>>>> Analytical results
>>>>> ==================
>>>>> Exact statistics for the realized stochastic processes
>>>>> are listed below, limited to the case of constant process parameters
>>>>> and with
>>>>> some further limitations to the parameters' domains.
>>>>> Function arguments are consistent with those of the corresponding
>>>>> processes.
>>>>> Suffixes `pdf`, `cdf` and `chf` stand respectively for probability
>>>>> distribution
>>>>> function, cumulative probability distribution function, and
>>>>> characteristic
>>>>> function. Black-Scholes formulae for the valuation of call and put
>>>>> options have been
>>>>> included (prefixed with `bs` below).
>>>>>
>>>>> .. autosummary::
>>>>>    :toctree: generated/
>>>>>
>>>>>    wiener_mean
>>>>>    wiener_var
>>>>>    wiener_std
>>>>>    wiener_pdf
>>>>>    wiener_cdf
>>>>>    wiener_chf
>>>>>
>>>>>    lognorm_mean
>>>>>    lognorm_var
>>>>>    lognorm_std
>>>>>    lognorm_pdf
>>>>>    lognorm_cdf
>>>>>    lognorm_log_chf
>>>>>
>>>>>    oruh_mean
>>>>>    oruh_var
>>>>>    oruh_std
>>>>>    oruh_pdf
>>>>>    oruh_cdf
>>>>>
>>>>>    hw2f_mean
>>>>>    hw2f_var
>>>>>    hw2f_std
>>>>>    hw2f_pdf
>>>>>    hw2f_cdf
>>>>>
>>>>>    cir_mean
>>>>>    cir_var
>>>>>    cir_std
>>>>>    cir_pdf
>>>>>
>>>>>    heston_log_mean
>>>>>    heston_log_var
>>>>>    heston_log_std
>>>>>    heston_log_pdf
>>>>>    heston_log_chf
>>>>>
>>>>>    mjd_log_pdf
>>>>>    mjd_log_chf
>>>>>
>>>>>    kou_mean
>>>>>    kou_log_pdf
>>>>>    kou_log_chf
>>>>>
>>>>>    bsd1d2
>>>>>    bscall
>>>>>    bscall_delta
>>>>>    bsput
>>>>>    bsput_delta
>>>>>
>>>>> Notes
>>>>> =====
>>>>> To the benefit of interactive and notebook sessions, and as an aid to
>>>>> fluently
>>>>> freeze or vary the plurality of parameters that define each stochastic
>>>>> process,
>>>>> all sources, process realizations and analytical results are objects
>>>>> with managed keywords (subclasses of `kfunc`): if the corresponding
>>>>> classes
>>>>> are instantiated with both variables (non keyword arguments) and
>>>>> parameters
>>>>> (keyword arguments), they behave as functions returning the
>>>>> computations'
>>>>> result; if called with parameters only, return an instance that stores
>>>>> the set
>>>>> parameters, and exposes the same calling pattern (call with variables
>>>>> and optionally with parameters to get results, call with parameters
>>>>> only
>>>>> to get a new instance storing modified parameters).
>>>>> Parameters that are not specified fall back to the class defaults, if
>>>>> calling
>>>>> the class, or to the instance's stored values, if calling an instance.
>>>>> """
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-Dev mailing list
>>>>> SciPy-Dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>

From rlucente at pipeline.com  Thu Jun 14 19:55:17 2018
From: rlucente at pipeline.com (Robert Lucente)
Date: Thu, 14 Jun 2018 19:55:17 -0400 (GMT-04:00)
Subject: [SciPy-Dev] scipy.sparse.linalg.svds
Message-ID: <141160967.9723.1529020517051@wamui-megara.atl.sa.earthlink.net>

https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.svds.html

states "This is a naive implementation using ARPACK as an eigensolver on A.H * A or A * A.H, depending on which one is more efficient."

One of my co-workers is struggling to find an implementation for SVD for a sparse matrix. I was thinking of suggesting that he look at SciPy. The phrase "naive implementation" has made me think twice about making the suggestion. I just don't want the first intro of Python/SciPy into the organization to be a negative one.

I poked around and found the following 2 articles

http://fa.bianp.net/blog/2012/singular-value-decomposition-in-scipy/

https://simplyml.com/benchmarking-the-singular-value-decomposition/

I realize that I should be suspicious of bench marks and that you have to run your own.

I would like to provide my co-worker the link to SciPy and then to 2 articles which can help guide him in creating his own bench mark. Any suggestions would be appreciated.

Please note that I am not an expert in SVD.

From maurizio.cipollina at gmail.com  Fri Jun 15 17:45:02 2018
From: maurizio.cipollina at gmail.com (Maurizio Cipollina)
Date: Fri, 15 Jun 2018 23:45:02 +0200
Subject: [SciPy-Dev] Stochastic calculus package
In-Reply-To: <CAMMTP+CjQnbgL+oCm1OnOudoO5JMBnORQhUS2u-PDhaY2=w2fg@mail.gmail.com>
References: <CAKfLqW08MOO+zQOMpxWO4jrxTHN-6BMDU_Qb=d0SUYp0iNi2Eg@mail.gmail.com>
 <CABL7CQg8_GeR1F6qeu65HJhx0xCN1FXav1avk4Zrp_YacDv21w@mail.gmail.com>
 <CAKfLqW2YDCg=+WSbAhzDNK9_Hz-2RC1UokZxtYXz2A-pq6TG4g@mail.gmail.com>
 <CAMMTP+BevuDHcDa5xw2Lj3niaZMYo_x=WYRUaW-Grj8eRK9xJw@mail.gmail.com>
 <CAKfLqW0rBsznR8kou4t15YG3qQn4D8VMN+PbN=QJ6SYBC33GOQ@mail.gmail.com>
 <CAMMTP+CjQnbgL+oCm1OnOudoO5JMBnORQhUS2u-PDhaY2=w2fg@mail.gmail.com>
Message-ID: <CAKfLqW0UDKxcN6SwYFwYWwfGz0S=__hdrgMayY8s7Z9O_FD1oA@mail.gmail.com>

Thanks Josef, point very well taken, the look is definitely old fashioned -
back from the good old times when a string wouldn?t be a String, and a list
wouldn?t be a List...

Maurizio

On Fri, 15 Jun 2018 at 20:39, <josef.pktd at gmail.com> wrote:

> Sounds good,
>
> One thing that immediately jumps into my eyes are the lower case class
> names.
> I think using pep-8 capital/camel case would make it look more modern.
> Only old fashioned packages like numpy and scipy still have the legacy
> spelling (when developers thought python has to look like matlab :)
>
> Josef
>
>
>
> On Fri, Jun 15, 2018 at 1:51 PM, Maurizio Cipollina
> <maurizio.cipollina at gmail.com> wrote:
> > Hi everybody,
> >
> > following up on this conversation of some time ago, a pre-release of the
> > package, renamed SdePy, is now available on pip
> > (https://pypi.org/project/sdepy  ) and as a project on github
> > (https://github.com/sdepy/sdepy ), version 1.0.0rc3. A short quickstart
> > guide (https://sdepy.readthedocs.io/en/latest/intro.html#id2 ) is the
> > easiest path to see what it does and what it does not. Any feedback,
> > comments and suggestions are very welcome!
> >
> > Since last September, the package has evolved quite a bit and had its
> fair
> > share of refactoring, going through several 0.x unpublished versions
> before
> > stabilizing in its current form (marked as beta). Here are a few
> takeaways I
> > got from this thread, which I acted upon:
> >
> > -  Integrating user defined SDEs turned out to be a main point of
> interest:
> > this ?odeint? like functionality has been considerably streamlined and
> > simplified - formerly, preset processes were mainly in focus, and
> getting a
> > new process up and running took a deep dive in a not-so-welcoming API,
> now
> > upgraded as well (the former ?integrator? has been superseded by a
> > ?paths_generator? parent class, and cooperating ?integrator? and ?SDE?
> > classes).
> >
> > -  The ?kfunc? beast was admittedly a hard sell: all undue dependencies
> of
> > the core modules on its machinery have been eliminated, it plays now the
> > role of an optional decorator for the shortcuts of frequently used stuff
> > (it?s fully functional though and, in my experience, quite handy).
> >
> > -  The question ?Do you REALLY need a ?process? class? clearly stood out
> in
> > your comments: after some experimentation, the main reason for answering
> in
> > the yes turned out, in my view, to be its seamless interoperability with
> the
> > rest of the package, hardly achievable without a dedicated design. Now
> > process instances can indeed be fed to integrators, both as
> time-dependent
> > and/or path-dependent SDE parameters, and as stochasticity sources: this
> > opened countless hacking opportunities when doing computations, and
> spoiling
> > them seemed a pity.
> >
> > In case any of you is willing to take a look, feel free to get in touch
> for
> > any problems/bugs/questions you might have
> >
> > Cheers
> > Maurizio
> >
> > On Mon, 11 Sep 2017 at 19:39, <josef.pktd at gmail.com> wrote:
> >>
> >>
> >>
> >> On Mon, Sep 11, 2017 at 12:35 PM, Maurizio Cipollina
> >> <maurizio.cipollina at gmail.com> wrote:
> >>>
> >>> Hi Ralf,
> >>>
> >>>
> >>> thanks for going through this, here are some follow up comments on your
> >>> points:
> >>>
> >>>
> >>>
> >>> On the two packages you found:
> >>>
> >>> Sdeint: provides several SDE integration algorithms but no specific
> >>> processes; does not cover processes with jumps; offers no explicit
> >>> provisions for handling solutions in a number of paths, which is what
> you
> >>> need since the integration output is random variables, not numbers.
> >>> SDE-higham is not a package or tool, but rather a collection of recipes
> >>> for a few textbook examples (no jumps either), in the form that an
> >>> interactive session might take (set globals, do calculations, plot and
> print
> >>> results).
> >>>
> >>>
> >>>
> >>> About your questions/concerns:
> >>>
> >>> ndarray subclassing: the integrator does not depend on the process
> class,
> >>> and process realizations may be easily tweaked to return an ndarray
> instead
> >>> of a process instance; however, a main point of the package is not
> just to
> >>> produce SDE solutions, but to allow the user to handle them (almost)
> >>> effortlessly once obtained: so one might possibly hide the process
> class
> >>> inside the testing suite, which depends on it, but hardly get rid of it
> >>> altogether. Indeed, I subclassed ndarray as safely as I could figure
> out,
> >>> but I will surely go through the scipy-dev archives and find out what
> >>> advised you against doing so.
> >>> odeint and scipy.integrate: my view is that odeint churns out numbers
> >>> dependant on numbers, while SDE integration churns out random variables
> >>> dependant on random variables (the stochasticity sources), so there is
> a
> >>> shift of paradigm and the two schemes do not fit smoothly one within
> the
> >>> other. The step by step integration of SDEs makes sense, and gets
> useful, if
> >>> you can easily control the inputs (eg. correlations among stochasticity
> >>> sources fed into different SDEs), generate the output in a decent
> number of
> >>> paths, and easily extract statistics of expressions dependent on it
> (hence
> >>> the flow ?get a number of paths packaged as a process instance? -> ?do
> numpy
> >>> algebra with it? -> ?estimate probability distributions etc. using the
> >>> process class methods? -> ?scale it up with the montecarlo class in
> case the
> >>> needed paths do not fit into memory?).
> >>> Montecarlo simulations is a big topic indeed, that goes far beyond this
> >>> package; what is called the montecarlo class is in fact a tool to
> cumulate
> >>> statistical information extracted from a number of realizations of a
> random
> >>> variable, and is focused on making easy to interpret the output of SDE
> >>> integration.
> >>> Black-Scholes assists one more tests in a rather extensive testing
> suite,
> >>> and might be painlessly dropped.
> >>>
> >>>
> >>>
> >>> This said, I see your general feeling is that this whole subject is too
> >>> specialized to fit into scipy at this stage. I suggested otherwise, in
> my
> >>> proposal, because stochastic calculus is regarded as one of the
> pillars of
> >>> modern probability theory, and covering it into scipy might be
> relevant to
> >>> lots of mathematicians and physicists, both practitioners and
> students, that
> >>> might otherwise hesitate to adopt, and build dependencies upon, a
> standalone
> >>> package. In case you reconsider let me know, I?d be happy to help (now
> or in
> >>> the future?).
> >>>
> >>>
> >>>
> >>> If you have further questions, let me know as well.
> >>>
> >>> Cheers
> >>>
> >>> Maurizio
> >>
> >>
> >>
> >> To partially follow up, similar to Ralph's view.
> >>
> >> I think it would be good to have this as a separate package, or at least
> >> the code on github.
> >> With distributions like conda it is now much easier to install
> additional
> >> packages, once they are established enough and are included in
> distributions
> >> or easily pip installable.
> >>
> >> The main reason that I think scipy is currently not the appropriate
> >> package for it is that the design and review would require a lot of
> work. I
> >> guess that there are not many maintainers and reviewers that are
> familiar
> >> with this area. I also know only of applications in finance and it is
> not
> >> clear whether or which parts would be of more general interest.
> >>
> >> In my opinion, some core tools for SDE and jump diffusions would fit in
> >> scipy. But the application support will not or might not fit well in a
> >> general purpose scientific numerical library. This would be similar to
> other
> >> areas where scipy provides the core computational tools, but
> applications
> >> are supported by other packages.
> >>
> >> For example, the process class might be similar in magnitude to scipy
> >> stats distributions or signal statespace classes, but I have no idea
> what
> >> design for it would fit in scipy.
> >> Some of the time aggregation properties sound similar to the
> corresponding
> >> pandas functions, and it is not clear whether users would want to use a
> >> separate class or just use pandas instead.
> >>
> >> Similarly, the kfunc class sounds like something that doesn't have a
> >> similar pattern in scipy.
> >>
> >> Design changes can be made more easily in a standalone package, while
> all
> >> refactorings of classes in scipy are difficult issues because of the
> >> backwards compatibility policy which requires that the design should be
> >> largely settled before a merge.
> >>
> >> Josef
> >>
> >> https://groups.google.com/d/msg/pystatsmodels/xsttiEiyqAg/maR6n4EeAgAJ
> >> (stackoverflow question has been deleted "Is there a library out there
> to
> >> calibrate an Ornstein-Uhlenbeck model?")
> >>
> >>
> >>
> https://github.com/statsmodels/statsmodels/blob/master/statsmodels/sandbox/tsa/diffusion.py
> >> and diffusion2.py
> >>
> >>
> >>
> >>>
> >>>
> >>>
> >>>
> >>> On 9 September 2017 at 03:31, Ralf Gommers <ralf.gommers at gmail.com>
> >>> wrote:
> >>>>
> >>>> Hi Maurizio,
> >>>>
> >>>>
> >>>> On Fri, Sep 8, 2017 at 7:55 PM, Maurizio Cipollina
> >>>> <maurizio.cipollina at gmail.com> wrote:
> >>>>>
> >>>>> Hi everybody, and first of all thanks for the great job you have been
> >>>>> doing, on which I relied *A LOT* over time.
> >>>>>
> >>>>> Stochastic calculus and Stochastic Differential Equations are a
> >>>>> significant branch of mathematics that to my understanding is
> >>>>> underrepresented in scipy, as well as in the open source python
> stack at
> >>>>> large. I may well be missing something, but as I can see one can
> find out
> >>>>> there many recipies or collections of recipies, and a number of
> >>>>> packages/projects skewed towards finance, but no structured
> framework for
> >>>>> generating and handling stochastic processes and numerically solving
> SDEs.
> >>>>
> >>>>
> >>>> I've never used these, but a quick search turns up two packages:
> >>>>   https://pypi.python.org/pypi/sdeint
> >>>>   https://github.com/alu042/SDE-higham
> >>>> Not very mature probably, but would be good to compare your code with
> >>>> those.
> >>>>
> >>>>>
> >>>>> This is a pity, since scipy/numpy provide efficen implementations of
> >>>>> all the basic ingredients needed to do the trick. Out of
> professional need,
> >>>>> and prompted by this gap, I have developed a general purpose package
> of
> >>>>> which I own the copyright, and which I would be happy to release
> under the
> >>>>> BSD license as part of scipy, in case there is a consensus among the
> scipy
> >>>>> community that it makes sense to do so (the package has no
> dependencies
> >>>>> beyond standard library, numpy and scipy, and might probably fit
> best as a
> >>>>> subpackage of scipy.stats).
> >>>>>
> >>>>> Before setting up a PR, I would be happy to hear your thoughts: I
> have
> >>>>> pasted below the package docstring, that should give an overview of
> its
> >>>>> scope and limitations. Please note that some of the realized
> processes stray
> >>>>> into mathematical finance territory, somehow inevitably since
> finance is one
> >>>>> major field of application of stochastic calculus, but the focus is
> the
> >>>>> latter, not the former, at least in my intentions.
> >>>>
> >>>>
> >>>> First thought: this looks like a useful addition to the scientific
> >>>> Python ecosystem, but is probably too specialized for SciPy (at least
> at
> >>>> this stage).
> >>>>
> >>>> Based on your docstring below, here are some questions/concerns:
> >>>> 1. We definitely don't want a new ndarray subclass (there's a pretty
> >>>> strong consensus at this point that subclassing ndarray is usually
> not a
> >>>> good solution), so your `process` doesn't sound attractive.
> >>>> 2. The `integrator` class sounds a lot like odeint, so maybe this
> would
> >>>> fit better with scipy.integrate?
> >>>> 3. Monte-Carlo simulation is a big and fairly technical topic. There
> are
> >>>> whole packages (e.g. PyMC3, emcee) dedicated to this. It may not fit
> well
> >>>> with SciPy.
> >>>> 4. Things specific to finance like Black-Scholes and put/call options
> >>>> are not a good fit.
> >>>> 5. Maintainers for this added code. The ODE integrators in
> >>>> scipy.integrate already suffer from lack of a dedicated maintainer,
> SDEs are
> >>>> more specialized so are likely to not be very well-maintained by
> other devs
> >>>> than you.
> >>>>
> >>>> Without seeing your code it's hard to say for sure, but it looks to me
> >>>> like it would be better to release your code as a standalone package.
> >>>>
> >>>> Cheers,
> >>>> Ralf
> >>>>
> >>>>
> >>>>>
> >>>>> Thanks in advance for taking your time to go through this, and for
> your
> >>>>> comments and suggestions.
> >>>>> Maurizio
> >>>>>
> >>>>> ???
> >>>>> ===========================================================
> >>>>> Stochastic - Monte Carlo simulation of stochastic processes
> >>>>> ===========================================================
> >>>>>
> >>>>> This package provides:
> >>>>>
> >>>>> 1.  A `process` class, a subclass of `numpy.ndarray` representing
> >>>>>     a sequence of values in time, realized in one or several paths.
> >>>>>     Algebraic manipulations and ufunc computations are supported for
> >>>>>     instances that share the same timeline and comply with numpy
> >>>>>     broadcasting rules. Interpolation along the timeline is supported
> >>>>>     via callability of `process` instances. Process-specific
> >>>>> functionalities,
> >>>>>     such as averaging and indexing along time or across paths, are
> >>>>> delegated
> >>>>>     to process-specific methods, attributes and properties (no
> >>>>> overriding
> >>>>>     of `numpy.ndarray` operations).
> >>>>>
> >>>>> 2.  The `source` class and its subclasses, stochasticity sources
> >>>>> providing
> >>>>>     numerical realizations of the differentials commonly found
> >>>>>     in stochastic differential equations (SDEs), with or without
> >>>>>     memory of formerly invoked realizations.
> >>>>>
> >>>>> 3.  The `integrator` class, a general framework for numerical SDE
> >>>>>     integration, intended for subclassing, to assist the definition
> of
> >>>>> step by
> >>>>>     step integration algorithms and the computation of numerical
> >>>>> realizations of
> >>>>>     stochastic processes.
> >>>>>
> >>>>> 4.  Several objects providing realizations of specific stochastic
> >>>>> processes
> >>>>>     along a given timeline and across a requested number of paths.
> >>>>>     Realizations are obtained via explicit formulae, in case the
> >>>>> relevant SDE
> >>>>>     has a closed form solution, or otherwise as `integrator`
> subclasses
> >>>>> performing
> >>>>>     Euler-Maruyama numerical integration.
> >>>>>
> >>>>> 5.  A `montecarlo` class, as an aid to cumulate the results of
> several
> >>>>>     Monte Carlo simulations of a given stochastic variable, and to
> >>>>> extract
> >>>>>     summary estimates for its probability distribution function and
> >>>>>     statistics, thus supporting simulations across a number of paths
> >>>>> that exceeds
> >>>>>     the maximum allowed by available memory.
> >>>>>
> >>>>> 6.  Several analytical results relating to the supported stochastic
> >>>>>     processes, as a general reference and for testing purpouses.
> >>>>>
> >>>>> For all sources and realizations, process values can take any shape,
> >>>>> scalar or multidimensional. Correlated multivariate stochasticity
> >>>>> sources are
> >>>>> supported. Time-varying process parameters (correlations, intensity
> of
> >>>>> Poisson
> >>>>> processes, volatilities etc.) are allowed whenever applicable.
> >>>>> `process` instances act as valid stochasticity source realizations
> (as
> >>>>> does
> >>>>> any callable object complying with a `source` protocol), and may be
> >>>>> passed as a source specification when computing the realization of a
> >>>>> given
> >>>>> process.
> >>>>>
> >>>>> Computations are fully vectorized across paths, providing an
> efficient
> >>>>> infrastructure for simulating a large number of process realizations.
> >>>>> Less so, for large number of time steps: integrating 1000 time steps
> >>>>> across 100000 paths takes seconds, one million time steps across
> >>>>> 100 paths takes minutes.
> >>>>>
> >>>>> General infrastructure
> >>>>> ======================
> >>>>> .. autosummary::
> >>>>>    :toctree: generated/
> >>>>>
> >>>>>    process     -- Array representation of a process (a subclass of
> >>>>>                   -- numpy.ndarray).
> >>>>>    kfunc       -- Base class for functions with managed keyword
> >>>>> arguments.
> >>>>>    source      -- Base class for stochasticity sources.
> >>>>>    true_source -- Base class for stochasticity sources with memory.
> >>>>>    integrator  -- General framework for numerical SDE integration.
> >>>>>    montecarlo  -- Summmary statistics of Monte Carlo simulations.
> >>>>>
> >>>>>
> >>>>> Stochasticity sources
> >>>>> =====================
> >>>>> .. autosummary::
> >>>>>    :toctree: generated/
> >>>>>
> >>>>>    wiener              -- dW, a source of standard Wiener process
> >>>>> (Brownian
> >>>>>                           -- motion) increments.
> >>>>>    symmetric_wiener    -- as above, with symmetric paths (averages
> >>>>> exactly 0).
> >>>>>    true_wiener         -- dW, a source of standard Wiener process
> >>>>> (Brownian
> >>>>>                           -- motion) increments, with memory.
> >>>>>    poisson             -- dN, a source of Poisson process increments.
> >>>>>    compound_poisson    -- dJ, a source of compound Poisson process
> >>>>> increments.
> >>>>>                           -- (jumps distributed in time as a Poisson
> >>>>> process,
> >>>>>                           -- and in size as a given `scipy.stats`
> >>>>> distribution).
> >>>>>    compound_poisson_rv -- Preset jump size distributions for compound
> >>>>> Poisson
> >>>>>                           -- process increments.
> >>>>>
> >>>>>
> >>>>> Stochastic process realizations
> >>>>> ===============================
> >>>>> .. autosummary::
> >>>>>    :toctree: generated/
> >>>>>
> >>>>>    const_wiener_process          -- Wiener process (Brownian motion),
> >>>>> with
> >>>>>                                     -- time-independent parameters.
> >>>>>    const_lognorm_process         -- Lognormal process, with
> >>>>> time-independent
> >>>>>                                     -- parameters.
> >>>>>    wiener_process                -- Wiener process (Brownian motion).
> >>>>>    lognorm_process               -- Lognormal process.
> >>>>>    ornstein_uhlenbeck_process    -- Ornstein-Uhlenbeck process
> >>>>> (mean-reverting
> >>>>>                                     -- Brownian motion).
> >>>>>    hull_white_process            -- F-factors Hull-White process (sum
> >>>>> of F
> >>>>>                                     -- correlated mean-reverting
> >>>>> Brownian
> >>>>>                                     -- motions).
> >>>>>    hull_white_1factor_process    -- 1-factor Hull-White process (F=1
> >>>>> Hull-White
> >>>>>                                     -- process with F-index collapsed
> >>>>> to a
> >>>>>                                     -- scalar).
> >>>>>    cox_ingersoll_ross_process    -- Cox-Ingersoll-Ross mean-reverting
> >>>>> process.
> >>>>>    full_heston_process           -- Heston stochastic volatility
> >>>>> process
> >>>>>                                     -- (returns both process and
> >>>>> volatility).
> >>>>>    heston_process                -- Heston stochastic volatility
> >>>>> process
> >>>>>                                     -- (stores and returns process
> >>>>> only).
> >>>>>    jump_diffusion_process        -- Jump-diffusion process (lognormal
> >>>>> process
> >>>>>                                     -- with compound Poisson jumps).
> >>>>>    merton_jump_diffusion_process -- Merton jump-diffusion process
> >>>>>                                     -- (jump-diffusion process with
> >>>>> normal
> >>>>>                                     -- jump size distribution).
> >>>>>    kou_jump_diffusion_process    -- Kou jump-diffusion process
> >>>>> (jump-diffusion
> >>>>>                                     -- process with double
> exponential
> >>>>>                                     -- jump size distribution).
> >>>>>
> >>>>> Shortcuts::
> >>>>>
> >>>>>    lognorm -- lognorm_process
> >>>>>    oruh    -- ornstein_uhlenbeck_process
> >>>>>    hwp     -- hull_white_process
> >>>>>    hw1f    -- hull_white_1factor_process
> >>>>>    cir     -- cox_ingersoll_ross_process
> >>>>>    heston  -- heston_process
> >>>>>    mjd     -- merton_jump_diffusion_process
> >>>>>    kou     -- kou_jump_diffusion_process
> >>>>>
> >>>>>
> >>>>> Analytical results
> >>>>> ==================
> >>>>> Exact statistics for the realized stochastic processes
> >>>>> are listed below, limited to the case of constant process parameters
> >>>>> and with
> >>>>> some further limitations to the parameters' domains.
> >>>>> Function arguments are consistent with those of the corresponding
> >>>>> processes.
> >>>>> Suffixes `pdf`, `cdf` and `chf` stand respectively for probability
> >>>>> distribution
> >>>>> function, cumulative probability distribution function, and
> >>>>> characteristic
> >>>>> function. Black-Scholes formulae for the valuation of call and put
> >>>>> options have been
> >>>>> included (prefixed with `bs` below).
> >>>>>
> >>>>> .. autosummary::
> >>>>>    :toctree: generated/
> >>>>>
> >>>>>    wiener_mean
> >>>>>    wiener_var
> >>>>>    wiener_std
> >>>>>    wiener_pdf
> >>>>>    wiener_cdf
> >>>>>    wiener_chf
> >>>>>
> >>>>>    lognorm_mean
> >>>>>    lognorm_var
> >>>>>    lognorm_std
> >>>>>    lognorm_pdf
> >>>>>    lognorm_cdf
> >>>>>    lognorm_log_chf
> >>>>>
> >>>>>    oruh_mean
> >>>>>    oruh_var
> >>>>>    oruh_std
> >>>>>    oruh_pdf
> >>>>>    oruh_cdf
> >>>>>
> >>>>>    hw2f_mean
> >>>>>    hw2f_var
> >>>>>    hw2f_std
> >>>>>    hw2f_pdf
> >>>>>    hw2f_cdf
> >>>>>
> >>>>>    cir_mean
> >>>>>    cir_var
> >>>>>    cir_std
> >>>>>    cir_pdf
> >>>>>
> >>>>>    heston_log_mean
> >>>>>    heston_log_var
> >>>>>    heston_log_std
> >>>>>    heston_log_pdf
> >>>>>    heston_log_chf
> >>>>>
> >>>>>    mjd_log_pdf
> >>>>>    mjd_log_chf
> >>>>>
> >>>>>    kou_mean
> >>>>>    kou_log_pdf
> >>>>>    kou_log_chf
> >>>>>
> >>>>>    bsd1d2
> >>>>>    bscall
> >>>>>    bscall_delta
> >>>>>    bsput
> >>>>>    bsput_delta
> >>>>>
> >>>>> Notes
> >>>>> =====
> >>>>> To the benefit of interactive and notebook sessions, and as an aid to
> >>>>> fluently
> >>>>> freeze or vary the plurality of parameters that define each
> stochastic
> >>>>> process,
> >>>>> all sources, process realizations and analytical results are objects
> >>>>> with managed keywords (subclasses of `kfunc`): if the corresponding
> >>>>> classes
> >>>>> are instantiated with both variables (non keyword arguments) and
> >>>>> parameters
> >>>>> (keyword arguments), they behave as functions returning the
> >>>>> computations'
> >>>>> result; if called with parameters only, return an instance that
> stores
> >>>>> the set
> >>>>> parameters, and exposes the same calling pattern (call with variables
> >>>>> and optionally with parameters to get results, call with parameters
> >>>>> only
> >>>>> to get a new instance storing modified parameters).
> >>>>> Parameters that are not specified fall back to the class defaults, if
> >>>>> calling
> >>>>> the class, or to the instance's stored values, if calling an
> instance.
> >>>>> """
> >>>>>
> >>>>> _______________________________________________
> >>>>> SciPy-Dev mailing list
> >>>>> SciPy-Dev at python.org
> >>>>> https://mail.python.org/mailman/listinfo/scipy-dev
> >>>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> SciPy-Dev mailing list
> >>>> SciPy-Dev at python.org
> >>>> https://mail.python.org/mailman/listinfo/scipy-dev
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> SciPy-Dev mailing list
> >>> SciPy-Dev at python.org
> >>> https://mail.python.org/mailman/listinfo/scipy-dev
> >>>
> >>
> >> _______________________________________________
> >> SciPy-Dev mailing list
> >> SciPy-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/scipy-dev
> >
> >
> > _______________________________________________
> > SciPy-Dev mailing list
> > SciPy-Dev at python.org
> > https://mail.python.org/mailman/listinfo/scipy-dev
> >
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180615/5c02026f/attachment-0001.html>

From ilhanpolat at gmail.com  Sat Jun 16 05:50:07 2018
From: ilhanpolat at gmail.com (Ilhan Polat)
Date: Sat, 16 Jun 2018 11:50:07 +0200
Subject: [SciPy-Dev] scipy.sparse.linalg.svds
In-Reply-To: <141160967.9723.1529020517051@wamui-megara.atl.sa.earthlink.net>
References: <141160967.9723.1529020517051@wamui-megara.atl.sa.earthlink.net>
Message-ID: <CAEBuzr-43tjdSQ9JXT1f2_2evteD3_F1BCgg-obYtA9MMeoBJw@mail.gmail.com>

Jake Vanderplas also followed up on Fabian's post around those times. For
reference, https://jakevdp.github.io/blog/2012/12/19/sparse-svds-in-python/

My guess is that PROPACK's license was ambiguous for a long time while
sparse library was being implemented and getting mature. There are more
experienced devs that can comment on that.

But maybe during the numpy.matrix revamp (
https://github.com/scipy/scipy/issues/8162) we can consider involving
PROPACK alternatives.


On Fri, Jun 15, 2018 at 1:55 AM, Robert Lucente <rlucente at pipeline.com>
wrote:

> https://docs.scipy.org/doc/scipy/reference/generated/
> scipy.sparse.linalg.svds.html
>
> states "This is a naive implementation using ARPACK as an eigensolver on
> A.H * A or A * A.H, depending on which one is more efficient."
>
> One of my co-workers is struggling to find an implementation for SVD for a
> sparse matrix. I was thinking of suggesting that he look at SciPy. The
> phrase "naive implementation" has made me think twice about making the
> suggestion. I just don't want the first intro of Python/SciPy into the
> organization to be a negative one.
>
> I poked around and found the following 2 articles
>
> http://fa.bianp.net/blog/2012/singular-value-decomposition-in-scipy/
>
> https://simplyml.com/benchmarking-the-singular-value-decomposition/
>
> I realize that I should be suspicious of bench marks and that you have to
> run your own.
>
> I would like to provide my co-worker the link to SciPy and then to 2
> articles which can help guide him in creating his own bench mark. Any
> suggestions would be appreciated.
>
> Please note that I am not an expert in SVD.
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180616/6c3c8125/attachment.html>

From charlesr.harris at gmail.com  Mon Jun 18 16:20:10 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 18 Jun 2018 14:20:10 -0600
Subject: [SciPy-Dev] rackspace ssl certificates
Message-ID: <CAB6mnxKMZBL4fxvrfM8x8EiSfGKdaKR_=Xzs1sU3Y8Gu4gG1Kg@mail.gmail.com>

Hi All,

I've been trying to put out the NumPy 1.15.0rc1, but cannot get
`numpy-wheels` to upload the wheels to rackspace on windows, there is a
certification problem. I note that that requirement was supposedly disabled:

 on_success:
  # Upload the generated wheel package to Rackspace
  # On Windows, Apache Libcloud cannot find a standard CA cert bundle so we
  # disable the ssl checks.

and nothing relevant seems to have changed in our `.appveyor.yml` since the
last successful run 7 days ago, 6 if we count 1.14.5, so I'm thinking a
policy has changed at either at rackspace or appveyor, but that is just a
guess. I'm experimenting with various changes to the script and the
`apache-libcloud` version to see if I can get success, but thought I'd ask
if anyone knew anything that might be helpful.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180618/f67816af/attachment.html>

From charlesr.harris at gmail.com  Mon Jun 18 16:42:31 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 18 Jun 2018 14:42:31 -0600
Subject: [SciPy-Dev] [Numpy-discussion] rackspace ssl certificates
In-Reply-To: <CAJXewOn9E1JvrLG8v2q0Vhp-AXKc_KKmKk8QQQeHH1TdHkiAhA@mail.gmail.com>
References: <CAB6mnxKMZBL4fxvrfM8x8EiSfGKdaKR_=Xzs1sU3Y8Gu4gG1Kg@mail.gmail.com>
 <CAJXewOn9E1JvrLG8v2q0Vhp-AXKc_KKmKk8QQQeHH1TdHkiAhA@mail.gmail.com>
Message-ID: <CAB6mnx+uBHNTMXtj-bHb9LodVqgtjzP0_b_4HUf=qoAkbEjaTw@mail.gmail.com>

On Mon, Jun 18, 2018 at 2:22 PM, Nathan Goldbaum <nathan12343 at gmail.com>
wrote:

> I think Matthew Brett needs to fix this.
>

That would be nice, but I'm not convinced it is helpful :) I note that
latest `apache-libcloud` does not install directly on windows, there seem
to be some missing dependencies.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180618/153954fc/attachment.html>

From cimrman3 at ntc.zcu.cz  Tue Jun 19 07:52:31 2018
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Tue, 19 Jun 2018 13:52:31 +0200
Subject: [SciPy-Dev] ANN: SfePy 2018.2
Message-ID: <31b88a31-b853-4a00-415a-55f935472ab3@ntc.zcu.cz>

I am pleased to announce release 2018.2 of SfePy.

Description
-----------

SfePy (simple finite elements in Python) is a software for solving systems of
coupled partial differential equations by the finite element method or by the
isogeometric analysis (limited support). It is distributed under the new BSD
license.

Home page: http://sfepy.org
Mailing list: https://mail.python.org/mm3/mailman3/lists/sfepy.python.org/
Git (source) repository, issue tracker: https://github.com/sfepy/sfepy

Highlights of this release
--------------------------

- generalized-alpha and velocity Verlet elastodynamics solvers
- terms for dispersion in fluids
- caching of reference coordinates for faster repeated use of probes
- new wrapper of MUMPS linear solver for parallel runs

For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1
(rather long and technical).

Cheers,
Robert Cimrman

---

Contributors to this release in alphabetical order:

Robert Cimrman
Lubos Kejzlar
Vladimir Lukes
Matyas Novak

From adibhar97 at gmail.com  Tue Jun 19 20:11:23 2018
From: adibhar97 at gmail.com (Aditya Bharti)
Date: Wed, 20 Jun 2018 05:41:23 +0530
Subject: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3
 dimensions
In-Reply-To: <CANRzPHXqGtT6d0VkDDYqqz9NQbKXV36T1jk9vCCf_oEZhqb5JA@mail.gmail.com>
References: <163213d9b99.e31ddfbf140246.3867213885941953730@zoho.com>
 <CANRzPHVN47mDJqc4DuUq70X9sAiRqBRWOSHszgtL3QyM1O6ihA@mail.gmail.com>
 <CANRzPHV0BswFBevgZ2fsf2vKm_J_LQEET=qtDZ1WO6H7dMDNyQ@mail.gmail.com>
 <CAB2ViTbimDdS0jZ3Qp71rorkhfALBhHQtTztv1OD11dY7V1ADg@mail.gmail.com>
 <CANRzPHUynj4LSmpUeBMzY-my1EmGKUp944mdy_tyqOLGY5Fmrw@mail.gmail.com>
 <CAB2ViTYKO285BtwFTjOR4brjQJE_3SJ0d4jSkWeaUp-DJc8ing@mail.gmail.com>
 <CANRzPHXqGtT6d0VkDDYqqz9NQbKXV36T1jk9vCCf_oEZhqb5JA@mail.gmail.com>
Message-ID: <CANRzPHW3DQ9ocL26jT_d=MVd0ymUi9uih-vjaiy=+ONCKLbCHg@mail.gmail.com>

Hi all,
So the last two weeks saw two major additions to the `Rotation` class:
rotation compositions and applications. The addition of a few helpful
functions such as indexing and inversions means that the current rotation
branch here
<https://github.com/adbugger/scipy/tree/rotation/scipy/spatial/transform>
is ready to be merged and will hopefully be available for everybody to use
in the near future.

This represents a major milestone as the first part of the project, the
main `Rotation` class, is now complete.

As always, the code can be found here
<https://github.com/adbugger/scipy/tree/rotation> and I invite you to check
out my blog here <https://blogs.python-gsoc.org/aditya-bharti>.

Cheers,
Aditya

On Tue, 5 Jun 2018 at 02:20, Aditya Bharti <adibhar97 at gmail.com> wrote:

> We do plan to create a tutorial with examples highlighting the major
> functionality of the module and some common use cases, but its exact nature
> and extent will depend upon how much time we have left after completing the
> implementation (including docstrings and tests).
>
> Best,
> Aditya
>
> On Mon, 4 Jun 2018 at 09:45, Phillip Feldman <phillip.m.feldman at gmail.com>
> wrote:
>
>> "Refer to docs for more details."  Will there be user documentation
>> beyond the doc strings in the various methods?
>>
>> On Tue, May 29, 2018 at 2:21 PM, Aditya Bharti <adibhar97 at gmail.com>
>> wrote:
>>
>>> Hi all,
>>> Continuing the work so far, the following have been implemented this
>>> week:
>>>
>>>    - `from_rotvec`, `as_rotvec`: Representing Euler angles as rotation
>>>    vectors, with appropriate Taylor series expansions for small angles
>>>    - `from_euler`: Initialization from Euler angles, along with a
>>>    string based axis sequence specification. Refer to docs for more details.
>>>
>>> As always, the project lives here
>>> <https://github.com/adbugger/scipy/tree/rotation/scipy/spatial/transform>,
>>> and my personal experiences can be found on the blog
>>> <https://blogs.python-gsoc.org/aditya-bharti>.
>>>
>>> Thanks,
>>> Aditya
>>>
>>> On Sun, 20 May 2018 at 13:28, Phillip Feldman <
>>> phillip.m.feldman at gmail.com> wrote:
>>>
>>>> When you say "discrete cosine matrix", I think that you mean "direction
>>>> cosine matrix" (see
>>>> https://en.wikiversity.org/wiki/PlanetPhysics/Direction_Cosine_Matrix).
>>>>
>>>> Phillip
>>>>
>>>> On Sat, May 19, 2018 at 12:23 PM, Aditya Bharti <adibhar97 at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>> So this concludes week 1 of GSoC 2018. I must say it was a great
>>>>> learning experience and I invite you all to check out my account of this
>>>>> week on the blog <https://blogs.python-gsoc.org/aditya-bharti/>. This
>>>>> email is more of a technical update.
>>>>>
>>>>>    - So, the main `Rotation` class will live under a new sub module
>>>>>    `scipy.spatial.transform`.
>>>>>    - Conversion between quaternions and discrete cosine matrices was
>>>>>    implemented.
>>>>>    - The rotation class now supports `from_quaternion`, `from_dcm`,
>>>>>    `as_quaternion` and `as_dcm`, with support for multiple rotations in one
>>>>>    call.
>>>>>
>>>>> The project currently lives in my own fork of scipy here
>>>>> <https://github.com/adbugger/scipy/tree/rotation/scipy/spatial/transform>.
>>>>> Stay tuned for more updates!
>>>>>
>>>>> Best,
>>>>> Aditya
>>>>>
>>>>> On Wed, 2 May 2018 at 21:03, Aditya Bharti <adibhar97 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Nikolay,
>>>>>>
>>>>>> I've used Wordpress only once before, so I don't know much about it.
>>>>>> From my limited experience, it is extremely customizable. You can customize
>>>>>> every thing from the look and feel to SEO characteristics. There are
>>>>>> apparently a lot of wordpress plugins for these kind of tasks. For this
>>>>>> particular blog, PSF had already setup an account for me with a site on it.
>>>>>> All I had to do was click on the 'New' button and open up the new post
>>>>>> page. There's a field for a header and body text, with options for adding
>>>>>> audio, video and hyperlinks.
>>>>>>
>>>>>> As regards to the post itself, sure I'll expand it with a brief
>>>>>> overview, motivation and an example. Note that the example will only show
>>>>>> sample usage, not any internals. I plan to borrow heavily from my proposal
>>>>>> for this purpose, I hope that's ok.
>>>>>>
>>>>>> Regards,
>>>>>> Aditya
>>>>>>
>>>>>> On 2 May 2018 at 19:54, Nikolay Mayorov <nikolay.mayorov at zoho.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi, Aditya!
>>>>>>>
>>>>>>> Glad that you set up the blog and good job on setting up the
>>>>>>> documentation build as well.
>>>>>>>
>>>>>>> Curious, what is this blogging platform like? How do you create
>>>>>>> posts in it?
>>>>>>>
>>>>>>> As for your first post: while not strictly necessary I think it
>>>>>>> would be nice to see a more thorough introductory post with a brief
>>>>>>> overview, motivation and/or an example. Do you want to work on it?
>>>>>>>
>>>>>>> Best,
>>>>>>> Nikolay
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SciPy-Dev mailing list
>>>>>>> SciPy-Dev at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> SciPy-Dev mailing list
>>>>> SciPy-Dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>>
>>>>>
>>>> _______________________________________________
>>>> SciPy-Dev mailing list
>>>> SciPy-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>>
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180620/ae1c3d4d/attachment.html>

From charlesr.harris at gmail.com  Thu Jun 21 13:34:08 2018
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 21 Jun 2018 11:34:08 -0600
Subject: [SciPy-Dev] NumPy 1.15.0rc1 released
Message-ID: <CAB6mnxJdmmoXD6uu5CD+2t+w_iYwzmk-0zZOacNwC3D+xGdq-Q@mail.gmail.com>

Hi All,

On behalf of the NumPy team I'm pleased to announce the release of NumPy
1.15.0rc1.
This release has an unusual number of cleanups, many deprecations of old
functions,
and improvements to many existing functions. A total of 423 pull reguests
were merged
for this release, please look at the release notes
<https://github.com/numpy/numpy/releases/tag/v1.15.0rc1>for details. Some
highlights are:

   - NumPy has switched to pytest for testing.
   - A new  `numpy.printoptions` context manager.
   - Many improvements to the histogram functions.
   - Support for unicode field names in python 2.7.
   - Improved support for PyPy.

The Python versions supported by this release are 2.7, 3.4-3.6.  The wheels
are linked with
OpenBLAS 3.0, which should fix some of the linalg problems reported for
NumPy 1.14,
and the source archives were created using Cython 0.28.2 and should work
with the upcoming
Python 3.7.

Wheels for this release can be downloaded from PyPI
<https://pypi.org/project/numpy/#description>, source archives are available
from Github <https://github.com/numpy/numpy/releases/tag/v1.15.0rc1>.

A total of 128 people contributed to this release.  People with a "+" by
their
names contributed a patch for the first time.

   - Aaron Critchley +
   - Aarthi +
   - Aarthi Agurusa +
   - Alex Thomas +
   - Alexander Belopolsky
   - Allan Haldane
   - Anas Khan +
   - Andras Deak
   - Andrey Portnoy +
   - Anna Chiara
   - Aurelien Jarno +
   - Baurzhan Muftakhidinov
   - Berend Kapelle +
   - Bernhard M. Wiedemann
   - Bjoern Thiel +
   - Bob Eldering
   - Cenny Wenner +
   - Charles Harris
   - ChloeColeongco +
   - Chris Billington +
   - Christopher +
   - Chun-Wei Yuan +
   - Claudio Freire +
   - Daniel Smith
   - Darcy Meyer +
   - David Abdurachmanov +
   - David Freese
   - Deepak Kumar Gouda +
   - Dennis Weyland +
   - Derrick Williams +
   - Dmitriy Shalyga +
   - Eric Cousineau +
   - Eric Larson
   - Eric Wieser
   - Evgeni Burovski
   - Frederick Lefebvre +
   - Gaspar Karm +
   - Geoffrey Irving
   - Gerhard Hobler +
   - Gerrit Holl
   - Guo Ci +
   - Hameer Abbasi +
   - Han Shen
   - Hiroyuki V. Yamazaki +
   - Hong Xu
   - Ihor Melnyk +
   - Jaime Fernandez
   - Jake VanderPlas +
   - James Tocknell +
   - Jarrod Millman
   - Jeff VanOss +
   - John Kirkham
   - Jonas Rauber +
   - Jonathan March +
   - Joseph Fox-Rabinovitz
   - Julian Taylor
   - Junjie Bai +
   - Juris Bogusevs +
   - J?rg D?pfert
   - Kenichi Maehashi +
   - Kevin Sheppard
   - Kimikazu Kato +
   - Kirit Thadaka +
   - Kritika Jalan +
   - Lakshay Garg +
   - Lars G +
   - Licht Takeuchi
   - Louis Potok +
   - Luke Zoltan Kelley
   - MSeifert04 +
   - Mads R. B. Kristensen +
   - Malcolm Smith +
   - Mark Harfouche +
   - Marten H. van Kerkwijk +
   - Marten van Kerkwijk
   - Matheus Vieira Portela +
   - Mathieu Lamarre
   - Mathieu Sornay +
   - Matthew Brett
   - Matthew Rocklin +
   - Matthias Bussonnier
   - Matti Picus
   - Michael Droettboom
   - Miguel S?nchez de Le?n Peque +
   - Mike Toews +
   - Milo +
   - Nathaniel J. Smith
   - Nelle Varoquaux
   - Nicholas Nadeau +
   - Nick Minkyu Lee +
   - Nikita +
   - Nikita Kartashov +
   - Nils Becker +
   - Oleg Zabluda
   - Orestis Floros +
   - Pat Gunn +
   - Paul van Mulbregt +
   - Pauli Virtanen
   - Pierre Chanial +
   - Ralf Gommers
   - Raunak Shah +
   - Robert Kern
   - Russell Keith-Magee +
   - Ryan Soklaski +
   - Samuel Jackson +
   - Sebastian Berg
   - Siavash Eliasi +
   - Simon Conseil
   - Simon Gibbons
   - Stefan Krah +
   - Stefan van der Walt
   - Stephan Hoyer
   - Subhendu +
   - Subhendu Ranjan Mishra +
   - Tai-Lin Wu +
   - Tobias Fischer +
   - Toshiki Kataoka +
   - Tyler Reddy +
   - Varun Nayyar
   - Victor Rodriguez +
   - Warren Weckesser
   - Zane Bradley +
   - fo40225
   - lumbric +
   - luzpaz +
   - mamrehn +
   - tynn +
   - xoviat

Cheers

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180621/49a9bd61/attachment.html>

From cournape at gmail.com  Sun Jun 24 10:03:18 2018
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 24 Jun 2018 23:03:18 +0900
Subject: [SciPy-Dev] scipy.sparse.linalg.svds
In-Reply-To: <CAEBuzr-43tjdSQ9JXT1f2_2evteD3_F1BCgg-obYtA9MMeoBJw@mail.gmail.com>
References: <141160967.9723.1529020517051@wamui-megara.atl.sa.earthlink.net>
 <CAEBuzr-43tjdSQ9JXT1f2_2evteD3_F1BCgg-obYtA9MMeoBJw@mail.gmail.com>
Message-ID: <CAGY4rcVndPkW3yghD475vWWHZu4es30_bK1FaFja7_uQRXqUNA@mail.gmail.com>

IIRC, I wrote that implementation at that time. Indeed, PROPACk was at that
time not usable because of licensing.

Instead, I implemented sparse SVD by solving the eigenvalues of the
(implicit) A^H * A. It is naive because of the loss of precision (because
of the squaring).

On Sat, Jun 16, 2018 at 6:50 PM, Ilhan Polat <ilhanpolat at gmail.com> wrote:

> Jake Vanderplas also followed up on Fabian's post around those times. For
> reference, https://jakevdp.github.io/blog/2012/12/19/sparse-svds-
> in-python/
>
> My guess is that PROPACK's license was ambiguous for a long time while
> sparse library was being implemented and getting mature. There are more
> experienced devs that can comment on that.
>
> But maybe during the numpy.matrix revamp (https://github.com/scipy/
> scipy/issues/8162) we can consider involving PROPACK alternatives.
>
>
> On Fri, Jun 15, 2018 at 1:55 AM, Robert Lucente <rlucente at pipeline.com>
> wrote:
>
>> https://docs.scipy.org/doc/scipy/reference/generated/scipy.
>> sparse.linalg.svds.html
>>
>> states "This is a naive implementation using ARPACK as an eigensolver on
>> A.H * A or A * A.H, depending on which one is more efficient."
>>
>> One of my co-workers is struggling to find an implementation for SVD for
>> a sparse matrix. I was thinking of suggesting that he look at SciPy. The
>> phrase "naive implementation" has made me think twice about making the
>> suggestion. I just don't want the first intro of Python/SciPy into the
>> organization to be a negative one.
>>
>> I poked around and found the following 2 articles
>>
>> http://fa.bianp.net/blog/2012/singular-value-decomposition-in-scipy/
>>
>> https://simplyml.com/benchmarking-the-singular-value-decomposition/
>>
>> I realize that I should be suspicious of bench marks and that you have to
>> run your own.
>>
>> I would like to provide my co-worker the link to SciPy and then to 2
>> articles which can help guide him in creating his own bench mark. Any
>> suggestions would be appreciated.
>>
>> Please note that I am not an expert in SVD.
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180624/53cab6e4/attachment.html>

From mikofski at berkeley.edu  Mon Jun 25 10:30:25 2018
From: mikofski at berkeley.edu (Mark Alexander Mikofski)
Date: Mon, 25 Jun 2018 07:30:25 -0700
Subject: [SciPy-Dev] Return residual and derivative in gradient descent
 methods like Newton
Message-ID: <CAEqRcW14b2mDHM8cn_Qk60niSyEWLxtwYSSKVcJn54XUBxKoYQ@mail.gmail.com>

Hi,

Would anyone disagree or would anyone be interested in a proposal to allow
the derivative to be returned from the user supplied function as an
optional second argument in gradient search method like Newton? EG

>>> lambda x,a: (x**3-a, 3*x*"2)
>>> newton(f, x0, fprime='f2', args=(a,))

Some simple tests show that this may have a 2X speed in cases where the
derivative expression requires the value of the original function call.

See this issue in which I proposed this idea and wore a sample test script
https://github.com/scipy/scipy/issues/8354

I proposed a way to keep the existing API and add this new feature.
* If `fprime` is a callable, then same as before
* If it's a string like "f2" the get fprime from second output

Thanks,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180625/9f57736f/attachment.html>

From josef.pktd at gmail.com  Mon Jun 25 10:57:21 2018
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 25 Jun 2018 10:57:21 -0400
Subject: [SciPy-Dev] Return residual and derivative in gradient descent
 methods like Newton
In-Reply-To: <CAEqRcW14b2mDHM8cn_Qk60niSyEWLxtwYSSKVcJn54XUBxKoYQ@mail.gmail.com>
References: <CAEqRcW14b2mDHM8cn_Qk60niSyEWLxtwYSSKVcJn54XUBxKoYQ@mail.gmail.com>
Message-ID: <CAMMTP+A-odpetBBhGZQ9K54hOYmapjH4ReW1Xn=p7zqz3_eEVQ@mail.gmail.com>

On Mon, Jun 25, 2018 at 10:30 AM, Mark Alexander Mikofski
<mikofski at berkeley.edu> wrote:
> Hi,
>
> Would anyone disagree or would anyone be interested in a proposal to allow
> the derivative to be returned from the user supplied function as an optional
> second argument in gradient search method like Newton? EG
>
>>>> lambda x,a: (x**3-a, 3*x*"2)
>>>> newton(f, x0, fprime='f2', args=(a,))
>
> Some simple tests show that this may have a 2X speed in cases where the
> derivative expression requires the value of the original function call.
>
> See this issue in which I proposed this idea and wore a sample test script
> https://github.com/scipy/scipy/issues/8354
>
> I proposed a way to keep the existing API and add this new feature.
> * If `fprime` is a callable, then same as before
> * If it's a string like "f2" the get fprime from second output

I think using something like a `full_output` option as in the fmin_xxx
function would be more explicit and more flexible.
e.g. the second optional return could be a dict or similar.

Josef


>
> Thanks,
> Mark
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>

From mikofski at berkeley.edu  Mon Jun 25 12:47:08 2018
From: mikofski at berkeley.edu (Mark Alexander Mikofski)
Date: Mon, 25 Jun 2018 09:47:08 -0700
Subject: [SciPy-Dev] Return residual and derivative in gradient descent
 methods like Newton
In-Reply-To: <CAMMTP+A-odpetBBhGZQ9K54hOYmapjH4ReW1Xn=p7zqz3_eEVQ@mail.gmail.com>
References: <CAEqRcW14b2mDHM8cn_Qk60niSyEWLxtwYSSKVcJn54XUBxKoYQ@mail.gmail.com>
 <CAMMTP+A-odpetBBhGZQ9K54hOYmapjH4ReW1Xn=p7zqz3_eEVQ@mail.gmail.com>
Message-ID: <CAEqRcW1Pgj46fm3F8YdDbueL27prxDyvBtdjRF21LnS+EGi9Rw@mail.gmail.com>

Hi Josef,

Sorry, I didn't phrase my question very well.

I meant from the callback function, not the output from the solver.

For example, say we want to solve:

y = exp(Ax+y)

dy/dx = A*y

My proposal is when supplying arguments to the solver instead of writing
two callbacks we just supply one:

>>> def f(x, y, A):
...     # callback returns both fval and jac
...     F = y - exp(Ax+y)  # residual
...     J = -A*y  # jacobian
...     return F, J

>>> newton(f, x0, fprime='anystring', args=(y0, A))

'anything' indicated to the solver that `fprime` is returned by the
callback `f`. So inside the solver instead of calling them separately it
just makes one call:

...     try:
...         J = fprime(x, *args)
...     except Exception:
...         F, J = f(x, *args)
...     else:
...         F = f(x, *args)

This is a common API in MATLAB, and my tests show that for some functions
with this can reduce the number of calls and improve the speed of the
solver.

Maybe I could get the same improvements by precalculating some things and
passing them in `args`?

On Mon, Jun 25, 2018, 7:58 AM <josef.pktd at gmail.com> wrote:

> On Mon, Jun 25, 2018 at 10:30 AM, Mark Alexander Mikofski
> <mikofski at berkeley.edu> wrote:
> > Hi,
> >
> > Would anyone disagree or would anyone be interested in a proposal to
> allow
> > the derivative to be returned from the user supplied function as an
> optional
> > second argument in gradient search method like Newton? EG
> >
> >>>> lambda x,a: (x**3-a, 3*x*"2)
> >>>> newton(f, x0, fprime='f2', args=(a,))
> >
> > Some simple tests show that this may have a 2X speed in cases where the
> > derivative expression requires the value of the original function call.
> >
> > See this issue in which I proposed this idea and wore a sample test
> script
> > https://github.com/scipy/scipy/issues/8354
> >
> > I proposed a way to keep the existing API and add this new feature.
> > * If `fprime` is a callable, then same as before
> > * If it's a string like "f2" the get fprime from second output
>
> I think using something like a `full_output` option as in the fmin_xxx
> function would be more explicit and more flexible.
> e.g. the second optional return could be a dict or similar.
>
> Josef
>
>
> >
> > Thanks,
> > Mark
> >
> > _______________________________________________
> > SciPy-Dev mailing list
> > SciPy-Dev at python.org
> > https://mail.python.org/mailman/listinfo/scipy-dev
> >
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180625/2b0f4f46/attachment.html>

From andyfaff at gmail.com  Mon Jun 25 18:12:20 2018
From: andyfaff at gmail.com (Andrew Nelson)
Date: Tue, 26 Jun 2018 08:12:20 +1000
Subject: [SciPy-Dev] Return residual and derivative in gradient descent
 methods like Newton
In-Reply-To: <CAEqRcW1Pgj46fm3F8YdDbueL27prxDyvBtdjRF21LnS+EGi9Rw@mail.gmail.com>
References: <CAEqRcW14b2mDHM8cn_Qk60niSyEWLxtwYSSKVcJn54XUBxKoYQ@mail.gmail.com>
 <CAMMTP+A-odpetBBhGZQ9K54hOYmapjH4ReW1Xn=p7zqz3_eEVQ@mail.gmail.com>
 <CAEqRcW1Pgj46fm3F8YdDbueL27prxDyvBtdjRF21LnS+EGi9Rw@mail.gmail.com>
Message-ID: <CAAbtOZeg4rC4yx2seqxB9Rgone8Ugeh=8DfRuqubOWx8B=Bh6A@mail.gmail.com>

>
> On Mon, Jun 25, 2018 at 10:30 AM, Mark Alexander Mikofski
>> <mikofski at berkeley.edu> wrote:
>> >
>> > Would anyone disagree or would anyone be interested in a proposal to
>> allow
>> > the derivative to be returned from the user supplied function as an
>> optional
>> > second argument in gradient search method like Newton? EG
>> >
>> >>>> lambda x,a: (x**3-a, 3*x*"2)
>> >>>> newton(f, x0, fprime='f2', args=(a,))
>> >
>> > Some simple tests show that this may have a 2X speed in cases where the
>> > derivative expression requires the value of the original function call.
>>
>
The `minimize` function does something like this for the `jac` keyword: "If
*jac* is a Boolean and is True, *fun* is assumed to return the gradient
along with the objective function.".
If something like this is desirable, then I suggest that the same minimize
pattern is used.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180626/b8f4e14b/attachment.html>

From jesse.livezey at gmail.com  Mon Jun 25 19:27:31 2018
From: jesse.livezey at gmail.com (Jesse Livezey)
Date: Mon, 25 Jun 2018 16:27:31 -0700
Subject: [SciPy-Dev] Interest in improvements to the cKDTree codebase?
In-Reply-To: <CABL7CQgwUUfphHM-65mdjgUMBQRabUOznOGPbwgVecnbUU1OeA@mail.gmail.com>
References: <CAH1y-JWppmXpuGoE05w5akV-hObjQ9gR+hs3iKg6Cjd4KkQf+Q@mail.gmail.com>
 <CABL7CQgwUUfphHM-65mdjgUMBQRabUOznOGPbwgVecnbUU1OeA@mail.gmail.com>
Message-ID: <CAH1y-JWMMQNBKJCopXpixSbjk1-bmT_4XMKDFgXj++QVsfXjxQ@mail.gmail.com>

Thanks for the feedback. PR for 'sorted' keyword is here
https://github.com/scipy/scipy/pull/8908

With no sorting, count_ball_point() vs. len(query_ball_point()) is
~3x faster for 1d problems,
~25% faster for 3d problems, and
~10% faster for 7d problems.

import scipy.spatial as ss
>
> x = np.random.randn(10000, 1)
> xtree = ss.cKDTree(x)
>
> %timeit xtree.count_ball_point(x, .02, p=np.inf)
> # 21.8 ms ? 190 ?s per loop (mean ? std. dev. of 7 runs, 10 loops each)
>
> %timeit [len(q) for q in xtree.query_ball_point(x, .02, p=np.inf)]
> # 66.9 ms ? 967 ?s per loop (mean ? std. dev. of 7 runs, 10 loops each)
>
>
> x = np.random.randn(10000, 3)
> xtree = ss.cKDTree(x)
>
> %timeit xtree.count_ball_point(x, .2, p=np.inf)
> # 42.3 ms ? 1.4 ms per loop (mean ? std. dev. of 7 runs, 10 loops each)
>
> %timeit [len(q) for q in xtree.query_ball_point(x, .2, p=np.inf)]
> # 53.1 ms ? 407 ?s per loop (mean ? std. dev. of 7 runs, 10 loops each)
>
>
> x = np.random.randn(10000, 7)
> xtree = ss.cKDTree(x)
>
> %timeit xtree.count_ball_point(x, 1., p=np.inf)
> # 537 ms ? 857 ?s per loop (mean ? std. dev. of 7 runs, 1 loop each)
>
> %timeit [len(q) for q in xtree.query_ball_point(x, 1., p=np.inf)]
> # 587 ms ? 6.17 ms per loop (mean ? std. dev. of 7 runs, 1 loop each)


--
Jesse Livezey
he/him/his

On Sat, Jun 9, 2018 at 1:12 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:

> Hi Jesse,
>
>
> On Tue, May 22, 2018 at 10:56 AM, Jesse Livezey <jesse.livezey at gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I'm using cKDTrees for a project and noticed two potential ways to
>> improve the code and have written one additional count method that might be
>> useful to others.
>>
>> I have written code and some tests for all three items and can contribute
>> if there is interest.
>>
>> 1) Allowing an array of rs in cKDTree.query_ball_point(). I started a PR
>> here <https://github.com/scipy/scipy/pull/8818>. In principle, this
>> should speed up multiple queries with different rs, but see 2.
>>
>> 2) I noticed that for the cases when cKDTree.query_ball_point() returns
>> large numbers of points (>100), it is faster to loop over queries in
>> python. This is true both with and without my PR. This is largely because
>> single point queries do not sort the return indices, but multi-point
>> queries DO sort them (see details here
>> <https://github.com/scipy/scipy/issues/8838>). Removing the sorted()
>> leads to considerable speedups, but is not backwards compatible. However,
>> the sorted behavior is not in the method description and is not even
>> internally consistent, so maybe it could just be removed or made optional?
>>
>
> I agree with Sturla's reply on the issue; adding a keyword that defaults
> to the current behavior and allows turning off sorting is probably the way
> to go.
>
>
>> 3) I have written a cKDTree.count_ball_point() method that behaves like
>> query_ball_point() but just returns the number of points rather than their
>> indices. This is much faster than calling len(query_ball_point()).
>>
>
> How much faster is it after adding the keyword for (2)?
>
> Cheers,
> Ralf
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180625/56eb0f2f/attachment-0001.html>

From p.vanmulbregt at comcast.net  Tue Jun 26 04:06:20 2018
From: p.vanmulbregt at comcast.net (Paul van Mulbregt)
Date: Tue, 26 Jun 2018 01:06:20 -0700
Subject: [SciPy-Dev] Return residual and derivative in gradient descent
 methods like Newton
In-Reply-To: <CAAbtOZeg4rC4yx2seqxB9Rgone8Ugeh=8DfRuqubOWx8B=Bh6A@mail.gmail.com>
References: <CAEqRcW14b2mDHM8cn_Qk60niSyEWLxtwYSSKVcJn54XUBxKoYQ@mail.gmail.com>
 <CAMMTP+A-odpetBBhGZQ9K54hOYmapjH4ReW1Xn=p7zqz3_eEVQ@mail.gmail.com>
 <CAEqRcW1Pgj46fm3F8YdDbueL27prxDyvBtdjRF21LnS+EGi9Rw@mail.gmail.com>
 <CAAbtOZeg4rC4yx2seqxB9Rgone8Ugeh=8DfRuqubOWx8B=Bh6A@mail.gmail.com>
Message-ID: <B890C103-4D67-4040-8215-84FBE8F2FD97@comcast.net>

It can indeed be desirable to return f and f? (and f??) in a single call, especially if the additional computation for "f?-while-computing-f" is very small.  Not all of the derivative-based methods  would necessarily benefit (e.g Homeier?s method which evaluates two f' and one f every iteration), but the base Newton?s method wouldn?t see any additional computation. 
There is an existing PR https://github.com/scipy/scipy/pull/8890 <https://github.com/scipy/scipy/pull/8890> to add
 `optimize.root_scalar(f, args=(), method=None, bracket=None, fprime=None, fprime2=None,x0=None, x1=None,xtol=None, rtol=None, maxiter=None,options=None)` 
This would be a 1-dim analog to match the multi-dimensional `optimize.root()`.  The `fprime` argument is either a callable or a Boolean.  If it is `True`, then calling `f(x, *args)` is expected to return both f and f?. Similarly if `fprime2` is True,  then calling `f(x, *args)` is expected to return f, f? and  f?'.

-Paul

> On Jun 25, 2018, at 3:12 PM, Andrew Nelson <andyfaff at gmail.com> wrote:
> 
> On Mon, Jun 25, 2018 at 10:30 AM, Mark Alexander Mikofski
> <mikofski at berkeley.edu <mailto:mikofski at berkeley.edu>> wrote:
> >
> > Would anyone disagree or would anyone be interested in a proposal to allow
> > the derivative to be returned from the user supplied function as an optional
> > second argument in gradient search method like Newton? EG
> >
> >>>> lambda x,a: (x**3-a, 3*x*"2)
> >>>> newton(f, x0, fprime='f2', args=(a,))
> >
> > Some simple tests show that this may have a 2X speed in cases where the
> > derivative expression requires the value of the original function call.
> 
> The `minimize` function does something like this for the `jac` keyword: "If jac is a Boolean and is True, fun is assumed to return the gradient along with the objective function.".
> If something like this is desirable, then I suggest that the same minimize pattern is used.
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180626/6b5e529d/attachment.html>