From sebastian at sipsolutions.net  Sat Jun  1 05:18:24 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Sat, 01 Jun 2013 11:18:24 +0200
Subject: [Numpy-discussion] Effect of deprecating non-integers for
 integer function arguments
In-Reply-To: <CAMMTP+CqBYnXEiiOYQgufD-L9mdqJexEWmgOxfH7=JCENh-07A@mail.gmail.com>
References: <1370023693.2584.25.camel@sebastian-laptop>
	<CAMMTP+CqBYnXEiiOYQgufD-L9mdqJexEWmgOxfH7=JCENh-07A@mail.gmail.com>
Message-ID: <1370078304.2584.32.camel@sebastian-laptop>

On Fri, 2013-05-31 at 16:32 -0400, josef.pktd at gmail.com wrote:
> On Fri, May 31, 2013 at 2:08 PM, Sebastian Berg
> <sebastian at sipsolutions.net> wrote:
> > Hi,
> >
> > the current numpy master has deprecated non-integers for the use of
> > indexing (not-fancy yet). However I think this should be moved further
> > down in the numpy machinery which means that the conversion utils
> > provided by numpy would generally raise warnings for non-integers.
> >

<snip>

> 
> I haven't checked for statsmodels, but I would consider it a bug on
> our side if we have implicit conversion to integers. We might still
> have some left in older code but changed already most of them.
> I just wrote code like this (upp are quantiles in (0, 1))
> 

Yeah, this is just a heads up that this will probably happen in 1.8 (and
you could test already if you like). I assume most projects will prefer
to have a warning, but I guess that most larger ones are likely to have
such oversights in the code, which will need to be fixed. Even if only
because of the truedivision change in python 3.

- Sebastian

> m_upp = [x[:int(np.round(upp_ * nobs))].mean(0) for upp_ in upp]
> 
> I don't think we use non-integers for reshape.
> 
> Josef
> 
> >
> > The branch implementing this can be found in the PR
> > https://github.com/numpy/numpy/pull/3243
> >
> > Regards,
> >
> > Sebastian
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From warren.weckesser at gmail.com  Sat Jun  1 18:50:12 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 1 Jun 2013 18:50:12 -0400
Subject: [Numpy-discussion] numpy tests errors and failures
Message-ID: <CAGzF1ufUO4JTXr8d7GQGWgGJfCpqb0b2c4svjFMU3-amZqxY5Q@mail.gmail.com>

I'm getting a failure and two errors with the latest master branch:

$ python -c "import numpy; numpy.test('full')"
Running unit tests for numpy
NumPy version 1.8.0.dev-dff8c94
NumPy is installed in
/home/warren/local_numpy/lib/python2.7/site-packages/numpy
Python version 2.7.4 |Anaconda 1.5.0 (64-bit)| (default, Apr 21 2013,
17:43:08) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)]
nose version 1.3.0
.....................F...................................................................................................................................................................................................................................................................................................................S...............................................................................................................................S..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K........................................E.E.................................................................................................................................................SSS...............................................................................................................................................................................................................................................................................................................................................K........................................................................................................K......................K...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SS..SS.....................................................................................................SSSSSSSS................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................
======================================================================
ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose([1e-08, 1,
1000020.0000000099], [0, nan, 1000000.0])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
line 197, in runTest
    self.test(*self.arg)
  File
"/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
line 1253, in tst_isclose_allclose
    assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
  File
"/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py",
line 2008, in allclose
    return all(less_equal(abs(x-y), atol + rtol * abs(y)))
RuntimeWarning: invalid value encountered in absolute

======================================================================
ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose(nan, [nan, nan,
nan])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
line 197, in runTest
    self.test(*self.arg)
  File
"/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
line 1253, in tst_isclose_allclose
    assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
  File
"/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py",
line 2008, in allclose
    return all(less_equal(abs(x-y), atol + rtol * abs(y)))
RuntimeWarning: invalid value encountered in absolute

======================================================================
FAIL: Test numpy dot with different order C, F
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
line 197, in runTest
    self.test(*self.arg)
  File
"/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.py",
line 114, in test_dot_array_order
    assert_almost_equal(a.dot(a), a.T.dot(a.T).T, decimal=30)
  File
"/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
line 458, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File
"/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
line 819, in assert_array_almost_equal
    header=('Arrays are not almost equal to %d decimals' % decimal))
  File
"/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
line 652, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 30 decimals

(mismatch 26.0%)
 x: array([[ 0.60970873,  1.6909554 , -1.08851945, -1.82058012, -3.95746613,
         1.52435613, -0.59853059, -3.72278619,  3.82375932,  5.51367044],
       [-3.58154893, -2.06231236, -0.0656727 ,  1.4737345 ,  2.6068747 ,...
 y: array([[ 0.60970873,  1.6909554 , -1.08851945, -1.82058001, -3.95746613,
         1.52435613, -0.59853059, -3.72278643,  3.82375932,  5.51367044],
       [-3.58154917, -2.06231236, -0.0656727 ,  1.4737345 ,  2.6068747 ,...

----------------------------------------------------------------------
Ran 5151 tests in 60.506s

FAILED (KNOWNFAIL=5, SKIP=17, errors=2, failures=1)



Warren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130601/384c0c5b/attachment.html>

From charlesr.harris at gmail.com  Sat Jun  1 19:47:25 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 1 Jun 2013 17:47:25 -0600
Subject: [Numpy-discussion] numpy tests errors and failures
In-Reply-To: <CAGzF1ufUO4JTXr8d7GQGWgGJfCpqb0b2c4svjFMU3-amZqxY5Q@mail.gmail.com>
References: <CAGzF1ufUO4JTXr8d7GQGWgGJfCpqb0b2c4svjFMU3-amZqxY5Q@mail.gmail.com>
Message-ID: <CAB6mnx+9fYmL6-_ke9MJ2W5yF89etMWebC2H7VmQ7PkVF-N7nw@mail.gmail.com>

On Sat, Jun 1, 2013 at 4:50 PM, Warren Weckesser <warren.weckesser at gmail.com
> wrote:

> I'm getting a failure and two errors with the latest master branch:
>
> $ python -c "import numpy; numpy.test('full')"
> Running unit tests for numpy
> NumPy version 1.8.0.dev-dff8c94
> NumPy is installed in
> /home/warren/local_numpy/lib/python2.7/site-packages/numpy
> Python version 2.7.4 |Anaconda 1.5.0 (64-bit)| (default, Apr 21 2013,
> 17:43:08) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)]
> nose version 1.3.0
>
> .....................F...................................................................................................................................................................................................................................................................................................................S...............................................................................................................................S..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K........................................E.E.................................................................................................................................................SSS...............................................................................................................................................................................................................................................................................................................................................K........................................................................................................K......................K...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SS..SS.....................................................................................................SSSSSSSS................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................
> ======================================================================
> ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose([1e-08, 1,
> 1000020.0000000099], [0, nan, 1000000.0])
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
> line 197, in runTest
>     self.test(*self.arg)
>   File
> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
> line 1253, in tst_isclose_allclose
>     assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
>   File
> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py",
> line 2008, in allclose
>     return all(less_equal(abs(x-y), atol + rtol * abs(y)))
> RuntimeWarning: invalid value encountered in absolute
>
> ======================================================================
> ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose(nan, [nan, nan,
> nan])
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
> line 197, in runTest
>     self.test(*self.arg)
>   File
> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
> line 1253, in tst_isclose_allclose
>     assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
>   File
> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py",
> line 2008, in allclose
>     return all(less_equal(abs(x-y), atol + rtol * abs(y)))
> RuntimeWarning: invalid value encountered in absolute
>
> ======================================================================
> FAIL: Test numpy dot with different order C, F
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
> line 197, in runTest
>     self.test(*self.arg)
>   File
> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.py",
> line 114, in test_dot_array_order
>     assert_almost_equal(a.dot(a), a.T.dot(a.T).T, decimal=30)
>   File
> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
> line 458, in assert_almost_equal
>     return assert_array_almost_equal(actual, desired, decimal, err_msg)
>   File
> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
> line 819, in assert_array_almost_equal
>     header=('Arrays are not almost equal to %d decimals' % decimal))
>   File
> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
> line 652, in assert_array_compare
>     raise AssertionError(msg)
> AssertionError:
> Arrays are not almost equal to 30 decimals
>

Thirty decimals? What architecture?


>
> (mismatch 26.0%)
>  x: array([[ 0.60970873,  1.6909554 , -1.08851945, -1.82058012,
> -3.95746613,
>          1.52435613, -0.59853059, -3.72278619,  3.82375932,  5.51367044],
>        [-3.58154893, -2.06231236, -0.0656727 ,  1.4737345 ,  2.6068747 ,...
>  y: array([[ 0.60970873,  1.6909554 , -1.08851945, -1.82058001,
> -3.95746613,
>          1.52435613, -0.59853059, -3.72278643,  3.82375932,  5.51367044],
>        [-3.58154917, -2.06231236, -0.0656727 ,  1.4737345 ,  2.6068747 ,...
>
> ----------------------------------------------------------------------
> Ran 5151 tests in 60.506s
>
> FAILED (KNOWNFAIL=5, SKIP=17, errors=2, failures=1)
>
>
>
>
I don't see any of these with

Running unit tests for numpy
NumPy version 1.8.0.dev-dff8c94
NumPy is installed in /home/charris/.local/lib/python2.7/site-packages/numpy
Python version 2.7.3 (default, Aug  9 2012, 17:23:57) [GCC 4.7.1 20120720
(Red Hat 4.7.1-5)]
nose version 1.3.0

I'll guess it's the ancient version of gcc.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130601/4169e38b/attachment.html>

From warren.weckesser at gmail.com  Sat Jun  1 20:09:00 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 1 Jun 2013 20:09:00 -0400
Subject: [Numpy-discussion] numpy tests errors and failures
In-Reply-To: <CAB6mnx+9fYmL6-_ke9MJ2W5yF89etMWebC2H7VmQ7PkVF-N7nw@mail.gmail.com>
References: <CAGzF1ufUO4JTXr8d7GQGWgGJfCpqb0b2c4svjFMU3-amZqxY5Q@mail.gmail.com>
	<CAB6mnx+9fYmL6-_ke9MJ2W5yF89etMWebC2H7VmQ7PkVF-N7nw@mail.gmail.com>
Message-ID: <CAGzF1ueX9+aZ=Xs_oer0ta6TaFOpegNvh31EibONA9aEG8R7dA@mail.gmail.com>

On Sat, Jun 1, 2013 at 7:47 PM, Charles R Harris
<charlesr.harris at gmail.com>wrote:

>
>
> On Sat, Jun 1, 2013 at 4:50 PM, Warren Weckesser <
> warren.weckesser at gmail.com> wrote:
>
>> I'm getting a failure and two errors with the latest master branch:
>>
>> $ python -c "import numpy; numpy.test('full')"
>> Running unit tests for numpy
>> NumPy version 1.8.0.dev-dff8c94
>> NumPy is installed in
>> /home/warren/local_numpy/lib/python2.7/site-packages/numpy
>> Python version 2.7.4 |Anaconda 1.5.0 (64-bit)| (default, Apr 21 2013,
>> 17:43:08) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)]
>> nose version 1.3.0
>>
>> .....................F...................................................................................................................................................................................................................................................................................................................S...............................................................................................................................S..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K........................................E.E.................................................................................................................................................SSS...............................................................................................................................................................................................................................................................................................................................................K........................................................................................................K......................K...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SS..SS.....................................................................................................SSSSSSSS................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................
>> ======================================================================
>> ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose([1e-08, 1,
>> 1000020.0000000099], [0, nan, 1000000.0])
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
>> line 197, in runTest
>>     self.test(*self.arg)
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
>> line 1253, in tst_isclose_allclose
>>     assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py",
>> line 2008, in allclose
>>     return all(less_equal(abs(x-y), atol + rtol * abs(y)))
>> RuntimeWarning: invalid value encountered in absolute
>>
>> ======================================================================
>> ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose(nan, [nan, nan,
>> nan])
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
>> line 197, in runTest
>>     self.test(*self.arg)
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
>> line 1253, in tst_isclose_allclose
>>     assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py",
>> line 2008, in allclose
>>     return all(less_equal(abs(x-y), atol + rtol * abs(y)))
>> RuntimeWarning: invalid value encountered in absolute
>>
>> ======================================================================
>> FAIL: Test numpy dot with different order C, F
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
>> line 197, in runTest
>>     self.test(*self.arg)
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.py",
>> line 114, in test_dot_array_order
>>     assert_almost_equal(a.dot(a), a.T.dot(a.T).T, decimal=30)
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
>> line 458, in assert_almost_equal
>>     return assert_array_almost_equal(actual, desired, decimal, err_msg)
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
>> line 819, in assert_array_almost_equal
>>     header=('Arrays are not almost equal to %d decimals' % decimal))
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
>> line 652, in assert_array_compare
>>     raise AssertionError(msg)
>> AssertionError:
>> Arrays are not almost equal to 30 decimals
>>
>
> Thirty decimals? What architecture?
>
>
>>
>> (mismatch 26.0%)
>>  x: array([[ 0.60970873,  1.6909554 , -1.08851945, -1.82058012,
>> -3.95746613,
>>          1.52435613, -0.59853059, -3.72278619,  3.82375932,  5.51367044],
>>        [-3.58154893, -2.06231236, -0.0656727 ,  1.4737345 ,  2.6068747
>> ,...
>>  y: array([[ 0.60970873,  1.6909554 , -1.08851945, -1.82058001,
>> -3.95746613,
>>          1.52435613, -0.59853059, -3.72278643,  3.82375932,  5.51367044],
>>        [-3.58154917, -2.06231236, -0.0656727 ,  1.4737345 ,  2.6068747
>> ,...
>>
>> ----------------------------------------------------------------------
>> Ran 5151 tests in 60.506s
>>
>> FAILED (KNOWNFAIL=5, SKIP=17, errors=2, failures=1)
>>
>>
>>
>>
> I don't see any of these with
>
>
> Running unit tests for numpy
> NumPy version 1.8.0.dev-dff8c94
> NumPy is installed in
> /home/charris/.local/lib/python2.7/site-packages/numpy
> Python version 2.7.3 (default, Aug  9 2012, 17:23:57) [GCC 4.7.1 20120720
> (Red Hat 4.7.1-5)]
> nose version 1.3.0
>
> I'll guess it's the ancient version of gcc.
>
>

I'm using Ubuntu 12.04, so I suspect I won't be the only one who sees these.

Warren


Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130601/cad78b89/attachment.html>

From warren.weckesser at gmail.com  Sat Jun  1 20:56:08 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 1 Jun 2013 20:56:08 -0400
Subject: [Numpy-discussion] numpy tests errors and failures
In-Reply-To: <CAB6mnx+9fYmL6-_ke9MJ2W5yF89etMWebC2H7VmQ7PkVF-N7nw@mail.gmail.com>
References: <CAGzF1ufUO4JTXr8d7GQGWgGJfCpqb0b2c4svjFMU3-amZqxY5Q@mail.gmail.com>
	<CAB6mnx+9fYmL6-_ke9MJ2W5yF89etMWebC2H7VmQ7PkVF-N7nw@mail.gmail.com>
Message-ID: <CAGzF1ucMO0UndAHvcgmykL6xZ6Hz3QxEEqfFrCs=Lx-6ooEvtg@mail.gmail.com>

On Sat, Jun 1, 2013 at 7:47 PM, Charles R Harris
<charlesr.harris at gmail.com>wrote:

>
>
> On Sat, Jun 1, 2013 at 4:50 PM, Warren Weckesser <
> warren.weckesser at gmail.com> wrote:
>
>> I'm getting a failure and two errors with the latest master branch:
>>
>> $ python -c "import numpy; numpy.test('full')"
>> Running unit tests for numpy
>> NumPy version 1.8.0.dev-dff8c94
>> NumPy is installed in
>> /home/warren/local_numpy/lib/python2.7/site-packages/numpy
>> Python version 2.7.4 |Anaconda 1.5.0 (64-bit)| (default, Apr 21 2013,
>> 17:43:08) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)]
>> nose version 1.3.0
>>
>> .....................F...................................................................................................................................................................................................................................................................................................................S...............................................................................................................................S..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K........................................E.E.................................................................................................................................................SSS...............................................................................................................................................................................................................................................................................................................................................K........................................................................................................K......................K...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SS..SS.....................................................................................................SSSSSSSS................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................
>> ======================================================================
>> ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose([1e-08, 1,
>> 1000020.0000000099], [0, nan, 1000000.0])
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
>> line 197, in runTest
>>     self.test(*self.arg)
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
>> line 1253, in tst_isclose_allclose
>>     assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py",
>> line 2008, in allclose
>>     return all(less_equal(abs(x-y), atol + rtol * abs(y)))
>> RuntimeWarning: invalid value encountered in absolute
>>
>> ======================================================================
>> ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose(nan, [nan, nan,
>> nan])
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
>> line 197, in runTest
>>     self.test(*self.arg)
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
>> line 1253, in tst_isclose_allclose
>>     assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py",
>> line 2008, in allclose
>>     return all(less_equal(abs(x-y), atol + rtol * abs(y)))
>> RuntimeWarning: invalid value encountered in absolute
>>
>> ======================================================================
>> FAIL: Test numpy dot with different order C, F
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
>> line 197, in runTest
>>     self.test(*self.arg)
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.py",
>> line 114, in test_dot_array_order
>>     assert_almost_equal(a.dot(a), a.T.dot(a.T).T, decimal=30)
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
>> line 458, in assert_almost_equal
>>     return assert_array_almost_equal(actual, desired, decimal, err_msg)
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
>> line 819, in assert_array_almost_equal
>>     header=('Arrays are not almost equal to %d decimals' % decimal))
>>   File
>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
>> line 652, in assert_array_compare
>>     raise AssertionError(msg)
>> AssertionError:
>> Arrays are not almost equal to 30 decimals
>>
>
> Thirty decimals? What architecture?
>


`decimal=30` is used in three places in numpy/core/tests/test_blasdot.py,
in the function `test_dot_array_order()`.  It looks like the argument
should be `decimal=prec`.

If I make that change, the above test passes, but I get a failure in a test
for array equality in the same function (lines 119-120):

            a_T = a.T.copy(order=a_order)
            assert_array_equal(a_T.dot(a_T), a.T.dot(a.T))


When the dtype of `a` is float32, this test fails.

Warren



>
>>
>> (mismatch 26.0%)
>>  x: array([[ 0.60970873,  1.6909554 , -1.08851945, -1.82058012,
>> -3.95746613,
>>          1.52435613, -0.59853059, -3.72278619,  3.82375932,  5.51367044],
>>        [-3.58154893, -2.06231236, -0.0656727 ,  1.4737345 ,  2.6068747
>> ,...
>>  y: array([[ 0.60970873,  1.6909554 , -1.08851945, -1.82058001,
>> -3.95746613,
>>          1.52435613, -0.59853059, -3.72278643,  3.82375932,  5.51367044],
>>        [-3.58154917, -2.06231236, -0.0656727 ,  1.4737345 ,  2.6068747
>> ,...
>>
>> ----------------------------------------------------------------------
>> Ran 5151 tests in 60.506s
>>
>> FAILED (KNOWNFAIL=5, SKIP=17, errors=2, failures=1)
>>
>>
>>
>>
> I don't see any of these with
>
>
> Running unit tests for numpy
> NumPy version 1.8.0.dev-dff8c94
> NumPy is installed in
> /home/charris/.local/lib/python2.7/site-packages/numpy
> Python version 2.7.3 (default, Aug  9 2012, 17:23:57) [GCC 4.7.1 20120720
> (Red Hat 4.7.1-5)]
> nose version 1.3.0
>
> I'll guess it's the ancient version of gcc.
>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130601/4918ae77/attachment.html>

From warren.weckesser at gmail.com  Sat Jun  1 21:06:24 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 1 Jun 2013 21:06:24 -0400
Subject: [Numpy-discussion] numpy tests errors and failures
In-Reply-To: <CAGzF1ucMO0UndAHvcgmykL6xZ6Hz3QxEEqfFrCs=Lx-6ooEvtg@mail.gmail.com>
References: <CAGzF1ufUO4JTXr8d7GQGWgGJfCpqb0b2c4svjFMU3-amZqxY5Q@mail.gmail.com>
	<CAB6mnx+9fYmL6-_ke9MJ2W5yF89etMWebC2H7VmQ7PkVF-N7nw@mail.gmail.com>
	<CAGzF1ucMO0UndAHvcgmykL6xZ6Hz3QxEEqfFrCs=Lx-6ooEvtg@mail.gmail.com>
Message-ID: <CAGzF1uedvcn-q_RsP7edX1H-UP_Se0_sX3T_ts7=VUKcG5n2mw@mail.gmail.com>

On Sat, Jun 1, 2013 at 8:56 PM, Warren Weckesser <warren.weckesser at gmail.com
> wrote:

>
>
>
> On Sat, Jun 1, 2013 at 7:47 PM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Sat, Jun 1, 2013 at 4:50 PM, Warren Weckesser <
>> warren.weckesser at gmail.com> wrote:
>>
>>> I'm getting a failure and two errors with the latest master branch:
>>>
>>> $ python -c "import numpy; numpy.test('full')"
>>> Running unit tests for numpy
>>> NumPy version 1.8.0.dev-dff8c94
>>> NumPy is installed in
>>> /home/warren/local_numpy/lib/python2.7/site-packages/numpy
>>> Python version 2.7.4 |Anaconda 1.5.0 (64-bit)| (default, Apr 21 2013,
>>> 17:43:08) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)]
>>> nose version 1.3.0
>>>
>>> .....................F...................................................................................................................................................................................................................................................................................................................S...............................................................................................................................S..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K........................................E.E.................................................................................................................................................SSS...............................................................................................................................................................................................................................................................................................................................................K........................................................................................................K......................K...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SS..SS.....................................................................................................SSSSSSSS................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................
>>> ======================================================================
>>> ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose([1e-08, 1,
>>> 1000020.0000000099], [0, nan, 1000000.0])
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
>>> line 197, in runTest
>>>     self.test(*self.arg)
>>>   File
>>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
>>> line 1253, in tst_isclose_allclose
>>>     assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
>>>   File
>>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py",
>>> line 2008, in allclose
>>>     return all(less_equal(abs(x-y), atol + rtol * abs(y)))
>>> RuntimeWarning: invalid value encountered in absolute
>>>
>>> ======================================================================
>>> ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose(nan, [nan, nan,
>>> nan])
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
>>> line 197, in runTest
>>>     self.test(*self.arg)
>>>   File
>>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
>>> line 1253, in tst_isclose_allclose
>>>     assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
>>>   File
>>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py",
>>> line 2008, in allclose
>>>     return all(less_equal(abs(x-y), atol + rtol * abs(y)))
>>> RuntimeWarning: invalid value encountered in absolute
>>>
>>> ======================================================================
>>> FAIL: Test numpy dot with different order C, F
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>   File "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py",
>>> line 197, in runTest
>>>     self.test(*self.arg)
>>>   File
>>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.py",
>>> line 114, in test_dot_array_order
>>>     assert_almost_equal(a.dot(a), a.T.dot(a.T).T, decimal=30)
>>>   File
>>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
>>> line 458, in assert_almost_equal
>>>     return assert_array_almost_equal(actual, desired, decimal, err_msg)
>>>   File
>>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
>>> line 819, in assert_array_almost_equal
>>>     header=('Arrays are not almost equal to %d decimals' % decimal))
>>>   File
>>> "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py",
>>> line 652, in assert_array_compare
>>>     raise AssertionError(msg)
>>> AssertionError:
>>> Arrays are not almost equal to 30 decimals
>>>
>>
>> Thirty decimals? What architecture?
>>
>
>
>  `decimal=30` is used in three places in
> numpy/core/tests/test_blasdot.py, in the function
> `test_dot_array_order()`.  It looks like the argument should be
> `decimal=prec`.
>
> If I make that change, the above test passes, but I get a failure in a
> test for array equality in the same function (lines 119-120):
>
>             a_T = a.T.copy(order=a_order)
>             assert_array_equal(a_T.dot(a_T), a.T.dot(a.T))
>
>
> When the dtype of `a` is float32, this test fails.
>
>
If I replace all tests of the form:

    assert_array_equal(X, Y)

with

    assert_almost_equal(X, Y, decimal=prec)

all the tests in the function test_doc_array_order() pass.

Warren



> Warren
>
>
>
>>
>>>
>>> (mismatch 26.0%)
>>>  x: array([[ 0.60970873,  1.6909554 , -1.08851945, -1.82058012,
>>> -3.95746613,
>>>          1.52435613, -0.59853059, -3.72278619,  3.82375932,  5.51367044],
>>>        [-3.58154893, -2.06231236, -0.0656727 ,  1.4737345 ,  2.6068747
>>> ,...
>>>  y: array([[ 0.60970873,  1.6909554 , -1.08851945, -1.82058001,
>>> -3.95746613,
>>>          1.52435613, -0.59853059, -3.72278643,  3.82375932,  5.51367044],
>>>        [-3.58154917, -2.06231236, -0.0656727 ,  1.4737345 ,  2.6068747
>>> ,...
>>>
>>> ----------------------------------------------------------------------
>>> Ran 5151 tests in 60.506s
>>>
>>> FAILED (KNOWNFAIL=5, SKIP=17, errors=2, failures=1)
>>>
>>>
>>>
>>>
>> I don't see any of these with
>>
>>
>> Running unit tests for numpy
>> NumPy version 1.8.0.dev-dff8c94
>> NumPy is installed in
>> /home/charris/.local/lib/python2.7/site-packages/numpy
>> Python version 2.7.3 (default, Aug  9 2012, 17:23:57) [GCC 4.7.1 20120720
>> (Red Hat 4.7.1-5)]
>> nose version 1.3.0
>>
>> I'll guess it's the ancient version of gcc.
>>
>> Chuck
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130601/d6bd35dc/attachment.html>

From sebastian at sipsolutions.net  Sun Jun  2 05:57:17 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Sun, 02 Jun 2013 11:57:17 +0200
Subject: [Numpy-discussion] numpy tests errors and failures
In-Reply-To: <CAB6mnx+9fYmL6-_ke9MJ2W5yF89etMWebC2H7VmQ7PkVF-N7nw@mail.gmail.com>
References: <CAGzF1ufUO4JTXr8d7GQGWgGJfCpqb0b2c4svjFMU3-amZqxY5Q@mail.gmail.com>
	<CAB6mnx+9fYmL6-_ke9MJ2W5yF89etMWebC2H7VmQ7PkVF-N7nw@mail.gmail.com>
Message-ID: <1370167037.15394.9.camel@sebastian-laptop>

On Sat, 2013-06-01 at 17:47 -0600, Charles R Harris wrote:
> 
> 
> On Sat, Jun 1, 2013 at 4:50 PM, Warren Weckesser
> <warren.weckesser at gmail.com> wrote:
>         I'm getting a failure and two errors with the latest master
>         branch:
>         
>         $ python -c "import numpy; numpy.test('full')"
>         Running unit tests for numpy
>         NumPy version 1.8.0.dev-dff8c94
>         NumPy is installed
>         in /home/warren/local_numpy/lib/python2.7/site-packages/numpy
>         Python version 2.7.4 |Anaconda 1.5.0 (64-bit)| (default, Apr
>         21 2013, 17:43:08) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)]
>         nose version 1.3.0
>         .....................F...................................................................................................................................................................................................................................................................................................................S...............................................................................................................................S..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K........................................E.E.................................................................................................................................................SSS...............................................................................................................................................................................................................................................................................................................................................K........................................................................................................K......................K...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SS..SS.....................................................................................................SSSSSSSS................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K....................................................
>         ======================================================================
>         ERROR:
>         test_numeric.TestIsclose.test_ip_isclose_allclose([1e-08, 1,
>         1000020.0000000099], [0, nan, 1000000.0])
>         ----------------------------------------------------------------------
>         Traceback (most recent call last):
>           File
>         "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
>             self.test(*self.arg)
>           File
>         "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", line 1253, in tst_isclose_allclose
>             assert_array_equal(isclose(x, y).all(), allclose(x, y),
>         msg % (x, y))
>           File
>         "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py", line 2008, in allclose
>             return all(less_equal(abs(x-y), atol + rtol * abs(y)))
>         RuntimeWarning: invalid value encountered in absolute
>         
>         ======================================================================
>         ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose(nan,
>         [nan, nan, nan])
>         ----------------------------------------------------------------------
>         Traceback (most recent call last):
>           File
>         "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
>             self.test(*self.arg)
>           File
>         "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", line 1253, in tst_isclose_allclose
>             assert_array_equal(isclose(x, y).all(), allclose(x, y),
>         msg % (x, y))
>           File
>         "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/numeric.py", line 2008, in allclose
>             return all(less_equal(abs(x-y), atol + rtol * abs(y)))
>         RuntimeWarning: invalid value encountered in absolute
>         
>         ======================================================================
>         FAIL: Test numpy dot with different order C, F
>         ----------------------------------------------------------------------
>         Traceback (most recent call last):
>           File
>         "/home/warren/anaconda/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
>             self.test(*self.arg)
>           File
>         "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.py", line 114, in test_dot_array_order
>             assert_almost_equal(a.dot(a), a.T.dot(a.T).T, decimal=30)
>           File
>         "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py", line 458, in assert_almost_equal
>             return assert_array_almost_equal(actual, desired, decimal,
>         err_msg)
>           File
>         "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py", line 819, in assert_array_almost_equal
>             header=('Arrays are not almost equal to %d decimals' %
>         decimal))
>           File
>         "/home/warren/local_numpy/lib/python2.7/site-packages/numpy/testing/utils.py", line 652, in assert_array_compare
>             raise AssertionError(msg)
>         AssertionError: 
>         Arrays are not almost equal to 30 decimals
>         
> 
> Thirty decimals? What architecture?
>  
> 
>         
>         (mismatch 26.0%)
>          x: array([[ 0.60970873,  1.6909554 , -1.08851945,
>         -1.82058012, -3.95746613,
>                  1.52435613, -0.59853059, -3.72278619,  3.82375932,
>         5.51367044],
>                [-3.58154893, -2.06231236, -0.0656727 ,  1.4737345 ,
>         2.6068747 ,...
>          y: array([[ 0.60970873,  1.6909554 , -1.08851945,
>         -1.82058001, -3.95746613,
>                  1.52435613, -0.59853059, -3.72278643,  3.82375932,
>         5.51367044],
>                [-3.58154917, -2.06231236, -0.0656727 ,  1.4737345 ,
>         2.6068747 ,...
>         
>         ----------------------------------------------------------------------
>         Ran 5151 tests in 60.506s
>         
>         FAILED (KNOWNFAIL=5, SKIP=17, errors=2, failures=1)
>         
>         
>         
>         
> 
> I don't see any of these with
> 
> Running unit tests for numpy
> NumPy version 1.8.0.dev-dff8c94
> NumPy is installed
> in /home/charris/.local/lib/python2.7/site-packages/numpy
> Python version 2.7.3 (default, Aug  9 2012, 17:23:57) [GCC 4.7.1
> 20120720 (Red Hat 4.7.1-5)]
> nose version 1.3.0
> 
> I'll guess it's the ancient version of gcc.
> 

The second failure should depend on the lapack/blas version used for
dot, so differences are to be expected I guess and the test simply needs
to be (further) relaxed probably. Not sure about the first thing, I
somewhat think we had similar failures when testing 1.7. but I don't
remember.

- Sebastian


> Chuck
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From ralf.gommers at gmail.com  Sun Jun  2 09:40:26 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 2 Jun 2013 15:40:26 +0200
Subject: [Numpy-discussion] 1.8 release update
In-Reply-To: <CAB6mnxJy=hOBvowjOg-TaDgXyGmt8X1HM-Pjq94PZ5NVB8huJA@mail.gmail.com>
References: <CAB6mnxJy=hOBvowjOg-TaDgXyGmt8X1HM-Pjq94PZ5NVB8huJA@mail.gmail.com>
Message-ID: <CABL7CQheiX+ucPnAuWi4Rfer+Evt9VENaMyE1fVPTzMbevH8mw@mail.gmail.com>

On Fri, May 31, 2013 at 11:47 PM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

> Hi All,
>
> Most of the PR's that were mentioned as desirable for the 1.8 release have
> been merged, or look to be merged in the next week or two. The current list
> of blockers<https://github.com/numpy/numpy/issues?milestone=1&page=1&state=open>doesn't look to severe to me but I suspect that it is incomplete.


Indeed nothing serious there. I added a ticket for the timezone issue,
https://github.com/numpy/numpy/issues/3388.


> The major outstanding issue I see is datetime and I'd like to get a small
> group together to work that out. As a start I think such a group should
> include Christopher Barker and Wes McKinney. Suggestions for other folks,
> or even volunteers, to be part of such a group are welcome.
>
> A lot of stuff has piled up over the last year for inclusion in the 1.8
> release, and I'm sure some bugs and regressions  have crept in along with
> the new code. On that account a lengthy wringing out period is probably
> going to be needed, so the sooner we can get the 1.8 branch tagged the
> better. I'd like to shoot for getting that done in 2-3 weeks.


How does that match with getting the datetime situation sorted out? I don't
think we're in a hurry with branching before we at least have an idea about
what to do about datetime. Other than that, soonish sounds good to me. We
should make an attempt to merge PRs that have been open for a while though.

Ralf


> If there look to be difficult issues remaining, perhaps they can be worked
> out at scipy2013 if there is enough interest.
>
> Thoughts?
>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130602/86d063bc/attachment.html>

From scopatz at gmail.com  Sun Jun  2 11:59:17 2013
From: scopatz at gmail.com (Anthony Scopatz)
Date: Sun, 2 Jun 2013 10:59:17 -0500
Subject: [Numpy-discussion] ANN: PyTables 3.0
Message-ID: <CAPk-6T4+VDQ36L0mt4fhdy+veJ=E3hH5MjiX7xtVJ-R+2BJeFg@mail.gmail.com>

===========================
  Announcing PyTables 3.0.0
===========================

We are happy to announce PyTables 3.0.0.

PyTables 3.0.0 comes after about 5 years from the last major release
(2.0) and 7 months since the last stable release (2.4.0).

This is new major release and an important milestone for the PyTables
project since it provides the long waited support for Python 3.x, which
has been around for 4 years.

Almost all of the core numeric/scientific packages for Python already
support Python 3 so we are very happy that now also PyTables can provide
this important feature.


What's new
==========

A short summary of main new features:

- Since this release, PyTables now provides full support to Python 3
- The entire code base is now more compliant with coding style
   guidelines described in PEP8.
- Basic support for HDF5 drivers.  It now is possible to open/create an
   HDF5 file using one of the SEC2, DIRECT, LOG, WINDOWS, STDIO or CORE
   drivers.
- Basic support for in-memory image files.  An HDF5 file can be set
   from or copied into a memory buffer.
- Implemented methods to get/set the user block size in a HDF5 file.
- All read methods now have an optional *out* argument that allows to
   pass a pre-allocated array to store data.
- Added support for the floating point data types with extended
   precision (Float96, Float128, Complex192 and Complex256).
- Consistent ``create_xxx()`` signatures.  Now it is possible to create
   all data sets Array, CArray, EArray, VLArray, and Table from existing
   Python objects.
- Complete rewrite of the `nodes.filenode` module. Now it is fully
   compliant with the interfaces defined in the standard `io` module.
   Only non-buffered binary I/O is supported currently.

Please refer to the RELEASE_NOTES document for a more detailed list of
changes in this release.

As always, a large amount of bugs have been addressed and squashed as well.

In case you want to know more in detail what has changed in this
version, please refer to: http://pytables.github.io/release_notes.html

You can download a source package with generated PDF and HTML docs, as
well as binaries for Windows, from:
http://sourceforge.net/projects/pytables/files/pytables/3.0.0

For an online version of the manual, visit:
http://pytables.github.io/usersguide/index.html


What it is?
===========

PyTables is a library for managing hierarchical datasets and
designed to efficiently cope with extremely large amounts of data with
support for full 64-bit file addressing.  PyTables runs on top of
the HDF5 library and NumPy package for achieving maximum throughput and
convenient use.  PyTables includes OPSI, a new indexing technology,
allowing to perform data lookups in tables exceeding 10 gigarows
(10**10 rows) in less than a tenth of a second.


Resources
=========

About PyTables: http://www.pytables.org

About the HDF5 library: http://hdfgroup.org/HDF5/

About NumPy: http://numpy.scipy.org/


Acknowledgments
===============

Thanks to many users who provided feature improvements, patches, bug
reports, support and suggestions.  See the ``THANKS`` file in the
distribution package for a (incomplete) list of contributors.  Most
specially, a lot of kudos go to the HDF5 and NumPy makers.
Without them, PyTables simply would not exist.


Share your experience
=====================

Let us know of any bugs, suggestions, gripes, kudos, etc. you may have.


----

   **Enjoy data!**

   -- The PyTables Developers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130602/2abd5d6d/attachment.html>

From Jerome.Kieffer at esrf.fr  Mon Jun  3 01:29:42 2013
From: Jerome.Kieffer at esrf.fr (Jerome Kieffer)
Date: Mon, 3 Jun 2013 07:29:42 +0200
Subject: [Numpy-discussion] About speed vs math...
Message-ID: <20130603072942.c92e846746a74f5229bba453@esrf.fr>

Hello,

I am giving some introduction tutorials to numpy and we notices a big
difference in speed between nuumpy and math for trigonometric
operations:

In [3]: %timeit numpy.sin(1)
100000 loops, best of 3: 2.27 us per loop

In [4]: %timeit math.sin(1)
10000000 loops, best of 3: 92.3 ns per loop

While not surprising, I did not expect numpy to be so much slower (25x)...

Is this normal ?
Cheers,
-- 
J?r?me Kieffer
Data analysis unit - ESRF


From cournape at gmail.com  Mon Jun  3 02:33:23 2013
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 3 Jun 2013 07:33:23 +0100
Subject: [Numpy-discussion] About speed vs math...
In-Reply-To: <20130603072942.c92e846746a74f5229bba453@esrf.fr>
References: <20130603072942.c92e846746a74f5229bba453@esrf.fr>
Message-ID: <CAGY4rcVbv=+Oh8BNn6OGCm0En5A4eprXYKwzayr3NRYO+KZjrA@mail.gmail.com>

On Mon, Jun 3, 2013 at 6:29 AM, Jerome Kieffer <Jerome.Kieffer at esrf.fr> wrote:
> Hello,
>
> I am giving some introduction tutorials to numpy and we notices a big
> difference in speed between nuumpy and math for trigonometric
> operations:
>
> In [3]: %timeit numpy.sin(1)
> 100000 loops, best of 3: 2.27 us per loop
>
> In [4]: %timeit math.sin(1)
> 10000000 loops, best of 3: 92.3 ns per loop
>
> While not surprising, I did not expect numpy to be so much slower (25x)...

It is a known limitation of numpy scalars. As soon as you use array
that are not tiny, the speed difference disappears and then favors
numpy arrays (around 50 on my own machine, but that's platform
specific).

David


From Jerome.Kieffer at esrf.fr  Mon Jun  3 03:30:02 2013
From: Jerome.Kieffer at esrf.fr (Jerome Kieffer)
Date: Mon, 3 Jun 2013 09:30:02 +0200
Subject: [Numpy-discussion] About speed vs math...
In-Reply-To: <CAGY4rcVbv=+Oh8BNn6OGCm0En5A4eprXYKwzayr3NRYO+KZjrA@mail.gmail.com>
References: <20130603072942.c92e846746a74f5229bba453@esrf.fr>
	<CAGY4rcVbv=+Oh8BNn6OGCm0En5A4eprXYKwzayr3NRYO+KZjrA@mail.gmail.com>
Message-ID: <20130603093002.7acda01c.Jerome.Kieffer@esrf.fr>

On Mon, 3 Jun 2013 07:33:23 +0100
David Cournapeau <cournape at gmail.com> wrote:

> > While not surprising, I did not expect numpy to be so much slower (25x)...
> It is a known limitation of numpy scalars. As soon as you use array
> that are not tiny, the speed difference disappears and then favors
> numpy arrays (around 50 on my own machine, but that's platform
> specific).

Thanks for the figure and the explanation.


-- 
J?r?me Kieffer
On-Line Data analysis / Software Group 
ISDD / ESRF
tel +33 476 882 445


From chaoyuejoy at gmail.com  Mon Jun  3 04:44:43 2013
From: chaoyuejoy at gmail.com (Chao YUE)
Date: Mon, 3 Jun 2013 10:44:43 +0200
Subject: [Numpy-discussion] easy way to collapse the last two dimension of
	an array?
Message-ID: <CAAN-aRE9db4GBkTQAfv36UtCS_0VSSm5B2uefjXcpuZZW-vOFg@mail.gmail.com>

Dear all,

I have an array with 4 dim:

In [24]: dd.shape
Out[24]: (12, 13, 120, 170)

I would like to collapse the last two dim for applying np.sum(axis=-1)

In [25]: dd.reshape(12,13,-1).shape
Out[25]: (12, 13, 20400)

is there a more general way to do this? something like
In [21]: dd.reshape(*dd.shape[0:2],-1).shape
------------------------------------------------------------
   File "<ipython console>", line 1
SyntaxError: only named arguments may follow *expression (<ipython
console>, line 1)

thanks a lot for any hints,

cheers,

Chao

-- 
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130603/bde9577d/attachment.html>

From robert.kern at gmail.com  Mon Jun  3 05:39:12 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 3 Jun 2013 10:39:12 +0100
Subject: [Numpy-discussion] easy way to collapse the last two dimension
 of an array?
In-Reply-To: <CAAN-aRE9db4GBkTQAfv36UtCS_0VSSm5B2uefjXcpuZZW-vOFg@mail.gmail.com>
References: <CAAN-aRE9db4GBkTQAfv36UtCS_0VSSm5B2uefjXcpuZZW-vOFg@mail.gmail.com>
Message-ID: <CAF6FJis8d9NWBrRD2F=OsCyUF=RhORyHhT+vHhd-ukLHHF+3HA@mail.gmail.com>

On Mon, Jun 3, 2013 at 9:44 AM, Chao YUE <chaoyuejoy at gmail.com> wrote:
> Dear all,
>
> I have an array with 4 dim:
>
> In [24]: dd.shape
> Out[24]: (12, 13, 120, 170)
>
> I would like to collapse the last two dim for applying np.sum(axis=-1)
>
> In [25]: dd.reshape(12,13,-1).shape
> Out[25]: (12, 13, 20400)
>
> is there a more general way to do this? something like
> In [21]: dd.reshape(*dd.shape[0:2],-1).shape
> ------------------------------------------------------------
>    File "<ipython console>", line 1
> SyntaxError: only named arguments may follow *expression (<ipython console>,
> line 1)

dd.reshape(dd.shape[:-2] + (-1,))

--
Robert Kern


From sebastian at sipsolutions.net  Mon Jun  3 05:53:27 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Mon, 03 Jun 2013 11:53:27 +0200
Subject: [Numpy-discussion] easy way to collapse the last two dimension
 of an array?
In-Reply-To: <CAAN-aRE9db4GBkTQAfv36UtCS_0VSSm5B2uefjXcpuZZW-vOFg@mail.gmail.com>
References: <CAAN-aRE9db4GBkTQAfv36UtCS_0VSSm5B2uefjXcpuZZW-vOFg@mail.gmail.com>
Message-ID: <1370253207.15394.10.camel@sebastian-laptop>

On Mon, 2013-06-03 at 10:44 +0200, Chao YUE wrote:
> Dear all,
> 
> I have an array with 4 dim:
> 
> In [24]: dd.shape
> Out[24]: (12, 13, 120, 170)
> 
> I would like to collapse the last two dim for applying np.sum(axis=-1)
> 
If you use Numpy >= 1.7. you can also just use dd.sum(axis=(-1,-2))

- Sebastian


> In [25]: dd.reshape(12,13,-1).shape
> Out[25]: (12, 13, 20400)
> 
> is there a more general way to do this? something like
> In [21]: dd.reshape(*dd.shape[0:2],-1).shape
> ------------------------------------------------------------
>    File "<ipython console>", line 1
> SyntaxError: only named arguments may follow *expression (<ipython
> console>, line 1)
> 
> thanks a lot for any hints,
> 
> cheers,
> 
> Chao
> 
> -- 
> ***********************************************************************************
> Chao YUE
> Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
> UMR 1572 CEA-CNRS-UVSQ
> Batiment 712 - Pe 119
> 91191 GIF Sur YVETTE Cedex
> Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
> 
> ************************************************************************************
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From davidmenhur at gmail.com  Mon Jun  3 06:15:33 2013
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Mon, 3 Jun 2013 12:15:33 +0200
Subject: [Numpy-discussion] About speed vs math...
In-Reply-To: <CAGY4rcVbv=+Oh8BNn6OGCm0En5A4eprXYKwzayr3NRYO+KZjrA@mail.gmail.com>
References: <20130603072942.c92e846746a74f5229bba453@esrf.fr>
	<CAGY4rcVbv=+Oh8BNn6OGCm0En5A4eprXYKwzayr3NRYO+KZjrA@mail.gmail.com>
Message-ID: <CAJhcF=30OTxvkN9J49LMitaOx78SZQm-aBBrwPVDWqLxrrMr7Q@mail.gmail.com>

On 3 June 2013 08:33, David Cournapeau <cournape at gmail.com> wrote:
> (around 50 on my own machine, but that's platform
> specific).

In my machine, it is around 10. You also have to be aware of the data
container: it is not the same to iterate over lists than over arrays.

In [6]: a = np.arange(50)

In [7]: %timeit np.sin(a)
100000 loops, best of 3: 3.95 us per loop

In [8]: %timeit np.array([math.sin(i) for i in a]) # We are working
with arrays, we keep the array output.
10000 loops, best of 3: 32.9 us per loop

In [9]: %timeit [math.sin(i) for i in a]    # Keep the list output.
10000 loops, best of 3: 21.3 us per loop



But, if you stick to lists:

In [12]: b = range(50)

In [13]: %timeit [math.sin(i) for i in b]
100000 loops, best of 3: 12.4 us per loop



In any case, switching between iterators and numpy functions is, in my
opinion, error prone. I would use math only for scalars, and numpy for
the rest, except if optimisation is needed (as a last step). For the
record, another interesting example:


In [18]: a = np.arange(10)

In [19]: b = range(10)

In [20]: %timeit a + b                              ## Mixed container
100000 loops, best of 3: 10.4 us per loop

In [21]: %timeit [i + j for i, j in zip(a, b)]
100000 loops, best of 3: 12 us per loop

In [22]: %timeit a + a                               ## Pure array
1000000 loops, best of 3: 1.23 us per loop

In [23]: %timeit [i + j for i, j in zip(a, a)]
100000 loops, best of 3: 7.21 us per loop

In [24]: %timeit b + b                              ## Pure list
1000000 loops, best of 3: 197 ns per loop ## OOOOPS

In [25]: %timeit [i + j for i, j in zip(b, b)]
100000 loops, best of 3: 1.68 us per loop

Here, b + b is just attaching the lists, not adding them. In a
program, that may be an error due to forgotten conversion.

Out of curiosity, VPython, a visual package oriented to beginners, has
its own functions that redirect to math if applied on scalars and to
numpy otherwise.



David.


From chaoyuejoy at gmail.com  Mon Jun  3 06:20:23 2013
From: chaoyuejoy at gmail.com (Chao YUE)
Date: Mon, 3 Jun 2013 12:20:23 +0200
Subject: [Numpy-discussion] easy way to collapse the last two dimension
 of an array?
In-Reply-To: <1370253207.15394.10.camel@sebastian-laptop>
References: <CAAN-aRE9db4GBkTQAfv36UtCS_0VSSm5B2uefjXcpuZZW-vOFg@mail.gmail.com>
	<1370253207.15394.10.camel@sebastian-laptop>
Message-ID: <CAAN-aRFZgwEdaiR4AfJo7Fp1OkaVgjnHX0OTRz74wghSzjF1Dg@mail.gmail.com>

Thanks to the two of you!!

Chao

On Mon, Jun 3, 2013 at 11:53 AM, Sebastian Berg
<sebastian at sipsolutions.net>wrote:

> On Mon, 2013-06-03 at 10:44 +0200, Chao YUE wrote:
> > Dear all,
> >
> > I have an array with 4 dim:
> >
> > In [24]: dd.shape
> > Out[24]: (12, 13, 120, 170)
> >
> > I would like to collapse the last two dim for applying np.sum(axis=-1)
> >
> If you use Numpy >= 1.7. you can also just use dd.sum(axis=(-1,-2))
>
> - Sebastian
>
>
> > In [25]: dd.reshape(12,13,-1).shape
> > Out[25]: (12, 13, 20400)
> >
> > is there a more general way to do this? something like
> > In [21]: dd.reshape(*dd.shape[0:2],-1).shape
> > ------------------------------------------------------------
> >    File "<ipython console>", line 1
> > SyntaxError: only named arguments may follow *expression (<ipython
> > console>, line 1)
> >
> > thanks a lot for any hints,
> >
> > cheers,
> >
> > Chao
> >
> > --
> >
> ***********************************************************************************
> > Chao YUE
> > Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
> > UMR 1572 CEA-CNRS-UVSQ
> > Batiment 712 - Pe 119
> > 91191 GIF Sur YVETTE Cedex
> > Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
> >
> >
> ************************************************************************************
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130603/354c90c0/attachment.html>

From antonio.valentino at tiscali.it  Mon Jun  3 13:18:09 2013
From: antonio.valentino at tiscali.it (Antonio Valentino)
Date: Mon, 03 Jun 2013 19:18:09 +0200
Subject: [Numpy-discussion] ANN: PyTables 3.0
Message-ID: <51ACCFD1.9080405@tiscali.it>

===========================
  Announcing PyTables 3.0.0
===========================

We are happy to announce PyTables 3.0.0.

PyTables 3.0.0 comes after about 5 years from the last major release
(2.0) and 7 months since the last stable release (2.4.0).

This is new major release and an important milestone for the PyTables 
project since it provides the long waited support for Python 3.x, which 
has been around for 4 years.

Almost all of the core numeric/scientific packages for Python already 
support Python 3 so we are very happy that now also PyTables can provide 
this important feature.


What's new
==========

A short summary of main new features:

- Since this release, PyTables now provides full support to Python 3
- The entire code base is now more compliant with coding style
   guidelines described in PEP8.
- Basic support for HDF5 drivers.  It now is possible to open/create an
   HDF5 file using one of the SEC2, DIRECT, LOG, WINDOWS, STDIO or CORE
   drivers.
- Basic support for in-memory image files.  An HDF5 file can be set
   from or copied into a memory buffer.
- Implemented methods to get/set the user block size in a HDF5 file.
- All read methods now have an optional *out* argument that allows to
   pass a pre-allocated array to store data.
- Added support for the floating point data types with extended
   precision (Float96, Float128, Complex192 and Complex256).
- Consistent ``create_xxx()`` signatures.  Now it is possible to create
   all data sets Array, CArray, EArray, VLArray, and Table from existing
   Python objects.
- Complete rewrite of the `nodes.filenode` module. Now it is fully
   compliant with the interfaces defined in the standard `io` module.
   Only non-buffered binary I/O is supported currently.

Please refer to the RELEASE_NOTES document for a more detailed list of
changes in this release.

As always, a large amount of bugs have been addressed and squashed as well.

In case you want to know more in detail what has changed in this
version, please refer to: http://pytables.github.io/release_notes.html

You can download a source package with generated PDF and HTML docs, as
well as binaries for Windows, from:
http://sourceforge.net/projects/pytables/files/pytables/3.0.0

For an online version of the manual, visit:
http://pytables.github.io/usersguide/index.html


What it is?
===========

PyTables is a library for managing hierarchical datasets and
designed to efficiently cope with extremely large amounts of data with
support for full 64-bit file addressing.  PyTables runs on top of
the HDF5 library and NumPy package for achieving maximum throughput and
convenient use.  PyTables includes OPSI, a new indexing technology,
allowing to perform data lookups in tables exceeding 10 gigarows
(10**10 rows) in less than a tenth of a second.


Resources
=========

About PyTables: http://www.pytables.org

About the HDF5 library: http://hdfgroup.org/HDF5/

About NumPy: http://numpy.scipy.org/


Acknowledgments
===============

Thanks to many users who provided feature improvements, patches, bug
reports, support and suggestions.  See the ``THANKS`` file in the
distribution package for a (incomplete) list of contributors.  Most
specially, a lot of kudos go to the HDF5 and NumPy makers.
Without them, PyTables simply would not exist.


Share your experience
=====================

Let us know of any bugs, suggestions, gripes, kudos, etc. you may have.


----

   **Enjoy data!**

   -- The PyTables Developers


From tim.burgess at noaa.gov  Tue Jun  4 01:20:40 2013
From: tim.burgess at noaa.gov (Tim Burgess)
Date: Tue, 04 Jun 2013 15:20:40 +1000
Subject: [Numpy-discussion] numpy tests errors and failures
In-Reply-To: <CAGzF1ueX9+aZ=Xs_oer0ta6TaFOpegNvh31EibONA9aEG8R7dA@mail.gmail.com>
References: <CAGzF1ufUO4JTXr8d7GQGWgGJfCpqb0b2c4svjFMU3-amZqxY5Q@mail.gmail.com>
	<CAB6mnx+9fYmL6-_ke9MJ2W5yF89etMWebC2H7VmQ7PkVF-N7nw@mail.gmail.com>
	<CAGzF1ueX9+aZ=Xs_oer0ta6TaFOpegNvh31EibONA9aEG8R7dA@mail.gmail.com>
Message-ID: <1370323240.29027.2.camel@typhoon>

On Sat, 2013-06-01 at 20:09 -0400, Warren Weckesser wrote:


> I'm using Ubuntu 12.04, so I suspect I won't be the only one who sees
> these.
> 
gcc on 12.04 (precise) should be 4.6.3

See
http://packages.ubuntu.com/search?keywords=gcc&searchon=names&suite=precise&section=all




From warren.weckesser at gmail.com  Tue Jun  4 07:52:08 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Tue, 4 Jun 2013 07:52:08 -0400
Subject: [Numpy-discussion] numpy tests errors and failures
In-Reply-To: <1370323240.29027.2.camel@typhoon>
References: <CAGzF1ufUO4JTXr8d7GQGWgGJfCpqb0b2c4svjFMU3-amZqxY5Q@mail.gmail.com>
	<CAB6mnx+9fYmL6-_ke9MJ2W5yF89etMWebC2H7VmQ7PkVF-N7nw@mail.gmail.com>
	<CAGzF1ueX9+aZ=Xs_oer0ta6TaFOpegNvh31EibONA9aEG8R7dA@mail.gmail.com>
	<1370323240.29027.2.camel@typhoon>
Message-ID: <CAGzF1ufBxB_aaxZkR35FS0Mpe3hpip8PDTRQjC_QQcqHKN6XHw@mail.gmail.com>

On Tue, Jun 4, 2013 at 1:20 AM, Tim Burgess <tim.burgess at noaa.gov> wrote:

> On Sat, 2013-06-01 at 20:09 -0400, Warren Weckesser wrote:
>
>
> > I'm using Ubuntu 12.04, so I suspect I won't be the only one who sees
> > these.
> >
> gcc on 12.04 (precise) should be 4.6.3
>

> See
>
> http://packages.ubuntu.com/search?keywords=gcc&searchon=names&suite=precise&section=all
>
>

Yes, that's what it is.   The python from the Anaconda package includes
"[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2" in its banner, so I
guess Anaconda is built with an even older gcc.

If I build python myself (with ubuntu's gcc 4.6.3), I get the same failure
and two errors that I originally reported.  In this case, I used python
3.3.1:

$ python3 -c "import numpy as np; np.test('full')"
Running unit tests for numpy
NumPy version 1.8.0.dev-e9e490a
NumPy is installed in
/home/warren/local_numpy/lib/python3.3/site-packages/numpy
Python version 3.3.1 (default, Apr 13 2013, 13:42:07) [GCC 4.6.3]
nose version 1.2.1

<snip>

======================================================================
ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose([1e-08, 1,
1000020.0000000099], [0, nan, 1000000.0])
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/warren/local_py331/lib/python3.3/site-packages/nose-1.2.1-py3.3.egg/nose/case.py",
line 198, in runTest
    self.test(*self.arg)
  File
"/home/warren/local_numpy/lib/python3.3/site-packages/numpy/core/tests/test_numeric.py",
line 1253, in tst_isclose_allclose
    assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
  File
"/home/warren/local_numpy/lib/python3.3/site-packages/numpy/core/numeric.py",
line 2008, in allclose
    return all(less_equal(abs(x-y), atol + rtol * abs(y)))
RuntimeWarning: invalid value encountered in absolute

======================================================================
ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose(nan, [nan, nan,
nan])
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/warren/local_py331/lib/python3.3/site-packages/nose-1.2.1-py3.3.egg/nose/case.py",
line 198, in runTest
    self.test(*self.arg)
  File
"/home/warren/local_numpy/lib/python3.3/site-packages/numpy/core/tests/test_numeric.py",
line 1253, in tst_isclose_allclose
    assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
  File
"/home/warren/local_numpy/lib/python3.3/site-packages/numpy/core/numeric.py",
line 2008, in allclose
    return all(less_equal(abs(x-y), atol + rtol * abs(y)))
RuntimeWarning: invalid value encountered in absolute

======================================================================
FAIL: Test numpy dot with different order C, F
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/warren/local_py331/lib/python3.3/site-packages/nose-1.2.1-py3.3.egg/nose/case.py",
line 198, in runTest
    self.test(*self.arg)
  File
"/home/warren/local_numpy/lib/python3.3/site-packages/numpy/core/tests/test_blasdot.py",
line 114, in test_dot_array_order
    assert_almost_equal(a.dot(a), a.T.dot(a.T).T, decimal=30)
  File
"/home/warren/local_numpy/lib/python3.3/site-packages/numpy/testing/utils.py",
line 458, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File
"/home/warren/local_numpy/lib/python3.3/site-packages/numpy/testing/utils.py",
line 819, in assert_array_almost_equal
    header=('Arrays are not almost equal to %d decimals' % decimal))
  File
"/home/warren/local_numpy/lib/python3.3/site-packages/numpy/testing/utils.py",
line 652, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 30 decimals

(mismatch 10.0%)
 x: array([[ 0.60970883,  1.6909554 , -1.0885194 , -1.82058004, -3.95746616,
         1.52435604, -0.59853062, -3.72278631,  3.82375941,  5.51367039],
       [-3.58154905, -2.0623123 , -0.06567267,  1.47373436,  2.60687462,...
 y: array([[ 0.60970883,  1.6909554 , -1.0885194 , -1.82058004, -3.95746616,
         1.52435604, -0.59853062, -3.72278631,  3.82375941,  5.51367039],
       [-3.58154905, -2.0623123 , -0.06567267,  1.47373436,  2.60687462,...

----------------------------------------------------------------------
Ran 5150 tests in 33.772s

FAILED (KNOWNFAIL=6, SKIP=18, errors=2, failures=1)


Warren



>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130604/408b1c2a/attachment.html>

From 183.amir at gmail.com  Tue Jun  4 12:18:19 2013
From: 183.amir at gmail.com (Amir Mohammadi)
Date: Tue, 4 Jun 2013 16:18:19 +0000 (UTC)
Subject: [Numpy-discussion] SVD errors
References: <34928.128.32.52.185.1233620513.squirrel@calmail.berkeley.edu>
	<3d375d730902021627x2ef54411j5bb9504927ff831@mail.gmail.com>
	<gm908k$d55$1@ger.gmane.org>
	<be3bc680902091425v3579660dnfef416d2006b8332@mail.gmail.com>
	<3d375d730902091440u23d72956le54b4894751a8dc7@mail.gmail.com>
Message-ID: <loom.20130604T181140-634@post.gmane.org>

Robert Kern <robert.kern <at> gmail.com> writes:

> 
> On Mon, Feb 9, 2009 at 16:25, M Trumpis <mtrumpis <at> berkeley.edu> wrote:
> > I played around with a C translation of that test program, and found
> > that dgesvd (but not dgesdd) happens to converge and return all
> > non-negative singular values for both operators I was having trouble
> > with. I'm also looking at the Octave code just now, and I think
> > they're using dgesvd also. Any one know why numpy uses dgesdd? speed?
> 
> Yes.
> 

Hello Everyone,

I am trying to write my own svd function to use the "dgesvd" method from 
lapack, but my problem is that I cannot find the "dgesvd" method from this 
import "from numpy.linalg import lapack_lite".

Do I have to install lapack on my system(Ubuntu 13.04 64-bit)? I want to get 
the same results as MATLAB 2011b which is why I am trying to write a slightly 
different svd function than the function from "numpy.linalg.svd".

I would appreciate your help, Thank you.
Amir





From robert.kern at gmail.com  Tue Jun  4 12:31:48 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 4 Jun 2013 17:31:48 +0100
Subject: [Numpy-discussion] SVD errors
In-Reply-To: <loom.20130604T181140-634@post.gmane.org>
References: <34928.128.32.52.185.1233620513.squirrel@calmail.berkeley.edu>
	<3d375d730902021627x2ef54411j5bb9504927ff831@mail.gmail.com>
	<gm908k$d55$1@ger.gmane.org>
	<be3bc680902091425v3579660dnfef416d2006b8332@mail.gmail.com>
	<3d375d730902091440u23d72956le54b4894751a8dc7@mail.gmail.com>
	<loom.20130604T181140-634@post.gmane.org>
Message-ID: <CAF6FJitZDwFbJsVOrGLKQ23FweKqCykC_iQSFF0d46+bFMTQ1w@mail.gmail.com>

On Tue, Jun 4, 2013 at 5:18 PM, Amir Mohammadi <183.amir at gmail.com> wrote:

> Hello Everyone,
>
> I am trying to write my own svd function to use the "dgesvd" method from
> lapack, but my problem is that I cannot find the "dgesvd" method from this
> import "from numpy.linalg import lapack_lite".
>
> Do I have to install lapack on my system(Ubuntu 13.04 64-bit)? I want to get
> the same results as MATLAB 2011b which is why I am trying to write a slightly
> different svd function than the function from "numpy.linalg.svd".

numpy does not wrap this LAPACK function. You will have to install
LAPACK yourself and write your own extension module that wraps the
DGESVD subroutine.

scipy also wraps *gesdd, and it would probably be a simple matter of
copy-paste-fix to get it to wrap *gesvd too, since the interfaces are
essentially the same.

https://github.com/scipy/scipy/blob/master/scipy/lib/lapack/flapack_esv.pyf.src#L231

--
Robert Kern


From warren.weckesser at gmail.com  Tue Jun  4 13:51:48 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Tue, 4 Jun 2013 13:51:48 -0400
Subject: [Numpy-discussion] numpy tests errors and failures
In-Reply-To: <CAGzF1ufBxB_aaxZkR35FS0Mpe3hpip8PDTRQjC_QQcqHKN6XHw@mail.gmail.com>
References: <CAGzF1ufUO4JTXr8d7GQGWgGJfCpqb0b2c4svjFMU3-amZqxY5Q@mail.gmail.com>
	<CAB6mnx+9fYmL6-_ke9MJ2W5yF89etMWebC2H7VmQ7PkVF-N7nw@mail.gmail.com>
	<CAGzF1ueX9+aZ=Xs_oer0ta6TaFOpegNvh31EibONA9aEG8R7dA@mail.gmail.com>
	<1370323240.29027.2.camel@typhoon>
	<CAGzF1ufBxB_aaxZkR35FS0Mpe3hpip8PDTRQjC_QQcqHKN6XHw@mail.gmail.com>
Message-ID: <CAGzF1udn_uwx0QZZ4QGe1BFRcu0taJirzewOv3Cq3TZwEQxwSw@mail.gmail.com>

On Tue, Jun 4, 2013 at 7:52 AM, Warren Weckesser <warren.weckesser at gmail.com
> wrote:

>
> On Tue, Jun 4, 2013 at 1:20 AM, Tim Burgess <tim.burgess at noaa.gov> wrote:
>
>> On Sat, 2013-06-01 at 20:09 -0400, Warren Weckesser wrote:
>>
>>
>> > I'm using Ubuntu 12.04, so I suspect I won't be the only one who sees
>> > these.
>> >
>> gcc on 12.04 (precise) should be 4.6.3
>>
>
>> See
>>
>> http://packages.ubuntu.com/search?keywords=gcc&searchon=names&suite=precise&section=all
>>
>>
>
> Yes, that's what it is.   The python from the Anaconda package includes
> "[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2" in its banner, so I
> guess Anaconda is built with an even older gcc.
>
> If I build python myself (with ubuntu's gcc 4.6.3), I get the same failure
> and two errors that I originally reported.  In this case, I used python
> 3.3.1:
>
> $ python3 -c "import numpy as np; np.test('full')"
>
> Running unit tests for numpy
> NumPy version 1.8.0.dev-e9e490a
> NumPy is installed in
> /home/warren/local_numpy/lib/python3.3/site-packages/numpy
> Python version 3.3.1 (default, Apr 13 2013, 13:42:07) [GCC 4.6.3]
> nose version 1.2.1
>
> <snip>
>
> ======================================================================
> ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose([1e-08, 1,
> 1000020.0000000099], [0, nan, 1000000.0])
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/home/warren/local_py331/lib/python3.3/site-packages/nose-1.2.1-py3.3.egg/nose/case.py",
> line 198, in runTest
>     self.test(*self.arg)
>   File
> "/home/warren/local_numpy/lib/python3.3/site-packages/numpy/core/tests/test_numeric.py",
> line 1253, in tst_isclose_allclose
>
>     assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
>   File
> "/home/warren/local_numpy/lib/python3.3/site-packages/numpy/core/numeric.py",
> line 2008, in allclose
>
>     return all(less_equal(abs(x-y), atol + rtol * abs(y)))
> RuntimeWarning: invalid value encountered in absolute
>
> ======================================================================
> ERROR: test_numeric.TestIsclose.test_ip_isclose_allclose(nan, [nan, nan,
> nan])
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/home/warren/local_py331/lib/python3.3/site-packages/nose-1.2.1-py3.3.egg/nose/case.py",
> line 198, in runTest
>     self.test(*self.arg)
>   File
> "/home/warren/local_numpy/lib/python3.3/site-packages/numpy/core/tests/test_numeric.py",
> line 1253, in tst_isclose_allclose
>
>     assert_array_equal(isclose(x, y).all(), allclose(x, y), msg % (x, y))
>   File
> "/home/warren/local_numpy/lib/python3.3/site-packages/numpy/core/numeric.py",
> line 2008, in allclose
>
>     return all(less_equal(abs(x-y), atol + rtol * abs(y)))
> RuntimeWarning: invalid value encountered in absolute
>
> ======================================================================
> FAIL: Test numpy dot with different order C, F
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/home/warren/local_py331/lib/python3.3/site-packages/nose-1.2.1-py3.3.egg/nose/case.py",
> line 198, in runTest
>     self.test(*self.arg)
>   File
> "/home/warren/local_numpy/lib/python3.3/site-packages/numpy/core/tests/test_blasdot.py",
> line 114, in test_dot_array_order
>     assert_almost_equal(a.dot(a), a.T.dot(a.T).T, decimal=30)
>   File
> "/home/warren/local_numpy/lib/python3.3/site-packages/numpy/testing/utils.py",
> line 458, in assert_almost_equal
>
>     return assert_array_almost_equal(actual, desired, decimal, err_msg)
>   File
> "/home/warren/local_numpy/lib/python3.3/site-packages/numpy/testing/utils.py",
> line 819, in assert_array_almost_equal
>
>     header=('Arrays are not almost equal to %d decimals' % decimal))
>   File
> "/home/warren/local_numpy/lib/python3.3/site-packages/numpy/testing/utils.py",
> line 652, in assert_array_compare
>
>     raise AssertionError(msg)
> AssertionError:
> Arrays are not almost equal to 30 decimals
>
> (mismatch 10.0%)
>  x: array([[ 0.60970883,  1.6909554 , -1.0885194 , -1.82058004,
> -3.95746616,
>          1.52435604, -0.59853062, -3.72278631,  3.82375941,  5.51367039],
>        [-3.58154905, -2.0623123 , -0.06567267,  1.47373436,  2.60687462,...
>  y: array([[ 0.60970883,  1.6909554 , -1.0885194 , -1.82058004,
> -3.95746616,
>          1.52435604, -0.59853062, -3.72278631,  3.82375941,  5.51367039],
>        [-3.58154905, -2.0623123 , -0.06567267,  1.47373436,  2.60687462,...
>
> ----------------------------------------------------------------------
> Ran 5150 tests in 33.772s
>
> FAILED (KNOWNFAIL=6, SKIP=18, errors=2, failures=1)
>
>
> Warren
>
>

I created an issue for this:  https://github.com/numpy/numpy/issues/3399

Warren



>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130604/6be0b6fa/attachment.html>

From jslavin at cfa.harvard.edu  Tue Jun  4 14:07:06 2013
From: jslavin at cfa.harvard.edu (Slavin, Jonathan)
Date: Tue, 4 Jun 2013 14:07:06 -0400
Subject: [Numpy-discussion] suggested change of behavior for interp
Message-ID: <CACcRS=fPYS2z6_X6TfEXhsc5gfqdKuUZzzUMKXtL-JFQS82gFQ@mail.gmail.com>

Hi,

I would like to suggest that the behavior of numpy.interp be changed
regarding treatment of situations in which the x-coordinates are not
monotonically increasing.  Specifically, it seems to me that interp should
work correctly when the x-coordinate is decreasing monotonically.  Clearly
it cannot work if the x-coordinate is not monotonic, but in that case it
should raise an exception.  Currently if x is not increasing it simply
silently fails, providing incorrect values.  This fix could be as simple as
a monotonicity test and inversion if necessary (plus a raise statement for
non-monotonic cases).

Jon
________________________________________________________
Jonathan D. Slavin                 Harvard-Smithsonian CfA
jslavin at cfa.harvard.edu       60 Garden Street, MS 83
phone: (617) 496-7981       Cambridge, MA 02138-1516
fax: (617) 496-7577            USA
________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130604/5e285005/attachment.html>

From ralf.gommers at gmail.com  Tue Jun  4 16:13:12 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 4 Jun 2013 22:13:12 +0200
Subject: [Numpy-discussion] request for SWIG numpy.i users
Message-ID: <CABL7CQhjMqzAYfDCpDfir0_=5fF7H7tYk9o5x=p7LaZNyJtCsg@mail.gmail.com>

Hi,

If you're using or are very familiar with SWIG and the numpy.i interface to
it, please help to test and/or review
https://github.com/numpy/numpy/pull/3148. It's a fairly major update to
numpy.i by Bill Spotz, containing the following:
  - support for 4D arrays and memory managed output arguments
  - rework for the deprecated API's in numpy 1.6 and 1.7
  - a bug fix in a 3D typemap
  - documentation improvements

It would be good to have this merged before branching 1.8.x. Not many of
the regular reviewers of numpy PRs are familiar with numpy.i, therefore
help would be much appreciated.

Thanks,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130604/482bd955/attachment.html>

From charlesr.harris at gmail.com  Tue Jun  4 20:05:24 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 4 Jun 2013 18:05:24 -0600
Subject: [Numpy-discussion] suggested change of behavior for interp
In-Reply-To: <CACcRS=fPYS2z6_X6TfEXhsc5gfqdKuUZzzUMKXtL-JFQS82gFQ@mail.gmail.com>
References: <CACcRS=fPYS2z6_X6TfEXhsc5gfqdKuUZzzUMKXtL-JFQS82gFQ@mail.gmail.com>
Message-ID: <CAB6mnxLEuiX7BAs21qKD4JXR2M8WBquFiXH9kxd3+O7o8asZdA@mail.gmail.com>

On Tue, Jun 4, 2013 at 12:07 PM, Slavin, Jonathan
<jslavin at cfa.harvard.edu>wrote:

> Hi,
>
> I would like to suggest that the behavior of numpy.interp be changed
> regarding treatment of situations in which the x-coordinates are not
> monotonically increasing.  Specifically, it seems to me that interp should
> work correctly when the x-coordinate is decreasing monotonically.  Clearly
> it cannot work if the x-coordinate is not monotonic, but in that case it
> should raise an exception.  Currently if x is not increasing it simply
> silently fails, providing incorrect values.  This fix could be as simple as
> a monotonicity test and inversion if necessary (plus a raise statement for
> non-monotonic cases).
>
>
Seems reasonable, although it might add a bit of execution time.

Chuck

>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130604/25526df7/attachment.html>

From efiring at hawaii.edu  Tue Jun  4 21:08:29 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Tue, 04 Jun 2013 15:08:29 -1000
Subject: [Numpy-discussion] suggested change of behavior for interp
In-Reply-To: <CAB6mnxLEuiX7BAs21qKD4JXR2M8WBquFiXH9kxd3+O7o8asZdA@mail.gmail.com>
References: <CACcRS=fPYS2z6_X6TfEXhsc5gfqdKuUZzzUMKXtL-JFQS82gFQ@mail.gmail.com>
	<CAB6mnxLEuiX7BAs21qKD4JXR2M8WBquFiXH9kxd3+O7o8asZdA@mail.gmail.com>
Message-ID: <51AE8F8D.4000601@hawaii.edu>

On 2013/06/04 2:05 PM, Charles R Harris wrote:
>
>
> On Tue, Jun 4, 2013 at 12:07 PM, Slavin, Jonathan
> <jslavin at cfa.harvard.edu <mailto:jslavin at cfa.harvard.edu>> wrote:
>
>     Hi,
>
>     I would like to suggest that the behavior of numpy.interp be changed
>     regarding treatment of situations in which the x-coordinates are not
>     monotonically increasing.  Specifically, it seems to me that interp
>     should work correctly when the x-coordinate is decreasing
>     monotonically.  Clearly it cannot work if the x-coordinate is not
>     monotonic, but in that case it should raise an exception.  Currently
>     if x is not increasing it simply silently fails, providing incorrect
>     values.  This fix could be as simple as a monotonicity test and
>     inversion if necessary (plus a raise statement for non-monotonic cases).
>
>
> Seems reasonable, although it might add a bit of execution time.

The monotonicity test should be an option if it is available at all; 
when interpolating a small number of points from a large pair of arrays, 
the single sweep through the whole array could dominate the execution 
time.  Checking for increasing versus decreasing, in contrast, can be 
done fast, so handling the decreasing case transparently is reasonable.

Eric

>
> Chuck



From ben.root at ou.edu  Tue Jun  4 22:15:21 2013
From: ben.root at ou.edu (Benjamin Root)
Date: Tue, 4 Jun 2013 22:15:21 -0400
Subject: [Numpy-discussion] suggested change of behavior for interp
In-Reply-To: <51AE8F8D.4000601@hawaii.edu>
References: <CACcRS=fPYS2z6_X6TfEXhsc5gfqdKuUZzzUMKXtL-JFQS82gFQ@mail.gmail.com>
	<CAB6mnxLEuiX7BAs21qKD4JXR2M8WBquFiXH9kxd3+O7o8asZdA@mail.gmail.com>
	<51AE8F8D.4000601@hawaii.edu>
Message-ID: <CANNq6FnmPy2bbH8A3n=PfVKqW5dCxXMa8GCsOEBPM3DEPyeTPA@mail.gmail.com>

Could non-monotonicity be detected as part of the interp process? Perhaps a
sign switch in the deltas?

I have been bitten by this problem too.

Cheers!
Ben Root

On Jun 4, 2013 9:08 PM, "Eric Firing" <efiring at hawaii.edu> wrote:
>
> On 2013/06/04 2:05 PM, Charles R Harris wrote:
> >
> >
> > On Tue, Jun 4, 2013 at 12:07 PM, Slavin, Jonathan
> > <jslavin at cfa.harvard.edu <mailto:jslavin at cfa.harvard.edu>> wrote:
> >
> >     Hi,
> >
> >     I would like to suggest that the behavior of numpy.interp be changed
> >     regarding treatment of situations in which the x-coordinates are not
> >     monotonically increasing.  Specifically, it seems to me that interp
> >     should work correctly when the x-coordinate is decreasing
> >     monotonically.  Clearly it cannot work if the x-coordinate is not
> >     monotonic, but in that case it should raise an exception.  Currently
> >     if x is not increasing it simply silently fails, providing incorrect
> >     values.  This fix could be as simple as a monotonicity test and
> >     inversion if necessary (plus a raise statement for non-monotonic
cases).
> >
> >
> > Seems reasonable, although it might add a bit of execution time.
>
> The monotonicity test should be an option if it is available at all;
> when interpolating a small number of points from a large pair of arrays,
> the single sweep through the whole array could dominate the execution
> time.  Checking for increasing versus decreasing, in contrast, can be
> done fast, so handling the decreasing case transparently is reasonable.
>
> Eric
>
> >
> > Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130604/fb8d6efa/attachment.html>

From efiring at hawaii.edu  Tue Jun  4 22:21:05 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Tue, 04 Jun 2013 16:21:05 -1000
Subject: [Numpy-discussion] suggested change of behavior for interp
In-Reply-To: <CANNq6FnmPy2bbH8A3n=PfVKqW5dCxXMa8GCsOEBPM3DEPyeTPA@mail.gmail.com>
References: <CACcRS=fPYS2z6_X6TfEXhsc5gfqdKuUZzzUMKXtL-JFQS82gFQ@mail.gmail.com>
	<CAB6mnxLEuiX7BAs21qKD4JXR2M8WBquFiXH9kxd3+O7o8asZdA@mail.gmail.com>
	<51AE8F8D.4000601@hawaii.edu>
	<CANNq6FnmPy2bbH8A3n=PfVKqW5dCxXMa8GCsOEBPM3DEPyeTPA@mail.gmail.com>
Message-ID: <51AEA091.8020507@hawaii.edu>

On 2013/06/04 4:15 PM, Benjamin Root wrote:
> Could non-monotonicity be detected as part of the interp process?
> Perhaps a sign switch in the deltas?

There are two code paths, depending on the number of points to be 
interpolated.  When it is greater than the size of the table, the deltas 
are pre-computed in a single sweep.  Non-monotonicity could be detected 
there at moderate cost.  In the other code path, for a smaller number of 
points, the deltas are computed only as needed, so monotonicity testing 
would require a separate sweep through the points.  That's the costly 
case that I think might reasonably be an option but that should not be 
required.

Eric

>
> I have been bitten by this problem too.
>
> Cheers!
> Ben Root
>
> On Jun 4, 2013 9:08 PM, "Eric Firing" <efiring at hawaii.edu
> <mailto:efiring at hawaii.edu>> wrote:
>  >
>  > On 2013/06/04 2:05 PM, Charles R Harris wrote:
>  > >
>  > >
>  > > On Tue, Jun 4, 2013 at 12:07 PM, Slavin, Jonathan
>  > > <jslavin at cfa.harvard.edu <mailto:jslavin at cfa.harvard.edu>
> <mailto:jslavin at cfa.harvard.edu <mailto:jslavin at cfa.harvard.edu>>> wrote:
>  > >
>  > >     Hi,
>  > >
>  > >     I would like to suggest that the behavior of numpy.interp be
> changed
>  > >     regarding treatment of situations in which the x-coordinates
> are not
>  > >     monotonically increasing.  Specifically, it seems to me that interp
>  > >     should work correctly when the x-coordinate is decreasing
>  > >     monotonically.  Clearly it cannot work if the x-coordinate is not
>  > >     monotonic, but in that case it should raise an exception.
>   Currently
>  > >     if x is not increasing it simply silently fails, providing
> incorrect
>  > >     values.  This fix could be as simple as a monotonicity test and
>  > >     inversion if necessary (plus a raise statement for
> non-monotonic cases).
>  > >
>  > >
>  > > Seems reasonable, although it might add a bit of execution time.
>  >
>  > The monotonicity test should be an option if it is available at all;
>  > when interpolating a small number of points from a large pair of arrays,
>  > the single sweep through the whole array could dominate the execution
>  > time.  Checking for increasing versus decreasing, in contrast, can be
>  > done fast, so handling the decreasing case transparently is reasonable.
>  >
>  > Eric
>  >
>  > >
>  > > Chuck
>  >
>  > _______________________________________________
>  > NumPy-Discussion mailing list
>  > NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>  > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



From njs at pobox.com  Wed Jun  5 03:54:53 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 5 Jun 2013 08:54:53 +0100
Subject: [Numpy-discussion] suggested change of behavior for interp
In-Reply-To: <51AEA091.8020507@hawaii.edu>
References: <CACcRS=fPYS2z6_X6TfEXhsc5gfqdKuUZzzUMKXtL-JFQS82gFQ@mail.gmail.com>
	<CAB6mnxLEuiX7BAs21qKD4JXR2M8WBquFiXH9kxd3+O7o8asZdA@mail.gmail.com>
	<51AE8F8D.4000601@hawaii.edu>
	<CANNq6FnmPy2bbH8A3n=PfVKqW5dCxXMa8GCsOEBPM3DEPyeTPA@mail.gmail.com>
	<51AEA091.8020507@hawaii.edu>
Message-ID: <CAPJVwBmNwWm5-3pQMvP9_N4uXJKhFg-B4a-AOxLvxLtXPOnogg@mail.gmail.com>

On 5 Jun 2013 03:21, "Eric Firing" <efiring at hawaii.edu> wrote:
>
> On 2013/06/04 4:15 PM, Benjamin Root wrote:
> > Could non-monotonicity be detected as part of the interp process?
> > Perhaps a sign switch in the deltas?
>
> There are two code paths, depending on the number of points to be
> interpolated.  When it is greater than the size of the table, the deltas
> are pre-computed in a single sweep.  Non-monotonicity could be detected
> there at moderate cost.  In the other code path, for a smaller number of
> points, the deltas are computed only as needed, so monotonicity testing
> would require a separate sweep through the points.  That's the costly
> case that I think might reasonably be an option but that should not be
> required.

Nonetheless, perhaps the function should at least be safe by default? I'm
worried by these multiple reports of people silently getting wrong answers
in practice...

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130605/b0e7c810/attachment.html>

From jslavin at cfa.harvard.edu  Wed Jun  5 10:16:12 2013
From: jslavin at cfa.harvard.edu (Slavin, Jonathan)
Date: Wed, 5 Jun 2013 10:16:12 -0400
Subject: [Numpy-discussion] suggested change of behavior for interp
Message-ID: <CACcRS=f1gy2J-enB5Ecif3q_fOkGN8fuQ9Bau83FizT1wR8hSA@mail.gmail.com>

The simplest monotonicity test that I've seen is:

dx = np.diff(x)
monotonic = np.all(dx < 0.) or np.all(dx > 0.)

I expect that this is pretty fast, though I haven't tested it yet.  If we
want to make checking optional, then I think the default should be to check
with the option to skip the check.

Jon

On Tue, Jun 4, 2013 at 9:03 PM, <numpy-discussion-request at scipy.org> wrote:

> From: Eric Firing <efiring at hawaii.edu>
> To: numpy-discussion at scipy.org
> Cc:
> Date: Tue, 04 Jun 2013 15:08:29 -1000
> Subject: Re: [Numpy-discussion] suggested change of behavior for interp
> On 2013/06/04 2:05 PM, Charles R Harris wrote:
>
>>
>>
>> On Tue, Jun 4, 2013 at 12:07 PM, Slavin, Jonathan
>> <jslavin at cfa.harvard.edu <mailto:jslavin at cfa.harvard.**edu<jslavin at cfa.harvard.edu>>>
>> wrote:
>>
>>     Hi,
>>
>>     I would like to suggest that the behavior of numpy.interp be changed
>>     regarding treatment of situations in which the x-coordinates are not
>>     monotonically increasing.  Specifically, it seems to me that interp
>>     should work correctly when the x-coordinate is decreasing
>>     monotonically.  Clearly it cannot work if the x-coordinate is not
>>     monotonic, but in that case it should raise an exception.  Currently
>>     if x is not increasing it simply silently fails, providing incorrect
>>     values.  This fix could be as simple as a monotonicity test and
>>     inversion if necessary (plus a raise statement for non-monotonic
>> cases).
>>
>>
>> Seems reasonable, although it might add a bit of execution time.
>>
>
> The monotonicity test should be an option if it is available at all; when
> interpolating a small number of points from a large pair of arrays, the
> single sweep through the whole array could dominate the execution time.
>  Checking for increasing versus decreasing, in contrast, can be done fast,
> so handling the decreasing case transparently is reasonable.
>
> Eric
>



________________________________________________________
Jonathan D. Slavin                 Harvard-Smithsonian CfA
jslavin at cfa.harvard.edu       60 Garden Street, MS 83
phone: (617) 496-7981       Cambridge, MA 02138-1516
fax: (617) 496-7577            USA
________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130605/e43100bb/attachment.html>

From njs at pobox.com  Wed Jun  5 10:33:55 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 5 Jun 2013 15:33:55 +0100
Subject: [Numpy-discussion] suggested change of behavior for interp
In-Reply-To: <CACcRS=f1gy2J-enB5Ecif3q_fOkGN8fuQ9Bau83FizT1wR8hSA@mail.gmail.com>
References: <CACcRS=f1gy2J-enB5Ecif3q_fOkGN8fuQ9Bau83FizT1wR8hSA@mail.gmail.com>
Message-ID: <CAPJVwB=f_gX9Qb-Piz4_078Ks7ODjtDq0ZSh65-0N-m5KmYUmQ@mail.gmail.com>

On Wed, Jun 5, 2013 at 3:16 PM, Slavin, Jonathan
<jslavin at cfa.harvard.edu> wrote:
> The simplest monotonicity test that I've seen is:
>
> dx = np.diff(x)
> monotonic = np.all(dx < 0.) or np.all(dx > 0.)
>
> I expect that this is pretty fast, though I haven't tested it yet.  If we
> want to make checking optional, then I think the default should be to check
> with the option to skip the check.

The slow down people are worried about is, suppose that 'xp' has
1,000,000 entries, and the user wants to interpolate 1 point. If we
can assume the array is sorted, then we can find which bin the 1 point
falls into by using binary search (np.searchsorted), which will
require examining ~log2(1,000,000) = 20 entries in the array. Checking
that it's sorted, though, will require examining all 1,000,000 points
-- it turns an O(log n) operation into an O(n) operation. And if this
is being called as part of some larger algorithm, this effect can
cascade -- if it gets called m times from a loop, now that's O(mn)
instead of (m log n), etc. That's often the difference between
tractable and intractable.

If you submit a PR that gives interp the option to check for
monotonicity, then we'll almost certainly merge it :-). If you also
make it the default then there might be some debate, though I'd argue
for it...

Whether interp should accept descending inputs is a separate issue and
probably more contentious; I'm weakly against it myself, as
unnecessary magic when you can just say np.interp(x, xp[::-1],
fp[::-1]).

-n


From archibald at astron.nl  Wed Jun  5 12:07:56 2013
From: archibald at astron.nl (Anne Archibald)
Date: Wed, 5 Jun 2013 18:07:56 +0200
Subject: [Numpy-discussion] supporting quad precision
Message-ID: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>

Hi folks,

I recently came across an application I needed quad precision for
(high-accuracy solution of a differential equation). I found a C++ library
(odeint) that worked for the integration itself, but unfortunately it
appears numpy is not able to work with quad precision arrays. For my
application the quad precision is only really needed for integrator state,
so I can manually convert my data to long doubles as I go to and from
numpy, but it's a pain. So quad precision support would be nice.

There's a thread discussing quad support:
http://mail.scipy.org/pipermail/numpy-discussion/2012-February/061080.html
Essentially, there isn't any, but since gcc >= 4.6 supports them on Intel
hardware (in software), it should be possible. (Then the thread got bogged
down in bike-shedding about what to call them.)

What would it take to support quads in numpy? I looked into the numpy base
dtype definitions, and it's a complex arrangement involving detection of
platform support and templatized C code; in the end I couldn't really see
where to start. But it seems to me all the basics are available: native C
syntax for basic arithmetic, "qabs"-style versions of all the basic
functions, NaNs and Infs. So how would one go about adding quad support?

Thanks,
Anne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130605/0427efe4/attachment.html>

From charlesr.harris at gmail.com  Wed Jun  5 12:21:29 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 5 Jun 2013 10:21:29 -0600
Subject: [Numpy-discussion] supporting quad precision
In-Reply-To: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>
References: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>
Message-ID: <CAB6mnxKzCGEz2=iBJrX0dHqcfXdugqRWHTVXOJ24KjUCBONTTw@mail.gmail.com>

Hi Anne,

Long time no see ;)

On Wed, Jun 5, 2013 at 10:07 AM, Anne Archibald <archibald at astron.nl> wrote:

> Hi folks,
>
> I recently came across an application I needed quad precision for
> (high-accuracy solution of a differential equation). I found a C++ library
> (odeint) that worked for the integration itself, but unfortunately it
> appears numpy is not able to work with quad precision arrays. For my
> application the quad precision is only really needed for integrator state,
> so I can manually convert my data to long doubles as I go to and from
> numpy, but it's a pain. So quad precision support would be nice.
>
> There's a thread discussing quad support:
> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/061080.html
> Essentially, there isn't any, but since gcc >= 4.6 supports them on Intel
> hardware (in software), it should be possible. (Then the thread got bogged
> down in bike-shedding about what to call them.)
>
> What would it take to support quads in numpy? I looked into the numpy base
> dtype definitions, and it's a complex arrangement involving detection of
> platform support and templatized C code; in the end I couldn't really see
> where to start. But it seems to me all the basics are available: native C
> syntax for basic arithmetic, "qabs"-style versions of all the basic
> functions, NaNs and Infs. So how would one go about adding quad support?
>
>
There are some improvements for user types committed in 1.8-dev. Perhaps
quad support could be added as a user type as it is still platform/compiler
dependent. The rational type added to numpy could supply a template for
adding the new type.

Long term, we need to have quad support. If you run on SUN hardware I think
it is already available as the extended precision type, but that doesn't
help the majority of users at this point and I don't think LAPACK/BLAS
supports it.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130605/82b37d39/attachment.html>

From rainexpected at theo.to  Wed Jun  5 12:52:11 2013
From: rainexpected at theo.to (Ted To)
Date: Wed, 05 Jun 2013 12:52:11 -0400
Subject: [Numpy-discussion] genfromtxt and gzip
Message-ID: <51AF6CBB.4020606@theo.to>

Hi all,

>From the list archives (2011), I noticed that there is a bug in the
python gzip module that causes genfromtxt to fail with python 2 but this
bug is not a problem for python 3.  When I tried to use genfromtxt and
python 3 with a gzip'ed csv file, I instead got:

IOError: Mode rbU not supported

Is this a bug?  I am using python 3.2.3 and numpy 1.7.1 from the
experimental Debian repository.

Thanks!


From jtaylor.debian at googlemail.com  Wed Jun  5 13:08:41 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Wed, 05 Jun 2013 19:08:41 +0200
Subject: [Numpy-discussion] suggested change of behavior for interp
In-Reply-To: <CAPJVwB=f_gX9Qb-Piz4_078Ks7ODjtDq0ZSh65-0N-m5KmYUmQ@mail.gmail.com>
References: <CACcRS=f1gy2J-enB5Ecif3q_fOkGN8fuQ9Bau83FizT1wR8hSA@mail.gmail.com>
	<CAPJVwB=f_gX9Qb-Piz4_078Ks7ODjtDq0ZSh65-0N-m5KmYUmQ@mail.gmail.com>
Message-ID: <51AF7099.2010200@googlemail.com>

On 05.06.2013 16:33, Nathaniel Smith wrote:
> On Wed, Jun 5, 2013 at 3:16 PM, Slavin, Jonathan
> <jslavin at cfa.harvard.edu> wrote:
>> The simplest monotonicity test that I've seen is:
>>
>> dx = np.diff(x)
>> monotonic = np.all(dx < 0.) or np.all(dx > 0.)
>>
>> I expect that this is pretty fast, though I haven't tested it yet.  If we
>> want to make checking optional, then I think the default should be to check
>> with the option to skip the check.
> 
> The slow down people are worried about is, suppose that 'xp' has
> 1,000,000 entries, and the user wants to interpolate 1 point. If we
> can assume the array is sorted, then we can find which bin the 1 point
> falls into by using binary search (np.searchsorted), which will
> require examining ~log2(1,000,000) = 20 entries in the array. Checking
> that it's sorted, though, will require examining all 1,000,000 points
> -- it turns an O(log n) operation into an O(n) operation. And if this
> is being called as part of some larger algorithm, this effect can
> cascade -- if it gets called m times from a loop, now that's O(mn)
> instead of (m log n), etc. That's often the difference between
> tractable and intractable.
> 
> If you submit a PR that gives interp the option to check for
> monotonicity, then we'll almost certainly merge it :-). If you also
> make it the default then there might be some debate, though I'd argue
> for it...

I would not like it when the performance goes from log to linear by default.
It is documented that the coordinates must be increasing after all.

How about as a compromise the function checks one or two closest
neighbors around the interpolation point and warns/errors if those are
not monotonic.

Its not fool proof but should at least catch the very wrong cases with
an acceptable performance loss.


From cournape at gmail.com  Wed Jun  5 13:10:30 2013
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 5 Jun 2013 18:10:30 +0100
Subject: [Numpy-discussion] supporting quad precision
In-Reply-To: <CAB6mnxKzCGEz2=iBJrX0dHqcfXdugqRWHTVXOJ24KjUCBONTTw@mail.gmail.com>
References: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>
	<CAB6mnxKzCGEz2=iBJrX0dHqcfXdugqRWHTVXOJ24KjUCBONTTw@mail.gmail.com>
Message-ID: <CAGY4rcV2_-HAzx-Wgfp96yHKKkjizx0UORXV1EWXtQ-NxijGbA@mail.gmail.com>

On Wed, Jun 5, 2013 at 5:21 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> Hi Anne,
>
> Long time no see ;)
>
> On Wed, Jun 5, 2013 at 10:07 AM, Anne Archibald <archibald at astron.nl> wrote:
>>
>> Hi folks,
>>
>> I recently came across an application I needed quad precision for
>> (high-accuracy solution of a differential equation). I found a C++ library
>> (odeint) that worked for the integration itself, but unfortunately it
>> appears numpy is not able to work with quad precision arrays. For my
>> application the quad precision is only really needed for integrator state,
>> so I can manually convert my data to long doubles as I go to and from numpy,
>> but it's a pain. So quad precision support would be nice.
>>
>> There's a thread discussing quad support:
>> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/061080.html
>> Essentially, there isn't any, but since gcc >= 4.6 supports them on Intel
>> hardware (in software), it should be possible. (Then the thread got bogged
>> down in bike-shedding about what to call them.)
>>
>> What would it take to support quads in numpy? I looked into the numpy base
>> dtype definitions, and it's a complex arrangement involving detection of
>> platform support and templatized C code; in the end I couldn't really see
>> where to start. But it seems to me all the basics are available: native C
>> syntax for basic arithmetic, "qabs"-style versions of all the basic
>> functions, NaNs and Infs. So how would one go about adding quad support?
>>
>
> There are some improvements for user types committed in 1.8-dev. Perhaps
> quad support could be added as a user type as it is still platform/compiler
> dependent. The rational type added to numpy could supply a template for
> adding the new type.

I would be in support of that direction as well: let it live
separately until CPU/compiler support is coming up.

Anne, will you be at scipy conference ? Improving user data type
internal API is something I'd like to work on as well

David


From njs at pobox.com  Wed Jun  5 13:48:21 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 5 Jun 2013 18:48:21 +0100
Subject: [Numpy-discussion] suggested change of behavior for interp
In-Reply-To: <51AF7099.2010200@googlemail.com>
References: <CACcRS=f1gy2J-enB5Ecif3q_fOkGN8fuQ9Bau83FizT1wR8hSA@mail.gmail.com>
	<CAPJVwB=f_gX9Qb-Piz4_078Ks7ODjtDq0ZSh65-0N-m5KmYUmQ@mail.gmail.com>
	<51AF7099.2010200@googlemail.com>
Message-ID: <CAPJVwBnTmoDh9MiXR+haJaELGg0dBdk2yeKvf2k64cWPurUo0w@mail.gmail.com>

On Wed, Jun 5, 2013 at 6:08 PM, Julian Taylor
<jtaylor.debian at googlemail.com> wrote:
> On 05.06.2013 16:33, Nathaniel Smith wrote:
>> The slow down people are worried about is, suppose that 'xp' has
>> 1,000,000 entries, and the user wants to interpolate 1 point. If we
>> can assume the array is sorted, then we can find which bin the 1 point
>> falls into by using binary search (np.searchsorted), which will
>> require examining ~log2(1,000,000) = 20 entries in the array. Checking
>> that it's sorted, though, will require examining all 1,000,000 points
>> -- it turns an O(log n) operation into an O(n) operation. And if this
>> is being called as part of some larger algorithm, this effect can
>> cascade -- if it gets called m times from a loop, now that's O(mn)
>> instead of (m log n), etc. That's often the difference between
>> tractable and intractable.
>>
>> If you submit a PR that gives interp the option to check for
>> monotonicity, then we'll almost certainly merge it :-). If you also
>> make it the default then there might be some debate, though I'd argue
>> for it...
>
> I would not like it when the performance goes from log to linear by default.
> It is documented that the coordinates must be increasing after all.

I agree, I don't like it either. But the problem is there are two
contradictory things and I don't like either of them:
1) performance going from log to linear by default (for a subset of
somewhat extreme cases)
2) people silently getting the wrong results (and according to reports
in this thread, the warning in the docs does not actually prevent
this)

The question is which one do we dislike more. My feeling is that in
the cases where (1) comes up, it will annoy people and get them to
check the docs and find the go-faster switch, while the first warning
of (2) is when your spaceship crashes, so we should worry more about
(2).

> How about as a compromise the function checks one or two closest
> neighbors around the interpolation point and warns/errors if those are
> not monotonic.
>
> Its not fool proof but should at least catch the very wrong cases with
> an acceptable performance loss.

There are tons of ways for data to accidentally end up sorted within
each local region, but unsorted overall, e.g., if you're combining
results from parallel simulation runs. Such data would systematically
pass this check, but still give nonsensical answers.

-n


From charlesr.harris at gmail.com  Wed Jun  5 13:59:09 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 5 Jun 2013 11:59:09 -0600
Subject: [Numpy-discussion] suggested change of behavior for interp
In-Reply-To: <CAPJVwBnTmoDh9MiXR+haJaELGg0dBdk2yeKvf2k64cWPurUo0w@mail.gmail.com>
References: <CACcRS=f1gy2J-enB5Ecif3q_fOkGN8fuQ9Bau83FizT1wR8hSA@mail.gmail.com>
	<CAPJVwB=f_gX9Qb-Piz4_078Ks7ODjtDq0ZSh65-0N-m5KmYUmQ@mail.gmail.com>
	<51AF7099.2010200@googlemail.com>
	<CAPJVwBnTmoDh9MiXR+haJaELGg0dBdk2yeKvf2k64cWPurUo0w@mail.gmail.com>
Message-ID: <CAB6mnxKmLWLkHnJA9=ML8fLOm-YhBg7AMAvGvCJrOo1yRDg+CA@mail.gmail.com>

On Wed, Jun 5, 2013 at 11:48 AM, Nathaniel Smith <njs at pobox.com> wrote:

> On Wed, Jun 5, 2013 at 6:08 PM, Julian Taylor
> <jtaylor.debian at googlemail.com> wrote:
> > On 05.06.2013 16:33, Nathaniel Smith wrote:
> >> The slow down people are worried about is, suppose that 'xp' has
> >> 1,000,000 entries, and the user wants to interpolate 1 point. If we
> >> can assume the array is sorted, then we can find which bin the 1 point
> >> falls into by using binary search (np.searchsorted), which will
> >> require examining ~log2(1,000,000) = 20 entries in the array. Checking
> >> that it's sorted, though, will require examining all 1,000,000 points
> >> -- it turns an O(log n) operation into an O(n) operation. And if this
> >> is being called as part of some larger algorithm, this effect can
> >> cascade -- if it gets called m times from a loop, now that's O(mn)
> >> instead of (m log n), etc. That's often the difference between
> >> tractable and intractable.
> >>
> >> If you submit a PR that gives interp the option to check for
> >> monotonicity, then we'll almost certainly merge it :-). If you also
> >> make it the default then there might be some debate, though I'd argue
> >> for it...
> >
> > I would not like it when the performance goes from log to linear by
> default.
> > It is documented that the coordinates must be increasing after all.
>
> I agree, I don't like it either. But the problem is there are two
> contradictory things and I don't like either of them:
> 1) performance going from log to linear by default (for a subset of
> somewhat extreme cases)
> 2) people silently getting the wrong results (and according to reports
> in this thread, the warning in the docs does not actually prevent
> this)
>
> The question is which one do we dislike more. My feeling is that in
> the cases where (1) comes up, it will annoy people and get them to
> check the docs and find the go-faster switch, while the first warning
> of (2) is when your spaceship crashes, so we should worry more about
> (2).
>
> > How about as a compromise the function checks one or two closest
> > neighbors around the interpolation point and warns/errors if those are
> > not monotonic.
> >
> > Its not fool proof but should at least catch the very wrong cases with
> > an acceptable performance loss.
>
> There are tons of ways for data to accidentally end up sorted within
> each local region, but unsorted overall, e.g., if you're combining
> results from parallel simulation runs. Such data would systematically
> pass this check, but still give nonsensical answers.
>

Some actual benchmarks might be useful to the discussion.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130605/8c3c674d/attachment.html>

From charlesr.harris at gmail.com  Wed Jun  5 14:00:57 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 5 Jun 2013 12:00:57 -0600
Subject: [Numpy-discussion] suggested change of behavior for interp
In-Reply-To: <CAB6mnxKmLWLkHnJA9=ML8fLOm-YhBg7AMAvGvCJrOo1yRDg+CA@mail.gmail.com>
References: <CACcRS=f1gy2J-enB5Ecif3q_fOkGN8fuQ9Bau83FizT1wR8hSA@mail.gmail.com>
	<CAPJVwB=f_gX9Qb-Piz4_078Ks7ODjtDq0ZSh65-0N-m5KmYUmQ@mail.gmail.com>
	<51AF7099.2010200@googlemail.com>
	<CAPJVwBnTmoDh9MiXR+haJaELGg0dBdk2yeKvf2k64cWPurUo0w@mail.gmail.com>
	<CAB6mnxKmLWLkHnJA9=ML8fLOm-YhBg7AMAvGvCJrOo1yRDg+CA@mail.gmail.com>
Message-ID: <CAB6mnxJyPTYzJ=ZXQi3EYA6q3nL+PSd4YWmUacKq6GJ+NL_8GQ@mail.gmail.com>

On Wed, Jun 5, 2013 at 11:59 AM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Wed, Jun 5, 2013 at 11:48 AM, Nathaniel Smith <njs at pobox.com> wrote:
>
>> On Wed, Jun 5, 2013 at 6:08 PM, Julian Taylor
>> <jtaylor.debian at googlemail.com> wrote:
>> > On 05.06.2013 16:33, Nathaniel Smith wrote:
>> >> The slow down people are worried about is, suppose that 'xp' has
>> >> 1,000,000 entries, and the user wants to interpolate 1 point. If we
>> >> can assume the array is sorted, then we can find which bin the 1 point
>> >> falls into by using binary search (np.searchsorted), which will
>> >> require examining ~log2(1,000,000) = 20 entries in the array. Checking
>> >> that it's sorted, though, will require examining all 1,000,000 points
>> >> -- it turns an O(log n) operation into an O(n) operation. And if this
>> >> is being called as part of some larger algorithm, this effect can
>> >> cascade -- if it gets called m times from a loop, now that's O(mn)
>> >> instead of (m log n), etc. That's often the difference between
>> >> tractable and intractable.
>> >>
>> >> If you submit a PR that gives interp the option to check for
>> >> monotonicity, then we'll almost certainly merge it :-). If you also
>> >> make it the default then there might be some debate, though I'd argue
>> >> for it...
>> >
>> > I would not like it when the performance goes from log to linear by
>> default.
>> > It is documented that the coordinates must be increasing after all.
>>
>> I agree, I don't like it either. But the problem is there are two
>> contradictory things and I don't like either of them:
>> 1) performance going from log to linear by default (for a subset of
>> somewhat extreme cases)
>> 2) people silently getting the wrong results (and according to reports
>> in this thread, the warning in the docs does not actually prevent
>> this)
>>
>> The question is which one do we dislike more. My feeling is that in
>> the cases where (1) comes up, it will annoy people and get them to
>> check the docs and find the go-faster switch, while the first warning
>> of (2) is when your spaceship crashes, so we should worry more about
>> (2).
>>
>> > How about as a compromise the function checks one or two closest
>> > neighbors around the interpolation point and warns/errors if those are
>> > not monotonic.
>> >
>> > Its not fool proof but should at least catch the very wrong cases with
>> > an acceptable performance loss.
>>
>> There are tons of ways for data to accidentally end up sorted within
>> each local region, but unsorted overall, e.g., if you're combining
>> results from parallel simulation runs. Such data would systematically
>> pass this check, but still give nonsensical answers.
>>
>
> Some actual benchmarks might be useful to the discussion.
>

And perhaps an inplace C function ismonotone would be generally useful.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130605/101aa11a/attachment.html>

From charlesr.harris at gmail.com  Wed Jun  5 14:08:45 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 5 Jun 2013 12:08:45 -0600
Subject: [Numpy-discussion] suggested change of behavior for interp
In-Reply-To: <CAB6mnxJyPTYzJ=ZXQi3EYA6q3nL+PSd4YWmUacKq6GJ+NL_8GQ@mail.gmail.com>
References: <CACcRS=f1gy2J-enB5Ecif3q_fOkGN8fuQ9Bau83FizT1wR8hSA@mail.gmail.com>
	<CAPJVwB=f_gX9Qb-Piz4_078Ks7ODjtDq0ZSh65-0N-m5KmYUmQ@mail.gmail.com>
	<51AF7099.2010200@googlemail.com>
	<CAPJVwBnTmoDh9MiXR+haJaELGg0dBdk2yeKvf2k64cWPurUo0w@mail.gmail.com>
	<CAB6mnxKmLWLkHnJA9=ML8fLOm-YhBg7AMAvGvCJrOo1yRDg+CA@mail.gmail.com>
	<CAB6mnxJyPTYzJ=ZXQi3EYA6q3nL+PSd4YWmUacKq6GJ+NL_8GQ@mail.gmail.com>
Message-ID: <CAB6mnxLB185Kt-zrUtWu-RcGfcBZ7=AncvARAu_yWvdX4OjJfw@mail.gmail.com>

On Wed, Jun 5, 2013 at 12:00 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Wed, Jun 5, 2013 at 11:59 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Wed, Jun 5, 2013 at 11:48 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>>> On Wed, Jun 5, 2013 at 6:08 PM, Julian Taylor
>>> <jtaylor.debian at googlemail.com> wrote:
>>> > On 05.06.2013 16:33, Nathaniel Smith wrote:
>>> >> The slow down people are worried about is, suppose that 'xp' has
>>> >> 1,000,000 entries, and the user wants to interpolate 1 point. If we
>>> >> can assume the array is sorted, then we can find which bin the 1 point
>>> >> falls into by using binary search (np.searchsorted), which will
>>> >> require examining ~log2(1,000,000) = 20 entries in the array. Checking
>>> >> that it's sorted, though, will require examining all 1,000,000 points
>>> >> -- it turns an O(log n) operation into an O(n) operation. And if this
>>> >> is being called as part of some larger algorithm, this effect can
>>> >> cascade -- if it gets called m times from a loop, now that's O(mn)
>>> >> instead of (m log n), etc. That's often the difference between
>>> >> tractable and intractable.
>>> >>
>>> >> If you submit a PR that gives interp the option to check for
>>> >> monotonicity, then we'll almost certainly merge it :-). If you also
>>> >> make it the default then there might be some debate, though I'd argue
>>> >> for it...
>>> >
>>> > I would not like it when the performance goes from log to linear by
>>> default.
>>> > It is documented that the coordinates must be increasing after all.
>>>
>>> I agree, I don't like it either. But the problem is there are two
>>> contradictory things and I don't like either of them:
>>> 1) performance going from log to linear by default (for a subset of
>>> somewhat extreme cases)
>>> 2) people silently getting the wrong results (and according to reports
>>> in this thread, the warning in the docs does not actually prevent
>>> this)
>>>
>>> The question is which one do we dislike more. My feeling is that in
>>> the cases where (1) comes up, it will annoy people and get them to
>>> check the docs and find the go-faster switch, while the first warning
>>> of (2) is when your spaceship crashes, so we should worry more about
>>> (2).
>>>
>>> > How about as a compromise the function checks one or two closest
>>> > neighbors around the interpolation point and warns/errors if those are
>>> > not monotonic.
>>> >
>>> > Its not fool proof but should at least catch the very wrong cases with
>>> > an acceptable performance loss.
>>>
>>> There are tons of ways for data to accidentally end up sorted within
>>> each local region, but unsorted overall, e.g., if you're combining
>>> results from parallel simulation runs. Such data would systematically
>>> pass this check, but still give nonsensical answers.
>>>
>>
>> Some actual benchmarks might be useful to the discussion.
>>
>
> And perhaps an inplace C function ismonotone would be generally useful.
>

Or make greater.reduce operate correctly.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130605/5be3a0f6/attachment.html>

From jkmacc at lanl.gov  Thu Jun  6 15:48:15 2013
From: jkmacc at lanl.gov (Maccarthy, Jonathan K)
Date: Thu, 6 Jun 2013 19:48:15 +0000
Subject: [Numpy-discussion] floats coerced to string with "{:f}".format() ?
Message-ID: <26E44CF1-EE48-4F9F-9768-9AE47DE08C45@lanl.gov>

Hi everyone,

I've looked in the mailing list archives and with the googles, but haven't yet found any hints with this question...

I have a float field in a NumPy record that looks like it's being substituted as a string in the Python "{:f}".format() mini-language, thus throwing an error:


In [1]: tmp = np.rec.array([('XYZZ', 2001123, -23.82396)], dtype=np.dtype([('sta', '|S6'), ('ondate', '<i8'), ('lat', '<f4')]))[0]

In [2]: type(tmp)
Out[3]: numpy.core.records.record

In [3]: tmp
Out[3]: ('XYZZ', 2001123, -23.823917388916016)

In [4]: tmp.sta, tmp.ondate, tmp.lat
Out[4]: ('XYZZ', 2001123, -23.823917)

# strings and integers work
In [5]: '{0.sta:6.6s} {0.ondate:8d}'.format(tmp)
Out[5]: 'XYZZ    2001123'

# "lat" is a float, but it seems to be coerced to a string first, and failing
In [6]: '{0.sta:6.6s} {0.ondate:8d} {0.lat:11.6f}'.format(tmp)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/Users/jkmacc/<ipython-input-312-bff8066cfde8> in <module>()
----> 1 '{0.sta:6.6s} {0.ondate:8d} {0.lat:>11.6f}'.format(tmp)

ValueError: Unknown format code 'f' for object of type 'str'

# string formatting for doesn't fail
In [7]: '{0.sta:6.6s} {0.ondate:8d} {0.lat:>11.6s}'.format(tmp)
Out[7]: 'XYZZ    2001123      -23.82'


This also fails:

In [7]: "{:f}".format(np.array(3.2, dtype='f4'))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/Users/jkmacc/<ipython-input-314-33119128e3e6> in <module>()
----> 1 "{:f}".format(np.array(3.2, dtype='f4'))

ValueError: Unknown format code 'f' for object of type 'str'



Does anyone understand what's happening?

Thanks for your help.

Best,
Jon

From ben.root at ou.edu  Thu Jun  6 15:53:50 2013
From: ben.root at ou.edu (Benjamin Root)
Date: Thu, 6 Jun 2013 15:53:50 -0400
Subject: [Numpy-discussion] floats coerced to string with
	"{:f}".format() ?
In-Reply-To: <26E44CF1-EE48-4F9F-9768-9AE47DE08C45@lanl.gov>
References: <26E44CF1-EE48-4F9F-9768-9AE47DE08C45@lanl.gov>
Message-ID: <CANNq6F=bb6JBZaK7=BDt+L5OCXQA8S_r8MTCfx7J_4TyMa+c3A@mail.gmail.com>

You can treat a record in a record array like a tuple or a dictionary when
it comes to formatting.  So, either refer to the index element you want
formatted as a float, or refer to it by name (in the formatting language).
By just doing {:f}, you are just grabbing the first one, which is "XXYYZZ"
and trying to format that.  But remember, you can only do this to a single
record at a time, not the entire record array at once.

Regular numpy arrays can not be formatted in this manner, hence your other
attempt failures.

Cheers!
Ben Root


On Thu, Jun 6, 2013 at 3:48 PM, Maccarthy, Jonathan K <jkmacc at lanl.gov>wrote:

> Hi everyone,
>
> I've looked in the mailing list archives and with the googles, but haven't
> yet found any hints with this question...
>
> I have a float field in a NumPy record that looks like it's being
> substituted as a string in the Python "{:f}".format() mini-language, thus
> throwing an error:
>
>
> In [1]: tmp = np.rec.array([('XYZZ', 2001123, -23.82396)],
> dtype=np.dtype([('sta', '|S6'), ('ondate', '<i8'), ('lat', '<f4')]))[0]
>
> In [2]: type(tmp)
> Out[3]: numpy.core.records.record
>
> In [3]: tmp
> Out[3]: ('XYZZ', 2001123, -23.823917388916016)
>
> In [4]: tmp.sta, tmp.ondate, tmp.lat
> Out[4]: ('XYZZ', 2001123, -23.823917)
>
> # strings and integers work
> In [5]: '{0.sta:6.6s} {0.ondate:8d}'.format(tmp)
> Out[5]: 'XYZZ    2001123'
>
> # "lat" is a float, but it seems to be coerced to a string first, and
> failing
> In [6]: '{0.sta:6.6s} {0.ondate:8d} {0.lat:11.6f}'.format(tmp)
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call last)
> /Users/jkmacc/<ipython-input-312-bff8066cfde8> in <module>()
> ----> 1 '{0.sta:6.6s} {0.ondate:8d} {0.lat:>11.6f}'.format(tmp)
>
> ValueError: Unknown format code 'f' for object of type 'str'
>
> # string formatting for doesn't fail
> In [7]: '{0.sta:6.6s} {0.ondate:8d} {0.lat:>11.6s}'.format(tmp)
> Out[7]: 'XYZZ    2001123      -23.82'
>
>
> This also fails:
>
> In [7]: "{:f}".format(np.array(3.2, dtype='f4'))
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call last)
> /Users/jkmacc/<ipython-input-314-33119128e3e6> in <module>()
> ----> 1 "{:f}".format(np.array(3.2, dtype='f4'))
>
> ValueError: Unknown format code 'f' for object of type 'str'
>
>
>
> Does anyone understand what's happening?
>
> Thanks for your help.
>
> Best,
> Jon
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130606/5746cab8/attachment.html>

From monocongo at gmail.com  Thu Jun  6 15:56:06 2013
From: monocongo at gmail.com (James Adams)
Date: Thu, 6 Jun 2013 15:56:06 -0400
Subject: [Numpy-discussion] ndarray: How to create and initialize with a
	value other than ones or zeros?
Message-ID: <CACafSVmfWrS6gcGy1wNjMaGXCdE-vjkE7fx332MUGAu8MZHFbw@mail.gmail.com>

I would like to create an array object and initialize the array's
values with an arbitrary fill value, like you can do using the ones()
and zeros() creation routines to create and initialize arrays with
ones or zeros.  Is there an easy way to do this?  If this isn't
possible then what is the most efficient way to initialize a numpy
array with an arbitrary fill value?

In order to provide such an array creation routine I can imagine that
it'd be as simple as taking the code for ones() and/or zeros() and
modifying that code so that it provides an additional fill value
argument and then within the section which does the initialization of
the array it could use that fill value instead of 1 or 0.  Is this a
naive assumption?

Thanks in advance for your help with this issue.

--James


From robert.kern at gmail.com  Thu Jun  6 15:56:03 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 6 Jun 2013 20:56:03 +0100
Subject: [Numpy-discussion] floats coerced to string with
	"{:f}".format() ?
In-Reply-To: <26E44CF1-EE48-4F9F-9768-9AE47DE08C45@lanl.gov>
References: <26E44CF1-EE48-4F9F-9768-9AE47DE08C45@lanl.gov>
Message-ID: <CAF6FJisYyDj8aEkqVFoTxete63gTY_ZzAKswm2D0d-zovfserA@mail.gmail.com>

On Thu, Jun 6, 2013 at 8:48 PM, Maccarthy, Jonathan K <jkmacc at lanl.gov> wrote:
> Hi everyone,
>
> I've looked in the mailing list archives and with the googles, but haven't yet found any hints with this question...
>
> I have a float field in a NumPy record that looks like it's being substituted as a string in the Python "{:f}".format() mini-language, thus throwing an error:
>
>
> In [1]: tmp = np.rec.array([('XYZZ', 2001123, -23.82396)], dtype=np.dtype([('sta', '|S6'), ('ondate', '<i8'), ('lat', '<f4')]))[0]
>
> In [2]: type(tmp)
> Out[3]: numpy.core.records.record
>
> In [3]: tmp
> Out[3]: ('XYZZ', 2001123, -23.823917388916016)
>
> In [4]: tmp.sta, tmp.ondate, tmp.lat
> Out[4]: ('XYZZ', 2001123, -23.823917)
>
> # strings and integers work
> In [5]: '{0.sta:6.6s} {0.ondate:8d}'.format(tmp)
> Out[5]: 'XYZZ    2001123'
>
> # "lat" is a float, but it seems to be coerced to a string first, and failing
> In [6]: '{0.sta:6.6s} {0.ondate:8d} {0.lat:11.6f}'.format(tmp)
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call last)
> /Users/jkmacc/<ipython-input-312-bff8066cfde8> in <module>()
> ----> 1 '{0.sta:6.6s} {0.ondate:8d} {0.lat:>11.6f}'.format(tmp)
>
> ValueError: Unknown format code 'f' for object of type 'str'
>
> # string formatting for doesn't fail
> In [7]: '{0.sta:6.6s} {0.ondate:8d} {0.lat:>11.6s}'.format(tmp)
> Out[7]: 'XYZZ    2001123      -23.82'
>
>
> This also fails:
>
> In [7]: "{:f}".format(np.array(3.2, dtype='f4'))
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call last)
> /Users/jkmacc/<ipython-input-314-33119128e3e6> in <module>()
> ----> 1 "{:f}".format(np.array(3.2, dtype='f4'))
>
> ValueError: Unknown format code 'f' for object of type 'str'
>
>
>
> Does anyone understand what's happening?

numpy.ndarray does not implement the __format__() method. Thus,
str.format() method falls back to object.__format__(). This is the
exception that object.__format__() raises. Why it says "object of type
'str'" is not clear to me. Similarly, numpy.float32 does not implement
the __format__() method. The string scalar type and the native integer
scalar type (I assume you are on a 64-bit platform, so Python ints are
64-bit to match your 'i8' field) inherit from the corresponding native
Python types, so they inherit their __format__() methods.

--
Robert Kern


From robert.kern at gmail.com  Thu Jun  6 16:03:57 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 6 Jun 2013 21:03:57 +0100
Subject: [Numpy-discussion] ndarray: How to create and initialize with a
 value other than ones or zeros?
In-Reply-To: <CACafSVmfWrS6gcGy1wNjMaGXCdE-vjkE7fx332MUGAu8MZHFbw@mail.gmail.com>
References: <CACafSVmfWrS6gcGy1wNjMaGXCdE-vjkE7fx332MUGAu8MZHFbw@mail.gmail.com>
Message-ID: <CAF6FJitcwq1WpxZyd1TGY_skTYmBTGVdZgv=JKGr0YT0B4QJJQ@mail.gmail.com>

On Thu, Jun 6, 2013 at 8:56 PM, James Adams <monocongo at gmail.com> wrote:
> I would like to create an array object and initialize the array's
> values with an arbitrary fill value, like you can do using the ones()
> and zeros() creation routines to create and initialize arrays with
> ones or zeros.  Is there an easy way to do this?  If this isn't
> possible then what is the most efficient way to initialize a numpy
> array with an arbitrary fill value?
>
> In order to provide such an array creation routine I can imagine that
> it'd be as simple as taking the code for ones() and/or zeros() and
> modifying that code so that it provides an additional fill value
> argument and then within the section which does the initialization of
> the array it could use that fill value instead of 1 or 0.  Is this a
> naive assumption?

Basically, yes. They both boil down to this:

  x = np.empty(...)
  x.fill(arbitrary_fill_value)

With just a little bit of extra help for structured dtypes (which is
relevant for zeros() but not much for you, I don't think).

--
Robert Kern

On Thu, Jun 6, 2013 at 8:56 PM, James Adams <monocongo at gmail.com> wrote:
> I would like to create an array object and initialize the array's
> values with an arbitrary fill value, like you can do using the ones()
> and zeros() creation routines to create and initialize arrays with
> ones or zeros.  Is there an easy way to do this?  If this isn't
> possible then what is the most efficient way to initialize a numpy
> array with an arbitrary fill value?
>
> In order to provide such an array creation routine I can imagine that
> it'd be as simple as taking the code for ones() and/or zeros() and
> modifying that code so that it provides an additional fill value
> argument and then within the section which does the initialization of
> the array it could use that fill value instead of 1 or 0.  Is this a
> naive assumption?
>
> Thanks in advance for your help with this issue.
>
> --James
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



-- 
Robert Kern


From jkmacc at lanl.gov  Thu Jun  6 16:18:20 2013
From: jkmacc at lanl.gov (Maccarthy, Jonathan K)
Date: Thu, 6 Jun 2013 20:18:20 +0000
Subject: [Numpy-discussion] floats coerced to string
	with	"{:f}".format() ?
In-Reply-To: <CAF6FJisYyDj8aEkqVFoTxete63gTY_ZzAKswm2D0d-zovfserA@mail.gmail.com>
References: <26E44CF1-EE48-4F9F-9768-9AE47DE08C45@lanl.gov>
	<CAF6FJisYyDj8aEkqVFoTxete63gTY_ZzAKswm2D0d-zovfserA@mail.gmail.com>
Message-ID: <CA9D2A2B-78DC-487F-9A25-B32BBAEA85E1@lanl.gov>

Ah, so np.int64 and np.str inherit the native Python __format__(), but np.float32/64 doesn't get __builtin__.float.__format__().  That's not intuitive, but I see now why this works:

In [8]: '{:6.6s} {:8d} {:11.6f}'.format(tmp.sta, tmp.ondate, float(tmp.lat))
Out[8]: 'XYZZ    2001123  -23.820000'

Thanks!

-Jon

On Jun 6, 2013, at 1:56 PM, Robert Kern wrote:

> On Thu, Jun 6, 2013 at 8:48 PM, Maccarthy, Jonathan K <jkmacc at lanl.gov> wrote:
>> Hi everyone,
>> 
>> I've looked in the mailing list archives and with the googles, but haven't yet found any hints with this question...
>> 
>> I have a float field in a NumPy record that looks like it's being substituted as a string in the Python "{:f}".format() mini-language, thus throwing an error:
>> 
>> 
>> In [1]: tmp = np.rec.array([('XYZZ', 2001123, -23.82396)], dtype=np.dtype([('sta', '|S6'), ('ondate', '<i8'), ('lat', '<f4')]))[0]
>> 
>> In [2]: type(tmp)
>> Out[3]: numpy.core.records.record
>> 
>> In [3]: tmp
>> Out[3]: ('XYZZ', 2001123, -23.823917388916016)
>> 
>> In [4]: tmp.sta, tmp.ondate, tmp.lat
>> Out[4]: ('XYZZ', 2001123, -23.823917)
>> 
>> # strings and integers work
>> In [5]: '{0.sta:6.6s} {0.ondate:8d}'.format(tmp)
>> Out[5]: 'XYZZ    2001123'
>> 
>> # "lat" is a float, but it seems to be coerced to a string first, and failing
>> In [6]: '{0.sta:6.6s} {0.ondate:8d} {0.lat:11.6f}'.format(tmp)
>> ---------------------------------------------------------------------------
>> ValueError                                Traceback (most recent call last)
>> /Users/jkmacc/<ipython-input-312-bff8066cfde8> in <module>()
>> ----> 1 '{0.sta:6.6s} {0.ondate:8d} {0.lat:>11.6f}'.format(tmp)
>> 
>> ValueError: Unknown format code 'f' for object of type 'str'
>> 
>> # string formatting for doesn't fail
>> In [7]: '{0.sta:6.6s} {0.ondate:8d} {0.lat:>11.6s}'.format(tmp)
>> Out[7]: 'XYZZ    2001123      -23.82'
>> 
>> 
>> This also fails:
>> 
>> In [7]: "{:f}".format(np.array(3.2, dtype='f4'))
>> ---------------------------------------------------------------------------
>> ValueError                                Traceback (most recent call last)
>> /Users/jkmacc/<ipython-input-314-33119128e3e6> in <module>()
>> ----> 1 "{:f}".format(np.array(3.2, dtype='f4'))
>> 
>> ValueError: Unknown format code 'f' for object of type 'str'
>> 
>> 
>> 
>> Does anyone understand what's happening?
> 
> numpy.ndarray does not implement the __format__() method. Thus,
> str.format() method falls back to object.__format__(). This is the
> exception that object.__format__() raises. Why it says "object of type
> 'str'" is not clear to me. Similarly, numpy.float32 does not implement
> the __format__() method. The string scalar type and the native integer
> scalar type (I assume you are on a 64-bit platform, so Python ints are
> 64-bit to match your 'i8' field) inherit from the corresponding native
> Python types, so they inherit their __format__() methods.
> 
> --
> Robert Kern
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Thu Jun  6 16:26:13 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 6 Jun 2013 21:26:13 +0100
Subject: [Numpy-discussion] floats coerced to string with
	"{:f}".format() ?
In-Reply-To: <CA9D2A2B-78DC-487F-9A25-B32BBAEA85E1@lanl.gov>
References: <26E44CF1-EE48-4F9F-9768-9AE47DE08C45@lanl.gov>
	<CAF6FJisYyDj8aEkqVFoTxete63gTY_ZzAKswm2D0d-zovfserA@mail.gmail.com>
	<CA9D2A2B-78DC-487F-9A25-B32BBAEA85E1@lanl.gov>
Message-ID: <CAF6FJivnEepN4712EXUpTFkOq_tquKNR1KfBNJ=4s=TLp3+_AA@mail.gmail.com>

On Thu, Jun 6, 2013 at 9:18 PM, Maccarthy, Jonathan K <jkmacc at lanl.gov> wrote:
> Ah, so np.int64 and np.str inherit the native Python __format__(), but np.float32/64 doesn't get __builtin__.float.__format__().  That's not intuitive, but I see now why this works:
>
> In [8]: '{:6.6s} {:8d} {:11.6f}'.format(tmp.sta, tmp.ondate, float(tmp.lat))
> Out[8]: 'XYZZ    2001123  -23.820000'

np.float64 works because it inherits from the Python float type
(Python floats are 64-bit floats). np.float32 doesn't inherit from the
Python float type because it can't; they don't represent the same kind
of data, so their memory layouts at the C level cannot coincide. Since
you are on a 64-bit platform, np.int64 represents the same kind of
integer as the Python int type, so it can subclass, but an np.int32
couldn't.

It's not necessarily intuitive, but it's the best we can do under the
constraints. The only thing more intuitive would be to disallow
subclassing from the Python builtin types entirely, but that's
*really* annoying.

--
Robert Kern


From jkmacc at lanl.gov  Thu Jun  6 16:50:42 2013
From: jkmacc at lanl.gov (Maccarthy, Jonathan K)
Date: Thu, 6 Jun 2013 20:50:42 +0000
Subject: [Numpy-discussion] floats coerced to string
	with	"{:f}".format() ?
In-Reply-To: <CAF6FJivnEepN4712EXUpTFkOq_tquKNR1KfBNJ=4s=TLp3+_AA@mail.gmail.com>
References: <26E44CF1-EE48-4F9F-9768-9AE47DE08C45@lanl.gov>
	<CAF6FJisYyDj8aEkqVFoTxete63gTY_ZzAKswm2D0d-zovfserA@mail.gmail.com>
	<CA9D2A2B-78DC-487F-9A25-B32BBAEA85E1@lanl.gov>
	<CAF6FJivnEepN4712EXUpTFkOq_tquKNR1KfBNJ=4s=TLp3+_AA@mail.gmail.com>
Message-ID: <AD3EA3B0-122C-48AF-AFFF-29E49B4CF8A6@lanl.gov>

I'm really showing my ignorance now, I think;  so, the old-style "fmt % (tuple)" must use a different mechanism, and perhaps that's why np.savetxt never choked on a float32 for me before (yes, I am on a 64-bit system).

In [8]: type(tmp.lat)
Out[8]: numpy.float32

In [9]: '%6s %8i %11.6f' % (tmp.sta, tmp.ondate, tmp.lat)
Out[9]: '  XYZZ  2001123  -23.820000'

Thanks again for your patience.

On Jun 6, 2013, at 2:26 PM, Robert Kern wrote:

> 
> np.float64 works because it inherits from the Python float type
> (Python floats are 64-bit floats). np.float32 doesn't inherit from the
> Python float type because it can't; they don't represent the same kind
> of data, so their memory layouts at the C level cannot coincide. Since
> you are on a 64-bit platform, np.int64 represents the same kind of
> integer as the Python int type, so it can subclass, but an np.int32
> couldn't.
> 
> It's not necessarily intuitive, but it's the best we can do under the
> constraints. The only thing more intuitive would be to disallow
> subclassing from the Python builtin types entirely, but that's
> *really* annoying.
> 
> --
> Robert Kern
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Thu Jun  6 16:57:20 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 6 Jun 2013 21:57:20 +0100
Subject: [Numpy-discussion] floats coerced to string with
	"{:f}".format() ?
In-Reply-To: <AD3EA3B0-122C-48AF-AFFF-29E49B4CF8A6@lanl.gov>
References: <26E44CF1-EE48-4F9F-9768-9AE47DE08C45@lanl.gov>
	<CAF6FJisYyDj8aEkqVFoTxete63gTY_ZzAKswm2D0d-zovfserA@mail.gmail.com>
	<CA9D2A2B-78DC-487F-9A25-B32BBAEA85E1@lanl.gov>
	<CAF6FJivnEepN4712EXUpTFkOq_tquKNR1KfBNJ=4s=TLp3+_AA@mail.gmail.com>
	<AD3EA3B0-122C-48AF-AFFF-29E49B4CF8A6@lanl.gov>
Message-ID: <CAF6FJitnGQxTUVch=nqskxDPef6pZsq9T6vMVOrv9cqFVpcptA@mail.gmail.com>

On Thu, Jun 6, 2013 at 9:50 PM, Maccarthy, Jonathan K <jkmacc at lanl.gov> wrote:
> I'm really showing my ignorance now, I think;  so, the old-style "fmt % (tuple)" must use a different mechanism, and perhaps that's why np.savetxt never choked on a float32 for me before (yes, I am on a 64-bit system).
>
> In [8]: type(tmp.lat)
> Out[8]: numpy.float32
>
> In [9]: '%6s %8i %11.6f' % (tmp.sta, tmp.ondate, tmp.lat)
> Out[9]: '  XYZZ  2001123  -23.820000'
>
> Thanks again for your patience.

Yes, the mechanism is quite different. With %f, str.__mod__() is in
control. When it is handling the %f code, it is expecting a Python
float object *or* one that can be converted into one via float(). All
of the numpy floating and integer types can be converted to a Python
float via float().

With str.format(), the control is inverted. The value being
represented is asked via its __format__() method to handle the 'f'
format code.

--
Robert Kern


From ewm at redtetrahedron.org  Thu Jun  6 18:12:10 2013
From: ewm at redtetrahedron.org (Eric Moore)
Date: Thu, 06 Jun 2013 18:12:10 -0400
Subject: [Numpy-discussion] ndarray: How to create and initialize with a
 value other than ones or zeros?
In-Reply-To: <CACafSVmfWrS6gcGy1wNjMaGXCdE-vjkE7fx332MUGAu8MZHFbw@mail.gmail.com>
References: <CACafSVmfWrS6gcGy1wNjMaGXCdE-vjkE7fx332MUGAu8MZHFbw@mail.gmail.com>
Message-ID: <51B1093A.4040008@redtetrahedron.org>

James Adams wrote:
> I would like to create an array object and initialize the array's
> values with an arbitrary fill value, like you can do using the ones()
> and zeros() creation routines to create and initialize arrays with
> ones or zeros.  Is there an easy way to do this?  If this isn't
> possible then what is the most efficient way to initialize a numpy
> array with an arbitrary fill value?
>
> In order to provide such an array creation routine I can imagine that
> it'd be as simple as taking the code for ones() and/or zeros() and
> modifying that code so that it provides an additional fill value
> argument and then within the section which does the initialization of
> the array it could use that fill value instead of 1 or 0.  Is this a
> naive assumption?
>
> Thanks in advance for your help with this issue.
>
> --James
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

There's also https://github.com/numpy/numpy/pull/2875 in the queue.

Eric


From ezindy at gmail.com  Fri Jun  7 01:57:13 2013
From: ezindy at gmail.com (Egor Zindy)
Date: Fri, 7 Jun 2013 06:57:13 +0100
Subject: [Numpy-discussion] request for SWIG numpy.i users
In-Reply-To: <CABL7CQhjMqzAYfDCpDfir0_=5fF7H7tYk9o5x=p7LaZNyJtCsg@mail.gmail.com>
References: <CABL7CQhjMqzAYfDCpDfir0_=5fF7H7tYk9o5x=p7LaZNyJtCsg@mail.gmail.com>
Message-ID: <CAEy_T3EnmMTMi7BTSaVU1OB5k+YbQSxrNpjLYwdWgy230yo0Pg@mail.gmail.com>

Hi Ralf,

Your post comes just on time! I implemented the memory managed arrays and
noticed a serious problem with my capsule creation code (post I sent to the
list about the update on the 12th of March in reply to Bill Spotz "Request
code review of numpy.i changes"). For some reason, the code I wrote works
under Linux (or at least on my 12.04 ubuntu machine) but crashes in Windows
/ mingw-64, something I just found out about this week.

Specifically, "SWIG_Python_DestroyModule" seems to be the wrong function
call for the capsule and cobject destructor and I am really not sure why
the code worked when I was using testing it on my Linux machine.

I should have a bit of time over the week-end to look into this and a
couple of other things.

Anyway, for PyCObject_FromVoidPtr(), the destructor should be a call to
free() and for PyCapsule_New(), the destructor code should be a call to a
free_cap() function with the following content:

%#ifdef SWIGPY_USE_CAPSULE
  void free_cap(PyObject * cap)
  {
    void* array = (void*) PyCapsule_GetPointer(cap,SWIGPY_CAPSULE_NAME);
    if (array != NULL) free(array);
  }
%#endif

This works both in Linux and Windows / mingw-64.

I'll ping the list when I'm done testing.

Kind regards,
Egor

On 4 June 2013 21:13, Ralf Gommers <ralf.gommers at gmail.com> wrote:

> Hi,
>
> If you're using or are very familiar with SWIG and the numpy.i interface
> to it, please help to test and/or review
> https://github.com/numpy/numpy/pull/3148. It's a fairly major update to
> numpy.i by Bill Spotz, containing the following:
>   - support for 4D arrays and memory managed output arguments
>   - rework for the deprecated API's in numpy 1.6 and 1.7
>   - a bug fix in a 3D typemap
>   - documentation improvements
>
> It would be good to have this merged before branching 1.8.x. Not many of
> the regular reviewers of numpy PRs are familiar with numpy.i, therefore
> help would be much appreciated.
>
> Thanks,
> Ralf
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130607/4498328b/attachment.html>

From lee.will at gmail.com  Fri Jun  7 12:30:18 2013
From: lee.will at gmail.com (Will Lee)
Date: Fri, 7 Jun 2013 11:30:18 -0500
Subject: [Numpy-discussion] What's the difference between calling __mul__
	and *?
Message-ID: <CAKOb=DT0DwqivjGyz=itk796Jr0goG+zoYKSUXZXe0p9xpOHJQ@mail.gmail.com>

Can somebody tell me why these operations are not the same in numpy?

In [2]: a = numpy.array([1, 2, 3.])

In [4]: matrix = numpy.matrix([[1, 2, 3.], [4, 5, 6], [7, 8, 9]])

In [5]: a.__mul__(matrix)

matrix([[  1.,   4.,   9.],

        [  4.,  10.,  18.],

        [  7.,  16.,  27.]])

In [6]: a * matrix

matrix([[ 30.,  36.,  42.]])


Essentially I'm trying to extend from numpy.ndarray.  From my subclass
__mul__ I'd like to call the parent's __mul__ method.  I ran into
problem when I'm trying to call super(SubArrayClass, self).__mul__()
method when working with a matrix.  I also can't think of a way to use
operator.mul() due to the subclass nature.  Is there any way to make
this work?

Any help is greatly appreciated.

Thanks,

Will


From evgeny.toder at jpmorgan.com  Fri Jun  7 12:38:22 2013
From: evgeny.toder at jpmorgan.com (Toder, Evgeny)
Date: Fri, 7 Jun 2013 16:38:22 +0000
Subject: [Numpy-discussion] What's the difference between calling
 __mul__	and *?
In-Reply-To: <CAKOb=DT0DwqivjGyz=itk796Jr0goG+zoYKSUXZXe0p9xpOHJQ@mail.gmail.com>
References: <CAKOb=DT0DwqivjGyz=itk796Jr0goG+zoYKSUXZXe0p9xpOHJQ@mail.gmail.com>
Message-ID: <C0B518458D7A2B479B898CA53524926E36223036@SBECMX010.exchad.jpmchase.net>

That's how it works in python:

"""
Note: If the right operand's type is a subclass of the left operand's type and that subclass provides the reflected method for the operation, this method will be called before the left operand's non-reflected method. This behavior allows subclasses to override their ancestors' operations.
"""
http://docs.python.org/2/reference/datamodel.html#emulating-numeric-types

Note that matrix is a subclass of ndarray.
Also note that __mul__ can return NotImplemented, in which case again the method of rhs argument will be used.

Eugene

-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Will Lee
Sent: Friday, June 07, 2013 12:30 PM
To: Discussion of Numerical Python
Subject: [Numpy-discussion] What's the difference between calling __mul__ and *?

Can somebody tell me why these operations are not the same in numpy?

In [2]: a = numpy.array([1, 2, 3.])

In [4]: matrix = numpy.matrix([[1, 2, 3.], [4, 5, 6], [7, 8, 9]])

In [5]: a.__mul__(matrix)

matrix([[  1.,   4.,   9.],

        [  4.,  10.,  18.],

        [  7.,  16.,  27.]])

In [6]: a * matrix

matrix([[ 30.,  36.,  42.]])


Essentially I'm trying to extend from numpy.ndarray.  From my subclass
__mul__ I'd like to call the parent's __mul__ method.  I ran into
problem when I'm trying to call super(SubArrayClass, self).__mul__()
method when working with a matrix.  I also can't think of a way to use
operator.mul() due to the subclass nature.  Is there any way to make
this work?

Any help is greatly appreciated.

Thanks,

Will
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion at scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  


From alan.isaac at gmail.com  Fri Jun  7 12:40:59 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 07 Jun 2013 12:40:59 -0400
Subject: [Numpy-discussion] What's the difference between calling
 __mul__ and *?
In-Reply-To: <CAKOb=DT0DwqivjGyz=itk796Jr0goG+zoYKSUXZXe0p9xpOHJQ@mail.gmail.com>
References: <CAKOb=DT0DwqivjGyz=itk796Jr0goG+zoYKSUXZXe0p9xpOHJQ@mail.gmail.com>
Message-ID: <51B20D1B.4090102@gmail.com>

On 6/7/2013 12:30 PM, Will Lee wrote:
> Can somebody tell me why these operations are not the same in numpy?


http://docs.python.org/2/reference/datamodel.html#object.__rmul__

hth,
Alan Isaac



From lee.will at gmail.com  Fri Jun  7 14:14:52 2013
From: lee.will at gmail.com (Will Lee)
Date: Fri, 7 Jun 2013 13:14:52 -0500
Subject: [Numpy-discussion] What's the difference between calling
 __mul__ and *?
In-Reply-To: <51B20D1B.4090102@gmail.com>
References: <CAKOb=DT0DwqivjGyz=itk796Jr0goG+zoYKSUXZXe0p9xpOHJQ@mail.gmail.com>
	<51B20D1B.4090102@gmail.com>
Message-ID: <CAKOb=DSLKxfHmybrdO_rECo=t+GxKtkN_yHA0pBpN3+kNkktNg@mail.gmail.com>

I think I've figured this out.  It seems like the rule in the "Note" box is
the problem.  Since a matrix is not a subclass of my custom class, the
__rmul__ of the matrix is not being called.

Thanks for the info.

Will


On Fri, Jun 7, 2013 at 11:40 AM, Alan G Isaac <alan.isaac at gmail.com> wrote:

> On 6/7/2013 12:30 PM, Will Lee wrote:
> > Can somebody tell me why these operations are not the same in numpy?
>
>
> http://docs.python.org/2/reference/datamodel.html#object.__rmul__
>
> hth,
> Alan Isaac
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130607/48777954/attachment.html>

From josef.pktd at gmail.com  Fri Jun  7 16:58:01 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 7 Jun 2013 16:58:01 -0400
Subject: [Numpy-discussion] adding booleans
Message-ID: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>

Interesting observation, (while lurking on a pull request)

>>> np.add.reduce(np.arange(5)<3)
3
>>> np.add((np.arange(5)<3), (np.arange(5)<3))
array([ True,  True,  True, False, False], dtype=bool)


I often use summing of an array of boolean but didn't know the second behavior

Josef


From njs at pobox.com  Fri Jun  7 19:48:07 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 8 Jun 2013 00:48:07 +0100
Subject: [Numpy-discussion] adding booleans
In-Reply-To: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>
References: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>
Message-ID: <CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>

On 7 Jun 2013 21:58, <josef.pktd at gmail.com> wrote:
>
> Interesting observation, (while lurking on a pull request)
>
> >>> np.add.reduce(np.arange(5)<3)
> 3
> >>> np.add((np.arange(5)<3), (np.arange(5)<3))
> array([ True,  True,  True, False, False], dtype=bool)
>
>
> I often use summing of an array of boolean but didn't know the second
behavior

...yeah weird. My gut reaction is that it's a bug. Addition on bools should
either be an error, undefined but doable via an implicit upcast to int
(analogous to calling np.sin on an int array triggering an upcast to
float), or xor (i.e., addition mod 2). But apparently we're inconsistent -
add.reduce upcasts, and add.__call__, uh... upcasts and then downcasts,
maybe? It's like if np.sin on an int array returned ints? I can't see how
to get the quoted behaviour in any conceptually coherent way. But maybe I'm
missing something.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130608/c5733fc1/attachment.html>

From josef.pktd at gmail.com  Fri Jun  7 20:08:30 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 7 Jun 2013 20:08:30 -0400
Subject: [Numpy-discussion] adding booleans
In-Reply-To: <CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
References: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>
	<CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
Message-ID: <CAMMTP+Bs_4QA1CT+DDF4c2AhwPZmNk+wR7Xxro_1JFRZMJ9QmQ@mail.gmail.com>

On Fri, Jun 7, 2013 at 7:48 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On 7 Jun 2013 21:58, <josef.pktd at gmail.com> wrote:
>>
>> Interesting observation, (while lurking on a pull request)
>>
>> >>> np.add.reduce(np.arange(5)<3)
>> 3
>> >>> np.add((np.arange(5)<3), (np.arange(5)<3))
>> array([ True,  True,  True, False, False], dtype=bool)
>>
>>
>> I often use summing of an array of boolean but didn't know the second
>> behavior
>
> ...yeah weird. My gut reaction is that it's a bug. Addition on bools should
> either be an error, undefined but doable via an implicit upcast to int
> (analogous to calling np.sin on an int array triggering an upcast to float),
> or xor (i.e., addition mod 2). But apparently we're inconsistent -
> add.reduce upcasts, and add.__call__, uh... upcasts and then downcasts,
> maybe? It's like if np.sin on an int array returned ints? I can't see how to
> get the quoted behaviour in any conceptually coherent way. But maybe I'm
> missing something.

The first case is perfectly good behavior. I always "knew"/assumed
that in python bool are 0-1 ints with all the calculation rules.
I only found the second one a surprise (found by Pauli)

>>> reduce(np.add, [ True,  True,  True, False, False])
True
>>> reduce(lambda x, y: x+y, [ True,  True,  True, False, False])
3


The following we use *very* often:

proportion = (x > 0).mean()
n_valid = isfinite(x).sum()

cond = cond1 * cond2

in python: trick indexing with 0-1 bool
>>> ["True", "False"][False]
'True'

Josef

>
> -n
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From josef.pktd at gmail.com  Fri Jun  7 20:29:01 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 7 Jun 2013 20:29:01 -0400
Subject: [Numpy-discussion] adding booleans
In-Reply-To: <CAMMTP+Bs_4QA1CT+DDF4c2AhwPZmNk+wR7Xxro_1JFRZMJ9QmQ@mail.gmail.com>
References: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>
	<CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
	<CAMMTP+Bs_4QA1CT+DDF4c2AhwPZmNk+wR7Xxro_1JFRZMJ9QmQ@mail.gmail.com>
Message-ID: <CAMMTP+DQHmK8gypebDYneRsTZ0tVUJy3hjfqKgaQFyReWH2zUw@mail.gmail.com>

On Fri, Jun 7, 2013 at 8:08 PM,  <josef.pktd at gmail.com> wrote:
> On Fri, Jun 7, 2013 at 7:48 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On 7 Jun 2013 21:58, <josef.pktd at gmail.com> wrote:
>>>
>>> Interesting observation, (while lurking on a pull request)
>>>
>>> >>> np.add.reduce(np.arange(5)<3)
>>> 3
>>> >>> np.add((np.arange(5)<3), (np.arange(5)<3))
>>> array([ True,  True,  True, False, False], dtype=bool)
>>>
>>>
>>> I often use summing of an array of boolean but didn't know the second
>>> behavior
>>
>> ...yeah weird. My gut reaction is that it's a bug. Addition on bools should
>> either be an error, undefined but doable via an implicit upcast to int
>> (analogous to calling np.sin on an int array triggering an upcast to float),
>> or xor (i.e., addition mod 2). But apparently we're inconsistent -
>> add.reduce upcasts, and add.__call__, uh... upcasts and then downcasts,
>> maybe? It's like if np.sin on an int array returned ints? I can't see how to
>> get the quoted behaviour in any conceptually coherent way. But maybe I'm
>> missing something.
>
> The first case is perfectly good behavior. I always "knew"/assumed
> that in python bool are 0-1 ints with all the calculation rules.
> I only found the second one a surprise (found by Pauli)
>
>>>> reduce(np.add, [ True,  True,  True, False, False])
> True
>>>> reduce(lambda x, y: x+y, [ True,  True,  True, False, False])
> 3
>
>
> The following we use *very* often:
>
> proportion = (x > 0).mean()
> n_valid = isfinite(x).sum()
>
> cond = cond1 * cond2
>
> in python: trick indexing with 0-1 bool
>>>> ["True", "False"][False]
> 'True'

python indexing with np.bool
>>> ["True", "False"][np.bool(False)]
'True'
>>> ["True", "False"][np.bool(True)]
'False'

operation between numbers and bool
>>> a = np.array([ True,  True,  True, False, False])
>>> a * range(5)
array([0, 1, 2, 0, 0])
>>> a * range(1, 6)
array([1, 2, 3, 0, 0])
>>> a + range(5)
array([1, 2, 3, 3, 4])


multiply and maximum don't need to upcast because the result stays 0-1

>>> reduce(np.multiply, [ True,  True,  True, False, False])
False
>>> np.multiply.reduce([ True,  True,  True, False, False])
0

>>> np.maximum.reduce([ True,  True,  True, False, False])
True
>>> np.maximum.accumulate([ True,  True,  True, False, False])
array([ True,  True,  True,  True,  True], dtype=bool)

also fine
>>> np.add.accumulate([ True,  True,  True, False, False])
array([1, 2, 3, 3, 3])

-----------

I think the only "weird" and inconsistent one is that bool1 + bool2
does not upcast to int

Josef

>
> Josef
>
>>
>> -n
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>


From arinkverma at gmail.com  Fri Jun  7 22:15:12 2013
From: arinkverma at gmail.com (Arink Verma)
Date: Sat, 8 Jun 2013 07:45:12 +0530
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
Message-ID: <CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>

I did profiling for
$python -m timeit -n 1000000 -s 'import numpy as np;x = np.asarray(1.0)'
'x+x'
with oprofilier, and used gprof2dot.py to create callgraph, but I got
graph[1] which doesn't create any meaning.

I tried to use pprof, but I can not find profiles to be used. like ls.prof
in "pprof /bin/ls ls.prof"
Does any one has used pprof or any other profiler for generating callgraph?

[1] https://docs.google.com/file/d/0B3Pqyp8kuQw0V042bmUwNHktZHM/edit


On Mon, May 6, 2013 at 4:56 AM, Nathaniel Smith <njs at pobox.com> wrote:

> On Sun, May 5, 2013 at 5:57 PM, David Cournapeau <cournape at gmail.com>
> wrote:
> >> perf is a fabulous framework and doesn't have any way to get full
> >> callgraph information out so IME it's been useless. They have
> >> reporting modes that claim to (like some "fractal" thing?) but AFAI
> >> been able to tell from docs/googling/mailing lists, there is nobody
> >> who understands how to interpret this output except the people who
> >> wrote it. Really a shame that it falls down in the last mile like
> >> that, hopefully they will fix this soon.
> >
> > Perf doc is written for Vulcan, but it does what I think you want, say:
> >
> > void work(int n) {
> >   volatile int i=0; //don't optimize away
> >   while(i++ < n);
> > }
> > void easy() { work(1000 * 1000 * 50); }
> > void hard() { work(1000*1000*1000); }
> > int main() { easy(); hard(); }
> >
> > compile with gcc -g -O0, and then:
> >
> > perf record -g -a -- ./a.out
> > perf report -g -a --stdio
> >
> > gives me
> >
> >   95.22%            a.out  a.out
> >   [.] work
> >                       |
> >                       --- work
> >                          |
> >                          |--89.84%-- hard
> >                          |          main
> >                          |          __libc_start_main
> >                          |
> >                           --5.38%-- easy
> >                                     main
> >                                     __libc_start_main
> >
> >
> > or maybe even better with the -G option
> >
> >  95.22%            a.out  a.out
> >  [.] work
> >                       |
> >                       --- __libc_start_main
> >                           main
> >                          |
> >                          |--94.35%-- hard
> >                          |          work
> >                          |
> >                           --5.65%-- easy
> >                                     work
> >
>
> Yeah I've seen these displays before and I can see the information is
> there, and (knowing the code you ran) that somehow the first number
> has to do with the time spent under 'hard' and the second to do with
> time spent under 'easy', but I have no idea how to generalize this to
> arbitrary samples of these output formats. That's what I meant.
>
> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Arink Verma
Indian Institute of Technology
www.arinkverma.in
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130608/fa750921/attachment.html>

From sebastian at sipsolutions.net  Sat Jun  8 05:18:53 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Sat, 08 Jun 2013 11:18:53 +0200
Subject: [Numpy-discussion] adding booleans
In-Reply-To: <CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
References: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>
	<CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
Message-ID: <1370683133.9298.15.camel@sebastian-laptop>

On Sat, 2013-06-08 at 00:48 +0100, Nathaniel Smith wrote:
> On 7 Jun 2013 21:58, <josef.pktd at gmail.com> wrote:
> >
> > Interesting observation, (while lurking on a pull request)
> >
> > >>> np.add.reduce(np.arange(5)<3)
> > 3
> > >>> np.add((np.arange(5)<3), (np.arange(5)<3))
> > array([ True,  True,  True, False, False], dtype=bool)
> >
> >
> > I often use summing of an array of boolean but didn't know the
> second behavior
> 
> ...yeah weird. My gut reaction is that it's a bug. Addition on bools
> should either be an error, undefined but doable via an implicit upcast
> to int (analogous to calling np.sin on an int array triggering an
> upcast to float), or xor (i.e., addition mod 2). But apparently we're
> inconsistent - add.reduce upcasts, and add.__call__, uh... upcasts and
> then downcasts, maybe? It's like if np.sin on an int array returned
> ints? I can't see how to get the quoted behaviour in any conceptually
> coherent way. But maybe I'm missing something.
> 
I believe these are implemented as ufunc-machinery special cases for the
add ufunc reduction methods. This ensures that the sums output integer
type is at least a `np.int_` if the input is integer or bool. With a
future warning, it seems plausible for bools to have `bool + bool = int`
(code might (ab)use it as a logical operation) I guess, but `int16 +
int16` should probably stay int16? While sum over it being larger avoids
overflow accidents.
Another (additionally good) thing would be to have better integer
overflow warnings here, but it seems likely that for the sake of
backward compatibility these special cases can't be easily changed in
any case.

- Sebastian

> -n
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From pav at iki.fi  Sat Jun  8 05:23:50 2013
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 08 Jun 2013 12:23:50 +0300
Subject: [Numpy-discussion] adding booleans
In-Reply-To: <CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
References: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>
	<CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
Message-ID: <kout71$n3u$1@ger.gmane.org>

08.06.2013 02:48, Nathaniel Smith kirjoitti:
[clip]
> ...yeah weird. My gut reaction is that it's a bug. Addition on bools
> should either be an error, undefined but doable via an implicit upcast
> to int (analogous to calling np.sin on an int array triggering an upcast
> to float), or xor (i.e., addition mod 2).

There are several more sensible choices:

http://en.wikipedia.org/wiki/Boolean_ring



From sebastian at sipsolutions.net  Sat Jun  8 05:36:54 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Sat, 08 Jun 2013 11:36:54 +0200
Subject: [Numpy-discussion] adding booleans
In-Reply-To: <CAMMTP+DQHmK8gypebDYneRsTZ0tVUJy3hjfqKgaQFyReWH2zUw@mail.gmail.com>
References: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>
	<CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
	<CAMMTP+Bs_4QA1CT+DDF4c2AhwPZmNk+wR7Xxro_1JFRZMJ9QmQ@mail.gmail.com>
	<CAMMTP+DQHmK8gypebDYneRsTZ0tVUJy3hjfqKgaQFyReWH2zUw@mail.gmail.com>
Message-ID: <1370684214.9298.30.camel@sebastian-laptop>

On Fri, 2013-06-07 at 20:29 -0400, josef.pktd at gmail.com wrote:
> On Fri, Jun 7, 2013 at 8:08 PM,  <josef.pktd at gmail.com> wrote:
> > On Fri, Jun 7, 2013 at 7:48 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >> On 7 Jun 2013 21:58, <josef.pktd at gmail.com> wrote:
> >>>
> >>> Interesting observation, (while lurking on a pull request)
> >>>
> >>> >>> np.add.reduce(np.arange(5)<3)
> >>> 3
> >>> >>> np.add((np.arange(5)<3), (np.arange(5)<3))
> >>> array([ True,  True,  True, False, False], dtype=bool)
> >>>
> >>>
> >>> I often use summing of an array of boolean but didn't know the second
> >>> behavior
> >>
> >> ...yeah weird. My gut reaction is that it's a bug. Addition on bools should
> >> either be an error, undefined but doable via an implicit upcast to int
> >> (analogous to calling np.sin on an int array triggering an upcast to float),
> >> or xor (i.e., addition mod 2). But apparently we're inconsistent -
> >> add.reduce upcasts, and add.__call__, uh... upcasts and then downcasts,
> >> maybe? It's like if np.sin on an int array returned ints? I can't see how to
> >> get the quoted behaviour in any conceptually coherent way. But maybe I'm
> >> missing something.
> >
> > The first case is perfectly good behavior. I always "knew"/assumed
> > that in python bool are 0-1 ints with all the calculation rules.
> > I only found the second one a surprise (found by Pauli)
> >
> >>>> reduce(np.add, [ True,  True,  True, False, False])
> > True
> >>>> reduce(lambda x, y: x+y, [ True,  True,  True, False, False])
> > 3
> >
> >
> > The following we use *very* often:
> >
> > proportion = (x > 0).mean()
> > n_valid = isfinite(x).sum()
> >
> > cond = cond1 * cond2
> >
> > in python: trick indexing with 0-1 bool
> >>>> ["True", "False"][False]
> > 'True'
> 
> python indexing with np.bool
> >>> ["True", "False"][np.bool(False)]
> 'True'
> >>> ["True", "False"][np.bool(True)]
> 'False'
> 

Uhh, damnit, I thought this was not the case. The correct code would be
`np.bool_(False)`, but it works as well. I guess for the sake of
compatibility with Python bools it must work...
Within NumPy indexing with bools however seems wrong to me. This creates
a bit of dilemma in my mind, because I thought it would be fine to just
say that python bools cannot be safely interpreted as integers (i.e.
ignore that they support the index operator), but seeing this we would
have to check for `np.bool_` too.
Also note that `operator.index(np.array(True))` actually fails
currently, which is a (harmless) inconsistency.

It seems to mean that we will either have a 0-d special case (allow
0darr[np.bool_] as boolean indexing but otherwise use it as int), or
need extra checks for `np.bool_` to disallow both python and numpy
bools.

- Sebastian

> operation between numbers and bool
> >>> a = np.array([ True,  True,  True, False, False])
> >>> a * range(5)
> array([0, 1, 2, 0, 0])
> >>> a * range(1, 6)
> array([1, 2, 3, 0, 0])
> >>> a + range(5)
> array([1, 2, 3, 3, 4])
> 
> 
> multiply and maximum don't need to upcast because the result stays 0-1
> 
> >>> reduce(np.multiply, [ True,  True,  True, False, False])
> False
> >>> np.multiply.reduce([ True,  True,  True, False, False])
> 0
> 
> >>> np.maximum.reduce([ True,  True,  True, False, False])
> True
> >>> np.maximum.accumulate([ True,  True,  True, False, False])
> array([ True,  True,  True,  True,  True], dtype=bool)
> 
> also fine
> >>> np.add.accumulate([ True,  True,  True, False, False])
> array([1, 2, 3, 3, 3])
> 
> -----------
> 
> I think the only "weird" and inconsistent one is that bool1 + bool2
> does not upcast to int
> 
> Josef
> 
> >
> > Josef
> >
> >>
> >> -n
> >>
> >>
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From josef.pktd at gmail.com  Sat Jun  8 07:40:49 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 8 Jun 2013 07:40:49 -0400
Subject: [Numpy-discussion] adding booleans
In-Reply-To: <1370684214.9298.30.camel@sebastian-laptop>
References: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>
	<CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
	<CAMMTP+Bs_4QA1CT+DDF4c2AhwPZmNk+wR7Xxro_1JFRZMJ9QmQ@mail.gmail.com>
	<CAMMTP+DQHmK8gypebDYneRsTZ0tVUJy3hjfqKgaQFyReWH2zUw@mail.gmail.com>
	<1370684214.9298.30.camel@sebastian-laptop>
Message-ID: <CAMMTP+AHa_c-hZAZomAOvpBfyYKGeZ24_t62HD8_2ecit3PFhA@mail.gmail.com>

Question about namespace

why is there bool and bool_ ?

>>> np.bool(True) + np.bool(True)
2
>>> np.bool_(True) + np.bool_(True)
True

>>> type(np.bool(True))
<type 'bool'>
>>> type(np.bool_(True))
<type 'numpy.bool_'>

I didn't pay attention to the trailing underline in Pauli's original example

Josef


From robert.kern at gmail.com  Sat Jun  8 07:54:56 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 8 Jun 2013 12:54:56 +0100
Subject: [Numpy-discussion] adding booleans
In-Reply-To: <CAMMTP+AHa_c-hZAZomAOvpBfyYKGeZ24_t62HD8_2ecit3PFhA@mail.gmail.com>
References: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>
	<CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
	<CAMMTP+Bs_4QA1CT+DDF4c2AhwPZmNk+wR7Xxro_1JFRZMJ9QmQ@mail.gmail.com>
	<CAMMTP+DQHmK8gypebDYneRsTZ0tVUJy3hjfqKgaQFyReWH2zUw@mail.gmail.com>
	<1370684214.9298.30.camel@sebastian-laptop>
	<CAMMTP+AHa_c-hZAZomAOvpBfyYKGeZ24_t62HD8_2ecit3PFhA@mail.gmail.com>
Message-ID: <CAF6FJist08CBg+ZAbTmVgX_N1CMk7HuQ86aAD2kRC6BwFCjgtg@mail.gmail.com>

On Sat, Jun 8, 2013 at 12:40 PM,  <josef.pktd at gmail.com> wrote:
> Question about namespace
>
> why is there bool and bool_ ?
>
>>>> np.bool(True) + np.bool(True)
> 2
>>>> np.bool_(True) + np.bool_(True)
> True
>
>>>> type(np.bool(True))
> <type 'bool'>
>>>> type(np.bool_(True))
> <type 'numpy.bool_'>
>
> I didn't pay attention to the trailing underline in Pauli's original example

`np.bool is __builtin__.bool`. It's a backwards-compatibility alias
for an old version of numpy that used the name `np.bool` for the
scalar type now called `np.bool_` (similarly for `np.float`, `np.int`,
and `np.complex`). Since `from numpy import *` caused name collisions,
we moved the scalar types to their current underscored versions.
However, since people had started to use `dtype=np.bool`, we just
aliased the builtin type objects to keep that code working. I would
love to be able to remove those since they are an unending source of
confusion. However, there is no good way to issue deprecation warnings
when you use them, so that means that upgrades would cause sudden,
unwarned breakage, which we like to avoid.

--
Robert Kern


From archibald at astron.nl  Sat Jun  8 08:35:27 2013
From: archibald at astron.nl (Anne Archibald)
Date: Sat, 8 Jun 2013 14:35:27 +0200
Subject: [Numpy-discussion] supporting quad precision
In-Reply-To: <CAGY4rcV2_-HAzx-Wgfp96yHKKkjizx0UORXV1EWXtQ-NxijGbA@mail.gmail.com>
References: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>
	<CAB6mnxKzCGEz2=iBJrX0dHqcfXdugqRWHTVXOJ24KjUCBONTTw@mail.gmail.com>
	<CAGY4rcV2_-HAzx-Wgfp96yHKKkjizx0UORXV1EWXtQ-NxijGbA@mail.gmail.com>
Message-ID: <CANm_+Zq9hMr0dca39_0uUsV6VeN8R+aDiY-XvCqGGU8CnxZa-Q@mail.gmail.com>

Looking at the rational module, I think you're right: it really shouldn't
be too hard to get quads working as a user type using gcc's __float128
type, which will provide hardware arithmetic in the unlikely case that the
user has hardware quads. Alternatively, probably more work, one could use a
package like qd to provide portable quad precision (and quad-double).

I'll take a look.

Anne
On Jun 5, 2013 7:10 PM, "David Cournapeau" <cournape at gmail.com> wrote:

> On Wed, Jun 5, 2013 at 5:21 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> > Hi Anne,
> >
> > Long time no see ;)
> >
> > On Wed, Jun 5, 2013 at 10:07 AM, Anne Archibald <archibald at astron.nl>
> wrote:
> >>
> >> Hi folks,
> >>
> >> I recently came across an application I needed quad precision for
> >> (high-accuracy solution of a differential equation). I found a C++
> library
> >> (odeint) that worked for the integration itself, but unfortunately it
> >> appears numpy is not able to work with quad precision arrays. For my
> >> application the quad precision is only really needed for integrator
> state,
> >> so I can manually convert my data to long doubles as I go to and from
> numpy,
> >> but it's a pain. So quad precision support would be nice.
> >>
> >> There's a thread discussing quad support:
> >>
> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/061080.html
> >> Essentially, there isn't any, but since gcc >= 4.6 supports them on
> Intel
> >> hardware (in software), it should be possible. (Then the thread got
> bogged
> >> down in bike-shedding about what to call them.)
> >>
> >> What would it take to support quads in numpy? I looked into the numpy
> base
> >> dtype definitions, and it's a complex arrangement involving detection of
> >> platform support and templatized C code; in the end I couldn't really
> see
> >> where to start. But it seems to me all the basics are available: native
> C
> >> syntax for basic arithmetic, "qabs"-style versions of all the basic
> >> functions, NaNs and Infs. So how would one go about adding quad support?
> >>
> >
> > There are some improvements for user types committed in 1.8-dev. Perhaps
> > quad support could be added as a user type as it is still
> platform/compiler
> > dependent. The rational type added to numpy could supply a template for
> > adding the new type.
>
> I would be in support of that direction as well: let it live
> separately until CPU/compiler support is coming up.
>
> Anne, will you be at scipy conference ? Improving user data type
> internal API is something I'd like to work on as well
>
> David
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130608/58acf4ae/attachment.html>

From josef.pktd at gmail.com  Sat Jun  8 08:48:37 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 8 Jun 2013 08:48:37 -0400
Subject: [Numpy-discussion] adding booleans
In-Reply-To: <1370683133.9298.15.camel@sebastian-laptop>
References: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>
	<CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
	<1370683133.9298.15.camel@sebastian-laptop>
Message-ID: <CAMMTP+CkT8Nip_CXRPRryi5ozA=Jw+L-OTAQV4MTCV_nV6G_eQ@mail.gmail.com>

On Sat, Jun 8, 2013 at 5:18 AM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> On Sat, 2013-06-08 at 00:48 +0100, Nathaniel Smith wrote:
>> On 7 Jun 2013 21:58, <josef.pktd at gmail.com> wrote:
>> >
>> > Interesting observation, (while lurking on a pull request)
>> >
>> > >>> np.add.reduce(np.arange(5)<3)
>> > 3
>> > >>> np.add((np.arange(5)<3), (np.arange(5)<3))
>> > array([ True,  True,  True, False, False], dtype=bool)
>> >
>> >
>> > I often use summing of an array of boolean but didn't know the
>> second behavior
>>
>> ...yeah weird. My gut reaction is that it's a bug. Addition on bools
>> should either be an error, undefined but doable via an implicit upcast
>> to int (analogous to calling np.sin on an int array triggering an
>> upcast to float), or xor (i.e., addition mod 2). But apparently we're
>> inconsistent - add.reduce upcasts, and add.__call__, uh... upcasts and
>> then downcasts, maybe? It's like if np.sin on an int array returned
>> ints? I can't see how to get the quoted behaviour in any conceptually
>> coherent way. But maybe I'm missing something.
>>
> I believe these are implemented as ufunc-machinery special cases for the
> add ufunc reduction methods. This ensures that the sums output integer
> type is at least a `np.int_` if the input is integer or bool. With a
> future warning, it seems plausible for bools to have `bool + bool = int`
> (code might (ab)use it as a logical operation) I guess, but `int16 +
> int16` should probably stay int16? While sum over it being larger avoids
> overflow accidents.

I think bool + bool with overflow result would be consistent with the rest.
if this is the rule
"reduce/accumulate operations upcast, simple operators on 2 identical
types don't"

but bool + bool has a result that is neither python nor overflow, AFAICS

I never looked carefully at what the expected overflow behavior is
with "small" dtypes (since I avoid them)

In python 3, division is now the odd case

>>> np.uint8(1) - np.uint8(2)

Warning (from warnings module):
  File "C:\Programs\Python32\Lib\idlelib\idle.pyw", line 1
    try:
RuntimeWarning: overflow encountered in ubyte_scalars
>>> 255
>>> np.uint8(1) / np.uint8(2)
0.5
>>> np.uint8(1) * np.uint8(2)
2
>>> np.uint8(150) * np.uint8(2)
44



Josef

> Another (additionally good) thing would be to have better integer
> overflow warnings here, but it seems likely that for the sake of
> backward compatibility these special cases can't be easily changed in
> any case.
>
> - Sebastian
>
>> -n
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From josef.pktd at gmail.com  Sat Jun  8 08:52:44 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 8 Jun 2013 08:52:44 -0400
Subject: [Numpy-discussion] np.asarray atleast_float64 ?
Message-ID: <CAMMTP+Cm6F17ooVaGM82L8skDzgGx-YaJQOdthk9EY+2GdpCrw@mail.gmail.com>

Is there anything to require a numpy array with a minimum numeric dtype?

To avoid lower precision calculations and be upwards compatible, something like

x = np.asarray(x, >=np.float64)

that converts ints, bool and lower precision to float64 but leaves
higher precision float and complex double alone.


Josef


From josef.pktd at gmail.com  Sat Jun  8 09:06:09 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 8 Jun 2013 09:06:09 -0400
Subject: [Numpy-discussion] adding booleans
In-Reply-To: <CAF6FJist08CBg+ZAbTmVgX_N1CMk7HuQ86aAD2kRC6BwFCjgtg@mail.gmail.com>
References: <CAMMTP+BwEDmT6izBF4gC1H=5ZtnQ9HbpDEQZ6tJjb5exAN2T2w@mail.gmail.com>
	<CAPJVwBmY-dYrfHd7YNXv0Ne_T8kABQ2eLrSDrpUeKBwTy6kcbQ@mail.gmail.com>
	<CAMMTP+Bs_4QA1CT+DDF4c2AhwPZmNk+wR7Xxro_1JFRZMJ9QmQ@mail.gmail.com>
	<CAMMTP+DQHmK8gypebDYneRsTZ0tVUJy3hjfqKgaQFyReWH2zUw@mail.gmail.com>
	<1370684214.9298.30.camel@sebastian-laptop>
	<CAMMTP+AHa_c-hZAZomAOvpBfyYKGeZ24_t62HD8_2ecit3PFhA@mail.gmail.com>
	<CAF6FJist08CBg+ZAbTmVgX_N1CMk7HuQ86aAD2kRC6BwFCjgtg@mail.gmail.com>
Message-ID: <CAMMTP+APYaaWwtxXr47B+xDLF7qaEw53QLLZtcDW2dheQdgjGQ@mail.gmail.com>

On Sat, Jun 8, 2013 at 7:54 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Sat, Jun 8, 2013 at 12:40 PM,  <josef.pktd at gmail.com> wrote:
>> Question about namespace
>>
>> why is there bool and bool_ ?
>>
>>>>> np.bool(True) + np.bool(True)
>> 2
>>>>> np.bool_(True) + np.bool_(True)
>> True
>>
>>>>> type(np.bool(True))
>> <type 'bool'>
>>>>> type(np.bool_(True))
>> <type 'numpy.bool_'>
>>
>> I didn't pay attention to the trailing underline in Pauli's original example
>
> `np.bool is __builtin__.bool`. It's a backwards-compatibility alias
> for an old version of numpy that used the name `np.bool` for the
> scalar type now called `np.bool_` (similarly for `np.float`, `np.int`,
> and `np.complex`). Since `from numpy import *` caused name collisions,
> we moved the scalar types to their current underscored versions.
> However, since people had started to use `dtype=np.bool`, we just
> aliased the builtin type objects to keep that code working. I would
> love to be able to remove those since they are an unending source of
> confusion. However, there is no good way to issue deprecation warnings
> when you use them, so that means that upgrades would cause sudden,
> unwarned breakage, which we like to avoid.

Thanks for the explanation.

I guess I never ran into a problem with this, and never saw the difference.

It's confusing, but if the behavior is the same (in all usual cases)
whether it's python or numpy, then it is less of a problem.

>>> np.int64(0) ** np.int64(-1)
inf
>>> np.int(0) ** np.int(-1)
Traceback (most recent call last):
  File "<pyshell#169>", line 1, in <module>
    np.int(0) ** np.int(-1)
ZeroDivisionError: 0.0 cannot be raised to a negative power

Another thing to watch out for.

Josef

>
> --
> Robert Kern
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From sebastian at sipsolutions.net  Sat Jun  8 19:05:55 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Sun, 09 Jun 2013 01:05:55 +0200
Subject: [Numpy-discussion] np.asarray atleast_float64 ?
In-Reply-To: <CAMMTP+Cm6F17ooVaGM82L8skDzgGx-YaJQOdthk9EY+2GdpCrw@mail.gmail.com>
References: <CAMMTP+Cm6F17ooVaGM82L8skDzgGx-YaJQOdthk9EY+2GdpCrw@mail.gmail.com>
Message-ID: <1370732755.2318.4.camel@sebastian-laptop>

On Sat, 2013-06-08 at 08:52 -0400, josef.pktd at gmail.com wrote:
> Is there anything to require a numpy array with a minimum numeric dtype?
> 
> To avoid lower precision calculations and be upwards compatible, something like
> 
> x = np.asarray(x, >=np.float64)

np.result_type(arr, np.float64) uses the usual numpy promotion rules.
But it doesn't do the "asarray" part. Its still the closest thing I can
think of right now.

- Sebastian

> 
> that converts ints, bool and lower precision to float64 but leaves
> higher precision float and complex double alone.
> 
> 
> Josef
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From josef.pktd at gmail.com  Sat Jun  8 19:31:16 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 8 Jun 2013 19:31:16 -0400
Subject: [Numpy-discussion] np.asarray atleast_float64 ?
In-Reply-To: <1370732755.2318.4.camel@sebastian-laptop>
References: <CAMMTP+Cm6F17ooVaGM82L8skDzgGx-YaJQOdthk9EY+2GdpCrw@mail.gmail.com>
	<1370732755.2318.4.camel@sebastian-laptop>
Message-ID: <CAMMTP+DPfG4_C4XHg1+wf65uE8V3-jX005GVY6A1cs0vhRV6nw@mail.gmail.com>

On Sat, Jun 8, 2013 at 7:05 PM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> On Sat, 2013-06-08 at 08:52 -0400, josef.pktd at gmail.com wrote:
>> Is there anything to require a numpy array with a minimum numeric dtype?
>>
>> To avoid lower precision calculations and be upwards compatible, something like
>>
>> x = np.asarray(x, >=np.float64)
>
> np.result_type(arr, np.float64) uses the usual numpy promotion rules.
> But it doesn't do the "asarray" part. Its still the closest thing I can
> think of right now.

Thank you, that looks close enough

And I will soon switch to numpy 1.6 and can read up on some old What's new.

Josef

>
> - Sebastian
>
>>
>> that converts ints, bool and lower precision to float64 but leaves
>> higher precision float and complex double alone.
>>
>>
>> Josef
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From thomas.p.krauss at gmail.com  Sat Jun  8 22:33:28 2013
From: thomas.p.krauss at gmail.com (Tom Krauss)
Date: Sat, 8 Jun 2013 21:33:28 -0500
Subject: [Numpy-discussion] request for SWIG numpy.i users
In-Reply-To: <CABL7CQhjMqzAYfDCpDfir0_=5fF7H7tYk9o5x=p7LaZNyJtCsg@mail.gmail.com>
References: <CABL7CQhjMqzAYfDCpDfir0_=5fF7H7tYk9o5x=p7LaZNyJtCsg@mail.gmail.com>
Message-ID: <CAFQELMyV0gsn8No7KLp1fx=x0uQ8SH8gdg5GLF9hFYFGxrRLhQ@mail.gmail.com>

Hi folks,

I just downloaded Bill's numpy.i
<https://github.com/wfspotz/numpy/blob/4dcb06796b290ae29d4e73ad995d219087f2e949/doc/swig/numpy.i>at
commit 4dcb0679, and tried it out a bit on some of my personal projects.
 It worked fine, although I use only a fraction of the capabilities that it
includes.

 And, it made the warning go away!

I used to get this warning

g++ -g -fPIC -c simple_wrap.cpp -I/usr/include/python2.7
-I/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include
In file included from
/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarraytypes.h:1722,
                 from
/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarrayobject.h:17,
                 from
/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/arrayobject.h:15,
                 from simple_wrap.cpp:3062:
/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
warning: #warning "Using deprecated NumPy API, disable it by #defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"

but not with this version.

You can see which version of numpy I am using there, and that I am on Mac
OS X 10.8. (10.8.4 specifically)  Python 2.7.2

I'd say SHIP IT!

Nice work, thanks for all your work on numpy and numpy.i.

- Tom Krauss



On Tue, Jun 4, 2013 at 3:13 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:

> Hi,
>
> If you're using or are very familiar with SWIG and the numpy.i interface
> to it, please help to test and/or review
> https://github.com/numpy/numpy/pull/3148. It's a fairly major update to
> numpy.i by Bill Spotz, containing the following:
>   - support for 4D arrays and memory managed output arguments
>   - rework for the deprecated API's in numpy 1.6 and 1.7
>   - a bug fix in a 3D typemap
>   - documentation improvements
>
> It would be good to have this merged before branching 1.8.x. Not many of
> the regular reviewers of numpy PRs are familiar with numpy.i, therefore
> help would be much appreciated.
>
> Thanks,
> Ralf
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130608/f73bf5bb/attachment.html>

From heng at cantab.net  Sun Jun  9 03:35:36 2013
From: heng at cantab.net (Henry Gomersall)
Date: Sun, 09 Jun 2013 08:35:36 +0100
Subject: [Numpy-discussion] supporting quad precision
In-Reply-To: <CANm_+Zq9hMr0dca39_0uUsV6VeN8R+aDiY-XvCqGGU8CnxZa-Q@mail.gmail.com>
References: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>
	<CAB6mnxKzCGEz2=iBJrX0dHqcfXdugqRWHTVXOJ24KjUCBONTTw@mail.gmail.com>
	<CAGY4rcV2_-HAzx-Wgfp96yHKKkjizx0UORXV1EWXtQ-NxijGbA@mail.gmail.com>
	<CANm_+Zq9hMr0dca39_0uUsV6VeN8R+aDiY-XvCqGGU8CnxZa-Q@mail.gmail.com>
Message-ID: <1370763336.28467.9.camel@farnsworth>

On Sat, 2013-06-08 at 14:35 +0200, Anne Archibald wrote:
> Looking at the rational module, I think you're right: it really
> shouldn't be too hard to get quads working as a user type using gcc's
> __float128 type, which will provide hardware arithmetic in the
> unlikely case that the user has hardware quads. Alternatively,
> probably more work, one could use a package like qd to provide
> portable quad precision (and quad-double).

In this vague area, and further to a question I asked a while ago on
StackOverflow (http://stackoverflow.com/q/9062562/709852), is there some
deep reason why on some platforms longdouble is float128 and on other
it's float96?

cheers,
Henry



From ezindy at gmail.com  Sun Jun  9 04:20:59 2013
From: ezindy at gmail.com (Egor Zindy)
Date: Sun, 9 Jun 2013 09:20:59 +0100
Subject: [Numpy-discussion] request for SWIG numpy.i users
In-Reply-To: <CAFQELMyV0gsn8No7KLp1fx=x0uQ8SH8gdg5GLF9hFYFGxrRLhQ@mail.gmail.com>
References: <CABL7CQhjMqzAYfDCpDfir0_=5fF7H7tYk9o5x=p7LaZNyJtCsg@mail.gmail.com>
	<CAFQELMyV0gsn8No7KLp1fx=x0uQ8SH8gdg5GLF9hFYFGxrRLhQ@mail.gmail.com>
Message-ID: <CAEy_T3HqPRzb_vfEtyrUzATD1ctjJ6JoZXYAJB+7cmHqHxD79w@mail.gmail.com>

Thanks Tom,

before we ship it, I'd love to have some feedback on the new ARGOUT_VIEWM
type.

I used to create my managed arrays using

    PyObject* cap = PyCObject_FromVoidPtr((void*)(*$1), free);

but since this function is deprecated, and because of Bill's background
work to bring numpy.i up to date, I now use capsules for this:

    PyObject* cap = PyCapsule_New((void*)(*$1), SWIGPY_CAPSULE_NAME,
free_cap);

... I'll admit it took longer than expected to get this right.

Would you mind testing my latest numpy.i changes hosted on github?
https://github.com/zindy/numpy/tree/numpy-swig/doc/swig

It's great that you are testing on a mac, I don't have one to test on yet.

> It worked fine, although I use only a fraction of the capabilities that
it includes.

Same here, but overall, it should be quit easy to choose the data type you
need. Narrow down it down to a type between IN_ARRAY / INPLACE_ / ARGOUT_ /
ARGOUT_VIEW/VIEWM
http://wiki.scipy.org/Cookbook/SWIG_NumPy_examples
http://wiki.scipy.org/Cookbook/SWIG_Memory_Deallocation (I'll update these
when I have a sec)

... and choose the number of dimensions you need (1/2/3/4). I can't comment
on the Fortran arrays data types though as I don't use them.

Also I've introduced a few of my more esoteric data types in this week, but
I have no idea how popular they will be. If you ever need to speed-up:

a = numpy.ones((1024,1024),numpy.uint8)
la = [a]*100
b = numpy.mean(numpy.array(la,float),axis=0).astype(numpy.uint8)

I have just the right type for that :)
DATA_TYPE** IN_ARRAY3, DIM_TYPE DIM1, DIM_TYPE DIM2, DIM_TYPE DIM3

Kind regards,
Egor

On 9 June 2013 03:33, Tom Krauss <thomas.p.krauss at gmail.com> wrote:

> Hi folks,
>
> I just downloaded Bill's numpy.i <https://github.com/wfspotz/numpy/blob/4dcb06796b290ae29d4e73ad995d219087f2e949/doc/swig/numpy.i>at
> commit 4dcb0679, and tried it out a bit on some of my personal projects.
>  It worked fine, although I use only a fraction of the capabilities that it
> includes.
>
>  And, it made the warning go away!
>
> I used to get this warning
>
> g++ -g -fPIC -c simple_wrap.cpp -I/usr/include/python2.7
> -I/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include
> In file included from
> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarraytypes.h:1722,
>                  from
> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarrayobject.h:17,
>                  from
> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/arrayobject.h:15,
>                  from simple_wrap.cpp:3062:
> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
> warning: #warning "Using deprecated NumPy API, disable it by #defining
> NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
>
> but not with this version.
>
> You can see which version of numpy I am using there, and that I am on Mac
> OS X 10.8. (10.8.4 specifically)  Python 2.7.2
>
> I'd say SHIP IT!
>
> Nice work, thanks for all your work on numpy and numpy.i.
>
> - Tom Krauss
>
>
>
> On Tue, Jun 4, 2013 at 3:13 PM, Ralf Gommers <ralf.gommers at gmail.com>wrote:
>
>> Hi,
>>
>> If you're using or are very familiar with SWIG and the numpy.i interface
>> to it, please help to test and/or review
>> https://github.com/numpy/numpy/pull/3148. It's a fairly major update to
>> numpy.i by Bill Spotz, containing the following:
>>   - support for 4D arrays and memory managed output arguments
>>   - rework for the deprecated API's in numpy 1.6 and 1.7
>>   - a bug fix in a 3D typemap
>>   - documentation improvements
>>
>> It would be good to have this merged before branching 1.8.x. Not many of
>> the regular reviewers of numpy PRs are familiar with numpy.i, therefore
>> help would be much appreciated.
>>
>> Thanks,
>> Ralf
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130609/52cdfcc3/attachment.html>

From josef.pktd at gmail.com  Sun Jun  9 06:10:54 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 9 Jun 2013 06:10:54 -0400
Subject: [Numpy-discussion] quickselect
In-Reply-To: <CAMMTP+D_hjjotGHcgG1PxFrS-yjEB2z5LXL=qYasi-E226NLZg@mail.gmail.com>
References: <CAB6mnxJtcm_VdqgLtMtxszQ1+MUW6kvf2d82noLW=fuxOe8EsA@mail.gmail.com>
	<CAMMTP+ArWn1hVswH6GXfJSbBtsAg3j9RZd2R52GzcjAeMjynEQ@mail.gmail.com>
	<51A62C0F.6040109@googlemail.com>
	<CAMMTP+D_hjjotGHcgG1PxFrS-yjEB2z5LXL=qYasi-E226NLZg@mail.gmail.com>
Message-ID: <CAMMTP+A5YttfZQ2iXv_VckDZH+FRgUp4D+CCE6AhF3CS-C4Hmg@mail.gmail.com>

On Wed, May 29, 2013 at 3:19 PM,  <josef.pktd at gmail.com> wrote:
> On Wed, May 29, 2013 at 12:25 PM, Julian Taylor
> <jtaylor.debian at googlemail.com> wrote:
>> On 05/29/2013 06:12 AM, josef.pktd at gmail.com wrote:
>>>
>>> On Tue, May 28, 2013 at 6:31 PM, Charles R Harris
>>> <charlesr.harris at gmail.com> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> There is a PR adding quickselect to numpy as a function `partition`.
>>>> Comments on name and exposure in the numpy API are welcome.
>>>
>>>
>>> I think the name is fine. It's possible to get used to it.
>>>
>>> One possible use I can think of is if I want to sort afterwards only
>>> one of the tails (like largest 15%).
>>>
>>> Limitations:
>>>
>>> medium: like percentiles it would be nice to partition into more than
>>> two sets, if that can be done more efficiently than calling the
>>> function twice.
>>
>>
>> I'm not aware of an algorithm that is more efficient to generate multiple
>> partitions opposed to partitioning twice.
>> Additional partitions can work on subranges so so smaller the remainign
>> range the faster it gets.
>> I did a coarse benchmark with percentile partitioning multiple times instead
>> of sorting once.
>> partition beats sorting even when you want to select 500 percentiles, so we
>> probably can change it completely.
>> If people frequently select even more we could add a heuristic cut off to
>> sort instead.
>
> I only know a few places in statistics where a medium large number of
> percentiles or partitions are used, most of the time it's either just
> a few or a full sort.
> This would work well then.
>
> Actually, I just thought about another application:
> If I want to do a coarse quantization of an array by percentiles, then
> I could partition it and assign categories, instead of doing a full
> sort. The number of bins could be large in this case.
>
>>
>>
>>>
>>> major:
>>> it won't work to calculate the median or other quantiles since those
>>> interpolate
>>> np.median: "When N is even, it is the average of the two middle values
>>> of V_sorted"
>>>
>>
>> this works fine, quickselect/partition moves the kth element into its final
>> sorted order, so to get the median of even elements you can just partition
>> twice, to get the two middle values. Then take the mean as usual.
>>
>> This is already implemented in the pull request.
>> It includes the additional optimization that the second middle value is just
>> a minimum of the second partition.
>>
>
> Thanks for the explanation. I was only looking for docstrings and
> didn't see the implementation of median.
> Using min on the second partitions looks like a nice trick, (one to remember).
>
> As a future user, I think this sounds good, faster median and
> percentiles and a nice new partition function.

Advertising for an other possible application for ``partition``

I'm just looking at trimmed means, where we want to remove the 10% or
20% largest and smallest values in an array, and base the mean on the
center 80% or 60% observations.

median is a special case with almost 50% trimming.

used for outlier detection and robust statistic (t-test with skewed
and heavy-tailed distributions)

This requires currently a full sort.

Josef


>
> Josef


From cournape at gmail.com  Sun Jun  9 07:23:14 2013
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 9 Jun 2013 12:23:14 +0100
Subject: [Numpy-discussion] supporting quad precision
In-Reply-To: <1370763336.28467.9.camel@farnsworth>
References: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>
	<CAB6mnxKzCGEz2=iBJrX0dHqcfXdugqRWHTVXOJ24KjUCBONTTw@mail.gmail.com>
	<CAGY4rcV2_-HAzx-Wgfp96yHKKkjizx0UORXV1EWXtQ-NxijGbA@mail.gmail.com>
	<CANm_+Zq9hMr0dca39_0uUsV6VeN8R+aDiY-XvCqGGU8CnxZa-Q@mail.gmail.com>
	<1370763336.28467.9.camel@farnsworth>
Message-ID: <CAGY4rcVY5wAq5OffeTx2vmtHRahQSO6t5dfhOBiU-zge=MHBgg@mail.gmail.com>

On Sun, Jun 9, 2013 at 8:35 AM, Henry Gomersall <heng at cantab.net> wrote:
> On Sat, 2013-06-08 at 14:35 +0200, Anne Archibald wrote:
>> Looking at the rational module, I think you're right: it really
>> shouldn't be too hard to get quads working as a user type using gcc's
>> __float128 type, which will provide hardware arithmetic in the
>> unlikely case that the user has hardware quads. Alternatively,
>> probably more work, one could use a package like qd to provide
>> portable quad precision (and quad-double).
>
> In this vague area, and further to a question I asked a while ago on
> StackOverflow (http://stackoverflow.com/q/9062562/709852), is there some
> deep reason why on some platforms longdouble is float128 and on other
> it's float96?

Long double is not standardized (like single/double are), so it is CPU
dependent. On Intel CPU, long double is generally translated into the
extended precision 80 bits. On 32 bits, it is aligned to 12 bytes (the
next multiple of 4 bytes), on 64 bits 16 bytes (the next multiple 8
bytes). MS compilers are a notable exception where long double ==
double.

So it depends on the CPU, the OS and the compiler. Using long double
for anything else than compatibility (e.g. binary files) is often a
mistake IMO, and highly unportable.

David


From jtaylor.debian at googlemail.com  Sun Jun  9 11:27:40 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Sun, 09 Jun 2013 17:27:40 +0200
Subject: [Numpy-discussion] quickselect
In-Reply-To: <CAMMTP+A5YttfZQ2iXv_VckDZH+FRgUp4D+CCE6AhF3CS-C4Hmg@mail.gmail.com>
References: <CAB6mnxJtcm_VdqgLtMtxszQ1+MUW6kvf2d82noLW=fuxOe8EsA@mail.gmail.com>
	<CAMMTP+ArWn1hVswH6GXfJSbBtsAg3j9RZd2R52GzcjAeMjynEQ@mail.gmail.com>
	<51A62C0F.6040109@googlemail.com>
	<CAMMTP+D_hjjotGHcgG1PxFrS-yjEB2z5LXL=qYasi-E226NLZg@mail.gmail.com>
	<CAMMTP+A5YttfZQ2iXv_VckDZH+FRgUp4D+CCE6AhF3CS-C4Hmg@mail.gmail.com>
Message-ID: <51B49EEC.6000400@googlemail.com>

On 09.06.2013 12:10, josef.pktd at gmail.com wrote:
> On Wed, May 29, 2013 at 3:19 PM,  <josef.pktd at gmail.com> wrote:
>> On Wed, May 29, 2013 at 12:25 PM, Julian Taylor
>> <jtaylor.debian at googlemail.com> wrote:
>>> On 05/29/2013 06:12 AM, josef.pktd at gmail.com wrote:
>>>>
>>>> On Tue, May 28, 2013 at 6:31 PM, Charles R Harris
>>>> <charlesr.harris at gmail.com> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> There is a PR adding quickselect to numpy as a function `partition`.
>>>>> Comments on name and exposure in the numpy API are welcome.
>>>>
>>>>


here a a quick status report on the PR
https://github.com/numpy/numpy/pull/3360

I now implemented partitioning via the introselect algorithm which is a
quickselect median of 3 pivot with a cutoff on recursion depth to a
median of median of 5 pivot for O(N) worst case complexity.
Additionally it can stores its pivots for reuse in other partitions on
the same data to reduce the space required to be partitioned next time,
this is useful e.g. for multiple percentiles.

It is functionally ready, but there are still some API/ABI issues to be
solved.
Mainly deciding if we put the selection algorithms in the ABI for 1.8 or
not. Currently the ABI is unchanged so user types cannot make use of the
algorithms (they will fall back to quicksort).

The python api is now:
np.partition(array, kth)
where kth is an integer or array of integers

it will move each index in kth into its final sorted position, so
np.partition(a, range(a.size)) results in a (inefficient) sort.
e.g.:

d = np.array([66, 81, 21, 75, 46, -6, 66, 86, 242, 47, 88, 79])
np.partition(d, (2, -2)) # (2, 8)
array([ -6,  21,  46,  47,  75,  66,  66,  79,  81,  86,  88, 242])

Multidimensional arrays will use the same array of kth, you cannot
partition each axis by different values, you would have to explicitly
loop to do that.

Median is implemented in terms of partitioning already, but percentile
is not.
I would suggest someone else than me gives a try at implementing
percentile in terms of partition to see if the documentation and api
make sense to others.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130609/ecd0011e/attachment.sig>

From heng at cantab.net  Mon Jun 10 02:49:27 2013
From: heng at cantab.net (Henry Gomersall)
Date: Mon, 10 Jun 2013 07:49:27 +0100
Subject: [Numpy-discussion] supporting quad precision
In-Reply-To: <CAGY4rcVY5wAq5OffeTx2vmtHRahQSO6t5dfhOBiU-zge=MHBgg@mail.gmail.com>
References: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>
	<CAB6mnxKzCGEz2=iBJrX0dHqcfXdugqRWHTVXOJ24KjUCBONTTw@mail.gmail.com>
	<CAGY4rcV2_-HAzx-Wgfp96yHKKkjizx0UORXV1EWXtQ-NxijGbA@mail.gmail.com>
	<CANm_+Zq9hMr0dca39_0uUsV6VeN8R+aDiY-XvCqGGU8CnxZa-Q@mail.gmail.com>
	<1370763336.28467.9.camel@farnsworth>
	<CAGY4rcVY5wAq5OffeTx2vmtHRahQSO6t5dfhOBiU-zge=MHBgg@mail.gmail.com>
Message-ID: <1370846967.4332.3.camel@farnsworth>

On Sun, 2013-06-09 at 12:23 +0100, David Cournapeau wrote:
> On Sun, Jun 9, 2013 at 8:35 AM, Henry Gomersall <heng at cantab.net>
> wrote:
> > On Sat, 2013-06-08 at 14:35 +0200, Anne Archibald wrote:
> >> Looking at the rational module, I think you're right: it really
> >> shouldn't be too hard to get quads working as a user type using
> gcc's
> >> __float128 type, which will provide hardware arithmetic in the
> >> unlikely case that the user has hardware quads. Alternatively,
> >> probably more work, one could use a package like qd to provide
> >> portable quad precision (and quad-double).
> >
> > In this vague area, and further to a question I asked a while ago on
> > StackOverflow (http://stackoverflow.com/q/9062562/709852), is there
> some
> > deep reason why on some platforms longdouble is float128 and on
> other
> > it's float96?
> 
> Long double is not standardized (like single/double are), so it is CPU
> dependent. On Intel CPU, long double is generally translated into the
> extended precision 80 bits. On 32 bits, it is aligned to 12 bytes (the
> next multiple of 4 bytes), on 64 bits 16 bytes (the next multiple 8
> bytes). MS compilers are a notable exception where long double ==
> double.
> 
> So it depends on the CPU, the OS and the compiler. Using long double
> for anything else than compatibility (e.g. binary files) is often a
> mistake IMO, and highly unportable.

Interesting. So long double consistently maps to the platform specific
long double?

With my work on https://github.com/hgomersall/pyFFTW, which supports
long double as one of the data types, numpy's long double is absolutely
the right way to do this. Certainly I've managed reasonable success
across the three main OSs with this approach.

Cheers,

Henry





From robert.kern at gmail.com  Mon Jun 10 08:21:09 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 10 Jun 2013 13:21:09 +0100
Subject: [Numpy-discussion] supporting quad precision
In-Reply-To: <1370846967.4332.3.camel@farnsworth>
References: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>
	<CAB6mnxKzCGEz2=iBJrX0dHqcfXdugqRWHTVXOJ24KjUCBONTTw@mail.gmail.com>
	<CAGY4rcV2_-HAzx-Wgfp96yHKKkjizx0UORXV1EWXtQ-NxijGbA@mail.gmail.com>
	<CANm_+Zq9hMr0dca39_0uUsV6VeN8R+aDiY-XvCqGGU8CnxZa-Q@mail.gmail.com>
	<1370763336.28467.9.camel@farnsworth>
	<CAGY4rcVY5wAq5OffeTx2vmtHRahQSO6t5dfhOBiU-zge=MHBgg@mail.gmail.com>
	<1370846967.4332.3.camel@farnsworth>
Message-ID: <CAF6FJitcXKg1s08r0TjSQshTY9FHDsCjtq79LfyP5gb_DM0NvQ@mail.gmail.com>

On Mon, Jun 10, 2013 at 7:49 AM, Henry Gomersall <heng at cantab.net> wrote:
> On Sun, 2013-06-09 at 12:23 +0100, David Cournapeau wrote:

>> So it depends on the CPU, the OS and the compiler. Using long double
>> for anything else than compatibility (e.g. binary files) is often a
>> mistake IMO, and highly unportable.
>
> Interesting. So long double consistently maps to the platform specific
> long double?

numpy's longdouble dtype should consistently map to the C "long
double" type used by the compiler that was used to build numpy (not
necessarily the same as the one used to build any other extension!).

> With my work on https://github.com/hgomersall/pyFFTW, which supports
> long double as one of the data types, numpy's long double is absolutely
> the right way to do this. Certainly I've managed reasonable success
> across the three main OSs with this approach.

It's certainly the best option in that situation, but you may run into
problems with people using different compilers to build pyFFTW than
what was used to build the numpy binary they have installed.

--
Robert Kern


From heng at cantab.net  Mon Jun 10 08:26:35 2013
From: heng at cantab.net (Henry Gomersall)
Date: Mon, 10 Jun 2013 13:26:35 +0100
Subject: [Numpy-discussion] supporting quad precision
In-Reply-To: <CAF6FJitcXKg1s08r0TjSQshTY9FHDsCjtq79LfyP5gb_DM0NvQ@mail.gmail.com>
References: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>
	<CAB6mnxKzCGEz2=iBJrX0dHqcfXdugqRWHTVXOJ24KjUCBONTTw@mail.gmail.com>
	<CAGY4rcV2_-HAzx-Wgfp96yHKKkjizx0UORXV1EWXtQ-NxijGbA@mail.gmail.com>
	<CANm_+Zq9hMr0dca39_0uUsV6VeN8R+aDiY-XvCqGGU8CnxZa-Q@mail.gmail.com>
	<1370763336.28467.9.camel@farnsworth>
	<CAGY4rcVY5wAq5OffeTx2vmtHRahQSO6t5dfhOBiU-zge=MHBgg@mail.gmail.com>
	<1370846967.4332.3.camel@farnsworth>
	<CAF6FJitcXKg1s08r0TjSQshTY9FHDsCjtq79LfyP5gb_DM0NvQ@mail.gmail.com>
Message-ID: <1370867195.4332.12.camel@farnsworth>

On Mon, 2013-06-10 at 13:21 +0100, Robert Kern wrote:
> > With my work on https://github.com/hgomersall/pyFFTW, which supports
> > long double as one of the data types, numpy's long double is
> absolutely
> > the right way to do this. Certainly I've managed reasonable success
> > across the three main OSs with this approach.
> 
> It's certainly the best option in that situation, but you may run into
> problems with people using different compilers to build pyFFTW than
> what was used to build the numpy binary they have installed.

No doubt, though that problem goes deeper than my wrappers. It's
something I'm aware of at least!

Cheers,

Henry



From njs at pobox.com  Mon Jun 10 08:52:22 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 10 Jun 2013 13:52:22 +0100
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
	<CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
Message-ID: <CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>

On Sat, Jun 8, 2013 at 3:15 AM, Arink Verma <arinkverma at gmail.com> wrote:
> I tried to use pprof, but I can not find profiles to be used. like ls.prof
> in "pprof /bin/ls ls.prof"

You're looking for the ProfilerStart/ProfilerStop functions, the
former takes a filename to write the profiler to (like "ls.prof" or
"x-plus-x.prof"):
  http://www.mail-archive.com/numpy-discussion at scipy.org/msg41451.html

-n


From njs at pobox.com  Mon Jun 10 15:47:28 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 10 Jun 2013 20:47:28 +0100
Subject: [Numpy-discussion] empty_like for masked arrays
Message-ID: <CAPJVwB=x5AiQG1iUywF9HK+ueYE83-D9pAM=8+VDJMCEzT9XEQ@mail.gmail.com>

Hi all,

Is there anyone out there using numpy masked arrays, who has an
opinion on how empty_like (and its friends ones_like, zeros_like)
should handle the mask?

Right now apparently if you call np.ma.empty_like on a masked array,
you get a new masked array that shares the original array's mask, so
modifying one modifies the other. That's almost certainly wrong. This
PR:
  https://github.com/numpy/numpy/pull/3404
makes it so instead the new array has values that are all set to
empty/zero/one, and a mask which is set to match the input array's
mask (so whenever something was masked in the original array, the
empty/zero/one in that place is also masked). We don't know if this is
the desired behaviour for these functions, though. Maybe it's more
intuitive for the new array to match the original array in shape and
dtype, but to always have an empty mask. Or maybe not. None of us
really use np.ma, so if you do and have an opinion then please speak
up...

-n


From aldcroft at head.cfa.harvard.edu  Mon Jun 10 16:17:36 2013
From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas)
Date: Mon, 10 Jun 2013 16:17:36 -0400
Subject: [Numpy-discussion] empty_like for masked arrays
In-Reply-To: <CAPJVwB=x5AiQG1iUywF9HK+ueYE83-D9pAM=8+VDJMCEzT9XEQ@mail.gmail.com>
References: <CAPJVwB=x5AiQG1iUywF9HK+ueYE83-D9pAM=8+VDJMCEzT9XEQ@mail.gmail.com>
Message-ID: <CAMtEP6y4pq6hD5ox5onAzYGdMSjQx1OBbZW-7XBZWhTJH8tMKA@mail.gmail.com>

On Mon, Jun 10, 2013 at 3:47 PM, Nathaniel Smith <njs at pobox.com> wrote:

> Hi all,
>
> Is there anyone out there using numpy masked arrays, who has an
> opinion on how empty_like (and its friends ones_like, zeros_like)
> should handle the mask?
>
> Right now apparently if you call np.ma.empty_like on a masked array,
> you get a new masked array that shares the original array's mask, so
> modifying one modifies the other. That's almost certainly wrong. This
> PR:
>   https://github.com/numpy/numpy/pull/3404
> makes it so instead the new array has values that are all set to
> empty/zero/one, and a mask which is set to match the input array's
> mask (so whenever something was masked in the original array, the
> empty/zero/one in that place is also masked). We don't know if this is
> the desired behaviour for these functions, though. Maybe it's more
> intuitive for the new array to match the original array in shape and
> dtype, but to always have an empty mask. Or maybe not. None of us
> really use np.ma, so if you do and have an opinion then please speak
> up...
>

I use np.ma, and for me the most intuitive would be the second option where
the new array matches the original array in shape and dtype, but always has
an empty mask.  I always think of the *_like() functions as just copying
shape and dtype, so it would be a bit surprising to get part of the data
(the mask) from the original.  If you do need the mask then on the next
line you have an explicit statement to copy the mask and the code and
intent will be clear.  Also, most of the time the mask is set because that
particular data value was bad or missing, so it seems like it would be a
less-common use case to want a new empty array with the same mask.

My 2c,
Tom


>
> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130610/9325b607/attachment.html>

From efiring at hawaii.edu  Mon Jun 10 17:07:14 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Mon, 10 Jun 2013 11:07:14 -1000
Subject: [Numpy-discussion] empty_like for masked arrays
In-Reply-To: <CAMtEP6y4pq6hD5ox5onAzYGdMSjQx1OBbZW-7XBZWhTJH8tMKA@mail.gmail.com>
References: <CAPJVwB=x5AiQG1iUywF9HK+ueYE83-D9pAM=8+VDJMCEzT9XEQ@mail.gmail.com>
	<CAMtEP6y4pq6hD5ox5onAzYGdMSjQx1OBbZW-7XBZWhTJH8tMKA@mail.gmail.com>
Message-ID: <51B64002.7020506@hawaii.edu>

On 2013/06/10 10:17 AM, Aldcroft, Thomas wrote:
> I use np.ma <http://np.ma>, and for me the most intuitive would be the
> second option where the new array matches the original array in shape
> and dtype, but always has an empty mask.  I always think of the *_like()
> functions as just copying shape and dtype, so it would be a bit
> surprising to get part of the data (the mask) from the original.  If you
> do need the mask then on the next line you have an explicit statement to
> copy the mask and the code and intent will be clear.  Also, most of the
> time the mask is set because that particular data value was bad or
> missing, so it seems like it would be a less-common use case to want a
> new empty array with the same mask.
>

I also use np.ma (and it is used internally in matplotlib).  I agree 
with Tom.  I think all of the *_like() functions should start with 
mask=False, meaning nothing is masked by default.  I don't see what the 
reasonable use cases would be for any alternative.

Eric


From pgmdevlist at gmail.com  Mon Jun 10 18:40:30 2013
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 11 Jun 2013 00:40:30 +0200
Subject: [Numpy-discussion] =?utf-8?q?_Re=3A__empty=5Flike_for_masked_arra?=
 =?utf-8?q?ys?=
In-Reply-To: <51B64002.7020506@hawaii.edu>
References: <CAPJVwB=x5AiQG1iUywF9HK+ueYE83-D9pAM=8+VDJMCEzT9XEQ@mail.gmail.com>
	<CAMtEP6y4pq6hD5ox5onAzYGdMSjQx1OBbZW-7XBZWhTJH8tMKA@mail.gmail.com>
	<51B64002.7020506@hawaii.edu>
Message-ID: <etPan.51b655de.6b8b4567.2f8@platypus.local>

On June 10, 2013 at 23:07:24 , Eric Firing (efiring at hawaii.edu) wrote:
On 2013/06/10 10:17 AM, Aldcroft, Thomas wrote:  
> I use np.ma <http://np.ma>, and for me the most intuitive would be the  
> second option where the new array matches the original array in shape  
> and dtype, but always has an empty mask. I always think of the *_like()  
> functions as just copying shape and dtype, so it would be a bit  
> surprising to get part of the data (the mask) from the original. If you  
> do need the mask then on the next line you have an explicit statement to  
> copy the mask and the code and intent will be clear. Also, most of the  
> time the mask is set because that particular data value was bad or  
> missing, so it seems like it would be a less-common use case to want a  
> new empty array with the same mask.  
>  

I also use np.ma (and it is used internally in matplotlib). I agree  
with Tom. I think all of the *_like() functions should start with  
mask=False, meaning nothing is masked by default. I don't see what the  
reasonable use cases would be for any alternative.  

I too agree with Eric and Tom: having the mask set to `np.ma.nomask` by default makes more sense for the `*_like` functions than having a mask matching the input.
AFAICR, these functions were introduced for the sake of completion. I must admit I can't recall why there's an implementation difference between `np.ma.empty` and `np.ma.empty_like`, though? There must have been a corner case at one point or another.
It should be relatively easy to force `_convert2ma` to set the mask of the output to `False`. Note that if that would solve the problem for `empty_like`, `ones_like` and `zero_like` would have to be derived from `_convert2ma` too. Right now, `ones_like` and `zero_like` are just their exact `np` counterparts.
The problem of the mask shared between the input and output comes from the fact that in `_convert2ma`, the mask of the output is just a view of the mask of the input, not a copy.


From pgmdevlist at gmail.com  Tue Jun 11 04:21:59 2013
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 11 Jun 2013 10:21:59 +0200
Subject: [Numpy-discussion] =?utf-8?q?_Re=3A__empty=5Flike_for_masked_arra?=
 =?utf-8?q?ys?=
In-Reply-To: <etPan.51b655de.6b8b4567.2f8@platypus.local>
References: <CAPJVwB=x5AiQG1iUywF9HK+ueYE83-D9pAM=8+VDJMCEzT9XEQ@mail.gmail.com>
	<CAMtEP6y4pq6hD5ox5onAzYGdMSjQx1OBbZW-7XBZWhTJH8tMKA@mail.gmail.com>
	<51B64002.7020506@hawaii.edu>
	<etPan.51b655de.6b8b4567.2f8@platypus.local>
Message-ID: <etPan.51b6de27.6b8b4567.1e4@franceix-pierre.local>

On June 11, 2013 at 00:40:31, Pierre GM (pgmdevlist at gmail.com) wrote:
On June 10, 2013 at 23:07:24 , Eric Firing (efiring at hawaii.edu) wrote:  
On 2013/06/10 10:17 AM, Aldcroft, Thomas wrote:  
> I use np.ma <http://np.ma>, and for me the most intuitive would be the  
> second option where the new array matches the original array in shape  
> and dtype, but always has an empty mask. I always think of the *_like()  
> functions as just copying shape and dtype, so it would be a bit  
> surprising to get part of the data (the mask) from the original. If you  
> do need the mask then on the next line you have an explicit statement to  
> copy the mask and the code and intent will be clear. Also, most of the  
> time the mask is set because that particular data value was bad or  
> missing, so it seems like it would be a less-common use case to want a  
> new empty array with the same mask.  
>  

I also use np.ma (and it is used internally in matplotlib). I agree  
with Tom. I think all of the *_like() functions should start with  
mask=False, meaning nothing is masked by default. I don't see what the  
reasonable use cases would be for any alternative.  


Thinking about it, a replacement to `_convert2ma` should work something like
`func(input.view(np.ndarray)).view(np.ma.MaskedArray)`

The `input.view(np.ndarray)` will get the `.data` part of a MaskedArray and work seamlessly with a regular ndarray, the `.view(MaskedArray)` will attach a `np.ma.nomask` to the result of the previous view (therefore making the output a MaskedArray). Depending on your decision, we could add a flag to copy the initial mask (if any)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130611/26782e42/attachment.html>

From pav at iki.fi  Tue Jun 11 04:22:37 2013
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 11 Jun 2013 08:22:37 +0000 (UTC)
Subject: [Numpy-discussion] Dropping support for Accelerate/veclib?
Message-ID: <loom.20130611T101447-656@post.gmane.org>

Hi,

numpy.distutils provides support for Accelerate+veclib on OSX,
but does not provide Fortran compiler flags that ensure that
the Fortran ABI used is compatible. As a result ASUM, SDOT,
(apparently also DDOT), and other common BLAS functions
return garbage when called with the wrong ABI.

This breaks parts of Scipy. (There's some code to hack around
this --- it worked on earlier versions of OSX, but apparently
is unreliable on 10.8, demonstrating that this approach is just
too brittle.)

Moreover, third-party module developers who use Fortran will not
be aware of this issue, so it may also break 3rd party code.

As far as I see, the options are:

1. Add -ff2c (or whatever is the correct flag for Accelerate)
   to the Fortran flags on OSX.

2. Drop support for Veclib/Accelerate.

I think Accelerate was also incompatible with multiprocessing,
which would weigh for option 2.

-- 
Pauli Virtanen



From cournape at gmail.com  Tue Jun 11 04:52:01 2013
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 11 Jun 2013 09:52:01 +0100
Subject: [Numpy-discussion] Dropping support for Accelerate/veclib?
In-Reply-To: <loom.20130611T101447-656@post.gmane.org>
References: <loom.20130611T101447-656@post.gmane.org>
Message-ID: <CAGY4rcXg-d4=F162BybzC+kV96XG4eH-9wiG4XK4CaFX-72_Pg@mail.gmail.com>

On Tue, Jun 11, 2013 at 9:22 AM, Pauli Virtanen <pav at iki.fi> wrote:
> Hi,
>
> numpy.distutils provides support for Accelerate+veclib on OSX,
> but does not provide Fortran compiler flags that ensure that
> the Fortran ABI used is compatible. As a result ASUM, SDOT,
> (apparently also DDOT), and other common BLAS functions
> return garbage when called with the wrong ABI.
>
> This breaks parts of Scipy. (There's some code to hack around
> this --- it worked on earlier versions of OSX, but apparently
> is unreliable on 10.8, demonstrating that this approach is just
> too brittle.)
>
> Moreover, third-party module developers who use Fortran will not
> be aware of this issue, so it may also break 3rd party code.
>
> As far as I see, the options are:
>
> 1. Add -ff2c (or whatever is the correct flag for Accelerate)
>    to the Fortran flags on OSX.

What is the default ABI used on homebrew ? I think we should just
follow that, given that Apple cannot figure it out.

>
> 2. Drop support for Veclib/Accelerate.
>
> I think Accelerate was also incompatible with multiprocessing,
> which would weigh for option 2.

As for multiprocessing support, the problem will happen with other
libraries as well. Essentially, the way multiprocessing work (fork wo
exec) is simply not supported on OS X. Olivier Grisel from scikits
learn knows more about this (and has some workaround to use
multiprocessing with numpy/scipy on os x).

I will look into the issue with Accelerate on 10.8 (am still on 10.7
myself), but I would not mind dropping support for it if it is too
much of an hassle. I will look into the bento issue that makes it fail
on 64 bits python, and maybe using openblas should become the default
?


David


From pav at iki.fi  Tue Jun 11 05:17:30 2013
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 11 Jun 2013 09:17:30 +0000 (UTC)
Subject: [Numpy-discussion] Dropping support for Accelerate/veclib?
References: <loom.20130611T101447-656@post.gmane.org>
	<CAGY4rcXg-d4=F162BybzC+kV96XG4eH-9wiG4XK4CaFX-72_Pg@mail.gmail.com>
Message-ID: <loom.20130611T110037-518@post.gmane.org>

David Cournapeau <cournape <at> gmail.com> writes:
[clip]
> What is the default ABI used on homebrew ? I think we should just
> follow that, given that Apple cannot figure it out.

I think for Scipy homebrew uses the Gfortran ABI:
https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile

But that's probably the wrong thing to do, it doesn't work:
http://trac.macports.org/ticket/36694

For Octave, they have -ff2c:
https://trac.macports.org/browser/trunk/dports/math/octave/Portfile

    ***

A third option (maybe the best one) could be to add an ABI check
to numpy.distutils BLAS/LAPACK detection --- compile a small test
program that checks SDOT/CDOTU/DDOT etc., and refuse to use the
BLAS/LAPACK libraries if they give incorrect results. After that,
we can also remove the sdot/cdotu wrappers.

This approach is used by Octave.

This leaves the problem of dealing with Fortran ABI to those in
charge of the build environment, e.g., macports, Enthought, ...,
who are also in the best position to pick the correct solution
per each platform supported.

AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS,
so doing something like

    export FOPT="-ff2c"

or

    export LDFLAGS="-ldotwrp -lblas"

works? This makes things a bit more complicated to the builder, an
issue that can be solved with documentation, and keeping that up to
date is easier than hardcoding stuff into numpy.distutils.

-- 
Pauli Virtanen



From matthew.brett at gmail.com  Tue Jun 11 08:09:09 2013
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 11 Jun 2013 08:09:09 -0400
Subject: [Numpy-discussion] Dropping support for Accelerate/veclib?
In-Reply-To: <loom.20130611T110037-518@post.gmane.org>
References: <loom.20130611T101447-656@post.gmane.org>
	<CAGY4rcXg-d4=F162BybzC+kV96XG4eH-9wiG4XK4CaFX-72_Pg@mail.gmail.com>
	<loom.20130611T110037-518@post.gmane.org>
Message-ID: <CAH6Pt5q85b_MU1n5JCw2KP962BmRfWQM5b9_wZsJaONaxamjOA@mail.gmail.com>

Hi,

On Tue, Jun 11, 2013 at 5:17 AM, Pauli Virtanen <pav at iki.fi> wrote:
> David Cournapeau <cournape <at> gmail.com> writes:
> [clip]
>> What is the default ABI used on homebrew ? I think we should just
>> follow that, given that Apple cannot figure it out.
>
> I think for Scipy homebrew uses the Gfortran ABI:
> https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile
>
> But that's probably the wrong thing to do, it doesn't work:
> http://trac.macports.org/ticket/36694
>
> For Octave, they have -ff2c:
> https://trac.macports.org/browser/trunk/dports/math/octave/Portfile
>
>     ***
>
> A third option (maybe the best one) could be to add an ABI check
> to numpy.distutils BLAS/LAPACK detection --- compile a small test
> program that checks SDOT/CDOTU/DDOT etc., and refuse to use the
> BLAS/LAPACK libraries if they give incorrect results. After that,
> we can also remove the sdot/cdotu wrappers.
>
> This approach is used by Octave.
>
> This leaves the problem of dealing with Fortran ABI to those in
> charge of the build environment, e.g., macports, Enthought, ...,
> who are also in the best position to pick the correct solution
> per each platform supported.
>
> AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS,
> so doing something like
>
>     export FOPT="-ff2c"
>
> or
>
>     export LDFLAGS="-ldotwrp -lblas"
>
> works? This makes things a bit more complicated to the builder, an
> issue that can be solved with documentation, and keeping that up to
> date is easier than hardcoding stuff into numpy.distutils.

What will be the performance drop for the default OSX installer
version of numpy, if we drop Accelerate / veclib support?

Cheers,

Matthew


From jjhelmus at gmail.com  Tue Jun 11 08:37:41 2013
From: jjhelmus at gmail.com (Jonathan J. Helmus)
Date: Tue, 11 Jun 2013 08:37:41 -0400
Subject: [Numpy-discussion] quickselect
In-Reply-To: <51B49EEC.6000400@googlemail.com>
References: <CAB6mnxJtcm_VdqgLtMtxszQ1+MUW6kvf2d82noLW=fuxOe8EsA@mail.gmail.com>
	<CAMMTP+ArWn1hVswH6GXfJSbBtsAg3j9RZd2R52GzcjAeMjynEQ@mail.gmail.com>
	<51A62C0F.6040109@googlemail.com>
	<CAMMTP+D_hjjotGHcgG1PxFrS-yjEB2z5LXL=qYasi-E226NLZg@mail.gmail.com>
	<CAMMTP+A5YttfZQ2iXv_VckDZH+FRgUp4D+CCE6AhF3CS-C4Hmg@mail.gmail.com>
	<51B49EEC.6000400@googlemail.com>
Message-ID: <E22CE11B-A88C-495B-957D-76E79CE40B74@gmail.com>


On Jun 9, 2013, at 11:27 AM, Julian Taylor <jtaylor.debian at googlemail.com> wrote:

> On 09.06.2013 12:10, josef.pktd at gmail.com wrote:
>> On Wed, May 29, 2013 at 3:19 PM,  <josef.pktd at gmail.com> wrote:
>>> On Wed, May 29, 2013 at 12:25 PM, Julian Taylor
>>> <jtaylor.debian at googlemail.com> wrote:
>>>> On 05/29/2013 06:12 AM, josef.pktd at gmail.com wrote:
>>>>> 
>>>>> On Tue, May 28, 2013 at 6:31 PM, Charles R Harris
>>>>> <charlesr.harris at gmail.com> wrote:
>>>>>> 
>>>>>> Hi All,
>>>>>> 
>>>>>> There is a PR adding quickselect to numpy as a function `partition`.
>>>>>> Comments on name and exposure in the numpy API are welcome.
>>>>> 
>>>>> 
> 
> 
> here a a quick status report on the PR
> https://github.com/numpy/numpy/pull/3360
> 
> I now implemented partitioning via the introselect algorithm which is a
> quickselect median of 3 pivot with a cutoff on recursion depth to a
> median of median of 5 pivot for O(N) worst case complexity.
> Additionally it can stores its pivots for reuse in other partitions on
> the same data to reduce the space required to be partitioned next time,
> this is useful e.g. for multiple percentiles.
> 
> It is functionally ready, but there are still some API/ABI issues to be
> solved.
> Mainly deciding if we put the selection algorithms in the ABI for 1.8 or
> not. Currently the ABI is unchanged so user types cannot make use of the
> algorithms (they will fall back to quicksort).
> 
> The python api is now:
> np.partition(array, kth)
> where kth is an integer or array of integers
> 
> it will move each index in kth into its final sorted position, so
> np.partition(a, range(a.size)) results in a (inefficient) sort.
> e.g.:
> 
> d = np.array([66, 81, 21, 75, 46, -6, 66, 86, 242, 47, 88, 79])
> np.partition(d, (2, -2)) # (2, 8)
> array([ -6,  21,  46,  47,  75,  66,  66,  79,  81,  86,  88, 242])
> 
> Multidimensional arrays will use the same array of kth, you cannot
> partition each axis by different values, you would have to explicitly
> loop to do that.
> 
> Median is implemented in terms of partitioning already, but percentile
> is not.
> I would suggest someone else than me gives a try at implementing
> percentile in terms of partition to see if the documentation and api
> make sense to others.
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

Julian,


Since I am the author of the current percentile PR (https://github.com/numpy/numpy/pull/2970), I'm willing to try reimplementing percentile with the new partition functionality.  I don't expect to have time to do this until the Scipy conference in two week, so if anyone else wants to give the implementation a try please feel free.  Julian will you be at Scipy this year if I have any questions?  

	Cheers,

	- Jonathan Helmus

From jaime.frio at gmail.com  Tue Jun 11 12:24:08 2013
From: jaime.frio at gmail.com (=?ISO-8859-1?Q?Jaime_Fern=E1ndez_del_R=EDo?=)
Date: Tue, 11 Jun 2013 09:24:08 -0700
Subject: [Numpy-discussion] Will transpose ever need to copy data?
Message-ID: <CAPOWHW=aKyiiKD0X=1cnzFLBPvUnRd+wO5NS5NuRfGYJ-52GJw@mail.gmail.com>

I noticed today that the documentation for np.transpose states, for the
return value, that "A view is returned whenever possible."

Is there really any situation where swapping axes around could trigger the
need to copy data, or will a view always be returned no matter what?

I can't think of any such situation, and was planning on relying on that
for some code: basically, I have an output array, which I would like to be
contiguous. So I preallocate it with the right shape, then take a view of
it moving a certain axis to the end to make computations easier, run all my
computations on the modified view, then return the original array, not the
view.

If I started with an array with the axis at the end, and then transposed
it, I would need to make a copy to make it contiguous, which is what I am
trying to avoid.

Is this a bad practice? Is that precaution in the documentation real?
Should I check that my view's base is the original array and trigger a
copy, or is it an unnecessary check?

Thanks in advance,

Jaime

-- 
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes
de dominaci?n mundial.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130611/e133698c/attachment.html>

From n.becker at amolf.nl  Tue Jun 11 13:22:14 2013
From: n.becker at amolf.nl (Nils Becker)
Date: Tue, 11 Jun 2013 19:22:14 +0200
Subject: [Numpy-discussion] Dropping support for,	Accelerate/veclib?
Message-ID: <51B75CC6.4090405@amolf.nl>


> I think for Scipy homebrew uses the Gfortran ABI:
> https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile

fwiw, homebrew is not macports. it's a more recent replacement that
seems to be taking over gradually.


From chris.barker at noaa.gov  Tue Jun 11 15:44:27 2013
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Tue, 11 Jun 2013 12:44:27 -0700
Subject: [Numpy-discussion] Dropping support for, Accelerate/veclib?
In-Reply-To: <51B75CC6.4090405@amolf.nl>
References: <51B75CC6.4090405@amolf.nl>
Message-ID: <CALGmxEJZgGO20u+r4FigVVb2ep0jgdxmMm8jXhQZ-4LmbrhCfA@mail.gmail.com>

On Tue, Jun 11, 2013 at 10:22 AM, Nils Becker <n.becker at amolf.nl> wrote:
> fwiw, homebrew is not macports. it's a more recent replacement that
> seems to be taking over gradually.

And then there is (or was) fink----

Anyway, it would be really nice if numpy could work well out-of-the
box with the pyton.org python (and or the Apple-supplied one) without
any need fro macports, homebrew, etc. It's  actually pretty cool that
Apple provides veclib, and and I liked that it got used by default
with a simple "setup.py build".

It would be nice to keep it that way.

If a user (or distributor of binaries) wants to build the whole scipy
stack, they'll need to figure out all this *&^%& anyway, changing a
config in numpy to build differently would be the easiest part.

And yes, there really are folks that use numpy a lot without scipy.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From ralf.gommers at gmail.com  Tue Jun 11 16:28:27 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 11 Jun 2013 22:28:27 +0200
Subject: [Numpy-discussion] Dropping support for, Accelerate/veclib?
In-Reply-To: <CALGmxEJZgGO20u+r4FigVVb2ep0jgdxmMm8jXhQZ-4LmbrhCfA@mail.gmail.com>
References: <51B75CC6.4090405@amolf.nl>
	<CALGmxEJZgGO20u+r4FigVVb2ep0jgdxmMm8jXhQZ-4LmbrhCfA@mail.gmail.com>
Message-ID: <CABL7CQjtWf8-b6k=XH2xDarF=JmFMavy+=FvXbLR3k22brKtUQ@mail.gmail.com>

On Tue, Jun 11, 2013 at 9:44 PM, Chris Barker - NOAA Federal <
chris.barker at noaa.gov> wrote:

> On Tue, Jun 11, 2013 at 10:22 AM, Nils Becker <n.becker at amolf.nl> wrote:
> > fwiw, homebrew is not macports. it's a more recent replacement that
> > seems to be taking over gradually.
>
> And then there is (or was) fink----
>
> Anyway, it would be really nice if numpy could work well out-of-the
> box with the pyton.org python (and or the Apple-supplied one) without
> any need fro macports, homebrew, etc.


The binaries will still be built against python.org Python, so there
shouldn't be an issue here. Same for building from source.


> It's  actually pretty cool that
> Apple provides veclib, and and I liked that it got used by default
> with a simple "setup.py build".
>

Yeah, I still have an OS X 10.6 machine and Accelerate works great there.
But they screwed it up pretty bad in 10.7, and then made it worse for 10.8.
Not so cool anymore....

It would be nice to keep it that way.
>

> If a user (or distributor of binaries) wants to build the whole scipy
> stack, they'll need to figure out all this *&^%& anyway, changing a
> config in numpy to build differently would be the easiest part.
>
> And yes, there really are folks that use numpy a lot without scipy.
>

I think we have to indeed keep it easy to build from source. Maybe
providing a script to automatically fetch all dependencies and then build
(like MPL does) would be a good way to go.

Ralf


> -Chris
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130611/ee90ff13/attachment.html>

From ralf.gommers at gmail.com  Tue Jun 11 16:45:05 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 11 Jun 2013 22:45:05 +0200
Subject: [Numpy-discussion] Dropping support for Accelerate/veclib?
In-Reply-To: <CAH6Pt5q85b_MU1n5JCw2KP962BmRfWQM5b9_wZsJaONaxamjOA@mail.gmail.com>
References: <loom.20130611T101447-656@post.gmane.org>
	<CAGY4rcXg-d4=F162BybzC+kV96XG4eH-9wiG4XK4CaFX-72_Pg@mail.gmail.com>
	<loom.20130611T110037-518@post.gmane.org>
	<CAH6Pt5q85b_MU1n5JCw2KP962BmRfWQM5b9_wZsJaONaxamjOA@mail.gmail.com>
Message-ID: <CABL7CQjCZ=YH_2FfVEOr+kmh-mbWgeeZ2ihKtwmfF06DkCbaAA@mail.gmail.com>

On Tue, Jun 11, 2013 at 2:09 PM, Matthew Brett <matthew.brett at gmail.com>wrote:

> Hi,
>
> On Tue, Jun 11, 2013 at 5:17 AM, Pauli Virtanen <pav at iki.fi> wrote:
> > David Cournapeau <cournape <at> gmail.com> writes:
> > [clip]
> >> What is the default ABI used on homebrew ? I think we should just
> >> follow that, given that Apple cannot figure it out.
> >
> > I think for Scipy homebrew uses the Gfortran ABI:
> > https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile
> >
> > But that's probably the wrong thing to do, it doesn't work:
> > http://trac.macports.org/ticket/36694
> >
> > For Octave, they have -ff2c:
> > https://trac.macports.org/browser/trunk/dports/math/octave/Portfile
> >
> >     ***
> >
> > A third option (maybe the best one) could be to add an ABI check
> > to numpy.distutils BLAS/LAPACK detection --- compile a small test
> > program that checks SDOT/CDOTU/DDOT etc., and refuse to use the
> > BLAS/LAPACK libraries if they give incorrect results. After that,
> > we can also remove the sdot/cdotu wrappers.
> >
> > This approach is used by Octave.
> >
> > This leaves the problem of dealing with Fortran ABI to those in
> > charge of the build environment, e.g., macports, Enthought, ...,
> > who are also in the best position to pick the correct solution
> > per each platform supported.
> >
> > AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS,
> > so doing something like
> >
> >     export FOPT="-ff2c"
> >
> > or
> >
> >     export LDFLAGS="-ldotwrp -lblas"
> >
> > works? This makes things a bit more complicated to the builder, an
> > issue that can be solved with documentation, and keeping that up to
> > date is easier than hardcoding stuff into numpy.distutils.
>
> What will be the performance drop for the default OSX installer
> version of numpy, if we drop Accelerate / veclib support?
>

Answer on scipy-dev:
http://article.gmane.org/gmane.comp.python.scientific.devel/17864
Summary: it depends.
If anyone knows of better benchmarks, please share.

Joern Hees just wrote up how to install with OpenBLAS, if you want to know
for your application you can give it a try:
http://joernhees.de/blog/2013/06/08/mac-os-x-10-8-scientific-python-with-homebrew/

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130611/3682cff5/attachment.html>

From chris.barker at noaa.gov  Tue Jun 11 17:11:34 2013
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Tue, 11 Jun 2013 14:11:34 -0700
Subject: [Numpy-discussion] Dropping support for, Accelerate/veclib?
In-Reply-To: <CABL7CQjtWf8-b6k=XH2xDarF=JmFMavy+=FvXbLR3k22brKtUQ@mail.gmail.com>
References: <51B75CC6.4090405@amolf.nl>
	<CALGmxEJZgGO20u+r4FigVVb2ep0jgdxmMm8jXhQZ-4LmbrhCfA@mail.gmail.com>
	<CABL7CQjtWf8-b6k=XH2xDarF=JmFMavy+=FvXbLR3k22brKtUQ@mail.gmail.com>
Message-ID: <CALGmxEKv-By0KmrWSJ0Ura0g=E3rvwoXGjw7ttF7OeWddV9_5w@mail.gmail.com>

On Tue, Jun 11, 2013 at 1:28 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
> The binaries will still be built against python.org Python, so there
> shouldn't be an issue here. Same for building from source.

My point was that it's nice to be able to have it build with an out of
teh box wetup.py with accelerated LAPACK and all... If whoever is
building binaries wants to get fancy, great.

> Yeah, I still have an OS X 10.6 machine and Accelerate works great there.
> But they screwed it up pretty bad in 10.7, and then made it worse for 10.8.
> Not so cool anymore....

oh well, not on 10.8 yet myself -- if it's broken, it's broken :-(

> I think we have to indeed keep it easy to build from source. Maybe providing
> a script to automatically fetch all dependencies and then build (like MPL
> does) would be a good way to go.

Maybe, but stand alone scripts like that get pretty fragile ( I know,
I have a few of them myself...)  I really want to have a better system
in place to build binaries for the mac -- ideally a system that uses
the same infrastructure to build a variety of Mac packages, rather
than each package having it's own build scripts that need to be
maintained and  probably overlap with each other.

I"m looking at gattai as a system to build on.
(http://sourceforge.net/projects/gattai/)

There is also the issue of third-party dependencies (libpng,
libfreetype, blas, etc, etc) I kind of like how Anaconda appears to be
doing it. They have packages that have the shared libs in them, and
then other packages can depend on those.

Anyway, lots to do!

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From ralf.gommers at gmail.com  Tue Jun 11 17:29:24 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 11 Jun 2013 23:29:24 +0200
Subject: [Numpy-discussion] Dropping support for Accelerate/veclib?
In-Reply-To: <loom.20130611T110037-518@post.gmane.org>
References: <loom.20130611T101447-656@post.gmane.org>
	<CAGY4rcXg-d4=F162BybzC+kV96XG4eH-9wiG4XK4CaFX-72_Pg@mail.gmail.com>
	<loom.20130611T110037-518@post.gmane.org>
Message-ID: <CABL7CQhQtD4iaPCitdd5Qbm22bZF1bdd3b0RDEFCkEJ_S0_ZQw@mail.gmail.com>

On Tue, Jun 11, 2013 at 11:17 AM, Pauli Virtanen <pav at iki.fi> wrote:

> David Cournapeau <cournape <at> gmail.com> writes:
> [clip]
> > What is the default ABI used on homebrew ? I think we should just
> > follow that, given that Apple cannot figure it out.
>
> I think for Scipy homebrew uses the Gfortran ABI:
> https://trac.macports.org/browser/trunk/dports/python/py-scipy/Portfile
>
> But that's probably the wrong thing to do, it doesn't work:
> http://trac.macports.org/ticket/36694
>
> For Octave, they have -ff2c:
> https://trac.macports.org/browser/trunk/dports/math/octave/Portfile
>
>     ***
>
> A third option (maybe the best one) could be to add an ABI check
> to numpy.distutils BLAS/LAPACK detection --- compile a small test
> program that checks SDOT/CDOTU/DDOT etc., and refuse to use the
> BLAS/LAPACK libraries if they give incorrect results. After that,
> we can also remove the sdot/cdotu wrappers.
>
> This approach is used by Octave.
>
> This leaves the problem of dealing with Fortran ABI to those in
> charge of the build environment, e.g., macports, Enthought, ...,
> who are also in the best position to pick the correct solution
> per each platform supported.
>
> AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS,
> so doing something like
>
>     export FOPT="-ff2c"
>
> or
>
>     export LDFLAGS="-ldotwrp -lblas"
>
> works? This makes things a bit more complicated to the builder, an
> issue that can be solved with documentation, and keeping that up to
> date is easier than hardcoding stuff into numpy.distutils.
>

Sounds like a good idea. Would still make sense to move Accelerate down in
the list of preferred libs, so that one can install ATLAS, MKL or OpenBLAS
once and be done, instead of always having to remember these envvars.

https://github.com/scipy/scipy/issues/2256#issuecomment-17028321 has a
sample test. Minor issue is that it segfaults with the wrong ABI, instead
of giving an incorrect result.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130611/7bf1d541/attachment.html>

From njs at pobox.com  Tue Jun 11 17:41:00 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 11 Jun 2013 22:41:00 +0100
Subject: [Numpy-discussion] Dropping support for Accelerate/veclib?
In-Reply-To: <CABL7CQhQtD4iaPCitdd5Qbm22bZF1bdd3b0RDEFCkEJ_S0_ZQw@mail.gmail.com>
References: <loom.20130611T101447-656@post.gmane.org>
	<CAGY4rcXg-d4=F162BybzC+kV96XG4eH-9wiG4XK4CaFX-72_Pg@mail.gmail.com>
	<loom.20130611T110037-518@post.gmane.org>
	<CABL7CQhQtD4iaPCitdd5Qbm22bZF1bdd3b0RDEFCkEJ_S0_ZQw@mail.gmail.com>
Message-ID: <CAPJVwBmU3WjapWqTp56XxR-jx1h2ziOuA16Y4GbdvBTPXay3hg@mail.gmail.com>

On 11 Jun 2013 22:31, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:
> Sounds like a good idea. Would still make sense to move Accelerate down
in the list of preferred libs, so that one can install ATLAS, MKL or
OpenBLAS once and be done, instead of always having to remember these
envvars.

These days it might make sense to just make OpenBLAS the default on all
platforms.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130611/1352199d/attachment.html>

From pav at iki.fi  Tue Jun 11 17:57:05 2013
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 12 Jun 2013 00:57:05 +0300
Subject: [Numpy-discussion] Dropping support for Accelerate/veclib?
In-Reply-To: <CABL7CQhQtD4iaPCitdd5Qbm22bZF1bdd3b0RDEFCkEJ_S0_ZQw@mail.gmail.com>
References: <loom.20130611T101447-656@post.gmane.org>
	<CAGY4rcXg-d4=F162BybzC+kV96XG4eH-9wiG4XK4CaFX-72_Pg@mail.gmail.com>
	<loom.20130611T110037-518@post.gmane.org>
	<CABL7CQhQtD4iaPCitdd5Qbm22bZF1bdd3b0RDEFCkEJ_S0_ZQw@mail.gmail.com>
Message-ID: <kp86fb$em0$1@ger.gmane.org>

12.06.2013 00:29, Ralf Gommers kirjoitti:
[clip]
>     AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS,
>     so doing something like
>
>          export FOPT="-ff2c"
[clip]
> Sounds like a good idea. Would still make sense to move Accelerate down
> in the list of preferred libs, so that one can install ATLAS, MKL or
> OpenBLAS once and be done, instead of always having to remember these
> envvars.

Btw, it would be interesting to actually check if some combination of 
"-ff2c" in FOPT/FFLAGS/LDFLAGS/... fixes the Scipy Arpack issues.

	Pauli




From jtaylor.debian at googlemail.com  Tue Jun 11 18:32:53 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Wed, 12 Jun 2013 00:32:53 +0200
Subject: [Numpy-discussion] quickselect
In-Reply-To: <E22CE11B-A88C-495B-957D-76E79CE40B74@gmail.com>
References: <CAB6mnxJtcm_VdqgLtMtxszQ1+MUW6kvf2d82noLW=fuxOe8EsA@mail.gmail.com>
	<CAMMTP+ArWn1hVswH6GXfJSbBtsAg3j9RZd2R52GzcjAeMjynEQ@mail.gmail.com>
	<51A62C0F.6040109@googlemail.com>
	<CAMMTP+D_hjjotGHcgG1PxFrS-yjEB2z5LXL=qYasi-E226NLZg@mail.gmail.com>
	<CAMMTP+A5YttfZQ2iXv_VckDZH+FRgUp4D+CCE6AhF3CS-C4Hmg@mail.gmail.com>
	<51B49EEC.6000400@googlemail.com>
	<E22CE11B-A88C-495B-957D-76E79CE40B74@gmail.com>
Message-ID: <51B7A595.2090306@googlemail.com>

On 11.06.2013 14:37, Jonathan J. Helmus wrote:
>
> 
> Julian,
> 
> 
> Since I am the author of the current percentile PR
> (https://github.com/numpy/numpy/pull/2970), I'm willing to try
> reimplementing percentile with the new partition functionality.  I
> don't expect to have time to do this until the Scipy conference in
> two week, so if anyone else wants to give the implementation a try
> please feel free.  Julian will you be at Scipy this year if I have
> any questions?

I wont be at the Scipy in June, but I can be reached via email or IRC
(jtaylor on freenode and oftc) in the evening (UTC).

btw. you don't need the minimum "trick" currently used in the branch for
even element medians any more, so don't copy it for percentile.
partition is specialized for small kth elements to be almost as fast as
minimum, sometimes even faster if iterative partitioning is used (e.g.
np.partition(d, (mid, mid + 1)))


From derek at astro.physik.uni-goettingen.de  Tue Jun 11 19:43:38 2013
From: derek at astro.physik.uni-goettingen.de (Derek Homeier)
Date: Tue, 11 Jun 2013 16:43:38 -0700
Subject: [Numpy-discussion] genfromtxt and gzip
In-Reply-To: <51AF6CBB.4020606@theo.to>
References: <51AF6CBB.4020606@theo.to>
Message-ID: <D01B7652-F5CD-47F1-831E-02C3706B9A54@astro.physik.uni-goettingen.de>

On 05.06.2013, at 9:52AM, Ted To <rainexpected at theo.to> wrote:

>> From the list archives (2011), I noticed that there is a bug in the
> python gzip module that causes genfromtxt to fail with python 2 but this
> bug is not a problem for python 3.  When I tried to use genfromtxt and
> python 3 with a gzip'ed csv file, I instead got:
> 
> IOError: Mode rbU not supported
> 
> Is this a bug?  I am using python 3.2.3 and numpy 1.7.1 from the
> experimental Debian repository.

Interesting, it used to be the other way round indeed - at least Python3's gzip
module was believed to work with 'U' mode (universal newline conversion).
This was apparently fixed in Python 2.7.3:
http://bugs.python.org/issue5148

but from the closing comment I'd take it should indeed _not_ be used in Python 3

"The data corruption issue is now fixed in the 2.7 branch.

In 3.x, using a mode containing 'U' results in an exception rather than silent data corruption. 
Additionally, gzip.open() has supported text modes ("rt"/"wt"/"at") and newline translation since 3.3"

Checking the various Python versions on OS X 10.8 I found:

2.6.8: fails similar to the older 2.x, i.e. gzip opens with 'rbU', but then fails upon reading
(possibly randomly) with
/sw/lib/python2.6/gzip.pyc in _read_eof(self)
    302         if crc32 != self.crc:
    303             raise IOError("CRC check failed %s != %s" % (hex(crc32),
--> 304                                                          hex(self.crc)))

2.7.5: works as to be expected with the resolution of 5148 above.

3.1.5: works as well, which could just mean that the exception mentioned above has not
made it into the 3.1.x branch?

3.2.5+3.3.2: gzip.open raises the exception as documented.

This looks like the original issue, that the universal newline conversion should not be passed
to gzip.open (where it is meaningless or even harmful) is still not resolved; ideally the 'U' flag
should probably be caught in _datasource.py.
I take it from the comments on issue 5148 that 3.3's gzip module offers alternative methods to
do the newline conversion, but for 3.1, 3.2 and 2.6 this might still have to be done within either
_datasource.py or genfromtxt; however I have no idea if anyone has come up with a good
solution for this by now?

Cheers,
						Derek




From ralf.gommers at gmail.com  Wed Jun 12 02:25:25 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Wed, 12 Jun 2013 08:25:25 +0200
Subject: [Numpy-discussion] Dropping support for Accelerate/veclib?
In-Reply-To: <kp86fb$em0$1@ger.gmane.org>
References: <loom.20130611T101447-656@post.gmane.org>
	<CAGY4rcXg-d4=F162BybzC+kV96XG4eH-9wiG4XK4CaFX-72_Pg@mail.gmail.com>
	<loom.20130611T110037-518@post.gmane.org>
	<CABL7CQhQtD4iaPCitdd5Qbm22bZF1bdd3b0RDEFCkEJ_S0_ZQw@mail.gmail.com>
	<kp86fb$em0$1@ger.gmane.org>
Message-ID: <CABL7CQh=kuDYLeHDComEBAuDJyDR5ZNv7G=SL6w-H9X=2TmhUg@mail.gmail.com>

On Tue, Jun 11, 2013 at 11:57 PM, Pauli Virtanen <pav at iki.fi> wrote:

> 12.06.2013 00:29, Ralf Gommers kirjoitti:
> [clip]
> >     AFAIK custom compiler flags can be injected via FOPT/FFLAGS/LDFLAGS,
> >     so doing something like
> >
> >          export FOPT="-ff2c"
> [clip]
> > Sounds like a good idea. Would still make sense to move Accelerate down
> > in the list of preferred libs, so that one can install ATLAS, MKL or
> > OpenBLAS once and be done, instead of always having to remember these
> > envvars.
>
> Btw, it would be interesting to actually check if some combination of
> "-ff2c" in FOPT/FFLAGS/LDFLAGS/... fixes the Scipy Arpack issues.


https://github.com/scipy/scipy/issues/2256#issuecomment-17028321

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/eaa9d224/attachment.html>

From dave.hirschfeld at gmail.com  Wed Jun 12 06:07:30 2013
From: dave.hirschfeld at gmail.com (Dave Hirschfeld)
Date: Wed, 12 Jun 2013 10:07:30 +0000 (UTC)
Subject: [Numpy-discussion] datetime64 constructor ignores dtype argument?
Message-ID: <loom.20130612T120459-638@post.gmane.org>

The example below demonstrates the fact that the datetime64 constructor 
ignores the dtype argument if passed in. Is this conscious design decision or 
a bug/oversight?


In [55]: from datetime import datetime
    ...: d = datetime.now()
    ...: 

In [56]: d
Out[56]: datetime.datetime(2013, 6, 12, 11, 3, 27, 861000)

In [57]: np.datetime64(d)
Out[57]: numpy.datetime64('2013-06-12T12:03:27.861000+0100')

In [58]: np.datetime64(d).dtype
Out[58]: dtype('<M8[us]')

In [59]: np.datetime64(d, dtype='M8[D]')
Out[59]: numpy.datetime64('2013-06-12T12:03:27.861000+0100')

In [60]: np.datetime64(d, dtype='M8[D]').dtype
Out[60]: dtype('<M8[us]')   <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ???

In [61]: np.datetime64(d).astype('M8[D]')
Out[61]: numpy.datetime64('2013-06-12')


Thanks,
Dave



From sudheer.joseph at yahoo.com  Wed Jun 12 06:17:19 2013
From: sudheer.joseph at yahoo.com (Sudheer Joseph)
Date: Wed, 12 Jun 2013 18:17:19 +0800 (SGT)
Subject: [Numpy-discussion] t-statistic
Message-ID: <1371032239.89194.YahooMailNeo@web193403.mail.sg3.yahoo.com>

Dear experts,
???????????????? I am doing a project involving regression of a model variable with observed variable and wanted to find t-values dynamically as the number of available observations involved in comparison changes. Is there a tool in numpy/scipy which gives the appropriate t-value if we give number of samples ?

t = 2.31				# appropriate t value (where n=9, two tailed 95%)

with best regards,
Sudheer 

***************************************************************
Sudheer Joseph 
Indian National Centre for Ocean Information Services
Ministry of Earth Sciences, Govt. of India
POST BOX NO: 21, IDA Jeedeemetla P.O.
Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55
Tel:+91-40-23886047(O),Fax:+91-40-23895011(O),
Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile)
E-mail:sjo.India at gmail.com;sudheer.joseph at yahoo.com
Web- http://oppamthadathil.tripod.com
***************************************************************


From pav at iki.fi  Wed Jun 12 06:18:02 2013
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 12 Jun 2013 10:18:02 +0000 (UTC)
Subject: [Numpy-discussion] datetime64 constructor ignores dtype
	argument?
References: <loom.20130612T120459-638@post.gmane.org>
Message-ID: <loom.20130612T121712-28@post.gmane.org>

Dave Hirschfeld <dave.hirschfeld <at> gmail.com> writes:
> The example below demonstrates the fact that the datetime64
> constructor ignores the dtype argument if passed in. Is this
> conscious design decision or a bug/oversight?

Bug. There's probably no dtype argument, but the constructor
just ignores all arguments:

>>> np.datetime64('2012', spam='parrot')
2012-01-01 00:00:00




From pav at iki.fi  Wed Jun 12 06:19:02 2013
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 12 Jun 2013 10:19:02 +0000 (UTC)
Subject: [Numpy-discussion] t-statistic
References: <1371032239.89194.YahooMailNeo@web193403.mail.sg3.yahoo.com>
Message-ID: <loom.20130612T121826-42@post.gmane.org>

Sudheer Joseph <sudheer.joseph <at> yahoo.com> writes:
> I am doing a project involving regression of a model variable with
> observed variable and wanted to find t-values dynamically as the number of 
available observations
> involved in comparison changes. Is there a tool in numpy/scipy which gives 
the appropriate t-value if we
> give number of samples ?

The scipy-user mailing list is more appropriate for this question.

-- 
Pauli Virtanen



From njs at pobox.com  Wed Jun 12 08:10:15 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 12 Jun 2013 13:10:15 +0100
Subject: [Numpy-discussion] numpy.filled, again
Message-ID: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>

Hi all,

It looks like we've gotten a bit confused and need to untangle
something. There's a PR to add new functions 'np.filled' and
'np.filled_like':
  https://github.com/numpy/numpy/pull/2875
And there was a discussion about this on the list back in January:
  http://thread.gmane.org/gmane.comp.python.numeric.general/52763

I think a reasonable summary of the opinions in the thread are:
- This functionality is great, ...
- ...but we can't call it 'np.filled' because there's also
'np.ma.filled' which does something else...
- ...but there really aren't any better names...
- ...so we should overload np.empty, like: 'np.empty(shape, fill=value)'

In the mean time the original submitter has continued puttering along
polishing the original patch, and it's ready to merge... except it's
still the original interface, somehow the thread discussion and the PR
discussion never met up.

So, we have to decide what to do.

Personally I think that overloading np.empty is horribly ugly, will
continue confusing newbies and everyone else indefinitely, and I'm
100% convinced that we'll regret implementing such a warty interface
for something that should be so idiomatic. (Unfortunately I got busy
and didn't actually say this in the previous thread though.) So I
think we should just merge the PR as is. The only downside is the
np.ma inconsistency, but, np.ma is already inconsistent (cf.
masked_array.fill versus masked_array.filled!), somewhat deprecated,
and AFAICT there are far more people who will benefit from a clean
np.filled idiom than who actually use np.ma (and in particular its
fill-value functionality). So there would be two
bad-but-IMHO-acceptable options: either live with an inconsistency
between np.filled and np.ma.filled, or deprecate np.ma.filled in favor
of masked_array.filled (which does exactly the same thing) and
eventually switch np.ma.filled to be consistent with the new
np.filled.

But, that's just my opinion.

-n


From sebastian at sipsolutions.net  Wed Jun 12 08:24:53 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Wed, 12 Jun 2013 14:24:53 +0200
Subject: [Numpy-discussion] Will transpose ever need to copy data?
In-Reply-To: <CAPOWHW=aKyiiKD0X=1cnzFLBPvUnRd+wO5NS5NuRfGYJ-52GJw@mail.gmail.com>
References: <CAPOWHW=aKyiiKD0X=1cnzFLBPvUnRd+wO5NS5NuRfGYJ-52GJw@mail.gmail.com>
Message-ID: <1371039893.29545.15.camel@sebastian-laptop>

On Tue, 2013-06-11 at 09:24 -0700, Jaime Fern?ndez del R?o wrote:
> I noticed today that the documentation for np.transpose states, for
> the return value, that "A view is returned whenever possible."
> 
> 
I guess a subclass could cause a copy (the code looks like subclassing
doing something fancy is not be anticipated, there may be a bug here),
other then that, I don't see a reason for this comment. Transposing is
always possible for strided memory patterns and a copy will never occur.

- Sebastian


> Is there really any situation where swapping axes around could trigger
> the need to copy data, or will a view always be returned no matter
> what?
> 
> 
> I can't think of any such situation, and was planning on relying on
> that for some code: basically, I have an output array, which I would
> like to be contiguous. So I preallocate it with the right shape, then
> take a view of it moving a certain axis to the end to make
> computations easier, run all my computations on the modified view,
> then return the original array, not the view.
> 
> 
> If I started with an array with the axis at the end, and then
> transposed it, I would need to make a copy to make it contiguous,
> which is what I am trying to avoid.
> 
> 
> Is this a bad practice? Is that precaution in the documentation real?
> Should I check that my view's base is the original array and trigger a
> copy, or is it an unnecessary check?
> 
> 
> Thanks in advance,
> 
> 
> Jaime
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From daniele at grinta.net  Wed Jun 12 08:26:58 2013
From: daniele at grinta.net (Daniele Nicolodi)
Date: Wed, 12 Jun 2013 14:26:58 +0200
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
Message-ID: <51B86912.4090303@grinta.net>

On 12/06/2013 14:10, Nathaniel Smith wrote:
> It looks like we've gotten a bit confused and need to untangle
> something. There's a PR to add new functions 'np.filled' and
> 'np.filled_like':
>   https://github.com/numpy/numpy/pull/2875
> And there was a discussion about this on the list back in January:
>   http://thread.gmane.org/gmane.comp.python.numeric.general/52763
> 
> I think a reasonable summary of the opinions in the thread are:
> - This functionality is great, ...
> - ...but we can't call it 'np.filled' because there's also
> 'np.ma.filled' which does something else...
> - ...but there really aren't any better names...
> - ...so we should overload np.empty, like: 'np.empty(shape, fill=value)'

There where the additional proposal (mostly neglected on the original
thread) to add the 'fill' optional parameter to the array constructor:

np.ndarray(shape, fill=value)

has an obvious meaning to me.

I also don't really see why an np.empty() constructor exists, it seems
to do the same thing that np.ndarray() does.  However, I don't know if
adding a np.ndarray_like() constructor would make much sense.

Cheers,
Daniele



From matthew.brett at gmail.com  Wed Jun 12 08:28:27 2013
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 12 Jun 2013 13:28:27 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
Message-ID: <CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>

Hi,

On Wed, Jun 12, 2013 at 1:10 PM, Nathaniel Smith <njs at pobox.com> wrote:
> Hi all,
>
> It looks like we've gotten a bit confused and need to untangle
> something. There's a PR to add new functions 'np.filled' and
> 'np.filled_like':
>   https://github.com/numpy/numpy/pull/2875
> And there was a discussion about this on the list back in January:
>   http://thread.gmane.org/gmane.comp.python.numeric.general/52763
>
> I think a reasonable summary of the opinions in the thread are:
> - This functionality is great, ...
> - ...but we can't call it 'np.filled' because there's also
> 'np.ma.filled' which does something else...
> - ...but there really aren't any better names...
> - ...so we should overload np.empty, like: 'np.empty(shape, fill=value)'
>
> In the mean time the original submitter has continued puttering along
> polishing the original patch, and it's ready to merge... except it's
> still the original interface, somehow the thread discussion and the PR
> discussion never met up.

:) - a temptation that is can be hard to resist.

> So, we have to decide what to do.
>
> Personally I think that overloading np.empty is horribly ugly, will
> continue confusing newbies and everyone else indefinitely, and I'm
> 100% convinced that we'll regret implementing such a warty interface
> for something that should be so idiomatic. (Unfortunately I got busy
> and didn't actually say this in the previous thread though.)

Maybe you could unpack this, as I seem to remember this was the option
with the most support previously.

Cheers,

Matthew


From robert.kern at gmail.com  Wed Jun 12 08:29:57 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 12 Jun 2013 13:29:57 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B86912.4090303@grinta.net>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<51B86912.4090303@grinta.net>
Message-ID: <CAF6FJitU9FwYTKJ04yMRbNAQZn0wEVDGGHY05sia+m4gqVb2NQ@mail.gmail.com>

On Wed, Jun 12, 2013 at 1:26 PM, Daniele Nicolodi <daniele at grinta.net> wrote:

> There where the additional proposal (mostly neglected on the original
> thread) to add the 'fill' optional parameter to the array constructor:
>
> np.ndarray(shape, fill=value)
>
> has an obvious meaning to me.
>
> I also don't really see why an np.empty() constructor exists, it seems
> to do the same thing that np.ndarray() does.

It predates numpy and the numpy.ndarray type.

> However, I don't know if
> adding a np.ndarray_like() constructor would make much sense.

Also that.

--
Robert Kern


From pgmdevlist at gmail.com  Wed Jun 12 08:31:34 2013
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 12 Jun 2013 14:31:34 +0200
Subject: [Numpy-discussion] =?utf-8?q?numpy=2Efilled=2C_again?=
In-Reply-To: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
Message-ID: <etPan.51b86a26.46e87ccd.239c@franceix-pierre.local>

--
Pierre GM
Sent with Airmail
On June 12, 2013 at 14:10:27, Nathaniel Smith (njs at pobox.com) wrote:
Hi all,  

It looks like we've gotten a bit confused and need to untangle  
something. There's a PR to add new functions 'np.filled' and  
'np.filled_like':  
https://github.com/numpy/numpy/pull/2875  
And there was a discussion about this on the list back in January:  
http://thread.gmane.org/gmane.comp.python.numeric.general/52763  

I think a reasonable summary of the opinions in the thread are:  
- This functionality is great, ...  
- ...but we can't call it 'np.filled' because there's also  
'np.ma.filled' which does something else...  

I don't think this is a problem, is it? Different namespaces

Personally I think that overloading np.empty is horribly ugly, will  
continue confusing newbies and everyone else indefinitely, and I'm  
100% convinced that we'll regret implementing such a warty interface  
for something that should be so idiomatic. (Unfortunately I got busy  
and didn't actually say this in the previous thread though.) So I  
think we should just merge the PR as is. The only downside is the  
np.ma inconsistency, but, np.ma is already inconsistent (cf.  
masked_array.fill versus masked_array.filled!), somewhat deprecated,  

in np.ma, `.fill` is directly inherited of ndarray: *all* the elements are replaced by some value. `.filled` is completely different beast, replacing *only the masked entries* by a filling value. The name `.filled` was kept for backward compatibility with the original implementation (we're talking pre 1.2, the one inherited from numarray)


and AFAICT there are far more people who will benefit from a clean  
np.filled idiom than who actually use np.ma (and in particular its  
fill-value functionality). So there would be two  
bad-but-IMHO-acceptable options: either live with an inconsistency  
between np.filled and np.ma.filled, or deprecate np.ma.filled in favor  
of masked_array.filled (which does exactly the same thing) and  
eventually switch np.ma.filled to be consistent with the new  
np.filled.  

But, that's just my opinion.  

Not that I have any strong opinion about that, but the inconsistency looks better (less worse) to me than changing the behaviour of `np.ma.filled`. After all, the name of the function says it all: it fills the holes in a masked array. Anyhow, given my level of involvement these days (month/years), I'll go along a consensus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/6c6eaaa5/attachment.html>

From robert.kern at gmail.com  Wed Jun 12 08:31:19 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 12 Jun 2013 13:31:19 +0100
Subject: [Numpy-discussion] Will transpose ever need to copy data?
In-Reply-To: <1371039893.29545.15.camel@sebastian-laptop>
References: <CAPOWHW=aKyiiKD0X=1cnzFLBPvUnRd+wO5NS5NuRfGYJ-52GJw@mail.gmail.com>
	<1371039893.29545.15.camel@sebastian-laptop>
Message-ID: <CAF6FJis_Brd5Ec3acTSu_a-xjwGzE7z_Xear6Qn5=drqVO8FZw@mail.gmail.com>

On Wed, Jun 12, 2013 at 1:24 PM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> On Tue, 2013-06-11 at 09:24 -0700, Jaime Fern?ndez del R?o wrote:
>> I noticed today that the documentation for np.transpose states, for
>> the return value, that "A view is returned whenever possible."
>>
> I guess a subclass could cause a copy (the code looks like subclassing
> doing something fancy is not be anticipated, there may be a bug here),
> other then that, I don't see a reason for this comment. Transposing is
> always possible for strided memory patterns and a copy will never occur.

Indeed, there is no codepath in PyArray_Transpose() that even
contemplates a memory copy.

--
Robert Kern


From daniele at grinta.net  Wed Jun 12 08:47:56 2013
From: daniele at grinta.net (Daniele Nicolodi)
Date: Wed, 12 Jun 2013 14:47:56 +0200
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAF6FJitU9FwYTKJ04yMRbNAQZn0wEVDGGHY05sia+m4gqVb2NQ@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<51B86912.4090303@grinta.net>
	<CAF6FJitU9FwYTKJ04yMRbNAQZn0wEVDGGHY05sia+m4gqVb2NQ@mail.gmail.com>
Message-ID: <51B86DFC.4050501@grinta.net>

On 12/06/2013 14:29, Robert Kern wrote:
> On Wed, Jun 12, 2013 at 1:26 PM, Daniele Nicolodi <daniele at grinta.net> wrote:
> 
>> There where the additional proposal (mostly neglected on the original
>> thread) to add the 'fill' optional parameter to the array constructor:
>>
>> np.ndarray(shape, fill=value)
>>
>> has an obvious meaning to me.
>>
>> I also don't really see why an np.empty() constructor exists, it seems
>> to do the same thing that np.ndarray() does.
> 
> It predates numpy and the numpy.ndarray type.

I see, but I'm surprised by the fact that np.empty() is not an alias for
np.ndarray() neither is implemented in term of np.ndarray().

Since the constructor exists and it would be the "most pythonic" way of
constructing an ndarray object I don't see why it's usage should not be
encouraged.

>> However, I don't know if
>> adding a np.ndarray_like() constructor would make much sense.
> 
> Also that.

I don't understand if you are saying that adding the fill parameter to
np.ndarray() is a good or a bad idea.  If it is a bad idea, what are the
drawbacks compared to the other two proposed solutions?

Cheers,
Daniele



From josef.pktd at gmail.com  Wed Jun 12 08:57:19 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 12 Jun 2013 08:57:19 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B86DFC.4050501@grinta.net>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<51B86912.4090303@grinta.net>
	<CAF6FJitU9FwYTKJ04yMRbNAQZn0wEVDGGHY05sia+m4gqVb2NQ@mail.gmail.com>
	<51B86DFC.4050501@grinta.net>
Message-ID: <CAMMTP+Bsq+eQ0-Bh=MLF7usVoWP7ssgcHp1D-vLUdS+0-YzkxQ@mail.gmail.com>

On Wed, Jun 12, 2013 at 8:47 AM, Daniele Nicolodi <daniele at grinta.net> wrote:
> On 12/06/2013 14:29, Robert Kern wrote:
>> On Wed, Jun 12, 2013 at 1:26 PM, Daniele Nicolodi <daniele at grinta.net> wrote:
>>
>>> There where the additional proposal (mostly neglected on the original
>>> thread) to add the 'fill' optional parameter to the array constructor:
>>>
>>> np.ndarray(shape, fill=value)
>>>
>>> has an obvious meaning to me.
>>>
>>> I also don't really see why an np.empty() constructor exists, it seems
>>> to do the same thing that np.ndarray() does.
>>
>> It predates numpy and the numpy.ndarray type.
>
> I see, but I'm surprised by the fact that np.empty() is not an alias for
> np.ndarray() neither is implemented in term of np.ndarray().
>
> Since the constructor exists and it would be the "most pythonic" way of
> constructing an ndarray object I don't see why it's usage should not be
> encouraged.
>
>>> However, I don't know if
>>> adding a np.ndarray_like() constructor would make much sense.
>>
>> Also that.
>
> I don't understand if you are saying that adding the fill parameter to
> np.ndarray() is a good or a bad idea.  If it is a bad idea, what are the
> drawbacks compared to the other two proposed solutions?

The documentation for ndarray says

"Arrays should be constructed using array, zeros or empty (refer to
the See Also section below). The parameters given here refer to a
low-level method (ndarray(...)) for instantiating an array."

I never used ``ndarray``. (5 years +)

Sounds like a change in recommendation and pattern if we start to use
``ndarray`` directly.
I've never seen an argument why we shouldn't use it, it's just not in
any examples or code that I have seen. AFAICR

Josef


>
> Cheers,
> Daniele
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From bakhtiyor_zokhidov at mail.ru  Wed Jun 12 09:21:58 2013
From: bakhtiyor_zokhidov at mail.ru (=?UTF-8?B?QmFraHRpeW9yIFpva2hpZG92?=)
Date: Wed, 12 Jun 2013 17:21:58 +0400
Subject: [Numpy-discussion] =?utf-8?q?Avoiding_messy_code_in_Python?=
Message-ID: <1371043318.106215414@f150.mail.ru>

 Hi,

I have some code more or less 500 lines, but very messy code. All codes containing several functions?are in one module, besides, without documentation and testing.

Could anyone give me some advice to organize my messy code in an accurate style including test function as well?

Any answers will be appreciated,

-- 
Bakhtiyor Zokhidov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/b8a88833/attachment.html>

From robert.kern at gmail.com  Wed Jun 12 09:31:51 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 12 Jun 2013 14:31:51 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B86DFC.4050501@grinta.net>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<51B86912.4090303@grinta.net>
	<CAF6FJitU9FwYTKJ04yMRbNAQZn0wEVDGGHY05sia+m4gqVb2NQ@mail.gmail.com>
	<51B86DFC.4050501@grinta.net>
Message-ID: <CAF6FJitrGtGOntaxidtSLEVYOUdyR95EPk174orP5Xn9KHhKoQ@mail.gmail.com>

On Wed, Jun 12, 2013 at 1:47 PM, Daniele Nicolodi <daniele at grinta.net> wrote:
> On 12/06/2013 14:29, Robert Kern wrote:
>> On Wed, Jun 12, 2013 at 1:26 PM, Daniele Nicolodi <daniele at grinta.net> wrote:
>>
>>> There where the additional proposal (mostly neglected on the original
>>> thread) to add the 'fill' optional parameter to the array constructor:
>>>
>>> np.ndarray(shape, fill=value)
>>>
>>> has an obvious meaning to me.
>>>
>>> I also don't really see why an np.empty() constructor exists, it seems
>>> to do the same thing that np.ndarray() does.
>>
>> It predates numpy and the numpy.ndarray type.
>
> I see, but I'm surprised by the fact that np.empty() is not an alias for
> np.ndarray() neither is implemented in term of np.ndarray().
>
> Since the constructor exists and it would be the "most pythonic" way of
> constructing an ndarray object I don't see why it's usage should not be
> encouraged.
>
>>> However, I don't know if
>>> adding a np.ndarray_like() constructor would make much sense.
>>
>> Also that.
>
> I don't understand if you are saying that adding the fill parameter to
> np.ndarray() is a good or a bad idea.

My point there was that the ones()/ones_like(), zeros()/zeros_like()
pairings are useful and extending that to empty()/empty_like() is a
good thing. Dropping empty() in favor of ndarray() breaks that
symmetry, and ndarray_like() doesn't make sense as a name.

> If it is a bad idea, what are the
> drawbacks compared to the other two proposed solutions?

I think it has more drawbacks than the others. ndarray() is quite
low-level. There is very little intelligence in it, which is a good
thing. It is used in places where that low-level capability is
necessary, e.g. in the memmap implementation where it is constructing
an ndarray object that points to specific memory. array(),
empty_like(), ones(), etc. are relatively high level and have more
intelligence built into them. The "filled" functionality, however you
spell it, is part of the latter category. Implementing it in the low
level part interferes with ndarray()'s current low-level uses. You
*could* drop the current empty() in favor ndarray() if you wanted to
as empty() is not so smart and the defaults for the other ndarray()
arguments are sensible. But if you extend the functionality of empty()
to implement filled(), then that no longer holds.

--
Robert Kern


From njs at pobox.com  Wed Jun 12 09:48:25 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 12 Jun 2013 14:48:25 +0100
Subject: [Numpy-discussion] Will transpose ever need to copy data?
In-Reply-To: <CAPOWHW=aKyiiKD0X=1cnzFLBPvUnRd+wO5NS5NuRfGYJ-52GJw@mail.gmail.com>
References: <CAPOWHW=aKyiiKD0X=1cnzFLBPvUnRd+wO5NS5NuRfGYJ-52GJw@mail.gmail.com>
Message-ID: <CAPJVwBm0N=BqUJAU8saRtUig65S+knpms5jKZ_FO3vpx-emn0Q@mail.gmail.com>

On Tue, Jun 11, 2013 at 5:24 PM, Jaime Fern?ndez del R?o
<jaime.frio at gmail.com> wrote:
> I noticed today that the documentation for np.transpose states, for the
> return value, that "A view is returned whenever possible."
>
> Is there really any situation where swapping axes around could trigger the
> need to copy data, or will a view always be returned no matter what?

Sounds like a doc bug. (Probably someone being over-careful -- the
default for many operations in numpy is that it's undefined whether
they return a view or not, so if it makes a difference to you you need
to take an explicit copy or similar.) Want to submit a PR to fix the
doc?

-n


From njs at pobox.com  Wed Jun 12 10:18:31 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 12 Jun 2013 15:18:31 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
Message-ID: <CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>

On Wed, Jun 12, 2013 at 1:28 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> On Wed, Jun 12, 2013 at 1:10 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> Personally I think that overloading np.empty is horribly ugly, will
>> continue confusing newbies and everyone else indefinitely, and I'm
>> 100% convinced that we'll regret implementing such a warty interface
>> for something that should be so idiomatic. (Unfortunately I got busy
>> and didn't actually say this in the previous thread though.)
>
> Maybe you could unpack this, as I seem to remember this was the option
> with the most support previously.

Indeed it was, which is why I brought it up :-).

I'm not sure what more there is to unpack, though. It's just...
offensive to every sense of API design I have, I don't know how to
explain more than I have. I speculate that it's only attraction is
that it showed up at the end of a 50 email thread and offered the
promise of ending things, but I don't know.

Well, here's maybe another way of getting at the ugliness.

Here's the current doc page listing all the options for creating an
array -- a very useful reference, esp. while learning:
  http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html

Now imagine a new version of this page, if we add 'filled'. There will
be a list at the top with functions named:
  empty
  filled
  ones
  zeros
It's immediately obvious what all of these things do, and how they
differ from each other, and in which situation you might want each,
just from the names, even before you read the one-line synopses. Even
more so if you know about the existence of np.fill(). The synopses for
'ones' and 'zeros' don't even have to change, they already use the
word 'filled' to describe what they do. It all just works.

Now imagine a different new version of this page, if we overload
'empty' to add a fill= option. I don't even know how we document that
on this page. The list will remain:
  empty
  ones
  zeros
So there will be no clue there how you get an array filled with NaNs
or whatever, or even any hint that it's possible. Then there's the
prose on the right. Right now the synopsis for 'empty' is:
  Return a new array of given shape and type, without initializing entries.
I guess we change this to
  Return a new array of given shape and type, without initializing
entries, OR return a new array of given shape and type, with values
initialized to some specific value.
? IMO probably the single best criterion for judging whether your API
is good, is whether you can write clean and pretty docs for it. This
fails that test horribly...

We probably should advertise the ndarray constructor more, and
possibly make it more generally useful, but the current situation for
better or worse is that we've spent many years telling people that
it's a weird low-level thing that they shouldn't use. (I didn't even
know how it worked until 10 minutes ago!) Adding this functionality
there means it'll still be hidden away, so it's not a great solution
to the 'filled' problem, and it doesn't really move us any closer to
having a coherent story on when you should use the ndarray constructor
either.

So IMO the best (least bad) solution on offer is still to just add a
'filled' function, and live with the np.ma inconsistency.

-n


From warren.weckesser at gmail.com  Wed Jun 12 10:45:49 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Wed, 12 Jun 2013 10:45:49 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
	<CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
Message-ID: <CAGzF1uf9=R9gvM3RBQtEwd3AxwiYAM+0hpA5To6jymTJPOG9dQ@mail.gmail.com>

On Wed, Jun 12, 2013 at 10:18 AM, Nathaniel Smith <njs at pobox.com> wrote:

> On Wed, Jun 12, 2013 at 1:28 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> > On Wed, Jun 12, 2013 at 1:10 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >> Personally I think that overloading np.empty is horribly ugly, will
> >> continue confusing newbies and everyone else indefinitely, and I'm
> >> 100% convinced that we'll regret implementing such a warty interface
> >> for something that should be so idiomatic. (Unfortunately I got busy
> >> and didn't actually say this in the previous thread though.)
> >
> > Maybe you could unpack this, as I seem to remember this was the option
> > with the most support previously.
>
> Indeed it was, which is why I brought it up :-).
>
> I'm not sure what more there is to unpack, though. It's just...
> offensive to every sense of API design I have, I don't know how to
> explain more than I have. I speculate that it's only attraction is
> that it showed up at the end of a 50 email thread and offered the
> promise of ending things, but I don't know.
>
> Well, here's maybe another way of getting at the ugliness.
>
> Here's the current doc page listing all the options for creating an
> array -- a very useful reference, esp. while learning:
>   http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html
>
> Now imagine a new version of this page, if we add 'filled'. There will
> be a list at the top with functions named:
>   empty
>   filled
>   ones
>   zeros
> It's immediately obvious what all of these things do, and how they
> differ from each other, and in which situation you might want each,
> just from the names, even before you read the one-line synopses. Even
> more so if you know about the existence of np.fill(). The synopses for
> 'ones' and 'zeros' don't even have to change, they already use the
> word 'filled' to describe what they do. It all just works.
>
> Now imagine a different new version of this page, if we overload
> 'empty' to add a fill= option. I don't even know how we document that
> on this page. The list will remain:
>   empty
>   ones
>   zeros
> So there will be no clue there how you get an array filled with NaNs
> or whatever, or even any hint that it's possible. Then there's the
> prose on the right. Right now the synopsis for 'empty' is:
>   Return a new array of given shape and type, without initializing entries.
> I guess we change this to
>   Return a new array of given shape and type, without initializing
> entries, OR return a new array of given shape and type, with values
> initialized to some specific value.
> ? IMO probably the single best criterion for judging whether your API
> is good, is whether you can write clean and pretty docs for it. This
> fails that test horribly...
>
> We probably should advertise the ndarray constructor more, and
> possibly make it more generally useful, but the current situation for
> better or worse is that we've spent many years telling people that
> it's a weird low-level thing that they shouldn't use. (I didn't even
> know how it worked until 10 minutes ago!) Adding this functionality
> there means it'll still be hidden away, so it's not a great solution
> to the 'filled' problem, and it doesn't really move us any closer to
> having a coherent story on when you should use the ndarray constructor
> either.
>
> So IMO the best (least bad) solution on offer is still to just add a
> 'filled' function, and live with the np.ma inconsistency.
>
>

Another idea (also imperfect): call the new functions `filledwith` and
`filledwith_like`.  Not as concise as `filled`, but the meaning is still
clear, and it avoids the clash with `ma.filled`.

Warren



> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/868e0ba8/attachment.html>

From pierre.haessig at crans.org  Wed Jun 12 10:49:11 2013
From: pierre.haessig at crans.org (Pierre Haessig)
Date: Wed, 12 Jun 2013 16:49:11 +0200
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
	<CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
Message-ID: <51B88A67.6020800@crans.org>

Hi,

Le 12/06/2013 16:18, Nathaniel Smith a ?crit :
> Now imagine a new version of this page, if we add 'filled'. There will
> be a list at the top with functions named:
>   empty
>   filled
>   ones
>   zeros
> It's immediately obvious what all of these things do, and how they
> differ from each other, and in which situation you might want each,
> just from the names, even before you read the one-line synopses. Even
> more so if you know about the existence of np.fill(). The synopses for
> 'ones' and 'zeros' don't even have to change, they already use the
> word 'filled' to describe what they do. It all just works.
I find the "docs would look nice and consistent" argument pretty convincing.
+1 for np.filled.

best,
Pierre


From josef.pktd at gmail.com  Wed Jun 12 11:29:26 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 12 Jun 2013 11:29:26 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAGzF1uf9=R9gvM3RBQtEwd3AxwiYAM+0hpA5To6jymTJPOG9dQ@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
	<CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
	<CAGzF1uf9=R9gvM3RBQtEwd3AxwiYAM+0hpA5To6jymTJPOG9dQ@mail.gmail.com>
Message-ID: <CAMMTP+A50jGRe_8yh-bqP3M_V6o9KyMd9Dkge_PVX9KwV_SBUw@mail.gmail.com>

On Wed, Jun 12, 2013 at 10:45 AM, Warren Weckesser
<warren.weckesser at gmail.com> wrote:
>
>
>
> On Wed, Jun 12, 2013 at 10:18 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Wed, Jun 12, 2013 at 1:28 PM, Matthew Brett <matthew.brett at gmail.com>
>> wrote:
>> > On Wed, Jun 12, 2013 at 1:10 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> >> Personally I think that overloading np.empty is horribly ugly, will
>> >> continue confusing newbies and everyone else indefinitely, and I'm
>> >> 100% convinced that we'll regret implementing such a warty interface
>> >> for something that should be so idiomatic. (Unfortunately I got busy
>> >> and didn't actually say this in the previous thread though.)
>> >
>> > Maybe you could unpack this, as I seem to remember this was the option
>> > with the most support previously.
>>
>> Indeed it was, which is why I brought it up :-).
>>
>> I'm not sure what more there is to unpack, though. It's just...
>> offensive to every sense of API design I have, I don't know how to
>> explain more than I have. I speculate that it's only attraction is
>> that it showed up at the end of a 50 email thread and offered the
>> promise of ending things, but I don't know.
>>
>> Well, here's maybe another way of getting at the ugliness.
>>
>> Here's the current doc page listing all the options for creating an
>> array -- a very useful reference, esp. while learning:
>>   http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html
>>
>> Now imagine a new version of this page, if we add 'filled'. There will
>> be a list at the top with functions named:
>>   empty
>>   filled
>>   ones
>>   zeros
>> It's immediately obvious what all of these things do, and how they
>> differ from each other, and in which situation you might want each,
>> just from the names, even before you read the one-line synopses. Even
>> more so if you know about the existence of np.fill(). The synopses for
>> 'ones' and 'zeros' don't even have to change, they already use the
>> word 'filled' to describe what they do. It all just works.
>>
>> Now imagine a different new version of this page, if we overload
>> 'empty' to add a fill= option. I don't even know how we document that
>> on this page. The list will remain:
>>   empty
>>   ones
>>   zeros
>> So there will be no clue there how you get an array filled with NaNs
>> or whatever, or even any hint that it's possible. Then there's the
>> prose on the right. Right now the synopsis for 'empty' is:
>>   Return a new array of given shape and type, without initializing
>> entries.
>> I guess we change this to
>>   Return a new array of given shape and type, without initializing
>> entries, OR return a new array of given shape and type, with values
>> initialized to some specific value.
>> ? IMO probably the single best criterion for judging whether your API
>> is good, is whether you can write clean and pretty docs for it. This
>> fails that test horribly...
>>
>> We probably should advertise the ndarray constructor more, and
>> possibly make it more generally useful, but the current situation for
>> better or worse is that we've spent many years telling people that
>> it's a weird low-level thing that they shouldn't use. (I didn't even
>> know how it worked until 10 minutes ago!) Adding this functionality
>> there means it'll still be hidden away, so it's not a great solution
>> to the 'filled' problem, and it doesn't really move us any closer to
>> having a coherent story on when you should use the ndarray constructor
>> either.
>>
>> So IMO the best (least bad) solution on offer is still to just add a
>> 'filled' function, and live with the np.ma inconsistency.

+2 on this.
I like a good name like `filled` for a function that I expect to use regularly.
it's short and descriptive.

Josef

>>
>
>
> Another idea (also imperfect): call the new functions `filledwith` and
> `filledwith_like`.  Not as concise as `filled`, but the meaning is still
> clear, and it avoids the clash with `ma.filled`.
>
> Warren
>
>
>>
>> -n
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From alan.isaac at gmail.com  Wed Jun 12 11:30:14 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Wed, 12 Jun 2013 11:30:14 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B88A67.6020800@crans.org>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
	<CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
	<51B88A67.6020800@crans.org>
Message-ID: <51B89406.2080701@gmail.com>

> Le 12/06/2013 16:18, Nathaniel Smith a ?crit :
>> Now imagine a new version of this page, if we add 'filled'. There will
>> be a list at the top with functions named:
>>    empty
>>    filled
>>    ones
>>    zeros
>> It's immediately obvious what all of these things do, and how they
>> differ from each other, and in which situation you might want each,
>> just from the names, even before you read the one-line synopses. Even
>> more so if you know about the existence of np.fill(). The synopses for
>> 'ones' and 'zeros' don't even have to change, they already use the
>> word 'filled' to describe what they do. It all just works.


On 6/12/2013 10:49 AM, Pierre Haessig wrote:
> I find the "docs would look nice and consistent" argument pretty convincing.
> +1 for np.filled.


In Maple and Mathematica, these are called constant arrays.
Thus (?) another possible name is:
np.constant

Alan Isaac

PS http://reference.wolfram.com/mathematica/ref/ConstantArray.html
http://www.maplesoft.com/support/help/Maple/view.aspx?path=Student/LinearAlgebra/ConstantMatrix



From njs at pobox.com  Wed Jun 12 11:45:28 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 12 Jun 2013 16:45:28 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B89406.2080701@gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
	<CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
	<51B88A67.6020800@crans.org> <51B89406.2080701@gmail.com>
Message-ID: <CAPJVwBmx2NsuUxew8cbktOf82f5GeHpx1+JD6hybHELteJP=Ww@mail.gmail.com>

On Wed, Jun 12, 2013 at 4:30 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> On 6/12/2013 10:49 AM, Pierre Haessig wrote:
>> I find the "docs would look nice and consistent" argument pretty convincing.
>> +1 for np.filled.
>
> In Maple and Mathematica, these are called constant arrays.
> Thus (?) another possible name is:
> np.constant

This came up in the previous thread as well... the problem is that
'constant' has another even stronger meaning (in computing and
science!) of 'fixed', 'unchanging', 'read-only' (cf. C's 'const', and
the scipy.constants module). And numpy arrays do in fact have a flag
that makes them read-only. But this would not be the function that
sets it.

-n


From warren.weckesser at gmail.com  Wed Jun 12 11:54:46 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Wed, 12 Jun 2013 11:54:46 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
	<CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
Message-ID: <CAGzF1uckgRyAHAkE+tNvo9bRH7bcWmgvgAzFT9LF--o9_MVZwA@mail.gmail.com>

On Wed, Jun 12, 2013 at 10:18 AM, Nathaniel Smith <njs at pobox.com> wrote:

> On Wed, Jun 12, 2013 at 1:28 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> > On Wed, Jun 12, 2013 at 1:10 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >> Personally I think that overloading np.empty is horribly ugly, will
> >> continue confusing newbies and everyone else indefinitely, and I'm
> >> 100% convinced that we'll regret implementing such a warty interface
> >> for something that should be so idiomatic. (Unfortunately I got busy
> >> and didn't actually say this in the previous thread though.)
> >
> > Maybe you could unpack this, as I seem to remember this was the option
> > with the most support previously.
>
> Indeed it was, which is why I brought it up :-).
>
> I'm not sure what more there is to unpack, though. It's just...
> offensive to every sense of API design I have, I don't know how to
> explain more than I have. I speculate that it's only attraction is
> that it showed up at the end of a 50 email thread and offered the
> promise of ending things, but I don't know.
>
> Well, here's maybe another way of getting at the ugliness.
>
> Here's the current doc page listing all the options for creating an
> array -- a very useful reference, esp. while learning:
>   http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html
>
> Now imagine a new version of this page, if we add 'filled'. There will
> be a list at the top with functions named:
>   empty
>   filled
>   ones
>   zeros
> It's immediately obvious what all of these things do, and how they
> differ from each other, and in which situation you might want each,
> just from the names, even before you read the one-line synopses. Even
> more so if you know about the existence of np.fill(). The synopses for
> 'ones' and 'zeros' don't even have to change, they already use the
> word 'filled' to describe what they do. It all just works.
>
> Now imagine a different new version of this page, if we overload
> 'empty' to add a fill= option. I don't even know how we document that
> on this page. The list will remain:
>   empty
>   ones
>   zeros
> So there will be no clue there how you get an array filled with NaNs
> or whatever, or even any hint that it's possible. Then there's the
> prose on the right. Right now the synopsis for 'empty' is:
>   Return a new array of given shape and type, without initializing entries.
> I guess we change this to
>   Return a new array of given shape and type, without initializing
> entries, OR return a new array of given shape and type, with values
> initialized to some specific value.
> ? IMO probably the single best criterion for judging whether your API
> is good, is whether you can write clean and pretty docs for it. This
> fails that test horribly...
>
> We probably should advertise the ndarray constructor more, and
> possibly make it more generally useful, but the current situation for
> better or worse is that we've spent many years telling people that
> it's a weird low-level thing that they shouldn't use. (I didn't even
> know how it worked until 10 minutes ago!) Adding this functionality
> there means it'll still be hidden away, so it's not a great solution
> to the 'filled' problem, and it doesn't really move us any closer to
> having a coherent story on when you should use the ndarray constructor
> either.
>
> So IMO the best (least bad) solution on offer is still to just add a
> 'filled' function, and live with the np.ma inconsistency.
>
>

But then what do you call the ma version of the new `filled` and
`filled_like` functions?  Presumably they should be implemented as part of
the pull request.

Warren



> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/56887fef/attachment.html>

From davidmenhur at gmail.com  Wed Jun 12 11:56:04 2013
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Wed, 12 Jun 2013 17:56:04 +0200
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAMMTP+A50jGRe_8yh-bqP3M_V6o9KyMd9Dkge_PVX9KwV_SBUw@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
	<CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
	<CAGzF1uf9=R9gvM3RBQtEwd3AxwiYAM+0hpA5To6jymTJPOG9dQ@mail.gmail.com>
	<CAMMTP+A50jGRe_8yh-bqP3M_V6o9KyMd9Dkge_PVX9KwV_SBUw@mail.gmail.com>
Message-ID: <CAJhcF=03OnpWXkZn7x7yKmghP1Q_Fkr-m786T_HSrjCg0_3t5Q@mail.gmail.com>

On 12 June 2013 17:29,  <josef.pktd at gmail.com> wrote:
> +2 on this.
> I like a good name like `filled` for a function that I expect to use regularly.
> it's short and descriptive.

FTIW, I am for this one too. That is not only clear, but a name that
one may try even before checking the docs.

Emtpy-filled sounds too much of an oximoron to me.


From uschmitt at mineway.de  Wed Jun 12 12:07:30 2013
From: uschmitt at mineway.de (Uwe Schmitt)
Date: Wed, 12 Jun 2013 18:07:30 +0200
Subject: [Numpy-discussion] svd + multiprocessing hangs
Message-ID: <51B89CC2.5070901@mineway.de>

Dear all,

the following code hangs on my Ubuntu machine.
I use self compiled numpy 1.7.1 and Python
2.7.3

-----

import numpy
import numpy.linalg
import multiprocessing

def classify():
    X = numpy.random.random((10,3))
    print "before svd"
    numpy.linalg.svd(X)
    print "this message is not printed"


if __name__ == "__main__":
    p = multiprocessing.Process(target=classify, args=())
    p.start()
    p.join()

-----

Regards, Uwe.

-- 
Dr. rer. nat. Uwe Schmitt
Leitung F/E Mathematik

mineway GmbH
Geb?ude 4
Im Helmerswald 2
66121 Saarbr?cken

Telefon: +49 (0)681 8390 5334
Telefax: +49 (0)681 830 4376

uschmitt at mineway.de
www.mineway.de

Gesch?ftsf?hrung: Dr.-Ing. Mathias Bauer
Amtsgericht Saarbr?cken HRB 12339







-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 553 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/077a08a3/attachment.sig>

From josef.pktd at gmail.com  Wed Jun 12 12:08:26 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 12 Jun 2013 12:08:26 -0400
Subject: [Numpy-discussion] np.asarray atleast_float64 ?
In-Reply-To: <CAMMTP+DPfG4_C4XHg1+wf65uE8V3-jX005GVY6A1cs0vhRV6nw@mail.gmail.com>
References: <CAMMTP+Cm6F17ooVaGM82L8skDzgGx-YaJQOdthk9EY+2GdpCrw@mail.gmail.com>
	<1370732755.2318.4.camel@sebastian-laptop>
	<CAMMTP+DPfG4_C4XHg1+wf65uE8V3-jX005GVY6A1cs0vhRV6nw@mail.gmail.com>
Message-ID: <CAMMTP+CcVO8ycFjdRNrkOuMDpeik=aXgDsOFnn9e9DSmcq+cOw@mail.gmail.com>

On Sat, Jun 8, 2013 at 7:31 PM,  <josef.pktd at gmail.com> wrote:
> On Sat, Jun 8, 2013 at 7:05 PM, Sebastian Berg
> <sebastian at sipsolutions.net> wrote:
>> On Sat, 2013-06-08 at 08:52 -0400, josef.pktd at gmail.com wrote:
>>> Is there anything to require a numpy array with a minimum numeric dtype?
>>>
>>> To avoid lower precision calculations and be upwards compatible, something like
>>>
>>> x = np.asarray(x, >=np.float64)
>>
>> np.result_type(arr, np.float64) uses the usual numpy promotion rules.
>> But it doesn't do the "asarray" part. Its still the closest thing I can
>> think of right now.
>
> Thank you, that looks close enough
>
> And I will soon switch to numpy 1.6 and can read up on some old What's new.

browsing the numpy 1.5 documentation for functions starting with f
(like fill), I found

numpy.find_common_type

which looks similar to `result_type`

Josef

>
> Josef
>
>>
>> - Sebastian
>>
>>>
>>> that converts ints, bool and lower precision to float64 but leaves
>>> higher precision float and complex double alone.
>>>
>>>
>>> Josef
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From pgmdevlist at gmail.com  Wed Jun 12 12:25:36 2013
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 12 Jun 2013 18:25:36 +0200
Subject: [Numpy-discussion] =?utf-8?q?numpy=2Efilled=2C_again?=
In-Reply-To: <CAJhcF=03OnpWXkZn7x7yKmghP1Q_Fkr-m786T_HSrjCg0_3t5Q@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
	<CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
	<CAGzF1uf9=R9gvM3RBQtEwd3AxwiYAM+0hpA5To6jymTJPOG9dQ@mail.gmail.com>
	<CAMMTP+A50jGRe_8yh-bqP3M_V6o9KyMd9Dkge_PVX9KwV_SBUw@mail.gmail.com>
	<CAJhcF=03OnpWXkZn7x7yKmghP1Q_Fkr-m786T_HSrjCg0_3t5Q@mail.gmail.com>
Message-ID: <etPan.51b8a100.5bd062c2.239c@franceix-pierre.local>

On June 12, 2013 at 17:56:33, Da?id (davidmenhur at gmail.com) wrote:
On 12 June 2013 17:29, <josef.pktd at gmail.com> wrote:  
> +2 on this.  
> I like a good name like `filled` for a function that I expect to use regularly.  
> it's short and descriptive.  

FTIW, I am for this one too. That is not only clear, but a name that  
one may try even before checking the docs.  

Emtpy-filled sounds too much of an oximoron to me.  


And what about `np.fill_with`, then ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/19454385/attachment.html>

From chris.barker at noaa.gov  Wed Jun 12 12:27:58 2013
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Wed, 12 Jun 2013 09:27:58 -0700
Subject: [Numpy-discussion] Avoiding messy code in Python
In-Reply-To: <1371043318.106215414@f150.mail.ru>
References: <1371043318.106215414@f150.mail.ru>
Message-ID: <CALGmxEL0hiy1q2m2AgMd2SvSUqsxJWVVZu_EJeoj54OqRO3X7g@mail.gmail.com>

> I have some code more or less 500 lines, but very messy code. All codes
> containing several functions are in one module, besides, without
> documentation and testing.
>
> Could anyone give me some advice to organize my messy code in an accurate
> style including test function as well?

This is a really broad, non-numpy specific question. You'd best read
up on good programming practices in general. But you've answered a
good portion of your questions already:

>All codes containing several functions are in one module

Nothing wrong with that -- functions that are likely to be used
together belong in one module -- and while Python supports namespaces
(one honking great idea), pythonic style supports keeping those
namespaces flat (flat is better than nested)

So do you need to do any different stucture? I'd look for two things:

1) the DRY principle (Don't Repeat Yourself) -- if you have
essentially the same code in multiple places, refactor that out into
utility functions, or classes, or...

2) Do classes make sense? Look for patterns like:
    function_a(data_structure_1, some, other, arguments)
    function_b(data_structure_1, some, other, arguments)
    function_c(data_structure_1, some, other, arguments)

that is the C procedural style -- nothing wring with that, but it may
be better suited to a class -- the class holds your data structure and
the functions that operate on it in one place.

But:
"if a class has only two methods, and one of them is __init__ -- you
don't need a class":

http://pyvideo.org/video/880/stop-writing-classes

also see:

http://pyvideo.org/video/879/the-art-of-subclassing

For why you would want to use classes...

> without documentation and testing.

This one is easy: add documentation and testing!

put docstings in your functions and classes -- if the package is big
enough to warrant it, use Sphinx to document it -- it can extract the
docstrings, and give you a nice framework for writing up more
extensive docs.

Testing:
This sounds like a collection of functions -- you want unit tests --
read up on unit testing (and test-driven development) and the write
them.

I suggest nose or pytest (I prefer pytest, it's got some nice
features, but either is fine). You can also use the stdlib unittest,
but I found it overly wordy and klunky.

HTH,
  -Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From chris.barker at noaa.gov  Wed Jun 12 12:36:28 2013
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Wed, 12 Jun 2013 09:36:28 -0700
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
Message-ID: <CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>

On Wed, Jun 12, 2013 at 5:10 AM, Nathaniel Smith <njs at pobox.com> wrote:

> Personally I think that overloading np.empty is horribly ugly, will
> continue confusing newbies and everyone else indefinitely, and I'm
> 100% convinced that we'll regret implementing such a warty interface
> for something that should be so idiomatic.
...
 deprecate np.ma.filled in favor
> of masked_array.filled (which does exactly the same thing) and
> eventually switch np.ma.filled to be consistent with the new
> np.filled.

+1

> I also don't really see why an np.empty() constructor exists, it seems
> to do the same thing that np.ndarray() does.

I had always assumed that np.ndarray() was a "low-level" interce that
you really don't want to use in regular code (maybe for subclassing
array...), as the docs say:

"""
Arrays should be constructed using `array`, `zeros` or `empty` (refer
to the See Also section below).  The parameters given here refer to
a low-level method (`ndarray(...)`) for instantiating an array.
"""

Am I wrong? is there any reason )other than history to have np.empty()

But in any case, I like np.filled(), as being analogous to ones(),
zeros() and empty()...

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From brad.froehle at gmail.com  Wed Jun 12 12:39:52 2013
From: brad.froehle at gmail.com (Bradley M. Froehle)
Date: Wed, 12 Jun 2013 09:39:52 -0700
Subject: [Numpy-discussion] svd + multiprocessing hangs
In-Reply-To: <51B89CC2.5070901@mineway.de>
References: <51B89CC2.5070901@mineway.de>
Message-ID: <CAHXv-Mg39xTiGSpAowmNp20Cr_hL=0Pw86TN46V8PDvCU+uwTA@mail.gmail.com>

Dear Uwe:

I can't reproduce this using the system default versions of Python and
NumPy on Ubuntu 13.04:

$ python uwe.py
before svd
this message is not printed

>>> sys.version_info
sys.version_info(major=2, minor=7, micro=4, releaselevel='final', serial=0)
>>> numpy.__version__
'1.7.1'

Any idea how your hand-compiled versions might differ from the system
provided versions?

-Brad


On Wed, Jun 12, 2013 at 9:07 AM, Uwe Schmitt <uschmitt at mineway.de> wrote:

> Dear all,
>
> the following code hangs on my Ubuntu machine.
> I use self compiled numpy 1.7.1 and Python
> 2.7.3
>
> -----
>
> import numpy
> import numpy.linalg
> import multiprocessing
>
> def classify():
>     X = numpy.random.random((10,3))
>     print "before svd"
>     numpy.linalg.svd(X)
>     print "this message is not printed"
>
>
> if __name__ == "__main__":
>     p = multiprocessing.Process(target=classify, args=())
>     p.start()
>     p.join()
>
> -----
>
> Regards, Uwe.
>
> --
> Dr. rer. nat. Uwe Schmitt
> Leitung F/E Mathematik
>
> mineway GmbH
> Geb?ude 4
> Im Helmerswald 2
> 66121 Saarbr?cken
>
> Telefon: +49 (0)681 8390 5334
> Telefax: +49 (0)681 830 4376
>
> uschmitt at mineway.de
> www.mineway.de
>
> Gesch?ftsf?hrung: Dr.-Ing. Mathias Bauer
> Amtsgericht Saarbr?cken HRB 12339
>
>
>
>
>
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/63ea9225/attachment.html>

From ralf.gommers at gmail.com  Wed Jun 12 13:20:16 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Wed, 12 Jun 2013 19:20:16 +0200
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
Message-ID: <CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>

On Wed, Jun 12, 2013 at 6:36 PM, Chris Barker - NOAA Federal <
chris.barker at noaa.gov> wrote:

> On Wed, Jun 12, 2013 at 5:10 AM, Nathaniel Smith <njs at pobox.com> wrote:
>
> > Personally I think that overloading np.empty is horribly ugly, will
> > continue confusing newbies and everyone else indefinitely, and I'm
> > 100% convinced that we'll regret implementing such a warty interface
> > for something that should be so idiomatic.
>

I agree.


> ...
>  deprecate np.ma.filled
>

Please don't. Rather just live with the inconsistency between numpy and
numpy.ma APIs. If that bothers you, just tell yourself that we'll get an NA
dtype at some point and that that will make numpy.ma much less important:)


> in favor
> > of masked_array.filled (which does exactly the same thing) and
> > eventually switch np.ma.filled to be consistent with the new
> > np.filled.
>
> +1
>
> > I also don't really see why an np.empty() constructor exists, it seems
> > to do the same thing that np.ndarray() does.
>
> I had always assumed that np.ndarray() was a "low-level" interce that
> you really don't want to use in regular code (maybe for subclassing
> array...), as the docs say:
>
> """
> Arrays should be constructed using `array`, `zeros` or `empty` (refer
> to the See Also section below).  The parameters given here refer to
> a low-level method (`ndarray(...)`) for instantiating an array.
> """
>
> Am I wrong? is there any reason )other than history to have np.empty()
>
> But in any case, I like np.filled(), as being analogous to ones(),
> zeros() and empty()...


I like np.filled as well. np.fill_with sounds fine too.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/39dbb372/attachment.html>

From jtaylor.debian at googlemail.com  Wed Jun 12 13:27:43 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Wed, 12 Jun 2013 19:27:43 +0200
Subject: [Numpy-discussion] svd + multiprocessing hangs
In-Reply-To: <51B89CC2.5070901@mineway.de>
References: <51B89CC2.5070901@mineway.de>
Message-ID: <51B8AF8F.4020708@googlemail.com>

On 12.06.2013 18:07, Uwe Schmitt wrote:
> Dear all,
> 
> the following code hangs on my Ubuntu machine.
> I use self compiled numpy 1.7.1 and Python
> 2.7.3
> 
> -----
> 
> import numpy
> import numpy.linalg
> import multiprocessing
> 
> def classify():
>     X = numpy.random.random((10,3))
>     print "before svd"
>     numpy.linalg.svd(X)
>     print "this message is not printed"
> 
> 
> if __name__ == "__main__":
>     p = multiprocessing.Process(target=classify, args=())
>     p.start()
>     p.join()
> 

I'm guessing you are using openblas?
check with:
ls -l /etc/alternatives/libblas.so.3

there are known hanging problems with openblas and multiprocessing.
you can work around them by disabling threading in openblas
(OPENBLAS_NUM_THREADS=1).


From njs at pobox.com  Wed Jun 12 14:00:26 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 12 Jun 2013 19:00:26 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
Message-ID: <CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>

On 12 Jun 2013 18:20, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:
>
>
>
>
> On Wed, Jun 12, 2013 at 6:36 PM, Chris Barker - NOAA Federal <
chris.barker at noaa.gov> wrote:
>>
>> On Wed, Jun 12, 2013 at 5:10 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> > Personally I think that overloading np.empty is horribly ugly, will
>> > continue confusing newbies and everyone else indefinitely, and I'm
>> > 100% convinced that we'll regret implementing such a warty interface
>> > for something that should be so idiomatic.
>
>
> I agree.

Sounds like we're pretty much reaching consensus. Phew.

>>
>> ...
>>  deprecate np.ma.filled
>
>
> Please don't. Rather just live with the inconsistency between numpy and
numpy.ma APIs. If that bothers you, just tell yourself that we'll get an NA
dtype at some point and that that will make numpy.ma much less important:)

Oh, I do tell myself that :-). With my committer/consensus-building hat on,
np.ma has users, so I want something they can live with, and was suggesting
some options. For myself I don't really care what np.ma does though since I
don't use it...

>> in favor
>> > of masked_array.filled (which does exactly the same thing) and
>> > eventually switch np.ma.filled to be consistent with the new
>> > np.filled.
>>
>> +1
>>
>> > I also don't really see why an np.empty() constructor exists, it seems
>> > to do the same thing that np.ndarray() does.
>>
>> I had always assumed that np.ndarray() was a "low-level" interce that
>> you really don't want to use in regular code (maybe for subclassing
>> array...), as the docs say:
>>
>> """
>> Arrays should be constructed using `array`, `zeros` or `empty` (refer
>> to the See Also section below).  The parameters given here refer to
>> a low-level method (`ndarray(...)`) for instantiating an array.
>> """
>>
>> Am I wrong? is there any reason )other than history to have np.empty()
>>
>> But in any case, I like np.filled(), as being analogous to ones(),
>> zeros() and empty()...
>
>
> I like np.filled as well. np.fill_with sounds fine too.

Grammatically, fill_with is an imperative, which suggests it needs an array
to operate on; it's synonymous with just plain 'fill'. Having 'fill' and
'fill_with' as different functions with different semantics would be pretty
confusing!

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/c86ceb63/attachment.html>

From warren.weckesser at gmail.com  Wed Jun 12 14:13:16 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Wed, 12 Jun 2013 14:13:16 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
Message-ID: <CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>

On Wed, Jun 12, 2013 at 2:00 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On 12 Jun 2013 18:20, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:
> >
> >
> >
> >
> > On Wed, Jun 12, 2013 at 6:36 PM, Chris Barker - NOAA Federal <
> chris.barker at noaa.gov> wrote:
> >>
> >> On Wed, Jun 12, 2013 at 5:10 AM, Nathaniel Smith <njs at pobox.com> wrote:
> >>
> >> > Personally I think that overloading np.empty is horribly ugly, will
> >> > continue confusing newbies and everyone else indefinitely, and I'm
> >> > 100% convinced that we'll regret implementing such a warty interface
> >> > for something that should be so idiomatic.
> >
> >
> > I agree.
>
> Sounds like we're pretty much reaching consensus. Phew.
>
> >>
> >> ...
> >>  deprecate np.ma.filled
> >
> >
> > Please don't. Rather just live with the inconsistency between numpy and
> numpy.ma APIs. If that bothers you, just tell yourself that we'll get an
> NA dtype at some point and that that will make numpy.ma much less
> important:)
>
> Oh, I do tell myself that :-). With my committer/consensus-building hat
> on, np.ma has users, so I want something they can live with, and was
> suggesting some options. For myself I don't really care what np.ma does
> though since I don't use it...
>
> >> in favor
> >> > of masked_array.filled (which does exactly the same thing) and
> >> > eventually switch np.ma.filled to be consistent with the new
> >> > np.filled.
> >>
> >> +1
> >>
> >> > I also don't really see why an np.empty() constructor exists, it seems
> >> > to do the same thing that np.ndarray() does.
> >>
> >> I had always assumed that np.ndarray() was a "low-level" interce that
> >> you really don't want to use in regular code (maybe for subclassing
> >> array...), as the docs say:
> >>
> >> """
> >> Arrays should be constructed using `array`, `zeros` or `empty` (refer
> >> to the See Also section below).  The parameters given here refer to
> >> a low-level method (`ndarray(...)`) for instantiating an array.
> >> """
> >>
> >> Am I wrong? is there any reason )other than history to have np.empty()
> >>
> >> But in any case, I like np.filled(), as being analogous to ones(),
> >> zeros() and empty()...
> >
> >
> > I like np.filled as well. np.fill_with sounds fine too.
>
> Grammatically, fill_with is an imperative, which suggests it needs an
> array to operate on; it's synonymous with just plain 'fill'. Having 'fill'
> and 'fill_with' as different functions with different semantics would be
> pretty confusing!
>
>
That's why I suggested 'filledwith' (add the underscore if you like).  This
also allows a corresponding masked implementation, 'ma.filledwith', without
clobbering the existing 'ma.filled'.

Warren

-n
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/efd6ab9e/attachment.html>

From efiring at hawaii.edu  Wed Jun 12 14:43:48 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Wed, 12 Jun 2013 08:43:48 -1000
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
Message-ID: <51B8C164.9000305@hawaii.edu>

On 2013/06/12 2:10 AM, Nathaniel Smith wrote:
> Hi all,
>
> It looks like we've gotten a bit confused and need to untangle
> something. There's a PR to add new functions 'np.filled' and
> 'np.filled_like':
>    https://github.com/numpy/numpy/pull/2875
> And there was a discussion about this on the list back in January:
>    http://thread.gmane.org/gmane.comp.python.numeric.general/52763
>
> I think a reasonable summary of the opinions in the thread are:
> - This functionality is great, ...
> - ...but we can't call it 'np.filled' because there's also
> 'np.ma.filled' which does something else...
> - ...but there really aren't any better names...

How about 'np.initialized'?

> - ...so we should overload np.empty, like: 'np.empty(shape, fill=value)'
>
> In the mean time the original submitter has continued puttering along
> polishing the original patch, and it's ready to merge... except it's
> still the original interface, somehow the thread discussion and the PR
> discussion never met up.
>
> So, we have to decide what to do.
>
> Personally I think that overloading np.empty is horribly ugly, will
> continue confusing newbies and everyone else indefinitely, and I'm
> 100% convinced that we'll regret implementing such a warty interface
> for something that should be so idiomatic. (Unfortunately I got busy
> and didn't actually say this in the previous thread though.) So I
> think we should just merge the PR as is. The only downside is the
> np.ma inconsistency, but, np.ma is already inconsistent (cf.
> masked_array.fill versus masked_array.filled!), somewhat deprecated,

"somewhat deprecated"?  Really?  Since when?  By whom?  Replaced by what?

> and AFAICT there are far more people who will benefit from a clean
> np.filled idiom than who actually use np.ma (and in particular its
> fill-value functionality). So there would be two

I think there are more np.ma users than you realize.  Everyone who uses 
matplotlib is using np.ma at least implicitly, if not explicitly.  Many 
of the matplotlib examples put np.ma to good use.  np.ma.filled is an 
essential long-standing part of the np.ma API.  I don't see any good 
rationale for generating a conflict with it, when an adequate 
non-conflicting alternative ('np.initialized', maybe others) exists.

Eric

> bad-but-IMHO-acceptable options: either live with an inconsistency
> between np.filled and np.ma.filled, or deprecate np.ma.filled in favor
> of masked_array.filled (which does exactly the same thing) and
> eventually switch np.ma.filled to be consistent with the new
> np.filled.
>
> But, that's just my opinion.
>
> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



From efiring at hawaii.edu  Wed Jun 12 14:49:52 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Wed, 12 Jun 2013 08:49:52 -1000
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
	<CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
Message-ID: <51B8C2D0.60200@hawaii.edu>

On 2013/06/12 4:18 AM, Nathaniel Smith wrote:
> Now imagine a different new version of this page, if we overload
> 'empty' to add a fill= option. I don't even know how we document that
> on this page. The list will remain:
>    empty
>    ones
>    zeros

Opposite of "empty": "full".  So that is another non-conflicting 
alternative to my earlier suggestion of "initialized".

Eric


From matthew.brett at gmail.com  Wed Jun 12 14:54:32 2013
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 12 Jun 2013 19:54:32 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B8C2D0.60200@hawaii.edu>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
	<CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
	<51B8C2D0.60200@hawaii.edu>
Message-ID: <CAH6Pt5oWLOwqdPexXXxxBRPeT4Nvznte1dk00NqhbiQ-Z39Ucw@mail.gmail.com>

Hi,

On Wed, Jun 12, 2013 at 7:49 PM, Eric Firing <efiring at hawaii.edu> wrote:
> On 2013/06/12 4:18 AM, Nathaniel Smith wrote:
>> Now imagine a different new version of this page, if we overload
>> 'empty' to add a fill= option. I don't even know how we document that
>> on this page. The list will remain:
>>    empty
>>    ones
>>    zeros
>
> Opposite of "empty": "full".  So that is another non-conflicting
> alternative to my earlier suggestion of "initialized".

I like 'full'.   'filledwith' is a little ugly, but still a whole lot
better than creating a clash with ma.filled,

Cheers,

Matthew


From efiring at hawaii.edu  Wed Jun 12 14:55:16 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Wed, 12 Jun 2013 08:55:16 -1000
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
Message-ID: <51B8C414.6080508@hawaii.edu>

On 2013/06/12 8:13 AM, Warren Weckesser wrote:
> That's why I suggested 'filledwith' (add the underscore if you like).
> This also allows a corresponding masked implementation, 'ma.filledwith',
> without clobbering the existing 'ma.filled'.

Consensus on np.filled? absolutely not, you do not have a consensus.

np.filledwith or filled_with: fine with me, maybe even with 
everyone--let's see.  I would prefer the underscore version.

Eric


From chris.barker at noaa.gov  Wed Jun 12 14:54:56 2013
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Wed, 12 Jun 2013 11:54:56 -0700
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B8C2D0.60200@hawaii.edu>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAH6Pt5rPG0dnsV4Lpa+i-1nHzJwjkG4pQDm0GT5wN4x7j8h5sw@mail.gmail.com>
	<CAPJVwBnesZMZcopxLf_KZC2PMGXv2xE4dZRWEK0GD3kq+oC38Q@mail.gmail.com>
	<51B8C2D0.60200@hawaii.edu>
Message-ID: <CALGmxEJp3KFRf_8AP1VfZYOSer+i5g_on3Vzh6ZxajKO9s345A@mail.gmail.com>

On Wed, Jun 12, 2013 at 11:49 AM, Eric Firing <efiring at hawaii.edu> wrote:
> On 2013/06/12 4:18 AM, Nathaniel Smith wrote:
>> Now imagine a different new version of this page, if we overload
>> 'empty' to add a fill= option. I don't even know how we document that
>> on this page. The list will remain:
>>    empty
>>    ones
>>    zeros

"values"

kind of fits with "ones" and "zeros"...

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From archibald at astron.nl  Wed Jun 12 14:55:51 2013
From: archibald at astron.nl (Anne Archibald)
Date: Wed, 12 Jun 2013 20:55:51 +0200
Subject: [Numpy-discussion] supporting quad precision
In-Reply-To: <CAGY4rcVY5wAq5OffeTx2vmtHRahQSO6t5dfhOBiU-zge=MHBgg@mail.gmail.com>
References: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>
	<CAB6mnxKzCGEz2=iBJrX0dHqcfXdugqRWHTVXOJ24KjUCBONTTw@mail.gmail.com>
	<CAGY4rcV2_-HAzx-Wgfp96yHKKkjizx0UORXV1EWXtQ-NxijGbA@mail.gmail.com>
	<CANm_+Zq9hMr0dca39_0uUsV6VeN8R+aDiY-XvCqGGU8CnxZa-Q@mail.gmail.com>
	<1370763336.28467.9.camel@farnsworth>
	<CAGY4rcVY5wAq5OffeTx2vmtHRahQSO6t5dfhOBiU-zge=MHBgg@mail.gmail.com>
Message-ID: <CANm_+ZpA-fqsxdUspFXhrWm=ZxbL-riLWoHt3UwqUo_D5miZ6g@mail.gmail.com>

On 9 June 2013 13:23, David Cournapeau <cournape at gmail.com> wrote:

> So it depends on the CPU, the OS and the compiler. Using long double
> for anything else than compatibility (e.g. binary files) is often a
> mistake IMO, and highly unportable.
>

Now this I have to comment on. Long double is highly questionable for
compatibility, especially with binary files, for the reasons you describe.
But if what you want is something like double but with a few extra decimal
places and hardware support, long double is exactly what you want.
Portably! Some platforms will fail to give you any extra decimal places,
and in-memory size may differ, but you can be pretty confident that it'll
give you approximately what you're looking for anywhere.

I am using it right now (and in fact one of the standard tools, tempo2 does
this also) for handling pulsar pulse arrival times. These are expressed in
Modified Julian Days, so typically are of order 50000, but we care about
nanosecond-level differences in these numbers (and yes, there are pulsars
where you can track every rotation for ten thousand days to less than a
microsecond). Doubles give you quarter-microsecond roundoff error on every
operation. Long doubles are better, though not enough for me to be really
comfortable. Hence pining after quads.

Anne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/e75d67ce/attachment.html>

From hodge at stsci.edu  Wed Jun 12 15:00:24 2013
From: hodge at stsci.edu (Phil Hodge)
Date: Wed, 12 Jun 2013 15:00:24 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B8C414.6080508@hawaii.edu>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
Message-ID: <51B8C548.7070301@stsci.edu>

On 06/12/2013 02:55 PM, Eric Firing wrote:
> Consensus on np.filled? absolutely not, you do not have a consensus.

I would interpret np.filled as a test, asking whether the array is 
filled.  If the function is supposed to do something related to 
assigning values, the name should be a verb.

Phil


From alex.eberspaecher at gmail.com  Wed Jun 12 15:21:35 2013
From: alex.eberspaecher at gmail.com (=?ISO-8859-15?Q?Alexander_Ebersp=E4cher?=)
Date: Wed, 12 Jun 2013 21:21:35 +0200
Subject: [Numpy-discussion] svd + multiprocessing hangs
In-Reply-To: <51B8AF8F.4020708@googlemail.com>
References: <51B89CC2.5070901@mineway.de> <51B8AF8F.4020708@googlemail.com>
Message-ID: <51B8CA3F.9030308@gmail.com>

On 06/12/2013 07:27 PM, Julian Taylor wrote:

> I'm guessing you are using openblas?

I can confirm hanging with OpenBLAS using OpenMP.

> there are known hanging problems with openblas and multiprocessing.
> you can work around them by disabling threading in openblas
> (OPENBLAS_NUM_THREADS=1).

This works.

Regards

Alex


From Catherine.M.Moroney at jpl.nasa.gov  Wed Jun 12 15:25:33 2013
From: Catherine.M.Moroney at jpl.nasa.gov (Moroney, Catherine M (398D))
Date: Wed, 12 Jun 2013 19:25:33 +0000
Subject: [Numpy-discussion] weird problem with subtracting ndarrays
Message-ID: <01762EEF-EB11-4F6D-B29F-029C341D22A1@jpl.nasa.gov>

Hello,

I've got two arrays of the same shape that I read in from a file, and I'm trying to
difference them.  Very simple stuff, but I'm getting weird answers.

Here is the code:

>>> counts1 = hfile1.read_grid_field("CFbA", "TerrainReferencedRCCMFraction_Num")
>>> counts2 = hfile2.read_grid_field("CFbA", "TerrainReferencedRCCMFraction_Num")
>>> counts1.max(), counts2.max()
(13, 13)
>>> counts1.min(), counts2.min()
(0, 0)
>>> numpy.all(counts1 == counts2)
False
>>> diff = counts1 - counts2
>>> diff.max()
4294967295      !! WHAT IS HAPPENING HERE ??
>>> sum = counts1 + counts2
>>> sum.max()
26

As you can see, the range of values in both arrays is 0 to 13, and the sum
behaves normally, but the difference gives this weird number.

When I create dummy arrays, the subtraction works fine.  So there must be some funny value
lurking in either the counts1 or counts2 array, but the numpy.isnan() test returns False.

Any ideas for how I debug this?  

Catherine




From warren.weckesser at gmail.com  Wed Jun 12 15:30:22 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Wed, 12 Jun 2013 15:30:22 -0400
Subject: [Numpy-discussion] weird problem with subtracting ndarrays
In-Reply-To: <01762EEF-EB11-4F6D-B29F-029C341D22A1@jpl.nasa.gov>
References: <01762EEF-EB11-4F6D-B29F-029C341D22A1@jpl.nasa.gov>
Message-ID: <CAGzF1udhTAmxPWwHsd9xQ386CBz2iXymmYC4kJJ3AOTRPZQzyg@mail.gmail.com>

On Wed, Jun 12, 2013 at 3:25 PM, Moroney, Catherine M (398D) <
Catherine.M.Moroney at jpl.nasa.gov> wrote:

> Hello,
>
> I've got two arrays of the same shape that I read in from a file, and I'm
> trying to
> difference them.  Very simple stuff, but I'm getting weird answers.
>
> Here is the code:
>
> >>> counts1 = hfile1.read_grid_field("CFbA",
> "TerrainReferencedRCCMFraction_Num")
> >>> counts2 = hfile2.read_grid_field("CFbA",
> "TerrainReferencedRCCMFraction_Num")
> >>> counts1.max(), counts2.max()
> (13, 13)
> >>> counts1.min(), counts2.min()
> (0, 0)
> >>> numpy.all(counts1 == counts2)
> False
> >>> diff = counts1 - counts2
> >>> diff.max()
> 4294967295      !! WHAT IS HAPPENING HERE ??
> >>> sum = counts1 + counts2
> >>> sum.max()
> 26
>
> As you can see, the range of values in both arrays is 0 to 13, and the sum
> behaves normally, but the difference gives this weird number.
>
> When I create dummy arrays, the subtraction works fine.  So there must be
> some funny value
> lurking in either the counts1 or counts2 array, but the numpy.isnan() test
> returns False.
>
> Any ideas for how I debug this?
>
> Catherine
>
>
Check the dtype of the arrays.  They are probably unsigned integers, and
the subtraction leads to wrap-around in some cases.

For example:

In [1]: x = np.array([0, 1, 2], dtype=np.uint32)

In [2]: y = np.array([1, 1, 1], dtype=np.uint32)

In [3]: x - y
Out[3]: array([4294967295,          0,          1], dtype=uint32)


Warren




> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/d555f98a/attachment.html>

From josef.pktd at gmail.com  Wed Jun 12 15:30:49 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 12 Jun 2013 15:30:49 -0400
Subject: [Numpy-discussion] weird problem with subtracting ndarrays
In-Reply-To: <01762EEF-EB11-4F6D-B29F-029C341D22A1@jpl.nasa.gov>
References: <01762EEF-EB11-4F6D-B29F-029C341D22A1@jpl.nasa.gov>
Message-ID: <CAMMTP+BxbA0e4bR1edjqb0NzZFr8RLccjKMiDYqkNw=-oOZuPg@mail.gmail.com>

On Wed, Jun 12, 2013 at 3:25 PM, Moroney, Catherine M (398D)
<Catherine.M.Moroney at jpl.nasa.gov> wrote:
> Hello,
>
> I've got two arrays of the same shape that I read in from a file, and I'm trying to
> difference them.  Very simple stuff, but I'm getting weird answers.
>
> Here is the code:
>
>>>> counts1 = hfile1.read_grid_field("CFbA", "TerrainReferencedRCCMFraction_Num")
>>>> counts2 = hfile2.read_grid_field("CFbA", "TerrainReferencedRCCMFraction_Num")
>>>> counts1.max(), counts2.max()
> (13, 13)
>>>> counts1.min(), counts2.min()
> (0, 0)
>>>> numpy.all(counts1 == counts2)
> False
>>>> diff = counts1 - counts2
>>>> diff.max()
> 4294967295      !! WHAT IS HAPPENING HERE ??
>>>> sum = counts1 + counts2
>>>> sum.max()
> 26
>
> As you can see, the range of values in both arrays is 0 to 13, and the sum
> behaves normally, but the difference gives this weird number.
>
> When I create dummy arrays, the subtraction works fine.  So there must be some funny value
> lurking in either the counts1 or counts2 array, but the numpy.isnan() test returns False.
>
> Any ideas for how I debug this?

check the dtype of the count variables, may be uint

diff =1 *  counts1 - counts2

Josef

>
> Catherine
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From Catherine.M.Moroney at jpl.nasa.gov  Wed Jun 12 15:38:58 2013
From: Catherine.M.Moroney at jpl.nasa.gov (Moroney, Catherine M (398D))
Date: Wed, 12 Jun 2013 19:38:58 +0000
Subject: [Numpy-discussion] Weird problem with subtracting ndarrays
In-Reply-To: <mailman.1263.1371065114.1077.numpy-discussion@scipy.org>
References: <mailman.1263.1371065114.1077.numpy-discussion@scipy.org>
Message-ID: <FFA82F12-A294-4C72-BC70-3E3AEEB31752@jpl.nasa.gov>

Thanks for the tip.  I thought it must be something simple like that.  When I convert
the arrays to numpy.int32 things behave normally.  

Another question though:  a numpy.all() on the signed-int arrays shows that they
are equal.  So why would the subtraction of the unsigned arrays wrap around like that?

Catherine

On Jun 12, 2013, at 12:25 PM, <numpy-discussion-request at scipy.org>
 wrote:

> Message: 9
> Date: Wed, 12 Jun 2013 15:30:22 -0400
> From: Warren Weckesser <warren.weckesser at gmail.com>
> Subject: Re: [Numpy-discussion] weird problem with subtracting
> 	ndarrays
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID:
> 	<CAGzF1udhTAmxPWwHsd9xQ386CBz2iXymmYC4kJJ3AOTRPZQzyg at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> On Wed, Jun 12, 2013 at 3:25 PM, Moroney, Catherine M (398D) <
> Catherine.M.Moroney at jpl.nasa.gov> wrote:
> 
>> Hello,
>> 
>> I've got two arrays of the same shape that I read in from a file, and I'm
>> trying to
>> difference them.  Very simple stuff, but I'm getting weird answers.
>> 
>> Here is the code:
>> 
>>>>> counts1 = hfile1.read_grid_field("CFbA",
>> "TerrainReferencedRCCMFraction_Num")
>>>>> counts2 = hfile2.read_grid_field("CFbA",
>> "TerrainReferencedRCCMFraction_Num")
>>>>> counts1.max(), counts2.max()
>> (13, 13)
>>>>> counts1.min(), counts2.min()
>> (0, 0)
>>>>> numpy.all(counts1 == counts2)
>> False
>>>>> diff = counts1 - counts2
>>>>> diff.max()
>> 4294967295      !! WHAT IS HAPPENING HERE ??
>>>>> sum = counts1 + counts2
>>>>> sum.max()
>> 26
>> 
>> As you can see, the range of values in both arrays is 0 to 13, and the sum
>> behaves normally, but the difference gives this weird number.
>> 
>> When I create dummy arrays, the subtraction works fine.  So there must be
>> some funny value
>> lurking in either the counts1 or counts2 array, but the numpy.isnan() test
>> returns False.
>> 
>> Any ideas for how I debug this?
>> 
>> Catherine
>> 
>> 
> Check the dtype of the arrays.  They are probably unsigned integers, and
> the subtraction leads to wrap-around in some cases.
> 
> For example:
> 
> In [1]: x = np.array([0, 1, 2], dtype=np.uint32)
> 
> In [2]: y = np.array([1, 1, 1], dtype=np.uint32)
> 
> In [3]: x - y
> Out[3]: array([4294967295,          0,          1], dtype=uint32)
> 
> 
> Warren
> 
> 



From josef.pktd at gmail.com  Wed Jun 12 15:44:32 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 12 Jun 2013 15:44:32 -0400
Subject: [Numpy-discussion] Weird problem with subtracting ndarrays
In-Reply-To: <FFA82F12-A294-4C72-BC70-3E3AEEB31752@jpl.nasa.gov>
References: <mailman.1263.1371065114.1077.numpy-discussion@scipy.org>
	<FFA82F12-A294-4C72-BC70-3E3AEEB31752@jpl.nasa.gov>
Message-ID: <CAMMTP+BCgC8p9fMJ6NsKVVfVhmRih8+bjRYQwneiY+VHHo-J8Q@mail.gmail.com>

On Wed, Jun 12, 2013 at 3:38 PM, Moroney, Catherine M (398D)
<Catherine.M.Moroney at jpl.nasa.gov> wrote:
> Thanks for the tip.  I thought it must be something simple like that.  When I convert
> the arrays to numpy.int32 things behave normally.
>
> Another question though:  a numpy.all() on the signed-int arrays shows that they
> are equal.  So why would the subtraction of the unsigned arrays wrap around like that?

in your code
>>> numpy.all(counts1 == counts2)
False

Josef

>
> Catherine
>
> On Jun 12, 2013, at 12:25 PM, <numpy-discussion-request at scipy.org>
>  wrote:
>
>> Message: 9
>> Date: Wed, 12 Jun 2013 15:30:22 -0400
>> From: Warren Weckesser <warren.weckesser at gmail.com>
>> Subject: Re: [Numpy-discussion] weird problem with subtracting
>>       ndarrays
>> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
>> Message-ID:
>>       <CAGzF1udhTAmxPWwHsd9xQ386CBz2iXymmYC4kJJ3AOTRPZQzyg at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> On Wed, Jun 12, 2013 at 3:25 PM, Moroney, Catherine M (398D) <
>> Catherine.M.Moroney at jpl.nasa.gov> wrote:
>>
>>> Hello,
>>>
>>> I've got two arrays of the same shape that I read in from a file, and I'm
>>> trying to
>>> difference them.  Very simple stuff, but I'm getting weird answers.
>>>
>>> Here is the code:
>>>
>>>>>> counts1 = hfile1.read_grid_field("CFbA",
>>> "TerrainReferencedRCCMFraction_Num")
>>>>>> counts2 = hfile2.read_grid_field("CFbA",
>>> "TerrainReferencedRCCMFraction_Num")
>>>>>> counts1.max(), counts2.max()
>>> (13, 13)
>>>>>> counts1.min(), counts2.min()
>>> (0, 0)
>>>>>> numpy.all(counts1 == counts2)
>>> False
>>>>>> diff = counts1 - counts2
>>>>>> diff.max()
>>> 4294967295      !! WHAT IS HAPPENING HERE ??
>>>>>> sum = counts1 + counts2
>>>>>> sum.max()
>>> 26
>>>
>>> As you can see, the range of values in both arrays is 0 to 13, and the sum
>>> behaves normally, but the difference gives this weird number.
>>>
>>> When I create dummy arrays, the subtraction works fine.  So there must be
>>> some funny value
>>> lurking in either the counts1 or counts2 array, but the numpy.isnan() test
>>> returns False.
>>>
>>> Any ideas for how I debug this?
>>>
>>> Catherine
>>>
>>>
>> Check the dtype of the arrays.  They are probably unsigned integers, and
>> the subtraction leads to wrap-around in some cases.
>>
>> For example:
>>
>> In [1]: x = np.array([0, 1, 2], dtype=np.uint32)
>>
>> In [2]: y = np.array([1, 1, 1], dtype=np.uint32)
>>
>> In [3]: x - y
>> Out[3]: array([4294967295,          0,          1], dtype=uint32)
>>
>>
>> Warren
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From jaime.frio at gmail.com  Wed Jun 12 16:43:54 2013
From: jaime.frio at gmail.com (=?ISO-8859-1?Q?Jaime_Fern=E1ndez_del_R=EDo?=)
Date: Wed, 12 Jun 2013 13:43:54 -0700
Subject: [Numpy-discussion] Will transpose ever need to copy data?
In-Reply-To: <CAPJVwBm0N=BqUJAU8saRtUig65S+knpms5jKZ_FO3vpx-emn0Q@mail.gmail.com>
References: <CAPOWHW=aKyiiKD0X=1cnzFLBPvUnRd+wO5NS5NuRfGYJ-52GJw@mail.gmail.com>
	<CAPJVwBm0N=BqUJAU8saRtUig65S+knpms5jKZ_FO3vpx-emn0Q@mail.gmail.com>
Message-ID: <CAPOWHW=66WE4CfuXgwgcZRQNVGBnLsB3Czc8a7i_h2JUOAzjHA@mail.gmail.com>

On Wed, Jun 12, 2013 at 6:48 AM, Nathaniel Smith <njs at pobox.com> wrote:

> Sounds like a doc bug. (Probably someone being over-careful -- the
> default for many operations in numpy is that it's undefined whether
> they return a view or not, so if it makes a difference to you you need
> to take an explicit copy or similar.) Want to submit a PR to fix the
> doc?


Is it just in the function's docstring that the correction has to be made?

I have also found a potential inconsistency related to this in np.rollaxis,
the following examples should demonstrate it:

>>> a = np.empty((2, 3))
>>> b = np.transpose(a, (0, 1))
>>> b.base is a
True
>>> c = np.rollaxis(a, 0, 1)
>>> c.base is a
False
>>> c is a
True

It seems that np.transpose always returns a view of the passed array, while
np.rollaxis either returns a view or the same array. I personally find this
to be an annoying behavior: similarly to np.nonzero always returning a
tuple, even when it holds a single item, this changing behavior can lead to
hard to track bugs. Once you know what's going on, it is easy to work
around by calling np.rollaxis on a[:] instead of a, but I doubt I will
remember that the next time I need to use it.

This is, by the way, also the behavior of np.ndarray.swapaxes, although it
adds another degree of annoyance to the way it works:

>>> d = a.swapaxes(0, 0)
>>> d.base is a
False
>>> d is a
True
>>> e = a.swapaxes(0, -2)
>>> e.base is a
True

I find that the right thing to do would be to eliminate the checks for
special cases from both np.rollaxis and np.ndarray.swapaxes. The least
would be, in np.ndarray.swapaxes, to move the check for equality of the
axes to be swapped until after negative axis values have been dealt with.
Should any of these changes be included in the doc change PR?

Jaime
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130612/85869777/attachment.html>

From chris.barker at noaa.gov  Wed Jun 12 17:15:48 2013
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Wed, 12 Jun 2013 14:15:48 -0700
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B8C548.7070301@stsci.edu>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu> <51B8C548.7070301@stsci.edu>
Message-ID: <CALGmxELgyX5-ngfynG+F2Uimiz2g6Lw3gNr--HocWteKzkvb5A@mail.gmail.com>

On Wed, Jun 12, 2013 at 12:00 PM, Phil Hodge <hodge at stsci.edu> wrote:
> On 06/12/2013 02:55 PM, Eric Firing wrote:

> I would interpret np.filled as a test, asking whether the array is
> filled.  If the function is supposed to do something related to
> assigning values, the name should be a verb.

or a noun: ones, zeros -- but I agree, better not an adjective.

-Chris




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From davidmenhur at gmail.com  Wed Jun 12 18:04:32 2013
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Thu, 13 Jun 2013 00:04:32 +0200
Subject: [Numpy-discussion] Avoiding messy code in Python
In-Reply-To: <CALGmxEL0hiy1q2m2AgMd2SvSUqsxJWVVZu_EJeoj54OqRO3X7g@mail.gmail.com>
References: <1371043318.106215414@f150.mail.ru>
	<CALGmxEL0hiy1q2m2AgMd2SvSUqsxJWVVZu_EJeoj54OqRO3X7g@mail.gmail.com>
Message-ID: <CAJhcF=1UmXA-s744eHdTCEg=qZXUjR7UYfeNaXYHRGUn04WaUA@mail.gmail.com>

On 12 June 2013 18:27, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
>> without documentation and testing.
>
> This one is easy: add documentation and testing!

This lecture may help you. It uses unittest, though, but the idea is
applicable to whatever system you use:

https://python.g-node.org/python-summerschool-2012/software_carpentry

And for a brief introduction to best practices, you could see this:
https://python.g-node.org/python-summerschool-2012/best_practices


David.


From cournape at gmail.com  Wed Jun 12 19:00:41 2013
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 13 Jun 2013 00:00:41 +0100
Subject: [Numpy-discussion] supporting quad precision
In-Reply-To: <CANm_+ZpA-fqsxdUspFXhrWm=ZxbL-riLWoHt3UwqUo_D5miZ6g@mail.gmail.com>
References: <CANm_+ZrrSvv5TMjMCQ9NNr4h=-EoafyecOn_k5=duZzBuMYq1A@mail.gmail.com>
	<CAB6mnxKzCGEz2=iBJrX0dHqcfXdugqRWHTVXOJ24KjUCBONTTw@mail.gmail.com>
	<CAGY4rcV2_-HAzx-Wgfp96yHKKkjizx0UORXV1EWXtQ-NxijGbA@mail.gmail.com>
	<CANm_+Zq9hMr0dca39_0uUsV6VeN8R+aDiY-XvCqGGU8CnxZa-Q@mail.gmail.com>
	<1370763336.28467.9.camel@farnsworth>
	<CAGY4rcVY5wAq5OffeTx2vmtHRahQSO6t5dfhOBiU-zge=MHBgg@mail.gmail.com>
	<CANm_+ZpA-fqsxdUspFXhrWm=ZxbL-riLWoHt3UwqUo_D5miZ6g@mail.gmail.com>
Message-ID: <CAGY4rcU7g=M-Qor1AFoNZccKMD1cCX-GwX424jYceAZhdR4=eQ@mail.gmail.com>

On Wed, Jun 12, 2013 at 7:55 PM, Anne Archibald <archibald at astron.nl> wrote:
> On 9 June 2013 13:23, David Cournapeau <cournape at gmail.com> wrote:
>>
>> So it depends on the CPU, the OS and the compiler. Using long double
>> for anything else than compatibility (e.g. binary files) is often a
>> mistake IMO, and highly unportable.
>
>
> Now this I have to comment on. Long double is highly questionable for
> compatibility, especially with binary files, for the reasons you describe.

By compatibiliy, I meant binary compatibility: if you have some binary
files or some API/ABI requiring long double, then using long double is
a requirement. Typical example, using structured dtypes to read some
binary headers with long double in it.

> But if what you want is something like double but with a few extra decimal
> places and hardware support, long double is exactly what you want. Portably!
> Some platforms will fail to give you any extra decimal places, and in-memory
> size may differ, but you can be pretty confident that it'll give you
> approximately what you're looking for anywhere.

Sure, if you want to have "at least double precision" behavior, it
gives you that. I fail to see how it could be useful in a portable
context if you actually require that extra precision: if it is ok to
fall back on double, why not using double in the first place ?

David


From uschmitt at mineway.de  Thu Jun 13 04:37:10 2013
From: uschmitt at mineway.de (Uwe Schmitt)
Date: Thu, 13 Jun 2013 10:37:10 +0200
Subject: [Numpy-discussion] svd + multiprocessing hangs
In-Reply-To: <51B8AF8F.4020708@googlemail.com>
References: <51B89CC2.5070901@mineway.de> <51B8AF8F.4020708@googlemail.com>
Message-ID: <51B984B6.9060108@mineway.de>

Am 12.06.2013 19:27, schrieb Julian Taylor:
> I'm guessing you are using openblas?
> check with:
> ls -l /etc/alternatives/libblas.so.3
>
> there are known hanging problems with openblas and multiprocessing.
> you can work around them by disabling threading in openblas
> (OPENBLAS_NUM_THREADS=1).

Thanks, that works !

Cheers, Uwe.

-- 
Dr. rer. nat. Uwe Schmitt
Leitung F/E Mathematik

mineway GmbH
Geb?ude 4
Im Helmerswald 2
66121 Saarbr?cken

Telefon: +49 (0)681 8390 5334
Telefax: +49 (0)681 830 4376

uschmitt at mineway.de
www.mineway.de

Gesch?ftsf?hrung: Dr.-Ing. Mathias Bauer
Amtsgericht Saarbr?cken HRB 12339





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 553 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130613/71732282/attachment.sig>

From a.h.jaffe at gmail.com  Thu Jun 13 07:25:38 2013
From: a.h.jaffe at gmail.com (Andrew Jaffe)
Date: Thu, 13 Jun 2013 12:25:38 +0100
Subject: [Numpy-discussion] Dropping support for, Accelerate/veclib?
In-Reply-To: <CALGmxEKv-By0KmrWSJ0Ura0g=E3rvwoXGjw7ttF7OeWddV9_5w@mail.gmail.com>
References: <51B75CC6.4090405@amolf.nl>
	<CALGmxEJZgGO20u+r4FigVVb2ep0jgdxmMm8jXhQZ-4LmbrhCfA@mail.gmail.com>
	<CABL7CQjtWf8-b6k=XH2xDarF=JmFMavy+=FvXbLR3k22brKtUQ@mail.gmail.com>
	<CALGmxEKv-By0KmrWSJ0Ura0g=E3rvwoXGjw7ttF7OeWddV9_5w@mail.gmail.com>
Message-ID: <51B9AC32.4060701@gmail.com>

On 11/06/2013 22:11, Chris Barker - NOAA Federal wrote:
> On Tue, Jun 11, 2013 at 1:28 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>> The binaries will still be built against python.org Python, so there
>> shouldn't be an issue here. Same for building from source.
>
> My point was that it's nice to be able to have it build with an out of
> teh box wetup.py with accelerated LAPACK and all... If whoever is
> building binaries wants to get fancy, great.

Yes, please. The current system does seem to work for at least some of 
us.  And, if I understand the thread in the scipy mailing list, it's not 
actually clear that there's a bug, as opposed to incompatible fortran 
ABIs (which doesn't seem like a bug to me).

But I guess the most important thing would be that it can be used with 
apple or python.org Python builds (my reading of some of the suggestions 
would be requiring one of homebrew/fink/macports), preferably 
out-of-the-box -- even if that meant restricting to prebuilt binaries. 
Being able to run non-obscure installers (i.e., from the main python.org 
and scipy.org sites) for Python + numpy + scipy + matplotlib and get 
optimized versions would be sufficient.


Andrew



From aldcroft at head.cfa.harvard.edu  Thu Jun 13 09:36:34 2013
From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas)
Date: Thu, 13 Jun 2013 09:36:34 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B8C414.6080508@hawaii.edu>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
Message-ID: <CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>

On Wed, Jun 12, 2013 at 2:55 PM, Eric Firing <efiring at hawaii.edu> wrote:

> On 2013/06/12 8:13 AM, Warren Weckesser wrote:
> > That's why I suggested 'filledwith' (add the underscore if you like).
> > This also allows a corresponding masked implementation, 'ma.filledwith',
> > without clobbering the existing 'ma.filled'.
>
> Consensus on np.filled? absolutely not, you do not have a consensus.
>
> np.filledwith or filled_with: fine with me, maybe even with
> everyone--let's see.  I would prefer the underscore version.
>

+1 on np.filled_with.  It's unique the meaning is extremely obvious.  We do
use np.ma.filled in astropy so a big -1 on deprecating that (which would
then require doing numpy version checks to get the right method).  Even
when there is an NA dtype the numpy.ma users won't go away anytime soon.


>
> Eric
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130613/1808af08/attachment.html>

From pietro.bonfa at fis.unipr.it  Thu Jun 13 10:50:44 2013
From: pietro.bonfa at fis.unipr.it (Pietro Bonfa')
Date: Thu, 13 Jun 2013 16:50:44 +0200
Subject: [Numpy-discussion] Memory leak
Message-ID: <51B9DC44.20602@fis.unipr.it>

Dear Numpy users,

I have a memory leak in my code. A simple way to reproduce my problem is:

import numpy

class test():
    def __init__(self):
        pass

    def t(self):
        temp = numpy.zeros([200,100,100])
        A = numpy.zeros([200], dtype = numpy.float)
        for i in range(200):
            A[i] = numpy.sum( temp[i].diagonal() )

        return A

a = test()
c = [a.t() for i in range(100)]

Running this script will require 1.5 Gb of memory since the 16 mb of
temp arrays are never deallocated.

How can I solve this problem?

Thanks in advances,
Pietro Bonfa'


P.S: I asked the same question also on stack overflow
(http://stackoverflow.com/questions/17085197/is-this-a-memory-leak-python-numpy
)

-- 
Pietro Bonfa' - PhD student
Dipartimento di Fisica e Scienze della Terra "Macedonio Melloni"
Viale delle Scienze 7A
43124 Parma - Italy


From aron at ahmadia.net  Thu Jun 13 10:56:17 2013
From: aron at ahmadia.net (Aron Ahmadia)
Date: Thu, 13 Jun 2013 15:56:17 +0100
Subject: [Numpy-discussion] Memory leak
In-Reply-To: <51B9DC44.20602@fis.unipr.it>
References: <51B9DC44.20602@fis.unipr.it>
Message-ID: <CAPhiW4gERhPSFDA21QtLms_qh38qwUCmEE+ReuoV5FpjVQxwPw@mail.gmail.com>

Hi Petro,

What version of numpy are you running?

A


On Thu, Jun 13, 2013 at 3:50 PM, Pietro Bonfa' <pietro.bonfa at fis.unipr.it>wrote:

> Dear Numpy users,
>
> I have a memory leak in my code. A simple way to reproduce my problem is:
>
> import numpy
>
> class test():
>     def __init__(self):
>         pass
>
>     def t(self):
>         temp = numpy.zeros([200,100,100])
>         A = numpy.zeros([200], dtype = numpy.float)
>         for i in range(200):
>             A[i] = numpy.sum( temp[i].diagonal() )
>
>         return A
>
> a = test()
> c = [a.t() for i in range(100)]
>
> Running this script will require 1.5 Gb of memory since the 16 mb of
> temp arrays are never deallocated.
>
> How can I solve this problem?
>
> Thanks in advances,
> Pietro Bonfa'
>
>
> P.S: I asked the same question also on stack overflow
> (
> http://stackoverflow.com/questions/17085197/is-this-a-memory-leak-python-numpy
> )
>
> --
> Pietro Bonfa' - PhD student
> Dipartimento di Fisica e Scienze della Terra "Macedonio Melloni"
> Viale delle Scienze 7A
> 43124 Parma - Italy
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130613/0719295f/attachment.html>

From pietro.bonfa at fis.unipr.it  Thu Jun 13 11:02:07 2013
From: pietro.bonfa at fis.unipr.it (Pietro Bonfa')
Date: Thu, 13 Jun 2013 17:02:07 +0200
Subject: [Numpy-discussion] Memory leak
In-Reply-To: <CAPhiW4gERhPSFDA21QtLms_qh38qwUCmEE+ReuoV5FpjVQxwPw@mail.gmail.com>
References: <51B9DC44.20602@fis.unipr.it>
	<CAPhiW4gERhPSFDA21QtLms_qh38qwUCmEE+ReuoV5FpjVQxwPw@mail.gmail.com>
Message-ID: <51B9DEEF.8040604@fis.unipr.it>

The numpy version is 1.7.0

In [5]: numpy.version.full_version
Out[5]: '1.7.0b2'

Thanks,
Pietro


On 06/13/13 16:56, Aron Ahmadia wrote:
> Hi Petro,
> 
> What version of numpy are you running?
> 
> A
> 
> 
> On Thu, Jun 13, 2013 at 3:50 PM, Pietro Bonfa'
> <pietro.bonfa at fis.unipr.it <mailto:pietro.bonfa at fis.unipr.it>> wrote:
> 
>     Dear Numpy users,
> 
>     I have a memory leak in my code. A simple way to reproduce my
>     problem is:
> 
>     import numpy
> 
>     class test():
>         def __init__(self):
>             pass
> 
>         def t(self):
>             temp = numpy.zeros([200,100,100])
>             A = numpy.zeros([200], dtype = numpy.float)
>             for i in range(200):
>                 A[i] = numpy.sum( temp[i].diagonal() )
> 
>             return A
> 
>     a = test()
>     c = [a.t() for i in range(100)]
> 
>     Running this script will require 1.5 Gb of memory since the 16 mb of
>     temp arrays are never deallocated.
> 
>     How can I solve this problem?
> 
>     Thanks in advances,
>     Pietro Bonfa'
> 
> 
>     P.S: I asked the same question also on stack overflow
>     (http://stackoverflow.com/questions/17085197/is-this-a-memory-leak-python-numpy
>     )
> 
>     --
>     Pietro Bonfa' - PhD student
>     Dipartimento di Fisica e Scienze della Terra "Macedonio Melloni"
>     Viale delle Scienze 7A
>     43124 Parma - Italy
>     _______________________________________________
>     NumPy-Discussion mailing list
>     NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 


-- 
Pietro Bonfa' - PhD student
Dipartimento di Fisica e Scienze della Terra "Macedonio Melloni"
Viale delle Scienze 7A
43124 Parma - Italy


From charlesr.harris at gmail.com  Thu Jun 13 11:04:40 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 13 Jun 2013 09:04:40 -0600
Subject: [Numpy-discussion] Memory leak
In-Reply-To: <CAPhiW4gERhPSFDA21QtLms_qh38qwUCmEE+ReuoV5FpjVQxwPw@mail.gmail.com>
References: <51B9DC44.20602@fis.unipr.it>
	<CAPhiW4gERhPSFDA21QtLms_qh38qwUCmEE+ReuoV5FpjVQxwPw@mail.gmail.com>
Message-ID: <CAB6mnxKeQFiVQTooKsEgUYZxH20HrcgasyE8caHRKFpWiOR25w@mail.gmail.com>

On Thu, Jun 13, 2013 at 8:56 AM, Aron Ahmadia <aron at ahmadia.net> wrote:

> Hi Petro,
>
> What version of numpy are you running?
>
> A
>
>
> On Thu, Jun 13, 2013 at 3:50 PM, Pietro Bonfa' <pietro.bonfa at fis.unipr.it>wrote:
>
>> Dear Numpy users,
>>
>> I have a memory leak in my code. A simple way to reproduce my problem is:
>>
>> import numpy
>>
>> class test():
>>     def __init__(self):
>>         pass
>>
>>     def t(self):
>>         temp = numpy.zeros([200,100,100])
>>         A = numpy.zeros([200], dtype = numpy.float)
>>         for i in range(200):
>>             A[i] = numpy.sum( temp[i].diagonal() )
>>
>>         return A
>>
>> a = test()
>> c = [a.t() for i in range(100)]
>>
>> Running this script will require 1.5 Gb of memory since the 16 mb of
>> temp arrays are never deallocated.
>>
>> How can I solve this problem?
>>
>> Thanks in advances,
>> Pietro Bonfa'
>>
>>
>> P.S: I asked the same question also on stack overflow
>> (
>> http://stackoverflow.com/questions/17085197/is-this-a-memory-leak-python-numpy
>> )
>>
>>
IIRC, there was a memory leak with diagonal in 1.7.0 that was fixed in
1.7.1.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130613/14c14e3d/attachment.html>

From sebastian at sipsolutions.net  Thu Jun 13 11:06:08 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Thu, 13 Jun 2013 17:06:08 +0200
Subject: [Numpy-discussion] Memory leak
In-Reply-To: <51B9DC44.20602@fis.unipr.it>
References: <51B9DC44.20602@fis.unipr.it>
Message-ID: <1371135968.2556.1.camel@sebastian-laptop>

On Thu, 2013-06-13 at 16:50 +0200, Pietro Bonfa' wrote:
> Dear Numpy users,
> 
> I have a memory leak in my code. A simple way to reproduce my problem is:
> 
> import numpy
> 
> class test():
>     def __init__(self):
>         pass
> 
>     def t(self):
>         temp = numpy.zeros([200,100,100])
>         A = numpy.zeros([200], dtype = numpy.float)
>         for i in range(200):
>             A[i] = numpy.sum( temp[i].diagonal() )
> 
>         return A
> 
> a = test()
> c = [a.t() for i in range(100)]
> 
> Running this script will require 1.5 Gb of memory since the 16 mb of
> temp arrays are never deallocated.
> 
> How can I solve this problem?
> 

Please upgrade your Numpy version, there was a problem in diagonal which
by a lot of bad luck managed to creep into the 1.7.0 version of NumPy.
And this is quite certainly what you are seeing.

Regards,

Sebastian

> Thanks in advances,
> Pietro Bonfa'
> 
> 
> P.S: I asked the same question also on stack overflow
> (http://stackoverflow.com/questions/17085197/is-this-a-memory-leak-python-numpy
> )
> 




From pietro.bonfa at fis.unipr.it  Thu Jun 13 11:06:43 2013
From: pietro.bonfa at fis.unipr.it (Pietro Bonfa')
Date: Thu, 13 Jun 2013 17:06:43 +0200
Subject: [Numpy-discussion] Memory leak
In-Reply-To: <1371135968.2556.1.camel@sebastian-laptop>
References: <51B9DC44.20602@fis.unipr.it>
	<1371135968.2556.1.camel@sebastian-laptop>
Message-ID: <51B9E003.9030204@fis.unipr.it>

Upgrading numpy solved the problem!

Thanks!


On 06/13/13 17:06, Sebastian Berg wrote:
> On Thu, 2013-06-13 at 16:50 +0200, Pietro Bonfa' wrote:
>> Dear Numpy users,
>>
>> I have a memory leak in my code. A simple way to reproduce my problem is:
>>
>> import numpy
>>
>> class test():
>>     def __init__(self):
>>         pass
>>
>>     def t(self):
>>         temp = numpy.zeros([200,100,100])
>>         A = numpy.zeros([200], dtype = numpy.float)
>>         for i in range(200):
>>             A[i] = numpy.sum( temp[i].diagonal() )
>>
>>         return A
>>
>> a = test()
>> c = [a.t() for i in range(100)]
>>
>> Running this script will require 1.5 Gb of memory since the 16 mb of
>> temp arrays are never deallocated.
>>
>> How can I solve this problem?
>>
> 
> Please upgrade your Numpy version, there was a problem in diagonal which
> by a lot of bad luck managed to creep into the 1.7.0 version of NumPy.
> And this is quite certainly what you are seeing.
> 
> Regards,
> 
> Sebastian
> 
>> Thanks in advances,
>> Pietro Bonfa'
>>
>>
>> P.S: I asked the same question also on stack overflow
>> (http://stackoverflow.com/questions/17085197/is-this-a-memory-leak-python-numpy
>> )
>>
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 


-- 
Pietro Bonfa' - PhD student
Dipartimento di Fisica e Scienze della Terra "Macedonio Melloni"
Viale delle Scienze 7A
43124 Parma - Italy


From jtaylor.debian at googlemail.com  Thu Jun 13 13:25:39 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Thu, 13 Jun 2013 19:25:39 +0200
Subject: [Numpy-discussion] strided copy unroll, benchmarks needed
Message-ID: <51BA0093.3090504@googlemail.com>

hi,
I posted a pull with a minor change instructing the GCC compiler to
unroll the strided copy loops (gcc will almost never do that on its own,
not even on O3).

https://github.com/numpy/numpy/pull/3429
It improves performance of these copies by 20%-50% depending on the size
of the data copied (if it goes out of all cpu caches you don't gain
anything anymore) on a couple machines (amd phenom x4, intel core2duo,
xeon 7xxx/5xxx)

As overriding the compiler decision is always dodgy, I would like some
numbers on a couple of cpu types to decide if its really a good idea.
So if you have the time please try the pull and the benchmark in the
first comment and report the difference in performance between the pull
and the unchanged numpy git head in the PR.
please include your cpu, gcc version and architecture (32 bit or 64 bit).
The benchmark can be run with ipython:
irunner --ipython bench.py


Cheers,
Julian


From pav at iki.fi  Thu Jun 13 16:00:20 2013
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 13 Jun 2013 23:00:20 +0300
Subject: [Numpy-discussion] Dropping support for Accelerate/veclib?
In-Reply-To: <CABL7CQhQtD4iaPCitdd5Qbm22bZF1bdd3b0RDEFCkEJ_S0_ZQw@mail.gmail.com>
References: <loom.20130611T101447-656@post.gmane.org>
	<CAGY4rcXg-d4=F162BybzC+kV96XG4eH-9wiG4XK4CaFX-72_Pg@mail.gmail.com>
	<loom.20130611T110037-518@post.gmane.org>
	<CABL7CQhQtD4iaPCitdd5Qbm22bZF1bdd3b0RDEFCkEJ_S0_ZQw@mail.gmail.com>
Message-ID: <kpd8ce$1ve$1@ger.gmane.org>

12.06.2013 00:29, Ralf Gommers kirjoitti:
[clip]
> Sounds like a good idea. Would still make sense to move Accelerate down
> in the list of preferred libs, so that one can install ATLAS, MKL or
> OpenBLAS once and be done, instead of always having to remember these
> envvars.

It goes like this:

     https://github.com/pv/numpy/tree/fortran-abicheck

Interested parties on OSX should check whether they manage to build this 
version of Numpy with Accelerate when this is enabled, and whether 
Scipy's ARPACK tests still fail when building against this version of Numpy.

-- 
Pauli Virtanen



From ben.root at ou.edu  Thu Jun 13 16:36:30 2013
From: ben.root at ou.edu (Benjamin Root)
Date: Thu, 13 Jun 2013 16:36:30 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
	<CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>
Message-ID: <CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>

On Thu, Jun 13, 2013 at 9:36 AM, Aldcroft, Thomas <
aldcroft at head.cfa.harvard.edu> wrote:

>
>
>
> On Wed, Jun 12, 2013 at 2:55 PM, Eric Firing <efiring at hawaii.edu> wrote:
>
>> On 2013/06/12 8:13 AM, Warren Weckesser wrote:
>> > That's why I suggested 'filledwith' (add the underscore if you like).
>> > This also allows a corresponding masked implementation, 'ma.filledwith',
>> > without clobbering the existing 'ma.filled'.
>>
>> Consensus on np.filled? absolutely not, you do not have a consensus.
>>
>> np.filledwith or filled_with: fine with me, maybe even with
>> everyone--let's see.  I would prefer the underscore version.
>>
>
> +1 on np.filled_with.  It's unique the meaning is extremely obvious.  We
> do use np.ma.filled in astropy so a big -1 on deprecating that (which would
> then require doing numpy version checks to get the right method).  Even
> when there is an NA dtype the numpy.ma users won't go away anytime soon.
>

I like np.filled_with(), but just to be devil's advocate, think of the
syntax:

np.filled_with((10, 24), np.nan)

As I read that, I am filling the array with (10, 24), not NaNs.  Minor
issue, for sure, but just thought I raise that.

-1 on deprecation of np.ma.filled().  -1 on np.filled() due to collision
with np.ma (both conceptually and programatically).

np.values() might be a decent alternative.

Cheers!
Ben Root
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130613/c2a738b9/attachment.html>

From efiring at hawaii.edu  Thu Jun 13 16:47:56 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Thu, 13 Jun 2013 10:47:56 -1000
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
	<CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>
	<CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>
Message-ID: <51BA2FFC.2060501@hawaii.edu>

On 2013/06/13 10:36 AM, Benjamin Root wrote:
>
> On Thu, Jun 13, 2013 at 9:36 AM, Aldcroft, Thomas
> <aldcroft at head.cfa.harvard.edu <mailto:aldcroft at head.cfa.harvard.edu>>
> wrote:
>
>
>
>
>     On Wed, Jun 12, 2013 at 2:55 PM, Eric Firing <efiring at hawaii.edu
>     <mailto:efiring at hawaii.edu>> wrote:
>
>         On 2013/06/12 8:13 AM, Warren Weckesser wrote:
>          > That's why I suggested 'filledwith' (add the underscore if
>         you like).
>          > This also allows a corresponding masked implementation,
>         'ma.filledwith',
>          > without clobbering the existing 'ma.filled'.
>
>         Consensus on np.filled? absolutely not, you do not have a consensus.
>
>         np.filledwith or filled_with: fine with me, maybe even with
>         everyone--let's see.  I would prefer the underscore version.
>
>
>     +1 on np.filled_with.  It's unique the meaning is extremely obvious.
>       We do use np.ma.filled in astropy so a big -1 on deprecating that
>     (which would then require doing numpy version checks to get the
>     right method).  Even when there is an NA dtype the numpy.ma
>     <http://numpy.ma> users won't go away anytime soon.
>
>
> I like np.filled_with(), but just to be devil's advocate, think of the
> syntax:
>
> np.filled_with((10, 24), np.nan)
>
> As I read that, I am filling the array with (10, 24), not NaNs.  Minor
> issue, for sure, but just thought I raise that.
>
> -1 on deprecation of np.ma.filled().  -1 on np.filled() due to collision
> with np.ma <http://np.ma> (both conceptually and programatically).
>
> np.values() might be a decent alternative.
>
> Cheers!
> Ben Root

Even if he is representing the devil, Ben raises a good point.  To 
summarize, the most recent set of suggestions that seem not to have been 
completely shot down include:

np.filled_with((10, 24), np.nan)
np.full((10, 24), np.nan)          # analogous to np.empty
np.values((10, 24), np.nan)        # seems clear, concise
np.initialized((10, 24), np.nan)   # a few more characters, but
                                    #  seems clear to me.

Personally, I like all of the last three better than the first.

Eric




From alan.isaac at gmail.com  Thu Jun 13 17:02:41 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Thu, 13 Jun 2013 17:02:41 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
	<CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>
	<CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>
Message-ID: <51BA3371.6090705@gmail.com>

On 6/13/2013 4:36 PM, Benjamin Root wrote:
> np.values() might be a decent alternative.


This could then reasonably support broadcasting
from the shape of the input to the shape of
the array.

Alan Isaac



From josef.pktd at gmail.com  Thu Jun 13 17:06:52 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 13 Jun 2013 17:06:52 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51BA2FFC.2060501@hawaii.edu>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
	<CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>
	<CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>
	<51BA2FFC.2060501@hawaii.edu>
Message-ID: <CAMMTP+B8o+i05=1d8ABA6sViym5up1a0YsSdNmr9YcY679THxg@mail.gmail.com>

On Thu, Jun 13, 2013 at 4:47 PM, Eric Firing <efiring at hawaii.edu> wrote:
> On 2013/06/13 10:36 AM, Benjamin Root wrote:
>>
>> On Thu, Jun 13, 2013 at 9:36 AM, Aldcroft, Thomas
>> <aldcroft at head.cfa.harvard.edu <mailto:aldcroft at head.cfa.harvard.edu>>
>> wrote:
>>
>>
>>
>>
>>     On Wed, Jun 12, 2013 at 2:55 PM, Eric Firing <efiring at hawaii.edu
>>     <mailto:efiring at hawaii.edu>> wrote:
>>
>>         On 2013/06/12 8:13 AM, Warren Weckesser wrote:
>>          > That's why I suggested 'filledwith' (add the underscore if
>>         you like).
>>          > This also allows a corresponding masked implementation,
>>         'ma.filledwith',
>>          > without clobbering the existing 'ma.filled'.
>>
>>         Consensus on np.filled? absolutely not, you do not have a consensus.
>>
>>         np.filledwith or filled_with: fine with me, maybe even with
>>         everyone--let's see.  I would prefer the underscore version.
>>
>>
>>     +1 on np.filled_with.  It's unique the meaning is extremely obvious.
>>       We do use np.ma.filled in astropy so a big -1 on deprecating that
>>     (which would then require doing numpy version checks to get the
>>     right method).  Even when there is an NA dtype the numpy.ma
>>     <http://numpy.ma> users won't go away anytime soon.
>>
>>
>> I like np.filled_with(), but just to be devil's advocate, think of the
>> syntax:
>>
>> np.filled_with((10, 24), np.nan)
>>
>> As I read that, I am filling the array with (10, 24), not NaNs.  Minor
>> issue, for sure, but just thought I raise that.
>>
>> -1 on deprecation of np.ma.filled().  -1 on np.filled() due to collision
>> with np.ma <http://np.ma> (both conceptually and programatically).
>>
>> np.values() might be a decent alternative.
>>
>> Cheers!
>> Ben Root
>
> Even if he is representing the devil, Ben raises a good point.  To
> summarize, the most recent set of suggestions that seem not to have been
> completely shot down include:
>
> np.filled_with((10, 24), np.nan)
> np.full((10, 24), np.nan)          # analogous to np.empty
> np.values((10, 24), np.nan)        # seems clear, concise
> np.initialized((10, 24), np.nan)   # a few more characters, but
>                                     #  seems clear to me.
>
> Personally, I like all of the last three better than the first.

np.values
sounds also good to me, a noun like np.ones, np.nans, np.infs, np.zeros

I don't like np.initialized because empty also initializes and array
(although) an empty one.

Josef

>
> Eric
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From arinkverma at gmail.com  Fri Jun 14 04:18:04 2013
From: arinkverma at gmail.com (Arink Verma)
Date: Fri, 14 Jun 2013 13:48:04 +0530
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
	<CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
	<CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>
Message-ID: <CAJAs4rSR05SB1AtgFCf1b+Ln-fhAEhgQ=QESrViVoB6Q_iQg3w@mail.gmail.com>

> You're looking for the ProfilerStart/ProfilerStop functions, the
> former takes a filename to write the profiler to (like "ls.prof" or
> "x-plus-x.prof"):
>   http://www.mail-archive.com/numpy-discussion at scipy.org/msg41451.html


I followed that and able to get a callgraph[1], but it doesnt contains all
the edges.

[1]
https://docs.google.com/file/d/0B3Pqyp8kuQw0eVM4ajVYLWtfZVU/edit?usp=sharing

Arink Verma
www.arinkverma.in
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130614/fafec1e1/attachment.html>

From njs at pobox.com  Fri Jun 14 07:15:09 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 14 Jun 2013 12:15:09 +0100
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAJAs4rSR05SB1AtgFCf1b+Ln-fhAEhgQ=QESrViVoB6Q_iQg3w@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
	<CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
	<CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>
	<CAJAs4rSR05SB1AtgFCf1b+Ln-fhAEhgQ=QESrViVoB6Q_iQg3w@mail.gmail.com>
Message-ID: <CAPJVwBnCVWAUVXcoHm0eCrB=9b86+ZHzz88vhCpKKQ4ejC1eRA@mail.gmail.com>

On 14 Jun 2013 09:18, "Arink Verma" <arinkverma at gmail.com> wrote:
>
>
>> You're looking for the ProfilerStart/ProfilerStop functions, the
>> former takes a filename to write the profiler to (like "ls.prof" or
>> "x-plus-x.prof"):
>>   http://www.mail-archive.com/numpy-discussion at scipy.org/msg41451.html
>
>
> I followed that and able to get a callgraph[1], but it doesnt contains
all the edges.
>
> [1]
https://docs.google.com/file/d/0B3Pqyp8kuQw0eVM4ajVYLWtfZVU/edit?usp=sharing

It's a probabilistic sampling profiler, so if it doesn't have enough
samples then it can miss things. 227 samples is way way too low. You need
to run the profiled code for longer (a few seconds at least), and if that's
not enough then maybe increase the sampling rate too (though be careful
because setting this too high can also add noise).

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130614/a0baf010/attachment.html>

From aldcroft at head.cfa.harvard.edu  Fri Jun 14 09:27:36 2013
From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas)
Date: Fri, 14 Jun 2013 09:27:36 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAMMTP+B8o+i05=1d8ABA6sViym5up1a0YsSdNmr9YcY679THxg@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
	<CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>
	<CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>
	<51BA2FFC.2060501@hawaii.edu>
	<CAMMTP+B8o+i05=1d8ABA6sViym5up1a0YsSdNmr9YcY679THxg@mail.gmail.com>
Message-ID: <CAMtEP6wAZq61h9oDy4mW+sX0Nv4dvEgNDchUpLGiMxAHKk_skw@mail.gmail.com>

On Thu, Jun 13, 2013 at 5:06 PM, <josef.pktd at gmail.com> wrote:

> On Thu, Jun 13, 2013 at 4:47 PM, Eric Firing <efiring at hawaii.edu> wrote:
> > On 2013/06/13 10:36 AM, Benjamin Root wrote:
> >>
> >> On Thu, Jun 13, 2013 at 9:36 AM, Aldcroft, Thomas
> >> <aldcroft at head.cfa.harvard.edu <mailto:aldcroft at head.cfa.harvard.edu>>
> >> wrote:
> >>
> >>
> >>
> >>
> >>     On Wed, Jun 12, 2013 at 2:55 PM, Eric Firing <efiring at hawaii.edu
> >>     <mailto:efiring at hawaii.edu>> wrote:
> >>
> >>         On 2013/06/12 8:13 AM, Warren Weckesser wrote:
> >>          > That's why I suggested 'filledwith' (add the underscore if
> >>         you like).
> >>          > This also allows a corresponding masked implementation,
> >>         'ma.filledwith',
> >>          > without clobbering the existing 'ma.filled'.
> >>
> >>         Consensus on np.filled? absolutely not, you do not have a
> consensus.
> >>
> >>         np.filledwith or filled_with: fine with me, maybe even with
> >>         everyone--let's see.  I would prefer the underscore version.
> >>
> >>
> >>     +1 on np.filled_with.  It's unique the meaning is extremely obvious.
> >>       We do use np.ma.filled in astropy so a big -1 on deprecating that
> >>     (which would then require doing numpy version checks to get the
> >>     right method).  Even when there is an NA dtype the numpy.ma
> >>     <http://numpy.ma> users won't go away anytime soon.
> >>
> >>
> >> I like np.filled_with(), but just to be devil's advocate, think of the
> >> syntax:
> >>
> >> np.filled_with((10, 24), np.nan)
> >>
> >> As I read that, I am filling the array with (10, 24), not NaNs.  Minor
> >> issue, for sure, but just thought I raise that.
> >>
> >> -1 on deprecation of np.ma.filled().  -1 on np.filled() due to collision
> >> with np.ma <http://np.ma> (both conceptually and programatically).
> >>
> >> np.values() might be a decent alternative.
> >>
> >> Cheers!
> >> Ben Root
> >
> > Even if he is representing the devil, Ben raises a good point.  To
> > summarize, the most recent set of suggestions that seem not to have been
> > completely shot down include:
> >
> > np.filled_with((10, 24), np.nan)
> > np.full((10, 24), np.nan)          # analogous to np.empty
> > np.values((10, 24), np.nan)        # seems clear, concise
> > np.initialized((10, 24), np.nan)   # a few more characters, but
> >                                     #  seems clear to me.
> >
> > Personally, I like all of the last three better than the first.
>

What about:

  np.filled_array((10, 24), np.nan)

If I just saw np.values(..) in some code I would never guess what it is
doing from the name, and beyond that I would guess that I can put in
multiple "values" in the second arg.  np.initialized() is more obvious to
me.

- Tom

>
> np.values
> sounds also good to me, a noun like np.ones, np.nans, np.infs, np.zeros
>
> I don't like np.initialized because empty also initializes and array
> (although) an empty one.
>
> Josef
>
> >
> > Eric
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130614/cca06886/attachment.html>

From alan.isaac at gmail.com  Fri Jun 14 11:15:13 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 14 Jun 2013 11:15:13 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAMtEP6wAZq61h9oDy4mW+sX0Nv4dvEgNDchUpLGiMxAHKk_skw@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
	<CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>
	<CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>
	<51BA2FFC.2060501@hawaii.edu>
	<CAMMTP+B8o+i05=1d8ABA6sViym5up1a0YsSdNmr9YcY679THxg@mail.gmail.com>
	<CAMtEP6wAZq61h9oDy4mW+sX0Nv4dvEgNDchUpLGiMxAHKk_skw@mail.gmail.com>
Message-ID: <51BB3381.4000508@gmail.com>

On 6/14/2013 9:27 AM, Aldcroft, Thomas wrote:
> If I just saw np.values(..) in some code I would never guess what it is doing from the name

That suggests np.fromvalues.
But more important than the name I think
is allowing broadcasting of the values,
based on NumPy's broadcasting rules.
Broadcasting a scalar is then a special case,
even if it is the case that has dominated this thread.

Alan Isaac



From njs at pobox.com  Fri Jun 14 11:34:03 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 14 Jun 2013 16:34:03 +0100
Subject: [Numpy-discussion] Will transpose ever need to copy data?
In-Reply-To: <CAPOWHW=66WE4CfuXgwgcZRQNVGBnLsB3Czc8a7i_h2JUOAzjHA@mail.gmail.com>
References: <CAPOWHW=aKyiiKD0X=1cnzFLBPvUnRd+wO5NS5NuRfGYJ-52GJw@mail.gmail.com>
	<CAPJVwBm0N=BqUJAU8saRtUig65S+knpms5jKZ_FO3vpx-emn0Q@mail.gmail.com>
	<CAPOWHW=66WE4CfuXgwgcZRQNVGBnLsB3Czc8a7i_h2JUOAzjHA@mail.gmail.com>
Message-ID: <CAPJVwB=OdLBDWSNfw=whUbqO0UnnK294-kk0HnyHEqAfmFpFVw@mail.gmail.com>

On Wed, Jun 12, 2013 at 9:43 PM, Jaime Fern?ndez del R?o
<jaime.frio at gmail.com> wrote:
> On Wed, Jun 12, 2013 at 6:48 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Sounds like a doc bug. (Probably someone being over-careful -- the
>> default for many operations in numpy is that it's undefined whether
>> they return a view or not, so if it makes a difference to you you need
>> to take an explicit copy or similar.) Want to submit a PR to fix the
>> doc?
>
>
> Is it just in the function's docstring that the correction has to be made?
>
> I have also found a potential inconsistency related to this in np.rollaxis,
> the following examples should demonstrate it:
>
>>>> a = np.empty((2, 3))
>>>> b = np.transpose(a, (0, 1))
>>>> b.base is a
> True
>>>> c = np.rollaxis(a, 0, 1)
>>>> c.base is a
> False
>>>> c is a
> True
>
> It seems that np.transpose always returns a view of the passed array, while
> np.rollaxis either returns a view or the same array. I personally find this
> to be an annoying behavior: similarly to np.nonzero always returning a
> tuple, even when it holds a single item, this changing behavior can lead to
> hard to track bugs. Once you know what's going on, it is easy to work around
> by calling np.rollaxis on a[:] instead of a, but I doubt I will remember
> that the next time I need to use it.
>
> This is, by the way, also the behavior of np.ndarray.swapaxes, although it
> adds another degree of annoyance to the way it works:
>
>>>> d = a.swapaxes(0, 0)
>>>> d.base is a
> False
>>>> d is a
> True
>>>> e = a.swapaxes(0, -2)
>>>> e.base is a
> True
>
> I find that the right thing to do would be to eliminate the checks for
> special cases from both np.rollaxis and np.ndarray.swapaxes. The least would
> be, in np.ndarray.swapaxes, to move the check for equality of the axes to be
> swapped until after negative axis values have been dealt with. Should any of
> these changes be included in the doc change PR?

I agree that rollaxis and swapaxes should always return a either a
copy or a view, never the original array.

Ideally this would be a separate PR from the changes to the
np.transpose docs, because one or the other might turn out to take
more review iterations etc., but it's not the most critical thing
ever.

Thanks!

-n


From njs at pobox.com  Fri Jun 14 12:28:01 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 14 Jun 2013 17:28:01 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B8C548.7070301@stsci.edu>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu> <51B8C548.7070301@stsci.edu>
Message-ID: <CAPJVwB=yUM7Fquv4T0oboH1w8TbDW94Mow+VtYhdtzskHqCK3Q@mail.gmail.com>

On Wed, Jun 12, 2013 at 8:00 PM, Phil Hodge <hodge at stsci.edu> wrote:
> I would interpret np.filled as a test, asking whether the array is
> filled.  If the function is supposed to do something related to
> assigning values, the name should be a verb.

That's a plausible convention, but it's not the convention that
actually exists for these sorts of functions. 'sorted' gives you an
object with the property of being sorted, it doesn't test for
sortedness. 'reversed' gives you an object with the property of being
reversed, it doesn't test for reversedness. Tests should be named like
'is_sorted'.

Conceptually, this function doesn't assign values -- for that we have
np.fill(), which is a side-effecting operation that fills an existing
array with some value, and does indeed have a verb as a name. The
function we're talking about is like 'sorted' in that it isn't
side-effecting, but instead gives you an object that has a particular
property.

-n


From efiring at hawaii.edu  Fri Jun 14 13:18:23 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Fri, 14 Jun 2013 07:18:23 -1000
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51BB3381.4000508@gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
	<CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>
	<CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>
	<51BA2FFC.2060501@hawaii.edu>
	<CAMMTP+B8o+i05=1d8ABA6sViym5up1a0YsSdNmr9YcY679THxg@mail.gmail.com>
	<CAMtEP6wAZq61h9oDy4mW+sX0Nv4dvEgNDchUpLGiMxAHKk_skw@mail.gmail.com>
	<51BB3381.4000508@gmail.com>
Message-ID: <51BB505F.6040107@hawaii.edu>

On 2013/06/14 5:15 AM, Alan G Isaac wrote:
> On 6/14/2013 9:27 AM, Aldcroft, Thomas wrote:
>> If I just saw np.values(..) in some code I would never guess what it is doing from the name
>
> That suggests np.fromvalues.
> But more important than the name I think
> is allowing broadcasting of the values,
> based on NumPy's broadcasting rules.
> Broadcasting a scalar is then a special case,
> even if it is the case that has dominated this thread.

True, but this looks to me like mission creep.  All of this fuss is 
about replacing two lines of user code with a single line.  If it can't 
be kept simple, both in implementation and in documentation, it 
shouldn't be done at all.  I'm not necessarily opposed to your 
suggestion, but I'm skeptical.

Eric

>
> Alan Isaac
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



From robert.kern at gmail.com  Fri Jun 14 13:21:42 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 14 Jun 2013 18:21:42 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51BB505F.6040107@hawaii.edu>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
	<CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>
	<CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>
	<51BA2FFC.2060501@hawaii.edu>
	<CAMMTP+B8o+i05=1d8ABA6sViym5up1a0YsSdNmr9YcY679THxg@mail.gmail.com>
	<CAMtEP6wAZq61h9oDy4mW+sX0Nv4dvEgNDchUpLGiMxAHKk_skw@mail.gmail.com>
	<51BB3381.4000508@gmail.com> <51BB505F.6040107@hawaii.edu>
Message-ID: <CAF6FJisCwfP+xFO+_LDVaVFi76r=RtTbpFcEnig9wHDp-c73iQ@mail.gmail.com>

On Fri, Jun 14, 2013 at 6:18 PM, Eric Firing <efiring at hawaii.edu> wrote:
> On 2013/06/14 5:15 AM, Alan G Isaac wrote:
>> On 6/14/2013 9:27 AM, Aldcroft, Thomas wrote:
>>> If I just saw np.values(..) in some code I would never guess what it is doing from the name
>>
>> That suggests np.fromvalues.
>> But more important than the name I think
>> is allowing broadcasting of the values,
>> based on NumPy's broadcasting rules.
>> Broadcasting a scalar is then a special case,
>> even if it is the case that has dominated this thread.
>
> True, but this looks to me like mission creep.  All of this fuss is
> about replacing two lines of user code with a single line.  If it can't
> be kept simple, both in implementation and in documentation, it
> shouldn't be done at all.  I'm not necessarily opposed to your
> suggestion, but I'm skeptical.

It's another two-liner:

[~]
|1> x = np.empty([3,4,5])

[~]
|2> x[...] = np.arange(5)

[~]
|3> x
array([[[ 0.,  1.,  2.,  3.,  4.],
        [ 0.,  1.,  2.,  3.,  4.],
        [ 0.,  1.,  2.,  3.,  4.],
        [ 0.,  1.,  2.,  3.,  4.]],

       [[ 0.,  1.,  2.,  3.,  4.],
        [ 0.,  1.,  2.,  3.,  4.],
        [ 0.,  1.,  2.,  3.,  4.],
        [ 0.,  1.,  2.,  3.,  4.]],

       [[ 0.,  1.,  2.,  3.,  4.],
        [ 0.,  1.,  2.,  3.,  4.],
        [ 0.,  1.,  2.,  3.,  4.],
        [ 0.,  1.,  2.,  3.,  4.]]])

It's wafer-thin!

--
Robert Kern


From njs at pobox.com  Fri Jun 14 13:22:44 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 14 Jun 2013 18:22:44 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51B8C164.9000305@hawaii.edu>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<51B8C164.9000305@hawaii.edu>
Message-ID: <CAPJVwBmoDseOK5hq_wW1sDe_dhpwaQ-OO7+PxMM4fp52UVXtPg@mail.gmail.com>

On Wed, Jun 12, 2013 at 7:43 PM, Eric Firing <efiring at hawaii.edu> wrote:
> On 2013/06/12 2:10 AM, Nathaniel Smith wrote:
>> Personally I think that overloading np.empty is horribly ugly, will
>> continue confusing newbies and everyone else indefinitely, and I'm
>> 100% convinced that we'll regret implementing such a warty interface
>> for something that should be so idiomatic. (Unfortunately I got busy
>> and didn't actually say this in the previous thread though.) So I
>> think we should just merge the PR as is. The only downside is the
>> np.ma inconsistency, but, np.ma is already inconsistent (cf.
>> masked_array.fill versus masked_array.filled!), somewhat deprecated,
>
> "somewhat deprecated"?  Really?  Since when?  By whom?  Replaced by what?

Sorry, not trying to start a fight, just trying to summarize the
situation. As far as I can tell:

Despite heroic efforts on the part of its authors, numpy.ma has a
number of weird quirks (masked data can still trigger invalid value
errors), misfeatures (hard versus soft masks), and just plain old pain
points (ongoing issues with whether any given operation will respect
or preserve the mask).

It's been in deep maintenance mode for some time; we merge the
occasional bug fix that people send in, and that's it. (To be fair,
numpy as a whole is fairly slow-moving, but numpy.ma still gets much
less attention.)

Even if there were active maintainers, no-one really has any idea how
to fix any of the problems above; they're not so much bugs as
intrinsic limitations of the design.

Therefore, my impression is that a majority (not all, but a majority)
of numpy developers strongly recommend against the use of numpy.ma in
new projects.

I could be wrong! And I know there's nothing to really replace it. I'd
like to fix that. But I think "semi-deprecated" is not an unfair
shorthand for the above.

(I'll even admit that I'd *like* to actually deprecate it. But what I
mean by that is, I don't think it's possible to fix it to the point
where it's actually a solid/clean/robust library, so I'd like to reach
a point where everyone who's currently using it is happier switching
to something else and is happy to sign off on deprecating it.)

>> and AFAICT there are far more people who will benefit from a clean
>> np.filled idiom than who actually use np.ma (and in particular its
>> fill-value functionality). So there would be two
>
> I think there are more np.ma users than you realize.  Everyone who uses
> matplotlib is using np.ma at least implicitly, if not explicitly.  Many
> of the matplotlib examples put np.ma to good use.  np.ma.filled is an
> essential long-standing part of the np.ma API.  I don't see any good
> rationale for generating a conflict with it, when an adequate
> non-conflicting alternative ('np.initialized', maybe others) exists.

I'm aware of that. If I didn't care about the opinions of numpy.ma
users, I wouldn't go starting long and annoying mailing list threads
about features that are only problematic because of their affect on
numpy.ma :-).

But, IMHO given the issues with numpy.ma, our number #1 priority ought
to be making numpy proper as clean and beautiful as possible; my
position that started this thread is basically just that we shouldn't
make numpy proper worse just for numpy.ma's sake. That's the tail
wagging the dog. And this 'conflict' seems a bit overstated given that
(1) np.ma.filled already has multiple names (and 3/4 of the uses in
matplotlib use the method version, not the function version), (2) even
if we give it a non-conflicting name, np.ma's lack of maintenance
means that it'd probably be years before someone got around to
actually adding a parallel function to np.ma. [Unless this thread
spurs someone into submitting one just to prove me wrong ;-).]

But anyway, that was when the comparison was between np.filled() and
np.empty(..., fill_value=...). Of the new things on the table:

- I agree with Tom that 'np.values(...)' is so generic as to be
unguessable. np.fromvalues() was also suggested, but this is even
worse, because it suggests that it's analogous to
np.from{buffer,file,function,regex,...}. But the analogous
fromvalues() function already has a name: np.array.

- np.filled_with and np.initialized are both gratuitously cumbersome.
(It's the gratuitous that bothers me more than the cumbersome. No-one
enjoys using APIs that feel like they're annoying for no good reason.)

- np.full is... huh. It's quirky, and compared to np.filled it's more
confusing (all arrays are full of *something*, but not all have been
filled with a particular value) and it's less consistent with things
like 'sorted'. But at least it's short, simple, and -- once you see it
-- memorable. And at least it isn't immediately obvious when looking
at it that it's a fallback choice because all the good names were
taken. I could probably live with np.full.

-n


From ben.root at ou.edu  Fri Jun 14 13:28:30 2013
From: ben.root at ou.edu (Benjamin Root)
Date: Fri, 14 Jun 2013 13:28:30 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAF6FJisCwfP+xFO+_LDVaVFi76r=RtTbpFcEnig9wHDp-c73iQ@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
	<CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>
	<CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>
	<51BA2FFC.2060501@hawaii.edu>
	<CAMMTP+B8o+i05=1d8ABA6sViym5up1a0YsSdNmr9YcY679THxg@mail.gmail.com>
	<CAMtEP6wAZq61h9oDy4mW+sX0Nv4dvEgNDchUpLGiMxAHKk_skw@mail.gmail.com>
	<51BB3381.4000508@gmail.com> <51BB505F.6040107@hawaii.edu>
	<CAF6FJisCwfP+xFO+_LDVaVFi76r=RtTbpFcEnig9wHDp-c73iQ@mail.gmail.com>
Message-ID: <CANNq6FmjbrpzTJ_WEO0822NJFw0951AgM4xaH0K6O3uM0q-kFQ@mail.gmail.com>

On Fri, Jun 14, 2013 at 1:21 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Fri, Jun 14, 2013 at 6:18 PM, Eric Firing <efiring at hawaii.edu> wrote:
> > On 2013/06/14 5:15 AM, Alan G Isaac wrote:
> >> On 6/14/2013 9:27 AM, Aldcroft, Thomas wrote:
> >>> If I just saw np.values(..) in some code I would never guess what it
> is doing from the name
> >>
> >> That suggests np.fromvalues.
> >> But more important than the name I think
> >> is allowing broadcasting of the values,
> >> based on NumPy's broadcasting rules.
> >> Broadcasting a scalar is then a special case,
> >> even if it is the case that has dominated this thread.
> >
> > True, but this looks to me like mission creep.  All of this fuss is
> > about replacing two lines of user code with a single line.  If it can't
> > be kept simple, both in implementation and in documentation, it
> > shouldn't be done at all.  I'm not necessarily opposed to your
> > suggestion, but I'm skeptical.
>
> It's another two-liner:
>
> [~]
> |1> x = np.empty([3,4,5])
>
> [~]
> |2> x[...] = np.arange(5)
>
> [~]
> |3> x
> array([[[ 0.,  1.,  2.,  3.,  4.],
>         [ 0.,  1.,  2.,  3.,  4.],
>         [ 0.,  1.,  2.,  3.,  4.],
>         [ 0.,  1.,  2.,  3.,  4.]],
>
>        [[ 0.,  1.,  2.,  3.,  4.],
>         [ 0.,  1.,  2.,  3.,  4.],
>         [ 0.,  1.,  2.,  3.,  4.],
>         [ 0.,  1.,  2.,  3.,  4.]],
>
>        [[ 0.,  1.,  2.,  3.,  4.],
>         [ 0.,  1.,  2.,  3.,  4.],
>         [ 0.,  1.,  2.,  3.,  4.],
>         [ 0.,  1.,  2.,  3.,  4.]]])
>
> It's wafer-thin!
>
>
True, but wouldn't we rather want to encourage the use of broadcasting in
the numerical operations rather than creating new arrays from broadcasted
arrays?

a = np.arange(5) + np.ones((3, 4, 5))
b = np.filled((3, 4, 5), np.arange(5)) + np.ones((3, 4, 5))

The first one is much easier to read, and is more efficient than the second
(theoretical) one because it needs to create two (3, 4, 5) arrays rather
than just one.  That being said, one could make a similar argument against
ones(), zeros(), etc.

Cheers!
Ben Root
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130614/677488b4/attachment.html>

From alan.isaac at gmail.com  Fri Jun 14 13:33:35 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 14 Jun 2013 13:33:35 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51BB505F.6040107@hawaii.edu>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CALGmxE+YKH=sDbL4r2Ex2Zw293n2SBT-evcLeGkCtrVj-HpZnA@mail.gmail.com>
	<CABL7CQgJV2POUFJuynbraMAKeza3du279Jd-oTYXUTeOqBtS6g@mail.gmail.com>
	<CAPJVwB=EC1GvGfohnLKk+iGoLySU_RNpOs_1AWF5UqFVQ=qPkg@mail.gmail.com>
	<CAGzF1uc==KOmyv5kTZy+LMDMvWWrSHLSZZD6u5Sctcx8Y3uodw@mail.gmail.com>
	<51B8C414.6080508@hawaii.edu>
	<CAMtEP6xH3FCA1BT4URS-GgG9Dvp7eTXgdNJiHq5cYnBTAOscOw@mail.gmail.com>
	<CANNq6F=xFJ=nmSRqkOz2sbrnG_fhsizG9GePFePtTBQ1xCOOmg@mail.gmail.com>
	<51BA2FFC.2060501@hawaii.edu>
	<CAMMTP+B8o+i05=1d8ABA6sViym5up1a0YsSdNmr9YcY679THxg@mail.gmail.com>
	<CAMtEP6wAZq61h9oDy4mW+sX0Nv4dvEgNDchUpLGiMxAHKk_skw@mail.gmail.com>
	<51BB3381.4000508@gmail.com> <51BB505F.6040107@hawaii.edu>
Message-ID: <51BB53EF.8050504@gmail.com>

> On 2013/06/14 5:15 AM, Alan G Isaac wrote:
>> But more important than the name I think
>> is allowing broadcasting of the values,
>> based on NumPy's broadcasting rules.
>> Broadcasting a scalar is then a special case,
>> even if it is the case that has dominated this thread.


On 6/14/2013 1:18 PM, Eric Firing wrote:
> All of this fuss is about replacing two lines of user code
> with a single line.  If it can't be kept simple, both in
> implementation and in documentation, it shouldn't be done
> at all.


Here's another perspective: if that's all we get
(one line instead of two), it isn't worth it at all.
But if we get something more useful (broadcast of values),
it might be worthwhile.

But hmm, once we go there, wouldn't the right thing to do
be to just make sure ``tile`` is efficient and add
a dtype argument?  We already have

>>> np.tile(0.5,(2,3))
array([[ 0.5,  0.5,  0.5],
        [ 0.5,  0.5,  0.5]])

Cheers,
Alan



From ben.root at ou.edu  Fri Jun 14 13:40:30 2013
From: ben.root at ou.edu (Benjamin Root)
Date: Fri, 14 Jun 2013 13:40:30 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBmoDseOK5hq_wW1sDe_dhpwaQ-OO7+PxMM4fp52UVXtPg@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<51B8C164.9000305@hawaii.edu>
	<CAPJVwBmoDseOK5hq_wW1sDe_dhpwaQ-OO7+PxMM4fp52UVXtPg@mail.gmail.com>
Message-ID: <CANNq6Fmd6_mVPEmnm4M2Lp1eDG8YmOcAs0VheJs=poNqJeyX6g@mail.gmail.com>

On Fri, Jun 14, 2013 at 1:22 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Wed, Jun 12, 2013 at 7:43 PM, Eric Firing <efiring at hawaii.edu> wrote:
> > On 2013/06/12 2:10 AM, Nathaniel Smith wrote:
> >> Personally I think that overloading np.empty is horribly ugly, will
> >> continue confusing newbies and everyone else indefinitely, and I'm
> >> 100% convinced that we'll regret implementing such a warty interface
> >> for something that should be so idiomatic. (Unfortunately I got busy
> >> and didn't actually say this in the previous thread though.) So I
> >> think we should just merge the PR as is. The only downside is the
> >> np.ma inconsistency, but, np.ma is already inconsistent (cf.
> >> masked_array.fill versus masked_array.filled!), somewhat deprecated,
> >
> > "somewhat deprecated"?  Really?  Since when?  By whom?  Replaced by what?
>
> Sorry, not trying to start a fight, just trying to summarize the
> situation. As far as I can tell:
>
>
Oh... (puts away iron knuckles)


> Despite heroic efforts on the part of its authors, numpy.ma has a
> number of weird quirks (masked data can still trigger invalid value
> errors), misfeatures (hard versus soft masks), and just plain old pain
> points (ongoing issues with whether any given operation will respect
> or preserve the mask).
>

Actually, now that we have a context manager for warning capture, we could
actually fix that.


>
> It's been in deep maintenance mode for some time; we merge the
> occasional bug fix that people send in, and that's it. (To be fair,
> numpy as a whole is fairly slow-moving, but numpy.ma still gets much
> less attention.)
>
> Even if there were active maintainers, no-one really has any idea how
> to fix any of the problems above; they're not so much bugs as
> intrinsic limitations of the design.
>

Therefore, my impression is that a majority (not all, but a majority)
> of numpy developers strongly recommend against the use of numpy.ma in
> new projects.
>
>
Such a recommendation should be in writing in the documentation and
elsewhere.  Furthermore, a proper replacement would also be needed.  Just
simiply deprecating it without some sort of decent alternative leaves
everybody in a lurch.  I have high hopes for NA to be that replacement, and
the sooner, the better.


> I could be wrong! And I know there's nothing to really replace it. I'd
> like to fix that. But I think "semi-deprecated" is not an unfair
> shorthand for the above.
>
>
You will have to pry np.ma from my cold, dead hands!  (or distract me with
a sufficiently shiny alternative)



> (I'll even admit that I'd *like* to actually deprecate it. But what I
> mean by that is, I don't think it's possible to fix it to the point
> where it's actually a solid/clean/robust library, so I'd like to reach
> a point where everyone who's currently using it is happier switching
> to something else and is happy to sign off on deprecating it.)
>

As far as many people are concerned, it is a solid, clean, robust library.


>
> >> and AFAICT there are far more people who will benefit from a clean
> >> np.filled idiom than who actually use np.ma (and in particular its
> >> fill-value functionality). So there would be two
> >
> > I think there are more np.ma users than you realize.  Everyone who uses
> > matplotlib is using np.ma at least implicitly, if not explicitly.  Many
> > of the matplotlib examples put np.ma to good use.  np.ma.filled is an
> > essential long-standing part of the np.ma API.  I don't see any good
> > rationale for generating a conflict with it, when an adequate
> > non-conflicting alternative ('np.initialized', maybe others) exists.
>
> I'm aware of that. If I didn't care about the opinions of numpy.ma
> users, I wouldn't go starting long and annoying mailing list threads
> about features that are only problematic because of their affect on
> numpy.ma :-).
>
> But, IMHO given the issues with numpy.ma, our number #1 priority ought
> to be making numpy proper as clean and beautiful as possible; my
> position that started this thread is basically just that we shouldn't
> make numpy proper worse just for numpy.ma's sake. That's the tail
> wagging the dog. And this 'conflict' seems a bit overstated given that
> (1) np.ma.filled already has multiple names (and 3/4 of the uses in
> matplotlib use the method version, not the function version), (2) even
> if we give it a non-conflicting name, np.ma's lack of maintenance
> means that it'd probably be years before someone got around to
> actually adding a parallel function to np.ma. [Unless this thread
> spurs someone into submitting one just to prove me wrong ;-).]
>
>
Actually, IIRC, np.ma does some sort of auto-wrapping of numpy functions.
This is why adding np.filled() would cause a namespace clobbering, I think.

Cheers!
Ben Root
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130614/1e5c1c67/attachment.html>

From njs at pobox.com  Fri Jun 14 14:23:03 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 14 Jun 2013 19:23:03 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CANNq6Fmd6_mVPEmnm4M2Lp1eDG8YmOcAs0VheJs=poNqJeyX6g@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<51B8C164.9000305@hawaii.edu>
	<CAPJVwBmoDseOK5hq_wW1sDe_dhpwaQ-OO7+PxMM4fp52UVXtPg@mail.gmail.com>
	<CANNq6Fmd6_mVPEmnm4M2Lp1eDG8YmOcAs0VheJs=poNqJeyX6g@mail.gmail.com>
Message-ID: <CAPJVwBnmD+3cgVc+8BuE2oL9HDaGiGzLU+7kSktHFTU29qZ_yw@mail.gmail.com>

On Fri, Jun 14, 2013 at 6:40 PM, Benjamin Root <ben.root at ou.edu> wrote:
> On Fri, Jun 14, 2013 at 1:22 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Wed, Jun 12, 2013 at 7:43 PM, Eric Firing <efiring at hawaii.edu> wrote:
>> > On 2013/06/12 2:10 AM, Nathaniel Smith wrote:
>> Despite heroic efforts on the part of its authors, numpy.ma has a
>> number of weird quirks (masked data can still trigger invalid value
>> errors), misfeatures (hard versus soft masks), and just plain old pain
>> points (ongoing issues with whether any given operation will respect
>> or preserve the mask).
>
>
> Actually, now that we have a context manager for warning capture, we could
> actually fix that.

PRs welcome!

>> It's been in deep maintenance mode for some time; we merge the
>> occasional bug fix that people send in, and that's it. (To be fair,
>> numpy as a whole is fairly slow-moving, but numpy.ma still gets much
>> less attention.)
>>
>> Even if there were active maintainers, no-one really has any idea how
>> to fix any of the problems above; they're not so much bugs as
>> intrinsic limitations of the design.
>>
>>
>> Therefore, my impression is that a majority (not all, but a majority)
>> of numpy developers strongly recommend against the use of numpy.ma in
>> new projects.
>>
>
> Such a recommendation should be in writing in the documentation and
> elsewhere.

Maybe, but someone needs to write it, and in a way that gets past the
list. Feel free to try :-).

> Furthermore, a proper replacement would also be needed.  Just
> simiply deprecating it without some sort of decent alternative leaves
> everybody in a lurch.  I have high hopes for NA to be that replacement, and
> the sooner, the better.

Again, this would be great, just, someone needs to do it :-).

>> But, IMHO given the issues with numpy.ma, our number #1 priority ought
>> to be making numpy proper as clean and beautiful as possible; my
>> position that started this thread is basically just that we shouldn't
>> make numpy proper worse just for numpy.ma's sake. That's the tail
>> wagging the dog. And this 'conflict' seems a bit overstated given that
>> (1) np.ma.filled already has multiple names (and 3/4 of the uses in
>> matplotlib use the method version, not the function version), (2) even
>> if we give it a non-conflicting name, np.ma's lack of maintenance
>> means that it'd probably be years before someone got around to
>> actually adding a parallel function to np.ma. [Unless this thread
>> spurs someone into submitting one just to prove me wrong ;-).]
>>
>
> Actually, IIRC, np.ma does some sort of auto-wrapping of numpy functions.
> This is why adding np.filled() would cause a namespace clobbering, I think.

Nope. np.ma manually wraps and re-exports every object that it
exports. Generally speaking it has to, because there's no way to look
at an arbitrary numpy function and know what the np.ma equivalent
would be. But even in cases where auto-wrapping would be possible and
useful, it doesn't do it. (E.g., it's easy to get a list of all ufunc
objects in the np.* namespace and wrap them all automatically, but
np.ma doesn't, it just has a big and possibly out-of-date list of all
the ufuncs that it knows about.)

-n


From efiring at hawaii.edu  Fri Jun 14 14:23:10 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Fri, 14 Jun 2013 08:23:10 -1000
Subject: [Numpy-discussion] NA, and replacement or reimplimentation of np.ma
In-Reply-To: <CAPJVwBmoDseOK5hq_wW1sDe_dhpwaQ-OO7+PxMM4fp52UVXtPg@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<51B8C164.9000305@hawaii.edu>
	<CAPJVwBmoDseOK5hq_wW1sDe_dhpwaQ-OO7+PxMM4fp52UVXtPg@mail.gmail.com>
Message-ID: <51BB5F8E.2070209@hawaii.edu>

On 2013/06/14 7:22 AM, Nathaniel Smith wrote:
> On Wed, Jun 12, 2013 at 7:43 PM, Eric Firing <efiring at hawaii.edu> wrote:
>> On 2013/06/12 2:10 AM, Nathaniel Smith wrote:
>>> Personally I think that overloading np.empty is horribly ugly, will
>>> continue confusing newbies and everyone else indefinitely, and I'm
>>> 100% convinced that we'll regret implementing such a warty interface
>>> for something that should be so idiomatic. (Unfortunately I got busy
>>> and didn't actually say this in the previous thread though.) So I
>>> think we should just merge the PR as is. The only downside is the
>>> np.ma inconsistency, but, np.ma is already inconsistent (cf.
>>> masked_array.fill versus masked_array.filled!), somewhat deprecated,
>>
>> "somewhat deprecated"?  Really?  Since when?  By whom?  Replaced by what?
>
> Sorry, not trying to start a fight, just trying to summarize the
> situation. As far as I can tell:
>
> Despite heroic efforts on the part of its authors, numpy.ma has a
> number of weird quirks (masked data can still trigger invalid value
> errors), misfeatures (hard versus soft masks), and just plain old pain
> points (ongoing issues with whether any given operation will respect
> or preserve the mask).
>
> It's been in deep maintenance mode for some time; we merge the
> occasional bug fix that people send in, and that's it. (To be fair,
> numpy as a whole is fairly slow-moving, but numpy.ma still gets much
> less attention.)
>
> Even if there were active maintainers, no-one really has any idea how
> to fix any of the problems above; they're not so much bugs as
> intrinsic limitations of the design.
>
> Therefore, my impression is that a majority (not all, but a majority)
> of numpy developers strongly recommend against the use of numpy.ma in
> new projects.
>
> I could be wrong! And I know there's nothing to really replace it. I'd
> like to fix that. But I think "semi-deprecated" is not an unfair
> shorthand for the above.
>
> (I'll even admit that I'd *like* to actually deprecate it. But what I
> mean by that is, I don't think it's possible to fix it to the point
> where it's actually a solid/clean/robust library, so I'd like to reach
> a point where everyone who's currently using it is happier switching
> to something else and is happy to sign off on deprecating it.)
>

Nathaniel,

I've been pondering when to bring this up again, but you did it for me, 
so here it is with a new title for the thread.  Maybe it will be short 
and sweet, maybe not.

I think we can agree that there is major interest in having good numpy 
support for one or more styles of missing/masked values.  You might not 
agree, but I will assert that the style of support provided by np.ma is 
*very* useful; it serves a real purpose in working code.  We do agree 
that np.ma has problems.  It is not at all clear to me, however, that 
those problems cannot or should not be fixed.  Even if they can't, I 
don't think they are so severe that it is wise to try to kill off np.ma 
*before* there is a good replacement.

In the NA branch, an attempt was made to lay the groundwork for solid 
missing/masked support.  I did not agree with every design aspect, but I 
thought it was nevertheless good as groundwork, and could be used to 
greatly improve np.ma, to provide a different style of support for those 
who require it, and perhaps to lead over the very long term to a 
withering away of the need for np.ma.

Some of the groundwork from the NA branch survived, but most of it is 
sitting off to the side.

Is there any way to revive this line of development?  To satisfy the 
needs of people coming from the R world *and* of people for whom np.ma 
is, despite its warts, an important tool?  This seems to me to be the 
single biggest area where numpy needs development.

It looks like this problem needs dedicated resources: a grant, a major 
corporate effort, or both.

Numpy is central to python in science, but it doesn't seem to have a 
corresponding level of direction and support.

Eric




From pgmdevlist at gmail.com  Fri Jun 14 15:35:05 2013
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 14 Jun 2013 21:35:05 +0200
Subject: [Numpy-discussion] NA,
	and replacement or reimplimentation of np.ma
In-Reply-To: <51BB5F8E.2070209@hawaii.edu>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<51B8C164.9000305@hawaii.edu>
	<CAPJVwBmoDseOK5hq_wW1sDe_dhpwaQ-OO7+PxMM4fp52UVXtPg@mail.gmail.com>
	<51BB5F8E.2070209@hawaii.edu>
Message-ID: <6975E850-2577-4A68-8EFA-B11C4ADF7C94@gmail.com>


On Jun 14, 2013, at 20:23 , Eric Firing <efiring at hawaii.edu> wrote:

> On 2013/06/14 7:22 AM, Nathaniel Smith wrote:
>> On Wed, Jun 12, 2013 at 7:43 PM, Eric Firing <efiring at hawaii.edu> wrote:
>>> On 2013/06/12 2:10 AM, Nathaniel Smith wrote:
>>>> Personally I think that overloading np.empty is horribly ugly, will
>>>> continue confusing newbies and everyone else indefinitely, and I'm
>>>> 100% convinced that we'll regret implementing such a warty interface
>>>> for something that should be so idiomatic. (Unfortunately I got busy
>>>> and didn't actually say this in the previous thread though.) So I
>>>> think we should just merge the PR as is. The only downside is the
>>>> np.ma inconsistency, but, np.ma is already inconsistent (cf.
>>>> masked_array.fill versus masked_array.filled!), somewhat deprecated,
>>> 
>>> "somewhat deprecated"?  Really?  Since when?  By whom?  Replaced by what?
>> 
>> Sorry, not trying to start a fight, just trying to summarize the
>> situation. As far as I can tell:
>> 
>> Despite heroic efforts on the part of its authors, numpy.ma has a
>> number of weird quirks (masked data can still trigger invalid value
>> errors), misfeatures (hard versus soft masks), and just plain old pain
>> points (ongoing issues with whether any given operation will respect
>> or preserve the mask).

The "invalid value errors" are a side-effect of some design decisions taken 6-7 years ago. It turned out to be more efficient in terms of speed to follow an approach "compute without the mask, put it back afterwards" than the original "mask before, fill the holes with some value, compute, put the mask back": some functions like `pow` that were not part of the very first implementations twisted my arm on this one. It's far from perfect, it's rather disappointing, but I don't see a workaround with the current "let's do it in python" approach. Any other implementation would have to be done directly in C (or maybe in Cython, it's been 5 years since I last touched it).

>> 
>> It's been in deep maintenance mode for some time; we merge the
>> occasional bug fix that people send in, and that's it. (To be fair,
>> numpy as a whole is fairly slow-moving, but numpy.ma still gets much
>> less attention.)

It never had a lot... 

>> Therefore, my impression is that a majority (not all, but a majority)
>> of numpy developers strongly recommend against the use of numpy.ma in
>> new projects.

And you take that from? OK, to be frank, *I* would advise against a very naive use of np.ma: there are plenty of tricks to know to be really efficient with masked arrays. Most of the functions of the module are just for convenience in interactive mode? 

> I think we can agree that there is major interest in having good numpy 
> support for one or more styles of missing/masked values.  You might not 
> agree, but I will assert that the style of support provided by np.ma is 
> *very* useful; it serves a real purpose in working code.  We do agree 
> that np.ma has problems.  It is not at all clear to me, however, that 
> those problems cannot or should not be fixed.  Even if they can't, I 
> don't think they are so severe that it is wise to try to kill off np.ma 
> *before* there is a good replacement.

Quite agreed with that
> 
> In the NA branch, an attempt was made to lay the groundwork for solid 
> missing/masked support.  I did not agree with every design aspect,

Talking about it, was a consensus (or at least a majority) reached about NA w/vs missing data ?

> but I 
> thought it was nevertheless good as groundwork, and could be used to 
> greatly improve np.ma, to provide a different style of support for those 
> who require it, and perhaps to lead over the very long term to a 
> withering away of the need for np.ma.

When I started rewriting np.ma, Paul Dubois wrote me that 'if he were to do it again, it'd be in C, and that he disagreed with my approach' (I'm paraphrasing, but the gist is here). Of course, like every kid, I thought I knew better. In retrospect, he was quite right. I'm no longer convinced that MaskedArray as a subclass of ndarray is a correct approach. It works, it worked well enough for my needs at the time, it was a very educational journey, but if I were to do it again...

> Is there any way to revive this line of development?  To satisfy the 
> needs of people coming from the R world *and* of people for whom np.ma 
> is, despite its warts, an important tool?  This seems to me to be the 
> single biggest area where numpy needs development.

I'm always surprised by the antagonism some people have towards np.ma? You can't always use NaN to represent the missing information you're doomed to meet in the real world.

> It looks like this problem needs dedicated resources: a grant, a major 
> corporate effort, or both.

<plug class="shameless">Fund me I'm yours</plug

> Numpy is central to python in science, but it doesn't seem to have a 
> corresponding level of direction and support.


Ecco.
More seriously, I'd be delighted to help. I can no longer work on it full time as I used to (even if I were not supposed to) but I can often explain why things were done the way they are and how we could improve them..



From ralf.gommers at gmail.com  Fri Jun 14 16:35:27 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 14 Jun 2013 22:35:27 +0200
Subject: [Numpy-discussion] Dropping support for, Accelerate/veclib?
In-Reply-To: <51B9AC32.4060701@gmail.com>
References: <51B75CC6.4090405@amolf.nl>
	<CALGmxEJZgGO20u+r4FigVVb2ep0jgdxmMm8jXhQZ-4LmbrhCfA@mail.gmail.com>
	<CABL7CQjtWf8-b6k=XH2xDarF=JmFMavy+=FvXbLR3k22brKtUQ@mail.gmail.com>
	<CALGmxEKv-By0KmrWSJ0Ura0g=E3rvwoXGjw7ttF7OeWddV9_5w@mail.gmail.com>
	<51B9AC32.4060701@gmail.com>
Message-ID: <CABL7CQiRZAS6Jef2cV5QSu8LTmHuAc3A2Vgi04smShacB7Ggbw@mail.gmail.com>

On Thu, Jun 13, 2013 at 1:25 PM, Andrew Jaffe <a.h.jaffe at gmail.com> wrote:

> On 11/06/2013 22:11, Chris Barker - NOAA Federal wrote:
> > On Tue, Jun 11, 2013 at 1:28 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >> The binaries will still be built against python.org Python, so there
> >> shouldn't be an issue here. Same for building from source.
> >
> > My point was that it's nice to be able to have it build with an out of
> > teh box wetup.py with accelerated LAPACK and all... If whoever is
> > building binaries wants to get fancy, great.
>
> Yes, please. The current system does seem to work for at least some of
> us.  And, if I understand the thread in the scipy mailing list, it's not
> actually clear that there's a bug, as opposed to incompatible fortran
> ABIs (which doesn't seem like a bug to me).
>
> But I guess the most important thing would be that it can be used with
> apple or python.org Python builds (my reading of some of the suggestions
> would be requiring one of homebrew/fink/macports),


No we should support all pythons; python.org being the one supported by our
binary installers.

Ralf


> preferably out-of-the-box -- even if that meant restricting to prebuilt
> binaries.
> Being able to run non-obscure installers (i.e., from the main python.org
> and scipy.org sites) for Python + numpy + scipy + matplotlib and get
> optimized versions would be sufficient.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130614/bf0ce322/attachment.html>

From efiring at hawaii.edu  Fri Jun 14 18:38:29 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Fri, 14 Jun 2013 12:38:29 -1000
Subject: [Numpy-discussion] time to revisit NA/ma ideas
Message-ID: <51BB9B65.9090807@hawaii.edu>

A nice summary of the discussions from a year ago is here:

http://www.numpy.org/NA-overview.html

It provides food for thought.

Eric


From a.kumar at alumni.iitm.ac.in  Fri Jun 14 20:59:03 2013
From: a.kumar at alumni.iitm.ac.in (Kumar Appaiah)
Date: Fri, 14 Jun 2013 20:59:03 -0400
Subject: [Numpy-discussion] Generating special polynomials (Chebyshev,
	Hermite etc.)
Message-ID: <20130615005903.GA3058@bluemoon.alumni.iitm.ac.in>

Dear Numpy Users,

I am trying to find out a way by which I can easily generate the n-th
order "special" polynomial, where "special" could refer to Hermite,
Chebyshev etc. Numpy 1.7 introduces several methods for such
polynomials, but I couldn't find a convenience function that gives me
a polynomial directly based on degree. For instance, I'd like:

hermite(3) to result in array([  0., -12.,   0.,   8.])
hermite(6) to result in array([-120.,    0.,  720.,    0., -480.,    0.,   64.])
and so on.

The quickest way I could come up with for this is:

def hermite(n):
    if n <= 0:
        return numpy.array([1.0])
    coeff_polynomial = [0.0] * n
    coeff_polynomial.extend([1])
    return numpy.polynomial.hermite.herm2poly(coeff_polynomial)

Now, if I am missing something, please let me know. If you think this
is a useful feature, I volunteer to patch all the polynomial modules
to generate such polynomials, if you could tell me appropriate
function names for such convenience functions.

Thanks!

Kumar
-- 
Kumar Appaiah


From a.kumar at alumni.iitm.ac.in  Fri Jun 14 21:02:39 2013
From: a.kumar at alumni.iitm.ac.in (Kumar Appaiah)
Date: Fri, 14 Jun 2013 21:02:39 -0400
Subject: [Numpy-discussion] Generating special polynomials (Chebyshev,
 Hermite etc.)
In-Reply-To: <20130615005903.GA3058@bluemoon.alumni.iitm.ac.in>
References: <20130615005903.GA3058@bluemoon.alumni.iitm.ac.in>
Message-ID: <20130615010239.GA3446@bluemoon.alumni.iitm.ac.in>

On Fri, Jun 14, 2013 at 08:59:03PM -0400, Kumar Appaiah wrote:
> Dear Numpy Users,
> 
> I am trying to find out a way by which I can easily generate the n-th
> order "special" polynomial, where "special" could refer to Hermite,
> Chebyshev etc. Numpy 1.7 introduces several methods for such
> polynomials, but I couldn't find a convenience function that gives me
> a polynomial directly based on degree. For instance, I'd like:
> 
> hermite(3) to result in array([  0., -12.,   0.,   8.])
> hermite(6) to result in array([-120.,    0.,  720.,    0., -480.,    0.,   64.])
> and so on.
> 
> The quickest way I could come up with for this is:
> 
> def hermite(n):
>     if n <= 0:
>         return numpy.array([1.0])

I should technically raise a ValueError here if n is below 0, but I'll
do the right thing in a patch if I am asked for one.

>     coeff_polynomial = [0.0] * n
>     coeff_polynomial.extend([1])
>     return numpy.polynomial.hermite.herm2poly(coeff_polynomial)
> 
> Now, if I am missing something, please let me know. If you think this
> is a useful feature, I volunteer to patch all the polynomial modules
> to generate such polynomials, if you could tell me appropriate
> function names for such convenience functions.

Thanks!

Kumar

-- 
Kumar Appaiah


From charlesr.harris at gmail.com  Fri Jun 14 22:07:57 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 14 Jun 2013 20:07:57 -0600
Subject: [Numpy-discussion] Generating special polynomials (Chebyshev,
 Hermite etc.)
In-Reply-To: <20130615005903.GA3058@bluemoon.alumni.iitm.ac.in>
References: <20130615005903.GA3058@bluemoon.alumni.iitm.ac.in>
Message-ID: <CAB6mnxK2Rf_G5Ei9xfMKKHbjFnFmA8zBbdhJnGzFp1RA3UtZYg@mail.gmail.com>

On Fri, Jun 14, 2013 at 6:59 PM, Kumar Appaiah <a.kumar at alumni.iitm.ac.in>wrote:

> Dear Numpy Users,
>
> I am trying to find out a way by which I can easily generate the n-th
> order "special" polynomial, where "special" could refer to Hermite,
> Chebyshev etc. Numpy 1.7 introduces several methods for such
> polynomials, but I couldn't find a convenience function that gives me
> a polynomial directly based on degree. For instance, I'd like:
>
> hermite(3) to result in array([  0., -12.,   0.,   8.])
> hermite(6) to result in array([-120.,    0.,  720.,    0., -480.,    0.,
> 64.])
> and so on.
>

Generally that is a bad idea, polynomials tend to be numerically unstable
and you lose all the virtue of the Hermite basis. However, you can do

In [1]: from numpy.polynomial import Polynomial, Hermite

In [2]: p = Hermite.basis(5)

In [3]: p.convert(kind=Polynomial)
Out[3]: Polynomial([   0.,  120.,    0., -160.,    0.,   32.], [-1.,  1.],
[-1.,  1.])

In [4]: Polynomial.cast(p)
Out[4]: Polynomial([   0.,  120.,    0., -160.,    0.,   32.], [-1.,  1.],
[-1.,  1.])

In [5]: from numpy.polynomial import Chebyshev

In [6]: Chebyshev.cast(p)
Out[6]: Chebyshev([  0.,  20.,   0., -30.,   0.,   2.], [-1.,  1.], [-1.,
1.])

Hmm, it should be possible to make the constructor take polynomials of
different kinds since they all derive from PolyBase and can be detected.
That could replace the cast method in a nice way.


> The quickest way I could come up with for this is:
>
> def hermite(n):
>     if n <= 0:
>         return numpy.array([1.0])
>     coeff_polynomial = [0.0] * n
>     coeff_polynomial.extend([1])
>     return numpy.polynomial.hermite.herm2poly(coeff_polynomial)
>
> Now, if I am missing something, please let me know. If you think this
> is a useful feature, I volunteer to patch all the polynomial modules
> to generate such polynomials, if you could tell me appropriate
> function names for such convenience functions.
>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130614/186db008/attachment.html>

From a.kumar at alumni.iitm.ac.in  Sat Jun 15 00:29:11 2013
From: a.kumar at alumni.iitm.ac.in (Kumar Appaiah)
Date: Sat, 15 Jun 2013 00:29:11 -0400
Subject: [Numpy-discussion] Generating special polynomials (Chebyshev,
 Hermite etc.)
In-Reply-To: <CAB6mnxK2Rf_G5Ei9xfMKKHbjFnFmA8zBbdhJnGzFp1RA3UtZYg@mail.gmail.com>
References: <20130615005903.GA3058@bluemoon.alumni.iitm.ac.in>
	<CAB6mnxK2Rf_G5Ei9xfMKKHbjFnFmA8zBbdhJnGzFp1RA3UtZYg@mail.gmail.com>
Message-ID: <20130615042911.GA24115@bluemoon.alumni.iitm.ac.in>

On Fri, Jun 14, 2013 at 08:07:57PM -0600, Charles R Harris wrote:
>      I am trying to find out a way by which I can easily generate the n-th
>      order "special" polynomial, where "special" could refer to Hermite,
>      Chebyshev etc. Numpy 1.7 introduces several methods for such
>      polynomials, but I couldn't find a convenience function that gives me
>      a polynomial directly based on degree. For instance, I'd like:
> 
>      hermite(3) to result in array([ ?0., -12., ? 0., ? 8.])
>      hermite(6) to result in array([-120., ? ?0., ?720., ? ?0., -480., ? ?0.,
>      ? 64.])
>      and so on.
> 
>    Generally that is a bad idea, polynomials tend to be numerically unstable
>    and you lose all the virtue of the Hermite basis. However, you can do

This may be true, but I am not concerned with the numerical
instability in my current application. Nevertheless, I understand it
if you don't want to introduce such a function.

>    In [1]: from numpy.polynomial import Polynomial, Hermite
> 
>    In [2]: p = Hermite.basis(5)
> 
>    In [3]: p.convert(kind=Polynomial)
>    Out[3]: Polynomial([?? 0.,? 120.,??? 0., -160.,??? 0.,?? 32.], [-1.,? 1.],
>    [-1.,? 1.])
> 
>    In [4]: Polynomial.cast(p)
>    Out[4]: Polynomial([?? 0.,? 120.,??? 0., -160.,??? 0.,?? 32.], [-1.,? 1.],
>    [-1.,? 1.])
> 
>    In [5]: from numpy.polynomial import Chebyshev
> 
>    In [6]: Chebyshev.cast(p)
>    Out[6]: Chebyshev([? 0.,? 20.,?? 0., -30.,?? 0.,?? 2.], [-1.,? 1.], [-1.,?
>    1.])
> 
>    Hmm, it should be possible to make the constructor take polynomials of
>    different kinds since they all derive from PolyBase and can be detected.
>    That could replace the cast method in a nice way.

I now see that the polynomial structure is intended to be "rich", as
opposed to the na?ve function that I proposed. In the least, though,
the documentation could reflect the example you gave me. I could send
a patch that adds an example for each of the polynomial types in the
documentation, much like yours, if that would be useful.

Thanks for the clarification.

Kumar
-- 
Kumar Appaiah


From a.kumar at alumni.iitm.ac.in  Sat Jun 15 00:44:37 2013
From: a.kumar at alumni.iitm.ac.in (Kumar Appaiah)
Date: Sat, 15 Jun 2013 00:44:37 -0400
Subject: [Numpy-discussion] Generating special polynomials (Chebyshev,
 Hermite etc.)
In-Reply-To: <20130615042911.GA24115@bluemoon.alumni.iitm.ac.in>
References: <20130615005903.GA3058@bluemoon.alumni.iitm.ac.in>
	<CAB6mnxK2Rf_G5Ei9xfMKKHbjFnFmA8zBbdhJnGzFp1RA3UtZYg@mail.gmail.com>
	<20130615042911.GA24115@bluemoon.alumni.iitm.ac.in>
Message-ID: <20130615044437.GA26733@bluemoon.alumni.iitm.ac.in>

On Sat, Jun 15, 2013 at 12:29:11AM -0400, Kumar Appaiah wrote:
> I now see that the polynomial structure is intended to be "rich", as
> opposed to the na?ve function that I proposed. In the least, though,
> the documentation could reflect the example you gave me. I could send
> a patch that adds an example for each of the polynomial types in the
> documentation, much like yours, if that would be useful.

Upon checking the latest numpy documentation, I found a hint for this
here:

http://docs.scipy.org/doc/numpy/reference/routines.polynomials.classes.html#other-polynomial-constructors

The documentation, when read as a whole, looks clear. I was
complaining earlier because I was referring to pre-1.7 documentation,
and hadn't familiarized myself with the new polynomial interface as
well.

Sorry for the noise. I'll bother you the list again if I run into
additional trouble.

Thanks!

Kumar
-- 
Kumar Appaiah


From sebastian at sipsolutions.net  Sat Jun 15 06:00:30 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Sat, 15 Jun 2013 12:00:30 +0200
Subject: [Numpy-discussion] Not checking github
Message-ID: <1371290430.19563.5.camel@sebastian-laptop>

Just to note, I disabled most github notifications for time reasons for
the next months, so if something pops up that you think I should look
at, use @mention.

- Sebastian



From klonuo at gmail.com  Sat Jun 15 06:47:00 2013
From: klonuo at gmail.com (klo uo)
Date: Sat, 15 Jun 2013 12:47:00 +0200
Subject: [Numpy-discussion] Reading data from structured text
Message-ID: <CAA-8Ld_Ha2dd4LA5Fd-wqJWc-O2=z_jF0T4x-CAJFYZ52mZeRw@mail.gmail.com>

Hi,

I have data reports in text files, where first 5 lines describe the
data following, which is actually continuous time series of 2048
values wrapped in 205 rows and 10 columns, and each file has 12 such
sets.

If I crop to first dataset and leave the headers (first 5 lines),
genfromtxt(skip_header=5) raises ValueError because last row contains
8 records instead 10 like the rest.

If I just try to read file as is, with same argument,
genfromtxt(skip_header=5), I get many errors as genfromtxt() doesn't
seem to be able to detect header present on every dataset.

So can I read such file with numpy and learn for future use, or should
I just write my reader?


From sudheer.joseph at yahoo.com  Sat Jun 15 08:22:24 2013
From: sudheer.joseph at yahoo.com (Sudheer Joseph)
Date: Sat, 15 Jun 2013 20:22:24 +0800 (SGT)
Subject: [Numpy-discussion] saving 3d array
Message-ID: <1371298944.8110.YahooMailNeo@web193403.mail.sg3.yahoo.com>

Dear Experts,
??????????????????? I was trying to save results of eof analysis to an npz file and see that it is not possible to save a 3d array as npz file variable. Is this true even today or are there developments which make this possible?? I am able to save all other variables except the 3d array.
with best regards,
Sudheer

?eof1.shape
Out[61]: (10, 140, 180)

pc1.shape
Out[62]: (2192, 10)

x.shape
Out[63]: (140, 180)

np.savez(outf,pcs=pc1,x=x,y=y,clevs=clevs)

works fine
but below do not

np.savez(outf,eof1=eof1)
---------------------------------------------------------------------------
NotImplementedError?????????????????????? Traceback (most recent call last)
/home/sjo/RAMA_20120807/adcp/<ipython-input-66-fe5940e44a77> in <module>()
----> 1 np.savez(outf,eof1=eof1)

/usr/local/lib/python2.7/dist-packages/numpy-1.7.0-py2.7-linux-x86_64.egg/numpy/lib/npyio.pyc in savez(file, *args, **kwds)
??? 514 
??? 515???? """
--> 516???? _savez(file, args, kwds, False)
??? 517 
??? 518 def savez_compressed(file, *args, **kwds):

/usr/local/lib/python2.7/dist-packages/numpy-1.7.0-py2.7-linux-x86_64.egg/numpy/lib/npyio.pyc in _savez(file, args, kwds, compress)
??? 573???????????? fid = open(tmpfile, 'wb')
??? 574???????????? try:
--> 575???????????????? format.write_array(fid, np.asanyarray(val))
??? 576???????????????? fid.close()
??? 577???????????????? fid = None

/usr/local/lib/python2.7/dist-packages/numpy-1.7.0-py2.7-linux-x86_64.egg/numpy/lib/format.pyc in write_array(fp, array, version)
??? 407???? else:
??? 408???????? if isfileobj(fp):
--> 409???????????? array.tofile(fp)
??? 410???????? else:
??? 411???????????? # XXX: We could probably chunk this using something like


/usr/local/lib/python2.7/dist-packages/numpy-1.7.0-py2.7-linux-x86_64.egg/numpy/ma/core.pyc in tofile(self, fid, sep, format)
?? 5380 
?? 5381???????? """
-> 5382???????? raise NotImplementedError("Not implemented yet, sorry...")
?? 5383 
?? 5384???? def toflex(self):

NotImplementedError: Not implemented yet, sorry...

***************************************************************
Sudheer Joseph 
Indian National Centre for Ocean Information Services
Ministry of Earth Sciences, Govt. of India
POST BOX NO: 21, IDA Jeedeemetla P.O.
Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55
Tel:+91-40-23886047(O),Fax:+91-40-23895011(O),
Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile)
E-mail:sjo.India at gmail.com;sudheer.joseph at yahoo.com
Web- http://oppamthadathil.tripod.com
***************************************************************


From matthew.brett at gmail.com  Sat Jun 15 08:28:43 2013
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 15 Jun 2013 13:28:43 +0100
Subject: [Numpy-discussion] saving 3d array
In-Reply-To: <1371298944.8110.YahooMailNeo@web193403.mail.sg3.yahoo.com>
References: <1371298944.8110.YahooMailNeo@web193403.mail.sg3.yahoo.com>
Message-ID: <CAH6Pt5o5BRqgAZQbCQ734rPiRKc2h31F8OgHNkvcbkXOrot9-w@mail.gmail.com>

Hi,

On Sat, Jun 15, 2013 at 1:22 PM, Sudheer Joseph
<sudheer.joseph at yahoo.com> wrote:
> Dear Experts,
>                     I was trying to save results of eof analysis to an npz file and see that it is not possible to save a 3d array as npz file variable. Is this true even today or are there developments which make this possible?  I am able to save all other variables except the 3d array.
> with best regards,
> Sudheer
>
>  eof1.shape
> Out[61]: (10, 140, 180)
>
> pc1.shape
> Out[62]: (2192, 10)
>
> x.shape
> Out[63]: (140, 180)
>
> np.savez(outf,pcs=pc1,x=x,y=y,clevs=clevs)
>
> works fine
> but below do not
>
> np.savez(outf,eof1=eof1)
> ---------------------------------------------------------------------------
> NotImplementedError                       Traceback (most recent call last)
> /home/sjo/RAMA_20120807/adcp/<ipython-input-66-fe5940e44a77> in <module>()
> ----> 1 np.savez(outf,eof1=eof1)
>
> /usr/local/lib/python2.7/dist-packages/numpy-1.7.0-py2.7-linux-x86_64.egg/numpy/lib/npyio.pyc in savez(file, *args, **kwds)
>     514
>     515     """
> --> 516     _savez(file, args, kwds, False)
>     517
>     518 def savez_compressed(file, *args, **kwds):
>
> /usr/local/lib/python2.7/dist-packages/numpy-1.7.0-py2.7-linux-x86_64.egg/numpy/lib/npyio.pyc in _savez(file, args, kwds, compress)
>     573             fid = open(tmpfile, 'wb')
>     574             try:
> --> 575                 format.write_array(fid, np.asanyarray(val))
>     576                 fid.close()
>     577                 fid = None
>
> /usr/local/lib/python2.7/dist-packages/numpy-1.7.0-py2.7-linux-x86_64.egg/numpy/lib/format.pyc in write_array(fp, array, version)
>     407     else:
>     408         if isfileobj(fp):
> --> 409             array.tofile(fp)
>     410         else:
>     411             # XXX: We could probably chunk this using something like
>
>
> /usr/local/lib/python2.7/dist-packages/numpy-1.7.0-py2.7-linux-x86_64.egg/numpy/ma/core.pyc in tofile(self, fid, sep, format)
>    5380
>    5381         """
> -> 5382         raise NotImplementedError("Not implemented yet, sorry...")
>    5383
>    5384     def toflex(self):
>
> NotImplementedError: Not implemented yet, sorry...

>From the traceback, it looks as though this is an error saving a
masked array, rather than a 3D array.   Can you convert to a
non-masked array before saving?  Maybe you could save the mask and the
unmasked array separately?

Best,

Matthew


From charlesr.harris at gmail.com  Sat Jun 15 09:25:40 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 15 Jun 2013 07:25:40 -0600
Subject: [Numpy-discussion] Not checking github
In-Reply-To: <1371290430.19563.5.camel@sebastian-laptop>
References: <1371290430.19563.5.camel@sebastian-laptop>
Message-ID: <CAB6mnxKusSS3kRMu1TC-WLb-YJ69AjgvvanCFP_p2t6pqPZk_Q@mail.gmail.com>

On Sat, Jun 15, 2013 at 4:00 AM, Sebastian Berg
<sebastian at sipsolutions.net>wrote:

> Just to note, I disabled most github notifications for time reasons for
> the next months, so if something pops up that you think I should look
> at, use @mention.
>

OK. Thanks for the work you have done. We'll look for your return.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/331a251c/attachment.html>

From sudheer.joseph at yahoo.com  Sat Jun 15 09:51:40 2013
From: sudheer.joseph at yahoo.com (Sudheer Joseph)
Date: Sat, 15 Jun 2013 21:51:40 +0800 (SGT)
Subject: [Numpy-discussion] saving 3d array
In-Reply-To: <CAH6Pt5o5BRqgAZQbCQ734rPiRKc2h31F8OgHNkvcbkXOrot9-w@mail.gmail.com>
Message-ID: <1371304300.53915.YahooMailMobile@web193404.mail.sg3.yahoo.com>

Thank you very much for this tip.
Is there a typical way to save masked and  the rest separately?. Not much familiar with array handling in numpy.
Sudheer
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/4c1fc73d/attachment.html>

From tmp50 at ukr.net  Sat Jun 15 11:02:01 2013
From: tmp50 at ukr.net (Dmitrey)
Date: Sat, 15 Jun 2013 18:02:01 +0300
Subject: [Numpy-discussion] new OpenOpt Suite release 0.50
Message-ID: <91637.1371308521.15870062949008277504@ffe6.ukr.net>




Hi all,

>

I'm glad to inform you about new OpenOpt Suite release 0.50
(2013-June-15):

>


>

* interalg (solver with specifiable accuracy) now works many times
(sometimes orders) faster on (possibly multidimensional) integration
problems (IP) and on some optimization problems
* Add modeling dense (MI)(QC)QP in FuncDesigner (alpha-version, rendering
may work slowly yet)
* Bugfix for cplex wrapper
* Some improvements for FuncDesigner interval analysis (and thus
interalg)
* Add FuncDesigner interval analysis for tan in range(-pi/2,pi/2)
* Some other bugfixes and improvements
* (Proprietary) FuncDesigner stochastic addon now is available as
standalone pyc-file, became available for Python3 as well

>


>

Regards, Dmitrey.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/9bcc9fd6/attachment.html>

From matthew.brett at gmail.com  Sat Jun 15 11:35:33 2013
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 15 Jun 2013 16:35:33 +0100
Subject: [Numpy-discussion] saving 3d array
In-Reply-To: <1371304300.53915.YahooMailMobile@web193404.mail.sg3.yahoo.com>
References: <CAH6Pt5o5BRqgAZQbCQ734rPiRKc2h31F8OgHNkvcbkXOrot9-w@mail.gmail.com>
	<1371304300.53915.YahooMailMobile@web193404.mail.sg3.yahoo.com>
Message-ID: <CAH6Pt5qJd0eRxxLK4=n6-STGaBZafEb3ZOvyQUP5fqb9FTgEUQ@mail.gmail.com>

Hi,

On Sat, Jun 15, 2013 at 2:51 PM, Sudheer Joseph
<sudheer.joseph at yahoo.com> wrote:
>
> Thank you very much for this tip.
> Is there a typical way to save masked and the rest separately?. Not much familiar with array handling in numpy.

I don't use masked array myself, but it looks like it would be something like:

eof1_unmasked = np.array(eof1)
eof1_mask = eof1.mask

then you could save those two.  Maybe a more maskey person could comment?

Cheers,

Matthew


From warren.weckesser at gmail.com  Sat Jun 15 11:43:23 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 15 Jun 2013 11:43:23 -0400
Subject: [Numpy-discussion] Seg. fault when running tests
Message-ID: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>

I'm getting a seg. fault in master when I run the tests.  I'm on Ubuntu
12.04 64 bit, with Python 3.3.2 (64 bits):

$ python3 -c "import numpy as np; np.test('full')"
Running unit tests for numpy
NumPy version 1.8.0.dev-fa5bc1c
NumPy is installed in
/home/warren/local_py332/lib/python3.3/site-packages/numpy
Python version 3.3.2 (default, Jun 14 2013, 12:12:22) [GCC 4.6.3]
nose version 1.3.0
.............................S.........................................................................................................................................................S.......................................................................................................................................................S..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............................................................................................................................................................................................SSS.................................................................................................K.....................................................................................................................................................................................Segmentation
fault

The seg. fault is occurring in ma/tests/test_mrecords.py:

$ nosetests test_mrecords.py
.Segmentation fault

More info:

$ python3
Python 3.3.2 (default, Jun 14 2013, 12:12:22)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.show_config()
atlas_threads_info:
    library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
    include_dirs = ['/usr/include/atlas']
    language = f77
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
atlas_blas_threads_info:
    library_dirs = ['/usr/lib/atlas-base']
    include_dirs = ['/usr/include/atlas']
    language = c
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
mkl_info:
  NOT AVAILABLE
lapack_opt_info:
    library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
    include_dirs = ['/usr/include/atlas']
    language = f77
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
blas_opt_info:
    library_dirs = ['/usr/lib/atlas-base']
    include_dirs = ['/usr/include/atlas']
    language = c
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
>>>

gdb:

$ gdb python3
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /home/warren/local_py332/bin/python3...done.
(gdb) run test_mrecords.py
Starting program: /home/warren/local_py332/bin/python3 test_mrecords.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
.
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff5f080a4 in _aligned_strided_to_contig_size8_srcstride0
(dst=<optimized out>,
    dst_stride=<optimized out>, src=<optimized out>,
__NPY_UNUSED_TAGGEDsrc_stride=<optimized out>,
    N=<optimized out>, __NPY_UNUSED_TAGGEDsrc_itemsize=<optimized out>,
__NPY_UNUSED_TAGGEDdata=0x0)
    at numpy/core/src/multiarray/lowlevel_strided_loops.c.src:190
190            *((@type@ *)dst) = temp;
(gdb)


Warren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/71216cd6/attachment.html>

From warren.weckesser at gmail.com  Sat Jun 15 11:50:43 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 15 Jun 2013 11:50:43 -0400
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
Message-ID: <CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>

On Sat, Jun 15, 2013 at 11:43 AM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

> I'm getting a seg. fault in master when I run the tests.  I'm on Ubuntu
> 12.04 64 bit, with Python 3.3.2 (64 bits):
>
> $ python3 -c "import numpy as np; np.test('full')"
> Running unit tests for numpy
> NumPy version 1.8.0.dev-fa5bc1c
> NumPy is installed in
> /home/warren/local_py332/lib/python3.3/site-packages/numpy
> Python version 3.3.2 (default, Jun 14 2013, 12:12:22) [GCC 4.6.3]
> nose version 1.3.0
> .............................S.........................................................................................................................................................S.......................................................................................................................................................S..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............................................................................................................................................................................................SSS.................................................................................................K.....................................................................................................................................................................................Segmentation
> fault
>
> The seg. fault is occurring in ma/tests/test_mrecords.py:
>
> $ nosetests test_mrecords.py
> .Segmentation fault
>
> More info:
>
> $ python3
> Python 3.3.2 (default, Jun 14 2013, 12:12:22)
> [GCC 4.6.3] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import numpy as np
> >>> np.show_config()
> atlas_threads_info:
>     library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
>     include_dirs = ['/usr/include/atlas']
>     language = f77
>     libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>     define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> atlas_blas_threads_info:
>     library_dirs = ['/usr/lib/atlas-base']
>     include_dirs = ['/usr/include/atlas']
>     language = c
>     libraries = ['ptf77blas', 'ptcblas', 'atlas']
>     define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> mkl_info:
>   NOT AVAILABLE
> lapack_opt_info:
>     library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
>     include_dirs = ['/usr/include/atlas']
>     language = f77
>     libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>     define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> blas_opt_info:
>     library_dirs = ['/usr/lib/atlas-base']
>     include_dirs = ['/usr/include/atlas']
>     language = c
>     libraries = ['ptf77blas', 'ptcblas', 'atlas']
>     define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> lapack_mkl_info:
>   NOT AVAILABLE
> blas_mkl_info:
>   NOT AVAILABLE
> >>>
>
> gdb:
>
> $ gdb python3
> GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
> Copyright (C) 2012 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> For bug reporting instructions, please see:
> <http://bugs.launchpad.net/gdb-linaro/>...
> Reading symbols from /home/warren/local_py332/bin/python3...done.
> (gdb) run test_mrecords.py
> Starting program: /home/warren/local_py332/bin/python3 test_mrecords.py
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> .
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff5f080a4 in _aligned_strided_to_contig_size8_srcstride0
> (dst=<optimized out>,
>     dst_stride=<optimized out>, src=<optimized out>,
> __NPY_UNUSED_TAGGEDsrc_stride=<optimized out>,
>     N=<optimized out>, __NPY_UNUSED_TAGGEDsrc_itemsize=<optimized out>,
> __NPY_UNUSED_TAGGEDdata=0x0)
>     at numpy/core/src/multiarray/lowlevel_strided_loops.c.src:190
> 190            *((@type@ *)dst) = temp;
> (gdb)
>
>

Log of the backtrace is attached.

Warren




>
> Warren
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/4d702dcc/attachment.html>
-------------- next part --------------
#0  0x00007ffff5f080a4 in _aligned_strided_to_contig_size8_srcstride0 (dst=<optimized out>, 
    dst_stride=<optimized out>, src=<optimized out>, __NPY_UNUSED_TAGGEDsrc_stride=<optimized out>, 
    N=<optimized out>, __NPY_UNUSED_TAGGEDsrc_itemsize=<optimized out>, __NPY_UNUSED_TAGGEDdata=0x0)
    at numpy/core/src/multiarray/lowlevel_strided_loops.c.src:190
#1  0x00007ffff5eb328d in raw_array_assign_scalar (ndim=1, shape=0xe89580, dst_dtype=<optimized out>, 
    dst_data=0xcbdd94 "\377\177", dst_strides=<optimized out>, src_dtype=<optimized out>, src_data=0xe87e40 "")
    at numpy/core/src/multiarray/array_assign_scalar.c:80
#2  0x00007ffff5eb3aaf in PyArray_AssignRawScalar (dst=0xe84280, src_dtype=0x7ffff61a4fa0, src_data=0xe87e40 "", 
    wheremask=0x0, casting=<optimized out>) at numpy/core/src/multiarray/array_assign_scalar.c:252
#3  0x00007ffff5eb4a23 in PyArray_AssignArray (dst=0xe84280, src=0xa9e860, wheremask=0x0, 
    casting=NPY_UNSAFE_CASTING) at numpy/core/src/multiarray/array_assign_array.c:231
#4  0x00007ffff5e82ed4 in PyArray_CopyObject (dest=0xe84280, src_object=0x7fffefb200d8)
    at numpy/core/src/multiarray/arrayobject.c:322
#5  0x00007ffff5e9901c in VOID_setitem (op=0x7fffefb200d8, ip=0xcbdd94 "\377\177", ap=0xe84170)
    at numpy/core/src/multiarray/arraytypes.c.src:726
#6  0x00007ffff5e98ed9 in VOID_setitem (op=0x7fffef644550, ip=0xcbdd90 "", ap=0xe84170)
    at numpy/core/src/multiarray/arraytypes.c.src:692
#7  0x00007ffff5ec6040 in PyArray_FromAny (op=0x7fffef644550, newtype=0x7fffef650298, min_depth=<optimized out>, 
    max_depth=<optimized out>, flags=<optimized out>, context=<optimized out>)
    at numpy/core/src/multiarray/ctors.c:1679
#8  0x00007ffff5ec6126 in PyArray_CheckFromAny (op=0x7fffef644550, descr=0x7fffef650298, 
    min_depth=<optimized out>, max_depth=<optimized out>, requires=112, context=<optimized out>)
    at numpy/core/src/multiarray/ctors.c:1772
#9  0x00007ffff5f28a91 in _array_fromobject (__NPY_UNUSED_TAGGEDignored=<optimized out>, args=<optimized out>, 
    kws=<optimized out>) at numpy/core/src/multiarray/multiarraymodule.c:1663
#10 0x000000000048159d in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffa4d0, 
    func=0x7ffff669fdd0) at Python/ceval.c:4280
#11 call_function (oparg=<optimized out>, pp_stack=0x7fffffffa4d0) at Python/ceval.c:4086
#12 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#13 0x0000000000481773 in fast_function (nk=<optimized out>, na=2, n=<optimized out>, pp_stack=0x7fffffffa640, 
    func=0x7ffff2086a70) at Python/ceval.c:4151
#14 call_function (oparg=<optimized out>, pp_stack=0x7fffffffa640) at Python/ceval.c:4084
#15 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#16 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7fffef640bd0, kwcount=0, defs=0x7ffff20897e8, 
    defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#17 0x0000000000481636 in fast_function (nk=<optimized out>, na=2, n=<optimized out>, pp_stack=0x7fffffffa860, 
    func=0x7ffff2096200) at Python/ceval.c:4161
#18 call_function (oparg=<optimized out>, pp_stack=0x7fffffffa860) at Python/ceval.c:4084
#19 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#20 0x0000000000481773 in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fffffffa9d0, 
    func=0x7fffefb22560) at Python/ceval.c:4151
#21 call_function (oparg=<optimized out>, pp_stack=0x7fffffffa9d0) at Python/ceval.c:4084
#22 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#23 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=3, kws=0xe892f8, kwcount=1, defs=0x7ffff4a39d68, 
    defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#24 0x0000000000481636 in fast_function (nk=<optimized out>, na=3, n=<optimized out>, pp_stack=0x7fffffffabf0, 
    func=0x7ffff4a46050) at Python/ceval.c:4161
#25 call_function (oparg=<optimized out>, pp_stack=0x7fffffffabf0) at Python/ceval.c:4084
#26 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#27 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7ffff7f92068, kwcount=0, defs=0x7ffff4a3c6a8, 
    defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#28 0x000000000053567c in function_call (func=0x7ffff4a460e0, arg=0x7fffefb06290, kw=0x7fffef647a70)
    at Objects/funcobject.c:633
#29 0x000000000050ac9a in PyObject_Call (func=0x7ffff4a460e0, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#30 0x000000000047dab5 in ext_do_call (nk=-273653104, na=1, flags=<optimized out>, pp_stack=0x7fffffffaee0, 
    func=0x7ffff4a460e0) at Python/ceval.c:4378
#31 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2720
#32 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, 
    kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#33 0x0000000000535575 in function_call (func=0x7ffff4a46200, arg=0x7fffefb053b0, kw=0x0)
    at Objects/funcobject.c:633
#34 0x000000000050ac9a in PyObject_Call (func=0x7ffff4a46200, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#35 0x00000000005213d4 in method_call (func=0x7ffff4a46200, arg=0x7fffefb053b0, kw=0x0)
    at Objects/classobject.c:323
#36 0x000000000050ac9a in PyObject_Call (func=0x7fffefb0fcb0, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#37 0x0000000000431481 in slot_tp_call (self=0x7fffefb109d0, args=0x7fffefb1de10, kwds=0x0)
    at Objects/typeobject.c:5365
#38 0x000000000050ac9a in PyObject_Call (func=0x7fffefb109d0, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#39 0x000000000047fae0 in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffb2c0, 
    func=0x7fffefb109d0) at Python/ceval.c:4283
#40 call_function (oparg=<optimized out>, pp_stack=0x7fffffffb2c0) at Python/ceval.c:4086
#41 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#42 0x0000000000481773 in fast_function (nk=<optimized out>, na=2, n=<optimized out>, pp_stack=0x7fffffffb430, 
    func=0x7ffff179ab00) at Python/ceval.c:4151
#43 call_function (oparg=<optimized out>, pp_stack=0x7fffffffb430) at Python/ceval.c:4084
#44 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#45 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7ffff7f92068, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#46 0x000000000053567c in function_call (func=0x7ffff179aa70, arg=0x7fffef645d88, kw=0x7fffef6479e0)
    at Objects/funcobject.c:633
#47 0x000000000050ac9a in PyObject_Call (func=0x7ffff179aa70, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#48 0x000000000047dab5 in ext_do_call (nk=-278635128, na=1, flags=<optimized out>, pp_stack=0x7fffffffb720, 
    func=0x7ffff179aa70) at Python/ceval.c:4378
#49 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2720
#50 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, 
    kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#51 0x0000000000535575 in function_call (func=0x7ffff179a560, arg=0x7fffef645d40, kw=0x0)
    at Objects/funcobject.c:633
#52 0x000000000050ac9a in PyObject_Call (func=0x7ffff179a560, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#53 0x00000000005213d4 in method_call (func=0x7ffff179a560, arg=0x7fffef645d40, kw=0x0)
    at Objects/classobject.c:323
#54 0x000000000050ac9a in PyObject_Call (func=0x7fffefb0fc68, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#55 0x0000000000431481 in slot_tp_call (self=0x7fffefb10990, args=0x7fffef6460d0, kwds=0x0)
    at Objects/typeobject.c:5365
#56 0x000000000050ac9a in PyObject_Call (func=0x7fffefb10990, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#57 0x000000000047fae0 in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffbb00, 
    func=0x7fffefb10990) at Python/ceval.c:4283
#58 call_function (oparg=<optimized out>, pp_stack=0x7fffffffbb00) at Python/ceval.c:4086
#59 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#60 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7ffff7f92068, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#61 0x000000000053567c in function_call (func=0x7ffff112b830, arg=0x7ffff67b2128, kw=0x7fffef647908)
    at Objects/funcobject.c:633
#62 0x000000000050ac9a in PyObject_Call (func=0x7ffff112b830, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#63 0x000000000047dab5 in ext_do_call (nk=-159702744, na=1, flags=<optimized out>, pp_stack=0x7fffffffbdf0, 
    func=0x7ffff112b830) at Python/ceval.c:4378
#64 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2720
#65 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, 
    kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#66 0x0000000000535575 in function_call (func=0x7ffff112b680, arg=0x7fffef645dd0, kw=0x0)
    at Objects/funcobject.c:633
#67 0x000000000050ac9a in PyObject_Call (func=0x7ffff112b680, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#68 0x00000000005213d4 in method_call (func=0x7ffff112b680, arg=0x7fffef645dd0, kw=0x0)
    at Objects/classobject.c:323
#69 0x000000000050ac9a in PyObject_Call (func=0x7fffefb06cf8, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#70 0x0000000000431481 in slot_tp_call (self=0x7fffefb1da90, args=0x7fffef646110, kwds=0x0)
    at Objects/typeobject.c:5365
#71 0x000000000050ac9a in PyObject_Call (func=0x7fffefb1da90, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#72 0x000000000047fae0 in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffc1d0, 
    func=0x7fffefb1da90) at Python/ceval.c:4283
#73 call_function (oparg=<optimized out>, pp_stack=0x7fffffffc1d0) at Python/ceval.c:4086
#74 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#75 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7ffff7f92068, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#76 0x000000000053567c in function_call (func=0x7ffff112b830, arg=0x7fffefb05128, kw=0x7fffef6478c0)
    at Objects/funcobject.c:633
#77 0x000000000050ac9a in PyObject_Call (func=0x7ffff112b830, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#78 0x000000000047dab5 in ext_do_call (nk=-273657560, na=1, flags=<optimized out>, pp_stack=0x7fffffffc4c0, 
    func=0x7ffff112b830) at Python/ceval.c:4378
#79 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2720
#80 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, 
    kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#81 0x0000000000535575 in function_call (func=0x7ffff112b680, arg=0x7fffefb055a8, kw=0x0)
    at Objects/funcobject.c:633
#82 0x000000000050ac9a in PyObject_Call (func=0x7ffff112b680, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#83 0x00000000005213d4 in method_call (func=0x7ffff112b680, arg=0x7fffefb055a8, kw=0x0)
    at Objects/classobject.c:323
#84 0x000000000050ac9a in PyObject_Call (func=0x7ffff11412d8, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#85 0x0000000000431481 in slot_tp_call (self=0x7fffef6464d0, args=0x7fffef646250, kwds=0x0)
    at Objects/typeobject.c:5365
#86 0x000000000050ac9a in PyObject_Call (func=0x7fffef6464d0, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#87 0x000000000047fae0 in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffc8a0, 
    func=0x7fffef6464d0) at Python/ceval.c:4283
#88 call_function (oparg=<optimized out>, pp_stack=0x7fffffffc8a0) at Python/ceval.c:4086
#89 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#90 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x7ffff7f92068, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#91 0x000000000053567c in function_call (func=0x7ffff112b830, arg=0x7ffff20b0ef0, kw=0x7fffef6477a0)
    at Objects/funcobject.c:633
#92 0x000000000050ac9a in PyObject_Call (func=0x7ffff112b830, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#93 0x000000000047dab5 in ext_do_call (nk=-234156304, na=1, flags=<optimized out>, pp_stack=0x7fffffffcb90, 
    func=0x7ffff112b830) at Python/ceval.c:4378
#94 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2720
#95 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, 
    kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#96 0x0000000000535575 in function_call (func=0x7ffff112b680, arg=0x7fffefb053f8, kw=0x0)
    at Objects/funcobject.c:633
#97 0x000000000050ac9a in PyObject_Call (func=0x7ffff112b680, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#98 0x00000000005213d4 in method_call (func=0x7ffff112b680, arg=0x7fffefb053f8, kw=0x0)
    at Objects/classobject.c:323
#99 0x000000000050ac9a in PyObject_Call (func=0x7fffefd46440, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#100 0x0000000000431481 in slot_tp_call (self=0x7fffefb104d0, args=0x7fffefb1dc90, kwds=0x0)
    at Objects/typeobject.c:5365
#101 0x000000000050ac9a in PyObject_Call (func=0x7fffefb104d0, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#102 0x000000000047fae0 in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffcf70, 
    func=0x7fffefb104d0) at Python/ceval.c:4283
#103 call_function (oparg=<optimized out>, pp_stack=0x7fffffffcf70) at Python/ceval.c:4086
#104 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#105 0x0000000000481773 in fast_function (nk=<optimized out>, na=2, n=<optimized out>, pp_stack=0x7fffffffd0e0, 
    func=0x7ffff112d950) at Python/ceval.c:4151
#106 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd0e0) at Python/ceval.c:4084
#107 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#108 0x0000000000481773 in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fffffffd250, 
    func=0x7ffff112dcb0) at Python/ceval.c:4151
#109 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd250) at Python/ceval.c:4084
#110 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#111 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=1, kws=0x7ffff112f3f8, kwcount=6, defs=0x7ffff4a17218, 
    defcount=11, kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#112 0x000000000053567c in function_call (func=0x7ffff4a17290, arg=0x7ffff677ad50, kw=0x7ffff1137368)
    at Objects/funcobject.c:633
#113 0x000000000050ac9a in PyObject_Call (func=0x7ffff4a17290, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#114 0x000000000047dab5 in ext_do_call (nk=-159929008, na=1, flags=<optimized out>, pp_stack=0x7fffffffd540, 
    func=0x7ffff4a17290) at Python/ceval.c:4378
#115 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2720
#116 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=1, kws=0x7ffff1135170, kwcount=2, defs=0x7ffff112da88, 
    defcount=11, kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#117 0x000000000053567c in function_call (func=0x7ffff112d9e0, arg=0x7ffff660fad0, kw=0x7ffff7ee9758)
    at Objects/funcobject.c:633
#118 0x000000000050ac9a in PyObject_Call (func=0x7ffff112d9e0, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#119 0x00000000005213d4 in method_call (func=0x7ffff112d9e0, arg=0x7ffff660fad0, kw=0x7ffff7ee9758)
    at Objects/classobject.c:323
#120 0x000000000050ac9a in PyObject_Call (func=0x7ffff11374d0, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#121 0x0000000000431189 in slot_tp_init (self=0x7ffff678a8d0, args=0x7ffff7f92050, kwds=0x7ffff7ee9758)
    at Objects/typeobject.c:5580
#122 0x000000000042cd87 in type_call (type=<optimized out>, args=0x7ffff7f92050, kwds=0x7ffff7ee9758)
    at Objects/typeobject.c:754
#123 0x000000000050ac9a in PyObject_Call (func=0xd55190, arg=<optimized out>, kw=<optimized out>)
    at Objects/abstract.c:2084
#124 0x000000000047dab5 in ext_do_call (nk=-134668208, na=0, flags=<optimized out>, pp_stack=0x7fffffffd940, 
    func=0xd55190) at Python/ceval.c:4378
#125 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2720
#126 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=0, kws=0x8b1030, kwcount=1, defs=0x0, defcount=0, 
    kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#127 0x0000000000481636 in fast_function (nk=<optimized out>, na=0, n=<optimized out>, pp_stack=0x7fffffffdb60, 
    func=0x7ffff20bf8c0) at Python/ceval.c:4161
#128 call_function (oparg=<optimized out>, pp_stack=0x7fffffffdb60) at Python/ceval.c:4084
#129 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#130 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=0, kws=0x8cca50, kwcount=0, defs=0x7ffff4a1e528, 
    defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#131 0x0000000000481636 in fast_function (nk=<optimized out>, na=0, n=<optimized out>, pp_stack=0x7fffffffdd80, 
    func=0x7ffff49b4290) at Python/ceval.c:4161
#132 call_function (oparg=<optimized out>, pp_stack=0x7fffffffdd80) at Python/ceval.c:4084
#133 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#134 0x0000000000479d54 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, 
    kwdefs=0x0, closure=0x0) at Python/ceval.c:3433
#135 0x0000000000479e1b in PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>)
    at Python/ceval.c:771
#136 0x00000000004a573d in run_mod (arena=0x91e990, flags=0x7fffffffe0b0, locals=0x7ffff7f136c8, 
    globals=0x7ffff7f136c8, filename=<optimized out>, mod=0x98fab0) at Python/pythonrun.c:1981
#137 PyRun_FileExFlags (fp=0x908dc0, filename=<optimized out>, start=<optimized out>, globals=0x7ffff7f136c8, 
    locals=0x7ffff7f136c8, closeit=1, flags=0x7fffffffe0b0) at Python/pythonrun.c:1937
#138 0x00000000004a67ad in PyRun_SimpleFileExFlags (fp=0x908dc0, filename=<optimized out>, closeit=1, 
    flags=0x7fffffffe0b0) at Python/pythonrun.c:1447
#139 0x00000000004bd332 in run_file (p_cf=0x7fffffffe0b0, filename=0x85b2d0 L"test_mrecords.py", fp=0x908dc0)
    at Modules/main.c:306
#140 Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:743
#141 0x00000000004195ee in main (argc=2, argv=0x7fffffffe228) at ./Modules/python.c:58

From pgmdevlist at gmail.com  Sat Jun 15 12:06:51 2013
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sat, 15 Jun 2013 18:06:51 +0200
Subject: [Numpy-discussion] saving 3d array
In-Reply-To: <CAH6Pt5qJd0eRxxLK4=n6-STGaBZafEb3ZOvyQUP5fqb9FTgEUQ@mail.gmail.com>
References: <CAH6Pt5o5BRqgAZQbCQ734rPiRKc2h31F8OgHNkvcbkXOrot9-w@mail.gmail.com>
	<1371304300.53915.YahooMailMobile@web193404.mail.sg3.yahoo.com>
	<CAH6Pt5qJd0eRxxLK4=n6-STGaBZafEb3ZOvyQUP5fqb9FTgEUQ@mail.gmail.com>
Message-ID: <4A26834D-E401-4E54-88D3-FBABA4957196@gmail.com>


On Jun 15, 2013, at 17:35 , Matthew Brett <matthew.brett at gmail.com> wrote:

> Hi,
> 
> On Sat, Jun 15, 2013 at 2:51 PM, Sudheer Joseph
> <sudheer.joseph at yahoo.com> wrote:
>> 
>> Thank you very much for this tip.
>> Is there a typical way to save masked and the rest separately?. Not much familiar with array handling in numpy.
> 
> I don't use masked array myself, but it looks like it would be something like:
> 
> eof1_unmasked = np.array(eof1)
> eof1_mask = eof1.mask
> 
> then you could save those two.  Maybe a more maskey person could comment?

Instead of `eof1_unmasked=np.array(eof1)`, you could do `eof1_unmasked = eof1.data`. The '.data' attribute points to  a view of the masked array as a simple ndarray.

You may also wanna try `eof1.torecords()` that will return a structured array with dtype `[('_data',type_of_eof1),('_mask', bool)]`. 

From sudheer.joseph at yahoo.com  Sat Jun 15 12:23:09 2013
From: sudheer.joseph at yahoo.com (Sudheer Joseph)
Date: Sun, 16 Jun 2013 00:23:09 +0800 (SGT)
Subject: [Numpy-discussion] saving 3d array
In-Reply-To: <4A26834D-E401-4E54-88D3-FBABA4957196@gmail.com>
References: <CAH6Pt5o5BRqgAZQbCQ734rPiRKc2h31F8OgHNkvcbkXOrot9-w@mail.gmail.com>
	<1371304300.53915.YahooMailMobile@web193404.mail.sg3.yahoo.com>
	<CAH6Pt5qJd0eRxxLK4=n6-STGaBZafEb3ZOvyQUP5fqb9FTgEUQ@mail.gmail.com>
	<4A26834D-E401-4E54-88D3-FBABA4957196@gmail.com>
Message-ID: <1371313389.34575.YahooMailNeo@web193406.mail.sg3.yahoo.com>


Thanks a lot,
????????????? Both suggestions helped me and I could save it as below.
np.savez('eof_output',eof=eof1.data,pcs=pc1,lags=lags,lons=lons,lats=lats,x=x,y=y,clevs=clevs,eof_mask=eof1.mask)

***************************************************************
Sudheer Joseph 
Indian National Centre for Ocean Information Services
Ministry of Earth Sciences, Govt. of India
POST BOX NO: 21, IDA Jeedeemetla P.O.
Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55
Tel:+91-40-23886047(O),Fax:+91-40-23895011(O),
Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile)
E-mail:sjo.India at gmail.com;sudheer.joseph at yahoo.com
Web- http://oppamthadathil.tripod.com
***************************************************************


>________________________________
> From: Pierre GM <pgmdevlist at gmail.com>
>To: Discussion of Numerical Python <numpy-discussion at scipy.org> 
>Sent: Saturday, 15 June 2013 9:36 PM
>Subject: Re: [Numpy-discussion] saving 3d array
> 
>
>
>On Jun 15, 2013, at 17:35 , Matthew Brett <matthew.brett at gmail.com> wrote:
>
>> Hi,
>> 
>> On Sat, Jun 15, 2013 at 2:51 PM, Sudheer Joseph
>> <sudheer.joseph at yahoo.com> wrote:
>>> 
>>> Thank you very much for this tip.
>>> Is there a typical way to save masked and the rest separately?. Not much familiar with array handling in numpy.
>> 
>> I don't use masked array myself, but it looks like it would be something like:
>> 
>> eof1_unmasked = np.array(eof1)
>> eof1_mask = eof1.mask
>> 
>> then you could save those two.? Maybe a more maskey person could comment?
>
>Instead of `eof1_unmasked=np.array(eof1)`, you could do `eof1_unmasked = eof1.data`. The '.data' attribute points to? a view of the masked array as a simple ndarray.
>
>You may also wanna try `eof1.torecords()` that will return a structured array with dtype `[('_data',type_of_eof1),('_mask', bool)]`. 
>_______________________________________________
>NumPy-Discussion mailing list
>NumPy-Discussion at scipy.org
>http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130616/e6c96a29/attachment.html>

From pav at iki.fi  Sat Jun 15 13:29:30 2013
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 15 Jun 2013 20:29:30 +0300
Subject: [Numpy-discussion] Dropping support for, Accelerate/veclib?
In-Reply-To: <CABL7CQiRZAS6Jef2cV5QSu8LTmHuAc3A2Vgi04smShacB7Ggbw@mail.gmail.com>
References: <51B75CC6.4090405@amolf.nl>
	<CALGmxEJZgGO20u+r4FigVVb2ep0jgdxmMm8jXhQZ-4LmbrhCfA@mail.gmail.com>
	<CABL7CQjtWf8-b6k=XH2xDarF=JmFMavy+=FvXbLR3k22brKtUQ@mail.gmail.com>
	<CALGmxEKv-By0KmrWSJ0Ura0g=E3rvwoXGjw7ttF7OeWddV9_5w@mail.gmail.com>
	<51B9AC32.4060701@gmail.com>
	<CABL7CQiRZAS6Jef2cV5QSu8LTmHuAc3A2Vgi04smShacB7Ggbw@mail.gmail.com>
Message-ID: <kpi89k$u1$1@ger.gmane.org>

FYI:

https://github.com/numpy/numpy/pull/3439

-- 
Pauli Virtanen



From efiring at hawaii.edu  Sat Jun 15 14:11:42 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Sat, 15 Jun 2013 08:11:42 -1000
Subject: [Numpy-discussion] saving 3d array
In-Reply-To: <4A26834D-E401-4E54-88D3-FBABA4957196@gmail.com>
References: <CAH6Pt5o5BRqgAZQbCQ734rPiRKc2h31F8OgHNkvcbkXOrot9-w@mail.gmail.com>
	<1371304300.53915.YahooMailMobile@web193404.mail.sg3.yahoo.com>
	<CAH6Pt5qJd0eRxxLK4=n6-STGaBZafEb3ZOvyQUP5fqb9FTgEUQ@mail.gmail.com>
	<4A26834D-E401-4E54-88D3-FBABA4957196@gmail.com>
Message-ID: <51BCAE5E.2080907@hawaii.edu>

On 2013/06/15 6:06 AM, Pierre GM wrote:
>
> On Jun 15, 2013, at 17:35 , Matthew Brett <matthew.brett at gmail.com> wrote:
>
>> Hi,
>>
>> On Sat, Jun 15, 2013 at 2:51 PM, Sudheer Joseph
>> <sudheer.joseph at yahoo.com> wrote:
>>>
>>> Thank you very much for this tip.
>>> Is there a typical way to save masked and the rest separately?. Not much familiar with array handling in numpy.
>>
>> I don't use masked array myself, but it looks like it would be something like:
>>
>> eof1_unmasked = np.array(eof1)
>> eof1_mask = eof1.mask
>>
>> then you could save those two.  Maybe a more maskey person could comment?
>
> Instead of `eof1_unmasked=np.array(eof1)`, you could do `eof1_unmasked = eof1.data`. The '.data' attribute points to  a view of the masked array as a simple ndarray.
>
> You may also wanna try `eof1.torecords()` that will return a structured array with dtype `[('_data',type_of_eof1),('_mask', bool)]`.

For automated saving and restoring, try this:

http://currents.soest.hawaii.edu/hgstage/pycurrents/file/686c2802a6c4/file/npzfile.py

Eric



From ben.root at ou.edu  Sat Jun 15 14:38:30 2013
From: ben.root at ou.edu (Benjamin Root)
Date: Sat, 15 Jun 2013 14:38:30 -0400
Subject: [Numpy-discussion] time to revisit NA/ma ideas
In-Reply-To: <51BB9B65.9090807@hawaii.edu>
References: <51BB9B65.9090807@hawaii.edu>
Message-ID: <CANNq6FmAZmDJm3m0WipMWMEE9JZqKCr2+u1HKTq707g0yc539g@mail.gmail.com>

On Fri, Jun 14, 2013 at 6:38 PM, Eric Firing <efiring at hawaii.edu> wrote:

> A nice summary of the discussions from a year ago is here:
>
> http://www.numpy.org/NA-overview.html
>
> It provides food for thought.
>
> Eric
>

Perhaps a BoF session should be put together for SciPy 2013, and possibly
even have a google hangout session for it to bring in interested parties to
the discussion?

Ben Root
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/9a7941c3/attachment.html>

From pgmdevlist at gmail.com  Sat Jun 15 15:04:01 2013
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sat, 15 Jun 2013 21:04:01 +0200
Subject: [Numpy-discussion] time to revisit NA/ma ideas
In-Reply-To: <CANNq6FmAZmDJm3m0WipMWMEE9JZqKCr2+u1HKTq707g0yc539g@mail.gmail.com>
References: <51BB9B65.9090807@hawaii.edu>
	<CANNq6FmAZmDJm3m0WipMWMEE9JZqKCr2+u1HKTq707g0yc539g@mail.gmail.com>
Message-ID: <6921CC95-F819-4F27-AB5A-3D95F5843027@gmail.com>


On Jun 15, 2013, at 20:38 , Benjamin Root <ben.root at ou.edu> wrote:

> 
> On Fri, Jun 14, 2013 at 6:38 PM, Eric Firing <efiring at hawaii.edu> wrote:
> A nice summary of the discussions from a year ago is here:
> 
> http://www.numpy.org/NA-overview.html
> 
> It provides food for thought.
> 
> Eric
> 
> Perhaps a BoF session should be put together for SciPy 2013, and possibly even have a google hangout session for it to bring in interested parties to the discussion?

Sounds like an Idea. I won't be able to make it to SciPy, and I doubt I'll be able to attend EuroSciPy, but I'd be grateful to be kept in the loop with a Google hangout session..
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/3ff6fd54/attachment.html>

From charlesr.harris at gmail.com  Sat Jun 15 15:12:51 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 15 Jun 2013 13:12:51 -0600
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
Message-ID: <CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>

On Sat, Jun 15, 2013 at 9:50 AM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

>
> On Sat, Jun 15, 2013 at 11:43 AM, Warren Weckesser <
> warren.weckesser at gmail.com> wrote:
>
>> I'm getting a seg. fault in master when I run the tests.  I'm on Ubuntu
>> 12.04 64 bit, with Python 3.3.2 (64 bits):
>>
>> $ python3 -c "import numpy as np; np.test('full')"
>> Running unit tests for numpy
>> NumPy version 1.8.0.dev-fa5bc1c
>> NumPy is installed in
>> /home/warren/local_py332/lib/python3.3/site-packages/numpy
>> Python version 3.3.2 (default, Jun 14 2013, 12:12:22) [GCC 4.6.3]
>> nose version 1.3.0
>> .............................S.........................................................................................................................................................S.......................................................................................................................................................S..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............................................................................................................................................................................................SSS.................................................................................................K.....................................................................................................................................................................................Segmentation
>> fault
>>
>> The seg. fault is occurring in ma/tests/test_mrecords.py:
>>
>> $ nosetests test_mrecords.py
>> .Segmentation fault
>>
>> More info:
>>
>> $ python3
>> Python 3.3.2 (default, Jun 14 2013, 12:12:22)
>> [GCC 4.6.3] on linux
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import numpy as np
>> >>> np.show_config()
>> atlas_threads_info:
>>     library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
>>     include_dirs = ['/usr/include/atlas']
>>     language = f77
>>     libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>>     define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> atlas_blas_threads_info:
>>     library_dirs = ['/usr/lib/atlas-base']
>>     include_dirs = ['/usr/include/atlas']
>>     language = c
>>     libraries = ['ptf77blas', 'ptcblas', 'atlas']
>>     define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> mkl_info:
>>   NOT AVAILABLE
>> lapack_opt_info:
>>     library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
>>     include_dirs = ['/usr/include/atlas']
>>     language = f77
>>     libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>>     define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> blas_opt_info:
>>     library_dirs = ['/usr/lib/atlas-base']
>>     include_dirs = ['/usr/include/atlas']
>>     language = c
>>     libraries = ['ptf77blas', 'ptcblas', 'atlas']
>>     define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> lapack_mkl_info:
>>   NOT AVAILABLE
>> blas_mkl_info:
>>   NOT AVAILABLE
>> >>>
>>
>> gdb:
>>
>> $ gdb python3
>> GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
>> Copyright (C) 2012 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <
>> http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://bugs.launchpad.net/gdb-linaro/>...
>> Reading symbols from /home/warren/local_py332/bin/python3...done.
>> (gdb) run test_mrecords.py
>> Starting program: /home/warren/local_py332/bin/python3 test_mrecords.py
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> .
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00007ffff5f080a4 in _aligned_strided_to_contig_size8_srcstride0
>> (dst=<optimized out>,
>>     dst_stride=<optimized out>, src=<optimized out>,
>> __NPY_UNUSED_TAGGEDsrc_stride=<optimized out>,
>>     N=<optimized out>, __NPY_UNUSED_TAGGEDsrc_itemsize=<optimized out>,
>> __NPY_UNUSED_TAGGEDdata=0x0)
>>     at numpy/core/src/multiarray/lowlevel_strided_loops.c.src:190
>> 190            *((@type@ *)dst) = temp;
>> (gdb)
>>
>>
>
>
Is this repeatable? Just asking as I don't see that here with gcc 4.7.2.
Looks like Julian's changes to lowlevel_strided_loops may be responsible.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/e0b28148/attachment.html>

From jtaylor.debian at googlemail.com  Sat Jun 15 15:15:53 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Sat, 15 Jun 2013 21:15:53 +0200
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
Message-ID: <51BCBD69.3080800@googlemail.com>

On 15.06.2013 21:12, Charles R Harris wrote:
> 
> 
> On Sat, Jun 15, 2013 at 9:50 AM, Warren Weckesser
> <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>> wrote:
> 
> 
>     On Sat, Jun 15, 2013 at 11:43 AM, Warren Weckesser
>     <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>> wrote:
> 
>         I'm getting a seg. fault in master when I run the tests.  I'm on
>         Ubuntu 12.04 64 bit, with Python 3.3.2 (64 bits):
> 
>         $ python3 -c "import numpy as np; np.test('full')"
>         Running unit tests for numpy
>         NumPy version 1.8.0.dev-fa5bc1c
>         NumPy is installed in
>         /home/warren/local_py332/lib/python3.3/site-packages/numpy
>         Python version 3.3.2 (default, Jun 14 2013, 12:12:22) [GCC 4.6.3]
>         nose version 1.3.0
>         .............................S.........................................................................................................................................................S.......................................................................................................................................................S.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............................................................................................................................................................................................SSS..........
.......................................................................................K.....................................................................................................................................................................................Segmentation
>         fault
> 
>         The seg. fault is occurring in ma/tests/test_mrecords.py:
> 
>         $ nosetests test_mrecords.py
>         .Segmentation fault
> 
>         More info:
> 
>         $ python3
>         Python 3.3.2 (default, Jun 14 2013, 12:12:22)
>         [GCC 4.6.3] on linux
>         Type "help", "copyright", "credits" or "license" for more
>         information.
>         >>> import numpy as np
>         >>> np.show_config()
>         atlas_threads_info:
>             library_dirs = ['/usr/lib/atlas-base/atlas',
>         '/usr/lib/atlas-base']
>             include_dirs = ['/usr/include/atlas']
>             language = f77
>             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>         atlas_blas_threads_info:
>             library_dirs = ['/usr/lib/atlas-base']
>             include_dirs = ['/usr/include/atlas']
>             language = c
>             libraries = ['ptf77blas', 'ptcblas', 'atlas']
>             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>         mkl_info:
>           NOT AVAILABLE
>         lapack_opt_info:
>             library_dirs = ['/usr/lib/atlas-base/atlas',
>         '/usr/lib/atlas-base']
>             include_dirs = ['/usr/include/atlas']
>             language = f77
>             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>         blas_opt_info:
>             library_dirs = ['/usr/lib/atlas-base']
>             include_dirs = ['/usr/include/atlas']
>             language = c
>             libraries = ['ptf77blas', 'ptcblas', 'atlas']
>             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>         lapack_mkl_info:
>           NOT AVAILABLE
>         blas_mkl_info:
>           NOT AVAILABLE
>         >>> 
> 
>         gdb:
> 
>         $ gdb python3
>         GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
>         Copyright (C) 2012 Free Software Foundation, Inc.
>         License GPLv3+: GNU GPL version 3 or later
>         <http://gnu.org/licenses/gpl.html>
>         This is free software: you are free to change and redistribute it.
>         There is NO WARRANTY, to the extent permitted by law.  Type
>         "show copying"
>         and "show warranty" for details.
>         This GDB was configured as "x86_64-linux-gnu".
>         For bug reporting instructions, please see:
>         <http://bugs.launchpad.net/gdb-linaro/>...
>         Reading symbols from /home/warren/local_py332/bin/python3...done.
>         (gdb) run test_mrecords.py
>         Starting program: /home/warren/local_py332/bin/python3
>         test_mrecords.py
>         [Thread debugging using libthread_db enabled]
>         Using host libthread_db library
>         "/lib/x86_64-linux-gnu/libthread_db.so.1".
>         .
>         Program received signal SIGSEGV, Segmentation fault.
>         0x00007ffff5f080a4 in
>         _aligned_strided_to_contig_size8_srcstride0 (dst=<optimized out>,
>             dst_stride=<optimized out>, src=<optimized out>,
>         __NPY_UNUSED_TAGGEDsrc_stride=<optimized out>,
>             N=<optimized out>,
>         __NPY_UNUSED_TAGGEDsrc_itemsize=<optimized out>,
>         __NPY_UNUSED_TAGGEDdata=0x0)
>             at numpy/core/src/multiarray/lowlevel_strided_loops.c.src:190
>         190            *((@type@ *)dst) = temp;
>         (gdb)
> 
> 
> 
> 
> Is this repeatable? Just asking as I don't see that here with gcc 4.7.2.
> Looks like Julian's changes to lowlevel_strided_loops may be responsible.
> 
> Chuck
> 

I can't reproduce it on ubuntu 12.04 with python 3.2 (don't have 3.3.2)
the changes made are very minimal, from looking at them again I don't
see how they could cause this (especially on a x86 platform).

@warren, can you please bisect the commit causing this?



From charlesr.harris at gmail.com  Sat Jun 15 15:26:27 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 15 Jun 2013 13:26:27 -0600
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <51BCBD69.3080800@googlemail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
	<51BCBD69.3080800@googlemail.com>
Message-ID: <CAB6mnx+GJTKUBfvvdKXBOWB2E5=1tH=eHLypd1iM3s9MarrRhA@mail.gmail.com>

On Sat, Jun 15, 2013 at 1:15 PM, Julian Taylor <
jtaylor.debian at googlemail.com> wrote:

> On 15.06.2013 21:12, Charles R Harris wrote:
> >
> >
> > On Sat, Jun 15, 2013 at 9:50 AM, Warren Weckesser
> > <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>> wrote:
> >
> >
> >     On Sat, Jun 15, 2013 at 11:43 AM, Warren Weckesser
> >     <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>>
> wrote:
> >
> >         I'm getting a seg. fault in master when I run the tests.  I'm on
> >         Ubuntu 12.04 64 bit, with Python 3.3.2 (64 bits):
> >
> >         $ python3 -c "import numpy as np; np.test('full')"
> >         Running unit tests for numpy
> >         NumPy version 1.8.0.dev-fa5bc1c
> >         NumPy is installed in
> >         /home/warren/local_py332/lib/python3.3/site-packages/numpy
> >         Python version 3.3.2 (default, Jun 14 2013, 12:12:22) [GCC 4.6.3]
> >         nose version 1.3.0
> >
> .............................S.........................................................................................................................................................S.......................................................................................................................................................S....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
>  .........
>
> .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............................................................................................................................................................................................SSS.
>  .........
>
> .......................................................................................K.....................................................................................................................................................................................Segmentation
> >         fault
> >
> >         The seg. fault is occurring in ma/tests/test_mrecords.py:
> >
> >         $ nosetests test_mrecords.py
> >         .Segmentation fault
> >
> >         More info:
> >
> >         $ python3
> >         Python 3.3.2 (default, Jun 14 2013, 12:12:22)
> >         [GCC 4.6.3] on linux
> >         Type "help", "copyright", "credits" or "license" for more
> >         information.
> >         >>> import numpy as np
> >         >>> np.show_config()
> >         atlas_threads_info:
> >             library_dirs = ['/usr/lib/atlas-base/atlas',
> >         '/usr/lib/atlas-base']
> >             include_dirs = ['/usr/include/atlas']
> >             language = f77
> >             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> >         atlas_blas_threads_info:
> >             library_dirs = ['/usr/lib/atlas-base']
> >             include_dirs = ['/usr/include/atlas']
> >             language = c
> >             libraries = ['ptf77blas', 'ptcblas', 'atlas']
> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> >         mkl_info:
> >           NOT AVAILABLE
> >         lapack_opt_info:
> >             library_dirs = ['/usr/lib/atlas-base/atlas',
> >         '/usr/lib/atlas-base']
> >             include_dirs = ['/usr/include/atlas']
> >             language = f77
> >             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> >         blas_opt_info:
> >             library_dirs = ['/usr/lib/atlas-base']
> >             include_dirs = ['/usr/include/atlas']
> >             language = c
> >             libraries = ['ptf77blas', 'ptcblas', 'atlas']
> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> >         lapack_mkl_info:
> >           NOT AVAILABLE
> >         blas_mkl_info:
> >           NOT AVAILABLE
> >         >>>
> >
> >         gdb:
> >
> >         $ gdb python3
> >         GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
> >         Copyright (C) 2012 Free Software Foundation, Inc.
> >         License GPLv3+: GNU GPL version 3 or later
> >         <http://gnu.org/licenses/gpl.html>
> >         This is free software: you are free to change and redistribute
> it.
> >         There is NO WARRANTY, to the extent permitted by law.  Type
> >         "show copying"
> >         and "show warranty" for details.
> >         This GDB was configured as "x86_64-linux-gnu".
> >         For bug reporting instructions, please see:
> >         <http://bugs.launchpad.net/gdb-linaro/>...
> >         Reading symbols from /home/warren/local_py332/bin/python3...done.
> >         (gdb) run test_mrecords.py
> >         Starting program: /home/warren/local_py332/bin/python3
> >         test_mrecords.py
> >         [Thread debugging using libthread_db enabled]
> >         Using host libthread_db library
> >         "/lib/x86_64-linux-gnu/libthread_db.so.1".
> >         .
> >         Program received signal SIGSEGV, Segmentation fault.
> >         0x00007ffff5f080a4 in
> >         _aligned_strided_to_contig_size8_srcstride0 (dst=<optimized out>,
> >             dst_stride=<optimized out>, src=<optimized out>,
> >         __NPY_UNUSED_TAGGEDsrc_stride=<optimized out>,
> >             N=<optimized out>,
> >         __NPY_UNUSED_TAGGEDsrc_itemsize=<optimized out>,
> >         __NPY_UNUSED_TAGGEDdata=0x0)
> >             at numpy/core/src/multiarray/lowlevel_strided_loops.c.src:190
> >         190            *((@type@ *)dst) = temp;
> >         (gdb)
> >
> >
> >
> >
> > Is this repeatable? Just asking as I don't see that here with gcc 4.7.2.
> > Looks like Julian's changes to lowlevel_strided_loops may be responsible.
> >
> > Chuck
> >
>
> I can't reproduce it on ubuntu 12.04 with python 3.2 (don't have 3.3.2)
> the changes made are very minimal, from looking at them again I don't
> see how they could cause this (especially on a x86 platform).
>
> @warren, can you please bisect the commit causing this?
>

Also, 32 or 64 bit? Intel hardware? Clean install? Works here with python
3.3.0.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/a375ac6b/attachment.html>

From warren.weckesser at gmail.com  Sat Jun 15 15:28:54 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 15 Jun 2013 15:28:54 -0400
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <CAB6mnx+GJTKUBfvvdKXBOWB2E5=1tH=eHLypd1iM3s9MarrRhA@mail.gmail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
	<51BCBD69.3080800@googlemail.com>
	<CAB6mnx+GJTKUBfvvdKXBOWB2E5=1tH=eHLypd1iM3s9MarrRhA@mail.gmail.com>
Message-ID: <CAGzF1ueE+83Au1+wbkoM+FKZE=01PFnvqw9h7AiKZ8Jmpy06mA@mail.gmail.com>

On Sat, Jun 15, 2013 at 3:26 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Sat, Jun 15, 2013 at 1:15 PM, Julian Taylor <
> jtaylor.debian at googlemail.com> wrote:
>
>> On 15.06.2013 21:12, Charles R Harris wrote:
>> >
>> >
>> > On Sat, Jun 15, 2013 at 9:50 AM, Warren Weckesser
>> > <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>> wrote:
>> >
>> >
>> >     On Sat, Jun 15, 2013 at 11:43 AM, Warren Weckesser
>> >     <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>>
>> wrote:
>> >
>> >         I'm getting a seg. fault in master when I run the tests.  I'm on
>> >         Ubuntu 12.04 64 bit, with Python 3.3.2 (64 bits):
>> >
>> >         $ python3 -c "import numpy as np; np.test('full')"
>> >         Running unit tests for numpy
>> >         NumPy version 1.8.0.dev-fa5bc1c
>> >         NumPy is installed in
>> >         /home/warren/local_py332/lib/python3.3/site-packages/numpy
>> >         Python version 3.3.2 (default, Jun 14 2013, 12:12:22) [GCC
>> 4.6.3]
>> >         nose version 1.3.0
>> >
>> .............................S.........................................................................................................................................................S.......................................................................................................................................................S....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
>>  .........
>>
>> .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............................................................................................................................................................................................SSS.
>>  .........
>>
>> .......................................................................................K.....................................................................................................................................................................................Segmentation
>> >         fault
>> >
>> >         The seg. fault is occurring in ma/tests/test_mrecords.py:
>> >
>> >         $ nosetests test_mrecords.py
>> >         .Segmentation fault
>> >
>> >         More info:
>> >
>> >         $ python3
>> >         Python 3.3.2 (default, Jun 14 2013, 12:12:22)
>> >         [GCC 4.6.3] on linux
>> >         Type "help", "copyright", "credits" or "license" for more
>> >         information.
>> >         >>> import numpy as np
>> >         >>> np.show_config()
>> >         atlas_threads_info:
>> >             library_dirs = ['/usr/lib/atlas-base/atlas',
>> >         '/usr/lib/atlas-base']
>> >             include_dirs = ['/usr/include/atlas']
>> >             language = f77
>> >             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> >         atlas_blas_threads_info:
>> >             library_dirs = ['/usr/lib/atlas-base']
>> >             include_dirs = ['/usr/include/atlas']
>> >             language = c
>> >             libraries = ['ptf77blas', 'ptcblas', 'atlas']
>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> >         mkl_info:
>> >           NOT AVAILABLE
>> >         lapack_opt_info:
>> >             library_dirs = ['/usr/lib/atlas-base/atlas',
>> >         '/usr/lib/atlas-base']
>> >             include_dirs = ['/usr/include/atlas']
>> >             language = f77
>> >             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> >         blas_opt_info:
>> >             library_dirs = ['/usr/lib/atlas-base']
>> >             include_dirs = ['/usr/include/atlas']
>> >             language = c
>> >             libraries = ['ptf77blas', 'ptcblas', 'atlas']
>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> >         lapack_mkl_info:
>> >           NOT AVAILABLE
>> >         blas_mkl_info:
>> >           NOT AVAILABLE
>> >         >>>
>> >
>> >         gdb:
>> >
>> >         $ gdb python3
>> >         GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
>> >         Copyright (C) 2012 Free Software Foundation, Inc.
>> >         License GPLv3+: GNU GPL version 3 or later
>> >         <http://gnu.org/licenses/gpl.html>
>> >         This is free software: you are free to change and redistribute
>> it.
>> >         There is NO WARRANTY, to the extent permitted by law.  Type
>> >         "show copying"
>> >         and "show warranty" for details.
>> >         This GDB was configured as "x86_64-linux-gnu".
>> >         For bug reporting instructions, please see:
>> >         <http://bugs.launchpad.net/gdb-linaro/>...
>> >         Reading symbols from
>> /home/warren/local_py332/bin/python3...done.
>> >         (gdb) run test_mrecords.py
>> >         Starting program: /home/warren/local_py332/bin/python3
>> >         test_mrecords.py
>> >         [Thread debugging using libthread_db enabled]
>> >         Using host libthread_db library
>> >         "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> >         .
>> >         Program received signal SIGSEGV, Segmentation fault.
>> >         0x00007ffff5f080a4 in
>> >         _aligned_strided_to_contig_size8_srcstride0 (dst=<optimized
>> out>,
>> >             dst_stride=<optimized out>, src=<optimized out>,
>> >         __NPY_UNUSED_TAGGEDsrc_stride=<optimized out>,
>> >             N=<optimized out>,
>> >         __NPY_UNUSED_TAGGEDsrc_itemsize=<optimized out>,
>> >         __NPY_UNUSED_TAGGEDdata=0x0)
>> >             at
>> numpy/core/src/multiarray/lowlevel_strided_loops.c.src:190
>> >         190            *((@type@ *)dst) = temp;
>> >         (gdb)
>> >
>> >
>> >
>> >
>> > Is this repeatable? Just asking as I don't see that here with gcc 4.7.2.
>> > Looks like Julian's changes to lowlevel_strided_loops may be
>> responsible.
>> >
>> > Chuck
>> >
>>
>> I can't reproduce it on ubuntu 12.04 with python 3.2 (don't have 3.3.2)
>> the changes made are very minimal, from looking at them again I don't
>> see how they could cause this (especially on a x86 platform).
>>
>> @warren, can you please bisect the commit causing this?
>>
>
> Also, 32 or 64 bit? Intel hardware? Clean install? Works here with python
> 3.3.0.
>
>
64 bit, intel, clean install.  Bisecting now...

Warren



> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/43af38e6/attachment.html>

From warren.weckesser at gmail.com  Sat Jun 15 15:57:01 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 15 Jun 2013 15:57:01 -0400
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <51BCBD69.3080800@googlemail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
	<51BCBD69.3080800@googlemail.com>
Message-ID: <CAGzF1ucQhZxOdTL=wGmWBVw_GBDt=MwDc3DxQkKedG3hO+kPpw@mail.gmail.com>

On Sat, Jun 15, 2013 at 3:15 PM, Julian Taylor <
jtaylor.debian at googlemail.com> wrote:

> On 15.06.2013 21:12, Charles R Harris wrote:
> >
> >
> > On Sat, Jun 15, 2013 at 9:50 AM, Warren Weckesser
> > <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>> wrote:
> >
> >
> >     On Sat, Jun 15, 2013 at 11:43 AM, Warren Weckesser
> >     <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>>
> wrote:
> >
> >         I'm getting a seg. fault in master when I run the tests.  I'm on
> >         Ubuntu 12.04 64 bit, with Python 3.3.2 (64 bits):
> >
> >         $ python3 -c "import numpy as np; np.test('full')"
> >         Running unit tests for numpy
> >         NumPy version 1.8.0.dev-fa5bc1c
> >         NumPy is installed in
> >         /home/warren/local_py332/lib/python3.3/site-packages/numpy
> >         Python version 3.3.2 (default, Jun 14 2013, 12:12:22) [GCC 4.6.3]
> >         nose version 1.3.0
> >
> .............................S.........................................................................................................................................................S.......................................................................................................................................................S....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
>  .........
>
> .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............................................................................................................................................................................................SSS.
>  .........
>
> .......................................................................................K.....................................................................................................................................................................................Segmentation
> >         fault
> >
> >         The seg. fault is occurring in ma/tests/test_mrecords.py:
> >
> >         $ nosetests test_mrecords.py
> >         .Segmentation fault
> >
> >         More info:
> >
> >         $ python3
> >         Python 3.3.2 (default, Jun 14 2013, 12:12:22)
> >         [GCC 4.6.3] on linux
> >         Type "help", "copyright", "credits" or "license" for more
> >         information.
> >         >>> import numpy as np
> >         >>> np.show_config()
> >         atlas_threads_info:
> >             library_dirs = ['/usr/lib/atlas-base/atlas',
> >         '/usr/lib/atlas-base']
> >             include_dirs = ['/usr/include/atlas']
> >             language = f77
> >             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> >         atlas_blas_threads_info:
> >             library_dirs = ['/usr/lib/atlas-base']
> >             include_dirs = ['/usr/include/atlas']
> >             language = c
> >             libraries = ['ptf77blas', 'ptcblas', 'atlas']
> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> >         mkl_info:
> >           NOT AVAILABLE
> >         lapack_opt_info:
> >             library_dirs = ['/usr/lib/atlas-base/atlas',
> >         '/usr/lib/atlas-base']
> >             include_dirs = ['/usr/include/atlas']
> >             language = f77
> >             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> >         blas_opt_info:
> >             library_dirs = ['/usr/lib/atlas-base']
> >             include_dirs = ['/usr/include/atlas']
> >             language = c
> >             libraries = ['ptf77blas', 'ptcblas', 'atlas']
> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
> >         lapack_mkl_info:
> >           NOT AVAILABLE
> >         blas_mkl_info:
> >           NOT AVAILABLE
> >         >>>
> >
> >         gdb:
> >
> >         $ gdb python3
> >         GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
> >         Copyright (C) 2012 Free Software Foundation, Inc.
> >         License GPLv3+: GNU GPL version 3 or later
> >         <http://gnu.org/licenses/gpl.html>
> >         This is free software: you are free to change and redistribute
> it.
> >         There is NO WARRANTY, to the extent permitted by law.  Type
> >         "show copying"
> >         and "show warranty" for details.
> >         This GDB was configured as "x86_64-linux-gnu".
> >         For bug reporting instructions, please see:
> >         <http://bugs.launchpad.net/gdb-linaro/>...
> >         Reading symbols from /home/warren/local_py332/bin/python3...done.
> >         (gdb) run test_mrecords.py
> >         Starting program: /home/warren/local_py332/bin/python3
> >         test_mrecords.py
> >         [Thread debugging using libthread_db enabled]
> >         Using host libthread_db library
> >         "/lib/x86_64-linux-gnu/libthread_db.so.1".
> >         .
> >         Program received signal SIGSEGV, Segmentation fault.
> >         0x00007ffff5f080a4 in
> >         _aligned_strided_to_contig_size8_srcstride0 (dst=<optimized out>,
> >             dst_stride=<optimized out>, src=<optimized out>,
> >         __NPY_UNUSED_TAGGEDsrc_stride=<optimized out>,
> >             N=<optimized out>,
> >         __NPY_UNUSED_TAGGEDsrc_itemsize=<optimized out>,
> >         __NPY_UNUSED_TAGGEDdata=0x0)
> >             at numpy/core/src/multiarray/lowlevel_strided_loops.c.src:190
> >         190            *((@type@ *)dst) = temp;
> >         (gdb)
> >
> >
> >
> >
> > Is this repeatable? Just asking as I don't see that here with gcc 4.7.2.
> > Looks like Julian's changes to lowlevel_strided_loops may be responsible.
> >
> > Chuck
> >
>
> I can't reproduce it on ubuntu 12.04 with python 3.2 (don't have 3.3.2)
> the changes made are very minimal, from looking at them again I don't
> see how they could cause this (especially on a x86 platform).
>
> @warren, can you please bisect the commit causing this?
>


Here's the culprit:

aef286debfd11a62f1c337dea55624cee7fd4d9e is the first bad commit
commit aef286debfd11a62f1c337dea55624cee7fd4d9e
Author: Julian Taylor <jtaylor.debian at googlemail.com>
Date:   Mon Jun 10 19:38:58 2013 +0200

    ENH: enable unaligned loads on x86

    x86 can handle unaligned load and there is no hand vectorized code in
    this file. It would be a serious compiler bug if it adds vectorization
    without checking for alignment.
    Enables fast complex128 copies which are unaligned on 32 bit gcc unless
    compiled with -malign-double.

:040000 040000 d0948d1e1d942d41d50ce9e57bdc430db9a16f9e
45a48f383353857b8d0dd24e542c7ab6f137448c M  numpy

Link:
https://github.com/numpy/numpy/commit/aef286debfd11a62f1c337dea55624cee7fd4d9e

Warren



>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/6082cba8/attachment.html>

From jtaylor.debian at googlemail.com  Sat Jun 15 16:03:20 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Sat, 15 Jun 2013 22:03:20 +0200
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <CAGzF1ucQhZxOdTL=wGmWBVw_GBDt=MwDc3DxQkKedG3hO+kPpw@mail.gmail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
	<51BCBD69.3080800@googlemail.com>
	<CAGzF1ucQhZxOdTL=wGmWBVw_GBDt=MwDc3DxQkKedG3hO+kPpw@mail.gmail.com>
Message-ID: <51BCC888.6030700@googlemail.com>

On 15.06.2013 21:57, Warren Weckesser wrote:
> 
> On Sat, Jun 15, 2013 at 3:15 PM, Julian Taylor

>     @warren, can you please bisect the commit causing this?
> 
> 
> 
> Here's the culprit:
> 
> aef286debfd11a62f1c337dea55624cee7fd4d9e is the first bad commit
> commit aef286debfd11a62f1c337dea55624cee7fd4d9e
> Author: Julian Taylor <jtaylor.debian at googlemail.com
> <mailto:jtaylor.debian at googlemail.com>>
> Date:   Mon Jun 10 19:38:58 2013 +0200
> 
>     ENH: enable unaligned loads on x86
>    
>     x86 can handle unaligned load and there is no hand vectorized code in
>     this file. It would be a serious compiler bug if it adds vectorization
>     without checking for alignment.
>     Enables fast complex128 copies which are unaligned on 32 bit gcc unless
>     compiled with -malign-double.
> 
> :040000 040000 d0948d1e1d942d41d50ce9e57bdc430db9a16f9e
> 45a48f383353857b8d0dd24e542c7ab6f137448c M  numpy
> 
> Link:
> https://github.com/numpy/numpy/commit/aef286debfd11a62f1c337dea55624cee7fd4d9e
> 

Interesting, possibly there is some inconsistencies in the macros in
this file.
But I don't understand why I can't reproduce it.
Does it happen with python 3.2 too?


From warren.weckesser at gmail.com  Sat Jun 15 16:03:37 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 15 Jun 2013 16:03:37 -0400
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <CAGzF1ucQhZxOdTL=wGmWBVw_GBDt=MwDc3DxQkKedG3hO+kPpw@mail.gmail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
	<51BCBD69.3080800@googlemail.com>
	<CAGzF1ucQhZxOdTL=wGmWBVw_GBDt=MwDc3DxQkKedG3hO+kPpw@mail.gmail.com>
Message-ID: <CAGzF1ueBUH-9qEAEFbOx2fCv4yQ6HfGcJW=H2BrCf=gpnUdRGA@mail.gmail.com>

On Sat, Jun 15, 2013 at 3:57 PM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

>
> On Sat, Jun 15, 2013 at 3:15 PM, Julian Taylor <
> jtaylor.debian at googlemail.com> wrote:
>
>> On 15.06.2013 21:12, Charles R Harris wrote:
>> >
>> >
>> > On Sat, Jun 15, 2013 at 9:50 AM, Warren Weckesser
>> > <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>> wrote:
>> >
>> >
>> >     On Sat, Jun 15, 2013 at 11:43 AM, Warren Weckesser
>> >     <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>>
>> wrote:
>> >
>> >         I'm getting a seg. fault in master when I run the tests.  I'm on
>> >         Ubuntu 12.04 64 bit, with Python 3.3.2 (64 bits):
>> >
>> >         $ python3 -c "import numpy as np; np.test('full')"
>> >         Running unit tests for numpy
>> >         NumPy version 1.8.0.dev-fa5bc1c
>> >         NumPy is installed in
>> >         /home/warren/local_py332/lib/python3.3/site-packages/numpy
>> >         Python version 3.3.2 (default, Jun 14 2013, 12:12:22) [GCC
>> 4.6.3]
>> >         nose version 1.3.0
>> >
>> .............................S.........................................................................................................................................................S.......................................................................................................................................................S....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
>>  .........
>>
>> .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............................................................................................................................................................................................SSS.
>>  .........
>>
>> .......................................................................................K.....................................................................................................................................................................................Segmentation
>> >         fault
>> >
>> >         The seg. fault is occurring in ma/tests/test_mrecords.py:
>> >
>> >         $ nosetests test_mrecords.py
>> >         .Segmentation fault
>> >
>> >         More info:
>> >
>> >         $ python3
>> >         Python 3.3.2 (default, Jun 14 2013, 12:12:22)
>> >         [GCC 4.6.3] on linux
>> >         Type "help", "copyright", "credits" or "license" for more
>> >         information.
>> >         >>> import numpy as np
>> >         >>> np.show_config()
>> >         atlas_threads_info:
>> >             library_dirs = ['/usr/lib/atlas-base/atlas',
>> >         '/usr/lib/atlas-base']
>> >             include_dirs = ['/usr/include/atlas']
>> >             language = f77
>> >             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> >         atlas_blas_threads_info:
>> >             library_dirs = ['/usr/lib/atlas-base']
>> >             include_dirs = ['/usr/include/atlas']
>> >             language = c
>> >             libraries = ['ptf77blas', 'ptcblas', 'atlas']
>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> >         mkl_info:
>> >           NOT AVAILABLE
>> >         lapack_opt_info:
>> >             library_dirs = ['/usr/lib/atlas-base/atlas',
>> >         '/usr/lib/atlas-base']
>> >             include_dirs = ['/usr/include/atlas']
>> >             language = f77
>> >             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> >         blas_opt_info:
>> >             library_dirs = ['/usr/lib/atlas-base']
>> >             include_dirs = ['/usr/include/atlas']
>> >             language = c
>> >             libraries = ['ptf77blas', 'ptcblas', 'atlas']
>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>> >         lapack_mkl_info:
>> >           NOT AVAILABLE
>> >         blas_mkl_info:
>> >           NOT AVAILABLE
>> >         >>>
>> >
>> >         gdb:
>> >
>> >         $ gdb python3
>> >         GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
>> >         Copyright (C) 2012 Free Software Foundation, Inc.
>> >         License GPLv3+: GNU GPL version 3 or later
>> >         <http://gnu.org/licenses/gpl.html>
>> >         This is free software: you are free to change and redistribute
>> it.
>> >         There is NO WARRANTY, to the extent permitted by law.  Type
>> >         "show copying"
>> >         and "show warranty" for details.
>> >         This GDB was configured as "x86_64-linux-gnu".
>> >         For bug reporting instructions, please see:
>> >         <http://bugs.launchpad.net/gdb-linaro/>...
>> >         Reading symbols from
>> /home/warren/local_py332/bin/python3...done.
>> >         (gdb) run test_mrecords.py
>> >         Starting program: /home/warren/local_py332/bin/python3
>> >         test_mrecords.py
>> >         [Thread debugging using libthread_db enabled]
>> >         Using host libthread_db library
>> >         "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> >         .
>> >         Program received signal SIGSEGV, Segmentation fault.
>> >         0x00007ffff5f080a4 in
>> >         _aligned_strided_to_contig_size8_srcstride0 (dst=<optimized
>> out>,
>> >             dst_stride=<optimized out>, src=<optimized out>,
>> >         __NPY_UNUSED_TAGGEDsrc_stride=<optimized out>,
>> >             N=<optimized out>,
>> >         __NPY_UNUSED_TAGGEDsrc_itemsize=<optimized out>,
>> >         __NPY_UNUSED_TAGGEDdata=0x0)
>> >             at
>> numpy/core/src/multiarray/lowlevel_strided_loops.c.src:190
>> >         190            *((@type@ *)dst) = temp;
>> >         (gdb)
>> >
>> >
>> >
>> >
>> > Is this repeatable? Just asking as I don't see that here with gcc 4.7.2.
>> > Looks like Julian's changes to lowlevel_strided_loops may be
>> responsible.
>> >
>> > Chuck
>> >
>>
>> I can't reproduce it on ubuntu 12.04 with python 3.2 (don't have 3.3.2)
>> the changes made are very minimal, from looking at them again I don't
>> see how they could cause this (especially on a x86 platform).
>>
>> @warren, can you please bisect the commit causing this?
>>
>
>
> Here's the culprit:
>
> aef286debfd11a62f1c337dea55624cee7fd4d9e is the first bad commit
> commit aef286debfd11a62f1c337dea55624cee7fd4d9e
> Author: Julian Taylor <jtaylor.debian at googlemail.com>
> Date:   Mon Jun 10 19:38:58 2013 +0200
>
>     ENH: enable unaligned loads on x86
>
>     x86 can handle unaligned load and there is no hand vectorized code in
>     this file. It would be a serious compiler bug if it adds vectorization
>     without checking for alignment.
>     Enables fast complex128 copies which are unaligned on 32 bit gcc unless
>     compiled with -malign-double.
>
> :040000 040000 d0948d1e1d942d41d50ce9e57bdc430db9a16f9e
> 45a48f383353857b8d0dd24e542c7ab6f137448c M  numpy
>
> Link:
> https://github.com/numpy/numpy/commit/aef286debfd11a62f1c337dea55624cee7fd4d9e
>
>
If I undo that change in master, I don't get the seg. fault.

Warren



> Warren
>
>
>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/d2c2a362/attachment.html>

From charlesr.harris at gmail.com  Sat Jun 15 16:17:42 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 15 Jun 2013 14:17:42 -0600
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <CAGzF1ueBUH-9qEAEFbOx2fCv4yQ6HfGcJW=H2BrCf=gpnUdRGA@mail.gmail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
	<51BCBD69.3080800@googlemail.com>
	<CAGzF1ucQhZxOdTL=wGmWBVw_GBDt=MwDc3DxQkKedG3hO+kPpw@mail.gmail.com>
	<CAGzF1ueBUH-9qEAEFbOx2fCv4yQ6HfGcJW=H2BrCf=gpnUdRGA@mail.gmail.com>
Message-ID: <CAB6mnxL6WoWjuJfzYOLiwc2abGhYmAxLNPoo=WQA1HDS0Db8Dg@mail.gmail.com>

On Sat, Jun 15, 2013 at 2:03 PM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

>
> On Sat, Jun 15, 2013 at 3:57 PM, Warren Weckesser <
> warren.weckesser at gmail.com> wrote:
>
>>
>> On Sat, Jun 15, 2013 at 3:15 PM, Julian Taylor <
>> jtaylor.debian at googlemail.com> wrote:
>>
>>> On 15.06.2013 21:12, Charles R Harris wrote:
>>> >
>>> >
>>> > On Sat, Jun 15, 2013 at 9:50 AM, Warren Weckesser
>>> > <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>>
>>> wrote:
>>> >
>>> >
>>> >     On Sat, Jun 15, 2013 at 11:43 AM, Warren Weckesser
>>> >     <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>>
>>> wrote:
>>> >
>>> >         I'm getting a seg. fault in master when I run the tests.  I'm
>>> on
>>> >         Ubuntu 12.04 64 bit, with Python 3.3.2 (64 bits):
>>> >
>>> >         $ python3 -c "import numpy as np; np.test('full')"
>>> >         Running unit tests for numpy
>>> >         NumPy version 1.8.0.dev-fa5bc1c
>>> >         NumPy is installed in
>>> >         /home/warren/local_py332/lib/python3.3/site-packages/numpy
>>> >         Python version 3.3.2 (default, Jun 14 2013, 12:12:22) [GCC
>>> 4.6.3]
>>> >         nose version 1.3.0
>>> >
>>> .............................S.........................................................................................................................................................S.......................................................................................................................................................S....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
>>>  .........
>>>
>>> .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K............................................................................................................................................................................................SSS.
>>>  .........
>>>
>>> .......................................................................................K.....................................................................................................................................................................................Segmentation
>>> >         fault
>>> >
>>> >         The seg. fault is occurring in ma/tests/test_mrecords.py:
>>> >
>>> >         $ nosetests test_mrecords.py
>>> >         .Segmentation fault
>>> >
>>> >         More info:
>>> >
>>> >         $ python3
>>> >         Python 3.3.2 (default, Jun 14 2013, 12:12:22)
>>> >         [GCC 4.6.3] on linux
>>> >         Type "help", "copyright", "credits" or "license" for more
>>> >         information.
>>> >         >>> import numpy as np
>>> >         >>> np.show_config()
>>> >         atlas_threads_info:
>>> >             library_dirs = ['/usr/lib/atlas-base/atlas',
>>> >         '/usr/lib/atlas-base']
>>> >             include_dirs = ['/usr/include/atlas']
>>> >             language = f77
>>> >             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>>> >         atlas_blas_threads_info:
>>> >             library_dirs = ['/usr/lib/atlas-base']
>>> >             include_dirs = ['/usr/include/atlas']
>>> >             language = c
>>> >             libraries = ['ptf77blas', 'ptcblas', 'atlas']
>>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>>> >         mkl_info:
>>> >           NOT AVAILABLE
>>> >         lapack_opt_info:
>>> >             library_dirs = ['/usr/lib/atlas-base/atlas',
>>> >         '/usr/lib/atlas-base']
>>> >             include_dirs = ['/usr/include/atlas']
>>> >             language = f77
>>> >             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>>> >         blas_opt_info:
>>> >             library_dirs = ['/usr/lib/atlas-base']
>>> >             include_dirs = ['/usr/include/atlas']
>>> >             language = c
>>> >             libraries = ['ptf77blas', 'ptcblas', 'atlas']
>>> >             define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')]
>>> >         lapack_mkl_info:
>>> >           NOT AVAILABLE
>>> >         blas_mkl_info:
>>> >           NOT AVAILABLE
>>> >         >>>
>>> >
>>> >         gdb:
>>> >
>>> >         $ gdb python3
>>> >         GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
>>> >         Copyright (C) 2012 Free Software Foundation, Inc.
>>> >         License GPLv3+: GNU GPL version 3 or later
>>> >         <http://gnu.org/licenses/gpl.html>
>>> >         This is free software: you are free to change and redistribute
>>> it.
>>> >         There is NO WARRANTY, to the extent permitted by law.  Type
>>> >         "show copying"
>>> >         and "show warranty" for details.
>>> >         This GDB was configured as "x86_64-linux-gnu".
>>> >         For bug reporting instructions, please see:
>>> >         <http://bugs.launchpad.net/gdb-linaro/>...
>>> >         Reading symbols from
>>> /home/warren/local_py332/bin/python3...done.
>>> >         (gdb) run test_mrecords.py
>>> >         Starting program: /home/warren/local_py332/bin/python3
>>> >         test_mrecords.py
>>> >         [Thread debugging using libthread_db enabled]
>>> >         Using host libthread_db library
>>> >         "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> >         .
>>> >         Program received signal SIGSEGV, Segmentation fault.
>>> >         0x00007ffff5f080a4 in
>>> >         _aligned_strided_to_contig_size8_srcstride0 (dst=<optimized
>>> out>,
>>> >             dst_stride=<optimized out>, src=<optimized out>,
>>> >         __NPY_UNUSED_TAGGEDsrc_stride=<optimized out>,
>>> >             N=<optimized out>,
>>> >         __NPY_UNUSED_TAGGEDsrc_itemsize=<optimized out>,
>>> >         __NPY_UNUSED_TAGGEDdata=0x0)
>>> >             at
>>> numpy/core/src/multiarray/lowlevel_strided_loops.c.src:190
>>> >         190            *((@type@ *)dst) = temp;
>>> >         (gdb)
>>> >
>>> >
>>> >
>>> >
>>> > Is this repeatable? Just asking as I don't see that here with gcc
>>> 4.7.2.
>>> > Looks like Julian's changes to lowlevel_strided_loops may be
>>> responsible.
>>> >
>>> > Chuck
>>> >
>>>
>>> I can't reproduce it on ubuntu 12.04 with python 3.2 (don't have 3.3.2)
>>> the changes made are very minimal, from looking at them again I don't
>>> see how they could cause this (especially on a x86 platform).
>>>
>>> @warren, can you please bisect the commit causing this?
>>>
>>
>>
>> Here's the culprit:
>>
>> aef286debfd11a62f1c337dea55624cee7fd4d9e is the first bad commit
>> commit aef286debfd11a62f1c337dea55624cee7fd4d9e
>> Author: Julian Taylor <jtaylor.debian at googlemail.com>
>> Date:   Mon Jun 10 19:38:58 2013 +0200
>>
>>     ENH: enable unaligned loads on x86
>>
>>     x86 can handle unaligned load and there is no hand vectorized code in
>>     this file. It would be a serious compiler bug if it adds vectorization
>>     without checking for alignment.
>>     Enables fast complex128 copies which are unaligned on 32 bit gcc
>> unless
>>     compiled with -malign-double.
>>
>> :040000 040000 d0948d1e1d942d41d50ce9e57bdc430db9a16f9e
>> 45a48f383353857b8d0dd24e542c7ab6f137448c M  numpy
>>
>> Link:
>> https://github.com/numpy/numpy/commit/aef286debfd11a62f1c337dea55624cee7fd4d9e
>>
>>
> If I undo that change in master, I don't get the seg. fault.
>

OK, I can duplicate this with -O3 along with another error that shows up.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/68ff9135/attachment.html>

From warren.weckesser at gmail.com  Sat Jun 15 16:23:57 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 15 Jun 2013 16:23:57 -0400
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <51BCC888.6030700@googlemail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
	<51BCBD69.3080800@googlemail.com>
	<CAGzF1ucQhZxOdTL=wGmWBVw_GBDt=MwDc3DxQkKedG3hO+kPpw@mail.gmail.com>
	<51BCC888.6030700@googlemail.com>
Message-ID: <CAGzF1ue2+KTeAT72Li0-LB2ARtm9sHOeoRcVLfS6XFqcj8QDUg@mail.gmail.com>

On Sat, Jun 15, 2013 at 4:03 PM, Julian Taylor <
jtaylor.debian at googlemail.com> wrote:

> On 15.06.2013 21:57, Warren Weckesser wrote:
> >
> > On Sat, Jun 15, 2013 at 3:15 PM, Julian Taylor
>
> >     @warren, can you please bisect the commit causing this?
> >
> >
> >
> > Here's the culprit:
> >
> > aef286debfd11a62f1c337dea55624cee7fd4d9e is the first bad commit
> > commit aef286debfd11a62f1c337dea55624cee7fd4d9e
> > Author: Julian Taylor <jtaylor.debian at googlemail.com
> > <mailto:jtaylor.debian at googlemail.com>>
> > Date:   Mon Jun 10 19:38:58 2013 +0200
> >
> >     ENH: enable unaligned loads on x86
> >
> >     x86 can handle unaligned load and there is no hand vectorized code in
> >     this file. It would be a serious compiler bug if it adds
> vectorization
> >     without checking for alignment.
> >     Enables fast complex128 copies which are unaligned on 32 bit gcc
> unless
> >     compiled with -malign-double.
> >
> > :040000 040000 d0948d1e1d942d41d50ce9e57bdc430db9a16f9e
> > 45a48f383353857b8d0dd24e542c7ab6f137448c M  numpy
> >
> > Link:
> >
> https://github.com/numpy/numpy/commit/aef286debfd11a62f1c337dea55624cee7fd4d9e
> >
>
> Interesting, possibly there is some inconsistencies in the macros in
> this file.
> But I don't understand why I can't reproduce it.
> Does it happen with python 3.2 too?
>


Yes, it happens with 3.3.2, 3.3.1, and 3.2.5.

Warren


> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/8a4670a0/attachment.html>

From charlesr.harris at gmail.com  Sat Jun 15 16:26:37 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 15 Jun 2013 14:26:37 -0600
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <CAGzF1ue2+KTeAT72Li0-LB2ARtm9sHOeoRcVLfS6XFqcj8QDUg@mail.gmail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
	<51BCBD69.3080800@googlemail.com>
	<CAGzF1ucQhZxOdTL=wGmWBVw_GBDt=MwDc3DxQkKedG3hO+kPpw@mail.gmail.com>
	<51BCC888.6030700@googlemail.com>
	<CAGzF1ue2+KTeAT72Li0-LB2ARtm9sHOeoRcVLfS6XFqcj8QDUg@mail.gmail.com>
Message-ID: <CAB6mnx+LMP08Ege-RgDnxR-YXxVt-XwaBt9ne8QjkbCCZxWHWg@mail.gmail.com>

On Sat, Jun 15, 2013 at 2:23 PM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

>
> On Sat, Jun 15, 2013 at 4:03 PM, Julian Taylor <
> jtaylor.debian at googlemail.com> wrote:
>
>> On 15.06.2013 21:57, Warren Weckesser wrote:
>> >
>> > On Sat, Jun 15, 2013 at 3:15 PM, Julian Taylor
>>
>> >     @warren, can you please bisect the commit causing this?
>> >
>> >
>> >
>> > Here's the culprit:
>> >
>> > aef286debfd11a62f1c337dea55624cee7fd4d9e is the first bad commit
>> > commit aef286debfd11a62f1c337dea55624cee7fd4d9e
>> > Author: Julian Taylor <jtaylor.debian at googlemail.com
>> > <mailto:jtaylor.debian at googlemail.com>>
>> > Date:   Mon Jun 10 19:38:58 2013 +0200
>> >
>> >     ENH: enable unaligned loads on x86
>> >
>> >     x86 can handle unaligned load and there is no hand vectorized code
>> in
>> >     this file. It would be a serious compiler bug if it adds
>> vectorization
>> >     without checking for alignment.
>> >     Enables fast complex128 copies which are unaligned on 32 bit gcc
>> unless
>> >     compiled with -malign-double.
>> >
>> > :040000 040000 d0948d1e1d942d41d50ce9e57bdc430db9a16f9e
>> > 45a48f383353857b8d0dd24e542c7ab6f137448c M  numpy
>> >
>> > Link:
>> >
>> https://github.com/numpy/numpy/commit/aef286debfd11a62f1c337dea55624cee7fd4d9e
>> >
>>
>> Interesting, possibly there is some inconsistencies in the macros in
>> this file.
>> But I don't understand why I can't reproduce it.
>> Does it happen with python 3.2 too?
>>
>
>
> Yes, it happens with 3.3.2, 3.3.1, and 3.2.5.
>

Also on 2.7. The -O3 flag seems to cause the problem.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/24b37112/attachment.html>

From charlesr.harris at gmail.com  Sat Jun 15 16:33:23 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 15 Jun 2013 14:33:23 -0600
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <CAB6mnx+LMP08Ege-RgDnxR-YXxVt-XwaBt9ne8QjkbCCZxWHWg@mail.gmail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
	<51BCBD69.3080800@googlemail.com>
	<CAGzF1ucQhZxOdTL=wGmWBVw_GBDt=MwDc3DxQkKedG3hO+kPpw@mail.gmail.com>
	<51BCC888.6030700@googlemail.com>
	<CAGzF1ue2+KTeAT72Li0-LB2ARtm9sHOeoRcVLfS6XFqcj8QDUg@mail.gmail.com>
	<CAB6mnx+LMP08Ege-RgDnxR-YXxVt-XwaBt9ne8QjkbCCZxWHWg@mail.gmail.com>
Message-ID: <CAB6mnxJQ618DYEgJ50q5-j3tTwAWgdVL-Z=ZeXAcyO3-z+2ycQ@mail.gmail.com>

On Sat, Jun 15, 2013 at 2:26 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Sat, Jun 15, 2013 at 2:23 PM, Warren Weckesser <
> warren.weckesser at gmail.com> wrote:
>
>>
>> On Sat, Jun 15, 2013 at 4:03 PM, Julian Taylor <
>> jtaylor.debian at googlemail.com> wrote:
>>
>>> On 15.06.2013 21:57, Warren Weckesser wrote:
>>> >
>>> > On Sat, Jun 15, 2013 at 3:15 PM, Julian Taylor
>>>
>>> >     @warren, can you please bisect the commit causing this?
>>> >
>>> >
>>> >
>>> > Here's the culprit:
>>> >
>>> > aef286debfd11a62f1c337dea55624cee7fd4d9e is the first bad commit
>>> > commit aef286debfd11a62f1c337dea55624cee7fd4d9e
>>> > Author: Julian Taylor <jtaylor.debian at googlemail.com
>>> > <mailto:jtaylor.debian at googlemail.com>>
>>> > Date:   Mon Jun 10 19:38:58 2013 +0200
>>> >
>>> >     ENH: enable unaligned loads on x86
>>> >
>>> >     x86 can handle unaligned load and there is no hand vectorized code
>>> in
>>> >     this file. It would be a serious compiler bug if it adds
>>> vectorization
>>> >     without checking for alignment.
>>> >     Enables fast complex128 copies which are unaligned on 32 bit gcc
>>> unless
>>> >     compiled with -malign-double.
>>> >
>>> > :040000 040000 d0948d1e1d942d41d50ce9e57bdc430db9a16f9e
>>> > 45a48f383353857b8d0dd24e542c7ab6f137448c M  numpy
>>> >
>>> > Link:
>>> >
>>> https://github.com/numpy/numpy/commit/aef286debfd11a62f1c337dea55624cee7fd4d9e
>>> >
>>>
>>> Interesting, possibly there is some inconsistencies in the macros in
>>> this file.
>>> But I don't understand why I can't reproduce it.
>>> Does it happen with python 3.2 too?
>>>
>>
>>
>> Yes, it happens with 3.3.2, 3.3.1, and 3.2.5.
>>
>
> Also on 2.7. The -O3 flag seems to cause the problem.
>
>
But I see it on test_3d_shaped_dtype.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130615/0fd85dcd/attachment.html>

From jtaylor.debian at googlemail.com  Sat Jun 15 16:34:15 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Sat, 15 Jun 2013 22:34:15 +0200
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <CAB6mnxJQ618DYEgJ50q5-j3tTwAWgdVL-Z=ZeXAcyO3-z+2ycQ@mail.gmail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
	<51BCBD69.3080800@googlemail.com>
	<CAGzF1ucQhZxOdTL=wGmWBVw_GBDt=MwDc3DxQkKedG3hO+kPpw@mail.gmail.com>
	<51BCC888.6030700@googlemail.com>
	<CAGzF1ue2+KTeAT72Li0-LB2ARtm9sHOeoRcVLfS6XFqcj8QDUg@mail.gmail.com>
	<CAB6mnx+LMP08Ege-RgDnxR-YXxVt-XwaBt9ne8QjkbCCZxWHWg@mail.gmail.com>
	<CAB6mnxJQ618DYEgJ50q5-j3tTwAWgdVL-Z=ZeXAcyO3-z+2ycQ@mail.gmail.com>
Message-ID: <51BCCFC7.5010204@googlemail.com>

On 15.06.2013 22:33, Charles R Harris wrote:
> 
> 
> On Sat, Jun 15, 2013 at 2:26 PM, Charles R Harris
> <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
> 
> 
> 
>     On Sat, Jun 15, 2013 at 2:23 PM, Warren Weckesser
>     <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>> wrote:
> 
> 
>         On Sat, Jun 15, 2013 at 4:03 PM, Julian Taylor
>         <jtaylor.debian at googlemail.com
>         <mailto:jtaylor.debian at googlemail.com>> wrote:
> 
>             On 15.06.2013 21:57, Warren Weckesser wrote:
>             >
>             > On Sat, Jun 15, 2013 at 3:15 PM, Julian Taylor
> 
>             >     @warren, can you please bisect the commit causing this?
>             >
>             >
>             >
>             > Here's the culprit:
>             >
>             > aef286debfd11a62f1c337dea55624cee7fd4d9e is the first bad
>             commit
>             > commit aef286debfd11a62f1c337dea55624cee7fd4d9e
>             > Author: Julian Taylor <jtaylor.debian at googlemail.com
>             <mailto:jtaylor.debian at googlemail.com>
>             > <mailto:jtaylor.debian at googlemail.com
>             <mailto:jtaylor.debian at googlemail.com>>>
>             > Date:   Mon Jun 10 19:38:58 2013 +0200
>             >
>             >     ENH: enable unaligned loads on x86
>             >
>             >     x86 can handle unaligned load and there is no hand
>             vectorized code in
>             >     this file. It would be a serious compiler bug if it
>             adds vectorization
>             >     without checking for alignment.
>             >     Enables fast complex128 copies which are unaligned on
>             32 bit gcc unless
>             >     compiled with -malign-double.
>             >
>             > :040000 040000 d0948d1e1d942d41d50ce9e57bdc430db9a16f9e
>             > 45a48f383353857b8d0dd24e542c7ab6f137448c M  numpy
>             >
>             > Link:
>             >
>             https://github.com/numpy/numpy/commit/aef286debfd11a62f1c337dea55624cee7fd4d9e
>             >
> 
>             Interesting, possibly there is some inconsistencies in the
>             macros in
>             this file.
>             But I don't understand why I can't reproduce it.
>             Does it happen with python 3.2 too?
> 
> 
> 
>         Yes, it happens with 3.3.2, 3.3.1, and 3.2.5.
> 
> 
>     Also on 2.7. The -O3 flag seems to cause the problem.
> 
> 
> But I see it on test_3d_shaped_dtype.
> 

I can reproduce it with O3, I'll have a look.



From jtaylor.debian at googlemail.com  Sat Jun 15 18:15:24 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Sun, 16 Jun 2013 00:15:24 +0200
Subject: [Numpy-discussion] Seg. fault when running tests
In-Reply-To: <CAB6mnx+LMP08Ege-RgDnxR-YXxVt-XwaBt9ne8QjkbCCZxWHWg@mail.gmail.com>
References: <CAGzF1ucVZGJnZ9GHC7XyTNXM8ePw425-9AZaF4d1E3bpGdXfkQ@mail.gmail.com>
	<CAGzF1ud4+nmQHvsHOWXPFgVmJJERiti6qu1_DMpnWofNMchK2g@mail.gmail.com>
	<CAB6mnxLShbDYYep=g_fK2dRpUXVoZEdLAs5259Csdd9ov6A+dg@mail.gmail.com>
	<51BCBD69.3080800@googlemail.com>
	<CAGzF1ucQhZxOdTL=wGmWBVw_GBDt=MwDc3DxQkKedG3hO+kPpw@mail.gmail.com>
	<51BCC888.6030700@googlemail.com>
	<CAGzF1ue2+KTeAT72Li0-LB2ARtm9sHOeoRcVLfS6XFqcj8QDUg@mail.gmail.com>
	<CAB6mnx+LMP08Ege-RgDnxR-YXxVt-XwaBt9ne8QjkbCCZxWHWg@mail.gmail.com>
Message-ID: <51BCE77C.6020602@googlemail.com>

On 15.06.2013 22:26, Charles R Harris wrote:
> 
> 
> On Sat, Jun 15, 2013 at 2:23 PM, Warren Weckesser
> <warren.weckesser at gmail.com <mailto:warren.weckesser at gmail.com>> wrote:
> 
> 
> Also on 2.7. The -O3 flag seems to cause the problem.
> 
> Chuck
> 

I was wrong when I changed this flag.
The compiler is allowed to assume the data is aligned to the size of the
type.
So when vectorizing it peels the wrong amount of data and ends up with
non 16-byte aligned SSE instructions.
I'll file a pull reverting the change.


From sudheer.joseph at yahoo.com  Sun Jun 16 02:48:34 2013
From: sudheer.joseph at yahoo.com (Sudheer Joseph)
Date: Sun, 16 Jun 2013 14:48:34 +0800 (SGT)
Subject: [Numpy-discussion] sampling arrays
Message-ID: <1371365314.38661.YahooMailNeo@web193405.mail.sg3.yahoo.com>

Dear experts,
?????????????? Is it possible to sample a 4D array of numpy at given dimensions with out writing loops? ie a smart python way?

for example,

I have a? data cube with cube=c(x,y,z,t) dimensions which are essentially latitude longitude depth and time. Is it possible to sample this array at a given dimensions ie

sample1= sample(c,:,:,1,1) this is sample at surface at time 1
sample2= sample(c,:,:,1,:) this is time series of surface? data
sample3= sample(c,1,1,:,1) this is a profile at given location and time
sample4= sample(c,1,1,:,:) this is time series at given location at all depths


with best regards,
Sudheer


***************************************************************
Sudheer Joseph 
Indian National Centre for Ocean Information Services
Ministry of Earth Sciences, Govt. of India
POST BOX NO: 21, IDA Jeedeemetla P.O.
Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55
Tel:+91-40-23886047(O),Fax:+91-40-23895011(O),
Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile)
E-mail:sjo.India at gmail.com;sudheer.joseph at yahoo.com
Web- http://oppamthadathil.tripod.com
***************************************************************?


From heng at cantab.net  Sun Jun 16 05:19:15 2013
From: heng at cantab.net (Henry Gomersall)
Date: Sun, 16 Jun 2013 10:19:15 +0100
Subject: [Numpy-discussion] sampling arrays
In-Reply-To: <1371365314.38661.YahooMailNeo@web193405.mail.sg3.yahoo.com>
References: <1371365314.38661.YahooMailNeo@web193405.mail.sg3.yahoo.com>
Message-ID: <1371374355.7804.8.camel@farnsworth>

On Sun, 2013-06-16 at 14:48 +0800, Sudheer Joseph wrote:
> Is it possible to sample a 4D array of numpy at given dimensions with
> out writing loops? ie a smart python way?

It's not clear how what you want to do is different from simply indexing
the array...?

Henry



From sudheer.joseph at yahoo.com  Sun Jun 16 05:26:13 2013
From: sudheer.joseph at yahoo.com (Sudheer Joseph)
Date: Sun, 16 Jun 2013 17:26:13 +0800 (SGT)
Subject: [Numpy-discussion] sampling arrays
In-Reply-To: <1371374355.7804.8.camel@farnsworth>
References: <1371365314.38661.YahooMailNeo@web193405.mail.sg3.yahoo.com>
	<1371374355.7804.8.camel@farnsworth>
Message-ID: <1371374773.4754.YahooMailNeo@web193402.mail.sg3.yahoo.com>

Thank you, 
???????????????? No if the location ( space time or depth) of choice is not available then the function I was looking for should give an interpolated value at the choice.
with best regards,
Sudheer
----- Original Message -----

> From: Henry Gomersall <heng at cantab.net>
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Cc: 
> Sent: Sunday, 16 June 2013 2:49 PM
> Subject: Re: [Numpy-discussion] sampling arrays
> 
> On Sun, 2013-06-16 at 14:48 +0800, Sudheer Joseph wrote:
>>  Is it possible to sample a 4D array of numpy at given dimensions with
>>  out writing loops? ie a smart python way?
> 
> It's not clear how what you want to do is different from simply indexing
> the array...?
> 
> Henry
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 


From sudheer.joseph at yahoo.com  Sun Jun 16 07:19:05 2013
From: sudheer.joseph at yahoo.com (Sudheer Joseph)
Date: Sun, 16 Jun 2013 19:19:05 +0800 (SGT)
Subject: [Numpy-discussion] sampling arrays
In-Reply-To: <1371374773.4754.YahooMailNeo@web193402.mail.sg3.yahoo.com>
References: <1371365314.38661.YahooMailNeo@web193405.mail.sg3.yahoo.com>
	<1371374355.7804.8.camel@farnsworth>
	<1371374773.4754.YahooMailNeo@web193402.mail.sg3.yahoo.com>
Message-ID: <1371381545.98570.YahooMailNeo@web193405.mail.sg3.yahoo.com>

Here is a solution I got.
I think I need to do the rest with loop. Though I am seeing an interpnd I could not find a documentation on that or example.
http://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.RectBivariateSpline.html#scipy.interpolate.RectBivariateSpline


Thank you
with best regards,
Sudheer

?
***************************************************************
Sudheer Joseph 
Indian National Centre for Ocean Information Services
Ministry of Earth Sciences, Govt. of India
POST BOX NO: 21, IDA Jeedeemetla P.O.
Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55
Tel:+91-40-23886047(O),Fax:+91-40-23895011(O),
Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile)
E-mail:sjo.India at gmail.com;sudheer.joseph at yahoo.com
Web- http://oppamthadathil.tripod.com
***************************************************************


----- Original Message -----
> From: Sudheer Joseph <sudheer.joseph at yahoo.com>
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Cc: 
> Sent: Sunday, 16 June 2013 2:56 PM
> Subject: Re: [Numpy-discussion] sampling arrays
> 
>T hank you, 
> ???????????????? No if the location ( space time or depth) of choice is not 
> available then the function I was looking for should give an interpolated value 
> at the choice.
> with best regards,
> Sudheer
> ----- Original Message -----
> 
>>  From: Henry Gomersall <heng at cantab.net>
>>  To: Discussion of Numerical Python <numpy-discussion at scipy.org>
>>  Cc: 
>>  Sent: Sunday, 16 June 2013 2:49 PM
>>  Subject: Re: [Numpy-discussion] sampling arrays
>> 
>>  On Sun, 2013-06-16 at 14:48 +0800, Sudheer Joseph wrote:
>>> ? Is it possible to sample a 4D array of numpy at given dimensions with
>>> ? out writing loops? ie a smart python way?
>> 
>>  It's not clear how what you want to do is different from simply 
> indexing
>>  the array...?
>> 
>>  Henry
>> 
>>  _______________________________________________
>>  NumPy-Discussion mailing list
>>  NumPy-Discussion at scipy.org
>>  http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 


From warren.weckesser at gmail.com  Sun Jun 16 12:56:03 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sun, 16 Jun 2013 12:56:03 -0400
Subject: [Numpy-discussion] TypeError when multiplying float64 and a big
	integer in Python 3.
Message-ID: <CAGzF1ueGFP=qHfCusO+evjzJMrV2F7DeftM+QD3PWXs_9mK9xQ@mail.gmail.com>

With Python 3.3.2 (64 bit), and numpy master:

>>> import numpy as np
>>> np.__version__
'1.8.0.dev-2a5c2c8'

>>> f = np.float64(1.0)
>>> i = 2**65
>>> f*i
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for *: 'numpy.float64' and 'int'

Is this the expected behavior?

The error does not occur with integers that fit in 64 bits:

>>> f*10
10.0

It also does not occur in numpy 1.7.1.

Warren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130616/dc1ebc99/attachment.html>

From warren.weckesser at gmail.com  Sun Jun 16 12:58:57 2013
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sun, 16 Jun 2013 12:58:57 -0400
Subject: [Numpy-discussion] TypeError when multiplying float64 and a big
	integer in Python 3.
In-Reply-To: <CAGzF1ueGFP=qHfCusO+evjzJMrV2F7DeftM+QD3PWXs_9mK9xQ@mail.gmail.com>
References: <CAGzF1ueGFP=qHfCusO+evjzJMrV2F7DeftM+QD3PWXs_9mK9xQ@mail.gmail.com>
Message-ID: <CAGzF1ufNkN=xqXDy8MFhZ-o_dYuRZdpU-SC1r-UnYiO26czbxw@mail.gmail.com>

On Sun, Jun 16, 2013 at 12:56 PM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

> With Python 3.3.2 (64 bit), and numpy master:
>
> >>> import numpy as np
> >>> np.__version__
> '1.8.0.dev-2a5c2c8'
>
> >>> f = np.float64(1.0)
> >>> i = 2**65
> >>> f*i
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: unsupported operand type(s) for *: 'numpy.float64' and 'int'
>
> Is this the expected behavior?
>
> The error does not occur with integers that fit in 64 bits:
>
> >>> f*10
> 10.0
>
> It also does not occur in numpy 1.7.1.
>
>

I should have checked the issues on github before mailing the list:
https://github.com/numpy/numpy/issues/3442

Warren



> Warren
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130616/a7398bbf/attachment.html>

From camilo.jimenezc at gmail.com  Sun Jun 16 14:07:13 2013
From: camilo.jimenezc at gmail.com (=?ISO-8859-1?Q?Camilo_Andr=E9s_Jimenez_Cruz?=)
Date: Sun, 16 Jun 2013 14:07:13 -0400
Subject: [Numpy-discussion] saving 3d array
In-Reply-To: <51BCAE5E.2080907@hawaii.edu>
References: <CAH6Pt5o5BRqgAZQbCQ734rPiRKc2h31F8OgHNkvcbkXOrot9-w@mail.gmail.com>
	<1371304300.53915.YahooMailMobile@web193404.mail.sg3.yahoo.com>
	<CAH6Pt5qJd0eRxxLK4=n6-STGaBZafEb3ZOvyQUP5fqb9FTgEUQ@mail.gmail.com>
	<4A26834D-E401-4E54-88D3-FBABA4957196@gmail.com>
	<51BCAE5E.2080907@hawaii.edu>
Message-ID: <CAKFp1o5XqZsp7nxq4Z+6FKpj=W6LOzytnxXqOJUaguwBZsc1xA@mail.gmail.com>

Hi!

I know it is pretty much the same as you did before, but has been useful
for me in the past. Instead of saving each array separately, just create a
dictionary and save the it, something like

d= {put_all_your_arrays_here}
savez_compressed('file.npz', **d)


-- Camilo Jim?nez


On Sat, Jun 15, 2013 at 2:11 PM, Eric Firing <efiring at hawaii.edu> wrote:

> On 2013/06/15 6:06 AM, Pierre GM wrote:
> >
> > On Jun 15, 2013, at 17:35 , Matthew Brett <matthew.brett at gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> On Sat, Jun 15, 2013 at 2:51 PM, Sudheer Joseph
> >> <sudheer.joseph at yahoo.com> wrote:
> >>>
> >>> Thank you very much for this tip.
> >>> Is there a typical way to save masked and the rest separately?. Not
> much familiar with array handling in numpy.
> >>
> >> I don't use masked array myself, but it looks like it would be
> something like:
> >>
> >> eof1_unmasked = np.array(eof1)
> >> eof1_mask = eof1.mask
> >>
> >> then you could save those two.  Maybe a more maskey person could
> comment?
> >
> > Instead of `eof1_unmasked=np.array(eof1)`, you could do `eof1_unmasked =
> eof1.data`. The '.data' attribute points to  a view of the masked array as
> a simple ndarray.
> >
> > You may also wanna try `eof1.torecords()` that will return a structured
> array with dtype `[('_data',type_of_eof1),('_mask', bool)]`.
>
> For automated saving and restoring, try this:
>
>
> http://currents.soest.hawaii.edu/hgstage/pycurrents/file/686c2802a6c4/file/npzfile.py
>
> Eric
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130616/a7e05170/attachment.html>

From efiring at hawaii.edu  Sun Jun 16 17:57:35 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Sun, 16 Jun 2013 11:57:35 -1000
Subject: [Numpy-discussion] bug fixes: which branch?
Message-ID: <51BE34CF.9070708@hawaii.edu>

What is the preferred strategy for handling bug fix PRs?  Initial fix on 
master, and then a separate PR to backport to v1.7.x?  Or the reverse? 
It doesn't look like v1.7.x is being merged into master regularly, so 
the matplotlib pattern (fix on maintenance, merge maintenance into 
master) seems not to be used here.

Thanks.

Eric


From njs at pobox.com  Sun Jun 16 18:31:35 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 16 Jun 2013 23:31:35 +0100
Subject: [Numpy-discussion] bug fixes: which branch?
In-Reply-To: <51BE34CF.9070708@hawaii.edu>
References: <51BE34CF.9070708@hawaii.edu>
Message-ID: <CAPJVwBm3=MEm9cHopaou7V9rWt=UNrPAtNVgBRM1ZK1fkunsow@mail.gmail.com>

On Sun, Jun 16, 2013 at 10:57 PM, Eric Firing <efiring at hawaii.edu> wrote:
> What is the preferred strategy for handling bug fix PRs?  Initial fix on
> master, and then a separate PR to backport to v1.7.x?  Or the reverse?
> It doesn't look like v1.7.x is being merged into master regularly, so
> the matplotlib pattern (fix on maintenance, merge maintenance into
> master) seems not to be used here.

Fix on master then backport is the current strategy.

-n


From efiring at hawaii.edu  Sun Jun 16 20:40:52 2013
From: efiring at hawaii.edu (Eric Firing)
Date: Sun, 16 Jun 2013 14:40:52 -1000
Subject: [Numpy-discussion] please close 611, 629, 2490, 2264
Message-ID: <51BE5B14.6090103@hawaii.edu>

Github issues 611, 629, and 2490 are duplicates.  611 included patches 
with a test and a fix, both of which were committed long ago, so all 
three issues should be closed.

Please see my comment on 2264 as to why that should be closed.

On 1417, please remove the "component:numpy.ma" label and add the 
"component:matrixlib" label.

Eric


From charlesr.harris at gmail.com  Sun Jun 16 20:57:22 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 16 Jun 2013 18:57:22 -0600
Subject: [Numpy-discussion] please close 611, 629, 2490, 2264
In-Reply-To: <51BE5B14.6090103@hawaii.edu>
References: <51BE5B14.6090103@hawaii.edu>
Message-ID: <CAB6mnx+5D_sDVOFXabjgZKZXuNidh4i8vhffKA1LGGwExU0eKw@mail.gmail.com>

On Sun, Jun 16, 2013 at 6:40 PM, Eric Firing <efiring at hawaii.edu> wrote:

> Github issues 611, 629, and 2490 are duplicates.  611 included patches
> with a test and a fix, both of which were committed long ago, so all
> three issues should be closed.
>
> Please see my comment on 2264 as to why that should be closed.
>
> On 1417, please remove the "component:numpy.ma" label and add the
> "component:matrixlib" label.
>

Done. Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130616/011d5fe3/attachment.html>

From sudheer.joseph at yahoo.com  Mon Jun 17 01:08:32 2013
From: sudheer.joseph at yahoo.com (Sudheer Joseph)
Date: Mon, 17 Jun 2013 13:08:32 +0800 (SGT)
Subject: [Numpy-discussion] saving 3d array
In-Reply-To: <CAKFp1o5XqZsp7nxq4Z+6FKpj=W6LOzytnxXqOJUaguwBZsc1xA@mail.gmail.com>
References: <CAH6Pt5o5BRqgAZQbCQ734rPiRKc2h31F8OgHNkvcbkXOrot9-w@mail.gmail.com>
	<1371304300.53915.YahooMailMobile@web193404.mail.sg3.yahoo.com>
	<CAH6Pt5qJd0eRxxLK4=n6-STGaBZafEb3ZOvyQUP5fqb9FTgEUQ@mail.gmail.com>
	<4A26834D-E401-4E54-88D3-FBABA4957196@gmail.com>
	<51BCAE5E.2080907@hawaii.edu>
	<CAKFp1o5XqZsp7nxq4Z+6FKpj=W6LOzytnxXqOJUaguwBZsc1xA@mail.gmail.com>
Message-ID: <1371445712.41651.YahooMailNeo@web193404.mail.sg3.yahoo.com>

Thank you Camilo,
???????????????????? It is indeed an easy way and also the link is a ready to use save method. Thanks a lot for sharing it 

with best regards,
Sudheer

?
***************************************************************
Sudheer Joseph 
Indian National Centre for Ocean Information Services
Ministry of Earth Sciences, Govt. of India
POST BOX NO: 21, IDA Jeedeemetla P.O.
Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55
Tel:+91-40-23886047(O),Fax:+91-40-23895011(O),
Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile)
E-mail:sjo.India at gmail.com;sudheer.joseph at yahoo.com
Web- http://oppamthadathil.tripod.com
***************************************************************


>________________________________
> From: Camilo Andr?s Jimenez Cruz <camilo.jimenezc at gmail.com>
>To: Discussion of Numerical Python <numpy-discussion at scipy.org> 
>Sent: Sunday, 16 June 2013 11:37 PM
>Subject: Re: [Numpy-discussion] saving 3d array
> 
>
>
>Hi!
>
>
>I know it is pretty much the same as you did before, but has been useful for me in the past. Instead of saving each array?separately, just create a dictionary and save the it, something like
>
>
>d= {put_all_your_arrays_here}
>savez_compressed('file.npz', **d)?
>
>
>
>-- Camilo Jim?nez
>
>
>
>On Sat, Jun 15, 2013 at 2:11 PM, Eric Firing <efiring at hawaii.edu> wrote:
>
>On 2013/06/15 6:06 AM, Pierre GM wrote:
>>>
>>> On Jun 15, 2013, at 17:35 , Matthew Brett <matthew.brett at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> On Sat, Jun 15, 2013 at 2:51 PM, Sudheer Joseph
>>>> <sudheer.joseph at yahoo.com> wrote:
>>>>>
>>>>> Thank you very much for this tip.
>>>>> Is there a typical way to save masked and the rest separately?. Not much familiar with array handling in numpy.
>>>>
>>>> I don't use masked array myself, but it looks like it would be something like:
>>>>
>>>> eof1_unmasked = np.array(eof1)
>>>> eof1_mask = eof1.mask
>>>>
>>>> then you could save those two. ?Maybe a more maskey person could comment?
>>>
>>> Instead of `eof1_unmasked=np.array(eof1)`, you could do `eof1_unmasked = eof1.data`. The '.data' attribute points to ?a view of the masked array as a simple ndarray.
>>>
>>> You may also wanna try `eof1.torecords()` that will return a structured array with dtype `[('_data',type_of_eof1),('_mask', bool)]`.
>>
>>For automated saving and restoring, try this:
>>
>>http://currents.soest.hawaii.edu/hgstage/pycurrents/file/686c2802a6c4/file/npzfile.py
>>
>>Eric
>>
>>
>>_______________________________________________
>>NumPy-Discussion mailing list
>>NumPy-Discussion at scipy.org
>>http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>_______________________________________________
>NumPy-Discussion mailing list
>NumPy-Discussion at scipy.org
>http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130617/0bfc3434/attachment.html>

From bala.biophysics at gmail.com  Mon Jun 17 07:07:34 2013
From: bala.biophysics at gmail.com (Bala subramanian)
Date: Mon, 17 Jun 2013 13:07:34 +0200
Subject: [Numpy-discussion] saving the lower half of matrix
Message-ID: <CA+WPOVOAv0fsPfiQDsn7szDr1jJa8EOAbHT1gS2RtfsxGQ26OA@mail.gmail.com>

Friends,
I have to save only the lower half of a symmetric matrix to a file. I used
numpy.tril to extract the lower half. However when i use 'numpy.savetxt',
numpy saves the whole matrix (with upper half values replaced as zeros)
rather than only the lower half. Any better idea to achieve this.

as an example
>>>import numpy as np
>>> d=np.arange(1,26,1)
>>> d=d.reshape(5,5)
>>> np.tril(d)
array([[ 1,  0,  0,  0,  0],
       [ 6,  7,  0,  0,  0],
       [11, 12, 13,  0,  0],
       [16, 17, 18, 19,  0],
       [21, 22, 23, 24, 25]])

>>> np.savetxt( 'test', np.tril(d))
output test also contains the zeros, what i want is only the lower half
like below.
1
6    7
11 12 13
 -     -     -    -  etc

Thanks,
Bala


-- 
C. Balasubramanian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130617/b27975dc/attachment.html>

From a.kumar at alumni.iitm.ac.in  Mon Jun 17 07:14:56 2013
From: a.kumar at alumni.iitm.ac.in (Kumar Appaiah)
Date: Mon, 17 Jun 2013 07:14:56 -0400
Subject: [Numpy-discussion] saving the lower half of matrix
In-Reply-To: <CA+WPOVOAv0fsPfiQDsn7szDr1jJa8EOAbHT1gS2RtfsxGQ26OA@mail.gmail.com>
References: <CA+WPOVOAv0fsPfiQDsn7szDr1jJa8EOAbHT1gS2RtfsxGQ26OA@mail.gmail.com>
Message-ID: <20130617111456.GA5145@bluemoon.alumni.iitm.ac.in>

On Mon, Jun 17, 2013 at 01:07:34PM +0200, Bala subramanian wrote:
>    Friends,
>    I have to save only the lower half of a symmetric matrix to a file. I used
>    numpy.tril to extract the lower half. However when i use 'numpy.savetxt',
>    numpy saves the whole matrix (with upper half values replaced as zeros)
>    rather than only the lower half. Any better idea to achieve this.
>    as an example
>    >>>import numpy as np
>    >>> d=np.arange(1,26,1)
>    >>> d=d.reshape(5,5)
>    >>> np.tril(d)
>    array([[ 1,? 0,? 0,? 0,? 0],
>    ?????? [ 6,? 7,? 0,? 0,? 0],
>    ?????? [11, 12, 13,? 0,? 0],
>    ?????? [16, 17, 18, 19,? 0],
>    ?????? [21, 22, 23, 24, 25]])
> 
>    >>> np.savetxt( 'test', np.tril(d))
>    output test also contains the zeros, what i want is only the lower half
>    like below.
>    1
>    6??? 7
>    11 12 13
>    ?-???? -???? -??? -? etc

How about saving numpy.concatenate([x[i,:i+1] for i in range(x.shape[0])])
instead? If you remove the concatenate, you'll get the individual
arrays that have the data you need as well.

Kumar
-- 
Kumar Appaiah


From matthew.brett at gmail.com  Mon Jun 17 07:19:55 2013
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 17 Jun 2013 12:19:55 +0100
Subject: [Numpy-discussion] saving the lower half of matrix
In-Reply-To: <CA+WPOVOAv0fsPfiQDsn7szDr1jJa8EOAbHT1gS2RtfsxGQ26OA@mail.gmail.com>
References: <CA+WPOVOAv0fsPfiQDsn7szDr1jJa8EOAbHT1gS2RtfsxGQ26OA@mail.gmail.com>
Message-ID: <CAH6Pt5pABKXXVF8XR1zkDS5UA6izKLkoDHu+wdo0=tJm8n1_pQ@mail.gmail.com>

Hi,

On Mon, Jun 17, 2013 at 12:07 PM, Bala subramanian
<bala.biophysics at gmail.com> wrote:
> Friends,
> I have to save only the lower half of a symmetric matrix to a file. I used
> numpy.tril to extract the lower half. However when i use 'numpy.savetxt',
> numpy saves the whole matrix (with upper half values replaced as zeros)
> rather than only the lower half. Any better idea to achieve this.
>
> as an example
>>>>import numpy as np
>>>> d=np.arange(1,26,1)
>>>> d=d.reshape(5,5)
>>>> np.tril(d)
> array([[ 1,  0,  0,  0,  0],
>        [ 6,  7,  0,  0,  0],
>        [11, 12, 13,  0,  0],
>        [16, 17, 18, 19,  0],
>        [21, 22, 23, 24, 25]])
>
>>>> np.savetxt( 'test', np.tril(d))
> output test also contains the zeros, what i want is only the lower half like
> below.
> 1
> 6    7
> 11 12 13
>  -     -     -    -  etc

Maybe you want something like:

e = d[np.tril_indices(5)]

?

Best,

Matthew


From nouiz at nouiz.org  Mon Jun 17 11:11:43 2013
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Mon, 17 Jun 2013 11:11:43 -0400
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
Message-ID: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>

Hi,

I saw that recently Julian Taylor is doing many low level optimization like
using SSE instruction. I think it is great.

Last year, Mark Florisson released the minivect[1] project that he worked
on during is master thesis. minivect is a compiler for element-wise
expression that do some of the same low level optimization that Julian is
doing in NumPy right now.

Mark did minivect in a way that allow it to be reused by other project. It
is used now by Cython and Numba I think. I had plan to reuse it in Theano,
but I didn't got the time to integrate it up to now.

What about reusing it in NumPy? I think that some of Julian optimization
aren't in minivect (I didn't check to confirm). But from I heard, minivect
don't implement reduction and there is a pull request to optimize this in
NumPy.

The advantage to concentrate some functionality in one common package is
that more project benefit from optimization done to it. (after the work to
use it first!)

How this could be done in NumPy? NumPy have its own code generator for many
dtype. We could call minivect code generator to replace some of it.

What do you think of this idea?


Sadly, I won't be able to spent time on the code for this, but I wanted to
raise the idea while people are working on that, in case it is helpful.

Fr?d?ric


[1] https://github.com/markflorisson88/minivect
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130617/058b18ed/attachment.html>

From arinkverma at gmail.com  Mon Jun 17 11:29:05 2013
From: arinkverma at gmail.com (Arink Verma)
Date: Mon, 17 Jun 2013 20:59:05 +0530
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAPJVwBnCVWAUVXcoHm0eCrB=9b86+ZHzz88vhCpKKQ4ejC1eRA@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
	<CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
	<CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>
	<CAJAs4rSR05SB1AtgFCf1b+Ln-fhAEhgQ=QESrViVoB6Q_iQg3w@mail.gmail.com>
	<CAPJVwBnCVWAUVXcoHm0eCrB=9b86+ZHzz88vhCpKKQ4ejC1eRA@mail.gmail.com>
Message-ID: <CAJAs4rRX-SJ6nVwyLhWbs_7VtD8cu6WMRzwDbGxFb0A0-iPhDg@mail.gmail.com>

Hi Nathaniel

It's a probabilistic sampling profiler, so if it doesn't have enough
> samples then it can miss things. 227 samples is way way too low. You need
> to run the profiled code for longer (a few seconds at least), and if that's
> not enough then maybe increase the sampling rate too (though be careful
> because setting this too high can also add noise).
>

I ran code '1000000000' times, which record 229115 samples. Callgraph[1]
generated is converging to *PyArray_DESCR*, and rest are unconnected.

Does it mean anything?
[1]
https://docs.google.com/file/d/0B3Pqyp8kuQw0MzRoTTNVbmlaNFE/edit?usp=sharing




-- 

Arink Verma
www.arinkverma.in
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130617/136a8266/attachment.html>

From charlesr.harris at gmail.com  Mon Jun 17 12:01:31 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 17 Jun 2013 10:01:31 -0600
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAJAs4rRX-SJ6nVwyLhWbs_7VtD8cu6WMRzwDbGxFb0A0-iPhDg@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
	<CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
	<CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>
	<CAJAs4rSR05SB1AtgFCf1b+Ln-fhAEhgQ=QESrViVoB6Q_iQg3w@mail.gmail.com>
	<CAPJVwBnCVWAUVXcoHm0eCrB=9b86+ZHzz88vhCpKKQ4ejC1eRA@mail.gmail.com>
	<CAJAs4rRX-SJ6nVwyLhWbs_7VtD8cu6WMRzwDbGxFb0A0-iPhDg@mail.gmail.com>
Message-ID: <CAB6mnxKkd7fN85wMeA0yNea2xxGU==TH4UKB5FeKsP1XXmz4Uw@mail.gmail.com>

On Mon, Jun 17, 2013 at 9:29 AM, Arink Verma <arinkverma at gmail.com> wrote:

> Hi Nathaniel
>
> It's a probabilistic sampling profiler, so if it doesn't have enough
>> samples then it can miss things. 227 samples is way way too low. You need
>> to run the profiled code for longer (a few seconds at least), and if that's
>> not enough then maybe increase the sampling rate too (though be careful
>> because setting this too high can also add noise).
>>
>
> I ran code '1000000000' times, which record 229115 samples. Callgraph[1]
> generated is converging to *PyArray_DESCR*, and rest are unconnected.
>

Not sure what you are profiling. The PyArray_DESCR call just returns a
pointer to the descr contained in an ndarray instance, so probably has
little relevance here.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130617/75dc7197/attachment.html>

From arinkverma at gmail.com  Mon Jun 17 12:08:54 2013
From: arinkverma at gmail.com (Arink Verma)
Date: Mon, 17 Jun 2013 21:38:54 +0530
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAB6mnxKkd7fN85wMeA0yNea2xxGU==TH4UKB5FeKsP1XXmz4Uw@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
	<CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
	<CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>
	<CAJAs4rSR05SB1AtgFCf1b+Ln-fhAEhgQ=QESrViVoB6Q_iQg3w@mail.gmail.com>
	<CAPJVwBnCVWAUVXcoHm0eCrB=9b86+ZHzz88vhCpKKQ4ejC1eRA@mail.gmail.com>
	<CAJAs4rRX-SJ6nVwyLhWbs_7VtD8cu6WMRzwDbGxFb0A0-iPhDg@mail.gmail.com>
	<CAB6mnxKkd7fN85wMeA0yNea2xxGU==TH4UKB5FeKsP1XXmz4Uw@mail.gmail.com>
Message-ID: <CAJAs4rSUK4HetK0DhcOUxmh5-bFG1naR24J59Ao-Kst6tty2ew@mail.gmail.com>

>
>
> Not sure what you are profiling. The PyArray_DESCR call just returns a
> pointer to the descr contained in an ndarray instance, so probably has
> little relevance here.
>

I am profiling following code

timeit.timeit('x+y',number=1000000000,setup='import numpy as np;x =
np.asarray(1.0);y = np.asarray(2.0);')

Arink Verma
www.arinkverma.in
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130617/a20e0ee5/attachment.html>

From njs at pobox.com  Mon Jun 17 12:34:47 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 17 Jun 2013 17:34:47 +0100
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAJAs4rRX-SJ6nVwyLhWbs_7VtD8cu6WMRzwDbGxFb0A0-iPhDg@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
	<CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
	<CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>
	<CAJAs4rSR05SB1AtgFCf1b+Ln-fhAEhgQ=QESrViVoB6Q_iQg3w@mail.gmail.com>
	<CAPJVwBnCVWAUVXcoHm0eCrB=9b86+ZHzz88vhCpKKQ4ejC1eRA@mail.gmail.com>
	<CAJAs4rRX-SJ6nVwyLhWbs_7VtD8cu6WMRzwDbGxFb0A0-iPhDg@mail.gmail.com>
Message-ID: <CAPJVwB=sr+vBpEa1bJDh9-R8iCSgb=_SRn9zd1CB4aAdNObUvg@mail.gmail.com>

On Mon, Jun 17, 2013 at 4:29 PM, Arink Verma <arinkverma at gmail.com> wrote:
> Hi Nathaniel
>
>> It's a probabilistic sampling profiler, so if it doesn't have enough
>> samples then it can miss things. 227 samples is way way too low. You need to
>> run the profiled code for longer (a few seconds at least), and if that's not
>> enough then maybe increase the sampling rate too (though be careful because
>> setting this too high can also add noise).
>
>
> I ran code '1000000000' times, which record 229115 samples. Callgraph[1]
> generated is converging to PyArray_DESCR, and rest are unconnected.
>
> Does it mean anything?
> [1]https://docs.google.com/file/d/0B3Pqyp8kuQw0MzRoTTNVbmlaNFE/edit?usp=sharing

I think it means that pprof is failing to walk to the stack to compute
callers. That's consistent with PyArray_DESCR being the only "call"
that it can find, because PyArray_DESCR isn't a real function, it
always gets inlined, so detecting the caller doesn't require walking
the stack.

Is your numpy compiled with -fomit-frame-pointer or something like
that? Any other weird build options used while building it? Is your
binary stripped? If you're using a distro version, do you have the
debug symbols installed? Did you compile this numpy yourself? (If not,
the simplest thing to do would be to just build it yourself using the
default settings and see if that helps.) What OS are you on?

When I run your profiling code (but being lazier and only running
12000 samples), then do 'google-pprof --pdf /path/to/python
/path/to/my/profile.prof', then I get the graph attached below.

-n
-------------- next part --------------
A non-text attachment was scrubbed...
Name: prof.pdf
Type: application/pdf
Size: 20466 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130617/02e8051b/attachment.pdf>

From jtaylor.debian at googlemail.com  Mon Jun 17 17:03:00 2013
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Mon, 17 Jun 2013 23:03:00 +0200
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
Message-ID: <51BF7984.9030205@googlemail.com>

On 17.06.2013 17:11, Fr?d?ric Bastien wrote:
> Hi,
> 
> I saw that recently Julian Taylor is doing many low level optimization
> like using SSE instruction. I think it is great.
> 
> Last year, Mark Florisson released the minivect[1] project that he
> worked on during is master thesis. minivect is a compiler for
> element-wise expression that do some of the same low level optimization
> that Julian is doing in NumPy right now.
> 
> Mark did minivect in a way that allow it to be reused by other project.
> It is used now by Cython and Numba I think. I had plan to reuse it in
> Theano, but I didn't got the time to integrate it up to now.
> 
> What about reusing it in NumPy? I think that some of Julian optimization
> aren't in minivect (I didn't check to confirm). But from I heard,
> minivect don't implement reduction and there is a pull request to
> optimize this in NumPy.

Hi,
what I vectorized is just the really easy cases of unit stride
continuous operations, so the min/max reductions which is now in numpy
is in essence pretty trivial.
minivect goes much further in optimizing general strided access and
broadcasting via loop optimizations (it seems to have a lot of overlap
with the graphite loop optimizer available in GCC [0]) so my code is
probably not of very much use to minivect.

The most interesting part in minivect for numpy is probably the
optimization of broadcasting loops which seem to be pretty inefficient
in numpy [0].

Concerning the rest I'm not sure how much of a bottleneck general
strided operations really are in common numpy using code.


I guess a similar discussion about adding an expression compiler to
numpy has already happened when numexpr was released?
If yes what was the outcome of that?


[0] http://gcc.gnu.org/wiki/Graphite
[1] ones((5000,100)) - ones((100,) spends about 40% of its time copying
stuff around in buffers


From arinkverma at gmail.com  Mon Jun 17 17:06:22 2013
From: arinkverma at gmail.com (Arink Verma)
Date: Tue, 18 Jun 2013 02:36:22 +0530
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAPJVwB=sr+vBpEa1bJDh9-R8iCSgb=_SRn9zd1CB4aAdNObUvg@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
	<CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
	<CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>
	<CAJAs4rSR05SB1AtgFCf1b+Ln-fhAEhgQ=QESrViVoB6Q_iQg3w@mail.gmail.com>
	<CAPJVwBnCVWAUVXcoHm0eCrB=9b86+ZHzz88vhCpKKQ4ejC1eRA@mail.gmail.com>
	<CAJAs4rRX-SJ6nVwyLhWbs_7VtD8cu6WMRzwDbGxFb0A0-iPhDg@mail.gmail.com>
	<CAPJVwB=sr+vBpEa1bJDh9-R8iCSgb=_SRn9zd1CB4aAdNObUvg@mail.gmail.com>
Message-ID: <CAJAs4rQv8b8TX5+3K6ED7hV+MNfMtTnAfxto3DJVnvUzdRHg-w@mail.gmail.com>

I am building numpy from source, python setup.py build --fcompiler=gnu95
then installation, python setup.py install --user, on ubuntu 13.04

for analysis results
pprof --svg /usr/bin/python py.prof



On Mon, Jun 17, 2013 at 10:04 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Mon, Jun 17, 2013 at 4:29 PM, Arink Verma <arinkverma at gmail.com> wrote:
> > Hi Nathaniel
> >
> >> It's a probabilistic sampling profiler, so if it doesn't have enough
> >> samples then it can miss things. 227 samples is way way too low. You
> need to
> >> run the profiled code for longer (a few seconds at least), and if
> that's not
> >> enough then maybe increase the sampling rate too (though be careful
> because
> >> setting this too high can also add noise).
> >
> >
> > I ran code '1000000000' times, which record 229115 samples. Callgraph[1]
> > generated is converging to PyArray_DESCR, and rest are unconnected.
> >
> > Does it mean anything?
> > [1]
> https://docs.google.com/file/d/0B3Pqyp8kuQw0MzRoTTNVbmlaNFE/edit?usp=sharing
>
> I think it means that pprof is failing to walk to the stack to compute
> callers. That's consistent with PyArray_DESCR being the only "call"
> that it can find, because PyArray_DESCR isn't a real function, it
> always gets inlined, so detecting the caller doesn't require walking
> the stack.
>
> Is your numpy compiled with -fomit-frame-pointer or something like
> that? Any other weird build options used while building it? Is your
> binary stripped? If you're using a distro version, do you have the
> debug symbols installed? Did you compile this numpy yourself? (If not,
> the simplest thing to do would be to just build it yourself using the
> default settings and see if that helps.) What OS are you on?
>
> When I run your profiling code (but being lazier and only running
> 12000 samples), then do 'google-pprof --pdf /path/to/python
> /path/to/my/profile.prof', then I get the graph attached below.
>
> -n
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

Arink Verma
www.arinkverma.in
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130618/9d994ba8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.py
Type: application/octet-stream
Size: 254 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130618/9d994ba8/attachment.obj>

From d.s.seljebotn at astro.uio.no  Mon Jun 17 17:29:09 2013
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 17 Jun 2013 23:29:09 +0200
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <51BF7984.9030205@googlemail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
Message-ID: <51BF7FA5.0@astro.uio.no>

On 06/17/2013 11:03 PM, Julian Taylor wrote:
> On 17.06.2013 17:11, Fr?d?ric Bastien wrote:
>> Hi,
>>
>> I saw that recently Julian Taylor is doing many low level optimization
>> like using SSE instruction. I think it is great.
>>
>> Last year, Mark Florisson released the minivect[1] project that he
>> worked on during is master thesis. minivect is a compiler for
>> element-wise expression that do some of the same low level optimization
>> that Julian is doing in NumPy right now.
>>
>> Mark did minivect in a way that allow it to be reused by other project.
>> It is used now by Cython and Numba I think. I had plan to reuse it in
>> Theano, but I didn't got the time to integrate it up to now.
>>
>> What about reusing it in NumPy? I think that some of Julian optimization
>> aren't in minivect (I didn't check to confirm). But from I heard,
>> minivect don't implement reduction and there is a pull request to
>> optimize this in NumPy.
>
> Hi,
> what I vectorized is just the really easy cases of unit stride
> continuous operations, so the min/max reductions which is now in numpy
> is in essence pretty trivial.
> minivect goes much further in optimizing general strided access and
> broadcasting via loop optimizations (it seems to have a lot of overlap
> with the graphite loop optimizer available in GCC [0]) so my code is
> probably not of very much use to minivect.
>
> The most interesting part in minivect for numpy is probably the
> optimization of broadcasting loops which seem to be pretty inefficient
> in numpy [0].

There's also related things like

arr + arr.T

which has much less than optimal performance in NumPy (unless there was 
recent changes). This example was one of the motivating examples for 
minivect.

Dag Sverre


From toddrjen at gmail.com  Tue Jun 18 05:33:39 2013
From: toddrjen at gmail.com (Todd)
Date: Tue, 18 Jun 2013 11:33:39 +0200
Subject: [Numpy-discussion] PEP 208 and upstreaming numpy
Message-ID: <CAFpSVpJoTkoFDWDehGjusrOwNO9bs25YZg-UYohzW877SHwhQA@mail.gmail.com>

I see that the plan to merge old Numeric into the python standard library,
PEP 208, is listed as withdrawn, although no reasons are given as far as I
could see.

Considering how mature Numpy has become, and how common it is as a
dependency for python packages, I was wondering if there were still plans
on the table to ultimately merge at least the ndarray class into the python
standard library, and if so where those plans stand.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130618/595e3327/attachment.html>

From robert.kern at gmail.com  Tue Jun 18 05:41:22 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 18 Jun 2013 10:41:22 +0100
Subject: [Numpy-discussion] PEP 208 and upstreaming numpy
In-Reply-To: <CAFpSVpJoTkoFDWDehGjusrOwNO9bs25YZg-UYohzW877SHwhQA@mail.gmail.com>
References: <CAFpSVpJoTkoFDWDehGjusrOwNO9bs25YZg-UYohzW877SHwhQA@mail.gmail.com>
Message-ID: <CAF6FJiu2f4=b7feKW60v0VNnVr42yGo1SrS_N8vitniS_CGqaQ@mail.gmail.com>

On Tue, Jun 18, 2013 at 10:33 AM, Todd <toddrjen at gmail.com> wrote:
> I see that the plan to merge old Numeric into the python standard library,
> PEP 208, is listed as withdrawn, although no reasons are given as far as I
> could see.
>
> Considering how mature Numpy has become, and how common it is as a
> dependency for python packages, I was wondering if there were still plans on
> the table to ultimately merge at least the ndarray class into the python
> standard library, and if so where those plans stand.

No, this will not be happening. Instead, the focus of effort went into
creating the enhanced buffer interface. This allows libraries to
communicate ndarray-like information using that interface instead of
sharing a particular concrete data structure.

http://www.python.org/dev/peps/pep-3118/

--
Robert Kern


From ezindy at gmail.com  Tue Jun 18 07:36:20 2013
From: ezindy at gmail.com (Egor Zindy)
Date: Tue, 18 Jun 2013 12:36:20 +0100
Subject: [Numpy-discussion] request for SWIG numpy.i users
In-Reply-To: <CAEy_T3HqPRzb_vfEtyrUzATD1ctjJ6JoZXYAJB+7cmHqHxD79w@mail.gmail.com>
References: <CABL7CQhjMqzAYfDCpDfir0_=5fF7H7tYk9o5x=p7LaZNyJtCsg@mail.gmail.com>
	<CAFQELMyV0gsn8No7KLp1fx=x0uQ8SH8gdg5GLF9hFYFGxrRLhQ@mail.gmail.com>
	<CAEy_T3HqPRzb_vfEtyrUzATD1ctjJ6JoZXYAJB+7cmHqHxD79w@mail.gmail.com>
Message-ID: <CAEy_T3G+OPMbkMuEuRfEYYTKZcCH=4LBUrrK1cW4rKdNPrtF+A@mail.gmail.com>

Dear all,

after some code clean-up / testing and a few additions, I've now sent
a pull request to numpy:master (#3451).
https://github.com/numpy/numpy/pull/3451

I also made a blog post to explain the new typemaps I would like included:
http://egorzindy.blogspot.co.uk/2013/06/new-numpyi-typemaps-for-working-with.html

Any comments appreciated.

Kind regards,
Egor


On 9 June 2013 09:20, Egor Zindy <ezindy at gmail.com> wrote:
> Thanks Tom,
>
> before we ship it, I'd love to have some feedback on the new ARGOUT_VIEWM
> type.
>
> I used to create my managed arrays using
>
>     PyObject* cap = PyCObject_FromVoidPtr((void*)(*$1), free);
>
> but since this function is deprecated, and because of Bill's background work
> to bring numpy.i up to date, I now use capsules for this:
>
>     PyObject* cap = PyCapsule_New((void*)(*$1), SWIGPY_CAPSULE_NAME,
> free_cap);
>
> ... I'll admit it took longer than expected to get this right.
>
> Would you mind testing my latest numpy.i changes hosted on github?
> https://github.com/zindy/numpy/tree/numpy-swig/doc/swig
>
> It's great that you are testing on a mac, I don't have one to test on yet.
>
>
>> It worked fine, although I use only a fraction of the capabilities that it
>> includes.
>
> Same here, but overall, it should be quit easy to choose the data type you
> need. Narrow down it down to a type between IN_ARRAY / INPLACE_ / ARGOUT_ /
> ARGOUT_VIEW/VIEWM
> http://wiki.scipy.org/Cookbook/SWIG_NumPy_examples
> http://wiki.scipy.org/Cookbook/SWIG_Memory_Deallocation (I'll update these
> when I have a sec)
>
> ... and choose the number of dimensions you need (1/2/3/4). I can't comment
> on the Fortran arrays data types though as I don't use them.
>
> Also I've introduced a few of my more esoteric data types in this week, but
> I have no idea how popular they will be. If you ever need to speed-up:
>
> a = numpy.ones((1024,1024),numpy.uint8)
> la = [a]*100
> b = numpy.mean(numpy.array(la,float),axis=0).astype(numpy.uint8)
>
> I have just the right type for that :)
> DATA_TYPE** IN_ARRAY3, DIM_TYPE DIM1, DIM_TYPE DIM2, DIM_TYPE DIM3
>
> Kind regards,
> Egor
>
>
> On 9 June 2013 03:33, Tom Krauss <thomas.p.krauss at gmail.com> wrote:
>>
>> Hi folks,
>>
>> I just downloaded Bill's numpy.i at commit 4dcb0679, and tried it out a
>> bit on some of my personal projects.  It worked fine, although I use only a
>> fraction of the capabilities that it includes.
>>
>>  And, it made the warning go away!
>>
>> I used to get this warning
>>
>> g++ -g -fPIC -c simple_wrap.cpp -I/usr/include/python2.7
>> -I/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include
>> In file included from
>> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarraytypes.h:1722,
>>                  from
>> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarrayobject.h:17,
>>                  from
>> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/arrayobject.h:15,
>>                  from simple_wrap.cpp:3062:
>>
>> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
>> warning: #warning "Using deprecated NumPy API, disable it by #defining
>> NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
>>
>> but not with this version.
>>
>> You can see which version of numpy I am using there, and that I am on Mac
>> OS X 10.8. (10.8.4 specifically)  Python 2.7.2
>>
>> I'd say SHIP IT!
>>
>> Nice work, thanks for all your work on numpy and numpy.i.
>>
>> - Tom Krauss
>>
>>
>>
>> On Tue, Jun 4, 2013 at 3:13 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> If you're using or are very familiar with SWIG and the numpy.i interface
>>> to it, please help to test and/or review
>>> https://github.com/numpy/numpy/pull/3148. It's a fairly major update to
>>> numpy.i by Bill Spotz, containing the following:
>>>   - support for 4D arrays and memory managed output arguments
>>>   - rework for the deprecated API's in numpy 1.6 and 1.7
>>>   - a bug fix in a 3D typemap
>>>   - documentation improvements
>>>
>>> It would be good to have this merged before branching 1.8.x. Not many of
>>> the regular reviewers of numpy PRs are familiar with numpy.i, therefore help
>>> would be much appreciated.
>>>
>>> Thanks,
>>> Ralf
>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>


From cournape at gmail.com  Tue Jun 18 07:40:27 2013
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 18 Jun 2013 06:40:27 -0500
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAJAs4rQv8b8TX5+3K6ED7hV+MNfMtTnAfxto3DJVnvUzdRHg-w@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
	<CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
	<CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>
	<CAJAs4rSR05SB1AtgFCf1b+Ln-fhAEhgQ=QESrViVoB6Q_iQg3w@mail.gmail.com>
	<CAPJVwBnCVWAUVXcoHm0eCrB=9b86+ZHzz88vhCpKKQ4ejC1eRA@mail.gmail.com>
	<CAJAs4rRX-SJ6nVwyLhWbs_7VtD8cu6WMRzwDbGxFb0A0-iPhDg@mail.gmail.com>
	<CAPJVwB=sr+vBpEa1bJDh9-R8iCSgb=_SRn9zd1CB4aAdNObUvg@mail.gmail.com>
	<CAJAs4rQv8b8TX5+3K6ED7hV+MNfMtTnAfxto3DJVnvUzdRHg-w@mail.gmail.com>
Message-ID: <CAGY4rcVAMV9gs5jom=bbYH0x9zC7bCOb5dTJdr2jh+Rfv1+68A@mail.gmail.com>

On Mon, Jun 17, 2013 at 4:06 PM, Arink Verma <arinkverma at gmail.com> wrote:
> I am building numpy from source, python setup.py build --fcompiler=gnu95
> then installation, python setup.py install --user, on ubuntu 13.04
>
> for analysis results
> pprof --svg /usr/bin/python py.prof

You can try using bento, with something like:

CFLAGS="-O0 -fno-omit-frame-pointer -g" bentomaker build -i -v -j4

There is a bit of overhead to set it up, though.

David


From njs at pobox.com  Tue Jun 18 08:26:12 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 18 Jun 2013 13:26:12 +0100
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAGY4rcVAMV9gs5jom=bbYH0x9zC7bCOb5dTJdr2jh+Rfv1+68A@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
	<CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
	<CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>
	<CAJAs4rSR05SB1AtgFCf1b+Ln-fhAEhgQ=QESrViVoB6Q_iQg3w@mail.gmail.com>
	<CAPJVwBnCVWAUVXcoHm0eCrB=9b86+ZHzz88vhCpKKQ4ejC1eRA@mail.gmail.com>
	<CAJAs4rRX-SJ6nVwyLhWbs_7VtD8cu6WMRzwDbGxFb0A0-iPhDg@mail.gmail.com>
	<CAPJVwB=sr+vBpEa1bJDh9-R8iCSgb=_SRn9zd1CB4aAdNObUvg@mail.gmail.com>
	<CAJAs4rQv8b8TX5+3K6ED7hV+MNfMtTnAfxto3DJVnvUzdRHg-w@mail.gmail.com>
	<CAGY4rcVAMV9gs5jom=bbYH0x9zC7bCOb5dTJdr2jh+Rfv1+68A@mail.gmail.com>
Message-ID: <CAPJVwBku7KXL=TtRGsvoqLmMGWOGRL_G=n1AdN3NDumStKb3Jg@mail.gmail.com>

On 18 Jun 2013 12:40, "David Cournapeau" <cournape at gmail.com> wrote:
>
> On Mon, Jun 17, 2013 at 4:06 PM, Arink Verma <arinkverma at gmail.com> wrote:
> > I am building numpy from source, python setup.py build --fcompiler=gnu95
> > then installation, python setup.py install --user, on ubuntu 13.04
> >
> > for analysis results
> > pprof --svg /usr/bin/python py.prof
>
> You can try using bento, with something like:
>
> CFLAGS="-O0 -fno-omit-frame-pointer -g" bentomaker build -i -v -j4

A good thing to try (and unfortunately I don't have time right now to try
and work out why my results are different than yours). Remember though that
once you get it working you'll want to re-enable optimization (-O2 or -O3
instead of -O0) before you use the profile results for anything serious.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130618/9848b9bb/attachment.html>

From zachary.pincus at yale.edu  Tue Jun 18 08:40:10 2013
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Tue, 18 Jun 2013 08:40:10 -0400
Subject: [Numpy-discussion] sampling arrays
In-Reply-To: <1371374773.4754.YahooMailNeo@web193402.mail.sg3.yahoo.com>
References: <1371365314.38661.YahooMailNeo@web193405.mail.sg3.yahoo.com>
	<1371374355.7804.8.camel@farnsworth>
	<1371374773.4754.YahooMailNeo@web193402.mail.sg3.yahoo.com>
Message-ID: <B36BEC95-63FC-4C33-9CD6-E0521504A2D9@yale.edu>

> Thank you, 
>                  No if the location ( space time or depth) of choice is not available then the function I was looking for should give an interpolated value at the choice.
> with best regards,
> Sudheer

scipy.ndimage.map_coordinates may be exactly what you want.


> ----- Original Message -----
> 
>> From: Henry Gomersall <heng at cantab.net>
>> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
>> Cc: 
>> Sent: Sunday, 16 June 2013 2:49 PM
>> Subject: Re: [Numpy-discussion] sampling arrays
>> 
>> On Sun, 2013-06-16 at 14:48 +0800, Sudheer Joseph wrote:
>>> Is it possible to sample a 4D array of numpy at given dimensions with
>>> out writing loops? ie a smart python way?
>> 
>> It's not clear how what you want to do is different from simply indexing
>> the array...?
>> 
>> Henry
>> 
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From sudheer.joseph at yahoo.com  Tue Jun 18 10:10:13 2013
From: sudheer.joseph at yahoo.com (Sudheer Joseph)
Date: Tue, 18 Jun 2013 22:10:13 +0800 (SGT)
Subject: [Numpy-discussion] sampling arrays
In-Reply-To: <B36BEC95-63FC-4C33-9CD6-E0521504A2D9@yale.edu>
References: <1371365314.38661.YahooMailNeo@web193405.mail.sg3.yahoo.com>
	<1371374355.7804.8.camel@farnsworth>
	<1371374773.4754.YahooMailNeo@web193402.mail.sg3.yahoo.com>
	<B36BEC95-63FC-4C33-9CD6-E0521504A2D9@yale.edu>
Message-ID: <1371564613.46395.YahooMailNeo@web193401.mail.sg3.yahoo.com>

Thank you? Zachary,
?????????????????? I will try this and see how this goes.

with best regards,
Sudheer


?
***************************************************************
Sudheer Joseph 
Indian National Centre for Ocean Information Services
Ministry of Earth Sciences, Govt. of India
POST BOX NO: 21, IDA Jeedeemetla P.O.
Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55
Tel:+91-40-23886047(O),Fax:+91-40-23895011(O),
Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile)
E-mail:sjo.India at gmail.com;sudheer.joseph at yahoo.com
Web- http://oppamthadathil.tripod.com
***************************************************************


>________________________________
> From: Zachary Pincus <zachary.pincus at yale.edu>
>To: Discussion of Numerical Python <numpy-discussion at scipy.org> 
>Sent: Tuesday, 18 June 2013 6:10 PM
>Subject: Re: [Numpy-discussion] sampling arrays
> 
>
>> Thank you, 
>>? ? ? ? ? ? ? ? ? No if the location ( space time or depth) of choice is not available then the function I was looking for should give an interpolated value at the choice.
>> with best regards,
>> Sudheer
>
>scipy.ndimage.map_coordinates may be exactly what you want.
>
>
>> ----- Original Message -----
>> 
>>> From: Henry Gomersall <heng at cantab.net>
>>> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
>>> Cc: 
>>> Sent: Sunday, 16 June 2013 2:49 PM
>>> Subject: Re: [Numpy-discussion] sampling arrays
>>> 
>>> On Sun, 2013-06-16 at 14:48 +0800, Sudheer Joseph wrote:
>>>> Is it possible to sample a 4D array of numpy at given dimensions with
>>>> out writing loops? ie a smart python way?
>>> 
>>> It's not clear how what you want to do is different from simply indexing
>>> the array...?
>>> 
>>> Henry
>>> 
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> 
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>_______________________________________________
>NumPy-Discussion mailing list
>NumPy-Discussion at scipy.org
>http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130618/7707239c/attachment.html>

From jeremy.solbrig at nrlmry.navy.mil  Tue Jun 18 15:19:34 2013
From: jeremy.solbrig at nrlmry.navy.mil (Jeremy Solbrig)
Date: Tue, 18 Jun 2013 12:19:34 -0700
Subject: [Numpy-discussion] f2py: Exposing parameters from "used" modules
In-Reply-To: <51BF8F4F.1070703@nrlmry.navy.mil>
References: <51BF8F4F.1070703@nrlmry.navy.mil>
Message-ID: <51C0B2C6.9060300@nrlmry.navy.mil>

Hi all,

I assume that this question is addressed somewhere, but I have spent an 
inordinate amount of time looking around for the answer including 
digging into the source code a bit.  I have tried to put the basic 
problem in the first paragraph.  The rest of the email shows a basic 
example of the problem.  This question is also available here:

http://stackoverflow.com/questions/17156125/f2py-exposing-parameters-from-used-modules

I am attempting to compile a module that contains a "use" statement that 
points to another, more general, module.  I would prefer to keep the 
used module separate so that it can be used in several "packages" as a 
set of general settings.  When I compile the two modules using f2py 
everything works as advertised from the fortran side, but from the 
python side the use statement appears to be ignored.  If I allow f2py to 
generate a signature file, the file contains a use statement as is 
appropriate, but if I complete the compilation and import from the 
resulting library the parameters from the used module are not available 
in the module that contains the use statement.  Below are two modules 
illustrating the situation:

#Contents of test.f90
MODULE test
     INTEGER, PARAMETER :: a = 1
END MODULE test

#Contents of test2.f90
MODULE test2
     USE test
     INTEGER, PARAMETER :: b = 2
END MODULE test2

In order to show the intermediate step I ran "f2py -h test.pyf test.f90 
test2.f90".  The following signature file is generated; note that the 
"test2" module contains "use test":

!    -*- f90 -*-
! Note: the context of this file is case sensitive.

python module test ! in
     interface  ! in :test
         module test ! in :test:test.f90
             integer, parameter,optional :: a=1
         end module test
         module test2 ! in :test:test2.f90
             use test
             integer, parameter,optional :: b=2
         end module test2
     end interface
end python module test

! This file was auto-generated with f2py (version:2).
! See http://cens.ioc.ee/projects/f2py2e/

If I now compile with "f2py --fcompiler=gfortran -c test.pyf test.f90 
test2.f90" I obtain test.so (same as running "f2py --fcompiler=gfortran 
-m test -c test.f90 test2.f90 without creating the signature file 
first).  Importing from this library in python exposes test.test.a and 
test.test2.b, but does not expose test.test2.a as can be seen here:

In [1]: import test

In [2]: print test.test.a
1

In [3]: print test.test2.b
2

In [4]: print test.test2.a
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/users/solbrig/svn_checkouts/inversion/satmet/branches/solbrig/rootpath/data/users/GeoIPS/src/test/<ipython-input-4-bffcf464e408> 
in <module>()
----> 1 print test.test2.a

AttributeError: a

Just to illustrate that "b" is defined properly in test2 from the 
perspective of fortran, the following code uses test2 and prints both 
"a" and "b":

SUBROUTINE run_test()
     USE test2
     IMPLICIT NONE
     print *, "a = ", a
     print *, "b = ", b
END SUBROUTINE run_test

After compiling with "f2py -m run_test -c test.f90 test2.f90 
run_test.f90" and obtaining run_test.so, run_test can be imported in 
python and works as expected:

In [1]: import run_test

In [2]: run_test.run_test()
  a =            1
  b =            2


Any help with this issue would be greatly appreciated.

Thanks,
Jeremy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130618/423103d2/attachment.html>

From nouiz at nouiz.org  Tue Jun 18 20:43:53 2013
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Tue, 18 Jun 2013 20:43:53 -0400
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <51BF7984.9030205@googlemail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
Message-ID: <CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>

Hi,


On Mon, Jun 17, 2013 at 5:03 PM, Julian Taylor <
jtaylor.debian at googlemail.com> wrote:

> On 17.06.2013 17:11, Fr?d?ric Bastien wrote:
> > Hi,
> >
> > I saw that recently Julian Taylor is doing many low level optimization
> > like using SSE instruction. I think it is great.
> >
> > Last year, Mark Florisson released the minivect[1] project that he
> > worked on during is master thesis. minivect is a compiler for
> > element-wise expression that do some of the same low level optimization
> > that Julian is doing in NumPy right now.
> >
> > Mark did minivect in a way that allow it to be reused by other project.
> > It is used now by Cython and Numba I think. I had plan to reuse it in
> > Theano, but I didn't got the time to integrate it up to now.
> >
> > What about reusing it in NumPy? I think that some of Julian optimization
> > aren't in minivect (I didn't check to confirm). But from I heard,
> > minivect don't implement reduction and there is a pull request to
> > optimize this in NumPy.
>
> Hi,
> what I vectorized is just the really easy cases of unit stride
> continuous operations, so the min/max reductions which is now in numpy
> is in essence pretty trivial.
> minivect goes much further in optimizing general strided access and
> broadcasting via loop optimizations (it seems to have a lot of overlap
> with the graphite loop optimizer available in GCC [0]) so my code is
> probably not of very much use to minivect.
>
> The most interesting part in minivect for numpy is probably the
> optimization of broadcasting loops which seem to be pretty inefficient
> in numpy [0].
>
> Concerning the rest I'm not sure how much of a bottleneck general
> strided operations really are in common numpy using code.
>
>
> I guess a similar discussion about adding an expression compiler to
> numpy has already happened when numexpr was released?
> If yes what was the outcome of that?
>

I don't recall a discussion when numexpr was done as this is before I read
this list. numexpr do optimization that can't be done by NumPy: fusing
element-wise operation in one call. So I don't see how it could be done to
reuse it in NumPy.

You call your optimization trivial, but I don't. In the git log of NumPy,
the first commit is in 2001. It is the first time someone do this in 12
years! Also, this give 1.5-8x speed up (from memory from your PR
description). This is not negligible. But how much time did you spend on
them? Also, some of them are processor dependent, how many people in this
list already have done this? I suppose not many.

Yes, your optimization don't cover all cases that minivect do. I see 2
level of optimization. 1) The inner loop/contiguous cases, 2) the strided,
broadcasted level. We don't need all optimization being done for them to be
useful. Any of them are useful.

So what I think is that we could reuse/share that work. NumPy have c code
generator. They could call minivect code generator for some of them when
compiling NumPy. This will make optimization done to those code generator
reused by more people. For example, when new processor are launched, we
will need only 1 place to change for many projects. Or for example, it the
call to MKL vector library is done there, more people will benefit from it.
Right now, only numexpr do it.

About the level 2 optimization (strides, broadcast), I never read NumPy
code that deal with that. Do someone that know it have an idea if it would
be possible to reuse minivect for this?

Fr?d?ric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130618/530f9738/attachment.html>

From matthew.brett at gmail.com  Wed Jun 19 07:45:42 2013
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 19 Jun 2013 12:45:42 +0100
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
	<CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
Message-ID: <CAH6Pt5q4eeDTP_4DY2pQ3Cy0Qz0j_mfzUMeDwT1ARZ8ajta56Q@mail.gmail.com>

Hi,

On Wed, Jun 19, 2013 at 1:43 AM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
> Hi,
>
>
> On Mon, Jun 17, 2013 at 5:03 PM, Julian Taylor
> <jtaylor.debian at googlemail.com> wrote:
>>
>> On 17.06.2013 17:11, Fr?d?ric Bastien wrote:
>> > Hi,
>> >
>> > I saw that recently Julian Taylor is doing many low level optimization
>> > like using SSE instruction. I think it is great.
>> >
>> > Last year, Mark Florisson released the minivect[1] project that he
>> > worked on during is master thesis. minivect is a compiler for
>> > element-wise expression that do some of the same low level optimization
>> > that Julian is doing in NumPy right now.
>> >
>> > Mark did minivect in a way that allow it to be reused by other project.
>> > It is used now by Cython and Numba I think. I had plan to reuse it in
>> > Theano, but I didn't got the time to integrate it up to now.
>> >
>> > What about reusing it in NumPy? I think that some of Julian optimization
>> > aren't in minivect (I didn't check to confirm). But from I heard,
>> > minivect don't implement reduction and there is a pull request to
>> > optimize this in NumPy.
>>
>> Hi,
>> what I vectorized is just the really easy cases of unit stride
>> continuous operations, so the min/max reductions which is now in numpy
>> is in essence pretty trivial.
>> minivect goes much further in optimizing general strided access and
>> broadcasting via loop optimizations (it seems to have a lot of overlap
>> with the graphite loop optimizer available in GCC [0]) so my code is
>> probably not of very much use to minivect.
>>
>> The most interesting part in minivect for numpy is probably the
>> optimization of broadcasting loops which seem to be pretty inefficient
>> in numpy [0].
>>
>> Concerning the rest I'm not sure how much of a bottleneck general
>> strided operations really are in common numpy using code.
>>
>>
>> I guess a similar discussion about adding an expression compiler to
>> numpy has already happened when numexpr was released?
>> If yes what was the outcome of that?
>
>
> I don't recall a discussion when numexpr was done as this is before I read
> this list. numexpr do optimization that can't be done by NumPy: fusing
> element-wise operation in one call. So I don't see how it could be done to
> reuse it in NumPy.
>
> You call your optimization trivial, but I don't. In the git log of NumPy,
> the first commit is in 2001. It is the first time someone do this in 12
> years! Also, this give 1.5-8x speed up (from memory from your PR
> description). This is not negligible. But how much time did you spend on
> them? Also, some of them are processor dependent, how many people in this
> list already have done this? I suppose not many.
>
> Yes, your optimization don't cover all cases that minivect do. I see 2 level
> of optimization. 1) The inner loop/contiguous cases, 2) the strided,
> broadcasted level. We don't need all optimization being done for them to be
> useful. Any of them are useful.
>
> So what I think is that we could reuse/share that work. NumPy have c code
> generator. They could call minivect code generator for some of them when
> compiling NumPy. This will make optimization done to those code generator
> reused by more people. For example, when new processor are launched, we will
> need only 1 place to change for many projects. Or for example, it the call
> to MKL vector library is done there, more people will benefit from it. Right
> now, only numexpr do it.
>
> About the level 2 optimization (strides, broadcast), I never read NumPy code
> that deal with that. Do someone that know it have an idea if it would be
> possible to reuse minivect for this?

Would someone be able to guide some of the numpy C experts into a room
to do some thinking / writing on this at the scipy conference?

I completely agree that these kind of optimizations and code sharing
seem likely to be very important for the future.

I'm not at the conference, but if there's anything I can do to help,
please someone let me know.

Cheers,

Matthew


From charlesr.harris at gmail.com  Wed Jun 19 08:48:32 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 19 Jun 2013 06:48:32 -0600
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <CAH6Pt5q4eeDTP_4DY2pQ3Cy0Qz0j_mfzUMeDwT1ARZ8ajta56Q@mail.gmail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
	<CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
	<CAH6Pt5q4eeDTP_4DY2pQ3Cy0Qz0j_mfzUMeDwT1ARZ8ajta56Q@mail.gmail.com>
Message-ID: <CAB6mnxLh2ens_omTWpWu9PUiMFnTWGDs-bsvN-_qV-UUX9vaGg@mail.gmail.com>

On Wed, Jun 19, 2013 at 5:45 AM, Matthew Brett <matthew.brett at gmail.com>wrote:

> Hi,
>
> On Wed, Jun 19, 2013 at 1:43 AM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
> > Hi,
> >
> >
> > On Mon, Jun 17, 2013 at 5:03 PM, Julian Taylor
> > <jtaylor.debian at googlemail.com> wrote:
> >>
> >> On 17.06.2013 17:11, Fr?d?ric Bastien wrote:
> >> > Hi,
> >> >
> >> > I saw that recently Julian Taylor is doing many low level optimization
> >> > like using SSE instruction. I think it is great.
> >> >
> >> > Last year, Mark Florisson released the minivect[1] project that he
> >> > worked on during is master thesis. minivect is a compiler for
> >> > element-wise expression that do some of the same low level
> optimization
> >> > that Julian is doing in NumPy right now.
> >> >
> >> > Mark did minivect in a way that allow it to be reused by other
> project.
> >> > It is used now by Cython and Numba I think. I had plan to reuse it in
> >> > Theano, but I didn't got the time to integrate it up to now.
> >> >
> >> > What about reusing it in NumPy? I think that some of Julian
> optimization
> >> > aren't in minivect (I didn't check to confirm). But from I heard,
> >> > minivect don't implement reduction and there is a pull request to
> >> > optimize this in NumPy.
> >>
> >> Hi,
> >> what I vectorized is just the really easy cases of unit stride
> >> continuous operations, so the min/max reductions which is now in numpy
> >> is in essence pretty trivial.
> >> minivect goes much further in optimizing general strided access and
> >> broadcasting via loop optimizations (it seems to have a lot of overlap
> >> with the graphite loop optimizer available in GCC [0]) so my code is
> >> probably not of very much use to minivect.
> >>
> >> The most interesting part in minivect for numpy is probably the
> >> optimization of broadcasting loops which seem to be pretty inefficient
> >> in numpy [0].
> >>
> >> Concerning the rest I'm not sure how much of a bottleneck general
> >> strided operations really are in common numpy using code.
> >>
> >>
> >> I guess a similar discussion about adding an expression compiler to
> >> numpy has already happened when numexpr was released?
> >> If yes what was the outcome of that?
> >
> >
> > I don't recall a discussion when numexpr was done as this is before I
> read
> > this list. numexpr do optimization that can't be done by NumPy: fusing
> > element-wise operation in one call. So I don't see how it could be done
> to
> > reuse it in NumPy.
> >
> > You call your optimization trivial, but I don't. In the git log of NumPy,
> > the first commit is in 2001. It is the first time someone do this in 12
> > years! Also, this give 1.5-8x speed up (from memory from your PR
> > description). This is not negligible. But how much time did you spend on
> > them? Also, some of them are processor dependent, how many people in this
> > list already have done this? I suppose not many.
> >
> > Yes, your optimization don't cover all cases that minivect do. I see 2
> level
> > of optimization. 1) The inner loop/contiguous cases, 2) the strided,
> > broadcasted level. We don't need all optimization being done for them to
> be
> > useful. Any of them are useful.
> >
> > So what I think is that we could reuse/share that work. NumPy have c code
> > generator. They could call minivect code generator for some of them when
> > compiling NumPy. This will make optimization done to those code generator
> > reused by more people. For example, when new processor are launched, we
> will
> > need only 1 place to change for many projects. Or for example, it the
> call
> > to MKL vector library is done there, more people will benefit from it.
> Right
> > now, only numexpr do it.
> >
> > About the level 2 optimization (strides, broadcast), I never read NumPy
> code
> > that deal with that. Do someone that know it have an idea if it would be
> > possible to reuse minivect for this?
>
> Would someone be able to guide some of the numpy C experts into a room
> to do some thinking / writing on this at the scipy conference?
>
> I completely agree that these kind of optimizations and code sharing
> seem likely to be very important for the future.
>
> I'm not at the conference, but if there's anything I can do to help,
> please someone let me know.
>

Concerning the future development of numpy, I'd also suggest that we look
at libdynd <https://github.com/ContinuumIO/libdynd>. It looks to me like it
is reaching a level of maturity where it is worth trying to plan out a long
term path to merger.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130619/38f0fba7/attachment.html>

From cournape at gmail.com  Wed Jun 19 22:07:16 2013
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 19 Jun 2013 21:07:16 -0500
Subject: [Numpy-discussion] Profiling (was GSoC : Performance parity
 between numpy arrays and Python scalars)
In-Reply-To: <CAPJVwBku7KXL=TtRGsvoqLmMGWOGRL_G=n1AdN3NDumStKb3Jg@mail.gmail.com>
References: <CAGY4rcVCN9v4fjWO8QtubQvvC8aTvxsYzBya+QzL3_82C5b4YQ@mail.gmail.com>
	<CAPJVwBkh0jda763ps+PF+yHrSJNNeSYdd7=bjUTi-jwjAp-vwg@mail.gmail.com>
	<CAGY4rcUGWCCqegE16bzShZnoQDJ3TtC6kEvXNPkh5nB5Z-N8Sw@mail.gmail.com>
	<CAPJVwB=22CD=SuHHK_BB+06sroaeVSMA+x5y62Cpg0iHgCyAQw@mail.gmail.com>
	<CAJAs4rTZTxbPg0QjtgbEE0r6XZxKGjnEJd=kfZ+JXLZfO-v7Yw@mail.gmail.com>
	<CAPJVwB=H0CZyLOxMN+8RMdw_oW5xoRDoDSKQCSkHCNW1hBcxgQ@mail.gmail.com>
	<CAJAs4rSR05SB1AtgFCf1b+Ln-fhAEhgQ=QESrViVoB6Q_iQg3w@mail.gmail.com>
	<CAPJVwBnCVWAUVXcoHm0eCrB=9b86+ZHzz88vhCpKKQ4ejC1eRA@mail.gmail.com>
	<CAJAs4rRX-SJ6nVwyLhWbs_7VtD8cu6WMRzwDbGxFb0A0-iPhDg@mail.gmail.com>
	<CAPJVwB=sr+vBpEa1bJDh9-R8iCSgb=_SRn9zd1CB4aAdNObUvg@mail.gmail.com>
	<CAJAs4rQv8b8TX5+3K6ED7hV+MNfMtTnAfxto3DJVnvUzdRHg-w@mail.gmail.com>
	<CAGY4rcVAMV9gs5jom=bbYH0x9zC7bCOb5dTJdr2jh+Rfv1+68A@mail.gmail.com>
	<CAPJVwBku7KXL=TtRGsvoqLmMGWOGRL_G=n1AdN3NDumStKb3Jg@mail.gmail.com>
Message-ID: <CAGY4rcUB1Uu0sDNsc=3cYSamaRFnK7-sSw_LrTbrOSDtoAN_qA@mail.gmail.com>

On Tue, Jun 18, 2013 at 7:26 AM, Nathaniel Smith <njs at pobox.com> wrote:

> On 18 Jun 2013 12:40, "David Cournapeau" <cournape at gmail.com> wrote:
> >
> > On Mon, Jun 17, 2013 at 4:06 PM, Arink Verma <arinkverma at gmail.com>
> wrote:
> > > I am building numpy from source, python setup.py build
> --fcompiler=gnu95
> > > then installation, python setup.py install --user, on ubuntu 13.04
> > >
> > > for analysis results
> > > pprof --svg /usr/bin/python py.prof
> >
> > You can try using bento, with something like:
> >
> > CFLAGS="-O0 -fno-omit-frame-pointer -g" bentomaker build -i -v -j4
>
> A good thing to try (and unfortunately I don't have time right now to try
> and work out why my results are different than yours). Remember though that
> once you get it working you'll want to re-enable optimization (-O2 or -O3
> instead of -O0) before you use the profile results for anything serious.
>

With the above compilation options, I get a flame graph that show similar
patterns to what you get as well:
https://s3.amazonaws.com/scipy-2013/diving_into_numpy/perf-numpy.svg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130619/50e10e98/attachment.html>

From thomas.p.krauss at gmail.com  Thu Jun 20 08:26:03 2013
From: thomas.p.krauss at gmail.com (Tom Krauss)
Date: Thu, 20 Jun 2013 07:26:03 -0500
Subject: [Numpy-discussion] request for SWIG numpy.i users
In-Reply-To: <CAEy_T3G+OPMbkMuEuRfEYYTKZcCH=4LBUrrK1cW4rKdNPrtF+A@mail.gmail.com>
References: <CABL7CQhjMqzAYfDCpDfir0_=5fF7H7tYk9o5x=p7LaZNyJtCsg@mail.gmail.com>
	<CAFQELMyV0gsn8No7KLp1fx=x0uQ8SH8gdg5GLF9hFYFGxrRLhQ@mail.gmail.com>
	<CAEy_T3HqPRzb_vfEtyrUzATD1ctjJ6JoZXYAJB+7cmHqHxD79w@mail.gmail.com>
	<CAEy_T3G+OPMbkMuEuRfEYYTKZcCH=4LBUrrK1cW4rKdNPrtF+A@mail.gmail.com>
Message-ID: <CAFQELMwbLA8USibQJ0838mLvq5JtZP8hVZWb41i-XW_RHLjxUA@mail.gmail.com>

Hi Egor - I just read through your blog post, thanks for describing those
new list-of-array type maps.  It helps to see your motivation and some
examples.  I'll keep it in mind if I ever have a list of large arrays to
process for which creating a numpy array first is not desirable.
- Tom


On Tue, Jun 18, 2013 at 6:36 AM, Egor Zindy <ezindy at gmail.com> wrote:

> Dear all,
>
> after some code clean-up / testing and a few additions, I've now sent
> a pull request to numpy:master (#3451).
> https://github.com/numpy/numpy/pull/3451
>
> I also made a blog post to explain the new typemaps I would like included:
>
> http://egorzindy.blogspot.co.uk/2013/06/new-numpyi-typemaps-for-working-with.html
>
> Any comments appreciated.
>
> Kind regards,
> Egor
>
>
> On 9 June 2013 09:20, Egor Zindy <ezindy at gmail.com> wrote:
> > Thanks Tom,
> >
> > before we ship it, I'd love to have some feedback on the new ARGOUT_VIEWM
> > type.
> >
> > I used to create my managed arrays using
> >
> >     PyObject* cap = PyCObject_FromVoidPtr((void*)(*$1), free);
> >
> > but since this function is deprecated, and because of Bill's background
> work
> > to bring numpy.i up to date, I now use capsules for this:
> >
> >     PyObject* cap = PyCapsule_New((void*)(*$1), SWIGPY_CAPSULE_NAME,
> > free_cap);
> >
> > ... I'll admit it took longer than expected to get this right.
> >
> > Would you mind testing my latest numpy.i changes hosted on github?
> > https://github.com/zindy/numpy/tree/numpy-swig/doc/swig
> >
> > It's great that you are testing on a mac, I don't have one to test on
> yet.
> >
> >
> >> It worked fine, although I use only a fraction of the capabilities that
> it
> >> includes.
> >
> > Same here, but overall, it should be quit easy to choose the data type
> you
> > need. Narrow down it down to a type between IN_ARRAY / INPLACE_ /
> ARGOUT_ /
> > ARGOUT_VIEW/VIEWM
> > http://wiki.scipy.org/Cookbook/SWIG_NumPy_examples
> > http://wiki.scipy.org/Cookbook/SWIG_Memory_Deallocation (I'll update
> these
> > when I have a sec)
> >
> > ... and choose the number of dimensions you need (1/2/3/4). I can't
> comment
> > on the Fortran arrays data types though as I don't use them.
> >
> > Also I've introduced a few of my more esoteric data types in this week,
> but
> > I have no idea how popular they will be. If you ever need to speed-up:
> >
> > a = numpy.ones((1024,1024),numpy.uint8)
> > la = [a]*100
> > b = numpy.mean(numpy.array(la,float),axis=0).astype(numpy.uint8)
> >
> > I have just the right type for that :)
> > DATA_TYPE** IN_ARRAY3, DIM_TYPE DIM1, DIM_TYPE DIM2, DIM_TYPE DIM3
> >
> > Kind regards,
> > Egor
> >
> >
> > On 9 June 2013 03:33, Tom Krauss <thomas.p.krauss at gmail.com> wrote:
> >>
> >> Hi folks,
> >>
> >> I just downloaded Bill's numpy.i at commit 4dcb0679, and tried it out a
> >> bit on some of my personal projects.  It worked fine, although I use
> only a
> >> fraction of the capabilities that it includes.
> >>
> >>  And, it made the warning go away!
> >>
> >> I used to get this warning
> >>
> >> g++ -g -fPIC -c simple_wrap.cpp -I/usr/include/python2.7
> >>
> -I/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include
> >> In file included from
> >>
> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarraytypes.h:1722,
> >>                  from
> >>
> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarrayobject.h:17,
> >>                  from
> >>
> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/arrayobject.h:15,
> >>                  from simple_wrap.cpp:3062:
> >>
> >>
> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
> >> warning: #warning "Using deprecated NumPy API, disable it by #defining
> >> NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
> >>
> >> but not with this version.
> >>
> >> You can see which version of numpy I am using there, and that I am on
> Mac
> >> OS X 10.8. (10.8.4 specifically)  Python 2.7.2
> >>
> >> I'd say SHIP IT!
> >>
> >> Nice work, thanks for all your work on numpy and numpy.i.
> >>
> >> - Tom Krauss
> >>
> >>
> >>
> >> On Tue, Jun 4, 2013 at 3:13 PM, Ralf Gommers <ralf.gommers at gmail.com>
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> If you're using or are very familiar with SWIG and the numpy.i
> interface
> >>> to it, please help to test and/or review
> >>> https://github.com/numpy/numpy/pull/3148. It's a fairly major update
> to
> >>> numpy.i by Bill Spotz, containing the following:
> >>>   - support for 4D arrays and memory managed output arguments
> >>>   - rework for the deprecated API's in numpy 1.6 and 1.7
> >>>   - a bug fix in a 3D typemap
> >>>   - documentation improvements
> >>>
> >>> It would be good to have this merged before branching 1.8.x. Not many
> of
> >>> the regular reviewers of numpy PRs are familiar with numpy.i,
> therefore help
> >>> would be much appreciated.
> >>>
> >>> Thanks,
> >>> Ralf
> >>>
> >>>
> >>> _______________________________________________
> >>> NumPy-Discussion mailing list
> >>> NumPy-Discussion at scipy.org
> >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>>
> >>
> >>
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130620/1b92a6a0/attachment.html>

From nouiz at nouiz.org  Thu Jun 20 15:45:36 2013
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Thu, 20 Jun 2013 15:45:36 -0400
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <CAB6mnxLh2ens_omTWpWu9PUiMFnTWGDs-bsvN-_qV-UUX9vaGg@mail.gmail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
	<CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
	<CAH6Pt5q4eeDTP_4DY2pQ3Cy0Qz0j_mfzUMeDwT1ARZ8ajta56Q@mail.gmail.com>
	<CAB6mnxLh2ens_omTWpWu9PUiMFnTWGDs-bsvN-_qV-UUX9vaGg@mail.gmail.com>
Message-ID: <CADKKbtjD+-VcHB28MGQ6sWm3Obr8bE-m1CKaKq0L=P3Nngom1g@mail.gmail.com>

I didn't know about this project. It is interresting.

Is some of you discuss this at the scipy conference, it would be
appreciated if you write here a summary of that. I won't be there this year.

thanks

Fr?d?ric


On Wed, Jun 19, 2013 at 8:48 AM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Wed, Jun 19, 2013 at 5:45 AM, Matthew Brett <matthew.brett at gmail.com>wrote:
>
>> Hi,
>>
>> On Wed, Jun 19, 2013 at 1:43 AM, Fr?d?ric Bastien <nouiz at nouiz.org>
>> wrote:
>> > Hi,
>> >
>> >
>> > On Mon, Jun 17, 2013 at 5:03 PM, Julian Taylor
>> > <jtaylor.debian at googlemail.com> wrote:
>> >>
>> >> On 17.06.2013 17:11, Fr?d?ric Bastien wrote:
>> >> > Hi,
>> >> >
>> >> > I saw that recently Julian Taylor is doing many low level
>> optimization
>> >> > like using SSE instruction. I think it is great.
>> >> >
>> >> > Last year, Mark Florisson released the minivect[1] project that he
>> >> > worked on during is master thesis. minivect is a compiler for
>> >> > element-wise expression that do some of the same low level
>> optimization
>> >> > that Julian is doing in NumPy right now.
>> >> >
>> >> > Mark did minivect in a way that allow it to be reused by other
>> project.
>> >> > It is used now by Cython and Numba I think. I had plan to reuse it in
>> >> > Theano, but I didn't got the time to integrate it up to now.
>> >> >
>> >> > What about reusing it in NumPy? I think that some of Julian
>> optimization
>> >> > aren't in minivect (I didn't check to confirm). But from I heard,
>> >> > minivect don't implement reduction and there is a pull request to
>> >> > optimize this in NumPy.
>> >>
>> >> Hi,
>> >> what I vectorized is just the really easy cases of unit stride
>> >> continuous operations, so the min/max reductions which is now in numpy
>> >> is in essence pretty trivial.
>> >> minivect goes much further in optimizing general strided access and
>> >> broadcasting via loop optimizations (it seems to have a lot of overlap
>> >> with the graphite loop optimizer available in GCC [0]) so my code is
>> >> probably not of very much use to minivect.
>> >>
>> >> The most interesting part in minivect for numpy is probably the
>> >> optimization of broadcasting loops which seem to be pretty inefficient
>> >> in numpy [0].
>> >>
>> >> Concerning the rest I'm not sure how much of a bottleneck general
>> >> strided operations really are in common numpy using code.
>> >>
>> >>
>> >> I guess a similar discussion about adding an expression compiler to
>> >> numpy has already happened when numexpr was released?
>> >> If yes what was the outcome of that?
>> >
>> >
>> > I don't recall a discussion when numexpr was done as this is before I
>> read
>> > this list. numexpr do optimization that can't be done by NumPy: fusing
>> > element-wise operation in one call. So I don't see how it could be done
>> to
>> > reuse it in NumPy.
>> >
>> > You call your optimization trivial, but I don't. In the git log of
>> NumPy,
>> > the first commit is in 2001. It is the first time someone do this in 12
>> > years! Also, this give 1.5-8x speed up (from memory from your PR
>> > description). This is not negligible. But how much time did you spend on
>> > them? Also, some of them are processor dependent, how many people in
>> this
>> > list already have done this? I suppose not many.
>> >
>> > Yes, your optimization don't cover all cases that minivect do. I see 2
>> level
>> > of optimization. 1) The inner loop/contiguous cases, 2) the strided,
>> > broadcasted level. We don't need all optimization being done for them
>> to be
>> > useful. Any of them are useful.
>> >
>> > So what I think is that we could reuse/share that work. NumPy have c
>> code
>> > generator. They could call minivect code generator for some of them when
>> > compiling NumPy. This will make optimization done to those code
>> generator
>> > reused by more people. For example, when new processor are launched, we
>> will
>> > need only 1 place to change for many projects. Or for example, it the
>> call
>> > to MKL vector library is done there, more people will benefit from it.
>> Right
>> > now, only numexpr do it.
>> >
>> > About the level 2 optimization (strides, broadcast), I never read NumPy
>> code
>> > that deal with that. Do someone that know it have an idea if it would be
>> > possible to reuse minivect for this?
>>
>> Would someone be able to guide some of the numpy C experts into a room
>> to do some thinking / writing on this at the scipy conference?
>>
>> I completely agree that these kind of optimizations and code sharing
>> seem likely to be very important for the future.
>>
>> I'm not at the conference, but if there's anything I can do to help,
>> please someone let me know.
>>
>
> Concerning the future development of numpy, I'd also suggest that we look
> at libdynd <https://github.com/ContinuumIO/libdynd>. It looks to me like
> it is reaching a level of maturity where it is worth trying to plan out a
> long term path to merger.
>
> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130620/d5140918/attachment.html>

From ezindy at gmail.com  Thu Jun 20 15:58:17 2013
From: ezindy at gmail.com (Egor Zindy)
Date: Thu, 20 Jun 2013 20:58:17 +0100
Subject: [Numpy-discussion] request for SWIG numpy.i users
In-Reply-To: <CAFQELMwbLA8USibQJ0838mLvq5JtZP8hVZWb41i-XW_RHLjxUA@mail.gmail.com>
References: <CABL7CQhjMqzAYfDCpDfir0_=5fF7H7tYk9o5x=p7LaZNyJtCsg@mail.gmail.com>
	<CAFQELMyV0gsn8No7KLp1fx=x0uQ8SH8gdg5GLF9hFYFGxrRLhQ@mail.gmail.com>
	<CAEy_T3HqPRzb_vfEtyrUzATD1ctjJ6JoZXYAJB+7cmHqHxD79w@mail.gmail.com>
	<CAEy_T3G+OPMbkMuEuRfEYYTKZcCH=4LBUrrK1cW4rKdNPrtF+A@mail.gmail.com>
	<CAFQELMwbLA8USibQJ0838mLvq5JtZP8hVZWb41i-XW_RHLjxUA@mail.gmail.com>
Message-ID: <CAEy_T3HoHB2ziJWW0vBn0OWiYhptPN=NsBGEsaekrJ1b2RNEkw@mail.gmail.com>

Thanks Tom,

> It helps to see your motivation and some
> examples.  I'll keep it in mind if I ever have a list of large arrays to
> process for which creating a numpy array first is not desirable.

by documenting numpy.i typemaps, I'm hoping to get a good feel for
whichever use they are best suited for. Doing it in the open means I
get valuable feedback in return.

Cheers,
Egor

On 20 June 2013 13:26, Tom Krauss <thomas.p.krauss at gmail.com> wrote:
> Hi Egor - I just read through your blog post, thanks for describing those
> new list-of-array type maps.  It helps to see your motivation and some
> examples.  I'll keep it in mind if I ever have a list of large arrays to
> process for which creating a numpy array first is not desirable.
> - Tom
>
>
> On Tue, Jun 18, 2013 at 6:36 AM, Egor Zindy <ezindy at gmail.com> wrote:
>>
>> Dear all,
>>
>> after some code clean-up / testing and a few additions, I've now sent
>> a pull request to numpy:master (#3451).
>> https://github.com/numpy/numpy/pull/3451
>>
>> I also made a blog post to explain the new typemaps I would like included:
>>
>> http://egorzindy.blogspot.co.uk/2013/06/new-numpyi-typemaps-for-working-with.html
>>
>> Any comments appreciated.
>>
>> Kind regards,
>> Egor
>>
>>
>> On 9 June 2013 09:20, Egor Zindy <ezindy at gmail.com> wrote:
>> > Thanks Tom,
>> >
>> > before we ship it, I'd love to have some feedback on the new
>> > ARGOUT_VIEWM
>> > type.
>> >
>> > I used to create my managed arrays using
>> >
>> >     PyObject* cap = PyCObject_FromVoidPtr((void*)(*$1), free);
>> >
>> > but since this function is deprecated, and because of Bill's background
>> > work
>> > to bring numpy.i up to date, I now use capsules for this:
>> >
>> >     PyObject* cap = PyCapsule_New((void*)(*$1), SWIGPY_CAPSULE_NAME,
>> > free_cap);
>> >
>> > ... I'll admit it took longer than expected to get this right.
>> >
>> > Would you mind testing my latest numpy.i changes hosted on github?
>> > https://github.com/zindy/numpy/tree/numpy-swig/doc/swig
>> >
>> > It's great that you are testing on a mac, I don't have one to test on
>> > yet.
>> >
>> >
>> >> It worked fine, although I use only a fraction of the capabilities that
>> >> it
>> >> includes.
>> >
>> > Same here, but overall, it should be quit easy to choose the data type
>> > you
>> > need. Narrow down it down to a type between IN_ARRAY / INPLACE_ /
>> > ARGOUT_ /
>> > ARGOUT_VIEW/VIEWM
>> > http://wiki.scipy.org/Cookbook/SWIG_NumPy_examples
>> > http://wiki.scipy.org/Cookbook/SWIG_Memory_Deallocation (I'll update
>> > these
>> > when I have a sec)
>> >
>> > ... and choose the number of dimensions you need (1/2/3/4). I can't
>> > comment
>> > on the Fortran arrays data types though as I don't use them.
>> >
>> > Also I've introduced a few of my more esoteric data types in this week,
>> > but
>> > I have no idea how popular they will be. If you ever need to speed-up:
>> >
>> > a = numpy.ones((1024,1024),numpy.uint8)
>> > la = [a]*100
>> > b = numpy.mean(numpy.array(la,float),axis=0).astype(numpy.uint8)
>> >
>> > I have just the right type for that :)
>> > DATA_TYPE** IN_ARRAY3, DIM_TYPE DIM1, DIM_TYPE DIM2, DIM_TYPE DIM3
>> >
>> > Kind regards,
>> > Egor
>> >
>> >
>> > On 9 June 2013 03:33, Tom Krauss <thomas.p.krauss at gmail.com> wrote:
>> >>
>> >> Hi folks,
>> >>
>> >> I just downloaded Bill's numpy.i at commit 4dcb0679, and tried it out a
>> >> bit on some of my personal projects.  It worked fine, although I use
>> >> only a
>> >> fraction of the capabilities that it includes.
>> >>
>> >>  And, it made the warning go away!
>> >>
>> >> I used to get this warning
>> >>
>> >> g++ -g -fPIC -c simple_wrap.cpp -I/usr/include/python2.7
>> >>
>> >> -I/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include
>> >> In file included from
>> >>
>> >> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarraytypes.h:1722,
>> >>                  from
>> >>
>> >> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarrayobject.h:17,
>> >>                  from
>> >>
>> >> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/arrayobject.h:15,
>> >>                  from simple_wrap.cpp:3062:
>> >>
>> >>
>> >> /Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
>> >> warning: #warning "Using deprecated NumPy API, disable it by #defining
>> >> NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
>> >>
>> >> but not with this version.
>> >>
>> >> You can see which version of numpy I am using there, and that I am on
>> >> Mac
>> >> OS X 10.8. (10.8.4 specifically)  Python 2.7.2
>> >>
>> >> I'd say SHIP IT!
>> >>
>> >> Nice work, thanks for all your work on numpy and numpy.i.
>> >>
>> >> - Tom Krauss
>> >>
>> >>
>> >>
>> >> On Tue, Jun 4, 2013 at 3:13 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> If you're using or are very familiar with SWIG and the numpy.i
>> >>> interface
>> >>> to it, please help to test and/or review
>> >>> https://github.com/numpy/numpy/pull/3148. It's a fairly major update
>> >>> to
>> >>> numpy.i by Bill Spotz, containing the following:
>> >>>   - support for 4D arrays and memory managed output arguments
>> >>>   - rework for the deprecated API's in numpy 1.6 and 1.7
>> >>>   - a bug fix in a 3D typemap
>> >>>   - documentation improvements
>> >>>
>> >>> It would be good to have this merged before branching 1.8.x. Not many
>> >>> of
>> >>> the regular reviewers of numpy PRs are familiar with numpy.i,
>> >>> therefore help
>> >>> would be much appreciated.
>> >>>
>> >>> Thanks,
>> >>> Ralf
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> NumPy-Discussion mailing list
>> >>> NumPy-Discussion at scipy.org
>> >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >>>
>> >>
>> >>
>> >> _______________________________________________
>> >> NumPy-Discussion mailing list
>> >> NumPy-Discussion at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >>
>> >
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From rhattersley at gmail.com  Fri Jun 21 09:09:53 2013
From: rhattersley at gmail.com (Richard Hattersley)
Date: Fri, 21 Jun 2013 14:09:53 +0100
Subject: [Numpy-discussion] Automatic custom dtype
Message-ID: <CAP=RS9kYQm6heKsY0k2J6EXgOdX0D7F7bzTAaD9uRZ91oATetQ@mail.gmail.com>

Hi all,

In my continuing adventures in the Land of Custom Dtypes I've come
across some rather disappointing behaviour in 1.7 & 1.8.

I've defined my own class `Time360`, and a corresponding dtype
`time360` which references Time360 as its scalar type.

Now with 1.6.2 I can do:
>>> t = Time360(2013, 6, 29)
>>> np.array([t]).dtype
dtype('Time360')

And since all the instances supplied to the function were instances of
the scalar type for my dtype, numpy automatically created an array
using my dtype. Happy days!

But in 1.7 and 1.8 I get:
>>> np.array([t]).dtype
dtype('O')

So now I just get a plain old object array. Boo! Hiss!

Is this expected? Desirable? An unwanted regression?

Richard


From charlesr.harris at gmail.com  Fri Jun 21 09:49:42 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 21 Jun 2013 07:49:42 -0600
Subject: [Numpy-discussion] Automatic custom dtype
In-Reply-To: <CAP=RS9kYQm6heKsY0k2J6EXgOdX0D7F7bzTAaD9uRZ91oATetQ@mail.gmail.com>
References: <CAP=RS9kYQm6heKsY0k2J6EXgOdX0D7F7bzTAaD9uRZ91oATetQ@mail.gmail.com>
Message-ID: <CAB6mnxJb-QeUEjDdhUH64ueiOv3h4uQtZQg9wCZ18FVyVonfmw@mail.gmail.com>

On Fri, Jun 21, 2013 at 7:09 AM, Richard Hattersley
<rhattersley at gmail.com>wrote:

> Hi all,
>
> In my continuing adventures in the Land of Custom Dtypes I've come
> across some rather disappointing behaviour in 1.7 & 1.8.
>
> I've defined my own class `Time360`, and a corresponding dtype
> `time360` which references Time360 as its scalar type.
>
> Now with 1.6.2 I can do:
> >>> t = Time360(2013, 6, 29)
> >>> np.array([t]).dtype
> dtype('Time360')
>
> And since all the instances supplied to the function were instances of
> the scalar type for my dtype, numpy automatically created an array
> using my dtype. Happy days!
>
> But in 1.7 and 1.8 I get:
> >>> np.array([t]).dtype
> dtype('O')
>
> So now I just get a plain old object array. Boo! Hiss!
>
> Is this expected? Desirable? An unwanted regression?
>
>
Bit short on detail here ;) How did you create/register the dtype?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130621/86957f4a/attachment.html>

From rhattersley at gmail.com  Fri Jun 21 11:18:27 2013
From: rhattersley at gmail.com (Richard Hattersley)
Date: Fri, 21 Jun 2013 16:18:27 +0100
Subject: [Numpy-discussion] Automatic custom dtype
In-Reply-To: <CAB6mnxJb-QeUEjDdhUH64ueiOv3h4uQtZQg9wCZ18FVyVonfmw@mail.gmail.com>
References: <CAP=RS9kYQm6heKsY0k2J6EXgOdX0D7F7bzTAaD9uRZ91oATetQ@mail.gmail.com>
	<CAB6mnxJb-QeUEjDdhUH64ueiOv3h4uQtZQg9wCZ18FVyVonfmw@mail.gmail.com>
Message-ID: <CAP=RS9=ZO7BufX2zgzrT=MD4hpyw-UbnFFRn2d6gQ7mocvCOMg@mail.gmail.com>

On 21 June 2013 14:49, Charles R Harris <charlesr.harris at gmail.com> wrote:
> Bit short on detail here ;) How did you create/register the dtype?

The dtype is created/registered during module initialisation with:
    dtype = PyObject_New(PyArray_Descr, &PyArrayDescr_Type);
    dtype->typeobj = &Time360Type;
    ...
    PyArray_RegisterDataType(dtype)

Where Time360Type is my new type definition:
    static PyTypeObject Time360Type = { ... }
which is initialised prior to the dtype creation.

If the detail matters then should I assume this is unexpected
behaviour and maybe I can fix my code so it works?

Richard


From charlesr.harris at gmail.com  Fri Jun 21 14:53:03 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 21 Jun 2013 12:53:03 -0600
Subject: [Numpy-discussion] Automatic custom dtype
In-Reply-To: <CAP=RS9=ZO7BufX2zgzrT=MD4hpyw-UbnFFRn2d6gQ7mocvCOMg@mail.gmail.com>
References: <CAP=RS9kYQm6heKsY0k2J6EXgOdX0D7F7bzTAaD9uRZ91oATetQ@mail.gmail.com>
	<CAB6mnxJb-QeUEjDdhUH64ueiOv3h4uQtZQg9wCZ18FVyVonfmw@mail.gmail.com>
	<CAP=RS9=ZO7BufX2zgzrT=MD4hpyw-UbnFFRn2d6gQ7mocvCOMg@mail.gmail.com>
Message-ID: <CAB6mnxJN9KOFN7szEdC=TMgkAtdZ4+FacCJkf4hA1d+c3QGipw@mail.gmail.com>

On Fri, Jun 21, 2013 at 9:18 AM, Richard Hattersley
<rhattersley at gmail.com>wrote:

> On 21 June 2013 14:49, Charles R Harris <charlesr.harris at gmail.com> wrote:
> > Bit short on detail here ;) How did you create/register the dtype?
>
> The dtype is created/registered during module initialisation with:
>     dtype = PyObject_New(PyArray_Descr, &PyArrayDescr_Type);
>     dtype->typeobj = &Time360Type;
>     ...
>     PyArray_RegisterDataType(dtype)
>
> Where Time360Type is my new type definition:
>     static PyTypeObject Time360Type = { ... }
> which is initialised prior to the dtype creation.
>
> If the detail matters then should I assume this is unexpected
> behaviour and maybe I can fix my code so it works?
>
>
Hmm, that part looks ok.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130621/33ffc640/attachment.html>

From charlesr.harris at gmail.com  Fri Jun 21 14:57:50 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 21 Jun 2013 12:57:50 -0600
Subject: [Numpy-discussion] Automatic custom dtype
In-Reply-To: <CAB6mnxJN9KOFN7szEdC=TMgkAtdZ4+FacCJkf4hA1d+c3QGipw@mail.gmail.com>
References: <CAP=RS9kYQm6heKsY0k2J6EXgOdX0D7F7bzTAaD9uRZ91oATetQ@mail.gmail.com>
	<CAB6mnxJb-QeUEjDdhUH64ueiOv3h4uQtZQg9wCZ18FVyVonfmw@mail.gmail.com>
	<CAP=RS9=ZO7BufX2zgzrT=MD4hpyw-UbnFFRn2d6gQ7mocvCOMg@mail.gmail.com>
	<CAB6mnxJN9KOFN7szEdC=TMgkAtdZ4+FacCJkf4hA1d+c3QGipw@mail.gmail.com>
Message-ID: <CAB6mnxKMAPOw2_zRkmN44n=WsDs66ZhkaTK+TGVSkeYrrLbY+w@mail.gmail.com>

On Fri, Jun 21, 2013 at 12:53 PM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

>
>
> On Fri, Jun 21, 2013 at 9:18 AM, Richard Hattersley <rhattersley at gmail.com
> > wrote:
>
>> On 21 June 2013 14:49, Charles R Harris <charlesr.harris at gmail.com>
>> wrote:
>> > Bit short on detail here ;) How did you create/register the dtype?
>>
>> The dtype is created/registered during module initialisation with:
>>     dtype = PyObject_New(PyArray_Descr, &PyArrayDescr_Type);
>>     dtype->typeobj = &Time360Type;
>>     ...
>>     PyArray_RegisterDataType(dtype)
>>
>> Where Time360Type is my new type definition:
>>     static PyTypeObject Time360Type = { ... }
>> which is initialised prior to the dtype creation.
>>
>> If the detail matters then should I assume this is unexpected
>> behaviour and maybe I can fix my code so it works?
>>
>>
> Hmm, that part looks ok.
>
>
You could check the numpy/core/src/umath/test_rational.c.src code to see if
you are missing something.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130621/90b1f688/attachment.html>

From scopatz at gmail.com  Mon Jun 24 03:15:16 2013
From: scopatz at gmail.com (Anthony Scopatz)
Date: Mon, 24 Jun 2013 02:15:16 -0500
Subject: [Numpy-discussion] ANN: XDress v0.2
Message-ID: <CAPk-6T5rJdMrT5Jd-iY4yNNz9qCtafRH5rbFzggqx3mQyx3Fow@mail.gmail.com>

Hello All,

I am pleased to announce the latest version of xdress, in preparation for
SciPy 2013.  For more information please visit the website:
http://xdress.org

Be Well
Anthony

========================
XDress 0.2 Release Notes
========================
XDress is an automatic wrapper generator for C/C++ written in pure
Python. Currently,
xdress may generate Python bindings (via Cython) for C++ classes, functions, and
certain variable types. It also contains idiomatic wrappers for C++
standard library
containers (sets, vectors, maps). In the future, other tools and
bindings will be
supported.

The main enabling feature of xdress is a dynamic type system that was
designed with
the purpose of API generation in mind.

Release highlights:

  - First class support for C via pycparser.
  - Python 3 support, by popular demand!
  - A plethora of awesome type system updates, including:
        - type matching
        - lambda-valued converters
        - function pointer support
  - Easy to use and implement plugin architecture.

Please visit the website for more information: http://xdress.org/

Ask questions on the mailing list: xdress at googlegroups.com

Download the code from GitHub: http://github.com/scopatz/xdress

XDress is free & open source (BSD 2-clause license) and requires Python 2.7,
NumPy 1.5+, Cython 0.19+, and optionally GCC-XML, pycparser, and lxml.

New Features
============

First Class C Support
---------------------
Wrapping C code is now fully handled through with the optional pycparser.
This means that you don't have to worry about whether or not the GCC-XML
parser will work on your particular bit of code.  Furthermore, C structs
and their members are now handled idiomatically.  (C++ structs are actually
treated as C++ classes, which means that their are allowed to have
constructors and other C++ concepts not present in C.)

Python 3 Support
----------------
The entire code base is built and tested under Python 3.3.  This uses the single
code base model, as most development takes pace in Python 2.7.  The Cython code
that is generated may be used by both Python 2 & 3.


Type System
-----------
The type system has been expanded and hardened to handle additional use cases,
largely motivated by the desire for realistic C support.

A new function pointer (``fucntion_pointer``) refinement type has been added.
When converting from C to Python, a new function object is created that wraps
the underlying call.  For converting from Python to C, a virtual table of
callback functions is constructed that have the same signature of the pointer
but hold a reference to a Python function.  The size of the table and thus how
many callbacks you can have before overwriting previous ones, is set by the
``max_callbacks`` key in the extra dictionary in class descriptions.  This
defaults to 8.

A new enum refinement type now also comes stock. These may be exposed to Python
in the ``rc.variables`` list.

The type system now also comes with basic type matching tools.  There is a new
``TypeMatcher`` class, a ``matches()`` function, and a singleton ``MatchAny``
that may be used for determining whether a type adheres to a pattern.  The
TypeMatcher class itself is immutable and hashable and therefore may be used
anywhere other type elements (tuples, str, int) may be used including as dict
keys!  This is helpful for specifying conversions for large groups of types.

Finally, the type system conversion dictionaries now accept callable objects
as values.  This was put in to handle templated types where the number of
argument is not in general know beforehand, e.g. enums and function pointers.
The values must be callable with only a single argument -- the type itself.
For example, ``lambda t: rtnvalue`` is valid.

Plugins!
--------
XDress is a suite of tools written on top of a type system.  Thus the entire
core has been refactored to implement a very nimble and easy plugin system.
The plugin mechanism enables external projects with their own code generators
to easily hook into the xdress tools.  Additionally, this allows users to
decide at run time which plugins they want to use.

Mini-FAQ
========
* Why not use an existing solution (eg, SWIG)?

    Their type systems don't support run-time, user provided refinement types,
    and thus are unsuited for verification & validation use cases that often
    arise in computational science.

    Furthermore, they tend to not handle C++ dependent types well
(i.e. vector<T>
    does not come back as a np.view(..., dtype=T)).

* Why GCC-XML and not Clang's AST?

    I tried using Clang's AST (and the remnants of a broken visitor class remain
    in the code base).  However, the official Clang AST Python bindings lack
    support for template argument types.  This is a really big deal.
Other C++ ASTs
    may be supported in the future -- including Clang's.

* I run xdress and it creates these files, now what?!

    It is your job to integrate the files created by xdress into your
build system.

Join in the Fun!
================
If you are interested in using xdress on your project (and need help),
contributing
back to xdress, starting up a development team, or writing your own
code generation
plugin tool on top of the type system and autodescriber, please let us know.
Participation is very welcome!

Authors
=======
- `Anthony Scopatz <http://scopatz.com/>`_
- Spencer Lyon *
- Gerald Dalley *
- Alexander Eisenhuth *

An * indicates a first time contributor.

Links
=====

1. Homepage - http://xdress.org/
2. Mailing List - xdress at googlegroups.com
3. GitHub Organization - https://github.com/xdress
4. Pycparser - https://pypi.python.org/pypi/pycparser
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130624/b3bb5ddf/attachment.html>

From Thomas.Goebel at th-nuernberg.de  Mon Jun 24 07:40:08 2013
From: Thomas.Goebel at th-nuernberg.de (Thomas Goebel)
Date: Mon, 24 Jun 2013 13:40:08 +0200
Subject: [Numpy-discussion] Computing least squares fit with masked array
Message-ID: <51C83018.3040600@th-nuernberg.de>

Hi list,

i want to calculate the least squares fit of some data with
missing values. So i masked all values using numpys masked array
module.

Unfortunately using linalg.lstsq i only get nan data back. Can
somebody help me solve this problem?

The output of np.linalg.lstsq ist:
(masked_array(data = [ nan  nan  nan  nan],
             mask = False,
       fill_value = 1e+20)
, masked_array(data = [ nan],
             mask = False,
       fill_value = 1e+20)
, 4, array([  2.08406074e+03,   1.73066556e+01,   1.32393450e+01,
         1.09943429e-01]))

Thanks for your help,
Tom

The python code i use is:

#!/usr/bin/env python2
# -*- coding: utf-8 -*-

import numpy as np

x = np.array([-30, -25, -20])
y = np.array([25, 20, 35])

z = np.array([
    [1285, 1613, 2190],
    [np.nan, 1283, 1897],
    [np.nan, np.nan, 1619]
    ])

N, M = len(x), len(y)

x_new = np.hstack([x] * N)
y_new = np.repeat(y, M)

z_new = np.repeat(z, 1)
z_new = np.ma.masked_array(z_new, np.isnan(z_new))

A = np.array([x_new, y_new, x_new*y_new, np.ones(N*M)])

out = np.linalg.lstsq(A.T, z_new)

print out


From rhattersley at gmail.com  Mon Jun 24 11:23:28 2013
From: rhattersley at gmail.com (Richard Hattersley)
Date: Mon, 24 Jun 2013 16:23:28 +0100
Subject: [Numpy-discussion] Automatic custom dtype
In-Reply-To: <CAB6mnxKMAPOw2_zRkmN44n=WsDs66ZhkaTK+TGVSkeYrrLbY+w@mail.gmail.com>
References: <CAP=RS9kYQm6heKsY0k2J6EXgOdX0D7F7bzTAaD9uRZ91oATetQ@mail.gmail.com>
	<CAB6mnxJb-QeUEjDdhUH64ueiOv3h4uQtZQg9wCZ18FVyVonfmw@mail.gmail.com>
	<CAP=RS9=ZO7BufX2zgzrT=MD4hpyw-UbnFFRn2d6gQ7mocvCOMg@mail.gmail.com>
	<CAB6mnxJN9KOFN7szEdC=TMgkAtdZ4+FacCJkf4hA1d+c3QGipw@mail.gmail.com>
	<CAB6mnxKMAPOw2_zRkmN44n=WsDs66ZhkaTK+TGVSkeYrrLbY+w@mail.gmail.com>
Message-ID: <CAP=RS9nuUJ-Fz=YOynqgkwXr+2Q9MJB2edyOkW2C4Ke6SPzMRA@mail.gmail.com>

On 21 June 2013 19:57, Charles R Harris <charlesr.harris at gmail.com> wrote:
> You could check the numpy/core/src/umath/test_rational.c.src code to see if
> you are missing something.

My code is based in large part on exactly those examples (I don't
think I could have got this far using the documentation alone!), but
I've rechecked and there's nothing obvious missing.

That said I think there may be something funny going on with error
handling within getitem and friends so I'm still following up on that.

Richard


From mwwiebe at gmail.com  Mon Jun 24 11:46:04 2013
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Mon, 24 Jun 2013 10:46:04 -0500
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <CAB6mnxLh2ens_omTWpWu9PUiMFnTWGDs-bsvN-_qV-UUX9vaGg@mail.gmail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
	<CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
	<CAH6Pt5q4eeDTP_4DY2pQ3Cy0Qz0j_mfzUMeDwT1ARZ8ajta56Q@mail.gmail.com>
	<CAB6mnxLh2ens_omTWpWu9PUiMFnTWGDs-bsvN-_qV-UUX9vaGg@mail.gmail.com>
Message-ID: <CAMRnEmrxHvw6Njewo4V=QZJeYpvubDZo7XHncgyE9N=Bk5O8mA@mail.gmail.com>

On Wed, Jun 19, 2013 at 7:48 AM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Wed, Jun 19, 2013 at 5:45 AM, Matthew Brett <matthew.brett at gmail.com>wrote:
>
>> Hi,
>>
>> On Wed, Jun 19, 2013 at 1:43 AM, Fr?d?ric Bastien <nouiz at nouiz.org>
>> wrote:
>> > Hi,
>> >
>> >
>> > On Mon, Jun 17, 2013 at 5:03 PM, Julian Taylor
>> > <jtaylor.debian at googlemail.com> wrote:
>> >>
>> >> On 17.06.2013 17:11, Fr?d?ric Bastien wrote:
>> >> > Hi,
>> >> >
>> >> > I saw that recently Julian Taylor is doing many low level
>> optimization
>> >> > like using SSE instruction. I think it is great.
>> >> >
>> >> > Last year, Mark Florisson released the minivect[1] project that he
>> >> > worked on during is master thesis. minivect is a compiler for
>> >> > element-wise expression that do some of the same low level
>> optimization
>> >> > that Julian is doing in NumPy right now.
>> >> >
>> >> > Mark did minivect in a way that allow it to be reused by other
>> project.
>> >> > It is used now by Cython and Numba I think. I had plan to reuse it in
>> >> > Theano, but I didn't got the time to integrate it up to now.
>> >> >
>> >> > What about reusing it in NumPy? I think that some of Julian
>> optimization
>> >> > aren't in minivect (I didn't check to confirm). But from I heard,
>> >> > minivect don't implement reduction and there is a pull request to
>> >> > optimize this in NumPy.
>> >>
>> >> Hi,
>> >> what I vectorized is just the really easy cases of unit stride
>> >> continuous operations, so the min/max reductions which is now in numpy
>> >> is in essence pretty trivial.
>> >> minivect goes much further in optimizing general strided access and
>> >> broadcasting via loop optimizations (it seems to have a lot of overlap
>> >> with the graphite loop optimizer available in GCC [0]) so my code is
>> >> probably not of very much use to minivect.
>> >>
>> >> The most interesting part in minivect for numpy is probably the
>> >> optimization of broadcasting loops which seem to be pretty inefficient
>> >> in numpy [0].
>> >>
>> >> Concerning the rest I'm not sure how much of a bottleneck general
>> >> strided operations really are in common numpy using code.
>> >>
>> >>
>> >> I guess a similar discussion about adding an expression compiler to
>> >> numpy has already happened when numexpr was released?
>> >> If yes what was the outcome of that?
>> >
>> >
>> > I don't recall a discussion when numexpr was done as this is before I
>> read
>> > this list. numexpr do optimization that can't be done by NumPy: fusing
>> > element-wise operation in one call. So I don't see how it could be done
>> to
>> > reuse it in NumPy.
>> >
>> > You call your optimization trivial, but I don't. In the git log of
>> NumPy,
>> > the first commit is in 2001. It is the first time someone do this in 12
>> > years! Also, this give 1.5-8x speed up (from memory from your PR
>> > description). This is not negligible. But how much time did you spend on
>> > them? Also, some of them are processor dependent, how many people in
>> this
>> > list already have done this? I suppose not many.
>> >
>> > Yes, your optimization don't cover all cases that minivect do. I see 2
>> level
>> > of optimization. 1) The inner loop/contiguous cases, 2) the strided,
>> > broadcasted level. We don't need all optimization being done for them
>> to be
>> > useful. Any of them are useful.
>> >
>> > So what I think is that we could reuse/share that work. NumPy have c
>> code
>> > generator. They could call minivect code generator for some of them when
>> > compiling NumPy. This will make optimization done to those code
>> generator
>> > reused by more people. For example, when new processor are launched, we
>> will
>> > need only 1 place to change for many projects. Or for example, it the
>> call
>> > to MKL vector library is done there, more people will benefit from it.
>> Right
>> > now, only numexpr do it.
>> >
>> > About the level 2 optimization (strides, broadcast), I never read NumPy
>> code
>> > that deal with that. Do someone that know it have an idea if it would be
>> > possible to reuse minivect for this?
>>
>> Would someone be able to guide some of the numpy C experts into a room
>> to do some thinking / writing on this at the scipy conference?
>>
>> I completely agree that these kind of optimizations and code sharing
>> seem likely to be very important for the future.
>>
>> I'm not at the conference, but if there's anything I can do to help,
>> please someone let me know.
>>
>
> Concerning the future development of numpy, I'd also suggest that we look
> at libdynd <https://github.com/ContinuumIO/libdynd>. It looks to me like
> it is reaching a level of maturity where it is worth trying to plan out a
> long term path to merger.
>

I'm in Austin for SciPy, and will giving a talk on the dynd library on
Thursday, please drop by if you can make it, I'm very interested in
cross-pollination of ideas between numpy, libdynd, blaze, and other array
programming projects. The Python exposure of dynd as it is now can
transform data to/from numpy via views very easily, where the data is
compatible, and I expect libdynd and numpy to live alongside each other for
quite some time. One possible way things could work is to think of libdynd
as a more rapidly changing "playground" for functionality that would be
nice to have in numpy, without the guarantees of C-level ABI or API
backwards compatibility that numpy has, at least before libdynd 1.0.

Cheers,
Mark


>
> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130624/46eec183/attachment.html>

From rodrigo at rodrigobotafogo.com  Mon Jun 24 13:41:31 2013
From: rodrigo at rodrigobotafogo.com (Rodrigo Botafogo)
Date: Mon, 24 Jun 2013 15:41:31 -0200
Subject: [Numpy-discussion] [ANN] Multidimensional Array - MDArray 0.5.3
Message-ID: <CAAKc=GCYdiN1favRdpZ-vCeteoz4UvL_iUvR5FnM23KXK07WQQ@mail.gmail.com>

*Announcement*

MDArray version 0.5.3 has Just been released. MDArray is a multi
dimensional array implemented for JRuby inspired by NumPy (www.numpy.org)
and Masahiro Tanaka?s Narray (narray.rubyforge.org).  MDArray stands on the
shoulders of Java-NetCDF and Parallel Colt.  At this point MDArray has
libraries for mathematical, trigonometric and descriptive statistics
methods.

NetCDF-Java Library is a Java interface to NetCDF
files<http://www.unidata.ucar.edu/software/netcdf/index.html>,
as well as to many other types of scientific data formats.  It is developed
and distributed by Unidata (http://www.unidata.ucar.edu).

Parallel Colt (
http://grepcode.com/snapshot/repo1.maven.org/maven2/net.sourceforge.parallelcolt/parallelcolt/0.10.0/)
is a multithreaded<http://en.wikipedia.org/wiki/Thread_%28computer_science%29>
version
of Colt <http://dsd.lbl.gov/~hoschek/colt/> (
http://acs.lbl.gov/software/colt/).  Colt provides a set of Open Source
Libraries for High Performance Scientific and Technical Computing in Java.
Scientific and technical computing is characterized by demanding problem
sizes and a need for high performance at reasonably small memory footprint.

*What?s new**:*

*Performance Improvement*

On previous versions, array operations were done by passing a Ruby Proc to
a loop for all elements of the given arrays.  For instance, adding two
MDArrays was done by passing Proc.new { |a, b| a + b } and looping through
all elements of the arrays.  Procs are very flexible in Ruby; however, from
my experience with MDArray, also very slow.

On this version, when available, instead of passing a Proc to the loop, we
pass a native Java method.  Available Java methods are those extracted from
Parallel Colt and listed below.  Note that Parallel Colt has native methods
for the following types only: ?double?, ?float?, ?long? and ?int?. With
this change, there was a performance improvement of over 90%, and using
MDArray operations is close to native Java operations.  We expect (but have
not yet benchmarking data) that this brings MDArray performance close to
similar solutions such as NArray, NMatrix and NumPy (please try it, and if
this assertion is false, I?ll be glad to change it in future announcements).

Methods not available in Parallel Colt but supported by Ruby, such as
?sinh?, ?tanh?, and ?add? for ?byte? type, etc. are still supported by
MDArray.  Again, to improve performance, instead of passing a Proc we now
create a class as follows

 class Add

*                        *def self.apply(a, b)

                                a + b

                         end

               end



This change brought performance improvement of over 60% for MDArray
operations with Ruby methods.

*Experimental Lazy Evaluation*

Usual MDArray operations are done eagerly, i.e., if @a, @b, @c are three
MDArrays then the following:

                @d = @a + @b + @c

will be evaluated as follows: first @a + @b is performed and stored in a
temporary variable, then this temporary variable is added to @c.  For large
expressions, temporary variables can have significant performance impact.

This version of MDArray introduces lazy evaluation of expressions.  Thus,
when in lazy mode:

                @lazy_d = @a + @b + @c

will not evaluate immediately.  Rather, the expression is preprocessed and
only executed when required.  Since at execution time the whole expression
is known, there is no need for temporary variables as the whole expression
is executed at once.  To put MDArray in lazy mode we only need to set its
mode to lazy with the following command ?MDArray.lazy = true?.  All
expressions after that are by default lazy.  In lazy mode, MDArray
resembles Numexpr, however, there is no need to write the expression as a
string, and there is no compilation involved.

MDArray does not implement broadcasting rules as NumPy.  As a result,
trying to operate on arrays of different shape raises an exception.  On
lazy mode, this exception is raise only at evaluation time, so it is
possible to have an invalid lazy array.  To evaluate a lazy array one
should use the ?[]? method as follows:

                @d = lazy_d[]

@d is now a normal MDArray.

Lazy MDArrays are really lazy, so let?s assume that @a = [1, 2, 3, 4] and
@b = [5, 6, 7, 8].  Lets also have @l_c = @a + @b.  Now doing @c = @l_c[],
will evaluate @c to [6, 8, 10, 12].  Now, let?s do @a[1] = 20 and then @d =
@l_c[].  Now @d evaluates to [25, 8, 10, 12] as the new value of @a is used.

Lazy arrays can be evaluated inside expressions:

                @l_c = (@a + @b)[] + @c

In this example, @l_c is a lazy array, but (@a + @b) is evaluated when the
?[]? method is called and then added to @c.  If now the value of @a or @b
is changed, the evaluation of @l_c will not be changed as in the previous
example.

Finally, laziness is contagious.  So, let?s assume that we have @l_c as
above, a lazy array and we do MDArray.lazy = false.  From this point on in
the code, operations will be done eagerly.  Now doing: @e = @d + @l_c, @e
is a lazy array as its construction involves a lazy array.  One should be
careful when in eager mode mixing lazy and eager arrays:

                @c = @l_a + (@b + @c)

then, with parenthesis, first (@b + @c) is evaluated eagerly and then added
lazily to @l_a, giving a lazy array.

In this version, Lazy evaluation is around 40% *less* efficient in one
machine I tested up to approximately the same performance in another
equipment than eager evaluation when only native Java methods (Parallel
Colt methods described below) are used in the expression. If expression
involves any Ruby method, evaluation of lazy expressions becomes much
slower than eager evaluation.  In order to improve performance, I believe
that compilation of expression will be necessary.

*MDArray and SciRuby**:*

MDArray subscribes fully to the SciRuby Manifesto (http://sciruby.com/).

*?**Ruby* <http://www.ruby-lang.org/>* has for some time had no equivalent
to the beautifully constructed **NumPy* <http://numpy.scipy.org/>*,
SciPy<http://www.scipy.org/>,
and **matplotlib* <http://matplotlib.sourceforge.net/>* libraries for **
Pytho <http://www.python.org/>n.*

*We believe that the time for a Ruby science and visualization package has
come. Sometimes when a solution of sugar and water becomes super-saturated,
from it precipitates a pure, delicious, and diabetes-inducing crystal of
sweetness, induced by no more than the tap of a finger. So is occurring
now, we believe, with numeric and visualization libraries for Ruby.?*



*MDArray main properties are**:*

?         Homogeneous multidimensional array, a table of elements (usually
numbers), all of the same type, indexed by a tuple of positive integers;

?         Easy calculation for large numerical multi dimensional arrays;

?         Basic types are: boolean, byte, short, int, long, float, double,
string, structure;

?         Based on JRuby, which allows importing Java libraries;

?         Operator: +,-,*,/,%,**, >, >=, etc.;

?         Functions: abs, ceil, floor, truncate, is_zero, square, cube,
fourth;

?         Binary Operators: &, |, ^, ~ (binary_ones_complement), <<, >>;

?         Ruby Math functions: acos, acosh, asin, asinh, atan, atan2,
atanh, cbrt, cos, erf, exp, gamma, hypot, ldexp, log, log10, log2, sin,
sinh, sqrt, tan, tanh, neg;

?          Boolean operations on boolean arrays: and, or, not;

?          Fast descriptive statistics from Parallel Colt (complete list
found bellow);

?          Easy manipulation of arrays: reshape, reduce dimension, permute,
section, slice, etc.;

?         Reading of two dimensional arrays from CSV files (mainly for
debugging and simple testing purposes);

?         StatList: a list that can grow/shrink and that can compute
Parallel Colt descriptive statistics;

?         Experimental lazy evaluation (still slower than eager evaluation).

*Descriptive statistics methods imported from Parallel Colt**:*

auto_correlation, correlation, covariance, durbin_watson, frequencies,
geometric_mean, harmonic_mean, kurtosis, lag1, max, mean, mean_deviation,
median, min, moment, moment3, moment4, pooled_mean, pooled_variance,
product, quantile, quantile_inverse, rank_interpolated, rms,
sample_covariance, sample_kurtosis, sample_kurtosis_standard_error,
sample_skew, sample_skew_standard_error, sample_standard_deviation,
sample_variance, sample_weighted_variance, skew,
split,  standard_deviation, standard_error, sum, sum_of_inversions,
sum_of_logarithms, sum_of_powers, sum_of_power_deviations, sum_of_squares,
sum_of_squared_deviations, trimmed_mean, variance, weighted_mean,
weighted_rms, weighted_sums, winsorized_mean.

*Double and Float methods from Parallel Colt*:

acos, asin, atan, atan2, ceil, cos, exp, floor, greater, IEEEremainder,
inv, less, lg, log, log2, rint, sin, sqrt, tan.

*Double, Float, Long and Int methods from Parallel Colt*:

abs, compare, div, divNeg, equals, isEqual (is_equal), isGreater
(is_greater), isles (is_less), max, min, minus, mod, mult, multNeg
(mult_neg), multSquare (mult_square), neg, plus (add), plusAbs (plus_abs),
pow (power), sign, square.

*Long and Int methods from Parallel Colt*

and, dec, factorial, inc, not, or, shiftLeft (shift_left), shiftRightSigned
(shift_right_signed), shiftRightUnsigned (shift_right_unsigned), xor.

*MDArray installation and download**:*

?         Install Jruby

?         jruby ?S gem install mdarray

*MDArray Homepages**:*

?         http://rubygems.org/gems/mdarray

?         https://github.com/rbotafogo/mdarray/wiki

*Contributors**:*

Contributors are welcome.

*MDArray History**:*

?    24/05/2013: Version 0.5.0 ? Over 90% Performance improvements for
methods imported from Parallel Colt and over 40% performance improvements
for all other methods (implemented in Ruby);

?     16/05/2013: Version 0.5.0 - All loops transferred to Java with over
50% performance improvements.  Descriptive statistics from Parallel Colt;

?         19/04/2013: Version 0.4.3 - Fixes a simple, but fatal bug in
0.4.2.  No new features;

?         17/04/2013: Version 0.4.2 - Adds simple statistics and boolean
operators;

?         05/04/2013: Version 0.4.0 ? Initial release.

-- 
Rodrigo Botafogo
Integrando TI ao seu neg?cio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130624/19d50394/attachment.html>

From rob at relentlessidiot.com  Tue Jun 25 02:52:11 2013
From: rob at relentlessidiot.com (Rob Ruana)
Date: Tue, 25 Jun 2013 02:52:11 -0400
Subject: [Numpy-discussion] Proposed enhancement to "Returns" section for
	numpy doc strings
Message-ID: <CADiRWGnzYRuzchD=opC+LMseMcu5i_qUA_A3C2vhbFd_H2n7pw@mail.gmail.com>

I am an admirer of the numpy doc string style. I'd like to propose a simple
enhancement to the Returns section that should make the common case more
readable, while maintaining backward compatibility with existing
documentation.

The current numpy documentation spec [1] requires that return values are
named. Most of the time I only care about the return *type* of a function,
and I can't be bothered to come up with a name. Of course, I can use the
default "out" return name, but that adds nothing of value to the
documentation, and takes away from the readability. I'd really like to just
write:

    Returns
    -------
    bool
        True if some condition, False otherwise.

Of course there are times when names can *enhance* the documentation, for
example numpy.meshgrid:

    Returns
    -------
    X1, X2,..., XN : ndarray
        For vectors `x1`, `x2`,..., 'xn' with lengths ``Ni=len(xi)`` ,
        return ``(N1, N2, N3,...Nn)`` shaped arrays if indexing='ij'
        or ``(N2, N1, N3,...Nn)`` shaped arrays if indexing='xy'
        with the elements of `xi` repeated to fill the matrix along
        the first dimension for `x1`, the second for `x2` and so on.

I suggest that the *name* of each return value should be optional, while
the *type* is required. This would allow for the common case, while still
supporting the documentation as it stands.

I've implemented this change in the numpydoc_returns_unnamed feature branch
on my github fork. The numpydoc sphinx extension has been updated, as well
as the reStructuredText example.py and the guide to numpy documentation.

Please review my pull request (https://github.com/numpy/numpy/pull/3468)
and let me know what you think.


[1]:
https://github.com/numpy/numpy/blob/7af95b0f779f587dfe56c44c5091731b122ceb1a/doc/HOWTO_DOCUMENT.rst.txt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130625/eba01970/attachment.html>

From tmp50 at ukr.net  Tue Jun 25 06:27:35 2013
From: tmp50 at ukr.net (Dmitrey)
Date: Tue, 25 Jun 2013 13:27:35 +0300
Subject: [Numpy-discussion] [ANN] Using some MATLAB optimization solvers
	from Python (OpenOpt/FuncDesigner)
Message-ID: <99974.1372156055.8305133911023747072@ffe16.ukr.net>




Hi all,

>


>

FYI some MATLAB solvers now can be involved with OpenOpt or FuncDesigner
:

>


* LP linprog
* QP quadprog
* LLSP lsqlin
* MILP bintprog

>


>

Sparsity handling is supported.

>


>

You should have
* MATLAB (or MATLAB Component Runtime)
* mlabwrap 

>


Unfortunately, it will hardly work out-of-the-box, you have to adjust
some paths and some environment variables.

>


>

As for nonlinear solvers, e.g. fmincon, probably they could be connected
via involving C MEX files, but it is not possible with current state of
mlabwrap yet.

>


>

Read MATLAB entry for details.

>


>

Regards, D.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130625/f15ee72a/attachment.html>

From nouiz at nouiz.org  Tue Jun 25 10:21:02 2013
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Tue, 25 Jun 2013 10:21:02 -0400
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <CAMRnEmrxHvw6Njewo4V=QZJeYpvubDZo7XHncgyE9N=Bk5O8mA@mail.gmail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
	<CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
	<CAH6Pt5q4eeDTP_4DY2pQ3Cy0Qz0j_mfzUMeDwT1ARZ8ajta56Q@mail.gmail.com>
	<CAB6mnxLh2ens_omTWpWu9PUiMFnTWGDs-bsvN-_qV-UUX9vaGg@mail.gmail.com>
	<CAMRnEmrxHvw6Njewo4V=QZJeYpvubDZo7XHncgyE9N=Bk5O8mA@mail.gmail.com>
Message-ID: <CADKKbtjB2R+4LHLQXeBrL9mmzvVmRMbh-cnuaTR1+E_Xj7u7=g@mail.gmail.com>

Hi,

I wasn't able to attend this year Scipy Conference. My tutorial proposal
was rejected and other deadline intefered with this conference date.

Will the presentation be recorded? If not, can you make the slide available?

What is your opinion on this question:

- Should other lib like NumPy/Theano/Cython/Numba base their elemwise
implemention (or part of it) on dynd or minivect? I know cython and Numba
do it, but it was before dynd and I don't know where dynd fit in the big
picture. Do dynd reuse minivect itself?

thanks

Fr?d?ric


On Mon, Jun 24, 2013 at 11:46 AM, Mark Wiebe <mwwiebe at gmail.com> wrote:

> On Wed, Jun 19, 2013 at 7:48 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Wed, Jun 19, 2013 at 5:45 AM, Matthew Brett <matthew.brett at gmail.com>wrote:
>>
>>> Hi,
>>>
>>> On Wed, Jun 19, 2013 at 1:43 AM, Fr?d?ric Bastien <nouiz at nouiz.org>
>>> wrote:
>>> > Hi,
>>> >
>>> >
>>> > On Mon, Jun 17, 2013 at 5:03 PM, Julian Taylor
>>> > <jtaylor.debian at googlemail.com> wrote:
>>> >>
>>> >> On 17.06.2013 17:11, Fr?d?ric Bastien wrote:
>>> >> > Hi,
>>> >> >
>>> >> > I saw that recently Julian Taylor is doing many low level
>>> optimization
>>> >> > like using SSE instruction. I think it is great.
>>> >> >
>>> >> > Last year, Mark Florisson released the minivect[1] project that he
>>> >> > worked on during is master thesis. minivect is a compiler for
>>> >> > element-wise expression that do some of the same low level
>>> optimization
>>> >> > that Julian is doing in NumPy right now.
>>> >> >
>>> >> > Mark did minivect in a way that allow it to be reused by other
>>> project.
>>> >> > It is used now by Cython and Numba I think. I had plan to reuse it
>>> in
>>> >> > Theano, but I didn't got the time to integrate it up to now.
>>> >> >
>>> >> > What about reusing it in NumPy? I think that some of Julian
>>> optimization
>>> >> > aren't in minivect (I didn't check to confirm). But from I heard,
>>> >> > minivect don't implement reduction and there is a pull request to
>>> >> > optimize this in NumPy.
>>> >>
>>> >> Hi,
>>> >> what I vectorized is just the really easy cases of unit stride
>>> >> continuous operations, so the min/max reductions which is now in numpy
>>> >> is in essence pretty trivial.
>>> >> minivect goes much further in optimizing general strided access and
>>> >> broadcasting via loop optimizations (it seems to have a lot of overlap
>>> >> with the graphite loop optimizer available in GCC [0]) so my code is
>>> >> probably not of very much use to minivect.
>>> >>
>>> >> The most interesting part in minivect for numpy is probably the
>>> >> optimization of broadcasting loops which seem to be pretty inefficient
>>> >> in numpy [0].
>>> >>
>>> >> Concerning the rest I'm not sure how much of a bottleneck general
>>> >> strided operations really are in common numpy using code.
>>> >>
>>> >>
>>> >> I guess a similar discussion about adding an expression compiler to
>>> >> numpy has already happened when numexpr was released?
>>> >> If yes what was the outcome of that?
>>> >
>>> >
>>> > I don't recall a discussion when numexpr was done as this is before I
>>> read
>>> > this list. numexpr do optimization that can't be done by NumPy: fusing
>>> > element-wise operation in one call. So I don't see how it could be
>>> done to
>>> > reuse it in NumPy.
>>> >
>>> > You call your optimization trivial, but I don't. In the git log of
>>> NumPy,
>>> > the first commit is in 2001. It is the first time someone do this in 12
>>> > years! Also, this give 1.5-8x speed up (from memory from your PR
>>> > description). This is not negligible. But how much time did you spend
>>> on
>>> > them? Also, some of them are processor dependent, how many people in
>>> this
>>> > list already have done this? I suppose not many.
>>> >
>>> > Yes, your optimization don't cover all cases that minivect do. I see 2
>>> level
>>> > of optimization. 1) The inner loop/contiguous cases, 2) the strided,
>>> > broadcasted level. We don't need all optimization being done for them
>>> to be
>>> > useful. Any of them are useful.
>>> >
>>> > So what I think is that we could reuse/share that work. NumPy have c
>>> code
>>> > generator. They could call minivect code generator for some of them
>>> when
>>> > compiling NumPy. This will make optimization done to those code
>>> generator
>>> > reused by more people. For example, when new processor are launched,
>>> we will
>>> > need only 1 place to change for many projects. Or for example, it the
>>> call
>>> > to MKL vector library is done there, more people will benefit from it.
>>> Right
>>> > now, only numexpr do it.
>>> >
>>> > About the level 2 optimization (strides, broadcast), I never read
>>> NumPy code
>>> > that deal with that. Do someone that know it have an idea if it would
>>> be
>>> > possible to reuse minivect for this?
>>>
>>> Would someone be able to guide some of the numpy C experts into a room
>>> to do some thinking / writing on this at the scipy conference?
>>>
>>> I completely agree that these kind of optimizations and code sharing
>>> seem likely to be very important for the future.
>>>
>>> I'm not at the conference, but if there's anything I can do to help,
>>> please someone let me know.
>>>
>>
>> Concerning the future development of numpy, I'd also suggest that we look
>> at libdynd <https://github.com/ContinuumIO/libdynd>. It looks to me like
>> it is reaching a level of maturity where it is worth trying to plan out a
>> long term path to merger.
>>
>
> I'm in Austin for SciPy, and will giving a talk on the dynd library on
> Thursday, please drop by if you can make it, I'm very interested in
> cross-pollination of ideas between numpy, libdynd, blaze, and other array
> programming projects. The Python exposure of dynd as it is now can
> transform data to/from numpy via views very easily, where the data is
> compatible, and I expect libdynd and numpy to live alongside each other for
> quite some time. One possible way things could work is to think of libdynd
> as a more rapidly changing "playground" for functionality that would be
> nice to have in numpy, without the guarantees of C-level ABI or API
> backwards compatibility that numpy has, at least before libdynd 1.0.
>
> Cheers,
> Mark
>
>
>>
>> Chuck
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130625/58e8c1ef/attachment.html>

From d.s.seljebotn at astro.uio.no  Wed Jun 26 04:05:11 2013
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 26 Jun 2013 10:05:11 +0200
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <CADKKbtjB2R+4LHLQXeBrL9mmzvVmRMbh-cnuaTR1+E_Xj7u7=g@mail.gmail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
	<CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
	<CAH6Pt5q4eeDTP_4DY2pQ3Cy0Qz0j_mfzUMeDwT1ARZ8ajta56Q@mail.gmail.com>
	<CAB6mnxLh2ens_omTWpWu9PUiMFnTWGDs-bsvN-_qV-UUX9vaGg@mail.gmail.com>
	<CAMRnEmrxHvw6Njewo4V=QZJeYpvubDZo7XHncgyE9N=Bk5O8mA@mail.gmail.com>
	<CADKKbtjB2R+4LHLQXeBrL9mmzvVmRMbh-cnuaTR1+E_Xj7u7=g@mail.gmail.com>
Message-ID: <51CAA0B7.6030005@astro.uio.no>

On 06/25/2013 04:21 PM, Fr?d?ric Bastien wrote:
> Hi,
>
> I wasn't able to attend this year Scipy Conference. My tutorial proposal
> was rejected and other deadline intefered with this conference date.
>
> Will the presentation be recorded? If not, can you make the slide available?
>
> What is your opinion on this question:
>
> - Should other lib like NumPy/Theano/Cython/Numba base their elemwise
> implemention (or part of it) on dynd or minivect? I know cython and
> Numba do it, but it was before dynd and I don't know where dynd fit in
> the big picture. Do dynd reuse minivect itself?

Actually, I think the Cython branch with minivect support was in the end 
not merged, due to lack of interest/manpower to maintain support for 
vectorization in the long term (so it was better to not add the feature 
than have a badly supported feature).

My understanding is that Numba is based on minivect and not on dynd, so 
it's more of a competitor.

Perhaps Mark Florisson will be able to comment.

Dag Sverre


From toddrjen at gmail.com  Wed Jun 26 05:43:07 2013
From: toddrjen at gmail.com (Todd)
Date: Wed, 26 Jun 2013 11:43:07 +0200
Subject: [Numpy-discussion] PEP 208 and upstreaming numpy
In-Reply-To: <CAF6FJiu2f4=b7feKW60v0VNnVr42yGo1SrS_N8vitniS_CGqaQ@mail.gmail.com>
References: <CAFpSVpJoTkoFDWDehGjusrOwNO9bs25YZg-UYohzW877SHwhQA@mail.gmail.com>
	<CAF6FJiu2f4=b7feKW60v0VNnVr42yGo1SrS_N8vitniS_CGqaQ@mail.gmail.com>
Message-ID: <CAFpSVp+hd8APXTOtMj8234TMpPC6gzbbpO7fVQHYp89VuTQC=Q@mail.gmail.com>

But wouldn't there be a benefit from integrating ndarrays directly into the
grammar like lists, tuples, and dictionaries?

On Jun 18, 2013 11:41 AM, "Robert Kern" <robert.kern at gmail.com> wrote:
>
> On Tue, Jun 18, 2013 at 10:33 AM, Todd <toddrjen at gmail.com> wrote:
> > I see that the plan to merge old Numeric into the python standard
library,
> > PEP 208, is listed as withdrawn, although no reasons are given as far
as I
> > could see.
> >
> > Considering how mature Numpy has become, and how common it is as a
> > dependency for python packages, I was wondering if there were still
plans on
> > the table to ultimately merge at least the ndarray class into the python
> > standard library, and if so where those plans stand.
>
> No, this will not be happening. Instead, the focus of effort went into
> creating the enhanced buffer interface. This allows libraries to
> communicate ndarray-like information using that interface instead of
> sharing a particular concrete data structure.
>
> http://www.python.org/dev/peps/pep-3118/
>
> --
> Robert Kern
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130626/6a616bcf/attachment.html>

From robert.kern at gmail.com  Wed Jun 26 05:50:18 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 26 Jun 2013 10:50:18 +0100
Subject: [Numpy-discussion] PEP 208 and upstreaming numpy
In-Reply-To: <CAFpSVp+hd8APXTOtMj8234TMpPC6gzbbpO7fVQHYp89VuTQC=Q@mail.gmail.com>
References: <CAFpSVpJoTkoFDWDehGjusrOwNO9bs25YZg-UYohzW877SHwhQA@mail.gmail.com>
	<CAF6FJiu2f4=b7feKW60v0VNnVr42yGo1SrS_N8vitniS_CGqaQ@mail.gmail.com>
	<CAFpSVp+hd8APXTOtMj8234TMpPC6gzbbpO7fVQHYp89VuTQC=Q@mail.gmail.com>
Message-ID: <CAF6FJivJgkRMNQiT+UqJEeNQH7tBNtYohe-5XNxGJGm7vPEaZg@mail.gmail.com>

On Wed, Jun 26, 2013 at 10:43 AM, Todd <toddrjen at gmail.com> wrote:

> But wouldn't there be a benefit from integrating ndarrays directly into
> the grammar like lists, tuples, and dictionaries?
>

That was never on the table. The benefit of ndarray literals is relatively
tiny and does not justify a change to the grammar.

-- 
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130626/f0da64e2/attachment.html>

From markflorisson88 at gmail.com  Wed Jun 26 07:29:38 2013
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 26 Jun 2013 12:29:38 +0100
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
	<CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
Message-ID: <CANg26EU8T+fR61m8uDzTvR4SC2D4gqwB4AAZDe4dTPRYYZ03gQ@mail.gmail.com>

On 19 June 2013 01:43, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
> Hi,
>
>
> On Mon, Jun 17, 2013 at 5:03 PM, Julian Taylor
> <jtaylor.debian at googlemail.com> wrote:
>>
>> On 17.06.2013 17:11, Fr?d?ric Bastien wrote:
>> > Hi,
>> >
>> > I saw that recently Julian Taylor is doing many low level optimization
>> > like using SSE instruction. I think it is great.
>> >
>> > Last year, Mark Florisson released the minivect[1] project that he
>> > worked on during is master thesis. minivect is a compiler for
>> > element-wise expression that do some of the same low level optimization
>> > that Julian is doing in NumPy right now.
>> >
>> > Mark did minivect in a way that allow it to be reused by other project.
>> > It is used now by Cython and Numba I think. I had plan to reuse it in
>> > Theano, but I didn't got the time to integrate it up to now.
>> >
>> > What about reusing it in NumPy? I think that some of Julian optimization
>> > aren't in minivect (I didn't check to confirm). But from I heard,
>> > minivect don't implement reduction and there is a pull request to
>> > optimize this in NumPy.
>>
>> Hi,
>> what I vectorized is just the really easy cases of unit stride
>> continuous operations, so the min/max reductions which is now in numpy
>> is in essence pretty trivial.
>> minivect goes much further in optimizing general strided access and
>> broadcasting via loop optimizations (it seems to have a lot of overlap
>> with the graphite loop optimizer available in GCC [0]) so my code is
>> probably not of very much use to minivect.
>>
>> The most interesting part in minivect for numpy is probably the
>> optimization of broadcasting loops which seem to be pretty inefficient
>> in numpy [0].
>>
>> Concerning the rest I'm not sure how much of a bottleneck general
>> strided operations really are in common numpy using code.
>>
>>
>> I guess a similar discussion about adding an expression compiler to
>> numpy has already happened when numexpr was released?
>> If yes what was the outcome of that?
>
>
> I don't recall a discussion when numexpr was done as this is before I read
> this list. numexpr do optimization that can't be done by NumPy: fusing
> element-wise operation in one call. So I don't see how it could be done to
> reuse it in NumPy.
>
> You call your optimization trivial, but I don't. In the git log of NumPy,
> the first commit is in 2001. It is the first time someone do this in 12
> years! Also, this give 1.5-8x speed up (from memory from your PR
> description). This is not negligible. But how much time did you spend on
> them? Also, some of them are processor dependent, how many people in this
> list already have done this? I suppose not many.
>
> Yes, your optimization don't cover all cases that minivect do. I see 2 level
> of optimization. 1) The inner loop/contiguous cases, 2) the strided,
> broadcasted level. We don't need all optimization being done for them to be
> useful. Any of them are useful.
>
> So what I think is that we could reuse/share that work. NumPy have c code
> generator. They could call minivect code generator for some of them when
> compiling NumPy. This will make optimization done to those code generator
> reused by more people. For example, when new processor are launched, we will
> need only 1 place to change for many projects. Or for example, it the call
> to MKL vector library is done there, more people will benefit from it. Right
> now, only numexpr do it.
>
> About the level 2 optimization (strides, broadcast), I never read NumPy code
> that deal with that. Do someone that know it have an idea if it would be
> possible to reuse minivect for this?

I wouldn't attempt to, it's not really maintained any longer, though
pykit will likely address what minivect did in the future (more in
following email). Many of the optimizations minivect will really only
shine in a runtime context where it can perform fusion, and where you
can hoist out repeated computation from inner loops. I like the code
reuse, especially between dynd/blaze/theano.

> Fr?d?ric
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From markflorisson88 at gmail.com  Wed Jun 26 07:30:17 2013
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 26 Jun 2013 12:30:17 +0100
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <51CAA0B7.6030005@astro.uio.no>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
	<CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
	<CAH6Pt5q4eeDTP_4DY2pQ3Cy0Qz0j_mfzUMeDwT1ARZ8ajta56Q@mail.gmail.com>
	<CAB6mnxLh2ens_omTWpWu9PUiMFnTWGDs-bsvN-_qV-UUX9vaGg@mail.gmail.com>
	<CAMRnEmrxHvw6Njewo4V=QZJeYpvubDZo7XHncgyE9N=Bk5O8mA@mail.gmail.com>
	<CADKKbtjB2R+4LHLQXeBrL9mmzvVmRMbh-cnuaTR1+E_Xj7u7=g@mail.gmail.com>
	<51CAA0B7.6030005@astro.uio.no>
Message-ID: <CANg26EUV-V8-jGW3VRMi_EAU8N6JWmpvYjHoZHJFFrv-pxvtYg@mail.gmail.com>

On 26 June 2013 09:05, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 06/25/2013 04:21 PM, Fr?d?ric Bastien wrote:
>>
>> Hi,
>>
>> I wasn't able to attend this year Scipy Conference. My tutorial proposal
>> was rejected and other deadline intefered with this conference date.
>>
>> Will the presentation be recorded? If not, can you make the slide
>> available?
>>
>> What is your opinion on this question:
>>
>> - Should other lib like NumPy/Theano/Cython/Numba base their elemwise
>> implemention (or part of it) on dynd or minivect? I know cython and
>> Numba do it, but it was before dynd and I don't know where dynd fit in
>> the big picture. Do dynd reuse minivect itself?
>
>
> Actually, I think the Cython branch with minivect support was in the end not
> merged, due to lack of interest/manpower to maintain support for
> vectorization in the long term (so it was better to not add the feature than
> have a badly supported feature).
>
> My understanding is that Numba is based on minivect and not on dynd, so it's
> more of a competitor.
>
> Perhaps Mark Florisson will be able to comment.
>
> Dag Sverre

Hey Dag,

Indeed, numba uses it for its array expression support, but it will
likely remove the minivect dependency and generate a simple loop nest
for now. I'm working on pykit now
(https://github.com/ContinuumIO/pykit) which similarly to minivect
defines its own intermediate representation, with array expressions in
the form of map/reduce/scan/etc functions. The project has a broader
scope than minivect, to be used by projects like numba, what but a
"minivect baked in".

As such, minivect isn't really maintained any longer, and I wouldn't
recommend anyone using the code at this point (just maybe some of the
ideas :)).


From josef.pktd at gmail.com  Wed Jun 26 11:30:04 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 26 Jun 2013 11:30:04 -0400
Subject: [Numpy-discussion] slicing with boolean in numpy master
Message-ID: <CAMMTP+CxX3o1ZZoQoFRH6OYgPHAL09S53bHdd33j50O01dLAiA@mail.gmail.com>

Is there a change in the behavior of boolean slicing in current master?

If not I have to find another candidate in numpy master.

(py27d) E:\Josef\testing\tox\py27d\Scripts>python
Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__version__
'1.7.1'
>>> x = np.ones((5,3))
>>> mask = np.arange(5) < 4
>>> x[mask, :]
array([[ 1.,  1.,  1.],
       [ 1.,  1.,  1.],
       [ 1.,  1.,  1.],
       [ 1.,  1.,  1.]])


We get errors like the following when running the statsmodels tests
with a current or recent numpy master, but not with numpy 1.7.1

======================================================================
ERROR: Failure: IndexError (too many indices)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
line 518, in makeTest
    return self._makeTest(obj, parent)
  File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
line 577, in _makeTest
    return MethodTestCase(obj)
  File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/case.py",
line 345, in __init__
    self.inst = self.cls()
  File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
line 19, in __init__
    super(Test_AFTModel, self).__init__()
  File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
line 12, in __init__
    self.mod1 = sm.emplike.emplikeAFT(endog, exog, data.censors)
  File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/aft_el.py",
line 248, in __init__
    self.uncens_endog = self.endog[np.bool_(self.censors), :].\
IndexError: too many indices

Thanks,

Josef


From sebastian at sipsolutions.net  Wed Jun 26 12:01:06 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Wed, 26 Jun 2013 18:01:06 +0200
Subject: [Numpy-discussion] slicing with boolean in numpy master
In-Reply-To: <CAMMTP+CxX3o1ZZoQoFRH6OYgPHAL09S53bHdd33j50O01dLAiA@mail.gmail.com>
References: <CAMMTP+CxX3o1ZZoQoFRH6OYgPHAL09S53bHdd33j50O01dLAiA@mail.gmail.com>
Message-ID: <1372262466.3006.1.camel@sebastian-laptop>

On Wed, 2013-06-26 at 11:30 -0400, josef.pktd at gmail.com wrote:
> Is there a change in the behavior of boolean slicing in current master?
> 

Yes, but I think this is probably a bug in statsmodel. I would expect
you should be using "..." and not ":" here, because ":" requires the
dimension to actually exist, and I *expect* that your mask actually has
the same dimensionality as the array itself.

I.e.:

x = np.arange(8).reshape(4,4)
mask = np.ones_like(x)
x[mask,:] # should NOT work, but this was buggy before current master.

- Sebastian

> If not I have to find another candidate in numpy master.
> 
> (py27d) E:\Josef\testing\tox\py27d\Scripts>python
> Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit
> (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import numpy as np
> >>> np.__version__
> '1.7.1'
> >>> x = np.ones((5,3))
> >>> mask = np.arange(5) < 4
> >>> x[mask, :]
> array([[ 1.,  1.,  1.],
>        [ 1.,  1.,  1.],
>        [ 1.,  1.,  1.],
>        [ 1.,  1.,  1.]])
> 
> 
> We get errors like the following when running the statsmodels tests
> with a current or recent numpy master, but not with numpy 1.7.1
> 
> ======================================================================
> ERROR: Failure: IndexError (too many indices)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
> line 518, in makeTest
>     return self._makeTest(obj, parent)
>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
> line 577, in _makeTest
>     return MethodTestCase(obj)
>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/case.py",
> line 345, in __init__
>     self.inst = self.cls()
>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
> line 19, in __init__
>     super(Test_AFTModel, self).__init__()
>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
> line 12, in __init__
>     self.mod1 = sm.emplike.emplikeAFT(endog, exog, data.censors)
>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/aft_el.py",
> line 248, in __init__
>     self.uncens_endog = self.endog[np.bool_(self.censors), :].\
> IndexError: too many indices
> 
> Thanks,
> 
> Josef
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From josef.pktd at gmail.com  Wed Jun 26 12:52:32 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 26 Jun 2013 12:52:32 -0400
Subject: [Numpy-discussion] slicing with boolean in numpy master
In-Reply-To: <1372262466.3006.1.camel@sebastian-laptop>
References: <CAMMTP+CxX3o1ZZoQoFRH6OYgPHAL09S53bHdd33j50O01dLAiA@mail.gmail.com>
	<1372262466.3006.1.camel@sebastian-laptop>
Message-ID: <CAMMTP+DwAU6wiFzdfiyijCu1WcApvuvwn9_DJPRUDyqdcgxR1g@mail.gmail.com>

On Wed, Jun 26, 2013 at 12:01 PM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> On Wed, 2013-06-26 at 11:30 -0400, josef.pktd at gmail.com wrote:
>> Is there a change in the behavior of boolean slicing in current master?
>>
>
> Yes, but I think this is probably a bug in statsmodel. I would expect
> you should be using "..." and not ":" here, because ":" requires the
> dimension to actually exist, and I *expect* that your mask actually has
> the same dimensionality as the array itself.
>
> I.e.:
>
> x = np.arange(8).reshape(4,4)
> mask = np.ones_like(x)
> x[mask,:] # should NOT work, but this was buggy before current master.

Why should this not work?

How do you select rows that don't have nans in them?

mask = np.isfinite(x).all(1)
x[mask, :]

or columns with switched axis.

>>> x[mask[:, None]]
array([ 1.,  1.,  1.,  1.])
???

(I have to check the usage in statsmodels, but I thought this is standard.)

Josef

>
> - Sebastian
>
>> If not I have to find another candidate in numpy master.
>>
>> (py27d) E:\Josef\testing\tox\py27d\Scripts>python
>> Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit
>> (Intel)] on win32
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import numpy as np
>> >>> np.__version__
>> '1.7.1'
>> >>> x = np.ones((5,3))
>> >>> mask = np.arange(5) < 4
>> >>> x[mask, :]
>> array([[ 1.,  1.,  1.],
>>        [ 1.,  1.,  1.],
>>        [ 1.,  1.,  1.],
>>        [ 1.,  1.,  1.]])
>>
>>
>> We get errors like the following when running the statsmodels tests
>> with a current or recent numpy master, but not with numpy 1.7.1
>>
>> ======================================================================
>> ERROR: Failure: IndexError (too many indices)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
>> line 518, in makeTest
>>     return self._makeTest(obj, parent)
>>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
>> line 577, in _makeTest
>>     return MethodTestCase(obj)
>>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/case.py",
>> line 345, in __init__
>>     self.inst = self.cls()
>>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
>> line 19, in __init__
>>     super(Test_AFTModel, self).__init__()
>>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
>> line 12, in __init__
>>     self.mod1 = sm.emplike.emplikeAFT(endog, exog, data.censors)
>>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/aft_el.py",
>> line 248, in __init__
>>     self.uncens_endog = self.endog[np.bool_(self.censors), :].\
>> IndexError: too many indices
>>
>> Thanks,
>>
>> Josef
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From sebastian at sipsolutions.net  Wed Jun 26 13:16:17 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Wed, 26 Jun 2013 19:16:17 +0200
Subject: [Numpy-discussion] slicing with boolean in numpy master
In-Reply-To: <CAMMTP+DwAU6wiFzdfiyijCu1WcApvuvwn9_DJPRUDyqdcgxR1g@mail.gmail.com>
References: <CAMMTP+CxX3o1ZZoQoFRH6OYgPHAL09S53bHdd33j50O01dLAiA@mail.gmail.com>
	<1372262466.3006.1.camel@sebastian-laptop>
	<CAMMTP+DwAU6wiFzdfiyijCu1WcApvuvwn9_DJPRUDyqdcgxR1g@mail.gmail.com>
Message-ID: <1372266977.3006.22.camel@sebastian-laptop>

On Wed, 2013-06-26 at 12:52 -0400, josef.pktd at gmail.com wrote:
> On Wed, Jun 26, 2013 at 12:01 PM, Sebastian Berg
> <sebastian at sipsolutions.net> wrote:
> > On Wed, 2013-06-26 at 11:30 -0400, josef.pktd at gmail.com wrote:
> >> Is there a change in the behavior of boolean slicing in current master?
> >>
> >
> > Yes, but I think this is probably a bug in statsmodel. I would expect
> > you should be using "..." and not ":" here, because ":" requires the
> > dimension to actually exist, and I *expect* that your mask actually has
> > the same dimensionality as the array itself.
> >
> > I.e.:
> >
> > x = np.arange(8).reshape(4,4)
> > mask = np.ones_like(x)
> > x[mask,:] # should NOT work, but this was buggy before current master.
> 
> Why should this not work?
> 
> How do you select rows that don't have nans in them?
> 
> mask = np.isfinite(x).all(1)
> x[mask, :]
> 
> or columns with switched axis.
> 
> >>> x[mask[:, None]]
> array([ 1.,  1.,  1.,  1.])
> ???
> 

I assume you wanted to write x[:, mask] there. Since boolean masks do
*not* broadcast, instead they eat away as many dimensions as they have.

Maybe these examples will help explain why the new behaviour is correct:

x = np.random.random((3,3))
mask = np.ones((3,3), dtype=np.bool_)

# Check slices:
x[:,:] # OK, result 2-d
x[:,:,:] # too many indices.

# replace first dimension with the mask:
x[mask[:,0], :] # OK, result 2-d
x[mask[:,0], :, :] # too many indices.

# replace *both* slices with a (single) mask:
x[mask] # OK, result 1-d (i.e. there nothing more then the mask)
x[mask, :] # too many indices! But it still works in 1.7.

# In fact we can make this absurd:
x[mask, :, :, :, :, :] # Too many slices even without the mask!

The last case used to work in pre-master due to a bug.

- Sebastian
 

> (I have to check the usage in statsmodels, but I thought this is standard.)
> 
> Josef
> 
> >
> > - Sebastian
> >
> >> If not I have to find another candidate in numpy master.
> >>
> >> (py27d) E:\Josef\testing\tox\py27d\Scripts>python
> >> Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit
> >> (Intel)] on win32
> >> Type "help", "copyright", "credits" or "license" for more information.
> >> >>> import numpy as np
> >> >>> np.__version__
> >> '1.7.1'
> >> >>> x = np.ones((5,3))
> >> >>> mask = np.arange(5) < 4
> >> >>> x[mask, :]
> >> array([[ 1.,  1.,  1.],
> >>        [ 1.,  1.,  1.],
> >>        [ 1.,  1.,  1.],
> >>        [ 1.,  1.,  1.]])
> >>
> >>
> >> We get errors like the following when running the statsmodels tests
> >> with a current or recent numpy master, but not with numpy 1.7.1
> >>
> >> ======================================================================
> >> ERROR: Failure: IndexError (too many indices)
> >> ----------------------------------------------------------------------
> >> Traceback (most recent call last):
> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
> >> line 518, in makeTest
> >>     return self._makeTest(obj, parent)
> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
> >> line 577, in _makeTest
> >>     return MethodTestCase(obj)
> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/case.py",
> >> line 345, in __init__
> >>     self.inst = self.cls()
> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
> >> line 19, in __init__
> >>     super(Test_AFTModel, self).__init__()
> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
> >> line 12, in __init__
> >>     self.mod1 = sm.emplike.emplikeAFT(endog, exog, data.censors)
> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/aft_el.py",
> >> line 248, in __init__
> >>     self.uncens_endog = self.endog[np.bool_(self.censors), :].\
> >> IndexError: too many indices
> >>
> >> Thanks,
> >>
> >> Josef
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From josef.pktd at gmail.com  Wed Jun 26 13:56:36 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 26 Jun 2013 13:56:36 -0400
Subject: [Numpy-discussion] slicing with boolean in numpy master
In-Reply-To: <1372266977.3006.22.camel@sebastian-laptop>
References: <CAMMTP+CxX3o1ZZoQoFRH6OYgPHAL09S53bHdd33j50O01dLAiA@mail.gmail.com>
	<1372262466.3006.1.camel@sebastian-laptop>
	<CAMMTP+DwAU6wiFzdfiyijCu1WcApvuvwn9_DJPRUDyqdcgxR1g@mail.gmail.com>
	<1372266977.3006.22.camel@sebastian-laptop>
Message-ID: <CAMMTP+BcG+-5TzY8mDi29iQK8Jk2yKiQKohHwsydUm7JkJvkBQ@mail.gmail.com>

On Wed, Jun 26, 2013 at 1:16 PM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> On Wed, 2013-06-26 at 12:52 -0400, josef.pktd at gmail.com wrote:
>> On Wed, Jun 26, 2013 at 12:01 PM, Sebastian Berg
>> <sebastian at sipsolutions.net> wrote:
>> > On Wed, 2013-06-26 at 11:30 -0400, josef.pktd at gmail.com wrote:
>> >> Is there a change in the behavior of boolean slicing in current master?
>> >>
>> >
>> > Yes, but I think this is probably a bug in statsmodel. I would expect
>> > you should be using "..." and not ":" here, because ":" requires the
>> > dimension to actually exist, and I *expect* that your mask actually has
>> > the same dimensionality as the array itself.
>> >
>> > I.e.:
>> >
>> > x = np.arange(8).reshape(4,4)
>> > mask = np.ones_like(x)
>> > x[mask,:] # should NOT work, but this was buggy before current master.
>>
>> Why should this not work?
>>
>> How do you select rows that don't have nans in them?
>>
>> mask = np.isfinite(x).all(1)
>> x[mask, :]
>>
>> or columns with switched axis.
>>
>> >>> x[mask[:, None]]
>> array([ 1.,  1.,  1.,  1.])
>> ???
>>
>
> I assume you wanted to write x[:, mask] there. Since boolean masks do
> *not* broadcast, instead they eat away as many dimensions as they have.
>
> Maybe these examples will help explain why the new behaviour is correct:
>
> x = np.random.random((3,3))
> mask = np.ones((3,3), dtype=np.bool_)
>
> # Check slices:
> x[:,:] # OK, result 2-d
> x[:,:,:] # too many indices.
>
> # replace first dimension with the mask:
> x[mask[:,0], :] # OK, result 2-d

Good, if this still works, then I go hunting in our code for the "too
many indices"

Thanks for the clarification,

Josef

> x[mask[:,0], :, :] # too many indices.
>
> # replace *both* slices with a (single) mask:
> x[mask] # OK, result 1-d (i.e. there nothing more then the mask)
> x[mask, :] # too many indices! But it still works in 1.7.
>
> # In fact we can make this absurd:
> x[mask, :, :, :, :, :] # Too many slices even without the mask!
>
> The last case used to work in pre-master due to a bug.
>
> - Sebastian
>
>
>> (I have to check the usage in statsmodels, but I thought this is standard.)
>>
>> Josef
>>
>> >
>> > - Sebastian
>> >
>> >> If not I have to find another candidate in numpy master.
>> >>
>> >> (py27d) E:\Josef\testing\tox\py27d\Scripts>python
>> >> Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit
>> >> (Intel)] on win32
>> >> Type "help", "copyright", "credits" or "license" for more information.
>> >> >>> import numpy as np
>> >> >>> np.__version__
>> >> '1.7.1'
>> >> >>> x = np.ones((5,3))
>> >> >>> mask = np.arange(5) < 4
>> >> >>> x[mask, :]
>> >> array([[ 1.,  1.,  1.],
>> >>        [ 1.,  1.,  1.],
>> >>        [ 1.,  1.,  1.],
>> >>        [ 1.,  1.,  1.]])
>> >>
>> >>
>> >> We get errors like the following when running the statsmodels tests
>> >> with a current or recent numpy master, but not with numpy 1.7.1
>> >>
>> >> ======================================================================
>> >> ERROR: Failure: IndexError (too many indices)
>> >> ----------------------------------------------------------------------
>> >> Traceback (most recent call last):
>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
>> >> line 518, in makeTest
>> >>     return self._makeTest(obj, parent)
>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
>> >> line 577, in _makeTest
>> >>     return MethodTestCase(obj)
>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/case.py",
>> >> line 345, in __init__
>> >>     self.inst = self.cls()
>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
>> >> line 19, in __init__
>> >>     super(Test_AFTModel, self).__init__()
>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
>> >> line 12, in __init__
>> >>     self.mod1 = sm.emplike.emplikeAFT(endog, exog, data.censors)
>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/aft_el.py",
>> >> line 248, in __init__
>> >>     self.uncens_endog = self.endog[np.bool_(self.censors), :].\
>> >> IndexError: too many indices
>> >>
>> >> Thanks,
>> >>
>> >> Josef
>> >> _______________________________________________
>> >> NumPy-Discussion mailing list
>> >> NumPy-Discussion at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >>
>> >
>> >
>> > _______________________________________________
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From josef.pktd at gmail.com  Wed Jun 26 14:16:53 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 26 Jun 2013 14:16:53 -0400
Subject: [Numpy-discussion] slicing with boolean in numpy master
In-Reply-To: <CAMMTP+BcG+-5TzY8mDi29iQK8Jk2yKiQKohHwsydUm7JkJvkBQ@mail.gmail.com>
References: <CAMMTP+CxX3o1ZZoQoFRH6OYgPHAL09S53bHdd33j50O01dLAiA@mail.gmail.com>
	<1372262466.3006.1.camel@sebastian-laptop>
	<CAMMTP+DwAU6wiFzdfiyijCu1WcApvuvwn9_DJPRUDyqdcgxR1g@mail.gmail.com>
	<1372266977.3006.22.camel@sebastian-laptop>
	<CAMMTP+BcG+-5TzY8mDi29iQK8Jk2yKiQKohHwsydUm7JkJvkBQ@mail.gmail.com>
Message-ID: <CAMMTP+BeUFZLPgiSzXPOQhEHcXj9t=nzPAJyn6gQ6qZ-MObKEg@mail.gmail.com>

On Wed, Jun 26, 2013 at 1:56 PM,  <josef.pktd at gmail.com> wrote:
> On Wed, Jun 26, 2013 at 1:16 PM, Sebastian Berg
> <sebastian at sipsolutions.net> wrote:
>> On Wed, 2013-06-26 at 12:52 -0400, josef.pktd at gmail.com wrote:
>>> On Wed, Jun 26, 2013 at 12:01 PM, Sebastian Berg
>>> <sebastian at sipsolutions.net> wrote:
>>> > On Wed, 2013-06-26 at 11:30 -0400, josef.pktd at gmail.com wrote:
>>> >> Is there a change in the behavior of boolean slicing in current master?
>>> >>
>>> >
>>> > Yes, but I think this is probably a bug in statsmodel. I would expect
>>> > you should be using "..." and not ":" here, because ":" requires the
>>> > dimension to actually exist, and I *expect* that your mask actually has
>>> > the same dimensionality as the array itself.
>>> >
>>> > I.e.:
>>> >
>>> > x = np.arange(8).reshape(4,4)
>>> > mask = np.ones_like(x)
>>> > x[mask,:] # should NOT work, but this was buggy before current master.
>>>
>>> Why should this not work?

that's fine, I didn't see that mask is 2d

>>>
>>> How do you select rows that don't have nans in them?
>>>
>>> mask = np.isfinite(x).all(1)
>>> x[mask, :]
>>>
>>> or columns with switched axis.
>>>
>>> >>> x[mask[:, None]]
>>> array([ 1.,  1.,  1.,  1.])
>>> ???
>>>
>>
>> I assume you wanted to write x[:, mask] there. Since boolean masks do
>> *not* broadcast, instead they eat away as many dimensions as they have.
>>
>> Maybe these examples will help explain why the new behaviour is correct:
>>
>> x = np.random.random((3,3))
>> mask = np.ones((3,3), dtype=np.bool_)
>>
>> # Check slices:
>> x[:,:] # OK, result 2-d
>> x[:,:,:] # too many indices.
>>
>> # replace first dimension with the mask:
>> x[mask[:,0], :] # OK, result 2-d
>
> Good, if this still works, then I go hunting in our code for the "too
> many indices"

Was easy to find given the clues, even without being able to get the
error messages form numpy > 1.7.1

Thanks,

Josef

>
> Thanks for the clarification,



>
> Josef
>
>> x[mask[:,0], :, :] # too many indices.
>>
>> # replace *both* slices with a (single) mask:
>> x[mask] # OK, result 1-d (i.e. there nothing more then the mask)
>> x[mask, :] # too many indices! But it still works in 1.7.
>>
>> # In fact we can make this absurd:
>> x[mask, :, :, :, :, :] # Too many slices even without the mask!
>>
>> The last case used to work in pre-master due to a bug.
>>
>> - Sebastian
>>
>>
>>> (I have to check the usage in statsmodels, but I thought this is standard.)
>>>
>>> Josef
>>>
>>> >
>>> > - Sebastian
>>> >
>>> >> If not I have to find another candidate in numpy master.
>>> >>
>>> >> (py27d) E:\Josef\testing\tox\py27d\Scripts>python
>>> >> Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit
>>> >> (Intel)] on win32
>>> >> Type "help", "copyright", "credits" or "license" for more information.
>>> >> >>> import numpy as np
>>> >> >>> np.__version__
>>> >> '1.7.1'
>>> >> >>> x = np.ones((5,3))
>>> >> >>> mask = np.arange(5) < 4
>>> >> >>> x[mask, :]
>>> >> array([[ 1.,  1.,  1.],
>>> >>        [ 1.,  1.,  1.],
>>> >>        [ 1.,  1.,  1.],
>>> >>        [ 1.,  1.,  1.]])
>>> >>
>>> >>
>>> >> We get errors like the following when running the statsmodels tests
>>> >> with a current or recent numpy master, but not with numpy 1.7.1
>>> >>
>>> >> ======================================================================
>>> >> ERROR: Failure: IndexError (too many indices)
>>> >> ----------------------------------------------------------------------
>>> >> Traceback (most recent call last):
>>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
>>> >> line 518, in makeTest
>>> >>     return self._makeTest(obj, parent)
>>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/loader.py",
>>> >> line 577, in _makeTest
>>> >>     return MethodTestCase(obj)
>>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/nose/case.py",
>>> >> line 345, in __init__
>>> >>     self.inst = self.cls()
>>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
>>> >> line 19, in __init__
>>> >>     super(Test_AFTModel, self).__init__()
>>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/tests/test_aft.py",
>>> >> line 12, in __init__
>>> >>     self.mod1 = sm.emplike.emplikeAFT(endog, exog, data.censors)
>>> >>   File "/Users/tom/python2.7/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-macosx-10.8-x86_64.egg/statsmodels/emplike/aft_el.py",
>>> >> line 248, in __init__
>>> >>     self.uncens_endog = self.endog[np.bool_(self.censors), :].\
>>> >> IndexError: too many indices
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Josef
>>> >> _______________________________________________
>>> >> NumPy-Discussion mailing list
>>> >> NumPy-Discussion at scipy.org
>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > NumPy-Discussion mailing list
>>> > NumPy-Discussion at scipy.org
>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From nouiz at nouiz.org  Wed Jun 26 20:48:56 2013
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Wed, 26 Jun 2013 20:48:56 -0400
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <CANg26EUV-V8-jGW3VRMi_EAU8N6JWmpvYjHoZHJFFrv-pxvtYg@mail.gmail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
	<CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
	<CAH6Pt5q4eeDTP_4DY2pQ3Cy0Qz0j_mfzUMeDwT1ARZ8ajta56Q@mail.gmail.com>
	<CAB6mnxLh2ens_omTWpWu9PUiMFnTWGDs-bsvN-_qV-UUX9vaGg@mail.gmail.com>
	<CAMRnEmrxHvw6Njewo4V=QZJeYpvubDZo7XHncgyE9N=Bk5O8mA@mail.gmail.com>
	<CADKKbtjB2R+4LHLQXeBrL9mmzvVmRMbh-cnuaTR1+E_Xj7u7=g@mail.gmail.com>
	<51CAA0B7.6030005@astro.uio.no>
	<CANg26EUV-V8-jGW3VRMi_EAU8N6JWmpvYjHoZHJFFrv-pxvtYg@mail.gmail.com>
Message-ID: <CADKKbthPsieT2D-6BTo1PF=vb7G5aWCQUOQGzoPzXomXDJxn8g@mail.gmail.com>

On Wed, Jun 26, 2013 at 7:30 AM, mark florisson
<markflorisson88 at gmail.com>wrote:

> On 26 June 2013 09:05, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no>
> wrote:
> > On 06/25/2013 04:21 PM, Fr?d?ric Bastien wrote:
> >>
> >> Hi,
> >>
> >> I wasn't able to attend this year Scipy Conference. My tutorial proposal
> >> was rejected and other deadline intefered with this conference date.
> >>
> >> Will the presentation be recorded? If not, can you make the slide
> >> available?
> >>
> >> What is your opinion on this question:
> >>
> >> - Should other lib like NumPy/Theano/Cython/Numba base their elemwise
> >> implemention (or part of it) on dynd or minivect? I know cython and
> >> Numba do it, but it was before dynd and I don't know where dynd fit in
> >> the big picture. Do dynd reuse minivect itself?
> >
> >
> > Actually, I think the Cython branch with minivect support was in the end
> not
> > merged, due to lack of interest/manpower to maintain support for
> > vectorization in the long term (so it was better to not add the feature
> than
> > have a badly supported feature).
> >
> > My understanding is that Numba is based on minivect and not on dynd, so
> it's
> > more of a competitor.
> >
> > Perhaps Mark Florisson will be able to comment.
> >
> > Dag Sverre
>
> Hey Dag,
>
> Indeed, numba uses it for its array expression support, but it will
> likely remove the minivect dependency and generate a simple loop nest
> for now. I'm working on pykit now
> (https://github.com/ContinuumIO/pykit) which similarly to minivect
> defines its own intermediate representation, with array expressions in
> the form of map/reduce/scan/etc functions. The project has a broader
> scope than minivect, to be used by projects like numba, what but a
> "minivect baked in".
>
> As such, minivect isn't really maintained any longer, and I wouldn't
> recommend anyone using the code at this point (just maybe some of the
> ideas :)).
>

Hi,

thanks for the information. I checked the repo rapidly and didn't found
information on how to use it the way I expect to use it. I would like to be
able to take a small Theano graph (like just elemwise operation) and make a
graph in it to have it generate the c code. Do you have some
tests/tests/doc that demonstrate something in that direction?

Ideally I would like to be able to implement something like this simple
example:

(x ** 2).sum(1) or (x ** 2).sum()

Is pykit or Numba IR ready for that?

thanks

Fr?d?ric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130626/80d0b8df/attachment.html>

From markflorisson88 at gmail.com  Thu Jun 27 07:36:46 2013
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 27 Jun 2013 12:36:46 +0100
Subject: [Numpy-discussion] low level optimization in NumPy and minivect
In-Reply-To: <CADKKbthPsieT2D-6BTo1PF=vb7G5aWCQUOQGzoPzXomXDJxn8g@mail.gmail.com>
References: <CADKKbti31EFKcX+0kw191EmrzB6XTf-T5HwVSsaimUt5d2TiwQ@mail.gmail.com>
	<51BF7984.9030205@googlemail.com>
	<CADKKbtgowHXdJLVLFG8qBxHyf+-L_-f8t-BY0tPdMCgfx+_ztQ@mail.gmail.com>
	<CAH6Pt5q4eeDTP_4DY2pQ3Cy0Qz0j_mfzUMeDwT1ARZ8ajta56Q@mail.gmail.com>
	<CAB6mnxLh2ens_omTWpWu9PUiMFnTWGDs-bsvN-_qV-UUX9vaGg@mail.gmail.com>
	<CAMRnEmrxHvw6Njewo4V=QZJeYpvubDZo7XHncgyE9N=Bk5O8mA@mail.gmail.com>
	<CADKKbtjB2R+4LHLQXeBrL9mmzvVmRMbh-cnuaTR1+E_Xj7u7=g@mail.gmail.com>
	<51CAA0B7.6030005@astro.uio.no>
	<CANg26EUV-V8-jGW3VRMi_EAU8N6JWmpvYjHoZHJFFrv-pxvtYg@mail.gmail.com>
	<CADKKbthPsieT2D-6BTo1PF=vb7G5aWCQUOQGzoPzXomXDJxn8g@mail.gmail.com>
Message-ID: <CANg26EUjuDyJF4ER9Sm-uneRFNzOya9yPS5+cgeMHQZHHKbqew@mail.gmail.com>

On 27 June 2013 01:48, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
>
>
>
> On Wed, Jun 26, 2013 at 7:30 AM, mark florisson <markflorisson88 at gmail.com>
> wrote:
>>
>> On 26 June 2013 09:05, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no>
>> wrote:
>> > On 06/25/2013 04:21 PM, Fr?d?ric Bastien wrote:
>> >>
>> >> Hi,
>> >>
>> >> I wasn't able to attend this year Scipy Conference. My tutorial
>> >> proposal
>> >> was rejected and other deadline intefered with this conference date.
>> >>
>> >> Will the presentation be recorded? If not, can you make the slide
>> >> available?
>> >>
>> >> What is your opinion on this question:
>> >>
>> >> - Should other lib like NumPy/Theano/Cython/Numba base their elemwise
>> >> implemention (or part of it) on dynd or minivect? I know cython and
>> >> Numba do it, but it was before dynd and I don't know where dynd fit in
>> >> the big picture. Do dynd reuse minivect itself?
>> >
>> >
>> > Actually, I think the Cython branch with minivect support was in the end
>> > not
>> > merged, due to lack of interest/manpower to maintain support for
>> > vectorization in the long term (so it was better to not add the feature
>> > than
>> > have a badly supported feature).
>> >
>> > My understanding is that Numba is based on minivect and not on dynd, so
>> > it's
>> > more of a competitor.
>> >
>> > Perhaps Mark Florisson will be able to comment.
>> >
>> > Dag Sverre
>>
>> Hey Dag,
>>
>> Indeed, numba uses it for its array expression support, but it will
>> likely remove the minivect dependency and generate a simple loop nest
>> for now. I'm working on pykit now
>> (https://github.com/ContinuumIO/pykit) which similarly to minivect
>> defines its own intermediate representation, with array expressions in
>> the form of map/reduce/scan/etc functions. The project has a broader
>> scope than minivect, to be used by projects like numba, what but a
>> "minivect baked in".
>>
>> As such, minivect isn't really maintained any longer, and I wouldn't
>> recommend anyone using the code at this point (just maybe some of the
>> ideas :)).
>
>
> Hi,
>
> thanks for the information. I checked the repo rapidly and didn't found
> information on how to use it the way I expect to use it. I would like to be
> able to take a small Theano graph (like just elemwise operation) and make a
> graph in it to have it generate the c code. Do you have some tests/tests/doc
> that demonstrate something in that direction?
>
> Ideally I would like to be able to implement something like this simple
> example:
>
> (x ** 2).sum(1) or (x ** 2).sum()
>
> Is pykit or Numba IR ready for that?
>
> thanks
>
> Fr?d?ric
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

Hey Fred,

It's in no way ready for public use, it doesn't really actually do
anything yet :) Numba doesn't really optimize reductions yet, so I
don't think it addresses any of your needs - but the input would be a
Python function (compiled from generated source code, or an AST).

I don't know how much further pykit would go beyond simple fusion and
perhaps tiling - I imagine it will defer to libraries like dynd to
perform actual work. This is offtopic for numpy itself, but it may be
useful to Theano in the future, I'll be sure to keep you in the loop
and bounce ideas of for feedback and collaboration.

Cheers,

Mark


From rhattersley at gmail.com  Fri Jun 28 06:27:49 2013
From: rhattersley at gmail.com (Richard Hattersley)
Date: Fri, 28 Jun 2013 11:27:49 +0100
Subject: [Numpy-discussion] Automatic custom dtype
In-Reply-To: <CAB6mnxKMAPOw2_zRkmN44n=WsDs66ZhkaTK+TGVSkeYrrLbY+w@mail.gmail.com>
References: <CAP=RS9kYQm6heKsY0k2J6EXgOdX0D7F7bzTAaD9uRZ91oATetQ@mail.gmail.com>
	<CAB6mnxJb-QeUEjDdhUH64ueiOv3h4uQtZQg9wCZ18FVyVonfmw@mail.gmail.com>
	<CAP=RS9=ZO7BufX2zgzrT=MD4hpyw-UbnFFRn2d6gQ7mocvCOMg@mail.gmail.com>
	<CAB6mnxJN9KOFN7szEdC=TMgkAtdZ4+FacCJkf4hA1d+c3QGipw@mail.gmail.com>
	<CAB6mnxKMAPOw2_zRkmN44n=WsDs66ZhkaTK+TGVSkeYrrLbY+w@mail.gmail.com>
Message-ID: <CAP=RS9nMZLNqTYB8r4k_i--sepikKCFchWYJ3rMtOGN+gSQoYw@mail.gmail.com>

On 21 June 2013 19:57, Charles R Harris <charlesr.harris at gmail.com> wrote:
> You could check the numpy/core/src/umath/test_rational.c.src code to see if
> you are missing something.

In v1.7+ the difference in behaviour between my code and the rational
test case is because my scalar type doesn't subclass np.generic (aka.
PyGenericArrType_Type).

In v1.6 this requirement doesn't exist ... mostly ... In other words,
it works as long as the supplied scalars are contained within a
sequence.
So:
    np.array([scalar]) => np.array([scalar], dtype=my_dtype)
But:
    np.array(scalar) => np.array(scalar, dtype=object)

For one of my scalar/dtype combos I can easily workaround the 1.7+
issue by just adding the subclass relationship. But another of my
dtypes is wrapping a third-party type so I can't modify the subclass
relationship. :-(

So I guess I have three questions.

Firstly, is there some cunning workaround when defining a dtype for a
third-party type?

Secondly, is the subclass-generic requirement in v1.7+ desirable
and/or intended? Or just an accidental regression?

And thirdly, assuming it's desirable to remove the subclass-generic
requirement, would it also make sense to make it work for scalars
which are not within a sequence?

NB. If we decide there's some work which needs doing here, then I
should be able to put time on it.

Thanks,
Richard


From cournape at gmail.com  Fri Jun 28 08:44:31 2013
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 28 Jun 2013 07:44:31 -0500
Subject: [Numpy-discussion] NumPy sprints @ scipy 2013
Message-ID: <CAGY4rcWkWx+FP0-g35U_MUGwJiYiXKeHRtJyAbzF2nrO6GNyKQ@mail.gmail.com>

Hi there,

It is very last minute, but I have set up a page to coordinate a bit scipy
2013's numpy sprints (Friday 28 and Saturday 29th, although I may not be
there the second day).

Depending on the audience, I will also look for low hanging fruits. Feel
free to edit the wiki at will

cheers,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130628/f720ed40/attachment.html>

From cournape at gmail.com  Fri Jun 28 09:02:36 2013
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 28 Jun 2013 08:02:36 -0500
Subject: [Numpy-discussion] NumPy sprints @ scipy 2013
In-Reply-To: <CAGY4rcWkWx+FP0-g35U_MUGwJiYiXKeHRtJyAbzF2nrO6GNyKQ@mail.gmail.com>
References: <CAGY4rcWkWx+FP0-g35U_MUGwJiYiXKeHRtJyAbzF2nrO6GNyKQ@mail.gmail.com>
Message-ID: <CAGY4rcVeGX88qmvwDTnoN6df8-4NbeORyGb67p3i2MhS9tv1Ag@mail.gmail.com>

It is better with a link, sorry about that:
https://github.com/numpy/numpy/wiki/Scipy-2013

On Fri, Jun 28, 2013 at 7:44 AM, David Cournapeau <cournape at gmail.com>wrote:

> Hi there,
>
> It is very last minute, but I have set up a page to coordinate a bit scipy
> 2013's numpy sprints (Friday 28 and Saturday 29th, although I may not be
> there the second day).
>
> Depending on the audience, I will also look for low hanging fruits. Feel
> free to edit the wiki at will
>
> cheers,
> David
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130628/844bdb74/attachment.html>

From sebastian at sipsolutions.net  Fri Jun 28 09:10:41 2013
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Fri, 28 Jun 2013 15:10:41 +0200
Subject: [Numpy-discussion] NumPy sprints @ scipy 2013
In-Reply-To: <CAGY4rcWkWx+FP0-g35U_MUGwJiYiXKeHRtJyAbzF2nrO6GNyKQ@mail.gmail.com>
References: <CAGY4rcWkWx+FP0-g35U_MUGwJiYiXKeHRtJyAbzF2nrO6GNyKQ@mail.gmail.com>
Message-ID: <1372425041.31359.38.camel@sebastian-laptop>

On Fri, 2013-06-28 at 07:44 -0500, David Cournapeau wrote:
> Hi there,
> 
> 
> It is very last minute, but I have set up a page to coordinate a bit
> scipy 2013's numpy sprints (Friday 28 and Saturday 29th, although I
> may not be there the second day).
> 
> 
> Depending on the audience, I will also look for low hanging fruits.
> Feel free to edit the wiki at will
> 
Just saw the "take vs. indexing" speed thing. To get it quite right, I
think you probably need to create a new mapiter (after all, take is not
ideal since it requires contiguity - though I guess it could be used as
a special case). Its only half related but this might be interesting if
someone wants to look into it:
https://github.com/seberg/numpy/compare/master...rewrite-index-parser

It does some smaller (but binary incompatible) tweaks to mapiter too,
though the speedup there is not nearly as big as should be possible (and
some fast special cases are not active). Mostly its a start of a
possible rewrite of the indexing machinery, but since the current
mapiter is rather organically interwoven with indexing...

Only a note, I would like to see it, but I doubt its a one day
project :).

- Sebastian

PS: Wondering about visiting EuroScipy (if my schedule works out), if
some of you are there, it would be something I could imagine working on
then (until then, I don't have any time).

> 
> cheers,
> David
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From charlesr.harris at gmail.com  Fri Jun 28 11:47:11 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 28 Jun 2013 10:47:11 -0500
Subject: [Numpy-discussion] Automatic custom dtype
In-Reply-To: <CAP=RS9nMZLNqTYB8r4k_i--sepikKCFchWYJ3rMtOGN+gSQoYw@mail.gmail.com>
References: <CAP=RS9kYQm6heKsY0k2J6EXgOdX0D7F7bzTAaD9uRZ91oATetQ@mail.gmail.com>
	<CAB6mnxJb-QeUEjDdhUH64ueiOv3h4uQtZQg9wCZ18FVyVonfmw@mail.gmail.com>
	<CAP=RS9=ZO7BufX2zgzrT=MD4hpyw-UbnFFRn2d6gQ7mocvCOMg@mail.gmail.com>
	<CAB6mnxJN9KOFN7szEdC=TMgkAtdZ4+FacCJkf4hA1d+c3QGipw@mail.gmail.com>
	<CAB6mnxKMAPOw2_zRkmN44n=WsDs66ZhkaTK+TGVSkeYrrLbY+w@mail.gmail.com>
	<CAP=RS9nMZLNqTYB8r4k_i--sepikKCFchWYJ3rMtOGN+gSQoYw@mail.gmail.com>
Message-ID: <CAB6mnxJHr8iRXEo1_4OQzfc==J6R1X7Xisf+vH_MdVpLOVn53Q@mail.gmail.com>

On Fri, Jun 28, 2013 at 5:27 AM, Richard Hattersley
<rhattersley at gmail.com>wrote:

> On 21 June 2013 19:57, Charles R Harris <charlesr.harris at gmail.com> wrote:
> > You could check the numpy/core/src/umath/test_rational.c.src code to see
> if
> > you are missing something.
>
> In v1.7+ the difference in behaviour between my code and the rational
> test case is because my scalar type doesn't subclass np.generic (aka.
> PyGenericArrType_Type).
>
> In v1.6 this requirement doesn't exist ... mostly ... In other words,
> it works as long as the supplied scalars are contained within a
> sequence.
> So:
>     np.array([scalar]) => np.array([scalar], dtype=my_dtype)
> But:
>     np.array(scalar) => np.array(scalar, dtype=object)
>

Thanks for tracking that down.

>
> For one of my scalar/dtype combos I can easily workaround the 1.7+
> issue by just adding the subclass relationship. But another of my
> dtypes is wrapping a third-party type so I can't modify the subclass
> relationship. :-(
>
> So I guess I have three questions.
>
> Firstly, is there some cunning workaround when defining a dtype for a
> third-party type?
>
> Secondly, is the subclass-generic requirement in v1.7+ desirable
> and/or intended? Or just an accidental regression?
>

I don't know ;) But we do try to keep backward compatibility so unless
there is a good reason it would be a regression. In any case, we should
look for a way to let the previous version work.

>
> And thirdly, assuming it's desirable to remove the subclass-generic
> requirement, would it also make sense to make it work for scalars
> which are not within a sequence?
>
> NB. If we decide there's some work which needs doing here, then I
> should be able to put time on it.
>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130628/ebff3232/attachment.html>

From charlesr.harris at gmail.com  Fri Jun 28 12:33:48 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 28 Jun 2013 11:33:48 -0500
Subject: [Numpy-discussion] Automatic custom dtype
In-Reply-To: <CAP=RS9nMZLNqTYB8r4k_i--sepikKCFchWYJ3rMtOGN+gSQoYw@mail.gmail.com>
References: <CAP=RS9kYQm6heKsY0k2J6EXgOdX0D7F7bzTAaD9uRZ91oATetQ@mail.gmail.com>
	<CAB6mnxJb-QeUEjDdhUH64ueiOv3h4uQtZQg9wCZ18FVyVonfmw@mail.gmail.com>
	<CAP=RS9=ZO7BufX2zgzrT=MD4hpyw-UbnFFRn2d6gQ7mocvCOMg@mail.gmail.com>
	<CAB6mnxJN9KOFN7szEdC=TMgkAtdZ4+FacCJkf4hA1d+c3QGipw@mail.gmail.com>
	<CAB6mnxKMAPOw2_zRkmN44n=WsDs66ZhkaTK+TGVSkeYrrLbY+w@mail.gmail.com>
	<CAP=RS9nMZLNqTYB8r4k_i--sepikKCFchWYJ3rMtOGN+gSQoYw@mail.gmail.com>
Message-ID: <CAB6mnxLTezMNEif4ZZ2E1V2WD_kvN-f-ytifphrNbncw_8Ynrw@mail.gmail.com>

On Fri, Jun 28, 2013 at 5:27 AM, Richard Hattersley
<rhattersley at gmail.com>wrote:

> On 21 June 2013 19:57, Charles R Harris <charlesr.harris at gmail.com> wrote:
> > You could check the numpy/core/src/umath/test_rational.c.src code to see
> if
> > you are missing something.
>
> In v1.7+ the difference in behaviour between my code and the rational
> test case is because my scalar type doesn't subclass np.generic (aka.
> PyGenericArrType_Type).
>
> In v1.6 this requirement doesn't exist ... mostly ... In other words,
> it works as long as the supplied scalars are contained within a
> sequence.
> So:
>     np.array([scalar]) => np.array([scalar], dtype=my_dtype)
> But:
>     np.array(scalar) => np.array(scalar, dtype=object)
>

So the scalar case (0 dimensional array) doesn't work right. Hmm, what
happens when you index the first array? Does subclassing the generic type
work in 1.6?

My impression is that subclassing the generic type should be required, but
I don't see where it is documented :( Anyway, what is the problem with the
third party code? Is there no chance that you can get hold of it to fix it?

>
> For one of my scalar/dtype combos I can easily workaround the 1.7+
> issue by just adding the subclass relationship. But another of my
> dtypes is wrapping a third-party type so I can't modify the subclass
> relationship. :-(
>
> So I guess I have three questions.
>
> Firstly, is there some cunning workaround when defining a dtype for a
> third-party type?
>
> Secondly, is the subclass-generic requirement in v1.7+ desirable
> and/or intended? Or just an accidental regression?
>
> And thirdly, assuming it's desirable to remove the subclass-generic
> requirement, would it also make sense to make it work for scalars
> which are not within a sequence?
>
> NB. If we decide there's some work which needs doing here, then I
> should be able to put time on it.
>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130628/1c926683/attachment.html>

From rhattersley at gmail.com  Fri Jun 28 15:11:30 2013
From: rhattersley at gmail.com (Richard Hattersley)
Date: Fri, 28 Jun 2013 20:11:30 +0100
Subject: [Numpy-discussion] Automatic custom dtype
In-Reply-To: <CAB6mnxLTezMNEif4ZZ2E1V2WD_kvN-f-ytifphrNbncw_8Ynrw@mail.gmail.com>
References: <CAP=RS9kYQm6heKsY0k2J6EXgOdX0D7F7bzTAaD9uRZ91oATetQ@mail.gmail.com>
	<CAB6mnxJb-QeUEjDdhUH64ueiOv3h4uQtZQg9wCZ18FVyVonfmw@mail.gmail.com>
	<CAP=RS9=ZO7BufX2zgzrT=MD4hpyw-UbnFFRn2d6gQ7mocvCOMg@mail.gmail.com>
	<CAB6mnxJN9KOFN7szEdC=TMgkAtdZ4+FacCJkf4hA1d+c3QGipw@mail.gmail.com>
	<CAB6mnxKMAPOw2_zRkmN44n=WsDs66ZhkaTK+TGVSkeYrrLbY+w@mail.gmail.com>
	<CAP=RS9nMZLNqTYB8r4k_i--sepikKCFchWYJ3rMtOGN+gSQoYw@mail.gmail.com>
	<CAB6mnxLTezMNEif4ZZ2E1V2WD_kvN-f-ytifphrNbncw_8Ynrw@mail.gmail.com>
Message-ID: <CAP=RS9kVjQ8oDKZ4EMB=pQ0f3ynNFabZCotNGnfW3xxoAfre-Q@mail.gmail.com>

On 28 June 2013 17:33, Charles R Harris <charlesr.harris at gmail.com> wrote:
> On Fri, Jun 28, 2013 at 5:27 AM, Richard Hattersley <rhattersley at gmail.com>
> wrote:
>> So:
>>     np.array([scalar]) => np.array([scalar], dtype=my_dtype)
>> But:
>>     np.array(scalar) => np.array(scalar, dtype=object)
>
> So the scalar case (0 dimensional array) doesn't work right. Hmm, what
> happens when you index the first array? Does subclassing the generic type
> work in 1.6?

Indexing into the first array works fine. So something like `a[0]`
calls my_dtype->f->getitem which creates a new scalar instance, and
something like `a[:1]` creates a new view with the correct dtype.


> My impression is that subclassing the generic type should be required, but I
> don't see where it is documented :(

Can you elaborate on why the generic type should be required? Do you
think it might cause problems elsewhere? (FYI I've also tested with a
patched version of v1.6.2 which fixes the typo which prevents the use
of user-defined dtypes with ufuncs, and that functionality seems to
work fine too.)


> Anyway, what is the problem with the
> third party code? Is there no chance that you can get hold of it to fix it?

Unfortunately it's out of my control.


Regards,
Richard


From josef.pktd at gmail.com  Sat Jun 29 00:03:13 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 29 Jun 2013 00:03:13 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
Message-ID: <CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>

On Wed, Jun 12, 2013 at 8:10 AM, Nathaniel Smith <njs at pobox.com> wrote:
> Hi all,
>
> It looks like we've gotten a bit confused and need to untangle
> something. There's a PR to add new functions 'np.filled' and
> 'np.filled_like':
>   https://github.com/numpy/numpy/pull/2875
> And there was a discussion about this on the list back in January:
>   http://thread.gmane.org/gmane.comp.python.numeric.general/52763
>
> I think a reasonable summary of the opinions in the thread are:
> - This functionality is great, ...
> - ...but we can't call it 'np.filled' because there's also
> 'np.ma.filled' which does something else...
> - ...but there really aren't any better names...
> - ...so we should overload np.empty, like: 'np.empty(shape, fill=value)'
>
> In the mean time the original submitter has continued puttering along
> polishing the original patch, and it's ready to merge... except it's
> still the original interface, somehow the thread discussion and the PR
> discussion never met up.
>
> So, we have to decide what to do.
>
> Personally I think that overloading np.empty is horribly ugly, will
> continue confusing newbies and everyone else indefinitely, and I'm
> 100% convinced that we'll regret implementing such a warty interface
> for something that should be so idiomatic. (Unfortunately I got busy
> and didn't actually say this in the previous thread though.) So I
> think we should just merge the PR as is. The only downside is the
> np.ma inconsistency, but, np.ma is already inconsistent (cf.
> masked_array.fill versus masked_array.filled!), somewhat deprecated,
> and AFAICT there are far more people who will benefit from a clean
> np.filled idiom than who actually use np.ma (and in particular its
> fill-value functionality). So there would be two
> bad-but-IMHO-acceptable options: either live with an inconsistency
> between np.filled and np.ma.filled, or deprecate np.ma.filled in favor
> of masked_array.filled (which does exactly the same thing) and
> eventually switch np.ma.filled to be consistent with the new
> np.filled.
>
> But, that's just my opinion.
>
> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


Why we need this animal whatever it is called

 Scientific Python @SciPyTip
Create a 2x3 array filled with integer 7's: 7*np.ones((2,3), int)

and I just did this yesterday,  np.nan * np.ones(3)
maybe I should have used np.zeros(3) / 0.

Josef


From njs at pobox.com  Sat Jun 29 05:37:55 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 29 Jun 2013 10:37:55 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
Message-ID: <CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>

So this petered off...any objections to np.full?
On 29 Jun 2013 05:03, <josef.pktd at gmail.com> wrote:

> On Wed, Jun 12, 2013 at 8:10 AM, Nathaniel Smith <njs at pobox.com> wrote:
> > Hi all,
> >
> > It looks like we've gotten a bit confused and need to untangle
> > something. There's a PR to add new functions 'np.filled' and
> > 'np.filled_like':
> >   https://github.com/numpy/numpy/pull/2875
> > And there was a discussion about this on the list back in January:
> >   http://thread.gmane.org/gmane.comp.python.numeric.general/52763
> >
> > I think a reasonable summary of the opinions in the thread are:
> > - This functionality is great, ...
> > - ...but we can't call it 'np.filled' because there's also
> > 'np.ma.filled' which does something else...
> > - ...but there really aren't any better names...
> > - ...so we should overload np.empty, like: 'np.empty(shape, fill=value)'
> >
> > In the mean time the original submitter has continued puttering along
> > polishing the original patch, and it's ready to merge... except it's
> > still the original interface, somehow the thread discussion and the PR
> > discussion never met up.
> >
> > So, we have to decide what to do.
> >
> > Personally I think that overloading np.empty is horribly ugly, will
> > continue confusing newbies and everyone else indefinitely, and I'm
> > 100% convinced that we'll regret implementing such a warty interface
> > for something that should be so idiomatic. (Unfortunately I got busy
> > and didn't actually say this in the previous thread though.) So I
> > think we should just merge the PR as is. The only downside is the
> > np.ma inconsistency, but, np.ma is already inconsistent (cf.
> > masked_array.fill versus masked_array.filled!), somewhat deprecated,
> > and AFAICT there are far more people who will benefit from a clean
> > np.filled idiom than who actually use np.ma (and in particular its
> > fill-value functionality). So there would be two
> > bad-but-IMHO-acceptable options: either live with an inconsistency
> > between np.filled and np.ma.filled, or deprecate np.ma.filled in favor
> > of masked_array.filled (which does exactly the same thing) and
> > eventually switch np.ma.filled to be consistent with the new
> > np.filled.
> >
> > But, that's just my opinion.
> >
> > -n
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> Why we need this animal whatever it is called
>
>  Scientific Python @SciPyTip
> Create a 2x3 array filled with integer 7's: 7*np.ones((2,3), int)
>
> and I just did this yesterday,  np.nan * np.ones(3)
> maybe I should have used np.zeros(3) / 0.
>
> Josef
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130629/efe18dba/attachment.html>

From matthew.brett at gmail.com  Sat Jun 29 06:39:21 2013
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 29 Jun 2013 11:39:21 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
Message-ID: <CAH6Pt5ogOiL+146oQ8+7Bzeu0=p_HeVKLhbgeygo2MRN0_4jUg@mail.gmail.com>

Hi,

On Sat, Jun 29, 2013 at 10:37 AM, Nathaniel Smith <njs at pobox.com> wrote:
> So this petered off...any objections to np.full?

full and filledwith and filled_with all seem OK to me.

On a meta note - anything we can do to stop threads petering off?  It
seems to happen rather often,

Cheers,

Matthew


From josef.pktd at gmail.com  Sat Jun 29 08:24:41 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 29 Jun 2013 08:24:41 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAH6Pt5ogOiL+146oQ8+7Bzeu0=p_HeVKLhbgeygo2MRN0_4jUg@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAH6Pt5ogOiL+146oQ8+7Bzeu0=p_HeVKLhbgeygo2MRN0_4jUg@mail.gmail.com>
Message-ID: <CAMMTP+Bg7iYFPzkij9c3LEq+pYYVcDyyXhGDUPZ9sE_W-yejrA@mail.gmail.com>

On Sat, Jun 29, 2013 at 6:39 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Sat, Jun 29, 2013 at 10:37 AM, Nathaniel Smith <njs at pobox.com> wrote:
>> So this petered off...any objections to np.full?
>
> full and filledwith and filled_with all seem OK to me.

same here, filled_with_like might have too many underlines.

>
> On a meta note - anything we can do to stop threads petering off?  It
> seems to happen rather often,

resurrect, especially if the PR is just waiting for a name to finish

Cheers,

Josef

>
> Cheers,
>
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From charlesr.harris at gmail.com  Sat Jun 29 12:04:23 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 29 Jun 2013 11:04:23 -0500
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
Message-ID: <CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>

On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs at pobox.com> wrote:

> So this petered off...any objections to np.full?
>

How about `np.inited` ?  `full` doesn't sound quite right to me. Although I
expect once it comes into use everyone will get used to it.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130629/9e66dc65/attachment.html>

From josef.pktd at gmail.com  Sat Jun 29 12:43:02 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 29 Jun 2013 12:43:02 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
Message-ID: <CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>

On Sat, Jun 29, 2013 at 12:04 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> So this petered off...any objections to np.full?
>
>
> How about `np.inited` ?  `full` doesn't sound quite right to me. Although I
> expect once it comes into use everyone will get used to it.

Is this pronounce like "ignited" ?

Josef
German speaker

>
> <snip>
>
> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From ralf.gommers at gmail.com  Sat Jun 29 15:00:35 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 29 Jun 2013 21:00:35 +0200
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
	<CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
Message-ID: <CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>

On Sat, Jun 29, 2013 at 6:43 PM, <josef.pktd at gmail.com> wrote:

> On Sat, Jun 29, 2013 at 12:04 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> >
> > On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs at pobox.com> wrote:
> >>
> >> So this petered off...any objections to np.full?
>

No objection.


> > How about `np.inited` ?  `full` doesn't sound quite right to me.
> Although I
> > expect once it comes into use everyone will get used to it.
>
> Is this pronounce like "ignited" ?
>

I guess ``__init__``-ed. Not very clear I'm afraid.

Ralf

Josef
> German speaker
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130629/ab33dbf0/attachment.html>

From alan.isaac at gmail.com  Sat Jun 29 16:55:10 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Sat, 29 Jun 2013 16:55:10 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
	<CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
	<CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
Message-ID: <51CF49AE.6050908@gmail.com>

On 6/29/2013 3:00 PM, Nathaniel wrote:
> any objections to np.full?

Still curious:
why isn't ``tile`` the right name?
(It already exists.)

 >>> import numpy as np
 >>> np.tile(3.0, (2,3))
array([[ 3.,  3.,  3.],
        [ 3.,  3.,  3.]])

If someone explained this, sorry to
have missed it.

Alan


From robert.kern at gmail.com  Sat Jun 29 17:25:22 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 29 Jun 2013 22:25:22 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51CF49AE.6050908@gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
	<CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
	<CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
	<51CF49AE.6050908@gmail.com>
Message-ID: <CAF6FJitJsaQLYQ0i8CYPmcDo7bxOZES18h48=1R4VHz8j8_1KA@mail.gmail.com>

On Sat, Jun 29, 2013 at 9:55 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
>
> On 6/29/2013 3:00 PM, Nathaniel wrote:
> > any objections to np.full?
>
> Still curious:
> why isn't ``tile`` the right name?
> (It already exists.)
>
>  >>> import numpy as np
>  >>> np.tile(3.0, (2,3))
> array([[ 3.,  3.,  3.],
>         [ 3.,  3.,  3.]])
>
> If someone explained this, sorry to
> have missed it.

It's implemented inefficiently.

It is aimed at a different use case (building up arrays from other arrays)
that only incidentally, and thus poorly, fulfils this one.

It has no relation to the empty()/ones()/zeros() line of functions. In
particular, tile_like() would make no sense.

Being aimed at a different use case, it will be more difficult to find for
people who are not familiar with numpy's entire API. It will be an isolated
idiom rather than a natural completion of the existing APIs.

I can probably think of others if I really get going, but I think we're
about at the point of diminishing returns.

-- 
Robert Kern


On Sat, Jun 29, 2013 at 9:55 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:

> On 6/29/2013 3:00 PM, Nathaniel wrote:
> > any objections to np.full?
>
> Still curious:
> why isn't ``tile`` the right name?
> (It already exists.)
>
>  >>> import numpy as np
>  >>> np.tile(3.0, (2,3))
> array([[ 3.,  3.,  3.],
>         [ 3.,  3.,  3.]])
>
> If someone explained this, sorry to
> have missed it.
>
> Alan
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130629/e9bc0047/attachment.html>

From alan.isaac at gmail.com  Sat Jun 29 17:53:48 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Sat, 29 Jun 2013 17:53:48 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAF6FJitJsaQLYQ0i8CYPmcDo7bxOZES18h48=1R4VHz8j8_1KA@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
	<CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
	<CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
	<51CF49AE.6050908@gmail.com>
	<CAF6FJitJsaQLYQ0i8CYPmcDo7bxOZES18h48=1R4VHz8j8_1KA@mail.gmail.com>
Message-ID: <51CF576C.8090702@gmail.com>

>> On 6/29/2013 3:00 PM, Nathaniel wrote:
>>> any objections to np.full?


> On Sat, Jun 29, 2013 at 9:55 PM, Alan G Isaac wrote:
>> Still curious:
>> why isn't ``tile`` the right name?
>> (It already exists.)

>>  >>> import numpy as np
>>  >>> np.tile(3.0, (2,3))
>> array([[ 3.,  3.,  3.],
>>         [ 3.,  3.,  3.]])

>> If someone explained this, sorry to
>> have missed it.


On 6/29/2013 5:25 PM, Robert Kern wrote:
> It's implemented inefficiently.

> It is aimed at a different use case (building up arrays
> from other arrays) that only incidentally, and thus
> poorly, fulfils this one.

> It has no relation to the empty()/ones()/zeros() line of
> functions. In particular, tile_like() would make no sense.

> Being aimed at a different use case, it will be more
> difficult to find for people who are not familiar with
> numpy's entire API. It will be an isolated idiom rather
> than a natural completion of the existing APIs.




I do not understand this reply.

1. efficiency: noted in my original post, and I believe
it could easily be fixed.  Am I wrong?

2. use case: I assume this is the efficiency argument in
a new guise?

3. grammar: if there were an obvious choice, this discussion
would have ceased long ago: ``tile`` is as good as any for
this reason, and I don't see the problem with ``tile_like``.

4. isolated idiom (i.e., 3. again): the only way around this
is to use ``constant``, which is in use in other languages.
This has been vetoed as confusing terminology.
The problems with each other suggestion have been covered
extensively in this and related threads.

Honestly, I don't have a dog in this race.  It just seems
that ``tile`` already exists, is easily fixed to do what is
wanted (or is this wrong?), and is tremendously easy to
remember once seen.

Cheers,
Alan Isaac



From robert.kern at gmail.com  Sat Jun 29 18:15:45 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 29 Jun 2013 23:15:45 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <51CF576C.8090702@gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
	<CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
	<CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
	<51CF49AE.6050908@gmail.com>
	<CAF6FJitJsaQLYQ0i8CYPmcDo7bxOZES18h48=1R4VHz8j8_1KA@mail.gmail.com>
	<51CF576C.8090702@gmail.com>
Message-ID: <CAF6FJivDykODHWgPOTdFspDvWoG6StL75ZWrv-90ao6buv4SLA@mail.gmail.com>

On Sat, Jun 29, 2013 at 10:53 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
>>>
>>> On 6/29/2013 3:00 PM, Nathaniel wrote:
>>>>
>>>> any objections to np.full?
>
>
>
>> On Sat, Jun 29, 2013 at 9:55 PM, Alan G Isaac wrote:
>>>
>>> Still curious:
>>> why isn't ``tile`` the right name?
>>> (It already exists.)
>
>
>>>  >>> import numpy as np
>>>  >>> np.tile(3.0, (2,3))
>>> array([[ 3.,  3.,  3.],
>>>         [ 3.,  3.,  3.]])
>
>
>>> If someone explained this, sorry to
>>> have missed it.
>
>
>
> On 6/29/2013 5:25 PM, Robert Kern wrote:
>>
>> It's implemented inefficiently.
>
>
>> It is aimed at a different use case (building up arrays
>> from other arrays) that only incidentally, and thus
>> poorly, fulfils this one.
>
>
>> It has no relation to the empty()/ones()/zeros() line of
>> functions. In particular, tile_like() would make no sense.
>
>
>> Being aimed at a different use case, it will be more
>> difficult to find for people who are not familiar with
>> numpy's entire API. It will be an isolated idiom rather
>> than a natural completion of the existing APIs.
>
>
>
>
>
> I do not understand this reply.
>
> 1. efficiency: noted in my original post, and I believe
> it could easily be fixed.  Am I wrong?

With more special cases, sure.

> 2. use case: I assume this is the efficiency argument in
> a new guise?

No. It does explain the inefficiency, though.

> 3. grammar: if there were an obvious choice, this discussion
> would have ceased long ago: ``tile`` is as good as any for
> this reason,

I'm not referring to the names. ones()/zeros()/empty() are designed and
documented for similar use cases. You describe them in very similar ways:
"in order to make an array of a given shape and dtype filled with
(1s/0s/nothing in particular), use ones()/zeros()/empty()". You should
never describe tile() with a similar sentence. It happens to do something
similar given a very specific corner case, but that is not what the
function does and that is not what it is for.

> and I don't see the problem with ``tile_like``.

It makes no sense except in the scalar case.

> 4. isolated idiom (i.e., 3. again): the only way around this
> is to use ``constant``, which is in use in other languages.
> This has been vetoed as confusing terminology.
> The problems with each other suggestion have been covered
> extensively in this and related threads.

Again, I'm not talking about the name or our relationship to other
languages. Abusing tile() for this use case (which is *not* the use case it
is designed and documented for) would make it a numpy idiom, just something
that you have to be told and remember, much like the current idiom that we
recommend:

  np.ones(shape) * value

It is the complete API of tile() and the fact that it only happens to meet
the use case by abusing a small corner of its API that makes it use for
this an idiom. No matter what we name the function in the PR, it won't be
an idiom in the way I mean here.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130629/8750e0ab/attachment.html>

From alan.isaac at gmail.com  Sat Jun 29 18:24:43 2013
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Sat, 29 Jun 2013 18:24:43 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAF6FJivDykODHWgPOTdFspDvWoG6StL75ZWrv-90ao6buv4SLA@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
	<CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
	<CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
	<51CF49AE.6050908@gmail.com>
	<CAF6FJitJsaQLYQ0i8CYPmcDo7bxOZES18h48=1R4VHz8j8_1KA@mail.gmail.com>
	<51CF576C.8090702@gmail.com>
	<CAF6FJivDykODHWgPOTdFspDvWoG6StL75ZWrv-90ao6buv4SLA@mail.gmail.com>
Message-ID: <51CF5EAB.4060102@gmail.com>

>> and I don't see the problem with ``tile_like``.


On 6/29/2013 6:15 PM, Robert Kern wrote:
> It makes no sense except in the scalar case.


I would think it makes sense in every case that
can be normally broadcast to the shape of the
paradigm array.

Anyway, I drop the suggestion.

Cheers,
Alan



From josef.pktd at gmail.com  Sat Jun 29 21:15:35 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 29 Jun 2013 21:15:35 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
	<CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
	<CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
Message-ID: <CAMMTP+CDbRku-4Fd3Bi_a3zzbvzRXfctXBQct8rXh0v3Lw92Vw@mail.gmail.com>

On Sat, Jun 29, 2013 at 3:00 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
>
> On Sat, Jun 29, 2013 at 6:43 PM, <josef.pktd at gmail.com> wrote:
>>
>> On Sat, Jun 29, 2013 at 12:04 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> >
>> > On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs at pobox.com> wrote:
>> >>
>> >> So this petered off...any objections to np.full?
>
>
> No objection.

full full full

np.full  full of what?

full of the number it's filledwith

no objections so far, from what I remember of this thread

we take an empty array, fill it and then it's "full"

Josef



>
>>
>> > How about `np.inited` ?  `full` doesn't sound quite right to me.
>> > Although I
>> > expect once it comes into use everyone will get used to it.
>>
>> Is this pronounce like "ignited" ?
>
>
> I guess ``__init__``-ed. Not very clear I'm afraid.
>
> Ralf
>
>> Josef
>> German speaker
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From charlesr.harris at gmail.com  Sat Jun 29 21:23:17 2013
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 29 Jun 2013 19:23:17 -0600
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAMMTP+CDbRku-4Fd3Bi_a3zzbvzRXfctXBQct8rXh0v3Lw92Vw@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
	<CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
	<CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
	<CAMMTP+CDbRku-4Fd3Bi_a3zzbvzRXfctXBQct8rXh0v3Lw92Vw@mail.gmail.com>
Message-ID: <CAB6mnxLQhRitnqN7uXGsdi2Os5_24AZ2E=PighbhBY4N+aNAuw@mail.gmail.com>

On Sat, Jun 29, 2013 at 7:15 PM, <josef.pktd at gmail.com> wrote:

> On Sat, Jun 29, 2013 at 3:00 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >
> >
> >
> > On Sat, Jun 29, 2013 at 6:43 PM, <josef.pktd at gmail.com> wrote:
> >>
> >> On Sat, Jun 29, 2013 at 12:04 PM, Charles R Harris
> >> <charlesr.harris at gmail.com> wrote:
> >> >
> >> >
> >> > On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs at pobox.com>
> wrote:
> >> >>
> >> >> So this petered off...any objections to np.full?
> >
> >
> > No objection.
>
> full full full
>
> np.full  full of what?
>
> full of the number it's filledwith
>
> no objections so far, from what I remember of this thread
>
> we take an empty array, fill it and then it's "full"
>

You mean "fulled"? ;)

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130629/b295fdec/attachment.html>

From josef.pktd at gmail.com  Sat Jun 29 21:31:56 2013
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 29 Jun 2013 21:31:56 -0400
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAB6mnxLQhRitnqN7uXGsdi2Os5_24AZ2E=PighbhBY4N+aNAuw@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
	<CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
	<CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
	<CAMMTP+CDbRku-4Fd3Bi_a3zzbvzRXfctXBQct8rXh0v3Lw92Vw@mail.gmail.com>
	<CAB6mnxLQhRitnqN7uXGsdi2Os5_24AZ2E=PighbhBY4N+aNAuw@mail.gmail.com>
Message-ID: <CAMMTP+B3L143Nivu1nXJYs3iHQJTGwz9+HkZr_V8qLTw7Lf+fA@mail.gmail.com>

On Sat, Jun 29, 2013 at 9:23 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Sat, Jun 29, 2013 at 7:15 PM, <josef.pktd at gmail.com> wrote:
>>
>> On Sat, Jun 29, 2013 at 3:00 PM, Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>> >
>> >
>> >
>> > On Sat, Jun 29, 2013 at 6:43 PM, <josef.pktd at gmail.com> wrote:
>> >>
>> >> On Sat, Jun 29, 2013 at 12:04 PM, Charles R Harris
>> >> <charlesr.harris at gmail.com> wrote:
>> >> >
>> >> >
>> >> > On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs at pobox.com>
>> >> > wrote:
>> >> >>
>> >> >> So this petered off...any objections to np.full?
>> >
>> >
>> > No objection.
>>
>> full full full
>>
>> np.full  full of what?
>>
>> full of the number it's filledwith
>>
>> no objections so far, from what I remember of this thread
>>
>> we take an empty array, fill it and then it's "full"
>
>
> You mean "fulled"? ;)

It's really "full", if you want to put something else in there, you
need to replace something that is already in there.

Fortunately, arrays are mutable and not "constant"

(However np.empty doesn't give you an array that's really empty.)

Josef


>
> <snip>
>
> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From njs at pobox.com  Sun Jun 30 13:11:00 2013
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 30 Jun 2013 18:11:00 +0100
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAMMTP+CDbRku-4Fd3Bi_a3zzbvzRXfctXBQct8rXh0v3Lw92Vw@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
	<CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
	<CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
	<CAMMTP+CDbRku-4Fd3Bi_a3zzbvzRXfctXBQct8rXh0v3Lw92Vw@mail.gmail.com>
Message-ID: <CAPJVwBksEB0NvSAvwjXoFp1OOOu0hPvhmA5Mjnci-=58jQF8EQ@mail.gmail.com>

On Sun, Jun 30, 2013 at 2:15 AM,  <josef.pktd at gmail.com> wrote:
> On Sat, Jun 29, 2013 at 3:00 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>> On Sat, Jun 29, 2013 at 6:43 PM, <josef.pktd at gmail.com> wrote:
>>>
>>> On Sat, Jun 29, 2013 at 12:04 PM, Charles R Harris
>>> <charlesr.harris at gmail.com> wrote:
>>> >
>>> >
>>> > On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>> >>
>>> >> So this petered off...any objections to np.full?
>>
>>
>> No objection.
>
> full full full

Great, then let's put an end to this! Merged.

-n


From ralf.gommers at gmail.com  Sun Jun 30 15:18:29 2013
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 30 Jun 2013 21:18:29 +0200
Subject: [Numpy-discussion] numpy.filled, again
In-Reply-To: <CAPJVwBksEB0NvSAvwjXoFp1OOOu0hPvhmA5Mjnci-=58jQF8EQ@mail.gmail.com>
References: <CAPJVwBkBKNgZmyDQqgpyYP1P=NgYL5=njpCbWVXc0Dv2DP041Q@mail.gmail.com>
	<CAMMTP+AmJ8_72Y61-wV8qrhZkzoTU76VhLS_f2X5x4hgohUrcQ@mail.gmail.com>
	<CAPJVwBnTS-0kdBaCtoUGJ52tMgOzXyzBWrszaWQi+Nh9BQH4Ug@mail.gmail.com>
	<CAB6mnx+SuML6rNJa+Qko_P1ox2b-xA6jRBU9puLe7XTvcgWzMQ@mail.gmail.com>
	<CAMMTP+Dty37OahfAibg+roCudtw85yZSVVmN1Hch9cjiUThouw@mail.gmail.com>
	<CABL7CQgNf1kjCJ0cqtEoAqsAWntQ5CojJZWMQ4vfd+ftVf6P2w@mail.gmail.com>
	<CAMMTP+CDbRku-4Fd3Bi_a3zzbvzRXfctXBQct8rXh0v3Lw92Vw@mail.gmail.com>
	<CAPJVwBksEB0NvSAvwjXoFp1OOOu0hPvhmA5Mjnci-=58jQF8EQ@mail.gmail.com>
Message-ID: <CABL7CQgH-uKUVWLAE4AZd1+9hvB2TfGv6BjM0CAJawjvXjBKUQ@mail.gmail.com>

On Sun, Jun 30, 2013 at 7:11 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Sun, Jun 30, 2013 at 2:15 AM,  <josef.pktd at gmail.com> wrote:
> > On Sat, Jun 29, 2013 at 3:00 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >> On Sat, Jun 29, 2013 at 6:43 PM, <josef.pktd at gmail.com> wrote:
> >>>
> >>> On Sat, Jun 29, 2013 at 12:04 PM, Charles R Harris
> >>> <charlesr.harris at gmail.com> wrote:
> >>> >
> >>> >
> >>> > On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs at pobox.com>
> wrote:
> >>> >>
> >>> >> So this petered off...any objections to np.full?
> >>
> >>
> >> No objection.
> >
> > full full full
>
> Great, then let's put an end to this! Merged.


Great.

Special thanks to Eric Firing for fixing some np.ma bugs in response to a
"that module is not maintained" statement.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130630/3b09b6db/attachment.html>