From ndbecker2 at gmail.com  Fri Nov  2 10:18:32 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 02 Nov 2012 10:18:32 -0400
Subject: [Numpy-discussion] yet another trivial question
Message-ID: <k70knm$fjl$1@ger.gmane.org>

I'm trying to convert some matlab code.  I see this:

b(1)=[];

AFAICT, this removes the first element of the array, shifting the others.

What is the preferred numpy equivalent?

I'm not sure if

b[:] = b[1:]

is safe or not



From jkington at wisc.edu  Fri Nov  2 11:16:21 2012
From: jkington at wisc.edu (Joe Kington)
Date: Fri, 02 Nov 2012 10:16:21 -0500
Subject: [Numpy-discussion] yet another trivial question
In-Reply-To: <CACe1pJdc6wMFA4sWUhQuPjdLJ5UZpRd7GSNBVtb0aBqLjst_hQ@mail.gmail.com>
References: <k70knm$fjl$1@ger.gmane.org>
	<CACe1pJdc6wMFA4sWUhQuPjdLJ5UZpRd7GSNBVtb0aBqLjst_hQ@mail.gmail.com>
Message-ID: <CACe1pJeqMqw+MW53kuwm43+juJKTADk3VrgeiX+_E=Q1v7jwDQ@mail.gmail.com>

On Fri, Nov 2, 2012 at 9:18 AM, Neal Becker <ndbecker2 at gmail.com> wrote:

> I'm trying to convert some matlab code.  I see this:
>
> b(1)=[];
>
> AFAICT, this removes the first element of the array, shifting the others.
>
> What is the preferred numpy equivalent?
>
> I'm not sure if
>
> b[:] = b[1:]
>

Unless I'm missing something, don't you just want:

    b = b[1:]


>
> is safe or not
>


It's not exactly the same as the matlab equivalent, as matlab will always
make a copy, and this will be a view of the same array.  For example, if
you do something like this:

    import numpy as np

    a = np.arange(10)

    b = a[1:]

    b[3] = 1000

    print a
    print b

You'll see that modifying "b" in-place will modify "a" as well, as "b" is
just a view into "a".  This wouldn't be the case in matlab (if I remember
correctly, anyway...).

> _________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121102/9f9c9171/attachment.html>

From d.warde.farley at gmail.com  Fri Nov  2 15:31:12 2012
From: d.warde.farley at gmail.com (David Warde-Farley)
Date: Fri, 2 Nov 2012 15:31:12 -0400
Subject: [Numpy-discussion] yet another trivial question
In-Reply-To: <CACe1pJeqMqw+MW53kuwm43+juJKTADk3VrgeiX+_E=Q1v7jwDQ@mail.gmail.com>
References: <k70knm$fjl$1@ger.gmane.org>
	<CACe1pJdc6wMFA4sWUhQuPjdLJ5UZpRd7GSNBVtb0aBqLjst_hQ@mail.gmail.com>
	<CACe1pJeqMqw+MW53kuwm43+juJKTADk3VrgeiX+_E=Q1v7jwDQ@mail.gmail.com>
Message-ID: <CALc6Xo7=TMYooC3Yids5GLQj6MGJpdjZHNWvyd=Yr+Jr+Vh1CQ@mail.gmail.com>

On Fri, Nov 2, 2012 at 11:16 AM, Joe Kington <jkington at wisc.edu> wrote:

> On Fri, Nov 2, 2012 at 9:18 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
>
>> I'm trying to convert some matlab code.  I see this:
>>
>> b(1)=[];
>>
>> AFAICT, this removes the first element of the array, shifting the others.
>>
>> What is the preferred numpy equivalent?
>>
>> I'm not sure if
>>
>> b[:] = b[1:]
>>
>
> Unless I'm missing something, don't you just want:
>
>     b = b[1:]
>
>
>>
>> is safe or not
>>
>
>
> It's not exactly the same as the matlab equivalent, as matlab will always
> make a copy, and this will be a view of the same array.  For example, if
> you do something like this:
>
>     import numpy as np
>
>     a = np.arange(10)
>
>     b = a[1:]
>
>     b[3] = 1000
>
>     print a
>     print b
>
> You'll see that modifying "b" in-place will modify "a" as well, as "b" is
> just a view into "a".  This wouldn't be the case in matlab (if I remember
> correctly, anyway...)
>

AFAIK that's correct. The closest thing to Matlab's semantics would
probably "b = b[1:].copy()".

b[:] = b[1:] would probably result in an error, as the RHS has a
first-dimension shape one less than the LHS.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121102/15e15f43/attachment.html>

From njs at pobox.com  Fri Nov  2 15:54:33 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 2 Nov 2012 19:54:33 +0000
Subject: [Numpy-discussion] yet another trivial question
In-Reply-To: <k70knm$fjl$1@ger.gmane.org>
References: <k70knm$fjl$1@ger.gmane.org>
Message-ID: <CAPJVwBmSHX3ALBt-twFqvJOorxzE8h5SPE5mDcGD1yf77USnCw@mail.gmail.com>

On Fri, Nov 2, 2012 at 2:18 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> I'm trying to convert some matlab code.  I see this:
>
> b(1)=[];
>
> AFAICT, this removes the first element of the array, shifting the others.
>
> What is the preferred numpy equivalent?

Perhaps np.delete is what you're looking for. (Esp. if you have
multiple items to remove like this, or want to remove something
besides the first element in the array.)

-n


From ralf.gommers at gmail.com  Sun Nov  4 14:31:35 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 4 Nov 2012 20:31:35 +0100
Subject: [Numpy-discussion] ticket 2228: Scientific package seeing ABI
 change in 1.6.x
In-Reply-To: <CAB6mnxLLvkrBAYUvkjfqgGDyGG2Yj_e1xCyK2Z1uOc2VKtfzSg@mail.gmail.com>
References: <CABL7CQja6OdMgtXZBVa7CdJh9Tf_5A4P4LDcrU_1GuVA=s0FcA@mail.gmail.com>
	<CAB6mnx+afjTR1ThMQZ5Cs-0xWO=r-jjTZTBvW4+_63uQ25bXBA@mail.gmail.com>
	<ADBF24E7-CEAE-450A-8E37-8A19D3337424@continuum.io>
	<CAB6mnxLLvkrBAYUvkjfqgGDyGG2Yj_e1xCyK2Z1uOc2VKtfzSg@mail.gmail.com>
Message-ID: <CABL7CQjTGhX1nAidkCcFB3q-EgBqxJcH3yTF=MaoZ+h+aEF_wg@mail.gmail.com>

On Wed, Oct 31, 2012 at 1:05 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Tue, Oct 30, 2012 at 9:26 PM, Travis Oliphant <travis at continuum.io>wrote:
>
>> The NPY_CHAR is not a "real type".   There are no type-coercion functions
>> attached to it nor ufuncs nor a full dtype object.      However, it is used
>> to mimic old Numeric character arrays (especially for copying a string).
>>
>> It should have been deprecated before changing the ABI.  I don't think it
>> was realized that it was part of the ABI (mostly for older codes that
>> depended on Numeric).   I think it was just another oversight that
>> inserting type-codes changes this part of the ABI.
>>
>> The positive side is that It's a small part of the ABI and not many codes
>> should depend on it.   At this point, I'm not sure what can be done, except
>> to document that NPY_CHAR has been deprecated in 1.7.0 and remove it in
>> 1.8.0 to avoid future ABI difficulties.
>>
>> The short answer, is that codes that use NPY_CHAR must be recompiled to
>> be compatible with 1.6.0.
>>
>>
> IIRC, it was proposed to remove it at one point, but the STScI folks
> wanted to keep it because their software depended on it.
>

I can't find that discussion in the list archives. If you know who from
STScI to ask about this, can you do so?

Is replacing NPY_CHAR with NPY_STRING supposed to just work?

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121104/d6382d68/attachment.html>

From travis at continuum.io  Sun Nov  4 21:47:12 2012
From: travis at continuum.io (Travis Oliphant)
Date: Sun, 4 Nov 2012 20:47:12 -0600
Subject: [Numpy-discussion] ticket 2228: Scientific package seeing ABI
	change in 1.6.x
In-Reply-To: <CABL7CQjTGhX1nAidkCcFB3q-EgBqxJcH3yTF=MaoZ+h+aEF_wg@mail.gmail.com>
References: <CABL7CQja6OdMgtXZBVa7CdJh9Tf_5A4P4LDcrU_1GuVA=s0FcA@mail.gmail.com>
	<CAB6mnx+afjTR1ThMQZ5Cs-0xWO=r-jjTZTBvW4+_63uQ25bXBA@mail.gmail.com>
	<ADBF24E7-CEAE-450A-8E37-8A19D3337424@continuum.io>
	<CAB6mnxLLvkrBAYUvkjfqgGDyGG2Yj_e1xCyK2Z1uOc2VKtfzSg@mail.gmail.com>
	<CABL7CQjTGhX1nAidkCcFB3q-EgBqxJcH3yTF=MaoZ+h+aEF_wg@mail.gmail.com>
Message-ID: <346E696A-CE25-48A0-9536-095C795851F5@continuum.io>


On Nov 4, 2012, at 1:31 PM, Ralf Gommers wrote:

> 
> 
> 
> On Wed, Oct 31, 2012 at 1:05 PM, Charles R Harris <charlesr.harris at gmail.com> wrote:
> 
> 
> On Tue, Oct 30, 2012 at 9:26 PM, Travis Oliphant <travis at continuum.io> wrote:
> The NPY_CHAR is not a "real type".   There are no type-coercion functions attached to it nor ufuncs nor a full dtype object.      However, it is used to mimic old Numeric character arrays (especially for copying a string).  
> 
> It should have been deprecated before changing the ABI.  I don't think it was realized that it was part of the ABI (mostly for older codes that depended on Numeric).   I think it was just another oversight that inserting type-codes changes this part of the ABI.     
> 
> The positive side is that It's a small part of the ABI and not many codes should depend on it.   At this point, I'm not sure what can be done, except to document that NPY_CHAR has been deprecated in 1.7.0 and remove it in 1.8.0 to avoid future ABI difficulties.    
> 
> The short answer, is that codes that use NPY_CHAR must be recompiled to be compatible with 1.6.0.
> 
> 
> IIRC, it was proposed to remove it at one point, but the STScI folks wanted to keep it because their software depended on it.
> 
> I can't find that discussion in the list archives. If you know who from STScI to ask about this, can you do so? 
> 
> Is replacing NPY_CHAR with NPY_STRING supposed to just work?

No, it's a little more complicated than that, but not too much.      Code that uses the NPY_CHAR type can be changed fairly easily to use the NPY_STRING type, but it does take some re-writing.   The NPY_CHAR field was added so that code written for Numeric (like ScientificPython's netcdf reader) would continue to "just work" with no changes and behave similarly to how it behaved with Numeric's character type.   

Unfortunately, adding it to the end of the TYPE list does prevent adding any more types without breaking at least this part of the ABI.  

-Travis


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121104/cc5253bf/attachment.html>

From travis at continuum.io  Tue Nov  6 02:33:41 2012
From: travis at continuum.io (Travis Oliphant)
Date: Tue, 6 Nov 2012 01:33:41 -0600
Subject: [Numpy-discussion] NumFOCUS has received 501(c)3 status
Message-ID: <746F39AF-FF3B-4E03-8AB2-332E94E6F5A2@continuum.io>

Hello all, 

I'm really happy to report that NumFOCUS has received it's 501(c)3 status from the IRS.   You can now make tax-deductible donations to NumFOCUS for the support of NumPy.  We will put a NumPy-specific button on the home-page of NumPy soon so you can specifically direct your funds.    But,  for now you can go to http://numfocus.org/donate and be confident that your funds will support: 

	1) Continuous integration 
	2) The John Hunter Technical fellowships (which are awards made to students and post-docs and their mentors who will contribute substantially to a supported project during a 3-18 month period).  
	3) Equipment grants
	4) Development sprints 
	5) Student travel to conferences 
        6) Project specific grants

For example, most of Ondrej's time to work on the release of NumPy 1.7.0 has been paid for by donations to NumFOCUS from Continuum Analytics. 

NumFOCUS is also seeking nominations for 5 new board members (to bring the total to 9).   If you would like to nominate someone please subscribe to numfocus at googlegroups.com (by sending an email to numfocus+subscribe at googlegroups.com) and then send your nomination.   Alternatively, you can email me or one of the other directors directly. 

Best, 

-Travis




From travis at continuum.io  Tue Nov  6 02:33:50 2012
From: travis at continuum.io (Travis Oliphant)
Date: Tue, 6 Nov 2012 01:33:50 -0600
Subject: [Numpy-discussion] 1.7.0 release
Message-ID: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>

Hey all, 

Ondrej has been tied up finishing his PhD for the past several weeks.  He is defending his work shortly and should be available to continue to help with the 1.7.0 release around the first of December.    He and I have been in contact during this process, and I've been helping where I can.   Fortunately, other NumPy developers have been active closing tickets and reviewing pull requests which has helped the process substantially. 

The release has taken us longer than we expected, but I'm really glad that we've received the bug-reports and issues that we have seen because it will help the 1.7.0 release be a more stable series.   Also, the merging of the Trac issues with Git has exposed over-looked problems as well and will hopefully encourage more Git-focused participation by users. 

We are targeting getting the final release of 1.7.0 out by mid December (based on Ondrej's availability).   But, I would like to find out which issues are seen as blockers by people on this list.   I think most of the issues that I had as blockers have been resolved.    If there are no more remaining blockers, then we may be able to accelerate the final release of 1.7.0 to just after Thanksgiving. 

Best regards,

-Travis






From lists at hilboll.de  Tue Nov  6 09:59:38 2012
From: lists at hilboll.de (Andreas Hilboll)
Date: Tue, 6 Nov 2012 15:59:38 +0100
Subject: [Numpy-discussion] pyhdf packaging
Message-ID: <bd3b168e7b9d9ed612f262aa7e426f16.squirrel@srv2.s4y.tournesol-consulting.eu>

Hi,

I would like to package pyhdf for Ubuntu and make the package publicly
available. Since the license is not totally clear to me (I cannot find any
information in the sources, and the cheeseshop says "public", which
doesn't mean anything to me), I tried to contact the maintainer, Andre
Gosselin, however the email bounces, so I guess he's gone.

Can anyone point me to how to proceed from here?

Cheers, Andreas.



From p.j.a.cock at googlemail.com  Tue Nov  6 12:49:39 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Tue, 6 Nov 2012 17:49:39 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
Message-ID: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>

Dear all,

Since the NumPy 1.7.0b2 release didn't include a Windows
(32 bit) installer for Python 3.3, I am considering compiling it
myself for local testing. What compiler is recommended?

Thanks,

Peter


From nouiz at nouiz.org  Tue Nov  6 16:16:23 2012
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Tue, 6 Nov 2012 16:16:23 -0500
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
Message-ID: <CADKKbtg5rfX0xQgN3kDhyQexcUQKOAR_MGxx+SSFJgtdaTP9Zw@mail.gmail.com>

Hi,

I updated the numpy master and recompiled it. I still have the compilation
error I got from Theano. I'll pop up that email thread again to have the
history and I made a PR for this.

Also, I think I said that numpy.ndindex changed its interface, in the past
numpy.ndindex() was valid, not this raise and error:

>>> import numpy
>>> a=numpy.ndindex()
>>> a.next()
()
>>> a.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/opt/lisa/os/epd-7.1.2/lib/python2.7/site-packages/numpy/lib/index_tricks.py",
line 577, in next
    raise StopIteration
StopIteration
>>> numpy.__version__
'1.6.1'


The error I have with master:

[...]
ValueError: __array_interface__ shape must be at least size 1

That is the only stopper I saw, but I didn't followed what was needed for
other people.

Fred


On Tue, Nov 6, 2012 at 2:33 AM, Travis Oliphant <travis at continuum.io> wrote:

> Hey all,
>
> Ondrej has been tied up finishing his PhD for the past several weeks.  He
> is defending his work shortly and should be available to continue to help
> with the 1.7.0 release around the first of December.    He and I have been
> in contact during this process, and I've been helping where I can.
> Fortunately, other NumPy developers have been active closing tickets and
> reviewing pull requests which has helped the process substantially.
>
> The release has taken us longer than we expected, but I'm really glad that
> we've received the bug-reports and issues that we have seen because it will
> help the 1.7.0 release be a more stable series.   Also, the merging of the
> Trac issues with Git has exposed over-looked problems as well and will
> hopefully encourage more Git-focused participation by users.
>
> We are targeting getting the final release of 1.7.0 out by mid December
> (based on Ondrej's availability).   But, I would like to find out which
> issues are seen as blockers by people on this list.   I think most of the
> issues that I had as blockers have been resolved.    If there are no more
> remaining blockers, then we may be able to accelerate the final release of
> 1.7.0 to just after Thanksgiving.
>
> Best regards,
>
> -Travis
>
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121106/6e4f7a6a/attachment.html>

From nouiz at nouiz.org  Tue Nov  6 16:17:09 2012
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Tue, 6 Nov 2012 16:17:09 -0500
Subject: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
In-Reply-To: <CAB6mnxLfZepA75=3mDR5=4yn_M_RrujHbRVF7D3icmnC96LBKg@mail.gmail.com>
References: <CADKKbtgG0RpoRd+Z1SSyAAAJ6QkwDVJSU4LGqLTaAtqzi_aTZQ@mail.gmail.com>
	<CAB6mnx+_07+zq0gEERq38mHkj1NgWQ2pUi4ghMv710N8rRDtzw@mail.gmail.com>
	<CADKKbtgJsaW9QqA4EO_gqUGyeVjV5NRKKH=zC20Rzk6z1TYabQ@mail.gmail.com>
	<CAB6mnxJgELd04bd5o_+z7JK8N9FTNrt72aTL-VsmJ4-yfC8cKw@mail.gmail.com>
	<CADKKbtiYKyxGVnnxCnanQM0c5k7R__cE8mMeNNaejy8NhOXdnA@mail.gmail.com>
	<CAB6mnxK8rex7dQTAH52U25z=mfZ4+XJCdiGCtfQrxehQDo1S0w@mail.gmail.com>
	<CAB6mnx+Uj_18yw0ViinqBSyfvoKPN=QouA+HGYsVkXagd1ZD5g@mail.gmail.com>
	<CADKKbtiPVda8gKanF8UOyEjCQxxaB2K5ua8wriTjSOjtqYz-JQ@mail.gmail.com>
	<CAB6mnxLfZepA75=3mDR5=4yn_M_RrujHbRVF7D3icmnC96LBKg@mail.gmail.com>
Message-ID: <CADKKbtgp3DXSve4NrCy7ODDBogUX7D8-R+dSwsjBb8LEMW2xXg@mail.gmail.com>

Hi,

I made a PR with my fix:

https://github.com/numpy/numpy/pull/2709

Fr?d?ric


On Tue, Oct 2, 2012 at 6:18 PM, Charles R Harris
<charlesr.harris at gmail.com>wrote:

>
>
> On Tue, Oct 2, 2012 at 1:44 PM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
>
>> With numpy 1.6.2, it is working. So this is an interface change. Are
>> you sure you want this? This break existing code.
>>
>> I do not understand what you mean by slot?
>>
>
> Pythonese for structure member ;)
>
>
>>
>> I'm not sure is the PyArray_SWAP is a good long term idea. I would not
>> make it if it is only for temporarily.
>>
>
> The C++ stdlib provides something similar for std::vector. One common use
> case would be to pass in a vector by reference that gets swapped with one
> on the stack. When the function exits the one on the stack is cleaned up
> and the vector that was passed in has the new data, but it has to be the
> same type.
>
> For PyArray_SWAP I was thinking of swapping everything: type, dims,
> strides, data, etc. That is what f2py does.
>
>
>> To set the base ptr, there is PyArray_SetBaseObject() fct that is new
>> in 1.7. Is a similar fct useful in the long term for numpy? In the
>> case where we implement differently the ndarray object, I think it
>> won't be useful. We will also need to know how the memory is laid out
>> by numpy for performance critical code. We we will need an attribute
>> that tell the intern structure used.
>>
>> So do you want to force this interface change in numpy 1.7 so that I
>> need to write codes now or can I wait to do it when you force the new
>> interface?
>>
>>
> Well, no we don't want to force you to use the new interface. If you don't
> define NPY_NO_DEPRECATED_API things should still work. Although if it is
> defined the function returns an rvalue, so some other method needs to be
> provided for what you are doing.
>
>
>> Currently the used code for PyArray_BYTES is:
>>
>> #define PyArray_BYTES(obj) ((char *)(((PyArrayObject_fields
>> *)(obj))->data))
>>
>> if I change it to
>>
>> #define PyArray_BYTES(obj) ((((PyArrayObject_fields *)(obj))->data))
>>
>> it work! I don't understand why removing the case make it work. the
>> data field is already an (char*), so this should not make a difference
>> to my underderstanding. But I'm missing something here, do someone
>> know?
>>
>
> What I find strange is that it is the same macro in 1.7 and 1.6, only the
> name of the structure was changed. Hmm... This looks almost like some
> compiler subtlety, I wonder if the compiler version/optimization flags have
> changed? In any case, I think the second form would be more correct for the
> lvalue since the structure member is, as you say, already a char*.
>
> We want things to work for you as they should, so we need to understand
> this and fix it.
>
> <snip>
>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121106/f34e924a/attachment.html>

From phillip.m.feldman at gmail.com  Tue Nov  6 21:27:21 2012
From: phillip.m.feldman at gmail.com (Phillip Feldman)
Date: Tue, 6 Nov 2012 18:27:21 -0800
Subject: [Numpy-discussion] strange behavior of numpy.unique
Message-ID: <CAB2ViTazH00kRhEuap1F+07DRaM9B+6Z1sVjWm4y5GJ3f6UOaA@mail.gmail.com>

numpy.unique behaves as I would expect for small inputs like the following:

In [12]: x= [0, 0, 1, 0, 1, 2, 0, 1, 2, 3]

In [13]: unique(x, return_index=True)
Out[13]: (array([0, 1, 2, 3]), array([0, 2, 5, 9], dtype=int64))

But, when I give it something larger, the return index values do not always
correspond to the first occurrences in the input. The documentation is
silent on the question of how the return index values are chosen when a
given element of x appears more than once. Either the documentation should
be
clarified, or better yet, the behavior should be changed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121106/36065c47/attachment.html>

From warren.weckesser at gmail.com  Tue Nov  6 21:52:24 2012
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Tue, 6 Nov 2012 20:52:24 -0600
Subject: [Numpy-discussion] strange behavior of numpy.unique
In-Reply-To: <CAB2ViTazH00kRhEuap1F+07DRaM9B+6Z1sVjWm4y5GJ3f6UOaA@mail.gmail.com>
References: <CAB2ViTazH00kRhEuap1F+07DRaM9B+6Z1sVjWm4y5GJ3f6UOaA@mail.gmail.com>
Message-ID: <CAGzF1udX9MCiKkSpKPpR_Xrgu6f2Svjb+c2UDuxcDfDQ=W9NBw@mail.gmail.com>

On Tue, Nov 6, 2012 at 8:27 PM, Phillip Feldman <phillip.m.feldman at gmail.com
> wrote:

> numpy.unique behaves as I would expect for small inputs like the following:
>
> In [12]: x= [0, 0, 1, 0, 1, 2, 0, 1, 2, 3]
>
> In [13]: unique(x, return_index=True)
> Out[13]: (array([0, 1, 2, 3]), array([0, 2, 5, 9], dtype=int64))
>
> But, when I give it something larger, the return index values do not
> always correspond to the first occurrences in the input. The documentation
> is silent on the question of how the return index values are chosen when a
> given element of x appears more than once. Either the documentation should
> be
> clarified, or better yet, the behavior should be changed.
>


In fact, it was changed (in the master branch on github) several months
ago, but there has not yet been a release with the changes.  The sort
method that np.unique passes to np.argsort is now 'mergesort', and the
docstring states that the indices returned are for the first occurrences of
the unique elements.  The new docstring is here:
http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.unique.html#numpy.unique

See
https://github.com/numpy/numpy/commit/dbf235169ed3386b359caaa9217f5280bf1d6749for
the commit, and
https://github.com/numpy/numpy/blob/master/numpy/lib/arraysetops.py for the
latest version of the source.

Warren



> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121106/c18cee8f/attachment.html>

From ndbecker2 at gmail.com  Wed Nov  7 07:35:24 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 07 Nov 2012 07:35:24 -0500
Subject: [Numpy-discussion] testing with amd libm/acml
Message-ID: <k7dkia$dii$1@ger.gmane.org>

I'm trying to do a bit of benchmarking to see if amd libm/acml will help me.

I got an idea that instead of building all of numpy/scipy and all of my custom 
modules against these libraries, I could simply use:

LD_PRELOAD=/opt/amdlibm-3.0.2/lib/dynamic/libamdlibm.so:/opt/acml5.2.0/gfortran64/lib/libacml.so 
<my program here>

I'm hoping that both numpy and my own dll's then will take advantage of these 
libraries.

Do you think this will work?



From cournape at gmail.com  Wed Nov  7 08:34:17 2012
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 7 Nov 2012 13:34:17 +0000
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <k7dkia$dii$1@ger.gmane.org>
References: <k7dkia$dii$1@ger.gmane.org>
Message-ID: <CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>

On Wed, Nov 7, 2012 at 12:35 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> I'm trying to do a bit of benchmarking to see if amd libm/acml will help me.
>
> I got an idea that instead of building all of numpy/scipy and all of my custom
> modules against these libraries, I could simply use:
>
> LD_PRELOAD=/opt/amdlibm-3.0.2/lib/dynamic/libamdlibm.so:/opt/acml5.2.0/gfortran64/lib/libacml.so
> <my program here>
>
> I'm hoping that both numpy and my own dll's then will take advantage of these
> libraries.
>
> Do you think this will work?

Quite unlikely depending on your configuration, because those
libraries are rarely if ever ABI compatible (that's why it is such a
pain to support).

David


From ndbecker2 at gmail.com  Wed Nov  7 08:56:41 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 07 Nov 2012 08:56:41 -0500
Subject: [Numpy-discussion] testing with amd libm/acml
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
Message-ID: <k7dpan$1ck$1@ger.gmane.org>

David Cournapeau wrote:

> On Wed, Nov 7, 2012 at 12:35 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>> I'm trying to do a bit of benchmarking to see if amd libm/acml will help me.
>>
>> I got an idea that instead of building all of numpy/scipy and all of my
>> custom modules against these libraries, I could simply use:
>>
>> 
LD_PRELOAD=/opt/amdlibm-3.0.2/lib/dynamic/libamdlibm.so:/opt/acml5.2.0/gfortran64/lib/libacml.so
>> <my program here>
>>
>> I'm hoping that both numpy and my own dll's then will take advantage of these
>> libraries.
>>
>> Do you think this will work?
> 
> Quite unlikely depending on your configuration, because those
> libraries are rarely if ever ABI compatible (that's why it is such a
> pain to support).
> 
> David

When you say quite unlikely (to work), you mean 

a) unlikely that libm/acml will be used to resolve symbols in numpy/dlls at 
runtime (e.g., exp)?

or 

b) program may produce wrong results and/or crash ?



From cournape at gmail.com  Wed Nov  7 09:06:41 2012
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 7 Nov 2012 14:06:41 +0000
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <k7dpan$1ck$1@ger.gmane.org>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
Message-ID: <CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>

On Wed, Nov 7, 2012 at 1:56 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> David Cournapeau wrote:
>
>> On Wed, Nov 7, 2012 at 12:35 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>>> I'm trying to do a bit of benchmarking to see if amd libm/acml will help me.
>>>
>>> I got an idea that instead of building all of numpy/scipy and all of my
>>> custom modules against these libraries, I could simply use:
>>>
>>>
> LD_PRELOAD=/opt/amdlibm-3.0.2/lib/dynamic/libamdlibm.so:/opt/acml5.2.0/gfortran64/lib/libacml.so
>>> <my program here>
>>>
>>> I'm hoping that both numpy and my own dll's then will take advantage of these
>>> libraries.
>>>
>>> Do you think this will work?
>>
>> Quite unlikely depending on your configuration, because those
>> libraries are rarely if ever ABI compatible (that's why it is such a
>> pain to support).
>>
>> David
>
> When you say quite unlikely (to work), you mean
>
> a) unlikely that libm/acml will be used to resolve symbols in numpy/dlls at
> runtime (e.g., exp)?
>
> or
>
> b) program may produce wrong results and/or crash ?

Both, actually. That's not something I would use myself. Did you try
openblas ? It is open source, simple to build, and is pretty fast,

David


From ndbecker2 at gmail.com  Wed Nov  7 09:28:32 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 07 Nov 2012 09:28:32 -0500
Subject: [Numpy-discussion] testing with amd libm/acml
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
Message-ID: <k7dr6e$jue$1@ger.gmane.org>

David Cournapeau wrote:

> On Wed, Nov 7, 2012 at 1:56 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>> David Cournapeau wrote:
>>
>>> On Wed, Nov 7, 2012 at 12:35 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>> I'm trying to do a bit of benchmarking to see if amd libm/acml will help
>>>> me.
>>>>
>>>> I got an idea that instead of building all of numpy/scipy and all of my
>>>> custom modules against these libraries, I could simply use:
>>>>
>>>>
>> 
LD_PRELOAD=/opt/amdlibm-3.0.2/lib/dynamic/libamdlibm.so:/opt/acml5.2.0/gfortran64/lib/libacml.so
>>>> <my program here>
>>>>
>>>> I'm hoping that both numpy and my own dll's then will take advantage of
>>>> these libraries.
>>>>
>>>> Do you think this will work?
>>>
>>> Quite unlikely depending on your configuration, because those
>>> libraries are rarely if ever ABI compatible (that's why it is such a
>>> pain to support).
>>>
>>> David
>>
>> When you say quite unlikely (to work), you mean
>>
>> a) unlikely that libm/acml will be used to resolve symbols in numpy/dlls at
>> runtime (e.g., exp)?
>>
>> or
>>
>> b) program may produce wrong results and/or crash ?
> 
> Both, actually. That's not something I would use myself. Did you try
> openblas ? It is open source, simple to build, and is pretty fast,
> 
> David

Actually, for my current work, I'm more concerned with speeding up operations 
such as exp, log and basic vector arithmetic.  Any thoughts on that?



From ndbecker2 at gmail.com  Wed Nov  7 09:30:28 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 07 Nov 2012 09:30:28 -0500
Subject: [Numpy-discussion] testing with amd libm/acml
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
Message-ID: <k7dra2$jue$2@ger.gmane.org>

David Cournapeau wrote:

> On Wed, Nov 7, 2012 at 1:56 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>> David Cournapeau wrote:
>>
>>> On Wed, Nov 7, 2012 at 12:35 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>> I'm trying to do a bit of benchmarking to see if amd libm/acml will help
>>>> me.
>>>>
>>>> I got an idea that instead of building all of numpy/scipy and all of my
>>>> custom modules against these libraries, I could simply use:
>>>>
>>>>
>> 
LD_PRELOAD=/opt/amdlibm-3.0.2/lib/dynamic/libamdlibm.so:/opt/acml5.2.0/gfortran64/lib/libacml.so
>>>> <my program here>
>>>>
>>>> I'm hoping that both numpy and my own dll's then will take advantage of
>>>> these libraries.
>>>>
>>>> Do you think this will work?
>>>
>>> Quite unlikely depending on your configuration, because those
>>> libraries are rarely if ever ABI compatible (that's why it is such a
>>> pain to support).
>>>
>>> David
>>
>> When you say quite unlikely (to work), you mean
>>
>> a) unlikely that libm/acml will be used to resolve symbols in numpy/dlls at
>> runtime (e.g., exp)?
>>
>> or
>>
>> b) program may produce wrong results and/or crash ?
> 
> Both, actually. That's not something I would use myself. Did you try
> openblas ? It is open source, simple to build, and is pretty fast,
> 
> David

In my current work, probably the largest bottlenecks are 'max*',  which are

log (\sum e^(x_i))




From d.s.seljebotn at astro.uio.no  Wed Nov  7 09:52:28 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 07 Nov 2012 15:52:28 +0100
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <k7dra2$jue$2@ger.gmane.org>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org>
Message-ID: <509A75AC.3010206@astro.uio.no>

On 11/07/2012 03:30 PM, Neal Becker wrote:
> David Cournapeau wrote:
>
>> On Wed, Nov 7, 2012 at 1:56 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>>> David Cournapeau wrote:
>>>
>>>> On Wed, Nov 7, 2012 at 12:35 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>>> I'm trying to do a bit of benchmarking to see if amd libm/acml will help
>>>>> me.
>>>>>
>>>>> I got an idea that instead of building all of numpy/scipy and all of my
>>>>> custom modules against these libraries, I could simply use:
>>>>>
>>>>>
>>>
> LD_PRELOAD=/opt/amdlibm-3.0.2/lib/dynamic/libamdlibm.so:/opt/acml5.2.0/gfortran64/lib/libacml.so
>>>>> <my program here>
>>>>>
>>>>> I'm hoping that both numpy and my own dll's then will take advantage of
>>>>> these libraries.
>>>>>
>>>>> Do you think this will work?
>>>>
>>>> Quite unlikely depending on your configuration, because those
>>>> libraries are rarely if ever ABI compatible (that's why it is such a
>>>> pain to support).
>>>>
>>>> David
>>>
>>> When you say quite unlikely (to work), you mean
>>>
>>> a) unlikely that libm/acml will be used to resolve symbols in numpy/dlls at
>>> runtime (e.g., exp)?
>>>
>>> or
>>>
>>> b) program may produce wrong results and/or crash ?
>>
>> Both, actually. That's not something I would use myself. Did you try
>> openblas ? It is open source, simple to build, and is pretty fast,
>>
>> David
>
> In my current work, probably the largest bottlenecks are 'max*',  which are
>
> log (\sum e^(x_i))

numexpr with Intel VML is the solution I know of that doesn't require 
you to dig into compiling C code yourself. Did you look into that or is 
using Intel VML/MKL not an option?

Fast exps depend on the CPU evaluating many exp's at the same time (both 
explicit through vector registers, and implicit through pipelining); 
even if you get what you try to work (which is unlikely I think) the 
approach is inherently slow, since just passing a single number at the 
time through the "exp" function can't be efficient.

Dag Sverre


From josef.pktd at gmail.com  Wed Nov  7 12:24:04 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 7 Nov 2012 12:24:04 -0500
Subject: [Numpy-discussion] strange behavior of numpy.unique
In-Reply-To: <CAGzF1udX9MCiKkSpKPpR_Xrgu6f2Svjb+c2UDuxcDfDQ=W9NBw@mail.gmail.com>
References: <CAB2ViTazH00kRhEuap1F+07DRaM9B+6Z1sVjWm4y5GJ3f6UOaA@mail.gmail.com>
	<CAGzF1udX9MCiKkSpKPpR_Xrgu6f2Svjb+c2UDuxcDfDQ=W9NBw@mail.gmail.com>
Message-ID: <CAMMTP+CXnxWZd7N4KZLdw0zMmVvoJw1nY+VJPKoQ=5PpR7Snrg@mail.gmail.com>

On Tue, Nov 6, 2012 at 9:52 PM, Warren Weckesser
<warren.weckesser at gmail.com> wrote:
>
>
> On Tue, Nov 6, 2012 at 8:27 PM, Phillip Feldman
> <phillip.m.feldman at gmail.com> wrote:
>>
>> numpy.unique behaves as I would expect for small inputs like the
>> following:
>>
>> In [12]: x= [0, 0, 1, 0, 1, 2, 0, 1, 2, 3]
>>
>> In [13]: unique(x, return_index=True)
>> Out[13]: (array([0, 1, 2, 3]), array([0, 2, 5, 9], dtype=int64))
>>
>> But, when I give it something larger, the return index values do not
>> always correspond to the first occurrences in the input. The documentation
>> is silent on the question of how the return index values are chosen when a
>> given element of x appears more than once. Either the documentation should
>> be
>> clarified, or better yet, the behavior should be changed.
>
>
>
> In fact, it was changed (in the master branch on github) several months ago,
> but there has not yet been a release with the changes.  The sort method that
> np.unique passes to np.argsort is now 'mergesort', and the docstring states
> that the indices returned are for the first occurrences of the unique
> elements.  The new docstring is here:
> http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.unique.html#numpy.unique
>
> See
> https://github.com/numpy/numpy/commit/dbf235169ed3386b359caaa9217f5280bf1d6749
> for the commit, and
> https://github.com/numpy/numpy/blob/master/numpy/lib/arraysetops.py for the
> latest version of the source.

I think it's in 1.6.2 and it broke return_index for structured dtypes, IIRC.

Josef


>
> Warren
>
>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From ocefpaf at gmail.com  Wed Nov  7 13:28:23 2012
From: ocefpaf at gmail.com (Filipe Pires Alvarenga Fernandes)
Date: Wed, 7 Nov 2012 16:28:23 -0200
Subject: [Numpy-discussion] Help compiling numpy with new gcc
Message-ID: <CAH2VmmCa0tppLEdndODca++9=Fpv8Jk5rpKwjDtT7Cv9LVbzDw@mail.gmail.com>

Hi I am trying to compile numpy with gcc 4.7.1 and I am having the
following issue.

"RuntimeError: Broken toolchain: cannot link a simple C program"

I noticed that I need to pass the flag '-fno-use-linker-plugin' to be
able to compile it.  However, even though I did pass it by exporting
the CFLAGS, it does not work.  I guess that numpy do not use the
CFLAGS for its internal extensions.  How can I pass that option to it?


Error below:
[   11s] compile options: '-Inumpy/core/src/private -Inumpy/core/src
-Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray
-Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include
-I/usr/include/python2.7 -c'
[   11s] gcc: _configtest.c
[   11s] gcc -pthread _configtest.o -o _configtest
[   11s] gcc: fatal error: -fuse-linker-plugin, but liblto_plugin.so not found
[   11s] compilation terminated.
[   11s] gcc: fatal error: -fuse-linker-plugin, but liblto_plugin.so not found
[   11s] compilation terminated.
[   11s] failure.
[   11s] removing: _configtest.c _configtest.o
[   11s] Traceback (most recent call last):
[   11s]   File "setup.py", line 214, in <module>
[   11s]     setup_package()
[   11s]   File "setup.py", line 207, in setup_package
[   11s]     configuration=configuration )
[   11s]   File
"/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/core.py", line 186,
in setup
[   11s]     return old_setup(**new_attr)
[   11s]   File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
[   11s]     dist.run_commands()
[   11s]   File "/usr/lib64/python2.7/distutils/dist.py", line 953, in
run_commands
[   11s]     self.run_command(cmd)
[   11s]   File "/usr/lib64/python2.7/distutils/dist.py", line 972, in
run_command
[   11s]     cmd_obj.run()
[   11s]   File
"/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build.py",
line 37, in run
[   11s]     old_build.run(self)
[   11s]   File "/usr/lib64/python2.7/distutils/command/build.py",
line 127, in run
[   11s]     self.run_command(cmd_name)
[   11s]   File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in
run_command
[   11s]     self.distribution.run_command(command)
[   11s]   File "/usr/lib64/python2.7/distutils/dist.py", line 972, in
run_command
[   11s]     cmd_obj.run()
[   11s]   File
"/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py",
line 152, in run
[   11s]     self.build_sources()
[   11s]   File
"/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py",
line 163, in build_sources
[   11s]     self.build_library_sources(*libname_info)
[   11s]   File
"/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py",
line 298, in build_library_sources
[   11s]     sources = self.generate_sources(sources, (lib_name, build_info))
[   11s]   File
"/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py",
line 385, in generate_sources
[   11s]     source = func(extension, build_dir)
[   11s]   File "numpy/core/setup.py", line 648, in get_mathlib_info
[   11s]     raise RuntimeError("Broken toolchain: cannot link a
simple C program")
[   11s] RuntimeError: Broken toolchain: cannot link a simple C program
[   11s] error: Bad exit status from /var/tmp/rpm-tmp.yO2SIE (%build)
[   11s]
[   11s]
[   11s] RPM build errors:
[   11s]     Bad exit status from /var/tmp/rpm-tmp.yO2SIE (%build)


From warren.weckesser at gmail.com  Wed Nov  7 14:18:08 2012
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Wed, 7 Nov 2012 13:18:08 -0600
Subject: [Numpy-discussion] strange behavior of numpy.unique
In-Reply-To: <CAMMTP+CXnxWZd7N4KZLdw0zMmVvoJw1nY+VJPKoQ=5PpR7Snrg@mail.gmail.com>
References: <CAB2ViTazH00kRhEuap1F+07DRaM9B+6Z1sVjWm4y5GJ3f6UOaA@mail.gmail.com>
	<CAGzF1udX9MCiKkSpKPpR_Xrgu6f2Svjb+c2UDuxcDfDQ=W9NBw@mail.gmail.com>
	<CAMMTP+CXnxWZd7N4KZLdw0zMmVvoJw1nY+VJPKoQ=5PpR7Snrg@mail.gmail.com>
Message-ID: <CAGzF1udE91z2YqBHG2H+vebpGr21vRnz1uXOWrcByJnKBdyXpg@mail.gmail.com>

On Wed, Nov 7, 2012 at 11:24 AM, <josef.pktd at gmail.com> wrote:

> On Tue, Nov 6, 2012 at 9:52 PM, Warren Weckesser
> <warren.weckesser at gmail.com> wrote:
> >
> >
> > On Tue, Nov 6, 2012 at 8:27 PM, Phillip Feldman
> > <phillip.m.feldman at gmail.com> wrote:
> >>
> >> numpy.unique behaves as I would expect for small inputs like the
> >> following:
> >>
> >> In [12]: x= [0, 0, 1, 0, 1, 2, 0, 1, 2, 3]
> >>
> >> In [13]: unique(x, return_index=True)
> >> Out[13]: (array([0, 1, 2, 3]), array([0, 2, 5, 9], dtype=int64))
> >>
> >> But, when I give it something larger, the return index values do not
> >> always correspond to the first occurrences in the input. The
> documentation
> >> is silent on the question of how the return index values are chosen
> when a
> >> given element of x appears more than once. Either the documentation
> should
> >> be
> >> clarified, or better yet, the behavior should be changed.
> >
> >
> >
> > In fact, it was changed (in the master branch on github) several months
> ago,
> > but there has not yet been a release with the changes.  The sort method
> that
> > np.unique passes to np.argsort is now 'mergesort', and the docstring
> states
> > that the indices returned are for the first occurrences of the unique
> > elements.  The new docstring is here:
> >
> http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.unique.html#numpy.unique
> >
> > See
> >
> https://github.com/numpy/numpy/commit/dbf235169ed3386b359caaa9217f5280bf1d6749
> > for the commit, and
> > https://github.com/numpy/numpy/blob/master/numpy/lib/arraysetops.py for
> the
> > latest version of the source.
>
> I think it's in 1.6.2 and it broke return_index for structured dtypes,
> IIRC.
>
>

You are correct, Josef, that change is in 1.6.2.  Thanks.

Warren


Josef
>
>
> >
> > Warren
> >
> >
> >>
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121107/cee05d16/attachment.html>

From ndbecker2 at gmail.com  Wed Nov  7 14:41:22 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 07 Nov 2012 14:41:22 -0500
Subject: [Numpy-discussion] testing with amd libm/acml
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
Message-ID: <k7edh1$7v0$1@ger.gmane.org>

Would you expect numexpr without MKL to give a significant boost?



From charlesr.harris at gmail.com  Wed Nov  7 16:48:05 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 7 Nov 2012 14:48:05 -0700
Subject: [Numpy-discussion] strange behavior of numpy.unique
In-Reply-To: <CAGzF1udX9MCiKkSpKPpR_Xrgu6f2Svjb+c2UDuxcDfDQ=W9NBw@mail.gmail.com>
References: <CAB2ViTazH00kRhEuap1F+07DRaM9B+6Z1sVjWm4y5GJ3f6UOaA@mail.gmail.com>
	<CAGzF1udX9MCiKkSpKPpR_Xrgu6f2Svjb+c2UDuxcDfDQ=W9NBw@mail.gmail.com>
Message-ID: <CAB6mnxJD1Yz7eVkNajOm7XK2EhDLmoB0BRPcm2LiXrZT_mt0xQ@mail.gmail.com>

On Tue, Nov 6, 2012 at 7:52 PM, Warren Weckesser <warren.weckesser at gmail.com
> wrote:

>
>
> On Tue, Nov 6, 2012 at 8:27 PM, Phillip Feldman <
> phillip.m.feldman at gmail.com> wrote:
>
>> numpy.unique behaves as I would expect for small inputs like the
>> following:
>>
>> In [12]: x= [0, 0, 1, 0, 1, 2, 0, 1, 2, 3]
>>
>> In [13]: unique(x, return_index=True)
>> Out[13]: (array([0, 1, 2, 3]), array([0, 2, 5, 9], dtype=int64))
>>
>> But, when I give it something larger, the return index values do not
>> always correspond to the first occurrences in the input. The documentation
>> is silent on the question of how the return index values are chosen when a
>> given element of x appears more than once. Either the documentation should
>> be
>> clarified, or better yet, the behavior should be changed.
>>
>
>
> In fact, it was changed (in the master branch on github) several months
> ago, but there has not yet been a release with the changes.  The sort
> method that np.unique passes to np.argsort is now 'mergesort', and the
> docstring states that the indices returned are for the first occurrences of
> the unique elements.  The new docstring is here:
> http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.unique.html#numpy.unique
>
> See
> https://github.com/numpy/numpy/commit/dbf235169ed3386b359caaa9217f5280bf1d6749for the commit, and
> https://github.com/numpy/numpy/blob/master/numpy/lib/arraysetops.py for
> the latest version of the source.
>
>
That change was backported to 1.6.2, but doesn't work for record/object
arrays. That oversight is fixed in master.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121107/c29138a5/attachment.html>

From scheffer.nicolas at gmail.com  Wed Nov  7 17:41:37 2012
From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER)
Date: Wed, 7 Nov 2012 14:41:37 -0800
Subject: [Numpy-discussion] Scipy dot
Message-ID: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>

Hi,

I've written a snippet of code that we could call scipy.dot, a drop-in
replacement for numpy.dot.
It's dead easy, and just answer the need of calling the right blas
function depending on the type of arrays, C or F order (aka slowness
of np.dot(A, A.T))

While this is not the scipy mailing list, I was wondering if this
snippet would relevant and/or useful to others, numpy folks, scipy
folks or could be integrated directly in numpy (so that we keep the
nice A.dot(B) syntax)
This bottleneck of temporary copies has been a problem for lots of
users and it seems everybody has their own snippets.
This code is probably not written as it should, I hope the community
can help improving it! ;)
First FIXME is to make it work for arrays of dimensions other than 2.

Suggestions highly appreciated!

Thanks!

===
Code (also on http://pastebin.com/QrRk0kEf)

def dot(A, B, out=None):
    """ A drop in replaement for numpy.dot
    Computes A.B optimized using fblas call
    note: unlike in numpy the returned array is in F order"""
    import scipy.linalg as sp
    gemm = sp.get_blas_funcs('gemm', arrays=(A,B))

    if out is None:
        lda, x, y, ldb = A.shape + B.shape
        if x != y:
            raise ValueError("matrices are not aligned")
        dtype = np.max([x.dtype for x in (A, B)])
        out = np.empty((lda, ldb), dtype, order='F')

    if A.flags.c_contiguous and B.flags.c_contiguous:
        gemm(alpha=1., a=A.T, b=B.T, trans_a=True, trans_b=True,
c=out, overwrite_c=True)
    if A.flags.c_contiguous and B.flags.f_contiguous:
        gemm(alpha=1., a=A.T, b=B, trans_a=True, c=out, overwrite_c=True)
    if A.flags.f_contiguous and B.flags.c_contiguous:
        gemm(alpha=1., a=A, b=B.T, trans_b=True, c=out, overwrite_c=True)
    if A.flags.f_contiguous and B.flags.f_contiguous:
        gemm(alpha=1., a=A, b=B, c=out, overwrite_c=True)
    return out
==


Timing (EPD, MKL):

In [15]: A = np.array(np.random.randn(1000, 1000), 'f')

In [16]: %timeit np.dot(A, A)
100 loops, best of 3: 7.19 ms per loop

In [17]: %timeit np.dot(A.T, A.T)
10 loops, best of 3: 27.7 ms per loop

In [18]: %timeit np.dot(A, A.T)
100 loops, best of 3: 18.3 ms per loop

In [19]: %timeit np.dot(A.T, A)
100 loops, best of 3: 18.7 ms per loop

In [20]: %timeit dot(A, A)
100 loops, best of 3: 7.16 ms per loop

In [21]: %timeit dot(A.T, A.T)
100 loops, best of 3: 6.67 ms per loop

In [22]: %timeit dot(A, A.T)
100 loops, best of 3: 6.79 ms per loop

In [23]: %timeit dot(A.T, A)
100 loops, best of 3: 7.02 ms per loop


From chris.barker at noaa.gov  Wed Nov  7 18:35:55 2012
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 7 Nov 2012 15:35:55 -0800
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <k7edh1$7v0$1@ger.gmane.org>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
	<k7edh1$7v0$1@ger.gmane.org>
Message-ID: <CALGmxELU9FSSPMaL_WQoDUu35=D2RBJ6AJDshR6sYV=TSqPk1w@mail.gmail.com>

On Wed, Nov 7, 2012 at 11:41 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
> Would you expect numexpr without MKL to give a significant boost?

It can, depending on the use case:
 -- It can remove a lot of uneccessary temporary creation.
 -- IIUC, it works on blocks of data at a time, and thus can keep
things in cache more when working with large data sets.
  -- It can (optionally) use multiple threads for easy parallelization.

All you can do is try it on your use-case and see what you get. It's a
pretty light lift to try.

-Chris






-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From francesc at continuum.io  Thu Nov  8 04:33:36 2012
From: francesc at continuum.io (Francesc Alted)
Date: Thu, 08 Nov 2012 10:33:36 +0100
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <k7edh1$7v0$1@ger.gmane.org>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
	<k7edh1$7v0$1@ger.gmane.org>
Message-ID: <509B7C70.9010508@continuum.io>

On 11/7/12 8:41 PM, Neal Becker wrote:
> Would you expect numexpr without MKL to give a significant boost?

Yes.  Have a look at how numexpr's own multi-threaded virtual machine 
compares with numexpr using VML:

http://code.google.com/p/numexpr/wiki/NumexprVML

As it can be seen, the best results are obtained by using the 
multi-threaded VM in numexpr in combination with a single-threaded VML 
engine.  Caution: I did these benchmarks some time ago (couple of 
years?), so it might be that multi-threaded VML would have improved by 
now.  If performance is critical, some experiments should be done first 
so as to find the optimal configuration.

At any rate, VML will let you to optimally leverage the SIMD 
instructions in the cores, allowing to compute, for example, exp() in 1 
or 2 clock cycles (depending on the vector length, the number of cores 
in your system and the data precision):

http://software.intel.com/sites/products/documentation/hpc/mkl/vml/functions/exp.html

Pretty amazing.

-- 
Francesc Alted



From francesc at continuum.io  Thu Nov  8 05:22:08 2012
From: francesc at continuum.io (Francesc Alted)
Date: Thu, 08 Nov 2012 11:22:08 +0100
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <CALGmxELU9FSSPMaL_WQoDUu35=D2RBJ6AJDshR6sYV=TSqPk1w@mail.gmail.com>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
	<k7edh1$7v0$1@ger.gmane.org>
	<CALGmxELU9FSSPMaL_WQoDUu35=D2RBJ6AJDshR6sYV=TSqPk1w@mail.gmail.com>
Message-ID: <509B87D0.60905@continuum.io>

On 11/8/12 12:35 AM, Chris Barker wrote:
> On Wed, Nov 7, 2012 at 11:41 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
>> Would you expect numexpr without MKL to give a significant boost?
> It can, depending on the use case:
>   -- It can remove a lot of uneccessary temporary creation.
>   -- IIUC, it works on blocks of data at a time, and thus can keep
> things in cache more when working with large data sets.

Well, the temporaries are still created, but the thing is that, by 
working with small blocks at a time, these temporaries fit in CPU cache, 
preventing copies into main memory.  I like to name this the 'blocking 
technique', as explained in slide 26 (and following) in:

https://python.g-node.org/wiki/_media/starving_cpu/starving-cpu.pdf

A better technique is to reduce the block size to the minimal expression 
(1 element), so temporaries are stored in registers in CPU instead of 
small blocks in cache, hence preventing copies even in *cache*.  Numba 
(https://github.com/numba/numba) follows this approach, which is pretty 
optimal as can be seen in slide 37 of the lecture above.

>    -- It can (optionally) use multiple threads for easy parallelization.

No, the *total* amount of cores detected in the system is the default in 
numexpr; if you want less, you will need to use 
set_num_threads(nthreads) function.  But agreed, sometimes using too 
many threads could effectively be counter-producing.

-- 
Francesc Alted



From njs at pobox.com  Thu Nov  8 06:28:21 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 8 Nov 2012 11:28:21 +0000
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
Message-ID: <CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>

On Wed, Nov 7, 2012 at 10:41 PM, Nicolas SCHEFFER
<scheffer.nicolas at gmail.com> wrote:
> Hi,
>
> I've written a snippet of code that we could call scipy.dot, a drop-in
> replacement for numpy.dot.
> It's dead easy, and just answer the need of calling the right blas
> function depending on the type of arrays, C or F order (aka slowness
> of np.dot(A, A.T))
>
> While this is not the scipy mailing list, I was wondering if this
> snippet would relevant and/or useful to others, numpy folks, scipy
> folks or could be integrated directly in numpy (so that we keep the
> nice A.dot(B) syntax)

I think everyone would be very happy to see numpy.dot modified to do
this automatically. But adding a scipy.dot IMHO would be fixing things
in the wrong place and just create extra confusion.

Is it possible to avoid changing the default output order from C to F?
(E.g. by transposing everything relative to what you have now?) That
seems like a change that would be good to avoid if it's easy.

-n


From gael.varoquaux at normalesup.org  Thu Nov  8 07:07:25 2012
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 8 Nov 2012 13:07:25 +0100
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
Message-ID: <20121108120725.GL313@phare.normalesup.org>

On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith wrote:
> I think everyone would be very happy to see numpy.dot modified to do
> this automatically. But adding a scipy.dot IMHO would be fixing things
> in the wrong place and just create extra confusion.

I am not sure I agree: numpy is often compiled without lapack support, as
it is not necessary. On the other hand scipy is always compiled with
lapack. Thus this makes more sens in scipy.

> Is it possible to avoid changing the default output order from C to F?

+1

G


From d.s.seljebotn at astro.uio.no  Thu Nov  8 07:12:08 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 08 Nov 2012 13:12:08 +0100
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <20121108120725.GL313@phare.normalesup.org>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
Message-ID: <509BA198.4070301@astro.uio.no>

On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith wrote:
>> I think everyone would be very happy to see numpy.dot modified to do
>> this automatically. But adding a scipy.dot IMHO would be fixing things
>> in the wrong place and just create extra confusion.
>
> I am not sure I agree: numpy is often compiled without lapack support, as
> it is not necessary. On the other hand scipy is always compiled with
> lapack. Thus this makes more sens in scipy.

Well, numpy.dot already contains multiple fallback cases for when it is 
compiled with BLAS and not. So I'm +1 on just making this an improvement 
on numpy.dot. I don't think there's a time when you would not want to 
use this (provided the output order issue is fixed), and it doesn't make 
sense to not have old codes take advantage of the speed improvement.

BTW, something this doesn't fix is that you still have to do 
"np.dot(x.conjugate().t, x)" in the complex case which needlessly copies 
the data since LAPACK can do the conjugation.

DS

>
>> Is it possible to avoid changing the default output order from C to F?
>
> +1
>
> G
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



From ndbecker2 at gmail.com  Thu Nov  8 07:37:53 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 08 Nov 2012 07:37:53 -0500
Subject: [Numpy-discussion] numexpr question
Message-ID: <k7g930$2mr$1@ger.gmane.org>

I'm interested in trying numexpr, but have a question (not sure where's the best 
forum to ask).

The examples I see use

ne.evaluate ("some string...")

When used within a loop, I would expect the compilation from the string form to 
add significant overhead.  I would have thought a pre-compiled form would be 
available, similar to a precompiled regexp.  yes/no?



From d.s.seljebotn at astro.uio.no  Thu Nov  8 07:41:45 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 08 Nov 2012 13:41:45 +0100
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <k7edh1$7v0$1@ger.gmane.org>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
	<k7edh1$7v0$1@ger.gmane.org>
Message-ID: <509BA889.9070102@astro.uio.no>

On 11/07/2012 08:41 PM, Neal Becker wrote:
> Would you expect numexpr without MKL to give a significant boost?

If you need higher performance than what numexpr can give without using 
MKL, you could look at code such as this:

https://github.com/herumi/fmath/blob/master/fmath.hpp#L480

But that means going to C (e.g., by wrapping that function in Cython). 
Pay attention to what range you evaluate the function in though (my eyes 
may deceive me but it seems that the test program only test for 
arguments drawn from the standard Gaussian which is a bit limited..)

Dag Sverre


From cournape at gmail.com  Thu Nov  8 08:06:53 2012
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 8 Nov 2012 13:06:53 +0000
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <509BA198.4070301@astro.uio.no>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
Message-ID: <CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>

On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
>> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith wrote:
>>> I think everyone would be very happy to see numpy.dot modified to do
>>> this automatically. But adding a scipy.dot IMHO would be fixing things
>>> in the wrong place and just create extra confusion.
>>
>> I am not sure I agree: numpy is often compiled without lapack support, as
>> it is not necessary. On the other hand scipy is always compiled with
>> lapack. Thus this makes more sens in scipy.
>
> Well, numpy.dot already contains multiple fallback cases for when it is
> compiled with BLAS and not. So I'm +1 on just making this an improvement
> on numpy.dot. I don't think there's a time when you would not want to
> use this (provided the output order issue is fixed), and it doesn't make
> sense to not have old codes take advantage of the speed improvement.

Indeed, there is no reason not to make this available in NumPy.

Nicolas, can you prepare a patch for numpy ?

David


From francesc at continuum.io  Thu Nov  8 10:17:23 2012
From: francesc at continuum.io (Francesc Alted)
Date: Thu, 08 Nov 2012 16:17:23 +0100
Subject: [Numpy-discussion] numexpr question
In-Reply-To: <k7g930$2mr$1@ger.gmane.org>
References: <k7g930$2mr$1@ger.gmane.org>
Message-ID: <509BCD03.1020900@continuum.io>

On 11/8/12 1:37 PM, Neal Becker wrote:
> I'm interested in trying numexpr, but have a question (not sure where's the best
> forum to ask).
>
> The examples I see use
>
> ne.evaluate ("some string...")
>
> When used within a loop, I would expect the compilation from the string form to
> add significant overhead.  I would have thought a pre-compiled form would be
> available, similar to a precompiled regexp.  yes/no?

numexpr comes with an internal cache for recent expressions, so if 
ne.evaluate() is in a loop, the compiled expression will be re-used 
without problems.  So you don't have to worry about caching it yourself.

The best forum for discussing numexpr is this:

https://groups.google.com/forum/?fromgroups#!forum/numexpr

-- 
Francesc Alted



From chris.barker at noaa.gov  Thu Nov  8 11:50:44 2012
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 8 Nov 2012 08:50:44 -0800
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <509B87D0.60905@continuum.io>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
	<k7edh1$7v0$1@ger.gmane.org>
	<CALGmxELU9FSSPMaL_WQoDUu35=D2RBJ6AJDshR6sYV=TSqPk1w@mail.gmail.com>
	<509B87D0.60905@continuum.io>
Message-ID: <CALGmxEKHnjEBJoSNCdUnDF8isaXieP_=-SA0ZYjmiC+2L9z6gw@mail.gmail.com>

On Thu, Nov 8, 2012 at 2:22 AM, Francesc Alted <francesc at continuum.io> wrote:

>>   -- It can remove a lot of uneccessary temporary creation.

> Well, the temporaries are still created, but the thing is that, by
> working with small blocks at a time, these temporaries fit in CPU cache,
> preventing copies into main memory.

hmm -- I thought it was "smart" enough to remove some unnecessary
temporaries altogether. Shows what I know. But apparently it does,
indeed, avoid creating the full-size temporary arrays.

pretty cool stuff, in any case.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From francesc at continuum.io  Thu Nov  8 12:06:17 2012
From: francesc at continuum.io (Francesc Alted)
Date: Thu, 08 Nov 2012 18:06:17 +0100
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <509BA889.9070102@astro.uio.no>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
	<k7edh1$7v0$1@ger.gmane.org> <509BA889.9070102@astro.uio.no>
Message-ID: <509BE689.9050804@continuum.io>

On 11/8/12 1:41 PM, Dag Sverre Seljebotn wrote:
> On 11/07/2012 08:41 PM, Neal Becker wrote:
>> Would you expect numexpr without MKL to give a significant boost?
> If you need higher performance than what numexpr can give without using
> MKL, you could look at code such as this:
>
> https://github.com/herumi/fmath/blob/master/fmath.hpp#L480

Hey, that's cool.  I was a bit disappointed not finding this sort of 
work in open space.  It seems that this lacks threading support, but 
that should be easy to implement by using OpenMP directives.

-- 
Francesc Alted



From scopatz at gmail.com  Thu Nov  8 12:29:19 2012
From: scopatz at gmail.com (Anthony Scopatz)
Date: Thu, 8 Nov 2012 11:29:19 -0600
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
Message-ID: <CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>

On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau <cournape at gmail.com> wrote:

> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith wrote:
> >>> I think everyone would be very happy to see numpy.dot modified to do
> >>> this automatically. But adding a scipy.dot IMHO would be fixing things
> >>> in the wrong place and just create extra confusion.
> >>
> >> I am not sure I agree: numpy is often compiled without lapack support,
> as
> >> it is not necessary. On the other hand scipy is always compiled with
> >> lapack. Thus this makes more sens in scipy.
> >
> > Well, numpy.dot already contains multiple fallback cases for when it is
> > compiled with BLAS and not. So I'm +1 on just making this an improvement
> > on numpy.dot. I don't think there's a time when you would not want to
> > use this (provided the output order issue is fixed), and it doesn't make
> > sense to not have old codes take advantage of the speed improvement.
>
> Indeed, there is no reason not to make this available in NumPy.
>
> Nicolas, can you prepare a patch for numpy ?
>

+1, I agree, this should be a fix in numpy, not scipy.

Be Well
Anthony


>
> David
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121108/84546885/attachment.html>

From d.s.seljebotn at astro.uio.no  Thu Nov  8 12:38:47 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 08 Nov 2012 18:38:47 +0100
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <509BE689.9050804@continuum.io>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
	<k7edh1$7v0$1@ger.gmane.org> <509BA889.9070102@astro.uio.no>
	<509BE689.9050804@continuum.io>
Message-ID: <509BEE27.4020307@astro.uio.no>

On 11/08/2012 06:06 PM, Francesc Alted wrote:
> On 11/8/12 1:41 PM, Dag Sverre Seljebotn wrote:
>> On 11/07/2012 08:41 PM, Neal Becker wrote:
>>> Would you expect numexpr without MKL to give a significant boost?
>> If you need higher performance than what numexpr can give without using
>> MKL, you could look at code such as this:
>>
>> https://github.com/herumi/fmath/blob/master/fmath.hpp#L480
>
> Hey, that's cool.  I was a bit disappointed not finding this sort of
> work in open space.  It seems that this lacks threading support, but
> that should be easy to implement by using OpenMP directives.

IMO this is the wrong place to introduce threading; each thread should 
call expd_v on its chunks. (Which I think is how you said numexpr 
currently uses VML anyway.)

Dag Sverre





From nouiz at nouiz.org  Thu Nov  8 12:42:47 2012
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Thu, 8 Nov 2012 12:42:47 -0500
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
Message-ID: <CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>

Hi,

I also think it should go into numpy.dot and that the output order should
not be changed.

A new point, what about the additional overhead for small ndarray? To
remove this, I would suggest to put this code into the C function that do
the actual work (at least, from memory it is a c function, not a python
one).

HTH

Fred


On Thu, Nov 8, 2012 at 12:29 PM, Anthony Scopatz <scopatz at gmail.com> wrote:

> On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau <cournape at gmail.com>wrote:
>
>> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
>> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith wrote:
>> >>> I think everyone would be very happy to see numpy.dot modified to do
>> >>> this automatically. But adding a scipy.dot IMHO would be fixing things
>> >>> in the wrong place and just create extra confusion.
>> >>
>> >> I am not sure I agree: numpy is often compiled without lapack support,
>> as
>> >> it is not necessary. On the other hand scipy is always compiled with
>> >> lapack. Thus this makes more sens in scipy.
>> >
>> > Well, numpy.dot already contains multiple fallback cases for when it is
>> > compiled with BLAS and not. So I'm +1 on just making this an improvement
>> > on numpy.dot. I don't think there's a time when you would not want to
>> > use this (provided the output order issue is fixed), and it doesn't make
>> > sense to not have old codes take advantage of the speed improvement.
>>
>> Indeed, there is no reason not to make this available in NumPy.
>>
>> Nicolas, can you prepare a patch for numpy ?
>>
>
> +1, I agree, this should be a fix in numpy, not scipy.
>
> Be Well
> Anthony
>
>
>>
>> David
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121108/8b0e4a11/attachment.html>

From francesc at continuum.io  Thu Nov  8 12:59:41 2012
From: francesc at continuum.io (Francesc Alted)
Date: Thu, 08 Nov 2012 18:59:41 +0100
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <509BEE27.4020307@astro.uio.no>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
	<k7edh1$7v0$1@ger.gmane.org> <509BA889.9070102@astro.uio.no>
	<509BE689.9050804@continuum.io> <509BEE27.4020307@astro.uio.no>
Message-ID: <509BF30D.8060903@continuum.io>

On 11/8/12 6:38 PM, Dag Sverre Seljebotn wrote:
> On 11/08/2012 06:06 PM, Francesc Alted wrote:
>> On 11/8/12 1:41 PM, Dag Sverre Seljebotn wrote:
>>> On 11/07/2012 08:41 PM, Neal Becker wrote:
>>>> Would you expect numexpr without MKL to give a significant boost?
>>> If you need higher performance than what numexpr can give without using
>>> MKL, you could look at code such as this:
>>>
>>> https://github.com/herumi/fmath/blob/master/fmath.hpp#L480
>> Hey, that's cool.  I was a bit disappointed not finding this sort of
>> work in open space.  It seems that this lacks threading support, but
>> that should be easy to implement by using OpenMP directives.
> IMO this is the wrong place to introduce threading; each thread should
> call expd_v on its chunks. (Which I think is how you said numexpr
> currently uses VML anyway.)

Oh sure, but then you need a blocked engine for performing the 
computations too.  And yes, by default numexpr uses its own threading 
code rather than the existing one in VML (but that can be changed by 
playing with set_num_threads/set_vml_num_threads).  It always stroked to 
me as a little strange that the internal threading in numexpr was more 
efficient than VML one, but I suppose this is because the latter is more 
optimized to deal with large blocks instead of those of medium size (4K) 
in numexpr.

-- 
Francesc Alted



From d.s.seljebotn at astro.uio.no  Thu Nov  8 13:55:20 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 08 Nov 2012 19:55:20 +0100
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <509BF30D.8060903@continuum.io>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
	<k7edh1$7v0$1@ger.gmane.org> <509BA889.9070102@astro.uio.no>
	<509BE689.9050804@continuum.io> <509BEE27.4020307@astro.uio.no>
	<509BF30D.8060903@continuum.io>
Message-ID: <509C0018.8020503@astro.uio.no>

On 11/08/2012 06:59 PM, Francesc Alted wrote:
> On 11/8/12 6:38 PM, Dag Sverre Seljebotn wrote:
>> On 11/08/2012 06:06 PM, Francesc Alted wrote:
>>> On 11/8/12 1:41 PM, Dag Sverre Seljebotn wrote:
>>>> On 11/07/2012 08:41 PM, Neal Becker wrote:
>>>>> Would you expect numexpr without MKL to give a significant boost?
>>>> If you need higher performance than what numexpr can give without using
>>>> MKL, you could look at code such as this:
>>>>
>>>> https://github.com/herumi/fmath/blob/master/fmath.hpp#L480
>>> Hey, that's cool.  I was a bit disappointed not finding this sort of
>>> work in open space.  It seems that this lacks threading support, but
>>> that should be easy to implement by using OpenMP directives.
>> IMO this is the wrong place to introduce threading; each thread should
>> call expd_v on its chunks. (Which I think is how you said numexpr
>> currently uses VML anyway.)
>
> Oh sure, but then you need a blocked engine for performing the
> computations too.  And yes, by default numexpr uses its own threading

I just meant that you can use a chunked OpenMP for-loop wherever in your 
code that you call expd_v. A "five-line blocked engine", if you like :-)

IMO that's the right location since entering/exiting OpenMP blocks takes 
some time.

> code rather than the existing one in VML (but that can be changed by
> playing with set_num_threads/set_vml_num_threads).  It always stroked to
> me as a little strange that the internal threading in numexpr was more
> efficient than VML one, but I suppose this is because the latter is more
> optimized to deal with large blocks instead of those of medium size (4K)
> in numexpr.

I don't know enough about numexpr to understand this :-)

I guess I just don't see the motivation to use VML threading or why it 
should be faster? If you pass a single 4K block to a threaded VML call 
then I could easily see lots of performance problems: a) 
starting/stopping threads or signalling the threads of a pool is a 
constant overhead per "parallel section", b) unless you're very careful 
to only have VML touch the data, and VML always schedules elements in 
the exact same way, you're going to have the cache lines of that 4K 
block shuffled between L1 caches of different cores for different 
operations...

As I said, I'm mostly ignorant about how numexpr works, that's probably 
showing :-)

Dag Sverre


From d.s.seljebotn at astro.uio.no  Thu Nov  8 13:56:27 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 08 Nov 2012 19:56:27 +0100
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <509C0018.8020503@astro.uio.no>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
	<k7edh1$7v0$1@ger.gmane.org> <509BA889.9070102@astro.uio.no>
	<509BE689.9050804@continuum.io> <509BEE27.4020307@astro.uio.no>
	<509BF30D.8060903@continuum.io> <509C0018.8020503@astro.uio.no>
Message-ID: <509C005B.2020305@astro.uio.no>

On 11/08/2012 07:55 PM, Dag Sverre Seljebotn wrote:
> On 11/08/2012 06:59 PM, Francesc Alted wrote:
>> On 11/8/12 6:38 PM, Dag Sverre Seljebotn wrote:
>>> On 11/08/2012 06:06 PM, Francesc Alted wrote:
>>>> On 11/8/12 1:41 PM, Dag Sverre Seljebotn wrote:
>>>>> On 11/07/2012 08:41 PM, Neal Becker wrote:
>>>>>> Would you expect numexpr without MKL to give a significant boost?
>>>>> If you need higher performance than what numexpr can give without
>>>>> using
>>>>> MKL, you could look at code such as this:
>>>>>
>>>>> https://github.com/herumi/fmath/blob/master/fmath.hpp#L480
>>>> Hey, that's cool.  I was a bit disappointed not finding this sort of
>>>> work in open space.  It seems that this lacks threading support, but
>>>> that should be easy to implement by using OpenMP directives.
>>> IMO this is the wrong place to introduce threading; each thread should
>>> call expd_v on its chunks. (Which I think is how you said numexpr
>>> currently uses VML anyway.)
>>
>> Oh sure, but then you need a blocked engine for performing the
>> computations too.  And yes, by default numexpr uses its own threading
>
> I just meant that you can use a chunked OpenMP for-loop wherever in your
> code that you call expd_v. A "five-line blocked engine", if you like :-)
>
> IMO that's the right location since entering/exiting OpenMP blocks takes
> some time.
>
>> code rather than the existing one in VML (but that can be changed by
>> playing with set_num_threads/set_vml_num_threads).  It always stroked to
>> me as a little strange that the internal threading in numexpr was more
>> efficient than VML one, but I suppose this is because the latter is more
>> optimized to deal with large blocks instead of those of medium size (4K)
>> in numexpr.
>
> I don't know enough about numexpr to understand this :-)
>
> I guess I just don't see the motivation to use VML threading or why it
> should be faster? If you pass a single 4K block to a threaded VML call
> then I could easily see lots of performance problems: a)
> starting/stopping threads or signalling the threads of a pool is a
> constant overhead per "parallel section", b) unless you're very careful
> to only have VML touch the data, and VML always schedules elements in
> the exact same way, you're going to have the cache lines of that 4K
> block shuffled between L1 caches of different cores for different
> operations...

c) Your "effective block size" is then 4KB/ncores.

(Unless you scale the block size by ncores).

DS


From francesc at continuum.io  Thu Nov  8 14:37:30 2012
From: francesc at continuum.io (Francesc Alted)
Date: Thu, 08 Nov 2012 20:37:30 +0100
Subject: [Numpy-discussion] testing with amd libm/acml
In-Reply-To: <509C0018.8020503@astro.uio.no>
References: <k7dkia$dii$1@ger.gmane.org>
	<CAGY4rcVg8xC_udV-1mNDTP3eO+NFqgJCyWhkNtoYvxxtny2nHw@mail.gmail.com>
	<k7dpan$1ck$1@ger.gmane.org>
	<CAGY4rcUYN-cWK5MURsYnRy6BRdFJ_De9s_iTC1JRsUeDtLB0eA@mail.gmail.com>
	<k7dra2$jue$2@ger.gmane.org> <509A75AC.3010206@astro.uio.no>
	<k7edh1$7v0$1@ger.gmane.org> <509BA889.9070102@astro.uio.no>
	<509BE689.9050804@continuum.io> <509BEE27.4020307@astro.uio.no>
	<509BF30D.8060903@continuum.io> <509C0018.8020503@astro.uio.no>
Message-ID: <509C09FA.8090409@continuum.io>

On 11/8/12 7:55 PM, Dag Sverre Seljebotn wrote:
> On 11/08/2012 06:59 PM, Francesc Alted wrote:
>> On 11/8/12 6:38 PM, Dag Sverre Seljebotn wrote:
>>> On 11/08/2012 06:06 PM, Francesc Alted wrote:
>>>> On 11/8/12 1:41 PM, Dag Sverre Seljebotn wrote:
>>>>> On 11/07/2012 08:41 PM, Neal Becker wrote:
>>>>>> Would you expect numexpr without MKL to give a significant boost?
>>>>> If you need higher performance than what numexpr can give without using
>>>>> MKL, you could look at code such as this:
>>>>>
>>>>> https://github.com/herumi/fmath/blob/master/fmath.hpp#L480
>>>> Hey, that's cool.  I was a bit disappointed not finding this sort of
>>>> work in open space.  It seems that this lacks threading support, but
>>>> that should be easy to implement by using OpenMP directives.
>>> IMO this is the wrong place to introduce threading; each thread should
>>> call expd_v on its chunks. (Which I think is how you said numexpr
>>> currently uses VML anyway.)
>> Oh sure, but then you need a blocked engine for performing the
>> computations too.  And yes, by default numexpr uses its own threading
> I just meant that you can use a chunked OpenMP for-loop wherever in your
> code that you call expd_v. A "five-line blocked engine", if you like :-)
>
> IMO that's the right location since entering/exiting OpenMP blocks takes
> some time.

Yes, I meant precisely this first hand.

>> code rather than the existing one in VML (but that can be changed by
>> playing with set_num_threads/set_vml_num_threads).  It always stroked to
>> me as a little strange that the internal threading in numexpr was more
>> efficient than VML one, but I suppose this is because the latter is more
>> optimized to deal with large blocks instead of those of medium size (4K)
>> in numexpr.
> I don't know enough about numexpr to understand this :-)
>
> I guess I just don't see the motivation to use VML threading or why it
> should be faster? If you pass a single 4K block to a threaded VML call
> then I could easily see lots of performance problems: a)
> starting/stopping threads or signalling the threads of a pool is a
> constant overhead per "parallel section", b) unless you're very careful
> to only have VML touch the data, and VML always schedules elements in
> the exact same way, you're going to have the cache lines of that 4K
> block shuffled between L1 caches of different cores for different
> operations...
>
> As I said, I'm mostly ignorant about how numexpr works, that's probably
> showing :-)

No, on the contrary, you rather hit the core of the issue (or part of 
it).  On one hand, VML needs large blocks in order to maximize the 
performance of the pipeline and in the other hand numexpr tries to 
minimize block size in order to make temporaries as small as possible 
(so avoiding the use of the higher level caches).  From this tension 
(and some benchmarking work) the size of 4K (btw, this is the number of 
*elements*, so the size is actually either 16 KB and 32 KB for single 
and double precision respectively) was derived.  Incidentally, for 
numexpr with no VML support, the size is reduced to 1K elements (and 
perhaps it could be reduced a bit more, but anyways).

Anyway, this is way too low level to be discussed here, although we can 
continue on the numexpr list if you are interested in more details.

-- 
Francesc Alted



From scheffer.nicolas at gmail.com  Thu Nov  8 14:38:31 2012
From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER)
Date: Thu, 8 Nov 2012 11:38:31 -0800
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
Message-ID: <CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>

Thanks for all the responses folks. This is indeed a nice problem to solve.

Few points:
I. Change the order from 'F' to 'C': I'll look into it.
II. Integration with scipy / numpy: opinions are diverging here.
Let's wait a bit to get more responses on what people think.
One thing though: I'd need the same functionality as get_blas_funcs in numpy.
Since numpy does not require lapack, what functions can I get?
III. Complex arrays
I unfortunately don't have enough knowledge here. If someone could
propose a fix, that'd be great.
IV. C
Writing this in C sounds like a good idea. I'm not sure I'd be the
right person to this though.
V. Patch in numpy
I'd love to do that and learn to do it as a byproduct.
Let's make sure we agree this can go in numpy first and that all FIXME
can be fixed.
Although I guess we can resolve fixmes using git.

Let me know how you'd like to proceed,

Thanks!

FIXMEs:
- Fix for ndim != 2
- Fix for dtype == np.complex*
- Fix order of output array

On Thu, Nov 8, 2012 at 9:42 AM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
> Hi,
>
> I also think it should go into numpy.dot and that the output order should
> not be changed.
>
> A new point, what about the additional overhead for small ndarray? To remove
> this, I would suggest to put this code into the C function that do the
> actual work (at least, from memory it is a c function, not a python one).
>
> HTH
>
> Fred
>
>
>
> On Thu, Nov 8, 2012 at 12:29 PM, Anthony Scopatz <scopatz at gmail.com> wrote:
>>
>> On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau <cournape at gmail.com>
>> wrote:
>>>
>>> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no> wrote:
>>> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
>>> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith wrote:
>>> >>> I think everyone would be very happy to see numpy.dot modified to do
>>> >>> this automatically. But adding a scipy.dot IMHO would be fixing
>>> >>> things
>>> >>> in the wrong place and just create extra confusion.
>>> >>
>>> >> I am not sure I agree: numpy is often compiled without lapack support,
>>> >> as
>>> >> it is not necessary. On the other hand scipy is always compiled with
>>> >> lapack. Thus this makes more sens in scipy.
>>> >
>>> > Well, numpy.dot already contains multiple fallback cases for when it is
>>> > compiled with BLAS and not. So I'm +1 on just making this an
>>> > improvement
>>> > on numpy.dot. I don't think there's a time when you would not want to
>>> > use this (provided the output order issue is fixed), and it doesn't
>>> > make
>>> > sense to not have old codes take advantage of the speed improvement.
>>>
>>> Indeed, there is no reason not to make this available in NumPy.
>>>
>>> Nicolas, can you prepare a patch for numpy ?
>>
>>
>> +1, I agree, this should be a fix in numpy, not scipy.
>>
>> Be Well
>> Anthony
>>
>>>
>>>
>>> David
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From scheffer.nicolas at gmail.com  Thu Nov  8 15:06:04 2012
From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER)
Date: Thu, 8 Nov 2012 12:06:04 -0800
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
Message-ID: <CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>

I've made the necessary changes to get the proper order for the output array.
Also, a pass of pep8 and some tests (fixmes are in failing tests)
http://pastebin.com/M8TfbURi

-n

On Thu, Nov 8, 2012 at 11:38 AM, Nicolas SCHEFFER
<scheffer.nicolas at gmail.com> wrote:
> Thanks for all the responses folks. This is indeed a nice problem to solve.
>
> Few points:
> I. Change the order from 'F' to 'C': I'll look into it.
> II. Integration with scipy / numpy: opinions are diverging here.
> Let's wait a bit to get more responses on what people think.
> One thing though: I'd need the same functionality as get_blas_funcs in numpy.
> Since numpy does not require lapack, what functions can I get?
> III. Complex arrays
> I unfortunately don't have enough knowledge here. If someone could
> propose a fix, that'd be great.
> IV. C
> Writing this in C sounds like a good idea. I'm not sure I'd be the
> right person to this though.
> V. Patch in numpy
> I'd love to do that and learn to do it as a byproduct.
> Let's make sure we agree this can go in numpy first and that all FIXME
> can be fixed.
> Although I guess we can resolve fixmes using git.
>
> Let me know how you'd like to proceed,
>
> Thanks!
>
> FIXMEs:
> - Fix for ndim != 2
> - Fix for dtype == np.complex*
> - Fix order of output array
>
> On Thu, Nov 8, 2012 at 9:42 AM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
>> Hi,
>>
>> I also think it should go into numpy.dot and that the output order should
>> not be changed.
>>
>> A new point, what about the additional overhead for small ndarray? To remove
>> this, I would suggest to put this code into the C function that do the
>> actual work (at least, from memory it is a c function, not a python one).
>>
>> HTH
>>
>> Fred
>>
>>
>>
>> On Thu, Nov 8, 2012 at 12:29 PM, Anthony Scopatz <scopatz at gmail.com> wrote:
>>>
>>> On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau <cournape at gmail.com>
>>> wrote:
>>>>
>>>> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
>>>> <d.s.seljebotn at astro.uio.no> wrote:
>>>> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
>>>> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith wrote:
>>>> >>> I think everyone would be very happy to see numpy.dot modified to do
>>>> >>> this automatically. But adding a scipy.dot IMHO would be fixing
>>>> >>> things
>>>> >>> in the wrong place and just create extra confusion.
>>>> >>
>>>> >> I am not sure I agree: numpy is often compiled without lapack support,
>>>> >> as
>>>> >> it is not necessary. On the other hand scipy is always compiled with
>>>> >> lapack. Thus this makes more sens in scipy.
>>>> >
>>>> > Well, numpy.dot already contains multiple fallback cases for when it is
>>>> > compiled with BLAS and not. So I'm +1 on just making this an
>>>> > improvement
>>>> > on numpy.dot. I don't think there's a time when you would not want to
>>>> > use this (provided the output order issue is fixed), and it doesn't
>>>> > make
>>>> > sense to not have old codes take advantage of the speed improvement.
>>>>
>>>> Indeed, there is no reason not to make this available in NumPy.
>>>>
>>>> Nicolas, can you prepare a patch for numpy ?
>>>
>>>
>>> +1, I agree, this should be a fix in numpy, not scipy.
>>>
>>> Be Well
>>> Anthony
>>>
>>>>
>>>>
>>>> David
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>


From scheffer.nicolas at gmail.com  Thu Nov  8 17:44:17 2012
From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER)
Date: Thu, 8 Nov 2012 14:44:17 -0800
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
	<CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
Message-ID: <CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>

Well, hinted by what Fabien said, I looked at the C level dot function.
Quite verbose!

But starting line 757, we can see that it shouldn't be too much work
to fix that bug (well there is even a comment there that states just
that)
https://github.com/numpy/numpy/blob/master/numpy/core/blasdot/_dotblas.c#L757
I now think that should be the cleanest.

This would only work for gemm though. I don't know what the benefit is
for gemv for instance, but we should make that kind of changes
everywhere we can.
The evil PyArray_Copy is there twice and that's what we want to get rid of.

I'm not sure, but it looks to me that removing the copy and doing the
following would do the work:
Order = CblasRowMajor;
Trans1 = CblasNoTrans;
Trans2 = CblasNoTrans;
if (!PyArray_ISCONTIGUOUS(ap1)) {
    Trans1 = CblasTrans;
}
if (!PyArray_ISCONTIGUOUS(ap2)) {
    Trans2 = CblasTrans;
}
might be too easy to be true.



On Thu, Nov 8, 2012 at 12:06 PM, Nicolas SCHEFFER
<scheffer.nicolas at gmail.com> wrote:
> I've made the necessary changes to get the proper order for the output array.
> Also, a pass of pep8 and some tests (fixmes are in failing tests)
> http://pastebin.com/M8TfbURi
>
> -n
>
> On Thu, Nov 8, 2012 at 11:38 AM, Nicolas SCHEFFER
> <scheffer.nicolas at gmail.com> wrote:
>> Thanks for all the responses folks. This is indeed a nice problem to solve.
>>
>> Few points:
>> I. Change the order from 'F' to 'C': I'll look into it.
>> II. Integration with scipy / numpy: opinions are diverging here.
>> Let's wait a bit to get more responses on what people think.
>> One thing though: I'd need the same functionality as get_blas_funcs in numpy.
>> Since numpy does not require lapack, what functions can I get?
>> III. Complex arrays
>> I unfortunately don't have enough knowledge here. If someone could
>> propose a fix, that'd be great.
>> IV. C
>> Writing this in C sounds like a good idea. I'm not sure I'd be the
>> right person to this though.
>> V. Patch in numpy
>> I'd love to do that and learn to do it as a byproduct.
>> Let's make sure we agree this can go in numpy first and that all FIXME
>> can be fixed.
>> Although I guess we can resolve fixmes using git.
>>
>> Let me know how you'd like to proceed,
>>
>> Thanks!
>>
>> FIXMEs:
>> - Fix for ndim != 2
>> - Fix for dtype == np.complex*
>> - Fix order of output array
>>
>> On Thu, Nov 8, 2012 at 9:42 AM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
>>> Hi,
>>>
>>> I also think it should go into numpy.dot and that the output order should
>>> not be changed.
>>>
>>> A new point, what about the additional overhead for small ndarray? To remove
>>> this, I would suggest to put this code into the C function that do the
>>> actual work (at least, from memory it is a c function, not a python one).
>>>
>>> HTH
>>>
>>> Fred
>>>
>>>
>>>
>>> On Thu, Nov 8, 2012 at 12:29 PM, Anthony Scopatz <scopatz at gmail.com> wrote:
>>>>
>>>> On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau <cournape at gmail.com>
>>>> wrote:
>>>>>
>>>>> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
>>>>> <d.s.seljebotn at astro.uio.no> wrote:
>>>>> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
>>>>> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith wrote:
>>>>> >>> I think everyone would be very happy to see numpy.dot modified to do
>>>>> >>> this automatically. But adding a scipy.dot IMHO would be fixing
>>>>> >>> things
>>>>> >>> in the wrong place and just create extra confusion.
>>>>> >>
>>>>> >> I am not sure I agree: numpy is often compiled without lapack support,
>>>>> >> as
>>>>> >> it is not necessary. On the other hand scipy is always compiled with
>>>>> >> lapack. Thus this makes more sens in scipy.
>>>>> >
>>>>> > Well, numpy.dot already contains multiple fallback cases for when it is
>>>>> > compiled with BLAS and not. So I'm +1 on just making this an
>>>>> > improvement
>>>>> > on numpy.dot. I don't think there's a time when you would not want to
>>>>> > use this (provided the output order issue is fixed), and it doesn't
>>>>> > make
>>>>> > sense to not have old codes take advantage of the speed improvement.
>>>>>
>>>>> Indeed, there is no reason not to make this available in NumPy.
>>>>>
>>>>> Nicolas, can you prepare a patch for numpy ?
>>>>
>>>>
>>>> +1, I agree, this should be a fix in numpy, not scipy.
>>>>
>>>> Be Well
>>>> Anthony
>>>>
>>>>>
>>>>>
>>>>> David
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>
>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>


From sebastian at sipsolutions.net  Thu Nov  8 18:24:43 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Fri, 09 Nov 2012 00:24:43 +0100
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
	<CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
	<CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>
Message-ID: <1352417083.30611.4.camel@sebastian-laptop>

Hey,

On Thu, 2012-11-08 at 14:44 -0800, Nicolas SCHEFFER wrote:
> Well, hinted by what Fabien said, I looked at the C level dot function.
> Quite verbose!
> 
> But starting line 757, we can see that it shouldn't be too much work
> to fix that bug (well there is even a comment there that states just
> that)
> https://github.com/numpy/numpy/blob/master/numpy/core/blasdot/_dotblas.c#L757
> I now think that should be the cleanest.
> 
> This would only work for gemm though. I don't know what the benefit is
> for gemv for instance, but we should make that kind of changes
> everywhere we can.
> The evil PyArray_Copy is there twice and that's what we want to get rid of.
> 
> I'm not sure, but it looks to me that removing the copy and doing the
> following would do the work:
> Order = CblasRowMajor;
> Trans1 = CblasNoTrans;
> Trans2 = CblasNoTrans;
> if (!PyArray_ISCONTIGUOUS(ap1)) {
>     Trans1 = CblasTrans;
> }
> if (!PyArray_ISCONTIGUOUS(ap2)) {
>     Trans2 = CblasTrans;
> }
> might be too easy to be true.
> 

Sounds nice, though don't forget that the array may also be neither C-
or F-Contiguous, in which case you need a copy in any case. So it would
probably be more like:

if (PyArray_IS_C_CONTIGUOUS(ap1)) {
    Trans1 = CblasNoTrans;
}
else if (PyArray_IS_F_CONTIGUOUS(ap1)) {
    Trans1 = CblasTrans;
}
else {
    Trans1 = CblasNoTrans;
    PyObject *new = PyArray_Copy(ap1);
    Py_DECREF(ap1);
    ap1 = (PyArrayObject *)new;
}

Regards,

Sebastian

> 
> 
> On Thu, Nov 8, 2012 at 12:06 PM, Nicolas SCHEFFER
> <scheffer.nicolas at gmail.com> wrote:
> > I've made the necessary changes to get the proper order for the output array.
> > Also, a pass of pep8 and some tests (fixmes are in failing tests)
> > http://pastebin.com/M8TfbURi
> >
> > -n
> >
> > On Thu, Nov 8, 2012 at 11:38 AM, Nicolas SCHEFFER
> > <scheffer.nicolas at gmail.com> wrote:
> >> Thanks for all the responses folks. This is indeed a nice problem to solve.
> >>
> >> Few points:
> >> I. Change the order from 'F' to 'C': I'll look into it.
> >> II. Integration with scipy / numpy: opinions are diverging here.
> >> Let's wait a bit to get more responses on what people think.
> >> One thing though: I'd need the same functionality as get_blas_funcs in numpy.
> >> Since numpy does not require lapack, what functions can I get?
> >> III. Complex arrays
> >> I unfortunately don't have enough knowledge here. If someone could
> >> propose a fix, that'd be great.
> >> IV. C
> >> Writing this in C sounds like a good idea. I'm not sure I'd be the
> >> right person to this though.
> >> V. Patch in numpy
> >> I'd love to do that and learn to do it as a byproduct.
> >> Let's make sure we agree this can go in numpy first and that all FIXME
> >> can be fixed.
> >> Although I guess we can resolve fixmes using git.
> >>
> >> Let me know how you'd like to proceed,
> >>
> >> Thanks!
> >>
> >> FIXMEs:
> >> - Fix for ndim != 2
> >> - Fix for dtype == np.complex*
> >> - Fix order of output array
> >>
> >> On Thu, Nov 8, 2012 at 9:42 AM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
> >>> Hi,
> >>>
> >>> I also think it should go into numpy.dot and that the output order should
> >>> not be changed.
> >>>
> >>> A new point, what about the additional overhead for small ndarray? To remove
> >>> this, I would suggest to put this code into the C function that do the
> >>> actual work (at least, from memory it is a c function, not a python one).
> >>>
> >>> HTH
> >>>
> >>> Fred
> >>>
> >>>
> >>>
> >>> On Thu, Nov 8, 2012 at 12:29 PM, Anthony Scopatz <scopatz at gmail.com> wrote:
> >>>>
> >>>> On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau <cournape at gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
> >>>>> <d.s.seljebotn at astro.uio.no> wrote:
> >>>>> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
> >>>>> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith wrote:
> >>>>> >>> I think everyone would be very happy to see numpy.dot modified to do
> >>>>> >>> this automatically. But adding a scipy.dot IMHO would be fixing
> >>>>> >>> things
> >>>>> >>> in the wrong place and just create extra confusion.
> >>>>> >>
> >>>>> >> I am not sure I agree: numpy is often compiled without lapack support,
> >>>>> >> as
> >>>>> >> it is not necessary. On the other hand scipy is always compiled with
> >>>>> >> lapack. Thus this makes more sens in scipy.
> >>>>> >
> >>>>> > Well, numpy.dot already contains multiple fallback cases for when it is
> >>>>> > compiled with BLAS and not. So I'm +1 on just making this an
> >>>>> > improvement
> >>>>> > on numpy.dot. I don't think there's a time when you would not want to
> >>>>> > use this (provided the output order issue is fixed), and it doesn't
> >>>>> > make
> >>>>> > sense to not have old codes take advantage of the speed improvement.
> >>>>>
> >>>>> Indeed, there is no reason not to make this available in NumPy.
> >>>>>
> >>>>> Nicolas, can you prepare a patch for numpy ?
> >>>>
> >>>>
> >>>> +1, I agree, this should be a fix in numpy, not scipy.
> >>>>
> >>>> Be Well
> >>>> Anthony
> >>>>
> >>>>>
> >>>>>
> >>>>> David
> >>>>> _______________________________________________
> >>>>> NumPy-Discussion mailing list
> >>>>> NumPy-Discussion at scipy.org
> >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> NumPy-Discussion mailing list
> >>>> NumPy-Discussion at scipy.org
> >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> NumPy-Discussion mailing list
> >>> NumPy-Discussion at scipy.org
> >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From sebastian at sipsolutions.net  Thu Nov  8 18:58:29 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Fri, 09 Nov 2012 00:58:29 +0100
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <1352417083.30611.4.camel@sebastian-laptop>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
	<CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
	<CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>
	<1352417083.30611.4.camel@sebastian-laptop>
Message-ID: <1352419109.30611.10.camel@sebastian-laptop>

On Fri, 2012-11-09 at 00:24 +0100, Sebastian Berg wrote:
> Hey,
> 
> On Thu, 2012-11-08 at 14:44 -0800, Nicolas SCHEFFER wrote:
> > Well, hinted by what Fabien said, I looked at the C level dot function.
> > Quite verbose!
> > 
> > But starting line 757, we can see that it shouldn't be too much work
> > to fix that bug (well there is even a comment there that states just
> > that)
> > https://github.com/numpy/numpy/blob/master/numpy/core/blasdot/_dotblas.c#L757
> > I now think that should be the cleanest.
> > 
> > This would only work for gemm though. I don't know what the benefit is
> > for gemv for instance, but we should make that kind of changes
> > everywhere we can.
> > The evil PyArray_Copy is there twice and that's what we want to get rid of.
> > 
> > I'm not sure, but it looks to me that removing the copy and doing the
> > following would do the work:
> > Order = CblasRowMajor;
> > Trans1 = CblasNoTrans;
> > Trans2 = CblasNoTrans;
> > if (!PyArray_ISCONTIGUOUS(ap1)) {
> >     Trans1 = CblasTrans;
> > }
> > if (!PyArray_ISCONTIGUOUS(ap2)) {
> >     Trans2 = CblasTrans;
> > }
> > might be too easy to be true.
> > 
> 
> Sounds nice, though don't forget that the array may also be neither C-
> or F-Contiguous, in which case you need a copy in any case. So it would
> probably be more like:
> 
> if (PyArray_IS_C_CONTIGUOUS(ap1)) {
>     Trans1 = CblasNoTrans;
> }
> else if (PyArray_IS_F_CONTIGUOUS(ap1)) {
>     Trans1 = CblasTrans;
> }
> else {
>     Trans1 = CblasNoTrans;
>     PyObject *new = PyArray_Copy(ap1);
>     Py_DECREF(ap1);
>     ap1 = (PyArrayObject *)new;
> }
> 

Well, of course I forgot error checking there, and maybe you need to set
some of the other parameters differently, but it looks like its probably
that easy, and I am sure everyone will welcome a PR with such changes.

> Regards,
> 
> Sebastian
> 
> > 
> > 
> > On Thu, Nov 8, 2012 at 12:06 PM, Nicolas SCHEFFER
> > <scheffer.nicolas at gmail.com> wrote:
> > > I've made the necessary changes to get the proper order for the output array.
> > > Also, a pass of pep8 and some tests (fixmes are in failing tests)
> > > http://pastebin.com/M8TfbURi
> > >
> > > -n
> > >
> > > On Thu, Nov 8, 2012 at 11:38 AM, Nicolas SCHEFFER
> > > <scheffer.nicolas at gmail.com> wrote:
> > >> Thanks for all the responses folks. This is indeed a nice problem to solve.
> > >>
> > >> Few points:
> > >> I. Change the order from 'F' to 'C': I'll look into it.
> > >> II. Integration with scipy / numpy: opinions are diverging here.
> > >> Let's wait a bit to get more responses on what people think.
> > >> One thing though: I'd need the same functionality as get_blas_funcs in numpy.
> > >> Since numpy does not require lapack, what functions can I get?
> > >> III. Complex arrays
> > >> I unfortunately don't have enough knowledge here. If someone could
> > >> propose a fix, that'd be great.
> > >> IV. C
> > >> Writing this in C sounds like a good idea. I'm not sure I'd be the
> > >> right person to this though.
> > >> V. Patch in numpy
> > >> I'd love to do that and learn to do it as a byproduct.
> > >> Let's make sure we agree this can go in numpy first and that all FIXME
> > >> can be fixed.
> > >> Although I guess we can resolve fixmes using git.
> > >>
> > >> Let me know how you'd like to proceed,
> > >>
> > >> Thanks!
> > >>
> > >> FIXMEs:
> > >> - Fix for ndim != 2
> > >> - Fix for dtype == np.complex*
> > >> - Fix order of output array
> > >>
> > >> On Thu, Nov 8, 2012 at 9:42 AM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
> > >>> Hi,
> > >>>
> > >>> I also think it should go into numpy.dot and that the output order should
> > >>> not be changed.
> > >>>
> > >>> A new point, what about the additional overhead for small ndarray? To remove
> > >>> this, I would suggest to put this code into the C function that do the
> > >>> actual work (at least, from memory it is a c function, not a python one).
> > >>>
> > >>> HTH
> > >>>
> > >>> Fred
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Nov 8, 2012 at 12:29 PM, Anthony Scopatz <scopatz at gmail.com> wrote:
> > >>>>
> > >>>> On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau <cournape at gmail.com>
> > >>>> wrote:
> > >>>>>
> > >>>>> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
> > >>>>> <d.s.seljebotn at astro.uio.no> wrote:
> > >>>>> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
> > >>>>> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith wrote:
> > >>>>> >>> I think everyone would be very happy to see numpy.dot modified to do
> > >>>>> >>> this automatically. But adding a scipy.dot IMHO would be fixing
> > >>>>> >>> things
> > >>>>> >>> in the wrong place and just create extra confusion.
> > >>>>> >>
> > >>>>> >> I am not sure I agree: numpy is often compiled without lapack support,
> > >>>>> >> as
> > >>>>> >> it is not necessary. On the other hand scipy is always compiled with
> > >>>>> >> lapack. Thus this makes more sens in scipy.
> > >>>>> >
> > >>>>> > Well, numpy.dot already contains multiple fallback cases for when it is
> > >>>>> > compiled with BLAS and not. So I'm +1 on just making this an
> > >>>>> > improvement
> > >>>>> > on numpy.dot. I don't think there's a time when you would not want to
> > >>>>> > use this (provided the output order issue is fixed), and it doesn't
> > >>>>> > make
> > >>>>> > sense to not have old codes take advantage of the speed improvement.
> > >>>>>
> > >>>>> Indeed, there is no reason not to make this available in NumPy.
> > >>>>>
> > >>>>> Nicolas, can you prepare a patch for numpy ?
> > >>>>
> > >>>>
> > >>>> +1, I agree, this should be a fix in numpy, not scipy.
> > >>>>
> > >>>> Be Well
> > >>>> Anthony
> > >>>>
> > >>>>>
> > >>>>>
> > >>>>> David
> > >>>>> _______________________________________________
> > >>>>> NumPy-Discussion mailing list
> > >>>>> NumPy-Discussion at scipy.org
> > >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> > >>>>
> > >>>>
> > >>>>
> > >>>> _______________________________________________
> > >>>> NumPy-Discussion mailing list
> > >>>> NumPy-Discussion at scipy.org
> > >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> > >>>>
> > >>>
> > >>>
> > >>> _______________________________________________
> > >>> NumPy-Discussion mailing list
> > >>> NumPy-Discussion at scipy.org
> > >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> > >>>
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> > 
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From scheffer.nicolas at gmail.com  Thu Nov  8 20:03:45 2012
From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER)
Date: Thu, 8 Nov 2012 17:03:45 -0800
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <1352419109.30611.10.camel@sebastian-laptop>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
	<CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
	<CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>
	<1352417083.30611.4.camel@sebastian-laptop>
	<1352419109.30611.10.camel@sebastian-laptop>
Message-ID: <CAHD5bXkym9FQxyKxeEgWMQWB8H8iH5p=dA_ieSyrVn14sy84kg@mail.gmail.com>

Thanks Sebastien, didn't think of that.

Well I went ahead and tried the change, and it's indeed straightforward.

I've run some tests, among which:
nosetests numpy/numpy/core/tests/test_blasdot.py
and it looks ok. I'm assuming this is good news.

I've copy-pasting the diff below, but I have that in my branch and can
create a PR if we agree on it.
I still cannot believe it's that easy (well this has been bugging me a
while... ;))
So I wouldn't mind waiting a day or two to see reactions on the list
before moving ahead.

diff --git a/numpy/core/blasdot/_dotblas.c b/numpy/core/blasdot/_dotblas.c
index c73dd6a..2b4be7c 100644
--- a/numpy/core/blasdot/_dotblas.c
+++ b/numpy/core/blasdot/_dotblas.c
@@ -770,7 +770,7 @@ dotblas_matrixproduct(PyObject *NPY_UNUSED(dummy),
PyObject *args, PyObject* kwa
          * using appropriate values of Order, Trans1, and Trans2.
          */

-        if (!PyArray_ISCONTIGUOUS(ap2)) {
+        if (!PyArray_IS_C_CONTIGUOUS(ap2) && !PyArray_IS_F_CONTIGUOUS(ap2)) {
             PyObject *new = PyArray_Copy(ap2);

             Py_DECREF(ap2);
@@ -779,7 +779,7 @@ dotblas_matrixproduct(PyObject *NPY_UNUSED(dummy),
PyObject *args, PyObject* kwa
                 goto fail;
             }
         }
-        if (!PyArray_ISCONTIGUOUS(ap1)) {
+        if (!PyArray_IS_C_CONTIGUOUS(ap1) && !PyArray_IS_F_CONTIGUOUS(ap1)) {
             PyObject *new = PyArray_Copy(ap1);

             Py_DECREF(ap1);
@@ -800,6 +800,19 @@ dotblas_matrixproduct(PyObject
*NPY_UNUSED(dummy), PyObject *args, PyObject* kwa
         lda = (PyArray_DIM(ap1, 1) > 1 ? PyArray_DIM(ap1, 1) : 1);
         ldb = (PyArray_DIM(ap2, 1) > 1 ? PyArray_DIM(ap2, 1) : 1);
         ldc = (PyArray_DIM(ret, 1) > 1 ? PyArray_DIM(ret, 1) : 1);
+
+        /*
+         * Avoid temporary copies for arrays in Fortran order
+         */
+        if (PyArray_IS_F_CONTIGUOUS(ap1)) {
+            Trans1 = CblasTrans;
+            lda = (PyArray_DIM(ap1, 0) > 1 ? PyArray_DIM(ap1, 0) : 1);
+        }
+        if (PyArray_IS_F_CONTIGUOUS(ap2)) {
+            Trans2 = CblasTrans;
+            ldb = (PyArray_DIM(ap2, 0) > 1 ? PyArray_DIM(ap2, 0) : 1);
+        }
+
         if (typenum == NPY_DOUBLE) {
             cblas_dgemm(Order, Trans1, Trans2,
                         L, N, M,

On Thu, Nov 8, 2012 at 3:58 PM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> On Fri, 2012-11-09 at 00:24 +0100, Sebastian Berg wrote:
>> Hey,
>>
>> On Thu, 2012-11-08 at 14:44 -0800, Nicolas SCHEFFER wrote:
>> > Well, hinted by what Fabien said, I looked at the C level dot function.
>> > Quite verbose!
>> >
>> > But starting line 757, we can see that it shouldn't be too much work
>> > to fix that bug (well there is even a comment there that states just
>> > that)
>> > https://github.com/numpy/numpy/blob/master/numpy/core/blasdot/_dotblas.c#L757
>> > I now think that should be the cleanest.
>> >
>> > This would only work for gemm though. I don't know what the benefit is
>> > for gemv for instance, but we should make that kind of changes
>> > everywhere we can.
>> > The evil PyArray_Copy is there twice and that's what we want to get rid of.
>> >
>> > I'm not sure, but it looks to me that removing the copy and doing the
>> > following would do the work:
>> > Order = CblasRowMajor;
>> > Trans1 = CblasNoTrans;
>> > Trans2 = CblasNoTrans;
>> > if (!PyArray_ISCONTIGUOUS(ap1)) {
>> >     Trans1 = CblasTrans;
>> > }
>> > if (!PyArray_ISCONTIGUOUS(ap2)) {
>> >     Trans2 = CblasTrans;
>> > }
>> > might be too easy to be true.
>> >
>>
>> Sounds nice, though don't forget that the array may also be neither C-
>> or F-Contiguous, in which case you need a copy in any case. So it would
>> probably be more like:
>>
>> if (PyArray_IS_C_CONTIGUOUS(ap1)) {
>>     Trans1 = CblasNoTrans;
>> }
>> else if (PyArray_IS_F_CONTIGUOUS(ap1)) {
>>     Trans1 = CblasTrans;
>> }
>> else {
>>     Trans1 = CblasNoTrans;
>>     PyObject *new = PyArray_Copy(ap1);
>>     Py_DECREF(ap1);
>>     ap1 = (PyArrayObject *)new;
>> }
>>
>
> Well, of course I forgot error checking there, and maybe you need to set
> some of the other parameters differently, but it looks like its probably
> that easy, and I am sure everyone will welcome a PR with such changes.
>
>> Regards,
>>
>> Sebastian
>>
>> >
>> >
>> > On Thu, Nov 8, 2012 at 12:06 PM, Nicolas SCHEFFER
>> > <scheffer.nicolas at gmail.com> wrote:
>> > > I've made the necessary changes to get the proper order for the output array.
>> > > Also, a pass of pep8 and some tests (fixmes are in failing tests)
>> > > http://pastebin.com/M8TfbURi
>> > >
>> > > -n
>> > >
>> > > On Thu, Nov 8, 2012 at 11:38 AM, Nicolas SCHEFFER
>> > > <scheffer.nicolas at gmail.com> wrote:
>> > >> Thanks for all the responses folks. This is indeed a nice problem to solve.
>> > >>
>> > >> Few points:
>> > >> I. Change the order from 'F' to 'C': I'll look into it.
>> > >> II. Integration with scipy / numpy: opinions are diverging here.
>> > >> Let's wait a bit to get more responses on what people think.
>> > >> One thing though: I'd need the same functionality as get_blas_funcs in numpy.
>> > >> Since numpy does not require lapack, what functions can I get?
>> > >> III. Complex arrays
>> > >> I unfortunately don't have enough knowledge here. If someone could
>> > >> propose a fix, that'd be great.
>> > >> IV. C
>> > >> Writing this in C sounds like a good idea. I'm not sure I'd be the
>> > >> right person to this though.
>> > >> V. Patch in numpy
>> > >> I'd love to do that and learn to do it as a byproduct.
>> > >> Let's make sure we agree this can go in numpy first and that all FIXME
>> > >> can be fixed.
>> > >> Although I guess we can resolve fixmes using git.
>> > >>
>> > >> Let me know how you'd like to proceed,
>> > >>
>> > >> Thanks!
>> > >>
>> > >> FIXMEs:
>> > >> - Fix for ndim != 2
>> > >> - Fix for dtype == np.complex*
>> > >> - Fix order of output array
>> > >>
>> > >> On Thu, Nov 8, 2012 at 9:42 AM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
>> > >>> Hi,
>> > >>>
>> > >>> I also think it should go into numpy.dot and that the output order should
>> > >>> not be changed.
>> > >>>
>> > >>> A new point, what about the additional overhead for small ndarray? To remove
>> > >>> this, I would suggest to put this code into the C function that do the
>> > >>> actual work (at least, from memory it is a c function, not a python one).
>> > >>>
>> > >>> HTH
>> > >>>
>> > >>> Fred
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Thu, Nov 8, 2012 at 12:29 PM, Anthony Scopatz <scopatz at gmail.com> wrote:
>> > >>>>
>> > >>>> On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau <cournape at gmail.com>
>> > >>>> wrote:
>> > >>>>>
>> > >>>>> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
>> > >>>>> <d.s.seljebotn at astro.uio.no> wrote:
>> > >>>>> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
>> > >>>>> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith wrote:
>> > >>>>> >>> I think everyone would be very happy to see numpy.dot modified to do
>> > >>>>> >>> this automatically. But adding a scipy.dot IMHO would be fixing
>> > >>>>> >>> things
>> > >>>>> >>> in the wrong place and just create extra confusion.
>> > >>>>> >>
>> > >>>>> >> I am not sure I agree: numpy is often compiled without lapack support,
>> > >>>>> >> as
>> > >>>>> >> it is not necessary. On the other hand scipy is always compiled with
>> > >>>>> >> lapack. Thus this makes more sens in scipy.
>> > >>>>> >
>> > >>>>> > Well, numpy.dot already contains multiple fallback cases for when it is
>> > >>>>> > compiled with BLAS and not. So I'm +1 on just making this an
>> > >>>>> > improvement
>> > >>>>> > on numpy.dot. I don't think there's a time when you would not want to
>> > >>>>> > use this (provided the output order issue is fixed), and it doesn't
>> > >>>>> > make
>> > >>>>> > sense to not have old codes take advantage of the speed improvement.
>> > >>>>>
>> > >>>>> Indeed, there is no reason not to make this available in NumPy.
>> > >>>>>
>> > >>>>> Nicolas, can you prepare a patch for numpy ?
>> > >>>>
>> > >>>>
>> > >>>> +1, I agree, this should be a fix in numpy, not scipy.
>> > >>>>
>> > >>>> Be Well
>> > >>>> Anthony
>> > >>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> David
>> > >>>>> _______________________________________________
>> > >>>>> NumPy-Discussion mailing list
>> > >>>>> NumPy-Discussion at scipy.org
>> > >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> _______________________________________________
>> > >>>> NumPy-Discussion mailing list
>> > >>>> NumPy-Discussion at scipy.org
>> > >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> > >>>>
>> > >>>
>> > >>>
>> > >>> _______________________________________________
>> > >>> NumPy-Discussion mailing list
>> > >>> NumPy-Discussion at scipy.org
>> > >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> > >>>
>> > _______________________________________________
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From nouiz at nouiz.org  Thu Nov  8 20:34:15 2012
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Thu, 8 Nov 2012 20:34:15 -0500
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHD5bXkym9FQxyKxeEgWMQWB8H8iH5p=dA_ieSyrVn14sy84kg@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
	<CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
	<CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>
	<1352417083.30611.4.camel@sebastian-laptop>
	<1352419109.30611.10.camel@sebastian-laptop>
	<CAHD5bXkym9FQxyKxeEgWMQWB8H8iH5p=dA_ieSyrVn14sy84kg@mail.gmail.com>
Message-ID: <CADKKbtiE0h8YLBtw764uKf0udOdOWav1qoDZ9wm1MtCd8-_cUw@mail.gmail.com>

Hi,

I suspect the current tests are not enought. You need to test all the
combination for the 3 inputs with thoses strides:

c-contiguous
f-contiguous
something else like strided.

Also, try with matrix with shape of 1 in each dimensions. Not all blas
libraries accept the strides that numpy use in that cases. Also, not all
blas version accept the same stuff, so if this isn't in the current
version, there will be probably some adjustment later on that side. What
blas do you use? I think ATLAS was one that was causing problem.


When we did this in Theano, it was more complicated then this diff... But
much of the code is boillerplate code.

Fred


On Thu, Nov 8, 2012 at 8:03 PM, Nicolas SCHEFFER <scheffer.nicolas at gmail.com
> wrote:

> Thanks Sebastien, didn't think of that.
>
> Well I went ahead and tried the change, and it's indeed straightforward.
>
> I've run some tests, among which:
> nosetests numpy/numpy/core/tests/test_blasdot.py
> and it looks ok. I'm assuming this is good news.
>
> I've copy-pasting the diff below, but I have that in my branch and can
> create a PR if we agree on it.
> I still cannot believe it's that easy (well this has been bugging me a
> while... ;))
> So I wouldn't mind waiting a day or two to see reactions on the list
> before moving ahead.
>
> diff --git a/numpy/core/blasdot/_dotblas.c b/numpy/core/blasdot/_dotblas.c
> index c73dd6a..2b4be7c 100644
> --- a/numpy/core/blasdot/_dotblas.c
> +++ b/numpy/core/blasdot/_dotblas.c
> @@ -770,7 +770,7 @@ dotblas_matrixproduct(PyObject *NPY_UNUSED(dummy),
> PyObject *args, PyObject* kwa
>           * using appropriate values of Order, Trans1, and Trans2.
>           */
>
> -        if (!PyArray_ISCONTIGUOUS(ap2)) {
> +        if (!PyArray_IS_C_CONTIGUOUS(ap2) &&
> !PyArray_IS_F_CONTIGUOUS(ap2)) {
>              PyObject *new = PyArray_Copy(ap2);
>
>              Py_DECREF(ap2);
> @@ -779,7 +779,7 @@ dotblas_matrixproduct(PyObject *NPY_UNUSED(dummy),
> PyObject *args, PyObject* kwa
>                  goto fail;
>              }
>          }
> -        if (!PyArray_ISCONTIGUOUS(ap1)) {
> +        if (!PyArray_IS_C_CONTIGUOUS(ap1) &&
> !PyArray_IS_F_CONTIGUOUS(ap1)) {
>              PyObject *new = PyArray_Copy(ap1);
>
>              Py_DECREF(ap1);
> @@ -800,6 +800,19 @@ dotblas_matrixproduct(PyObject
> *NPY_UNUSED(dummy), PyObject *args, PyObject* kwa
>          lda = (PyArray_DIM(ap1, 1) > 1 ? PyArray_DIM(ap1, 1) : 1);
>          ldb = (PyArray_DIM(ap2, 1) > 1 ? PyArray_DIM(ap2, 1) : 1);
>          ldc = (PyArray_DIM(ret, 1) > 1 ? PyArray_DIM(ret, 1) : 1);
> +
> +        /*
> +         * Avoid temporary copies for arrays in Fortran order
> +         */
> +        if (PyArray_IS_F_CONTIGUOUS(ap1)) {
> +            Trans1 = CblasTrans;
> +            lda = (PyArray_DIM(ap1, 0) > 1 ? PyArray_DIM(ap1, 0) : 1);
> +        }
> +        if (PyArray_IS_F_CONTIGUOUS(ap2)) {
> +            Trans2 = CblasTrans;
> +            ldb = (PyArray_DIM(ap2, 0) > 1 ? PyArray_DIM(ap2, 0) : 1);
> +        }
> +
>          if (typenum == NPY_DOUBLE) {
>              cblas_dgemm(Order, Trans1, Trans2,
>                          L, N, M,
>
> On Thu, Nov 8, 2012 at 3:58 PM, Sebastian Berg
> <sebastian at sipsolutions.net> wrote:
> > On Fri, 2012-11-09 at 00:24 +0100, Sebastian Berg wrote:
> >> Hey,
> >>
> >> On Thu, 2012-11-08 at 14:44 -0800, Nicolas SCHEFFER wrote:
> >> > Well, hinted by what Fabien said, I looked at the C level dot
> function.
> >> > Quite verbose!
> >> >
> >> > But starting line 757, we can see that it shouldn't be too much work
> >> > to fix that bug (well there is even a comment there that states just
> >> > that)
> >> >
> https://github.com/numpy/numpy/blob/master/numpy/core/blasdot/_dotblas.c#L757
> >> > I now think that should be the cleanest.
> >> >
> >> > This would only work for gemm though. I don't know what the benefit is
> >> > for gemv for instance, but we should make that kind of changes
> >> > everywhere we can.
> >> > The evil PyArray_Copy is there twice and that's what we want to get
> rid of.
> >> >
> >> > I'm not sure, but it looks to me that removing the copy and doing the
> >> > following would do the work:
> >> > Order = CblasRowMajor;
> >> > Trans1 = CblasNoTrans;
> >> > Trans2 = CblasNoTrans;
> >> > if (!PyArray_ISCONTIGUOUS(ap1)) {
> >> >     Trans1 = CblasTrans;
> >> > }
> >> > if (!PyArray_ISCONTIGUOUS(ap2)) {
> >> >     Trans2 = CblasTrans;
> >> > }
> >> > might be too easy to be true.
> >> >
> >>
> >> Sounds nice, though don't forget that the array may also be neither C-
> >> or F-Contiguous, in which case you need a copy in any case. So it would
> >> probably be more like:
> >>
> >> if (PyArray_IS_C_CONTIGUOUS(ap1)) {
> >>     Trans1 = CblasNoTrans;
> >> }
> >> else if (PyArray_IS_F_CONTIGUOUS(ap1)) {
> >>     Trans1 = CblasTrans;
> >> }
> >> else {
> >>     Trans1 = CblasNoTrans;
> >>     PyObject *new = PyArray_Copy(ap1);
> >>     Py_DECREF(ap1);
> >>     ap1 = (PyArrayObject *)new;
> >> }
> >>
> >
> > Well, of course I forgot error checking there, and maybe you need to set
> > some of the other parameters differently, but it looks like its probably
> > that easy, and I am sure everyone will welcome a PR with such changes.
> >
> >> Regards,
> >>
> >> Sebastian
> >>
> >> >
> >> >
> >> > On Thu, Nov 8, 2012 at 12:06 PM, Nicolas SCHEFFER
> >> > <scheffer.nicolas at gmail.com> wrote:
> >> > > I've made the necessary changes to get the proper order for the
> output array.
> >> > > Also, a pass of pep8 and some tests (fixmes are in failing tests)
> >> > > http://pastebin.com/M8TfbURi
> >> > >
> >> > > -n
> >> > >
> >> > > On Thu, Nov 8, 2012 at 11:38 AM, Nicolas SCHEFFER
> >> > > <scheffer.nicolas at gmail.com> wrote:
> >> > >> Thanks for all the responses folks. This is indeed a nice problem
> to solve.
> >> > >>
> >> > >> Few points:
> >> > >> I. Change the order from 'F' to 'C': I'll look into it.
> >> > >> II. Integration with scipy / numpy: opinions are diverging here.
> >> > >> Let's wait a bit to get more responses on what people think.
> >> > >> One thing though: I'd need the same functionality as
> get_blas_funcs in numpy.
> >> > >> Since numpy does not require lapack, what functions can I get?
> >> > >> III. Complex arrays
> >> > >> I unfortunately don't have enough knowledge here. If someone could
> >> > >> propose a fix, that'd be great.
> >> > >> IV. C
> >> > >> Writing this in C sounds like a good idea. I'm not sure I'd be the
> >> > >> right person to this though.
> >> > >> V. Patch in numpy
> >> > >> I'd love to do that and learn to do it as a byproduct.
> >> > >> Let's make sure we agree this can go in numpy first and that all
> FIXME
> >> > >> can be fixed.
> >> > >> Although I guess we can resolve fixmes using git.
> >> > >>
> >> > >> Let me know how you'd like to proceed,
> >> > >>
> >> > >> Thanks!
> >> > >>
> >> > >> FIXMEs:
> >> > >> - Fix for ndim != 2
> >> > >> - Fix for dtype == np.complex*
> >> > >> - Fix order of output array
> >> > >>
> >> > >> On Thu, Nov 8, 2012 at 9:42 AM, Fr?d?ric Bastien <nouiz at nouiz.org>
> wrote:
> >> > >>> Hi,
> >> > >>>
> >> > >>> I also think it should go into numpy.dot and that the output
> order should
> >> > >>> not be changed.
> >> > >>>
> >> > >>> A new point, what about the additional overhead for small
> ndarray? To remove
> >> > >>> this, I would suggest to put this code into the C function that
> do the
> >> > >>> actual work (at least, from memory it is a c function, not a
> python one).
> >> > >>>
> >> > >>> HTH
> >> > >>>
> >> > >>> Fred
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> On Thu, Nov 8, 2012 at 12:29 PM, Anthony Scopatz <
> scopatz at gmail.com> wrote:
> >> > >>>>
> >> > >>>> On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau <
> cournape at gmail.com>
> >> > >>>> wrote:
> >> > >>>>>
> >> > >>>>> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
> >> > >>>>> <d.s.seljebotn at astro.uio.no> wrote:
> >> > >>>>> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
> >> > >>>>> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith
> wrote:
> >> > >>>>> >>> I think everyone would be very happy to see numpy.dot
> modified to do
> >> > >>>>> >>> this automatically. But adding a scipy.dot IMHO would be
> fixing
> >> > >>>>> >>> things
> >> > >>>>> >>> in the wrong place and just create extra confusion.
> >> > >>>>> >>
> >> > >>>>> >> I am not sure I agree: numpy is often compiled without
> lapack support,
> >> > >>>>> >> as
> >> > >>>>> >> it is not necessary. On the other hand scipy is always
> compiled with
> >> > >>>>> >> lapack. Thus this makes more sens in scipy.
> >> > >>>>> >
> >> > >>>>> > Well, numpy.dot already contains multiple fallback cases for
> when it is
> >> > >>>>> > compiled with BLAS and not. So I'm +1 on just making this an
> >> > >>>>> > improvement
> >> > >>>>> > on numpy.dot. I don't think there's a time when you would not
> want to
> >> > >>>>> > use this (provided the output order issue is fixed), and it
> doesn't
> >> > >>>>> > make
> >> > >>>>> > sense to not have old codes take advantage of the speed
> improvement.
> >> > >>>>>
> >> > >>>>> Indeed, there is no reason not to make this available in NumPy.
> >> > >>>>>
> >> > >>>>> Nicolas, can you prepare a patch for numpy ?
> >> > >>>>
> >> > >>>>
> >> > >>>> +1, I agree, this should be a fix in numpy, not scipy.
> >> > >>>>
> >> > >>>> Be Well
> >> > >>>> Anthony
> >> > >>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> David
> >> > >>>>> _______________________________________________
> >> > >>>>> NumPy-Discussion mailing list
> >> > >>>>> NumPy-Discussion at scipy.org
> >> > >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >> > >>>>
> >> > >>>>
> >> > >>>>
> >> > >>>> _______________________________________________
> >> > >>>> NumPy-Discussion mailing list
> >> > >>>> NumPy-Discussion at scipy.org
> >> > >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >> > >>>>
> >> > >>>
> >> > >>>
> >> > >>> _______________________________________________
> >> > >>> NumPy-Discussion mailing list
> >> > >>> NumPy-Discussion at scipy.org
> >> > >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >> > >>>
> >> > _______________________________________________
> >> > NumPy-Discussion mailing list
> >> > NumPy-Discussion at scipy.org
> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >> >
> >>
> >>
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121108/6de4c7d2/attachment.html>

From scheffer.nicolas at gmail.com  Fri Nov  9 01:18:32 2012
From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER)
Date: Thu, 8 Nov 2012 22:18:32 -0800
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CADKKbtiE0h8YLBtw764uKf0udOdOWav1qoDZ9wm1MtCd8-_cUw@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
	<CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
	<CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>
	<1352417083.30611.4.camel@sebastian-laptop>
	<1352419109.30611.10.camel@sebastian-laptop>
	<CAHD5bXkym9FQxyKxeEgWMQWB8H8iH5p=dA_ieSyrVn14sy84kg@mail.gmail.com>
	<CADKKbtiE0h8YLBtw764uKf0udOdOWav1qoDZ9wm1MtCd8-_cUw@mail.gmail.com>
Message-ID: <CAHD5bXkHZxcL59boEyWrhS+TKZmJEoew0mZRNyvN=jczUO2Q1g@mail.gmail.com>

Fred,

Thanks for the advice.
The code will only affect the part in _dotblas.c where gemm is called.
There's tons of check before that make sure both matrices are of ndim 2.
We should check though if we can do these tricks in other parts of the function.

Otherwise:
- I've built against ATLAS 3.10
- I'm happy to add a couple more test for C and F-contiguous. I'm not
sure how to get the third type (strided), would you have an example?

The following test for instance checks integrity against
multiarray.dot, which I believe is default when not compiled with
BLAS.
Dot is a hard function to test imho, so if anybody has ideas on what
kind of test they'd like to see, please let me know.

If that's ok I might now be able to:
- Check for more bugs, I need to dig a bit more in the gemm call, make
sure everything is ok.
- Create an issue on github and link to this discussion
- Make a commit in a seperate branch
- Move forward like that.

==
import numpy as np
from time import time
from numpy.testing import assert_almost_equal

def test_dot_regression():
    """ Test numpy dot by comparing with multiarray dot
    """
    np.random.seed(7)
    a = np.random.randn(3, 3)
    b = np.random.randn(3, 2)
    c = np.random.randn(2, 3)

    _dot = np.core.multiarray.dot

    assert_almost_equal(np.dot(a, a), _dot(a, a))
    assert_almost_equal(np.dot(b, c), _dot(b, c))
    assert_almost_equal(np.dot(b.T, c.T), _dot(b.T, c.T))

    assert_almost_equal(np.dot(a.T, a), _dot(a.T, a))
    assert_almost_equal(np.dot(a, a.T), _dot(a, a.T))
    assert_almost_equal(np.dot(a.T, a.T), _dot(a.T, a.T))

On Thu, Nov 8, 2012 at 5:34 PM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
> Hi,
>
> I suspect the current tests are not enought. You need to test all the
> combination for the 3 inputs with thoses strides:
>
> c-contiguous
> f-contiguous
> something else like strided.
>
> Also, try with matrix with shape of 1 in each dimensions. Not all blas
> libraries accept the strides that numpy use in that cases. Also, not all
> blas version accept the same stuff, so if this isn't in the current version,
> there will be probably some adjustment later on that side. What blas do you
> use? I think ATLAS was one that was causing problem.
>
>
> When we did this in Theano, it was more complicated then this diff... But
> much of the code is boillerplate code.
>
> Fred
>
>
>
> On Thu, Nov 8, 2012 at 8:03 PM, Nicolas SCHEFFER
> <scheffer.nicolas at gmail.com> wrote:
>>
>> Thanks Sebastien, didn't think of that.
>>
>> Well I went ahead and tried the change, and it's indeed straightforward.
>>
>> I've run some tests, among which:
>> nosetests numpy/numpy/core/tests/test_blasdot.py
>> and it looks ok. I'm assuming this is good news.
>>
>> I've copy-pasting the diff below, but I have that in my branch and can
>> create a PR if we agree on it.
>> I still cannot believe it's that easy (well this has been bugging me a
>> while... ;))
>> So I wouldn't mind waiting a day or two to see reactions on the list
>> before moving ahead.
>>
>> diff --git a/numpy/core/blasdot/_dotblas.c b/numpy/core/blasdot/_dotblas.c
>> index c73dd6a..2b4be7c 100644
>> --- a/numpy/core/blasdot/_dotblas.c
>> +++ b/numpy/core/blasdot/_dotblas.c
>> @@ -770,7 +770,7 @@ dotblas_matrixproduct(PyObject *NPY_UNUSED(dummy),
>> PyObject *args, PyObject* kwa
>>           * using appropriate values of Order, Trans1, and Trans2.
>>           */
>>
>> -        if (!PyArray_ISCONTIGUOUS(ap2)) {
>> +        if (!PyArray_IS_C_CONTIGUOUS(ap2) &&
>> !PyArray_IS_F_CONTIGUOUS(ap2)) {
>>              PyObject *new = PyArray_Copy(ap2);
>>
>>              Py_DECREF(ap2);
>> @@ -779,7 +779,7 @@ dotblas_matrixproduct(PyObject *NPY_UNUSED(dummy),
>> PyObject *args, PyObject* kwa
>>                  goto fail;
>>              }
>>          }
>> -        if (!PyArray_ISCONTIGUOUS(ap1)) {
>> +        if (!PyArray_IS_C_CONTIGUOUS(ap1) &&
>> !PyArray_IS_F_CONTIGUOUS(ap1)) {
>>              PyObject *new = PyArray_Copy(ap1);
>>
>>              Py_DECREF(ap1);
>> @@ -800,6 +800,19 @@ dotblas_matrixproduct(PyObject
>> *NPY_UNUSED(dummy), PyObject *args, PyObject* kwa
>>          lda = (PyArray_DIM(ap1, 1) > 1 ? PyArray_DIM(ap1, 1) : 1);
>>          ldb = (PyArray_DIM(ap2, 1) > 1 ? PyArray_DIM(ap2, 1) : 1);
>>          ldc = (PyArray_DIM(ret, 1) > 1 ? PyArray_DIM(ret, 1) : 1);
>> +
>> +        /*
>> +         * Avoid temporary copies for arrays in Fortran order
>> +         */
>> +        if (PyArray_IS_F_CONTIGUOUS(ap1)) {
>> +            Trans1 = CblasTrans;
>> +            lda = (PyArray_DIM(ap1, 0) > 1 ? PyArray_DIM(ap1, 0) : 1);
>> +        }
>> +        if (PyArray_IS_F_CONTIGUOUS(ap2)) {
>> +            Trans2 = CblasTrans;
>> +            ldb = (PyArray_DIM(ap2, 0) > 1 ? PyArray_DIM(ap2, 0) : 1);
>> +        }
>> +
>>          if (typenum == NPY_DOUBLE) {
>>              cblas_dgemm(Order, Trans1, Trans2,
>>                          L, N, M,
>>
>> On Thu, Nov 8, 2012 at 3:58 PM, Sebastian Berg
>> <sebastian at sipsolutions.net> wrote:
>> > On Fri, 2012-11-09 at 00:24 +0100, Sebastian Berg wrote:
>> >> Hey,
>> >>
>> >> On Thu, 2012-11-08 at 14:44 -0800, Nicolas SCHEFFER wrote:
>> >> > Well, hinted by what Fabien said, I looked at the C level dot
>> >> > function.
>> >> > Quite verbose!
>> >> >
>> >> > But starting line 757, we can see that it shouldn't be too much work
>> >> > to fix that bug (well there is even a comment there that states just
>> >> > that)
>> >> >
>> >> > https://github.com/numpy/numpy/blob/master/numpy/core/blasdot/_dotblas.c#L757
>> >> > I now think that should be the cleanest.
>> >> >
>> >> > This would only work for gemm though. I don't know what the benefit
>> >> > is
>> >> > for gemv for instance, but we should make that kind of changes
>> >> > everywhere we can.
>> >> > The evil PyArray_Copy is there twice and that's what we want to get
>> >> > rid of.
>> >> >
>> >> > I'm not sure, but it looks to me that removing the copy and doing the
>> >> > following would do the work:
>> >> > Order = CblasRowMajor;
>> >> > Trans1 = CblasNoTrans;
>> >> > Trans2 = CblasNoTrans;
>> >> > if (!PyArray_ISCONTIGUOUS(ap1)) {
>> >> >     Trans1 = CblasTrans;
>> >> > }
>> >> > if (!PyArray_ISCONTIGUOUS(ap2)) {
>> >> >     Trans2 = CblasTrans;
>> >> > }
>> >> > might be too easy to be true.
>> >> >
>> >>
>> >> Sounds nice, though don't forget that the array may also be neither C-
>> >> or F-Contiguous, in which case you need a copy in any case. So it would
>> >> probably be more like:
>> >>
>> >> if (PyArray_IS_C_CONTIGUOUS(ap1)) {
>> >>     Trans1 = CblasNoTrans;
>> >> }
>> >> else if (PyArray_IS_F_CONTIGUOUS(ap1)) {
>> >>     Trans1 = CblasTrans;
>> >> }
>> >> else {
>> >>     Trans1 = CblasNoTrans;
>> >>     PyObject *new = PyArray_Copy(ap1);
>> >>     Py_DECREF(ap1);
>> >>     ap1 = (PyArrayObject *)new;
>> >> }
>> >>
>> >
>> > Well, of course I forgot error checking there, and maybe you need to set
>> > some of the other parameters differently, but it looks like its probably
>> > that easy, and I am sure everyone will welcome a PR with such changes.
>> >
>> >> Regards,
>> >>
>> >> Sebastian
>> >>
>> >> >
>> >> >
>> >> > On Thu, Nov 8, 2012 at 12:06 PM, Nicolas SCHEFFER
>> >> > <scheffer.nicolas at gmail.com> wrote:
>> >> > > I've made the necessary changes to get the proper order for the
>> >> > > output array.
>> >> > > Also, a pass of pep8 and some tests (fixmes are in failing tests)
>> >> > > http://pastebin.com/M8TfbURi
>> >> > >
>> >> > > -n
>> >> > >
>> >> > > On Thu, Nov 8, 2012 at 11:38 AM, Nicolas SCHEFFER
>> >> > > <scheffer.nicolas at gmail.com> wrote:
>> >> > >> Thanks for all the responses folks. This is indeed a nice problem
>> >> > >> to solve.
>> >> > >>
>> >> > >> Few points:
>> >> > >> I. Change the order from 'F' to 'C': I'll look into it.
>> >> > >> II. Integration with scipy / numpy: opinions are diverging here.
>> >> > >> Let's wait a bit to get more responses on what people think.
>> >> > >> One thing though: I'd need the same functionality as
>> >> > >> get_blas_funcs in numpy.
>> >> > >> Since numpy does not require lapack, what functions can I get?
>> >> > >> III. Complex arrays
>> >> > >> I unfortunately don't have enough knowledge here. If someone could
>> >> > >> propose a fix, that'd be great.
>> >> > >> IV. C
>> >> > >> Writing this in C sounds like a good idea. I'm not sure I'd be the
>> >> > >> right person to this though.
>> >> > >> V. Patch in numpy
>> >> > >> I'd love to do that and learn to do it as a byproduct.
>> >> > >> Let's make sure we agree this can go in numpy first and that all
>> >> > >> FIXME
>> >> > >> can be fixed.
>> >> > >> Although I guess we can resolve fixmes using git.
>> >> > >>
>> >> > >> Let me know how you'd like to proceed,
>> >> > >>
>> >> > >> Thanks!
>> >> > >>
>> >> > >> FIXMEs:
>> >> > >> - Fix for ndim != 2
>> >> > >> - Fix for dtype == np.complex*
>> >> > >> - Fix order of output array
>> >> > >>
>> >> > >> On Thu, Nov 8, 2012 at 9:42 AM, Fr?d?ric Bastien <nouiz at nouiz.org>
>> >> > >> wrote:
>> >> > >>> Hi,
>> >> > >>>
>> >> > >>> I also think it should go into numpy.dot and that the output
>> >> > >>> order should
>> >> > >>> not be changed.
>> >> > >>>
>> >> > >>> A new point, what about the additional overhead for small
>> >> > >>> ndarray? To remove
>> >> > >>> this, I would suggest to put this code into the C function that
>> >> > >>> do the
>> >> > >>> actual work (at least, from memory it is a c function, not a
>> >> > >>> python one).
>> >> > >>>
>> >> > >>> HTH
>> >> > >>>
>> >> > >>> Fred
>> >> > >>>
>> >> > >>>
>> >> > >>>
>> >> > >>> On Thu, Nov 8, 2012 at 12:29 PM, Anthony Scopatz
>> >> > >>> <scopatz at gmail.com> wrote:
>> >> > >>>>
>> >> > >>>> On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau
>> >> > >>>> <cournape at gmail.com>
>> >> > >>>> wrote:
>> >> > >>>>>
>> >> > >>>>> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
>> >> > >>>>> <d.s.seljebotn at astro.uio.no> wrote:
>> >> > >>>>> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
>> >> > >>>>> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith
>> >> > >>>>> >> wrote:
>> >> > >>>>> >>> I think everyone would be very happy to see numpy.dot
>> >> > >>>>> >>> modified to do
>> >> > >>>>> >>> this automatically. But adding a scipy.dot IMHO would be
>> >> > >>>>> >>> fixing
>> >> > >>>>> >>> things
>> >> > >>>>> >>> in the wrong place and just create extra confusion.
>> >> > >>>>> >>
>> >> > >>>>> >> I am not sure I agree: numpy is often compiled without
>> >> > >>>>> >> lapack support,
>> >> > >>>>> >> as
>> >> > >>>>> >> it is not necessary. On the other hand scipy is always
>> >> > >>>>> >> compiled with
>> >> > >>>>> >> lapack. Thus this makes more sens in scipy.
>> >> > >>>>> >
>> >> > >>>>> > Well, numpy.dot already contains multiple fallback cases for
>> >> > >>>>> > when it is
>> >> > >>>>> > compiled with BLAS and not. So I'm +1 on just making this an
>> >> > >>>>> > improvement
>> >> > >>>>> > on numpy.dot. I don't think there's a time when you would not
>> >> > >>>>> > want to
>> >> > >>>>> > use this (provided the output order issue is fixed), and it
>> >> > >>>>> > doesn't
>> >> > >>>>> > make
>> >> > >>>>> > sense to not have old codes take advantage of the speed
>> >> > >>>>> > improvement.
>> >> > >>>>>
>> >> > >>>>> Indeed, there is no reason not to make this available in NumPy.
>> >> > >>>>>
>> >> > >>>>> Nicolas, can you prepare a patch for numpy ?
>> >> > >>>>
>> >> > >>>>
>> >> > >>>> +1, I agree, this should be a fix in numpy, not scipy.
>> >> > >>>>
>> >> > >>>> Be Well
>> >> > >>>> Anthony
>> >> > >>>>
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>> David
>> >> > >>>>> _______________________________________________
>> >> > >>>>> NumPy-Discussion mailing list
>> >> > >>>>> NumPy-Discussion at scipy.org
>> >> > >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >> > >>>>
>> >> > >>>>
>> >> > >>>>
>> >> > >>>> _______________________________________________
>> >> > >>>> NumPy-Discussion mailing list
>> >> > >>>> NumPy-Discussion at scipy.org
>> >> > >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >> > >>>>
>> >> > >>>
>> >> > >>>
>> >> > >>> _______________________________________________
>> >> > >>> NumPy-Discussion mailing list
>> >> > >>> NumPy-Discussion at scipy.org
>> >> > >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >> > >>>
>> >> > _______________________________________________
>> >> > NumPy-Discussion mailing list
>> >> > NumPy-Discussion at scipy.org
>> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >> >
>> >>
>> >>
>> >> _______________________________________________
>> >> NumPy-Discussion mailing list
>> >> NumPy-Discussion at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> >
>> > _______________________________________________
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From njs at pobox.com  Fri Nov  9 03:32:52 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 9 Nov 2012 08:32:52 +0000
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHD5bXkHZxcL59boEyWrhS+TKZmJEoew0mZRNyvN=jczUO2Q1g@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
	<CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
	<CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>
	<1352417083.30611.4.camel@sebastian-laptop>
	<1352419109.30611.10.camel@sebastian-laptop>
	<CAHD5bXkym9FQxyKxeEgWMQWB8H8iH5p=dA_ieSyrVn14sy84kg@mail.gmail.com>
	<CADKKbtiE0h8YLBtw764uKf0udOdOWav1qoDZ9wm1MtCd8-_cUw@mail.gmail.com>
	<CAHD5bXkHZxcL59boEyWrhS+TKZmJEoew0mZRNyvN=jczUO2Q1g@mail.gmail.com>
Message-ID: <CAPJVwBk5RHziKTFkH7gPTk4neafwrfM+vWTuLv-aWKq=vs0aFA@mail.gmail.com>

On Fri, Nov 9, 2012 at 6:18 AM, Nicolas SCHEFFER
<scheffer.nicolas at gmail.com> wrote:
> Fred,
>
> Thanks for the advice.
> The code will only affect the part in _dotblas.c where gemm is called.
> There's tons of check before that make sure both matrices are of ndim 2.
> We should check though if we can do these tricks in other parts of the function.
>
> Otherwise:
> - I've built against ATLAS 3.10
> - I'm happy to add a couple more test for C and F-contiguous. I'm not
> sure how to get the third type (strided), would you have an example?

def with_memory_order(a, order):
    assert order in ("C", "F", "discontig")
    assert a.ndim == 2
    if order in ("C", "F"):
        return np.asarray(a, order=order)
    else:
        buf = np.empty((a.shape[0] * 2, a.shape[1] * 2), dtype=a.dtype)
        buf[::2, ::2] = a
        # This returns a view onto every other element of 'buf':
        result = buf[::2, ::2]
        assert not result.flags.c_contiguous and not result.flags.f_contiguous
        return result

> The following test for instance checks integrity against
> multiarray.dot, which I believe is default when not compiled with
> BLAS.
> Dot is a hard function to test imho, so if anybody has ideas on what
> kind of test they'd like to see, please let me know.
>
> If that's ok I might now be able to:
> - Check for more bugs, I need to dig a bit more in the gemm call, make
> sure everything is ok.
> - Create an issue on github and link to this discussion
> - Make a commit in a seperate branch
> - Move forward like that.
>
> ==
> import numpy as np
> from time import time
> from numpy.testing import assert_almost_equal
>
> def test_dot_regression():
>     """ Test numpy dot by comparing with multiarray dot
>     """
>     np.random.seed(7)
>     a = np.random.randn(3, 3)
>     b = np.random.randn(3, 2)
>     c = np.random.randn(2, 3)
>
>     _dot = np.core.multiarray.dot
>
>     assert_almost_equal(np.dot(a, a), _dot(a, a))
>     assert_almost_equal(np.dot(b, c), _dot(b, c))
>     assert_almost_equal(np.dot(b.T, c.T), _dot(b.T, c.T))
>
>     assert_almost_equal(np.dot(a.T, a), _dot(a.T, a))
>     assert_almost_equal(np.dot(a, a.T), _dot(a, a.T))
>     assert_almost_equal(np.dot(a.T, a.T), _dot(a.T, a.T))

You should check that the result is C-contiguous in all cases too.

for a_order in ("C", "F", "discontig"):
    for b_order in ("C", "F", "discontig"):
        this_a = with_memory_order(a, a_order)
        this_b = with_memory_order(b, b_order)
        result = np.dot(this_a, this_b)
        assert_almost_equal(result, expected)
        assert result.flags.c_contiguous

You could also wrap the above in yet another loop to try a few
different combinations of a and b matrices (perhaps after sticking the
code into a utility function, like run_dot_tests(a, b, expected), so
the indentation doesn't get out of hand ;-)). Then you can easily test
some of the edge cases, like Nx1 matrices.

-n

> On Thu, Nov 8, 2012 at 5:34 PM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
>> Hi,
>>
>> I suspect the current tests are not enought. You need to test all the
>> combination for the 3 inputs with thoses strides:
>>
>> c-contiguous
>> f-contiguous
>> something else like strided.
>>
>> Also, try with matrix with shape of 1 in each dimensions. Not all blas
>> libraries accept the strides that numpy use in that cases. Also, not all
>> blas version accept the same stuff, so if this isn't in the current version,
>> there will be probably some adjustment later on that side. What blas do you
>> use? I think ATLAS was one that was causing problem.
>>
>>
>> When we did this in Theano, it was more complicated then this diff... But
>> much of the code is boillerplate code.
>>
>> Fred
>>
>>
>>
>> On Thu, Nov 8, 2012 at 8:03 PM, Nicolas SCHEFFER
>> <scheffer.nicolas at gmail.com> wrote:
>>>
>>> Thanks Sebastien, didn't think of that.
>>>
>>> Well I went ahead and tried the change, and it's indeed straightforward.
>>>
>>> I've run some tests, among which:
>>> nosetests numpy/numpy/core/tests/test_blasdot.py
>>> and it looks ok. I'm assuming this is good news.
>>>
>>> I've copy-pasting the diff below, but I have that in my branch and can
>>> create a PR if we agree on it.
>>> I still cannot believe it's that easy (well this has been bugging me a
>>> while... ;))
>>> So I wouldn't mind waiting a day or two to see reactions on the list
>>> before moving ahead.
>>>
>>> diff --git a/numpy/core/blasdot/_dotblas.c b/numpy/core/blasdot/_dotblas.c
>>> index c73dd6a..2b4be7c 100644
>>> --- a/numpy/core/blasdot/_dotblas.c
>>> +++ b/numpy/core/blasdot/_dotblas.c
>>> @@ -770,7 +770,7 @@ dotblas_matrixproduct(PyObject *NPY_UNUSED(dummy),
>>> PyObject *args, PyObject* kwa
>>>           * using appropriate values of Order, Trans1, and Trans2.
>>>           */
>>>
>>> -        if (!PyArray_ISCONTIGUOUS(ap2)) {
>>> +        if (!PyArray_IS_C_CONTIGUOUS(ap2) &&
>>> !PyArray_IS_F_CONTIGUOUS(ap2)) {
>>>              PyObject *new = PyArray_Copy(ap2);
>>>
>>>              Py_DECREF(ap2);
>>> @@ -779,7 +779,7 @@ dotblas_matrixproduct(PyObject *NPY_UNUSED(dummy),
>>> PyObject *args, PyObject* kwa
>>>                  goto fail;
>>>              }
>>>          }
>>> -        if (!PyArray_ISCONTIGUOUS(ap1)) {
>>> +        if (!PyArray_IS_C_CONTIGUOUS(ap1) &&
>>> !PyArray_IS_F_CONTIGUOUS(ap1)) {
>>>              PyObject *new = PyArray_Copy(ap1);
>>>
>>>              Py_DECREF(ap1);
>>> @@ -800,6 +800,19 @@ dotblas_matrixproduct(PyObject
>>> *NPY_UNUSED(dummy), PyObject *args, PyObject* kwa
>>>          lda = (PyArray_DIM(ap1, 1) > 1 ? PyArray_DIM(ap1, 1) : 1);
>>>          ldb = (PyArray_DIM(ap2, 1) > 1 ? PyArray_DIM(ap2, 1) : 1);
>>>          ldc = (PyArray_DIM(ret, 1) > 1 ? PyArray_DIM(ret, 1) : 1);
>>> +
>>> +        /*
>>> +         * Avoid temporary copies for arrays in Fortran order
>>> +         */
>>> +        if (PyArray_IS_F_CONTIGUOUS(ap1)) {
>>> +            Trans1 = CblasTrans;
>>> +            lda = (PyArray_DIM(ap1, 0) > 1 ? PyArray_DIM(ap1, 0) : 1);
>>> +        }
>>> +        if (PyArray_IS_F_CONTIGUOUS(ap2)) {
>>> +            Trans2 = CblasTrans;
>>> +            ldb = (PyArray_DIM(ap2, 0) > 1 ? PyArray_DIM(ap2, 0) : 1);
>>> +        }
>>> +
>>>          if (typenum == NPY_DOUBLE) {
>>>              cblas_dgemm(Order, Trans1, Trans2,
>>>                          L, N, M,
>>>
>>> On Thu, Nov 8, 2012 at 3:58 PM, Sebastian Berg
>>> <sebastian at sipsolutions.net> wrote:
>>> > On Fri, 2012-11-09 at 00:24 +0100, Sebastian Berg wrote:
>>> >> Hey,
>>> >>
>>> >> On Thu, 2012-11-08 at 14:44 -0800, Nicolas SCHEFFER wrote:
>>> >> > Well, hinted by what Fabien said, I looked at the C level dot
>>> >> > function.
>>> >> > Quite verbose!
>>> >> >
>>> >> > But starting line 757, we can see that it shouldn't be too much work
>>> >> > to fix that bug (well there is even a comment there that states just
>>> >> > that)
>>> >> >
>>> >> > https://github.com/numpy/numpy/blob/master/numpy/core/blasdot/_dotblas.c#L757
>>> >> > I now think that should be the cleanest.
>>> >> >
>>> >> > This would only work for gemm though. I don't know what the benefit
>>> >> > is
>>> >> > for gemv for instance, but we should make that kind of changes
>>> >> > everywhere we can.
>>> >> > The evil PyArray_Copy is there twice and that's what we want to get
>>> >> > rid of.
>>> >> >
>>> >> > I'm not sure, but it looks to me that removing the copy and doing the
>>> >> > following would do the work:
>>> >> > Order = CblasRowMajor;
>>> >> > Trans1 = CblasNoTrans;
>>> >> > Trans2 = CblasNoTrans;
>>> >> > if (!PyArray_ISCONTIGUOUS(ap1)) {
>>> >> >     Trans1 = CblasTrans;
>>> >> > }
>>> >> > if (!PyArray_ISCONTIGUOUS(ap2)) {
>>> >> >     Trans2 = CblasTrans;
>>> >> > }
>>> >> > might be too easy to be true.
>>> >> >
>>> >>
>>> >> Sounds nice, though don't forget that the array may also be neither C-
>>> >> or F-Contiguous, in which case you need a copy in any case. So it would
>>> >> probably be more like:
>>> >>
>>> >> if (PyArray_IS_C_CONTIGUOUS(ap1)) {
>>> >>     Trans1 = CblasNoTrans;
>>> >> }
>>> >> else if (PyArray_IS_F_CONTIGUOUS(ap1)) {
>>> >>     Trans1 = CblasTrans;
>>> >> }
>>> >> else {
>>> >>     Trans1 = CblasNoTrans;
>>> >>     PyObject *new = PyArray_Copy(ap1);
>>> >>     Py_DECREF(ap1);
>>> >>     ap1 = (PyArrayObject *)new;
>>> >> }
>>> >>
>>> >
>>> > Well, of course I forgot error checking there, and maybe you need to set
>>> > some of the other parameters differently, but it looks like its probably
>>> > that easy, and I am sure everyone will welcome a PR with such changes.
>>> >
>>> >> Regards,
>>> >>
>>> >> Sebastian
>>> >>
>>> >> >
>>> >> >
>>> >> > On Thu, Nov 8, 2012 at 12:06 PM, Nicolas SCHEFFER
>>> >> > <scheffer.nicolas at gmail.com> wrote:
>>> >> > > I've made the necessary changes to get the proper order for the
>>> >> > > output array.
>>> >> > > Also, a pass of pep8 and some tests (fixmes are in failing tests)
>>> >> > > http://pastebin.com/M8TfbURi
>>> >> > >
>>> >> > > -n
>>> >> > >
>>> >> > > On Thu, Nov 8, 2012 at 11:38 AM, Nicolas SCHEFFER
>>> >> > > <scheffer.nicolas at gmail.com> wrote:
>>> >> > >> Thanks for all the responses folks. This is indeed a nice problem
>>> >> > >> to solve.
>>> >> > >>
>>> >> > >> Few points:
>>> >> > >> I. Change the order from 'F' to 'C': I'll look into it.
>>> >> > >> II. Integration with scipy / numpy: opinions are diverging here.
>>> >> > >> Let's wait a bit to get more responses on what people think.
>>> >> > >> One thing though: I'd need the same functionality as
>>> >> > >> get_blas_funcs in numpy.
>>> >> > >> Since numpy does not require lapack, what functions can I get?
>>> >> > >> III. Complex arrays
>>> >> > >> I unfortunately don't have enough knowledge here. If someone could
>>> >> > >> propose a fix, that'd be great.
>>> >> > >> IV. C
>>> >> > >> Writing this in C sounds like a good idea. I'm not sure I'd be the
>>> >> > >> right person to this though.
>>> >> > >> V. Patch in numpy
>>> >> > >> I'd love to do that and learn to do it as a byproduct.
>>> >> > >> Let's make sure we agree this can go in numpy first and that all
>>> >> > >> FIXME
>>> >> > >> can be fixed.
>>> >> > >> Although I guess we can resolve fixmes using git.
>>> >> > >>
>>> >> > >> Let me know how you'd like to proceed,
>>> >> > >>
>>> >> > >> Thanks!
>>> >> > >>
>>> >> > >> FIXMEs:
>>> >> > >> - Fix for ndim != 2
>>> >> > >> - Fix for dtype == np.complex*
>>> >> > >> - Fix order of output array
>>> >> > >>
>>> >> > >> On Thu, Nov 8, 2012 at 9:42 AM, Fr?d?ric Bastien <nouiz at nouiz.org>
>>> >> > >> wrote:
>>> >> > >>> Hi,
>>> >> > >>>
>>> >> > >>> I also think it should go into numpy.dot and that the output
>>> >> > >>> order should
>>> >> > >>> not be changed.
>>> >> > >>>
>>> >> > >>> A new point, what about the additional overhead for small
>>> >> > >>> ndarray? To remove
>>> >> > >>> this, I would suggest to put this code into the C function that
>>> >> > >>> do the
>>> >> > >>> actual work (at least, from memory it is a c function, not a
>>> >> > >>> python one).
>>> >> > >>>
>>> >> > >>> HTH
>>> >> > >>>
>>> >> > >>> Fred
>>> >> > >>>
>>> >> > >>>
>>> >> > >>>
>>> >> > >>> On Thu, Nov 8, 2012 at 12:29 PM, Anthony Scopatz
>>> >> > >>> <scopatz at gmail.com> wrote:
>>> >> > >>>>
>>> >> > >>>> On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau
>>> >> > >>>> <cournape at gmail.com>
>>> >> > >>>> wrote:
>>> >> > >>>>>
>>> >> > >>>>> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
>>> >> > >>>>> <d.s.seljebotn at astro.uio.no> wrote:
>>> >> > >>>>> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
>>> >> > >>>>> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith
>>> >> > >>>>> >> wrote:
>>> >> > >>>>> >>> I think everyone would be very happy to see numpy.dot
>>> >> > >>>>> >>> modified to do
>>> >> > >>>>> >>> this automatically. But adding a scipy.dot IMHO would be
>>> >> > >>>>> >>> fixing
>>> >> > >>>>> >>> things
>>> >> > >>>>> >>> in the wrong place and just create extra confusion.
>>> >> > >>>>> >>
>>> >> > >>>>> >> I am not sure I agree: numpy is often compiled without
>>> >> > >>>>> >> lapack support,
>>> >> > >>>>> >> as
>>> >> > >>>>> >> it is not necessary. On the other hand scipy is always
>>> >> > >>>>> >> compiled with
>>> >> > >>>>> >> lapack. Thus this makes more sens in scipy.
>>> >> > >>>>> >
>>> >> > >>>>> > Well, numpy.dot already contains multiple fallback cases for
>>> >> > >>>>> > when it is
>>> >> > >>>>> > compiled with BLAS and not. So I'm +1 on just making this an
>>> >> > >>>>> > improvement
>>> >> > >>>>> > on numpy.dot. I don't think there's a time when you would not
>>> >> > >>>>> > want to
>>> >> > >>>>> > use this (provided the output order issue is fixed), and it
>>> >> > >>>>> > doesn't
>>> >> > >>>>> > make
>>> >> > >>>>> > sense to not have old codes take advantage of the speed
>>> >> > >>>>> > improvement.
>>> >> > >>>>>
>>> >> > >>>>> Indeed, there is no reason not to make this available in NumPy.
>>> >> > >>>>>
>>> >> > >>>>> Nicolas, can you prepare a patch for numpy ?
>>> >> > >>>>
>>> >> > >>>>
>>> >> > >>>> +1, I agree, this should be a fix in numpy, not scipy.
>>> >> > >>>>
>>> >> > >>>> Be Well
>>> >> > >>>> Anthony
>>> >> > >>>>
>>> >> > >>>>>
>>> >> > >>>>>
>>> >> > >>>>> David
>>> >> > >>>>> _______________________________________________
>>> >> > >>>>> NumPy-Discussion mailing list
>>> >> > >>>>> NumPy-Discussion at scipy.org
>>> >> > >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >> > >>>>
>>> >> > >>>>
>>> >> > >>>>
>>> >> > >>>> _______________________________________________
>>> >> > >>>> NumPy-Discussion mailing list
>>> >> > >>>> NumPy-Discussion at scipy.org
>>> >> > >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >> > >>>>
>>> >> > >>>
>>> >> > >>>
>>> >> > >>> _______________________________________________
>>> >> > >>> NumPy-Discussion mailing list
>>> >> > >>> NumPy-Discussion at scipy.org
>>> >> > >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >> > >>>
>>> >> > _______________________________________________
>>> >> > NumPy-Discussion mailing list
>>> >> > NumPy-Discussion at scipy.org
>>> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >> >
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> NumPy-Discussion mailing list
>>> >> NumPy-Discussion at scipy.org
>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >
>>> >
>>> > _______________________________________________
>>> > NumPy-Discussion mailing list
>>> > NumPy-Discussion at scipy.org
>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From gael.varoquaux at normalesup.org  Fri Nov  9 03:48:26 2012
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 9 Nov 2012 09:48:26 +0100
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
Message-ID: <20121109084826.GB19371@phare.normalesup.org>

On Thu, Nov 08, 2012 at 11:29:19AM -0600, Anthony Scopatz wrote:
>     Indeed, there is no reason not to make this available in NumPy.

> +1, I agree, this should be a fix in numpy, not scipy.

I agree.

My point was a bit of a side issue: given a user's computer, I have no
garantee that numpy will be compiled with an optimized blas. In fact, I
have often seen on computing clusters where people tend to compile stuff
manually numpy using the embedded lapack_lite. On the other hand, scipy
is necessarily compiled using a blas. This is why, as a developer, I
always write code that uses scipy.linalg rather than numpy.linalg.
However, for 'dot', I have no way of ensuring that the blas used to
compile scipy is used.

Cheers,

Ga?l


From gnurser at gmail.com  Fri Nov  9 04:32:53 2012
From: gnurser at gmail.com (George Nurser)
Date: Fri, 9 Nov 2012 09:32:53 +0000
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHD5bXkHZxcL59boEyWrhS+TKZmJEoew0mZRNyvN=jczUO2Q1g@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
	<CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
	<CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>
	<1352417083.30611.4.camel@sebastian-laptop>
	<1352419109.30611.10.camel@sebastian-laptop>
	<CAHD5bXkym9FQxyKxeEgWMQWB8H8iH5p=dA_ieSyrVn14sy84kg@mail.gmail.com>
	<CADKKbtiE0h8YLBtw764uKf0udOdOWav1qoDZ9wm1MtCd8-_cUw@mail.gmail.com>
	<CAHD5bXkHZxcL59boEyWrhS+TKZmJEoew0mZRNyvN=jczUO2Q1g@mail.gmail.com>
Message-ID: <CAAyCRnQ+VpjBuxYxY1s0j0=AJXWC00dZoST_TwUHu5xOxM9F7A@mail.gmail.com>

Hi,
It's really good to see this work being done.

It would be great if this could somehow be put also into the np.einsum
function, which currently doesn't even use blas, and is consequently
substantially slower than current np.dot (timings on
http://mail.scipy.org/pipermail/numpy-discussion/2012-October/064259.html ).

I really do think that the Einstein summation convention is a really clear
way of doing array operations, and its use should be encouraged.

--George


On 9 November 2012 06:18, Nicolas SCHEFFER <scheffer.nicolas at gmail.com>wrote:

> Fred,
>
> Thanks for the advice.
> The code will only affect the part in _dotblas.c where gemm is called.
> There's tons of check before that make sure both matrices are of ndim 2.
> We should check though if we can do these tricks in other parts of the
> function.
>
> Otherwise:
> - I've built against ATLAS 3.10
> - I'm happy to add a couple more test for C and F-contiguous. I'm not
> sure how to get the third type (strided), would you have an example?
>
> The following test for instance checks integrity against
> multiarray.dot, which I believe is default when not compiled with
> BLAS.
> Dot is a hard function to test imho, so if anybody has ideas on what
> kind of test they'd like to see, please let me know.
>
> If that's ok I might now be able to:
> - Check for more bugs, I need to dig a bit more in the gemm call, make
> sure everything is ok.
> - Create an issue on github and link to this discussion
> - Make a commit in a seperate branch
> - Move forward like that.
>
> ==
> import numpy as np
> from time import time
> from numpy.testing import assert_almost_equal
>
> def test_dot_regression():
>     """ Test numpy dot by comparing with multiarray dot
>     """
>     np.random.seed(7)
>     a = np.random.randn(3, 3)
>     b = np.random.randn(3, 2)
>     c = np.random.randn(2, 3)
>
>     _dot = np.core.multiarray.dot
>
>     assert_almost_equal(np.dot(a, a), _dot(a, a))
>     assert_almost_equal(np.dot(b, c), _dot(b, c))
>     assert_almost_equal(np.dot(b.T, c.T), _dot(b.T, c.T))
>
>     assert_almost_equal(np.dot(a.T, a), _dot(a.T, a))
>     assert_almost_equal(np.dot(a, a.T), _dot(a, a.T))
>     assert_almost_equal(np.dot(a.T, a.T), _dot(a.T, a.T))
>
> On Thu, Nov 8, 2012 at 5:34 PM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
> > Hi,
> >
> > I suspect the current tests are not enought. You need to test all the
> > combination for the 3 inputs with thoses strides:
> >
> > c-contiguous
> > f-contiguous
> > something else like strided.
> >
> > Also, try with matrix with shape of 1 in each dimensions. Not all blas
> > libraries accept the strides that numpy use in that cases. Also, not all
> > blas version accept the same stuff, so if this isn't in the current
> version,
> > there will be probably some adjustment later on that side. What blas do
> you
> > use? I think ATLAS was one that was causing problem.
> >
> >
> > When we did this in Theano, it was more complicated then this diff... But
> > much of the code is boillerplate code.
> >
> > Fred
> >
> >
> >
> > On Thu, Nov 8, 2012 at 8:03 PM, Nicolas SCHEFFER
> > <scheffer.nicolas at gmail.com> wrote:
> >>
> >> Thanks Sebastien, didn't think of that.
> >>
> >> Well I went ahead and tried the change, and it's indeed straightforward.
> >>
> >> I've run some tests, among which:
> >> nosetests numpy/numpy/core/tests/test_blasdot.py
> >> and it looks ok. I'm assuming this is good news.
> >>
> >> I've copy-pasting the diff below, but I have that in my branch and can
> >> create a PR if we agree on it.
> >> I still cannot believe it's that easy (well this has been bugging me a
> >> while... ;))
> >> So I wouldn't mind waiting a day or two to see reactions on the list
> >> before moving ahead.
> >>
> >> diff --git a/numpy/core/blasdot/_dotblas.c
> b/numpy/core/blasdot/_dotblas.c
> >> index c73dd6a..2b4be7c 100644
> >> --- a/numpy/core/blasdot/_dotblas.c
> >> +++ b/numpy/core/blasdot/_dotblas.c
> >> @@ -770,7 +770,7 @@ dotblas_matrixproduct(PyObject *NPY_UNUSED(dummy),
> >> PyObject *args, PyObject* kwa
> >>           * using appropriate values of Order, Trans1, and Trans2.
> >>           */
> >>
> >> -        if (!PyArray_ISCONTIGUOUS(ap2)) {
> >> +        if (!PyArray_IS_C_CONTIGUOUS(ap2) &&
> >> !PyArray_IS_F_CONTIGUOUS(ap2)) {
> >>              PyObject *new = PyArray_Copy(ap2);
> >>
> >>              Py_DECREF(ap2);
> >> @@ -779,7 +779,7 @@ dotblas_matrixproduct(PyObject *NPY_UNUSED(dummy),
> >> PyObject *args, PyObject* kwa
> >>                  goto fail;
> >>              }
> >>          }
> >> -        if (!PyArray_ISCONTIGUOUS(ap1)) {
> >> +        if (!PyArray_IS_C_CONTIGUOUS(ap1) &&
> >> !PyArray_IS_F_CONTIGUOUS(ap1)) {
> >>              PyObject *new = PyArray_Copy(ap1);
> >>
> >>              Py_DECREF(ap1);
> >> @@ -800,6 +800,19 @@ dotblas_matrixproduct(PyObject
> >> *NPY_UNUSED(dummy), PyObject *args, PyObject* kwa
> >>          lda = (PyArray_DIM(ap1, 1) > 1 ? PyArray_DIM(ap1, 1) : 1);
> >>          ldb = (PyArray_DIM(ap2, 1) > 1 ? PyArray_DIM(ap2, 1) : 1);
> >>          ldc = (PyArray_DIM(ret, 1) > 1 ? PyArray_DIM(ret, 1) : 1);
> >> +
> >> +        /*
> >> +         * Avoid temporary copies for arrays in Fortran order
> >> +         */
> >> +        if (PyArray_IS_F_CONTIGUOUS(ap1)) {
> >> +            Trans1 = CblasTrans;
> >> +            lda = (PyArray_DIM(ap1, 0) > 1 ? PyArray_DIM(ap1, 0) : 1);
> >> +        }
> >> +        if (PyArray_IS_F_CONTIGUOUS(ap2)) {
> >> +            Trans2 = CblasTrans;
> >> +            ldb = (PyArray_DIM(ap2, 0) > 1 ? PyArray_DIM(ap2, 0) : 1);
> >> +        }
> >> +
> >>          if (typenum == NPY_DOUBLE) {
> >>              cblas_dgemm(Order, Trans1, Trans2,
> >>                          L, N, M,
> >>
> >> On Thu, Nov 8, 2012 at 3:58 PM, Sebastian Berg
> >> <sebastian at sipsolutions.net> wrote:
> >> > On Fri, 2012-11-09 at 00:24 +0100, Sebastian Berg wrote:
> >> >> Hey,
> >> >>
> >> >> On Thu, 2012-11-08 at 14:44 -0800, Nicolas SCHEFFER wrote:
> >> >> > Well, hinted by what Fabien said, I looked at the C level dot
> >> >> > function.
> >> >> > Quite verbose!
> >> >> >
> >> >> > But starting line 757, we can see that it shouldn't be too much
> work
> >> >> > to fix that bug (well there is even a comment there that states
> just
> >> >> > that)
> >> >> >
> >> >> >
> https://github.com/numpy/numpy/blob/master/numpy/core/blasdot/_dotblas.c#L757
> >> >> > I now think that should be the cleanest.
> >> >> >
> >> >> > This would only work for gemm though. I don't know what the benefit
> >> >> > is
> >> >> > for gemv for instance, but we should make that kind of changes
> >> >> > everywhere we can.
> >> >> > The evil PyArray_Copy is there twice and that's what we want to get
> >> >> > rid of.
> >> >> >
> >> >> > I'm not sure, but it looks to me that removing the copy and doing
> the
> >> >> > following would do the work:
> >> >> > Order = CblasRowMajor;
> >> >> > Trans1 = CblasNoTrans;
> >> >> > Trans2 = CblasNoTrans;
> >> >> > if (!PyArray_ISCONTIGUOUS(ap1)) {
> >> >> >     Trans1 = CblasTrans;
> >> >> > }
> >> >> > if (!PyArray_ISCONTIGUOUS(ap2)) {
> >> >> >     Trans2 = CblasTrans;
> >> >> > }
> >> >> > might be too easy to be true.
> >> >> >
> >> >>
> >> >> Sounds nice, though don't forget that the array may also be neither
> C-
> >> >> or F-Contiguous, in which case you need a copy in any case. So it
> would
> >> >> probably be more like:
> >> >>
> >> >> if (PyArray_IS_C_CONTIGUOUS(ap1)) {
> >> >>     Trans1 = CblasNoTrans;
> >> >> }
> >> >> else if (PyArray_IS_F_CONTIGUOUS(ap1)) {
> >> >>     Trans1 = CblasTrans;
> >> >> }
> >> >> else {
> >> >>     Trans1 = CblasNoTrans;
> >> >>     PyObject *new = PyArray_Copy(ap1);
> >> >>     Py_DECREF(ap1);
> >> >>     ap1 = (PyArrayObject *)new;
> >> >> }
> >> >>
> >> >
> >> > Well, of course I forgot error checking there, and maybe you need to
> set
> >> > some of the other parameters differently, but it looks like its
> probably
> >> > that easy, and I am sure everyone will welcome a PR with such changes.
> >> >
> >> >> Regards,
> >> >>
> >> >> Sebastian
> >> >>
> >> >> >
> >> >> >
> >> >> > On Thu, Nov 8, 2012 at 12:06 PM, Nicolas SCHEFFER
> >> >> > <scheffer.nicolas at gmail.com> wrote:
> >> >> > > I've made the necessary changes to get the proper order for the
> >> >> > > output array.
> >> >> > > Also, a pass of pep8 and some tests (fixmes are in failing tests)
> >> >> > > http://pastebin.com/M8TfbURi
> >> >> > >
> >> >> > > -n
> >> >> > >
> >> >> > > On Thu, Nov 8, 2012 at 11:38 AM, Nicolas SCHEFFER
> >> >> > > <scheffer.nicolas at gmail.com> wrote:
> >> >> > >> Thanks for all the responses folks. This is indeed a nice
> problem
> >> >> > >> to solve.
> >> >> > >>
> >> >> > >> Few points:
> >> >> > >> I. Change the order from 'F' to 'C': I'll look into it.
> >> >> > >> II. Integration with scipy / numpy: opinions are diverging here.
> >> >> > >> Let's wait a bit to get more responses on what people think.
> >> >> > >> One thing though: I'd need the same functionality as
> >> >> > >> get_blas_funcs in numpy.
> >> >> > >> Since numpy does not require lapack, what functions can I get?
> >> >> > >> III. Complex arrays
> >> >> > >> I unfortunately don't have enough knowledge here. If someone
> could
> >> >> > >> propose a fix, that'd be great.
> >> >> > >> IV. C
> >> >> > >> Writing this in C sounds like a good idea. I'm not sure I'd be
> the
> >> >> > >> right person to this though.
> >> >> > >> V. Patch in numpy
> >> >> > >> I'd love to do that and learn to do it as a byproduct.
> >> >> > >> Let's make sure we agree this can go in numpy first and that all
> >> >> > >> FIXME
> >> >> > >> can be fixed.
> >> >> > >> Although I guess we can resolve fixmes using git.
> >> >> > >>
> >> >> > >> Let me know how you'd like to proceed,
> >> >> > >>
> >> >> > >> Thanks!
> >> >> > >>
> >> >> > >> FIXMEs:
> >> >> > >> - Fix for ndim != 2
> >> >> > >> - Fix for dtype == np.complex*
> >> >> > >> - Fix order of output array
> >> >> > >>
> >> >> > >> On Thu, Nov 8, 2012 at 9:42 AM, Fr?d?ric Bastien <
> nouiz at nouiz.org>
> >> >> > >> wrote:
> >> >> > >>> Hi,
> >> >> > >>>
> >> >> > >>> I also think it should go into numpy.dot and that the output
> >> >> > >>> order should
> >> >> > >>> not be changed.
> >> >> > >>>
> >> >> > >>> A new point, what about the additional overhead for small
> >> >> > >>> ndarray? To remove
> >> >> > >>> this, I would suggest to put this code into the C function that
> >> >> > >>> do the
> >> >> > >>> actual work (at least, from memory it is a c function, not a
> >> >> > >>> python one).
> >> >> > >>>
> >> >> > >>> HTH
> >> >> > >>>
> >> >> > >>> Fred
> >> >> > >>>
> >> >> > >>>
> >> >> > >>>
> >> >> > >>> On Thu, Nov 8, 2012 at 12:29 PM, Anthony Scopatz
> >> >> > >>> <scopatz at gmail.com> wrote:
> >> >> > >>>>
> >> >> > >>>> On Thu, Nov 8, 2012 at 7:06 AM, David Cournapeau
> >> >> > >>>> <cournape at gmail.com>
> >> >> > >>>> wrote:
> >> >> > >>>>>
> >> >> > >>>>> On Thu, Nov 8, 2012 at 12:12 PM, Dag Sverre Seljebotn
> >> >> > >>>>> <d.s.seljebotn at astro.uio.no> wrote:
> >> >> > >>>>> > On 11/08/2012 01:07 PM, Gael Varoquaux wrote:
> >> >> > >>>>> >> On Thu, Nov 08, 2012 at 11:28:21AM +0000, Nathaniel Smith
> >> >> > >>>>> >> wrote:
> >> >> > >>>>> >>> I think everyone would be very happy to see numpy.dot
> >> >> > >>>>> >>> modified to do
> >> >> > >>>>> >>> this automatically. But adding a scipy.dot IMHO would be
> >> >> > >>>>> >>> fixing
> >> >> > >>>>> >>> things
> >> >> > >>>>> >>> in the wrong place and just create extra confusion.
> >> >> > >>>>> >>
> >> >> > >>>>> >> I am not sure I agree: numpy is often compiled without
> >> >> > >>>>> >> lapack support,
> >> >> > >>>>> >> as
> >> >> > >>>>> >> it is not necessary. On the other hand scipy is always
> >> >> > >>>>> >> compiled with
> >> >> > >>>>> >> lapack. Thus this makes more sens in scipy.
> >> >> > >>>>> >
> >> >> > >>>>> > Well, numpy.dot already contains multiple fallback cases
> for
> >> >> > >>>>> > when it is
> >> >> > >>>>> > compiled with BLAS and not. So I'm +1 on just making this
> an
> >> >> > >>>>> > improvement
> >> >> > >>>>> > on numpy.dot. I don't think there's a time when you would
> not
> >> >> > >>>>> > want to
> >> >> > >>>>> > use this (provided the output order issue is fixed), and it
> >> >> > >>>>> > doesn't
> >> >> > >>>>> > make
> >> >> > >>>>> > sense to not have old codes take advantage of the speed
> >> >> > >>>>> > improvement.
> >> >> > >>>>>
> >> >> > >>>>> Indeed, there is no reason not to make this available in
> NumPy.
> >> >> > >>>>>
> >> >> > >>>>> Nicolas, can you prepare a patch for numpy ?
> >> >> > >>>>
> >> >> > >>>>
> >> >> > >>>> +1, I agree, this should be a fix in numpy, not scipy.
> >> >> > >>>>
> >> >> > >>>> Be Well
> >> >> > >>>> Anthony
> >> >> > >>>>
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>> David
> >> >> > >>>>> _______________________________________________
> >> >> > >>>>> NumPy-Discussion mailing list
> >> >> > >>>>> NumPy-Discussion at scipy.org
> >> >> > >>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >> >> > >>>>
> >> >> > >>>>
> >> >> > >>>>
> >> >> > >>>> _______________________________________________
> >> >> > >>>> NumPy-Discussion mailing list
> >> >> > >>>> NumPy-Discussion at scipy.org
> >> >> > >>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >> >> > >>>>
> >> >> > >>>
> >> >> > >>>
> >> >> > >>> _______________________________________________
> >> >> > >>> NumPy-Discussion mailing list
> >> >> > >>> NumPy-Discussion at scipy.org
> >> >> > >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >> >> > >>>
> >> >> > _______________________________________________
> >> >> > NumPy-Discussion mailing list
> >> >> > NumPy-Discussion at scipy.org
> >> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >> >> >
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> NumPy-Discussion mailing list
> >> >> NumPy-Discussion at scipy.org
> >> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >> >
> >> >
> >> > _______________________________________________
> >> > NumPy-Discussion mailing list
> >> > NumPy-Discussion at scipy.org
> >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121109/99ca829c/attachment.html>

From alan.isaac at gmail.com  Fri Nov  9 09:17:46 2012
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 09 Nov 2012 09:17:46 -0500
Subject: [Numpy-discussion] random.choice
Message-ID: <509D108A.1060405@gmail.com>

I just noticed that 1.7 is scheduled to add a random.choice function.
I wonder if the best structure has been chosen.  Specifically, it does
not provide for array flattening, and it does not provide for subarray
choice.

Back in 2006 (?) Robert Kern suggested something like the below
(forgive any mistranslation).  I think the included functionality
does belong in a choice function.  I do like the provision for
repeated sampling in the current proposal, however.

Cheers,
Alan Isaac

def choice(x, axis=None):
	"""Select an element or subarray uniformly randomly.
	If axis is None, then a single element is chosen from the entire array.
	Otherwise, a subarray is chosen from the given axis.
	"""
	x = np.asarray(x)
	if axis is None:
		length = np.multiply.reduce(x.shape)
		n = random.randint(length)
		return x.flat[n]
	else:
		n = random.randint(x.shape[axis])
		idx = map(slice, x.shape)
		idx[axis] = n
		return x[tuple(idx)]


From nouiz at nouiz.org  Fri Nov  9 09:45:35 2012
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Fri, 9 Nov 2012 09:45:35 -0500
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAPJVwBk5RHziKTFkH7gPTk4neafwrfM+vWTuLv-aWKq=vs0aFA@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
	<CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
	<CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>
	<1352417083.30611.4.camel@sebastian-laptop>
	<1352419109.30611.10.camel@sebastian-laptop>
	<CAHD5bXkym9FQxyKxeEgWMQWB8H8iH5p=dA_ieSyrVn14sy84kg@mail.gmail.com>
	<CADKKbtiE0h8YLBtw764uKf0udOdOWav1qoDZ9wm1MtCd8-_cUw@mail.gmail.com>
	<CAHD5bXkHZxcL59boEyWrhS+TKZmJEoew0mZRNyvN=jczUO2Q1g@mail.gmail.com>
	<CAPJVwBk5RHziKTFkH7gPTk4neafwrfM+vWTuLv-aWKq=vs0aFA@mail.gmail.com>
Message-ID: <CADKKbtiFifwbaJBxtEJkvEyVehZ56n3tSmfb_oagn4H5FWmBzA@mail.gmail.com>

On Fri, Nov 9, 2012 at 3:32 AM, Nathaniel Smith <njs at pobox.com> wrote:

> On Fri, Nov 9, 2012 at 6:18 AM, Nicolas SCHEFFER
> <scheffer.nicolas at gmail.com> wrote:
> > Fred,
> >
> > Thanks for the advice.
> > The code will only affect the part in _dotblas.c where gemm is called.
> > There's tons of check before that make sure both matrices are of ndim 2.
> > We should check though if we can do these tricks in other parts of the
> function.
> >
> > Otherwise:
> > - I've built against ATLAS 3.10
> > - I'm happy to add a couple more test for C and F-contiguous. I'm not
> > sure how to get the third type (strided), would you have an example?
>
> def with_memory_order(a, order):
>     assert order in ("C", "F", "discontig")
>     assert a.ndim == 2
>     if order in ("C", "F"):
>         return np.asarray(a, order=order)
>     else:
>         buf = np.empty((a.shape[0] * 2, a.shape[1] * 2), dtype=a.dtype)
>         buf[::2, ::2] = a
>         # This returns a view onto every other element of 'buf':
>         result = buf[::2, ::2]
>         assert not result.flags.c_contiguous and not
> result.flags.f_contiguous
>         return result
>
> > The following test for instance checks integrity against
> > multiarray.dot, which I believe is default when not compiled with
> > BLAS.
> > Dot is a hard function to test imho, so if anybody has ideas on what
> > kind of test they'd like to see, please let me know.
> >
> > If that's ok I might now be able to:
> > - Check for more bugs, I need to dig a bit more in the gemm call, make
> > sure everything is ok.
> > - Create an issue on github and link to this discussion
> > - Make a commit in a seperate branch
> > - Move forward like that.
> >
> > ==
> > import numpy as np
> > from time import time
> > from numpy.testing import assert_almost_equal
> >
> > def test_dot_regression():
> >     """ Test numpy dot by comparing with multiarray dot
> >     """
> >     np.random.seed(7)
> >     a = np.random.randn(3, 3)
> >     b = np.random.randn(3, 2)
> >     c = np.random.randn(2, 3)
> >
> >     _dot = np.core.multiarray.dot
> >
> >     assert_almost_equal(np.dot(a, a), _dot(a, a))
> >     assert_almost_equal(np.dot(b, c), _dot(b, c))
> >     assert_almost_equal(np.dot(b.T, c.T), _dot(b.T, c.T))
> >
> >     assert_almost_equal(np.dot(a.T, a), _dot(a.T, a))
> >     assert_almost_equal(np.dot(a, a.T), _dot(a, a.T))
> >     assert_almost_equal(np.dot(a.T, a.T), _dot(a.T, a.T))
>
> You should check that the result is C-contiguous in all cases too.
>
> for a_order in ("C", "F", "discontig"):
>     for b_order in ("C", "F", "discontig"):
>         this_a = with_memory_order(a, a_order)
>         this_b = with_memory_order(b, b_order)
>         result = np.dot(this_a, this_b)
>         assert_almost_equal(result, expected)
>         assert result.flags.c_contiguous
>
> You could also wrap the above in yet another loop to try a few
> different combinations of a and b matrices (perhaps after sticking the
> code into a utility function, like run_dot_tests(a, b, expected), so
> the indentation doesn't get out of hand ;-)). Then you can easily test
> some of the edge cases, like Nx1 matrices.
>

I agree that tests are needed the for Nx1 and variant cases. I saw blas
error being raised with some blas version.

You also need to test with the output provided, so there is 3 loops:

for a_order in ("C", "F", "discontig", "neg"):
    for b_order in ("C", "F", "discontig", "neg"):
        for c_order in ("C", "F", "discontig", "neg"):

I also added the stride type "neg", I'm not sure if it is needed, but that
is other corner cases. neg =>  result = buf[::-1, ::-1]

I just looked again at our code and there is another constrain: that the
strides are multiple of the elemsize. Theano do not support not aligned
array, but numpy does, so there is a need for test for this. You can make
an unaligned array like this:

dtype = "b1,f4"
a = numpy.empty(1e4, dtype=dtype)['f1']

I just saw the strides problems that affect only some. I think that the
best explaination is our code:

        /* create appropriate strides for malformed matrices that are
row or column
         * vectors, or empty matrices.
         * In that case, the value of the stride does not really matter, but
         * some versions of BLAS insist that:
         *  - they are not smaller than the number of elements in the array,
         *  - they are not 0.
         */
        sx_0 = (Nx[0] > 1) ? Sx[0]/type_size : (Nx[1] + 1);
        sx_1 = (Nx[1] > 1) ? Sx[1]/type_size : (Nx[0] + 1);
        sy_0 = (Ny[0] > 1) ? Sy[0]/type_size : (Ny[1] + 1);
        sy_1 = (Ny[1] > 1) ? Sy[1]/type_size : (Ny[0] + 1);
        sz_0 = (Nz[0] > 1) ? Sz[0]/type_size : (Nz[1] + 1);
        sz_1 = (Nz[1] > 1) ? Sz[1]/type_size : (Nz[0] + 1);


So this ask for test with empty matrices too.

HTH

Fred
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121109/94d5e6b2/attachment.html>

From njs at pobox.com  Fri Nov  9 10:12:42 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 9 Nov 2012 15:12:42 +0000
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <20121109084826.GB19371@phare.normalesup.org>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
Message-ID: <CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>

On 9 Nov 2012 08:48, "Gael Varoquaux" <gael.varoquaux at normalesup.org> wrote:
>
> On Thu, Nov 08, 2012 at 11:29:19AM -0600, Anthony Scopatz wrote:
> >     Indeed, there is no reason not to make this available in NumPy.
>
> > +1, I agree, this should be a fix in numpy, not scipy.
>
> I agree.
>
> My point was a bit of a side issue: given a user's computer, I have no
> garantee that numpy will be compiled with an optimized blas. In fact, I
> have often seen on computing clusters where people tend to compile stuff
> manually numpy using the embedded lapack_lite. On the other hand, scipy
> is necessarily compiled using a blas. This is why, as a developer, I
> always write code that uses scipy.linalg rather than numpy.linalg.
> However, for 'dot', I have no way of ensuring that the blas used to
> compile scipy is used.

But what if someone compiles numpy against an optimized blas (mkl,
say) and then compiles SciPy against the reference blas? What do you
do then!? ;-)

I mean, this would be dumb. But not really dumber than installing an
optimized blas for SciPy and then configuring numpy to *not* use it...

As a higher-level point, having a scipy.dot would be a serious
violation of There's Only One Way To Do It. I think that if people who
are writing code against numpy/scipy have to care about our
difficulties with build systems, then that's a bug in numpy/scipy and
we should do a better job rather than pushing the responsibility off
on our users to figure out the best combination of functions to use on
each end-user's machine.

(Honestly I feel that the existence of scipy.linalg in the first place
is sort of a low-level embarrassment that we should try to move away
from in the long run. At least as far as the public API goes, and for
all of the near-duplicate functions that are in both numpy and scipy,
like qr.)

-n

> Cheers,
>
> Ga?l
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From njs at pobox.com  Fri Nov  9 10:20:35 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 9 Nov 2012 15:20:35 +0000
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CADKKbtiFifwbaJBxtEJkvEyVehZ56n3tSmfb_oagn4H5FWmBzA@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
	<CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
	<CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>
	<1352417083.30611.4.camel@sebastian-laptop>
	<1352419109.30611.10.camel@sebastian-laptop>
	<CAHD5bXkym9FQxyKxeEgWMQWB8H8iH5p=dA_ieSyrVn14sy84kg@mail.gmail.com>
	<CADKKbtiE0h8YLBtw764uKf0udOdOWav1qoDZ9wm1MtCd8-_cUw@mail.gmail.com>
	<CAHD5bXkHZxcL59boEyWrhS+TKZmJEoew0mZRNyvN=jczUO2Q1g@mail.gmail.com>
	<CAPJVwBk5RHziKTFkH7gPTk4neafwrfM+vWTuLv-aWKq=vs0aFA@mail.gmail.com>
	<CADKKbtiFifwbaJBxtEJkvEyVehZ56n3tSmfb_oagn4H5FWmBzA@mail.gmail.com>
Message-ID: <CAPJVwBmc9wzHyOCQPs_Cf5_2K0XE_wq2rPEAL9tEXBbSW-H1Vw@mail.gmail.com>

On Fri, Nov 9, 2012 at 2:45 PM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
>
>
>
> On Fri, Nov 9, 2012 at 3:32 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Fri, Nov 9, 2012 at 6:18 AM, Nicolas SCHEFFER
>> <scheffer.nicolas at gmail.com> wrote:
>> > Fred,
>> >
>> > Thanks for the advice.
>> > The code will only affect the part in _dotblas.c where gemm is called.
>> > There's tons of check before that make sure both matrices are of ndim 2.
>> > We should check though if we can do these tricks in other parts of the
>> > function.
>> >
>> > Otherwise:
>> > - I've built against ATLAS 3.10
>> > - I'm happy to add a couple more test for C and F-contiguous. I'm not
>> > sure how to get the third type (strided), would you have an example?
>>
>> def with_memory_order(a, order):
>>     assert order in ("C", "F", "discontig")
>>     assert a.ndim == 2
>>     if order in ("C", "F"):
>>         return np.asarray(a, order=order)
>>     else:
>>         buf = np.empty((a.shape[0] * 2, a.shape[1] * 2), dtype=a.dtype)
>>         buf[::2, ::2] = a
>>         # This returns a view onto every other element of 'buf':
>>         result = buf[::2, ::2]
>>         assert not result.flags.c_contiguous and not
>> result.flags.f_contiguous
>>         return result
>>
>> > The following test for instance checks integrity against
>> > multiarray.dot, which I believe is default when not compiled with
>> > BLAS.
>> > Dot is a hard function to test imho, so if anybody has ideas on what
>> > kind of test they'd like to see, please let me know.
>> >
>> > If that's ok I might now be able to:
>> > - Check for more bugs, I need to dig a bit more in the gemm call, make
>> > sure everything is ok.
>> > - Create an issue on github and link to this discussion
>> > - Make a commit in a seperate branch
>> > - Move forward like that.
>> >
>> > ==
>> > import numpy as np
>> > from time import time
>> > from numpy.testing import assert_almost_equal
>> >
>> > def test_dot_regression():
>> >     """ Test numpy dot by comparing with multiarray dot
>> >     """
>> >     np.random.seed(7)
>> >     a = np.random.randn(3, 3)
>> >     b = np.random.randn(3, 2)
>> >     c = np.random.randn(2, 3)
>> >
>> >     _dot = np.core.multiarray.dot
>> >
>> >     assert_almost_equal(np.dot(a, a), _dot(a, a))
>> >     assert_almost_equal(np.dot(b, c), _dot(b, c))
>> >     assert_almost_equal(np.dot(b.T, c.T), _dot(b.T, c.T))
>> >
>> >     assert_almost_equal(np.dot(a.T, a), _dot(a.T, a))
>> >     assert_almost_equal(np.dot(a, a.T), _dot(a, a.T))
>> >     assert_almost_equal(np.dot(a.T, a.T), _dot(a.T, a.T))
>>
>> You should check that the result is C-contiguous in all cases too.
>>
>> for a_order in ("C", "F", "discontig"):
>>     for b_order in ("C", "F", "discontig"):
>>         this_a = with_memory_order(a, a_order)
>>         this_b = with_memory_order(b, b_order)
>>         result = np.dot(this_a, this_b)
>>         assert_almost_equal(result, expected)
>>         assert result.flags.c_contiguous
>>
>> You could also wrap the above in yet another loop to try a few
>> different combinations of a and b matrices (perhaps after sticking the
>> code into a utility function, like run_dot_tests(a, b, expected), so
>> the indentation doesn't get out of hand ;-)). Then you can easily test
>> some of the edge cases, like Nx1 matrices.
>
>
> I agree that tests are needed the for Nx1 and variant cases. I saw blas
> error being raised with some blas version.
>
> You also need to test with the output provided, so there is 3 loops:
>
> for a_order in ("C", "F", "discontig", "neg"):
>     for b_order in ("C", "F", "discontig", "neg"):
>         for c_order in ("C", "F", "discontig", "neg"):
>
> I also added the stride type "neg", I'm not sure if it is needed, but that
> is other corner cases. neg =>  result = buf[::-1, ::-1]
>
> I just looked again at our code and there is another constrain: that the
> strides are multiple of the elemsize. Theano do not support not aligned
> array, but numpy does, so there is a need for test for this. You can make an
> unaligned array like this:
>
> dtype = "b1,f4"
> a = numpy.empty(1e4, dtype=dtype)['f1']

I think you're mixing up two issues here... requiring that the strides
and the itemsize match is part of the requirement for C- and
F-contiguity, the example code you give here produces a non-contiguous
array, so it's already checked for by the function we're talking
about.

However, there can be contiguous arrays that are not aligned, like:

In [25]: a = np.empty(100, dtype="i1")[1:-3].view(np.float32)

In [26]: a.flags.c_contiguous
Out[26]: True

In [27]: a.flags.aligned
Out[27]: False

I suspect the np.dot code that Nicolas is looking at already checks
for the ALIGNED flag and makes a copy if necessary, but it would be
good to have an array like this in the tests to be sure.

-n

> I just saw the strides problems that affect only some. I think that the best
> explaination is our code:
>
>
>
>         /* create appropriate strides for malformed matrices that are row or
> column
>          * vectors, or empty matrices.
>
>          * In that case, the value of the stride does not really matter, but
>          * some versions of BLAS insist that:
>
>          *  - they are not smaller than the number of elements in the array,
>          *  - they are not 0.
>          */
>
>         sx_0 = (Nx[0] > 1) ? Sx[0]/type_size : (Nx[1] + 1);
>         sx_1 = (Nx[1] > 1) ? Sx[1]/type_size : (Nx[0] + 1);
>
>         sy_0 = (Ny[0] > 1) ? Sy[0]/type_size : (Ny[1] + 1);
>         sy_1 = (Ny[1] > 1) ? Sy[1]/type_size : (Ny[0] + 1);
>
>         sz_0 = (Nz[0] > 1) ? Sz[0]/type_size : (Nz[1] + 1);
>         sz_1 = (Nz[1] > 1) ? Sz[1]/type_size : (Nz[0] + 1);
>
>
> So this ask for test with empty matrices too.
>
> HTH
>
> Fred
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From nouiz at nouiz.org  Fri Nov  9 11:17:27 2012
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Fri, 9 Nov 2012 11:17:27 -0500
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAPJVwBmc9wzHyOCQPs_Cf5_2K0XE_wq2rPEAL9tEXBbSW-H1Vw@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<CADKKbtiZrM=UtFd5Nuh3f-A-Jk+ec4PZ+-1O+MyCHho2adQ--A@mail.gmail.com>
	<CAHD5bX=BSkmXwz=k1a4DYU2gjzarmEF_PnQ-TN36iSYbDhpUZA@mail.gmail.com>
	<CAHD5bXmVcCcLGusRYDQh=d9QdKd93=XQc+CPR3dazFW5z2rNzg@mail.gmail.com>
	<CAHD5bXko3=s5xTfduAmq-n60SyhJYSnbBG2o=nviDwTSm5OX9A@mail.gmail.com>
	<1352417083.30611.4.camel@sebastian-laptop>
	<1352419109.30611.10.camel@sebastian-laptop>
	<CAHD5bXkym9FQxyKxeEgWMQWB8H8iH5p=dA_ieSyrVn14sy84kg@mail.gmail.com>
	<CADKKbtiE0h8YLBtw764uKf0udOdOWav1qoDZ9wm1MtCd8-_cUw@mail.gmail.com>
	<CAHD5bXkHZxcL59boEyWrhS+TKZmJEoew0mZRNyvN=jczUO2Q1g@mail.gmail.com>
	<CAPJVwBk5RHziKTFkH7gPTk4neafwrfM+vWTuLv-aWKq=vs0aFA@mail.gmail.com>
	<CADKKbtiFifwbaJBxtEJkvEyVehZ56n3tSmfb_oagn4H5FWmBzA@mail.gmail.com>
	<CAPJVwBmc9wzHyOCQPs_Cf5_2K0XE_wq2rPEAL9tEXBbSW-H1Vw@mail.gmail.com>
Message-ID: <CADKKbtifmC44WBimf+iYXQgHUb-6dV1ynGPjH6_zZguskgMCxw@mail.gmail.com>

On Fri, Nov 9, 2012 at 10:20 AM, Nathaniel Smith <njs at pobox.com> wrote:

> On Fri, Nov 9, 2012 at 2:45 PM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:
> >
> >
> >
> > On Fri, Nov 9, 2012 at 3:32 AM, Nathaniel Smith <njs at pobox.com> wrote:
> >>
> >> On Fri, Nov 9, 2012 at 6:18 AM, Nicolas SCHEFFER
> >> <scheffer.nicolas at gmail.com> wrote:
> >> > Fred,
> >> >
> >> > Thanks for the advice.
> >> > The code will only affect the part in _dotblas.c where gemm is called.
> >> > There's tons of check before that make sure both matrices are of ndim
> 2.
> >> > We should check though if we can do these tricks in other parts of the
> >> > function.
> >> >
> >> > Otherwise:
> >> > - I've built against ATLAS 3.10
> >> > - I'm happy to add a couple more test for C and F-contiguous. I'm not
> >> > sure how to get the third type (strided), would you have an example?
> >>
> >> def with_memory_order(a, order):
> >>     assert order in ("C", "F", "discontig")
> >>     assert a.ndim == 2
> >>     if order in ("C", "F"):
> >>         return np.asarray(a, order=order)
> >>     else:
> >>         buf = np.empty((a.shape[0] * 2, a.shape[1] * 2), dtype=a.dtype)
> >>         buf[::2, ::2] = a
> >>         # This returns a view onto every other element of 'buf':
> >>         result = buf[::2, ::2]
> >>         assert not result.flags.c_contiguous and not
> >> result.flags.f_contiguous
> >>         return result
> >>
> >> > The following test for instance checks integrity against
> >> > multiarray.dot, which I believe is default when not compiled with
> >> > BLAS.
> >> > Dot is a hard function to test imho, so if anybody has ideas on what
> >> > kind of test they'd like to see, please let me know.
> >> >
> >> > If that's ok I might now be able to:
> >> > - Check for more bugs, I need to dig a bit more in the gemm call, make
> >> > sure everything is ok.
> >> > - Create an issue on github and link to this discussion
> >> > - Make a commit in a seperate branch
> >> > - Move forward like that.
> >> >
> >> > ==
> >> > import numpy as np
> >> > from time import time
> >> > from numpy.testing import assert_almost_equal
> >> >
> >> > def test_dot_regression():
> >> >     """ Test numpy dot by comparing with multiarray dot
> >> >     """
> >> >     np.random.seed(7)
> >> >     a = np.random.randn(3, 3)
> >> >     b = np.random.randn(3, 2)
> >> >     c = np.random.randn(2, 3)
> >> >
> >> >     _dot = np.core.multiarray.dot
> >> >
> >> >     assert_almost_equal(np.dot(a, a), _dot(a, a))
> >> >     assert_almost_equal(np.dot(b, c), _dot(b, c))
> >> >     assert_almost_equal(np.dot(b.T, c.T), _dot(b.T, c.T))
> >> >
> >> >     assert_almost_equal(np.dot(a.T, a), _dot(a.T, a))
> >> >     assert_almost_equal(np.dot(a, a.T), _dot(a, a.T))
> >> >     assert_almost_equal(np.dot(a.T, a.T), _dot(a.T, a.T))
> >>
> >> You should check that the result is C-contiguous in all cases too.
> >>
> >> for a_order in ("C", "F", "discontig"):
> >>     for b_order in ("C", "F", "discontig"):
> >>         this_a = with_memory_order(a, a_order)
> >>         this_b = with_memory_order(b, b_order)
> >>         result = np.dot(this_a, this_b)
> >>         assert_almost_equal(result, expected)
> >>         assert result.flags.c_contiguous
> >>
> >> You could also wrap the above in yet another loop to try a few
> >> different combinations of a and b matrices (perhaps after sticking the
> >> code into a utility function, like run_dot_tests(a, b, expected), so
> >> the indentation doesn't get out of hand ;-)). Then you can easily test
> >> some of the edge cases, like Nx1 matrices.
> >
> >
> > I agree that tests are needed the for Nx1 and variant cases. I saw blas
> > error being raised with some blas version.
> >
> > You also need to test with the output provided, so there is 3 loops:
> >
> > for a_order in ("C", "F", "discontig", "neg"):
> >     for b_order in ("C", "F", "discontig", "neg"):
> >         for c_order in ("C", "F", "discontig", "neg"):
> >
> > I also added the stride type "neg", I'm not sure if it is needed, but
> that
> > is other corner cases. neg =>  result = buf[::-1, ::-1]
> >
> > I just looked again at our code and there is another constrain: that the
> > strides are multiple of the elemsize. Theano do not support not aligned
> > array, but numpy does, so there is a need for test for this. You can
> make an
> > unaligned array like this:
> >
> > dtype = "b1,f4"
> > a = numpy.empty(1e4, dtype=dtype)['f1']
>
> I think you're mixing up two issues here... requiring that the strides
> and the itemsize match is part of the requirement for C- and
> F-contiguity, the example code you give here produces a non-contiguous
> array, so it's already checked for by the function we're talking
> about.
>
> However, there can be contiguous arrays that are not aligned, like:
>
> In [25]: a = np.empty(100, dtype="i1")[1:-3].view(np.float32)
>
> In [26]: a.flags.c_contiguous
> Out[26]: True
>
> In [27]: a.flags.aligned
> Out[27]: False
>
> I suspect the np.dot code that Nicolas is looking at already checks
> for the ALIGNED flag and makes a copy if necessary, but it would be
> good to have an array like this in the tests to be sure.
>

You are right, my test wasn't good and your example is good.

Nicolas, I hope I don't de-motivate you with all those details, this need
to be done. When dealing with blas lib, the devil is in the detail... and
the fact that not all lib accept the same inputs... We hit a few of them. I
just try to tell you our experience to make the implementation easier and
shorter to stabilize.

Fred
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121109/b23ea0cf/attachment.html>

From gael.varoquaux at normalesup.org  Fri Nov  9 11:25:50 2012
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 9 Nov 2012 17:25:50 +0100
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
Message-ID: <20121109162550.GQ19465@phare.normalesup.org>

On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith wrote:
> But what if someone compiles numpy against an optimized blas (mkl,
> say) and then compiles SciPy against the reference blas? What do you
> do then!? ;-)

This could happen. But the converse happens very often. What happens is
that users (eg on shared computing resource) ask for a scientific python
environment. The administrator than installs the package starting from
the most basic one, to the most advanced one, thus starting with numpy
that can very well build without any external blas. When he gets to scipy
he hits the problem that the build system does not detect properly the
blas, and he solves that problem.

Also, it used to be that on the major linux distributions, numpy would not
be build with an optimize lapack because numpy was in the 'base' set of
packages, but not lapack. On the contrary, scipy being in the 'contrib'
set, it could depend on lapack. I just checked, and this has been fixed
in the major distributions (Fedora, Debian, Ubuntu).

Now we can discuss with such problems should not happen, and put the
blame on the users/administrators, the fact is that they happen often. I
keep seeing environments in which np.linalg is unreasonnably slow.

Gael


From ralf.gommers at gmail.com  Fri Nov  9 11:44:57 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 9 Nov 2012 17:44:57 +0100
Subject: [Numpy-discussion] Help compiling numpy with new gcc
In-Reply-To: <CAH2VmmCa0tppLEdndODca++9=Fpv8Jk5rpKwjDtT7Cv9LVbzDw@mail.gmail.com>
References: <CAH2VmmCa0tppLEdndODca++9=Fpv8Jk5rpKwjDtT7Cv9LVbzDw@mail.gmail.com>
Message-ID: <CABL7CQg0GVA++HBPrs5eSqDy3cHUTbPehmNXt9Gr9-RWfdf_3w@mail.gmail.com>

On Wed, Nov 7, 2012 at 7:28 PM, Filipe Pires Alvarenga Fernandes <
ocefpaf at gmail.com> wrote:

> Hi I am trying to compile numpy with gcc 4.7.1 and I am having the
> following issue.
>
> "RuntimeError: Broken toolchain: cannot link a simple C program"
>

Do you have Python development headers installed? Usually a separate
package called python-dev on linux. If so, can you compile with an older
gcc?

Ralf


> I noticed that I need to pass the flag '-fno-use-linker-plugin' to be
> able to compile it.  However, even though I did pass it by exporting
> the CFLAGS, it does not work.  I guess that numpy do not use the
> CFLAGS for its internal extensions.  How can I pass that option to it?
>
>
> Error below:
> [   11s] compile options: '-Inumpy/core/src/private -Inumpy/core/src
> -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray
> -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include
> -I/usr/include/python2.7 -c'
> [   11s] gcc: _configtest.c
> [   11s] gcc -pthread _configtest.o -o _configtest
> [   11s] gcc: fatal error: -fuse-linker-plugin, but liblto_plugin.so not
> found
> [   11s] compilation terminated.
> [   11s] gcc: fatal error: -fuse-linker-plugin, but liblto_plugin.so not
> found
> [   11s] compilation terminated.
> [   11s] failure.
> [   11s] removing: _configtest.c _configtest.o
> [   11s] Traceback (most recent call last):
> [   11s]   File "setup.py", line 214, in <module>
> [   11s]     setup_package()
> [   11s]   File "setup.py", line 207, in setup_package
> [   11s]     configuration=configuration )
> [   11s]   File
> "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/core.py", line 186,
> in setup
> [   11s]     return old_setup(**new_attr)
> [   11s]   File "/usr/lib64/python2.7/distutils/core.py", line 152, in
> setup
> [   11s]     dist.run_commands()
> [   11s]   File "/usr/lib64/python2.7/distutils/dist.py", line 953, in
> run_commands
> [   11s]     self.run_command(cmd)
> [   11s]   File "/usr/lib64/python2.7/distutils/dist.py", line 972, in
> run_command
> [   11s]     cmd_obj.run()
> [   11s]   File
> "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build.py",
> line 37, in run
> [   11s]     old_build.run(self)
> [   11s]   File "/usr/lib64/python2.7/distutils/command/build.py",
> line 127, in run
> [   11s]     self.run_command(cmd_name)
> [   11s]   File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in
> run_command
> [   11s]     self.distribution.run_command(command)
> [   11s]   File "/usr/lib64/python2.7/distutils/dist.py", line 972, in
> run_command
> [   11s]     cmd_obj.run()
> [   11s]   File
> "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py",
> line 152, in run
> [   11s]     self.build_sources()
> [   11s]   File
> "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py",
> line 163, in build_sources
> [   11s]     self.build_library_sources(*libname_info)
> [   11s]   File
> "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py",
> line 298, in build_library_sources
> [   11s]     sources = self.generate_sources(sources, (lib_name,
> build_info))
> [   11s]   File
> "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py",
> line 385, in generate_sources
> [   11s]     source = func(extension, build_dir)
> [   11s]   File "numpy/core/setup.py", line 648, in get_mathlib_info
> [   11s]     raise RuntimeError("Broken toolchain: cannot link a
> simple C program")
> [   11s] RuntimeError: Broken toolchain: cannot link a simple C program
> [   11s] error: Bad exit status from /var/tmp/rpm-tmp.yO2SIE (%build)
> [   11s]
> [   11s]
> [   11s] RPM build errors:
> [   11s]     Bad exit status from /var/tmp/rpm-tmp.yO2SIE (%build)
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121109/9cea89cf/attachment.html>

From charlesr.harris at gmail.com  Fri Nov  9 11:47:34 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 9 Nov 2012 09:47:34 -0700
Subject: [Numpy-discussion] Help compiling numpy with new gcc
In-Reply-To: <CABL7CQg0GVA++HBPrs5eSqDy3cHUTbPehmNXt9Gr9-RWfdf_3w@mail.gmail.com>
References: <CAH2VmmCa0tppLEdndODca++9=Fpv8Jk5rpKwjDtT7Cv9LVbzDw@mail.gmail.com>
	<CABL7CQg0GVA++HBPrs5eSqDy3cHUTbPehmNXt9Gr9-RWfdf_3w@mail.gmail.com>
Message-ID: <CAB6mnx+T80hR9vCyDu05kKR0ZyBpGRHJZ-YDmwwB-0HpzUg-zA@mail.gmail.com>

On Fri, Nov 9, 2012 at 9:44 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:

>
>
>
> On Wed, Nov 7, 2012 at 7:28 PM, Filipe Pires Alvarenga Fernandes <
> ocefpaf at gmail.com> wrote:
>
>> Hi I am trying to compile numpy with gcc 4.7.1 and I am having the
>> following issue.
>>
>> "RuntimeError: Broken toolchain: cannot link a simple C program"
>>
>
> Do you have Python development headers installed? Usually a separate
> package called python-dev on linux. If so, can you compile with an older
> gcc?
>
>
I haven't had any problems with gcc 4.7.2.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121109/04fa52ad/attachment.html>

From njs at pobox.com  Fri Nov  9 11:56:02 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 9 Nov 2012 16:56:02 +0000
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <20121109162550.GQ19465@phare.normalesup.org>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
Message-ID: <CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>

On Fri, Nov 9, 2012 at 4:25 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith wrote:
>> But what if someone compiles numpy against an optimized blas (mkl,
>> say) and then compiles SciPy against the reference blas? What do you
>> do then!? ;-)
>
> This could happen. But the converse happens very often. What happens is
> that users (eg on shared computing resource) ask for a scientific python
> environment. The administrator than installs the package starting from
> the most basic one, to the most advanced one, thus starting with numpy
> that can very well build without any external blas. When he gets to scipy
> he hits the problem that the build system does not detect properly the
> blas, and he solves that problem.
>
> Also, it used to be that on the major linux distributions, numpy would not
> be build with an optimize lapack because numpy was in the 'base' set of
> packages, but not lapack. On the contrary, scipy being in the 'contrib'
> set, it could depend on lapack. I just checked, and this has been fixed
> in the major distributions (Fedora, Debian, Ubuntu).
>
> Now we can discuss with such problems should not happen, and put the
> blame on the users/administrators, the fact is that they happen often. I
> keep seeing environments in which np.linalg is unreasonnably slow.

If this is something that's been a problem for you, maybe we should
start another thread on things we could do to fix it directly? Improve
build instructions, advertise build systems that set up the whole
environment (and thus do the right thing), make numpy's setup.py
scream and yell if blas isn't available...?

-n


From njs at pobox.com  Fri Nov  9 12:21:13 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 9 Nov 2012 17:21:13 +0000
Subject: [Numpy-discussion] random.choice
In-Reply-To: <509D108A.1060405@gmail.com>
References: <509D108A.1060405@gmail.com>
Message-ID: <CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>

On Fri, Nov 9, 2012 at 2:17 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> I just noticed that 1.7 is scheduled to add a random.choice function.
> I wonder if the best structure has been chosen.  Specifically, it does
> not provide for array flattening, and it does not provide for subarray
> choice.

I think in terms of the function currently in numpy master:
  http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.random.choice.html

You write flattening as
  np.random.choice(a.ravel(), ...)
and subarray choice as
  np.take(a, np.random.choice(a.shape[ax], ...), axis=ax)
?

That said, since it (claims to) only work on 1-d arrays right now, we
could always add either or both of these features later without
breaking compatibility. So I don't think there's any urgent need to
fix this before releasing.

(If you're worried though then you might want to double-check that the
np.random.choice in 1.7 actually *does* give an error if the input
array is not 1-d.)

-n


From will at thearete.co.uk  Fri Nov  9 12:16:46 2012
From: will at thearete.co.uk (Will Furnass)
Date: Fri, 9 Nov 2012 17:16:46 +0000 (UTC)
Subject: [Numpy-discussion] Double-ended queues
References: <CAJJEUT+ym4GUCuZ6RSgoq2dMCa6w4gXdys8VE+EmibH6Hk_JGw@mail.gmail.com>
	<CAPJVwBnqNo63QGEt=uFzP7pCwySUwJD-M4sNmx4aXUpcOkSkqA@mail.gmail.com>
	<5061962F.5000809@molden.no>
	<CAPJVwBn_ce+qGG1JjbPCKgx39s+Vcq+xxv0+LquNH8p9Sg4aSQ@mail.gmail.com>
	<CAB6mnxLZgtyV9c_fu9Si1drq+=r3LLAocRkkfhDm=-T2V+sYUg@mail.gmail.com>
Message-ID: <k7jdpu$3kp$1@ger.gmane.org>

On Tue, 25 Sep 2012 07:23:54 -0600, Charles R Harris wrote:

> On Tue, Sep 25, 2012 at 6:50 AM, Nathaniel Smith <njs at pobox.com> wrote:
> 
>> On Tue, Sep 25, 2012 at 12:31 PM, Sturla Molden <sturla at molden.no>
>> wrote:
>> > On 25.09.2012 11:38, Nathaniel Smith wrote:
>> >
>> >> Implementing a ring buffer on top of ndarray would be pretty
>> >> straightforward and probably work better than a linked-list
>> >> implementation.
>> >
>> > Amazingly, many do not know that a ringbuffer is simply an array
>> > indexed modulus its length:
>> >
>> > foo = np.zeros(n)
>> > i = 0 while 1:
>> >    foo[i % n]  # access ringbuffer i += 1
>>
>> Good trick, but to be reliable I think you need to either be willing
>> for i to overflow into a long (arbitrary width) integer, or else make
>> sure that i is an unsigned integer and that n is 2**k where k <=
>> sizeof(i)? Just doing i %= n on each pass through the loop might be
>> less error-prone.
>>
>> > Also, instead of writing a linked list, consider collections.deque.
>> > A deque is by definition a double-ended queue. It is just waste of
>> > time to implement a deque (double-ended queue) and hope it will
>> > perform better than Python's standard lib collections.deque object.
>>
>> The original poster is using collections.deque now, but wants a version
>> that supports efficient vectorized operations.
>>
>>
> The C++ stdlib has an efficient deque object, but it moves through
> memory.
> Hmm, it wouldn't be easy to make that work with numpy arrays what with
> views and all.
> 
> Efficient circular lists are often implemented using powers of two so
> that modulo indexing can be done using a mask.

What is meant by 'moves through memory'?  Are the reasons for avoiding 
creating ndarray/ufunc-like interfaces to Cython-wrapped STL objects to 
do with the complexity of coding such interfaces or to efficiency of 
execution?

Given that I'm simply wanting to go from

for t in timestep_arr:
    for i in len(deque1):
        deque2[i] += some_stuff[t] * deque1[i] / some_value
possibly_push_front(deque1, deque2)
possibly_pop_back_(deque1, deque2)

to 

for t in timestep_arr:
    deque2 += some_stuff[t] * deque1 / some_value
possibly_push_front(deque1, deque2)
possibly_pop_back_(deque1, deque2)

would it make sense to use a cython wrapped STL deque class for deque1 
and deque2 but somehow also include in __add__, __div__ and __mul__ 
methods to provide ufunc-like capabilities?

Apologies if I'm missing something fairly obvious - I'm fairly new to 
Cython.

Regards,

Will Furnass



From scheffer.nicolas at gmail.com  Fri Nov  9 17:13:51 2012
From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER)
Date: Fri, 9 Nov 2012 14:13:51 -0800
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
Message-ID: <CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>

I too encourage users to use scipy.linalg for speed and robustness
(hence calling this scipy.dot), but it just brings so much confusion!
When using the scipy + numpy ecosystem, you'd almost want everything
be done with scipy so that you get the best implementation in all
cases: scipy.zeros(), scipy.array(), scipy.dot(), scipy.linalg.inv().

Anyway this is indeed for another thread, the confusion we'd like to
fix here is that users shouldn't have to understand the C/F contiguous
concepts to get the maximum speed for np.dot()

To summarize:
- The python snippet I posted is still valid and can speed up your
code if you can change all your dot() calls.
- The change in dotblas.c is a bit more problematic because it's very
core. I'm having issues right now to replicate the timings, I've got
better timing for a.dot(a.T) than for a.dot(a). There might be a bug.

It's a pain to test because I cannot do the test in a single python session.
I'm going to try to integrate most of your suggestions, I cannot
guarantee I'll have time to do them all though.

-nicolas
On Fri, Nov 9, 2012 at 8:56 AM, Nathaniel Smith <njs at pobox.com> wrote:
> On Fri, Nov 9, 2012 at 4:25 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
>> On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith wrote:
>>> But what if someone compiles numpy against an optimized blas (mkl,
>>> say) and then compiles SciPy against the reference blas? What do you
>>> do then!? ;-)
>>
>> This could happen. But the converse happens very often. What happens is
>> that users (eg on shared computing resource) ask for a scientific python
>> environment. The administrator than installs the package starting from
>> the most basic one, to the most advanced one, thus starting with numpy
>> that can very well build without any external blas. When he gets to scipy
>> he hits the problem that the build system does not detect properly the
>> blas, and he solves that problem.
>>
>> Also, it used to be that on the major linux distributions, numpy would not
>> be build with an optimize lapack because numpy was in the 'base' set of
>> packages, but not lapack. On the contrary, scipy being in the 'contrib'
>> set, it could depend on lapack. I just checked, and this has been fixed
>> in the major distributions (Fedora, Debian, Ubuntu).
>>
>> Now we can discuss with such problems should not happen, and put the
>> blame on the users/administrators, the fact is that they happen often. I
>> keep seeing environments in which np.linalg is unreasonnably slow.
>
> If this is something that's been a problem for you, maybe we should
> start another thread on things we could do to fix it directly? Improve
> build instructions, advertise build systems that set up the whole
> environment (and thus do the right thing), make numpy's setup.py
> scream and yell if blas isn't available...?
>
> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From scheffer.nicolas at gmail.com  Fri Nov  9 17:52:22 2012
From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER)
Date: Fri, 9 Nov 2012 14:52:22 -0800
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
	<CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
Message-ID: <CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>

Ok: comparing apples to apples. I'm clueless on my observations and
would need input from you guys.

Using ATLAS 3.10, numpy with and without my changes, I'm getting these
timings and comparisons.

#
#I. Generate matrices using regular dot:
#
big = np.array(np.random.randn(2000, 2000), 'f');
np.savez('out', big=big, none=big.dot(big), both=big.T.dot(big.T),
left=big.T.dot(big), right=big.dot(big.T))"

#
#II. Timings with regular dot
#
In [3]: %timeit np.dot(big, big)
10 loops, best of 3: 138 ms per loop

In [4]: %timeit np.dot(big, big.T)
10 loops, best of 3: 166 ms per loop

In [5]: %timeit np.dot(big.T, big.T)
10 loops, best of 3: 193 ms per loop

In [6]: %timeit np.dot(big.T, big)
10 loops, best of 3: 165 ms per loop

#
#III. I load these arrays and time again with the "fast" dot
#
In [21]: %timeit np.dot(big, big)
10 loops, best of 3: 138 ms per loop

In [22]: %timeit np.dot(big.T, big)
10 loops, best of 3: 104 ms per loop

In [23]: %timeit np.dot(big.T, big.T)
10 loops, best of 3: 138 ms per loop

In [24]: %timeit np.dot(big, big.T)
10 loops, best of 3: 102 ms per loop

1. A'.A': great!
2. A.A' becomes faster than A.A !?!

#
#IV. MSE on differences
#
In [25]: np.sqrt(((arr['none'] - none)**2).sum())
Out[25]: 0.0

In [26]: np.sqrt(((arr['both'] - both)**2).sum())
Out[26]: 0.0

In [27]: np.sqrt(((arr['left'] - left)**2).sum())
Out[27]: 0.015900515

In [28]: np.sqrt(((arr['right'] - right)**2).sum())
Out[28]: 0.015331409

#
# CCl
#
While the MSE are small, I'm wondering whether:
- It's a bug: it should be exactly the same
- It's a feature: BLAS is taking shortcuts when you have A.A'. The
difference is not significant. Quick: PR that asap!

I don't have enough expertise to answer that...

Thanks much!

-nicolas
On Fri, Nov 9, 2012 at 2:13 PM, Nicolas SCHEFFER
<scheffer.nicolas at gmail.com> wrote:
> I too encourage users to use scipy.linalg for speed and robustness
> (hence calling this scipy.dot), but it just brings so much confusion!
> When using the scipy + numpy ecosystem, you'd almost want everything
> be done with scipy so that you get the best implementation in all
> cases: scipy.zeros(), scipy.array(), scipy.dot(), scipy.linalg.inv().
>
> Anyway this is indeed for another thread, the confusion we'd like to
> fix here is that users shouldn't have to understand the C/F contiguous
> concepts to get the maximum speed for np.dot()
>
> To summarize:
> - The python snippet I posted is still valid and can speed up your
> code if you can change all your dot() calls.
> - The change in dotblas.c is a bit more problematic because it's very
> core. I'm having issues right now to replicate the timings, I've got
> better timing for a.dot(a.T) than for a.dot(a). There might be a bug.
>
> It's a pain to test because I cannot do the test in a single python session.
> I'm going to try to integrate most of your suggestions, I cannot
> guarantee I'll have time to do them all though.
>
> -nicolas
> On Fri, Nov 9, 2012 at 8:56 AM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Fri, Nov 9, 2012 at 4:25 PM, Gael Varoquaux
>> <gael.varoquaux at normalesup.org> wrote:
>>> On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith wrote:
>>>> But what if someone compiles numpy against an optimized blas (mkl,
>>>> say) and then compiles SciPy against the reference blas? What do you
>>>> do then!? ;-)
>>>
>>> This could happen. But the converse happens very often. What happens is
>>> that users (eg on shared computing resource) ask for a scientific python
>>> environment. The administrator than installs the package starting from
>>> the most basic one, to the most advanced one, thus starting with numpy
>>> that can very well build without any external blas. When he gets to scipy
>>> he hits the problem that the build system does not detect properly the
>>> blas, and he solves that problem.
>>>
>>> Also, it used to be that on the major linux distributions, numpy would not
>>> be build with an optimize lapack because numpy was in the 'base' set of
>>> packages, but not lapack. On the contrary, scipy being in the 'contrib'
>>> set, it could depend on lapack. I just checked, and this has been fixed
>>> in the major distributions (Fedora, Debian, Ubuntu).
>>>
>>> Now we can discuss with such problems should not happen, and put the
>>> blame on the users/administrators, the fact is that they happen often. I
>>> keep seeing environments in which np.linalg is unreasonnably slow.
>>
>> If this is something that's been a problem for you, maybe we should
>> start another thread on things we could do to fix it directly? Improve
>> build instructions, advertise build systems that set up the whole
>> environment (and thus do the right thing), make numpy's setup.py
>> scream and yell if blas isn't available...?
>>
>> -n
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From matthieu.brucher at gmail.com  Fri Nov  9 17:57:10 2012
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 9 Nov 2012 23:57:10 +0100
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
	<CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
	<CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
Message-ID: <CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>

Hi,

A.A slower than A.A' is not a surprise for me. The latter is far more cache
friendly that the former. Everything follows cache lines, so it is faster
than something that will use one element from each cache line. In fact it
is exactly what "proves" that the new version is correct.
Good job (if all the tests were made and still pass ;) )

Cheers,

Matthieu


2012/11/9 Nicolas SCHEFFER <scheffer.nicolas at gmail.com>

> Ok: comparing apples to apples. I'm clueless on my observations and
> would need input from you guys.
>
> Using ATLAS 3.10, numpy with and without my changes, I'm getting these
> timings and comparisons.
>
> #
> #I. Generate matrices using regular dot:
> #
> big = np.array(np.random.randn(2000, 2000), 'f');
> np.savez('out', big=big, none=big.dot(big), both=big.T.dot(big.T),
> left=big.T.dot(big), right=big.dot(big.T))"
>
> #
> #II. Timings with regular dot
> #
> In [3]: %timeit np.dot(big, big)
> 10 loops, best of 3: 138 ms per loop
>
> In [4]: %timeit np.dot(big, big.T)
> 10 loops, best of 3: 166 ms per loop
>
> In [5]: %timeit np.dot(big.T, big.T)
> 10 loops, best of 3: 193 ms per loop
>
> In [6]: %timeit np.dot(big.T, big)
> 10 loops, best of 3: 165 ms per loop
>
> #
> #III. I load these arrays and time again with the "fast" dot
> #
> In [21]: %timeit np.dot(big, big)
> 10 loops, best of 3: 138 ms per loop
>
> In [22]: %timeit np.dot(big.T, big)
> 10 loops, best of 3: 104 ms per loop
>
> In [23]: %timeit np.dot(big.T, big.T)
> 10 loops, best of 3: 138 ms per loop
>
> In [24]: %timeit np.dot(big, big.T)
> 10 loops, best of 3: 102 ms per loop
>
> 1. A'.A': great!
> 2. A.A' becomes faster than A.A !?!
>
> #
> #IV. MSE on differences
> #
> In [25]: np.sqrt(((arr['none'] - none)**2).sum())
> Out[25]: 0.0
>
> In [26]: np.sqrt(((arr['both'] - both)**2).sum())
> Out[26]: 0.0
>
> In [27]: np.sqrt(((arr['left'] - left)**2).sum())
> Out[27]: 0.015900515
>
> In [28]: np.sqrt(((arr['right'] - right)**2).sum())
> Out[28]: 0.015331409
>
> #
> # CCl
> #
> While the MSE are small, I'm wondering whether:
> - It's a bug: it should be exactly the same
> - It's a feature: BLAS is taking shortcuts when you have A.A'. The
> difference is not significant. Quick: PR that asap!
>
> I don't have enough expertise to answer that...
>
> Thanks much!
>
> -nicolas
> On Fri, Nov 9, 2012 at 2:13 PM, Nicolas SCHEFFER
> <scheffer.nicolas at gmail.com> wrote:
> > I too encourage users to use scipy.linalg for speed and robustness
> > (hence calling this scipy.dot), but it just brings so much confusion!
> > When using the scipy + numpy ecosystem, you'd almost want everything
> > be done with scipy so that you get the best implementation in all
> > cases: scipy.zeros(), scipy.array(), scipy.dot(), scipy.linalg.inv().
> >
> > Anyway this is indeed for another thread, the confusion we'd like to
> > fix here is that users shouldn't have to understand the C/F contiguous
> > concepts to get the maximum speed for np.dot()
> >
> > To summarize:
> > - The python snippet I posted is still valid and can speed up your
> > code if you can change all your dot() calls.
> > - The change in dotblas.c is a bit more problematic because it's very
> > core. I'm having issues right now to replicate the timings, I've got
> > better timing for a.dot(a.T) than for a.dot(a). There might be a bug.
> >
> > It's a pain to test because I cannot do the test in a single python
> session.
> > I'm going to try to integrate most of your suggestions, I cannot
> > guarantee I'll have time to do them all though.
> >
> > -nicolas
> > On Fri, Nov 9, 2012 at 8:56 AM, Nathaniel Smith <njs at pobox.com> wrote:
> >> On Fri, Nov 9, 2012 at 4:25 PM, Gael Varoquaux
> >> <gael.varoquaux at normalesup.org> wrote:
> >>> On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith wrote:
> >>>> But what if someone compiles numpy against an optimized blas (mkl,
> >>>> say) and then compiles SciPy against the reference blas? What do you
> >>>> do then!? ;-)
> >>>
> >>> This could happen. But the converse happens very often. What happens is
> >>> that users (eg on shared computing resource) ask for a scientific
> python
> >>> environment. The administrator than installs the package starting from
> >>> the most basic one, to the most advanced one, thus starting with numpy
> >>> that can very well build without any external blas. When he gets to
> scipy
> >>> he hits the problem that the build system does not detect properly the
> >>> blas, and he solves that problem.
> >>>
> >>> Also, it used to be that on the major linux distributions, numpy would
> not
> >>> be build with an optimize lapack because numpy was in the 'base' set of
> >>> packages, but not lapack. On the contrary, scipy being in the 'contrib'
> >>> set, it could depend on lapack. I just checked, and this has been fixed
> >>> in the major distributions (Fedora, Debian, Ubuntu).
> >>>
> >>> Now we can discuss with such problems should not happen, and put the
> >>> blame on the users/administrators, the fact is that they happen often.
> I
> >>> keep seeing environments in which np.linalg is unreasonnably slow.
> >>
> >> If this is something that's been a problem for you, maybe we should
> >> start another thread on things we could do to fix it directly? Improve
> >> build instructions, advertise build systems that set up the whole
> >> environment (and thus do the right thing), make numpy's setup.py
> >> scream and yell if blas isn't available...?
> >>
> >> -n
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121109/162caeba/attachment.html>

From matthieu.brucher at gmail.com  Fri Nov  9 17:58:59 2012
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 9 Nov 2012 23:58:59 +0100
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
	<CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
	<CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
	<CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>
Message-ID: <CAHCaCkJDongGikFK4O0OJx1AQy7rbj_QYjdJOpz63c+KqLircw@mail.gmail.com>

Oh, about the differences. If there is something like cache blocking inside
ATLAS (which would make sense), the MAD are not in exactly the same order
and this would lead to some differences. You need to compare the MSE/sum of
values squared to the machine precision.

Cheers,


2012/11/9 Matthieu Brucher <matthieu.brucher at gmail.com>

> Hi,
>
> A.A slower than A.A' is not a surprise for me. The latter is far more
> cache friendly that the former. Everything follows cache lines, so it is
> faster than something that will use one element from each cache line. In
> fact it is exactly what "proves" that the new version is correct.
> Good job (if all the tests were made and still pass ;) )
>
> Cheers,
>
> Matthieu
>
>
> 2012/11/9 Nicolas SCHEFFER <scheffer.nicolas at gmail.com>
>
>> Ok: comparing apples to apples. I'm clueless on my observations and
>> would need input from you guys.
>>
>> Using ATLAS 3.10, numpy with and without my changes, I'm getting these
>> timings and comparisons.
>>
>> #
>> #I. Generate matrices using regular dot:
>> #
>> big = np.array(np.random.randn(2000, 2000), 'f');
>> np.savez('out', big=big, none=big.dot(big), both=big.T.dot(big.T),
>> left=big.T.dot(big), right=big.dot(big.T))"
>>
>> #
>> #II. Timings with regular dot
>> #
>> In [3]: %timeit np.dot(big, big)
>> 10 loops, best of 3: 138 ms per loop
>>
>> In [4]: %timeit np.dot(big, big.T)
>> 10 loops, best of 3: 166 ms per loop
>>
>> In [5]: %timeit np.dot(big.T, big.T)
>> 10 loops, best of 3: 193 ms per loop
>>
>> In [6]: %timeit np.dot(big.T, big)
>> 10 loops, best of 3: 165 ms per loop
>>
>> #
>> #III. I load these arrays and time again with the "fast" dot
>> #
>> In [21]: %timeit np.dot(big, big)
>> 10 loops, best of 3: 138 ms per loop
>>
>> In [22]: %timeit np.dot(big.T, big)
>> 10 loops, best of 3: 104 ms per loop
>>
>> In [23]: %timeit np.dot(big.T, big.T)
>> 10 loops, best of 3: 138 ms per loop
>>
>> In [24]: %timeit np.dot(big, big.T)
>> 10 loops, best of 3: 102 ms per loop
>>
>> 1. A'.A': great!
>> 2. A.A' becomes faster than A.A !?!
>>
>> #
>> #IV. MSE on differences
>> #
>> In [25]: np.sqrt(((arr['none'] - none)**2).sum())
>> Out[25]: 0.0
>>
>> In [26]: np.sqrt(((arr['both'] - both)**2).sum())
>> Out[26]: 0.0
>>
>> In [27]: np.sqrt(((arr['left'] - left)**2).sum())
>> Out[27]: 0.015900515
>>
>> In [28]: np.sqrt(((arr['right'] - right)**2).sum())
>> Out[28]: 0.015331409
>>
>> #
>> # CCl
>> #
>> While the MSE are small, I'm wondering whether:
>> - It's a bug: it should be exactly the same
>> - It's a feature: BLAS is taking shortcuts when you have A.A'. The
>> difference is not significant. Quick: PR that asap!
>>
>> I don't have enough expertise to answer that...
>>
>> Thanks much!
>>
>> -nicolas
>> On Fri, Nov 9, 2012 at 2:13 PM, Nicolas SCHEFFER
>> <scheffer.nicolas at gmail.com> wrote:
>> > I too encourage users to use scipy.linalg for speed and robustness
>> > (hence calling this scipy.dot), but it just brings so much confusion!
>> > When using the scipy + numpy ecosystem, you'd almost want everything
>> > be done with scipy so that you get the best implementation in all
>> > cases: scipy.zeros(), scipy.array(), scipy.dot(), scipy.linalg.inv().
>> >
>> > Anyway this is indeed for another thread, the confusion we'd like to
>> > fix here is that users shouldn't have to understand the C/F contiguous
>> > concepts to get the maximum speed for np.dot()
>> >
>> > To summarize:
>> > - The python snippet I posted is still valid and can speed up your
>> > code if you can change all your dot() calls.
>> > - The change in dotblas.c is a bit more problematic because it's very
>> > core. I'm having issues right now to replicate the timings, I've got
>> > better timing for a.dot(a.T) than for a.dot(a). There might be a bug.
>> >
>> > It's a pain to test because I cannot do the test in a single python
>> session.
>> > I'm going to try to integrate most of your suggestions, I cannot
>> > guarantee I'll have time to do them all though.
>> >
>> > -nicolas
>> > On Fri, Nov 9, 2012 at 8:56 AM, Nathaniel Smith <njs at pobox.com> wrote:
>> >> On Fri, Nov 9, 2012 at 4:25 PM, Gael Varoquaux
>> >> <gael.varoquaux at normalesup.org> wrote:
>> >>> On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith wrote:
>> >>>> But what if someone compiles numpy against an optimized blas (mkl,
>> >>>> say) and then compiles SciPy against the reference blas? What do you
>> >>>> do then!? ;-)
>> >>>
>> >>> This could happen. But the converse happens very often. What happens
>> is
>> >>> that users (eg on shared computing resource) ask for a scientific
>> python
>> >>> environment. The administrator than installs the package starting from
>> >>> the most basic one, to the most advanced one, thus starting with numpy
>> >>> that can very well build without any external blas. When he gets to
>> scipy
>> >>> he hits the problem that the build system does not detect properly the
>> >>> blas, and he solves that problem.
>> >>>
>> >>> Also, it used to be that on the major linux distributions, numpy
>> would not
>> >>> be build with an optimize lapack because numpy was in the 'base' set
>> of
>> >>> packages, but not lapack. On the contrary, scipy being in the
>> 'contrib'
>> >>> set, it could depend on lapack. I just checked, and this has been
>> fixed
>> >>> in the major distributions (Fedora, Debian, Ubuntu).
>> >>>
>> >>> Now we can discuss with such problems should not happen, and put the
>> >>> blame on the users/administrators, the fact is that they happen
>> often. I
>> >>> keep seeing environments in which np.linalg is unreasonnably slow.
>> >>
>> >> If this is something that's been a problem for you, maybe we should
>> >> start another thread on things we could do to fix it directly? Improve
>> >> build instructions, advertise build systems that set up the whole
>> >> environment (and thus do the right thing), make numpy's setup.py
>> >> scream and yell if blas isn't available...?
>> >>
>> >> -n
>> >> _______________________________________________
>> >> NumPy-Discussion mailing list
>> >> NumPy-Discussion at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
> Information System Engineer, Ph.D.
> Blog: http://matt.eifelle.com
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> Music band: http://liliejay.com/
>
>


-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121109/9dc933a0/attachment.html>

From sebastian at sipsolutions.net  Fri Nov  9 18:02:01 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Sat, 10 Nov 2012 00:02:01 +0100
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
	<CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
	<CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
Message-ID: <1352502121.18462.8.camel@sebastian-laptop>

On Fri, 2012-11-09 at 14:52 -0800, Nicolas SCHEFFER wrote:
> Ok: comparing apples to apples. I'm clueless on my observations and
> would need input from you guys.
> 
> Using ATLAS 3.10, numpy with and without my changes, I'm getting these
> timings and comparisons.
> 
> #
> #I. Generate matrices using regular dot:
> #
> big = np.array(np.random.randn(2000, 2000), 'f');
> np.savez('out', big=big, none=big.dot(big), both=big.T.dot(big.T),
> left=big.T.dot(big), right=big.dot(big.T))"
> 
> #
> #II. Timings with regular dot
> #
> In [3]: %timeit np.dot(big, big)
> 10 loops, best of 3: 138 ms per loop
> 
> In [4]: %timeit np.dot(big, big.T)
> 10 loops, best of 3: 166 ms per loop
> 
> In [5]: %timeit np.dot(big.T, big.T)
> 10 loops, best of 3: 193 ms per loop
> 
> In [6]: %timeit np.dot(big.T, big)
> 10 loops, best of 3: 165 ms per loop
> 
> #
> #III. I load these arrays and time again with the "fast" dot
> #
> In [21]: %timeit np.dot(big, big)
> 10 loops, best of 3: 138 ms per loop
> 
> In [22]: %timeit np.dot(big.T, big)
> 10 loops, best of 3: 104 ms per loop
> 
> In [23]: %timeit np.dot(big.T, big.T)
> 10 loops, best of 3: 138 ms per loop
> 
> In [24]: %timeit np.dot(big, big.T)
> 10 loops, best of 3: 102 ms per loop
> 
> 1. A'.A': great!
> 2. A.A' becomes faster than A.A !?!
> 
> #
> #IV. MSE on differences
> #
> In [25]: np.sqrt(((arr['none'] - none)**2).sum())
> Out[25]: 0.0
> 
> In [26]: np.sqrt(((arr['both'] - both)**2).sum())
> Out[26]: 0.0
> 
> In [27]: np.sqrt(((arr['left'] - left)**2).sum())
> Out[27]: 0.015900515
> 
> In [28]: np.sqrt(((arr['right'] - right)**2).sum())
> Out[28]: 0.015331409
> 
> #
> # CCl
> #
> While the MSE are small, I'm wondering whether:
> - It's a bug: it should be exactly the same
> - It's a feature: BLAS is taking shortcuts when you have A.A'. The
> difference is not significant. Quick: PR that asap!
> 

I think its a feature because memory can be accessed in a more regular
fashion in the case of one array being Fortran and the other C
contiguous. IE. if its A.B' then fetching the first row is perfect,
since its all behind each other in memory (C-order), while it has to be
multiplied with the first column from B' which, due to being transposed,
is fortran order and the column thus also one chunk in memory.

> I don't have enough expertise to answer that...
> 
> Thanks much!
> 
> -nicolas
> On Fri, Nov 9, 2012 at 2:13 PM, Nicolas SCHEFFER
> <scheffer.nicolas at gmail.com> wrote:
> > I too encourage users to use scipy.linalg for speed and robustness
> > (hence calling this scipy.dot), but it just brings so much confusion!
> > When using the scipy + numpy ecosystem, you'd almost want everything
> > be done with scipy so that you get the best implementation in all
> > cases: scipy.zeros(), scipy.array(), scipy.dot(), scipy.linalg.inv().
> >
> > Anyway this is indeed for another thread, the confusion we'd like to
> > fix here is that users shouldn't have to understand the C/F contiguous
> > concepts to get the maximum speed for np.dot()
> >
> > To summarize:
> > - The python snippet I posted is still valid and can speed up your
> > code if you can change all your dot() calls.
> > - The change in dotblas.c is a bit more problematic because it's very
> > core. I'm having issues right now to replicate the timings, I've got
> > better timing for a.dot(a.T) than for a.dot(a). There might be a bug.
> >
> > It's a pain to test because I cannot do the test in a single python session.
> > I'm going to try to integrate most of your suggestions, I cannot
> > guarantee I'll have time to do them all though.
> >
> > -nicolas
> > On Fri, Nov 9, 2012 at 8:56 AM, Nathaniel Smith <njs at pobox.com> wrote:
> >> On Fri, Nov 9, 2012 at 4:25 PM, Gael Varoquaux
> >> <gael.varoquaux at normalesup.org> wrote:
> >>> On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith wrote:
> >>>> But what if someone compiles numpy against an optimized blas (mkl,
> >>>> say) and then compiles SciPy against the reference blas? What do you
> >>>> do then!? ;-)
> >>>
> >>> This could happen. But the converse happens very often. What happens is
> >>> that users (eg on shared computing resource) ask for a scientific python
> >>> environment. The administrator than installs the package starting from
> >>> the most basic one, to the most advanced one, thus starting with numpy
> >>> that can very well build without any external blas. When he gets to scipy
> >>> he hits the problem that the build system does not detect properly the
> >>> blas, and he solves that problem.
> >>>
> >>> Also, it used to be that on the major linux distributions, numpy would not
> >>> be build with an optimize lapack because numpy was in the 'base' set of
> >>> packages, but not lapack. On the contrary, scipy being in the 'contrib'
> >>> set, it could depend on lapack. I just checked, and this has been fixed
> >>> in the major distributions (Fedora, Debian, Ubuntu).
> >>>
> >>> Now we can discuss with such problems should not happen, and put the
> >>> blame on the users/administrators, the fact is that they happen often. I
> >>> keep seeing environments in which np.linalg is unreasonnably slow.
> >>
> >> If this is something that's been a problem for you, maybe we should
> >> start another thread on things we could do to fix it directly? Improve
> >> build instructions, advertise build systems that set up the whole
> >> environment (and thus do the right thing), make numpy's setup.py
> >> scream and yell if blas isn't available...?
> >>
> >> -n
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From njs at pobox.com  Fri Nov  9 18:05:34 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 9 Nov 2012 23:05:34 +0000
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
	<CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
	<CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
	<CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>
Message-ID: <CAPJVwB=fGtm2U-_Jj=WMrohOKUa==p538e-H3s95Nk9XJaneMQ@mail.gmail.com>

In this case it's even possible the blas is being smart enough to notice
that you've passed it the same pointer twice with the transpose switch on
one of them, and is just skipping half the multiplies since the output is
provably symmetric. Don't know if they actually do that but...

-n
On 9 Nov 2012 22:57, "Matthieu Brucher" <matthieu.brucher at gmail.com> wrote:

> Hi,
>
> A.A slower than A.A' is not a surprise for me. The latter is far more
> cache friendly that the former. Everything follows cache lines, so it is
> faster than something that will use one element from each cache line. In
> fact it is exactly what "proves" that the new version is correct.
> Good job (if all the tests were made and still pass ;) )
>
> Cheers,
>
> Matthieu
>
>
> 2012/11/9 Nicolas SCHEFFER <scheffer.nicolas at gmail.com>
>
>> Ok: comparing apples to apples. I'm clueless on my observations and
>> would need input from you guys.
>>
>> Using ATLAS 3.10, numpy with and without my changes, I'm getting these
>> timings and comparisons.
>>
>> #
>> #I. Generate matrices using regular dot:
>> #
>> big = np.array(np.random.randn(2000, 2000), 'f');
>> np.savez('out', big=big, none=big.dot(big), both=big.T.dot(big.T),
>> left=big.T.dot(big), right=big.dot(big.T))"
>>
>> #
>> #II. Timings with regular dot
>> #
>> In [3]: %timeit np.dot(big, big)
>> 10 loops, best of 3: 138 ms per loop
>>
>> In [4]: %timeit np.dot(big, big.T)
>> 10 loops, best of 3: 166 ms per loop
>>
>> In [5]: %timeit np.dot(big.T, big.T)
>> 10 loops, best of 3: 193 ms per loop
>>
>> In [6]: %timeit np.dot(big.T, big)
>> 10 loops, best of 3: 165 ms per loop
>>
>> #
>> #III. I load these arrays and time again with the "fast" dot
>> #
>> In [21]: %timeit np.dot(big, big)
>> 10 loops, best of 3: 138 ms per loop
>>
>> In [22]: %timeit np.dot(big.T, big)
>> 10 loops, best of 3: 104 ms per loop
>>
>> In [23]: %timeit np.dot(big.T, big.T)
>> 10 loops, best of 3: 138 ms per loop
>>
>> In [24]: %timeit np.dot(big, big.T)
>> 10 loops, best of 3: 102 ms per loop
>>
>> 1. A'.A': great!
>> 2. A.A' becomes faster than A.A !?!
>>
>> #
>> #IV. MSE on differences
>> #
>> In [25]: np.sqrt(((arr['none'] - none)**2).sum())
>> Out[25]: 0.0
>>
>> In [26]: np.sqrt(((arr['both'] - both)**2).sum())
>> Out[26]: 0.0
>>
>> In [27]: np.sqrt(((arr['left'] - left)**2).sum())
>> Out[27]: 0.015900515
>>
>> In [28]: np.sqrt(((arr['right'] - right)**2).sum())
>> Out[28]: 0.015331409
>>
>> #
>> # CCl
>> #
>> While the MSE are small, I'm wondering whether:
>> - It's a bug: it should be exactly the same
>> - It's a feature: BLAS is taking shortcuts when you have A.A'. The
>> difference is not significant. Quick: PR that asap!
>>
>> I don't have enough expertise to answer that...
>>
>> Thanks much!
>>
>> -nicolas
>> On Fri, Nov 9, 2012 at 2:13 PM, Nicolas SCHEFFER
>> <scheffer.nicolas at gmail.com> wrote:
>> > I too encourage users to use scipy.linalg for speed and robustness
>> > (hence calling this scipy.dot), but it just brings so much confusion!
>> > When using the scipy + numpy ecosystem, you'd almost want everything
>> > be done with scipy so that you get the best implementation in all
>> > cases: scipy.zeros(), scipy.array(), scipy.dot(), scipy.linalg.inv().
>> >
>> > Anyway this is indeed for another thread, the confusion we'd like to
>> > fix here is that users shouldn't have to understand the C/F contiguous
>> > concepts to get the maximum speed for np.dot()
>> >
>> > To summarize:
>> > - The python snippet I posted is still valid and can speed up your
>> > code if you can change all your dot() calls.
>> > - The change in dotblas.c is a bit more problematic because it's very
>> > core. I'm having issues right now to replicate the timings, I've got
>> > better timing for a.dot(a.T) than for a.dot(a). There might be a bug.
>> >
>> > It's a pain to test because I cannot do the test in a single python
>> session.
>> > I'm going to try to integrate most of your suggestions, I cannot
>> > guarantee I'll have time to do them all though.
>> >
>> > -nicolas
>> > On Fri, Nov 9, 2012 at 8:56 AM, Nathaniel Smith <njs at pobox.com> wrote:
>> >> On Fri, Nov 9, 2012 at 4:25 PM, Gael Varoquaux
>> >> <gael.varoquaux at normalesup.org> wrote:
>> >>> On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith wrote:
>> >>>> But what if someone compiles numpy against an optimized blas (mkl,
>> >>>> say) and then compiles SciPy against the reference blas? What do you
>> >>>> do then!? ;-)
>> >>>
>> >>> This could happen. But the converse happens very often. What happens
>> is
>> >>> that users (eg on shared computing resource) ask for a scientific
>> python
>> >>> environment. The administrator than installs the package starting from
>> >>> the most basic one, to the most advanced one, thus starting with numpy
>> >>> that can very well build without any external blas. When he gets to
>> scipy
>> >>> he hits the problem that the build system does not detect properly the
>> >>> blas, and he solves that problem.
>> >>>
>> >>> Also, it used to be that on the major linux distributions, numpy
>> would not
>> >>> be build with an optimize lapack because numpy was in the 'base' set
>> of
>> >>> packages, but not lapack. On the contrary, scipy being in the
>> 'contrib'
>> >>> set, it could depend on lapack. I just checked, and this has been
>> fixed
>> >>> in the major distributions (Fedora, Debian, Ubuntu).
>> >>>
>> >>> Now we can discuss with such problems should not happen, and put the
>> >>> blame on the users/administrators, the fact is that they happen
>> often. I
>> >>> keep seeing environments in which np.linalg is unreasonnably slow.
>> >>
>> >> If this is something that's been a problem for you, maybe we should
>> >> start another thread on things we could do to fix it directly? Improve
>> >> build instructions, advertise build systems that set up the whole
>> >> environment (and thus do the right thing), make numpy's setup.py
>> >> scream and yell if blas isn't available...?
>> >>
>> >> -n
>> >> _______________________________________________
>> >> NumPy-Discussion mailing list
>> >> NumPy-Discussion at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
> Information System Engineer, Ph.D.
> Blog: http://matt.eifelle.com
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> Music band: http://liliejay.com/
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121109/5acc8f84/attachment.html>

From scheffer.nicolas at gmail.com  Fri Nov  9 19:02:51 2012
From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER)
Date: Fri, 9 Nov 2012 16:02:51 -0800
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAPJVwB=fGtm2U-_Jj=WMrohOKUa==p538e-H3s95Nk9XJaneMQ@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
	<CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
	<CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
	<CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>
	<CAPJVwB=fGtm2U-_Jj=WMrohOKUa==p538e-H3s95Nk9XJaneMQ@mail.gmail.com>
Message-ID: <CAHD5bX=O+ygDUYmUN1hY8PgZOs7StZCsih8z+ws9HT3-X0aPLQ@mail.gmail.com>

Well I have proof we're going things right! And it also shows the
power of MKL...

I checked with EPD and my python snippet (see also my first email),
and all 4 timings are the same...
However, if this behavior is the right one, then my python snippet
should get the same trend. But it's not.

So I though that indeed MKL could be tricking me, so I built scipy and
time again with the same config (ATLAS 3.10).
I call the 'dot' function I put on pastebin as dot and here are the timings:
In [8]: %timeit dot(big, big)
10 loops, best of 3: 133 ms per loop

In [9]: %timeit dot(big.T, big.T)
10 loops, best of 3: 133 ms per loop

In [10]: %timeit dot(big.T, big)
10 loops, best of 3: 97.9 ms per loop

In [11]: %timeit dot(big, big.T)
10 loops, best of 3: 96.4 ms per loop

So you guys were right, it's cache friendly.
We're doing the right thing, and the differences are most likely not
significant (or the snippet is wrong too)!
In [14]: np.sqrt(((dot(big, big.T) - np.dot(big, big.T))**2).sum())
Out[14]: 0.015331409

In [15]: np.sqrt(((dot(big, big) - np.dot(big, big))**2).sum())
Out[15]: 0.0


All this to say that we are getting a 2x speed improvement on
a.dot(a.T), which is awesome.

-nicolas

Nath: same trend if arrays are different.

On Fri, Nov 9, 2012 at 3:05 PM, Nathaniel Smith <njs at pobox.com> wrote:
> In this case it's even possible the blas is being smart enough to notice
> that you've passed it the same pointer twice with the transpose switch on
> one of them, and is just skipping half the multiplies since the output is
> provably symmetric. Don't know if they actually do that but...
>
> -n
>
> On 9 Nov 2012 22:57, "Matthieu Brucher" <matthieu.brucher at gmail.com> wrote:
>>
>> Hi,
>>
>> A.A slower than A.A' is not a surprise for me. The latter is far more
>> cache friendly that the former. Everything follows cache lines, so it is
>> faster than something that will use one element from each cache line. In
>> fact it is exactly what "proves" that the new version is correct.
>> Good job (if all the tests were made and still pass ;) )
>>
>> Cheers,
>>
>> Matthieu
>>
>>
>> 2012/11/9 Nicolas SCHEFFER <scheffer.nicolas at gmail.com>
>>>
>>> Ok: comparing apples to apples. I'm clueless on my observations and
>>> would need input from you guys.
>>>
>>> Using ATLAS 3.10, numpy with and without my changes, I'm getting these
>>> timings and comparisons.
>>>
>>> #
>>> #I. Generate matrices using regular dot:
>>> #
>>> big = np.array(np.random.randn(2000, 2000), 'f');
>>> np.savez('out', big=big, none=big.dot(big), both=big.T.dot(big.T),
>>> left=big.T.dot(big), right=big.dot(big.T))"
>>>
>>> #
>>> #II. Timings with regular dot
>>> #
>>> In [3]: %timeit np.dot(big, big)
>>> 10 loops, best of 3: 138 ms per loop
>>>
>>> In [4]: %timeit np.dot(big, big.T)
>>> 10 loops, best of 3: 166 ms per loop
>>>
>>> In [5]: %timeit np.dot(big.T, big.T)
>>> 10 loops, best of 3: 193 ms per loop
>>>
>>> In [6]: %timeit np.dot(big.T, big)
>>> 10 loops, best of 3: 165 ms per loop
>>>
>>> #
>>> #III. I load these arrays and time again with the "fast" dot
>>> #
>>> In [21]: %timeit np.dot(big, big)
>>> 10 loops, best of 3: 138 ms per loop
>>>
>>> In [22]: %timeit np.dot(big.T, big)
>>> 10 loops, best of 3: 104 ms per loop
>>>
>>> In [23]: %timeit np.dot(big.T, big.T)
>>> 10 loops, best of 3: 138 ms per loop
>>>
>>> In [24]: %timeit np.dot(big, big.T)
>>> 10 loops, best of 3: 102 ms per loop
>>>
>>> 1. A'.A': great!
>>> 2. A.A' becomes faster than A.A !?!
>>>
>>> #
>>> #IV. MSE on differences
>>> #
>>> In [25]: np.sqrt(((arr['none'] - none)**2).sum())
>>> Out[25]: 0.0
>>>
>>> In [26]: np.sqrt(((arr['both'] - both)**2).sum())
>>> Out[26]: 0.0
>>>
>>> In [27]: np.sqrt(((arr['left'] - left)**2).sum())
>>> Out[27]: 0.015900515
>>>
>>> In [28]: np.sqrt(((arr['right'] - right)**2).sum())
>>> Out[28]: 0.015331409
>>>
>>> #
>>> # CCl
>>> #
>>> While the MSE are small, I'm wondering whether:
>>> - It's a bug: it should be exactly the same
>>> - It's a feature: BLAS is taking shortcuts when you have A.A'. The
>>> difference is not significant. Quick: PR that asap!
>>>
>>> I don't have enough expertise to answer that...
>>>
>>> Thanks much!
>>>
>>> -nicolas
>>> On Fri, Nov 9, 2012 at 2:13 PM, Nicolas SCHEFFER
>>> <scheffer.nicolas at gmail.com> wrote:
>>> > I too encourage users to use scipy.linalg for speed and robustness
>>> > (hence calling this scipy.dot), but it just brings so much confusion!
>>> > When using the scipy + numpy ecosystem, you'd almost want everything
>>> > be done with scipy so that you get the best implementation in all
>>> > cases: scipy.zeros(), scipy.array(), scipy.dot(), scipy.linalg.inv().
>>> >
>>> > Anyway this is indeed for another thread, the confusion we'd like to
>>> > fix here is that users shouldn't have to understand the C/F contiguous
>>> > concepts to get the maximum speed for np.dot()
>>> >
>>> > To summarize:
>>> > - The python snippet I posted is still valid and can speed up your
>>> > code if you can change all your dot() calls.
>>> > - The change in dotblas.c is a bit more problematic because it's very
>>> > core. I'm having issues right now to replicate the timings, I've got
>>> > better timing for a.dot(a.T) than for a.dot(a). There might be a bug.
>>> >
>>> > It's a pain to test because I cannot do the test in a single python
>>> > session.
>>> > I'm going to try to integrate most of your suggestions, I cannot
>>> > guarantee I'll have time to do them all though.
>>> >
>>> > -nicolas
>>> > On Fri, Nov 9, 2012 at 8:56 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>> >> On Fri, Nov 9, 2012 at 4:25 PM, Gael Varoquaux
>>> >> <gael.varoquaux at normalesup.org> wrote:
>>> >>> On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith wrote:
>>> >>>> But what if someone compiles numpy against an optimized blas (mkl,
>>> >>>> say) and then compiles SciPy against the reference blas? What do you
>>> >>>> do then!? ;-)
>>> >>>
>>> >>> This could happen. But the converse happens very often. What happens
>>> >>> is
>>> >>> that users (eg on shared computing resource) ask for a scientific
>>> >>> python
>>> >>> environment. The administrator than installs the package starting
>>> >>> from
>>> >>> the most basic one, to the most advanced one, thus starting with
>>> >>> numpy
>>> >>> that can very well build without any external blas. When he gets to
>>> >>> scipy
>>> >>> he hits the problem that the build system does not detect properly
>>> >>> the
>>> >>> blas, and he solves that problem.
>>> >>>
>>> >>> Also, it used to be that on the major linux distributions, numpy
>>> >>> would not
>>> >>> be build with an optimize lapack because numpy was in the 'base' set
>>> >>> of
>>> >>> packages, but not lapack. On the contrary, scipy being in the
>>> >>> 'contrib'
>>> >>> set, it could depend on lapack. I just checked, and this has been
>>> >>> fixed
>>> >>> in the major distributions (Fedora, Debian, Ubuntu).
>>> >>>
>>> >>> Now we can discuss with such problems should not happen, and put the
>>> >>> blame on the users/administrators, the fact is that they happen
>>> >>> often. I
>>> >>> keep seeing environments in which np.linalg is unreasonnably slow.
>>> >>
>>> >> If this is something that's been a problem for you, maybe we should
>>> >> start another thread on things we could do to fix it directly? Improve
>>> >> build instructions, advertise build systems that set up the whole
>>> >> environment (and thus do the right thing), make numpy's setup.py
>>> >> scream and yell if blas isn't available...?
>>> >>
>>> >> -n
>>> >> _______________________________________________
>>> >> NumPy-Discussion mailing list
>>> >> NumPy-Discussion at scipy.org
>>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>>
>>
>> --
>> Information System Engineer, Ph.D.
>> Blog: http://matt.eifelle.com
>> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>> Music band: http://liliejay.com/
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From d.s.seljebotn at astro.uio.no  Fri Nov  9 19:38:31 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sat, 10 Nov 2012 01:38:31 +0100
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
	<CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
	<CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
	<CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>
Message-ID: <509DA207.1000302@astro.uio.no>

On 11/09/2012 11:57 PM, Matthieu Brucher wrote:
> Hi,
>
> A.A slower than A.A' is not a surprise for me. The latter is far more
> cache friendly that the former. Everything follows cache lines, so it is
> faster than something that will use one element from each cache line. In
> fact it is exactly what "proves" that the new version is correct.
> Good job (if all the tests were made and still pass ;) )

Cache lines shouldn't matter much with a decent BLAS?

http://dl.acm.org/citation.cfm?id=1356053

(Googling for "Anatomy of High-Performance Matrix Multiplication" will 
give you a preprint outside of paywall, but Google appears to not want 
to give me the URL of a too long search result so I can't paste it).

Dag Sverre

>
> Cheers,
>
> Matthieu
>
>
> 2012/11/9 Nicolas SCHEFFER <scheffer.nicolas at gmail.com
> <mailto:scheffer.nicolas at gmail.com>>
>
>     Ok: comparing apples to apples. I'm clueless on my observations and
>     would need input from you guys.
>
>     Using ATLAS 3.10, numpy with and without my changes, I'm getting these
>     timings and comparisons.
>
>     #
>     #I. Generate matrices using regular dot:
>     #
>     big = np.array(np.random.randn(2000, 2000), 'f');
>     np.savez('out', big=big, none=big.dot(big), both=big.T.dot(big.T),
>     left=big.T.dot(big), right=big.dot(big.T))"
>
>     #
>     #II. Timings with regular dot
>     #
>     In [3]: %timeit np.dot(big, big)
>     10 loops, best of 3: 138 ms per loop
>
>     In [4]: %timeit np.dot(big, big.T)
>     10 loops, best of 3: 166 ms per loop
>
>     In [5]: %timeit np.dot(big.T, big.T)
>     10 loops, best of 3: 193 ms per loop
>
>     In [6]: %timeit np.dot(big.T, big)
>     10 loops, best of 3: 165 ms per loop
>
>     #
>     #III. I load these arrays and time again with the "fast" dot
>     #
>     In [21]: %timeit np.dot(big, big)
>     10 loops, best of 3: 138 ms per loop
>
>     In [22]: %timeit np.dot(big.T, big)
>     10 loops, best of 3: 104 ms per loop
>
>     In [23]: %timeit np.dot(big.T, big.T)
>     10 loops, best of 3: 138 ms per loop
>
>     In [24]: %timeit np.dot(big, big.T)
>     10 loops, best of 3: 102 ms per loop
>
>     1. A'.A': great!
>     2. A.A' becomes faster than A.A !?!
>
>     #
>     #IV. MSE on differences
>     #
>     In [25]: np.sqrt(((arr['none'] - none)**2).sum())
>     Out[25]: 0.0
>
>     In [26]: np.sqrt(((arr['both'] - both)**2).sum())
>     Out[26]: 0.0
>
>     In [27]: np.sqrt(((arr['left'] - left)**2).sum())
>     Out[27]: 0.015900515
>
>     In [28]: np.sqrt(((arr['right'] - right)**2).sum())
>     Out[28]: 0.015331409
>
>     #
>     # CCl
>     #
>     While the MSE are small, I'm wondering whether:
>     - It's a bug: it should be exactly the same
>     - It's a feature: BLAS is taking shortcuts when you have A.A'. The
>     difference is not significant. Quick: PR that asap!
>
>     I don't have enough expertise to answer that...
>
>     Thanks much!
>
>     -nicolas
>     On Fri, Nov 9, 2012 at 2:13 PM, Nicolas SCHEFFER
>     <scheffer.nicolas at gmail.com <mailto:scheffer.nicolas at gmail.com>> wrote:
>      > I too encourage users to use scipy.linalg for speed and robustness
>      > (hence calling this scipy.dot), but it just brings so much confusion!
>      > When using the scipy + numpy ecosystem, you'd almost want everything
>      > be done with scipy so that you get the best implementation in all
>      > cases: scipy.zeros(), scipy.array(), scipy.dot(), scipy.linalg.inv().
>      >
>      > Anyway this is indeed for another thread, the confusion we'd like to
>      > fix here is that users shouldn't have to understand the C/F
>     contiguous
>      > concepts to get the maximum speed for np.dot()
>      >
>      > To summarize:
>      > - The python snippet I posted is still valid and can speed up your
>      > code if you can change all your dot() calls.
>      > - The change in dotblas.c is a bit more problematic because it's very
>      > core. I'm having issues right now to replicate the timings, I've got
>      > better timing for a.dot(a.T) than for a.dot(a). There might be a bug.
>      >
>      > It's a pain to test because I cannot do the test in a single
>     python session.
>      > I'm going to try to integrate most of your suggestions, I cannot
>      > guarantee I'll have time to do them all though.
>      >
>      > -nicolas
>      > On Fri, Nov 9, 2012 at 8:56 AM, Nathaniel Smith <njs at pobox.com
>     <mailto:njs at pobox.com>> wrote:
>      >> On Fri, Nov 9, 2012 at 4:25 PM, Gael Varoquaux
>      >> <gael.varoquaux at normalesup.org
>     <mailto:gael.varoquaux at normalesup.org>> wrote:
>      >>> On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith wrote:
>      >>>> But what if someone compiles numpy against an optimized blas (mkl,
>      >>>> say) and then compiles SciPy against the reference blas? What
>     do you
>      >>>> do then!? ;-)
>      >>>
>      >>> This could happen. But the converse happens very often. What
>     happens is
>      >>> that users (eg on shared computing resource) ask for a
>     scientific python
>      >>> environment. The administrator than installs the package
>     starting from
>      >>> the most basic one, to the most advanced one, thus starting
>     with numpy
>      >>> that can very well build without any external blas. When he
>     gets to scipy
>      >>> he hits the problem that the build system does not detect
>     properly the
>      >>> blas, and he solves that problem.
>      >>>
>      >>> Also, it used to be that on the major linux distributions,
>     numpy would not
>      >>> be build with an optimize lapack because numpy was in the
>     'base' set of
>      >>> packages, but not lapack. On the contrary, scipy being in the
>     'contrib'
>      >>> set, it could depend on lapack. I just checked, and this has
>     been fixed
>      >>> in the major distributions (Fedora, Debian, Ubuntu).
>      >>>
>      >>> Now we can discuss with such problems should not happen, and
>     put the
>      >>> blame on the users/administrators, the fact is that they happen
>     often. I
>      >>> keep seeing environments in which np.linalg is unreasonnably slow.
>      >>
>      >> If this is something that's been a problem for you, maybe we should
>      >> start another thread on things we could do to fix it directly?
>     Improve
>      >> build instructions, advertise build systems that set up the whole
>      >> environment (and thus do the right thing), make numpy's setup.py
>      >> scream and yell if blas isn't available...?
>      >>
>      >> -n
>      >> _______________________________________________
>      >> NumPy-Discussion mailing list
>      >> NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>      >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>     _______________________________________________
>     NumPy-Discussion mailing list
>     NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>
> --
> Information System Engineer, Ph.D.
> Blog: http://matt.eifelle.com
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> Music band: http://liliejay.com/
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



From gnurser at gmail.com  Sat Nov 10 07:39:03 2012
From: gnurser at gmail.com (George Nurser)
Date: Sat, 10 Nov 2012 12:39:03 +0000
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <509DA207.1000302@astro.uio.no>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
	<CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
	<CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
	<CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>
	<509DA207.1000302@astro.uio.no>
Message-ID: <CAAyCRnS5fSn1JXXCa6goA9VVNxs0wZO86J9S3WquYN7m99=9Rg@mail.gmail.com>

Note also that OpenBlas claims performance as good as MKL with Sandy Bridge
processors.

https://github.com/xianyi/OpenBLAS/wiki/faq#wiki-sandybridge_perf

George Nurser.


On 10 November 2012 00:38, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no>wrote:

> On 11/09/2012 11:57 PM, Matthieu Brucher wrote:
> > Hi,
> >
> > A.A slower than A.A' is not a surprise for me. The latter is far more
> > cache friendly that the former. Everything follows cache lines, so it is
> > faster than something that will use one element from each cache line. In
> > fact it is exactly what "proves" that the new version is correct.
> > Good job (if all the tests were made and still pass ;) )
>
> Cache lines shouldn't matter much with a decent BLAS?
>
> http://dl.acm.org/citation.cfm?id=1356053
>
> (Googling for "Anatomy of High-Performance Matrix Multiplication" will
> give you a preprint outside of paywall, but Google appears to not want
> to give me the URL of a too long search result so I can't paste it).
>
> Dag Sverre
>
> >
> > Cheers,
> >
> > Matthieu
> >
> >
> > 2012/11/9 Nicolas SCHEFFER <scheffer.nicolas at gmail.com
> > <mailto:scheffer.nicolas at gmail.com>>
> >
> >     Ok: comparing apples to apples. I'm clueless on my observations and
> >     would need input from you guys.
> >
> >     Using ATLAS 3.10, numpy with and without my changes, I'm getting
> these
> >     timings and comparisons.
> >
> >     #
> >     #I. Generate matrices using regular dot:
> >     #
> >     big = np.array(np.random.randn(2000, 2000), 'f');
> >     np.savez('out', big=big, none=big.dot(big), both=big.T.dot(big.T),
> >     left=big.T.dot(big), right=big.dot(big.T))"
> >
> >     #
> >     #II. Timings with regular dot
> >     #
> >     In [3]: %timeit np.dot(big, big)
> >     10 loops, best of 3: 138 ms per loop
> >
> >     In [4]: %timeit np.dot(big, big.T)
> >     10 loops, best of 3: 166 ms per loop
> >
> >     In [5]: %timeit np.dot(big.T, big.T)
> >     10 loops, best of 3: 193 ms per loop
> >
> >     In [6]: %timeit np.dot(big.T, big)
> >     10 loops, best of 3: 165 ms per loop
> >
> >     #
> >     #III. I load these arrays and time again with the "fast" dot
> >     #
> >     In [21]: %timeit np.dot(big, big)
> >     10 loops, best of 3: 138 ms per loop
> >
> >     In [22]: %timeit np.dot(big.T, big)
> >     10 loops, best of 3: 104 ms per loop
> >
> >     In [23]: %timeit np.dot(big.T, big.T)
> >     10 loops, best of 3: 138 ms per loop
> >
> >     In [24]: %timeit np.dot(big, big.T)
> >     10 loops, best of 3: 102 ms per loop
> >
> >     1. A'.A': great!
> >     2. A.A' becomes faster than A.A !?!
> >
> >     #
> >     #IV. MSE on differences
> >     #
> >     In [25]: np.sqrt(((arr['none'] - none)**2).sum())
> >     Out[25]: 0.0
> >
> >     In [26]: np.sqrt(((arr['both'] - both)**2).sum())
> >     Out[26]: 0.0
> >
> >     In [27]: np.sqrt(((arr['left'] - left)**2).sum())
> >     Out[27]: 0.015900515
> >
> >     In [28]: np.sqrt(((arr['right'] - right)**2).sum())
> >     Out[28]: 0.015331409
> >
> >     #
> >     # CCl
> >     #
> >     While the MSE are small, I'm wondering whether:
> >     - It's a bug: it should be exactly the same
> >     - It's a feature: BLAS is taking shortcuts when you have A.A'. The
> >     difference is not significant. Quick: PR that asap!
> >
> >     I don't have enough expertise to answer that...
> >
> >     Thanks much!
> >
> >     -nicolas
> >     On Fri, Nov 9, 2012 at 2:13 PM, Nicolas SCHEFFER
> >     <scheffer.nicolas at gmail.com <mailto:scheffer.nicolas at gmail.com>>
> wrote:
> >      > I too encourage users to use scipy.linalg for speed and robustness
> >      > (hence calling this scipy.dot), but it just brings so much
> confusion!
> >      > When using the scipy + numpy ecosystem, you'd almost want
> everything
> >      > be done with scipy so that you get the best implementation in all
> >      > cases: scipy.zeros(), scipy.array(), scipy.dot(),
> scipy.linalg.inv().
> >      >
> >      > Anyway this is indeed for another thread, the confusion we'd like
> to
> >      > fix here is that users shouldn't have to understand the C/F
> >     contiguous
> >      > concepts to get the maximum speed for np.dot()
> >      >
> >      > To summarize:
> >      > - The python snippet I posted is still valid and can speed up your
> >      > code if you can change all your dot() calls.
> >      > - The change in dotblas.c is a bit more problematic because it's
> very
> >      > core. I'm having issues right now to replicate the timings, I've
> got
> >      > better timing for a.dot(a.T) than for a.dot(a). There might be a
> bug.
> >      >
> >      > It's a pain to test because I cannot do the test in a single
> >     python session.
> >      > I'm going to try to integrate most of your suggestions, I cannot
> >      > guarantee I'll have time to do them all though.
> >      >
> >      > -nicolas
> >      > On Fri, Nov 9, 2012 at 8:56 AM, Nathaniel Smith <njs at pobox.com
> >     <mailto:njs at pobox.com>> wrote:
> >      >> On Fri, Nov 9, 2012 at 4:25 PM, Gael Varoquaux
> >      >> <gael.varoquaux at normalesup.org
> >     <mailto:gael.varoquaux at normalesup.org>> wrote:
> >      >>> On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith wrote:
> >      >>>> But what if someone compiles numpy against an optimized blas
> (mkl,
> >      >>>> say) and then compiles SciPy against the reference blas? What
> >     do you
> >      >>>> do then!? ;-)
> >      >>>
> >      >>> This could happen. But the converse happens very often. What
> >     happens is
> >      >>> that users (eg on shared computing resource) ask for a
> >     scientific python
> >      >>> environment. The administrator than installs the package
> >     starting from
> >      >>> the most basic one, to the most advanced one, thus starting
> >     with numpy
> >      >>> that can very well build without any external blas. When he
> >     gets to scipy
> >      >>> he hits the problem that the build system does not detect
> >     properly the
> >      >>> blas, and he solves that problem.
> >      >>>
> >      >>> Also, it used to be that on the major linux distributions,
> >     numpy would not
> >      >>> be build with an optimize lapack because numpy was in the
> >     'base' set of
> >      >>> packages, but not lapack. On the contrary, scipy being in the
> >     'contrib'
> >      >>> set, it could depend on lapack. I just checked, and this has
> >     been fixed
> >      >>> in the major distributions (Fedora, Debian, Ubuntu).
> >      >>>
> >      >>> Now we can discuss with such problems should not happen, and
> >     put the
> >      >>> blame on the users/administrators, the fact is that they happen
> >     often. I
> >      >>> keep seeing environments in which np.linalg is unreasonnably
> slow.
> >      >>
> >      >> If this is something that's been a problem for you, maybe we
> should
> >      >> start another thread on things we could do to fix it directly?
> >     Improve
> >      >> build instructions, advertise build systems that set up the whole
> >      >> environment (and thus do the right thing), make numpy's setup.py
> >      >> scream and yell if blas isn't available...?
> >      >>
> >      >> -n
> >      >> _______________________________________________
> >      >> NumPy-Discussion mailing list
> >      >> NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
> >      >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >     _______________________________________________
> >     NumPy-Discussion mailing list
> >     NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
> >     http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> >
> > --
> > Information System Engineer, Ph.D.
> > Blog: http://matt.eifelle.com
> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
> > Music band: http://liliejay.com/
> >
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121110/23702894/attachment.html>

From ralf.gommers at gmail.com  Sat Nov 10 12:47:41 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 10 Nov 2012 18:47:41 +0100
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
In-Reply-To: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
References: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
Message-ID: <CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>

On Tue, Nov 6, 2012 at 6:49 PM, Peter Cock <p.j.a.cock at googlemail.com>wrote:

> Dear all,
>
> Since the NumPy 1.7.0b2 release didn't include a Windows
> (32 bit) installer for Python 3.3, I am considering compiling it
> myself for local testing. What compiler is recommended?
>

Either MSVC or MinGW 3.4.5. For the latter see
https://github.com/certik/numpy-vendor

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121110/bf2df909/attachment.html>

From p.j.a.cock at googlemail.com  Sat Nov 10 18:04:31 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Sat, 10 Nov 2012 23:04:31 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
In-Reply-To: <CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
References: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
	<CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
Message-ID: <CAKVJ-_69z5exXi3Oazm2tCgttEu9tZjqZQfUt6gvNhVsm8EdGw@mail.gmail.com>

On Sat, Nov 10, 2012 at 5:47 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
>
> On Tue, Nov 6, 2012 at 6:49 PM, Peter Cock <p.j.a.cock at googlemail.com>
> wrote:
>>
>> Dear all,
>>
>> Since the NumPy 1.7.0b2 release didn't include a Windows
>> (32 bit) installer for Python 3.3, I am considering compiling it
>> myself for local testing. What compiler is recommended?
>
>
> Either MSVC or MinGW 3.4.5. For the latter see
> https://github.com/certik/numpy-vendor

Thanks Ralf,

I was trying with MSVC 9.0 installed, but got this cryptic error:

   C:\Downloads\numpy-1.7.0b2 > C:\python33\python setup.py build
   ...
   error: Unable to find vcvarsall.bat

After sprinkling distutils with debug statements, I found it was
looking for MSVC v10 (not numpy's fault but the error is most
unhelpful).

Presumably "Microsoft Visual C++ 2010 Express Edition" is
the appropriate thing to download?
http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express

Thanks,

Peter


From p.j.a.cock at googlemail.com  Sat Nov 10 18:13:56 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Sat, 10 Nov 2012 23:13:56 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
In-Reply-To: <CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
References: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
	<CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
Message-ID: <CAKVJ-_7q4WW3sD-p3zE8tUTs7y9JaUbzr1MnF5AiYPV=iHy_sQ@mail.gmail.com>

On Sat, Nov 10, 2012 at 5:47 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
> On Tue, Nov 6, 2012 at 6:49 PM, Peter Cock wrote:
>>
>> Dear all,
>>
>> Since the NumPy 1.7.0b2 release didn't include a Windows
>> (32 bit) installer for Python 3.3, I am considering compiling it
>> myself for local testing. What compiler is recommended?
>
>
> Either MSVC or MinGW 3.4.5. For the latter see
> https://github.com/certik/numpy-vendor
>
> Ralf

I was trying with mingw32 via cygwin with gcc 2.4.4,
which also failed with a cryptic error:

C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\system_info.py:1406: UserW
arning:
    Lapack (http://www.netlib.org/lapack/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [lapack]) or by setting
    the LAPACK environment variable.
  warnings.warn(LapackNotFoundError.__doc__)
lapack_src_info:
  NOT AVAILABLE

C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\system_info.py:1409: UserW
arning:
    Lapack (http://www.netlib.org/lapack/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [lapack_src]) or by setting
    the LAPACK_SRC environment variable.
  warnings.warn(LapackSrcNotFoundError.__doc__)
  NOT AVAILABLE

running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler opti
ons
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler opt
ions
running build_src
build_src
building py_modules sources
building library "npymath" sources
Traceback (most recent call last):
  File "setup.py", line 214, in <module>
    setup_package()
  File "setup.py", line 207, in setup_package
    configuration=configuration )
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\core.py", line 186
, in setup
    return old_setup(**new_attr)
  File "c:\python33\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "c:\python33\lib\distutils\dist.py", line 917, in run_commands
    self.run_command(cmd)
  File "c:\python33\lib\distutils\dist.py", line 936, in run_command
    cmd_obj.run()
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build.py",
 line 37, in run
    old_build.run(self)
  File "c:\python33\lib\distutils\command\build.py", line 126, in run
    self.run_command(cmd_name)
  File "c:\python33\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "c:\python33\lib\distutils\dist.py", line 936, in run_command
    cmd_obj.run()
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 152, in run
    self.build_sources()
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 163, in build_sources
    self.build_library_sources(*libname_info)
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 298, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 385, in generate_sources
    source = func(extension, build_dir)
  File "numpy\core\setup.py", line 646, in get_mathlib_info
    st = config_cmd.try_link('int main(void) { return 0;}')
  File "c:\python33\lib\distutils\command\config.py", line 243, in try_link
    self._check_compiler()
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\config.py"
, line 45, in _check_compiler
    old_config._check_compiler(self)
  File "c:\python33\lib\distutils\command\config.py", line 98, in _check_compile
r
    dry_run=self.dry_run, force=1)
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\ccompiler.py", lin
e 560, in new_compiler
    compiler = klass(None, dry_run, force)
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.p
y", line 94, in __init__
    msvcr_success = build_msvcr_library()
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.p
y", line 336, in build_msvcr_library
    if int(msvcr_name.lstrip('msvcr')) < 80:
AttributeError: 'NoneType' object has no attribute 'lstrip'

I am updating cygwin to see if anything changes...

Peter


From p.j.a.cock at googlemail.com  Sat Nov 10 18:24:00 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Sat, 10 Nov 2012 23:24:00 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
In-Reply-To: <CAKVJ-_7q4WW3sD-p3zE8tUTs7y9JaUbzr1MnF5AiYPV=iHy_sQ@mail.gmail.com>
References: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
	<CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
	<CAKVJ-_7q4WW3sD-p3zE8tUTs7y9JaUbzr1MnF5AiYPV=iHy_sQ@mail.gmail.com>
Message-ID: <CAKVJ-_4XauwyYLED0Gy4xoDeomV+xHUNQ9zpe+QYnr5X4Kwhhw@mail.gmail.com>

I meant to click on "save" not "send", anyway:

On Sat, Nov 10, 2012 at 11:13 PM, Peter Cock <p.j.a.cock at googlemail.com> wrote:
>>
>> Either MSVC or MinGW 3.4.5. For the latter see
>> https://github.com/certik/numpy-vendor
>>
>> Ralf
>
> I was trying with mingw32 via cygwin with gcc 2.4.4,

Typo, gcc 3.4.4

> which also failed with a cryptic error:
>
> ...
>   File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.p
> y", line 94, in __init__
>     msvcr_success = build_msvcr_library()
>   File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.p
> y", line 336, in build_msvcr_library
>     if int(msvcr_name.lstrip('msvcr')) < 80:
> AttributeError: 'NoneType' object has no attribute 'lstrip'

I think part of the problem could be in numpy/distutils/misc_util.py
where there is no code to detect MSCV 10,

def msvc_runtime_library():
    "Return name of MSVC runtime library if Python was built with MSVC >= 7"
    msc_pos = sys.version.find('MSC v.')
    if msc_pos != -1:
        msc_ver = sys.version[msc_pos+6:msc_pos+10]
        lib = {'1300' : 'msvcr70',    # MSVC 7.0
               '1310' : 'msvcr71',    # MSVC 7.1
               '1400' : 'msvcr80',    # MSVC 8
               '1500' : 'msvcr90',    # MSVC 9 (VS 2008)
              }.get(msc_ver, None)
    else:
        lib = None
    return lib

https://github.com/numpy/numpy/blob/master/numpy/distutils/misc_util.py#L353

Under Python 3.3, we have:

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600
32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)]'

i.e. It looks to me like that dictionary needs another entry for key '1600'.

Peter


From p.j.a.cock at googlemail.com  Sat Nov 10 18:43:54 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Sat, 10 Nov 2012 23:43:54 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
In-Reply-To: <CAKVJ-_4XauwyYLED0Gy4xoDeomV+xHUNQ9zpe+QYnr5X4Kwhhw@mail.gmail.com>
References: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
	<CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
	<CAKVJ-_7q4WW3sD-p3zE8tUTs7y9JaUbzr1MnF5AiYPV=iHy_sQ@mail.gmail.com>
	<CAKVJ-_4XauwyYLED0Gy4xoDeomV+xHUNQ9zpe+QYnr5X4Kwhhw@mail.gmail.com>
Message-ID: <CAKVJ-_5AhQD9sKLpW5RDA1a6=S_Wcs_ELi5BKSJkNmA2YvYmkw@mail.gmail.com>

On Sat, Nov 10, 2012 at 11:24 PM, Peter Cock <p.j.a.cock at googlemail.com> wrote:
>
> I think part of the problem could be in numpy/distutils/misc_util.py
> where there is no code to detect MSCV 10,
>
> def msvc_runtime_library():
>     "Return name of MSVC runtime library if Python was built with MSVC >= 7"
>     msc_pos = sys.version.find('MSC v.')
>     if msc_pos != -1:
>         msc_ver = sys.version[msc_pos+6:msc_pos+10]
>         lib = {'1300' : 'msvcr70',    # MSVC 7.0
>                '1310' : 'msvcr71',    # MSVC 7.1
>                '1400' : 'msvcr80',    # MSVC 8
>                '1500' : 'msvcr90',    # MSVC 9 (VS 2008)
>               }.get(msc_ver, None)
>     else:
>         lib = None
>     return lib
>
> https://github.com/numpy/numpy/blob/master/numpy/distutils/misc_util.py#L353
>
> Under Python 3.3, we have:
>
> Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600
> 32 bit (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> sys.version
> '3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)]'
>
> i.e. It looks to me like that dictionary needs another entry for key '1600'.
>

Adding this line seems to help,

               '1600' : 'msvcr100',   # MSVC 10 (aka 2010)

Now my compile gets further, but runs into another issue:

  File "c:\python33\lib\distutils\command\config.py", line 246, in try_link
    libraries, library_dirs, lang)
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\config.py",
line 146, in _link
    generate_manifest(self)
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py",
line 562, in generate_manifest
    check_embedded_msvcr_match_linked(msver)
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py",
line 541, in check_embedded_msvcr_match_linked
    "(%d)" % (int(msver), maj))
ValueError: Discrepancy between linked msvcr (10) and the one about to
be embedded (1)

My hunch was version something about the new three digit version
number is breaking things... which appears to be breaking here:

def check_embedded_msvcr_match_linked(msver):
    """msver is the ms runtime version used for the MANIFEST."""
    # check msvcr major version are the same for linking and
    # embedding
    msvcv = msvc_runtime_library()
    if msvcv:
        maj = int(msvcv[5:6])
        if not maj == int(msver):
            raise ValueError(
                  "Discrepancy between linked msvcr " \
                  "(%d) and the one about to be embedded " \
                  "(%d)" % (int(msver), maj))

https://github.com/numpy/numpy/blob/master/numpy/distutils/mingw32ccompiler.py#L530

As you can see, to get the major version number from the
string it looks at the first digit. When the string was something
like "81" or "90" that was fine, but now it is "100". Instead it
should look at all the digits up to the final one, i.e. use:

        maj = int(msvcv[5:-1])

Now (finally), I get an understandable (but hopefully wrong)
error message from trying to build NumPy 1.7.0b2 under
Python 3.3 on Windows XP,

  File "numpy\core\setup.py", line 646, in get_mathlib_info
    st = config_cmd.try_link('int main(void) { return 0;}')
  File "c:\python33\lib\distutils\command\config.py", line 246, in try_link
    libraries, library_dirs, lang)
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\config.py",
line 146, in _link
    generate_manifest(self)
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py",
line 568, in generate_manifest
    manxml = msvc_manifest_xml(ma, mi)
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py",
line 484, in msvc_manifest_xml
    % (maj, min))
ValueError: Version 10,0 of MSVCRT not supported yet

Presumably those two changes I have described are worth
committing to the trunk anyway? I can prepare a patch or
pull request, but currently I've been working on this Windows
box remotely and I'd prefer to wait until next week when I
can do it directly on the machine concerned.

Files affected:
numpy/distutils/misc_util.py function msvc_runtime_library
numpy/distutils/mingw32ccompiler.py function check_embedded_msvcr_match_linked

Peter


From ralf.gommers at gmail.com  Sun Nov 11 15:48:32 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 11 Nov 2012 21:48:32 +0100
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
Message-ID: <CABL7CQj+Nrqzo5k9Z+tELVSpXsiQgbE+5x2xi5CxUWMQPSV2AQ@mail.gmail.com>

On Tue, Nov 6, 2012 at 8:33 AM, Travis Oliphant <travis at continuum.io> wrote:

> Hey all,
>
> Ondrej has been tied up finishing his PhD for the past several weeks.  He
> is defending his work shortly and should be available to continue to help
> with the 1.7.0 release around the first of December.    He and I have been
> in contact during this process, and I've been helping where I can.
> Fortunately, other NumPy developers have been active closing tickets and
> reviewing pull requests which has helped the process substantially.
>
> The release has taken us longer than we expected, but I'm really glad that
> we've received the bug-reports and issues that we have seen because it will
> help the 1.7.0 release be a more stable series.   Also, the merging of the
> Trac issues with Git has exposed over-looked problems as well and will
> hopefully encourage more Git-focused participation by users.
>
> We are targeting getting the final release of 1.7.0 out by mid December
> (based on Ondrej's availability).   But, I would like to find out which
> issues are seen as blockers by people on this list.   I think most of the
> issues that I had as blockers have been resolved.    If there are no more
> remaining blockers, then we may be able to accelerate the final release of
> 1.7.0 to just after Thanksgiving.
>

Still the same ones I brought up last time (datetime + MinGW & segfault on
SPARC):
http://thread.gmane.org/gmane.comp.python.numeric.general/50907/focus=50944

There are some more issues listed at
https://github.com/numpy/numpy/issues?labels=&milestone=3&page=1&state=open.
Most of the Debian issues have disappeared, but there's some work left to
do there.

A Python 3.3 Windows binary would also be nice to have.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121111/2c21f52e/attachment.html>

From ralf.gommers at gmail.com  Sun Nov 11 16:05:32 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 11 Nov 2012 22:05:32 +0100
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
In-Reply-To: <CAKVJ-_5AhQD9sKLpW5RDA1a6=S_Wcs_ELi5BKSJkNmA2YvYmkw@mail.gmail.com>
References: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
	<CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
	<CAKVJ-_7q4WW3sD-p3zE8tUTs7y9JaUbzr1MnF5AiYPV=iHy_sQ@mail.gmail.com>
	<CAKVJ-_4XauwyYLED0Gy4xoDeomV+xHUNQ9zpe+QYnr5X4Kwhhw@mail.gmail.com>
	<CAKVJ-_5AhQD9sKLpW5RDA1a6=S_Wcs_ELi5BKSJkNmA2YvYmkw@mail.gmail.com>
Message-ID: <CABL7CQgjsHYPTyLhgePRNSG88cksJcodM2mA1E-eETvj9ZxWLw@mail.gmail.com>

On Sun, Nov 11, 2012 at 12:43 AM, Peter Cock <p.j.a.cock at googlemail.com>wrote:

> On Sat, Nov 10, 2012 at 11:24 PM, Peter Cock <p.j.a.cock at googlemail.com>
> wrote:
> >
> > I think part of the problem could be in numpy/distutils/misc_util.py
> > where there is no code to detect MSCV 10,
> >
> > def msvc_runtime_library():
> >     "Return name of MSVC runtime library if Python was built with MSVC
> >= 7"
> >     msc_pos = sys.version.find('MSC v.')
> >     if msc_pos != -1:
> >         msc_ver = sys.version[msc_pos+6:msc_pos+10]
> >         lib = {'1300' : 'msvcr70',    # MSVC 7.0
> >                '1310' : 'msvcr71',    # MSVC 7.1
> >                '1400' : 'msvcr80',    # MSVC 8
> >                '1500' : 'msvcr90',    # MSVC 9 (VS 2008)
> >               }.get(msc_ver, None)
> >     else:
> >         lib = None
> >     return lib
> >
> >
> https://github.com/numpy/numpy/blob/master/numpy/distutils/misc_util.py#L353
> >
> > Under Python 3.3, we have:
> >
> > Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600
> > 32 bit (Intel)] on win32
> > Type "help", "copyright", "credits" or "license" for more information.
> >>>> import sys
> >>>> sys.version
> > '3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit
> (Intel)]'
> >
> > i.e. It looks to me like that dictionary needs another entry for key
> '1600'.
> >
>
> Adding this line seems to help,
>
>                '1600' : 'msvcr100',   # MSVC 10 (aka 2010)
>
> Now my compile gets further, but runs into another issue:
>
>   File "c:\python33\lib\distutils\command\config.py", line 246, in try_link
>     libraries, library_dirs, lang)
>   File
> "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\config.py",
> line 146, in _link
>     generate_manifest(self)
>   File
> "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py",
> line 562, in generate_manifest
>     check_embedded_msvcr_match_linked(msver)
>   File
> "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py",
> line 541, in check_embedded_msvcr_match_linked
>     "(%d)" % (int(msver), maj))
> ValueError: Discrepancy between linked msvcr (10) and the one about to
> be embedded (1)
>
> My hunch was version something about the new three digit version
> number is breaking things... which appears to be breaking here:
>
> def check_embedded_msvcr_match_linked(msver):
>     """msver is the ms runtime version used for the MANIFEST."""
>     # check msvcr major version are the same for linking and
>     # embedding
>     msvcv = msvc_runtime_library()
>     if msvcv:
>         maj = int(msvcv[5:6])
>         if not maj == int(msver):
>             raise ValueError(
>                   "Discrepancy between linked msvcr " \
>                   "(%d) and the one about to be embedded " \
>                   "(%d)" % (int(msver), maj))
>
>
> https://github.com/numpy/numpy/blob/master/numpy/distutils/mingw32ccompiler.py#L530
>
> As you can see, to get the major version number from the
> string it looks at the first digit. When the string was something
> like "81" or "90" that was fine, but now it is "100". Instead it
> should look at all the digits up to the final one, i.e. use:
>
>         maj = int(msvcv[5:-1])
>
> Now (finally), I get an understandable (but hopefully wrong)
> error message from trying to build NumPy 1.7.0b2 under
> Python 3.3 on Windows XP,
>
>   File "numpy\core\setup.py", line 646, in get_mathlib_info
>     st = config_cmd.try_link('int main(void) { return 0;}')
>   File "c:\python33\lib\distutils\command\config.py", line 246, in try_link
>     libraries, library_dirs, lang)
>   File
> "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\config.py",
> line 146, in _link
>     generate_manifest(self)
>   File
> "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py",
> line 568, in generate_manifest
>     manxml = msvc_manifest_xml(ma, mi)
>   File
> "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\mingw32ccompiler.py",
> line 484, in msvc_manifest_xml
>     % (maj, min))
> ValueError: Version 10,0 of MSVCRT not supported yet
>
> Presumably those two changes I have described are worth
> committing to the trunk anyway? I can prepare a patch or
> pull request, but currently I've been working on this Windows
> box remotely and I'd prefer to wait until next week when I
> can do it directly on the machine concerned.
>

Those changes look correct, a PR would be great.

Fixing the next error also seems straightforward; around line 465 of
mingw32ccompiler a check is needed for 10.0 (not sure which one) and then a
line like
    _MSVCRVER_TO_FULLVER['10'] = "10.0.xxxxx.x"
needs to be added.

Ralf


> Files affected:
> numpy/distutils/misc_util.py function msvc_runtime_library
> numpy/distutils/mingw32ccompiler.py function
> check_embedded_msvcr_match_linked
>
> Peter
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121111/50ec94bf/attachment.html>

From p.j.a.cock at googlemail.com  Sun Nov 11 18:20:48 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Sun, 11 Nov 2012 23:20:48 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
In-Reply-To: <CABL7CQgjsHYPTyLhgePRNSG88cksJcodM2mA1E-eETvj9ZxWLw@mail.gmail.com>
References: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
	<CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
	<CAKVJ-_7q4WW3sD-p3zE8tUTs7y9JaUbzr1MnF5AiYPV=iHy_sQ@mail.gmail.com>
	<CAKVJ-_4XauwyYLED0Gy4xoDeomV+xHUNQ9zpe+QYnr5X4Kwhhw@mail.gmail.com>
	<CAKVJ-_5AhQD9sKLpW5RDA1a6=S_Wcs_ELi5BKSJkNmA2YvYmkw@mail.gmail.com>
	<CABL7CQgjsHYPTyLhgePRNSG88cksJcodM2mA1E-eETvj9ZxWLw@mail.gmail.com>
Message-ID: <CAKVJ-_44xa3x5ZKp7wdozh5HD0ZL9NskTQ95umdmC+3s3f5h1A@mail.gmail.com>

On Sun, Nov 11, 2012 at 9:05 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
> Those changes look correct, a PR would be great.
>

I'll do that later this week - but feel free to do it yourself immediately
if more convenient.

> Fixing the next error also seems straightforward; around line 465 of
> mingw32ccompiler a check is needed for 10.0 (not sure which one)
> and then a line like
>     _MSVCRVER_TO_FULLVER['10'] = "10.0.xxxxx.x"
> needs to be added.

I'd got that far last night before calling turning in - my question was
where do I get the last two parts of the version number. Presumably
I can just use the values from the msvcr100.dll on my machine?

Thanks,

Peter


From p.j.a.cock at googlemail.com  Mon Nov 12 07:31:05 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Mon, 12 Nov 2012 12:31:05 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
In-Reply-To: <CAKVJ-_44xa3x5ZKp7wdozh5HD0ZL9NskTQ95umdmC+3s3f5h1A@mail.gmail.com>
References: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
	<CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
	<CAKVJ-_7q4WW3sD-p3zE8tUTs7y9JaUbzr1MnF5AiYPV=iHy_sQ@mail.gmail.com>
	<CAKVJ-_4XauwyYLED0Gy4xoDeomV+xHUNQ9zpe+QYnr5X4Kwhhw@mail.gmail.com>
	<CAKVJ-_5AhQD9sKLpW5RDA1a6=S_Wcs_ELi5BKSJkNmA2YvYmkw@mail.gmail.com>
	<CABL7CQgjsHYPTyLhgePRNSG88cksJcodM2mA1E-eETvj9ZxWLw@mail.gmail.com>
	<CAKVJ-_44xa3x5ZKp7wdozh5HD0ZL9NskTQ95umdmC+3s3f5h1A@mail.gmail.com>
Message-ID: <CAKVJ-_4zJmKcUbmXMFLnzQ+_agPMEct7dUxBRcf6WbF4mPwyDg@mail.gmail.com>

On Sun, Nov 11, 2012 at 11:20 PM, Peter Cock <p.j.a.cock at googlemail.com> wrote:
> On Sun, Nov 11, 2012 at 9:05 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>
>> Those changes look correct, a PR would be great.
>>
>
> I'll do that later this week - but feel free to do it yourself immediately
> if more convenient.
>

Hi again Ralf,

OK, new branch from the master here:
https://github.com/peterjc/numpy/tree/msvc10

The first two commits are the changes already discussed.
https://github.com/peterjc/numpy/commit/d8ead4c83364f9c7d4543690eb12e4543c608372
https://github.com/peterjc/numpy/commit/1d498f54668a9a87286fa31f2779acbb048edd39

>> Fixing the next error also seems straightforward; around line 465 of
>> mingw32ccompiler a check is needed for 10.0 (not sure which one)
>> and then a line like
>>     _MSVCRVER_TO_FULLVER['10'] = "10.0.xxxxx.x"
>> needs to be added.
>
> I'd got that far last night before calling turning in - my question was
> where do I get the last two parts of the version number. Presumably
> I can just use the values from the msvcr100.dll on my machine?

The third commit provides a fall back value (based on the DLL on
my machine) as done for MSVC v8 and v9, and updates the live
lookup via msvcrt.CRT_ASSEMBLY_VERSION to actually check
what version that is (the current code wrongly assumes it is going
to be for MSCV v9):
https://github.com/peterjc/numpy/commit/24523565b5dbb23d6de0591ef2a4c1d014722c5d

Do you want a pull request for that work so far? Or can you
suggest what I should investigate next since there is at least
one more hurdle to clear before I can build it (build output below).

Regards,

Peter

------


I now get further trying to build with mingw32 (from this branch
from master, not the numpy-1.7.0b2 files this time), an excerpt
from which this line may be note-worthy:

Cannot build msvcr library: "msvcr100d.dll" not found

This omits the failure to detect ALTAS etc which seems irrelevant.
I would guess the MismatchCAPIWarning is an unrelated issue
from working from the master branch this time.

C:\repositories\numpy>c:\python33\python setup.py build --compiler=mingw32
Converting to Python3 via 2to3...
<snip>
numpy\core\setup_common.py:86: MismatchCAPIWarning: API mismatch
detected, the C API version numbers have to be updated. Current C api
version is 8, with checksum f4362353e2d72f889fda0128aa015037, but
recorded checksum for C API version 8 in codegen_dir/cversions.txt is
17321775fc884de0b1eda478cd61c74b. If functions were added in the C
API, you have to update C_API_VERSION  in numpy\core\setup_common.py.
MismatchCAPIWarning)
<snip>
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands
--compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands
--fcompiler options
running build_src
build_src
building py_modules sources
building library "npymath" sources
Cannot build msvcr library: "msvcr100d.dll" not found
Unable to find productdir in registry
Checking environ VS100COMNTOOLS
customize GnuFCompiler
Could not locate executable g77
Could not locate executable f77
customize IntelVisualFCompiler
Could not locate executable ifort
Could not locate executable ifl
customize AbsoftFCompiler
Could not locate executable f90
customize CompaqVisualFCompiler
Found executable C:\cygwin\bin\DF.exe
Found executable C:\cygwin\bin\DF.exe
customize IntelItaniumVisualFCompiler
Could not locate executable efl
customize Gnu95FCompiler
Could not locate executable gfortran
Could not locate executable f95
customize G95FCompiler
Could not locate executable g95
customize IntelEM64VisualFCompiler
customize IntelEM64TFCompiler
Could not locate executable efort
Could not locate executable efc
don't know how to compile Fortran code on platform 'nt'
C compiler: gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes

compile options: '-DNPY_MINGW_USE_CUSTOM_MSVCR
-D__MSVCRT_VERSION__=0x1000 -Inumpy\core\src\private -Inumpy\core\src
-Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray
-Inumpy\core\src\umath -Inumpy\core\src\npysort -Inumpy\core\include
-Ic:\python33\include -Ic:\python33\include -c'
gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes
-DNPY_MINGW_USE_CUSTOM_MSVCR -D__MSVCRT_VERSION__=0x1000
-Inumpy\core\src\private -Inumpy\core\src -Inumpy\core
-Inumpy\core\src\npymath -Inumpy\core\src\multiarray
-Inumpy\core\src\umath -Inumpy\core\src\npysort -Inumpy\core\include
-Ic:\python33\include -Ic:\python33\include -c _configtest.c -o
_configtest.o
Found executable C:\cygwin\usr\bin\gcc.exe
g++ -mno-cygwin _configtest.o -lmsvcr100 -o _configtest.exe
Could not locate executable g++
Executable g++ does not exist

failure.
removing: _configtest.exe.manifest _configtest.c _configtest.o
Traceback (most recent call last):
  File "setup.py", line 214, in <module>
    setup_package()
  File "setup.py", line 207, in setup_package
    configuration=configuration )
  File "C:\repositories\numpy\build\py3k\numpy\distutils\core.py", line 186, in
setup
    return old_setup(**new_attr)
  File "c:\python33\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "c:\python33\lib\distutils\dist.py", line 917, in run_commands
    self.run_command(cmd)
  File "c:\python33\lib\distutils\dist.py", line 936, in run_command
    cmd_obj.run()
  File "C:\repositories\numpy\build\py3k\numpy\distutils\command\build.py", line
 37, in run
    old_build.run(self)
  File "c:\python33\lib\distutils\command\build.py", line 126, in run
    self.run_command(cmd_name)
  File "c:\python33\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "c:\python33\lib\distutils\dist.py", line 936, in run_command
    cmd_obj.run()
  File "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py",
line 152, in run
    self.build_sources()
  File "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py",
line 163, in build_sources
    self.build_library_sources(*libname_info)
  File "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py",
line 298, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py",
line 385, in generate_sources
    source = func(extension, build_dir)
  File "numpy\core\setup.py", line 648, in get_mathlib_info
    raise RuntimeError("Broken toolchain: cannot link a simple C program")
RuntimeError: Broken toolchain: cannot link a simple C program


Peter


From alan.isaac at gmail.com  Mon Nov 12 08:48:04 2012
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Mon, 12 Nov 2012 08:48:04 -0500
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
Message-ID: <50A0FE14.6050605@gmail.com>

On 11/9/2012 12:21 PM, Nathaniel Smith wrote:
> you might want to double-check that the
> np.random.choice in 1.7 actually*does*  give an error if the input
> array is not 1-d


Any idea where I can look at the code?
I browsed github after failing to find
a productive search string, but failed
to find it.

Which remind me: it would be nice if the
docs linked to the source.

Thanks,
Alan


From sebastian at sipsolutions.net  Mon Nov 12 08:59:42 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Mon, 12 Nov 2012 14:59:42 +0100
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <50A0FE14.6050605@gmail.com>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com>
Message-ID: <1352728782.3411.2.camel@sebastian-laptop>

Hey,

On Mon, 2012-11-12 at 08:48 -0500, Alan G Isaac wrote:
> On 11/9/2012 12:21 PM, Nathaniel Smith wrote:
> > you might want to double-check that the
> > np.random.choice in 1.7 actually*does*  give an error if the input
> > array is not 1-d
> 
> 
> Any idea where I can look at the code?
> I browsed github after failing to find
> a productive search string, but failed
> to find it.
> 

its here:
https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/mtrand.pyx#L919

Sounds like it should be pretty simple to add axis=None which would
change the current behavior very little, it would stop give an error
anymore for none 1-d arrays though.

Regards,

Sebastian


> Which remind me: it would be nice if the
> docs linked to the source.
> 
> Thanks,
> Alan
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From njs at pobox.com  Mon Nov 12 09:03:56 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 12 Nov 2012 15:03:56 +0100
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <50A0FE14.6050605@gmail.com>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com>
Message-ID: <CAPJVwBmv272mLFqyLZoLm1n+gwtBxpHr=o8AR+Uu8Le2xE8k3A@mail.gmail.com>

On Mon, Nov 12, 2012 at 2:48 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> On 11/9/2012 12:21 PM, Nathaniel Smith wrote:
>> you might want to double-check that the
>> np.random.choice in 1.7 actually*does*  give an error if the input
>> array is not 1-d
>
> Any idea where I can look at the code?
> I browsed github after failing to find
> a productive search string, but failed
> to find it.

Looks like it's in numpy/random/mtrand/mtrand.pyx (cython code)

I was actually thinking you would check by just trying it, since
that's an easier and more reliable way to determine what code actually
does than reading it :-). (Or even better, writing a test?)

> Which remind me: it would be nice if the
> docs linked to the source.

True, though it's a difficult problem for code like this that goes
Cython file -> C -> .so file -> Python. I'm not sure Cython actually
preserves any metadata that would let us look at the np.random.choice
object in the interpreter and map that back to a line of a source
file.

-n


From alan.isaac at gmail.com  Mon Nov 12 09:50:26 2012
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Mon, 12 Nov 2012 09:50:26 -0500
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <1352728782.3411.2.camel@sebastian-laptop>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com>
	<1352728782.3411.2.camel@sebastian-laptop>
Message-ID: <50A10CB2.80209@gmail.com>

On 11/12/2012 8:59 AM, Sebastian Berg wrote:
> https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/mtrand.pyx#L919
>
> Sounds like it should be pretty simple to add axis=None which would
> change the current behavior very little, it would stop give an error
> anymore for none 1-d arrays though.


I believe Nathaniel suggested *retaining* this error
until two things were supported: flattening ndarrays,
and choice of subarrays for ndarrays.  If I understand
you, you are suggesting the first is easily supported.
Or are you also suggesting the 2nd is easily supported?

Alan Isaac

PS I'll repost the code (or similar) that Robert Kern
posted when this function was discussed in 2006.
However it did not support multiple samples.

def choice(x, axis=None):
     x = np.asarray(x)
     if axis is None:
         length = np.multiply.reduce(x.shape)
         n = random.randint(length)
         return x.flat[n]
     else:
         n = random.randint(x.shape[axis])
         idx = map(slice, x.shape)
         idx[axis] = n
         return x[tuple(idx)]



From njs at pobox.com  Mon Nov 12 10:00:10 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 12 Nov 2012 16:00:10 +0100
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <50A10CB2.80209@gmail.com>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com>
	<1352728782.3411.2.camel@sebastian-laptop>
	<50A10CB2.80209@gmail.com>
Message-ID: <CAPJVwBm5UCvLWiw3Fb-Vnrop+L6B7jjm3x0qzyoNmqT7zTOHLw@mail.gmail.com>

On Mon, Nov 12, 2012 at 3:50 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> On 11/12/2012 8:59 AM, Sebastian Berg wrote:
>> https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/mtrand.pyx#L919
>>
>> Sounds like it should be pretty simple to add axis=None which would
>> change the current behavior very little, it would stop give an error
>> anymore for none 1-d arrays though.
>
> I believe Nathaniel suggested *retaining* this error
> until two things were supported: flattening ndarrays,
> and choice of subarrays for ndarrays.

Just to be clear, I don't really have an opinion on whether those
things should be supported, or what the right API should be; I haven't
really thought about it. Maybe others on the list have opinions. I was
just saying that we have plenty of time to decide about these things,
it's not an emergency that should hold up the 1.7 release (or risk
locking in a bad API).

-n


From alan.isaac at gmail.com  Mon Nov 12 10:09:25 2012
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Mon, 12 Nov 2012 10:09:25 -0500
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <CAPJVwBm5UCvLWiw3Fb-Vnrop+L6B7jjm3x0qzyoNmqT7zTOHLw@mail.gmail.com>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com>
	<1352728782.3411.2.camel@sebastian-laptop>
	<50A10CB2.80209@gmail.com>
	<CAPJVwBm5UCvLWiw3Fb-Vnrop+L6B7jjm3x0qzyoNmqT7zTOHLw@mail.gmail.com>
Message-ID: <50A11125.1020106@gmail.com>

On 11/12/2012 10:00 AM, Nathaniel Smith wrote:
> I don't really have an opinion on whether those
> things should be supported, or what the right API should be; I haven't
> really thought about it. Maybe others on the list have opinions. I was
> just saying that we have plenty of time to decide about these things


OK. For now I've opened an issue for this:
https://github.com/numpy/numpy/issues/2724
I assume that's the right place to accumulate
comments on the proposal.

Alan



From alan.isaac at gmail.com  Mon Nov 12 11:31:03 2012
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Mon, 12 Nov 2012 11:31:03 -0500
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <1352728782.3411.2.camel@sebastian-laptop>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com>
	<1352728782.3411.2.camel@sebastian-laptop>
Message-ID: <50A12447.6050604@gmail.com>

In a comment on the issue https://github.com/numpy/numpy/issues/2724 Sebastian notes:
"it could also be reasonable to have size=None as default and have it return a scalar/the given axes removed in that case. That would be a real change 
in functionality unfortunately, but it would make sense for similarity to import random; random.choice mostly."

If this is under serious consider, then perhaps
random.choice should not be in 1.7 unless a decision can
be quickly made about the default for `size`.
(Allowing an axis argument can however be postponed.)

I am inclined to think that Sebastian's suggestion
is correct.

Alan Isaac



From njs at pobox.com  Mon Nov 12 11:52:24 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 12 Nov 2012 17:52:24 +0100
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <50A12447.6050604@gmail.com>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com>
	<1352728782.3411.2.camel@sebastian-laptop>
	<50A12447.6050604@gmail.com>
Message-ID: <CAPJVwB=_tXi5qMVNJ-ZfBXAPLvkgy1SG6Pnn73-ESvtZ6HcWjA@mail.gmail.com>

On Mon, Nov 12, 2012 at 5:31 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> In a comment on the issue https://github.com/numpy/numpy/issues/2724 Sebastian notes:
> "it could also be reasonable to have size=None as default and have it return a scalar/the given axes removed in that case. That would be a real change
> in functionality unfortunately, but it would make sense for similarity to import random; random.choice mostly."
>
> If this is under serious consider, then perhaps
> random.choice should not be in 1.7 unless a decision can
> be quickly made about the default for `size`.
> (Allowing an axis argument can however be postponed.)
>
> I am inclined to think that Sebastian's suggestion
> is correct.

For anyone else trying to follow, here's the current function:
  http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.random.choice.html

I'm afraid I don't understand what Sebastian is suggesting should
happen by default. size=None doesn't have any intuitive meaning to me,
and I don't know what "a scalar/the given axes removed" means.

-n


From sebastian at sipsolutions.net  Mon Nov 12 12:16:11 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Mon, 12 Nov 2012 18:16:11 +0100
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <CAPJVwB=_tXi5qMVNJ-ZfBXAPLvkgy1SG6Pnn73-ESvtZ6HcWjA@mail.gmail.com>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com> <1352728782.3411.2.camel@sebastian-laptop>
	<50A12447.6050604@gmail.com>
	<CAPJVwB=_tXi5qMVNJ-ZfBXAPLvkgy1SG6Pnn73-ESvtZ6HcWjA@mail.gmail.com>
Message-ID: <1352740571.3411.19.camel@sebastian-laptop>

On Mon, 2012-11-12 at 17:52 +0100, Nathaniel Smith wrote:
> On Mon, Nov 12, 2012 at 5:31 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> > In a comment on the issue https://github.com/numpy/numpy/issues/2724 Sebastian notes:
> > "it could also be reasonable to have size=None as default and have it return a scalar/the given axes removed in that case. That would be a real change
> > in functionality unfortunately, but it would make sense for similarity to import random; random.choice mostly."
> >
> > If this is under serious consider, then perhaps
> > random.choice should not be in 1.7 unless a decision can
> > be quickly made about the default for `size`.
> > (Allowing an axis argument can however be postponed.)
> >
> > I am inclined to think that Sebastian's suggestion
> > is correct.
> 
> For anyone else trying to follow, here's the current function:
>   http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.random.choice.html
> 
> I'm afraid I don't understand what Sebastian is suggesting should
> happen by default. size=None doesn't have any intuitive meaning to me,
> and I don't know what "a scalar/the given axes removed" means.
> 
None is a little awkward I agree (but I don't think there is something
better), but basically what I meant is this:

>>> random.choice([1, 1])
1
>>> np.random.choice([1, 2])
array([1]) # its 1-D not 0-D.

So instead of taking a sequence of length 1, take an element as default.

> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From matthew.brett at gmail.com  Mon Nov 12 14:54:06 2012
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 12 Nov 2012 11:54:06 -0800
Subject: [Numpy-discussion] Do we want scalar casting to behave as it does
	at the moment?
Message-ID: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>

Hi,

I wanted to check that everyone knows about and is happy with the
scalar casting changes from 1.6.0.

Specifically, the rules for (array, scalar) casting have changed such
that the resulting dtype depends on the _value_ of the scalar.

Mark W has documented these changes here:

http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html

Specifically, as of 1.6.0:

In [19]: arr = np.array([1.], dtype=np.float32)

In [20]: (arr + (2**16-1)).dtype
Out[20]: dtype('float32')

In [21]: (arr + (2**16)).dtype
Out[21]: dtype('float64')

In [25]: arr = np.array([1.], dtype=np.int8)

In [26]: (arr + 127).dtype
Out[26]: dtype('int8')

In [27]: (arr + 128).dtype
Out[27]: dtype('int16')

There's discussion about the changes here:

http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html
http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html

It seems to me that this change is hard to explain, and does what you
want only some of the time, making it a false friend.

Is it the right behavior for numpy 2.0?

Cheers,

Matthew


From scheffer.nicolas at gmail.com  Mon Nov 12 15:08:39 2012
From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER)
Date: Mon, 12 Nov 2012 12:08:39 -0800
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAAyCRnS5fSn1JXXCa6goA9VVNxs0wZO86J9S3WquYN7m99=9Rg@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
	<CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
	<CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
	<CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>
	<509DA207.1000302@astro.uio.no>
	<CAAyCRnS5fSn1JXXCa6goA9VVNxs0wZO86J9S3WquYN7m99=9Rg@mail.gmail.com>
Message-ID: <CAHD5bXkn8ddzVU1s19cex9zOdkspzTe7=hZXBydKaHrr0e+D+Q@mail.gmail.com>

I've pushed my code to a branch here
https://github.com/leschef/numpy/tree/faster_dot
with the commit
https://github.com/leschef/numpy/commit/ea037770e03f23aca1a06274a1a8e8bf0e0e2ee4

Let me know if that's enough to create a pull request.

Thanks,

-nicolas

On Sat, Nov 10, 2012 at 4:39 AM, George Nurser <gnurser at gmail.com> wrote:
> Note also that OpenBlas claims performance as good as MKL with Sandy Bridge
> processors.
>
> https://github.com/xianyi/OpenBLAS/wiki/faq#wiki-sandybridge_perf
>
> George Nurser.
>
>
> On 10 November 2012 00:38, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no>
> wrote:
>>
>> On 11/09/2012 11:57 PM, Matthieu Brucher wrote:
>> > Hi,
>> >
>> > A.A slower than A.A' is not a surprise for me. The latter is far more
>> > cache friendly that the former. Everything follows cache lines, so it is
>> > faster than something that will use one element from each cache line. In
>> > fact it is exactly what "proves" that the new version is correct.
>> > Good job (if all the tests were made and still pass ;) )
>>
>> Cache lines shouldn't matter much with a decent BLAS?
>>
>> http://dl.acm.org/citation.cfm?id=1356053
>>
>> (Googling for "Anatomy of High-Performance Matrix Multiplication" will
>> give you a preprint outside of paywall, but Google appears to not want
>> to give me the URL of a too long search result so I can't paste it).
>>
>> Dag Sverre
>>
>> >
>> > Cheers,
>> >
>> > Matthieu
>> >
>> >
>> > 2012/11/9 Nicolas SCHEFFER <scheffer.nicolas at gmail.com
>> > <mailto:scheffer.nicolas at gmail.com>>
>> >
>> >     Ok: comparing apples to apples. I'm clueless on my observations and
>> >     would need input from you guys.
>> >
>> >     Using ATLAS 3.10, numpy with and without my changes, I'm getting
>> > these
>> >     timings and comparisons.
>> >
>> >     #
>> >     #I. Generate matrices using regular dot:
>> >     #
>> >     big = np.array(np.random.randn(2000, 2000), 'f');
>> >     np.savez('out', big=big, none=big.dot(big), both=big.T.dot(big.T),
>> >     left=big.T.dot(big), right=big.dot(big.T))"
>> >
>> >     #
>> >     #II. Timings with regular dot
>> >     #
>> >     In [3]: %timeit np.dot(big, big)
>> >     10 loops, best of 3: 138 ms per loop
>> >
>> >     In [4]: %timeit np.dot(big, big.T)
>> >     10 loops, best of 3: 166 ms per loop
>> >
>> >     In [5]: %timeit np.dot(big.T, big.T)
>> >     10 loops, best of 3: 193 ms per loop
>> >
>> >     In [6]: %timeit np.dot(big.T, big)
>> >     10 loops, best of 3: 165 ms per loop
>> >
>> >     #
>> >     #III. I load these arrays and time again with the "fast" dot
>> >     #
>> >     In [21]: %timeit np.dot(big, big)
>> >     10 loops, best of 3: 138 ms per loop
>> >
>> >     In [22]: %timeit np.dot(big.T, big)
>> >     10 loops, best of 3: 104 ms per loop
>> >
>> >     In [23]: %timeit np.dot(big.T, big.T)
>> >     10 loops, best of 3: 138 ms per loop
>> >
>> >     In [24]: %timeit np.dot(big, big.T)
>> >     10 loops, best of 3: 102 ms per loop
>> >
>> >     1. A'.A': great!
>> >     2. A.A' becomes faster than A.A !?!
>> >
>> >     #
>> >     #IV. MSE on differences
>> >     #
>> >     In [25]: np.sqrt(((arr['none'] - none)**2).sum())
>> >     Out[25]: 0.0
>> >
>> >     In [26]: np.sqrt(((arr['both'] - both)**2).sum())
>> >     Out[26]: 0.0
>> >
>> >     In [27]: np.sqrt(((arr['left'] - left)**2).sum())
>> >     Out[27]: 0.015900515
>> >
>> >     In [28]: np.sqrt(((arr['right'] - right)**2).sum())
>> >     Out[28]: 0.015331409
>> >
>> >     #
>> >     # CCl
>> >     #
>> >     While the MSE are small, I'm wondering whether:
>> >     - It's a bug: it should be exactly the same
>> >     - It's a feature: BLAS is taking shortcuts when you have A.A'. The
>> >     difference is not significant. Quick: PR that asap!
>> >
>> >     I don't have enough expertise to answer that...
>> >
>> >     Thanks much!
>> >
>> >     -nicolas
>> >     On Fri, Nov 9, 2012 at 2:13 PM, Nicolas SCHEFFER
>> >     <scheffer.nicolas at gmail.com <mailto:scheffer.nicolas at gmail.com>>
>> > wrote:
>> >      > I too encourage users to use scipy.linalg for speed and
>> > robustness
>> >      > (hence calling this scipy.dot), but it just brings so much
>> > confusion!
>> >      > When using the scipy + numpy ecosystem, you'd almost want
>> > everything
>> >      > be done with scipy so that you get the best implementation in all
>> >      > cases: scipy.zeros(), scipy.array(), scipy.dot(),
>> > scipy.linalg.inv().
>> >      >
>> >      > Anyway this is indeed for another thread, the confusion we'd like
>> > to
>> >      > fix here is that users shouldn't have to understand the C/F
>> >     contiguous
>> >      > concepts to get the maximum speed for np.dot()
>> >      >
>> >      > To summarize:
>> >      > - The python snippet I posted is still valid and can speed up
>> > your
>> >      > code if you can change all your dot() calls.
>> >      > - The change in dotblas.c is a bit more problematic because it's
>> > very
>> >      > core. I'm having issues right now to replicate the timings, I've
>> > got
>> >      > better timing for a.dot(a.T) than for a.dot(a). There might be a
>> > bug.
>> >      >
>> >      > It's a pain to test because I cannot do the test in a single
>> >     python session.
>> >      > I'm going to try to integrate most of your suggestions, I cannot
>> >      > guarantee I'll have time to do them all though.
>> >      >
>> >      > -nicolas
>> >      > On Fri, Nov 9, 2012 at 8:56 AM, Nathaniel Smith <njs at pobox.com
>> >     <mailto:njs at pobox.com>> wrote:
>> >      >> On Fri, Nov 9, 2012 at 4:25 PM, Gael Varoquaux
>> >      >> <gael.varoquaux at normalesup.org
>> >     <mailto:gael.varoquaux at normalesup.org>> wrote:
>> >      >>> On Fri, Nov 09, 2012 at 03:12:42PM +0000, Nathaniel Smith
>> > wrote:
>> >      >>>> But what if someone compiles numpy against an optimized blas
>> > (mkl,
>> >      >>>> say) and then compiles SciPy against the reference blas? What
>> >     do you
>> >      >>>> do then!? ;-)
>> >      >>>
>> >      >>> This could happen. But the converse happens very often. What
>> >     happens is
>> >      >>> that users (eg on shared computing resource) ask for a
>> >     scientific python
>> >      >>> environment. The administrator than installs the package
>> >     starting from
>> >      >>> the most basic one, to the most advanced one, thus starting
>> >     with numpy
>> >      >>> that can very well build without any external blas. When he
>> >     gets to scipy
>> >      >>> he hits the problem that the build system does not detect
>> >     properly the
>> >      >>> blas, and he solves that problem.
>> >      >>>
>> >      >>> Also, it used to be that on the major linux distributions,
>> >     numpy would not
>> >      >>> be build with an optimize lapack because numpy was in the
>> >     'base' set of
>> >      >>> packages, but not lapack. On the contrary, scipy being in the
>> >     'contrib'
>> >      >>> set, it could depend on lapack. I just checked, and this has
>> >     been fixed
>> >      >>> in the major distributions (Fedora, Debian, Ubuntu).
>> >      >>>
>> >      >>> Now we can discuss with such problems should not happen, and
>> >     put the
>> >      >>> blame on the users/administrators, the fact is that they happen
>> >     often. I
>> >      >>> keep seeing environments in which np.linalg is unreasonnably
>> > slow.
>> >      >>
>> >      >> If this is something that's been a problem for you, maybe we
>> > should
>> >      >> start another thread on things we could do to fix it directly?
>> >     Improve
>> >      >> build instructions, advertise build systems that set up the
>> > whole
>> >      >> environment (and thus do the right thing), make numpy's setup.py
>> >      >> scream and yell if blas isn't available...?
>> >      >>
>> >      >> -n
>> >      >> _______________________________________________
>> >      >> NumPy-Discussion mailing list
>> >      >> NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>> >      >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >     _______________________________________________
>> >     NumPy-Discussion mailing list
>> >     NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>> >     http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> >
>> >
>> >
>> > --
>> > Information System Engineer, Ph.D.
>> > Blog: http://matt.eifelle.com
>> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
>> > Music band: http://liliejay.com/
>> >
>> >
>> >
>> > _______________________________________________
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From njs at pobox.com  Mon Nov 12 15:59:31 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 12 Nov 2012 21:59:31 +0100
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAHD5bXkn8ddzVU1s19cex9zOdkspzTe7=hZXBydKaHrr0e+D+Q@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
	<CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
	<CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
	<CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>
	<509DA207.1000302@astro.uio.no>
	<CAAyCRnS5fSn1JXXCa6goA9VVNxs0wZO86J9S3WquYN7m99=9Rg@mail.gmail.com>
	<CAHD5bXkn8ddzVU1s19cex9zOdkspzTe7=hZXBydKaHrr0e+D+Q@mail.gmail.com>
Message-ID: <CAPJVwBnfFBMCByquApwt6TW5TqHZuN+LXQOhrycbd2YNm08c1Q@mail.gmail.com>

On Mon, Nov 12, 2012 at 9:08 PM, Nicolas SCHEFFER
<scheffer.nicolas at gmail.com> wrote:
> I've pushed my code to a branch here
> https://github.com/leschef/numpy/tree/faster_dot
> with the commit
> https://github.com/leschef/numpy/commit/ea037770e03f23aca1a06274a1a8e8bf0e0e2ee4
>
> Let me know if that's enough to create a pull request.

"Pull request" basically means "this code is good enough to look at"
-- the name is a bit misleading. It just creates a discussion thread
where it's easy to look at the change, comment on pieces, etc., and
you can continue to update the code after you start the pull request.

-n


From njs at pobox.com  Mon Nov 12 16:11:52 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 12 Nov 2012 22:11:52 +0100
Subject: [Numpy-discussion] Do we want scalar casting to behave as it
 does at the moment?
In-Reply-To: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>
References: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>
Message-ID: <CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>

On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> I wanted to check that everyone knows about and is happy with the
> scalar casting changes from 1.6.0.
>
> Specifically, the rules for (array, scalar) casting have changed such
> that the resulting dtype depends on the _value_ of the scalar.
>
> Mark W has documented these changes here:
>
> http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html
>
> Specifically, as of 1.6.0:
>
> In [19]: arr = np.array([1.], dtype=np.float32)
>
> In [20]: (arr + (2**16-1)).dtype
> Out[20]: dtype('float32')
>
> In [21]: (arr + (2**16)).dtype
> Out[21]: dtype('float64')
>
> In [25]: arr = np.array([1.], dtype=np.int8)
>
> In [26]: (arr + 127).dtype
> Out[26]: dtype('int8')
>
> In [27]: (arr + 128).dtype
> Out[27]: dtype('int16')
>
> There's discussion about the changes here:
>
> http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html
> http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html
>
> It seems to me that this change is hard to explain, and does what you
> want only some of the time, making it a false friend.

The old behaviour was that in these cases, the scalar was always cast
to the type of the array, right? So
  np.array([1], dtype=np.int8) + 256
returned 1? Is that the behaviour you prefer?

I agree that the 1.6 behaviour is surprising and somewhat
inconsistent. There are many places where you can get an overflow in
numpy, and in all the other cases we just let the overflow happen. And
in fact you can still get an overflow with arr + scalar operations, so
this doesn't really fix anything.

I find the specific handling of unsigned -> signed and float32 ->
float64 upcasting confusing as well. (Sure, 2**16 isn't exactly
representable as a float32, but it doesn't *overflow*, it just gives
you 2.0**16... if I'm using float32 then I presumably don't care that
much about exact representability, so it's surprising that numpy is
working to enforce it, and definitely a separate decision from what to
do about overflow.)

None of those threads seem to really get into the question of what the
best behaviour here *is*, though.

Possibly the most defensible choice is to treat ufunc(arr, scalar)
operations as performing an implicit cast of the scalar to arr's
dtype, and using the standard implicit casting rules -- which I think
means, raising an error if !can_cast(scalar, arr.dtype,
casting="safe")

-n


From ralf.gommers at gmail.com  Mon Nov 12 16:20:26 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 12 Nov 2012 22:20:26 +0100
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
In-Reply-To: <CAKVJ-_4zJmKcUbmXMFLnzQ+_agPMEct7dUxBRcf6WbF4mPwyDg@mail.gmail.com>
References: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
	<CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
	<CAKVJ-_7q4WW3sD-p3zE8tUTs7y9JaUbzr1MnF5AiYPV=iHy_sQ@mail.gmail.com>
	<CAKVJ-_4XauwyYLED0Gy4xoDeomV+xHUNQ9zpe+QYnr5X4Kwhhw@mail.gmail.com>
	<CAKVJ-_5AhQD9sKLpW5RDA1a6=S_Wcs_ELi5BKSJkNmA2YvYmkw@mail.gmail.com>
	<CABL7CQgjsHYPTyLhgePRNSG88cksJcodM2mA1E-eETvj9ZxWLw@mail.gmail.com>
	<CAKVJ-_44xa3x5ZKp7wdozh5HD0ZL9NskTQ95umdmC+3s3f5h1A@mail.gmail.com>
	<CAKVJ-_4zJmKcUbmXMFLnzQ+_agPMEct7dUxBRcf6WbF4mPwyDg@mail.gmail.com>
Message-ID: <CABL7CQiHvF+GuKuobvzzm9XOeFLbKrbYA563HWUyAMXd90KbBA@mail.gmail.com>

On Mon, Nov 12, 2012 at 1:31 PM, Peter Cock <p.j.a.cock at googlemail.com>wrote:

> On Sun, Nov 11, 2012 at 11:20 PM, Peter Cock <p.j.a.cock at googlemail.com>
> wrote:
> > On Sun, Nov 11, 2012 at 9:05 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >>
> >> Those changes look correct, a PR would be great.
> >>
> >
> > I'll do that later this week - but feel free to do it yourself
> immediately
> > if more convenient.
> >
>
> Hi again Ralf,
>
> OK, new branch from the master here:
> https://github.com/peterjc/numpy/tree/msvc10
>
> The first two commits are the changes already discussed.
>
> https://github.com/peterjc/numpy/commit/d8ead4c83364f9c7d4543690eb12e4543c608372
>
> https://github.com/peterjc/numpy/commit/1d498f54668a9a87286fa31f2779acbb048edd39
>
> >> Fixing the next error also seems straightforward; around line 465 of
> >> mingw32ccompiler a check is needed for 10.0 (not sure which one)
> >> and then a line like
> >>     _MSVCRVER_TO_FULLVER['10'] = "10.0.xxxxx.x"
> >> needs to be added.
> >
> > I'd got that far last night before calling turning in - my question was
> > where do I get the last two parts of the version number. Presumably
> > I can just use the values from the msvcr100.dll on my machine?
>
> The third commit provides a fall back value (based on the DLL on
> my machine) as done for MSVC v8 and v9, and updates the live
> lookup via msvcrt.CRT_ASSEMBLY_VERSION to actually check
> what version that is (the current code wrongly assumes it is going
> to be for MSCV v9):
>
> https://github.com/peterjc/numpy/commit/24523565b5dbb23d6de0591ef2a4c1d014722c5d
>
> Do you want a pull request for that work so far? Or can you
> suggest what I should investigate next since there is at least
> one more hurdle to clear before I can build it (build output below).
>

It would be easier to only have to test this once, so let's try to get it
all resolved.


>
>
> I now get further trying to build with mingw32 (from this branch
> from master, not the numpy-1.7.0b2 files this time), an excerpt
> from which this line may be note-worthy:
>
> Cannot build msvcr library: "msvcr100d.dll" not found
>
> This omits the failure to detect ALTAS etc which seems irrelevant.
> I would guess the MismatchCAPIWarning is an unrelated issue
> from working from the master branch this time.
>

That's indeed unrelated. You get that warning each time something is added
to the C API without an update to the API version number. That version
number update is usually only done right before a release.

>
> C:\repositories\numpy>c:\python33\python setup.py build --compiler=mingw32
> Converting to Python3 via 2to3...
> <snip>
> numpy\core\setup_common.py:86: MismatchCAPIWarning: API mismatch
> detected, the C API version numbers have to be updated. Current C api
> version is 8, with checksum f4362353e2d72f889fda0128aa015037, but
> recorded checksum for C API version 8 in codegen_dir/cversions.txt is
> 17321775fc884de0b1eda478cd61c74b. If functions were added in the C
> API, you have to update C_API_VERSION  in numpy\core\setup_common.py.
> MismatchCAPIWarning)
> <snip>
> running build
> running config_cc
> unifing config_cc, config, build_clib, build_ext, build commands
> --compiler options
> running config_fc
> unifing config_fc, config, build_clib, build_ext, build commands
> --fcompiler options
> running build_src
> build_src
> building py_modules sources
> building library "npymath" sources
> Cannot build msvcr library: "msvcr100d.dll" not found
> Unable to find productdir in registry
> Checking environ VS100COMNTOOLS
> customize GnuFCompiler
> Could not locate executable g77
> Could not locate executable f77
> customize IntelVisualFCompiler
> Could not locate executable ifort
> Could not locate executable ifl
> customize AbsoftFCompiler
> Could not locate executable f90
> customize CompaqVisualFCompiler
> Found executable C:\cygwin\bin\DF.exe
> Found executable C:\cygwin\bin\DF.exe
> customize IntelItaniumVisualFCompiler
> Could not locate executable efl
> customize Gnu95FCompiler
> Could not locate executable gfortran
> Could not locate executable f95
> customize G95FCompiler
> Could not locate executable g95
> customize IntelEM64VisualFCompiler
> customize IntelEM64TFCompiler
> Could not locate executable efort
> Could not locate executable efc
> don't know how to compile Fortran code on platform 'nt'
> C compiler: gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes
>
> compile options: '-DNPY_MINGW_USE_CUSTOM_MSVCR
> -D__MSVCRT_VERSION__=0x1000 -Inumpy\core\src\private -Inumpy\core\src
> -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray
> -Inumpy\core\src\umath -Inumpy\core\src\npysort -Inumpy\core\include
> -Ic:\python33\include -Ic:\python33\include -c'
> gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes
> -DNPY_MINGW_USE_CUSTOM_MSVCR -D__MSVCRT_VERSION__=0x1000
> -Inumpy\core\src\private -Inumpy\core\src -Inumpy\core
> -Inumpy\core\src\npymath -Inumpy\core\src\multiarray
> -Inumpy\core\src\umath -Inumpy\core\src\npysort -Inumpy\core\include
> -Ic:\python33\include -Ic:\python33\include -c _configtest.c -o
> _configtest.o
> Found executable C:\cygwin\usr\bin\gcc.exe
> g++ -mno-cygwin _configtest.o -lmsvcr100 -o _configtest.exe
> Could not locate executable g++
> Executable g++ does not exist
>

A C++ compiler shouldn't be needed for numpy, so it shouldn't try to find
one. On Linux it doesn't try this. It looks like on Windows (with MinGW at
least) it does. I looked at a build log from a compile with MinGW and g++
available (
http://projects.scipy.org/numpy/attachment/ticket/1909/dc6b601-builld.log),
and g++ is used exactly once: for the linking stage of
numpy.core._dummy.pyd.

Looking at core/setup.py that is related to __STDC_FORMAT_MACROS. That led
me to https://github.com/numpy/numpy/commit/64423fdb18e8. So the easy way
forward is to install g++. The hard way is figuring out why that define was
necessary and find a way around it.

Ralf


> failure.
> removing: _configtest.exe.manifest _configtest.c _configtest.o
> Traceback (most recent call last):
>   File "setup.py", line 214, in <module>
>     setup_package()
>   File "setup.py", line 207, in setup_package
>     configuration=configuration )
>   File "C:\repositories\numpy\build\py3k\numpy\distutils\core.py", line
> 186, in
> setup
>     return old_setup(**new_attr)
>   File "c:\python33\lib\distutils\core.py", line 148, in setup
>     dist.run_commands()
>   File "c:\python33\lib\distutils\dist.py", line 917, in run_commands
>     self.run_command(cmd)
>   File "c:\python33\lib\distutils\dist.py", line 936, in run_command
>     cmd_obj.run()
>   File
> "C:\repositories\numpy\build\py3k\numpy\distutils\command\build.py", line
>  37, in run
>     old_build.run(self)
>   File "c:\python33\lib\distutils\command\build.py", line 126, in run
>     self.run_command(cmd_name)
>   File "c:\python33\lib\distutils\cmd.py", line 313, in run_command
>     self.distribution.run_command(command)
>   File "c:\python33\lib\distutils\dist.py", line 936, in run_command
>     cmd_obj.run()
>   File
> "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py",
> line 152, in run
>     self.build_sources()
>   File
> "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py",
> line 163, in build_sources
>     self.build_library_sources(*libname_info)
>   File
> "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py",
> line 298, in build_library_sources
>     sources = self.generate_sources(sources, (lib_name, build_info))
>   File
> "C:\repositories\numpy\build\py3k\numpy\distutils\command\build_src.py",
> line 385, in generate_sources
>     source = func(extension, build_dir)
>   File "numpy\core\setup.py", line 648, in get_mathlib_info
>     raise RuntimeError("Broken toolchain: cannot link a simple C program")
> RuntimeError: Broken toolchain: cannot link a simple C program
>
>
> Peter
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121112/c6bfde2d/attachment.html>

From matthew.brett at gmail.com  Mon Nov 12 16:27:27 2012
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 12 Nov 2012 13:27:27 -0800
Subject: [Numpy-discussion] Do we want scalar casting to behave as it
 does at the moment?
In-Reply-To: <CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>
References: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>
	<CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>
Message-ID: <CAH6Pt5pMLivffPBh2yT+xsqnOEBAJR2NZ=c=AvjfACTGs-bqCA@mail.gmail.com>

Hi,

On Mon, Nov 12, 2012 at 1:11 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Hi,
>>
>> I wanted to check that everyone knows about and is happy with the
>> scalar casting changes from 1.6.0.
>>
>> Specifically, the rules for (array, scalar) casting have changed such
>> that the resulting dtype depends on the _value_ of the scalar.
>>
>> Mark W has documented these changes here:
>>
>> http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html
>>
>> Specifically, as of 1.6.0:
>>
>> In [19]: arr = np.array([1.], dtype=np.float32)
>>
>> In [20]: (arr + (2**16-1)).dtype
>> Out[20]: dtype('float32')
>>
>> In [21]: (arr + (2**16)).dtype
>> Out[21]: dtype('float64')
>>
>> In [25]: arr = np.array([1.], dtype=np.int8)
>>
>> In [26]: (arr + 127).dtype
>> Out[26]: dtype('int8')
>>
>> In [27]: (arr + 128).dtype
>> Out[27]: dtype('int16')
>>
>> There's discussion about the changes here:
>>
>> http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html
>> http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
>> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html
>>
>> It seems to me that this change is hard to explain, and does what you
>> want only some of the time, making it a false friend.
>
> The old behaviour was that in these cases, the scalar was always cast
> to the type of the array, right? So
>   np.array([1], dtype=np.int8) + 256
> returned 1? Is that the behaviour you prefer?

Right.  In that case of course, I'm getting something a bit nasty.
But if you're working with int8, I think you expect to be careful of
overflow.  And you may well not want an automatic and maybe surprising
upcast to int16.

> I agree that the 1.6 behaviour is surprising and somewhat
> inconsistent. There are many places where you can get an overflow in
> numpy, and in all the other cases we just let the overflow happen. And
> in fact you can still get an overflow with arr + scalar operations, so
> this doesn't really fix anything.

Right - it's a half-fix, which seems to me worse than no fix.

> I find the specific handling of unsigned -> signed and float32 ->
> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly
> representable as a float32, but it doesn't *overflow*, it just gives
> you 2.0**16... if I'm using float32 then I presumably don't care that
> much about exact representability, so it's surprising that numpy is
> working to enforce it, and definitely a separate decision from what to
> do about overflow.)
>
> None of those threads seem to really get into the question of what the
> best behaviour here *is*, though.
>
> Possibly the most defensible choice is to treat ufunc(arr, scalar)
> operations as performing an implicit cast of the scalar to arr's
> dtype, and using the standard implicit casting rules -- which I think
> means, raising an error if !can_cast(scalar, arr.dtype,
> casting="safe")

You mean:

In [25]: arr = np.array([1.], dtype=np.int8)

In [27]: arr + 128

ValueError - cannot safely cast 128 to array dtype int8?

That would be a major change.  If I really wanted to do that, would
you then suggest I cast to an array?

arr + np.array([128])

It would be very good to make a well-argued long-term decision,
whatever the chosen outcome.  Maybe this is the place for a partly
retrospective NEP?

Best,

Matthew


From njs at pobox.com  Mon Nov 12 16:34:59 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 12 Nov 2012 22:34:59 +0100
Subject: [Numpy-discussion] Do we want scalar casting to behave as it
 does at the moment?
In-Reply-To: <CAH6Pt5pMLivffPBh2yT+xsqnOEBAJR2NZ=c=AvjfACTGs-bqCA@mail.gmail.com>
References: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>
	<CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>
	<CAH6Pt5pMLivffPBh2yT+xsqnOEBAJR2NZ=c=AvjfACTGs-bqCA@mail.gmail.com>
Message-ID: <CAPJVwBnHcMw4CoHS--k9THbD-LH=U9FAtoLUkG=dYO_S8RNYXA@mail.gmail.com>

On Mon, Nov 12, 2012 at 10:27 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Mon, Nov 12, 2012 at 1:11 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>>> Hi,
>>>
>>> I wanted to check that everyone knows about and is happy with the
>>> scalar casting changes from 1.6.0.
>>>
>>> Specifically, the rules for (array, scalar) casting have changed such
>>> that the resulting dtype depends on the _value_ of the scalar.
>>>
>>> Mark W has documented these changes here:
>>>
>>> http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
>>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
>>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html
>>>
>>> Specifically, as of 1.6.0:
>>>
>>> In [19]: arr = np.array([1.], dtype=np.float32)
>>>
>>> In [20]: (arr + (2**16-1)).dtype
>>> Out[20]: dtype('float32')
>>>
>>> In [21]: (arr + (2**16)).dtype
>>> Out[21]: dtype('float64')
>>>
>>> In [25]: arr = np.array([1.], dtype=np.int8)
>>>
>>> In [26]: (arr + 127).dtype
>>> Out[26]: dtype('int8')
>>>
>>> In [27]: (arr + 128).dtype
>>> Out[27]: dtype('int16')
>>>
>>> There's discussion about the changes here:
>>>
>>> http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html
>>> http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
>>> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html
>>>
>>> It seems to me that this change is hard to explain, and does what you
>>> want only some of the time, making it a false friend.
>>
>> The old behaviour was that in these cases, the scalar was always cast
>> to the type of the array, right? So
>>   np.array([1], dtype=np.int8) + 256
>> returned 1? Is that the behaviour you prefer?
>
> Right.  In that case of course, I'm getting something a bit nasty.
> But if you're working with int8, I think you expect to be careful of
> overflow.  And you may well not want an automatic and maybe surprising
> upcast to int16.
>
>> I agree that the 1.6 behaviour is surprising and somewhat
>> inconsistent. There are many places where you can get an overflow in
>> numpy, and in all the other cases we just let the overflow happen. And
>> in fact you can still get an overflow with arr + scalar operations, so
>> this doesn't really fix anything.
>
> Right - it's a half-fix, which seems to me worse than no fix.
>
>> I find the specific handling of unsigned -> signed and float32 ->
>> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly
>> representable as a float32, but it doesn't *overflow*, it just gives
>> you 2.0**16... if I'm using float32 then I presumably don't care that
>> much about exact representability, so it's surprising that numpy is
>> working to enforce it, and definitely a separate decision from what to
>> do about overflow.)
>>
>> None of those threads seem to really get into the question of what the
>> best behaviour here *is*, though.
>>
>> Possibly the most defensible choice is to treat ufunc(arr, scalar)
>> operations as performing an implicit cast of the scalar to arr's
>> dtype, and using the standard implicit casting rules -- which I think
>> means, raising an error if !can_cast(scalar, arr.dtype,
>> casting="safe")
>
> You mean:
>
> In [25]: arr = np.array([1.], dtype=np.int8)
>
> In [27]: arr + 128
>
> ValueError - cannot safely cast 128 to array dtype int8?
>
> That would be a major change.  If I really wanted to do that, would
> you then suggest I cast to an array?
>
> arr + np.array([128])

No, that will upcast, I think. (It should, anyway -- scalars are a
special case.) Maybe you meant np.array([128], dtype=np.int8)?

Anyway, you'd cast to an int8 scalar:
  arr + np.int8(128)
(or a scalar array, np.array(128, dtype=np.int8), I think would also
count as a scalar for these purposes?)

I don't see how this would be *that* major a change, the change in 1.6
(which silently changed the meaning of people's code) is larger, I
would say :-).

> It would be very good to make a well-argued long-term decision,
> whatever the chosen outcome.  Maybe this is the place for a partly
> retrospective NEP?

Couldn't hurt.

-n


From ondrej.certik at gmail.com  Mon Nov 12 17:27:02 2012
From: ondrej.certik at gmail.com (=?UTF-8?B?T25kxZllaiDEjGVydMOtaw==?=)
Date: Mon, 12 Nov 2012 14:27:02 -0800
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
Message-ID: <CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>

Hi,

On Mon, Nov 5, 2012 at 11:33 PM, Travis Oliphant <travis at continuum.io> wrote:
> Hey all,
>
> Ondrej has been tied up finishing his PhD for the past several weeks.  He is defending his work shortly and should be available to continue to help with the 1.7.0 release around the first of December.    He and I have been in contact during this process, and I've been helping where I can.   Fortunately, other NumPy developers have been active closing tickets and reviewing pull requests which has helped the process substantially.
>
> The release has taken us longer than we expected, but I'm really glad that we've received the bug-reports and issues that we have seen because it will help the 1.7.0 release be a more stable series.   Also, the merging of the Trac issues with Git has exposed over-looked problems as well and will hopefully encourage more Git-focused participation by users.
>
> We are targeting getting the final release of 1.7.0 out by mid December (based on Ondrej's availability).   But, I would like to find out which issues are seen as blockers by people on this list.   I think most of the issues that I had as blockers have been resolved.    If there are no more remaining blockers, then we may be able to accelerate the final release of 1.7.0 to just after Thanksgiving.



I successfully defended my Ph.D. thesis last Thursday, I just need to
do some changes to it and submit it and I am done.
So I started to work on the release again (my apologies that it got
delayed, I had to devote my full attention to finishing my school
first).

Here is a list of issues that need to be fixed before the release:

https://github.com/numpy/numpy/issues?milestone=3&state=open

If anyone wants to help, we just need to get through them and submit a
PR for each, or close it if it doesn't apply anymore.
This is what I am doing now.

Ondrej


From alan.isaac at gmail.com  Mon Nov 12 17:34:23 2012
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Mon, 12 Nov 2012 17:34:23 -0500
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <1352740571.3411.19.camel@sebastian-laptop>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com>
	<1352728782.3411.2.camel@sebastian-laptop>
	<50A12447.6050604@gmail.com>
	<CAPJVwB=_tXi5qMVNJ-ZfBXAPLvkgy1SG6Pnn73-ESvtZ6HcWjA@mail.gmail.com>
	<1352740571.3411.19.camel@sebastian-laptop>
Message-ID: <50A1796F.4010006@gmail.com>

On 11/12/2012 12:16 PM, Sebastian Berg wrote:
> So instead of taking a sequence of length 1, take an element as default.


Sebastien has proposed that np.random.choice return
a single *element* by default, not a 1d array of length 1.
He proposes to associate this with a default value of `size=None`.

The motivation: it is more natural, and in particular,
it would behave more like Python's random.choice by default.

This decision should be made before this function
is part of a release.

Cheers,
Alan



From njs at pobox.com  Mon Nov 12 17:46:10 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 12 Nov 2012 23:46:10 +0100
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <50A1796F.4010006@gmail.com>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com>
	<1352728782.3411.2.camel@sebastian-laptop>
	<50A12447.6050604@gmail.com>
	<CAPJVwB=_tXi5qMVNJ-ZfBXAPLvkgy1SG6Pnn73-ESvtZ6HcWjA@mail.gmail.com>
	<1352740571.3411.19.camel@sebastian-laptop>
	<50A1796F.4010006@gmail.com>
Message-ID: <CAPJVwBkJHcYRpjdrgfNV9S74sO0hhfzjrsfRWXeA+MeqKRTx9w@mail.gmail.com>

On Mon, Nov 12, 2012 at 11:34 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> On 11/12/2012 12:16 PM, Sebastian Berg wrote:
>> So instead of taking a sequence of length 1, take an element as default.
>
> Sebastien has proposed that np.random.choice return
> a single *element* by default, not a 1d array of length 1.
> He proposes to associate this with a default value of `size=None`.
>
> The motivation: it is more natural, and in particular,
> it would behave more like Python's random.choice by default.
>
> This decision should be made before this function
> is part of a release.

I see, so right now we have

  >>> np.random.choice([1, 2, 3])
  array([2])

but you're suggesting

  >>> np.random.choice([1, 2, 3])
  2
  >>> np.random.choice([1, 2, 3], size=1)
  array([2])

That does seem like an obvious improvement to me, since all the other
random functions work that way, e.g.:

In [2]: np.random.normal()
Out[2]: -0.8752867990041713

In [4]: np.random.normal(size=1)
Out[4]: array([ 1.92803487])

Want to make a pull request?

-n


From cournape at gmail.com  Mon Nov 12 17:48:36 2012
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 12 Nov 2012 22:48:36 +0000
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
	<CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
Message-ID: <CAGY4rcVry+_Odd6g45yeMvdzPW8WYKisJ3FsrZ6iM4oQByw_HQ@mail.gmail.com>

On Mon, Nov 12, 2012 at 10:27 PM, Ond?ej ?ert?k <ondrej.certik at gmail.com> wrote:
> Hi,
>
> On Mon, Nov 5, 2012 at 11:33 PM, Travis Oliphant <travis at continuum.io> wrote:
>> Hey all,
>>
>> Ondrej has been tied up finishing his PhD for the past several weeks.  He is defending his work shortly and should be available to continue to help with the 1.7.0 release around the first of December.    He and I have been in contact during this process, and I've been helping where I can.   Fortunately, other NumPy developers have been active closing tickets and reviewing pull requests which has helped the process substantially.
>>
>> The release has taken us longer than we expected, but I'm really glad that we've received the bug-reports and issues that we have seen because it will help the 1.7.0 release be a more stable series.   Also, the merging of the Trac issues with Git has exposed over-looked problems as well and will hopefully encourage more Git-focused participation by users.
>>
>> We are targeting getting the final release of 1.7.0 out by mid December (based on Ondrej's availability).   But, I would like to find out which issues are seen as blockers by people on this list.   I think most of the issues that I had as blockers have been resolved.    If there are no more remaining blockers, then we may be able to accelerate the final release of 1.7.0 to just after Thanksgiving.
>
>
>
> I successfully defended my Ph.D. thesis last Thursday, I just need to
> do some changes to it and submit it and I am done.

Congrats, (almost) Dr. ?ert?k :)

cheers,
David


From jeffspencerd at gmail.com  Mon Nov 12 18:01:58 2012
From: jeffspencerd at gmail.com (Jeffrey Spencer)
Date: Tue, 13 Nov 2012 10:01:58 +1100
Subject: [Numpy-discussion] (no subject)
Message-ID: <CALmFPZ1S8rrYzsdcnXo=hxfTQY-zD=hLkuhU3w728S9V9mGfMw@mail.gmail.com>

 http://www.a1bailbondsmd.com/wp-includes/ugoogle.html


From alan.isaac at gmail.com  Mon Nov 12 18:36:48 2012
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Mon, 12 Nov 2012 18:36:48 -0500
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <CAPJVwBkJHcYRpjdrgfNV9S74sO0hhfzjrsfRWXeA+MeqKRTx9w@mail.gmail.com>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com>
	<1352728782.3411.2.camel@sebastian-laptop>
	<50A12447.6050604@gmail.com>
	<CAPJVwB=_tXi5qMVNJ-ZfBXAPLvkgy1SG6Pnn73-ESvtZ6HcWjA@mail.gmail.com>
	<1352740571.3411.19.camel@sebastian-laptop>
	<50A1796F.4010006@gmail.com>
	<CAPJVwBkJHcYRpjdrgfNV9S74sO0hhfzjrsfRWXeA+MeqKRTx9w@mail.gmail.com>
Message-ID: <50A18810.6090200@gmail.com>

On 11/12/2012 5:46 PM, Nathaniel Smith wrote:
> Want to make a pull request?


Well, I'd be happy to help Sebastien to change the
code, but I'm not a git user.

And I'd have some questions.  E.g., with `size=None`,
couldn't we just call Python's random.choice?  And for
sampling without replacement, wouldn't it be faster to
just call Python's random.sample (rather than
implement this as currently done)?

Alan


From charlesr.harris at gmail.com  Mon Nov 12 18:47:57 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 12 Nov 2012 16:47:57 -0700
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
	<CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
Message-ID: <CAB6mnx+Xjw-Fr0TzfpNZFPmENM14J3mizO5JaRnK9=5u3aks3g@mail.gmail.com>

On Mon, Nov 12, 2012 at 3:27 PM, Ond?ej ?ert?k <ondrej.certik at gmail.com>wrote:

> Hi,
>
> On Mon, Nov 5, 2012 at 11:33 PM, Travis Oliphant <travis at continuum.io>
> wrote:
> > Hey all,
> >
> > Ondrej has been tied up finishing his PhD for the past several weeks.
>  He is defending his work shortly and should be available to continue to
> help with the 1.7.0 release around the first of December.    He and I have
> been in contact during this process, and I've been helping where I can.
> Fortunately, other NumPy developers have been active closing tickets and
> reviewing pull requests which has helped the process substantially.
> >
> > The release has taken us longer than we expected, but I'm really glad
> that we've received the bug-reports and issues that we have seen because it
> will help the 1.7.0 release be a more stable series.   Also, the merging of
> the Trac issues with Git has exposed over-looked problems as well and will
> hopefully encourage more Git-focused participation by users.
> >
> > We are targeting getting the final release of 1.7.0 out by mid December
> (based on Ondrej's availability).   But, I would like to find out which
> issues are seen as blockers by people on this list.   I think most of the
> issues that I had as blockers have been resolved.    If there are no more
> remaining blockers, then we may be able to accelerate the final release of
> 1.7.0 to just after Thanksgiving.
>
>
>
> I successfully defended my Ph.D. thesis last Thursday, I just need to
> do some changes to it and submit it and I am done.
> So I started to work on the release again (my apologies that it got
> delayed, I had to devote my full attention to finishing my school
> first).
>
> Here is a list of issues that need to be fixed before the release:
>
> https://github.com/numpy/numpy/issues?milestone=3&state=open
>
> If anyone wants to help, we just need to get through them and submit a
> PR for each, or close it if it doesn't apply anymore.
> This is what I am doing now.
>
>
Concrats. There are some commits already due for backports. Do you want to
do that, or will it be OK for others to do it? Another possibility would be
to open new PR's for the backports and tag them as 1.7.0.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121112/e13f5f47/attachment.html>

From p.j.a.cock at googlemail.com  Mon Nov 12 19:36:22 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Tue, 13 Nov 2012 00:36:22 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
In-Reply-To: <CABL7CQiHvF+GuKuobvzzm9XOeFLbKrbYA563HWUyAMXd90KbBA@mail.gmail.com>
References: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
	<CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
	<CAKVJ-_7q4WW3sD-p3zE8tUTs7y9JaUbzr1MnF5AiYPV=iHy_sQ@mail.gmail.com>
	<CAKVJ-_4XauwyYLED0Gy4xoDeomV+xHUNQ9zpe+QYnr5X4Kwhhw@mail.gmail.com>
	<CAKVJ-_5AhQD9sKLpW5RDA1a6=S_Wcs_ELi5BKSJkNmA2YvYmkw@mail.gmail.com>
	<CABL7CQgjsHYPTyLhgePRNSG88cksJcodM2mA1E-eETvj9ZxWLw@mail.gmail.com>
	<CAKVJ-_44xa3x5ZKp7wdozh5HD0ZL9NskTQ95umdmC+3s3f5h1A@mail.gmail.com>
	<CAKVJ-_4zJmKcUbmXMFLnzQ+_agPMEct7dUxBRcf6WbF4mPwyDg@mail.gmail.com>
	<CABL7CQiHvF+GuKuobvzzm9XOeFLbKrbYA563HWUyAMXd90KbBA@mail.gmail.com>
Message-ID: <CAKVJ-_5HiLkS-PPC6BY40H5247kKwMVd2FUyGWLpbkd6BACmcA@mail.gmail.com>

On Mon, Nov 12, 2012 at 9:20 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>> ...
>> Found executable C:\cygwin\usr\bin\gcc.exe
>> g++ -mno-cygwin _configtest.o -lmsvcr100 -o _configtest.exe
>> Could not locate executable g++
>> Executable g++ does not exist
>
>
> A C++ compiler shouldn't be needed for numpy, so it shouldn't try to find
> one. On Linux it doesn't try this. It looks like on Windows (with MinGW at
> least) it does. I looked at a build log from a compile with MinGW and g++
> available
> (http://projects.scipy.org/numpy/attachment/ticket/1909/dc6b601-builld.log),
> and g++ is used exactly once: for the linking stage of
> numpy.core._dummy.pyd.
>
> Looking at core/setup.py that is related to __STDC_FORMAT_MACROS. That led
> me to https://github.com/numpy/numpy/commit/64423fdb18e8. So the easy way
> forward is to install g++. The hard way is figuring out why that define was
> necessary and find a way around it.
>
> Ralf

Very strange (wanting a C++ compiler).

I had some trouble updating my cygwin installation (probably in part
due to issues stemming from a change in Windows username) and
gave up and reinstalled cygwin.

This presented a new issue that gcc and g++ wouldn't work from
the Windows cmd prompt (due to being symlinks), so I deleted
the symlinks /usr/bin/gcc[.exe] and /usr/bin/g++[.exe] and just
replaced them with copies of the binaries /usr/bin/gcc-3.exe and
/usr/bin/g++-3.exe - which seems to work. [I don't recall having to
do that before, perhaps a relatively recent regression in cygwin?]

Now, back to numpy, ...

C:\repositories\numpy>c:\python33\python setup.py build --compiler=mingw32
<snip>
gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes
-DNPY_NEEDS_MINGW_TIME_WORKAROUND -DNPY_MINGW_USE_CUSTOM_MSVCR
-D__MSVCRT_VERSION__=0x1000 -Inumpy\core\include
-Ibuild\src.win32-3.3\numpy\core\include/numpy
-Inumpy\core\src\private -Inumpy\core\src -Inumpy\core
-Inumpy\core\src\npymath -Inumpy\core\src\multiarray
-Inumpy\core\src\umath -Inumpy\core\src\npysort -Inumpy\core\include
-Ic:\python33\include -Ic:\python33\include
-Ibuild\src.win32-3.3\numpy\core\src\multiarray
-Ibuild\src.win32-3.3\numpy\core\src\umath -c
numpy\random\mtrand\distributions.c -o
build\temp.win32-3.3\Release\numpy\random\mtrand\distributions.o
g++ -mno-cygwin -shared
build\temp.win32-3.3\Release\numpy\random\mtrand\mtrand.o
build\temp.win32-3.3\Release\numpy\random\mtrand\randomkit.o
build\temp.win32-3.3\Release\numpy\random\mtrand\initarray.o
build\temp.win32-3.3\Release\numpy\random\mtrand\distributions.o
-Lc:\python33\libs -Lc:\python33\PCbuild -Lbuild\temp.win32-3.3
-lpython33 -lmsvcr100 -o build\lib.win32-3.3\numpy\random\mtrand.pyd
running scons
running build_scripts
creating build\scripts.win32-3.3
Creating build\scripts.win32-3.3\f2py.py
  adding 'build\scripts.win32-3.3\f2py.py' to scripts

C:\repositories\numpy>

Horray! It appears to work. I guess I need nose for the tests now...

However, what is the correct way to now install this build? I have a
workaround (given below), but the obvious routes failed:

-------------------------------------------------------------

C:\repositories\numpy>c:\python33\python setup.py install --compiler=mingw32
Converting to Python3 via 2to3...
Running from numpy source directory.
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --compiler not recognized

-------------------------------------------------------------

Yet the only other likely choice of leaving out the compiler option also fails:

C:\repositories\numpy>c:\python33\python setup.py install
Converting to Python3 via 2to3...
Running from numpy source directory.
non-existing path in 'numpy\\distutils': 'site.cfg'
F2PY Version 2
numpy\core\setup_common.py:86: MismatchCAPIWarning: API mismatch detected, the C
 API version numbers have to be updated. Current C api version is 8, with checks
um f4362353e2d72f889fda0128aa015037, but recorded checksum for C API version 8 i
n codegen_dir/cversions.txt is 17321775fc884de0b1eda478cd61c74b. If functions we
re added in the C API, you have to update C_API_VERSION  in numpy\core\setup_com
mon.py.
  MismatchCAPIWarning)
blas_opt_info:
<snip>
running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands
--compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands
--fcompiler options
running build_src
build_src
building py_modules sources
building library "npymath" sources
No module named 'numpy.distutils.msvccompiler' in numpy.distutils;
trying from distutils
Unable to find productdir in registry
Checking environ VS100COMNTOOLS
error: Unable to find vcvarsall.bat

C:\repositories\numpy>

-------------------------------------------------------------

I went with plan (B), creating C:\Python33\Lib\distutils\distutils.cfg
containing:

[build]
compiler=mingw32

Now I don't have to give a compiler switch and distutils will
default to using mingw32 with Python 3.3, and build and
install seem to work.

I've not yet run the numpy tests yet, but I think this means
my github branches are worth merging:

https://github.com/peterjc/numpy/commits/msvc10

Regards,

Peter

P.S. I am really looking forward to official Windows installers for
NumPy on Python 3.3 on Windows...


From ondrej.certik at gmail.com  Mon Nov 12 19:38:02 2012
From: ondrej.certik at gmail.com (=?UTF-8?B?T25kxZllaiDEjGVydMOtaw==?=)
Date: Mon, 12 Nov 2012 16:38:02 -0800
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <CAB6mnx+Xjw-Fr0TzfpNZFPmENM14J3mizO5JaRnK9=5u3aks3g@mail.gmail.com>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
	<CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
	<CAB6mnx+Xjw-Fr0TzfpNZFPmENM14J3mizO5JaRnK9=5u3aks3g@mail.gmail.com>
Message-ID: <CADDwiVAsASC=NTMmhiXYHk9u0EwoN02zW5qwqmgcFqnMTpsswA@mail.gmail.com>

On Mon, Nov 12, 2012 at 3:47 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Mon, Nov 12, 2012 at 3:27 PM, Ond?ej ?ert?k <ondrej.certik at gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Mon, Nov 5, 2012 at 11:33 PM, Travis Oliphant <travis at continuum.io>
>> wrote:
>> > Hey all,
>> >
>> > Ondrej has been tied up finishing his PhD for the past several weeks.
>> > He is defending his work shortly and should be available to continue to help
>> > with the 1.7.0 release around the first of December.    He and I have been
>> > in contact during this process, and I've been helping where I can.
>> > Fortunately, other NumPy developers have been active closing tickets and
>> > reviewing pull requests which has helped the process substantially.
>> >
>> > The release has taken us longer than we expected, but I'm really glad
>> > that we've received the bug-reports and issues that we have seen because it
>> > will help the 1.7.0 release be a more stable series.   Also, the merging of
>> > the Trac issues with Git has exposed over-looked problems as well and will
>> > hopefully encourage more Git-focused participation by users.
>> >
>> > We are targeting getting the final release of 1.7.0 out by mid December
>> > (based on Ondrej's availability).   But, I would like to find out which
>> > issues are seen as blockers by people on this list.   I think most of the
>> > issues that I had as blockers have been resolved.    If there are no more
>> > remaining blockers, then we may be able to accelerate the final release of
>> > 1.7.0 to just after Thanksgiving.
>>
>>
>>
>> I successfully defended my Ph.D. thesis last Thursday, I just need to
>> do some changes to it and submit it and I am done.
>> So I started to work on the release again (my apologies that it got
>> delayed, I had to devote my full attention to finishing my school
>> first).
>>
>> Here is a list of issues that need to be fixed before the release:
>>
>> https://github.com/numpy/numpy/issues?milestone=3&state=open
>>
>> If anyone wants to help, we just need to get through them and submit a
>> PR for each, or close it if it doesn't apply anymore.
>> This is what I am doing now.
>>
>
> Concrats. There are some commits already due for backports. Do you want to
> do that, or will it be OK for others to do it? Another possibility would be
> to open new PR's for the backports and tag them as 1.7.0.

I'll be happy to do that. If you know which commits are those, let me know.
I actually like to open a PR just for the backports as well, to see if tests
pass and to have more eyes look on it to minimize mistakes.

Ondrej


From p.j.a.cock at googlemail.com  Mon Nov 12 19:46:14 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Tue, 13 Nov 2012 00:46:14 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
In-Reply-To: <CAKVJ-_5HiLkS-PPC6BY40H5247kKwMVd2FUyGWLpbkd6BACmcA@mail.gmail.com>
References: <CAKVJ-_7P_+KFBzXdPTzn8r+mLyAs7UdmnuUeNvKPBk0G3Q3ahg@mail.gmail.com>
	<CABL7CQif5P5cePvYQVdUt6_cxek9qDkPnU13sqhTQwungmO-Jw@mail.gmail.com>
	<CAKVJ-_7q4WW3sD-p3zE8tUTs7y9JaUbzr1MnF5AiYPV=iHy_sQ@mail.gmail.com>
	<CAKVJ-_4XauwyYLED0Gy4xoDeomV+xHUNQ9zpe+QYnr5X4Kwhhw@mail.gmail.com>
	<CAKVJ-_5AhQD9sKLpW5RDA1a6=S_Wcs_ELi5BKSJkNmA2YvYmkw@mail.gmail.com>
	<CABL7CQgjsHYPTyLhgePRNSG88cksJcodM2mA1E-eETvj9ZxWLw@mail.gmail.com>
	<CAKVJ-_44xa3x5ZKp7wdozh5HD0ZL9NskTQ95umdmC+3s3f5h1A@mail.gmail.com>
	<CAKVJ-_4zJmKcUbmXMFLnzQ+_agPMEct7dUxBRcf6WbF4mPwyDg@mail.gmail.com>
	<CABL7CQiHvF+GuKuobvzzm9XOeFLbKrbYA563HWUyAMXd90KbBA@mail.gmail.com>
	<CAKVJ-_5HiLkS-PPC6BY40H5247kKwMVd2FUyGWLpbkd6BACmcA@mail.gmail.com>
Message-ID: <CAKVJ-_5_=LhnJOrSUrKOFsny2e=gZ7Bzs1WLEWjE9hr9Hmh2aQ@mail.gmail.com>

On Tue, Nov 13, 2012 at 12:36 AM, Peter Cock <p.j.a.cock at googlemail.com> wrote:
>
> I've not yet run the numpy tests yet, but I think this means
> my github branches are worth merging:
>
> https://github.com/peterjc/numpy/commits/msvc10
>

Hi Ralf,

Pull request filed, assuming this gets applied to the master could
you also flag it for backporting to NumPy 1.7 as well? Thanks:
https://github.com/numpy/numpy/pull/2726

I hope to run the test suite later this week (tomorrow ideally).

Peter


From sebastian at sipsolutions.net  Mon Nov 12 20:18:41 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Tue, 13 Nov 2012 02:18:41 +0100
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <50A18810.6090200@gmail.com>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com> <1352728782.3411.2.camel@sebastian-laptop>
	<50A12447.6050604@gmail.com>
	<CAPJVwB=_tXi5qMVNJ-ZfBXAPLvkgy1SG6Pnn73-ESvtZ6HcWjA@mail.gmail.com>
	<1352740571.3411.19.camel@sebastian-laptop>
	<50A1796F.4010006@gmail.com>
	<CAPJVwBkJHcYRpjdrgfNV9S74sO0hhfzjrsfRWXeA+MeqKRTx9w@mail.gmail.com>
	<50A18810.6090200@gmail.com>
Message-ID: <1352769521.19117.16.camel@sebastian-laptop>

On Mon, 2012-11-12 at 18:36 -0500, Alan G Isaac wrote:
> On 11/12/2012 5:46 PM, Nathaniel Smith wrote:
> > Want to make a pull request?
> 
> 
> Well, I'd be happy to help Sebastien to change the
> code, but I'm not a git user.
> 

I have created a pull request, but tests are still needed... If you like
it would be very nice if you can check it and maybe also write some
tests. Git is relatively simple (and likely worth to learn) but even
otherwise posting code would be nice.

> And I'd have some questions.  E.g., with `size=None`,
> couldn't we just call Python's random.choice?  And for
> sampling without replacement, wouldn't it be faster to
> just call Python's random.sample (rather than
> implement this as currently done)?
> 

I don't think the difference should be really noticeable. But even if, I
doubt its worth special casing. If someone cares a lot about speed they
probably should not use it to get single values anyway. Also for this
case of random numbers, it would be a bad idea since you would use a
different random number generator and a different seed!

Regards,

Sebastian

> Alan
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From scheffer.nicolas at gmail.com  Mon Nov 12 20:39:55 2012
From: scheffer.nicolas at gmail.com (Nicolas SCHEFFER)
Date: Mon, 12 Nov 2012 17:39:55 -0800
Subject: [Numpy-discussion] Scipy dot
In-Reply-To: <CAPJVwBnfFBMCByquApwt6TW5TqHZuN+LXQOhrycbd2YNm08c1Q@mail.gmail.com>
References: <CAHD5bXmCdv6-v-8DOfiau1OcO0F=LxWCdtOuJSg1+u-Kmr3oKw@mail.gmail.com>
	<CAPJVwBmGxP4YV9MCCyCUVoPsoK52=gguAm0n0vm2kvir15qnNQ@mail.gmail.com>
	<20121108120725.GL313@phare.normalesup.org>
	<509BA198.4070301@astro.uio.no>
	<CAGY4rcWpxHNLbr1Cg_roTAe2Ed_1CMh9+8gpLybLVbZKjUTYRg@mail.gmail.com>
	<CAPk-6T75ZPZz4Rey3xEpXCc14=a-1MjRZBHVojxcWdnWmvwpAQ@mail.gmail.com>
	<20121109084826.GB19371@phare.normalesup.org>
	<CAPJVwB=HQ824RxbQej4D0pFBpr3vpavTC+dHhVX1z1BsXu1yqA@mail.gmail.com>
	<20121109162550.GQ19465@phare.normalesup.org>
	<CAPJVwBkNYgbH1f-kcWzoPZnnXFKuEdcD_QRTFjCEmd7TvBRLHg@mail.gmail.com>
	<CAHD5bXmTWuLAnjQPW+_tcrHXZDdVAiuAVVN0m5QrZ2oYqFy-2Q@mail.gmail.com>
	<CAHD5bXn6=A7b1xbeyJcbQRbGnj_bJntQ5ubqy-dHBJ7RKPObRw@mail.gmail.com>
	<CAHCaCkK+nqXZtaEZsd1ZVpSy5aMqnHjaQLRifU+jWKbhH4AQVw@mail.gmail.com>
	<509DA207.1000302@astro.uio.no>
	<CAAyCRnS5fSn1JXXCa6goA9VVNxs0wZO86J9S3WquYN7m99=9Rg@mail.gmail.com>
	<CAHD5bXkn8ddzVU1s19cex9zOdkspzTe7=hZXBydKaHrr0e+D+Q@mail.gmail.com>
	<CAPJVwBnfFBMCByquApwt6TW5TqHZuN+LXQOhrycbd2YNm08c1Q@mail.gmail.com>
Message-ID: <CAHD5bX=4ekO2jK=QeXYZxNdgkfoR=y_PeR22kSRXURiivQKd4g@mail.gmail.com>

Yep exactly.

I just want to make sure that we talked enough on the principle first
(ie. goals and technical approach), and that indeed the code is good
enough to look at.

I get it from your answer that it is, so I went ahead
https://github.com/numpy/numpy/pull/2730

Thanks

-nicolas

On Mon, Nov 12, 2012 at 12:59 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Mon, Nov 12, 2012 at 9:08 PM, Nicolas SCHEFFER
> <scheffer.nicolas at gmail.com> wrote:
>> I've pushed my code to a branch here
>> https://github.com/leschef/numpy/tree/faster_dot
>> with the commit
>> https://github.com/leschef/numpy/commit/ea037770e03f23aca1a06274a1a8e8bf0e0e2ee4
>>
>> Let me know if that's enough to create a pull request.
>
> "Pull request" basically means "this code is good enough to look at"
> -- the name is a bit misleading. It just creates a discussion thread
> where it's easy to look at the change, comment on pieces, etc., and
> you can continue to update the code after you start the pull request.
>
> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From shish at keba.be  Mon Nov 12 21:21:21 2012
From: shish at keba.be (Olivier Delalleau)
Date: Mon, 12 Nov 2012 21:21:21 -0500
Subject: [Numpy-discussion] Do we want scalar casting to behave as it
 does at the moment?
In-Reply-To: <CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>
References: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>
	<CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>
Message-ID: <CAFXk4boMkSgGJmHHcjUoERYt9_-dMk4CuqKG1ULEOy1cW5q0vA@mail.gmail.com>

2012/11/12 Nathaniel Smith <njs at pobox.com>

> On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> > Hi,
> >
> > I wanted to check that everyone knows about and is happy with the
> > scalar casting changes from 1.6.0.
> >
> > Specifically, the rules for (array, scalar) casting have changed such
> > that the resulting dtype depends on the _value_ of the scalar.
> >
> > Mark W has documented these changes here:
> >
> > http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
> >
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
> >
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html
> >
> > Specifically, as of 1.6.0:
> >
> > In [19]: arr = np.array([1.], dtype=np.float32)
> >
> > In [20]: (arr + (2**16-1)).dtype
> > Out[20]: dtype('float32')
> >
> > In [21]: (arr + (2**16)).dtype
> > Out[21]: dtype('float64')
> >
> > In [25]: arr = np.array([1.], dtype=np.int8)
> >
> > In [26]: (arr + 127).dtype
> > Out[26]: dtype('int8')
> >
> > In [27]: (arr + 128).dtype
> > Out[27]: dtype('int16')
> >
> > There's discussion about the changes here:
> >
> >
> http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html
> > http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
> >
> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html
> >
> > It seems to me that this change is hard to explain, and does what you
> > want only some of the time, making it a false friend.
>
> The old behaviour was that in these cases, the scalar was always cast
> to the type of the array, right? So
>   np.array([1], dtype=np.int8) + 256
> returned 1? Is that the behaviour you prefer?
>
> I agree that the 1.6 behaviour is surprising and somewhat
> inconsistent. There are many places where you can get an overflow in
> numpy, and in all the other cases we just let the overflow happen. And
> in fact you can still get an overflow with arr + scalar operations, so
> this doesn't really fix anything.
>
> I find the specific handling of unsigned -> signed and float32 ->
> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly
> representable as a float32, but it doesn't *overflow*, it just gives
> you 2.0**16... if I'm using float32 then I presumably don't care that
> much about exact representability, so it's surprising that numpy is
> working to enforce it, and definitely a separate decision from what to
> do about overflow.)
>
> None of those threads seem to really get into the question of what the
> best behaviour here *is*, though.
>
> Possibly the most defensible choice is to treat ufunc(arr, scalar)
> operations as performing an implicit cast of the scalar to arr's
> dtype, and using the standard implicit casting rules -- which I think
> means, raising an error if !can_cast(scalar, arr.dtype,
> casting="safe")


I like this suggestion. It may break some existing code, but I think it'd
be for the best. The current behavior can be very confusing.

-=- Olivier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121112/76ebdb82/attachment.html>

From Robert.Lagace at fsaa.ulaval.ca  Mon Nov 12 21:32:34 2012
From: Robert.Lagace at fsaa.ulaval.ca (=?utf-8?B?Um9iZXJ0IExhZ2Fjw6k=?=)
Date: Mon, 12 Nov 2012 21:32:34 -0500
Subject: [Numpy-discussion] RE :  1.7.0 release
In-Reply-To: <CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>,
	<CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
Message-ID: <53541A1E9E53A4449C6558A1B0FEAE5A106132B5AF@EXCH-MBX-C.ulaval.ca>

Congratulation.
________________________________________
De : numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] de la part de Ond?ej ?ert?k [ondrej.certik at gmail.com]
Date d'envoi : 12 novembre 2012 17:27
? : Discussion of Numerical Python
Objet : Re: [Numpy-discussion] 1.7.0 release

Hi,

On Mon, Nov 5, 2012 at 11:33 PM, Travis Oliphant <travis at continuum.io> wrote:
> Hey all,
>
> Ondrej has been tied up finishing his PhD for the past several weeks.  He is defending his work shortly and should be available to continue to help with the 1.7.0 release around the first of December.    He and I have been in contact during this process, and I've been helping where I can.   Fortunately, other NumPy developers have been active closing tickets and reviewing pull requests which has helped the process substantially.
>
> The release has taken us longer than we expected, but I'm really glad that we've received the bug-reports and issues that we have seen because it will help the 1.7.0 release be a more stable series.   Also, the merging of the Trac issues with Git has exposed over-looked problems as well and will hopefully encourage more Git-focused participation by users.
>
> We are targeting getting the final release of 1.7.0 out by mid December (based on Ondrej's availability).   But, I would like to find out which issues are seen as blockers by people on this list.   I think most of the issues that I had as blockers have been resolved.    If there are no more remaining blockers, then we may be able to accelerate the final release of 1.7.0 to just after Thanksgiving.



I successfully defended my Ph.D. thesis last Thursday, I just need to
do some changes to it and submit it and I am done.
So I started to work on the release again (my apologies that it got
delayed, I had to devote my full attention to finishing my school
first).

Here is a list of issues that need to be fixed before the release:

https://github.com/numpy/numpy/issues?milestone=3&state=open

If anyone wants to help, we just need to get through them and submit a
PR for each, or close it if it doesn't apply anymore.
This is what I am doing now.

Ondrej
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion at scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

From alan.isaac at gmail.com  Mon Nov 12 22:44:58 2012
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Mon, 12 Nov 2012 22:44:58 -0500
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <1352769521.19117.16.camel@sebastian-laptop>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com>
	<1352728782.3411.2.camel@sebastian-laptop>
	<50A12447.6050604@gmail.com>
	<CAPJVwB=_tXi5qMVNJ-ZfBXAPLvkgy1SG6Pnn73-ESvtZ6HcWjA@mail.gmail.com>
	<1352740571.3411.19.camel@sebastian-laptop>
	<50A1796F.4010006@gmail.com>
	<CAPJVwBkJHcYRpjdrgfNV9S74sO0hhfzjrsfRWXeA+MeqKRTx9w@mail.gmail.com>
	<50A18810.6090200@gmail.com>
	<1352769521.19117.16.camel@sebastian-laptop>
Message-ID: <50A1C23A.5010506@gmail.com>

On 11/12/2012 8:18 PM, Sebastian Berg wrote:
> I have created a pull request


This is still a bit different than I thought you intended.
With `size=None` we don't get an element,
but rather a 0d array.

I thought the idea was to return an element in this case?

Alan



From ben.root at ou.edu  Mon Nov 12 23:15:48 2012
From: ben.root at ou.edu (Benjamin Root)
Date: Mon, 12 Nov 2012 23:15:48 -0500
Subject: [Numpy-discussion] Do we want scalar casting to behave as it
 does at the moment?
In-Reply-To: <CAFXk4boMkSgGJmHHcjUoERYt9_-dMk4CuqKG1ULEOy1cW5q0vA@mail.gmail.com>
References: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>
	<CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>
	<CAFXk4boMkSgGJmHHcjUoERYt9_-dMk4CuqKG1ULEOy1cW5q0vA@mail.gmail.com>
Message-ID: <CANNq6F=XH6Hj9yhWrwid8FeF7JDvdYyRSnM0nwZ6ao1u1_cCOA@mail.gmail.com>

On Monday, November 12, 2012, Olivier Delalleau wrote:

> 2012/11/12 Nathaniel Smith <njs at pobox.com <javascript:_e({}, 'cvml',
> 'njs at pobox.com');>>
>
>> On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <matthew.brett at gmail.com<javascript:_e({}, 'cvml', 'matthew.brett at gmail.com');>>
>> wrote:
>> > Hi,
>> >
>> > I wanted to check that everyone knows about and is happy with the
>> > scalar casting changes from 1.6.0.
>> >
>> > Specifically, the rules for (array, scalar) casting have changed such
>> > that the resulting dtype depends on the _value_ of the scalar.
>> >
>> > Mark W has documented these changes here:
>> >
>> > http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
>> >
>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
>> >
>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html
>> >
>> > Specifically, as of 1.6.0:
>> >
>> > In [19]: arr = np.array([1.], dtype=np.float32)
>> >
>> > In [20]: (arr + (2**16-1)).dtype
>> > Out[20]: dtype('float32')
>> >
>> > In [21]: (arr + (2**16)).dtype
>> > Out[21]: dtype('float64')
>> >
>> > In [25]: arr = np.array([1.], dtype=np.int8)
>> >
>> > In [26]: (arr + 127).dtype
>> > Out[26]: dtype('int8')
>> >
>> > In [27]: (arr + 128).dtype
>> > Out[27]: dtype('int16')
>> >
>> > There's discussion about the changes here:
>> >
>> >
>> http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html
>> > http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
>> >
>> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html
>> >
>> > It seems to me that this change is hard to explain, and does what you
>> > want only some of the time, making it a false friend.
>>
>> The old behaviour was that in these cases, the scalar was always cast
>> to the type of the array, right? So
>>   np.array([1], dtype=np.int8) + 256
>> returned 1? Is that the behaviour you prefer?
>>
>> I agree that the 1.6 behaviour is surprising and somewhat
>> inconsistent. There are many places where you can get an overflow in
>> numpy, and in all the other cases we just let the overflow happen. And
>> in fact you can still get an overflow with arr + scalar operations, so
>> this doesn't really fix anything.
>>
>> I find the specific handling of unsigned -> signed and float32 ->
>> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly
>> representable as a float32, but it doesn't *overflow*, it just gives
>> you 2.0**16... if I'm using float32 then I presumably don't care that
>> much about exact representability, so it's surprising that numpy is
>> working to enforce it, and definitely a separate decision from what to
>> do about overflow.)
>>
>> None of those threads seem to really get into the question of what the
>> best behaviour here *is*, though.
>>
>> Possibly the most defensible choice is to treat ufunc(arr, scalar)
>> operations as performing an implicit cast of the scalar to arr's
>> dtype, and using the standard implicit casting rules -- which I think
>> means, raising an error if !can_cast(scalar, arr.dtype,
>> casting="safe")
>
>
> I like this suggestion. It may break some existing code, but I think it'd
> be for the best. The current behavior can be very confusing.
>
> -=- Olivier
>


"break some existing code"

I really should set up an email filter for this phrase and have it send
back an email automatically: "Are you nuts?!"

We just resolved an issue where the "safe" casting rule unexpectedly broke
existing code with regards to unplaced operations.  The solution was to
warn about the change in the upcoming release and to throw errors in a
later release.  Playing around with fundemental things like this need to be
done methodically and carefully.

Cheers!
Ben Root
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121112/c767bf3b/attachment.html>

From ben.root at ou.edu  Mon Nov 12 23:17:11 2012
From: ben.root at ou.edu (Benjamin Root)
Date: Mon, 12 Nov 2012 23:17:11 -0500
Subject: [Numpy-discussion] Do we want scalar casting to behave as it
 does at the moment?
In-Reply-To: <CANNq6F=XH6Hj9yhWrwid8FeF7JDvdYyRSnM0nwZ6ao1u1_cCOA@mail.gmail.com>
References: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>
	<CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>
	<CAFXk4boMkSgGJmHHcjUoERYt9_-dMk4CuqKG1ULEOy1cW5q0vA@mail.gmail.com>
	<CANNq6F=XH6Hj9yhWrwid8FeF7JDvdYyRSnM0nwZ6ao1u1_cCOA@mail.gmail.com>
Message-ID: <CANNq6FkxV14yG8hY8xqa6xD7DwOG_uF+sAAufxhztpnusUeGrg@mail.gmail.com>

On Monday, November 12, 2012, Benjamin Root wrote:

>
>
> On Monday, November 12, 2012, Olivier Delalleau wrote:
>
>> 2012/11/12 Nathaniel Smith <njs at pobox.com>
>>
>>> On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <matthew.brett at gmail.com>
>>> wrote:
>>> > Hi,
>>> >
>>> > I wanted to check that everyone knows about and is happy with the
>>> > scalar casting changes from 1.6.0.
>>> >
>>> > Specifically, the rules for (array, scalar) casting have changed such
>>> > that the resulting dtype depends on the _value_ of the scalar.
>>> >
>>> > Mark W has documented these changes here:
>>> >
>>> > http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
>>> >
>>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
>>> >
>>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html
>>> >
>>> > Specifically, as of 1.6.0:
>>> >
>>> > In [19]: arr = np.array([1.], dtype=np.float32)
>>> >
>>> > In [20]: (arr + (2**16-1)).dtype
>>> > Out[20]: dtype('float32')
>>> >
>>> > In [21]: (arr + (2**16)).dtype
>>> > Out[21]: dtype('float64')
>>> >
>>> > In [25]: arr = np.array([1.], dtype=np.int8)
>>> >
>>> > In [26]: (arr + 127).dtype
>>> > Out[26]: dtype('int8')
>>> >
>>> > In [27]: (arr + 128).dtype
>>> > Out[27]: dtype('int16')
>>> >
>>> > There's discussion about the changes here:
>>> >
>>> >
>>> http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html
>>> >
>>> http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
>>> >
>>> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html
>>> >
>>> > It seems to me that this change is hard to explain, and does what you
>>> > want only some of the time, making it a false friend.
>>>
>>> The old behaviour was that in these cases, the scalar was always cast
>>> to the type of the array, right? So
>>>   np.array([1], dtype=np.int8) + 256
>>> returned 1? Is that the behaviour you prefer?
>>>
>>> I agree that the 1.6 behaviour is surprising and somewhat
>>> inconsistent. There are many places where you can get an overflow in
>>> numpy, and in all the other cases we just let the overflow happen. And
>>> in fact you can still get an overflow with arr + scalar operations, so
>>> this doesn't really fix anything.
>>>
>>> I find the specific handling of unsigned -> signed and float32 ->
>>> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly
>>> representable as a float32, but it doesn't *overflow*, it just gives
>>> you 2.0**16... if I'm using float32 then I presumably don't care that
>>> much about exact representability, so it's surprising that numpy is
>>> working to enforce it, and definitely a separate decision from what to
>>> do about overflow.)
>>>
>>> None of those threads seem to really get into the question of what the
>>> best behaviour here *is*, though.
>>>
>>> Possibly the most defensible choice is to treat ufunc(arr, scalar)
>>> operations as performing an implicit cast of the scalar to arr's
>>> dtype, and using the standard implicit casting rules -- which I think
>>> means, raising an error if !can_cast(scalar, arr.dtype,
>>> casting="safe")
>>
>>
>> I like this suggestion. It may break some existing code, but I think it'd
>> be for the best. The current behavior can be very confusing.
>>
>> -=- Olivier
>>
>
>
> "break some existing code"
>
> I really should set up an email filter for this phrase and have it send
> back an email automatically: "Are you nuts?!"
>
> We just resolved an issue where the "safe" casting rule unexpectedly broke
> existing code with regards to unplaced operations.  The solution was to
> warn about the change in the upcoming release and to throw errors in a
> later release.  Playing around with fundemental things like this need to be
> done methodically and carefully.
>
> Cheers!
> Ben Root
>


Stupid autocorrect:  unplaced --> inplace
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121112/5e3ab0d7/attachment.html>

From matthew.brett at gmail.com  Mon Nov 12 23:39:29 2012
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 12 Nov 2012 20:39:29 -0800
Subject: [Numpy-discussion] Do we want scalar casting to behave as it
 does at the moment?
In-Reply-To: <CANNq6F=XH6Hj9yhWrwid8FeF7JDvdYyRSnM0nwZ6ao1u1_cCOA@mail.gmail.com>
References: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>
	<CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>
	<CAFXk4boMkSgGJmHHcjUoERYt9_-dMk4CuqKG1ULEOy1cW5q0vA@mail.gmail.com>
	<CANNq6F=XH6Hj9yhWrwid8FeF7JDvdYyRSnM0nwZ6ao1u1_cCOA@mail.gmail.com>
Message-ID: <CAH6Pt5qeXRB-ZfqLh-07DAT1zL6mM=Zqhcjcbd63MGPXfLw+1g@mail.gmail.com>

Hi,

On Mon, Nov 12, 2012 at 8:15 PM, Benjamin Root <ben.root at ou.edu> wrote:
>
>
> On Monday, November 12, 2012, Olivier Delalleau wrote:
>>
>> 2012/11/12 Nathaniel Smith <njs at pobox.com>
>>>
>>> On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <matthew.brett at gmail.com>
>>> wrote:
>>> > Hi,
>>> >
>>> > I wanted to check that everyone knows about and is happy with the
>>> > scalar casting changes from 1.6.0.
>>> >
>>> > Specifically, the rules for (array, scalar) casting have changed such
>>> > that the resulting dtype depends on the _value_ of the scalar.
>>> >
>>> > Mark W has documented these changes here:
>>> >
>>> > http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
>>> >
>>> > http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
>>> >
>>> > http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html
>>> >
>>> > Specifically, as of 1.6.0:
>>> >
>>> > In [19]: arr = np.array([1.], dtype=np.float32)
>>> >
>>> > In [20]: (arr + (2**16-1)).dtype
>>> > Out[20]: dtype('float32')
>>> >
>>> > In [21]: (arr + (2**16)).dtype
>>> > Out[21]: dtype('float64')
>>> >
>>> > In [25]: arr = np.array([1.], dtype=np.int8)
>>> >
>>> > In [26]: (arr + 127).dtype
>>> > Out[26]: dtype('int8')
>>> >
>>> > In [27]: (arr + 128).dtype
>>> > Out[27]: dtype('int16')
>>> >
>>> > There's discussion about the changes here:
>>> >
>>> >
>>> > http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html
>>> > http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
>>> >
>>> > http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html
>>> >
>>> > It seems to me that this change is hard to explain, and does what you
>>> > want only some of the time, making it a false friend.
>>>
>>> The old behaviour was that in these cases, the scalar was always cast
>>> to the type of the array, right? So
>>>   np.array([1], dtype=np.int8) + 256
>>> returned 1? Is that the behaviour you prefer?
>>>
>>> I agree that the 1.6 behaviour is surprising and somewhat
>>> inconsistent. There are many places where you can get an overflow in
>>> numpy, and in all the other cases we just let the overflow happen. And
>>> in fact you can still get an overflow with arr + scalar operations, so
>>> this doesn't really fix anything.
>>>
>>> I find the specific handling of unsigned -> signed and float32 ->
>>> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly
>>> representable as a float32, but it doesn't *overflow*, it just gives
>>> you 2.0**16... if I'm using float32 then I presumably don't care that
>>> much about exact representability, so it's surprising that numpy is
>>> working to enforce it, and definitely a separate decision from what to
>>> do about overflow.)
>>>
>>> None of those threads seem to really get into the question of what the
>>> best behaviour here *is*, though.
>>>
>>> Possibly the most defensible choice is to treat ufunc(arr, scalar)
>>> operations as performing an implicit cast of the scalar to arr's
>>> dtype, and using the standard implicit casting rules -- which I think
>>> means, raising an error if !can_cast(scalar, arr.dtype,
>>> casting="safe")
>>
>>
>> I like this suggestion. It may break some existing code, but I think it'd
>> be for the best. The current behavior can be very confusing.
>>
>> -=- Olivier
>
>
>
> "break some existing code"
>
> I really should set up an email filter for this phrase and have it send back
> an email automatically: "Are you nuts?!"

Well, hold on though, I was asking earlier in the thread what we
thought the behavior should be in 2.0 or maybe better put, sometime in
the future.

If we know what we think the best answer is, and we think the best
answer is worth shooting for, then we can try to think of sensible
ways of getting there.

I guess that's what Nathaniel and Olivier were thinking of but they
can correct me if I'm wrong...

Cheers,

Matthew


From ben.root at ou.edu  Tue Nov 13 00:13:17 2012
From: ben.root at ou.edu (Benjamin Root)
Date: Tue, 13 Nov 2012 00:13:17 -0500
Subject: [Numpy-discussion] Do we want scalar casting to behave as it
 does at the moment?
In-Reply-To: <CAH6Pt5qeXRB-ZfqLh-07DAT1zL6mM=Zqhcjcbd63MGPXfLw+1g@mail.gmail.com>
References: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>
	<CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>
	<CAFXk4boMkSgGJmHHcjUoERYt9_-dMk4CuqKG1ULEOy1cW5q0vA@mail.gmail.com>
	<CANNq6F=XH6Hj9yhWrwid8FeF7JDvdYyRSnM0nwZ6ao1u1_cCOA@mail.gmail.com>
	<CAH6Pt5qeXRB-ZfqLh-07DAT1zL6mM=Zqhcjcbd63MGPXfLw+1g@mail.gmail.com>
Message-ID: <CANNq6Fn-fwef7koGZTUDo=M=2tkRU4DzY-f62fSwOAZAEnWfMQ@mail.gmail.com>

On Monday, November 12, 2012, Matthew Brett wrote:

> Hi,
>
> On Mon, Nov 12, 2012 at 8:15 PM, Benjamin Root <ben.root at ou.edu> wrote:
> >
> >
> > On Monday, November 12, 2012, Olivier Delalleau wrote:
> >>
> >> 2012/11/12 Nathaniel Smith <njs at pobox.com>
> >>>
> >>> On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <
> matthew.brett at gmail.com>
> >>> wrote:
> >>> > Hi,
> >>> >
> >>> > I wanted to check that everyone knows about and is happy with the
> >>> > scalar casting changes from 1.6.0.
> >>> >
> >>> > Specifically, the rules for (array, scalar) casting have changed such
> >>> > that the resulting dtype depends on the _value_ of the scalar.
> >>> >
> >>> > Mark W has documented these changes here:
> >>> >
> >>> > http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
> >>> >
> >>> >
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
> >>> >
> >>> >
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html
> >>> >
> >>> > Specifically, as of 1.6.0:
> >>> >
> >>> > In [19]: arr = np.array([1.], dtype=np.float32)
> >>> >
> >>> > In [20]: (arr + (2**16-1)).dtype
> >>> > Out[20]: dtype('float32')
> >>> >
> >>> > In [21]: (arr + (2**16)).dtype
> >>> > Out[21]: dtype('float64')
> >>> >
> >>> > In [25]: arr = np.array([1.], dtype=np.int8)
> >>> >
> >>> > In [26]: (arr + 127).dtype
> >>> > Out[26]: dtype('int8')
> >>> >
> >>> > In [27]: (arr + 128).dtype
> >>> > Out[27]: dtype('int16')
> >>> >
> >>> > There's discussion about the changes here:
> >>> >
> >>> >
> >>> >
> http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html
> >>> >
> http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
> >>> >
> >>> >
> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html
> >>> >
> >>> > It seems to me that this change is hard to explain, and does what you
> >>> > want only some of the time, making it a false friend.
> >>>
> >>> The old behaviour was that in these cases, the scalar was always cast
> >>> to the type of the array, right? So
> >>>   np.array([1], dtype=np.int8) + 256
> >>> returned 1? Is that the behaviour you prefer?
> >>>
> >>> I agree that the 1.6 behaviour is surprising and somewhat
> >>> inconsistent. There are many places where you can get an overflow in
> >>> numpy, and in all the other cases we just let the overflow happen. And
> >>> in fact you can still get an overflow with arr + scalar operations, so
> >>> this doesn't really fix anything.
> >>>
> >>> I find the specific handling of unsigned -> signed and float32 ->
> >>> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly
> >>> representable as a float32, but it doesn't *overflow*, it just gives
> >>> you 2.0**16... if I'm using float32 then I presumably don't care that
> >>> much about exact representability, so it's surprising that numpy is
> >>> working to enforce it, and definitely a separate decision from what to
> >>> do about overflow.)
> >>>
> >>> None of those threads seem to really get into the question of what the
> >>> best behaviour here *is*, though.
> >>>
> >>> Possibly the moWell, hold on though, I was asking earlier in the
> thread what we
> thought the behavior should be in 2.0 or maybe better put, sometime in
> the future.
>
> If we know what we think the best answer is, and we think the best
> answer is worth shooting for, then we can try to think of sensible
> ways of getting there.
>
> I guess that's what Nathaniel and Olivier were thinking of but they
> can correct me if I'm wrong...
>
> Cheers,
>
> Matthew


I am fine with migrating to better solutions (I have yet to decide on this
current situation, though), but whatever change is adopted must go through
a deprecation process, which was my point.  Outright breaking of code as a
first step is the wrong choice, and I was merely nipping it in the bud.

Cheers!
Ben Root
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121113/2aa3c475/attachment.html>

From austin.bingham at gmail.com  Tue Nov 13 02:26:14 2012
From: austin.bingham at gmail.com (Austin Bingham)
Date: Tue, 13 Nov 2012 08:26:14 +0100
Subject: [Numpy-discussion] Numpy's policy for releasing memory
Message-ID: <CAEZidR0ERxkW8c2DxvQ-iWfSE54Qvf_jeOCk0uzyOY50NOCv6g@mail.gmail.com>

I'm trying to understand how numpy decides when to release memory and
whether it's possible to exert any control over that. The situation is that
I'm profiling memory usage on a system in which a great deal of the overall
memory is tied up in ndarrays. Since numpy manages ndarray memory on its
own (i.e. without the python gc, or so it seems), I'm finding that I can't
do much to convince numpy to release memory when things get tight. For
python object, for example, I can explicitly run gc.collect().

So, in an effort to at least understand the system better, can anyone tell
me how/when numpy decides to release memory? And is there any way via
either the Python or C-API to explicitly request release? Thanks.

Austin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121113/e9cddc7d/attachment.html>

From njs at pobox.com  Tue Nov 13 03:41:37 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 13 Nov 2012 09:41:37 +0100
Subject: [Numpy-discussion] Numpy's policy for releasing memory
In-Reply-To: <CAEZidR0ERxkW8c2DxvQ-iWfSE54Qvf_jeOCk0uzyOY50NOCv6g@mail.gmail.com>
References: <CAEZidR0ERxkW8c2DxvQ-iWfSE54Qvf_jeOCk0uzyOY50NOCv6g@mail.gmail.com>
Message-ID: <CAPJVwB=3uwU-dfZsv4wbN=ggYzxiLVXGY1nrizyy7nFKGyfyqw@mail.gmail.com>

On Tue, Nov 13, 2012 at 8:26 AM, Austin Bingham
<austin.bingham at gmail.com> wrote:
> I'm trying to understand how numpy decides when to release memory and
> whether it's possible to exert any control over that. The situation is that
> I'm profiling memory usage on a system in which a great deal of the overall
> memory is tied up in ndarrays. Since numpy manages ndarray memory on its own
> (i.e. without the python gc, or so it seems), I'm finding that I can't do
> much to convince numpy to release memory when things get tight. For python
> object, for example, I can explicitly run gc.collect().
>
> So, in an effort to at least understand the system better, can anyone tell
> me how/when numpy decides to release memory? And is there any way via either
> the Python or C-API to explicitly request release? Thanks.

Numpy array memory is released when the corresponding Python objects
are deleted, so it exactly follows Python's rules. You can't
explicitly request release, because by definition, if memory is not
released, then it means that it's still accessible somehow, so
releasing it could create segfaults. Perhaps you have stray references
sitting around that you have forgotten to clear -- that's a common
cause of memory leaks in Python. gc.get_referrers() can be useful to
debug such things.

Some things to note:
- Numpy uses malloc() instead of going through the Python low-level
memory allocation layer (which itself is a wrapper around malloc with
various optimizations for small objects). This is really only relevant
because it might create some artifacts depending on how your memory
profiler gathers data.
- gc.collect() doesn't do that much in Python... it only matters if
you have circular references. Mostly Python releases the memory
associated with objects as soon as the object becomes unreferenced.
You could try avoiding circular references, and then gc.collect()
won't even do anything.
- If you have multiple views of the same memory in numpy, then they
share the same underlying memory, so that memory won't be released
until all of the views objects are released. (The one thing to watch
out for is you can do something like 'huge_array = np.zeros((2,
10000000)); tiny_array = a[:, 100]' and now since tiny_array is a view
onto huge_array, so long as a reference to tiny_array exists the full
big memory allocation will remain.)

-n


From njs at pobox.com  Tue Nov 13 03:42:52 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 13 Nov 2012 09:42:52 +0100
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
	<CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
Message-ID: <CAPJVwBkkC6_ga-j+z5CVdYgS8MsWUA+C+UhgJUjTamuEvnR9wA@mail.gmail.com>

On Mon, Nov 12, 2012 at 11:27 PM, Ond?ej ?ert?k <ondrej.certik at gmail.com> wrote:
> I successfully defended my Ph.D. thesis last Thursday, I just need to
> do some changes to it and submit it and I am done.

Congratulations!


From austin.bingham at gmail.com  Tue Nov 13 04:27:14 2012
From: austin.bingham at gmail.com (Austin Bingham)
Date: Tue, 13 Nov 2012 10:27:14 +0100
Subject: [Numpy-discussion] Numpy's policy for releasing memory
In-Reply-To: <CAPJVwB=3uwU-dfZsv4wbN=ggYzxiLVXGY1nrizyy7nFKGyfyqw@mail.gmail.com>
References: <CAEZidR0ERxkW8c2DxvQ-iWfSE54Qvf_jeOCk0uzyOY50NOCv6g@mail.gmail.com>
	<CAPJVwB=3uwU-dfZsv4wbN=ggYzxiLVXGY1nrizyy7nFKGyfyqw@mail.gmail.com>
Message-ID: <CAEZidR3URSsG1akqcvH49mibZ0tpqH8UA2K0L8ZnqJq-byukkA@mail.gmail.com>

OK, if numpy is just subject to Python's behavior then what I'm seeing must
be due to the vagaries of Python. I've noticed that things like removing a
particular line of code or reordering seemingly unrelated calls (unrelated
to the memory issue, that is) can affect when memory is reported as free.
I'll just assume that everything is in order and carry on. Thanks!

Austin


On Tue, Nov 13, 2012 at 9:41 AM, Nathaniel Smith <njs at pobox.com> wrote:

> On Tue, Nov 13, 2012 at 8:26 AM, Austin Bingham
> <austin.bingham at gmail.com> wrote:
> > I'm trying to understand how numpy decides when to release memory and
> > whether it's possible to exert any control over that. The situation is
> that
> > I'm profiling memory usage on a system in which a great deal of the
> overall
> > memory is tied up in ndarrays. Since numpy manages ndarray memory on its
> own
> > (i.e. without the python gc, or so it seems), I'm finding that I can't do
> > much to convince numpy to release memory when things get tight. For
> python
> > object, for example, I can explicitly run gc.collect().
> >
> > So, in an effort to at least understand the system better, can anyone
> tell
> > me how/when numpy decides to release memory? And is there any way via
> either
> > the Python or C-API to explicitly request release? Thanks.
>
> Numpy array memory is released when the corresponding Python objects
> are deleted, so it exactly follows Python's rules. You can't
> explicitly request release, because by definition, if memory is not
> released, then it means that it's still accessible somehow, so
> releasing it could create segfaults. Perhaps you have stray references
> sitting around that you have forgotten to clear -- that's a common
> cause of memory leaks in Python. gc.get_referrers() can be useful to
> debug such things.
>
> Some things to note:
> - Numpy uses malloc() instead of going through the Python low-level
> memory allocation layer (which itself is a wrapper around malloc with
> various optimizations for small objects). This is really only relevant
> because it might create some artifacts depending on how your memory
> profiler gathers data.
> - gc.collect() doesn't do that much in Python... it only matters if
> you have circular references. Mostly Python releases the memory
> associated with objects as soon as the object becomes unreferenced.
> You could try avoiding circular references, and then gc.collect()
> won't even do anything.
> - If you have multiple views of the same memory in numpy, then they
> share the same underlying memory, so that memory won't be released
> until all of the views objects are released. (The one thing to watch
> out for is you can do something like 'huge_array = np.zeros((2,
> 10000000)); tiny_array = a[:, 100]' and now since tiny_array is a view
> onto huge_array, so long as a reference to tiny_array exists the full
> big memory allocation will remain.)
>
> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121113/3cd4db8d/attachment.html>

From sebastian at sipsolutions.net  Tue Nov 13 04:29:57 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Tue, 13 Nov 2012 10:29:57 +0100
Subject: [Numpy-discussion] fix random.choice for 1.7?
In-Reply-To: <50A1C23A.5010506@gmail.com>
References: <509D108A.1060405@gmail.com>
	<CAPJVwBkUrCEKCW+jCPE5LbLAvHCTpT8G7Et+43XxFdaNi-zNCw@mail.gmail.com>
	<50A0FE14.6050605@gmail.com> <1352728782.3411.2.camel@sebastian-laptop>
	<50A12447.6050604@gmail.com>
	<CAPJVwB=_tXi5qMVNJ-ZfBXAPLvkgy1SG6Pnn73-ESvtZ6HcWjA@mail.gmail.com>
	<1352740571.3411.19.camel@sebastian-laptop>
	<50A1796F.4010006@gmail.com>
	<CAPJVwBkJHcYRpjdrgfNV9S74sO0hhfzjrsfRWXeA+MeqKRTx9w@mail.gmail.com>
	<50A18810.6090200@gmail.com>
	<1352769521.19117.16.camel@sebastian-laptop>
	<50A1C23A.5010506@gmail.com>
Message-ID: <1352798997.19117.23.camel@sebastian-laptop>

On Mon, 2012-11-12 at 22:44 -0500, Alan G Isaac wrote:
> On 11/12/2012 8:18 PM, Sebastian Berg wrote:
> > I have created a pull request
> 
> 
> This is still a bit different than I thought you intended.
> With `size=None` we don't get an element,
> but rather a 0d array.
> 
You are right, it should not be a 0d array. I overlooked that tuple()
does not give the same as None at least least for the random functions.

> I thought the idea was to return an element in this case?
> 
> Alan
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From francesc at continuum.io  Tue Nov 13 04:33:52 2012
From: francesc at continuum.io (Francesc Alted)
Date: Tue, 13 Nov 2012 10:33:52 +0100
Subject: [Numpy-discussion] Numpy's policy for releasing memory
In-Reply-To: <CAEZidR3URSsG1akqcvH49mibZ0tpqH8UA2K0L8ZnqJq-byukkA@mail.gmail.com>
References: <CAEZidR0ERxkW8c2DxvQ-iWfSE54Qvf_jeOCk0uzyOY50NOCv6g@mail.gmail.com>
	<CAPJVwB=3uwU-dfZsv4wbN=ggYzxiLVXGY1nrizyy7nFKGyfyqw@mail.gmail.com>
	<CAEZidR3URSsG1akqcvH49mibZ0tpqH8UA2K0L8ZnqJq-byukkA@mail.gmail.com>
Message-ID: <50A21400.6010105@continuum.io>

On 11/13/12 10:27 AM, Austin Bingham wrote:
> OK, if numpy is just subject to Python's behavior then what I'm seeing 
> must be due to the vagaries of Python. I've noticed that things like 
> removing a particular line of code or reordering seemingly unrelated 
> calls (unrelated to the memory issue, that is) can affect when memory 
> is reported as free. I'll just assume that everything is in order and 
> carry on. Thanks!

Profiling memory can be tricky because the operating system may not 
return memory *immediately* as requested, and it might mislead you in 
some situations.  So do not trust too much in memory profilers to be too 
exact and rather focus on the big picture (i.e. my app is reclaiming a 
lot of memory for a large amount o time?  if yes, then start worrying, 
but not before).

-- 
Francesc Alted



From njs at pobox.com  Tue Nov 13 05:12:40 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 13 Nov 2012 11:12:40 +0100
Subject: [Numpy-discussion] Do we want scalar casting to behave as it
 does at the moment?
In-Reply-To: <CANNq6Fn-fwef7koGZTUDo=M=2tkRU4DzY-f62fSwOAZAEnWfMQ@mail.gmail.com>
References: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>
	<CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>
	<CAFXk4boMkSgGJmHHcjUoERYt9_-dMk4CuqKG1ULEOy1cW5q0vA@mail.gmail.com>
	<CANNq6F=XH6Hj9yhWrwid8FeF7JDvdYyRSnM0nwZ6ao1u1_cCOA@mail.gmail.com>
	<CAH6Pt5qeXRB-ZfqLh-07DAT1zL6mM=Zqhcjcbd63MGPXfLw+1g@mail.gmail.com>
	<CANNq6Fn-fwef7koGZTUDo=M=2tkRU4DzY-f62fSwOAZAEnWfMQ@mail.gmail.com>
Message-ID: <CAPJVwBnDQ7XuQs5LKkGfYkSOsdBkeuXfHuj-ZTTEwANYTcJSSg@mail.gmail.com>

On Tue, Nov 13, 2012 at 6:13 AM, Benjamin Root <ben.root at ou.edu> wrote:
>
> On Monday, November 12, 2012, Matthew Brett wrote:
>>
>> Hi,
>>
>> On Mon, Nov 12, 2012 at 8:15 PM, Benjamin Root <ben.root at ou.edu> wrote:
>> >
>> >
>> > On Monday, November 12, 2012, Olivier Delalleau wrote:
>> >>
>> >> 2012/11/12 Nathaniel Smith <njs at pobox.com>
>> >>>
>> >>> On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett
>> >>> <matthew.brett at gmail.com>
>> >>> wrote:
>> >>> > Hi,
>> >>> >
>> >>> > I wanted to check that everyone knows about and is happy with the
>> >>> > scalar casting changes from 1.6.0.
>> >>> >
>> >>> > Specifically, the rules for (array, scalar) casting have changed
>> >>> > such
>> >>> > that the resulting dtype depends on the _value_ of the scalar.
>> >>> >
>> >>> > Mark W has documented these changes here:
>> >>> >
>> >>> > http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
>> >>> >
>> >>> >
>> >>> > http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
>> >>> >
>> >>> >
>> >>> > http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html
>> >>> >
>> >>> > Specifically, as of 1.6.0:
>> >>> >
>> >>> > In [19]: arr = np.array([1.], dtype=np.float32)
>> >>> >
>> >>> > In [20]: (arr + (2**16-1)).dtype
>> >>> > Out[20]: dtype('float32')
>> >>> >
>> >>> > In [21]: (arr + (2**16)).dtype
>> >>> > Out[21]: dtype('float64')
>> >>> >
>> >>> > In [25]: arr = np.array([1.], dtype=np.int8)
>> >>> >
>> >>> > In [26]: (arr + 127).dtype
>> >>> > Out[26]: dtype('int8')
>> >>> >
>> >>> > In [27]: (arr + 128).dtype
>> >>> > Out[27]: dtype('int16')
>> >>> >
>> >>> > There's discussion about the changes here:
>> >>> >
>> >>> >
>> >>> >
>> >>> > http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html
>> >>> >
>> >>> > http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
>> >>> >
>> >>> >
>> >>> > http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html
>> >>> >
>> >>> > It seems to me that this change is hard to explain, and does what
>> >>> > you
>> >>> > want only some of the time, making it a false friend.
>> >>>
>> >>> The old behaviour was that in these cases, the scalar was always cast
>> >>> to the type of the array, right? So
>> >>>   np.array([1], dtype=np.int8) + 256
>> >>> returned 1? Is that the behaviour you prefer?
>> >>>
>> >>> I agree that the 1.6 behaviour is surprising and somewhat
>> >>> inconsistent. There are many places where you can get an overflow in
>> >>> numpy, and in all the other cases we just let the overflow happen. And
>> >>> in fact you can still get an overflow with arr + scalar operations, so
>> >>> this doesn't really fix anything.
>> >>>
>> >>> I find the specific handling of unsigned -> signed and float32 ->
>> >>> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly
>> >>> representable as a float32, but it doesn't *overflow*, it just gives
>> >>> you 2.0**16... if I'm using float32 then I presumably don't care that
>> >>> much about exact representability, so it's surprising that numpy is
>> >>> working to enforce it, and definitely a separate decision from what to
>> >>> do about overflow.)
>> >>>
>> >>> None of those threads seem to really get into the question of what the
>> >>> best behaviour here *is*, though.
>> >>>
>> >>> Possibly the moWell, hold on though, I was asking earlier in the
>> >>> thread what we
>>
>> thought the behavior should be in 2.0 or maybe better put, sometime in
>> the future.
>>
>> If we know what we think the best answer is, and we think the best
>> answer is worth shooting for, then we can try to think of sensible
>> ways of getting there.
>>
>> I guess that's what Nathaniel and Olivier were thinking of but they
>> can correct me if I'm wrong...
>>
>> Cheers,
>>
>> Matthew
>
>
> I am fine with migrating to better solutions (I have yet to decide on this
> current situation, though), but whatever change is adopted must go through a
> deprecation process, which was my point.  Outright breaking of code as a
> first step is the wrong choice, and I was merely nipping it in the bud.

Thanks for your vigilance.

Unfortunately in this case AFAICT 1.6 already silently broke people's
code in this rather weird case, so that will presumably affect
whatever migration strategy we go with, but yes, the goal is to end up
in the right place and to get there in the right way...

-n


From shish at keba.be  Tue Nov 13 06:55:52 2012
From: shish at keba.be (Olivier Delalleau)
Date: Tue, 13 Nov 2012 06:55:52 -0500
Subject: [Numpy-discussion] Do we want scalar casting to behave as it
 does at the moment?
In-Reply-To: <CAH6Pt5qeXRB-ZfqLh-07DAT1zL6mM=Zqhcjcbd63MGPXfLw+1g@mail.gmail.com>
References: <CAH6Pt5rZU9XoxMWp4Uyx6rBMoFDBEgfzWZVXDaVMSsSc+13rgg@mail.gmail.com>
	<CAPJVwBn_yzquXFXXWLhdgxszgQg0ON=xmtGmnd2C3rgPSzmmsw@mail.gmail.com>
	<CAFXk4boMkSgGJmHHcjUoERYt9_-dMk4CuqKG1ULEOy1cW5q0vA@mail.gmail.com>
	<CANNq6F=XH6Hj9yhWrwid8FeF7JDvdYyRSnM0nwZ6ao1u1_cCOA@mail.gmail.com>
	<CAH6Pt5qeXRB-ZfqLh-07DAT1zL6mM=Zqhcjcbd63MGPXfLw+1g@mail.gmail.com>
Message-ID: <CAFXk4bqMuyBwNSduRhCBe_BTTfyGTcVN2NnxSUyWLb4H34ZScw@mail.gmail.com>

2012/11/12 Matthew Brett <matthew.brett at gmail.com>

> Hi,
>
> On Mon, Nov 12, 2012 at 8:15 PM, Benjamin Root <ben.root at ou.edu> wrote:
> >
> >
> > On Monday, November 12, 2012, Olivier Delalleau wrote:
> >>
> >> 2012/11/12 Nathaniel Smith <njs at pobox.com>
> >>>
> >>> On Mon, Nov 12, 2012 at 8:54 PM, Matthew Brett <
> matthew.brett at gmail.com>
> >>> wrote:
> >>> > Hi,
> >>> >
> >>> > I wanted to check that everyone knows about and is happy with the
> >>> > scalar casting changes from 1.6.0.
> >>> >
> >>> > Specifically, the rules for (array, scalar) casting have changed such
> >>> > that the resulting dtype depends on the _value_ of the scalar.
> >>> >
> >>> > Mark W has documented these changes here:
> >>> >
> >>> > http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
> >>> >
> >>> >
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
> >>> >
> >>> >
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.promote_types.html
> >>> >
> >>> > Specifically, as of 1.6.0:
> >>> >
> >>> > In [19]: arr = np.array([1.], dtype=np.float32)
> >>> >
> >>> > In [20]: (arr + (2**16-1)).dtype
> >>> > Out[20]: dtype('float32')
> >>> >
> >>> > In [21]: (arr + (2**16)).dtype
> >>> > Out[21]: dtype('float64')
> >>> >
> >>> > In [25]: arr = np.array([1.], dtype=np.int8)
> >>> >
> >>> > In [26]: (arr + 127).dtype
> >>> > Out[26]: dtype('int8')
> >>> >
> >>> > In [27]: (arr + 128).dtype
> >>> > Out[27]: dtype('int16')
> >>> >
> >>> > There's discussion about the changes here:
> >>> >
> >>> >
> >>> >
> http://mail.scipy.org/pipermail/numpy-discussion/2011-September/058563.html
> >>> >
> http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
> >>> >
> >>> >
> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060381.html
> >>> >
> >>> > It seems to me that this change is hard to explain, and does what you
> >>> > want only some of the time, making it a false friend.
> >>>
> >>> The old behaviour was that in these cases, the scalar was always cast
> >>> to the type of the array, right? So
> >>>   np.array([1], dtype=np.int8) + 256
> >>> returned 1? Is that the behaviour you prefer?
> >>>
> >>> I agree that the 1.6 behaviour is surprising and somewhat
> >>> inconsistent. There are many places where you can get an overflow in
> >>> numpy, and in all the other cases we just let the overflow happen. And
> >>> in fact you can still get an overflow with arr + scalar operations, so
> >>> this doesn't really fix anything.
> >>>
> >>> I find the specific handling of unsigned -> signed and float32 ->
> >>> float64 upcasting confusing as well. (Sure, 2**16 isn't exactly
> >>> representable as a float32, but it doesn't *overflow*, it just gives
> >>> you 2.0**16... if I'm using float32 then I presumably don't care that
> >>> much about exact representability, so it's surprising that numpy is
> >>> working to enforce it, and definitely a separate decision from what to
> >>> do about overflow.)
> >>>
> >>> None of those threads seem to really get into the question of what the
> >>> best behaviour here *is*, though.
> >>>
> >>> Possibly the most defensible choice is to treat ufunc(arr, scalar)
> >>> operations as performing an implicit cast of the scalar to arr's
> >>> dtype, and using the standard implicit casting rules -- which I think
> >>> means, raising an error if !can_cast(scalar, arr.dtype,
> >>> casting="safe")
> >>
> >>
> >> I like this suggestion. It may break some existing code, but I think
> it'd
> >> be for the best. The current behavior can be very confusing.
> >>
> >> -=- Olivier
> >
> >
> >
> > "break some existing code"
> >
> > I really should set up an email filter for this phrase and have it send
> back
> > an email automatically: "Are you nuts?!"
>
> Well, hold on though, I was asking earlier in the thread what we
> thought the behavior should be in 2.0 or maybe better put, sometime in
> the future.
>
> If we know what we think the best answer is, and we think the best
> answer is worth shooting for, then we can try to think of sensible
> ways of getting there.
>
> I guess that's what Nathaniel and Olivier were thinking of but they
> can correct me if I'm wrong...
>
> Cheers,
>
> Matthew
>

This is indeed what I had in mind, thanks.
I definitely agree a (long) period with a deprecation warning would be
needed if this is changed.

-=- Olivier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121113/eb88ece2/attachment.html>

From shish at keba.be  Tue Nov 13 07:08:32 2012
From: shish at keba.be (Olivier Delalleau)
Date: Tue, 13 Nov 2012 07:08:32 -0500
Subject: [Numpy-discussion] Numpy's policy for releasing memory
In-Reply-To: <CAEZidR3URSsG1akqcvH49mibZ0tpqH8UA2K0L8ZnqJq-byukkA@mail.gmail.com>
References: <CAEZidR0ERxkW8c2DxvQ-iWfSE54Qvf_jeOCk0uzyOY50NOCv6g@mail.gmail.com>
	<CAPJVwB=3uwU-dfZsv4wbN=ggYzxiLVXGY1nrizyy7nFKGyfyqw@mail.gmail.com>
	<CAEZidR3URSsG1akqcvH49mibZ0tpqH8UA2K0L8ZnqJq-byukkA@mail.gmail.com>
Message-ID: <CAFXk4bqe7imbofthQM4ELd9rirk2brHAYg_azeVhvC8VXYdggg@mail.gmail.com>

How are you monitoring memory usage?
Personally I've been using psutil and it seems to work well, although I've
used it only on Windows and not in applications with large numpy arrays, so
I can't tell whether it would work you.

Also, keep in mind that:
- The "auto-delete object when it goes out of scope" behavior is specific
to the CPython implementation and not part of the Python standard, so if
you're actually using a different implementation you may see a different
behavior.
- CPython deals with small objects in a special way, not actually releasing
allocated memory. For more info:
http://deeplearning.net/software/theano/tutorial/python-memory-management.html#internal-memory-management

-=- Olivier

2012/11/13 Austin Bingham <austin.bingham at gmail.com>

> OK, if numpy is just subject to Python's behavior then what I'm seeing
> must be due to the vagaries of Python. I've noticed that things like
> removing a particular line of code or reordering seemingly unrelated calls
> (unrelated to the memory issue, that is) can affect when memory is reported
> as free. I'll just assume that everything is in order and carry on. Thanks!
>
> Austin
>
>
> On Tue, Nov 13, 2012 at 9:41 AM, Nathaniel Smith <njs at pobox.com> wrote:
>
>> On Tue, Nov 13, 2012 at 8:26 AM, Austin Bingham
>> <austin.bingham at gmail.com> wrote:
>> > I'm trying to understand how numpy decides when to release memory and
>> > whether it's possible to exert any control over that. The situation is
>> that
>> > I'm profiling memory usage on a system in which a great deal of the
>> overall
>> > memory is tied up in ndarrays. Since numpy manages ndarray memory on
>> its own
>> > (i.e. without the python gc, or so it seems), I'm finding that I can't
>> do
>> > much to convince numpy to release memory when things get tight. For
>> python
>> > object, for example, I can explicitly run gc.collect().
>> >
>> > So, in an effort to at least understand the system better, can anyone
>> tell
>> > me how/when numpy decides to release memory? And is there any way via
>> either
>> > the Python or C-API to explicitly request release? Thanks.
>>
>> Numpy array memory is released when the corresponding Python objects
>> are deleted, so it exactly follows Python's rules. You can't
>> explicitly request release, because by definition, if memory is not
>> released, then it means that it's still accessible somehow, so
>> releasing it could create segfaults. Perhaps you have stray references
>> sitting around that you have forgotten to clear -- that's a common
>> cause of memory leaks in Python. gc.get_referrers() can be useful to
>> debug such things.
>>
>> Some things to note:
>> - Numpy uses malloc() instead of going through the Python low-level
>> memory allocation layer (which itself is a wrapper around malloc with
>> various optimizations for small objects). This is really only relevant
>> because it might create some artifacts depending on how your memory
>> profiler gathers data.
>> - gc.collect() doesn't do that much in Python... it only matters if
>> you have circular references. Mostly Python releases the memory
>> associated with objects as soon as the object becomes unreferenced.
>> You could try avoiding circular references, and then gc.collect()
>> won't even do anything.
>> - If you have multiple views of the same memory in numpy, then they
>> share the same underlying memory, so that memory won't be released
>> until all of the views objects are released. (The one thing to watch
>> out for is you can do something like 'huge_array = np.zeros((2,
>> 10000000)); tiny_array = a[:, 100]' and now since tiny_array is a view
>> onto huge_array, so long as a reference to tiny_array exists the full
>> big memory allocation will remain.)
>>
>> -n
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121113/6a4d6dd2/attachment.html>

From gael.varoquaux at normalesup.org  Tue Nov 13 07:18:45 2012
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 13 Nov 2012 13:18:45 +0100
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
	<CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
Message-ID: <20121113121845.GB15981@phare.normalesup.org>

On Mon, Nov 12, 2012 at 02:27:02PM -0800, Ond?ej ?ert?k wrote:
> I successfully defended my Ph.D. thesis last Thursday, I just need to
> do some changes to it and submit it and I am done.

Yey! Tag and release (the thesis, I mean)!

Congratulation.

G

PS: It seems to me that it was yesterday that you moved to the state. You
didn't loose time with your PhD. Very impressive!


From austin.bingham at gmail.com  Tue Nov 13 07:31:34 2012
From: austin.bingham at gmail.com (Austin Bingham)
Date: Tue, 13 Nov 2012 13:31:34 +0100
Subject: [Numpy-discussion] Numpy's policy for releasing memory
In-Reply-To: <CAFXk4bqe7imbofthQM4ELd9rirk2brHAYg_azeVhvC8VXYdggg@mail.gmail.com>
References: <CAEZidR0ERxkW8c2DxvQ-iWfSE54Qvf_jeOCk0uzyOY50NOCv6g@mail.gmail.com>
	<CAPJVwB=3uwU-dfZsv4wbN=ggYzxiLVXGY1nrizyy7nFKGyfyqw@mail.gmail.com>
	<CAEZidR3URSsG1akqcvH49mibZ0tpqH8UA2K0L8ZnqJq-byukkA@mail.gmail.com>
	<CAFXk4bqe7imbofthQM4ELd9rirk2brHAYg_azeVhvC8VXYdggg@mail.gmail.com>
Message-ID: <CAEZidR0upK633BUk-GTRFrHwSCLm0MaFvet-cDAk54xu3_FgRg@mail.gmail.com>

I've been using psutil, pmap (linux command), and resource in various
capacities, all on cpython. When I wasn't seeing memory freed when I
expected, I got to wondering if maybe numpy was maintaining pools of
buffers for reuse or something like that. It sounds like that's not the
case, though, so I'm following up other possibilities.

Austin


On Tue, Nov 13, 2012 at 1:08 PM, Olivier Delalleau <shish at keba.be> wrote:

> How are you monitoring memory usage?
> Personally I've been using psutil and it seems to work well, although I've
> used it only on Windows and not in applications with large numpy arrays, so
> I can't tell whether it would work you.
>
> Also, keep in mind that:
> - The "auto-delete object when it goes out of scope" behavior is specific
> to the CPython implementation and not part of the Python standard, so if
> you're actually using a different implementation you may see a different
> behavior.
> - CPython deals with small objects in a special way, not actually
> releasing allocated memory. For more info:
> http://deeplearning.net/software/theano/tutorial/python-memory-management.html#internal-memory-management
>
> -=- Olivier
>
> 2012/11/13 Austin Bingham <austin.bingham at gmail.com>
>
>> OK, if numpy is just subject to Python's behavior then what I'm seeing
>> must be due to the vagaries of Python. I've noticed that things like
>> removing a particular line of code or reordering seemingly unrelated calls
>> (unrelated to the memory issue, that is) can affect when memory is reported
>> as free. I'll just assume that everything is in order and carry on. Thanks!
>>
>> Austin
>>
>>
>> On Tue, Nov 13, 2012 at 9:41 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>>> On Tue, Nov 13, 2012 at 8:26 AM, Austin Bingham
>>> <austin.bingham at gmail.com> wrote:
>>> > I'm trying to understand how numpy decides when to release memory and
>>> > whether it's possible to exert any control over that. The situation is
>>> that
>>> > I'm profiling memory usage on a system in which a great deal of the
>>> overall
>>> > memory is tied up in ndarrays. Since numpy manages ndarray memory on
>>> its own
>>> > (i.e. without the python gc, or so it seems), I'm finding that I can't
>>> do
>>> > much to convince numpy to release memory when things get tight. For
>>> python
>>> > object, for example, I can explicitly run gc.collect().
>>> >
>>> > So, in an effort to at least understand the system better, can anyone
>>> tell
>>> > me how/when numpy decides to release memory? And is there any way via
>>> either
>>> > the Python or C-API to explicitly request release? Thanks.
>>>
>>> Numpy array memory is released when the corresponding Python objects
>>> are deleted, so it exactly follows Python's rules. You can't
>>> explicitly request release, because by definition, if memory is not
>>> released, then it means that it's still accessible somehow, so
>>> releasing it could create segfaults. Perhaps you have stray references
>>> sitting around that you have forgotten to clear -- that's a common
>>> cause of memory leaks in Python. gc.get_referrers() can be useful to
>>> debug such things.
>>>
>>> Some things to note:
>>> - Numpy uses malloc() instead of going through the Python low-level
>>> memory allocation layer (which itself is a wrapper around malloc with
>>> various optimizations for small objects). This is really only relevant
>>> because it might create some artifacts depending on how your memory
>>> profiler gathers data.
>>> - gc.collect() doesn't do that much in Python... it only matters if
>>> you have circular references. Mostly Python releases the memory
>>> associated with objects as soon as the object becomes unreferenced.
>>> You could try avoiding circular references, and then gc.collect()
>>> won't even do anything.
>>> - If you have multiple views of the same memory in numpy, then they
>>> share the same underlying memory, so that memory won't be released
>>> until all of the views objects are released. (The one thing to watch
>>> out for is you can do something like 'huge_array = np.zeros((2,
>>> 10000000)); tiny_array = a[:, 100]' and now since tiny_array is a view
>>> onto huge_array, so long as a reference to tiny_array exists the full
>>> big memory allocation will remain.)
>>>
>>> -n
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121113/6a73b524/attachment.html>

From njs at pobox.com  Tue Nov 13 07:46:53 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 13 Nov 2012 13:46:53 +0100
Subject: [Numpy-discussion] Numpy's policy for releasing memory
In-Reply-To: <CAEZidR0upK633BUk-GTRFrHwSCLm0MaFvet-cDAk54xu3_FgRg@mail.gmail.com>
References: <CAEZidR0ERxkW8c2DxvQ-iWfSE54Qvf_jeOCk0uzyOY50NOCv6g@mail.gmail.com>
	<CAPJVwB=3uwU-dfZsv4wbN=ggYzxiLVXGY1nrizyy7nFKGyfyqw@mail.gmail.com>
	<CAEZidR3URSsG1akqcvH49mibZ0tpqH8UA2K0L8ZnqJq-byukkA@mail.gmail.com>
	<CAFXk4bqe7imbofthQM4ELd9rirk2brHAYg_azeVhvC8VXYdggg@mail.gmail.com>
	<CAEZidR0upK633BUk-GTRFrHwSCLm0MaFvet-cDAk54xu3_FgRg@mail.gmail.com>
Message-ID: <CAPJVwBn7Rpwcgmo5km=kYO88YoaENq=3LhRs2SaHvbcF2BS5NQ@mail.gmail.com>

On Tue, Nov 13, 2012 at 1:31 PM, Austin Bingham
<austin.bingham at gmail.com> wrote:
> I've been using psutil, pmap (linux command), and resource in various
> capacities, all on cpython. When I wasn't seeing memory freed when I
> expected, I got to wondering if maybe numpy was maintaining pools of buffers
> for reuse or something like that. It sounds like that's not the case,
> though, so I'm following up other possibilities.

Those tools show how much memory the OS has allocated to the process.
In general, processes can request memory from the OS, but *they cannot
give it back*. At the C level, if you call free(), then what actually
happens is that the memory management library in your process makes a
note for itself that that memory is not used, and may return it from a
future malloc(), but from the OS's point of view it is still
"allocated". (And python uses another similar system on top for
malloc()/free(), but this doesn't really change anything.) So the OS
memory usage you see is generally a "high water mark", the maximum
amount of memory that your process ever needed.

The exception is that for large single allocations (e.g. if you create
a multi-megabyte array), a different mechanism is used. Such large
memory allocations *can* be released back to the OS. So it might
specifically be the non-numpy parts of your program that are producing
the issues you see.

-n


From charlesr.harris at gmail.com  Tue Nov 13 09:52:54 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 13 Nov 2012 07:52:54 -0700
Subject: [Numpy-discussion] Numpy's policy for releasing memory
In-Reply-To: <CAEZidR3URSsG1akqcvH49mibZ0tpqH8UA2K0L8ZnqJq-byukkA@mail.gmail.com>
References: <CAEZidR0ERxkW8c2DxvQ-iWfSE54Qvf_jeOCk0uzyOY50NOCv6g@mail.gmail.com>
	<CAPJVwB=3uwU-dfZsv4wbN=ggYzxiLVXGY1nrizyy7nFKGyfyqw@mail.gmail.com>
	<CAEZidR3URSsG1akqcvH49mibZ0tpqH8UA2K0L8ZnqJq-byukkA@mail.gmail.com>
Message-ID: <CAB6mnxKj2E3E3QbX9e=1xdGJuh+2NYLKFJro3yS2v3tLu1KcJQ@mail.gmail.com>

On Tue, Nov 13, 2012 at 2:27 AM, Austin Bingham <austin.bingham at gmail.com>wrote:

> OK, if numpy is just subject to Python's behavior then what I'm seeing
> must be due to the vagaries of Python. I've noticed that things like
> removing a particular line of code or reordering seemingly unrelated calls
> (unrelated to the memory issue, that is) can affect when memory is reported
> as free. I'll just assume that everything is in order and carry on. Thanks!
>
>
If you are running interactively in IPython, references will be kept to
return values. That can eventually eat up memory if you are working with a
lot of big arrays.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121113/f6c518a7/attachment.html>

From ondrej.certik at gmail.com  Tue Nov 13 11:01:20 2012
From: ondrej.certik at gmail.com (=?UTF-8?B?T25kxZllaiDEjGVydMOtaw==?=)
Date: Tue, 13 Nov 2012 08:01:20 -0800
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <20121113121845.GB15981@phare.normalesup.org>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
	<CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
	<20121113121845.GB15981@phare.normalesup.org>
Message-ID: <CADDwiVAzSKoGOa6fibVACsf5ZBMaq4qOr7Z8Ch_Zo5yKih6xcA@mail.gmail.com>

On Tue, Nov 13, 2012 at 4:18 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Mon, Nov 12, 2012 at 02:27:02PM -0800, Ond?ej ?ert?k wrote:
>> I successfully defended my Ph.D. thesis last Thursday, I just need to
>> do some changes to it and submit it and I am done.
>
> Yey! Tag and release (the thesis, I mean)!
>
> Congratulation.
>
> G
>
> PS: It seems to me that it was yesterday that you moved to the state. You
> didn't loose time with your PhD. Very impressive!

Thanks everybody. Gael, it feels like yesterday, but it's been 4 years ago.
And I had an M.S. degree already from Prague which took 2 years, so the total
time is 6, and I had to wait 0.5 years after finishing my M.S. and before coming
to the US (paperwork), so this gives 6.5 years since finishing my B.C.
degree. The average
time for physics Ph.D. seems to be 5.5 years, at least according to [1], so
I am 1 year late, but I had to overcome a few bumps along the way ---
maybe I'll write
a blog post later.

Ondrej

[1] http://physics.uchicago.edu/prospective/graduate/index.html


From charlesr.harris at gmail.com  Tue Nov 13 11:43:10 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 13 Nov 2012 09:43:10 -0700
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <CADDwiVAzSKoGOa6fibVACsf5ZBMaq4qOr7Z8Ch_Zo5yKih6xcA@mail.gmail.com>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
	<CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
	<20121113121845.GB15981@phare.normalesup.org>
	<CADDwiVAzSKoGOa6fibVACsf5ZBMaq4qOr7Z8Ch_Zo5yKih6xcA@mail.gmail.com>
Message-ID: <CAB6mnxLatRoY63=7-BGDEEADdWr2b16QBvANAP32vMr8ah4K3A@mail.gmail.com>

On Tue, Nov 13, 2012 at 9:01 AM, Ond?ej ?ert?k <ondrej.certik at gmail.com>wrote:

> On Tue, Nov 13, 2012 at 4:18 AM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
> > On Mon, Nov 12, 2012 at 02:27:02PM -0800, Ond?ej ?ert?k wrote:
> >> I successfully defended my Ph.D. thesis last Thursday, I just need to
> >> do some changes to it and submit it and I am done.
> >
> > Yey! Tag and release (the thesis, I mean)!
> >
> > Congratulation.
> >
> > G
> >
> > PS: It seems to me that it was yesterday that you moved to the state. You
> > didn't loose time with your PhD. Very impressive!
>
> Thanks everybody. Gael, it feels like yesterday, but it's been 4 years ago.
> And I had an M.S. degree already from Prague which took 2 years, so the
> total
> time is 6, and I had to wait 0.5 years after finishing my M.S. and before
> coming
> to the US (paperwork), so this gives 6.5 years since finishing my B.C.
> degree. The average
> time for physics Ph.D. seems to be 5.5 years, at least according to [1], so
> I am 1 year late, but I had to overcome a few bumps along the way ---
> maybe I'll write
> a blog post later.
>

When I was at Columbia I think it was more like 7 for experimental physics.
But then there were the theoreticians who got by in a year or two with
short dissertations, and in later years picked up their Nobel prize...

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121113/fc7ffaa9/attachment.html>

From ondrej.certik at gmail.com  Tue Nov 13 23:24:36 2012
From: ondrej.certik at gmail.com (=?UTF-8?B?T25kxZllaiDEjGVydMOtaw==?=)
Date: Tue, 13 Nov 2012 20:24:36 -0800
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
	<CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
Message-ID: <CADDwiVDd_U7iZKHzj=qt8WqY_pT1HzR9UqptYaMxcvRbnWXxog@mail.gmail.com>

On Mon, Nov 12, 2012 at 2:27 PM, Ond?ej ?ert?k <ondrej.certik at gmail.com> wrote:
[...]
> Here is a list of issues that need to be fixed before the release:
>
> https://github.com/numpy/numpy/issues?milestone=3&state=open
>
> If anyone wants to help, we just need to get through them and submit a
> PR for each, or close it if it doesn't apply anymore.
> This is what I am doing now.

Ok, I went over all the issues, closed fixed issues and sent PRs for about
6 issues. Just go to the link above to see them all (or go to issues and
click on "NumPy 1.7" milestone) and left comments on most issues.

Most of the minor problems are fixed. There are only 3 big issues that
need to be fixed so I set them "priority high":

https://github.com/numpy/numpy/issues?labels=priority%3A+high&milestone=3&page=1&state=open

in particular:

https://github.com/numpy/numpy/issues/568
https://github.com/numpy/numpy/issues/2668
https://github.com/numpy/numpy/issues/606

after that I think we should be good to go. If you have some spare
cycles, just concentrate on these 3.

Ondrej


From p.j.a.cock at googlemail.com  Wed Nov 14 08:34:36 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Wed, 14 Nov 2012 13:34:36 +0000
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <CADDwiVDd_U7iZKHzj=qt8WqY_pT1HzR9UqptYaMxcvRbnWXxog@mail.gmail.com>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
	<CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
	<CADDwiVDd_U7iZKHzj=qt8WqY_pT1HzR9UqptYaMxcvRbnWXxog@mail.gmail.com>
Message-ID: <CAKVJ-_4vfCDrAbsaJAv6vbzicavaUsW3Ft1e-kFJyC-vbjMcVA@mail.gmail.com>

On Wed, Nov 14, 2012 at 4:24 AM, Ond?ej ?ert?k <ondrej.certik at gmail.com>wrote:

> On Mon, Nov 12, 2012 at 2:27 PM, Ond?ej ?ert?k <ondrej.certik at gmail.com>
> wrote:
> [...]
> > Here is a list of issues that need to be fixed before the release:
> >
> > https://github.com/numpy/numpy/issues?milestone=3&state=open
> >
> > If anyone wants to help, we just need to get through them and submit a
> > PR for each, or close it if it doesn't apply anymore.
> > This is what I am doing now.
>
> Ok, I went over all the issues, closed fixed issues and sent PRs for about
> 6 issues. Just go to the link above to see them all (or go to issues and
> click on "NumPy 1.7" milestone) and left comments on most issues.
>
> Most of the minor problems are fixed. There are only 3 big issues that
> need to be fixed so I set them "priority high":
>
>
> https://github.com/numpy/numpy/issues?labels=priority%3A+high&milestone=3&page=1&state=open
>
> in particular:
>
> https://github.com/numpy/numpy/issues/568
> https://github.com/numpy/numpy/issues/2668
> https://github.com/numpy/numpy/issues/606
>
> after that I think we should be good to go. If you have some spare
> cycles, just concentrate on these 3.
>
> Ondrej
>

Hi all,

Having looked at the README.txt and INSTALL.txt files on the
branch, I see no mention of which Python 3.x versions are supported:

https://github.com/numpy/numpy/blob/maintenance/1.7.x/README.txt
https://github.com/numpy/numpy/blob/maintenance/1.7.x/INSTALL.txt

Is NumPy 1.7 intended to support Python 3.3?

My impression from this thread is probably yes:
http://mail.scipy.org/pipermail/numpy-discussion/2012-July/063483.html
...
http://mail.scipy.org/pipermail/numpy-discussion/2012-August/063597.html

If so, then under Windows (32 bit at least) where the Python.org
provided Python 3.3 is compiled with MSCV 2010 there are some
problems - see: https://github.com/numpy/numpy/pull/2726
Should an issue be filed for this?

Thanks,

Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121114/bc11dc27/attachment.html>

From ondrej.certik at gmail.com  Wed Nov 14 13:02:50 2012
From: ondrej.certik at gmail.com (=?UTF-8?B?T25kxZllaiDEjGVydMOtaw==?=)
Date: Wed, 14 Nov 2012 10:02:50 -0800
Subject: [Numpy-discussion] 1.7.0 release
In-Reply-To: <CAKVJ-_4vfCDrAbsaJAv6vbzicavaUsW3Ft1e-kFJyC-vbjMcVA@mail.gmail.com>
References: <EF46F58F-EEE8-401F-9B12-D4A279030A2A@continuum.io>
	<CADDwiVCe3u3vu_GOmkk=vNL3HhmWinkdFafx1SCy31VJmdMbZw@mail.gmail.com>
	<CADDwiVDd_U7iZKHzj=qt8WqY_pT1HzR9UqptYaMxcvRbnWXxog@mail.gmail.com>
	<CAKVJ-_4vfCDrAbsaJAv6vbzicavaUsW3Ft1e-kFJyC-vbjMcVA@mail.gmail.com>
Message-ID: <CADDwiVA1xSMZvaKQF=K7WpiqNSPQ1S3Z=+o=vV0u5gogR5PzRQ@mail.gmail.com>

> Hi all,
>
> Having looked at the README.txt and INSTALL.txt files on the
> branch, I see no mention of which Python 3.x versions are supported:
>
> https://github.com/numpy/numpy/blob/maintenance/1.7.x/README.txt
> https://github.com/numpy/numpy/blob/maintenance/1.7.x/INSTALL.txt
>
> Is NumPy 1.7 intended to support Python 3.3?
>
> My impression from this thread is probably yes:
> http://mail.scipy.org/pipermail/numpy-discussion/2012-July/063483.html
> ...
> http://mail.scipy.org/pipermail/numpy-discussion/2012-August/063597.html
>
> If so, then under Windows (32 bit at least) where the Python.org
> provided Python 3.3 is compiled with MSCV 2010 there are some
> problems - see: https://github.com/numpy/numpy/pull/2726

Thanks Peter for the email, yes I think we should support Python 3.3.

> Should an issue be filed for this?

I just did:

https://github.com/numpy/numpy/issues/2743


Btw, I also bumbed this issue as priority high:

https://github.com/numpy/numpy/issues/2738

Because it segfaults numpy.

Ondrej


From p.j.a.cock at googlemail.com  Wed Nov 14 13:10:24 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Wed, 14 Nov 2012 18:10:24 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3 with
	MSVC 2010
Message-ID: <CAKVJ-_4aO2zTXm7iPB0wzuWvNDo5PohqCWNPZ1vG5FOAz18aYw@mail.gmail.com>

Changing title to reflect the fact this thread is now about using
the Microsoft compiler rather than mingw32 as in the old thread.

On Sat, Nov 10, 2012 at 11:04 PM, Peter Cock <p.j.a.cock at googlemail.com> wrote:
> On Sat, Nov 10, 2012 at 5:47 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>
>>
>>
>> On Tue, Nov 6, 2012 at 6:49 PM, Peter Cock <p.j.a.cock at googlemail.com>
>> wrote:
>>>
>>> Dear all,
>>>
>>> Since the NumPy 1.7.0b2 release didn't include a Windows
>>> (32 bit) installer for Python 3.3, I am considering compiling it
>>> myself for local testing. What compiler is recommended?
>>
>>
>> Either MSVC or MinGW 3.4.5. For the latter see
>> https://github.com/certik/numpy-vendor
>
> Thanks Ralf,
>
> I was trying with MSVC 9.0 installed, but got this cryptic error:
>
>    C:\Downloads\numpy-1.7.0b2 > C:\python33\python setup.py build
>    ...
>    error: Unable to find vcvarsall.bat
>
> After sprinkling distutils with debug statements, I found it was
> looking for MSVC v10 (not numpy's fault but the error is most
> unhelpful).
>
> Presumably "Microsoft Visual C++ 2010 Express Edition" is
> the appropriate thing to download?
> http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express
>

I would have tried this earlier, but it required Windows XP SP3
and there was something amiss with the permissions on my
work Windows machine preventing that update. Solved now,
and this file now exists:

C:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat

I've tried my branch (focussed on mingw32 fixes), and numpy-1.7.0b2
but both fail at the same point. I did remove the old build directory first.

C:\Downloads\numpy-1.7.0b2>c:\python33\python setup.py build
Converting to Python3 via 2to3...
<snip>
Could not locate executable efc
don't know how to compile Fortran code on platform 'nt'
C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo
/Ox /MD /W3 /GS- /DNDEBUG -Inumpy\core\src\private -Inumpy\core\src
-Inumpy\core -Inumpy\
core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath
-Inumpy\core\src\npysort -Inumpy\core\include -Ic:\python33\include
-Ic:\python33\include /Tc_configtest.c /Fo_configtest.obj
Found executable C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe
C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /nologo
/INCREMENTAL:NO _configtest.obj /OUT:_configtest.exe
/MANIFESTFILE:_configtest.exe.manifest
Found executable C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe
mt.exe -nologo -manifest _configtest.exe.manifest
-outputresource:_configtest.exe;1
Found executable C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\mt.exe

_configtest.exe.manifest : general error c1010070: Failed to load and
parse the manifest. The system cannot find the file specified.

failure.
removing: _configtest.c _configtest.obj
Traceback (most recent call last):
  File "setup.py", line 214, in <module>
    setup_package()
  File "setup.py", line 207, in setup_package
    configuration=configuration )
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\core.py", line 186
, in setup
    return old_setup(**new_attr)
  File "c:\python33\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "c:\python33\lib\distutils\dist.py", line 917, in run_commands
    self.run_command(cmd)
  File "c:\python33\lib\distutils\dist.py", line 936, in run_command
    cmd_obj.run()
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build.py",
 line 37, in run
    old_build.run(self)
  File "c:\python33\lib\distutils\command\build.py", line 126, in run
    self.run_command(cmd_name)
  File "c:\python33\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "c:\python33\lib\distutils\dist.py", line 936, in run_command
    cmd_obj.run()
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 152, in run
    self.build_sources()
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 163, in build_sources
    self.build_library_sources(*libname_info)
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 298, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 385, in generate_sources
    source = func(extension, build_dir)
  File "numpy\core\setup.py", line 648, in get_mathlib_info
    raise RuntimeError("Broken toolchain: cannot link a simple C program")
RuntimeError: Broken toolchain: cannot link a simple C program

It appears a similar issue was raised before:
http://mail.scipy.org/pipermail/numpy-discussion/2012-June/062866.html

Any tips?

Peter


From cgohlke at uci.edu  Wed Nov 14 14:35:27 2012
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Wed, 14 Nov 2012 11:35:27 -0800
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
 with MSVC 2010
In-Reply-To: <CAKVJ-_4aO2zTXm7iPB0wzuWvNDo5PohqCWNPZ1vG5FOAz18aYw@mail.gmail.com>
References: <CAKVJ-_4aO2zTXm7iPB0wzuWvNDo5PohqCWNPZ1vG5FOAz18aYw@mail.gmail.com>
Message-ID: <50A3F27F.70605@uci.edu>

On 11/14/2012 10:10 AM, Peter Cock wrote:
> Changing title to reflect the fact this thread is now about using
> the Microsoft compiler rather than mingw32 as in the old thread.
>
> On Sat, Nov 10, 2012 at 11:04 PM, Peter Cock <p.j.a.cock at googlemail.com> wrote:
>> On Sat, Nov 10, 2012 at 5:47 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>
>>>
>>>
>>> On Tue, Nov 6, 2012 at 6:49 PM, Peter Cock <p.j.a.cock at googlemail.com>
>>> wrote:
>>>>
>>>> Dear all,
>>>>
>>>> Since the NumPy 1.7.0b2 release didn't include a Windows
>>>> (32 bit) installer for Python 3.3, I am considering compiling it
>>>> myself for local testing. What compiler is recommended?
>>>
>>>
>>> Either MSVC or MinGW 3.4.5. For the latter see
>>> https://github.com/certik/numpy-vendor
>>
>> Thanks Ralf,
>>
>> I was trying with MSVC 9.0 installed, but got this cryptic error:
>>
>>     C:\Downloads\numpy-1.7.0b2 > C:\python33\python setup.py build
>>     ...
>>     error: Unable to find vcvarsall.bat
>>
>> After sprinkling distutils with debug statements, I found it was
>> looking for MSVC v10 (not numpy's fault but the error is most
>> unhelpful).
>>
>> Presumably "Microsoft Visual C++ 2010 Express Edition" is
>> the appropriate thing to download?
>> http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express
>>
>
> I would have tried this earlier, but it required Windows XP SP3
> and there was something amiss with the permissions on my
> work Windows machine preventing that update. Solved now,
> and this file now exists:
>
> C:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat
>
> I've tried my branch (focussed on mingw32 fixes), and numpy-1.7.0b2
> but both fail at the same point. I did remove the old build directory first.
>
> C:\Downloads\numpy-1.7.0b2>c:\python33\python setup.py build
> Converting to Python3 via 2to3...
> <snip>
> Could not locate executable efc
> don't know how to compile Fortran code on platform 'nt'
> C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo
> /Ox /MD /W3 /GS- /DNDEBUG -Inumpy\core\src\private -Inumpy\core\src
> -Inumpy\core -Inumpy\
> core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath
> -Inumpy\core\src\npysort -Inumpy\core\include -Ic:\python33\include
> -Ic:\python33\include /Tc_configtest.c /Fo_configtest.obj
> Found executable C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe
> C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /nologo
> /INCREMENTAL:NO _configtest.obj /OUT:_configtest.exe
> /MANIFESTFILE:_configtest.exe.manifest
> Found executable C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe
> mt.exe -nologo -manifest _configtest.exe.manifest
> -outputresource:_configtest.exe;1
> Found executable C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\mt.exe
>
> _configtest.exe.manifest : general error c1010070: Failed to load and
> parse the manifest. The system cannot find the file specified.
>
> failure.
> removing: _configtest.c _configtest.obj
> Traceback (most recent call last):
>    File "setup.py", line 214, in <module>
>      setup_package()
>    File "setup.py", line 207, in setup_package
>      configuration=configuration )
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\core.py", line 186
> , in setup
>      return old_setup(**new_attr)
>    File "c:\python33\lib\distutils\core.py", line 148, in setup
>      dist.run_commands()
>    File "c:\python33\lib\distutils\dist.py", line 917, in run_commands
>      self.run_command(cmd)
>    File "c:\python33\lib\distutils\dist.py", line 936, in run_command
>      cmd_obj.run()
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build.py",
>   line 37, in run
>      old_build.run(self)
>    File "c:\python33\lib\distutils\command\build.py", line 126, in run
>      self.run_command(cmd_name)
>    File "c:\python33\lib\distutils\cmd.py", line 313, in run_command
>      self.distribution.run_command(command)
>    File "c:\python33\lib\distutils\dist.py", line 936, in run_command
>      cmd_obj.run()
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
> py", line 152, in run
>      self.build_sources()
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
> py", line 163, in build_sources
>      self.build_library_sources(*libname_info)
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
> py", line 298, in build_library_sources
>      sources = self.generate_sources(sources, (lib_name, build_info))
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
> py", line 385, in generate_sources
>      source = func(extension, build_dir)
>    File "numpy\core\setup.py", line 648, in get_mathlib_info
>      raise RuntimeError("Broken toolchain: cannot link a simple C program")
> RuntimeError: Broken toolchain: cannot link a simple C program
>
> It appears a similar issue was raised before:
> http://mail.scipy.org/pipermail/numpy-discussion/2012-June/062866.html
>
> Any tips?
>
> Peter

Try changing line 648 in Python33\Lib\distutils\msvc9compiler.py to 
`mfinfo = None`.

http://hg.python.org/cpython/file/tip/Lib/distutils/msvc9compiler.py#l648

Christoph


From thomas.robitaille at gmail.com  Wed Nov 14 14:46:53 2012
From: thomas.robitaille at gmail.com (Thomas Robitaille)
Date: Wed, 14 Nov 2012 20:46:53 +0100
Subject: [Numpy-discussion] PRs for MaskedArray bugs
Message-ID: <CAGMHX_3bSuP3SkKHuuAdJNUwpOfD1mgJ3JSZXdLvVwnNdPwBOw@mail.gmail.com>

I've recently opened a couple of pull requests that fix bugs with
MaskedArray - these are pretty straightforward, so would it be
possible to consider them in time for 1.7?

https://github.com/numpy/numpy/pull/2703

https://github.com/numpy/numpy/pull/2733

Thanks!
Tom


From ondrej.certik at gmail.com  Wed Nov 14 18:13:48 2012
From: ondrej.certik at gmail.com (=?UTF-8?B?T25kxZllaiDEjGVydMOtaw==?=)
Date: Wed, 14 Nov 2012 15:13:48 -0800
Subject: [Numpy-discussion] Should abs([nan]) be supported?
In-Reply-To: <CABL7CQg6UUB0aALP7kMhKhn0usUEK+3mj2RcCynQ7qFWNr4mfA@mail.gmail.com>
References: <CADDwiVB9NrL0+pY6Ge-h7rtJpvDZ1eTPU64nygyRyRfkamgY3w@mail.gmail.com>
	<B4FABD4F-6807-49A9-87D2-73AD62CCDAB3@continuum.io>
	<CADDwiVBvKsM3nAhwjwJnxkW-Em9z7Qg1jDN7NnHvSC93__EeFA@mail.gmail.com>
	<2F3BC0D2-E111-478C-B733-F414BFFAA76A@continuum.io>
	<CABL7CQg6UUB0aALP7kMhKhn0usUEK+3mj2RcCynQ7qFWNr4mfA@mail.gmail.com>
Message-ID: <CADDwiVCD-boYajkzWKRxgXU6kn1RgJTnzr+MGevT5=Jfz1DzJw@mail.gmail.com>

On Fri, Sep 7, 2012 at 7:54 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
> On Wed, Sep 5, 2012 at 7:06 AM, Travis Oliphant <travis at continuum.io> wrote:
>>
>> The framework for catching errors relies on hardware flags getting set and
>> our C code making the right calls to detect those flags.
>>
>> This has usually worked correctly in the past --- but it is an area where
>> changes in compilers or platforms could create problems.
>
>
> I don't think it ever did, for less common platforms at least. See all the
> Debian test issues that were filed by Sandro this week. And even between
> Windows and Linux, there are some inconsistencies.
>
>>
>>
>> We should test to be sure that the correct warnings are issued, I would
>> think.    Perhaps using a catch_warnings context would be helpful (from
>> http://docs.python.org/library/warnings.html)
>
>
> There are some tests for that already, in core/test_numeric.py. For example:
>
> ======================================================================
> FAIL: test_default (test_numeric.TestSeterr)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/Users/rgommers/Code/numpy/numpy/core/tests/test_numeric.py", line
> 231, in test_default
>     under='ignore',
> AssertionError: {'over': 'ignore', 'divide': 'ignore', 'invalid': 'ignore',
> 'under': 'ignore'} != {'over': 'warn', 'divide': 'warn', 'invalid': 'warn',
> 'under': 'ignore'}
>
> ----------------------------------------------------------------------
>
> They're not exhaustive though.
>
>>
>>
>> import warnings
>>
>> def fxn():
>>     warnings.warn("deprecated", DeprecationWarning)
>>
>> with warnings.catch_warnings(record=True) as w:
>>     # Cause all warnings to always be triggered.
>>     warnings.simplefilter("always")
>>     # Trigger a warning.
>>     fxn()
>>     # Verify some things
>>     assert len(w) == 1
>>     assert issubclass(w[-1].category, DeprecationWarning)
>>     assert "deprecated" in str(w[-1].message)
>>
>>
>>
>
> Use ``from numpy.testing import WarningManager`` for a 2.4-compatible
> version of catch_warnings (with explicitly calling its __enter__ and
> __exit__ methods).


In the release 1.7.x branch these warnings are suppressed, but in
master they frequently make the Travis tests fail. That's extremely
annoying to me, so I started working on a fix here:

https://github.com/numpy/numpy/pull/2745

using the recommendation from Travis above.

Ondrej


From p.j.a.cock at googlemail.com  Thu Nov 15 09:24:28 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Thu, 15 Nov 2012 14:24:28 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
 with MSVC 2010
In-Reply-To: <50A3F27F.70605@uci.edu>
References: <CAKVJ-_4aO2zTXm7iPB0wzuWvNDo5PohqCWNPZ1vG5FOAz18aYw@mail.gmail.com>
	<50A3F27F.70605@uci.edu>
Message-ID: <CAKVJ-_6QnF+4bGkGS=i+5cLWxg405Ffjy3g4MBRT015yJ2UxGQ@mail.gmail.com>

On Wed, Nov 14, 2012 at 7:35 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:
>> ...
>> RuntimeError: Broken toolchain: cannot link a simple C program
>>
>> It appears a similar issue was raised before:
>> http://mail.scipy.org/pipermail/numpy-discussion/2012-June/062866.html
>>
>> Any tips?
>>
>> Peter
>
> Try changing line 648 in Python33\Lib\distutils\msvc9compiler.py to
> `mfinfo = None`.
>
> http://hg.python.org/cpython/file/tip/Lib/distutils/msvc9compiler.py#l648
>
> Christoph

Hi Christoph,

That was very precise advice and seems to solve this. Presumably
you've faced something like this before. Is there an open issue for
this in Python itself?

Line 648 didn't seem sensible (and I guess the tip has changed), but
I tried replacing this bit of Python33\Lib\distutils\msvc9compiler.py
which was near by:

            # embed the manifest
            # XXX - this is somewhat fragile - if mt.exe fails, distutils
            # will still consider the DLL up-to-date, but it will not have a
            # manifest.  Maybe we should link to a temp file?  OTOH, that
            # implies a build environment error that shouldn't go undetected.
            mfinfo = self.manifest_get_embed_info(target_desc, ld_args)

with your suggestion of 'mfinfo = None', and this did seem enough
to get NumPy to compile with Python 3.3 and MSCV v10. I could
then build and test a library using NumPy (C and Python APIs), but
I've not yet installed nose to run NumPy's own tests.

Looking at the code for the manifest_get_embed_info method,
I don't see any obvious 9 vs 10 issues like the problems I hit
before. However there are some regular expressions in the
method _remove_visual_c_ref which it calls which look more
likely - looking for two digits when perhaps it needs to be three
under MSVC 10...

As a general point, if MSVC 10 is sufficiently different from 9,
does it make sense to introduce distutils/msvc10compiler.py
(subclassing/reusing most of distutils/msvc9compiler.py) in
Python itself? Or in NumPy's distutils?

Thanks,

Peter


From will at thearete.co.uk  Thu Nov 15 11:02:17 2012
From: will at thearete.co.uk (Will Furnass)
Date: Thu, 15 Nov 2012 16:02:17 +0000 (UTC)
Subject: [Numpy-discussion] numpy.power vs pylab.power
Message-ID: <k833m9$hfi$1@ger.gmane.org>

On my machine these are rather confusingly different functions, with the 
latter corresponding to numpy.random.power.  I appreciate that pylab 
imports everything from both the numpy and numpy.random modules but 
wouldn't it make sense if pylab.power were the frequently used power 
function rather than a means for sampling from the power distribution?

Regards,

Will Furnass



From robert.kern at gmail.com  Thu Nov 15 11:05:48 2012
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 15 Nov 2012 16:05:48 +0000
Subject: [Numpy-discussion] numpy.power vs pylab.power
In-Reply-To: <k833m9$hfi$1@ger.gmane.org>
References: <k833m9$hfi$1@ger.gmane.org>
Message-ID: <CAF6FJiudP90sa2EXNjscMfh1pC4ouuFobG+oWCZ9y1sxJGZ=Lg@mail.gmail.com>

On Thu, Nov 15, 2012 at 4:02 PM, Will Furnass <will at thearete.co.uk> wrote:
> On my machine these are rather confusingly different functions, with the
> latter corresponding to numpy.random.power.  I appreciate that pylab
> imports everything from both the numpy and numpy.random modules but
> wouldn't it make sense if pylab.power were the frequently used power
> function rather than a means for sampling from the power distribution?

Matplotlib may be discussed over here:

https://lists.sourceforge.net/lists/listinfo/matplotlib-users

--
Robert Kern


From ppmime at gmail.com  Thu Nov 15 12:37:08 2012
From: ppmime at gmail.com (=?ISO-8859-1?Q?Jose_Miguel_Ib=E1=F1ez?=)
Date: Thu, 15 Nov 2012 18:37:08 +0100
Subject: [Numpy-discussion] Numpydoc warnings for methods
Message-ID: <CAFUEvSPs7Wth2W0dTNrHGnD7iUaxrqHG2BQPhg=FcFad1DXDhg@mail.gmail.com>

Hello,

Now, I am having the same problem, and although I have tried the
Pauili fix (see below) I still have the same problem
when using numpydoc extension.

Does anyone have more information or suggestions about it ?


Thanks,
Jose


On Sun, Jul 17, 2011 at 7:15 PM, Tony Yu <tsyu80 at gmail.com
<http://mail.scipy.org/mailman/listinfo/numpy-discussion>> wrote:

>* I'm building documentation using Sphinx, and it seems that numpydoc is*>* raising*>* a lot of warnings. Specifically, the warnings look like "failed to import*>* <method_name>", "toctree*>* references unknown document u'<method_name>'", "toctree contains reference*>* to nonexisting document '<method_name>'---for each method defined. The*>* example below reproduces the issue on my system (Sphinx 1.0.7, numpy HEAD).*>* These warnings appear in my build of the numpy docs, as well.*>**>* Removing numpydoc from the list of Sphinx extensions gets rid of these*>* warnings*>* (but, of course, adds new warnings if headings for 'Parameters', 'Returns',*>* etc. are present).*>**>* Am I doing something wrong here?*>**>* You're not, it's a Sphinx bug that Pauli already has a fix for. See*http://projects.scipy.org/numpy/ticket/1772

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121115/c030fbf0/attachment.html>

From cgohlke at uci.edu  Thu Nov 15 13:15:35 2012
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Thu, 15 Nov 2012 10:15:35 -0800
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
 with MSVC 2010
In-Reply-To: <CAKVJ-_6QnF+4bGkGS=i+5cLWxg405Ffjy3g4MBRT015yJ2UxGQ@mail.gmail.com>
References: <CAKVJ-_4aO2zTXm7iPB0wzuWvNDo5PohqCWNPZ1vG5FOAz18aYw@mail.gmail.com>
	<50A3F27F.70605@uci.edu>
	<CAKVJ-_6QnF+4bGkGS=i+5cLWxg405Ffjy3g4MBRT015yJ2UxGQ@mail.gmail.com>
Message-ID: <50A53147.6060608@uci.edu>

On 11/15/2012 6:24 AM, Peter Cock wrote:
> On Wed, Nov 14, 2012 at 7:35 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:
>>> ...
>>> RuntimeError: Broken toolchain: cannot link a simple C program
>>>
>>> It appears a similar issue was raised before:
>>> http://mail.scipy.org/pipermail/numpy-discussion/2012-June/062866.html
>>>
>>> Any tips?
>>>
>>> Peter
>>
>> Try changing line 648 in Python33\Lib\distutils\msvc9compiler.py to
>> `mfinfo = None`.
>>
>> http://hg.python.org/cpython/file/tip/Lib/distutils/msvc9compiler.py#l648
>>
>> Christoph
>
> Hi Christoph,
>
> That was very precise advice and seems to solve this. Presumably
> you've faced something like this before. Is there an open issue for
> this in Python itself?
>
> Line 648 didn't seem sensible (and I guess the tip has changed), but
> I tried replacing this bit of Python33\Lib\distutils\msvc9compiler.py
> which was near by:
>
>              # embed the manifest
>              # XXX - this is somewhat fragile - if mt.exe fails, distutils
>              # will still consider the DLL up-to-date, but it will not have a
>              # manifest.  Maybe we should link to a temp file?  OTOH, that
>              # implies a build environment error that shouldn't go undetected.
>              mfinfo = self.manifest_get_embed_info(target_desc, ld_args)
>
> with your suggestion of 'mfinfo = None', and this did seem enough
> to get NumPy to compile with Python 3.3 and MSCV v10. I could
> then build and test a library using NumPy (C and Python APIs), but
> I've not yet installed nose to run NumPy's own tests.
>
> Looking at the code for the manifest_get_embed_info method,
> I don't see any obvious 9 vs 10 issues like the problems I hit
> before. However there are some regular expressions in the
> method _remove_visual_c_ref which it calls which look more
> likely - looking for two digits when perhaps it needs to be three
> under MSVC 10...
>
> As a general point, if MSVC 10 is sufficiently different from 9,
> does it make sense to introduce distutils/msvc10compiler.py
> (subclassing/reusing most of distutils/msvc9compiler.py) in
> Python itself? Or in NumPy's distutils?
>
> Thanks,
>
> Peter
>

Naturally the file would be named msvc10compiler.py but the name may be 
kept for compatibility reasons. AFAIK msvc10 does not use manifests any 
longer for the CRT dependencies and all the code handling msvc9 
manifests could be removed for Python 3.3. I have been building 
extensions for Python 3.3 with msvc10 and this distutils patch for some 
months and did not notice any issues.

Christoph


From gokhansever at gmail.com  Thu Nov 15 22:24:38 2012
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Thu, 15 Nov 2012 20:24:38 -0700
Subject: [Numpy-discussion] float32 to float64 casting
Message-ID: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>

Hello,

Could someone briefly explain why are these two operations are casting my
float32 arrays to float64?

I1 (np.arange(5, dtype='float32')).dtype
O1 dtype('float32')

I2 (100000*np.arange(5, dtype='float32')).dtype
O2 dtype('float64')



I3 (np.arange(5, dtype='float32')[0]).dtype
O3 dtype('float32')

I4 (1*np.arange(5, dtype='float32')[0]).dtype
O4 dtype('float64')



Thanks.




-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121115/ddfb3270/attachment.html>

From charlesr.harris at gmail.com  Fri Nov 16 01:37:03 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 15 Nov 2012 23:37:03 -0700
Subject: [Numpy-discussion] float32 to float64 casting
In-Reply-To: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
References: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
Message-ID: <CAB6mnxK8skiXUerba3fFCmbvd=s8eQ29J_JB5pgxSo=82cbq_g@mail.gmail.com>

On Thu, Nov 15, 2012 at 8:24 PM, G?khan Sever <gokhansever at gmail.com> wrote:

> Hello,
>
> Could someone briefly explain why are these two operations are casting my
> float32 arrays to float64?
>
> I1 (np.arange(5, dtype='float32')).dtype
> O1 dtype('float32')
>
> I2 (100000*np.arange(5, dtype='float32')).dtype
> O2 dtype('float64')
>

This one is depends on the size of the multiplier and is first present in
1.6.0. I suspect it is a side effect of making the type conversion code
sensitive to magnitude.


>
>
>
> I3 (np.arange(5, dtype='float32')[0]).dtype
> O3 dtype('float32')
>
> I4 (1*np.arange(5, dtype='float32')[0]).dtype
> O4 dtype('float64')
>

This one probably depends on the fact that the element is a scalar, but
doesn't look right. Scalars are promoted differently. Also holds in numpy
1.5.0 so is of old provenance.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121115/20dc8810/attachment.html>

From charlesr.harris at gmail.com  Fri Nov 16 02:07:08 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 16 Nov 2012 00:07:08 -0700
Subject: [Numpy-discussion] float32 to float64 casting
In-Reply-To: <CAB6mnxK8skiXUerba3fFCmbvd=s8eQ29J_JB5pgxSo=82cbq_g@mail.gmail.com>
References: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
	<CAB6mnxK8skiXUerba3fFCmbvd=s8eQ29J_JB5pgxSo=82cbq_g@mail.gmail.com>
Message-ID: <CAB6mnxKQGK4Xs2k6CppXB3iXvntHQHBvABA=geneq+85Bs06Vg@mail.gmail.com>

On Thu, Nov 15, 2012 at 11:37 PM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

>
>
> On Thu, Nov 15, 2012 at 8:24 PM, G?khan Sever <gokhansever at gmail.com>wrote:
>
>> Hello,
>>
>> Could someone briefly explain why are these two operations are casting my
>> float32 arrays to float64?
>>
>> I1 (np.arange(5, dtype='float32')).dtype
>> O1 dtype('float32')
>>
>> I2 (100000*np.arange(5, dtype='float32')).dtype
>> O2 dtype('float64')
>>
>
> This one is depends on the size of the multiplier and is first present in
> 1.6.0. I suspect it is a side effect of making the type conversion code
> sensitive to magnitude.
>
>
>>
>>
>>
>> I3 (np.arange(5, dtype='float32')[0]).dtype
>> O3 dtype('float32')
>>
>> I4 (1*np.arange(5, dtype='float32')[0]).dtype
>> O4 dtype('float64')
>>
>
> This one probably depends on the fact that the element is a scalar, but
> doesn't look right. Scalars are promoted differently. Also holds in numpy
> 1.5.0 so is of old provenance.
>
>
This one has always bothered me:

In [3]: (-1*arange(5, dtype=uint64)).dtype
Out[3]: dtype('float64')

And there is another where the result is an object array, but I can't
recall it at the moment.

Chuck



> Chuck
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121116/1d338fea/attachment.html>

From p.j.a.cock at googlemail.com  Fri Nov 16 04:28:08 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Fri, 16 Nov 2012 09:28:08 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
 with MSVC 2010
In-Reply-To: <50A53147.6060608@uci.edu>
References: <CAKVJ-_4aO2zTXm7iPB0wzuWvNDo5PohqCWNPZ1vG5FOAz18aYw@mail.gmail.com>
	<50A3F27F.70605@uci.edu>
	<CAKVJ-_6QnF+4bGkGS=i+5cLWxg405Ffjy3g4MBRT015yJ2UxGQ@mail.gmail.com>
	<50A53147.6060608@uci.edu>
Message-ID: <CAKVJ-_7W2mmT=m9_WpHxOL+WBDzaMZ+fThNpWC5rTdVwVtuagw@mail.gmail.com>

On Thu, Nov 15, 2012 at 6:15 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:
>
> Naturally the file would be named msvc10compiler.py but the name may be
> kept for compatibility reasons. AFAIK msvc10 does not use manifests any
> longer for the CRT dependencies and all the code handling msvc9
> manifests could be removed for Python 3.3. I have been building
> extensions for Python 3.3 with msvc10 and this distutils patch for some
> months and did not notice any issues.
>

Sounds Python 3.3 needs a fix then - have you reported this?
If not, could you report it (since you know far more about the
Windows build system than I do)?

If it will be fixed in Python itself, then perhaps a manual hack like
this will be enough for NumPy in the short term. Otherwise, maybe
numpy needs to include its own copy of msvc9compiler.py (or
msvc10compiler.py)?

Thanks,

Peter


From cgohlke at uci.edu  Fri Nov 16 05:08:09 2012
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Fri, 16 Nov 2012 02:08:09 -0800
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
 with MSVC 2010
In-Reply-To: <CAKVJ-_7W2mmT=m9_WpHxOL+WBDzaMZ+fThNpWC5rTdVwVtuagw@mail.gmail.com>
References: <CAKVJ-_4aO2zTXm7iPB0wzuWvNDo5PohqCWNPZ1vG5FOAz18aYw@mail.gmail.com>
	<50A3F27F.70605@uci.edu>
	<CAKVJ-_6QnF+4bGkGS=i+5cLWxg405Ffjy3g4MBRT015yJ2UxGQ@mail.gmail.com>
	<50A53147.6060608@uci.edu>
	<CAKVJ-_7W2mmT=m9_WpHxOL+WBDzaMZ+fThNpWC5rTdVwVtuagw@mail.gmail.com>
Message-ID: <50A61089.2050906@uci.edu>

On 11/16/2012 1:28 AM, Peter Cock wrote:
> On Thu, Nov 15, 2012 at 6:15 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:
>>
>> Naturally the file would be named msvc10compiler.py but the name may be
>> kept for compatibility reasons. AFAIK msvc10 does not use manifests any
>> longer for the CRT dependencies and all the code handling msvc9
>> manifests could be removed for Python 3.3. I have been building
>> extensions for Python 3.3 with msvc10 and this distutils patch for some
>> months and did not notice any issues.
>>
>
> Sounds Python 3.3 needs a fix then - have you reported this?
> If not, could you report it (since you know far more about the
> Windows build system than I do)?
>
> If it will be fixed in Python itself, then perhaps a manual hack like
> this will be enough for NumPy in the short term. Otherwise, maybe
> numpy needs to include its own copy of msvc9compiler.py (or
> msvc10compiler.py)?
>
> Thanks,
>
> Peter

Could be related to <http://bugs.python.org/issue16296>.

Christoph


From shish at keba.be  Fri Nov 16 06:47:42 2012
From: shish at keba.be (Olivier Delalleau)
Date: Fri, 16 Nov 2012 06:47:42 -0500
Subject: [Numpy-discussion] float32 to float64 casting
In-Reply-To: <CAB6mnxK8skiXUerba3fFCmbvd=s8eQ29J_JB5pgxSo=82cbq_g@mail.gmail.com>
References: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
	<CAB6mnxK8skiXUerba3fFCmbvd=s8eQ29J_JB5pgxSo=82cbq_g@mail.gmail.com>
Message-ID: <CAFXk4bpP4C6Da3tTNVQCJY7ofdPJhL_KbsOtoSBRaO_1_jESug@mail.gmail.com>

2012/11/16 Charles R Harris <charlesr.harris at gmail.com>

>
>
> On Thu, Nov 15, 2012 at 8:24 PM, G?khan Sever <gokhansever at gmail.com>wrote:
>
>> Hello,
>>
>> Could someone briefly explain why are these two operations are casting my
>> float32 arrays to float64?
>>
>> I1 (np.arange(5, dtype='float32')).dtype
>> O1 dtype('float32')
>>
>> I2 (100000*np.arange(5, dtype='float32')).dtype
>> O2 dtype('float64')
>>
>
> This one is depends on the size of the multiplier and is first present in
> 1.6.0. I suspect it is a side effect of making the type conversion code
> sensitive to magnitude.
>
>
>>
>>
>>
>> I3 (np.arange(5, dtype='float32')[0]).dtype
>> O3 dtype('float32')
>>
>> I4 (1*np.arange(5, dtype='float32')[0]).dtype
>> O4 dtype('float64')
>>
>
> This one probably depends on the fact that the element is a scalar, but
> doesn't look right. Scalars are promoted differently. Also holds in numpy
> 1.5.0 so is of old provenance.
>
> Chuck
>

My understanding is that non-mixed operations (scalar/scalar or
array/array) use casting rules that don't depend on magnitude, and the
upcast of int{32,64} mixed with float32 has always been float64 (probably
because the result has to be a kind of float, and float64 makes it possible
to represent exactly a larger integer range than float32). Note that if you
cast 1 into int16 the result will be float32 (I guess float32 can represent
exactly all int16 integers).

-=- Olivier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121116/68a2699b/attachment.html>

From shish at keba.be  Fri Nov 16 06:56:18 2012
From: shish at keba.be (Olivier Delalleau)
Date: Fri, 16 Nov 2012 06:56:18 -0500
Subject: [Numpy-discussion] float32 to float64 casting
In-Reply-To: <CAFXk4bo_9kiXjXKwTF45R9sB9Z1u0=k-AbdPZrug-WHSrimf8w@mail.gmail.com>
References: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
	<CAB6mnxK8skiXUerba3fFCmbvd=s8eQ29J_JB5pgxSo=82cbq_g@mail.gmail.com>
	<CAB6mnxKQGK4Xs2k6CppXB3iXvntHQHBvABA=geneq+85Bs06Vg@mail.gmail.com>
	<CAFXk4bo_9kiXjXKwTF45R9sB9Z1u0=k-AbdPZrug-WHSrimf8w@mail.gmail.com>
Message-ID: <CAFXk4bqKPyzVo9yAEHj4F7iNrq7nMV=jFOzVax8j=wYBQAY0Vg@mail.gmail.com>

2012/11/16 Olivier Delalleau <olivier.delalleau at gmail.com>

> 2012/11/16 Charles R Harris <charlesr.harris at gmail.com>
>
>>
>>
>> On Thu, Nov 15, 2012 at 11:37 PM, Charles R Harris <
>> charlesr.harris at gmail.com> wrote:
>>
>>>
>>>
>>> On Thu, Nov 15, 2012 at 8:24 PM, G?khan Sever <gokhansever at gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> Could someone briefly explain why are these two operations are casting
>>>> my float32 arrays to float64?
>>>>
>>>> I1 (np.arange(5, dtype='float32')).dtype
>>>> O1 dtype('float32')
>>>>
>>>> I2 (100000*np.arange(5, dtype='float32')).dtype
>>>> O2 dtype('float64')
>>>>
>>>
>>> This one is depends on the size of the multiplier and is first present
>>> in 1.6.0. I suspect it is a side effect of making the type conversion code
>>> sensitive to magnitude.
>>>
>>>
>>>>
>>>>
>>>>
>>>> I3 (np.arange(5, dtype='float32')[0]).dtype
>>>> O3 dtype('float32')
>>>>
>>>> I4 (1*np.arange(5, dtype='float32')[0]).dtype
>>>> O4 dtype('float64')
>>>>
>>>
>>> This one probably depends on the fact that the element is a scalar, but
>>> doesn't look right. Scalars are promoted differently. Also holds in numpy
>>> 1.5.0 so is of old provenance.
>>>
>>>
>> This one has always bothered me:
>>
>> In [3]: (-1*arange(5, dtype=uint64)).dtype
>> Out[3]: dtype('float64')
>>
>
> My interpretation here is that since the possible results when multiplying
> an int64 with an uint64 can be signed, and can go beyond the range of
> int64, numpy prefers to cast everything to float64, which can represent
> (even if approximately) a larger range of signed values.
>

Actually, thinking about it a bit more, I suspect the logic is not related
to the result of the operation, but to the fact numpy needs to cast both
arguments into a common dtype before doing the operation, and it has no
integer dtype available that can hold both int64 and uint64 numbers, so it
uses float64 instead.

-=- Olivier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121116/1592bbbe/attachment.html>

From njs at pobox.com  Fri Nov 16 07:51:47 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 16 Nov 2012 12:51:47 +0000
Subject: [Numpy-discussion] float32 to float64 casting
In-Reply-To: <CAB6mnxK8skiXUerba3fFCmbvd=s8eQ29J_JB5pgxSo=82cbq_g@mail.gmail.com>
References: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
	<CAB6mnxK8skiXUerba3fFCmbvd=s8eQ29J_JB5pgxSo=82cbq_g@mail.gmail.com>
Message-ID: <CAPJVwB=5Kn8Y1VstWa1sbGqimHK=Q7aZ8RWXd9oj2DtSkiQb3g@mail.gmail.com>

On Fri, Nov 16, 2012 at 6:37 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> On Thu, Nov 15, 2012 at 8:24 PM, G?khan Sever <gokhansever at gmail.com> wrote:
>>
>> Hello,
>>
>> Could someone briefly explain why are these two operations are casting my
>> float32 arrays to float64?
>>
>> I1 (np.arange(5, dtype='float32')).dtype
>> O1 dtype('float32')
>>
>> I2 (100000*np.arange(5, dtype='float32')).dtype
>> O2 dtype('float64')
>
>
> This one is depends on the size of the multiplier and is first present in
> 1.6.0. I suspect it is a side effect of making the type conversion code
> sensitive to magnitude.

Right, this is the problem:

In [22]: np.can_cast(10000, np.float32, "safe")
Out[22]: True

In [23]: np.can_cast(100000, np.float32, "safe")
Out[23]: False

But... that said... this makes NO SENSE. 100000 is exactly
representable as a float32! can_cast is just wrong, yes?

https://en.wikipedia.org/wiki/Single-precision_floating-point_format
bin(100000) == 0b11000011010100000
Sign: 0
Exponent for an integer: 23 to make the fractional part into an
integer, -7 to shift the leading 1 bit so it's in position 23 instead
of position 16, +127 to correct for the bias
  bin(23 - 7 + 127) == 0b10001111
Fraction = 100000, shifted so that the top bit lands in position 23
  bin(100000 << 7) == 0b110000110101000000000000
Throw away the top bit and concatenate:

  In [74]: np.uint32(0b01000111110000110101000000000000).view(np.float32)
Out[74]: 100000.0

Looks good to me. So... numpy just doesn't know how integer<->float
conversion works...?

>> I3 (np.arange(5, dtype='float32')[0]).dtype
>> O3 dtype('float32')
>>
>> I4 (1*np.arange(5, dtype='float32')[0]).dtype
>> O4 dtype('float64')
>
>
> This one probably depends on the fact that the element is a scalar, but
> doesn't look right. Scalars are promoted differently. Also holds in numpy
> 1.5.0 so is of old provenance.

Yeah, I missed at first that this is scalar * scalar, probably clearer to write:

In [85]: (1 * np.float32(1)).dtype
Out[85]: dtype('float64')

For this numpy just uses ordinary find-a-common-type rules and ignores
the values, and neither int32 nor float32 is a superset of the other,
so it goes for float64.

It's a bit disconcerting that in this case numpy's find-a-common-type
rules don't match C's, though... (C's rule is: if you have an
expression involving floating point, find the widest floating point
type involved, and convert everything to that type. If your expression
only involves integers, then... well, then things get kind of bizarre.
First you upcast anything smaller than an int to an int. Then, find
the widest integer types involved. If one of them is signed and can
represent everything that the other can, you use that. Otherwise you
cast it to unsigned (!!!) and use that type. So uint32 + int32 ->
uint32. This is 6.3.1.8 in C99. I'm not saying we should necessarily
follow the weirdo integer rules, but for floats it's a bit
surprising.)

-n


From aldcroft at head.cfa.harvard.edu  Fri Nov 16 09:19:53 2012
From: aldcroft at head.cfa.harvard.edu (Tom Aldcroft)
Date: Fri, 16 Nov 2012 09:19:53 -0500
Subject: [Numpy-discussion] Segfault for Numpy 1.4.1 on 64-bit CentOS-5
Message-ID: <CAGsQ-NupywVYPJeAzO3ME94Y4jdd6bKhv47TBT6_NM2FCawHLQ@mail.gmail.com>

For library compatibility testing I'm trying to use numpy 1.4.1 with Python
2.7.3 on a 64-bit CentOS-5 platform.  I installed a clean Python from
source (basically "./configure --prefix=$prefix ; make install") and then
installed numpy 1.4.1 with "python setup.py install".

The crash message begins with [1]:

*** glibc detected *** /home/aldcroft/vpy/py27_np141/bin/python: free():
invalid next size (fast): 0x000000001a9fcf30 ***
======= Backtrace: =========
/lib64/libc.so.6[0x31a90711df]
/lib64/libc.so.6(cfree+0x4b)[0x31a907163b]
/home/aldcroft/vpy/py27_np141/lib/python2.7/site-packages/numpy/core/multiarray.so[0x2aaab025ebbe]

A problem which seems related is that fancy indexing is failing:

>>> idx = np.array([1])
>>> idx.dtype
dtype('int64')
>>> np.arange(5)[idx]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: index 210453397505 out of bounds 0<=index<5

Does anyone have suggestions for compilation flags or workarounds when
building numpy or Python that might fix this?

Thanks,
Tom

[1] I don't know the exact code that triggered this crash, it's buried in
some units tests.  If it was useful I could dig it out.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121116/ac612104/attachment.html>

From gokhansever at gmail.com  Fri Nov 16 16:53:07 2012
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Fri, 16 Nov 2012 14:53:07 -0700
Subject: [Numpy-discussion] float32 to float64 casting
In-Reply-To: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
References: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
Message-ID: <CAE5kuyinNs5eJ6KtYW38XigY+oKnocw=E4zkYh+M4dU0qFgX4g@mail.gmail.com>

Thanks for the explanations.

For either case, I was expecting to get float32 as a resulting data type.
Since, float32 is large enough to contain the result. I am wondering if
changing casting rule this way, requires a lot of modification in the NumPy
code. Maybe as an alternative to the current casting mechanism?

I like the way that NumPy can convert to float64. As if these data-types
are continuation of each other. But just the conversation might happen too
early --at least in my opinion, as demonstrated in my example.

For instance comparing this example to IDL surprises me:

I16 np.float32(5555)*5e38
O16 2.7774999999999998e+42

I17 (np.float32(5555)*5e38).dtype
O17 dtype('float64')

IDL> help, 5e38*float(5555)
<Expression>    FLOAT     =           Inf

In IDL, the expression doesn't get converted to DOUBLE. Perhaps, its a
design decision.


On Thu, Nov 15, 2012 at 8:24 PM, G?khan Sever <gokhansever at gmail.com> wrote:

> Hello,
>
> Could someone briefly explain why are these two operations are casting my
> float32 arrays to float64?
>
> I1 (np.arange(5, dtype='float32')).dtype
> O1 dtype('float32')
>
> I2 (100000*np.arange(5, dtype='float32')).dtype
> O2 dtype('float64')
>
>
>
> I3 (np.arange(5, dtype='float32')[0]).dtype
> O3 dtype('float32')
>
> I4 (1*np.arange(5, dtype='float32')[0]).dtype
> O4 dtype('float64')
>
>
>
> Thanks.
>
>
>
>
> --
> G?khan
>



-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121116/1dc72674/attachment.html>

From chaoyuejoy at gmail.com  Sat Nov 17 08:28:43 2012
From: chaoyuejoy at gmail.com (Chao YUE)
Date: Sat, 17 Nov 2012 14:28:43 +0100
Subject: [Numpy-discussion] the fast way to loop over ndarray elements?
Message-ID: <CAAN-aRHEyFe3QaCOXObs3Uu=F1N89KvuiMcwnm1gPnPiLtxfNA@mail.gmail.com>

Dear all,

I need to make a linear contrast of the 2D numpy array "data" from an
interval to another, the approach is:
I have another two list: "base" & "target", then I check for each ndarray
element "data[i,j]",
if   base[m] =< data[i,j] <= base[m+1], then it will be linearly converted
to be in the interval of (target[m], target[m+1]),
using another function called "lintrans".


#The way I do is to loop each row and column of the 2D array, and finally
loop the intervals constituted by base list:

for row in range(data.shape[0]):
    for col in range(data.shape[1]):
        for i in range(len(base)-1):
            if data[row,col]>=base[i] and data[row,col]<=base[i+1]:

data[row,col]=lintrans(data[row,col],(base[i],base[i+1]),(target[i],target[i+1]))
                break  #use break to jump out of loop as the data have to
be ONLY transferred ONCE.


Now the profiling result shows that most of the time has been used in this
loop over the array ("plot_array_transg"),
and less time in calling the linear transformation fuction "lintrans":

   ncalls     tottime  percall    cumtime    percall
filename:lineno(function)
  18047    0.110    0.000      0.110        0.000
mathex.py:132(lintrans)
  1            12.495  12.495   19.061      19.061
mathex.py:196(plot_array_transg)


so is there anyway I can speed up this loop?  Thanks for any suggestions!!

best,

Chao

-- 
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121117/67a7c325/attachment.html>

From ben.root at ou.edu  Sat Nov 17 08:40:46 2012
From: ben.root at ou.edu (Benjamin Root)
Date: Sat, 17 Nov 2012 08:40:46 -0500
Subject: [Numpy-discussion] the fast way to loop over ndarray elements?
In-Reply-To: <CAAN-aRHEyFe3QaCOXObs3Uu=F1N89KvuiMcwnm1gPnPiLtxfNA@mail.gmail.com>
References: <CAAN-aRHEyFe3QaCOXObs3Uu=F1N89KvuiMcwnm1gPnPiLtxfNA@mail.gmail.com>
Message-ID: <CANNq6FmT=kS5M7gRc+SOfLooX_mTi8C71Nyqm_pKfMdSkjBy=Q@mail.gmail.com>

On Saturday, November 17, 2012, Chao YUE wrote:

> Dear all,
>
> I need to make a linear contrast of the 2D numpy array "data" from an
> interval to another, the approach is:
> I have another two list: "base" & "target", then I check for each ndarray
> element "data[i,j]",
> if   base[m] =< data[i,j] <= base[m+1], then it will be linearly converted
> to be in the interval of (target[m], target[m+1]),
> using another function called "lintrans".
>
>
> #The way I do is to loop each row and column of the 2D array, and finally
> loop the intervals constituted by base list:
>
> for row in range(data.shape[0]):
>     for col in range(data.shape[1]):
>         for i in range(len(base)-1):
>             if data[row,col]>=base[i] and data[row,col]<=base[i+1]:
>
> data[row,col]=lintrans(data[row,col],(base[i],base[i+1]),(target[i],target[i+1]))
>                 break  #use break to jump out of loop as the data have to
> be ONLY transferred ONCE.
>
>
> Now the profiling result shows that most of the time has been used in this
> loop over the array ("plot_array_transg"),
> and less time in calling the linear transformation fuction "lintrans":
>
>    ncalls     tottime  percall    cumtime    percall
> filename:lineno(function)
>   18047    0.110    0.000      0.110        0.000
> mathex.py:132(lintrans)
>   1            12.495  12.495   19.061      19.061
> mathex.py:196(plot_array_transg)
>
>
> so is there anyway I can speed up this loop?  Thanks for any suggestions!!
>
> best,
>
> Chao
>
>
If the values in base are ascending, you can use searchsorted() to find out
where values from data can be placed into base while maintaining order.
 Don't know if it is faster, but it would certainly be easier to read.

Cheers!
Ben Root
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121117/c0aa7046/attachment.html>

From chaoyuejoy at gmail.com  Sat Nov 17 08:43:04 2012
From: chaoyuejoy at gmail.com (Chao YUE)
Date: Sat, 17 Nov 2012 14:43:04 +0100
Subject: [Numpy-discussion] the fast way to loop over ndarray elements?
In-Reply-To: <CANNq6FmT=kS5M7gRc+SOfLooX_mTi8C71Nyqm_pKfMdSkjBy=Q@mail.gmail.com>
References: <CAAN-aRHEyFe3QaCOXObs3Uu=F1N89KvuiMcwnm1gPnPiLtxfNA@mail.gmail.com>
	<CANNq6FmT=kS5M7gRc+SOfLooX_mTi8C71Nyqm_pKfMdSkjBy=Q@mail.gmail.com>
Message-ID: <CAAN-aRGbON1CTbWYwK14mYJZ5JrCmzT9x_ULjDEmMcsUr=AT6g@mail.gmail.com>

Yes, both the "base" and "target" are ascending.  Thanks!

Chao

On Sat, Nov 17, 2012 at 2:40 PM, Benjamin Root <ben.root at ou.edu> wrote:

>
>
> On Saturday, November 17, 2012, Chao YUE wrote:
>
>> Dear all,
>>
>> I need to make a linear contrast of the 2D numpy array "data" from an
>> interval to another, the approach is:
>> I have another two list: "base" & "target", then I check for each ndarray
>> element "data[i,j]",
>> if   base[m] =< data[i,j] <= base[m+1], then it will be linearly
>> converted to be in the interval of (target[m], target[m+1]),
>> using another function called "lintrans".
>>
>>
>> #The way I do is to loop each row and column of the 2D array, and finally
>> loop the intervals constituted by base list:
>>
>> for row in range(data.shape[0]):
>>     for col in range(data.shape[1]):
>>         for i in range(len(base)-1):
>>             if data[row,col]>=base[i] and data[row,col]<=base[i+1]:
>>
>> data[row,col]=lintrans(data[row,col],(base[i],base[i+1]),(target[i],target[i+1]))
>>                 break  #use break to jump out of loop as the data have to
>> be ONLY transferred ONCE.
>>
>>
>> Now the profiling result shows that most of the time has been used in
>> this loop over the array ("plot_array_transg"),
>> and less time in calling the linear transformation fuction "lintrans":
>>
>>    ncalls     tottime  percall    cumtime    percall
>> filename:lineno(function)
>>   18047    0.110    0.000      0.110        0.000
>> mathex.py:132(lintrans)
>>   1            12.495  12.495   19.061      19.061
>> mathex.py:196(plot_array_transg)
>>
>>
>> so is there anyway I can speed up this loop?  Thanks for any suggestions!!
>>
>> best,
>>
>> Chao
>>
>>
> If the values in base are ascending, you can use searchsorted() to find
> out where values from data can be placed into base while maintaining order.
>  Don't know if it is faster, but it would certainly be easier to read.
>
> Cheers!
> Ben Root
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121117/db77d19c/attachment.html>

From ewm at redtetrahedron.org  Sat Nov 17 09:25:48 2012
From: ewm at redtetrahedron.org (Eric Moore)
Date: Sat, 17 Nov 2012 09:25:48 -0500
Subject: [Numpy-discussion] Assignment function with a signature similar to
	take?
Message-ID: <50A79E6C.7000709@redtetrahedron.org>

Is there a function that operates like 'take' but does assignment? 
Specifically that takes indices and an axis?  As far as I can tell no 
such function exists.  Is there any particular reason?

One can fake such a thing by doing (code untested):

s = len(a.shape)*[np.s_[:]]
s[axis] = np.s_[1::2]
a[s] = b.take(np.arange(1,b.shape[axis],2), axis)

Or by using np.rollaxis:

a = np.rollaxis(a, axis, len(a.shape))
a[..., 1::2] = b[..., 1::2]
a = np.rollaxis(a, len(a.shape)-1, axis)

But I don't really think that either of these are particularly clear, 
but probably prefer the rollaxis solution.

Also, while I'm here, what about having take also be able to use a slice 
object in lieu of a collection of indices?

-Eric


From njs at pobox.com  Sat Nov 17 11:47:39 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 17 Nov 2012 16:47:39 +0000
Subject: [Numpy-discussion] float32 to float64 casting
In-Reply-To: <CAE5kuyinNs5eJ6KtYW38XigY+oKnocw=E4zkYh+M4dU0qFgX4g@mail.gmail.com>
References: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
	<CAE5kuyinNs5eJ6KtYW38XigY+oKnocw=E4zkYh+M4dU0qFgX4g@mail.gmail.com>
Message-ID: <CAPJVwBkd=cJG1pYvu7c+WDUUqVXxz3O0HK=jvc2HzXVsLkW+Ug@mail.gmail.com>

On Fri, Nov 16, 2012 at 9:53 PM, G?khan Sever <gokhansever at gmail.com> wrote:
> Thanks for the explanations.
>
> For either case, I was expecting to get float32 as a resulting data type.
> Since, float32 is large enough to contain the result. I am wondering if
> changing casting rule this way, requires a lot of modification in the NumPy
> code. Maybe as an alternative to the current casting mechanism?
>
> I like the way that NumPy can convert to float64. As if these data-types are
> continuation of each other. But just the conversation might happen too early
> --at least in my opinion, as demonstrated in my example.
>
> For instance comparing this example to IDL surprises me:
>
> I16 np.float32(5555)*5e38
> O16 2.7774999999999998e+42
>
> I17 (np.float32(5555)*5e38).dtype
> O17 dtype('float64')

In this case, what's going on is that 5e38 is a Python float object,
and Python float objects have double-precision, i.e., they're
equivalent to np.float64's. So you're multiplying a float32 and a
float64. I think most people will agree that in this situation it's
better to use float64 for the output?

-n


From gokhansever at gmail.com  Sat Nov 17 14:49:44 2012
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Sat, 17 Nov 2012 12:49:44 -0700
Subject: [Numpy-discussion] float32 to float64 casting
In-Reply-To: <CAPJVwBkd=cJG1pYvu7c+WDUUqVXxz3O0HK=jvc2HzXVsLkW+Ug@mail.gmail.com>
References: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
	<CAE5kuyinNs5eJ6KtYW38XigY+oKnocw=E4zkYh+M4dU0qFgX4g@mail.gmail.com>
	<CAPJVwBkd=cJG1pYvu7c+WDUUqVXxz3O0HK=jvc2HzXVsLkW+Ug@mail.gmail.com>
Message-ID: <CAE5kuyhyeHj4JNT5c8LPWzFtcA_Crj52LzN0-kqf3zg7v-jtpQ@mail.gmail.com>

On Sat, Nov 17, 2012 at 9:47 AM, Nathaniel Smith <njs at pobox.com> wrote:

> On Fri, Nov 16, 2012 at 9:53 PM, G?khan Sever <gokhansever at gmail.com>
> wrote:
> > Thanks for the explanations.
> >
> > For either case, I was expecting to get float32 as a resulting data type.
> > Since, float32 is large enough to contain the result. I am wondering if
> > changing casting rule this way, requires a lot of modification in the
> NumPy
> > code. Maybe as an alternative to the current casting mechanism?
> >
> > I like the way that NumPy can convert to float64. As if these data-types
> are
> > continuation of each other. But just the conversation might happen too
> early
> > --at least in my opinion, as demonstrated in my example.
> >
> > For instance comparing this example to IDL surprises me:
> >
> > I16 np.float32(5555)*5e38
> > O16 2.7774999999999998e+42
> >
> > I17 (np.float32(5555)*5e38).dtype
> > O17 dtype('float64')
>
> In this case, what's going on is that 5e38 is a Python float object,
> and Python float objects have double-precision, i.e., they're
> equivalent to np.float64's. So you're multiplying a float32 and a
> float64. I think most people will agree that in this situation it's
> better to use float64 for the output?
>
> -n
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

OK, I see your point. Python numeric data objects and NumPy data objects
mixed operations require more attention.

The following causes float32 overflow --rather than casting to float64 as
in the case for Python float multiplication, and behaves like in IDL.

I3 (np.float32(5555)*np.float32(5e38))
O3 inf

However, these two still surprises me:

I5 (np.float32(5555)*1).dtype
O5 dtype('float64')

I6 (np.float32(5555)*np.int32(1)).dtype
O6 dtype('float64')
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121117/e6814c4c/attachment.html>

From shish at keba.be  Sat Nov 17 15:00:14 2012
From: shish at keba.be (Olivier Delalleau)
Date: Sat, 17 Nov 2012 15:00:14 -0500
Subject: [Numpy-discussion] float32 to float64 casting
In-Reply-To: <CAE5kuyhyeHj4JNT5c8LPWzFtcA_Crj52LzN0-kqf3zg7v-jtpQ@mail.gmail.com>
References: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
	<CAE5kuyinNs5eJ6KtYW38XigY+oKnocw=E4zkYh+M4dU0qFgX4g@mail.gmail.com>
	<CAPJVwBkd=cJG1pYvu7c+WDUUqVXxz3O0HK=jvc2HzXVsLkW+Ug@mail.gmail.com>
	<CAE5kuyhyeHj4JNT5c8LPWzFtcA_Crj52LzN0-kqf3zg7v-jtpQ@mail.gmail.com>
Message-ID: <CAFXk4bregDOEHFH34WKPCjDUuEJDAwaSQr2-fLRZf=ML0D95Vg@mail.gmail.com>

2012/11/17 G?khan Sever <gokhansever at gmail.com>

>
>
> On Sat, Nov 17, 2012 at 9:47 AM, Nathaniel Smith <njs at pobox.com> wrote:
>
>> On Fri, Nov 16, 2012 at 9:53 PM, G?khan Sever <gokhansever at gmail.com>
>> wrote:
>> > Thanks for the explanations.
>> >
>> > For either case, I was expecting to get float32 as a resulting data
>> type.
>> > Since, float32 is large enough to contain the result. I am wondering if
>> > changing casting rule this way, requires a lot of modification in the
>> NumPy
>> > code. Maybe as an alternative to the current casting mechanism?
>> >
>> > I like the way that NumPy can convert to float64. As if these
>> data-types are
>> > continuation of each other. But just the conversation might happen too
>> early
>> > --at least in my opinion, as demonstrated in my example.
>> >
>> > For instance comparing this example to IDL surprises me:
>> >
>> > I16 np.float32(5555)*5e38
>> > O16 2.7774999999999998e+42
>> >
>> > I17 (np.float32(5555)*5e38).dtype
>> > O17 dtype('float64')
>>
>> In this case, what's going on is that 5e38 is a Python float object,
>> and Python float objects have double-precision, i.e., they're
>> equivalent to np.float64's. So you're multiplying a float32 and a
>> float64. I think most people will agree that in this situation it's
>> better to use float64 for the output?
>>
>> -n
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> OK, I see your point. Python numeric data objects and NumPy data objects
> mixed operations require more attention.
>
> The following causes float32 overflow --rather than casting to float64 as
> in the case for Python float multiplication, and behaves like in IDL.
>
> I3 (np.float32(5555)*np.float32(5e38))
> O3 inf
>
> However, these two still surprises me:
>
> I5 (np.float32(5555)*1).dtype
> O5 dtype('float64')
>
> I6 (np.float32(5555)*np.int32(1)).dtype
> O6 dtype('float64')
>

That's because the current way of finding out the result's dtype is based
on input dtypes only (not on numeric values), and numpy.can_cast('int32',
'float32') is False, while numpy.can_cast('int32', 'float64') is True (and
same for int64).
Thus it decides to cast to float64.

-=- Olivier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121117/70ee105f/attachment.html>

From charlesr.harris at gmail.com  Sat Nov 17 15:18:56 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 17 Nov 2012 13:18:56 -0700
Subject: [Numpy-discussion] float32 to float64 casting
In-Reply-To: <CAFXk4bregDOEHFH34WKPCjDUuEJDAwaSQr2-fLRZf=ML0D95Vg@mail.gmail.com>
References: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
	<CAE5kuyinNs5eJ6KtYW38XigY+oKnocw=E4zkYh+M4dU0qFgX4g@mail.gmail.com>
	<CAPJVwBkd=cJG1pYvu7c+WDUUqVXxz3O0HK=jvc2HzXVsLkW+Ug@mail.gmail.com>
	<CAE5kuyhyeHj4JNT5c8LPWzFtcA_Crj52LzN0-kqf3zg7v-jtpQ@mail.gmail.com>
	<CAFXk4bregDOEHFH34WKPCjDUuEJDAwaSQr2-fLRZf=ML0D95Vg@mail.gmail.com>
Message-ID: <CAB6mnxL8k+1s7qhoKv4+e+JXWE6wd_6j+ZgqTmBnE0vp6sCT_A@mail.gmail.com>

On Sat, Nov 17, 2012 at 1:00 PM, Olivier Delalleau <shish at keba.be> wrote:

> 2012/11/17 G?khan Sever <gokhansever at gmail.com>
>
>>
>>
>> On Sat, Nov 17, 2012 at 9:47 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>>> On Fri, Nov 16, 2012 at 9:53 PM, G?khan Sever <gokhansever at gmail.com>
>>> wrote:
>>> > Thanks for the explanations.
>>> >
>>> > For either case, I was expecting to get float32 as a resulting data
>>> type.
>>> > Since, float32 is large enough to contain the result. I am wondering if
>>> > changing casting rule this way, requires a lot of modification in the
>>> NumPy
>>> > code. Maybe as an alternative to the current casting mechanism?
>>> >
>>> > I like the way that NumPy can convert to float64. As if these
>>> data-types are
>>> > continuation of each other. But just the conversation might happen too
>>> early
>>> > --at least in my opinion, as demonstrated in my example.
>>> >
>>> > For instance comparing this example to IDL surprises me:
>>> >
>>> > I16 np.float32(5555)*5e38
>>> > O16 2.7774999999999998e+42
>>> >
>>> > I17 (np.float32(5555)*5e38).dtype
>>> > O17 dtype('float64')
>>>
>>> In this case, what's going on is that 5e38 is a Python float object,
>>> and Python float objects have double-precision, i.e., they're
>>> equivalent to np.float64's. So you're multiplying a float32 and a
>>> float64. I think most people will agree that in this situation it's
>>> better to use float64 for the output?
>>>
>>> -n
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>> OK, I see your point. Python numeric data objects and NumPy data objects
>> mixed operations require more attention.
>>
>> The following causes float32 overflow --rather than casting to float64 as
>> in the case for Python float multiplication, and behaves like in IDL.
>>
>> I3 (np.float32(5555)*np.float32(5e38))
>> O3 inf
>>
>> However, these two still surprises me:
>>
>> I5 (np.float32(5555)*1).dtype
>> O5 dtype('float64')
>>
>> I6 (np.float32(5555)*np.int32(1)).dtype
>> O6 dtype('float64')
>>
>
> That's because the current way of finding out the result's dtype is based
> on input dtypes only (not on numeric values), and numpy.can_cast('int32',
> 'float32') is False, while numpy.can_cast('int32', 'float64') is True (and
> same for int64).
> Thus it decides to cast to float64.
>

It might be nice to revisit all the casting rules at some point, but
current experience suggests that any changes will lead to cries of pain and
outrage ;)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121117/cc4159dd/attachment.html>

From ben.root at ou.edu  Sat Nov 17 15:52:11 2012
From: ben.root at ou.edu (Benjamin Root)
Date: Sat, 17 Nov 2012 15:52:11 -0500
Subject: [Numpy-discussion] float32 to float64 casting
In-Reply-To: <CAB6mnxL8k+1s7qhoKv4+e+JXWE6wd_6j+ZgqTmBnE0vp6sCT_A@mail.gmail.com>
References: <CAE5kuyi2xgpb3BLVSxWs+suKStAPMkoqq=YF0Ti-3oKzqxWd_w@mail.gmail.com>
	<CAE5kuyinNs5eJ6KtYW38XigY+oKnocw=E4zkYh+M4dU0qFgX4g@mail.gmail.com>
	<CAPJVwBkd=cJG1pYvu7c+WDUUqVXxz3O0HK=jvc2HzXVsLkW+Ug@mail.gmail.com>
	<CAE5kuyhyeHj4JNT5c8LPWzFtcA_Crj52LzN0-kqf3zg7v-jtpQ@mail.gmail.com>
	<CAFXk4bregDOEHFH34WKPCjDUuEJDAwaSQr2-fLRZf=ML0D95Vg@mail.gmail.com>
	<CAB6mnxL8k+1s7qhoKv4+e+JXWE6wd_6j+ZgqTmBnE0vp6sCT_A@mail.gmail.com>
Message-ID: <CANNq6Fk7WP2Qb8HX3=toiikaOUa4h_rmZ+xUt8xpsvGAsX4oeg@mail.gmail.com>

On Saturday, November 17, 2012, Charles R Harris wrote:

>
>
> On Sat, Nov 17, 2012 at 1:00 PM, Olivier Delalleau <shish at keba.be<javascript:_e({}, 'cvml', 'shish at keba.be');>
> > wrote:
>
>> 2012/11/17 G?khan Sever <gokhansever at gmail.com <javascript:_e({},
>> 'cvml', 'gokhansever at gmail.com');>>
>>
>>>
>>>
>>> On Sat, Nov 17, 2012 at 9:47 AM, Nathaniel Smith <njs at pobox.com<javascript:_e({}, 'cvml', 'njs at pobox.com');>
>>> > wrote:
>>>
>>>> On Fri, Nov 16, 2012 at 9:53 PM, G?khan Sever <gokhansever at gmail.com<javascript:_e({}, 'cvml', 'gokhansever at gmail.com');>>
>>>> wrote:
>>>> > Thanks for the explanations.
>>>> >
>>>> > For either case, I was expecting to get float32 as a resulting data
>>>> type.
>>>> > Since, float32 is large enough to contain the result. I am wondering
>>>> if
>>>> > changing casting rule this way, requires a lot of modification in the
>>>> NumPy
>>>> > code. Maybe as an alternative to the current casting mechanism?
>>>> >
>>>> > I like the way that NumPy can convert to float64. As if these
>>>> data-types are
>>>> > continuation of each other. But just the conversation might happen
>>>> too early
>>>> > --at least in my opinion, as demonstrated in my example.
>>>> >
>>>> > For instance comparing this example to IDL surprises me:
>>>> >
>>>> > I16 np.float32(5555)*5e38
>>>> > O16 2.7774999999999998e+42
>>>> >
>>>> > I17 (np.float32(5555)*5e38).dtype
>>>> > O17 dtype('float64')
>>>>
>>>> In this case, what's going on is that 5e38 is a Python float object,
>>>> and Python float objects have double-precision, i.e., they're
>>>> equivalent to np.float64's. So you're multiplying a float32 and a
>>>> float64. I think most people will agree that in this situation it's
>>>> better to use float64 for the output?
>>>>
>>>> -n
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at scipy.org <javascript:_e({}, 'cvml',
>>>> 'NumPy-Discussion at scipy.org');>
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>
>>>
>>> OK, I see your point. Python numeric data objects and NumPy data objects
>>> mixed operations require more attention.
>>>
>>> The following causes float32 overflow --rather than casting to float64
>>> as in the case for Python float multiplication, and behaves like in IDL.
>>>
>>> I3 (np.float32(5555)*np.float32(5e38))
>>> O3 inf
>>>
>>> However, these two still surprises me:
>>>
>>> I5 (np.float32(5555)*1).dtype
>>> O5 dtype('float64')
>>>
>>> I6 (np.float32(5555)*np.int32(1)).dtype
>>> O6 dtype('float64')
>>>
>>
>> That's because the current way of finding out the result's dtype is based
>> on input dtypes only (not on numeric values), and numpy.can_cast('int32',
>> 'float32') is False, while numpy.can_cast('int32', 'float64') is True (and
>> same for int64).
>> Thus it decides to cast to float64.
>>
>
> It might be nice to revisit all the casting rules at some point, but
> current experience suggests that any changes will lead to cries of pain and
> outrage ;)
>
> Chuck
>
>
Can we at least put these examples into the tests?  Also, I think the
bigger issue was that, unlike deprecation of a function, it is much harder
to grep for particular operations, especially in a dynamic language like
python. What were intended as minor bugfixes ended up becoming much larger.

Has the casting table been added to the tests?  I think that will bring
much more confidence and assurances for future changes going forward.

Cheers!
Ben Root
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121117/6c4b40c7/attachment.html>

From d.warde.farley at gmail.com  Sun Nov 18 17:10:17 2012
From: d.warde.farley at gmail.com (David Warde-Farley)
Date: Sun, 18 Nov 2012 17:10:17 -0500
Subject: [Numpy-discussion] the fast way to loop over ndarray elements?
In-Reply-To: <CAAN-aRHEyFe3QaCOXObs3Uu=F1N89KvuiMcwnm1gPnPiLtxfNA@mail.gmail.com>
References: <CAAN-aRHEyFe3QaCOXObs3Uu=F1N89KvuiMcwnm1gPnPiLtxfNA@mail.gmail.com>
Message-ID: <CALc6Xo6wR8NyAUAaLp0Q2NzTJnx45uusv_dyO_DgY9=Onhjriw@mail.gmail.com>

This looks like a prime candidate for using Cython:

http://docs.cython.org/src/userguide/numpy_tutorial.html


On Sat, Nov 17, 2012 at 8:28 AM, Chao YUE <chaoyuejoy at gmail.com> wrote:

> Dear all,
>
> I need to make a linear contrast of the 2D numpy array "data" from an
> interval to another, the approach is:
> I have another two list: "base" & "target", then I check for each ndarray
> element "data[i,j]",
> if   base[m] =< data[i,j] <= base[m+1], then it will be linearly converted
> to be in the interval of (target[m], target[m+1]),
> using another function called "lintrans".
>
>
> #The way I do is to loop each row and column of the 2D array, and finally
> loop the intervals constituted by base list:
>
> for row in range(data.shape[0]):
>     for col in range(data.shape[1]):
>         for i in range(len(base)-1):
>             if data[row,col]>=base[i] and data[row,col]<=base[i+1]:
>
> data[row,col]=lintrans(data[row,col],(base[i],base[i+1]),(target[i],target[i+1]))
>                 break  #use break to jump out of loop as the data have to
> be ONLY transferred ONCE.
>
>
> Now the profiling result shows that most of the time has been used in this
> loop over the array ("plot_array_transg"),
> and less time in calling the linear transformation fuction "lintrans":
>
>    ncalls     tottime  percall    cumtime    percall
> filename:lineno(function)
>   18047    0.110    0.000      0.110        0.000
> mathex.py:132(lintrans)
>   1            12.495  12.495   19.061      19.061
> mathex.py:196(plot_array_transg)
>
>
> so is there anyway I can speed up this loop?  Thanks for any suggestions!!
>
> best,
>
> Chao
>
> --
>
> ***********************************************************************************
> Chao YUE
> Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
> UMR 1572 CEA-CNRS-UVSQ
> Batiment 712 - Pe 119
> 91191 GIF Sur YVETTE Cedex
> Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
>
> ************************************************************************************
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121118/8837c04a/attachment.html>

From thouis at gmail.com  Mon Nov 19 09:08:38 2012
From: thouis at gmail.com (Thouis (Ray) Jones)
Date: Mon, 19 Nov 2012 09:08:38 -0500
Subject: [Numpy-discussion] the fast way to loop over ndarray elements?
In-Reply-To: <CAAN-aRHEyFe3QaCOXObs3Uu=F1N89KvuiMcwnm1gPnPiLtxfNA@mail.gmail.com>
References: <CAAN-aRHEyFe3QaCOXObs3Uu=F1N89KvuiMcwnm1gPnPiLtxfNA@mail.gmail.com>
Message-ID: <CAHGWWxH2FXLPnCkse09h-KMUcmGHK7yPX7MBOy+B4skG4DNFTg@mail.gmail.com>

On Sat, Nov 17, 2012 at 8:28 AM, Chao YUE <chaoyuejoy at gmail.com> wrote:
> Dear all,
>
> I need to make a linear contrast of the 2D numpy array "data" from an
> interval to another, the approach is:
> I have another two list: "base" & "target", then I check for each ndarray
> element "data[i,j]",
> if   base[m] =< data[i,j] <= base[m+1], then it will be linearly converted
> to be in the interval of (target[m], target[m+1]),
> using another function called "lintrans".
>
>
> #The way I do is to loop each row and column of the 2D array, and finally
> loop the intervals constituted by base list:
>
> for row in range(data.shape[0]):
>     for col in range(data.shape[1]):
>         for i in range(len(base)-1):
>             if data[row,col]>=base[i] and data[row,col]<=base[i+1]:
>
> data[row,col]=lintrans(data[row,col],(base[i],base[i+1]),(target[i],target[i+1]))
>                 break  #use break to jump out of loop as the data have to be
> ONLY transferred ONCE.
>
>
> Now the profiling result shows that most of the time has been used in this
> loop over the array ("plot_array_transg"),
> and less time in calling the linear transformation fuction "lintrans":
>
>    ncalls     tottime  percall    cumtime    percall
> filename:lineno(function)
>   18047    0.110    0.000      0.110        0.000
> mathex.py:132(lintrans)
>   1            12.495  12.495   19.061      19.061
> mathex.py:196(plot_array_transg)
>
>
> so is there anyway I can speed up this loop?  Thanks for any suggestions!!
>
> best,
>
> Chao

If lintrans() is a linear interpolation, could you use interp?

http://docs.scipy.org/doc/numpy/reference/generated/numpy.interp.html


From sturla at molden.no  Mon Nov 19 12:12:23 2012
From: sturla at molden.no (Sturla Molden)
Date: Mon, 19 Nov 2012 18:12:23 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
Message-ID: <50AA6877.7010806@molden.no>

I think NumPy and SciPy should consider to use OpenBLAS (a fork of 
GotoBLAS2) instead of ATLAS or f2c'd Netlib BLAS for the binary releases.

Here are its virtues:

* Very easy to build: Just a makefile, no configuration script or 
special build tools.

* Building ATLAS can be a PITA. So why bother?

* Faster than ATLAS, sometimes faster than MKL.

* Multithreaded BLAS kernels: OpenMP on Unix, Windows threads on Windows.

* The quality of its ancestor GotoBLAS is undisputed. I was the BLAS 
implementation of choice for major HPC projects around the World.

* Free as in BSD licensed.

* Funded and developed for use in major Chinese HPC projects. Actively 
maintained. (GotoBLAS2 is abandonware.)

* Open source. The C sources are a pleasure to read, and very easy to 
verify.

* No OpenMP on Windows means no dependency on pthreads-win32 (an LGPL 
library) when building with MinGW.

* Builds on Windows with MinGW and MSYS, and perhaps even without MSYS.

* Cygwin is not needed on Windows (this is just BS from the GotoBLAS 
documentation). Thus, 64-buit builds are possible (I've built it using 
TDM-GCC for Win64 and 32-bit MSYS).


Sturla


From daniele at grinta.net  Mon Nov 19 12:20:10 2012
From: daniele at grinta.net (Daniele Nicolodi)
Date: Mon, 19 Nov 2012 18:20:10 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50AA6877.7010806@molden.no>
References: <50AA6877.7010806@molden.no>
Message-ID: <50AA6A4A.1060708@grinta.net>

On 19/11/2012 18:12, Sturla Molden wrote:
> I think NumPy and SciPy should consider to use OpenBLAS (a fork of 
> GotoBLAS2) instead of ATLAS or f2c'd Netlib BLAS for the binary releases.
...
> * Funded and developed for use in major Chinese HPC projects. Actively 
> maintained. (GotoBLAS2 is abandonware.)

Hello Sturla,

do you know why GotoBLAS2 is not worked on anymore? It looked like a
project with serious university backing and its web page does not
reflect the fact that it has been discontinued by its authors.

Thanks.

Cheers,
Daniele



From nouiz at nouiz.org  Mon Nov 19 12:30:02 2012
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Mon, 19 Nov 2012 12:30:02 -0500
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50AA6A4A.1060708@grinta.net>
References: <50AA6877.7010806@molden.no> <50AA6A4A.1060708@grinta.net>
Message-ID: <CADKKbthsdUjbJ_mUGobsHw7WoowBAg=Ypj_YXSWeQ=3jFX-=qQ@mail.gmail.com>

Hi,

it was mainly developed by one person and he left for Intel. They
relicensed GotoBLAS to BSD 3 clauses and some Chinese people forked it to
OpenBLAS. There was another fork, but I didn't got news of it. Maybe I just
missed the news.

Fred



On Mon, Nov 19, 2012 at 12:20 PM, Daniele Nicolodi <daniele at grinta.net>wrote:

> On 19/11/2012 18:12, Sturla Molden wrote:
> > I think NumPy and SciPy should consider to use OpenBLAS (a fork of
> > GotoBLAS2) instead of ATLAS or f2c'd Netlib BLAS for the binary releases.
> ...
> > * Funded and developed for use in major Chinese HPC projects. Actively
> > maintained. (GotoBLAS2 is abandonware.)
>
> Hello Sturla,
>
> do you know why GotoBLAS2 is not worked on anymore? It looked like a
> project with serious university backing and its web page does not
> reflect the fact that it has been discontinued by its authors.
>
> Thanks.
>
> Cheers,
> Daniele
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121119/a5bb42a3/attachment.html>

From d.s.seljebotn at astro.uio.no  Mon Nov 19 12:42:58 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Mon, 19 Nov 2012 18:42:58 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50AA6877.7010806@molden.no>
References: <50AA6877.7010806@molden.no>
Message-ID: <50AA6FA2.4010101@astro.uio.no>

On 11/19/2012 06:12 PM, Sturla Molden wrote:
> I think NumPy and SciPy should consider to use OpenBLAS (a fork of
> GotoBLAS2) instead of ATLAS or f2c'd Netlib BLAS for the binary releases.
>
> Here are its virtues:
>
> * Very easy to build: Just a makefile, no configuration script or
> special build tools.
>
> * Building ATLAS can be a PITA. So why bother?
>
> * Faster than ATLAS, sometimes faster than MKL.
>
> * Multithreaded BLAS kernels: OpenMP on Unix, Windows threads on Windows.
>
> * The quality of its ancestor GotoBLAS is undisputed. I was the BLAS
> implementation of choice for major HPC projects around the World.
>
> * Free as in BSD licensed.
>
> * Funded and developed for use in major Chinese HPC projects. Actively
> maintained. (GotoBLAS2 is abandonware.)
>
> * Open source. The C sources are a pleasure to read, and very easy to
> verify.
>
> * No OpenMP on Windows means no dependency on pthreads-win32 (an LGPL
> library) when building with MinGW.
>
> * Builds on Windows with MinGW and MSYS, and perhaps even without MSYS.
>
> * Cygwin is not needed on Windows (this is just BS from the GotoBLAS
> documentation). Thus, 64-buit builds are possible (I've built it using
> TDM-GCC for Win64 and 32-bit MSYS).

+1.

Even on CPUs that are not directly supported, this is at least better 
than reference BLAS.

(On our AMD CPUs, which are too new to have a separate OpenBLAS 
implementation, the implementations for older AMD CPUs still outperform 
at least Intel MKL, because MKL does so poorly on these -- although ACML 
beats them both by a factor 2. And of course on supported CPUs 
(everything Intel and older AMD) OpenBLAS is wonderful.

Dag Sverre


From sturla at molden.no  Mon Nov 19 13:27:38 2012
From: sturla at molden.no (Sturla Molden)
Date: Mon, 19 Nov 2012 19:27:38 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50AA6FA2.4010101@astro.uio.no>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
Message-ID: <50AA7A1A.4070307@molden.no>

On 19.11.2012 18:42, Dag Sverre Seljebotn wrote:

> Even on CPUs that are not directly supported, this is at least better
> than reference BLAS.
>
> (On our AMD CPUs, which are too new to have a separate OpenBLAS
> implementation, the implementations for older AMD CPUs still outperform
> at least Intel MKL, because MKL does so poorly on these -- although ACML
> beats them both by a factor 2. And of course on supported CPUs
> (everything Intel and older AMD) OpenBLAS is wonderful.

Indeed.

I have a laptop with a quadcore AMD Phenom II, but ACML consistently 
segfaults when used with MinGW (gcc or gfortran).

With another set of compilers (Microsoft C and Absoft Fortran), ACML 
usually works, but sometimes decides my CPU is not supported, and just 
terminates the process.

I'm not sure NumPy even builds against ACML, e.g. it has a non-standard 
C interface for BLAS and LAPACK. Also, it wouldn't work with GNU 
compilers on Windows. Dependency on Intel or PGI run-time DLLs would 
also make it useless for the binary superpacks.

OTOH, OpenBLAS works wonderfully :)


Sturla



From ralf.gommers at gmail.com  Mon Nov 19 15:54:24 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 19 Nov 2012 21:54:24 +0100
Subject: [Numpy-discussion] PRs for MaskedArray bugs
In-Reply-To: <CAGMHX_3bSuP3SkKHuuAdJNUwpOfD1mgJ3JSZXdLvVwnNdPwBOw@mail.gmail.com>
References: <CAGMHX_3bSuP3SkKHuuAdJNUwpOfD1mgJ3JSZXdLvVwnNdPwBOw@mail.gmail.com>
Message-ID: <CABL7CQhGUdpQOvh6yqYswKKinP5vEb4Rvz+Hh7mKdz3wPNXkMA@mail.gmail.com>

On Wed, Nov 14, 2012 at 8:46 PM, Thomas Robitaille <
thomas.robitaille at gmail.com> wrote:

> I've recently opened a couple of pull requests that fix bugs with
> MaskedArray - these are pretty straightforward, so would it be
> possible to consider them in time for 1.7?
>
> https://github.com/numpy/numpy/pull/2703
>

This one could be considered a change in behavior, but it looks more like a
bug to me. Should be OK to backport imho.


> https://github.com/numpy/numpy/pull/2733
>

This one is definitely OK to backport.

Ralf


>
> Thanks!
> Tom
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121119/3141d5dd/attachment.html>

From vs at it.uu.se  Mon Nov 19 18:17:53 2012
From: vs at it.uu.se (Virgil Stokes)
Date: Tue, 20 Nov 2012 00:17:53 +0100
Subject: [Numpy-discussion] Confused with qr decomposition function
Message-ID: <50AABE21.3000307@it.uu.se>

I am using the latest versions of numpy (from 
numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from 
scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit) 
platform.

I have used

import numpy as np
q,r = np.linalg.qr(A)

and compared the results obtained  from MATLAB (R2010B)

[q,r] = qr(A)

The q,r returned from numpy are both the negative of theq,r returned 
from MATLAB for the same matrix A. I believe that the q,r returned from 
MATLAB are correct. Why am I getting their negative from numpy?

Note, I have tried this on several different matrices ---  numpy always 
gives the negative of MATLAB's q,r values.
[I mistakenly have already sent a similar email to the scipy list --- 
please excuse this mistake.]


From charlesr.harris at gmail.com  Mon Nov 19 18:30:53 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 19 Nov 2012 16:30:53 -0700
Subject: [Numpy-discussion] Confused with qr decomposition function
In-Reply-To: <50AABE21.3000307@it.uu.se>
References: <50AABE21.3000307@it.uu.se>
Message-ID: <CAB6mnxJcx0_KU+n3oUQtixKax7+d8rbWap+QJq_dxat=uT+E-w@mail.gmail.com>

On Mon, Nov 19, 2012 at 4:17 PM, Virgil Stokes <vs at it.uu.se> wrote:

> I am using the latest versions of numpy (from
> numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from
> scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit)
> platform.
>
> I have used
>
> import numpy as np
> q,r = np.linalg.qr(A)
>
> and compared the results obtained  from MATLAB (R2010B)
>
> [q,r] = qr(A)
>
> The q,r returned from numpy are both the negative of theq,r returned
> from MATLAB for the same matrix A. I believe that the q,r returned from
> MATLAB are correct. Why am I getting their negative from numpy?
>
> Note, I have tried this on several different matrices ---  numpy always
> gives the negative of MATLAB's q,r values.
> [I mistakenly have already sent a similar email to the scipy list ---
> please excuse this mistake.]
>

They are both correct, the decomposition isn't unique. In particular, if
both algorithms use Housholder reflections there are two possible
reflection planes at each step, one of which is more numerically stable
than the other, and the two choices lead to different signs at each step.
That said, MATLAB may be normalizing the result in some way or using some
other algorithm.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121119/e326b009/attachment.html>

From vs at it.uu.se  Tue Nov 20 08:24:48 2012
From: vs at it.uu.se (Virgil Stokes)
Date: Tue, 20 Nov 2012 14:24:48 +0100
Subject: [Numpy-discussion] Confused with qr decomposition function
In-Reply-To: <CAB6mnxJcx0_KU+n3oUQtixKax7+d8rbWap+QJq_dxat=uT+E-w@mail.gmail.com>
References: <50AABE21.3000307@it.uu.se>
	<CAB6mnxJcx0_KU+n3oUQtixKax7+d8rbWap+QJq_dxat=uT+E-w@mail.gmail.com>
Message-ID: <50AB84A0.6030609@it.uu.se>

On 20-Nov-2012 00:30, Charles R Harris wrote:
>
>
> On Mon, Nov 19, 2012 at 4:17 PM, Virgil Stokes <vs at it.uu.se 
> <mailto:vs at it.uu.se>> wrote:
>
>     I am using the latest versions of numpy (from
>     numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from
>     scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit)
>     platform.
>
>     I have used
>
>     import numpy as np
>     q,r = np.linalg.qr(A)
>
>     and compared the results obtained  from MATLAB (R2010B)
>
>     [q,r] = qr(A)
>
>     The q,r returned from numpy are both the negative of theq,r returned
>     from MATLAB for the same matrix A. I believe that the q,r returned from
>     MATLAB are correct. Why am I getting their negative from numpy?
>
>     Note, I have tried this on several different matrices ---  numpy always
>     gives the negative of MATLAB's q,r values.
>     [I mistakenly have already sent a similar email to the scipy list ---
>     please excuse this mistake.]
>
>
> They are both correct, the decomposition isn't unique. In particular, if both 
> algorithms use Housholder reflections there are two possible reflection planes 
> at each step, one of which is more numerically stable than the other, and the 
> two choices lead to different signs at each step. That said, MATLAB may be 
> normalizing the result in some way or using some other algorithm.
>
> Chuck
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
I am aware that they are both correct; but, if you are doing covariance QR 
decomposition then you almost surely are interested in the positive results 
(which is the default for MATLAB and most papers/books on this subject).

This problem came up recently when converting MATLAB code to python/numpy, with 
some Kalman Filter/Smooth code. It took some time to isolate the problem. It 
would useful if the documentation (numpy) stated explicitly what the default 
"signs" are for q,r.

Thanks for your interest in the problem, Chuck.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121120/9e0cf37c/attachment.html>

From mdroe at stsci.edu  Tue Nov 20 08:32:55 2012
From: mdroe at stsci.edu (Michael Droettboom)
Date: Tue, 20 Nov 2012 08:32:55 -0500
Subject: [Numpy-discussion] PRs for MaskedArray bugs
In-Reply-To: <CAGMHX_3bSuP3SkKHuuAdJNUwpOfD1mgJ3JSZXdLvVwnNdPwBOw@mail.gmail.com>
References: <CAGMHX_3bSuP3SkKHuuAdJNUwpOfD1mgJ3JSZXdLvVwnNdPwBOw@mail.gmail.com>
Message-ID: <50AB8687.5050005@stsci.edu>

If I may also point out another simple (but critical for astropy) bugfix 
related to masked arrays:

https://github.com/numpy/numpy/pull/2747

Mike

On 11/14/2012 02:46 PM, Thomas Robitaille wrote:
> I've recently opened a couple of pull requests that fix bugs with
> MaskedArray - these are pretty straightforward, so would it be
> possible to consider them in time for 1.7?
>
> https://github.com/numpy/numpy/pull/2703
>
> https://github.com/numpy/numpy/pull/2733
>
> Thanks!
> Tom
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121120/2691fccf/attachment.html>

From p.j.a.cock at googlemail.com  Tue Nov 20 08:59:08 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Tue, 20 Nov 2012 13:59:08 +0000
Subject: [Numpy-discussion] Compiling NumPy on Windows for Python 3.3
 with MSVC 2010
In-Reply-To: <50A61089.2050906@uci.edu>
References: <CAKVJ-_4aO2zTXm7iPB0wzuWvNDo5PohqCWNPZ1vG5FOAz18aYw@mail.gmail.com>
	<50A3F27F.70605@uci.edu>
	<CAKVJ-_6QnF+4bGkGS=i+5cLWxg405Ffjy3g4MBRT015yJ2UxGQ@mail.gmail.com>
	<50A53147.6060608@uci.edu>
	<CAKVJ-_7W2mmT=m9_WpHxOL+WBDzaMZ+fThNpWC5rTdVwVtuagw@mail.gmail.com>
	<50A61089.2050906@uci.edu>
Message-ID: <CAKVJ-_6GN-_R8MSJrJn44rtJVuB5g-Mi0KN7vFh2pykaN-2LxQ@mail.gmail.com>

On Fri, Nov 16, 2012 at 10:08 AM, Christoph Gohlke <cgohlke at uci.edu> wrote:
> On 11/16/2012 1:28 AM, Peter Cock wrote:
>> On Thu, Nov 15, 2012 at 6:15 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:
>>>
>>> Naturally the file would be named msvc10compiler.py but the name may be
>>> kept for compatibility reasons. AFAIK msvc10 does not use manifests any
>>> longer for the CRT dependencies and all the code handling msvc9
>>> manifests could be removed for Python 3.3. I have been building
>>> extensions for Python 3.3 with msvc10 and this distutils patch for some
>>> months and did not notice any issues.
>>>
>>
>> Sounds Python 3.3 needs a fix then - have you reported this?
>> If not, could you report it (since you know far more about the
>> Windows build system than I do)?
>>
>> If it will be fixed in Python itself, then perhaps a manual hack like
>> this will be enough for NumPy in the short term. Otherwise, maybe
>> numpy needs to include its own copy of msvc9compiler.py (or
>> msvc10compiler.py)?
>>
>> Thanks,
>>
>> Peter
>
> Could be related to <http://bugs.python.org/issue16296>.
>
> Christoph

Thanks Christoph, you're probably right this is linked to
http://bugs.python.org/issue16296

Note here's an example of the manifest file, obtained from a hack
to Python 3.3's distutitls/msvc9compiler.py - looks like there are
no MSVC version numbers in here that we would need to
worry about:

build\temp.win32-3.3\Release\numpy\core\src\_dummy.pyd.manifest
<?xml version='1.0' encoding='UTF-8' standalone='yes'?>
<assembly xmlns='urn:schemas-microsoft-com:asm.v1' manifestVersion='1.0'>
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
    <security>
      <requestedPrivileges>
        <requestedExecutionLevel level='asInvoker' uiAccess='false' />
      </requestedPrivileges>
    </security>
  </trustInfo>
</assembly>

I tried the patch from http://bugs.python.org/issue16296
applied by hand (in place of Christoph's one line change),
and it also seemed to work. I wanted to double check this,
so started by reverting to an unmodified copy of Python 3.3.

I just removed Python 3.3, and reinstalled it afresh using
python-3.3.0.msi, then updated to the latest commit on the
master branch of numpy, as it happens Ralf merging my
fixes to get mingw32  to compile numpy Python 3.3:
724da615902b9feb140cb6f7307ff1b1c2596a40

Now a clean numpy build under Python 3.3 with MSVC 10
"just worked", the error "Broken toolchain: cannot link a simple
C program" has gone. The comments in msvc9compiler.py did
mention this manifest stuff was fragile... but I am puzzled.

My hunch right now is that the order of installation of
MSVC 2010 and Python 3.3 could be important. Either
that, or something else changed on the numpy master
which had an impact?

Regards,

Peter


From cournape at gmail.com  Tue Nov 20 09:38:27 2012
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 20 Nov 2012 14:38:27 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50AA6FA2.4010101@astro.uio.no>
References: <50AA6877.7010806@molden.no>
	<50AA6FA2.4010101@astro.uio.no>
Message-ID: <CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>

On Mon, Nov 19, 2012 at 5:42 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 11/19/2012 06:12 PM, Sturla Molden wrote:
>> I think NumPy and SciPy should consider to use OpenBLAS (a fork of
>> GotoBLAS2) instead of ATLAS or f2c'd Netlib BLAS for the binary releases.
>>
>> Here are its virtues:
>>
>> * Very easy to build: Just a makefile, no configuration script or
>> special build tools.
>>
>> * Building ATLAS can be a PITA. So why bother?
>>
>> * Faster than ATLAS, sometimes faster than MKL.
>>
>> * Multithreaded BLAS kernels: OpenMP on Unix, Windows threads on Windows.
>>
>> * The quality of its ancestor GotoBLAS is undisputed. I was the BLAS
>> implementation of choice for major HPC projects around the World.
>>
>> * Free as in BSD licensed.
>>
>> * Funded and developed for use in major Chinese HPC projects. Actively
>> maintained. (GotoBLAS2 is abandonware.)
>>
>> * Open source. The C sources are a pleasure to read, and very easy to
>> verify.
>>
>> * No OpenMP on Windows means no dependency on pthreads-win32 (an LGPL
>> library) when building with MinGW.
>>
>> * Builds on Windows with MinGW and MSYS, and perhaps even without MSYS.
>>
>> * Cygwin is not needed on Windows (this is just BS from the GotoBLAS
>> documentation). Thus, 64-buit builds are possible (I've built it using
>> TDM-GCC for Win64 and 32-bit MSYS).
>
> +1.
>
> Even on CPUs that are not directly supported, this is at least better
> than reference BLAS.
>
> (On our AMD CPUs, which are too new to have a separate OpenBLAS
> implementation, the implementations for older AMD CPUs still outperform
> at least Intel MKL, because MKL does so poorly on these -- although ACML
> beats them both by a factor 2. And of course on supported CPUs
> (everything Intel and older AMD) OpenBLAS is wonderful.

I support this as well in principle for our binary release: one issue
is that we don't have the infrastructure on mac to build an installer
with multi-arch support, and we can't assume every mac out there has
SSE 3 or 4 available.

We would need more testing first, as this is not a change to make lightly.

cheers,
David


From chaoyuejoy at gmail.com  Tue Nov 20 11:16:32 2012
From: chaoyuejoy at gmail.com (Chao YUE)
Date: Tue, 20 Nov 2012 17:16:32 +0100
Subject: [Numpy-discussion] the fast way to loop over ndarray elements?
In-Reply-To: <CAHGWWxH2FXLPnCkse09h-KMUcmGHK7yPX7MBOy+B4skG4DNFTg@mail.gmail.com>
References: <CAAN-aRHEyFe3QaCOXObs3Uu=F1N89KvuiMcwnm1gPnPiLtxfNA@mail.gmail.com>
	<CAHGWWxH2FXLPnCkse09h-KMUcmGHK7yPX7MBOy+B4skG4DNFTg@mail.gmail.com>
Message-ID: <CAAN-aRFYo7UZdma5aaEyao4T07jT4wPnQdUhm=YQLHX9G6y=Gg@mail.gmail.com>

Dear Thouis,

I take some time to check, before I tried with cython, I tried the
np.interp first, and very luckily, it's exeactly what I need.

And with the old written by me, it spend 20 seconds, now it's 0.2 seconds!

Thanks a lot to all you guys.

Chao

On Mon, Nov 19, 2012 at 3:08 PM, Thouis (Ray) Jones <thouis at gmail.com>wrote:

> On Sat, Nov 17, 2012 at 8:28 AM, Chao YUE <chaoyuejoy at gmail.com> wrote:
> > Dear all,
> >
> > I need to make a linear contrast of the 2D numpy array "data" from an
> > interval to another, the approach is:
> > I have another two list: "base" & "target", then I check for each ndarray
> > element "data[i,j]",
> > if   base[m] =< data[i,j] <= base[m+1], then it will be linearly
> converted
> > to be in the interval of (target[m], target[m+1]),
> > using another function called "lintrans".
> >
> >
> > #The way I do is to loop each row and column of the 2D array, and finally
> > loop the intervals constituted by base list:
> >
> > for row in range(data.shape[0]):
> >     for col in range(data.shape[1]):
> >         for i in range(len(base)-1):
> >             if data[row,col]>=base[i] and data[row,col]<=base[i+1]:
> >
> >
> data[row,col]=lintrans(data[row,col],(base[i],base[i+1]),(target[i],target[i+1]))
> >                 break  #use break to jump out of loop as the data have
> to be
> > ONLY transferred ONCE.
> >
> >
> > Now the profiling result shows that most of the time has been used in
> this
> > loop over the array ("plot_array_transg"),
> > and less time in calling the linear transformation fuction "lintrans":
> >
> >    ncalls     tottime  percall    cumtime    percall
> > filename:lineno(function)
> >   18047    0.110    0.000      0.110        0.000
> > mathex.py:132(lintrans)
> >   1            12.495  12.495   19.061      19.061
> > mathex.py:196(plot_array_transg)
> >
> >
> > so is there anyway I can speed up this loop?  Thanks for any
> suggestions!!
> >
> > best,
> >
> > Chao
>
> If lintrans() is a linear interpolation, could you use interp?
>
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.interp.html
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121120/e909e4b6/attachment.html>

From sturla at molden.no  Tue Nov 20 12:03:03 2012
From: sturla at molden.no (Sturla Molden)
Date: Tue, 20 Nov 2012 18:03:03 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
Message-ID: <50ABB7C7.2000509@molden.no>

On 20.11.2012 15:38, David Cournapeau wrote:

> I support this as well in principle for our binary release: one issue
> is that we don't have the infrastructure on mac to build an installer
> with multi-arch support, and we can't assume every mac out there has
> SSE 3 or 4 available.

Perhaps we could check the CPU architecture at run-time, and then load 
(or install) the correct extension module? OpenBLAS does have functions 
for testing if SSE 3 or 4 are available, which we could adapt:

https://github.com/xianyi/OpenBLAS/blob/master/cpuid.h
https://github.com/xianyi/OpenBLAS/blob/master/cpuid_x86.c

cf. the last function in cpuid_x86.c called "get_sse".

> We would need more testing first, as this is not a change to make lightly.

I'm not saying it will be easy :)


Sturla


From pav at iki.fi  Tue Nov 20 12:09:16 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 20 Nov 2012 19:09:16 +0200
Subject: [Numpy-discussion] Confused with qr decomposition function
In-Reply-To: <50AB84A0.6030609@it.uu.se>
References: <50AABE21.3000307@it.uu.se>
	<CAB6mnxJcx0_KU+n3oUQtixKax7+d8rbWap+QJq_dxat=uT+E-w@mail.gmail.com>
	<50AB84A0.6030609@it.uu.se>
Message-ID: <k8gdfq$do0$1@ger.gmane.org>

20.11.2012 15:24, Virgil Stokes kirjoitti:
[clip]
> I am aware that they are both correct; but, if you are doing covariance
> QR decomposition then you almost surely are interested in the positive
> results (which is the default for MATLAB and most papers/books on this
> subject).

I get exactly identical results from MATLAB (R2011b), Octave, Numpy, and
Scipy. Can you give an example matrix which behaves differently?

Note that Numpy and Scipy return exactly what LAPACK's *GEQRF routines
give, and Octave seems also to do this.

-- 
Pauli Virtanen



From cournape at gmail.com  Tue Nov 20 12:22:53 2012
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 20 Nov 2012 17:22:53 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50ABB7C7.2000509@molden.no>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
Message-ID: <CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>

On Tue, Nov 20, 2012 at 5:03 PM, Sturla Molden <sturla at molden.no> wrote:
> On 20.11.2012 15:38, David Cournapeau wrote:
>
>> I support this as well in principle for our binary release: one issue
>> is that we don't have the infrastructure on mac to build an installer
>> with multi-arch support, and we can't assume every mac out there has
>> SSE 3 or 4 available.
>
> Perhaps we could check the CPU architecture at run-time, and then load
> (or install) the correct extension module? OpenBLAS does have functions
> for testing if SSE 3 or 4 are available, which we could adapt:

Doing at runtime would be really hard. On windows, our installer does
it at install time, and openblas should be pretty much the same than
atlas there.

We would need a solution on mac (where I really want to get away from
Accelerate which has a very nasty limitation w.r.t GD vs fork, making
multiprocessing with our official numpy  binary essentially useless).

David


From vs at it.uu.se  Tue Nov 20 13:03:17 2012
From: vs at it.uu.se (Virgil Stokes)
Date: Tue, 20 Nov 2012 19:03:17 +0100
Subject: [Numpy-discussion] Confused with qr decomposition function
In-Reply-To: <k8gdfq$do0$1@ger.gmane.org>
References: <50AABE21.3000307@it.uu.se>
	<CAB6mnxJcx0_KU+n3oUQtixKax7+d8rbWap+QJq_dxat=uT+E-w@mail.gmail.com>
	<50AB84A0.6030609@it.uu.se> <k8gdfq$do0$1@ger.gmane.org>
Message-ID: <50ABC5E5.5030002@it.uu.se>

On 2012-11-20 18:09, Pauli Virtanen wrote:
> 20.11.2012 15:24, Virgil Stokes kirjoitti:
> [clip]
>> I am aware that they are both correct; but, if you are doing covariance
>> QR decomposition then you almost surely are interested in the positive
>> results (which is the default for MATLAB and most papers/books on this
>> subject).
> I get exactly identical results from MATLAB (R2011b), Octave, Numpy, and
> Scipy. Can you give an example matrix which behaves differently?
>
> Note that Numpy and Scipy return exactly what LAPACK's *GEQRF routines
> give, and Octave seems also to do this.
>
Here are two that had opposite signs compared to MATLAB:

array([[  7.07106781e+02,  -2.32273270e+04,  -2.46173719e+04],
        [ -3.53553391e+01,  -2.31566171e+04,  -2.46173719e+04],
        [  2.32273276e+04,  -3.97555166e+00,  -1.39003725e+03],
        [  2.25202208e+04,  -6.48214647e-04,  -1.39004432e+03],
        [  2.46527272e+04,   1.31933390e+03,   1.66675481e-19],
        [  2.46173719e+04,   1.39401993e+03,  -7.07106781e-03],
        [  0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
        [  0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
        [  0.00000000e+00,   0.00000000e+00,   0.00000000e+00]])


array([[  3.66711160e+04,   3.36224799e+04,   7.60569110e+02, 
-1.19158202e+03],
        [  3.24652853e+03,   0.00000000e+00,  -2.32192233e+04, 
-2.46276301e+04],
        [ -1.71055253e+04,   0.00000000e+00,   0.00000000e+00, 
-8.47443620e+01],
        [  1.15905933e+04,  -3.36224799e+04,  -7.60569110e+02, 
1.19158202e+03],
        [ -1.72015604e+04,   0.00000000e+00,   2.32192233e+04, 
2.46276301e+04],
        [ -1.72015604e+04,   0.00000000e+00,   0.00000000e+00, 
8.47443620e+01],
        [  3.00000000e+01,   0.00000000e+00,   0.00000000e+00, 
0.00000000e+00]])


From pav at iki.fi  Tue Nov 20 13:11:42 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 20 Nov 2012 20:11:42 +0200
Subject: [Numpy-discussion] Confused with qr decomposition function
In-Reply-To: <50ABC5E5.5030002@it.uu.se>
References: <50AABE21.3000307@it.uu.se>
	<CAB6mnxJcx0_KU+n3oUQtixKax7+d8rbWap+QJq_dxat=uT+E-w@mail.gmail.com>
	<50AB84A0.6030609@it.uu.se> <k8gdfq$do0$1@ger.gmane.org>
	<50ABC5E5.5030002@it.uu.se>
Message-ID: <k8gh4s$i51$1@ger.gmane.org>

20.11.2012 20:03, Virgil Stokes kirjoitti:
[clip]
> Here are two that had opposite signs compared to MATLAB:
[clip]

Interestingly, I get the same signs also for these matrices.

This probably means that the issue is specific to the Windows binaries.
Probably, the ATLAS+LAPACK combination shipped with it is somehow strange.

-- 
Pauli Virtanen



From ndbecker2 at gmail.com  Tue Nov 20 14:11:38 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 20 Nov 2012 14:11:38 -0500
Subject: [Numpy-discussion] confused about univaritespline
Message-ID: <k8gkl8$in6$1@ger.gmane.org>

I don't understand why the plot of the spline continues on a negative slope at 
the end, but the plot of the integral of it flattens.

-----------------------------------------------------

import numpy as np
import matplotlib.pyplot as plt

ibo = np.array ((12, 14, 16, 18, 20, 22, 24, 26, 28, 29, 29.8, 30.2))
gain_deriv = np.array ((0, 0,  0,  0,   0, 0,  .2, .4, .5, .5, 0,    -2))

import scipy.interpolate
s = scipy.interpolate.UnivariateSpline(ibo, gain_deriv, s=0.1)

xs = linspace(12, 31, 100)
gain = np.vectorize (lambda x: s.integral (12, x)) (xs)

plot (xs, s(xs))
plot (xs, gain)



From pav at iki.fi  Tue Nov 20 14:17:13 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 20 Nov 2012 21:17:13 +0200
Subject: [Numpy-discussion] confused about univaritespline
In-Reply-To: <k8gkl8$in6$1@ger.gmane.org>
References: <k8gkl8$in6$1@ger.gmane.org>
Message-ID: <k8gkvo$muk$1@ger.gmane.org>

20.11.2012 21:11, Neal Becker kirjoitti:
> import numpy as np
> import matplotlib.pyplot as plt
> 
> ibo = np.array ((12, 14, 16, 18, 20, 22, 24, 26, 28, 29, 29.8, 30.2))
> gain_deriv = np.array ((0, 0,  0,  0,   0, 0,  .2, .4, .5, .5, 0,    -2))
> 
> import scipy.interpolate
> s = scipy.interpolate.UnivariateSpline(ibo, gain_deriv, s=0.1)
> 
> xs = linspace(12, 31, 100)
> gain = np.vectorize (lambda x: s.integral (12, x)) (xs)
> 
> plot (xs, s(xs))
> plot (xs, gain)

>From fitpack/splint.f:

c           s(x) is considered to be identically zero outside
c           the interval (t(k+1),t(n-k)).

Not documented in the Python wrapper, though.

-- 
Pauli Virtanen



From ndbecker2 at gmail.com  Tue Nov 20 14:30:04 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 20 Nov 2012 14:30:04 -0500
Subject: [Numpy-discussion] confused about univaritespline
References: <k8gkl8$in6$1@ger.gmane.org> <k8gkvo$muk$1@ger.gmane.org>
Message-ID: <k8glnq$sel$1@ger.gmane.org>

Pauli Virtanen wrote:

> 20.11.2012 21:11, Neal Becker kirjoitti:
>> import numpy as np
>> import matplotlib.pyplot as plt
>> 
>> ibo = np.array ((12, 14, 16, 18, 20, 22, 24, 26, 28, 29, 29.8, 30.2))
>> gain_deriv = np.array ((0, 0,  0,  0,   0, 0,  .2, .4, .5, .5, 0,    -2))
>> 
>> import scipy.interpolate
>> s = scipy.interpolate.UnivariateSpline(ibo, gain_deriv, s=0.1)
>> 
>> xs = linspace(12, 31, 100)
>> gain = np.vectorize (lambda x: s.integral (12, x)) (xs)
>> 
>> plot (xs, s(xs))
>> plot (xs, gain)
> 
>>From fitpack/splint.f:
> 
> c           s(x) is considered to be identically zero outside
> c           the interval (t(k+1),t(n-k)).
> 
> Not documented in the Python wrapper, though.
> 

Ah, thanks!  Bitten one more by the fact that spline is not meant for 
extrapolation, which here I had done inadvertantly.

IMO, if spline isn't intended to be used to extrapolate, I'd prefer it to throw.



From d.s.seljebotn at astro.uio.no  Tue Nov 20 14:35:51 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 20 Nov 2012 20:35:51 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
Message-ID: <50ABDB97.4050500@astro.uio.no>

On 11/20/2012 06:22 PM, David Cournapeau wrote:
> On Tue, Nov 20, 2012 at 5:03 PM, Sturla Molden <sturla at molden.no> wrote:
>> On 20.11.2012 15:38, David Cournapeau wrote:
>>
>>> I support this as well in principle for our binary release: one issue
>>> is that we don't have the infrastructure on mac to build an installer
>>> with multi-arch support, and we can't assume every mac out there has
>>> SSE 3 or 4 available.
>>
>> Perhaps we could check the CPU architecture at run-time, and then load
>> (or install) the correct extension module? OpenBLAS does have functions
>> for testing if SSE 3 or 4 are available, which we could adapt:
>
> Doing at runtime would be really hard. On windows, our installer does
> it at install time, and openblas should be pretty much the same than
> atlas there.

Is there a specific reason it *has* to happen at compile-time? I'd think 
one could do something like just shipping a lot of separate Python 
extensions which are really just the same module linked with different 
versions of the library, and then

if cpu_is_nehalem:
     import blas_nehalem as blas
elif ...

I'm asking a question about whether this would work in principle, I 
realize it would perhaps not fit that well in the current NumPy codebase.

One thing is one would want to propagate the BLAS/LAPACK to other 
extensions through a function pointer table much like the current NumPy 
C API.

Dag Sverre


From d.s.seljebotn at astro.uio.no  Tue Nov 20 14:36:18 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 20 Nov 2012 20:36:18 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50ABDB97.4050500@astro.uio.no>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
Message-ID: <50ABDBB2.1050101@astro.uio.no>

On 11/20/2012 08:35 PM, Dag Sverre Seljebotn wrote:
> On 11/20/2012 06:22 PM, David Cournapeau wrote:
>> On Tue, Nov 20, 2012 at 5:03 PM, Sturla Molden <sturla at molden.no> wrote:
>>> On 20.11.2012 15:38, David Cournapeau wrote:
>>>
>>>> I support this as well in principle for our binary release: one issue
>>>> is that we don't have the infrastructure on mac to build an installer
>>>> with multi-arch support, and we can't assume every mac out there has
>>>> SSE 3 or 4 available.
>>>
>>> Perhaps we could check the CPU architecture at run-time, and then load
>>> (or install) the correct extension module? OpenBLAS does have functions
>>> for testing if SSE 3 or 4 are available, which we could adapt:
>>
>> Doing at runtime would be really hard. On windows, our installer does
>> it at install time, and openblas should be pretty much the same than
>> atlas there.
>
> Is there a specific reason it *has* to happen at compile-time? I'd think

Sorry, I meant install-time.

DS

> one could do something like just shipping a lot of separate Python
> extensions which are really just the same module linked with different
> versions of the library, and then
>
> if cpu_is_nehalem:
>      import blas_nehalem as blas
> elif ...
>
> I'm asking a question about whether this would work in principle, I
> realize it would perhaps not fit that well in the current NumPy codebase.
>
> One thing is one would want to propagate the BLAS/LAPACK to other
> extensions through a function pointer table much like the current NumPy
> C API.
>
> Dag Sverre



From gnurser at gmail.com  Tue Nov 20 14:52:46 2012
From: gnurser at gmail.com (George Nurser)
Date: Tue, 20 Nov 2012 19:52:46 +0000
Subject: [Numpy-discussion] Building numpy with OpenBLAS using bento
Message-ID: <CAAyCRnSsabRc24XA2_X0S49ahVAwd1KLysMyV-Xe8B5arU0JEw@mail.gmail.com>

Hi,
I've tried to build numpy with OpenBLAS following on from the blog
http://cournape.wordpress.com/2012/10/10/notes-on-building-numpyscipy-with-openblas/

I've cloned and installed an up to date version of bento from git.

Problem is that the bentomaker command doesn't seem to know about
the --with-blas-lapack-libdir option. The only place this option is defined
in the code seems to be in the blas_lapack.py routine .... but I can't see
how to use this routine -- it isn't imported into any other file.

(I've tried sending this smessage to the bento list, but it doesn't seem to
be appearing on it as far as I can see.)

Best regards, George Nurser
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121120/be488ce6/attachment.html>

From heng at cantab.net  Tue Nov 20 15:52:54 2012
From: heng at cantab.net (Henry Gomersall)
Date: Tue, 20 Nov 2012 20:52:54 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50ABDB97.4050500@astro.uio.no>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
Message-ID: <1353444774.3664.1.camel@farnsworth>

On Tue, 2012-11-20 at 20:35 +0100, Dag Sverre Seljebotn wrote:
> Is there a specific reason it *has* to happen at compile-time? I'd
> think 
> one could do something like just shipping a lot of separate Python 
> extensions which are really just the same module linked with
> different 
> versions of the library, and then
> 
> if cpu_is_nehalem:
>      import blas_nehalem as blas
> elif ...
> 
> I'm asking a question about whether this would work in principle, I 
> realize it would perhaps not fit that well in the current NumPy
> codebase. 

I was wondering this in the context of a previous discussion. Could we
not have an autotune module, that just runs a load of test scripts and
picks the best library to link against?

Cheers,

Henry



From charlesr.harris at gmail.com  Tue Nov 20 17:29:27 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 20 Nov 2012 15:29:27 -0700
Subject: [Numpy-discussion] confused about univaritespline
In-Reply-To: <k8glnq$sel$1@ger.gmane.org>
References: <k8gkl8$in6$1@ger.gmane.org> <k8gkvo$muk$1@ger.gmane.org>
	<k8glnq$sel$1@ger.gmane.org>
Message-ID: <CAB6mnx+wgKjFRAdMH0=+iK6TeehbtPtuj+Q-HJhwcMj3Kf+e3Q@mail.gmail.com>

On Tue, Nov 20, 2012 at 12:30 PM, Neal Becker <ndbecker2 at gmail.com> wrote:

> Pauli Virtanen wrote:
>
> > 20.11.2012 21:11, Neal Becker kirjoitti:
> >> import numpy as np
> >> import matplotlib.pyplot as plt
> >>
> >> ibo = np.array ((12, 14, 16, 18, 20, 22, 24, 26, 28, 29, 29.8, 30.2))
> >> gain_deriv = np.array ((0, 0,  0,  0,   0, 0,  .2, .4, .5, .5, 0,
>  -2))
> >>
> >> import scipy.interpolate
> >> s = scipy.interpolate.UnivariateSpline(ibo, gain_deriv, s=0.1)
> >>
> >> xs = linspace(12, 31, 100)
> >> gain = np.vectorize (lambda x: s.integral (12, x)) (xs)
> >>
> >> plot (xs, s(xs))
> >> plot (xs, gain)
> >
> >>From fitpack/splint.f:
> >
> > c           s(x) is considered to be identically zero outside
> > c           the interval (t(k+1),t(n-k)).
> >
> > Not documented in the Python wrapper, though.
> >
>
> Ah, thanks!  Bitten one more by the fact that spline is not meant for
> extrapolation, which here I had done inadvertantly.
>
> IMO, if spline isn't intended to be used to extrapolate, I'd prefer it to
> throw.
>

That can be tricky at the ends due to round off errors. I added a flag to
splev for this back when:

outine splev evaluates in a number of points x(i),i=1,2,...,m
c  a spline s(x) of degree k, given in its b-spline representation.
c
c  calling sequence:
c     call splev(t,n,c,k,x,y,m,e,ier)
c
c  input parameters:
c    t    : array,length n, which contains the position of the knots.
c    n    : integer, giving the total number of knots of s(x).
c    c    : array,length n, which contains the b-spline coefficients.
c    k    : integer, giving the degree of s(x).
c    x    : array,length m, which contains the points where s(x) must
c           be evaluated.
c    m    : integer, giving the number of points where s(x) must be
c           evaluated.
c    e    : integer, if 0 the spline is extrapolated from the end
c           spans for points not in the support, if 1 the spline
c           evaluates to zero for those points, and if 2 ier is set to
c           1 and the subroutine returns.


Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121120/10d88df5/attachment.html>

From cournape at gmail.com  Tue Nov 20 19:38:48 2012
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 21 Nov 2012 00:38:48 +0000
Subject: [Numpy-discussion] Building numpy with OpenBLAS using bento
In-Reply-To: <CAAyCRnSsabRc24XA2_X0S49ahVAwd1KLysMyV-Xe8B5arU0JEw@mail.gmail.com>
References: <CAAyCRnSsabRc24XA2_X0S49ahVAwd1KLysMyV-Xe8B5arU0JEw@mail.gmail.com>
Message-ID: <CAGY4rcU+Rj3uB0o5H9zL+pzmg3b6VzvXdkgoW6veCY1khhSxXg@mail.gmail.com>

On Tue, Nov 20, 2012 at 7:52 PM, George Nurser <gnurser at gmail.com> wrote:
> Hi,
> I've tried to build numpy with OpenBLAS following on from the blog
> http://cournape.wordpress.com/2012/10/10/notes-on-building-numpyscipy-with-openblas/
>
> I've cloned and installed an up to date version of bento from git.
>
> Problem is that the bentomaker command doesn't seem to know about the
> --with-blas-lapack-libdir option. The only place this option is defined in
> the code seems to be in the blas_lapack.py routine .... but I can't see how
> to use this routine -- it isn't imported into any other file.

Can you describe the exact steps you followed ? For example, just
cloning bento won't get you far enough, you need to have waf installed
as well, and you get an explicit error message otherwise.

>
> (I've tried sending this smessage to the bento list, but it doesn't seem to
> be appearing on it as far as I can see.)

Yeah, it looks like librelist is fundamentally broken, many messages
never appear. I need to move to something sane.

cheers,
David


From cournape at gmail.com  Tue Nov 20 19:44:37 2012
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 21 Nov 2012 00:44:37 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <1353444774.3664.1.camel@farnsworth>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
Message-ID: <CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>

On Tue, Nov 20, 2012 at 8:52 PM, Henry Gomersall <heng at cantab.net> wrote:
> On Tue, 2012-11-20 at 20:35 +0100, Dag Sverre Seljebotn wrote:
>> Is there a specific reason it *has* to happen at compile-time? I'd
>> think
>> one could do something like just shipping a lot of separate Python
>> extensions which are really just the same module linked with
>> different
>> versions of the library, and then
>>
>> if cpu_is_nehalem:
>>      import blas_nehalem as blas
>> elif ...
>>
>> I'm asking a question about whether this would work in principle, I
>> realize it would perhaps not fit that well in the current NumPy
>> codebase.
>
> I was wondering this in the context of a previous discussion. Could we
> not have an autotune module, that just runs a load of test scripts and
> picks the best library to link against?

You can't easily relink at install time, so what you want is pick up
the best library at runtime. It is more or less impossible to do this
in a portable way (e.g. there is no solution that I know of on windows
< windows 2008, short of requiring to install some dlls with admin
privileges).

cheers,
David


From cournape at gmail.com  Tue Nov 20 19:43:02 2012
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 21 Nov 2012 00:43:02 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50ABDB97.4050500@astro.uio.no>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
Message-ID: <CAGY4rcWhm5ugQW4V_hSm6Ui0DpXYuXTDWFHtBwOdE7CdZ2Tn2w@mail.gmail.com>

On Tue, Nov 20, 2012 at 7:35 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 11/20/2012 06:22 PM, David Cournapeau wrote:
>> On Tue, Nov 20, 2012 at 5:03 PM, Sturla Molden <sturla at molden.no> wrote:
>>> On 20.11.2012 15:38, David Cournapeau wrote:
>>>
>>>> I support this as well in principle for our binary release: one issue
>>>> is that we don't have the infrastructure on mac to build an installer
>>>> with multi-arch support, and we can't assume every mac out there has
>>>> SSE 3 or 4 available.
>>>
>>> Perhaps we could check the CPU architecture at run-time, and then load
>>> (or install) the correct extension module? OpenBLAS does have functions
>>> for testing if SSE 3 or 4 are available, which we could adapt:
>>
>> Doing at runtime would be really hard. On windows, our installer does
>> it at install time, and openblas should be pretty much the same than
>> atlas there.
>
> Is there a specific reason it *has* to happen at compile-time? I'd think
> one could do something like just shipping a lot of separate Python
> extensions which are really just the same module linked with different
> versions of the library, and then
>
> if cpu_is_nehalem:
>      import blas_nehalem as blas
> elif ...

For this to work, we would need to to compile not just the library
itself for each arch, but also every extension that link to it (e.g.
for blas, every extension linking to blas would need to be linked
against each arch library). That would be a nightmare to manage.

>
> One thing is one would want to propagate the BLAS/LAPACK to other
> extensions through a function pointer table much like the current NumPy
> C API.

That would be the better solution, but doing this in such a way that
would work on every platform is a significant effort.

David


From tjligocki at lbl.gov  Wed Nov 21 04:12:35 2012
From: tjligocki at lbl.gov (Terry J. Ligocki)
Date: Wed, 21 Nov 2012 01:12:35 -0800
Subject: [Numpy-discussion] Crash using "reshape"...
Message-ID: <50AC9B03.50504@lbl.gov>

I am having a problem with "reshape" crashing:

     > python
    Python 2.6.4 (r264:75706, Jan 16 2010, 21:11:47)
    [GCC 4.3.2] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
     >>> import numpy
     >>> numpy.version.version
    '1.6.2'
     >>> npData = numpy.ones([701,701,7899],dtype=numpy.dtype('b'))
     >>> npDataSubset = npData[[slice(0,700),slice(0,700),slice(0,5000)]]
     >>> npDataOutput = npDataSubset.reshape([700*700*5000],order='F')
    Segmentation fault

If I change the "5000" to a "4000", everything is fine.  I'm not running 
out of memory - my system had 48 GB of memory and nothing else is using 
a significant portion of this memory.

Note:  700x700x4000 = 1,960,000,000 < 2^31 and 700x700x5000 = 2450000000 
 > 2^31.  I suspect somewhere in the underlying code there is a signed 
32-bit integer being used for an index/pointer offset (this is running 
on a 64-bit machine).

I did some searching of the archives and didn't find a match for this 
problem.  Thank you for any and all help!

         Terry J. (Ligocki, tjligocki at lbl.gov)
------------------------------------------------------------------------
*Wishes
*
/When you wish upon a falling star, your dreams can come true.
Unless it's really a meteorite hurtling to the Earth which will destroy 
all life.
Then you're pretty much hosed no matter what you wish for.
Unless it's death by meteor.
/
(Despair, Inc.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121121/46730129/attachment.html>

From francesc at continuum.io  Wed Nov 21 05:14:58 2012
From: francesc at continuum.io (Francesc Alted)
Date: Wed, 21 Nov 2012 11:14:58 +0100
Subject: [Numpy-discussion] Crash using "reshape"...
In-Reply-To: <50AC9B03.50504@lbl.gov>
References: <50AC9B03.50504@lbl.gov>
Message-ID: <50ACA9A2.3070707@continuum.io>

On 11/21/12 10:12 AM, Terry J. Ligocki wrote:
> I am having a problem with "reshape" crashing:
>
>     > python
>     Python 2.6.4 (r264:75706, Jan 16 2010, 21:11:47)
>     [GCC 4.3.2] on linux2
>     Type "help", "copyright", "credits" or "license" for more information.
>     >>> import numpy
>     >>> numpy.version.version
>     '1.6.2'
>     >>> npData = numpy.ones([701,701,7899],dtype=numpy.dtype('b'))
>     >>> npDataSubset = npData[[slice(0,700),slice(0,700),slice(0,5000)]]
>     >>> npDataOutput = npDataSubset.reshape([700*700*5000],order='F')
>     Segmentation fault
>
> If I change the "5000" to a "4000", everything is fine.  I'm not 
> running out of memory - my system had 48 GB of memory and nothing else 
> is using a significant portion of this memory.
>
> Note:  700x700x4000 = 1,960,000,000 < 2^31 and 700x700x5000 = 
> 2450000000 > 2^31.  I suspect somewhere in the underlying code there 
> is a signed 32-bit integer being used for an index/pointer offset 
> (this is running on a 64-bit machine).

Yes, looks like a 32-bit issue.  Sometimes you can have 32-bit software 
installed in 64-bit machines, so that might be your problem.  What's the 
equivalent of numpy.intp in your machine?  Mine is:

In []: import numpy as np

In []: np.intp
Out[]: numpy.int64

If you see 'numpy.int32' here then that is the problem.

-- 
Francesc Alted



From heng at cantab.net  Wed Nov 21 05:16:29 2012
From: heng at cantab.net (Henry Gomersall)
Date: Wed, 21 Nov 2012 10:16:29 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no> <1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
Message-ID: <1353492989.3664.2.camel@farnsworth>

On Wed, 2012-11-21 at 00:44 +0000, David Cournapeau wrote:
> On Tue, Nov 20, 2012 at 8:52 PM, Henry Gomersall <heng at cantab.net>
> wrote:
> > On Tue, 2012-11-20 at 20:35 +0100, Dag Sverre Seljebotn wrote:
> >> Is there a specific reason it *has* to happen at compile-time? I'd
> >> think
> >> one could do something like just shipping a lot of separate Python
> >> extensions which are really just the same module linked with
> >> different
> >> versions of the library, and then
> >>
> >> if cpu_is_nehalem:
> >>      import blas_nehalem as blas
> >> elif ...
> >>
> >> I'm asking a question about whether this would work in principle, I
> >> realize it would perhaps not fit that well in the current NumPy
> >> codebase.
> >
> > I was wondering this in the context of a previous discussion. Could
> we
> > not have an autotune module, that just runs a load of test scripts
> and
> > picks the best library to link against?
> 
> You can't easily relink at install time, so what you want is pick up
> the best library at runtime. It is more or less impossible to do this
> in a portable way (e.g. there is no solution that I know of on windows
> < windows 2008, short of requiring to install some dlls with admin
> privileges). 

I meant it as Dag did, at install time ;)



From tjligocki at lbl.gov  Wed Nov 21 05:25:44 2012
From: tjligocki at lbl.gov (Terry J. Ligocki)
Date: Wed, 21 Nov 2012 02:25:44 -0800
Subject: [Numpy-discussion] Crash using "reshape"...
In-Reply-To: <50ACA9A2.3070707@continuum.io>
References: <50AC9B03.50504@lbl.gov> <50ACA9A2.3070707@continuum.io>
Message-ID: <50ACAC28.30401@lbl.gov>

I just checked, "numpy.intp" is "<type 'numpy.int64'>" in my 
installation of Python and NumPy.  It was a good thing to check but it 
looks like there's still may be a signed 32-bit integer somewhere in the 
code (or my build(s))...

                                 Terry J.
> On 11/21/12 10:12 AM, Terry J. Ligocki wrote:
>> I am having a problem with "reshape" crashing:
>>
>>      > python
>>      Python 2.6.4 (r264:75706, Jan 16 2010, 21:11:47)
>>      [GCC 4.3.2] on linux2
>>      Type "help", "copyright", "credits" or "license" for more information.
>>      >>> import numpy
>>      >>> numpy.version.version
>>      '1.6.2'
>>      >>> npData = numpy.ones([701,701,7899],dtype=numpy.dtype('b'))
>>      >>> npDataSubset = npData[[slice(0,700),slice(0,700),slice(0,5000)]]
>>      >>> npDataOutput = npDataSubset.reshape([700*700*5000],order='F')
>>      Segmentation fault
>>
>> If I change the "5000" to a "4000", everything is fine.  I'm not
>> running out of memory - my system had 48 GB of memory and nothing else
>> is using a significant portion of this memory.
>>
>> Note:  700x700x4000 = 1,960,000,000 < 2^31 and 700x700x5000 =
>> 2450000000 > 2^31.  I suspect somewhere in the underlying code there
>> is a signed 32-bit integer being used for an index/pointer offset
>> (this is running on a 64-bit machine).
> Yes, looks like a 32-bit issue.  Sometimes you can have 32-bit software
> installed in 64-bit machines, so that might be your problem.  What's the
> equivalent of numpy.intp in your machine?  Mine is:
>
> In []: import numpy as np
>
> In []: np.intp
> Out[]: numpy.int64
>
> If you see 'numpy.int32' here then that is the problem.
------------------------------------------------------------------------
/She's got to be this town's best mess
But it ain't nothin' that her face would suggest
/
- God Don't Make Lonely Girls (The Wallflowers)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121121/31c0f442/attachment.html>

From francesc at continuum.io  Wed Nov 21 05:41:06 2012
From: francesc at continuum.io (Francesc Alted)
Date: Wed, 21 Nov 2012 11:41:06 +0100
Subject: [Numpy-discussion] Crash using "reshape"...
In-Reply-To: <50ACAC28.30401@lbl.gov>
References: <50AC9B03.50504@lbl.gov> <50ACA9A2.3070707@continuum.io>
	<50ACAC28.30401@lbl.gov>
Message-ID: <50ACAFC2.3020701@continuum.io>

On 11/21/12 11:25 AM, Terry J. Ligocki wrote:
> I just checked, "numpy.intp" is "<type 'numpy.int64'>" in my 
> installation of Python and NumPy.  It was a good thing to check but it 
> looks like there's still may be a signed 32-bit integer somewhere in 
> the code (or my build(s))...

Okay.  I can reproduce that too (using 1.6.1).  Could you please file a 
ticket for this?  Smells like a bug to me.

-- 
Francesc Alted



From cournape at gmail.com  Wed Nov 21 05:49:35 2012
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 21 Nov 2012 10:49:35 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <1353492989.3664.2.camel@farnsworth>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
Message-ID: <CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>

That's already what we do (on.windows anyway). The binary installer
contains multiple arch binaries, and we pick the bewt one.
Le 21 nov. 2012 10:16, "Henry Gomersall" <heng at cantab.net> a ?crit :

> On Wed, 2012-11-21 at 00:44 +0000, David Cournapeau wrote:
> > On Tue, Nov 20, 2012 at 8:52 PM, Henry Gomersall <heng at cantab.net>
> > wrote:
> > > On Tue, 2012-11-20 at 20:35 +0100, Dag Sverre Seljebotn wrote:
> > >> Is there a specific reason it *has* to happen at compile-time? I'd
> > >> think
> > >> one could do something like just shipping a lot of separate Python
> > >> extensions which are really just the same module linked with
> > >> different
> > >> versions of the library, and then
> > >>
> > >> if cpu_is_nehalem:
> > >>      import blas_nehalem as blas
> > >> elif ...
> > >>
> > >> I'm asking a question about whether this would work in principle, I
> > >> realize it would perhaps not fit that well in the current NumPy
> > >> codebase.
> > >
> > > I was wondering this in the context of a previous discussion. Could
> > we
> > > not have an autotune module, that just runs a load of test scripts
> > and
> > > picks the best library to link against?
> >
> > You can't easily relink at install time, so what you want is pick up
> > the best library at runtime. It is more or less impossible to do this
> > in a portable way (e.g. there is no solution that I know of on windows
> > < windows 2008, short of requiring to install some dlls with admin
> > privileges).
>
> I meant it as Dag did, at install time ;)
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121121/0d035756/attachment.html>

From heng at cantab.net  Wed Nov 21 05:56:54 2012
From: heng at cantab.net (Henry Gomersall)
Date: Wed, 21 Nov 2012 10:56:54 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no> <1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
Message-ID: <1353495414.3664.3.camel@farnsworth>

On Wed, 2012-11-21 at 10:49 +0000, David Cournapeau wrote:
> That's already what we do (on.windows anyway). The binary installer
> contains multiple arch binaries, and we pick the bewt one.

Interesting. Does it (or can it) extend to different algorithmic
implementations?

Henry



From cournape at gmail.com  Wed Nov 21 06:44:58 2012
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 21 Nov 2012 11:44:58 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <1353495414.3664.3.camel@farnsworth>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
	<1353495414.3664.3.camel@farnsworth>
Message-ID: <CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>

On Wed, Nov 21, 2012 at 10:56 AM, Henry Gomersall <heng at cantab.net> wrote:
> On Wed, 2012-11-21 at 10:49 +0000, David Cournapeau wrote:
>> That's already what we do (on.windows anyway). The binary installer
>> contains multiple arch binaries, and we pick the bewt one.
>
> Interesting. Does it (or can it) extend to different algorithmic
> implementations?

It is rather simple, in that it expects a different, full installer
for each new combination. I would rather not complicate things too
much there, as it is working well for its purpose and is not easy to
modify. We would need something similar on Mac to make openblas a good
replacement on that platform.

David


From sturla at molden.no  Wed Nov 21 07:00:41 2012
From: sturla at molden.no (Sturla Molden)
Date: Wed, 21 Nov 2012 13:00:41 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
	<1353495414.3664.3.camel@farnsworth>
	<CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
Message-ID: <3B2D7559-F2DE-473E-8C89-91A19F6AD33D@molden.no>

But do we need a binary OpenBLAS on Mac? Isn't there an accelerate framework with BLAS and LAPACK on that platform?

Sturla

Sendt fra min iPad

Den 21. nov. 2012 kl. 12:44 skrev David Cournapeau <cournape at gmail.com>:

> On Wed, Nov 21, 2012 at 10:56 AM, Henry Gomersall <heng at cantab.net> wrote:
>> On Wed, 2012-11-21 at 10:49 +0000, David Cournapeau wrote:
>>> That's already what we do (on.windows anyway). The binary installer
>>> contains multiple arch binaries, and we pick the bewt one.
>> 
>> Interesting. Does it (or can it) extend to different algorithmic
>> implementations?
> 
> It is rather simple, in that it expects a different, full installer
> for each new combination. I would rather not complicate things too
> much there, as it is working well for its purpose and is not easy to
> modify. We would need something similar on Mac to make openblas a good
> replacement on that platform.
> 
> David
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From cournape at gmail.com  Wed Nov 21 09:01:15 2012
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 21 Nov 2012 14:01:15 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <3B2D7559-F2DE-473E-8C89-91A19F6AD33D@molden.no>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
	<1353495414.3664.3.camel@farnsworth>
	<CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
	<3B2D7559-F2DE-473E-8C89-91A19F6AD33D@molden.no>
Message-ID: <CAGY4rcWDG8Yf6bgqNTWKvu6J_Myuf_Rc+3exNX_y3dn_wYHQuA@mail.gmail.com>

On Wed, Nov 21, 2012 at 12:00 PM, Sturla Molden <sturla at molden.no> wrote:
> But do we need a binary OpenBLAS on Mac? Isn't there an accelerate framework with BLAS and LAPACK on that platform?

Because of this: https://gist.github.com/2027412

Numpy + accelerate is essentially unusable with multiprocessing.

David


From sturla at molden.no  Wed Nov 21 09:31:35 2012
From: sturla at molden.no (Sturla Molden)
Date: Wed, 21 Nov 2012 15:31:35 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <CAGY4rcWDG8Yf6bgqNTWKvu6J_Myuf_Rc+3exNX_y3dn_wYHQuA@mail.gmail.com>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
	<1353495414.3664.3.camel@farnsworth>
	<CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
	<3B2D7559-F2DE-473E-8C89-91A19F6AD33D@molden.no>
	<CAGY4rcWDG8Yf6bgqNTWKvu6J_Myuf_Rc+3exNX_y3dn_wYHQuA@mail.gmail.com>
Message-ID: <50ACE5C7.2030703@molden.no>

On 21.11.2012 15:01, David Cournapeau wrote:
> On Wed, Nov 21, 2012 at 12:00 PM, Sturla Molden<sturla at molden.no>  wrote:
>> But do we need a binary OpenBLAS on Mac? Isn't there an accelerate framework with BLAS and LAPACK on that platform?
>
> Because of this: https://gist.github.com/2027412
>
> Numpy + accelerate is essentially unusable with multiprocessing.

See this too:

http://stackoverflow.com/questions/8748790/memory-leak-in-dgemm

If they put a memory leak in DGEMM, basically most of LAPACK is useless 
as well. That is very sloppy of Apple.

And how could they fail to detect that DGEMM crashes on fork()? Amazing...

It doesn't seem the Grand Central Dispatch is very robust then...

Sturla


From cournape at gmail.com  Wed Nov 21 09:37:25 2012
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 21 Nov 2012 14:37:25 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50ACE5C7.2030703@molden.no>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
	<1353495414.3664.3.camel@farnsworth>
	<CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
	<3B2D7559-F2DE-473E-8C89-91A19F6AD33D@molden.no>
	<CAGY4rcWDG8Yf6bgqNTWKvu6J_Myuf_Rc+3exNX_y3dn_wYHQuA@mail.gmail.com>
	<50ACE5C7.2030703@molden.no>
Message-ID: <CAGY4rcUA0b=rmimgK9v3GHiNTxjHV_E0W7TPeQUBqQKxGrshxw@mail.gmail.com>

On Wed, Nov 21, 2012 at 2:31 PM, Sturla Molden <sturla at molden.no> wrote:
> On 21.11.2012 15:01, David Cournapeau wrote:
>> On Wed, Nov 21, 2012 at 12:00 PM, Sturla Molden<sturla at molden.no>  wrote:
>>> But do we need a binary OpenBLAS on Mac? Isn't there an accelerate framework with BLAS and LAPACK on that platform?
>>
>> Because of this: https://gist.github.com/2027412
>>
>> Numpy + accelerate is essentially unusable with multiprocessing.
>
> See this too:
>
> http://stackoverflow.com/questions/8748790/memory-leak-in-dgemm
>
> If they put a memory leak in DGEMM, basically most of LAPACK is useless
> as well. That is very sloppy of Apple.
>
> And how could they fail to detect that DGEMM crashes on fork()? Amazing...
>
> It doesn't seem the Grand Central Dispatch is very robust then...

I don't think it has anything to do with robustness as much as
multiprocessing doing things not supported by GCD. The answer we got
from Apple when we looked into this with Olivier Grisel at pycon was:

"""
For API outside of POSIX, including GCD and technologies like
Accelerate, we do not support usage on both sides of a fork(). For
this reason among others, use of fork() without exec is discouraged in
general in processes that use layers above POSIX.

We recommend that you either restrict usage of blas to the parent or
the child process but not both, or that you switch to using GCD or
pthreads rather than forking to create parallelism.

Please also see the following in the fork(2) manpage:

CAVEATS
     There are limits to what you can do in the child process.  To be
totally safe you should restrict your-
     self to only executing async-signal safe operations until such
time as one of the exec functions is
     called.  All APIs, including global data symbols, in any
framework or library should be assumed to be
     unsafe after a fork() unless explicitly documented to be safe or
async-signal safe.  If you need to use
     these frameworks in the child process, you must exec.  In this
situation it is reasonable to exec your-
     self.
"""

cheers,
David


From sebastian at sipsolutions.net  Wed Nov 21 09:42:17 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Wed, 21 Nov 2012 15:42:17 +0100
Subject: [Numpy-discussion] Crash using "reshape"...
In-Reply-To: <50AC9B03.50504@lbl.gov>
References: <50AC9B03.50504@lbl.gov>
Message-ID: <1353508937.2529.6.camel@sebastian-laptop>

Hey,

On Wed, 2012-11-21 at 01:12 -0800, Terry J. Ligocki wrote:
> I am having a problem with "reshape" crashing:
>         > python
>         Python 2.6.4 (r264:75706, Jan 16 2010, 21:11:47) 
>         [GCC 4.3.2] on linux2
>         Type "help", "copyright", "credits" or "license" for more
>         information.
>         >>> import numpy
>         >>> numpy.version.version
>         '1.6.2'
>         >>> npData = numpy.ones([701,701,7899],dtype=numpy.dtype('b'))
>         >>> npDataSubset =
>         npData[[slice(0,700),slice(0,700),slice(0,5000)]]
>         >>> npDataOutput =
>         npDataSubset.reshape([700*700*5000],order='F')
>         Segmentation fault
> If I change the "5000" to a "4000", everything is fine.  I'm not
> running out of memory - my system had 48 GB of memory and nothing else
> is using a significant portion of this memory.
> 
> Note:  700x700x4000 = 1,960,000,000 < 2^31 and 700x700x5000 =
> 2450000000 > 2^31.  I suspect somewhere in the underlying code there
> is a signed 32-bit integer being used for an index/pointer offset
> (this is running on a 64-bit machine).

yes, int is used for npy_intp in one occasione. I have created a PR that
fixes the issue:
https://github.com/numpy/numpy/pull/2754

Regards,

Sebastian

> I did some searching of the archives and didn't find a match for this
> problem.  Thank you for any and all help!
> 
>         Terry J. (Ligocki, tjligocki at lbl.gov)
> 
> ______________________________________________________________________
> Wishes
> 
> When you wish upon a falling star, your dreams can come true.
> Unless it's really a meteorite hurtling to the Earth which will
> destroy all life.
> Then you're pretty much hosed no matter what you wish for.
> Unless it's death by meteor.
> 
> (Despair, Inc.)
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From sturla at molden.no  Wed Nov 21 09:45:09 2012
From: sturla at molden.no (Sturla Molden)
Date: Wed, 21 Nov 2012 15:45:09 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <CAGY4rcUA0b=rmimgK9v3GHiNTxjHV_E0W7TPeQUBqQKxGrshxw@mail.gmail.com>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
	<1353495414.3664.3.camel@farnsworth>
	<CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
	<3B2D7559-F2DE-473E-8C89-91A19F6AD33D@molden.no>
	<CAGY4rcWDG8Yf6bgqNTWKvu6J_Myuf_Rc+3exNX_y3dn_wYHQuA@mail.gmail.com>
	<50ACE5C7.2030703@molden.no>
	<CAGY4rcUA0b=rmimgK9v3GHiNTxjHV_E0W7TPeQUBqQKxGrshxw@mail.gmail.com>
Message-ID: <50ACE8F5.2040300@molden.no>

Ok, so using BLAS on each side of a fork requires an exec. That means 
multiprocessing on Mac must behave as it does on Windows, with the 
restrictions there. I.e. no fork, no closures as target kwarg to 
Process, etc. But that is also why multiprocessing feels crippled on 
Windows, and why the GIL sucks 10x more on Windows than Mac or Linux.

Sturla


On 21.11.2012 15:37, David Cournapeau wrote:
> On Wed, Nov 21, 2012 at 2:31 PM, Sturla Molden<sturla at molden.no>  wrote:
>> On 21.11.2012 15:01, David Cournapeau wrote:
>>> On Wed, Nov 21, 2012 at 12:00 PM, Sturla Molden<sturla at molden.no>   wrote:
>>>> But do we need a binary OpenBLAS on Mac? Isn't there an accelerate framework with BLAS and LAPACK on that platform?
>>>
>>> Because of this: https://gist.github.com/2027412
>>>
>>> Numpy + accelerate is essentially unusable with multiprocessing.
>>
>> See this too:
>>
>> http://stackoverflow.com/questions/8748790/memory-leak-in-dgemm
>>
>> If they put a memory leak in DGEMM, basically most of LAPACK is useless
>> as well. That is very sloppy of Apple.
>>
>> And how could they fail to detect that DGEMM crashes on fork()? Amazing...
>>
>> It doesn't seem the Grand Central Dispatch is very robust then...
>
> I don't think it has anything to do with robustness as much as
> multiprocessing doing things not supported by GCD. The answer we got
> from Apple when we looked into this with Olivier Grisel at pycon was:
>
> """
> For API outside of POSIX, including GCD and technologies like
> Accelerate, we do not support usage on both sides of a fork(). For
> this reason among others, use of fork() without exec is discouraged in
> general in processes that use layers above POSIX.
>
> We recommend that you either restrict usage of blas to the parent or
> the child process but not both, or that you switch to using GCD or
> pthreads rather than forking to create parallelism.
>
> Please also see the following in the fork(2) manpage:
>
> CAVEATS
>       There are limits to what you can do in the child process.  To be
> totally safe you should restrict your-
>       self to only executing async-signal safe operations until such
> time as one of the exec functions is
>       called.  All APIs, including global data symbols, in any
> framework or library should be assumed to be
>       unsafe after a fork() unless explicitly documented to be safe or
> async-signal safe.  If you need to use
>       these frameworks in the child process, you must exec.  In this
> situation it is reasonable to exec your-
>       self.
> """
>
> cheers,
> David
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From sturla at molden.no  Wed Nov 21 09:52:29 2012
From: sturla at molden.no (Sturla Molden)
Date: Wed, 21 Nov 2012 15:52:29 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50ACE8F5.2040300@molden.no>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
	<1353495414.3664.3.camel@farnsworth>
	<CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
	<3B2D7559-F2DE-473E-8C89-91A19F6AD33D@molden.no>
	<CAGY4rcWDG8Yf6bgqNTWKvu6J_Myuf_Rc+3exNX_y3dn_wYHQuA@mail.gmail.com>
	<50ACE5C7.2030703@molden.no>
	<CAGY4rcUA0b=rmimgK9v3GHiNTxjHV_E0W7TPeQUBqQKxGrshxw@mail.gmail.com>
	<50ACE8F5.2040300@molden.no>
Message-ID: <50ACEAAD.4090409@molden.no>

But if this issue is in the GCD, it will probably affect other 
applications that uses the GCD and fork without exec as well. Unless you 
are certain the GCD is not used, a fork would never be safe without an exec.

Sturla

On 21.11.2012 15:45, Sturla Molden wrote:
> Ok, so using BLAS on each side of a fork requires an exec. That means
> multiprocessing on Mac must behave as it does on Windows, with the
> restrictions there. I.e. no fork, no closures as target kwarg to
> Process, etc. But that is also why multiprocessing feels crippled on
> Windows, and why the GIL sucks 10x more on Windows than Mac or Linux.
>
> Sturla
>
>
> On 21.11.2012 15:37, David Cournapeau wrote:
>> On Wed, Nov 21, 2012 at 2:31 PM, Sturla Molden<sturla at molden.no>   wrote:
>>> On 21.11.2012 15:01, David Cournapeau wrote:
>>>> On Wed, Nov 21, 2012 at 12:00 PM, Sturla Molden<sturla at molden.no>    wrote:
>>>>> But do we need a binary OpenBLAS on Mac? Isn't there an accelerate framework with BLAS and LAPACK on that platform?
>>>>
>>>> Because of this: https://gist.github.com/2027412
>>>>
>>>> Numpy + accelerate is essentially unusable with multiprocessing.
>>>
>>> See this too:
>>>
>>> http://stackoverflow.com/questions/8748790/memory-leak-in-dgemm
>>>
>>> If they put a memory leak in DGEMM, basically most of LAPACK is useless
>>> as well. That is very sloppy of Apple.
>>>
>>> And how could they fail to detect that DGEMM crashes on fork()? Amazing...
>>>
>>> It doesn't seem the Grand Central Dispatch is very robust then...
>>
>> I don't think it has anything to do with robustness as much as
>> multiprocessing doing things not supported by GCD. The answer we got
>> from Apple when we looked into this with Olivier Grisel at pycon was:
>>
>> """
>> For API outside of POSIX, including GCD and technologies like
>> Accelerate, we do not support usage on both sides of a fork(). For
>> this reason among others, use of fork() without exec is discouraged in
>> general in processes that use layers above POSIX.
>>
>> We recommend that you either restrict usage of blas to the parent or
>> the child process but not both, or that you switch to using GCD or
>> pthreads rather than forking to create parallelism.
>>
>> Please also see the following in the fork(2) manpage:
>>
>> CAVEATS
>>        There are limits to what you can do in the child process.  To be
>> totally safe you should restrict your-
>>        self to only executing async-signal safe operations until such
>> time as one of the exec functions is
>>        called.  All APIs, including global data symbols, in any
>> framework or library should be assumed to be
>>        unsafe after a fork() unless explicitly documented to be safe or
>> async-signal safe.  If you need to use
>>        these frameworks in the child process, you must exec.  In this
>> situation it is reasonable to exec your-
>>        self.
>> """
>>
>> cheers,
>> David
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From njs at pobox.com  Wed Nov 21 09:55:18 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 21 Nov 2012 14:55:18 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50ACEAAD.4090409@molden.no>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
	<1353495414.3664.3.camel@farnsworth>
	<CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
	<3B2D7559-F2DE-473E-8C89-91A19F6AD33D@molden.no>
	<CAGY4rcWDG8Yf6bgqNTWKvu6J_Myuf_Rc+3exNX_y3dn_wYHQuA@mail.gmail.com>
	<50ACE5C7.2030703@molden.no>
	<CAGY4rcUA0b=rmimgK9v3GHiNTxjHV_E0W7TPeQUBqQKxGrshxw@mail.gmail.com>
	<50ACE8F5.2040300@molden.no> <50ACEAAD.4090409@molden.no>
Message-ID: <CAPJVwB=jrfi+DjswBN2LRuND_ZbDC4xEqiGRtPOZP3CiD6mOow@mail.gmail.com>

On Wed, Nov 21, 2012 at 2:52 PM, Sturla Molden <sturla at molden.no> wrote:
> But if this issue is in the GCD, it will probably affect other
> applications that uses the GCD and fork without exec as well. Unless you
> are certain the GCD is not used, a fork would never be safe without an exec.

I think the point is that it's easy for programmers to decide to avoid
GCD if they want to use multiprocessing. But it's not so easy for them
to decide to avoid BLAS.

-n


From sturla at molden.no  Wed Nov 21 10:01:48 2012
From: sturla at molden.no (Sturla Molden)
Date: Wed, 21 Nov 2012 16:01:48 +0100
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <CAPJVwB=jrfi+DjswBN2LRuND_ZbDC4xEqiGRtPOZP3CiD6mOow@mail.gmail.com>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
	<1353495414.3664.3.camel@farnsworth>
	<CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
	<3B2D7559-F2DE-473E-8C89-91A19F6AD33D@molden.no>
	<CAGY4rcWDG8Yf6bgqNTWKvu6J_Myuf_Rc+3exNX_y3dn_wYHQuA@mail.gmail.com>
	<50ACE5C7.2030703@molden.no>
	<CAGY4rcUA0b=rmimgK9v3GHiNTxjHV_E0W7TPeQUBqQKxGrshxw@mail.gmail.com>
	<50ACE8F5.2040300@molden.no> <50ACEAAD.4090409@molden.no>
	<CAPJVwB=jrfi+DjswBN2LRuND_ZbDC4xEqiGRtPOZP3CiD6mOow@mail.gmail.com>
Message-ID: <50ACECDC.80100@molden.no>

On 21.11.2012 15:55, Nathaniel Smith wrote:

> I think the point is that it's easy for programmers to decide to avoid
> GCD if they want to use multiprocessing. But it's not so easy for them
> to decide to avoid BLAS.

Actually the answer from Apple was that no API except POSIX is supported 
on both sides of a fork without calling exec.

That certainly affects more than the GCD.



Sturla




From njs at pobox.com  Wed Nov 21 10:29:16 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 21 Nov 2012 15:29:16 +0000
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <50ACECDC.80100@molden.no>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no>
	<1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
	<1353495414.3664.3.camel@farnsworth>
	<CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
	<3B2D7559-F2DE-473E-8C89-91A19F6AD33D@molden.no>
	<CAGY4rcWDG8Yf6bgqNTWKvu6J_Myuf_Rc+3exNX_y3dn_wYHQuA@mail.gmail.com>
	<50ACE5C7.2030703@molden.no>
	<CAGY4rcUA0b=rmimgK9v3GHiNTxjHV_E0W7TPeQUBqQKxGrshxw@mail.gmail.com>
	<50ACE8F5.2040300@molden.no> <50ACEAAD.4090409@molden.no>
	<CAPJVwB=jrfi+DjswBN2LRuND_ZbDC4xEqiGRtPOZP3CiD6mOow@mail.gmail.com>
	<50ACECDC.80100@molden.no>
Message-ID: <CAPJVwBkrMGUwi3TOEzh3m8y2yB7nhSO7C4+p-4iTT7WXLTf6Bg@mail.gmail.com>

On Wed, Nov 21, 2012 at 3:01 PM, Sturla Molden <sturla at molden.no> wrote:
> On 21.11.2012 15:55, Nathaniel Smith wrote:
>
>> I think the point is that it's easy for programmers to decide to avoid
>> GCD if they want to use multiprocessing. But it's not so easy for them
>> to decide to avoid BLAS.
>
> Actually the answer from Apple was that no API except POSIX is supported
> on both sides of a fork without calling exec.
>
> That certainly affects more than the GCD.

Yes, but still only the Apple-specific APIs like Cocoa, Core Audio,
etc. The bottom line is that a script that runs on Linux will almost
certainly run on OS X... unless it uses BLAS and your numpy is linked
to Accelerate. numpy linking to Accelerate breaks things.

-n


From Christos.Siopis at ulb.ac.be  Wed Nov 21 10:27:40 2012
From: Christos.Siopis at ulb.ac.be (Christos Siopis)
Date: Wed, 21 Nov 2012 16:27:40 +0100
Subject: [Numpy-discussion] FOSS for scientists devroom at FOSDEM 2013
Message-ID: <20121121152740.GA6241@ulb.ac.be>

Dear members of the Numerical Python ecosystem
(with apologies for cross-postings),

A day-long session ("devroom") on Free/Libre and Open Source Software
(FLOSS) for scientists will be held during the next FOSDEM conference,
Brussels, 2-3 February 2013 (http://fosdem.org/2013).

We aim at having a dozen or two short talks introducing projects,
advertising brand new features of established tools, discussing issues
relevant to the development of software for scientific computing, and
touching on the interdependence of FLOSS and open science.

You can find more info on the call for talks at:
http://slayoo.github.com/fosdem2013/

The deadline for sending talk proposals is December 16th 2012.

Please send your submissions or comments to:
foss4scientists-devroom at lists.fosdem.org

Please do forward this message to anyone potentially interested.
Please also let us know if you have any suggestions for what would you
like to hear about in the devroom.

Looking forward to meeting you in Brussels.
Thanks in advance.

The conveners,
Sylwester Arabas, Juan Antonio A?el, Christos Siopis

P.S. There are open calls for main-track talks, lightning talks, and
stands at FOSDEM as well, see: http://fosdem.org/2013/

--------------

I would like to add to the above general announcement, that it would be 
great if a main track talk were to be given at FOSDEM about the 
importance of scientific open source software in science and engineering 
today. Main track talks last 50 minutes, and are addressed to all FOSDEM 
participants, something that would add to the visibility of scientific 
software. As an extra bonus, main track speakers have their travel and 
hotel expenses covered by FOSDEM.

I think that the numerical python "ecosystem" could serve as an 
excellent "case study" of the data processing and visualisation 
workflow, while adding an interesting historical dimension, being one of 
the oldest projects of its sort.

If you decide to respond to the call for main track speakers, you should 
start here:

https://fosdem.org/2013/call_for_main_speakers/

Please note the December 1 deadline. I urge you to let us (the science 
software devroom conveners) know about your proposed talk, so that we 
may send a word of recommendation to the FOSDEM committee who will make 
the ultimate selection.

We thank you in advance for your expressions of interest and 
participation!
_______________________________________________
foss4scientists-devroom mailing list
foss4scientists-devroom at lists.fosdem.org
https://lists.fosdem.org/listinfo/foss4scientists-devroom


From chris.barker at noaa.gov  Wed Nov 21 12:39:48 2012
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Wed, 21 Nov 2012 09:39:48 -0800
Subject: [Numpy-discussion] Use OpenBLAS for the binary releases?
In-Reply-To: <CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
References: <50AA6877.7010806@molden.no> <50AA6FA2.4010101@astro.uio.no>
	<CAGY4rcU20mfs_eNhQHB2c2==5JOL=RykWNzC=jXmNEnqP1UfGg@mail.gmail.com>
	<50ABB7C7.2000509@molden.no>
	<CAGY4rcXWkoQeQTL6b9g9Q0YdcGtzgSjn-O0LcbmKYUcb2ASk+Q@mail.gmail.com>
	<50ABDB97.4050500@astro.uio.no> <1353444774.3664.1.camel@farnsworth>
	<CAGY4rcXmBLwA0oqmU+fwW_K2pf3G2k1LH7SkLtnXt=rZ=YBRRg@mail.gmail.com>
	<1353492989.3664.2.camel@farnsworth>
	<CAGY4rcXvtQyBTPXk6bpfcoFvjm+5XDzqgEuzN2_PaEBykXcASQ@mail.gmail.com>
	<1353495414.3664.3.camel@farnsworth>
	<CAGY4rcU+EZ5=rBJxuP21M_0XC2AYs8BUdFMgmOCn2D5jgYAMqg@mail.gmail.com>
Message-ID: <CALGmxE+PFU-kzYCAq2p_dhTpoc0ZuzUzhMwNsu2mJE=cermnDw@mail.gmail.com>

On Wed, Nov 21, 2012 at 3:44 AM, David Cournapeau <cournape at gmail.com> wrote:

> It is rather simple, in that it expects a different, full installer
> for each new combination. I would rather not complicate things too
> much there, as it is working well for its purpose and is not easy to
> modify. We would need something similar on Mac to make openblas a good
> replacement on that platform.

One would think this would be fairly simple on OS-X -- after all,
there is a pretty limited set of hardware (officially) supported by
OS-X. All Intel, and none of the ancient ones.

I'm assuming that the standard *.pkg installers allow some checking of
hardware versions...

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From charlesr.harris at gmail.com  Wed Nov 21 20:51:50 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 21 Nov 2012 18:51:50 -0700
Subject: [Numpy-discussion] the mean, var, std of empty arrays
Message-ID: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>

What should be the value of the mean, var, and std of empty arrays?
Currently

In [12]: a
Out[12]: array([], dtype=int64)

In [13]: a.mean()
Out[13]: nan

In [14]: a.std()
Out[14]: nan

In [15]: a.var()
Out[15]: nan

I think the nan comes from 0/0. All of these also raise warnings the first
time they are called.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121121/b92478dd/attachment.html>

From shish at keba.be  Wed Nov 21 21:22:17 2012
From: shish at keba.be (Olivier Delalleau)
Date: Wed, 21 Nov 2012 21:22:17 -0500
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
Message-ID: <CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>

Current behavior looks sensible to me. I personally would prefer no warning
but I think it makes sense to have one as it can be helpful to detect
issues faster.

-=- Olivier

2012/11/21 Charles R Harris <charlesr.harris at gmail.com>

> What should be the value of the mean, var, and std of empty arrays?
> Currently
>
> In [12]: a
> Out[12]: array([], dtype=int64)
>
> In [13]: a.mean()
> Out[13]: nan
>
> In [14]: a.std()
> Out[14]: nan
>
> In [15]: a.var()
> Out[15]: nan
>
> I think the nan comes from 0/0. All of these also raise warnings the first
> time they are called.
>
> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121121/dc94cf97/attachment.html>

From charlesr.harris at gmail.com  Wed Nov 21 21:33:58 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 21 Nov 2012 19:33:58 -0700
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
	<CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
Message-ID: <CAB6mnxLJGvjdZCRtQn5UyFYxtHOnA78_UVdm=Mry+2aoPM2Yiw@mail.gmail.com>

Hi Olivier,

Please don't top post, it isn't the custom on this list.

On Wed, Nov 21, 2012 at 7:22 PM, Olivier Delalleau <shish at keba.be> wrote:

> Current behavior looks sensible to me. I personally would prefer no
> warning but I think it makes sense to have one as it can be helpful to
> detect issues faster.
>
> -=- Olivier
>
> 2012/11/21 Charles R Harris <charlesr.harris at gmail.com>
>
>> What should be the value of the mean, var, and std of empty arrays?
>> Currently
>>
>> In [12]: a
>> Out[12]: array([], dtype=int64)
>>
>> In [13]: a.mean()
>> Out[13]: nan
>>
>> In [14]: a.std()
>> Out[14]: nan
>>
>> In [15]: a.var()
>> Out[15]: nan
>>
>> I think the nan comes from 0/0. All of these also raise warnings the
>> first time they are called.
>>
>>
The warnings vary and don't directly give information on the cause, i.e.,
empty arrays. If we do go with warnings I think they should be more
specific.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121121/d9b4d634/attachment.html>

From jsseabold at gmail.com  Wed Nov 21 21:38:13 2012
From: jsseabold at gmail.com (Skipper Seabold)
Date: Wed, 21 Nov 2012 21:38:13 -0500
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
	<CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
Message-ID: <CAKF=DjvDOJZsVOYtrf=AtA=3qvRX5P5ut428-Tduwk2xsWQbqA@mail.gmail.com>

On Wed, Nov 21, 2012 at 9:22 PM, Olivier Delalleau <shish at keba.be> wrote:

> Current behavior looks sensible to me. I personally would prefer no
> warning but I think it makes sense to have one as it can be helpful to
> detect issues faster.
>
> -=- Olivier
>

It's configurable.

[~/]
[1]: np.seterr(all='ignore')

[1]: {'divide': 'ignore', 'invalid': 'ignore', 'over': 'ignore', 'under':
'ignore'}

[~/]
[2]: np.array([]).mean()

[2]: nan

[~/]
[3]: np.seterr(all='warn')

[3]: {'divide': 'ignore', 'invalid': 'ignore', 'over': 'ignore', 'under':
'ignore'}

[~/]
[4]: np.array([]).mean()

/usr/local/lib/python2.7/dist-packages/numpy/core/_methods.py:57:
RuntimeWarning: invalid value encountered in double_scalars
  ret = ret / float(rcount)
[4]: nan

Skipper


> 2012/11/21 Charles R Harris <charlesr.harris at gmail.com>
>
>> What should be the value of the mean, var, and std of empty arrays?
>> Currently
>>
>> In [12]: a
>> Out[12]: array([], dtype=int64)
>>
>> In [13]: a.mean()
>> Out[13]: nan
>>
>> In [14]: a.std()
>> Out[14]: nan
>>
>> In [15]: a.var()
>> Out[15]: nan
>>
>> I think the nan comes from 0/0. All of these also raise warnings the
>> first time they are called.
>>
>> Chuck
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121121/cb318fac/attachment.html>

From josef.pktd at gmail.com  Wed Nov 21 21:45:13 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 21 Nov 2012 21:45:13 -0500
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
	<CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
Message-ID: <CAMMTP+AiGLcfsne4SF2fm=1jPvo-rM-1_V30mzW1_FJuiBocNg@mail.gmail.com>

On Wed, Nov 21, 2012 at 9:22 PM, Olivier Delalleau <shish at keba.be> wrote:
> Current behavior looks sensible to me. I personally would prefer no warning
> but I think it makes sense to have one as it can be helpful to detect issues
> faster.

I agree that nan should be the correct answer.
(I gave up trying to define a default for 0/0 in scipy.stats ttests.)

some funnier cases

>>> np.var([1], ddof=1)
0.0
>>> np.var([1], ddof=5)
-0
>>> np.var([1,2], ddof=5)
-0.16666666666666666
>>> np.std([1,2], ddof=5)
nan

But maybe my numpy is too old on my open interpreter
>>> np.__version__
'1.5.1'

Josef

>
> -=- Olivier
>
> 2012/11/21 Charles R Harris <charlesr.harris at gmail.com>
>>
>> What should be the value of the mean, var, and std of empty arrays?
>> Currently
>>
>> In [12]: a
>> Out[12]: array([], dtype=int64)
>>
>> In [13]: a.mean()
>> Out[13]: nan
>>
>> In [14]: a.std()
>> Out[14]: nan
>>
>> In [15]: a.var()
>> Out[15]: nan
>>
>> I think the nan comes from 0/0. All of these also raise warnings the first
>> time they are called.
>>
>> Chuck
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From charlesr.harris at gmail.com  Wed Nov 21 22:35:39 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 21 Nov 2012 20:35:39 -0700
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <CAMMTP+AiGLcfsne4SF2fm=1jPvo-rM-1_V30mzW1_FJuiBocNg@mail.gmail.com>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
	<CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
	<CAMMTP+AiGLcfsne4SF2fm=1jPvo-rM-1_V30mzW1_FJuiBocNg@mail.gmail.com>
Message-ID: <CAB6mnxKq7+av4t-hEkPpk42G8-60wovnqGF0wrX2G5zQNohYrA@mail.gmail.com>

On Wed, Nov 21, 2012 at 7:45 PM, <josef.pktd at gmail.com> wrote:

> On Wed, Nov 21, 2012 at 9:22 PM, Olivier Delalleau <shish at keba.be> wrote:
> > Current behavior looks sensible to me. I personally would prefer no
> warning
> > but I think it makes sense to have one as it can be helpful to detect
> issues
> > faster.
>
> I agree that nan should be the correct answer.
> (I gave up trying to define a default for 0/0 in scipy.stats ttests.)
>
> some funnier cases
>
> >>> np.var([1], ddof=1)
> 0.0
>

This one is a nan in development.


> >>> np.var([1], ddof=5)
> -0
> >>> np.var([1,2], ddof=5)
> -0.16666666666666666
> >>> np.std([1,2], ddof=5)
> nan
>
>
These still do this. Also

In [10]: var([], ddof=1)
Out[10]: -0

Which suggests that the nan is pretty much an accidental byproduct of
division by zero. I think it might make sense to have a definite policy for
these corner cases.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121121/d101b5db/attachment.html>

From josef.pktd at gmail.com  Wed Nov 21 22:58:47 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 21 Nov 2012 22:58:47 -0500
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <CAB6mnxKq7+av4t-hEkPpk42G8-60wovnqGF0wrX2G5zQNohYrA@mail.gmail.com>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
	<CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
	<CAMMTP+AiGLcfsne4SF2fm=1jPvo-rM-1_V30mzW1_FJuiBocNg@mail.gmail.com>
	<CAB6mnxKq7+av4t-hEkPpk42G8-60wovnqGF0wrX2G5zQNohYrA@mail.gmail.com>
Message-ID: <CAMMTP+DdPiXhpgsi8nuvAMVYkmuaWpdfN3LKgRcbJSQdfmztmA@mail.gmail.com>

On Wed, Nov 21, 2012 at 10:35 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Wed, Nov 21, 2012 at 7:45 PM, <josef.pktd at gmail.com> wrote:
>>
>> On Wed, Nov 21, 2012 at 9:22 PM, Olivier Delalleau <shish at keba.be> wrote:
>> > Current behavior looks sensible to me. I personally would prefer no
>> > warning
>> > but I think it makes sense to have one as it can be helpful to detect
>> > issues
>> > faster.
>>
>> I agree that nan should be the correct answer.
>> (I gave up trying to define a default for 0/0 in scipy.stats ttests.)
>>
>> some funnier cases
>>
>> >>> np.var([1], ddof=1)
>> 0.0
>
>
> This one is a nan in development.
>
>>
>> >>> np.var([1], ddof=5)
>> -0
>> >>> np.var([1,2], ddof=5)
>> -0.16666666666666666
>> >>> np.std([1,2], ddof=5)
>> nan
>>
>
> These still do this. Also
>
> In [10]: var([], ddof=1)
> Out[10]: -0
>
> Which suggests that the nan is pretty much an accidental byproduct of
> division by zero. I think it might make sense to have a definite policy for
> these corner cases.

It would also be consistent with the usual pattern to raise a
ValueError on this. ddof too large, size too small.
It wouldn't be the case that for some columns or rows we get valid
answers in this case, as long as we don't allow for missing values.


quick check with np.ma

looks correct except when delegating to numpy ?

>>> s = np.ma.var(np.ma.masked_invalid([[1.,2],[1,np.nan]]), ddof=5, axis=0)
>>> s
masked_array(data = [-- --],
             mask = [ True  True],
       fill_value = 1e+20)

>>> s = np.ma.var(np.ma.masked_invalid([[1.,2],[1,np.nan]]), ddof=1, axis=0)
>>> s
masked_array(data = [0.0 --],
             mask = [False  True],
       fill_value = 1e+20)

>>> s = np.ma.std([1,2], ddof=5)
>>> s
masked
>>> type(s)
<class 'numpy.ma.core.MaskedConstant'>

>>> np.ma.var([1,2], ddof=5)
-0.16666666666666666


Josef

>
> <snip>
>
> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From fperez.net at gmail.com  Wed Nov 21 23:17:45 2012
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 21 Nov 2012 20:17:45 -0800
Subject: [Numpy-discussion] In case anybody wants them: py4science.*
	domains...
Message-ID: <CAHAreOrVR1050-mmTF67pYwPO-xmJ4s0pO-LdbaUT67vvL6k_w@mail.gmail.com>

Hi folks,

years ago, John Hunter and I bought the py4science.{com, org, info}
domains thinking they might be useful.  We never did anything with
them, and with his passing I realized I'm not really in the mood to
keep renewing them without a clear goal in mind.

Does anybody here want to do anything with these?  They expire
December 3, 2012.  I can just let them lapse, but I figured I'd give a
heads-up in case anybody has a concrete use for them I'd rather
transfer them than let them go to a domain squatter.

Basically, if you want them, they're yours to have as of 12/3/12.

Cheers,

f


From josef.pktd at gmail.com  Wed Nov 21 23:20:14 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 21 Nov 2012 23:20:14 -0500
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <CAMMTP+DdPiXhpgsi8nuvAMVYkmuaWpdfN3LKgRcbJSQdfmztmA@mail.gmail.com>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
	<CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
	<CAMMTP+AiGLcfsne4SF2fm=1jPvo-rM-1_V30mzW1_FJuiBocNg@mail.gmail.com>
	<CAB6mnxKq7+av4t-hEkPpk42G8-60wovnqGF0wrX2G5zQNohYrA@mail.gmail.com>
	<CAMMTP+DdPiXhpgsi8nuvAMVYkmuaWpdfN3LKgRcbJSQdfmztmA@mail.gmail.com>
Message-ID: <CAMMTP+BD_O6J3iVqPtLQdKzW+oGQVxrRr0sDOQSGgqzgPjfqpQ@mail.gmail.com>

On Wed, Nov 21, 2012 at 10:58 PM,  <josef.pktd at gmail.com> wrote:
> On Wed, Nov 21, 2012 at 10:35 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>>
>> On Wed, Nov 21, 2012 at 7:45 PM, <josef.pktd at gmail.com> wrote:
>>>
>>> On Wed, Nov 21, 2012 at 9:22 PM, Olivier Delalleau <shish at keba.be> wrote:
>>> > Current behavior looks sensible to me. I personally would prefer no
>>> > warning
>>> > but I think it makes sense to have one as it can be helpful to detect
>>> > issues
>>> > faster.
>>>
>>> I agree that nan should be the correct answer.
>>> (I gave up trying to define a default for 0/0 in scipy.stats ttests.)
>>>
>>> some funnier cases
>>>
>>> >>> np.var([1], ddof=1)
>>> 0.0
>>
>>
>> This one is a nan in development.
>>
>>>
>>> >>> np.var([1], ddof=5)
>>> -0
>>> >>> np.var([1,2], ddof=5)
>>> -0.16666666666666666
>>> >>> np.std([1,2], ddof=5)
>>> nan
>>>
>>
>> These still do this. Also
>>
>> In [10]: var([], ddof=1)
>> Out[10]: -0
>>
>> Which suggests that the nan is pretty much an accidental byproduct of
>> division by zero. I think it might make sense to have a definite policy for
>> these corner cases.
>
> It would also be consistent with the usual pattern to raise a
> ValueError on this. ddof too large, size too small.
> It wouldn't be the case that for some columns or rows we get valid
> answers in this case, as long as we don't allow for missing values.

I think I prefer NaNs to an exception, they propagate nicer to
downstream functions.

I'm in favor of a policy instead of nans or wrong numbers by accident.

>
>
> quick check with np.ma
>
> looks correct except when delegating to numpy ?
>
>>>> s = np.ma.var(np.ma.masked_invalid([[1.,2],[1,np.nan]]), ddof=5, axis=0)
>>>> s
> masked_array(data = [-- --],
>              mask = [ True  True],
>        fill_value = 1e+20)
>
>>>> s = np.ma.var(np.ma.masked_invalid([[1.,2],[1,np.nan]]), ddof=1, axis=0)
>>>> s
> masked_array(data = [0.0 --],
>              mask = [False  True],
>        fill_value = 1e+20)
>
>>>> s = np.ma.std([1,2], ddof=5)
>>>> s
> masked
>>>> type(s)
> <class 'numpy.ma.core.MaskedConstant'>
>
>>>> np.ma.var([1,2], ddof=5)
> -0.16666666666666666

and cov:

>>> np.cov([1.],[3],bias=True, rowvar=False)   #looks fine
array([[ 0.,  0.],
       [ 0.,  0.]])
>>> np.cov([1.],[3],bias=False, rowvar=False)
array([[ nan,  nan],
       [ nan,  nan]])

>>> np.cov([[1.],[3]],bias=False, rowvar=True)
array([[ nan,  nan],
       [ nan,  nan]])

>>> np.cov([],[],bias=False, rowvar=False)     #should be nan
array([[-0., -0.],
       [-0., -0.]])
>>> np.cov([],[],bias=True, rowvar=False)
array([[ nan,  nan],
       [ nan,  nan]])


np.corrcoef seems to have nans in the right places in the examples I tried

Josef

>
>
> Josef
>
>>
>> <snip>
>>
>> Chuck
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>


From p.j.a.cock at googlemail.com  Thu Nov 22 04:30:02 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Thu, 22 Nov 2012 09:30:02 +0000
Subject: [Numpy-discussion] In case anybody wants them: py4science.*
	domains...
In-Reply-To: <CAHAreOrVR1050-mmTF67pYwPO-xmJ4s0pO-LdbaUT67vvL6k_w@mail.gmail.com>
References: <CAHAreOrVR1050-mmTF67pYwPO-xmJ4s0pO-LdbaUT67vvL6k_w@mail.gmail.com>
Message-ID: <CAKVJ-_4YJ=P3Aaox5tUgLQDK9WtrWtudx2174UZapzgQKAf5sg@mail.gmail.com>

On Thu, Nov 22, 2012 at 4:17 AM, Fernando Perez <fperez.net at gmail.com> wrote:
> Hi folks,
>
> years ago, John Hunter and I bought the py4science.{com, org, info}
> domains thinking they might be useful.  We never did anything with
> them, and with his passing I realized I'm not really in the mood to
> keep renewing them without a clear goal in mind.
>
> Does anybody here want to do anything with these?  They expire
> December 3, 2012.  I can just let them lapse, but I figured I'd give a
> heads-up in case anybody has a concrete use for them I'd rather
> transfer them than let them go to a domain squatter.
>
> Basically, if you want them, they're yours to have as of 12/3/12.
>
> Cheers,
>
> f

Perhaps http://numfocus.org/ could take them on, or the PSF?
(even if they don't have a specific use in mind immediately)
For the short them I'd just have them redirect to www.scipy.org ;)

Peter


From sebastian at sipsolutions.net  Thu Nov 22 07:14:34 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Thu, 22 Nov 2012 13:14:34 +0100
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <CAMMTP+DdPiXhpgsi8nuvAMVYkmuaWpdfN3LKgRcbJSQdfmztmA@mail.gmail.com>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
	<CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
	<CAMMTP+AiGLcfsne4SF2fm=1jPvo-rM-1_V30mzW1_FJuiBocNg@mail.gmail.com>
	<CAB6mnxKq7+av4t-hEkPpk42G8-60wovnqGF0wrX2G5zQNohYrA@mail.gmail.com>
	<CAMMTP+DdPiXhpgsi8nuvAMVYkmuaWpdfN3LKgRcbJSQdfmztmA@mail.gmail.com>
Message-ID: <1353586474.21512.19.camel@sebastian-laptop>

On Wed, 2012-11-21 at 22:58 -0500, josef.pktd at gmail.com wrote:
> On Wed, Nov 21, 2012 at 10:35 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> >
> > On Wed, Nov 21, 2012 at 7:45 PM, <josef.pktd at gmail.com> wrote:
> >>
> >> On Wed, Nov 21, 2012 at 9:22 PM, Olivier Delalleau <shish at keba.be> wrote:
> >> > Current behavior looks sensible to me. I personally would prefer no
> >> > warning
> >> > but I think it makes sense to have one as it can be helpful to detect
> >> > issues
> >> > faster.
> >>
> >> I agree that nan should be the correct answer.
> >> (I gave up trying to define a default for 0/0 in scipy.stats ttests.)
> >>
> >> some funnier cases
> >>
> >> >>> np.var([1], ddof=1)
> >> 0.0
> >
> >
> > This one is a nan in development.
> >
> >>
> >> >>> np.var([1], ddof=5)
> >> -0
> >> >>> np.var([1,2], ddof=5)
> >> -0.16666666666666666
> >> >>> np.std([1,2], ddof=5)
> >> nan
> >>
> >
> > These still do this. Also
> >
> > In [10]: var([], ddof=1)
> > Out[10]: -0
> >
> > Which suggests that the nan is pretty much an accidental byproduct of
> > division by zero. I think it might make sense to have a definite policy for
> > these corner cases.
> 
> It would also be consistent with the usual pattern to raise a
> ValueError on this. ddof too large, size too small.
> It wouldn't be the case that for some columns or rows we get valid
> answers in this case, as long as we don't allow for missing values.
> 

It seems to me that nan is the reasonable result for these operations
(reduce like operations that do not have an identity). Though actually
reduce operations without an identity throw a ValueError (ie.
`np.minimum.reduce([])`), but then mean/std/var seem special enough to
be different from other reduce operations (for example their result is
always floating point). As for usability I think for example when
plotting errorbars using std, it would be rather annoying to get a
ValueError, so if anything the reduce machinery could give more special
results for empty floating point reductions.

In any case the warning should be clearer and for too large ddof's I
would say it should return nan+Warning as well.

Sebastian

> 
> quick check with np.ma
> 
> looks correct except when delegating to numpy ?
> 
> >>> s = np.ma.var(np.ma.masked_invalid([[1.,2],[1,np.nan]]), ddof=5, axis=0)
> >>> s
> masked_array(data = [-- --],
>              mask = [ True  True],
>        fill_value = 1e+20)
> 
> >>> s = np.ma.var(np.ma.masked_invalid([[1.,2],[1,np.nan]]), ddof=1, axis=0)
> >>> s
> masked_array(data = [0.0 --],
>              mask = [False  True],
>        fill_value = 1e+20)
> 
> >>> s = np.ma.std([1,2], ddof=5)
> >>> s
> masked
> >>> type(s)
> <class 'numpy.ma.core.MaskedConstant'>
> 
> >>> np.ma.var([1,2], ddof=5)
> -0.16666666666666666
> 
> 
> Josef
> 
> >
> > <snip>
> >
> > Chuck
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From chaoyuejoy at gmail.com  Thu Nov 22 07:41:34 2012
From: chaoyuejoy at gmail.com (Chao YUE)
Date: Thu, 22 Nov 2012 13:41:34 +0100
Subject: [Numpy-discussion] the difference between "+" and np.add?
Message-ID: <CAAN-aREVeYj8J0KOYKfv7fr=UttsxcSXQ=dKoabWLBgT-aGk7w@mail.gmail.com>

Dear all,

if I have two ndarray arr1 and arr2 (with the same shape), is there some
difference when I do:

arr = arr1 + arr2

and

arr = np.add(arr1, arr2),

and then if I have more than 2 arrays: arr1, arr2, arr3, arr4, arr5, then I
cannot use np.add anymore as it only recieves 2 arguments.
then what's the best practice to add these arrays? should I do

arr = arr1 + arr2 + arr3 + arr4 + arr5

or I do

arr = np.sum(np.array([arr1, arr2, arr3, arr4, arr5]), axis=0)?

because I just noticed recently that there are functions like np.add,
np.divide, np.substract... before I am using all like directly arr1/arr2,
rather than np.divide(arr1,arr2).

best regards,

Chao

-- 
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121122/e556cce8/attachment.html>

From njs at pobox.com  Thu Nov 22 07:51:23 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 22 Nov 2012 12:51:23 +0000
Subject: [Numpy-discussion] the difference between "+" and np.add?
In-Reply-To: <CAAN-aREVeYj8J0KOYKfv7fr=UttsxcSXQ=dKoabWLBgT-aGk7w@mail.gmail.com>
References: <CAAN-aREVeYj8J0KOYKfv7fr=UttsxcSXQ=dKoabWLBgT-aGk7w@mail.gmail.com>
Message-ID: <CAPJVwBkS6vYK3nZ_pgPDeP2i=aH=PeQY0j2Vj=c3O2_w76L+Gg@mail.gmail.com>

On Thu, Nov 22, 2012 at 12:41 PM, Chao YUE <chaoyuejoy at gmail.com> wrote:
> Dear all,
>
> if I have two ndarray arr1 and arr2 (with the same shape), is there some
> difference when I do:
>
> arr = arr1 + arr2
>
> and
>
> arr = np.add(arr1, arr2),
>
> and then if I have more than 2 arrays: arr1, arr2, arr3, arr4, arr5, then I
> cannot use np.add anymore as it only recieves 2 arguments.
> then what's the best practice to add these arrays? should I do
>
> arr = arr1 + arr2 + arr3 + arr4 + arr5
>
> or I do
>
> arr = np.sum(np.array([arr1, arr2, arr3, arr4, arr5]), axis=0)?
>
> because I just noticed recently that there are functions like np.add,
> np.divide, np.substract... before I am using all like directly arr1/arr2,
> rather than np.divide(arr1,arr2).

For numpy arrays, a + b just calls np.add(a, b) internally. You can
use whichever looks nicer to you. Usually people just use +

np.add can be more flexible, though. For instance, you can write
  np.add(a, b, out=c)
but there's no way to pass extra arguments to the "+" operator.

In fact np.add is not just a function, it's a "ufunc object" (see the
numpy documentation for some more details). So it also provides
methods like
  np.add.reduce(a) # the same as np.sum (except, sadly, with different defaults)
  np.add.accumulate(a) # like np.cumsum
  np.add.reduceat(a, indices) # complicated, see docs
And there are lots of these ufunc objects, all of which provide these
same interfaces. Some of them have associated operators like "+", but
others don't.

-n


From francesc at continuum.io  Thu Nov 22 09:20:39 2012
From: francesc at continuum.io (Francesc Alted)
Date: Thu, 22 Nov 2012 15:20:39 +0100
Subject: [Numpy-discussion] the difference between "+" and np.add?
In-Reply-To: <CAAN-aREVeYj8J0KOYKfv7fr=UttsxcSXQ=dKoabWLBgT-aGk7w@mail.gmail.com>
References: <CAAN-aREVeYj8J0KOYKfv7fr=UttsxcSXQ=dKoabWLBgT-aGk7w@mail.gmail.com>
Message-ID: <50AE34B7.3070401@continuum.io>

On 11/22/12 1:41 PM, Chao YUE wrote:
> Dear all,
>
> if I have two ndarray arr1 and arr2 (with the same shape), is there 
> some difference when I do:
>
> arr = arr1 + arr2
>
> and
>
> arr = np.add(arr1, arr2),
>
> and then if I have more than 2 arrays: arr1, arr2, arr3, arr4, arr5, 
> then I cannot use np.add anymore as it only recieves 2 arguments.
> then what's the best practice to add these arrays? should I do
>
> arr = arr1 + arr2 + arr3 + arr4 + arr5
>
> or I do
>
> arr = np.sum(np.array([arr1, arr2, arr3, arr4, arr5]), axis=0)?
>
> because I just noticed recently that there are functions like np.add, 
> np.divide, np.substract... before I am using all like directly 
> arr1/arr2, rather than np.divide(arr1,arr2).

As Nathaniel said, there is not a difference in terms of *what* is 
computed.  However, the methods that you suggested actually differ on 
*how* they are computed, and that has dramatic effects on the time 
used.  For example:

In []: arr1, arr2, arr3, arr4, arr5 = [np.arange(1e7) for x in range(5)]

In []: %time arr1 + arr2 + arr3 + arr4 + arr5
CPU times: user 0.05 s, sys: 0.10 s, total: 0.14 s
Wall time: 0.15 s
Out[]:
array([  0.00000000e+00,   5.00000000e+00,   1.00000000e+01, ...,
          4.99999850e+07,   4.99999900e+07,   4.99999950e+07])

In []: %time np.sum(np.array([arr1, arr2, arr3, arr4, arr5]), axis=0)
CPU times: user 2.98 s, sys: 0.15 s, total: 3.13 s
Wall time: 3.14 s
Out[]:
array([  0.00000000e+00,   5.00000000e+00,   1.00000000e+01, ...,
          4.99999850e+07,   4.99999900e+07,   4.99999950e+07])

The difference is how memory is used.  In the first case, the additional 
memory was just a temporary with the size of the operands, while for the 
second case a big temporary has to be created, so the difference in is 
speed is pretty large.

There are also ways to minimize the size of temporaries, and numexpr is 
one of the simplests:

In []: import numexpr as ne

In []: %time ne.evaluate('arr1 + arr2 + arr3 + arr4 + arr5')
CPU times: user 0.04 s, sys: 0.04 s, total: 0.08 s
Wall time: 0.04 s
Out[]:
array([  0.00000000e+00,   5.00000000e+00,   1.00000000e+01, ...,
          4.99999850e+07,   4.99999900e+07,   4.99999950e+07])

Again, the computations are the same, but how you manage memory is critical.

-- 
Francesc Alted



From josef.pktd at gmail.com  Thu Nov 22 09:54:28 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 22 Nov 2012 09:54:28 -0500
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <1353586474.21512.19.camel@sebastian-laptop>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
	<CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
	<CAMMTP+AiGLcfsne4SF2fm=1jPvo-rM-1_V30mzW1_FJuiBocNg@mail.gmail.com>
	<CAB6mnxKq7+av4t-hEkPpk42G8-60wovnqGF0wrX2G5zQNohYrA@mail.gmail.com>
	<CAMMTP+DdPiXhpgsi8nuvAMVYkmuaWpdfN3LKgRcbJSQdfmztmA@mail.gmail.com>
	<1353586474.21512.19.camel@sebastian-laptop>
Message-ID: <CAMMTP+AH-hhpMhT-Of8ZnFUQ984x+iWvTw_kwDh0iC+48xQm4g@mail.gmail.com>

On Thu, Nov 22, 2012 at 7:14 AM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> On Wed, 2012-11-21 at 22:58 -0500, josef.pktd at gmail.com wrote:
>> On Wed, Nov 21, 2012 at 10:35 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> >
>> > On Wed, Nov 21, 2012 at 7:45 PM, <josef.pktd at gmail.com> wrote:
>> >>
>> >> On Wed, Nov 21, 2012 at 9:22 PM, Olivier Delalleau <shish at keba.be> wrote:
>> >> > Current behavior looks sensible to me. I personally would prefer no
>> >> > warning
>> >> > but I think it makes sense to have one as it can be helpful to detect
>> >> > issues
>> >> > faster.
>> >>
>> >> I agree that nan should be the correct answer.
>> >> (I gave up trying to define a default for 0/0 in scipy.stats ttests.)
>> >>
>> >> some funnier cases
>> >>
>> >> >>> np.var([1], ddof=1)
>> >> 0.0
>> >
>> >
>> > This one is a nan in development.
>> >
>> >>
>> >> >>> np.var([1], ddof=5)
>> >> -0
>> >> >>> np.var([1,2], ddof=5)
>> >> -0.16666666666666666
>> >> >>> np.std([1,2], ddof=5)
>> >> nan
>> >>
>> >
>> > These still do this. Also
>> >
>> > In [10]: var([], ddof=1)
>> > Out[10]: -0
>> >
>> > Which suggests that the nan is pretty much an accidental byproduct of
>> > division by zero. I think it might make sense to have a definite policy for
>> > these corner cases.
>>
>> It would also be consistent with the usual pattern to raise a
>> ValueError on this. ddof too large, size too small.
>> It wouldn't be the case that for some columns or rows we get valid
>> answers in this case, as long as we don't allow for missing values.
>>
>
> It seems to me that nan is the reasonable result for these operations
> (reduce like operations that do not have an identity). Though actually
> reduce operations without an identity throw a ValueError (ie.
> `np.minimum.reduce([])`), but then mean/std/var seem special enough to
> be different from other reduce operations (for example their result is
> always floating point). As for usability I think for example when
> plotting errorbars using std, it would be rather annoying to get a
> ValueError, so if anything the reduce machinery could give more special
> results for empty floating point reductions.
>
> In any case the warning should be clearer and for too large ddof's I
> would say it should return nan+Warning as well.


Why don't operations on empty arrays not return empty arrays?

but this looks ok

>>> (np.array([]) - np.array([]).mean()) / np.array([]).std()
array([], dtype=float64)
>>> (np.array([]) - np.array([]).mean()) / np.array([]).std(0)
array([], dtype=float64)

>>> (np.array([]) - np.array([]).mean(0)) / np.array([]).std(0)
array([], dtype=float64)
>>> (np.array([]) - np.array([]).mean(0)) / np.array([])
array([], dtype=float64)

>>> np.array([[]]) - np.expand_dims(np.array([[]]).mean(1),1)
array([], shape=(1, 0), dtype=float64)

>>> np.array([[]]) - np.expand_dims(np.array([]),1)
array([], shape=(0, 0), dtype=float64)

>>> np.array([]) - np.expand_dims(np.array([]),0)
array([], shape=(1, 0), dtype=float64)


(But I doubt I will rely in many cases on correct "calculations" with
empty arrays.)

Josef


>
> Sebastian
>
>>
>> quick check with np.ma
>>
>> looks correct except when delegating to numpy ?
>>
>> >>> s = np.ma.var(np.ma.masked_invalid([[1.,2],[1,np.nan]]), ddof=5, axis=0)
>> >>> s
>> masked_array(data = [-- --],
>>              mask = [ True  True],
>>        fill_value = 1e+20)
>>
>> >>> s = np.ma.var(np.ma.masked_invalid([[1.,2],[1,np.nan]]), ddof=1, axis=0)
>> >>> s
>> masked_array(data = [0.0 --],
>>              mask = [False  True],
>>        fill_value = 1e+20)
>>
>> >>> s = np.ma.std([1,2], ddof=5)
>> >>> s
>> masked
>> >>> type(s)
>> <class 'numpy.ma.core.MaskedConstant'>
>>
>> >>> np.ma.var([1,2], ddof=5)
>> -0.16666666666666666
>>
>>
>> Josef
>>
>> >
>> > <snip>
>> >
>> > Chuck
>> >
>> >
>> > _______________________________________________
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From davidmenhur at gmail.com  Thu Nov 22 10:05:11 2012
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Thu, 22 Nov 2012 16:05:11 +0100
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <CAMMTP+AH-hhpMhT-Of8ZnFUQ984x+iWvTw_kwDh0iC+48xQm4g@mail.gmail.com>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
	<CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
	<CAMMTP+AiGLcfsne4SF2fm=1jPvo-rM-1_V30mzW1_FJuiBocNg@mail.gmail.com>
	<CAB6mnxKq7+av4t-hEkPpk42G8-60wovnqGF0wrX2G5zQNohYrA@mail.gmail.com>
	<CAMMTP+DdPiXhpgsi8nuvAMVYkmuaWpdfN3LKgRcbJSQdfmztmA@mail.gmail.com>
	<1353586474.21512.19.camel@sebastian-laptop>
	<CAMMTP+AH-hhpMhT-Of8ZnFUQ984x+iWvTw_kwDh0iC+48xQm4g@mail.gmail.com>
Message-ID: <CAJhcF=0Y5QrWTgOj2h-F=hLgZs028ZKK9boghP6D5U=B2Memrg@mail.gmail.com>

On Thu, Nov 22, 2012 at 3:54 PM,  <josef.pktd at gmail.com> wrote:
> Why don't operations on empty arrays not return empty arrays?

Because functions like mean or std are expected to return a scalar.
Functions that are piecewiese can (and should) return an empty array,
but not the mean.


From sebastian at sipsolutions.net  Thu Nov 22 10:15:54 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Thu, 22 Nov 2012 16:15:54 +0100
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <CAJhcF=0Y5QrWTgOj2h-F=hLgZs028ZKK9boghP6D5U=B2Memrg@mail.gmail.com>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
	<CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
	<CAMMTP+AiGLcfsne4SF2fm=1jPvo-rM-1_V30mzW1_FJuiBocNg@mail.gmail.com>
	<CAB6mnxKq7+av4t-hEkPpk42G8-60wovnqGF0wrX2G5zQNohYrA@mail.gmail.com>
	<CAMMTP+DdPiXhpgsi8nuvAMVYkmuaWpdfN3LKgRcbJSQdfmztmA@mail.gmail.com>
	<1353586474.21512.19.camel@sebastian-laptop>
	<CAMMTP+AH-hhpMhT-Of8ZnFUQ984x+iWvTw_kwDh0iC+48xQm4g@mail.gmail.com>
	<CAJhcF=0Y5QrWTgOj2h-F=hLgZs028ZKK9boghP6D5U=B2Memrg@mail.gmail.com>
Message-ID: <1353597354.21512.27.camel@sebastian-laptop>

On Thu, 2012-11-22 at 16:05 +0100, Da?id wrote:
> On Thu, Nov 22, 2012 at 3:54 PM,  <josef.pktd at gmail.com> wrote:
> > Why don't operations on empty arrays not return empty arrays?
> 
> Because functions like mean or std are expected to return a scalar.
> Functions that are piecewiese can (and should) return an empty array,
> but not the mean.

I agree, this makes sense, note that:

In [2]: a = np.empty((5,0))

In [3]: a.std(0)
Out[3]: array([], dtype=float64)

In [4]: a.std(1)
/usr/bin/ipython:1: RuntimeWarning: invalid value encountered in divide
  #!/usr/bin/env python
Out[4]: array([ nan,  nan,  nan,  nan,  nan])

However you are reducing, and with reducing you expect exactly 1 scalar
result (along that dimension).

> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




From josef.pktd at gmail.com  Thu Nov 22 10:31:28 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 22 Nov 2012 10:31:28 -0500
Subject: [Numpy-discussion] the mean, var, std of empty arrays
In-Reply-To: <1353597354.21512.27.camel@sebastian-laptop>
References: <CAB6mnxKP71qRUHb5yqGMMUg544-zJWoSiZsw3KGwrPYdCwu+sA@mail.gmail.com>
	<CAFXk4brG6uL71A1DKyfK3PwzZg89keF3an_TcpZkzpi4xMRmOw@mail.gmail.com>
	<CAMMTP+AiGLcfsne4SF2fm=1jPvo-rM-1_V30mzW1_FJuiBocNg@mail.gmail.com>
	<CAB6mnxKq7+av4t-hEkPpk42G8-60wovnqGF0wrX2G5zQNohYrA@mail.gmail.com>
	<CAMMTP+DdPiXhpgsi8nuvAMVYkmuaWpdfN3LKgRcbJSQdfmztmA@mail.gmail.com>
	<1353586474.21512.19.camel@sebastian-laptop>
	<CAMMTP+AH-hhpMhT-Of8ZnFUQ984x+iWvTw_kwDh0iC+48xQm4g@mail.gmail.com>
	<CAJhcF=0Y5QrWTgOj2h-F=hLgZs028ZKK9boghP6D5U=B2Memrg@mail.gmail.com>
	<1353597354.21512.27.camel@sebastian-laptop>
Message-ID: <CAMMTP+C4voPVoSertwY-YKXgAWSmvfJ=ZHC0dm+CmojR3o30xQ@mail.gmail.com>

On Thu, Nov 22, 2012 at 10:15 AM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> On Thu, 2012-11-22 at 16:05 +0100, Da?id wrote:
>> On Thu, Nov 22, 2012 at 3:54 PM,  <josef.pktd at gmail.com> wrote:
>> > Why don't operations on empty arrays not return empty arrays?
>>
>> Because functions like mean or std are expected to return a scalar.
>> Functions that are piecewiese can (and should) return an empty array,
>> but not the mean.
>
> I agree, this makes sense, note that:
>
> In [2]: a = np.empty((5,0))
>
> In [3]: a.std(0)
> Out[3]: array([], dtype=float64)
>
> In [4]: a.std(1)
> /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in divide
>   #!/usr/bin/env python
> Out[4]: array([ nan,  nan,  nan,  nan,  nan])
>
> However you are reducing, and with reducing you expect exactly 1 scalar
> result (along that dimension).

Ok, I see.
we cannot have an empty 1-D array shape (5,)

Josef

>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


From chaoyuejoy at gmail.com  Thu Nov 22 13:09:02 2012
From: chaoyuejoy at gmail.com (Chao YUE)
Date: Thu, 22 Nov 2012 19:09:02 +0100
Subject: [Numpy-discussion] the difference between "+" and np.add?
In-Reply-To: <50AE34B7.3070401@continuum.io>
References: <CAAN-aREVeYj8J0KOYKfv7fr=UttsxcSXQ=dKoabWLBgT-aGk7w@mail.gmail.com>
	<50AE34B7.3070401@continuum.io>
Message-ID: <CAAN-aRGMmof96mtF42b7FR_w4acwn0NB1=mjZP1qHcw0NxC-hw@mail.gmail.com>

Thanks for the explanations. Yes, what I am thinking is basically the same
but I didn't test the time.

I never try numexpr, but it would be nice to try it.

Chao

On Thu, Nov 22, 2012 at 3:20 PM, Francesc Alted <francesc at continuum.io>wrote:

> On 11/22/12 1:41 PM, Chao YUE wrote:
> > Dear all,
> >
> > if I have two ndarray arr1 and arr2 (with the same shape), is there
> > some difference when I do:
> >
> > arr = arr1 + arr2
> >
> > and
> >
> > arr = np.add(arr1, arr2),
> >
> > and then if I have more than 2 arrays: arr1, arr2, arr3, arr4, arr5,
> > then I cannot use np.add anymore as it only recieves 2 arguments.
> > then what's the best practice to add these arrays? should I do
> >
> > arr = arr1 + arr2 + arr3 + arr4 + arr5
> >
> > or I do
> >
> > arr = np.sum(np.array([arr1, arr2, arr3, arr4, arr5]), axis=0)?
> >
> > because I just noticed recently that there are functions like np.add,
> > np.divide, np.substract... before I am using all like directly
> > arr1/arr2, rather than np.divide(arr1,arr2).
>
> As Nathaniel said, there is not a difference in terms of *what* is
> computed.  However, the methods that you suggested actually differ on
> *how* they are computed, and that has dramatic effects on the time
> used.  For example:
>
> In []: arr1, arr2, arr3, arr4, arr5 = [np.arange(1e7) for x in range(5)]
>
> In []: %time arr1 + arr2 + arr3 + arr4 + arr5
> CPU times: user 0.05 s, sys: 0.10 s, total: 0.14 s
> Wall time: 0.15 s
> Out[]:
> array([  0.00000000e+00,   5.00000000e+00,   1.00000000e+01, ...,
>           4.99999850e+07,   4.99999900e+07,   4.99999950e+07])
>
> In []: %time np.sum(np.array([arr1, arr2, arr3, arr4, arr5]), axis=0)
> CPU times: user 2.98 s, sys: 0.15 s, total: 3.13 s
> Wall time: 3.14 s
> Out[]:
> array([  0.00000000e+00,   5.00000000e+00,   1.00000000e+01, ...,
>           4.99999850e+07,   4.99999900e+07,   4.99999950e+07])
>
> The difference is how memory is used.  In the first case, the additional
> memory was just a temporary with the size of the operands, while for the
> second case a big temporary has to be created, so the difference in is
> speed is pretty large.
>
> There are also ways to minimize the size of temporaries, and numexpr is
> one of the simplests:
>
> In []: import numexpr as ne
>
> In []: %time ne.evaluate('arr1 + arr2 + arr3 + arr4 + arr5')
> CPU times: user 0.04 s, sys: 0.04 s, total: 0.08 s
> Wall time: 0.04 s
> Out[]:
> array([  0.00000000e+00,   5.00000000e+00,   1.00000000e+01, ...,
>           4.99999850e+07,   4.99999900e+07,   4.99999950e+07])
>
> Again, the computations are the same, but how you manage memory is
> critical.
>
> --
> Francesc Alted
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121122/6ffb8843/attachment.html>

From chaoyuejoy at gmail.com  Thu Nov 22 16:08:50 2012
From: chaoyuejoy at gmail.com (Chao YUE)
Date: Thu, 22 Nov 2012 22:08:50 +0100
Subject: [Numpy-discussion] Is apply_over_axes working for np.ma function?
Message-ID: <CAAN-aRFbzZ5oVY=O4u772E7JdzApawgdMjhZKk4jr1PA4-JR8A@mail.gmail.com>

Dear all,

I tried the np.apply_over_axes and np.ma.apply_over_axes, it seems that
they are not working for the masked array?
I searched the wiki and there are two tickets (1480,8417) related with
this, it seems that it's a solved issue?

the example is below:

In [67]: a = np.arange(60.).reshape(3,4,5)

In [68]: tempm = np.tile(np.array([True,True,False,False,False]),(3,4,1))

In [69]: b = np.ma.masked_array(a,mask=tempm)

In [70]: np.apply_over_axes(np.ma.sum,a,[1,2])
Out[70]:
array([[[ 190.]],

       [[ 590.]],

       [[ 990.]]])

In [71]: np.ma.sum(np.ma.sum(a,axis=1),axis=1)
Out[71]: array([ 190.,  590.,  990.])

In [72]: np.ma.sum(np.ma.sum(b,axis=1),axis=1)
Out[72]:
masked_array(data = [126.0 366.0 606.0],
             mask = [False False False],
       fill_value = 1e+20)


In [73]: np.apply_over_axes(np.ma.sum,b,[1,2])
Out[73]:
array([[[ 190.]],

       [[ 590.]],

       [[ 990.]]])

In [74]: np.apply_over_axes(np.sum,b,[1,2])
Out[74]:
array([[[ 190.]],

       [[ 590.]],

       [[ 990.]]])

In [75]: np.ma.apply_over_axes(np.sum,b,[1,2])
Out[75]:
array([[[ 190.]],

       [[ 590.]],

       [[ 990.]]])

In [76]: np.ma.apply_over_axes(np.ma.sum,b,[1,2])
Out[76]:
array([[[ 190.]],

       [[ 590.]],

       [[ 990.]]])


thanks,

Chao

-- 
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121122/f96379fc/attachment.html>

From chaoyuejoy at gmail.com  Thu Nov 22 16:13:18 2012
From: chaoyuejoy at gmail.com (Chao YUE)
Date: Thu, 22 Nov 2012 22:13:18 +0100
Subject: [Numpy-discussion] Is apply_over_axes working for np.ma
	function?
In-Reply-To: <CAAN-aRFbzZ5oVY=O4u772E7JdzApawgdMjhZKk4jr1PA4-JR8A@mail.gmail.com>
References: <CAAN-aRFbzZ5oVY=O4u772E7JdzApawgdMjhZKk4jr1PA4-JR8A@mail.gmail.com>
Message-ID: <CAAN-aRFwQVJ9-4+G3_eCFOGO3Tb4J9ePtEP2c4EqOXFhHdjp+A@mail.gmail.com>

I am using version 1.6.2

In [77]: np.__version__
Out[77]: '1.6.2'


On Thu, Nov 22, 2012 at 10:08 PM, Chao YUE <chaoyuejoy at gmail.com> wrote:

> Dear all,
>
> I tried the np.apply_over_axes and np.ma.apply_over_axes, it seems that
> they are not working for the masked array?
> I searched the wiki and there are two tickets (1480,8417) related with
> this, it seems that it's a solved issue?
>
> the example is below:
>
> In [67]: a = np.arange(60.).reshape(3,4,5)
>
> In [68]: tempm = np.tile(np.array([True,True,False,False,False]),(3,4,1))
>
> In [69]: b = np.ma.masked_array(a,mask=tempm)
>
> In [70]: np.apply_over_axes(np.ma.sum,a,[1,2])
> Out[70]:
> array([[[ 190.]],
>
>        [[ 590.]],
>
>        [[ 990.]]])
>
> In [71]: np.ma.sum(np.ma.sum(a,axis=1),axis=1)
> Out[71]: array([ 190.,  590.,  990.])
>
> In [72]: np.ma.sum(np.ma.sum(b,axis=1),axis=1)
> Out[72]:
> masked_array(data = [126.0 366.0 606.0],
>              mask = [False False False],
>        fill_value = 1e+20)
>
>
> In [73]: np.apply_over_axes(np.ma.sum,b,[1,2])
> Out[73]:
> array([[[ 190.]],
>
>        [[ 590.]],
>
>        [[ 990.]]])
>
> In [74]: np.apply_over_axes(np.sum,b,[1,2])
> Out[74]:
> array([[[ 190.]],
>
>        [[ 590.]],
>
>        [[ 990.]]])
>
> In [75]: np.ma.apply_over_axes(np.sum,b,[1,2])
> Out[75]:
> array([[[ 190.]],
>
>        [[ 590.]],
>
>        [[ 990.]]])
>
> In [76]: np.ma.apply_over_axes(np.ma.sum,b,[1,2])
> Out[76]:
> array([[[ 190.]],
>
>        [[ 590.]],
>
>        [[ 990.]]])
>
>
> thanks,
>
> Chao
>
> --
>
> ***********************************************************************************
> Chao YUE
> Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
> UMR 1572 CEA-CNRS-UVSQ
> Batiment 712 - Pe 119
> 91191 GIF Sur YVETTE Cedex
> Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
>
> ************************************************************************************
>
>


-- 
***********************************************************************************
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121122/0f2e05e4/attachment.html>

From charlesr.harris at gmail.com  Thu Nov 22 22:34:27 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 22 Nov 2012 20:34:27 -0700
Subject: [Numpy-discussion] When are 0-d arrays writeable?
Message-ID: <CAB6mnx+s4YuE1Q5er64R9j8NO9vgek6jQ6_yK60=Gc+B1XnsCw@mail.gmail.com>

Examples,

In [13]: ones(()).flags.writeable
Out[13]: True

In [14]: (-ones(())).flags.writeable
Out[14]: False

In [15]: (-1*ones(())).flags.writeable
Out[15]: False

In [16]: (1 + ones(())).flags.writeable
Out[16]: False

In [17]: array(1)
Out[17]: array(1)

In [18]: array(1).shape
Out[18]: ()

In [19]: array(1).flags.writeable
Out[19]: True


Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121122/b32f3811/attachment.html>

From njs at pobox.com  Fri Nov 23 05:49:37 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 23 Nov 2012 10:49:37 +0000
Subject: [Numpy-discussion] When are 0-d arrays writeable?
In-Reply-To: <CAB6mnx+s4YuE1Q5er64R9j8NO9vgek6jQ6_yK60=Gc+B1XnsCw@mail.gmail.com>
References: <CAB6mnx+s4YuE1Q5er64R9j8NO9vgek6jQ6_yK60=Gc+B1XnsCw@mail.gmail.com>
Message-ID: <CAPJVwB=S5sEF-=c_N4SMR7+XJZ-i_yji7cW+tSUcLfKoE-jL+w@mail.gmail.com>

On 23 Nov 2012 03:34, "Charles R Harris" <charlesr.harris at gmail.com> wrote:
>
> Examples,
>
> In [13]: ones(()).flags.writeable
> Out[13]: True
>
> In [14]: (-ones(())).flags.writeable
> Out[14]: False
>
> In [15]: (-1*ones(())).flags.writeable
> Out[15]: False
>
> In [16]: (1 + ones(())).flags.writeable
> Out[16]: False
>
> In [17]: array(1)
> Out[17]: array(1)
>
> In [18]: array(1).shape
> Out[18]: ()
>
> In [19]: array(1).flags.writeable
> Out[19]: True

Looks like a bug in the ufunc output value setup code or something?

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121123/fd8c922b/attachment.html>

From rjd4+numpy at cam.ac.uk  Fri Nov 23 06:38:25 2012
From: rjd4+numpy at cam.ac.uk (Bob Dowling)
Date: Fri, 23 Nov 2012 11:38:25 +0000
Subject: [Numpy-discussion] Unexpected RuntimeWarning
Message-ID: <50AF6031.2010503@cam.ac.uk>

I have a simple function defined in the following snippet:

  --- start ---
import numpy

def chebyshev(x, m):
     '''Calculates Chebyshev functions of the first kind using the 
trigonometric identities.'''

     theta = numpy.where(
         numpy.abs(x)<=1.0,
         numpy.arccos(x),
         numpy.arccosh(numpy.abs(x))
         )

     y = numpy.where(
         numpy.abs(x)<=1.0,
         numpy.cos(m*theta),
         numpy.cosh(m*theta) * numpy.where(
             x > 0.0,
             1.0,
             -1.0
             )**m
         )

     return y


if __name__ == '__main__':
     x = numpy.linspace(-2.0, 2.0, 21)
     y = chebyshev(x,3)
     print(y)

  --- end ---


I'm using the numpy.where() call to extract only legal values for the 
circular and hyperbolic trigonometric functions.  But I still get 
warnings that I'm passing invalid anrguments:

--- start---

$ python3 demo.py
demo.py:8: RuntimeWarning: invalid value encountered in arccos
   numpy.arccos(x),
demo.py:9: RuntimeWarning: invalid value encountered in arccosh
   numpy.arccosh(numpy.abs(x))
[ -2.60000000e+01  -1.79280000e+01  -1.15840000e+01  -6.77600000e+00
   -3.31200000e+00  -1.00000000e+00   3.52000000e-01   9.36000000e-01
    9.44000000e-01   5.68000000e-01  -1.83697020e-16  -5.68000000e-01
   -9.44000000e-01  -9.36000000e-01  -3.52000000e-01   1.00000000e+00
    3.31200000e+00   6.77600000e+00   1.15840000e+01   1.79280000e+01
    2.60000000e+01]

  --- end ---

(I get the same with Python 2 so don't get excited about that.)


I'm guessing that numpy.where() is evaluating the complete arccos and 
arccosh arrays and therefore getting invalid arguments.

Now, I can turn off the warnings with numpy.seterr(invalid='ignore') but 
that's not what I would regard as good practice.

Is there a "numpythonic" way to address the issue?


From pav at iki.fi  Fri Nov 23 06:42:12 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 23 Nov 2012 11:42:12 +0000 (UTC)
Subject: [Numpy-discussion] Unexpected RuntimeWarning
References: <50AF6031.2010503@cam.ac.uk>
Message-ID: <loom.20121123T124043-647@post.gmane.org>

Bob Dowling <rjd4+numpy <at> cam.ac.uk> writes:
[clip]
> I'm guessing that numpy.where() is evaluating the complete arccos and 
> arccosh arrays and therefore getting invalid arguments.
> 
> Now, I can turn off the warnings with numpy.seterr(invalid='ignore') but 
> that's not what I would regard as good practice.
> 
> Is there a "numpythonic" way to address the issue?

Correct, the arguments are evaluated first, due to how Python semantics work.

You may want to use this:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.piecewise.html



From rjd4+numpy at cam.ac.uk  Fri Nov 23 07:07:51 2012
From: rjd4+numpy at cam.ac.uk (Bob Dowling)
Date: Fri, 23 Nov 2012 12:07:51 +0000
Subject: [Numpy-discussion] Unexpected RuntimeWarning
In-Reply-To: <loom.20121123T124043-647@post.gmane.org>
References: <50AF6031.2010503@cam.ac.uk>
	<loom.20121123T124043-647@post.gmane.org>
Message-ID: <50AF6717.6040605@cam.ac.uk>

> You may want to use this:
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.piecewise.html

Thank you.  That's just what I needed.

Works a treat:
  --- start ---

import numpy

def chebyshev(x, m):
     '''Calculates Chebyshev functions of the first kind using the 
trigonometric identities.'''

     zone_a = x < -1.0
     zone_b = numpy.logical_and(-1.0 <= x, x <= 1.0)
     zone_c = 1.0 < x

     theta = numpy.piecewise(
         x,
         [zone_a, zone_b, zone_c],
         [lambda z: numpy.arccosh(numpy.abs(z)), numpy.arccos, 
numpy.arccosh]
         )

     y = numpy.piecewise(
         theta,
         [zone_a, zone_b, zone_c],
         [lambda z: (-1.0)**m*numpy.cosh(m*z), lambda z: numpy.cos(m*z), 
lambda z: numpy.cosh(m*z)]
         )

     return y


if __name__ == '__main__':
     x = numpy.linspace(-2.0, 2.0, 21)
     y = chebyshev(x,3)
     print(y)

  --- end---


From sebastian at sipsolutions.net  Fri Nov 23 09:53:19 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Fri, 23 Nov 2012 15:53:19 +0100
Subject: [Numpy-discussion] When are 0-d arrays writeable?
In-Reply-To: <CAPJVwB=S5sEF-=c_N4SMR7+XJZ-i_yji7cW+tSUcLfKoE-jL+w@mail.gmail.com>
References: <CAB6mnx+s4YuE1Q5er64R9j8NO9vgek6jQ6_yK60=Gc+B1XnsCw@mail.gmail.com>
	<CAPJVwB=S5sEF-=c_N4SMR7+XJZ-i_yji7cW+tSUcLfKoE-jL+w@mail.gmail.com>
Message-ID: <1353682399.15250.11.camel@sebastian-laptop>

On Fri, 2012-11-23 at 10:49 +0000, Nathaniel Smith wrote:
> On 23 Nov 2012 03:34, "Charles R Harris" <charlesr.harris at gmail.com>
> wrote:
> >
> > Examples,
> >
> > In [13]: ones(()).flags.writeable
> > Out[13]: True
> >
> > In [14]: (-ones(())).flags.writeable
> > Out[14]: False
> >
> > In [15]: (-1*ones(())).flags.writeable
> > Out[15]: False
> >
> > In [16]: (1 + ones(())).flags.writeable
> > Out[16]: False
> >
> > In [17]: array(1)
> > Out[17]: array(1)
> >
> > In [18]: array(1).shape
> > Out[18]: ()
> >
> > In [19]: array(1).flags.writeable
> > Out[19]: True
> 
> Looks like a bug in the ufunc output value setup code or something?
> 
It might be possible to rethink when (or if) to convert 0-d array to a
scalar. However, at the moment as far as I understand many functions
generally do not return 0-d arrays but scalars. Which makes sense
because mostly we would rather use scalars then 0-d arrays as they are
closer to typical python (hashable and subclasses of python types).

Of course the way this is done is not aware of what is put in (scalar
vs. 0-d array), since all input is converted to an array normally, which
means that most (all?) functions either return 0-d arrays or scalars and
are never aware if the original input was a scalar or an array. Maybe
there could be a np.asarray_or_scalar or such so that its easier to give
the same output type as the original input type?

Regards,

Sebastian

> -n
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From tmp50 at ukr.net  Fri Nov 23 11:22:28 2012
From: tmp50 at ukr.net (Dmitrey)
Date: Fri, 23 Nov 2012 18:22:28 +0200
Subject: [Numpy-discussion] [ANN] Stochastic programming and optimization
	addon for FuncDesigner v. 0.421
Message-ID: <31182.1353687748.10255534963624443904@ffe16.ukr.net>




hi all,
I'm glad to inform you that stochastic programming and optimization addon
for FuncDesigner v. 0.421 has been released.

Now you can use gradient-based solvers for numerical optimization, such
as ALGENCAN, IPOPT, ralg, gsubg etc. Usually they work faster than
derivative-free (such as scipy_cobyla, BOBYQA)  or global (GLP) solvers,
e.g. on this example ALGENCAN time elapsed is less than 1 second while
scipy_cobyla spend ~20 sec. However pay attention that having function
P() in your problem may bring nonconvexity or some other issues to the
solver optimization trajectory, thus sometimes you'll have to use
derivative-free or GLP solvers (e.g. de) instead.

FuncDesigner is free (BSD license) cross-platform Python language written
software, while its stochastic programming and optimization addon,
written by same authors, is free for small-scaled problems with
educational or research purposes only.

For more details visit our website http://openopt.org

-------------------------------------
Regards, D.
http://openopt.org/Dmitrey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121123/24305e6c/attachment.html>

From charlesr.harris at gmail.com  Fri Nov 23 12:55:59 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 23 Nov 2012 10:55:59 -0700
Subject: [Numpy-discussion] When are 0-d arrays writeable?
In-Reply-To: <1353682399.15250.11.camel@sebastian-laptop>
References: <CAB6mnx+s4YuE1Q5er64R9j8NO9vgek6jQ6_yK60=Gc+B1XnsCw@mail.gmail.com>
	<CAPJVwB=S5sEF-=c_N4SMR7+XJZ-i_yji7cW+tSUcLfKoE-jL+w@mail.gmail.com>
	<1353682399.15250.11.camel@sebastian-laptop>
Message-ID: <CAB6mnxJdEG-SA_H5_uv+hMhkhnXwK_qCPuPNy-wegjv4wkE4sA@mail.gmail.com>

On Fri, Nov 23, 2012 at 7:53 AM, Sebastian Berg
<sebastian at sipsolutions.net>wrote:

> On Fri, 2012-11-23 at 10:49 +0000, Nathaniel Smith wrote:
> > On 23 Nov 2012 03:34, "Charles R Harris" <charlesr.harris at gmail.com>
> > wrote:
> > >
> > > Examples,
> > >
> > > In [13]: ones(()).flags.writeable
> > > Out[13]: True
> > >
> > > In [14]: (-ones(())).flags.writeable
> > > Out[14]: False
> > >
> > > In [15]: (-1*ones(())).flags.writeable
> > > Out[15]: False
> > >
> > > In [16]: (1 + ones(())).flags.writeable
> > > Out[16]: False
> > >
> > > In [17]: array(1)
> > > Out[17]: array(1)
> > >
> > > In [18]: array(1).shape
> > > Out[18]: ()
> > >
> > > In [19]: array(1).flags.writeable
> > > Out[19]: True
> >
> > Looks like a bug in the ufunc output value setup code or something?
> >
> It might be possible to rethink when (or if) to convert 0-d array to a
> scalar. However, at the moment as far as I understand many functions
> generally do not return 0-d arrays but scalars. Which makes sense
> because mostly we would rather use scalars then 0-d arrays as they are
> closer to typical python (hashable and subclasses of python types).
>
> Of course the way this is done is not aware of what is put in (scalar
> vs. 0-d array), since all input is converted to an array normally, which
> means that most (all?) functions either return 0-d arrays or scalars and
> are never aware if the original input was a scalar or an array. Maybe
> there could be a np.asarray_or_scalar or such so that its easier to give
> the same output type as the original input type?
>
>
Yes, that is what looks to be the case. It breaks the scalar + array ->
array rule that applies to higher dimensional arrays. I'm not sure what
should be done for this corner case, especially as the code in question is
rather involved...

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121123/4f35eb3f/attachment.html>

From charlesr.harris at gmail.com  Fri Nov 23 13:01:13 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 23 Nov 2012 11:01:13 -0700
Subject: [Numpy-discussion] Unexpected RuntimeWarning
In-Reply-To: <50AF6031.2010503@cam.ac.uk>
References: <50AF6031.2010503@cam.ac.uk>
Message-ID: <CAB6mnxKX3LLb_5hpAgPQ2+zSiF3TMAmJ_yCakpa157Mw34_NSA@mail.gmail.com>

On Fri, Nov 23, 2012 at 4:38 AM, Bob Dowling <rjd4+numpy at cam.ac.uk> wrote:

> I have a simple function defined in the following snippet:
>
>   --- start ---
> import numpy
>
> def chebyshev(x, m):
>      '''Calculates Chebyshev functions of the first kind using the
> trigonometric identities.'''
>
>      theta = numpy.where(
>          numpy.abs(x)<=1.0,
>          numpy.arccos(x),
>          numpy.arccosh(numpy.abs(x))
>          )
>
>      y = numpy.where(
>          numpy.abs(x)<=1.0,
>          numpy.cos(m*theta),
>          numpy.cosh(m*theta) * numpy.where(
>              x > 0.0,
>              1.0,
>              -1.0
>              )**m
>          )
>
>      return y
>
>
> if __name__ == '__main__':
>      x = numpy.linspace(-2.0, 2.0, 21)
>      y = chebyshev(x,3)
>      print(y)
>
>   --- end ---
>
>
> I'm using the numpy.where() call to extract only legal values for the
> circular and hyperbolic trigonometric functions.  But I still get
> warnings that I'm passing invalid anrguments:
>
> --- start---
>
> $ python3 demo.py
> demo.py:8: RuntimeWarning: invalid value encountered in arccos
>    numpy.arccos(x),
> demo.py:9: RuntimeWarning: invalid value encountered in arccosh
>    numpy.arccosh(numpy.abs(x))
> [ -2.60000000e+01  -1.79280000e+01  -1.15840000e+01  -6.77600000e+00
>    -3.31200000e+00  -1.00000000e+00   3.52000000e-01   9.36000000e-01
>     9.44000000e-01   5.68000000e-01  -1.83697020e-16  -5.68000000e-01
>    -9.44000000e-01  -9.36000000e-01  -3.52000000e-01   1.00000000e+00
>     3.31200000e+00   6.77600000e+00   1.15840000e+01   1.79280000e+01
>     2.60000000e+01]
>
>   --- end ---
>
> (I get the same with Python 2 so don't get excited about that.)
>
>
> I'm guessing that numpy.where() is evaluating the complete arccos and
> arccosh arrays and therefore getting invalid arguments.
>
> Now, I can turn off the warnings with numpy.seterr(invalid='ignore') but
> that's not what I would regard as good practice.
>
> Is there a "numpythonic" way to address the issue?
>

Are you aware of the Chebyshev series in numpy ?

In [1]: from numpy.polynomial import Chebyshev as T

In [2]: p = T([1,2,3])

In [3]: p(linspace(-2,2, 10))
Out[3]:
array([ 18.        ,   9.40740741,   3.18518519,  -0.66666667,
        -2.14814815,  -1.25925926,   2.        ,   7.62962963,
        15.62962963,  26.        ])




> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121123/7af101e8/attachment.html>

From chris.barker at noaa.gov  Fri Nov 23 14:00:35 2012
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Fri, 23 Nov 2012 11:00:35 -0800
Subject: [Numpy-discussion] the difference between "+" and np.add?
In-Reply-To: <50AE34B7.3070401@continuum.io>
References: <CAAN-aREVeYj8J0KOYKfv7fr=UttsxcSXQ=dKoabWLBgT-aGk7w@mail.gmail.com>
	<50AE34B7.3070401@continuum.io>
Message-ID: <CALGmxEL-J5rOo9-YwQuRvXO+7R2AOUX56R_y45gfbTK=ZoRP4w@mail.gmail.com>

On Thu, Nov 22, 2012 at 6:20 AM, Francesc Alted <francesc at continuum.io> wrote:
> As Nathaniel said, there is not a difference in terms of *what* is
> computed.  However, the methods that you suggested actually differ on
> *how* they are computed, and that has dramatic effects on the time
> used.  For example:
>
> In []: arr1, arr2, arr3, arr4, arr5 = [np.arange(1e7) for x in range(5)]
>
> In []: %time arr1 + arr2 + arr3 + arr4 + arr5
> CPU times: user 0.05 s, sys: 0.10 s, total: 0.14 s
> Wall time: 0.15 s

> There are also ways to minimize the size of temporaries, and numexpr is
> one of the simplests:

but you can also use np.add (and friends) to reduce the number of
temporaries. It can make a difference:

In [11]: def add_5_arrays(arr1, arr2, arr3, arr4, arr5):
   ....:     result = arr1 + arr2
   ....:     np.add(result, arr3, out=result)
   ....:     np.add(result, arr4, out=result)
   ....:     np.add(result, arr5, out=result)

In [13]: timeit arr1 + arr2 + arr3 + arr4 + arr5
1 loops, best of 3: 528 ms per loop

In [17]: timeit add_5_arrays(arr1, arr2, arr3, arr4, arr5)
1 loops, best of 3: 293 ms per loop

(don't have numexpr on this machine for a comparison)

NOTE: no point in going through all this unless this operation is
really a bottleneck in your code -- profile, profile, profile!

-Chris

PS: you can put a loop in the function to make it more generic:

In [18]: def add_n_arrays(*args):
   ....:     result = args[0] + args[1]
   ....:     for arr in args[2:]:
   ....:         np.add(result, arr, result)
   ....:     return result

In [21]: timeit add_n_arrays(arr1, arr2, arr3, arr4, arr5)
1 loops, best of 3: 317 ms per loop




> In []: import numexpr as ne
>
> In []: %time ne.evaluate('arr1 + arr2 + arr3 + arr4 + arr5')
> CPU times: user 0.04 s, sys: 0.04 s, total: 0.08 s
> Wall time: 0.04 s
> Out[]:
> array([  0.00000000e+00,   5.00000000e+00,   1.00000000e+01, ...,
>           4.99999850e+07,   4.99999900e+07,   4.99999950e+07])
>
> Again, the computations are the same, but how you manage memory is critical.
>
> --
> Francesc Alted
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From gamblin2 at llnl.gov  Sat Nov 24 14:34:18 2012
From: gamblin2 at llnl.gov (Gamblin, Todd)
Date: Sat, 24 Nov 2012 19:34:18 +0000
Subject: [Numpy-discussion] Hierarchical vs non-hierarchical ndarray.base
 and __array_interface__
Message-ID: <2B413C1254F1B44F908C9BE85A5DA0AD21B7FA@PRDEXMBX-04.the-lab.llnl.gov>

Hi all,

I posted on the change in semantics of ndarray.base here:

	https://github.com/numpy/numpy/commit/6c0ad59#commitcomment-2153047

And some folks asked me to post my question to the numpy mailing list.  I've implemented a tool for mapping processes in parallel applications to nodes in cartesian networks.  It uses hierarchies of numpy arrays to represent the domain decomposition of the application, as well as corresponding groups of processes on the network.  You can "map" an application to the network using assignment of through views.  The tool is here if anyone is curious:  https://github.com/tgamblin/rubik.  I used numpy to implement this because I wanted to be able to do mappings for arbitrary-dimensional networks.  Blue Gene/Q, for example, has a 5-D network.

The reason I bring this up is because I rely on the ndarray.base pointer and some of the semantics in __array_interface__ to translate indices within my hierarchy of views.  e.g., if a value is at (0,0) in a view I want to know that it's actually at (4,4) in its immediate parent array.

After looking over the commit I linked to above, I realized I'm actually relying on a lot of stuff that's not guaranteed by numpy.  I rely on .base pointing to its closest parent, and I rely on __array_interface__.data containing the address of the array's memory and its strides.  None of these is guaranteed by the API docs:

	http://docs.scipy.org/doc/numpy/reference/arrays.interface.html

So I guess I have a few questions:

1. Is translating indices between base arrays and views something that would be useful to other people?

2. Is there some better way to do this than using ndarray.base and __array_interface__?

3. What's the numpy philosophy on this?  Should views know about their parents or not?  They obviously have to know a little bit about their memory, but whether or not they know how they were derived from their owning array is a different question.  There was some discussion on the vagueness of .base here:

	http://thread.gmane.org/gmane.comp.python.numeric.general/51688/focus=51703

But it doesn't look like you're deprecating .base in 1.7, only changing its behavior, which I tend to agree is worse than deprecating it.

After thinking about all this, I'm not sure what I would like to happen.  I can see the value of not keeping extra references around within numpy, and my domain is pretty different from the ways that I imagine people use numpy.  I wouldn't have to change my code much to make it work without .base, but I do rely on __array_interface__.  If that doesn't include the address and strides, t think I'm screwed as far as translating indices go.

Any suggestions?

Thanks!
-Todd



______________________________________________________________________
Todd Gamblin, tgamblin at llnl.gov, http://people.llnl.gov/gamblin2
CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA



From gamblin2 at llnl.gov  Sat Nov 24 15:03:06 2012
From: gamblin2 at llnl.gov (Gamblin, Todd)
Date: Sat, 24 Nov 2012 20:03:06 +0000
Subject: [Numpy-discussion] Z-ordering (Morton ordering) for numpy
Message-ID: <2B413C1254F1B44F908C9BE85A5DA0AD21C086@PRDEXMBX-04.the-lab.llnl.gov>

Hi all,

In the course of developing a network mapping tool I'm working on, I also developed some python code to do arbitrary-dimensional z-order (morton order) for ndarrays.  The code is here:

	https://github.com/tgamblin/rubik/blob/master/rubik/zorder.py

There is a function to put the elements of an array in Z order, and another one to enumerate an array's elements in Z order.  There is also a ZEncoder class that can generate Z-codes for arbitrary dimensions and bit widths.

I figure this is something that would be generally useful.  Any interest in having this in numpy?  If so, what should the interface look like and can you point me to a good spot in the code to add it?

I was thinking it might make sense to have a Z-order iterator for ndarrays, kind of like ndarray.flat.  i.e.:

	arr = np.empty([4,4], dtype=int)
	arr.flat = range(arr.size)
	for elt in arr.zorder:
		print elt,
	0 4 1 5 8 12 9 13 2 6 3 7 10 14 11 15

Or an equivalent to ndindex:

	arr = np.empty(4,4, dtype=int)
	arr.flat = range(arr.size)
	for ix in np.zindex(arr.shape):
		print ix,
	(0, 0) (1, 0) (0, 1) (1, 1) (2, 0) (3, 0) (2, 1) (3, 1) (0, 2) (1, 2) (0, 3) (1, 3) (2, 2) (3, 2) (2, 3) (3, 3)

Thoughts?

-Todd
______________________________________________________________________
Todd Gamblin, tgamblin at llnl.gov, http://people.llnl.gov/gamblin2
CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA



From travis at continuum.io  Sat Nov 24 15:10:48 2012
From: travis at continuum.io (Travis Oliphant)
Date: Sat, 24 Nov 2012 13:10:48 -0700
Subject: [Numpy-discussion] Z-ordering (Morton ordering) for numpy
In-Reply-To: <2B413C1254F1B44F908C9BE85A5DA0AD21C086@PRDEXMBX-04.the-lab.llnl.gov>
References: <2B413C1254F1B44F908C9BE85A5DA0AD21C086@PRDEXMBX-04.the-lab.llnl.gov>
Message-ID: <85D6E672-825B-4DA7-9A7D-68232015B4B8@continuum.io>

This is pretty cool.    Something like this would be interesting to play with.  There are some algorithms that are faster with z-order arrays.    The code is simple enough and small enough that I could see putting it in NumPy.   What do others think?

-Travis



On Nov 24, 2012, at 1:03 PM, Gamblin, Todd wrote:

> Hi all,
> 
> In the course of developing a network mapping tool I'm working on, I also developed some python code to do arbitrary-dimensional z-order (morton order) for ndarrays.  The code is here:
> 
> 	https://github.com/tgamblin/rubik/blob/master/rubik/zorder.py
> 
> There is a function to put the elements of an array in Z order, and another one to enumerate an array's elements in Z order.  There is also a ZEncoder class that can generate Z-codes for arbitrary dimensions and bit widths.
> 
> I figure this is something that would be generally useful.  Any interest in having this in numpy?  If so, what should the interface look like and can you point me to a good spot in the code to add it?
> 
> I was thinking it might make sense to have a Z-order iterator for ndarrays, kind of like ndarray.flat.  i.e.:
> 
> 	arr = np.empty([4,4], dtype=int)
> 	arr.flat = range(arr.size)
> 	for elt in arr.zorder:
> 		print elt,
> 	0 4 1 5 8 12 9 13 2 6 3 7 10 14 11 15
> 
> Or an equivalent to ndindex:
> 
> 	arr = np.empty(4,4, dtype=int)
> 	arr.flat = range(arr.size)
> 	for ix in np.zindex(arr.shape):
> 		print ix,
> 	(0, 0) (1, 0) (0, 1) (1, 1) (2, 0) (3, 0) (2, 1) (3, 1) (0, 2) (1, 2) (0, 3) (1, 3) (2, 2) (3, 2) (2, 3) (3, 3)
> 
> Thoughts?
> 
> -Todd
> ______________________________________________________________________
> Todd Gamblin, tgamblin at llnl.gov, http://people.llnl.gov/gamblin2
> CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From aron at ahmadia.net  Sat Nov 24 15:17:17 2012
From: aron at ahmadia.net (Aron Ahmadia)
Date: Sat, 24 Nov 2012 20:17:17 +0000
Subject: [Numpy-discussion] Z-ordering (Morton ordering) for numpy
In-Reply-To: <85D6E672-825B-4DA7-9A7D-68232015B4B8@continuum.io>
References: <2B413C1254F1B44F908C9BE85A5DA0AD21C086@PRDEXMBX-04.the-lab.llnl.gov>
	<85D6E672-825B-4DA7-9A7D-68232015B4B8@continuum.io>
Message-ID: <CAPhiW4gLKDup2QkxCvi4Z06eb3z7v+aVkjDPraLdh+QdKwddmg@mail.gmail.com>

Todd,

I am optimistic and I think it would be a good idea to put this in.  A
couple previous studies [1] haven't found any useful speedups from in-core
applications for Morton-order, and if you have results for real scientific
applications using numpy this would not only be great, but the resulting
paper would have quite a bit of impact.  I'm sure you're already connected
to the right people at LLNL, but I can think of a couple other projects
which might be interested in trying this sort of thing out.

http://www.cs.utexas.edu/~pingali/CS395T/2012sp/papers/co.pdf

Cheers,
Aron


On Sat, Nov 24, 2012 at 8:10 PM, Travis Oliphant <travis at continuum.io>wrote:

> This is pretty cool.    Something like this would be interesting to play
> with.  There are some algorithms that are faster with z-order arrays.
>  The code is simple enough and small enough that I could see putting it in
> NumPy.   What do others think?
>
> -Travis
>
>
>
> On Nov 24, 2012, at 1:03 PM, Gamblin, Todd wrote:
>
> > Hi all,
> >
> > In the course of developing a network mapping tool I'm working on, I
> also developed some python code to do arbitrary-dimensional z-order (morton
> order) for ndarrays.  The code is here:
> >
> >       https://github.com/tgamblin/rubik/blob/master/rubik/zorder.py
> >
> > There is a function to put the elements of an array in Z order, and
> another one to enumerate an array's elements in Z order.  There is also a
> ZEncoder class that can generate Z-codes for arbitrary dimensions and bit
> widths.
> >
> > I figure this is something that would be generally useful.  Any interest
> in having this in numpy?  If so, what should the interface look like and
> can you point me to a good spot in the code to add it?
> >
> > I was thinking it might make sense to have a Z-order iterator for
> ndarrays, kind of like ndarray.flat.  i.e.:
> >
> >       arr = np.empty([4,4], dtype=int)
> >       arr.flat = range(arr.size)
> >       for elt in arr.zorder:
> >               print elt,
> >       0 4 1 5 8 12 9 13 2 6 3 7 10 14 11 15
> >
> > Or an equivalent to ndindex:
> >
> >       arr = np.empty(4,4, dtype=int)
> >       arr.flat = range(arr.size)
> >       for ix in np.zindex(arr.shape):
> >               print ix,
> >       (0, 0) (1, 0) (0, 1) (1, 1) (2, 0) (3, 0) (2, 1) (3, 1) (0, 2) (1,
> 2) (0, 3) (1, 3) (2, 2) (3, 2) (2, 3) (3, 3)
> >
> > Thoughts?
> >
> > -Todd
> > ______________________________________________________________________
> > Todd Gamblin, tgamblin at llnl.gov, http://people.llnl.gov/gamblin2
> > CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121124/ff7e9a02/attachment.html>

From gamblin2 at llnl.gov  Sat Nov 24 15:30:54 2012
From: gamblin2 at llnl.gov (Gamblin, Todd)
Date: Sat, 24 Nov 2012 20:30:54 +0000
Subject: [Numpy-discussion] Z-ordering (Morton ordering) for numpy
In-Reply-To: <CAPhiW4gLKDup2QkxCvi4Z06eb3z7v+aVkjDPraLdh+QdKwddmg@mail.gmail.com>
References: <2B413C1254F1B44F908C9BE85A5DA0AD21C086@PRDEXMBX-04.the-lab.llnl.gov>
	<85D6E672-825B-4DA7-9A7D-68232015B4B8@continuum.io>
	<CAPhiW4gLKDup2QkxCvi4Z06eb3z7v+aVkjDPraLdh+QdKwddmg@mail.gmail.com>
Message-ID: <2B413C1254F1B44F908C9BE85A5DA0AD21C9F7@PRDEXMBX-04.the-lab.llnl.gov>

So, just FYI, my usage of this is for Rubik, where it's a communication latency optimization for the code being mapped to the network.  I haven't tested it as an optimization for particular in-core algorithms.  However, there was some work on this at LLNL maybe a couple years ago -- I think it was for the solvers.  I'll ask around for an example and/or a paper, or maybe Travis has examples.

Just from an ease-of-use point of view, though, if you make it simple to do zordering, you might see more people using it :).  That's why I wanted to get this into numpy.

This brings up another point.  This is pure python, so it won't be super-fast.  What's the typical way things are integrated and optimized in numpy?  Do you contribute something like this in python first then convert to cython/C as necessary?  Or would you want it in C to begin with?

-Todd


On Nov 24, 2012, at 12:17 PM, Aron Ahmadia <aron at ahmadia.net>
 wrote:

> Todd,
> 
> I am optimistic and I think it would be a good idea to put this in.  A couple previous studies [1] haven't found any useful speedups from in-core applications for Morton-order, and if you have results for real scientific applications using numpy this would not only be great, but the resulting paper would have quite a bit of impact.  I'm sure you're already connected to the right people at LLNL, but I can think of a couple other projects which might be interested in trying this sort of thing out.
> 
> http://www.cs.utexas.edu/~pingali/CS395T/2012sp/papers/co.pdf
> 
> Cheers,
> Aron
> 
> 
> On Sat, Nov 24, 2012 at 8:10 PM, Travis Oliphant <travis at continuum.io> wrote:
> This is pretty cool.    Something like this would be interesting to play with.  There are some algorithms that are faster with z-order arrays.    The code is simple enough and small enough that I could see putting it in NumPy.   What do others think?
> 
> -Travis
> 
> 
> 
> On Nov 24, 2012, at 1:03 PM, Gamblin, Todd wrote:
> 
> > Hi all,
> >
> > In the course of developing a network mapping tool I'm working on, I also developed some python code to do arbitrary-dimensional z-order (morton order) for ndarrays.  The code is here:
> >
> >       https://github.com/tgamblin/rubik/blob/master/rubik/zorder.py
> >
> > There is a function to put the elements of an array in Z order, and another one to enumerate an array's elements in Z order.  There is also a ZEncoder class that can generate Z-codes for arbitrary dimensions and bit widths.
> >
> > I figure this is something that would be generally useful.  Any interest in having this in numpy?  If so, what should the interface look like and can you point me to a good spot in the code to add it?
> >
> > I was thinking it might make sense to have a Z-order iterator for ndarrays, kind of like ndarray.flat.  i.e.:
> >
> >       arr = np.empty([4,4], dtype=int)
> >       arr.flat = range(arr.size)
> >       for elt in arr.zorder:
> >               print elt,
> >       0 4 1 5 8 12 9 13 2 6 3 7 10 14 11 15
> >
> > Or an equivalent to ndindex:
> >
> >       arr = np.empty(4,4, dtype=int)
> >       arr.flat = range(arr.size)
> >       for ix in np.zindex(arr.shape):
> >               print ix,
> >       (0, 0) (1, 0) (0, 1) (1, 1) (2, 0) (3, 0) (2, 1) (3, 1) (0, 2) (1, 2) (0, 3) (1, 3) (2, 2) (3, 2) (2, 3) (3, 3)
> >
> > Thoughts?
> >
> > -Todd
> > ______________________________________________________________________
> > Todd Gamblin, tgamblin at llnl.gov, http://people.llnl.gov/gamblin2
> > CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

______________________________________________________________________
Todd Gamblin, tgamblin at llnl.gov, http://people.llnl.gov/gamblin2
CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA



From sgonzi at staffmail.ed.ac.uk  Sat Nov 24 15:36:45 2012
From: sgonzi at staffmail.ed.ac.uk (Siegfried Gonzi)
Date: Sat, 24 Nov 2012 20:36:45 +0000
Subject: [Numpy-discussion] numpy where function on different sized arrays
Message-ID: <D5167012-3E99-473A-BA50-9876F961240D@staffmail.ed.ac.uk>

Hi all
 
This must have been answered in the past but my google search capabilities are not the best.
 
Given an array A say of dimension 40x60 and given another array/vector B of dimension 20 (the values in B occur only once).
 
What I would like to do is the following which of course does not work (by the way doesn't work in IDL either):
 
indx=where(A == B)
 
I understand A and B are both of different dimensions. So my question: what would the fastest or proper way to accomplish this (I found a solution but think is rather awkward and not very scipy/numpy-tonic tough).
 
Thanks
-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



From sgonzi at staffmail.ed.ac.uk  Sat Nov 24 15:51:07 2012
From: sgonzi at staffmail.ed.ac.uk (Siegfried Gonzi)
Date: Sat, 24 Nov 2012 20:51:07 +0000
Subject: [Numpy-discussion] numpy where function on different size
Message-ID: <A8850854-01C9-4C12-853E-8541AC7686BC@staffmail.ed.ac.uk>

> Message: 6
> Date: Sat, 24 Nov 2012 20:36:45 +0000
> From: Siegfried Gonzi <sgonzi at staffmail.ed.ac.uk>
> Subject: [Numpy-discussion] numpy where function on different size
> Hi all
>This must have been answered in the past but my google search 
> capabilities are not the best.
>Given an array A say of dimension 40x60 and given another array/vector B 
> of dimension 20 (the values in B occur only once).
>What I would like to do is the following which of course does not work 
> (by the way doesn't work in IDL either):
>indx=where(A == B)
>I understand A and B are both of different dimensions. So my question: 
> what would the fastest or proper way to accomplish this (I found a 
> solution but think is rather awkward and not very scipy/numpy-tonic 
> tough).


I should clarify: where(A==B, C, A) 

C is of equal dimension than B. Basically: everywhere where A equals e.g. B[0] replace by C[0]; or where B[4] equals A replace very occurence in A by C[4].


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



From d.warde.farley at gmail.com  Sat Nov 24 16:19:27 2012
From: d.warde.farley at gmail.com (David Warde-Farley)
Date: Sat, 24 Nov 2012 16:19:27 -0500
Subject: [Numpy-discussion] numpy where function on different sized
	arrays
In-Reply-To: <D5167012-3E99-473A-BA50-9876F961240D@staffmail.ed.ac.uk>
References: <D5167012-3E99-473A-BA50-9876F961240D@staffmail.ed.ac.uk>
Message-ID: <CALc6Xo6Q2i6gJAPM6Bc7j=NtJ51upy+RrVgxh25yJ9EvZ80+Qw@mail.gmail.com>

M = A[..., np.newaxis] == B

will give you a 40x60x20 boolean 3d-array where M[..., i] gives you a
boolean mask for all the occurrences of B[i] in A.

If you wanted all the (i, j) pairs for each value in B, you could do
something like

import numpy as np
from itertools import izip, groupby
from operator import itemgetter

id1, id2, id3 = np.where(A[..., np.newaxis] == B)
order = np.argsort(id3)
triples_iter = izip(id3[order], id1[order], id2[order])
grouped = groupby(triples_iter, itemgetter(0))
d = dict((b_value, [idx[1:] for idx in indices]) for b_value, indices in
grouped)

Then d[value] is a list of all the (i, j) pairs where A[i, j] == value, and
the keys of d are every value in B.



On Sat, Nov 24, 2012 at 3:36 PM, Siegfried Gonzi
<sgonzi at staffmail.ed.ac.uk>wrote:

> Hi all
>
> This must have been answered in the past but my google search capabilities
> are not the best.
>
> Given an array A say of dimension 40x60 and given another array/vector B
> of dimension 20 (the values in B occur only once).
>
> What I would like to do is the following which of course does not work (by
> the way doesn't work in IDL either):
>
> indx=where(A == B)
>
> I understand A and B are both of different dimensions. So my question:
> what would the fastest or proper way to accomplish this (I found a solution
> but think is rather awkward and not very scipy/numpy-tonic tough).
>
> Thanks
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121124/c6678a26/attachment.html>

From davidmenhur at gmail.com  Sat Nov 24 17:23:36 2012
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Sat, 24 Nov 2012 23:23:36 +0100
Subject: [Numpy-discussion] numpy where function on different sized
	arrays
In-Reply-To: <CALc6Xo6Q2i6gJAPM6Bc7j=NtJ51upy+RrVgxh25yJ9EvZ80+Qw@mail.gmail.com>
References: <D5167012-3E99-473A-BA50-9876F961240D@staffmail.ed.ac.uk>
	<CALc6Xo6Q2i6gJAPM6Bc7j=NtJ51upy+RrVgxh25yJ9EvZ80+Qw@mail.gmail.com>
Message-ID: <CAJhcF=2nmVeeQUo5Q45OUcQK+7KnQ_u0tvzUyOSXGpQbLSUy+Q@mail.gmail.com>

A pure Python approach could be:

for i, x in enumerate(a):
	for j, y in enumerate(x):
		if y in b:
			idx.append((i,j))

Of course, it is slow if the arrays are large, but it is very
readable, and probably very fast if cythonised.


David.

On Sat, Nov 24, 2012 at 10:19 PM, David Warde-Farley
<d.warde.farley at gmail.com> wrote:
> M = A[..., np.newaxis] == B
>
> will give you a 40x60x20 boolean 3d-array where M[..., i] gives you a
> boolean mask for all the occurrences of B[i] in A.
>
> If you wanted all the (i, j) pairs for each value in B, you could do
> something like
>
> import numpy as np
> from itertools import izip, groupby
> from operator import itemgetter
>
> id1, id2, id3 = np.where(A[..., np.newaxis] == B)
> order = np.argsort(id3)
> triples_iter = izip(id3[order], id1[order], id2[order])
> grouped = groupby(triples_iter, itemgetter(0))
> d = dict((b_value, [idx[1:] for idx in indices]) for b_value, indices in
> grouped)
>
> Then d[value] is a list of all the (i, j) pairs where A[i, j] == value, and
> the keys of d are every value in B.
>
>
>
> On Sat, Nov 24, 2012 at 3:36 PM, Siegfried Gonzi <sgonzi at staffmail.ed.ac.uk>
> wrote:
>>
>> Hi all
>>
>> This must have been answered in the past but my google search capabilities
>> are not the best.
>>
>> Given an array A say of dimension 40x60 and given another array/vector B
>> of dimension 20 (the values in B occur only once).
>>
>> What I would like to do is the following which of course does not work (by
>> the way doesn't work in IDL either):
>>
>> indx=where(A == B)
>>
>> I understand A and B are both of different dimensions. So my question:
>> what would the fastest or proper way to accomplish this (I found a solution
>> but think is rather awkward and not very scipy/numpy-tonic tough).
>>
>> Thanks
>> --
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From charlesr.harris at gmail.com  Sat Nov 24 18:04:53 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 24 Nov 2012 16:04:53 -0700
Subject: [Numpy-discussion] Z-ordering (Morton ordering) for numpy
In-Reply-To: <2B413C1254F1B44F908C9BE85A5DA0AD21C9F7@PRDEXMBX-04.the-lab.llnl.gov>
References: <2B413C1254F1B44F908C9BE85A5DA0AD21C086@PRDEXMBX-04.the-lab.llnl.gov>
	<85D6E672-825B-4DA7-9A7D-68232015B4B8@continuum.io>
	<CAPhiW4gLKDup2QkxCvi4Z06eb3z7v+aVkjDPraLdh+QdKwddmg@mail.gmail.com>
	<2B413C1254F1B44F908C9BE85A5DA0AD21C9F7@PRDEXMBX-04.the-lab.llnl.gov>
Message-ID: <CAB6mnx+5t_QPy9jpTeoyNbJcQUsF7eu=ZYBNjHOjZpjHwnhFzw@mail.gmail.com>

On Sat, Nov 24, 2012 at 1:30 PM, Gamblin, Todd <gamblin2 at llnl.gov> wrote:

> So, just FYI, my usage of this is for Rubik, where it's a communication
> latency optimization for the code being mapped to the network.  I haven't
> tested it as an optimization for particular in-core algorithms.  However,
> there was some work on this at LLNL maybe a couple years ago -- I think it
> was for the solvers.  I'll ask around for an example and/or a paper, or
> maybe Travis has examples.
>
> Just from an ease-of-use point of view, though, if you make it simple to
> do zordering, you might see more people using it :).  That's why I wanted
> to get this into numpy.
>
> This brings up another point.  This is pure python, so it won't be
> super-fast.  What's the typical way things are integrated and optimized in
> numpy?  Do you contribute something like this in python first then convert
> to cython/C as necessary?  Or would you want it in C to begin with?
>
>
I think Python is a good start as it allows easy modification and can be
converted to Cython later on. OTOH, if the new function looks settled from
the get go, Cython is an option.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121124/41919f1b/attachment.html>

From d.warde.farley at gmail.com  Sat Nov 24 19:08:40 2012
From: d.warde.farley at gmail.com (David Warde-Farley)
Date: Sat, 24 Nov 2012 19:08:40 -0500
Subject: [Numpy-discussion] numpy where function on different sized
	arrays
In-Reply-To: <CAJhcF=2nmVeeQUo5Q45OUcQK+7KnQ_u0tvzUyOSXGpQbLSUy+Q@mail.gmail.com>
References: <D5167012-3E99-473A-BA50-9876F961240D@staffmail.ed.ac.uk>
	<CALc6Xo6Q2i6gJAPM6Bc7j=NtJ51upy+RrVgxh25yJ9EvZ80+Qw@mail.gmail.com>
	<CAJhcF=2nmVeeQUo5Q45OUcQK+7KnQ_u0tvzUyOSXGpQbLSUy+Q@mail.gmail.com>
Message-ID: <CALc6Xo5C2vDJTJ+zSrem6cps9MRcnW6hSO-P0Op4P4sfhsNZDw@mail.gmail.com>

I think that would lose information as to which value in B was at each
position. I think you want:



On Sat, Nov 24, 2012 at 5:23 PM, Da?id <davidmenhur at gmail.com> wrote:

> A pure Python approach could be:
>
> for i, x in enumerate(a):
>         for j, y in enumerate(x):
>                 if y in b:
>                         idx.append((i,j))
>
> Of course, it is slow if the arrays are large, but it is very
> readable, and probably very fast if cythonised.
>
>
> David.
>
> On Sat, Nov 24, 2012 at 10:19 PM, David Warde-Farley
> <d.warde.farley at gmail.com> wrote:
> > M = A[..., np.newaxis] == B
> >
> > will give you a 40x60x20 boolean 3d-array where M[..., i] gives you a
> > boolean mask for all the occurrences of B[i] in A.
> >
> > If you wanted all the (i, j) pairs for each value in B, you could do
> > something like
> >
> > import numpy as np
> > from itertools import izip, groupby
> > from operator import itemgetter
> >
> > id1, id2, id3 = np.where(A[..., np.newaxis] == B)
> > order = np.argsort(id3)
> > triples_iter = izip(id3[order], id1[order], id2[order])
> > grouped = groupby(triples_iter, itemgetter(0))
> > d = dict((b_value, [idx[1:] for idx in indices]) for b_value, indices in
> > grouped)
> >
> > Then d[value] is a list of all the (i, j) pairs where A[i, j] == value,
> and
> > the keys of d are every value in B.
> >
> >
> >
> > On Sat, Nov 24, 2012 at 3:36 PM, Siegfried Gonzi <
> sgonzi at staffmail.ed.ac.uk>
> > wrote:
> >>
> >> Hi all
> >>
> >> This must have been answered in the past but my google search
> capabilities
> >> are not the best.
> >>
> >> Given an array A say of dimension 40x60 and given another array/vector B
> >> of dimension 20 (the values in B occur only once).
> >>
> >> What I would like to do is the following which of course does not work
> (by
> >> the way doesn't work in IDL either):
> >>
> >> indx=where(A == B)
> >>
> >> I understand A and B are both of different dimensions. So my question:
> >> what would the fastest or proper way to accomplish this (I found a
> solution
> >> but think is rather awkward and not very scipy/numpy-tonic tough).
> >>
> >> Thanks
> >> --
> >> The University of Edinburgh is a charitable body, registered in
> >> Scotland, with registration number SC005336.
> >>
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121124/48c51716/attachment.html>

From d.warde.farley at gmail.com  Sat Nov 24 19:34:17 2012
From: d.warde.farley at gmail.com (David Warde-Farley)
Date: Sat, 24 Nov 2012 19:34:17 -0500
Subject: [Numpy-discussion] numpy where function on different sized
	arrays
In-Reply-To: <CALc6Xo5C2vDJTJ+zSrem6cps9MRcnW6hSO-P0Op4P4sfhsNZDw@mail.gmail.com>
References: <D5167012-3E99-473A-BA50-9876F961240D@staffmail.ed.ac.uk>
	<CALc6Xo6Q2i6gJAPM6Bc7j=NtJ51upy+RrVgxh25yJ9EvZ80+Qw@mail.gmail.com>
	<CAJhcF=2nmVeeQUo5Q45OUcQK+7KnQ_u0tvzUyOSXGpQbLSUy+Q@mail.gmail.com>
	<CALc6Xo5C2vDJTJ+zSrem6cps9MRcnW6hSO-P0Op4P4sfhsNZDw@mail.gmail.com>
Message-ID: <CALc6Xo5cPLFV8RLNQy+qjF6OSPLUb-fmWaTJsd+gjy8nfyxdYw@mail.gmail.com>

On Sat, Nov 24, 2012 at 7:08 PM, David Warde-Farley <
d.warde.farley at gmail.com> wrote:

> I think that would lose information as to which value in B was at each
> position. I think you want:
>
>
(premature send, stupid Gmail...)

idx = {}
for i, x in enumerate(a):
        for j, y in enumerate(x):
                if y in B:
                        idx.setdefault(y, []).append((i,j))

On the problem size the OP specified, this is about 4x slower than the
NumPy version I posted above. However with a small modification:

idx = {}
set_b = set(B)  # makes 'if y in B' lookups much faster
for i, x in enumerate(a):
        for j, y in enumerate(x):
                if y in set_b:
                        idx.setdefault(y, []).append((i,j))


It actually beats my solution. With inputs: np.random.seed(0); A =
np.random.random_integers(40, 59, size=(40, 60)); B = np.arange(40, 60)

In [115]: timeit foo_py_orig(A, B)
100 loops, best of 3: 16.5 ms per loop

In [116]: timeit foo_py(A, B)
100 loops, best of 3: 2.5 ms per loop

In [117]: timeit foo_numpy(A, B)
100 loops, best of 3: 4.15 ms per loop

Depending on the specifics of the inputs, a collections.DefaultDict could
also help things.


> On Sat, Nov 24, 2012 at 5:23 PM, Da?id <davidmenhur at gmail.com> wrote:
>
>> A pure Python approach could be:
>>
>> for i, x in enumerate(a):
>>         for j, y in enumerate(x):
>>                 if y in b:
>>                         idx.append((i,j))
>>
>> Of course, it is slow if the arrays are large, but it is very
>> readable, and probably very fast if cythonised.
>>
>>
>> David.
>>
>> On Sat, Nov 24, 2012 at 10:19 PM, David Warde-Farley
>> <d.warde.farley at gmail.com> wrote:
>> > M = A[..., np.newaxis] == B
>> >
>> > will give you a 40x60x20 boolean 3d-array where M[..., i] gives you a
>> > boolean mask for all the occurrences of B[i] in A.
>> >
>> > If you wanted all the (i, j) pairs for each value in B, you could do
>> > something like
>> >
>> > import numpy as np
>> > from itertools import izip, groupby
>> > from operator import itemgetter
>> >
>> > id1, id2, id3 = np.where(A[..., np.newaxis] == B)
>> > order = np.argsort(id3)
>> > triples_iter = izip(id3[order], id1[order], id2[order])
>> > grouped = groupby(triples_iter, itemgetter(0))
>> > d = dict((b_value, [idx[1:] for idx in indices]) for b_value, indices in
>> > grouped)
>> >
>> > Then d[value] is a list of all the (i, j) pairs where A[i, j] == value,
>> and
>> > the keys of d are every value in B.
>> >
>> >
>> >
>> > On Sat, Nov 24, 2012 at 3:36 PM, Siegfried Gonzi <
>> sgonzi at staffmail.ed.ac.uk>
>> > wrote:
>> >>
>> >> Hi all
>> >>
>> >> This must have been answered in the past but my google search
>> capabilities
>> >> are not the best.
>> >>
>> >> Given an array A say of dimension 40x60 and given another array/vector
>> B
>> >> of dimension 20 (the values in B occur only once).
>> >>
>> >> What I would like to do is the following which of course does not work
>> (by
>> >> the way doesn't work in IDL either):
>> >>
>> >> indx=where(A == B)
>> >>
>> >> I understand A and B are both of different dimensions. So my question:
>> >> what would the fastest or proper way to accomplish this (I found a
>> solution
>> >> but think is rather awkward and not very scipy/numpy-tonic tough).
>> >>
>> >> Thanks
>> >> --
>> >> The University of Edinburgh is a charitable body, registered in
>> >> Scotland, with registration number SC005336.
>> >>
>> >> _______________________________________________
>> >> NumPy-Discussion mailing list
>> >> NumPy-Discussion at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> >
>> >
>> > _______________________________________________
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121124/44da9fb8/attachment.html>

From sgonzi at staffmail.ed.ac.uk  Sun Nov 25 04:22:58 2012
From: sgonzi at staffmail.ed.ac.uk (Siegfried Gonzi)
Date: Sun, 25 Nov 2012 09:22:58 +0000
Subject: [Numpy-discussion] numpy where function on different sized
	arrays
In-Reply-To: <mailman.433.1353803380.1031.numpy-discussion@scipy.org>
References: <mailman.433.1353803380.1031.numpy-discussion@scipy.org>
Message-ID: <9AEC027B-50E2-4B8F-8C8A-7F0BADBA0D59@staffmail.ed.ac.uk>


On 25 Nov 2012, at 00:29, numpy-discussion-request at scipy.org wrote:
> 
> Message: 3
> Date: Sat, 24 Nov 2012 23:23:36 +0100
> From: Da?id <davidmenhur at gmail.com>
> Subject: Re: [Numpy-discussion] numpy where function on different
> 	sized	arrays
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID:
> 	<CAJhcF=2nmVeeQUo5Q45OUcQK+7KnQ_u0tvzUyOSXGpQbLSUy+Q at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
> 
> A pure Python approach could be:
> 
> for i, x in enumerate(a):
> 	for j, y in enumerate(x):
> 		if y in b:
> 			idx.append((i,j))
> 
> Of course, it is slow if the arrays are large, but it is very
> readable, and probably very fast if cythonised.


Thanks for all the answers. In that particular case speed is not important (A is 360x720 and b and c is lower than 10 in terms of dimension). However, I stumbled across similar comparison problems in IDL a couple of times where speed was crucial.

My own solution or attempt was this:

==
def fu(A, b, c):
    for x, y in zip(b,c):
        indx = np.where(A == x)
        A[indx] = y
    return A
==

> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121125/49bbc51a/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121125/49bbc51a/attachment.ksh>

From ralf.gommers at gmail.com  Sun Nov 25 14:04:05 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 25 Nov 2012 20:04:05 +0100
Subject: [Numpy-discussion] adding rfftfreq
Message-ID: <CABL7CQgKQ4OMVfWwsn2pwLK8wu6ku8pG4BEZtx_pFXi4WcD3JA@mail.gmail.com>

Hi,

https://github.com/numpy/numpy/pull/484 adds an rfftfreq function to
numpy.fft. Looks logical to me, but before merging I want to check anyway
if no one objects to that.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121125/1403de45/attachment.html>

From charlesr.harris at gmail.com  Sun Nov 25 15:06:25 2012
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 25 Nov 2012 13:06:25 -0700
Subject: [Numpy-discussion] adding rfftfreq
In-Reply-To: <CABL7CQgKQ4OMVfWwsn2pwLK8wu6ku8pG4BEZtx_pFXi4WcD3JA@mail.gmail.com>
References: <CABL7CQgKQ4OMVfWwsn2pwLK8wu6ku8pG4BEZtx_pFXi4WcD3JA@mail.gmail.com>
Message-ID: <CAB6mnxLb7U-75HPPELQvf72aVbTvTdyTsbau_KsCJzCEkcFBtw@mail.gmail.com>

On Sun, Nov 25, 2012 at 12:04 PM, Ralf Gommers <ralf.gommers at gmail.com>wrote:

> Hi,
>
> https://github.com/numpy/numpy/pull/484 adds an rfftfreq function to
> numpy.fft. Looks logical to me, but before merging I want to check anyway
> if no one objects to that.
>
>
No objection here.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121125/1c7b8bb4/attachment.html>

From tom.bennett at mail.zyzhu.net  Sun Nov 25 21:24:46 2012
From: tom.bennett at mail.zyzhu.net (Tom Bennett)
Date: Sun, 25 Nov 2012 20:24:46 -0600
Subject: [Numpy-discussion] How to Keep An Array Two Dimensional
Message-ID: <CAEYEPaTaK49Zj_30z+xhWY27gefvL9Tk2qqmy+jtTij-jtwXiA@mail.gmail.com>

Hi,

I am trying to extract n columns from an 2D array and then operate on the
extracted columns. Below is the code:

A is an MxN 2D array.

u = A[:,:n] #extract the first n columns from A

B = np.dot(u, u.T) #take outer product.

This code works when n>1. However, when n=1, u becomes an 1D array instead
of an Mx1 2D array and the code breaks down.

I wonder if there is any way to keep u=A[:,:n] an Mxn array no matter what
value n takes. I do not want to use matrix because array is more convenient
in other places.

Thanks,
Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121125/ef8bf902/attachment.html>

From warren.weckesser at gmail.com  Sun Nov 25 21:35:06 2012
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sun, 25 Nov 2012 20:35:06 -0600
Subject: [Numpy-discussion] How to Keep An Array Two Dimensional
In-Reply-To: <CAEYEPaTaK49Zj_30z+xhWY27gefvL9Tk2qqmy+jtTij-jtwXiA@mail.gmail.com>
References: <CAEYEPaTaK49Zj_30z+xhWY27gefvL9Tk2qqmy+jtTij-jtwXiA@mail.gmail.com>
Message-ID: <CAGzF1ufep6p-7BrPh9K7_tq3Ykfm+GXPewjLux4rSyv9=HhE3Q@mail.gmail.com>

On Sun, Nov 25, 2012 at 8:24 PM, Tom Bennett <tom.bennett at mail.zyzhu.net>wrote:

> Hi,
>
> I am trying to extract n columns from an 2D array and then operate on the
> extracted columns. Below is the code:
>
> A is an MxN 2D array.
>
> u = A[:,:n] #extract the first n columns from A
>
> B = np.dot(u, u.T) #take outer product.
>
> This code works when n>1. However, when n=1, u becomes an 1D array instead
> of an Mx1 2D array and the code breaks down.
>
> I wonder if there is any way to keep u=A[:,:n] an Mxn array no matter what
> value n takes. I do not want to use matrix because array is more convenient
> in other places.
>
>
Tom,

Your example works for me:

In [1]: np.__version__
Out[1]: '1.6.2'

In [2]: A = arange(15).reshape(3,5)

In [3]: A
Out[3]:
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])

In [4]: u = A[:,:1]

In [5]: u
Out[5]:
array([[ 0],
       [ 5],
       [10]])

In [6]: B = np.dot(u, u.T)

In [7]: B
Out[7]:
array([[  0,   0,   0],
       [  0,  25,  50],
       [  0,  50, 100]])



Warren



> Thanks,
> Tom
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121125/0a3a54a1/attachment.html>

From tom.bennett at mail.zyzhu.net  Sun Nov 25 21:47:18 2012
From: tom.bennett at mail.zyzhu.net (Tom Bennett)
Date: Sun, 25 Nov 2012 20:47:18 -0600
Subject: [Numpy-discussion] How to Keep An Array Two Dimensional
In-Reply-To: <CAGzF1ufep6p-7BrPh9K7_tq3Ykfm+GXPewjLux4rSyv9=HhE3Q@mail.gmail.com>
References: <CAEYEPaTaK49Zj_30z+xhWY27gefvL9Tk2qqmy+jtTij-jtwXiA@mail.gmail.com>
	<CAGzF1ufep6p-7BrPh9K7_tq3Ykfm+GXPewjLux4rSyv9=HhE3Q@mail.gmail.com>
Message-ID: <CAEYEPaQgBqgDX8f6pFgGm7=P1S2iVLQFLTuz=_BwJO8PNcVWaw@mail.gmail.com>

Thanks for the quick response.

Ah, I see. There is a difference between A[:,:1] and A[:,0]. The former
returns an Mx1 2D array whereas the latter returns an M element 1D array. I
was using A[:,0] in the code but A[:,:1] in the example.


On Sun, Nov 25, 2012 at 8:35 PM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

>
>
> On Sun, Nov 25, 2012 at 8:24 PM, Tom Bennett <tom.bennett at mail.zyzhu.net>wrote:
>
>> Hi,
>>
>> I am trying to extract n columns from an 2D array and then operate on the
>> extracted columns. Below is the code:
>>
>> A is an MxN 2D array.
>>
>> u = A[:,:n] #extract the first n columns from A
>>
>> B = np.dot(u, u.T) #take outer product.
>>
>> This code works when n>1. However, when n=1, u becomes an 1D array
>> instead of an Mx1 2D array and the code breaks down.
>>
>> I wonder if there is any way to keep u=A[:,:n] an Mxn array no matter
>> what value n takes. I do not want to use matrix because array is more
>> convenient in other places.
>>
>>
> Tom,
>
> Your example works for me:
>
> In [1]: np.__version__
> Out[1]: '1.6.2'
>
> In [2]: A = arange(15).reshape(3,5)
>
> In [3]: A
> Out[3]:
> array([[ 0,  1,  2,  3,  4],
>        [ 5,  6,  7,  8,  9],
>        [10, 11, 12, 13, 14]])
>
> In [4]: u = A[:,:1]
>
> In [5]: u
> Out[5]:
> array([[ 0],
>        [ 5],
>        [10]])
>
> In [6]: B = np.dot(u, u.T)
>
> In [7]: B
> Out[7]:
> array([[  0,   0,   0],
>        [  0,  25,  50],
>        [  0,  50, 100]])
>
>
>
> Warren
>
>
>
>> Thanks,
>> Tom
>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121125/bd94ba99/attachment.html>

From d.warde.farley at gmail.com  Mon Nov 26 01:07:13 2012
From: d.warde.farley at gmail.com (David Warde-Farley)
Date: Mon, 26 Nov 2012 01:07:13 -0500
Subject: [Numpy-discussion] How to Keep An Array Two Dimensional
In-Reply-To: <CAEYEPaQgBqgDX8f6pFgGm7=P1S2iVLQFLTuz=_BwJO8PNcVWaw@mail.gmail.com>
References: <CAEYEPaTaK49Zj_30z+xhWY27gefvL9Tk2qqmy+jtTij-jtwXiA@mail.gmail.com>
	<CAGzF1ufep6p-7BrPh9K7_tq3Ykfm+GXPewjLux4rSyv9=HhE3Q@mail.gmail.com>
	<CAEYEPaQgBqgDX8f6pFgGm7=P1S2iVLQFLTuz=_BwJO8PNcVWaw@mail.gmail.com>
Message-ID: <CALc6Xo4WNzEJQBshx=LyXG7SKyxruPnqHn1L_z7Z83pfs5gOqw@mail.gmail.com>

On Sun, Nov 25, 2012 at 9:47 PM, Tom Bennett <tom.bennett at mail.zyzhu.net> wrote:
> Thanks for the quick response.
>
> Ah, I see. There is a difference between A[:,:1] and A[:,0]. The former
> returns an Mx1 2D array whereas the latter returns an M element 1D array. I
> was using A[:,0] in the code but A[:,:1] in the example.

You'll notice that Python lists and tuples work the same way: foo[0]
on a list or tuple gives you the first element whereas foo[:1] gives
you a list or tuple containing only the first element.

To clarify what's going on in the case of NumPy: when you use the [:,
0] syntax, the interpreter is calling A.__getitem__ with the tuple
(slice(None), 0) as the argument. When you use [:, :1], the argument
is (slice(None), slice(None, 1)). You can try this out with
A[(slice(None), slice(None, 1))] -- it does the same thing (creating
index tuples explicitly like this can be very handy in certain cases).

The rule that NumPy follows for index tuples is (approximately) that
scalar indices always squash the corresponding dimension, whereas
slices or iterables (in the case of fancy indexing) preserve the
dimension with an appropriate size. Notably, A[:, [0]] will also
return an (A.shape[0], 1) array. But the semantics here are different:
because using a sequence as an "advanced" indexing operation, a copy
is made, whereas A[:, :1] will return a view.

Hope that makes things less mysterious,

David


From mogus.mochena at famu.edu  Sat Nov 24 16:36:48 2012
From: mogus.mochena at famu.edu (Mogus Mochena)
Date: Sat, 24 Nov 2012 16:36:48 -0500
Subject: [Numpy-discussion] unsubscribe
Message-ID: <CCD6A820.9F72%mogus.mochena@famu.edu>






From jsseabold at gmail.com  Mon Nov 26 13:54:01 2012
From: jsseabold at gmail.com (Skipper Seabold)
Date: Mon, 26 Nov 2012 13:54:01 -0500
Subject: [Numpy-discussion] result shape from dot for 0d, 1d, 2d scalar
Message-ID: <CAKF=DjsMN9di=QZKdj_u_OZgJe2kDhAoxqvNOFShwFUVO9f3Tg@mail.gmail.com>

I discovered this because scipy.optimize.fmin_powell appears to squeeze 1d
argmin to 0d unlike the other optimizers, but that's a different story.

I would expect the 0d array to behave like the 1d array not the 2d as it
does below. Thoughts? Maybe too big of a pain to change this behavior if
indeed it's not desired, but I found it to be unexpected.

[255]: np.version.full_version # same on 1.5.1
[255]: '1.8.0.dev-8e0a542'

[262]: arr = np.random.random((25,1))

[~/]
[263]: np.dot(arr, np.array([1.])).shape
[263]: (25,)

[~/]
[264]: np.dot(arr, np.array([[1.]])).shape
[264]: (25, 1)

[~/]
[265]: np.dot(arr, np.array(1.)).shape
[265]: (25, 1)

[~/]
[271]: np.dot(arr.squeeze(), np.array(1.)).shape
[271]: (25,)

Huh? 0d arrays broadcast with dot?

[~]
[279]: arr = np.random.random((25,2))

[~/]
[280]: np.dot(arr.squeeze(), np.array(2.)).shape
[280]: (25, 2)

Skipper
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121126/7d9fba40/attachment.html>

From thomas.robitaille at gmail.com  Tue Nov 27 07:27:06 2012
From: thomas.robitaille at gmail.com (Thomas Robitaille)
Date: Tue, 27 Nov 2012 13:27:06 +0100
Subject: [Numpy-discussion] Numpy on Travis with Python 3
Message-ID: <CAGMHX_26bi-g4cRrpKHQ=ceLcmAhh0OsgOj=iepua_+6CPuvNQ@mail.gmail.com>

Hi everyone,

I'm currently having issues with installing Numpy 1.6.2 with Python
3.1 and 3.2 using pip in Travis builds - see for example:

https://travis-ci.org/astropy/astropy/jobs/3379866

The build aborts with a cryptic message:

ValueError: underlying buffer has been detached

Has anyone seen this kind of issue before?

Thanks for any help,

Cheers,
Tom


From p.j.a.cock at googlemail.com  Tue Nov 27 07:35:44 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Tue, 27 Nov 2012 12:35:44 +0000
Subject: [Numpy-discussion] Numpy on Travis with Python 3
In-Reply-To: <CAGMHX_26bi-g4cRrpKHQ=ceLcmAhh0OsgOj=iepua_+6CPuvNQ@mail.gmail.com>
References: <CAGMHX_26bi-g4cRrpKHQ=ceLcmAhh0OsgOj=iepua_+6CPuvNQ@mail.gmail.com>
Message-ID: <CAKVJ-_7inrzdbUjYWdK+rH5FkVp8G4veH1XVaREAML5XrfUrhA@mail.gmail.com>

On Tue, Nov 27, 2012 at 12:27 PM, Thomas Robitaille
<thomas.robitaille at gmail.com> wrote:
> Hi everyone,
>
> I'm currently having issues with installing Numpy 1.6.2 with Python
> 3.1 and 3.2 using pip in Travis builds - see for example:
>
> https://travis-ci.org/astropy/astropy/jobs/3379866
>
> The build aborts with a cryptic message:
>
> ValueError: underlying buffer has been detached
>
> Has anyone seen this kind of issue before?
>
> Thanks for any help,
>
> Cheers,
> Tom

Hi Tom,

Yes, a similar error has been reported with virtualenv 1.8.3,
see https://github.com/pypa/virtualenv/issues/359

If you were not aware, the TravisCI team are currently actively
trying to get NumPy preinstalled on the virtual machines. As I
write this, Python 2.5, 2.6 and 2.7 now have NumPy while
Python 3.1 is being dropped (it was only ever an unofficially
supported platform), and NumPy under Python 3.2 and 3.3
is still in progress. See:

https://github.com/travis-ci/travis-cookbooks/issues/48
https://github.com/travis-ci/travis-cookbooks/issues/89

This "ValueError: underlying buffer has been detached" was
one of the issues the TravisCI team had faced.

Peter


From deil.christoph at googlemail.com  Tue Nov 27 07:37:01 2012
From: deil.christoph at googlemail.com (Christoph Deil)
Date: Tue, 27 Nov 2012 13:37:01 +0100
Subject: [Numpy-discussion] Numpy on Travis with Python 3
In-Reply-To: <CAGMHX_26bi-g4cRrpKHQ=ceLcmAhh0OsgOj=iepua_+6CPuvNQ@mail.gmail.com>
References: <CAGMHX_26bi-g4cRrpKHQ=ceLcmAhh0OsgOj=iepua_+6CPuvNQ@mail.gmail.com>
Message-ID: <DF2BA2A9-1DD8-4BB2-B5FD-C4CEB7784AE3@gmail.com>

Hi Tom,

you can start reading here:
https://github.com/numpy/numpy/issues/2761#issuecomment-10672029

:-)

Christoph

On Nov 27, 2012, at 1:27 PM, Thomas Robitaille <thomas.robitaille at gmail.com> wrote:

> Hi everyone,
> 
> I'm currently having issues with installing Numpy 1.6.2 with Python
> 3.1 and 3.2 using pip in Travis builds - see for example:
> 
> https://travis-ci.org/astropy/astropy/jobs/3379866
> 
> The build aborts with a cryptic message:
> 
> ValueError: underlying buffer has been detached
> 
> Has anyone seen this kind of issue before?
> 
> Thanks for any help,
> 
> Cheers,
> Tom
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From sebastian at sipsolutions.net  Tue Nov 27 11:16:31 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Tue, 27 Nov 2012 17:16:31 +0100
Subject: [Numpy-discussion] result shape from dot for 0d, 1d, 2d scalar
In-Reply-To: <CAKF=DjsMN9di=QZKdj_u_OZgJe2kDhAoxqvNOFShwFUVO9f3Tg@mail.gmail.com>
References: <CAKF=DjsMN9di=QZKdj_u_OZgJe2kDhAoxqvNOFShwFUVO9f3Tg@mail.gmail.com>
Message-ID: <1354032991.16990.3.camel@sebastian-laptop>

On Mon, 2012-11-26 at 13:54 -0500, Skipper Seabold wrote:
> I discovered this because scipy.optimize.fmin_powell appears to
> squeeze 1d argmin to 0d unlike the other optimizers, but that's a
> different story.
> 
> 
> I would expect the 0d array to behave like the 1d array not the 2d as
> it does below. Thoughts? Maybe too big of a pain to change this
> behavior if indeed it's not desired, but I found it to be unexpected.

I don't quite understand why it is unexpected. A 1-d array is considered
a vector, a 0-d array is a scalar.

> [255]: np.version.full_version # same on 1.5.1
> [255]: '1.8.0.dev-8e0a542'
> 
> 
> [262]: arr = np.random.random((25,1))
> 
> 
> [~/]
> [263]: np.dot(arr, np.array([1.])).shape
> [263]: (25,)
> 
Matrix times vector = vector
> 
> [~/]
> [264]: np.dot(arr, np.array([[1.]])).shape
> [264]: (25, 1)
> 
Matrix times matrix = matrix
> 
> [~/]
> [265]: np.dot(arr, np.array(1.)).shape
> [265]: (25, 1)
> 
matrix times scalar = matrix (of same shape)
> 
> [~/]
> [271]: np.dot(arr.squeeze(), np.array(1.)).shape
> [271]: (25,)
> 
vector times scalar = vector (of same shape)
> 
> Huh? 0d arrays broadcast with dot?
> 
Remember a 0-d array is a scalar, there is no actual broadcasting
involved here. (except that vectors (1-d arrays) are special)

> [~]
> [279]: arr = np.random.random((25,2))
> 
> 
> [~/]
> [280]: np.dot(arr.squeeze(), np.array(2.)).shape
> [280]: (25, 2)
> 
> 
> Skipper
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From fperez.net at gmail.com  Tue Nov 27 15:08:08 2012
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 27 Nov 2012 12:08:08 -0800
Subject: [Numpy-discussion] In case anybody wants them: py4science.*
	domains...
In-Reply-To: <CAKVJ-_4YJ=P3Aaox5tUgLQDK9WtrWtudx2174UZapzgQKAf5sg@mail.gmail.com>
References: <CAHAreOrVR1050-mmTF67pYwPO-xmJ4s0pO-LdbaUT67vvL6k_w@mail.gmail.com>
	<CAKVJ-_4YJ=P3Aaox5tUgLQDK9WtrWtudx2174UZapzgQKAf5sg@mail.gmail.com>
Message-ID: <CAHAreOpa2q0+B6Ptvs2O6V2kmx2XRVoiDH8aVXL-MsARWxEC9A@mail.gmail.com>

On Thu, Nov 22, 2012 at 1:30 AM, Peter Cock <p.j.a.cock at googlemail.com> wrote:

> Perhaps http://numfocus.org/ could take them on, or the PSF?
> (even if they don't have a specific use in mind immediately)
> For the short them I'd just have them redirect to www.scipy.org ;)

I asked on the numfocus list and nobody was really interested, and I
floated the question at a board meeting and folks also agreed that
with the limited time/resources numfocus has right now, there were
more important things to do.

So I'll just let them lapse, if anybody cares, they'll be open for the
taking come December 3 :)

Cheers,

f


From scopatz at gmail.com  Tue Nov 27 16:25:13 2012
From: scopatz at gmail.com (Anthony Scopatz)
Date: Tue, 27 Nov 2012 15:25:13 -0600
Subject: [Numpy-discussion] In case anybody wants them: py4science.*
	domains...
In-Reply-To: <CAHAreOpa2q0+B6Ptvs2O6V2kmx2XRVoiDH8aVXL-MsARWxEC9A@mail.gmail.com>
References: <CAHAreOrVR1050-mmTF67pYwPO-xmJ4s0pO-LdbaUT67vvL6k_w@mail.gmail.com>
	<CAKVJ-_4YJ=P3Aaox5tUgLQDK9WtrWtudx2174UZapzgQKAf5sg@mail.gmail.com>
	<CAHAreOpa2q0+B6Ptvs2O6V2kmx2XRVoiDH8aVXL-MsARWxEC9A@mail.gmail.com>
Message-ID: <CAPk-6T6EXKAKDnM-Jt2evaH0ou+z5UKgs93EWnj4_YQZe-djrA@mail.gmail.com>

On Tue, Nov 27, 2012 at 2:08 PM, Fernando Perez <fperez.net at gmail.com>wrote:

> On Thu, Nov 22, 2012 at 1:30 AM, Peter Cock <p.j.a.cock at googlemail.com>
> wrote:
>
> > Perhaps http://numfocus.org/ could take them on, or the PSF?
> > (even if they don't have a specific use in mind immediately)
> > For the short them I'd just have them redirect to www.scipy.org ;)
>
> I asked on the numfocus list and nobody was really interested, and I
> floated the question at a board meeting and folks also agreed that
> with the limited time/resources numfocus has right now, there were
> more important things to do.
>
> So I'll just let them lapse, if anybody cares, they'll be open for the
> taking come December 3 :)
>

Gah! Sorry for missing this.  I actually think that the redirection idea is
a
really good one.  It can't be that expensive to just maintain these
indefinitely.
I'd rather us have them than someone else.  I'll make a motion for this
on the NumFOCUS list.

Be Well
Anthony


>
> Cheers,
>
> f
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121127/883279b9/attachment.html>

From p.j.a.cock at googlemail.com  Tue Nov 27 16:51:11 2012
From: p.j.a.cock at googlemail.com (Peter Cock)
Date: Tue, 27 Nov 2012 21:51:11 +0000
Subject: [Numpy-discussion] In case anybody wants them: py4science.*
	domains...
In-Reply-To: <CAPk-6T6EXKAKDnM-Jt2evaH0ou+z5UKgs93EWnj4_YQZe-djrA@mail.gmail.com>
References: <CAHAreOrVR1050-mmTF67pYwPO-xmJ4s0pO-LdbaUT67vvL6k_w@mail.gmail.com>
	<CAKVJ-_4YJ=P3Aaox5tUgLQDK9WtrWtudx2174UZapzgQKAf5sg@mail.gmail.com>
	<CAHAreOpa2q0+B6Ptvs2O6V2kmx2XRVoiDH8aVXL-MsARWxEC9A@mail.gmail.com>
	<CAPk-6T6EXKAKDnM-Jt2evaH0ou+z5UKgs93EWnj4_YQZe-djrA@mail.gmail.com>
Message-ID: <CAKVJ-_508KLqpSRxxJvzUnFnzbR94Mk-WGhJ=beXhaSQBFZfoQ@mail.gmail.com>

On Tue, Nov 27, 2012 at 9:25 PM, Anthony Scopatz <scopatz at gmail.com> wrote:
> On Tue, Nov 27, 2012 at 2:08 PM, Fernando Perez <fperez.net at gmail.com>
> wrote:
>>
>> On Thu, Nov 22, 2012 at 1:30 AM, Peter Cock <p.j.a.cock at googlemail.com>
>> wrote:
>>
>> > Perhaps http://numfocus.org/ could take them on, or the PSF?
>> > (even if they don't have a specific use in mind immediately)
>> > For the short them I'd just have them redirect to www.scipy.org ;)
>>
>> I asked on the numfocus list and nobody was really interested, and I
>> floated the question at a board meeting and folks also agreed that
>> with the limited time/resources numfocus has right now, there were
>> more important things to do.
>>
>> So I'll just let them lapse, if anybody cares, they'll be open for the
>> taking come December 3 :)
>
>
> Gah! Sorry for missing this.  I actually think that the redirection idea is
> a really good one.

It seems more worthwhile than just letting a domain squatter use it.

> It can't be that expensive to just maintain these indefinitely.
> I'd rather us have them than someone else.  I'll make a motion for this
> on the NumFOCUS list.

The domain registration cost is minimal, and if NumFOCUS are
already looking after existing domains the extra admin should
be minimal.

Could you keep us informed if the domains still need a home?
Might make a good domain name for a blog... hmm.

Thanks,

Peter


From scopatz at gmail.com  Tue Nov 27 16:56:19 2012
From: scopatz at gmail.com (Anthony Scopatz)
Date: Tue, 27 Nov 2012 15:56:19 -0600
Subject: [Numpy-discussion] In case anybody wants them: py4science.*
	domains...
In-Reply-To: <CAKVJ-_508KLqpSRxxJvzUnFnzbR94Mk-WGhJ=beXhaSQBFZfoQ@mail.gmail.com>
References: <CAHAreOrVR1050-mmTF67pYwPO-xmJ4s0pO-LdbaUT67vvL6k_w@mail.gmail.com>
	<CAKVJ-_4YJ=P3Aaox5tUgLQDK9WtrWtudx2174UZapzgQKAf5sg@mail.gmail.com>
	<CAHAreOpa2q0+B6Ptvs2O6V2kmx2XRVoiDH8aVXL-MsARWxEC9A@mail.gmail.com>
	<CAPk-6T6EXKAKDnM-Jt2evaH0ou+z5UKgs93EWnj4_YQZe-djrA@mail.gmail.com>
	<CAKVJ-_508KLqpSRxxJvzUnFnzbR94Mk-WGhJ=beXhaSQBFZfoQ@mail.gmail.com>
Message-ID: <CAPk-6T6qx3T8H3azLtjW+aKyFd_AYhPJ9NNC6OABQDmv8vwLXw@mail.gmail.com>

On Tue, Nov 27, 2012 at 3:51 PM, Peter Cock <p.j.a.cock at googlemail.com>wrote:

> On Tue, Nov 27, 2012 at 9:25 PM, Anthony Scopatz <scopatz at gmail.com>
> wrote:
> > On Tue, Nov 27, 2012 at 2:08 PM, Fernando Perez <fperez.net at gmail.com>
> > wrote:
> >>
> >> On Thu, Nov 22, 2012 at 1:30 AM, Peter Cock <p.j.a.cock at googlemail.com>
> >> wrote:
> >>
> >> > Perhaps http://numfocus.org/ could take them on, or the PSF?
> >> > (even if they don't have a specific use in mind immediately)
> >> > For the short them I'd just have them redirect to www.scipy.org ;)
> >>
> >> I asked on the numfocus list and nobody was really interested, and I
> >> floated the question at a board meeting and folks also agreed that
> >> with the limited time/resources numfocus has right now, there were
> >> more important things to do.
> >>
> >> So I'll just let them lapse, if anybody cares, they'll be open for the
> >> taking come December 3 :)
> >
> >
> > Gah! Sorry for missing this.  I actually think that the redirection idea
> is
> > a really good one.
>
> It seems more worthwhile than just letting a domain squatter use it.
>
> > It can't be that expensive to just maintain these indefinitely.
> > I'd rather us have them than someone else.  I'll make a motion for this
> > on the NumFOCUS list.
>
> The domain registration cost is minimal, and if NumFOCUS are
> already looking after existing domains the extra admin should
> be minimal.
>
> Could you keep us informed if the domains still need a home?
>

Yup, I'll be sure to keep everyone posted.


> Might make a good domain name for a blog... hmm.


> Thanks,
>
> Peter
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121127/028cafec/attachment.html>

From francesc at continuum.io  Wed Nov 28 06:34:38 2012
From: francesc at continuum.io (Francesc Alted)
Date: Wed, 28 Nov 2012 12:34:38 +0100
Subject: [Numpy-discussion] the difference between "+" and np.add?
In-Reply-To: <CALGmxEL-J5rOo9-YwQuRvXO+7R2AOUX56R_y45gfbTK=ZoRP4w@mail.gmail.com>
References: <CAAN-aREVeYj8J0KOYKfv7fr=UttsxcSXQ=dKoabWLBgT-aGk7w@mail.gmail.com>
	<50AE34B7.3070401@continuum.io>
	<CALGmxEL-J5rOo9-YwQuRvXO+7R2AOUX56R_y45gfbTK=ZoRP4w@mail.gmail.com>
Message-ID: <50B5F6CE.9090708@continuum.io>

On 11/23/12 8:00 PM, Chris Barker - NOAA Federal wrote:
> On Thu, Nov 22, 2012 at 6:20 AM, Francesc Alted <francesc at continuum.io> wrote:
>> As Nathaniel said, there is not a difference in terms of *what* is
>> computed.  However, the methods that you suggested actually differ on
>> *how* they are computed, and that has dramatic effects on the time
>> used.  For example:
>>
>> In []: arr1, arr2, arr3, arr4, arr5 = [np.arange(1e7) for x in range(5)]
>>
>> In []: %time arr1 + arr2 + arr3 + arr4 + arr5
>> CPU times: user 0.05 s, sys: 0.10 s, total: 0.14 s
>> Wall time: 0.15 s
>> There are also ways to minimize the size of temporaries, and numexpr is
>> one of the simplests:
> but you can also use np.add (and friends) to reduce the number of
> temporaries. It can make a difference:
>
> In [11]: def add_5_arrays(arr1, arr2, arr3, arr4, arr5):
>     ....:     result = arr1 + arr2
>     ....:     np.add(result, arr3, out=result)
>     ....:     np.add(result, arr4, out=result)
>     ....:     np.add(result, arr5, out=result)
>
> In [13]: timeit arr1 + arr2 + arr3 + arr4 + arr5
> 1 loops, best of 3: 528 ms per loop
>
> In [17]: timeit add_5_arrays(arr1, arr2, arr3, arr4, arr5)
> 1 loops, best of 3: 293 ms per loop
>
> (don't have numexpr on this machine for a comparison)

Yes, you are right.  However, numexpr still can beat this:

In [8]: timeit arr1 + arr2 + arr3 + arr4 + arr5
10 loops, best of 3: 138 ms per loop

In [9]: timeit add_5_arrays(arr1, arr2, arr3, arr4, arr5)
10 loops, best of 3: 74.3 ms per loop

In [10]: timeit ne.evaluate("arr1 + arr2 + arr3 + arr4 + arr5")
10 loops, best of 3: 20.8 ms per loop

The reason is that numexpr is multithreaded (using 6 cores above), and 
for memory-bounded problems like this one, fetching data in different 
threads is more efficient than using a single thread:

In [12]: timeit arr1.copy()
10 loops, best of 3: 41 ms per loop

In [13]: ne.set_num_threads(1)
Out[13]: 6

In [14]: timeit ne.evaluate("arr1")
10 loops, best of 3: 30.7 ms per loop

In [15]: ne.set_num_threads(6)
Out[15]: 1

In [16]: timeit ne.evaluate("arr1")
100 loops, best of 3: 13.4 ms per loop

I.e., the joy of multi-threading is that it not only buys you CPU speed, 
but can also bring your data from memory faster.  So yeah, modern 
applications *do* need multi-threading for getting good performance.

-- 
Francesc Alted



From mail at telenczuk.pl  Wed Nov 28 07:47:18 2012
From: mail at telenczuk.pl (Bartosz)
Date: Wed, 28 Nov 2012 13:47:18 +0100
Subject: [Numpy-discussion] Conditional update of recarray field
Message-ID: <50B607D6.6000303@telenczuk.pl>

Hi,

I try to update values in a single field of numpy record array based on 
a condition defined in another array. I found that that the result 
depends on the order in which I apply the boolean indices/field names.

For example:

cond = np.zeros(5, dtype=np.bool)
cond[2:] = True
X = np.rec.fromarrays([np.arange(5)], names='a')
X[cond]['a'] = -1
print X

returns:  [(0,) (1,) (2,) (3,) (4,)] (the values were not updated)

X['a'][cond] = -1
print X

returns: [(0,) (1,) (-1,) (-1,) (-1,)] (it worked this time).

I find this behaviour very confusing. Is it expected? Would it be 
possible to emit a warning message in the case of "faulty" assignments?

Bartosz


From francesc at continuum.io  Wed Nov 28 09:05:37 2012
From: francesc at continuum.io (Francesc Alted)
Date: Wed, 28 Nov 2012 15:05:37 +0100
Subject: [Numpy-discussion] Conditional update of recarray field
In-Reply-To: <50B607D6.6000303@telenczuk.pl>
References: <50B607D6.6000303@telenczuk.pl>
Message-ID: <50B61A31.3070903@continuum.io>

On 11/28/12 1:47 PM, Bartosz wrote:
> Hi,
>
> I try to update values in a single field of numpy record array based on
> a condition defined in another array. I found that that the result
> depends on the order in which I apply the boolean indices/field names.
>
> For example:
>
> cond = np.zeros(5, dtype=np.bool)
> cond[2:] = True
> X = np.rec.fromarrays([np.arange(5)], names='a')
> X[cond]['a'] = -1
> print X
>
> returns:  [(0,) (1,) (2,) (3,) (4,)] (the values were not updated)
>
> X['a'][cond] = -1
> print X
>
> returns: [(0,) (1,) (-1,) (-1,) (-1,)] (it worked this time).
>
> I find this behaviour very confusing. Is it expected?

Yes, it is.  In the first idiom, X[cond] is a fancy indexing operation 
and the result is not a view, so what you are doing is basically 
modifying the temporary object that results from the indexing.  In the 
second idiom, X['a'] is returning a *view* of the original object, so 
this is why it works.

>   Would it be
> possible to emit a warning message in the case of "faulty" assignments?

The only solution that I can see for this is that the fancy indexing 
would return a view, and not a different object, but NumPy containers 
are not prepared for this.

-- 
Francesc Alted



From mail at telenczuk.pl  Wed Nov 28 09:26:48 2012
From: mail at telenczuk.pl (Bartosz)
Date: Wed, 28 Nov 2012 15:26:48 +0100
Subject: [Numpy-discussion] Conditional update of recarray field
In-Reply-To: <50B61A31.3070903@continuum.io>
References: <50B607D6.6000303@telenczuk.pl> <50B61A31.3070903@continuum.io>
Message-ID: <50B61F28.1040804@telenczuk.pl>

Thanks for answer, Francesc.

I understand now that fancy indexing returns a copy of a recarray. Is 
it also true for standard ndarrays? If so, I do not understand why 
X['a'][cond]=-1 should work.

Cheers,

Bartosz

On Wed 28 Nov 2012 03:05:37 PM CET, Francesc Alted wrote:
> On 11/28/12 1:47 PM, Bartosz wrote:
>> Hi,
>>
>> I try to update values in a single field of numpy record array based on
>> a condition defined in another array. I found that that the result
>> depends on the order in which I apply the boolean indices/field names.
>>
>> For example:
>>
>> cond = np.zeros(5, dtype=np.bool)
>> cond[2:] = True
>> X = np.rec.fromarrays([np.arange(5)], names='a')
>> X[cond]['a'] = -1
>> print X
>>
>> returns:  [(0,) (1,) (2,) (3,) (4,)] (the values were not updated)
>>
>> X['a'][cond] = -1
>> print X
>>
>> returns: [(0,) (1,) (-1,) (-1,) (-1,)] (it worked this time).
>>
>> I find this behaviour very confusing. Is it expected?
>
> Yes, it is.  In the first idiom, X[cond] is a fancy indexing operation
> and the result is not a view, so what you are doing is basically
> modifying the temporary object that results from the indexing.  In the
> second idiom, X['a'] is returning a *view* of the original object, so
> this is why it works.
>
>>    Would it be
>> possible to emit a warning message in the case of "faulty" assignments?
>
> The only solution that I can see for this is that the fancy indexing
> would return a view, and not a different object, but NumPy containers
> are not prepared for this.
>


From francesc at continuum.io  Wed Nov 28 09:36:50 2012
From: francesc at continuum.io (Francesc Alted)
Date: Wed, 28 Nov 2012 15:36:50 +0100
Subject: [Numpy-discussion] Conditional update of recarray field
In-Reply-To: <50B61F28.1040804@telenczuk.pl>
References: <50B607D6.6000303@telenczuk.pl> <50B61A31.3070903@continuum.io>
	<50B61F28.1040804@telenczuk.pl>
Message-ID: <50B62182.3090005@continuum.io>

Hey Bartosz,

On 11/28/12 3:26 PM, Bartosz wrote:
> Thanks for answer, Francesc.
>
> I understand now that fancy indexing returns a copy of a recarray. Is
> it also true for standard ndarrays? If so, I do not understand why
> X['a'][cond]=-1 should work.

Yes, that's a good question.  No, in this case the boolean array `cond` 
is passed to the __setitem__() of the original view, so this is why this 
works.  The first idiom is concatenating the fancy indexing with another 
indexing operation, and NumPy needs to create a temporary for executing 
this, so the second indexing operation acts over a copy, not a view.

And yes, fancy indexing returning a copy is standard for all ndarrays.

Hope it is clearer now (although admittedly it is a bit strange at first 
sight),

-- 
Francesc Alted



From mail at telenczuk.pl  Wed Nov 28 10:15:52 2012
From: mail at telenczuk.pl (Bartosz)
Date: Wed, 28 Nov 2012 16:15:52 +0100
Subject: [Numpy-discussion] Conditional update of recarray field
In-Reply-To: <50B62182.3090005@continuum.io>
References: <50B607D6.6000303@telenczuk.pl> <50B61A31.3070903@continuum.io>
	<50B61F28.1040804@telenczuk.pl> <50B62182.3090005@continuum.io>
Message-ID: <50B62AA8.5020902@telenczuk.pl>

I got it. Thanks! Now I see why this is non-trivial to fix it.

However, it might be also a source of very-hard-to-find bugs. It might 
be worth discussing this non-intuitive example in the documentation.

Cheers,

Bartosz

>> Thanks for answer, Francesc.
>>
>> I understand now that fancy indexing returns a copy of a recarray. Is
>> it also true for standard ndarrays? If so, I do not understand why
>> X['a'][cond]=-1 should work.
>
> Yes, that's a good question.  No, in this case the boolean array `cond`
> is passed to the __setitem__() of the original view, so this is why this
> works.  The first idiom is concatenating the fancy indexing with another
> indexing operation, and NumPy needs to create a temporary for executing
> this, so the second indexing operation acts over a copy, not a view.
>
> And yes, fancy indexing returning a copy is standard for all ndarrays.
>
> Hope it is clearer now (although admittedly it is a bit strange at first
> sight),
>


From jsseabold at gmail.com  Wed Nov 28 11:11:02 2012
From: jsseabold at gmail.com (Skipper Seabold)
Date: Wed, 28 Nov 2012 11:11:02 -0500
Subject: [Numpy-discussion] result shape from dot for 0d, 1d, 2d scalar
In-Reply-To: <1354032991.16990.3.camel@sebastian-laptop>
References: <CAKF=DjsMN9di=QZKdj_u_OZgJe2kDhAoxqvNOFShwFUVO9f3Tg@mail.gmail.com>
	<1354032991.16990.3.camel@sebastian-laptop>
Message-ID: <CAKF=Djt78j8F0m7Jy+PhR6yZMtJeKC0Jkm8-DzyoJyMgKXRnag@mail.gmail.com>

On Tue, Nov 27, 2012 at 11:16 AM, Sebastian Berg <sebastian at sipsolutions.net
> wrote:

> On Mon, 2012-11-26 at 13:54 -0500, Skipper Seabold wrote:
> > I discovered this because scipy.optimize.fmin_powell appears to
> > squeeze 1d argmin to 0d unlike the other optimizers, but that's a
> > different story.
> >
> >
> > I would expect the 0d array to behave like the 1d array not the 2d as
> > it does below. Thoughts? Maybe too big of a pain to change this
> > behavior if indeed it's not desired, but I found it to be unexpected.
>
> I don't quite understand why it is unexpected. A 1-d array is considered
> a vector, a 0-d array is a scalar.
>
>
When you put it like this I guess it makes sense. I don't encounter 0d
arrays often and never think of a 0d array as truly a scalar like
np.array(1.).item(). See below for my intuition.


> > [255]: np.version.full_version # same on 1.5.1
> > [255]: '1.8.0.dev-8e0a542'
> >
> >
> > [262]: arr = np.random.random((25,1))
> >
> >
> > [~/]
> > [263]: np.dot(arr, np.array([1.])).shape
> > [263]: (25,)
> >
> Matrix times vector = vector
> >
> > [~/]
> > [264]: np.dot(arr, np.array([[1.]])).shape
> > [264]: (25, 1)
> >
> Matrix times matrix = matrix
> >
> > [~/]
> > [265]: np.dot(arr, np.array(1.)).shape
> > [265]: (25, 1)
> >
> matrix times scalar = matrix (of same shape)
> >
> > [~/]
> > [271]: np.dot(arr.squeeze(), np.array(1.)).shape
> > [271]: (25,)
> >
> vector times scalar = vector (of same shape)
> >
> > Huh? 0d arrays broadcast with dot?
> >
> Remember a 0-d array is a scalar, there is no actual broadcasting
> involved here. (except that vectors (1-d arrays) are special)
>
>
Maybe I'm misunderstanding. How do you mean there is no broadcasting?
They're clearly not conformable. Is vector.scalar specially defined (I have
no idea)? I recall arguing once and submitting a patch such that
np.linalg.det(5) and np.linalg.inv(5) should be well-defined and work but
the counter-argument was that a scalar is not the same as a scalar matrix.
This seems to be an exception.

Here, I guess, following that counterargument, I'd expected the scalar to
fail in dot. I certainly don't expect a (N,2).scalar -> (N,2). Or I'd
expect it to follow the rules of matrix notation and be treated like the 1d
scalar vector so that (N,1).scalar -> (N,). To my mind, this follows more
closely to the expectation that (J,K).(M,N) -> (J,N), i.e., the second
dimension of the result is the same as the second dimension of whatever is
post-multiplying where the first dimension is inferred if necessary (or
should fail if non-existent). So my expectations are (were)

(N,).() -> (N,)
(N,1).() -> (N,)
(N,1).(1,) -> (N,)
(N,1).(1,1) -> (N,1)
(N,2).() -> Error

Skipper

> [~]
> > [279]: arr = np.random.random((25,2))
> >
> >
> > [~/]
> > [280]: np.dot(arr.squeeze(), np.array(2.)).shape
> > [280]: (25, 2)
> >
> >
> > Skipper
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121128/459f01f1/attachment.html>

From sebastian at sipsolutions.net  Wed Nov 28 12:31:25 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Wed, 28 Nov 2012 18:31:25 +0100
Subject: [Numpy-discussion] result shape from dot for 0d, 1d, 2d scalar
In-Reply-To: <CAKF=Djt78j8F0m7Jy+PhR6yZMtJeKC0Jkm8-DzyoJyMgKXRnag@mail.gmail.com>
References: <CAKF=DjsMN9di=QZKdj_u_OZgJe2kDhAoxqvNOFShwFUVO9f3Tg@mail.gmail.com>
	<1354032991.16990.3.camel@sebastian-laptop>
	<CAKF=Djt78j8F0m7Jy+PhR6yZMtJeKC0Jkm8-DzyoJyMgKXRnag@mail.gmail.com>
Message-ID: <1354123885.13193.42.camel@sebastian-laptop>

On Wed, 2012-11-28 at 11:11 -0500, Skipper Seabold wrote:
> On Tue, Nov 27, 2012 at 11:16 AM, Sebastian Berg
> <sebastian at sipsolutions.net> wrote:
>         On Mon, 2012-11-26 at 13:54 -0500, Skipper Seabold wrote:
>         > I discovered this because scipy.optimize.fmin_powell appears
>         to
>         > squeeze 1d argmin to 0d unlike the other optimizers, but
>         that's a
>         > different story.
>         >
>         >
>         > I would expect the 0d array to behave like the 1d array not
>         the 2d as
>         > it does below. Thoughts? Maybe too big of a pain to change
>         this
>         > behavior if indeed it's not desired, but I found it to be
>         unexpected.
>         
>         
>         I don't quite understand why it is unexpected. A 1-d array is
>         considered
>         a vector, a 0-d array is a scalar.
>         
>         
> 
> 
> When you put it like this I guess it makes sense. I don't encounter 0d
> arrays often and never think of a 0d array as truly a scalar like
> np.array(1.).item(). See below for my intuition.
>  
I think you should see them as a scalar though for mathematical
operations. The differences are fine in any case, and numpy typically
silently converts scalars -> 0d arrays on function calls and back again
to return scalars.

<snip>
> 
> Maybe I'm misunderstanding. How do you mean there is no broadcasting?

Broadcasting adds dimensions to the start. To handle a vector like a
matrix product in dot, you do not always add the dimension at the start.
For matrix.vector the vector (N,) is much like (N,1). Also the result of
dot is not necessarily 2-d which it should be in your reasoning and if
you think about what happens in broadcasting terms.

>  They're clearly not conformable. Is vector.scalar specially defined
> (I have no idea)? I recall arguing once and submitting a patch such
> that np.linalg.det(5) and np.linalg.inv(5) should be well-defined and
> work but the counter-argument was that a scalar is not the same as a
> scalar matrix. This seems to be an exception.

I do not see an exception, in all cases there is no implicit
(broadcasting like) adding of extra dimensions (leading to an error in
most linear algebra functions if the input is not 2-d) which is good
since "explicit is better then implicit".

> Here, I guess, following that counterargument, I'd expected the scalar
> to fail in dot. I certainly don't expect a (N,2).scalar -> (N,2). Or

If you say dot is strictly a matrix product yes (though it should also
throw errors for vectors then). I think it simply is trying to be more
like the dot that I would write down on paper and thus special cases
vectors and scalars and this generalization only replaces what should
otherwise be an error in a matrix product!

Maybe a strict matrix product would make sense too, but the dot function
behavior cannot be changed in any case, so its pointless to argue about
it. Just make sure your arrays are 2-d (or matrices) if you want a
matrix product, which will give the behavior you expect in a much more
controlled fashion anyway.

>  I'd expect it to follow the rules of matrix notation and be treated
> like the 1d scalar vector so that (N,1).scalar -> (N,). To my mind,
> this follows more closely to the expectation that (J,K).(M,N) ->
> (J,N), i.e., the second dimension of the result is the same as the
> second dimension of whatever is post-multiplying where the first
> dimension is inferred if necessary (or should fail if non-existent).
> So my expectations are (were)
> 
> 
> (N,).() -> (N,)
> (N,1).() -> (N,)
> (N,1).(1,) -> (N,)
> (N,1).(1,1) -> (N,1)
> (N,2).() -> Error
>  
> Skipper
> 
> 
>         > [~]
>         > [279]: arr = np.random.random((25,2))
>         >
>         >
>         > [~/]
>         > [280]: np.dot(arr.squeeze(), np.array(2.)).shape
>         > [280]: (25, 2)
>         >
>         >
>         > Skipper
>         >
>         >
>         
>         > _______________________________________________
>         > NumPy-Discussion mailing list
>         > NumPy-Discussion at scipy.org
>         > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>         
>         
>         _______________________________________________
>         NumPy-Discussion mailing list
>         NumPy-Discussion at scipy.org
>         http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion




From jsseabold at gmail.com  Wed Nov 28 12:41:57 2012
From: jsseabold at gmail.com (Skipper Seabold)
Date: Wed, 28 Nov 2012 12:41:57 -0500
Subject: [Numpy-discussion] result shape from dot for 0d, 1d, 2d scalar
In-Reply-To: <1354123885.13193.42.camel@sebastian-laptop>
References: <CAKF=DjsMN9di=QZKdj_u_OZgJe2kDhAoxqvNOFShwFUVO9f3Tg@mail.gmail.com>
	<1354032991.16990.3.camel@sebastian-laptop>
	<CAKF=Djt78j8F0m7Jy+PhR6yZMtJeKC0Jkm8-DzyoJyMgKXRnag@mail.gmail.com>
	<1354123885.13193.42.camel@sebastian-laptop>
Message-ID: <CAKF=Djt3oSWFRPK6PKhrBn8CFaumpNWz5cX+uB3-uYr+sEaqEQ@mail.gmail.com>

On Wed, Nov 28, 2012 at 12:31 PM, Sebastian Berg <sebastian at sipsolutions.net
> wrote:

> Maybe a strict matrix product would make sense too, but the dot function
> behavior cannot be changed in any case, so its pointless to argue about
> it. Just make sure your arrays are 2-d (or matrices) if you want a
> matrix product, which will give the behavior you expect in a much more
> controlled fashion anyway.
>
>
I'm not arguing anything. I was just stating why I was surprised and was
looking for guidance to update my expectations, which you've provided.
Thanks. Assuring input dimensions is my solution.

Skipper
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121128/d34b5b46/attachment.html>

From jim at jgssebl.net  Wed Nov 28 16:16:10 2012
From: jim at jgssebl.net (Jim O'Brien)
Date: Wed, 28 Nov 2012 14:16:10 -0700
Subject: [Numpy-discussion] Windows installation problem
Message-ID: <C7B2B1D4C0AB4181B7D7B96249A41A89@traveler>

I have tried to install the 1.6.2 win32 superpack on my Windows 7 Pro (64
bit) system which has ActiveState ActivePython 2.7.2.5 (64 bit) installed.
 
However, I get an error that Python 2.7 is required and can't be found in
the Registry.
 
I only need numpy as it is a pre-requisite for another package and numpy is
the only pre-requisite that won't install.
 
Other are specific to Python 2.7 and they install.  Some are Win64 and  some
are Win32.
 
Is there a work around for this?
 
I have no facilities available to build numpy.
 
Regards,
Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121128/ec750f68/attachment.html>

From ralf.gommers at gmail.com  Wed Nov 28 16:32:11 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Wed, 28 Nov 2012 22:32:11 +0100
Subject: [Numpy-discussion] Windows installation problem
In-Reply-To: <C7B2B1D4C0AB4181B7D7B96249A41A89@traveler>
References: <C7B2B1D4C0AB4181B7D7B96249A41A89@traveler>
Message-ID: <CABL7CQiAE7_pHMUhF0HcwcLh2idazQd6wLt2GwKp04ukqu_Csg@mail.gmail.com>

On Wed, Nov 28, 2012 at 10:16 PM, Jim O'Brien <jim at jgssebl.net> wrote:

> **
> I have tried to install the 1.6.2 win32 superpack on my Windows 7 Pro (64
> bit) system which has ActiveState ActivePython 2.7.2.5 (64 bit) installed.
>
> However, I get an error that Python 2.7 is required and can't be found in
> the Registry.
>
> I only need numpy as it is a pre-requisite for another package and numpy
> is the only pre-requisite that won't install.
>
> Other are specific to Python 2.7 and they install.  Some are Win64 and
> some are Win32.
>
> Is there a work around for this?
>

You need to use 64-bit numpy if you have 64-bit Python. You can find one at
http://www.lfd.uci.edu/~gohlke/pythonlibs/.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121128/609b3443/attachment.html>

From jim at jgssebl.net  Wed Nov 28 17:59:20 2012
From: jim at jgssebl.net (Jim O'Brien)
Date: Wed, 28 Nov 2012 15:59:20 -0700
Subject: [Numpy-discussion] Windows installation problem
In-Reply-To: <CABL7CQiAE7_pHMUhF0HcwcLh2idazQd6wLt2GwKp04ukqu_Csg@mail.gmail.com>
References: <C7B2B1D4C0AB4181B7D7B96249A41A89@traveler>
	<CABL7CQiAE7_pHMUhF0HcwcLh2idazQd6wLt2GwKp04ukqu_Csg@mail.gmail.com>
Message-ID: <B81FA29BA51D49BDAA6B7FFA2892C05A@traveler>

Ralf,
 
Thanks.
 
I downloaded the 1.6.2 release for win64 and tried to install.
 
I am still being told that it requires 2.7 and that was not found in the
registry.
 
I know I have Python 2.7 as other packages find it just fine.
 
Is there a way to get around the check that is done by the installer?
 
Regards,
Jim

  _____  

From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Ralf Gommers
Sent: Wed, Nov 28, 2012 2:32 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Windows installation problem





On Wed, Nov 28, 2012 at 10:16 PM, Jim O'Brien <jim at jgssebl.net> wrote:



I have tried to install the 1.6.2 win32 superpack on my Windows 7 Pro (64
bit) system which has ActiveState ActivePython 2.7.2.5 (64 bit) installed.
 
However, I get an error that Python 2.7 is required and can't be found in
the Registry.
 
I only need numpy as it is a pre-requisite for another package and numpy is
the only pre-requisite that won't install.
 
Other are specific to Python 2.7 and they install.  Some are Win64 and  some
are Win32.
 
Is there a work around for this?


You need to use 64-bit numpy if you have 64-bit Python. You can find one at
http://www.lfd.uci.edu/~gohlke/pythonlibs/.

Ralf
 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121128/0ca355b1/attachment.html>

From jim at jgssebl.net  Wed Nov 28 18:04:16 2012
From: jim at jgssebl.net (Jim O'Brien)
Date: Wed, 28 Nov 2012 16:04:16 -0700
Subject: [Numpy-discussion] Windows installation problem
In-Reply-To: <CABL7CQiAE7_pHMUhF0HcwcLh2idazQd6wLt2GwKp04ukqu_Csg@mail.gmail.com>
References: <C7B2B1D4C0AB4181B7D7B96249A41A89@traveler>
	<CABL7CQiAE7_pHMUhF0HcwcLh2idazQd6wLt2GwKp04ukqu_Csg@mail.gmail.com>
Message-ID: <C4D8119CF36A4B608EA377A895EB1331@traveler>

Forget the last post. 
 
I was one the wrong machine!
 
The 64 bit release installed fine.
 
Regards,
Jim

  _____  

From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Ralf Gommers
Sent: Wed, Nov 28, 2012 2:32 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Windows installation problem





On Wed, Nov 28, 2012 at 10:16 PM, Jim O'Brien <jim at jgssebl.net> wrote:



I have tried to install the 1.6.2 win32 superpack on my Windows 7 Pro (64
bit) system which has ActiveState ActivePython 2.7.2.5 (64 bit) installed.
 
However, I get an error that Python 2.7 is required and can't be found in
the Registry.
 
I only need numpy as it is a pre-requisite for another package and numpy is
the only pre-requisite that won't install.
 
Other are specific to Python 2.7 and they install.  Some are Win64 and  some
are Win32.
 
Is there a work around for this?


You need to use 64-bit numpy if you have 64-bit Python. You can find one at
http://www.lfd.uci.edu/~gohlke/pythonlibs/.

Ralf
 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121128/bb1e475e/attachment.html>

From rblove_lists at comcast.net  Wed Nov 28 19:21:08 2012
From: rblove_lists at comcast.net (Robert Love)
Date: Wed, 28 Nov 2012 18:21:08 -0600
Subject: [Numpy-discussion] Simple Loadtxt question
Message-ID: <BB6C38B5-E96C-43E9-9E75-2C6AF22C0374@comcast.net>

I have a file with thousands of lines like this:

Signal was returned in 204 microseconds
Signal was returned in 184 microseconds
Signal was returned in 199 microseconds
Signal was returned in 4274 microseconds
Signal was returned in 202 microseconds
Signal was returned in 189 microseconds

I try to read it like this:


data = np.loadtxt('dummy.data', dtype={'names':('label','times','musec'), 'fmts':('|S23','i8','|S13')})

It fails, I think, because it wants a string format and field for each of the words 'Signal' 'was' 'returned' etc.

Can I make it treat that whole string before the number as one string, one field?  All I really care about is the numbers anyway.

Any advice appreciated.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121128/d2f0912f/attachment.html>

From derek at astro.physik.uni-goettingen.de  Wed Nov 28 19:29:27 2012
From: derek at astro.physik.uni-goettingen.de (Derek Homeier)
Date: Thu, 29 Nov 2012 01:29:27 +0100
Subject: [Numpy-discussion] Simple Loadtxt question
In-Reply-To: <BB6C38B5-E96C-43E9-9E75-2C6AF22C0374@comcast.net>
References: <BB6C38B5-E96C-43E9-9E75-2C6AF22C0374@comcast.net>
Message-ID: <863083A1-C4A3-4D27-B8CA-D2E92C11BEC5@astro.physik.uni-goettingen.de>

On 29.11.2012, at 1:21AM, Robert Love wrote:

> I have a file with thousands of lines like this:
> 
> Signal was returned in 204 microseconds
> Signal was returned in 184 microseconds
> Signal was returned in 199 microseconds
> Signal was returned in 4274 microseconds
> Signal was returned in 202 microseconds
> Signal was returned in 189 microseconds
> 
> I try to read it like this:
> 
> 
> data = np.loadtxt('dummy.data', dtype={'names':('label','times','musec'), 'fmts':('|S23','i8','|S13')})
> 
> It fails, I think, because it wants a string format and field for each of the words 'Signal' 'was' 'returned' etc.
> 
> Can I make it treat that whole string before the number as one string, one field?  All I really care about is the numbers anyway.
> 
Then how about 

np.loadtxt('dummy.data', usecols=(4, ))

Cheers,
					Derek



From denis-bz-gg at t-online.de  Thu Nov 29 06:52:17 2012
From: denis-bz-gg at t-online.de (denis)
Date: Thu, 29 Nov 2012 11:52:17 +0000 (UTC)
Subject: [Numpy-discussion] install numpy 1.6.2 .dmg on macosx 10.7,
	check for python 2.7
Message-ID: <loom.20121129T124459-287@post.gmane.org>

Trying to install numpy 1.6.2 on a mac osx 10.7.4 from this .dmg
    9323135 numpy-1.6.2-py2.7-python.org-macosx10.3.dmg
gives
    "numpy 1.6.2 can't be installed on this disk.
    numpy requires python.org Python 2.7 to install."

But python 2.7.3 *is* installed from python.org
    18761950 python-2.7.3-macosx10.6.dmg
    and /usr/bin/python is linked as described in
    http://wolfpaulus.com/journal/mac/installing_python_osx
python -c 'import sys; print sys.version'
    2.7.3 (v2.7.3:70274d53c1dd, Apr  9 2012, 20:52:43)
    [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]

How is the code in the .dmg checking for python.org python ?

Thanks,
cheers
  -- denis



From robert.kern at gmail.com  Thu Nov 29 07:37:14 2012
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 29 Nov 2012 13:37:14 +0100
Subject: [Numpy-discussion] install numpy 1.6.2 .dmg on macosx 10.7,
 check for python 2.7
In-Reply-To: <loom.20121129T124459-287@post.gmane.org>
References: <loom.20121129T124459-287@post.gmane.org>
Message-ID: <CAF6FJiukO5r-7KAvTx67ZZobWzkg-UJjosbUmqVE8BLryJSBmg@mail.gmail.com>

On Thu, Nov 29, 2012 at 12:52 PM, denis <denis-bz-gg at t-online.de> wrote:
> Trying to install numpy 1.6.2 on a mac osx 10.7.4 from this .dmg
>     9323135 numpy-1.6.2-py2.7-python.org-macosx10.3.dmg
> gives
>     "numpy 1.6.2 can't be installed on this disk.
>     numpy requires python.org Python 2.7 to install."
>
> But python 2.7.3 *is* installed from python.org
>     18761950 python-2.7.3-macosx10.6.dmg
>     and /usr/bin/python is linked as described in
>     http://wolfpaulus.com/journal/mac/installing_python_osx

Ouch! I'm sorry, but these are wildly *dangerous* instructions. Please
disregard them. The author is reckless.

You can't undo the damage without reinstalling OS X, but there is a
good chance you will not run into the parts of OS X that depended on
the /System Python being unmolested. However, you do need to reinstall
the python.org Python framework again. Just don't mv it from where it
gets installed. Then the numpy-1.6.2-py2.7-python.org-macosx10.3.dmg
will recognize it.

--
Robert Kern


From denis-bz-gg at t-online.de  Thu Nov 29 10:05:10 2012
From: denis-bz-gg at t-online.de (denis)
Date: Thu, 29 Nov 2012 15:05:10 +0000 (UTC)
Subject: [Numpy-discussion] install numpy 1.6.2 .dmg on macosx 10.7,
	check for python 2.7
References: <loom.20121129T124459-287@post.gmane.org>
	<CAF6FJiukO5r-7KAvTx67ZZobWzkg-UJjosbUmqVE8BLryJSBmg@mail.gmail.com>
Message-ID: <loom.20121129T160211-226@post.gmane.org>

Robert Kern <robert.kern <at> gmail.com> writes:
...
> >     http://wolfpaulus.com/journal/mac/installing_python_osx
> 
> Ouch! I'm sorry, but these are wildly *dangerous* instructions. Please
> disregard them. The author is reckless.

Thank you Robert, that worked just fine
(for scipy and matplotlib .dmg s too)
cheers
  -- denis



From jim22k at gmail.com  Thu Nov 29 11:24:56 2012
From: jim22k at gmail.com (Jim Kitchen)
Date: Thu, 29 Nov 2012 11:24:56 -0500
Subject: [Numpy-discussion] Truth value of ndarray not Pythonic
Message-ID: <CAN3FdhmtrTWwuvLBUT8sOd3vawmwbqHFR7du_2uXinOKna+-2w@mail.gmail.com>

I understand the historical reason for the "Truth value of an array" error,
avoiding the pitfalls of:
>>> a = np.arange(10)
>>> if a==0: # this is ambiguous, use any() or all()

I also understand the issues with logical and/or:
>>> if (a<10 and a > 5): # this will break due to "and" vs "&"

However the main point in this thread from 3 years
ago<http://thread.gmane.org/gmane.comp.python.numeric.general/32653>is
very valid.  If I write code that uses lists and then convert that to
an
array for efficiency or more powerful computation, I have my own pitfall
trying to do:
>>> if a: # why doesn't this just check for size?

My Proposal
------------------
It seems to me that an elegant solution to this dilemma is to separate the
behavior of ndarrays of type bool from all other ndarrays.  Keep the
current behavior for ndarrays of type bool, but let the __nonzero__ for all
other ndarrays be based on size.

>>> if a==0: # still raises Truth error because it's of dtype bool

>>> if (a<10 and a>5): # still raises Truth error because it's of dtype bool

>>> if a: # works fine because dtype is int64

This solution avoids all the primary pitfalls of ambiguity where someone
would need any() or all() because they're working with bools at the element
level.  But for cases where a function may return data or None, I really
like to use the normal Python truth test for that instead of:
>>> if a is not None and len(a) > 0: # that was a chore to find out

The only problem I see with this solution is with the case of the
single-element array.
>>> s = np.array([[0]])
>>> if s: # works today, returns False

With my proposal,
>>> if s: # still works, but returns True because array is not empty

It's a wart to be sure, but it would make ndarrays much easier to work with
when converting from standard Python containers.  Maybe we need something
like this (probably not possible):
>>> from numpy.__future__ import truthiness

I've especially found this Truth error a challenge converting from
dictionaries of lists to pandas DataFrames.  It raises the same error,
tracing back to this ambiguity in ndarrays.  If it's too big of a change to
make for ndarrays, maybe the same proposal could be implemented in pandas.

Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121129/fbee9515/attachment.html>

From njs at pobox.com  Thu Nov 29 14:19:08 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 29 Nov 2012 19:19:08 +0000
Subject: [Numpy-discussion] Truth value of ndarray not Pythonic
In-Reply-To: <CAN3FdhmtrTWwuvLBUT8sOd3vawmwbqHFR7du_2uXinOKna+-2w@mail.gmail.com>
References: <CAN3FdhmtrTWwuvLBUT8sOd3vawmwbqHFR7du_2uXinOKna+-2w@mail.gmail.com>
Message-ID: <CAPJVwB=X0msB-1uu-r3-cgRaEKP6C26GpuE0kdv+RtdB86UjBw@mail.gmail.com>

On 29 Nov 2012 16:27, "Jim Kitchen" <jim22k at gmail.com> wrote:
>
> I understand the historical reason for the "Truth value of an array" error, avoiding the pitfalls of:
> >>> a = np.arange(10)
> >>> if a==0: # this is ambiguous, use any() or all()
>
> I also understand the issues with logical and/or:
> >>> if (a<10 and a > 5): # this will break due to "and" vs "&"
>
> However the main point in this thread from 3 years ago is very valid.  If I write code that uses lists and then convert that to an array for efficiency or more powerful computation, I have my own pitfall trying to do:
> >>> if a: # why doesn't this just check for size?
>
> My Proposal
> ------------------
> It seems to me that an elegant solution to this dilemma is to separate the behavior of ndarrays of type bool from all other ndarrays.  Keep the current behavior for ndarrays of type bool, but let the __nonzero__ for all other ndarrays be based on size.
>
> >>> if a==0: # still raises Truth error because it's of dtype bool
>
> >>> if (a<10 and a>5): # still raises Truth error because it's of dtype bool
>
> >>> if a: # works fine because dtype is int64

I see what you mean, but I think this change would be dangerously
confusing. The problem is that an ndarray of ints follows the
conventions of both Python lists and Python ints. E.g., it acts like a
list for len() and iteration, but like an int for arithmetic (+ does
addition, not concatenation). So your suggestion makes sense if you're
thinking of the list analogy first, but there are other people out
there who are going to think of the int analogy first instead. Python
has two very well established conventions for how boolean casting
works (emptiness for containers, zero-ness for scalars), and they
conflict here. The current behaviour follows the scalar convention
when possible, and almost always throws a safe error if people wrote
code expecting the container. Your version would be just as confusing
to a different set of people, but wouldn't even fail safe by raising
an error, it would just silently do the wrong thing.

> This solution avoids all the primary pitfalls of ambiguity where someone would need any() or all() because they're working with bools at the element level.  But for cases where a function may return data or None, I really like to use the normal Python truth test for that instead of:
> >>> if a is not None and len(a) > 0: # that was a chore to find out
>
> The only problem I see with this solution is with the case of the single-element array.
> >>> s = np.array([[0]])
> >>> if s: # works today, returns False
>
> With my proposal,
> >>> if s: # still works, but returns True because array is not empty

This would break huge amounts of already existing code. So that means
that to get this change through, you'd have to not just convince
everyone that it was a good idea, but that bit was such an improvement
that it'd be worth auditing all that code (and it's not even
greppable).

It strikes me as unusual, though, that you're testing for both None
and emptiness and treating them the same in your if statement. If your
function is returning an empty array as a 'special' value to signal
that something funny has happened, then perhaps it could just return
None in this case instead? If it's returning an empty array as an
ordinary value (e.g. when you happen to have zero data points that
fall into some category), then usually you don't need to check for
this explicitly, since numpy functions like sum() etc. will do the
right thing? Of course you might have some situation where everything
happens to line up so that what you wrote is the best solution, but
you might want to revisit it to check. Or post a longer example here
and see if anyone has suggestions for how to make it more
"numpythonic".

Hope that helps,
-n

> It's a wart to be sure, but it would make ndarrays much easier to work with when converting from standard Python containers.  Maybe we need something like this (probably not possible):
> >>> from numpy.__future__ import truthiness
>
> I've especially found this Truth error a challenge converting from dictionaries of lists to pandas DataFrames.  It raises the same error, tracing back to this ambiguity in ndarrays.  If it's too big of a change to make for ndarrays, maybe the same proposal could be implemented in pandas.
>
> Jim
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From rowen at uw.edu  Thu Nov 29 14:25:25 2012
From: rowen at uw.edu (Russell E. Owen)
Date: Thu, 29 Nov 2012 11:25:25 -0800
Subject: [Numpy-discussion] install numpy 1.6.2 .dmg on macosx 10.7,
	check for python 2.7
References: <loom.20121129T124459-287@post.gmane.org>
Message-ID: <rowen-67F4A9.11252529112012@news.gmane.org>

In article <loom.20121129T124459-287 at post.gmane.org>,
 denis <denis-bz-gg at t-online.de> wrote:

> Trying to install numpy 1.6.2 on a mac osx 10.7.4 from this .dmg
>     9323135 numpy-1.6.2-py2.7-python.org-macosx10.3.dmg
> gives
>     "numpy 1.6.2 can't be installed on this disk.
>     numpy requires python.org Python 2.7 to install."
> 
> But python 2.7.3 *is* installed from python.org
>     18761950 python-2.7.3-macosx10.6.dmg
>     and /usr/bin/python is linked as described in
>     http://wolfpaulus.com/journal/mac/installing_python_osx
> python -c 'import sys; print sys.version'
>     2.7.3 (v2.7.3:70274d53c1dd, Apr  9 2012, 20:52:43)
>     [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]

These are not compatible. You have installed python.org's macosx10.6 
installed (a very reasonable choice). But you must install binary 
packages labelled 10.6, never packages labelled 10.3 (which are only for 
the 10.3 version of python.org's python).

I'm glad you got an error message (however opaque), since if the install 
had succeeded the results would not have worked.

-- Russell

P.S. the difference is:
- 10.6 (which requires MacOS X 10.6 or later) is 64-bit and requires 
intel
- 10.3 (which requires MacOS X 10.3.9 or later) is 32-bit and includes 
PPC support



From chris.barker at noaa.gov  Thu Nov 29 16:54:57 2012
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Thu, 29 Nov 2012 13:54:57 -0800
Subject: [Numpy-discussion] install numpy 1.6.2 .dmg on macosx 10.7,
 check for python 2.7
In-Reply-To: <rowen-67F4A9.11252529112012@news.gmane.org>
References: <loom.20121129T124459-287@post.gmane.org>
	<rowen-67F4A9.11252529112012@news.gmane.org>
Message-ID: <-4379392438098142818@unknownmsgid>

On Nov 29, 2012, at 11:25 AM, "Russell E. Owen" <rowen at uw.edu> wrote:

>
>
> P.S. the difference is:
> - 10.6 (which requires MacOS X 10.6 or later) is 64-bit and requires
>
Slight clarification -- the 10.6 build is 32 and 64 bit Intel.

-Chris


From martin.raspaud at smhi.se  Fri Nov 30 03:07:11 2012
From: martin.raspaud at smhi.se (Martin Raspaud)
Date: Fri, 30 Nov 2012 09:07:11 +0100
Subject: [Numpy-discussion] allclose changed behaviour in 1.6.2 ?
Message-ID: <50B8692F.4050706@smhi.se>

Hi,

We noticed that comparing arrays of different shapes with allclose
doesn't work anymore in numpy 1.6.2.

Is this a feature or a bug ? :)

See the output in both 1.6.1 and 1.6.2 at the end of this mail.

Best regards,
Martin

1.6.1::

 In [1]: import numpy as np

 In [2]: np.__version__
 Out[2]: '1.6.1'

 In [3]: a = np.array([1, 2, 3])

 In [4]: b = np.array([1, 2, 3, 4])

 In [5]: np.allclose(a, b)
 Out[5]: False


1.6.2::

 In[1]: import numpy as np

 In[2]: np.__version__
 Out[2]: '1.6.2'

 In [3]: a = np.array([1, 2, 3])

 In[4]: b = np.array([1, 2, 3, 4])

 In[5]: np.allclose(a, b)
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File
"/home/maarten/pytroll/local/lib/python2.7/site-packages/numpy-1.6.2-py2.7-linux-x86_64.egg/numpy/core/numeric.py",
line 1936, in allclose
     return all(less_equal(abs(x-y), atol + rtol * abs(y)))
 ValueError: operands could not be broadcast together with shapes (3) (4)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: martin_raspaud.vcf
Type: text/x-vcard
Size: 303 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121130/2ac51866/attachment.vcf>

From brad.froehle at gmail.com  Fri Nov 30 15:13:58 2012
From: brad.froehle at gmail.com (Bradley M. Froehle)
Date: Fri, 30 Nov 2012 12:13:58 -0800
Subject: [Numpy-discussion] site.cfg: Custom BLAS / LAPACK configuration
Message-ID: <CAHXv-MjuNXNCry3W4b+wL-7QshNs3C=yzCvDvJTddwSdMo4QEg@mail.gmail.com>

I recently installed NumPy 1.6.2 on a new computer and wanted to use ACML
as the BLAS/LAPACK library.  [I'm aware that ACML doesn't provide CBLAS,
but that is easy to work around by compiling it yourself to produce
libcblas.a or libcblas.so].

I experienced a great bit of difficulty in getting NumPy to use ACML
(-lcblas -lacml), primarily stemming from the fact that there was a working
ATLAS installation already in /usr/lib64.

As far as I can tell, it's IMPOSSIBLE to create a site.cfg which will link
to ACML when a system installed ATLAS is present.

The detection routine for blas_opt (and similarly for lapack_opt) seem to
operate as:
* Is MKL present?  If so, use it.
* Is ATLAS present? If so, use it.
* Use [blas] section from site.cfg.

Instead I would have expected the detection routine to be more like:
* Is [blas_opt] present in site.cfg? If so, use it.
* Is MKL present? ...
* Is ATLAS present? ...
* Use [blas] section from site.cfg.

This is not just a problem with ACML.  I've also experienced this when
using NumPy on some cray supercomputers where the default C compiler
automatically links a preferred BLAS/LAPACK.

I created a GitHub issue for this:
https://github.com/numpy/numpy/issues/2728.  In addition, I created a pull
request with a "works for me" solution, but which should have needs some
wider visibility https://github.com/numpy/numpy/pull/2751.

I'd appreciate any reviews, workarounds, or other general feedback.  If you
want to test our the library detection mechanism you can run the following
from within the NumPy source directory::

    import __builtin__ as builtins
    builtins.__NUMPY_SETUP__ = True
    import numpy.distutils.system_info as si
    print si.get_info('blas_opt')

Thanks,
Brad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121130/dfc759bb/attachment.html>