From pav at iki.fi  Fri Apr  1 04:12:11 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 1 Apr 2011 08:12:11 +0000 (UTC)
Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug?
References: <AANLkTinSTXcSv9pqDXmth4Jxr1uX0XcO4vpRrW8Qyqgp@mail.gmail.com>
	<AANLkTikoeR+2_u9cgzzJRJ7584FZOUKYW+F4cLirWt3=@mail.gmail.com>
	<AANLkTi=oOHWzL13Z2VZ-yO7haHtQbXcNAaixYqpDU6yu@mail.gmail.com>
	<AANLkTik-aQ3RfQYu7Vz4g1o1=-Z7pvZBasGz5GayaN=D@mail.gmail.com>
	<AANLkTi=H4nC1Fg8CiJSksfKG2x-Q-qhbLMsBKRmBNen8@mail.gmail.com>
Message-ID: <in41cr$429$1@dough.gmane.org>

Thu, 31 Mar 2011 11:15:08 -0700, Mark Wiebe wrote:
[clip]
> My reading of Pauli's thoughts was that putting it in unilaterally is
> undesirable, something I definitely agree with. I think with Eli doing
> the legwork of getting input and acceptance from the relevant parties,
> we should help him out as much as possible.

Yes, this was the main concern. However, reading the thread on 
python-ideas, apart from the choice of the letter there does not seem to 
be much resistance, especially as we have a volunteer for implementing 
any changes required. :)

> Not getting this change into 1.6 makes the Cython support much more
> difficult because of their design based around the buffer protocol.
>
> Looking at the thread on python-users, I see mostly concern that the
> type be standard or have precedent elsewhere, which as an IEEE standard
> it definitely satisfies. The other question is on the chosen letter - we
> picked 'e' as 'h' was taken and it is close to 'f' and 'd', the other
> float types. If a letter change is deemed necessary, it would be good to
> get that into 1.6 as well, since this kind of change is easy now, but
> much more difficult later.

The buffer interface format string does not need to match what Numpy 
internally uses (although that would be clearer), so changing that later 
on, if the Python devs strongly disagree with the choice made, maybe does 
not have *too* much inertia. However, it could be good to come to an 
agreement soon, if possible.

	Pauli



From ndbecker2 at gmail.com  Fri Apr  1 09:18:22 2011
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 01 Apr 2011 09:18:22 -0400
Subject: [Numpy-discussion] can't find doc for as_strided
Message-ID: <in4jav$kph$1@dough.gmane.org>

Visiting http://docs.scipy.org/doc/numpy/reference/, as search for

as_strided

or

stride_tricks

shows nothing (useful).

For that matter, I don't see a reference to numpy.lib.



From dsdale24 at gmail.com  Fri Apr  1 14:19:52 2011
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 1 Apr 2011 14:19:52 -0400
Subject: [Numpy-discussion] nearing a milestone
Message-ID: <AANLkTikrZnKZ-Upq6JztBAj9GvUvrRyLeYUmcV_BuuTB@mail.gmail.com>

Numpy is nearing a milestone:
http://sourceforge.net/projects/numpy/files/NumPy/stats/timeline?dates=2007-09-25+to+2011-04-01


From ben.root at ou.edu  Fri Apr  1 14:24:27 2011
From: ben.root at ou.edu (Benjamin Root)
Date: Fri, 1 Apr 2011 13:24:27 -0500
Subject: [Numpy-discussion] nearing a milestone
In-Reply-To: <AANLkTikrZnKZ-Upq6JztBAj9GvUvrRyLeYUmcV_BuuTB@mail.gmail.com>
References: <AANLkTikrZnKZ-Upq6JztBAj9GvUvrRyLeYUmcV_BuuTB@mail.gmail.com>
Message-ID: <AANLkTin4e0aO3rM0dduJQCGXjhhj-WMB6iLETRgAVinC@mail.gmail.com>

Yeah, but they have been downhill since November...

On Fri, Apr 1, 2011 at 1:19 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> Numpy is nearing a milestone:
>
> http://sourceforge.net/projects/numpy/files/NumPy/stats/timeline?dates=2007-09-25+to+2011-04-01
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110401/859a1b24/attachment.html>

From ralf.gommers at googlemail.com  Fri Apr  1 15:41:54 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Fri, 1 Apr 2011 21:41:54 +0200
Subject: [Numpy-discussion] can't find doc for as_strided
In-Reply-To: <in4jav$kph$1@dough.gmane.org>
References: <in4jav$kph$1@dough.gmane.org>
Message-ID: <AANLkTi=BDPbwe4mA0DzHZPF0v7nSz428NBuRnJT4U2to@mail.gmail.com>

On Fri, Apr 1, 2011 at 3:18 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> Visiting http://docs.scipy.org/doc/numpy/reference/, as search for
>
> as_strided
>
> or
>
> stride_tricks
>
> shows nothing (useful).

That's because as_strided is not exposed. It's a private function (see
commit f912322e) used in broadcast_arrays, which is there:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast_arrays.html

>
> For that matter, I don't see a reference to numpy.lib.

Things in numpy.lib that are not private are available in the numpy
namespace (see numpy.lib.function_base.histogram in the doc wiki vs.
numpy.histogram in the reference guide).

Ralf


From ralf.gommers at googlemail.com  Fri Apr  1 15:51:54 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Fri, 1 Apr 2011 21:51:54 +0200
Subject: [Numpy-discussion] np.histogramdd of empty data
In-Reply-To: <AANLkTinB=eprmhTm1Y6+1W2Jg6-y_tCGErCkjWfjwJNw@mail.gmail.com>
References: <4D9435C8.5040709@amolf.nl>
	<AANLkTinZ+P8OiH+YppfDDodPhoamELoCHrTy657Gg+qv@mail.gmail.com>
	<4D94586D.4080300@amolf.nl>
	<AANLkTi=hOTS-i-Ly1nQz2LBNjFMqy6VesEwU-iGoS8Tr@mail.gmail.com>
	<AANLkTikO-6pMAJ04kGms5TpByg8G8=tLdMGsjtC3Sye_@mail.gmail.com>
	<AANLkTinB=eprmhTm1Y6+1W2Jg6-y_tCGErCkjWfjwJNw@mail.gmail.com>
Message-ID: <AANLkTinqW6+tdSTnFJ_gaCqMCfOFAGUW8qbvkAYKHxVy@mail.gmail.com>

On Thu, Mar 31, 2011 at 8:00 PM, Benjamin Root <ben.root at ou.edu> wrote:
>
>
> On Thu, Mar 31, 2011 at 12:32 PM, Ralf Gommers <ralf.gommers at googlemail.com>
> wrote:
>>
>> ---------- Forwarded message ----------
>> From: Ralf Gommers <ralf.gommers at googlemail.com>
>> Date: Thu, Mar 31, 2011 at 7:31 PM
>> Subject: Re: [Numpy-discussion] np.histogramdd of empty data
>> To: Nils Becker <n.becker at amolf.nl>
>>
>>
>> On Thu, Mar 31, 2011 at 12:33 PM, Nils Becker <n.becker at amolf.nl> wrote:
>> > Hi Ralf,
>> >
>> > I cloned numpy/master and played around a little.
>> >
>> > when giving the bins explicitely, now histogram2d and histogramdd work
>> > as expected in all tests i tried.
>> >
>> >
>> > However, some of the cases with missing bin specification appear
>> > somewhat inconsistent.
>> >
>> > The first question is if creating arbitrary bins for empty data and
>> > empty bin specification is better than raising an Exception:
>> >
>> > Specifically:
>>
>> Bins of size 0 should give a meaningful error, I was just fixing that
>> as part of #1788 in
>> https://github.com/rgommers/numpy/tree/ticket-1788-histogramdd
>>
>> > numpy.histogram2d([],[],bins=[0,0])
>> >> (array([ 0., ?0.]), array([ 0.]), array([ 0.]))
>>
>> Now gives:
>> ? ?ValueError: Element at index 0 in `bins` should be a positive integer.
>>
>> > numpy.histogram([],bins=0)
>> >> ValueError: zero-size array to minimum.reduce without identity
>>
>> Now gives:
>> ? ?ValueError: `bins` should be a positive integer.
>>
>>
>> > so 1-d and 2-d behave not quite the same.
>> >
>> > also, these work (although with arbitrary bin edges):
>> >
>> > numpy.histogram2d([],[],bins=[1,1])
>> >> (array([ 0., ?0.]), array([ 0., ?1.]), array([ 0., ?1.]))
>> >
>> > numpy.histogram2d([],[],bins=[0,1])
>> >> (array([ 0., ?0.]), array([ 0.]), array([ 0., ?1.]))
>> >
>> > while this raises an error:
>> >
>> > numpy.histogram([],bins=1)
>> >> ValueError: zero-size array to minimum.reduce without identity
>>
>> Now gives:
>> ? ?(array([0]), array([ 0., ?1.]))
>>
>
> Just for consistency's sake, maybe the same should be done for np.bincount()
> and np.digitize() for empty data (but known bins)?? I don't know if your fix
> to histogram does this or not, but the latest pull from numpy master doesn't
> do this.

It doesn't. There was a ticket for that already, #1387. I added the
desired behavior there. Since those functions are written in C it's a
little more time-consuming to fix.

Ralf


From seb.haase at gmail.com  Fri Apr  1 15:51:52 2011
From: seb.haase at gmail.com (Sebastian Haase)
Date: Fri, 1 Apr 2011 21:51:52 +0200
Subject: [Numpy-discussion] nearing a milestone
In-Reply-To: <AANLkTin4e0aO3rM0dduJQCGXjhhj-WMB6iLETRgAVinC@mail.gmail.com>
References: <AANLkTikrZnKZ-Upq6JztBAj9GvUvrRyLeYUmcV_BuuTB@mail.gmail.com>
	<AANLkTin4e0aO3rM0dduJQCGXjhhj-WMB6iLETRgAVinC@mail.gmail.com>
Message-ID: <AANLkTinMTJ02=PrOePSv3DuJJcVmMUhx=vsYmgx-jE+K@mail.gmail.com>

I think he is talking about 1e6 downloads total.
-S.


On Fri, Apr 1, 2011 at 8:24 PM, Benjamin Root <ben.root at ou.edu> wrote:
> Yeah, but they have been downhill since November...
>
> On Fri, Apr 1, 2011 at 1:19 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>>
>> Numpy is nearing a milestone:
>>
>> http://sourceforge.net/projects/numpy/files/NumPy/stats/timeline?dates=2007-09-25+to+2011-04-01
>> _______________________________________________


From ben.root at ou.edu  Fri Apr  1 16:08:24 2011
From: ben.root at ou.edu (Benjamin Root)
Date: Fri, 1 Apr 2011 15:08:24 -0500
Subject: [Numpy-discussion] nearing a milestone
In-Reply-To: <AANLkTinMTJ02=PrOePSv3DuJJcVmMUhx=vsYmgx-jE+K@mail.gmail.com>
References: <AANLkTikrZnKZ-Upq6JztBAj9GvUvrRyLeYUmcV_BuuTB@mail.gmail.com>
	<AANLkTin4e0aO3rM0dduJQCGXjhhj-WMB6iLETRgAVinC@mail.gmail.com>
	<AANLkTinMTJ02=PrOePSv3DuJJcVmMUhx=vsYmgx-jE+K@mail.gmail.com>
Message-ID: <AANLkTinTBMkw-9FMHeH2KmOvBgJcCYc9FASvXQvOOiuS@mail.gmail.com>

Whoosh!

Ben Root

P.S. -- In case it is needed to be said, that is 1e6 downloads from
sourceforge only.  NumPy is now on github...

Happy April Fools!


On Fri, Apr 1, 2011 at 2:51 PM, Sebastian Haase <seb.haase at gmail.com> wrote:

> I think he is talking about 1e6 downloads total.
> -S.
>
>
> On Fri, Apr 1, 2011 at 8:24 PM, Benjamin Root <ben.root at ou.edu> wrote:
> > Yeah, but they have been downhill since November...
> >
> > On Fri, Apr 1, 2011 at 1:19 PM, Darren Dale <dsdale24 at gmail.com> wrote:
> >>
> >> Numpy is nearing a milestone:
> >>
> >>
> http://sourceforge.net/projects/numpy/files/NumPy/stats/timeline?dates=2007-09-25+to+2011-04-01
> >> _______________________________________________
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110401/7bcac1a2/attachment.html>

From dsdale24 at gmail.com  Fri Apr  1 16:32:21 2011
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 1 Apr 2011 16:32:21 -0400
Subject: [Numpy-discussion] nearing a milestone
In-Reply-To: <AANLkTinTBMkw-9FMHeH2KmOvBgJcCYc9FASvXQvOOiuS@mail.gmail.com>
References: <AANLkTikrZnKZ-Upq6JztBAj9GvUvrRyLeYUmcV_BuuTB@mail.gmail.com>
	<AANLkTin4e0aO3rM0dduJQCGXjhhj-WMB6iLETRgAVinC@mail.gmail.com>
	<AANLkTinMTJ02=PrOePSv3DuJJcVmMUhx=vsYmgx-jE+K@mail.gmail.com>
	<AANLkTinTBMkw-9FMHeH2KmOvBgJcCYc9FASvXQvOOiuS@mail.gmail.com>
Message-ID: <AANLkTinbuW094vwxsOK_1KH=Pyw12sDo3UtONUgiuMES@mail.gmail.com>

On Fri, Apr 1, 2011 at 4:08 PM, Benjamin Root <ben.root at ou.edu> wrote:
> Whoosh!
>
> Ben Root
>
> P.S. -- In case it is needed to be said, that is 1e6 downloads from
> sourceforge only.? NumPy is now on github...

The releases are still distributed through sourceforge.

Maybe the SciPy2011 Program Committee could prepare a mounted gold
record to mark the occasion. I could probably dig up and donate an old
copy of the Footloose soundtrack.

Darren


From ralf.gommers at googlemail.com  Fri Apr  1 16:56:07 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Fri, 1 Apr 2011 22:56:07 +0200
Subject: [Numpy-discussion] 1.6.0b1 installs stuff in
	/usr/lib/python2.7/site-packages/doc
In-Reply-To: <AANLkTinG5NA8kV_qfZZ8utdK3Pn78eTsNrXkOEZVVyD8@mail.gmail.com>
References: <4D94ACDC.40701@cora.nwra.com>
	<AANLkTinE8=QLOcMLNdfVRYZy0_7P1b1QHpFuPgzbnzy9@mail.gmail.com>
	<AANLkTim9RBiym0WhpdtT4A5-Z_6t1j06vwsWxgUyq0zx@mail.gmail.com>
	<AANLkTi=BzEFjUPP4DX1vTUKixP13asU-dAPs7vNU-jv5@mail.gmail.com>
	<AANLkTimiwaE+u9wZ9dEUbnRO+6kXhdaCnHfk_PVVLfnU@mail.gmail.com>
	<AANLkTinoxEzPCSoOSsPV4vTcg17B1ET8peWr1ueM_XoP@mail.gmail.com>
	<AANLkTinKkgPXESRhn93gn1xg5toC+MPdEE5=PJGTcTuS@mail.gmail.com>
	<4D94D304.1050401@cora.nwra.com>
	<AANLkTinG5NA8kV_qfZZ8utdK3Pn78eTsNrXkOEZVVyD8@mail.gmail.com>
Message-ID: <AANLkTikaxKxcFwri7_U1yWi3QY=4H6X+kwD8J05skB97@mail.gmail.com>

On Thu, Mar 31, 2011 at 9:21 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Mar 31, 2011 at 14:16, Orion Poplawski <orion at cora.nwra.com> wrote:
>> On 03/31/2011 01:12 PM, Robert Kern wrote:
>>> Well, they're meant to be copied into your own code, which is why they
>>> end up under a doc/ directory. Lots of things like this tend to end up
>>> in doc/ directories.
>>
>> perhaps /usr/share/doc/numpy then.
>
> This is up to the various Linux packagers of numpy. They have their
> own tools to do this according to their own policies. Doing *anything*
> in our setup.py to install these files will just make their jobs
> harder. That's why I think that this needs to be handled in the binary
> installers, not the setup.py.

Okay, makes sense. For now I'll just delete those lines from setup.py
then and open a ticket. Will try to find some time later to add this
for the binary installers for Windows and OS X. People who do a source
install will just have to grab those files from a tarball then.

Ralf


From mwwiebe at gmail.com  Fri Apr  1 20:28:53 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Fri, 1 Apr 2011 17:28:53 -0700
Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug?
In-Reply-To: <AANLkTin+1TNx7u5ftJ2+fT4GgWnKQAcJoo-1EM7C5MN-@mail.gmail.com>
References: <AANLkTinSTXcSv9pqDXmth4Jxr1uX0XcO4vpRrW8Qyqgp@mail.gmail.com>
	<AANLkTikoeR+2_u9cgzzJRJ7584FZOUKYW+F4cLirWt3=@mail.gmail.com>
	<AANLkTi=oOHWzL13Z2VZ-yO7haHtQbXcNAaixYqpDU6yu@mail.gmail.com>
	<AANLkTik-aQ3RfQYu7Vz4g1o1=-Z7pvZBasGz5GayaN=D@mail.gmail.com>
	<AANLkTi=yheMqDaD5D3JoCisbz0y+Z+_8CTJ0pecCbWqL@mail.gmail.com>
	<AANLkTim9-k_3QvV8+F9MNW-93cM7Sh7p16SOCxJ98zzF@mail.gmail.com>
	<AANLkTin+1TNx7u5ftJ2+fT4GgWnKQAcJoo-1EM7C5MN-@mail.gmail.com>
Message-ID: <AANLkTiko3siqmNuZf3nQ-5znv_e3-9JDWvs-k47H2-ix@mail.gmail.com>

On Thu, Mar 31, 2011 at 8:12 PM, Eli Stevens (Gmail)
<wickedgrey at gmail.com>wrote:

> On Thu, Mar 31, 2011 at 11:45 AM, Ralf Gommers
> <ralf.gommers at googlemail.com> wrote:
> > On Thu, Mar 31, 2011 at 8:37 PM, Eli Stevens (Gmail)
> > <wickedgrey at gmail.com> wrote:
> >> Next up for me is to get a patch onto the CPython issue tracker, but
> >> as soon as I have that done, I'll start working on adding unit tests
> >> to my numpy fork.  I anticipate being able to get that done today.
> >
> > Sounds good.
>
> CPython issue here:
> http://bugs.python.org/issue11734
>
> My numpy changes so far:
>
> https://github.com/wickedgrey/numpy/compare/23965b6b665985390d7a...maintenance%2F1.6.x
>
> >From my last commit message:
> "Note that my changes to _internal were mostly of the "make it work"
> variety; I'm not really understanding the wider scope.  Very much
> needs review by wiser eyes."
>
> I'd also appreciate some feedback on the unit test coverage; the
> test_roundtrip_half function uncovered a problem that the other two
> didn't (which surprised me), and I'd like to make sure that there
> aren't more gaps in the tests that I've written.
>

Your changes look fine to me, and I think the tests are reasonable as is. I
would suggest tidying up your commit messages a bit, creating an enhancement
bug in the bug tracker, and making a pull request. Some info about the pull
requests is here:

http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html

-Mark


> Thanks for all the help!
> Eli
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110401/ff9cdfcd/attachment.html>

From wickedgrey at gmail.com  Sat Apr  2 00:42:11 2011
From: wickedgrey at gmail.com (Eli Stevens (Gmail))
Date: Fri, 1 Apr 2011 21:42:11 -0700
Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug?
In-Reply-To: <AANLkTiko3siqmNuZf3nQ-5znv_e3-9JDWvs-k47H2-ix@mail.gmail.com>
References: <AANLkTinSTXcSv9pqDXmth4Jxr1uX0XcO4vpRrW8Qyqgp@mail.gmail.com>
	<AANLkTikoeR+2_u9cgzzJRJ7584FZOUKYW+F4cLirWt3=@mail.gmail.com>
	<AANLkTi=oOHWzL13Z2VZ-yO7haHtQbXcNAaixYqpDU6yu@mail.gmail.com>
	<AANLkTik-aQ3RfQYu7Vz4g1o1=-Z7pvZBasGz5GayaN=D@mail.gmail.com>
	<AANLkTi=yheMqDaD5D3JoCisbz0y+Z+_8CTJ0pecCbWqL@mail.gmail.com>
	<AANLkTim9-k_3QvV8+F9MNW-93cM7Sh7p16SOCxJ98zzF@mail.gmail.com>
	<AANLkTin+1TNx7u5ftJ2+fT4GgWnKQAcJoo-1EM7C5MN-@mail.gmail.com>
	<AANLkTiko3siqmNuZf3nQ-5znv_e3-9JDWvs-k47H2-ix@mail.gmail.com>
Message-ID: <BANLkTi=QeLDqD30JgSrJDpHhYRqk08nSLg@mail.gmail.com>

On Fri, Apr 1, 2011 at 5:28 PM, Mark Wiebe <mwwiebe at gmail.com> wrote:
> Your changes look fine to me, and I think the tests are reasonable as is. I
> would suggest tidying up your commit messages a bit, creating an enhancement
> bug in the bug tracker, and making a pull request. Some info about the pull
> requests is here:
> http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html

Done and done.  :)

http://projects.scipy.org/numpy/ticket/1789
https://github.com/numpy/numpy/pull/68

Now to figure out how I can speed things up on the CPython side of things...

Cheers,
Eli


From pav at iki.fi  Sat Apr  2 06:41:47 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 2 Apr 2011 10:41:47 +0000 (UTC)
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1
References: <AANLkTi=eeg8KL7639imRTL-iHg1ncqYoLDDSiD5tfYvs@mail.gmail.com>
Message-ID: <in6uhb$the$1@dough.gmane.org>

Hi,

I'd like to backport the following fixes to the 1.6.x branch for b2:

	ef46a091
	89db53b1

They're bugs in PEP 3118 format string generation, of the brown paper-pag 
variety.

	Pauli



From ralf.gommers at googlemail.com  Sat Apr  2 06:49:24 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sat, 2 Apr 2011 12:49:24 +0200
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 1
In-Reply-To: <in6uhb$the$1@dough.gmane.org>
References: <AANLkTi=eeg8KL7639imRTL-iHg1ncqYoLDDSiD5tfYvs@mail.gmail.com>
	<in6uhb$the$1@dough.gmane.org>
Message-ID: <AANLkTi=CUYWcNaje5am_MrPLr4p18YgH20jNseh83Fi1@mail.gmail.com>

On Sat, Apr 2, 2011 at 12:41 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Hi,
>
> I'd like to backport the following fixes to the 1.6.x branch for b2:
>
> ? ? ? ?ef46a091
> ? ? ? ?89db53b1
>
> They're bugs in PEP 3118 format string generation, of the brown paper-pag
> variety.

Go ahead. For bug fixes I think you can always just backport (at least
before the RC stage), no need to check.

Cheers,
Ralf


From ralf.gommers at googlemail.com  Sat Apr  2 07:40:01 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sat, 2 Apr 2011 13:40:01 +0200
Subject: [Numpy-discussion] 1.6.0b1 installs stuff in
	/usr/lib/python2.7/site-packages/doc
In-Reply-To: <AANLkTikaxKxcFwri7_U1yWi3QY=4H6X+kwD8J05skB97@mail.gmail.com>
References: <4D94ACDC.40701@cora.nwra.com>
	<AANLkTinE8=QLOcMLNdfVRYZy0_7P1b1QHpFuPgzbnzy9@mail.gmail.com>
	<AANLkTim9RBiym0WhpdtT4A5-Z_6t1j06vwsWxgUyq0zx@mail.gmail.com>
	<AANLkTi=BzEFjUPP4DX1vTUKixP13asU-dAPs7vNU-jv5@mail.gmail.com>
	<AANLkTimiwaE+u9wZ9dEUbnRO+6kXhdaCnHfk_PVVLfnU@mail.gmail.com>
	<AANLkTinoxEzPCSoOSsPV4vTcg17B1ET8peWr1ueM_XoP@mail.gmail.com>
	<AANLkTinKkgPXESRhn93gn1xg5toC+MPdEE5=PJGTcTuS@mail.gmail.com>
	<4D94D304.1050401@cora.nwra.com>
	<AANLkTinG5NA8kV_qfZZ8utdK3Pn78eTsNrXkOEZVVyD8@mail.gmail.com>
	<AANLkTikaxKxcFwri7_U1yWi3QY=4H6X+kwD8J05skB97@mail.gmail.com>
Message-ID: <AANLkTimoFA-avZwMkhgSmgCM9RgamhXue=XM6Kr=b1J_@mail.gmail.com>

On Fri, Apr 1, 2011 at 10:56 PM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
> On Thu, Mar 31, 2011 at 9:21 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Thu, Mar 31, 2011 at 14:16, Orion Poplawski <orion at cora.nwra.com> wrote:
>>> On 03/31/2011 01:12 PM, Robert Kern wrote:
>>>> Well, they're meant to be copied into your own code, which is why they
>>>> end up under a doc/ directory. Lots of things like this tend to end up
>>>> in doc/ directories.
>>>
>>> perhaps /usr/share/doc/numpy then.
>>
>> This is up to the various Linux packagers of numpy. They have their
>> own tools to do this according to their own policies. Doing *anything*
>> in our setup.py to install these files will just make their jobs
>> harder. That's why I think that this needs to be handled in the binary
>> installers, not the setup.py.
>
> Okay, makes sense. For now I'll just delete those lines from setup.py
> then and open a ticket. Will try to find some time later to add this
> for the binary installers for Windows and OS X. People who do a source
> install will just have to grab those files from a tarball then.

http://projects.scipy.org/numpy/ticket/1791
https://github.com/numpy/numpy/pull/69


From mwwiebe at gmail.com  Sat Apr  2 14:39:01 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Sat, 2 Apr 2011 11:39:01 -0700
Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug?
In-Reply-To: <AANLkTim9-k_3QvV8+F9MNW-93cM7Sh7p16SOCxJ98zzF@mail.gmail.com>
References: <AANLkTinSTXcSv9pqDXmth4Jxr1uX0XcO4vpRrW8Qyqgp@mail.gmail.com>
	<AANLkTikoeR+2_u9cgzzJRJ7584FZOUKYW+F4cLirWt3=@mail.gmail.com>
	<AANLkTi=oOHWzL13Z2VZ-yO7haHtQbXcNAaixYqpDU6yu@mail.gmail.com>
	<AANLkTik-aQ3RfQYu7Vz4g1o1=-Z7pvZBasGz5GayaN=D@mail.gmail.com>
	<AANLkTi=yheMqDaD5D3JoCisbz0y+Z+_8CTJ0pecCbWqL@mail.gmail.com>
	<AANLkTim9-k_3QvV8+F9MNW-93cM7Sh7p16SOCxJ98zzF@mail.gmail.com>
Message-ID: <AANLkTin_HXc-bZJdLw6PnF6-jct3jfGRbnxvQkD4XdVO@mail.gmail.com>

On Thu, Mar 31, 2011 at 11:45 AM, Ralf Gommers
<ralf.gommers at googlemail.com>wrote:

> On Thu, Mar 31, 2011 at 8:37 PM, Eli Stevens (Gmail)
> <wickedgrey at gmail.com> wrote:
> > On Thu, Mar 31, 2011 at 10:40 AM, Ralf Gommers
> > <ralf.gommers at googlemail.com> wrote:
> >> It would definitely need to get into beta 2, and even then I'm a
> >> little hesitant to push in such a change at the last moment.
> >
> > If I miss the 1.6.0b2 cutoff, would a change like this be appropriate
> > for 1.6.1, or will it need to wait until 1.7.0?  I wish I'd known that
> > half floats were going to be in 1.6.0 (the bug tracker says 2.0); I'd
> > have started working on this a few weeks earlier.  :)
>
> There likely will not be a 1.7.0, if things with 2.0 go according to
> plan. It probably wouldn't be a very appropriate change for 1.6.1.
>
> >
> >> It would
> >> need some testing with other libraries that use the numpy buffer
> >> protocol.
> >
> > I am currently working on a patch for cython to accept half-floats.
> > ATM, it is able to accept the data from numpy (though it does not yet
> > handle the bit-fiddling to expand the values into a float32 on read,
> > nor collapse it from a float32 back into a float16 on write).  The
> > code change to numpy was literally a one-line change to add NPY_HALF
> > to a switch statement (though I haven't tried to import half floats
> > yet).  Is one other library enough, or will I need to find another
> > library to patch as well?  I think that OpenCL would probably be my
> > next choice, since I think that we'll eventually want float16 support
> > there too.
>
> If it works with Cython I think that's enough. With other libraries I
> did not mean that you should write patches for those, just some
> testing to make sure the change is 100% backwards compatible.
>
> >> Also, reading back the thread, Pauli seemed to disagree with
> >> this.
> >
> > On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen <pav at iki.fi> wrote:
> >> The buffer interface cannot be used to export the half-float types,
> since
> >> the type is not specified in PEP 3118. Numpy cannot unilaterally add
> >> nonstandard format codes to the spec.
> >
> > I'm currently in the process of submitting a patch to CPython to
> > update the struct module to handle half-floats.  So far, the response
> > on python-ideas has been generally positive (a couple +1s from people
> > with commit bits, GVR has chimed in on the thread with an aside (but
> > not an objection), etc.).  Unless I'm misunderstanding the objection,
> > making half-floats an official part of the spec (the discussion on
> > python-ideas explicitly notes that numpy driving force behind the
> > change) would satisfy Pauli's concern.
> >
> > Next up for me is to get a patch onto the CPython issue tracker, but
> > as soon as I have that done, I'll start working on adding unit tests
> > to my numpy fork.  I anticipate being able to get that done today.
>
> Sounds good.
>
> I've committed the patch adding 'e' to the buffer protocol and back-ported
it to the 1.6.x branch. This leaves the testing with Cython and getting
consensus from CPython about the letter to use in the struct module/buffer
interface.

-Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110402/468ac6be/attachment.html>

From charlesr.harris at gmail.com  Sat Apr  2 15:59:06 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 2 Apr 2011 13:59:06 -0600
Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug?
In-Reply-To: <AANLkTin_HXc-bZJdLw6PnF6-jct3jfGRbnxvQkD4XdVO@mail.gmail.com>
References: <AANLkTinSTXcSv9pqDXmth4Jxr1uX0XcO4vpRrW8Qyqgp@mail.gmail.com>
	<AANLkTikoeR+2_u9cgzzJRJ7584FZOUKYW+F4cLirWt3=@mail.gmail.com>
	<AANLkTi=oOHWzL13Z2VZ-yO7haHtQbXcNAaixYqpDU6yu@mail.gmail.com>
	<AANLkTik-aQ3RfQYu7Vz4g1o1=-Z7pvZBasGz5GayaN=D@mail.gmail.com>
	<AANLkTi=yheMqDaD5D3JoCisbz0y+Z+_8CTJ0pecCbWqL@mail.gmail.com>
	<AANLkTim9-k_3QvV8+F9MNW-93cM7Sh7p16SOCxJ98zzF@mail.gmail.com>
	<AANLkTin_HXc-bZJdLw6PnF6-jct3jfGRbnxvQkD4XdVO@mail.gmail.com>
Message-ID: <AANLkTi=E4mvDNZupOhV04LmeV3-9W3M_O_9=RK3L34JE@mail.gmail.com>

On Sat, Apr 2, 2011 at 12:39 PM, Mark Wiebe <mwwiebe at gmail.com> wrote:

> On Thu, Mar 31, 2011 at 11:45 AM, Ralf Gommers <
> ralf.gommers at googlemail.com> wrote:
>
>> On Thu, Mar 31, 2011 at 8:37 PM, Eli Stevens (Gmail)
>> <wickedgrey at gmail.com> wrote:
>> > On Thu, Mar 31, 2011 at 10:40 AM, Ralf Gommers
>> > <ralf.gommers at googlemail.com> wrote:
>> >> It would definitely need to get into beta 2, and even then I'm a
>> >> little hesitant to push in such a change at the last moment.
>> >
>> > If I miss the 1.6.0b2 cutoff, would a change like this be appropriate
>> > for 1.6.1, or will it need to wait until 1.7.0?  I wish I'd known that
>> > half floats were going to be in 1.6.0 (the bug tracker says 2.0); I'd
>> > have started working on this a few weeks earlier.  :)
>>
>> There likely will not be a 1.7.0, if things with 2.0 go according to
>> plan. It probably wouldn't be a very appropriate change for 1.6.1.
>>
>> >
>> >> It would
>> >> need some testing with other libraries that use the numpy buffer
>> >> protocol.
>> >
>> > I am currently working on a patch for cython to accept half-floats.
>> > ATM, it is able to accept the data from numpy (though it does not yet
>> > handle the bit-fiddling to expand the values into a float32 on read,
>> > nor collapse it from a float32 back into a float16 on write).  The
>> > code change to numpy was literally a one-line change to add NPY_HALF
>> > to a switch statement (though I haven't tried to import half floats
>> > yet).  Is one other library enough, or will I need to find another
>> > library to patch as well?  I think that OpenCL would probably be my
>> > next choice, since I think that we'll eventually want float16 support
>> > there too.
>>
>> If it works with Cython I think that's enough. With other libraries I
>> did not mean that you should write patches for those, just some
>> testing to make sure the change is 100% backwards compatible.
>>
>> >> Also, reading back the thread, Pauli seemed to disagree with
>> >> this.
>> >
>> > On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen <pav at iki.fi> wrote:
>> >> The buffer interface cannot be used to export the half-float types,
>> since
>> >> the type is not specified in PEP 3118. Numpy cannot unilaterally add
>> >> nonstandard format codes to the spec.
>> >
>> > I'm currently in the process of submitting a patch to CPython to
>> > update the struct module to handle half-floats.  So far, the response
>> > on python-ideas has been generally positive (a couple +1s from people
>> > with commit bits, GVR has chimed in on the thread with an aside (but
>> > not an objection), etc.).  Unless I'm misunderstanding the objection,
>> > making half-floats an official part of the spec (the discussion on
>> > python-ideas explicitly notes that numpy driving force behind the
>> > change) would satisfy Pauli's concern.
>> >
>> > Next up for me is to get a patch onto the CPython issue tracker, but
>> > as soon as I have that done, I'll start working on adding unit tests
>> > to my numpy fork.  I anticipate being able to get that done today.
>>
>> Sounds good.
>>
>> I've committed the patch adding 'e' to the buffer protocol and back-ported
> it to the 1.6.x branch. This leaves the testing with Cython and getting
> consensus from CPython about the letter to use in the struct module/buffer
> interface.
>
>
I didn't see any usable suggestions for an alternative to 'e' and not much
resistance, so I expect  'e' will stick.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110402/9b707892/attachment.html>

From pierre.raybaut at gmail.com  Sat Apr  2 17:01:07 2011
From: pierre.raybaut at gmail.com (Pierre Raybaut)
Date: Sat, 2 Apr 2011 23:01:07 +0200
Subject: [Numpy-discussion] ANN: Spyder v2.0.9
Message-ID: <D8AACF39741D47939BABCDF00D62EACD@asus>

Hi all,

I am pleased to announced that Spyder v2.0.9 has just been released.
As this is mostly a maintenance release, a lot of bugs were fixed (see
below) and some minor features were added.

One of the most significative change since v2.0.8 is a major performance
improvement for the editor's code introspection features (i.e. code
completion, calltips and get definition location -- all three powered by the
"rope" library) provided by an exclusive patch for "rope".

Embedding an interactive Python console into your GUI-based application
(Qt):
This version includes an example of application using the Spyder's internal
shell as a debugging console which also demonstrates the py2exe deployment
procedure.

Spyder (previously known as Pydee) is a free open-source Python development
environment providing MATLAB-like features in a simple and light-weighted
software, available for Windows XP/Vista/7, GNU/Linux and MacOS X:
http://spyderlib.googlecode.com/
Spyder is part of spyderlib, a Python module based on PyQt4, pyflakes and
rope (QScintilla's dependency has been removed in version 2.0 and
rope features have been integrated since this version as well).

Some of Spyder basic features:
    * Python, C/C++, Fortran source editor with class/function browser, code
completion and calltips
    * consoles:
          o open as many Python interpreters, IPython consoles or command
windows as you need
          o code completion and calltips
          o variable explorer with GUI-based editors for a lot of data types
(numbers, strings, lists, arrays, dictionaries, ...)
    * object inspector: provide documentation or source code on any Python
object (class, function, module, ...)
    * online documentation: automatically generated html documentation on
installed Python modules
    * find in files
    * file explorer
    * project manager
    * MATLAB-like PYTHONPATH management dialog box (works with all consoles)
    * Windows only: current user environment variables editor
    * direct links to documentation (Python, Qt, Matplotlib, NumPy, Scipy,
etc.)
    * direct link to Python(x,y) launcher
    * direct links to QtDesigner, QtLinguist and QtAssistant (Qt
documentation)


Bug fixes (since v2.0.8):
Console: added option to ignore PyQt/sip errors when trying to set sip API
(fixed Enthought Tool Suite 3.6.0 compatibility issue)
utils.dochelpers.getargtxt/bugfix: retrieving builtin function arguments was
no longer working
(Fixes Issue 499) Editor-related keyboard shortcuts were not applied after
opening files
(Fixes Issue 575) Tab scroll buttons were not shown on OS X resulting in
clamped/changing window sizes
(Fixes Issue 574) Debugging: Spyder only synced at debugger breakpoints
(Fixes Issue 576) "Source / Remove trailing spaces" was removing newline at
the end of file (+ added support for "undo")
(Fixes Issue 582) Console: changing font preferences was requiring a restart
to be fully taken into account
(Fixes Issue 562) Spyder was unable to restore editor's outline explorer
tree when mixed ' and " characters were found in tree entries
(Fixes Issue 590) Shell/"Clear line" shortcut was not correct: this is
actually "Shift+Escape" (not "Escape")
(Fixes Issue 591) History log was systematically erased when updating Spyder
version
Outline explorer/bugfix: when opening file, the 'show/hide all files' option
was not applied (this was then applied when switching from a file to
another)
(Fixes Issue 602) Backported from v2.1 a couple of bugfixes related to
Editor and multiple panels
Object inspector: when raised automatically above other dockwidgets, plugin
refresh was unnecessarily triggered
Editor/code completion-bugfix: some key events (e.g. Ctrl+V) were lost
during code completion-related hang-up
(Fixes Issue 599) Multiline text pasting was not working in a newly opened
console (i.e. first prompt)

Enhancements (since v2.0.8):
Major change/Translations: moved from 'QtLinguist' to 'gettext' (localizing
Spyder should now be easier)
Console: increased default maximum line count (buffer depth) up to 10,000
lines (instead of only 300 lines)
Editor's rope-based introspection features (code completion, calltips, go to
definition): new rope monkey-patch providing major performance improvements
File explorer/Project explorer - opening file with associated application:
now supported on all platforms
Added action "Reset window layout" in "View" menu to reset main window
layout to default
Documentation: added page on debugging
Editor: added syntax highlighters for diff/patch files (.diff, .patch, .rej)
and gettext files (.po, .pot)
(Fixes Issue 537) Global working directory toolbar: removed label
considering the fact that the toolbar widgets are quite explicit on its role
(and the combo box tooltip is explaining it in detail)
(Fixes Issue 598) Added a .desktop file in source package
(Fixes Issue 87) Editor plugin's title now show the current script filename

Cheers,
Pierre



From fonnesbeck at gmail.com  Sat Apr  2 18:12:56 2011
From: fonnesbeck at gmail.com (Chris Fonnesbeck)
Date: Sat, 2 Apr 2011 17:12:56 -0500
Subject: [Numpy-discussion] Indexing bug in structured arrays(?)
Message-ID: <BANLkTi=u1FrGKtLatfSa=M02hNzvx1CyxA@mail.gmail.com>

I am either getting a nasty bug when indexing structured arrays, or I
don't really understand how they work. I have imported some data using
genfromtxt and an associated list of dtypes:

ndtype=[('include', int), ('year', int), ('month', int), ('day', int),
('deg_day_north', int), ('deg_day_south', int), ('water_north',
float), ('water_south', float), ('obs1', int), ('obs2', int), ('obs3',
int), ('obs4', int)]

survey_data = genfromtxt("Data/man33.out", dtype=ndtype)

This yields a nice structured array as follows:

array([(1, 85, 1, 6, 34, 20, 20.0, 22.5, 6, 30, 26, 79),
       (1, 85, 1, 10, 87, 38, 17.0, 22.0, 11, 43, 40, 113),
       (1, 85, 1, 14, 137, 60, 14.0, 22.0, 4, 87, 18, 70),
       (1, 85, 1, 19, 126, 42, 16.0, 22.0, 31, 192, 32, 301),
       (1, 85, 1, 22, 170, 80, 13.0, 21.0, 6, 316, 2, 118),
       (1, 85, 1, 27, 170, 99, 13.0, 21.0, 2, 373, 12, 124),
       (0, 85, 2, 2, 106, 34, 17.0, 22.0, 0, 52, 93, 87),
  ...
       (1, 0, 1, 16, 18, 2, 18.0, 22.5, 45, 158, 18, 141),
       (1, 0, 1, 27, 103, 70, 15.0, 22.0, 86, 503, 40, 162),
       (1, 1, 12, 21, 64, 33, 18.0, 24.5, 24, 103, 51, 72),
       (1, 1, 1, 1, 113, 56, 15.0, 22.5, 65, 399, 32, 259),
       (1, 1, 1, 5, 169, 104, 13.0, 22.5, 120, 677, 24, 390),
       (1, 1, 1, 24, 68, 43, 16.0, 21.0, 282, 658, 6, 298)],
      dtype=[('include', '<i8'), ('year', '<i8'), ('month', '<i8'),
('day', '<i8'), ('deg_day_north', '<i8'), ('deg_day_south', '<i8'),
('water_north', '<f8'), ('water_south', '<f8'), ('obs1', '<i8'),
('obs2', '<i8'), ('obs3', '<i8'), ('obs4', '<i8')])

The first column ('include') is an indicator for inclusion on a
particular analysis, so I want to filter those out. So, I tried:

In [30]: survey_data['include'==1]
Out[30]: (1, 85, 1, 6, 34, 20, 20.0, 22.5, 6, 30, 26, 79)

Is this the expected behavior? If so, please point me to any
documentation that explains it -- I did not see it on the pages
describing structured arrays on the Numpy site. I finally was able to
index these out using an array of booleans, but I would have thought
the above would work.

Thanks in advance.
cf


From fonnesbeck at gmail.com  Sat Apr  2 18:15:10 2011
From: fonnesbeck at gmail.com (Chris)
Date: Sat, 2 Apr 2011 22:15:10 +0000 (UTC)
Subject: [Numpy-discussion] Indexing bug in structured arrays(?)
References: <BANLkTi=u1FrGKtLatfSa=M02hNzvx1CyxA@mail.gmail.com>
Message-ID: <loom.20110403T001423-21@post.gmane.org>


Figured out why this does not work. Sorry and thanks.

cf






From wickedgrey at gmail.com  Sat Apr  2 22:50:36 2011
From: wickedgrey at gmail.com (Eli Stevens (Gmail))
Date: Sat, 2 Apr 2011 19:50:36 -0700
Subject: [Numpy-discussion] 1.6.0b1 half float buffer bug?
In-Reply-To: <AANLkTin_HXc-bZJdLw6PnF6-jct3jfGRbnxvQkD4XdVO@mail.gmail.com>
References: <AANLkTinSTXcSv9pqDXmth4Jxr1uX0XcO4vpRrW8Qyqgp@mail.gmail.com>
	<AANLkTikoeR+2_u9cgzzJRJ7584FZOUKYW+F4cLirWt3=@mail.gmail.com>
	<AANLkTi=oOHWzL13Z2VZ-yO7haHtQbXcNAaixYqpDU6yu@mail.gmail.com>
	<AANLkTik-aQ3RfQYu7Vz4g1o1=-Z7pvZBasGz5GayaN=D@mail.gmail.com>
	<AANLkTi=yheMqDaD5D3JoCisbz0y+Z+_8CTJ0pecCbWqL@mail.gmail.com>
	<AANLkTim9-k_3QvV8+F9MNW-93cM7Sh7p16SOCxJ98zzF@mail.gmail.com>
	<AANLkTin_HXc-bZJdLw6PnF6-jct3jfGRbnxvQkD4XdVO@mail.gmail.com>
Message-ID: <BANLkTimkLU8zaUASnqGzkMSjiJ+sdNKrTg@mail.gmail.com>

On Sat, Apr 2, 2011 at 11:39 AM, Mark Wiebe <mwwiebe at gmail.com> wrote:
> I've committed the patch adding 'e' to the buffer protocol and back-ported
> it to the 1.6.x branch.

Awesome!  Thanks a bunch.  :)

> This leaves the testing with Cython and getting
> consensus from CPython about the letter to use in the struct module/buffer
> interface.

I've done some informal poking around using half-floats with my
patched numpy and Cython and things seem to be okay; let me know what
you're looking for in more detail and I'd be happy to do the legwork.
:)

Cheers,
Eli


From ralf.gommers at googlemail.com  Sun Apr  3 10:10:25 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sun, 3 Apr 2011 16:10:25 +0200
Subject: [Numpy-discussion] tests with generators & nose: a warning
Message-ID: <AANLkTinXnSki+aUUuzE+OCQXm+kd6rtVGKwtQoo84dM5@mail.gmail.com>

Hi all,

I just came across tests for allclose that were using generators and
were in a subclass of unittest.TestCase. These tests won't run, nose
does not support this (see last line of
http://somethingaboutorange.com/mrl/projects/nose/1.0.0/writing_tests.html).
So if you write tests with "yield", don't subclass from TestCase!

I'll open a ticket for this and check more, but I thought this was
important and obscure enough to send a warning.

Cheers,
Ralf


From fperez.net at gmail.com  Sun Apr  3 15:04:19 2011
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 3 Apr 2011 12:04:19 -0700
Subject: [Numpy-discussion] tests with generators & nose: a warning
In-Reply-To: <AANLkTinXnSki+aUUuzE+OCQXm+kd6rtVGKwtQoo84dM5@mail.gmail.com>
References: <AANLkTinXnSki+aUUuzE+OCQXm+kd6rtVGKwtQoo84dM5@mail.gmail.com>
Message-ID: <BANLkTi==M86FQbX14FBrc2D42RbF_==+1Q@mail.gmail.com>

Hey,

On Sun, Apr 3, 2011 at 7:10 AM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
> Hi all,
>
> I just came across tests for allclose that were using generators and
> were in a subclass of unittest.TestCase. These tests won't run, nose
> does not support this (see last line of
> http://somethingaboutorange.com/mrl/projects/nose/1.0.0/writing_tests.html).
> So if you write tests with "yield", don't subclass from TestCase!
>
> I'll open a ticket for this and check more, but I thought this was
> important and obscure enough to send a warning.

FWIW, in ipython we do have support, in the form of decorators, for
parametric tests both as functions and as TestCase subclasses:

the python2/3 implementations:
https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy2.py
https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy3.py

the enclosing public api:
https://github.com/ipython/ipython/blob/master/IPython/testing/decorators.py

usage examples from the test suite for this functionality:
https://github.com/ipython/ipython/blob/master/IPython/testing/tests/test_decorators.py

I need to add that these tests lose one key feature of nose parametric
tests: even though all tests are run and counted if they succeed, when
there's a failure the suite stops at the first failure.

We've accepted that unfortunate limitation because the upside is that
our version is infinitely easier to debug than nose's approach.  In
nose you write

yield callable, arg1, arg2, ...

which means that when things fail, the stack you see (and the one you
have for debugging via --pdb) is the nose execution stack instead of
your own.  That makes debugging parametric nose tests insanely hard,
while at least in our form, where you write

yield callable(arg1, arg2, ...)

you get to debug off your real stack.

So, not ideal, but if you guys want any of it, as usual just grab it.

Cheers,

f


From ralf.gommers at googlemail.com  Sun Apr  3 16:37:03 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sun, 3 Apr 2011 22:37:03 +0200
Subject: [Numpy-discussion] tests with generators & nose: a warning
In-Reply-To: <BANLkTi==M86FQbX14FBrc2D42RbF_==+1Q@mail.gmail.com>
References: <AANLkTinXnSki+aUUuzE+OCQXm+kd6rtVGKwtQoo84dM5@mail.gmail.com>
	<BANLkTi==M86FQbX14FBrc2D42RbF_==+1Q@mail.gmail.com>
Message-ID: <AANLkTimd=1LaSr9ywUkauApvVresuGMg3mrHn5TZM7+5@mail.gmail.com>

On Sun, Apr 3, 2011 at 9:04 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> Hey,
>
> On Sun, Apr 3, 2011 at 7:10 AM, Ralf Gommers
> <ralf.gommers at googlemail.com> wrote:
>> Hi all,
>>
>> I just came across tests for allclose that were using generators and
>> were in a subclass of unittest.TestCase. These tests won't run, nose
>> does not support this (see last line of
>> http://somethingaboutorange.com/mrl/projects/nose/1.0.0/writing_tests.html).
>> So if you write tests with "yield", don't subclass from TestCase!
>>
>> I'll open a ticket for this and check more, but I thought this was
>> important and obscure enough to send a warning.
>
> FWIW, in ipython we do have support, in the form of decorators, for
> parametric tests both as functions and as TestCase subclasses:
>
> the python2/3 implementations:
> https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy2.py
> https://github.com/ipython/ipython/blob/master/IPython/testing/_paramtestpy3.py
>
> the enclosing public api:
> https://github.com/ipython/ipython/blob/master/IPython/testing/decorators.py
>
> usage examples from the test suite for this functionality:
> https://github.com/ipython/ipython/blob/master/IPython/testing/tests/test_decorators.py
>
> I need to add that these tests lose one key feature of nose parametric
> tests: even though all tests are run and counted if they succeed, when
> there's a failure the suite stops at the first failure.
>
> We've accepted that unfortunate limitation because the upside is that
> our version is infinitely easier to debug than nose's approach. ?In
> nose you write
>
> yield callable, arg1, arg2, ...
>
> which means that when things fail, the stack you see (and the one you
> have for debugging via --pdb) is the nose execution stack instead of
> your own. ?That makes debugging parametric nose tests insanely hard,
> while at least in our form, where you write
>
> yield callable(arg1, arg2, ...)
>
> you get to debug off your real stack.
>
> So, not ideal, but if you guys want any of it, as usual just grab it.

That indeed looks much more useful than the nose version. I searched a
bit on the ipython-devel list but couldn't find too much discussion
about the implementation. Would it be at all possible to let it not
stop after the first failure? Either way, I think we should copy it.

Ralf


From fperez.net at gmail.com  Sun Apr  3 16:47:56 2011
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 3 Apr 2011 13:47:56 -0700
Subject: [Numpy-discussion] tests with generators & nose: a warning
In-Reply-To: <AANLkTimd=1LaSr9ywUkauApvVresuGMg3mrHn5TZM7+5@mail.gmail.com>
References: <AANLkTinXnSki+aUUuzE+OCQXm+kd6rtVGKwtQoo84dM5@mail.gmail.com>
	<BANLkTi==M86FQbX14FBrc2D42RbF_==+1Q@mail.gmail.com>
	<AANLkTimd=1LaSr9ywUkauApvVresuGMg3mrHn5TZM7+5@mail.gmail.com>
Message-ID: <BANLkTiktRgQgpKdCo9G4ahiqfQ7UaQO7XA@mail.gmail.com>

On Sun, Apr 3, 2011 at 1:37 PM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
> That indeed looks much more useful than the nose version. I searched a
> bit on the ipython-devel list but couldn't find too much discussion
> about the implementation. Would it be at all possible to let it not
> stop after the first failure? Either way, I think we should copy it.

I tried very hard and never managed to find a way to make it work in
that mode, unfortunately. Not saying it's categorically impossible,
but given the current architecture of unittest and nose, it wasn't
easy to do with code that wasn't super intrusive.

You'd need to modify a lot of how unittest and nose execute tests and
record failures to be able to change that, I'm afraid.  Since I wanted
a *lightweight* extension that would work cleanly on top of
nose/unittest and would be easy to maintain moving forward, I settled
for this.

Ideally this would be done right at the unittest level, but I don't
have the bandwidth to pursue that, I'm afraid.

HTH,

f


From david.crisp at gmail.com  Sun Apr  3 20:03:49 2011
From: david.crisp at gmail.com (David Crisp)
Date: Mon, 4 Apr 2011 10:03:49 +1000
Subject: [Numpy-discussion] Matrix to Array Transformation assistance
	required
Message-ID: <BANLkTi=VPjvUR3SbRd3WGohAoK7NA4V8nw@mail.gmail.com>

Hello,

I am trying to convert a numpy matrix to an array containing
coordinate pairs,  and I am having difficulty understanding the
transformation process.   I know there is a fairly simply way of doing
this transformation but am unsure of the commands to use.

The matrix I have is created with the following command

vegetation_matrix = numpy.empty(shape=(x_bins,y_bins))

x_bins and y_bins are dynamic in size depending on the input data and
the required precision.   Some tests I need to perform could result in
a 10 by 10 matrix and some could result in a 4000,5000 matrix,
However,  these values are dynamic and could be anything.  For
instance they COULD be 300,124 or 400,900 etc.. these are unknown
values until the data has been collected.

A sample matrix view will be:

>>> a
array([['x', 'o', 'o'],
      ['o', 'x', 'x'],
      ['o', 'x', 'o']],
     dtype='|S1')

I wish to convert this to:
0,0,x
0,1,o
0,2,o
...
2,2,o

I have been given the following example which works on a known size of data.

>>> a
array([['x', 'o', 'o'],
      ['o', 'x', 'x'],
      ['o', 'x', 'o']],
     dtype='|S1')

>>> numoy.array([numpy.arange(9)//3, numpy.arange(9)%3, a.flatten()]).transpose() array([['0', '0', 'x'],
      ['0', '1', 'o'],
      ['0', '2', 'o'],
      ['1', '0', 'o'],
      ['1', '1', 'x'],
      ['1', '2', 'x'],
      ['2', '0', 'o'],
      ['2', '1', 'x'],
      ['2', '2', 'o']],
     dtype='|S8')

But when I try and convert that line to work on my own matrix with its
larger size,   I do not understand how to make it work.

Would somebody be able to provide a little assistance on how I can proceed here.

(I am a little new to python and part of the problem I have is using
the correct terminology for various processes etc)

Regards,
David


From charlesr.harris at gmail.com  Sun Apr  3 20:20:24 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 3 Apr 2011 18:20:24 -0600
Subject: [Numpy-discussion] Remove deprecated skiprows
Message-ID: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>

Should skiprows be removed?

    if skiprows:
        warnings.warn(\
            "The use of `skiprows` is deprecated, it will be removed in
numpy 2.0.\n" \
            "Please use `skip_header` instead.",
            DeprecationWarning)
        skip_header = skiprows

Its been deprecated since 1.4. Personally, I like skiprows better than
skip_header ;)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110403/219641b5/attachment.html>

From josef.pktd at gmail.com  Mon Apr  4 00:04:13 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 4 Apr 2011 00:04:13 -0400
Subject: [Numpy-discussion] Matrix to Array Transformation assistance
	required
In-Reply-To: <BANLkTi=VPjvUR3SbRd3WGohAoK7NA4V8nw@mail.gmail.com>
References: <BANLkTi=VPjvUR3SbRd3WGohAoK7NA4V8nw@mail.gmail.com>
Message-ID: <BANLkTimwdGyxNmZPri5GUZJPz+e9cQRAvA@mail.gmail.com>

On Sun, Apr 3, 2011 at 8:03 PM, David Crisp <david.crisp at gmail.com> wrote:
> Hello,
>
> I am trying to convert a numpy matrix to an array containing
> coordinate pairs, ?and I am having difficulty understanding the
> transformation process. ? I know there is a fairly simply way of doing
> this transformation but am unsure of the commands to use.
>
> The matrix I have is created with the following command
>
> vegetation_matrix = numpy.empty(shape=(x_bins,y_bins))
>
> x_bins and y_bins are dynamic in size depending on the input data and
> the required precision. ? Some tests I need to perform could result in
> a 10 by 10 matrix and some could result in a 4000,5000 matrix,
> However, ?these values are dynamic and could be anything. ?For
> instance they COULD be 300,124 or 400,900 etc.. these are unknown
> values until the data has been collected.
>
> A sample matrix view will be:
>
>>>> a
> array([['x', 'o', 'o'],
> ? ? ?['o', 'x', 'x'],
> ? ? ?['o', 'x', 'o']],
> ? ? dtype='|S1')
>
> I wish to convert this to:
> 0,0,x
> 0,1,o
> 0,2,o
> ...
> 2,2,o
>
> I have been given the following example which works on a known size of data.
>
>>>> a
> array([['x', 'o', 'o'],
> ? ? ?['o', 'x', 'x'],
> ? ? ?['o', 'x', 'o']],
> ? ? dtype='|S1')
>
>>>> numoy.array([numpy.arange(9)//3, numpy.arange(9)%3, a.flatten()]).transpose() array([['0', '0', 'x'],
> ? ? ?['0', '1', 'o'],
> ? ? ?['0', '2', 'o'],
> ? ? ?['1', '0', 'o'],
> ? ? ?['1', '1', 'x'],
> ? ? ?['1', '2', 'x'],
> ? ? ?['2', '0', 'o'],
> ? ? ?['2', '1', 'x'],
> ? ? ?['2', '2', 'o']],
> ? ? dtype='|S8')
>
> But when I try and convert that line to work on my own matrix with its
> larger size, ? I do not understand how to make it work.

It looks to me that you just have to replace the 9 and the 3 with your
2d array size and shape, for example

>>> a = np.arange(12).reshape(3,4)
>>> a
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
>>> np.column_stack((np.arange(a.size)//a.shape[1], np.arange(a.size)%a.shape[1], a.ravel()))
array([[ 0,  0,  0],
       [ 0,  1,  1],
       [ 0,  2,  2],
       [ 0,  3,  3],
       [ 1,  0,  4],
       [ 1,  1,  5],
       [ 1,  2,  6],
       [ 1,  3,  7],
       [ 2,  0,  8],
       [ 2,  1,  9],
       [ 2,  2, 10],
       [ 2,  3, 11]])

Or do I misunderstand the question?

Josef

>
> Would somebody be able to provide a little assistance on how I can proceed here.
>
> (I am a little new to python and part of the problem I have is using
> the correct terminology for various processes etc)
>
> Regards,
> David
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From paul.anton.letnes at gmail.com  Sun Apr  3 06:35:08 2011
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Sun, 3 Apr 2011 12:35:08 +0200
Subject: [Numpy-discussion] Import patch
Message-ID: <4A5FDFED-67FE-4782-9399-02D91D415C34@gmail.com>

Hi.

When looking at the loadtxt/savetxt tickets, I noticed that the 're' module is imported in an odd place. I therefore suggest that this import is moved to the top of the file, in order to gather these as much as possible. I find the code easier to read then. After all, there is no 'try / catch' or similar to check if the module exists. See patch below. I do not believe any tests or tickets are needed - correct me if I am wrong.

Cheers,
Paul.


--- a/numpy/lib/npyio.py	Sat Apr 02 20:19:55 2011 -0600
+++ b/numpy/lib/npyio.py	Sun Apr 03 12:30:02 2011 +0200
@@ -6,6 +6,7 @@
 import format
 import sys
 import os
+import re
 import sys
 import itertools
 import warnings
@@ -956,7 +957,6 @@
         if own_fh:
             fh.close()
 
-import re
 def fromregex(file, regexp, dtype):
     """
     Construct an array from a text file, using regular expression parsing.

From ater1980 at gmail.com  Mon Apr  4 06:49:54 2011
From: ater1980 at gmail.com (Alex Ter-Sarkissov)
Date: Mon, 4 Apr 2011 22:49:54 +1200
Subject: [Numpy-discussion] division operator
Message-ID: <BANLkTimt4pm2-UY5y7ogOpyLLvw4C4-ePQ@mail.gmail.com>

I have 2 variables, say var1=10,var2=100. To divide I do either
divide(float(var1),float(var2)) or simply float(var1)/float(var2). I'm just
wondering if there's a smarter way of doing this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110404/d56708c3/attachment.html>

From andyfaff at gmail.com  Mon Apr  4 06:58:52 2011
From: andyfaff at gmail.com (andrew nelson)
Date: Mon, 4 Apr 2011 20:58:52 +1000
Subject: [Numpy-discussion] division operator
In-Reply-To: <BANLkTimt4pm2-UY5y7ogOpyLLvw4C4-ePQ@mail.gmail.com>
References: <BANLkTimt4pm2-UY5y7ogOpyLLvw4C4-ePQ@mail.gmail.com>
Message-ID: <F68CC3A6-BF62-43DE-B6B8-5D8A4FFE5F96@gmail.com>

>>> var1 = 10
>>> var2 = 100
>>> import numpy as np
>>> np.divide(var1,var2)
0
>>> np.divide(10,100 * 1.)
0.10000000000000001
>>> var1/var2
0
>>>var1*1./var2
0.1


On 04/04/2011, at 8:49 PM, Alex Ter-Sarkissov wrote:

> I have 2 variables, say var1=10,var2=100. To divide I do either divide(float(var1),float(var2)) or simply float(var1)/float(var2). I'm just wondering if there's a smarter way of doing this? 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

_________________
Dr Andrew Nelson
andyfaff at gmail.com
_________________



From pearu.peterson at gmail.com  Mon Apr  4 06:57:02 2011
From: pearu.peterson at gmail.com (Pearu Peterson)
Date: Mon, 04 Apr 2011 13:57:02 +0300
Subject: [Numpy-discussion] division operator
In-Reply-To: <BANLkTimt4pm2-UY5y7ogOpyLLvw4C4-ePQ@mail.gmail.com>
References: <BANLkTimt4pm2-UY5y7ogOpyLLvw4C4-ePQ@mail.gmail.com>
Message-ID: <4D99A3FE.8080106@cens.ioc.ee>



On 04/04/2011 01:49 PM, Alex Ter-Sarkissov wrote:
> I have 2 variables, say var1=10,var2=100. To divide I do either
> divide(float(var1),float(var2)) or simply float(var1)/float(var2). I'm
> just wondering if there's a smarter way of doing this?

 >>> from __future__ import division
 >>> var1 = 10
 >>> var2 = 100
 >>> print var1/var2
     0.1

HTH,
Pearu


From alan.isaac at gmail.com  Mon Apr  4 07:49:24 2011
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Mon, 04 Apr 2011 07:49:24 -0400
Subject: [Numpy-discussion] division operator
In-Reply-To: <BANLkTimt4pm2-UY5y7ogOpyLLvw4C4-ePQ@mail.gmail.com>
References: <BANLkTimt4pm2-UY5y7ogOpyLLvw4C4-ePQ@mail.gmail.com>
Message-ID: <4D99B044.4040702@gmail.com>

On 4/4/2011 6:49 AM, Alex Ter-Sarkissov wrote:
> divide(float(var1),float(var2))

http://docs.scipy.org/doc/numpy/reference/generated/numpy.true_divide.html

hth,
Alan Isaac


From charlesr.harris at gmail.com  Mon Apr  4 09:34:05 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 4 Apr 2011 07:34:05 -0600
Subject: [Numpy-discussion] Import patch
In-Reply-To: <4A5FDFED-67FE-4782-9399-02D91D415C34@gmail.com>
References: <4A5FDFED-67FE-4782-9399-02D91D415C34@gmail.com>
Message-ID: <BANLkTi==n6Lgq_MvLK7EQoeWFqBEz6b8sQ@mail.gmail.com>

On Sun, Apr 3, 2011 at 4:35 AM, Paul Anton Letnes <
paul.anton.letnes at gmail.com> wrote:

> Hi.
>
> When looking at the loadtxt/savetxt tickets, I noticed that the 're' module
> is imported in an odd place. I therefore suggest that this import is moved
> to the top of the file, in order to gather these as much as possible. I find
> the code easier to read then. After all, there is no 'try / catch' or
> similar to check if the module exists. See patch below. I do not believe any
> tests or tickets are needed - correct me if I am wrong.
>
> Cheers,
> Paul.
>
>
> --- a/numpy/lib/npyio.py        Sat Apr 02 20:19:55 2011 -0600
> +++ b/numpy/lib/npyio.py        Sun Apr 03 12:30:02 2011 +0200
> @@ -6,6 +6,7 @@
>  import format
>  import sys
>  import os
> +import re
>  import sys
>  import itertools
>  import warnings
> @@ -956,7 +957,6 @@
>         if own_fh:
>             fh.close()
>
> -import re
>  def fromregex(file, regexp, dtype):
>     """
>     Construct an array from a text file, using regular expression parsing.


If you want to see a lot of other small things for cleanups, run pyflakes or
pylint on the files in numpy/lib

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110404/8008056b/attachment.html>

From paul.anton.letnes at gmail.com  Mon Apr  4 10:29:51 2011
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Mon, 4 Apr 2011 16:29:51 +0200
Subject: [Numpy-discussion] Import patch
In-Reply-To: <BANLkTi==n6Lgq_MvLK7EQoeWFqBEz6b8sQ@mail.gmail.com>
References: <4A5FDFED-67FE-4782-9399-02D91D415C34@gmail.com>
	<BANLkTi==n6Lgq_MvLK7EQoeWFqBEz6b8sQ@mail.gmail.com>
Message-ID: <75CA4071-26EC-4B08-8965-66C855330CF2@gmail.com>


On 4. apr. 2011, at 15.34, Charles R Harris wrote:

> 
> 
> On Sun, Apr 3, 2011 at 4:35 AM, Paul Anton Letnes <paul.anton.letnes at gmail.com> wrote:
> Hi.
> 
> When looking at the loadtxt/savetxt tickets, I noticed that the 're' module is imported in an odd place. I therefore suggest that this import is moved to the top of the file, in order to gather these as much as possible. I find the code easier to read then. After all, there is no 'try / catch' or similar to check if the module exists. See patch below. I do not believe any tests or tickets are needed - correct me if I am wrong.
> 
> Cheers,
> Paul.
> 
> 
> --- a/numpy/lib/npyio.py        Sat Apr 02 20:19:55 2011 -0600
> +++ b/numpy/lib/npyio.py        Sun Apr 03 12:30:02 2011 +0200
> @@ -6,6 +6,7 @@
>  import format
>  import sys
>  import os
> +import re
>  import sys
>  import itertools
>  import warnings
> @@ -956,7 +957,6 @@
>         if own_fh:
>             fh.close()
> 
> -import re
>  def fromregex(file, regexp, dtype):
>     """
>     Construct an array from a text file, using regular expression parsing.
> 
> If you want to see a lot of other small things for cleanups, run pyflakes or pylint on the files in numpy/lib
> 
> Chuck
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

Are you suggesting that I should do this and submit one or more patches?

By the way - what is a suggested form of submitting patches? Using e-mail seems a bit clumsy to me.

Paul.




From charlesr.harris at gmail.com  Mon Apr  4 10:42:47 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 4 Apr 2011 08:42:47 -0600
Subject: [Numpy-discussion] Import patch
In-Reply-To: <75CA4071-26EC-4B08-8965-66C855330CF2@gmail.com>
References: <4A5FDFED-67FE-4782-9399-02D91D415C34@gmail.com>
	<BANLkTi==n6Lgq_MvLK7EQoeWFqBEz6b8sQ@mail.gmail.com>
	<75CA4071-26EC-4B08-8965-66C855330CF2@gmail.com>
Message-ID: <BANLkTim_=LoyLm9yOb=sgwGBjHSNRbTiMw@mail.gmail.com>

On Mon, Apr 4, 2011 at 8:29 AM, Paul Anton Letnes <
paul.anton.letnes at gmail.com> wrote:

>
> On 4. apr. 2011, at 15.34, Charles R Harris wrote:
>
> >
> >
> > On Sun, Apr 3, 2011 at 4:35 AM, Paul Anton Letnes <
> paul.anton.letnes at gmail.com> wrote:
> > Hi.
> >
> > When looking at the loadtxt/savetxt tickets, I noticed that the 're'
> module is imported in an odd place. I therefore suggest that this import is
> moved to the top of the file, in order to gather these as much as possible.
> I find the code easier to read then. After all, there is no 'try / catch' or
> similar to check if the module exists. See patch below. I do not believe any
> tests or tickets are needed - correct me if I am wrong.
> >
> > Cheers,
> > Paul.
> >
> >
> > --- a/numpy/lib/npyio.py        Sat Apr 02 20:19:55 2011 -0600
> > +++ b/numpy/lib/npyio.py        Sun Apr 03 12:30:02 2011 +0200
> > @@ -6,6 +6,7 @@
> >  import format
> >  import sys
> >  import os
> > +import re
> >  import sys
> >  import itertools
> >  import warnings
> > @@ -956,7 +957,6 @@
> >         if own_fh:
> >             fh.close()
> >
> > -import re
> >  def fromregex(file, regexp, dtype):
> >     """
> >     Construct an array from a text file, using regular expression
> parsing.
> >
> > If you want to see a lot of other small things for cleanups, run pyflakes
> or pylint on the files in numpy/lib
> >
> > Chuck
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> Are you suggesting that I should do this and submit one or more patches?
>

That would be great, and a good way to get into numpy development.


>
> By the way - what is a suggested form of submitting patches? Using e-mail
> seems a bit clumsy to me.
>
>
The best thing would be to setup to do your work on github and then issue
pull requests, see the directions here:
http://docs.scipy.org/doc/numpy/dev/gitwash/development_setup.html. If you
are on linux or mac getting set up should be pretty easy. Windows may be
trickier, I don't have experience doing development on that platform.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110404/bb2f5c17/attachment.html>

From paul.anton.letnes at gmail.com  Mon Apr  4 10:57:48 2011
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Mon, 4 Apr 2011 16:57:48 +0200
Subject: [Numpy-discussion] Import patch
In-Reply-To: <BANLkTim_=LoyLm9yOb=sgwGBjHSNRbTiMw@mail.gmail.com>
References: <4A5FDFED-67FE-4782-9399-02D91D415C34@gmail.com>
	<BANLkTi==n6Lgq_MvLK7EQoeWFqBEz6b8sQ@mail.gmail.com>
	<75CA4071-26EC-4B08-8965-66C855330CF2@gmail.com>
	<BANLkTim_=LoyLm9yOb=sgwGBjHSNRbTiMw@mail.gmail.com>
Message-ID: <13EC5F35-9256-4213-B2D6-59F6CE25446B@gmail.com>


On 4. apr. 2011, at 16.42, Charles R Harris wrote:

> 
> 
> On Mon, Apr 4, 2011 at 8:29 AM, Paul Anton Letnes <paul.anton.letnes at gmail.com> wrote:
> 
> On 4. apr. 2011, at 15.34, Charles R Harris wrote:
> 
> >
> >
> > On Sun, Apr 3, 2011 at 4:35 AM, Paul Anton Letnes <paul.anton.letnes at gmail.com> wrote:
> > Hi.
> >
> > When looking at the loadtxt/savetxt tickets, I noticed that the 're' module is imported in an odd place. I therefore suggest that this import is moved to the top of the file, in order to gather these as much as possible. I find the code easier to read then. After all, there is no 'try / catch' or similar to check if the module exists. See patch below. I do not believe any tests or tickets are needed - correct me if I am wrong.
> >
> > Cheers,
> > Paul.
> >
> >
> > --- a/numpy/lib/npyio.py        Sat Apr 02 20:19:55 2011 -0600
> > +++ b/numpy/lib/npyio.py        Sun Apr 03 12:30:02 2011 +0200
> > @@ -6,6 +6,7 @@
> >  import format
> >  import sys
> >  import os
> > +import re
> >  import sys
> >  import itertools
> >  import warnings
> > @@ -956,7 +957,6 @@
> >         if own_fh:
> >             fh.close()
> >
> > -import re
> >  def fromregex(file, regexp, dtype):
> >     """
> >     Construct an array from a text file, using regular expression parsing.
> >
> > If you want to see a lot of other small things for cleanups, run pyflakes or pylint on the files in numpy/lib
> >
> > Chuck
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> Are you suggesting that I should do this and submit one or more patches?
> 
> That would be great, and a good way to get into numpy development.
>  
> 
> By the way - what is a suggested form of submitting patches? Using e-mail seems a bit clumsy to me.
> 
> 
> The best thing would be to setup to do your work on github and then issue pull requests, see the directions here: http://docs.scipy.org/doc/numpy/dev/gitwash/development_setup.html. If you are on linux or mac getting set up should be pretty easy. Windows may be trickier, I don't have experience doing development on that platform.

I'm on a mac, and I've got git. The skills to use it will have to grow with time.

I submitted a pull request on this particular patch - let me know if I got it right!

Cheers,
Paul.





From charlesr.harris at gmail.com  Mon Apr  4 11:33:49 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 4 Apr 2011 09:33:49 -0600
Subject: [Numpy-discussion] Import patch
In-Reply-To: <13EC5F35-9256-4213-B2D6-59F6CE25446B@gmail.com>
References: <4A5FDFED-67FE-4782-9399-02D91D415C34@gmail.com>
	<BANLkTi==n6Lgq_MvLK7EQoeWFqBEz6b8sQ@mail.gmail.com>
	<75CA4071-26EC-4B08-8965-66C855330CF2@gmail.com>
	<BANLkTim_=LoyLm9yOb=sgwGBjHSNRbTiMw@mail.gmail.com>
	<13EC5F35-9256-4213-B2D6-59F6CE25446B@gmail.com>
Message-ID: <BANLkTikuHsH+E2Ed45ZwNzMSz9o3VZ+JrQ@mail.gmail.com>

On Mon, Apr 4, 2011 at 8:57 AM, Paul Anton Letnes <
paul.anton.letnes at gmail.com> wrote:

>
> On 4. apr. 2011, at 16.42, Charles R Harris wrote:
>
> >
> >
> > On Mon, Apr 4, 2011 at 8:29 AM, Paul Anton Letnes <
> paul.anton.letnes at gmail.com> wrote:
> >
> > On 4. apr. 2011, at 15.34, Charles R Harris wrote:
> >
> > >
> > >
> > > On Sun, Apr 3, 2011 at 4:35 AM, Paul Anton Letnes <
> paul.anton.letnes at gmail.com> wrote:
> > > Hi.
> > >
> > > When looking at the loadtxt/savetxt tickets, I noticed that the 're'
> module is imported in an odd place. I therefore suggest that this import is
> moved to the top of the file, in order to gather these as much as possible.
> I find the code easier to read then. After all, there is no 'try / catch' or
> similar to check if the module exists. See patch below. I do not believe any
> tests or tickets are needed - correct me if I am wrong.
> > >
> > > Cheers,
> > > Paul.
> > >
> > >
> > > --- a/numpy/lib/npyio.py        Sat Apr 02 20:19:55 2011 -0600
> > > +++ b/numpy/lib/npyio.py        Sun Apr 03 12:30:02 2011 +0200
> > > @@ -6,6 +6,7 @@
> > >  import format
> > >  import sys
> > >  import os
> > > +import re
> > >  import sys
> > >  import itertools
> > >  import warnings
> > > @@ -956,7 +957,6 @@
> > >         if own_fh:
> > >             fh.close()
> > >
> > > -import re
> > >  def fromregex(file, regexp, dtype):
> > >     """
> > >     Construct an array from a text file, using regular expression
> parsing.
> > >
> > > If you want to see a lot of other small things for cleanups, run
> pyflakes or pylint on the files in numpy/lib
> > >
> > > Chuck
> > > _______________________________________________
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion at scipy.org
> > > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> > Are you suggesting that I should do this and submit one or more patches?
> >
> > That would be great, and a good way to get into numpy development.
> >
> >
> > By the way - what is a suggested form of submitting patches? Using e-mail
> seems a bit clumsy to me.
> >
> >
> > The best thing would be to setup to do your work on github and then issue
> pull requests, see the directions here:
> http://docs.scipy.org/doc/numpy/dev/gitwash/development_setup.html. If you
> are on linux or mac getting set up should be pretty easy. Windows may be
> trickier, I don't have experience doing development on that platform.
>
> I'm on a mac, and I've got git. The skills to use it will have to grow with
> time.
>
> I submitted a pull request on this particular patch - let me know if I got
> it right!
>
>
Looked good, I went ahead and applied it. Mind, we aren't always so
responsive ;)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110404/aff118ad/attachment.html>

From charlesr.harris at gmail.com  Mon Apr  4 11:47:27 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 4 Apr 2011 09:47:27 -0600
Subject: [Numpy-discussion] Import patch
In-Reply-To: <BANLkTikuHsH+E2Ed45ZwNzMSz9o3VZ+JrQ@mail.gmail.com>
References: <4A5FDFED-67FE-4782-9399-02D91D415C34@gmail.com>
	<BANLkTi==n6Lgq_MvLK7EQoeWFqBEz6b8sQ@mail.gmail.com>
	<75CA4071-26EC-4B08-8965-66C855330CF2@gmail.com>
	<BANLkTim_=LoyLm9yOb=sgwGBjHSNRbTiMw@mail.gmail.com>
	<13EC5F35-9256-4213-B2D6-59F6CE25446B@gmail.com>
	<BANLkTikuHsH+E2Ed45ZwNzMSz9o3VZ+JrQ@mail.gmail.com>
Message-ID: <BANLkTi=gX4Nad6KfCb14OGngL16R97=UuQ@mail.gmail.com>

On Mon, Apr 4, 2011 at 9:33 AM, Charles R Harris
<charlesr.harris at gmail.com>wrote:

>
>
> On Mon, Apr 4, 2011 at 8:57 AM, Paul Anton Letnes <
> paul.anton.letnes at gmail.com> wrote:
>
>>
>> On 4. apr. 2011, at 16.42, Charles R Harris wrote:
>>
>> >
>> >
>> > On Mon, Apr 4, 2011 at 8:29 AM, Paul Anton Letnes <
>> paul.anton.letnes at gmail.com> wrote:
>> >
>> > On 4. apr. 2011, at 15.34, Charles R Harris wrote:
>> >
>> > >
>> > >
>> > > On Sun, Apr 3, 2011 at 4:35 AM, Paul Anton Letnes <
>> paul.anton.letnes at gmail.com> wrote:
>> > > Hi.
>> > >
>> > > When looking at the loadtxt/savetxt tickets, I noticed that the 're'
>> module is imported in an odd place. I therefore suggest that this import is
>> moved to the top of the file, in order to gather these as much as possible.
>> I find the code easier to read then. After all, there is no 'try / catch' or
>> similar to check if the module exists. See patch below. I do not believe any
>> tests or tickets are needed - correct me if I am wrong.
>> > >
>> > > Cheers,
>> > > Paul.
>> > >
>> > >
>> > > --- a/numpy/lib/npyio.py        Sat Apr 02 20:19:55 2011 -0600
>> > > +++ b/numpy/lib/npyio.py        Sun Apr 03 12:30:02 2011 +0200
>> > > @@ -6,6 +6,7 @@
>> > >  import format
>> > >  import sys
>> > >  import os
>> > > +import re
>> > >  import sys
>> > >  import itertools
>> > >  import warnings
>> > > @@ -956,7 +957,6 @@
>> > >         if own_fh:
>> > >             fh.close()
>> > >
>> > > -import re
>> > >  def fromregex(file, regexp, dtype):
>> > >     """
>> > >     Construct an array from a text file, using regular expression
>> parsing.
>> > >
>> > > If you want to see a lot of other small things for cleanups, run
>> pyflakes or pylint on the files in numpy/lib
>> > >
>> > > Chuck
>> > > _______________________________________________
>> > > NumPy-Discussion mailing list
>> > > NumPy-Discussion at scipy.org
>> > > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>> > Are you suggesting that I should do this and submit one or more patches?
>> >
>> > That would be great, and a good way to get into numpy development.
>> >
>> >
>> > By the way - what is a suggested form of submitting patches? Using
>> e-mail seems a bit clumsy to me.
>> >
>> >
>> > The best thing would be to setup to do your work on github and then
>> issue pull requests, see the directions here:
>> http://docs.scipy.org/doc/numpy/dev/gitwash/development_setup.html. If
>> you are on linux or mac getting set up should be pretty easy. Windows may be
>> trickier, I don't have experience doing development on that platform.
>>
>> I'm on a mac, and I've got git. The skills to use it will have to grow
>> with time.
>>
>> I submitted a pull request on this particular patch - let me know if I got
>> it right!
>>
>>
> Looked good, I went ahead and applied it. Mind, we aren't always so
> responsive ;)
>
>
One other thing, you should do even the trivial stuff in a branch, keeping
the master branch of your numpy fork updated from the numpy repository
directly. That way your diffs will based on numpy instead of your own fork.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110404/c24a21a9/attachment.html>

From bsouthey at gmail.com  Mon Apr  4 11:59:07 2011
From: bsouthey at gmail.com (Bruce Southey)
Date: Mon, 04 Apr 2011 10:59:07 -0500
Subject: [Numpy-discussion] loadtxt/savetxt tickets
In-Reply-To: <CB9C54E6-4124-49EB-965A-6C9BAA2C7D1E@astro.physik.uni-goettingen.de>
References: <AANLkTi=gJtN2E2+5YqzYSSH_2XnjOdX8AKudv0An43M7@mail.gmail.com>	<2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com>	<5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de>	<F5B15BD5-88D9-4AEF-BD02-2516338B037B@gmail.com>	<AANLkTimFaYbWqUMFkc0GqE80O86ehFCyqvxMed=iy5SC@mail.gmail.com>	<AANLkTinezEqay7zg1n3FiLRjVfNNmUMsTVfRqp6-W61=@mail.gmail.com>
	<CB9C54E6-4124-49EB-965A-6C9BAA2C7D1E@astro.physik.uni-goettingen.de>
Message-ID: <4D99EACB.6080000@gmail.com>

On 03/31/2011 12:02 PM, Derek Homeier wrote:
> On 31 Mar 2011, at 17:03, Bruce Southey wrote:
>
>> This is an invalid ticket because the docstring clearly states that in
>> 3 different, yet critical places, that missing values are not handled
>> here:
>>
>> "Each row in the text file must have the same number of values."
>> "genfromtxt : Load data with missing values handled as specified."
>> "   This function aims to be a fast reader for simply formatted
>> files.  The
>>     `genfromtxt` function provides more sophisticated handling of,
>> e.g.,
>>     lines with missing values."
>>
>> Really I am trying to separate the usage of loadtxt and genfromtxt to
>> avoid unnecessary duplication and confusion. Part of this is
>> historical because loadtxt was added in 2007 and genfromtxt was added
>> in 2009. So really certain features of loadtxt have been  'kept' for
>> backwards compatibility purposes yet these features can be 'abused' to
>> handle missing data. But I really consider that any missing values
>> should cause loadtxt to fail.
>>
> OK, I was not aware of the design issues of loadtxt vs. genfromtxt -
> you could probably say also for historical reasons since I have not
> used genfromtxt much so far.
> Anyway the docstring statement "Converters can also be used to
>           provide a default value for missing data:"
> then appears quite misleading, or an invitation to abuse, if you will.
> This should better be removed from the documentation then, or users
> explicitly discouraged from using converters instead of genfromtxt
> (I don't see how you could completely prevent using converters in
> this way).
>
>> The patch is incorrect because it should not include a space in the
>> split() as indicated in the comment by the original reporter. Of
> The split('\r\n') alone caused test_dtype_with_object(self) to fail,
> probably
> because it relies on stripping the blanks. But maybe the test is ill-
> formed?
>
>> course a corrected patch alone still is not sufficient to address the
>> problem without the user providing the correct converter. Also you
>> start to run into problems with multiple delimiters (such as one space
>> versus two spaces) so you start down the path to add all the features
>> that duplicate genfromtxt.
> Given that genfromtxt provides that functionality more conveniently,
> I agree again users should be encouraged to use this instead of
> converters.
> But the actual tab-problem causes in fact an issue not related to
> missing
> values at all (well, depending on what you call a missing value).
> I am describing an example on the ticket.
>
> Cheers,
> 					Derek
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
Okay I see that 1071 got closed which I am fine with.

I think that your following example should be a test because the two 
spaces should not be removed with a tab delimiter:
np.loadtxt(StringIO("aa\tbb\n \t \ncc\t"), delimiter='\t', 
dtype=np.dtype([('label', 'S4'), ('comment', 'S4')]))

Thanks very much for fixing this!
Bruce





From charlesr.harris at gmail.com  Mon Apr  4 12:20:33 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 4 Apr 2011 10:20:33 -0600
Subject: [Numpy-discussion] loadtxt/savetxt tickets
In-Reply-To: <4D99EACB.6080000@gmail.com>
References: <AANLkTi=gJtN2E2+5YqzYSSH_2XnjOdX8AKudv0An43M7@mail.gmail.com>
	<2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com>
	<5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de>
	<F5B15BD5-88D9-4AEF-BD02-2516338B037B@gmail.com>
	<AANLkTimFaYbWqUMFkc0GqE80O86ehFCyqvxMed=iy5SC@mail.gmail.com>
	<AANLkTinezEqay7zg1n3FiLRjVfNNmUMsTVfRqp6-W61=@mail.gmail.com>
	<CB9C54E6-4124-49EB-965A-6C9BAA2C7D1E@astro.physik.uni-goettingen.de>
	<4D99EACB.6080000@gmail.com>
Message-ID: <BANLkTimjT1kAjUSoWuFs84Pq6F9biTCZDQ@mail.gmail.com>

On Mon, Apr 4, 2011 at 9:59 AM, Bruce Southey <bsouthey at gmail.com> wrote:

> On 03/31/2011 12:02 PM, Derek Homeier wrote:
> > On 31 Mar 2011, at 17:03, Bruce Southey wrote:
> >
> >> This is an invalid ticket because the docstring clearly states that in
> >> 3 different, yet critical places, that missing values are not handled
> >> here:
> >>
> >> "Each row in the text file must have the same number of values."
> >> "genfromtxt : Load data with missing values handled as specified."
> >> "   This function aims to be a fast reader for simply formatted
> >> files.  The
> >>     `genfromtxt` function provides more sophisticated handling of,
> >> e.g.,
> >>     lines with missing values."
> >>
> >> Really I am trying to separate the usage of loadtxt and genfromtxt to
> >> avoid unnecessary duplication and confusion. Part of this is
> >> historical because loadtxt was added in 2007 and genfromtxt was added
> >> in 2009. So really certain features of loadtxt have been  'kept' for
> >> backwards compatibility purposes yet these features can be 'abused' to
> >> handle missing data. But I really consider that any missing values
> >> should cause loadtxt to fail.
> >>
> > OK, I was not aware of the design issues of loadtxt vs. genfromtxt -
> > you could probably say also for historical reasons since I have not
> > used genfromtxt much so far.
> > Anyway the docstring statement "Converters can also be used to
> >           provide a default value for missing data:"
> > then appears quite misleading, or an invitation to abuse, if you will.
> > This should better be removed from the documentation then, or users
> > explicitly discouraged from using converters instead of genfromtxt
> > (I don't see how you could completely prevent using converters in
> > this way).
> >
> >> The patch is incorrect because it should not include a space in the
> >> split() as indicated in the comment by the original reporter. Of
> > The split('\r\n') alone caused test_dtype_with_object(self) to fail,
> > probably
> > because it relies on stripping the blanks. But maybe the test is ill-
> > formed?
> >
> >> course a corrected patch alone still is not sufficient to address the
> >> problem without the user providing the correct converter. Also you
> >> start to run into problems with multiple delimiters (such as one space
> >> versus two spaces) so you start down the path to add all the features
> >> that duplicate genfromtxt.
> > Given that genfromtxt provides that functionality more conveniently,
> > I agree again users should be encouraged to use this instead of
> > converters.
> > But the actual tab-problem causes in fact an issue not related to
> > missing
> > values at all (well, depending on what you call a missing value).
> > I am describing an example on the ticket.
> >
> > Cheers,
> >                                       Derek
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> Okay I see that 1071 got closed which I am fine with.
>
> I think that your following example should be a test because the two
> spaces should not be removed with a tab delimiter:
> np.loadtxt(StringIO("aa\tbb\n \t \ncc\t"), delimiter='\t',
> dtype=np.dtype([('label', 'S4'), ('comment', 'S4')]))
>
>
Make a test and we'll put it in.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110404/e20a461b/attachment.html>

From pgmdevlist at gmail.com  Mon Apr  4 12:29:17 2011
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 4 Apr 2011 18:29:17 +0200
Subject: [Numpy-discussion] loadtxt/savetxt tickets
In-Reply-To: <4D99EACB.6080000@gmail.com>
References: <AANLkTi=gJtN2E2+5YqzYSSH_2XnjOdX8AKudv0An43M7@mail.gmail.com>	<2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com>	<5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de>	<F5B15BD5-88D9-4AEF-BD02-2516338B037B@gmail.com>	<AANLkTimFaYbWqUMFkc0GqE80O86ehFCyqvxMed=iy5SC@mail.gmail.com>	<AANLkTinezEqay7zg1n3FiLRjVfNNmUMsTVfRqp6-W61=@mail.gmail.com>
	<CB9C54E6-4124-49EB-965A-6C9BAA2C7D1E@astro.physik.uni-goettingen.de>
	<4D99EACB.6080000@gmail.com>
Message-ID: <08782B95-3084-4EC7-A507-D18A2733BBA5@gmail.com>

Y'all,
Just a quick word before disappearing again:
genfromtxt was designed to support files that don't have a well-defined structure (eg, the number and format of the column is unknown, some entries may be missing, and so forth). That was initially just an improved rec2csv from matplotlib back then, which eventually got rewritten a lot. 
On the other hand, loadtxt was designed to support the simpler case of well-structred files (you know the number of columns beforehand, you know their type...). It's simpler, but also far faster than genfromtxt, and it should stay like that. Missing values (and I'm talking about empty "cells") should not be covered by loadtxt, that's a job for genfromtxt: just raise an exception asking the user to try with another function. 
Anyhow, sorry for not being responsive, I don't have much time to work on numpy these days (which saddens me but pays my bills) but don't hesitate to contact me if there's a real issue somewhere.
P.




From bsouthey at gmail.com  Mon Apr  4 13:01:19 2011
From: bsouthey at gmail.com (Bruce Southey)
Date: Mon, 04 Apr 2011 12:01:19 -0500
Subject: [Numpy-discussion] loadtxt/savetxt tickets
In-Reply-To: <BANLkTimjT1kAjUSoWuFs84Pq6F9biTCZDQ@mail.gmail.com>
References: <AANLkTi=gJtN2E2+5YqzYSSH_2XnjOdX8AKudv0An43M7@mail.gmail.com>	<2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com>	<5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de>	<F5B15BD5-88D9-4AEF-BD02-2516338B037B@gmail.com>	<AANLkTimFaYbWqUMFkc0GqE80O86ehFCyqvxMed=iy5SC@mail.gmail.com>	<AANLkTinezEqay7zg1n3FiLRjVfNNmUMsTVfRqp6-W61=@mail.gmail.com>	<CB9C54E6-4124-49EB-965A-6C9BAA2C7D1E@astro.physik.uni-goettingen.de>	<4D99EACB.6080000@gmail.com>
	<BANLkTimjT1kAjUSoWuFs84Pq6F9biTCZDQ@mail.gmail.com>
Message-ID: <4D99F95F.7090200@gmail.com>

On 04/04/2011 11:20 AM, Charles R Harris wrote:
>
>
> On Mon, Apr 4, 2011 at 9:59 AM, Bruce Southey <bsouthey at gmail.com 
> <mailto:bsouthey at gmail.com>> wrote:
>
>     On 03/31/2011 12:02 PM, Derek Homeier wrote:
>     > On 31 Mar 2011, at 17:03, Bruce Southey wrote:
>     >
>     >> This is an invalid ticket because the docstring clearly states
>     that in
>     >> 3 different, yet critical places, that missing values are not
>     handled
>     >> here:
>     >>
>     >> "Each row in the text file must have the same number of values."
>     >> "genfromtxt : Load data with missing values handled as specified."
>     >> "   This function aims to be a fast reader for simply formatted
>     >> files.  The
>     >>     `genfromtxt` function provides more sophisticated handling of,
>     >> e.g.,
>     >>     lines with missing values."
>     >>
>     >> Really I am trying to separate the usage of loadtxt and
>     genfromtxt to
>     >> avoid unnecessary duplication and confusion. Part of this is
>     >> historical because loadtxt was added in 2007 and genfromtxt was
>     added
>     >> in 2009. So really certain features of loadtxt have been
>      'kept' for
>     >> backwards compatibility purposes yet these features can be
>     'abused' to
>     >> handle missing data. But I really consider that any missing values
>     >> should cause loadtxt to fail.
>     >>
>     > OK, I was not aware of the design issues of loadtxt vs. genfromtxt -
>     > you could probably say also for historical reasons since I have not
>     > used genfromtxt much so far.
>     > Anyway the docstring statement "Converters can also be used to
>     >           provide a default value for missing data:"
>     > then appears quite misleading, or an invitation to abuse, if you
>     will.
>     > This should better be removed from the documentation then, or users
>     > explicitly discouraged from using converters instead of genfromtxt
>     > (I don't see how you could completely prevent using converters in
>     > this way).
>     >
>     >> The patch is incorrect because it should not include a space in the
>     >> split() as indicated in the comment by the original reporter. Of
>     > The split('\r\n') alone caused test_dtype_with_object(self) to fail,
>     > probably
>     > because it relies on stripping the blanks. But maybe the test is
>     ill-
>     > formed?
>     >
>     >> course a corrected patch alone still is not sufficient to
>     address the
>     >> problem without the user providing the correct converter. Also you
>     >> start to run into problems with multiple delimiters (such as
>     one space
>     >> versus two spaces) so you start down the path to add all the
>     features
>     >> that duplicate genfromtxt.
>     > Given that genfromtxt provides that functionality more conveniently,
>     > I agree again users should be encouraged to use this instead of
>     > converters.
>     > But the actual tab-problem causes in fact an issue not related to
>     > missing
>     > values at all (well, depending on what you call a missing value).
>     > I am describing an example on the ticket.
>     >
>     > Cheers,
>     >                                       Derek
>     >
>     > _______________________________________________
>     > NumPy-Discussion mailing list
>     > NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>     Okay I see that 1071 got closed which I am fine with.
>
>     I think that your following example should be a test because the two
>     spaces should not be removed with a tab delimiter:
>     np.loadtxt(StringIO("aa\tbb\n \t \ncc\t"), delimiter='\t',
>     dtype=np.dtype([('label', 'S4'), ('comment', 'S4')]))
>
>
> Make a test and we'll put it in.
>
> Chuck
>
>
I know!
Trying to write one made me realize that loadtxt is not handling string 
arrays correctly. So I have to check more on this as I think loadtxt is 
giving a 1-d array instead of a 2-d array.

I do agree with you Pierre but this is a nice corner case that Derek 
raised where a space does not necessarily mean a missing value when 
there is a tab delimiter:

data = StringIO("aa\tbb\n \t \ncc\tdd")
dt=np.dtype([('label', 'S2'), ('comment', 'S2')])
test = np.loadtxt(data, delimiter="\t", dtype=dt)
control = np.array([['aa','bb'], [' ', ' '],['cc','dd']], dtype=dt)

So 'test' and 'control' should give the same array.

Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110404/a31b74c9/attachment.html>

From charlesr.harris at gmail.com  Mon Apr  4 13:38:29 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 4 Apr 2011 11:38:29 -0600
Subject: [Numpy-discussion] loadtxt/savetxt tickets
In-Reply-To: <4D99F95F.7090200@gmail.com>
References: <AANLkTi=gJtN2E2+5YqzYSSH_2XnjOdX8AKudv0An43M7@mail.gmail.com>
	<2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com>
	<5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de>
	<F5B15BD5-88D9-4AEF-BD02-2516338B037B@gmail.com>
	<AANLkTimFaYbWqUMFkc0GqE80O86ehFCyqvxMed=iy5SC@mail.gmail.com>
	<AANLkTinezEqay7zg1n3FiLRjVfNNmUMsTVfRqp6-W61=@mail.gmail.com>
	<CB9C54E6-4124-49EB-965A-6C9BAA2C7D1E@astro.physik.uni-goettingen.de>
	<4D99EACB.6080000@gmail.com>
	<BANLkTimjT1kAjUSoWuFs84Pq6F9biTCZDQ@mail.gmail.com>
	<4D99F95F.7090200@gmail.com>
Message-ID: <BANLkTinHe8jpNcrTP1MhKCf_yO5knhCKoA@mail.gmail.com>

On Mon, Apr 4, 2011 at 11:01 AM, Bruce Southey <bsouthey at gmail.com> wrote:

>  On 04/04/2011 11:20 AM, Charles R Harris wrote:
>
>
>
> On Mon, Apr 4, 2011 at 9:59 AM, Bruce Southey <bsouthey at gmail.com> wrote:
>
>> On 03/31/2011 12:02 PM, Derek Homeier wrote:
>>  > On 31 Mar 2011, at 17:03, Bruce Southey wrote:
>> >
>> >> This is an invalid ticket because the docstring clearly states that in
>> >> 3 different, yet critical places, that missing values are not handled
>> >> here:
>> >>
>> >> "Each row in the text file must have the same number of values."
>> >> "genfromtxt : Load data with missing values handled as specified."
>> >> "   This function aims to be a fast reader for simply formatted
>> >> files.  The
>> >>     `genfromtxt` function provides more sophisticated handling of,
>> >> e.g.,
>> >>     lines with missing values."
>> >>
>> >> Really I am trying to separate the usage of loadtxt and genfromtxt to
>> >> avoid unnecessary duplication and confusion. Part of this is
>> >> historical because loadtxt was added in 2007 and genfromtxt was added
>> >> in 2009. So really certain features of loadtxt have been  'kept' for
>> >> backwards compatibility purposes yet these features can be 'abused' to
>> >> handle missing data. But I really consider that any missing values
>> >> should cause loadtxt to fail.
>> >>
>> > OK, I was not aware of the design issues of loadtxt vs. genfromtxt -
>> > you could probably say also for historical reasons since I have not
>> > used genfromtxt much so far.
>> > Anyway the docstring statement "Converters can also be used to
>> >           provide a default value for missing data:"
>> > then appears quite misleading, or an invitation to abuse, if you will.
>> > This should better be removed from the documentation then, or users
>> > explicitly discouraged from using converters instead of genfromtxt
>> > (I don't see how you could completely prevent using converters in
>> > this way).
>> >
>> >> The patch is incorrect because it should not include a space in the
>> >> split() as indicated in the comment by the original reporter. Of
>> > The split('\r\n') alone caused test_dtype_with_object(self) to fail,
>> > probably
>> > because it relies on stripping the blanks. But maybe the test is ill-
>> > formed?
>> >
>> >> course a corrected patch alone still is not sufficient to address the
>> >> problem without the user providing the correct converter. Also you
>> >> start to run into problems with multiple delimiters (such as one space
>> >> versus two spaces) so you start down the path to add all the features
>> >> that duplicate genfromtxt.
>> > Given that genfromtxt provides that functionality more conveniently,
>> > I agree again users should be encouraged to use this instead of
>> > converters.
>> > But the actual tab-problem causes in fact an issue not related to
>> > missing
>> > values at all (well, depending on what you call a missing value).
>> > I am describing an example on the ticket.
>> >
>> > Cheers,
>> >                                       Derek
>> >
>> > _______________________________________________
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>  Okay I see that 1071 got closed which I am fine with.
>>
>> I think that your following example should be a test because the two
>> spaces should not be removed with a tab delimiter:
>> np.loadtxt(StringIO("aa\tbb\n \t \ncc\t"), delimiter='\t',
>> dtype=np.dtype([('label', 'S4'), ('comment', 'S4')]))
>>
>>
> Make a test and we'll put it in.
>
> Chuck
>
>
>  I know!
> Trying to write one made me realize that loadtxt is not handling string
> arrays correctly. So I have to check more on this as I think loadtxt is
> giving a 1-d array instead of a 2-d array.
>
>
Tests often have that side effect.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110404/3e91360d/attachment.html>

From bsouthey at gmail.com  Mon Apr  4 15:01:10 2011
From: bsouthey at gmail.com (Bruce Southey)
Date: Mon, 04 Apr 2011 14:01:10 -0500
Subject: [Numpy-discussion] How to create structured char arrays?
Message-ID: <4D9A1576.90709@gmail.com>

Hi,
I think that I have resolved my issue down to creating a structured 
string array.
I am using numpy version '2.0.0.dev-3c338cb'.

Without a structured array, it should be a 2 by 2 array:
 >>> np.array([('a','b'),('c','d')])
array([['a', 'b'],
        ['c', 'd']],
       dtype='|S1')
 >>> np.array([('a','b'),('c','d')], dtype='a2')
array([['a', 'b'],
        ['c', 'd']],
       dtype='|S2')


It does not work as a structured array:
 >>> np.array([('a','b'),('c','d')], dtype=[('label','a2'), ('Value', 
'a4')])
array([('a', 'b'), ('c', 'd')],
       dtype=[('label', '|S2'), ('Value', '|S4')])

What is the correct dtype argument to use in this case?

Thanks
Bruce




From dsdale24 at gmail.com  Mon Apr  4 15:15:37 2011
From: dsdale24 at gmail.com (Darren Dale)
Date: Mon, 4 Apr 2011 15:15:37 -0400
Subject: [Numpy-discussion] python3 setup.py install fails with git
	maint/1.6.x
Message-ID: <BANLkTimEwBpvnZe+h6UT+9Qg03L1r+a4QQ@mail.gmail.com>

I just checkout out the 1.6 branch, attempted to install with python3:

RefactoringTool: Line 695: You should use a for loop here
Running from numpy source directory.Traceback (most recent call last):
  File "setup.py", line 196, in <module>
    setup_package()
  File "setup.py", line 170, in setup_package
    write_version_py()
  File "setup.py", line 117, in write_version_py
    from numpy.version import git_revision as GIT_REVISION
ImportError: cannot import name git_revision

Next, I built and installed with python2, with no problems. Then I
attempted to install with python3 again, at which point git_revision
was importable, presumably because it was provided during the python2
build.

Darren


From jsseabold at gmail.com  Mon Apr  4 15:22:04 2011
From: jsseabold at gmail.com (Skipper Seabold)
Date: Mon, 4 Apr 2011 15:22:04 -0400
Subject: [Numpy-discussion] "Best" dot product along an axis?
Message-ID: <BANLkTikOWCiELunSGPkmXC5714UYcO-n+g@mail.gmail.com>

Going through some of the recent threads on similar problems, I'm
trying to discern which is "best."

I have X is T x i, theta is T x i x j. I want a T by j array that
contains X[t].T.dot(theta[t]) along axis 0.

Say,

T,i,j = 166,7,3

X = np.random.random((T,i))
theta = np.random.random((T,i,j))

# Using sum

F1 = (X[:,:,None] * theta).sum(1)

# Using a loop

F2 = np.zeros((T,j))
for t in range(T):
    for j_idx in range(j):
        for i_idx in range(7):
            F2[t,j_idx] += X[t,i_idx] * theta[t,i_idx,j_idx]

# One way with dot, keeps an extra T index
F3 = np.squeeze(np.dot(X[:,None,:],theta)).reshape(-1,3)[::167]

# the above is way fast than
F3_2 = np.dot(X,theta).reshape(-1,3)[::167]

# zipped arrays
F4 = np.asarray([np.dot(x,theta_i) for x,theta_i in zip(X,theta)])

It seems that F3 is the fastest way to do this given T,i,j, but it
seems inefficient to compute things I don't want. Does it make sense
that F3 is faster than F3_2? I'm using generic Ubuntu ATLAS on this
machine. Is there a more efficient way to do this? I'm not coming up
with one.

Skipper


From derek at astro.physik.uni-goettingen.de  Mon Apr  4 15:30:59 2011
From: derek at astro.physik.uni-goettingen.de (Derek Homeier)
Date: Mon, 4 Apr 2011 21:30:59 +0200
Subject: [Numpy-discussion] How to create structured char arrays?
In-Reply-To: <4D9A1576.90709@gmail.com>
References: <4D9A1576.90709@gmail.com>
Message-ID: <F805E695-D1BB-4397-B93A-AFD51044446F@astro.physik.uni-goettingen.de>

Hi Bruce,

> I think that I have resolved my issue down to creating a structured
> string array.
> I am using numpy version '2.0.0.dev-3c338cb'.
>
> Without a structured array, it should be a 2 by 2 array:
>>>> np.array([('a','b'),('c','d')])
> array([['a', 'b'],
>        ['c', 'd']],
>       dtype='|S1')
>>>> np.array([('a','b'),('c','d')], dtype='a2')
> array([['a', 'b'],
>        ['c', 'd']],
>       dtype='|S2')
>
>
> It does not work as a structured array:
>>>> np.array([('a','b'),('c','d')], dtype=[('label','a2'), ('Value',
> 'a4')])
> array([('a', 'b'), ('c', 'd')],
>       dtype=[('label', '|S2'), ('Value', '|S4')])
>
> What is the correct dtype argument to use in this case?
>
what do you mean by "does not work" - this should give you a 1 by 2  
array
with 2 fields, i.e.

 >>> c['label']
array(['a', 'c'],
       dtype='|S2')
 >>> c['Value']
array(['b', 'd'],
       dtype='|S4')

and this should be the same as returned by loadtxt for

 >>> dt=np.dtype([('label', 'S2'), ('Value', 'S4')])
 >>> d = np.loadtxt(StringIO("a\tb\nc\td"), delimiter="\t", dtype=dt)

right? Maybe I should modify my test in #1071 to compare to the full  
array, like

a = np.array([(1,2,3, asbytes('start ')),
          (4,5,6, asbytes('  ')), (7,8,9.5, asbytes(''))])
assert_array_equal(x, a)


HTH,
					Derek



From ralf.gommers at googlemail.com  Mon Apr  4 15:31:36 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Mon, 4 Apr 2011 21:31:36 +0200
Subject: [Numpy-discussion] python3 setup.py install fails with git
	maint/1.6.x
In-Reply-To: <BANLkTimEwBpvnZe+h6UT+9Qg03L1r+a4QQ@mail.gmail.com>
References: <BANLkTimEwBpvnZe+h6UT+9Qg03L1r+a4QQ@mail.gmail.com>
Message-ID: <BANLkTi=1Xm5HUpSASBfbp46TOP+LSwxrqQ@mail.gmail.com>

On Mon, Apr 4, 2011 at 9:15 PM, Darren Dale <dsdale24 at gmail.com> wrote:
> I just checkout out the 1.6 branch, attempted to install with python3:

I hope you mean the 1.6.0b1 tarball, not the current branch head? This
problem is (or should have been) fixed.

Just tried again with python3.2 and 1.6.0b2, installs fine. The line
it fails on is only reached when a numpy/version.py exists, which is
the case for source releases or if you did not clean your local git
repo before building.

If it is still a problem, can you check what's in version.py and give
us details of OS etc.?

Cheers,
Ralf


> RefactoringTool: Line 695: You should use a for loop here
> Running from numpy source directory.Traceback (most recent call last):
> ?File "setup.py", line 196, in <module>
> ? ?setup_package()
> ?File "setup.py", line 170, in setup_package
> ? ?write_version_py()
> ?File "setup.py", line 117, in write_version_py
> ? ?from numpy.version import git_revision as GIT_REVISION
> ImportError: cannot import name git_revision
>
> Next, I built and installed with python2, with no problems. Then I
> attempted to install with python3 again, at which point git_revision
> was importable, presumably because it was provided during the python2
> build.
>
> Darren
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From dsdale24 at gmail.com  Mon Apr  4 15:41:17 2011
From: dsdale24 at gmail.com (Darren Dale)
Date: Mon, 4 Apr 2011 15:41:17 -0400
Subject: [Numpy-discussion] python3 setup.py install fails with git
	maint/1.6.x
In-Reply-To: <BANLkTi=1Xm5HUpSASBfbp46TOP+LSwxrqQ@mail.gmail.com>
References: <BANLkTimEwBpvnZe+h6UT+9Qg03L1r+a4QQ@mail.gmail.com>
	<BANLkTi=1Xm5HUpSASBfbp46TOP+LSwxrqQ@mail.gmail.com>
Message-ID: <BANLkTikq4Ya_EU9xqa1zHktz9UPiiVA=7Q@mail.gmail.com>

On Mon, Apr 4, 2011 at 3:31 PM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
> On Mon, Apr 4, 2011 at 9:15 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>> I just checkout out the 1.6 branch, attempted to install with python3:
>
> I hope you mean the 1.6.0b1 tarball, not the current branch head? This
> problem is (or should have been) fixed.

It was the branch HEAD...

> Just tried again with python3.2 and 1.6.0b2, installs fine. The line
> it fails on is only reached when a numpy/version.py exists, which is
> the case for source releases or if you did not clean your local git
> repo before building.

... but I think this was the case. I just deleted numpy/version.py,
and build/, and now everything is ok.

Darren


From charlesr.harris at gmail.com  Mon Apr  4 15:44:14 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 4 Apr 2011 13:44:14 -0600
Subject: [Numpy-discussion] python3 setup.py install fails with git
	maint/1.6.x
In-Reply-To: <BANLkTi=1Xm5HUpSASBfbp46TOP+LSwxrqQ@mail.gmail.com>
References: <BANLkTimEwBpvnZe+h6UT+9Qg03L1r+a4QQ@mail.gmail.com>
	<BANLkTi=1Xm5HUpSASBfbp46TOP+LSwxrqQ@mail.gmail.com>
Message-ID: <BANLkTimWWii-MASyHxy5+YU_3k1LGUEnag@mail.gmail.com>

On Mon, Apr 4, 2011 at 1:31 PM, Ralf Gommers <ralf.gommers at googlemail.com>wrote:

> On Mon, Apr 4, 2011 at 9:15 PM, Darren Dale <dsdale24 at gmail.com> wrote:
> > I just checkout out the 1.6 branch, attempted to install with python3:
>
> I hope you mean the 1.6.0b1 tarball, not the current branch head? This
> problem is (or should have been) fixed.
>
> Just tried again with python3.2 and 1.6.0b2, installs fine. The line
> it fails on is only reached when a numpy/version.py exists, which is
> the case for source releases or if you did not clean your local git
> repo before building.
>
>
I had to remove the build directory to get past that problem.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110404/46a5aad5/attachment.html>

From derek at astro.physik.uni-goettingen.de  Mon Apr  4 15:50:58 2011
From: derek at astro.physik.uni-goettingen.de (Derek Homeier)
Date: Mon, 4 Apr 2011 21:50:58 +0200
Subject: [Numpy-discussion] python3 setup.py install fails with git
	maint/1.6.x
In-Reply-To: <BANLkTikq4Ya_EU9xqa1zHktz9UPiiVA=7Q@mail.gmail.com>
References: <BANLkTimEwBpvnZe+h6UT+9Qg03L1r+a4QQ@mail.gmail.com>
	<BANLkTi=1Xm5HUpSASBfbp46TOP+LSwxrqQ@mail.gmail.com>
	<BANLkTikq4Ya_EU9xqa1zHktz9UPiiVA=7Q@mail.gmail.com>
Message-ID: <3392D51C-1527-4D1B-B0B1-95F37D083DA5@astro.physik.uni-goettingen.de>

On 4 Apr 2011, at 21:41, Darren Dale wrote:

>> Just tried again with python3.2 and 1.6.0b2, installs fine. The line
>> it fails on is only reached when a numpy/version.py exists, which is
>> the case for source releases or if you did not clean your local git
>> repo before building.
>
> ... but I think this was the case. I just deleted numpy/version.py,
> and build/, and now everything is ok.

I just tried the 1.6.0b2 tarball (which was literally just appearing on
sourceforge while your mails were coming in ;-) and setup.py is doing
ok there, too.

Cheers,
						Derek



From ralf.gommers at googlemail.com  Mon Apr  4 16:04:25 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Mon, 4 Apr 2011 22:04:25 +0200
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
Message-ID: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>

Hi,

I am pleased to announce the availability of the second beta of NumPy
1.6.0. Due to the extensive changes in the Numpy core for this
release, the beta testing phase will last at least one month. Please
test this beta and report any problems on the Numpy mailing list.

Many bug fixes went in since beta 1, among which:
- fix installation of source release with python 3
- f2py fixes for assumed shape arrays
- several loadtxt bug fixes and enhancements
- change default floating point error handling from "print" to "warn"
- much more

I quickly counted in the timeline, and in the last few weeks the
number of open tickets has been decreased by over 100. Thanks to
everyone who contributed to this spring cleaning!

Sources and binaries can be found at
http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b2/
For (preliminary) release notes see below.

Enjoy,
Ralf



Note: NumPy 1.6.0 is not yet released.


=========================
NumPy 1.6.0 Release Notes
=========================

This release includes several new features as well as numerous bug fixes and
improved documentation.  It is backward compatible with the 1.5.0 release, and
supports Python 2.4 - 2.7 and 3.1 - 3.2.


Highlights
==========

* Re-introduction of datetime dtype support to deal with dates in arrays.

* A new 16-bit floating point type.

* A new iterator, which improves performance of many functions.


New features
============

New 16-bit floating point type
------------------------------

This release adds support for the IEEE 754-2008 binary16 format, available as
the data type ``numpy.half``.  Within Python, the type behaves similarly to
`float` or `double`, and C extensions can add support for it with the exposed
half-float API.


New iterator
------------

A new iterator has been added, replacing the functionality of the
existing iterator and multi-iterator with a single object and API.
This iterator works well with general memory layouts different from
C or Fortran contiguous, and handles both standard NumPy and
customized broadcasting. The buffering, automatic data type
conversion, and optional output parameters, offered by
ufuncs but difficult to replicate elsewhere, are now exposed by this
iterator.


Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial``
-------------------------------------------------------------------------

Extend the number of polynomials available in the polynomial package. In
addition, a new ``window`` attribute has been added to the classes in
order to specify the range the ``domain`` maps to. This is mostly useful
for the Laguerre, Hermite, and HermiteE polynomials whose natural domains
are infinite and provides a more intuitive way to get the correct mapping
of values without playing unnatural tricks with the domain.


Fortran assumed shape array and size function support in ``numpy.f2py``
-----------------------------------------------------------------------

F2py now supports wrapping Fortran 90 routines that use assumed shape
arrays.  Before such routines could be called from Python but the
corresponding Fortran routines received assumed shape arrays as zero
length arrays which caused unpredicted results. Thanks to Lorenz
H?depohl for pointing out the correct way to interface routines with
assumed shape arrays.

In addition, f2py interprets Fortran expression ``size(array, dim)``
as ``shape(array, dim-1)`` which makes it possible to automatically
wrap Fortran routines that use two argument ``size`` function in
dimension specifications. Before users were forced to apply this
mapping manually.


Other new functions
-------------------

``numpy.ravel_multi_index`` : Converts a multi-index tuple into
an array of flat indices, applying boundary modes to the indices.

``numpy.einsum`` : Evaluate the Einstein summation convention.  Using the
Einstein summation convention, many common multi-dimensional array operations
can be represented in a simple fashion.  This function provides a way compute
such summations.

``numpy.count_nonzero`` : Counts the number of non-zero elements in an array.

``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose
the underlying type promotion used by the ufuncs and other operations to
determine the types of outputs. These improve upon the ``numpy.common_type``
and ``numpy.mintypecode`` which provide similar functionality but do
not match the ufunc implementation.


Changes
=======

Changes and improvements in the numpy core
------------------------------------------

``default error handling``
--------------------------

The default error handling has been change from ``print`` to ``warn`` for
all except for ``underflow``, which remains as ``ignore``.


``numpy.distutils``
-------------------

Several new compilers are supported for building Numpy: the Portland Group
Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C
compiler on Linux.


``numpy.testing``
-----------------

The testing framework gained ``numpy.testing.assert_allclose``, which provides
a more convenient way to compare floating point arrays than
`assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`.


``C API``
---------

In addition to the APIs for the new iterator and half data type, a number
of other additions have been made to the C API. The type promotion
mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``,
``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration
``NPY_CASTING`` has been added which controls what types of casts are
permitted. This is used by the new functions ``PyArray_CanCastArrayTo``
and ``PyArray_CanCastTypeTo``.  A more flexible way to handle
conversion of arbitrary python objects into arrays is exposed by
``PyArray_GetArrayParamsFromObject``.


Deprecated features
===================

The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality
will be replaced by the new "density" keyword.


Removed features
================

``numpy.fft``
-------------

The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`,
which were aliases for the same functions without the 'e' in the name, were
removed.


``numpy.memmap``
----------------

The `sync()` and `close()` methods of memmap were removed.  Use `flush()` and
"del memmap" instead.


``numpy.lib``
-------------

The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``,
``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed.


``numpy.ma``
------------

Several deprecated items were removed from the ``numpy.ma`` module::

  * ``numpy.ma.MaskedArray`` "raw_data" method
  * ``numpy.ma.MaskedArray`` constructor "flag" keyword
  * ``numpy.ma.make_mask`` "flag" keyword
  * ``numpy.ma.allclose`` "fill_value" keyword


``numpy.distutils``
-------------------

The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include``
instead.


From derek at astro.physik.uni-goettingen.de  Mon Apr  4 22:42:35 2011
From: derek at astro.physik.uni-goettingen.de (Derek Homeier)
Date: Tue, 5 Apr 2011 04:42:35 +0200
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
Message-ID: <8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>

Hi all,

On 4 Apr 2011, at 22:04, Ralf Gommers wrote:

> I am pleased to announce the availability of the second beta of NumPy
> 1.6.0. Due to the extensive changes in the Numpy core for this
> release, the beta testing phase will last at least one month. Please
> test this beta and report any problems on the Numpy mailing list.
>
the tests have a number of Python2.4-incompatibilities, one for a file
opening mode and the rest for class declaration styles.

Cheers,
						Derek

Running unit tests for numpy
NumPy version 1.6.0b2
NumPy is installed in /sw/lib/python2.4/site-packages/numpy
Python version 2.4.4 (#1, Jan  5 2011, 03:05:41) [GCC 4.0.1 (Apple  
Inc. build 5493)]
nose version 1.0.0
.....
======================================================================
ERROR: Failure: SyntaxError (invalid syntax (test_multiarray.py, line  
1023))
...
   File "/sw/lib/python2.4/site-packages/numpy/core/tests/ 
test_multiarray.py", line 1023
     class TestPutmask():
                       ^
SyntaxError: invalid syntax

======================================================================
ERROR: Failure: SyntaxError (invalid syntax (test_numeric.py, line  
1068))
...
   File "/sw/lib/python2.4/site-packages/numpy/core/tests/ 
test_numeric.py", line 1068
     class TestAllclose():
                        ^
SyntaxError: invalid syntax

======================================================================
ERROR: Failure: SyntaxError (invalid syntax (test_scalarmath.py, line  
84))
...
   File "/sw/lib/python2.4/site-packages/numpy/core/tests/ 
test_scalarmath.py", line 84
     class TestRepr():
                    ^
SyntaxError: invalid syntax

======================================================================
ERROR: Failure: SyntaxError (invalid syntax (test_twodim_base.py, line  
280))
...
   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/ 
test_twodim_base.py", line 280
     class TestTriuIndices():
                           ^
SyntaxError: invalid syntax

======================================================================
ERROR: Failure: SyntaxError (invalid syntax (test_linalg.py, line 243))
...
   File "/sw/lib/python2.4/site-packages/numpy/linalg/tests/ 
test_linalg.py", line 243
     class TestMatrixPower():
                           ^
SyntaxError: invalid syntax

======================================================================
ERROR: test_gft_filename (test_io.TestFromTxt)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/test_io.py",  
line 1327, in test_gft_filename
     assert_array_equal(np.genfromtxt(name), exp_res)
   File "/sw/lib/python2.4/site-packages/numpy/lib/npyio.py", line  
1235, in genfromtxt
     fhd = iter(np.lib._datasource.open(fname, 'Ub'))
   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",  
line 145, in open
     return ds.open(path, mode)
   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",  
line 477, in open
     return _file_openers[ext](found, mode=mode)
IOError: invalid mode: Ub

----------------------------------------------------------------------
Ran 2648 tests in 28.573s

FAILED (KNOWNFAIL=3, SKIP=21, errors=6)



From charlesr.harris at gmail.com  Mon Apr  4 23:42:35 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 4 Apr 2011 21:42:35 -0600
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
Message-ID: <BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>

On Mon, Apr 4, 2011 at 8:42 PM, Derek Homeier <
derek at astro.physik.uni-goettingen.de> wrote:

> Hi all,
>
> On 4 Apr 2011, at 22:04, Ralf Gommers wrote:
>
> > I am pleased to announce the availability of the second beta of NumPy
> > 1.6.0. Due to the extensive changes in the Numpy core for this
> > release, the beta testing phase will last at least one month. Please
> > test this beta and report any problems on the Numpy mailing list.
> >
> the tests have a number of Python2.4-incompatibilities, one for a file
> opening mode and the rest for class declaration styles.
>
> Cheers,
>                                                Derek
>
> Running unit tests for numpy
> NumPy version 1.6.0b2
> NumPy is installed in /sw/lib/python2.4/site-packages/numpy
> Python version 2.4.4 (#1, Jan  5 2011, 03:05:41) [GCC 4.0.1 (Apple
> Inc. build 5493)]
> nose version 1.0.0
> .....
> ======================================================================
> ERROR: Failure: SyntaxError (invalid syntax (test_multiarray.py, line
> 1023))
> ...
>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
> test_multiarray.py", line 1023
>     class TestPutmask():
>                       ^
> SyntaxError: invalid syntax
>
> ======================================================================
> ERROR: Failure: SyntaxError (invalid syntax (test_numeric.py, line
> 1068))
> ...
>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
> test_numeric.py", line 1068
>     class TestAllclose():
>                        ^
> SyntaxError: invalid syntax
>
> ======================================================================
> ERROR: Failure: SyntaxError (invalid syntax (test_scalarmath.py, line
> 84))
> ...
>   File "/sw/lib/python2.4/site-packages/numpy/core/tests/
> test_scalarmath.py", line 84
>     class TestRepr():
>                    ^
> SyntaxError: invalid syntax
>
> ======================================================================
> ERROR: Failure: SyntaxError (invalid syntax (test_twodim_base.py, line
> 280))
> ...
>   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/
> test_twodim_base.py", line 280
>     class TestTriuIndices():
>                           ^
> SyntaxError: invalid syntax
>
> ======================================================================
> ERROR: Failure: SyntaxError (invalid syntax (test_linalg.py, line 243))
> ...
>   File "/sw/lib/python2.4/site-packages/numpy/linalg/tests/
> test_linalg.py", line 243
>     class TestMatrixPower():
>                           ^
> SyntaxError: invalid syntax
>
>
Subclassing object should fix those, I think.


> ======================================================================
> ERROR: test_gft_filename (test_io.TestFromTxt)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/sw/lib/python2.4/site-packages/numpy/lib/tests/test_io.py",
> line 1327, in test_gft_filename
>     assert_array_equal(np.genfromtxt(name), exp_res)
>   File "/sw/lib/python2.4/site-packages/numpy/lib/npyio.py", line
> 1235, in genfromtxt
>     fhd = iter(np.lib._datasource.open(fname, 'Ub'))
>   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
> line 145, in open
>     return ds.open(path, mode)
>   File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
> line 477, in open
>     return _file_openers[ext](found, mode=mode)
> IOError: invalid mode: Ub
>
>
Guess that wasn't tested before ;) I thought that was strange when I saw it.
The source of the problem is line 2035 in npyio.py. Additionally, Since
genloadtxt needs to have byte strings the 'rb" mode should probably be used.
That works on linux, both for python 2 and python 3, but doing that might
uncover genfromtxt problems on other platforms.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110404/204efdb0/attachment.html>

From josef.pktd at gmail.com  Tue Apr  5 00:03:43 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 5 Apr 2011 00:03:43 -0400
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
Message-ID: <BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>

On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Mon, Apr 4, 2011 at 8:42 PM, Derek Homeier
> <derek at astro.physik.uni-goettingen.de> wrote:
>>
>> Hi all,
>>
>> On 4 Apr 2011, at 22:04, Ralf Gommers wrote:
>>
>> > I am pleased to announce the availability of the second beta of NumPy
>> > 1.6.0. Due to the extensive changes in the Numpy core for this
>> > release, the beta testing phase will last at least one month. Please
>> > test this beta and report any problems on the Numpy mailing list.
>> >
>> the tests have a number of Python2.4-incompatibilities, one for a file
>> opening mode and the rest for class declaration styles.
>>
>> Cheers,
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Derek
>>
>> Running unit tests for numpy
>> NumPy version 1.6.0b2
>> NumPy is installed in /sw/lib/python2.4/site-packages/numpy
>> Python version 2.4.4 (#1, Jan ?5 2011, 03:05:41) [GCC 4.0.1 (Apple
>> Inc. build 5493)]
>> nose version 1.0.0
>> .....
>> ======================================================================
>> ERROR: Failure: SyntaxError (invalid syntax (test_multiarray.py, line
>> 1023))
>> ...
>> ? File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>> test_multiarray.py", line 1023
>> ? ? class TestPutmask():
>> ? ? ? ? ? ? ? ? ? ? ? ^
>> SyntaxError: invalid syntax
>>
>> ======================================================================
>> ERROR: Failure: SyntaxError (invalid syntax (test_numeric.py, line
>> 1068))
>> ...
>> ? File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>> test_numeric.py", line 1068
>> ? ? class TestAllclose():
>> ? ? ? ? ? ? ? ? ? ? ? ?^
>> SyntaxError: invalid syntax
>>
>> ======================================================================
>> ERROR: Failure: SyntaxError (invalid syntax (test_scalarmath.py, line
>> 84))
>> ...
>> ? File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>> test_scalarmath.py", line 84
>> ? ? class TestRepr():
>> ? ? ? ? ? ? ? ? ? ?^
>> SyntaxError: invalid syntax
>>
>> ======================================================================
>> ERROR: Failure: SyntaxError (invalid syntax (test_twodim_base.py, line
>> 280))
>> ...
>> ? File "/sw/lib/python2.4/site-packages/numpy/lib/tests/
>> test_twodim_base.py", line 280
>> ? ? class TestTriuIndices():
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ^
>> SyntaxError: invalid syntax
>>
>> ======================================================================
>> ERROR: Failure: SyntaxError (invalid syntax (test_linalg.py, line 243))
>> ...
>> ? File "/sw/lib/python2.4/site-packages/numpy/linalg/tests/
>> test_linalg.py", line 243
>> ? ? class TestMatrixPower():
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ^
>> SyntaxError: invalid syntax
>>
>
> Subclassing object should fix those, I think.
>
>>
>> ======================================================================
>> ERROR: test_gft_filename (test_io.TestFromTxt)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File "/sw/lib/python2.4/site-packages/numpy/lib/tests/test_io.py",
>> line 1327, in test_gft_filename
>> ? ? assert_array_equal(np.genfromtxt(name), exp_res)
>> ? File "/sw/lib/python2.4/site-packages/numpy/lib/npyio.py", line
>> 1235, in genfromtxt
>> ? ? fhd = iter(np.lib._datasource.open(fname, 'Ub'))
>> ? File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>> line 145, in open
>> ? ? return ds.open(path, mode)
>> ? File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>> line 477, in open
>> ? ? return _file_openers[ext](found, mode=mode)
>> IOError: invalid mode: Ub
>>
>
> Guess that wasn't tested before ;) I thought that was strange when I saw it.
> The source of the problem is line 2035 in npyio.py. Additionally, Since
> genloadtxt needs to have byte strings the 'rb" mode should probably be used.
> That works on linux, both for python 2 and python 3, but doing that might
> uncover genfromtxt problems on other platforms.

"rb" is fine on Windows with python 3.2, (that's what I tested
initially for this bug)

Josef

>
> Chuck
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


From josef.pktd at gmail.com  Tue Apr  5 00:09:29 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 5 Apr 2011 00:09:29 -0400
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
Message-ID: <BANLkTimQw=EopPC9rx_DH2pt3+imWQgNbQ@mail.gmail.com>

On Tue, Apr 5, 2011 at 12:03 AM,  <josef.pktd at gmail.com> wrote:
> On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>>
>> On Mon, Apr 4, 2011 at 8:42 PM, Derek Homeier
>> <derek at astro.physik.uni-goettingen.de> wrote:
>>>
>>> Hi all,
>>>
>>> On 4 Apr 2011, at 22:04, Ralf Gommers wrote:
>>>
>>> > I am pleased to announce the availability of the second beta of NumPy
>>> > 1.6.0. Due to the extensive changes in the Numpy core for this
>>> > release, the beta testing phase will last at least one month. Please
>>> > test this beta and report any problems on the Numpy mailing list.
>>> >
>>> the tests have a number of Python2.4-incompatibilities, one for a file
>>> opening mode and the rest for class declaration styles.
>>>
>>> Cheers,
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Derek
>>>
>>> Running unit tests for numpy
>>> NumPy version 1.6.0b2
>>> NumPy is installed in /sw/lib/python2.4/site-packages/numpy
>>> Python version 2.4.4 (#1, Jan ?5 2011, 03:05:41) [GCC 4.0.1 (Apple
>>> Inc. build 5493)]
>>> nose version 1.0.0
>>> .....
>>> ======================================================================
>>> ERROR: Failure: SyntaxError (invalid syntax (test_multiarray.py, line
>>> 1023))
>>> ...
>>> ? File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>>> test_multiarray.py", line 1023
>>> ? ? class TestPutmask():
>>> ? ? ? ? ? ? ? ? ? ? ? ^
>>> SyntaxError: invalid syntax
>>>
>>> ======================================================================
>>> ERROR: Failure: SyntaxError (invalid syntax (test_numeric.py, line
>>> 1068))
>>> ...
>>> ? File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>>> test_numeric.py", line 1068
>>> ? ? class TestAllclose():
>>> ? ? ? ? ? ? ? ? ? ? ? ?^
>>> SyntaxError: invalid syntax
>>>
>>> ======================================================================
>>> ERROR: Failure: SyntaxError (invalid syntax (test_scalarmath.py, line
>>> 84))
>>> ...
>>> ? File "/sw/lib/python2.4/site-packages/numpy/core/tests/
>>> test_scalarmath.py", line 84
>>> ? ? class TestRepr():
>>> ? ? ? ? ? ? ? ? ? ?^
>>> SyntaxError: invalid syntax
>>>
>>> ======================================================================
>>> ERROR: Failure: SyntaxError (invalid syntax (test_twodim_base.py, line
>>> 280))
>>> ...
>>> ? File "/sw/lib/python2.4/site-packages/numpy/lib/tests/
>>> test_twodim_base.py", line 280
>>> ? ? class TestTriuIndices():
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ^
>>> SyntaxError: invalid syntax
>>>
>>> ======================================================================
>>> ERROR: Failure: SyntaxError (invalid syntax (test_linalg.py, line 243))
>>> ...
>>> ? File "/sw/lib/python2.4/site-packages/numpy/linalg/tests/
>>> test_linalg.py", line 243
>>> ? ? class TestMatrixPower():
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ^
>>> SyntaxError: invalid syntax
>>>
>>
>> Subclassing object should fix those, I think.
>>
>>>
>>> ======================================================================
>>> ERROR: test_gft_filename (test_io.TestFromTxt)
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>> ? File "/sw/lib/python2.4/site-packages/numpy/lib/tests/test_io.py",
>>> line 1327, in test_gft_filename
>>> ? ? assert_array_equal(np.genfromtxt(name), exp_res)
>>> ? File "/sw/lib/python2.4/site-packages/numpy/lib/npyio.py", line
>>> 1235, in genfromtxt
>>> ? ? fhd = iter(np.lib._datasource.open(fname, 'Ub'))
>>> ? File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>>> line 145, in open
>>> ? ? return ds.open(path, mode)
>>> ? File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>>> line 477, in open
>>> ? ? return _file_openers[ext](found, mode=mode)
>>> IOError: invalid mode: Ub
>>>
>>
>> Guess that wasn't tested before ;) I thought that was strange when I saw it.
>> The source of the problem is line 2035 in npyio.py. Additionally, Since
>> genloadtxt needs to have byte strings the 'rb" mode should probably be used.
>> That works on linux, both for python 2 and python 3, but doing that might
>> uncover genfromtxt problems on other platforms.
>
> "rb" is fine on Windows with python 3.2, (that's what I tested
> initially for this bug)

Sorry, I take this back. All our files have \n file endings not \r\n.
So I didn't test the latter case.

Josef

>
> Josef
>
>>
>> Chuck
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>


From cgohlke at uci.edu  Tue Apr  5 00:10:11 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Mon, 04 Apr 2011 21:10:11 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
Message-ID: <4D9A9623.7080601@uci.edu>



On 4/4/2011 1:04 PM, Ralf Gommers wrote:
> Hi,
>
> I am pleased to announce the availability of the second beta of NumPy
> 1.6.0. Due to the extensive changes in the Numpy core for this
> release, the beta testing phase will last at least one month. Please
> test this beta and report any problems on the Numpy mailing list.
>
> Many bug fixes went in since beta 1, among which:
> - fix installation of source release with python 3
> - f2py fixes for assumed shape arrays
> - several loadtxt bug fixes and enhancements
> - change default floating point error handling from "print" to "warn"
> - much more
>
> I quickly counted in the timeline, and in the last few weeks the
> number of open tickets has been decreased by over 100. Thanks to
> everyone who contributed to this spring cleaning!
>
> Sources and binaries can be found at
> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b2/
> For (preliminary) release notes see below.
>
> Enjoy,
> Ralf
>

Thank you.

numpy 1.6.0b2 builds well on Windows with msvc9/MKL.

No license file is included in the installers.

The installers for Python 3 include 2to3 backup files. This has already 
been fixed in scipy 
<https://github.com/scipy/scipy/commit/f7dae4f21593d94735a0377a1af3a9275413b889>.

All numpy tests pass on win32.

A few numpy tests fail on win-amd64:

======================================================================
ERROR: Ticket #99
----------------------------------------------------------------------
Traceback (most recent call last):
   File "X:\Python26-x64\lib\site-packages\numpy\testing\decorators.py", 
line 215, in knownfailer
     return f(*args, **kwargs)
   File 
"X:\Python26-x64\lib\site-packages\numpy\core\tests\test_regression.py", 
line 148, in test_intp
     np.intp('0x' + 'f'*i_width,16)
TypeError: function takes at most 1 argument (2 given)

======================================================================
FAIL: test_iterator.test_iter_broadcasting_errors
----------------------------------------------------------------------
Traceback (most recent call last):
   File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187, in 
runTest
     self.test(*self.arg)
   File 
"X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py", 
line 639, in test_iter_broadcasting_errors
     'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
34, in assert_
     raise AssertionError(msg)
AssertionError: Message "non-broadcastable output operand with shape 
(%lld,%lld)
  doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain 
operand shape (2,3)

======================================================================
FAIL: test_noncentral_f (test_random.TestRandomDist)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"X:\Python26-x64\lib\site-packages\numpy\random\tests\test_random.py", 
line 297, in test_noncentral_f
     np.testing.assert_array_almost_equal(actual, desired, decimal=14)
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
800, in assert_array_almost_equal
     header=('Arrays are not almost equal to %d decimals' % decimal))
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
636, in assert_array_compare
     raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 14 decimals

(mismatch 100.0%)
  x: array([[ 1.62003345,  1.7253997 ],
        [ 0.96735921,  0.42933718],
        [ 0.71714872,  6.24979552]])
  y: array([[ 1.405981  ,  0.34207973],
        [ 3.57715069,  7.92632663],
        [ 0.43741599,  1.17742088]])



One scipy 0.9.0 test fails (32 and 64 bit):

======================================================================
FAIL: test_expon (test_morestats.TestAnderson)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"X:\Python26-x64\lib\site-packages\scipy\stats\tests\test_morestats.py", 
line 72, in test_expon
     assert_array_less(crit[:-1], A)
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
869, in assert_array_less
     header='Arrays are not less-ordered')
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
613, in assert_array_compare
     chk_same_position(x_id, y_id, hasval='inf')
   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line 
588, in chk_same_position
     raise AssertionError(msg)
AssertionError:
Arrays are not less-ordered

x and y inf location mismatch:
  x: array([ 0.911,  1.065,  1.325,  1.587])
  y: array(inf)



Christoph


From josef.pktd at gmail.com  Tue Apr  5 01:03:50 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 5 Apr 2011 01:03:50 -0400
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <4D9A9623.7080601@uci.edu>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<4D9A9623.7080601@uci.edu>
Message-ID: <BANLkTi=gn9hp-qzwvL0YEhp39WQ84EAy5w@mail.gmail.com>

On Tue, Apr 5, 2011 at 12:10 AM, Christoph Gohlke <cgohlke at uci.edu> wrote:
>
>
> On 4/4/2011 1:04 PM, Ralf Gommers wrote:
>> Hi,
>>
>> I am pleased to announce the availability of the second beta of NumPy
>> 1.6.0. Due to the extensive changes in the Numpy core for this
>> release, the beta testing phase will last at least one month. Please
>> test this beta and report any problems on the Numpy mailing list.
>>
>> Many bug fixes went in since beta 1, among which:
>> - fix installation of source release with python 3
>> - f2py fixes for assumed shape arrays
>> - several loadtxt bug fixes and enhancements
>> - change default floating point error handling from "print" to "warn"
>> - much more
>>
>> I quickly counted in the timeline, and in the last few weeks the
>> number of open tickets has been decreased by over 100. Thanks to
>> everyone who contributed to this spring cleaning!
>>
>> Sources and binaries can be found at
>> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b2/
>> For (preliminary) release notes see below.
>>
>> Enjoy,
>> Ralf
>>
>
> Thank you.
>
> numpy 1.6.0b2 builds well on Windows with msvc9/MKL.
>
> No license file is included in the installers.
>
> The installers for Python 3 include 2to3 backup files. This has already
> been fixed in scipy
> <https://github.com/scipy/scipy/commit/f7dae4f21593d94735a0377a1af3a9275413b889>.
>
> All numpy tests pass on win32.
>
> A few numpy tests fail on win-amd64:
>
> ======================================================================
> ERROR: Ticket #99
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File "X:\Python26-x64\lib\site-packages\numpy\testing\decorators.py",
> line 215, in knownfailer
> ? ? return f(*args, **kwargs)
> ? File
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_regression.py",
> line 148, in test_intp
> ? ? np.intp('0x' + 'f'*i_width,16)
> TypeError: function takes at most 1 argument (2 given)
>
> ======================================================================
> FAIL: test_iterator.test_iter_broadcasting_errors
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187, in
> runTest
> ? ? self.test(*self.arg)
> ? File
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> line 639, in test_iter_broadcasting_errors
> ? ? 'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
> ? File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 34, in assert_
> ? ? raise AssertionError(msg)
> AssertionError: Message "non-broadcastable output operand with shape
> (%lld,%lld)
> ?doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain
> operand shape (2,3)
>
> ======================================================================
> FAIL: test_noncentral_f (test_random.TestRandomDist)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "X:\Python26-x64\lib\site-packages\numpy\random\tests\test_random.py",
> line 297, in test_noncentral_f
> ? ? np.testing.assert_array_almost_equal(actual, desired, decimal=14)
> ? File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 800, in assert_array_almost_equal
> ? ? header=('Arrays are not almost equal to %d decimals' % decimal))
> ? File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 636, in assert_array_compare
> ? ? raise AssertionError(msg)
> AssertionError:
> Arrays are not almost equal to 14 decimals
>
> (mismatch 100.0%)
> ?x: array([[ 1.62003345, ?1.7253997 ],
> ? ? ? ?[ 0.96735921, ?0.42933718],
> ? ? ? ?[ 0.71714872, ?6.24979552]])
> ?y: array([[ 1.405981 ?, ?0.34207973],
> ? ? ? ?[ 3.57715069, ?7.92632663],
> ? ? ? ?[ 0.43741599, ?1.17742088]])
>
>
>
> One scipy 0.9.0 test fails (32 and 64 bit):
>
> ======================================================================
> FAIL: test_expon (test_morestats.TestAnderson)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "X:\Python26-x64\lib\site-packages\scipy\stats\tests\test_morestats.py",
> line 72, in test_expon
> ? ? assert_array_less(crit[:-1], A)
> ? File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 869, in assert_array_less
> ? ? header='Arrays are not less-ordered')
> ? File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 613, in assert_array_compare
> ? ? chk_same_position(x_id, y_id, hasval='inf')
> ? File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 588, in chk_same_position
> ? ? raise AssertionError(msg)
> AssertionError:
> Arrays are not less-ordered
>
> x and y inf location mismatch:
> ?x: array([ 0.911, ?1.065, ?1.325, ?1.587])
> ?y: array(inf)

I think this is from the change how inf are now handled in the
asserts, but I cannot check because I don't have numpy 1.6 yet

>>> np.testing.assert_array_less(2,3)
>>> np.testing.assert_array_less(2,np.inf)
>>> np.testing.assert_array_less(np.inf,np.inf)

the anderson test really states:  assert  array([ 0.911,  1.065,
1.325,  1.587]) < np.inf
which is correct but not a good test.

Josef


>
>
>
> Christoph
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From Chris.Barker at noaa.gov  Tue Apr  5 01:22:04 2011
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Mon, 04 Apr 2011 22:22:04 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
Message-ID: <4D9AA6FC.1010904@noaa.gov>

On 4/4/11 9:03 PM, josef.pktd at gmail.com wrote:
> On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris

>>>    File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>>> line 477, in open
>>>      return _file_openers[ext](found, mode=mode)
>>> IOError: invalid mode: Ub
>>>
>>
>> Guess that wasn't tested before ;) I thought that was strange when I saw it.
>> The source of the problem is line 2035 in npyio.py. Additionally, Since
>> genloadtxt needs to have byte strings the 'rb" mode should probably be used.
>> That works on linux, both for python 2 and python 3, but doing that might
>> uncover genfromtxt problems on other platforms.

> "rb" is fine on Windows with python 3.2, (that's what I tested
> initially for this bug)

IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no 
sense when used with "b" for binary. I looked at the code a ways back, 
and I can't remember the resolution order, but there isn't any checking 
for incompatible flags.

I'd expect that genfromtxt, being txt, and line oriented, should use 
'rU'. but if it wants the raw line endings (why would it?) then rb 
should be fine.

Note that if you only test with unix (\n) and dos (\r\n) line endings, 
it may work with 'b', if it's splitting on '\n', but not if you try it 
with Mac endings (\r). Of course with OS-X mac endings are getting 
pretty uncommon.

-Chris




From charlesr.harris at gmail.com  Tue Apr  5 01:35:08 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 4 Apr 2011 23:35:08 -0600
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <4D9AA6FC.1010904@noaa.gov>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
Message-ID: <BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>

On Mon, Apr 4, 2011 at 11:22 PM, Chris Barker <Chris.Barker at noaa.gov> wrote:

> On 4/4/11 9:03 PM, josef.pktd at gmail.com wrote:
> > On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris
>
> >>>    File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
> >>> line 477, in open
> >>>      return _file_openers[ext](found, mode=mode)
> >>> IOError: invalid mode: Ub
> >>>
> >>
> >> Guess that wasn't tested before ;) I thought that was strange when I saw
> it.
> >> The source of the problem is line 2035 in npyio.py. Additionally, Since
> >> genloadtxt needs to have byte strings the 'rb" mode should probably be
> used.
> >> That works on linux, both for python 2 and python 3, but doing that
> might
> >> uncover genfromtxt problems on other platforms.
>
> > "rb" is fine on Windows with python 3.2, (that's what I tested
> > initially for this bug)
>
> IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
> sense when used with "b" for binary. I looked at the code a ways back,
> and I can't remember the resolution order, but there isn't any checking
> for incompatible flags.
>
> I'd expect that genfromtxt, being txt, and line oriented, should use
> 'rU'. but if it wants the raw line endings (why would it?) then rb
> should be fine.
>
>
"U" has been kept around for backwards compatibility, the python
documentation recommends that it not be used for new code. Whatever 'new'
means in this case ;) If it is unneeded for python 2.4 I say drop it.

Note that if you only test with unix (\n) and dos (\r\n) line endings,
> it may work with 'b', if it's splitting on '\n', but not if you try it
> with Mac endings (\r). Of course with OS-X mac endings are getting
> pretty uncommon.
>


I suppose we could try the different line endings on a single platform and
see what happens. It would be nice to know just how portable text really is.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110404/b8cb2335/attachment.html>

From tmp50 at ukr.net  Tue Apr  5 05:33:54 2011
From: tmp50 at ukr.net (Dmitrey)
Date: Tue, 05 Apr 2011 12:33:54 +0300
Subject: [Numpy-discussion] bug with numpy nanargmax/nanargmin
Message-ID: <E1Q72e2-0000JP-09@ffe15.ukr.net>

 >>> from numpy import *
   >>> nanargmax([nan,nan])
   nan # ok
   >>> nanargmax([nan,nan],0)
   nan # ok
   >>> nanargmax([[1,nan],[1,nan]],0)
   Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File
   "/usr/local/lib/python2.6/site-packages/numpy/lib/function_base.py",
   line 1606, in nanargmax
   return _nanop(np.argmax, -np.inf, a, axis)
   File
   "/usr/local/lib/python2.6/site-packages/numpy/lib/function_base.py",
   line 1346, in _nanop
   res[mask_all_along_axis] = np.nan
   ValueError: cannot convert float NaN to integer
   >>> __version__
   '2.0.0.dev-1fe8136'

   D.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/48614f97/attachment.html>

From francois.steinmetz at gmail.com  Tue Apr  5 05:49:13 2011
From: francois.steinmetz at gmail.com (=?UTF-8?Q?Fran=C3=A7ois_Steinmetz?=)
Date: Tue, 5 Apr 2011 11:49:13 +0200
Subject: [Numpy-discussion] Strange behavior of operator *=
Message-ID: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>

Hi all,

I have encountered the following strangeness :
>>> from numpy import *
>>> __version__
'1.5.1'
>>> a = eye(2, dtype='int')
>>> a *= 1.0
>>> a ; a.dtype
array([[1, 0],
       [0, 1]])
dtype('int64')
>>> a = a * 1.0
>>> a ; a.dtype
array([[ 1.,  0.],
       [ 0.,  1.]])
dtype('float64')

So, in this case "a *= x" is not equivalent to "a = a*x" ?

Fran?ois
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/5ca6e506/attachment.html>

From matthieu.brucher at gmail.com  Tue Apr  5 05:55:04 2011
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 5 Apr 2011 11:55:04 +0200
Subject: [Numpy-discussion] Strange behavior of operator *=
In-Reply-To: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>
References: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>
Message-ID: <BANLkTikNONASvTxJkg2QtDWEREXQj=MKXA@mail.gmail.com>

Indeed, it is not. In the first case, you keep your original object and each
(integer) element is multiplied by 1.0. In the second example, you are
creating a temporary object a*x, and as x is a float and a an array of
integers, the result will be an array of floats, which will be assigned to
a.

Matthieu

2011/4/5 Fran?ois Steinmetz <francois.steinmetz at gmail.com>

> Hi all,
>
> I have encountered the following strangeness :
> >>> from numpy import *
> >>> __version__
> '1.5.1'
> >>> a = eye(2, dtype='int')
> >>> a *= 1.0
> >>> a ; a.dtype
> array([[1, 0],
>        [0, 1]])
> dtype('int64')
> >>> a = a * 1.0
> >>> a ; a.dtype
> array([[ 1.,  0.],
>        [ 0.,  1.]])
> dtype('float64')
>
> So, in this case "a *= x" is not equivalent to "a = a*x" ?
>
> Fran?ois
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/96137278/attachment.html>

From francois.steinmetz at gmail.com  Tue Apr  5 06:41:05 2011
From: francois.steinmetz at gmail.com (=?UTF-8?Q?Fran=C3=A7ois_Steinmetz?=)
Date: Tue, 5 Apr 2011 12:41:05 +0200
Subject: [Numpy-discussion] Strange behavior of operator *=
In-Reply-To: <BANLkTikNONASvTxJkg2QtDWEREXQj=MKXA@mail.gmail.com>
References: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>
	<BANLkTikNONASvTxJkg2QtDWEREXQj=MKXA@mail.gmail.com>
Message-ID: <BANLkTikkyrS5ZMGMJ-=ZXk+6UX_3kiSHBg@mail.gmail.com>

Ok, thanks for the clarification !

Fran?ois

On Tue, Apr 5, 2011 at 11:55, Matthieu Brucher
<matthieu.brucher at gmail.com>wrote:

> Indeed, it is not. In the first case, you keep your original object and
> each (integer) element is multiplied by 1.0. In the second example, you are
> creating a temporary object a*x, and as x is a float and a an array of
> integers, the result will be an array of floats, which will be assigned to
> a.
>
> Matthieu
>
> 2011/4/5 Fran?ois Steinmetz <francois.steinmetz at gmail.com>
>
>> Hi all,
>>
>> I have encountered the following strangeness :
>> >>> from numpy import *
>> >>> __version__
>> '1.5.1'
>> >>> a = eye(2, dtype='int')
>> >>> a *= 1.0
>> >>> a ; a.dtype
>> array([[1, 0],
>>        [0, 1]])
>> dtype('int64')
>> >>> a = a * 1.0
>> >>> a ; a.dtype
>> array([[ 1.,  0.],
>>        [ 0.,  1.]])
>> dtype('float64')
>>
>> So, in this case "a *= x" is not equivalent to "a = a*x" ?
>>
>>  Fran?ois
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
>
> --
> Information System Engineer, Ph.D.
> Blog: http://matt.eifelle.com
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/aca06f42/attachment.html>

From alan.isaac at gmail.com  Tue Apr  5 09:02:21 2011
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Tue, 05 Apr 2011 09:02:21 -0400
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <4D9A9623.7080601@uci.edu>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<4D9A9623.7080601@uci.edu>
Message-ID: <4D9B12DD.1010208@gmail.com>

Superpack is OK on Windows Vista 64bit with Python 2.7 32bit.
Alan Isaac

Ran 2993 tests in 25.707s

OK (KNOWNFAIL=9, SKIP=5)
Running unit tests for numpy
NumPy version 1.6.0b2
NumPy is installed in c:\Python27\lib\site-packages\numpy
Python version 2.7 (r27:82525, Jul  4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)]
nose version 0.11.0


From alan.isaac at gmail.com  Tue Apr  5 09:09:35 2011
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Tue, 05 Apr 2011 09:09:35 -0400
Subject: [Numpy-discussion] Strange behavior of operator *=
In-Reply-To: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>
References: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>
Message-ID: <4D9B148F.2060008@gmail.com>

On 4/5/2011 5:49 AM, Fran?ois Steinmetz wrote:
>  >>> a = eye(2, dtype='int')
>  >>> a *= 1.0
>  >>> a ; a.dtype
> array([[1, 0],
>         [0, 1]])
> dtype('int64')


This in-place (!) multiplication should not change
the dtype of a.  I suspect you did not exactly cut
and paste...

Alan Isaac



From bsouthey at gmail.com  Tue Apr  5 09:07:37 2011
From: bsouthey at gmail.com (Bruce Southey)
Date: Tue, 05 Apr 2011 08:07:37 -0500
Subject: [Numpy-discussion] How to create structured char arrays?
In-Reply-To: <F805E695-D1BB-4397-B93A-AFD51044446F@astro.physik.uni-goettingen.de>
References: <4D9A1576.90709@gmail.com>
	<F805E695-D1BB-4397-B93A-AFD51044446F@astro.physik.uni-goettingen.de>
Message-ID: <4D9B1419.5090904@gmail.com>

On 04/04/2011 02:30 PM, Derek Homeier wrote:
> Hi Bruce,
>
>> I think that I have resolved my issue down to creating a structured
>> string array.
>> I am using numpy version '2.0.0.dev-3c338cb'.
>>
>> Without a structured array, it should be a 2 by 2 array:
>>>>> np.array([('a','b'),('c','d')])
>> array([['a', 'b'],
>>         ['c', 'd']],
>>        dtype='|S1')
>>>>> np.array([('a','b'),('c','d')], dtype='a2')
>> array([['a', 'b'],
>>         ['c', 'd']],
>>        dtype='|S2')
>>
>>
>> It does not work as a structured array:
>>>>> np.array([('a','b'),('c','d')], dtype=[('label','a2'), ('Value',
>> 'a4')])
>> array([('a', 'b'), ('c', 'd')],
>>        dtype=[('label', '|S2'), ('Value', '|S4')])
>>
>> What is the correct dtype argument to use in this case?
>>
> what do you mean by "does not work" - this should give you a 1 by 2
> array
> with 2 fields, i.e.
>
>   >>>  c['label']
> array(['a', 'c'],
>         dtype='|S2')
>   >>>  c['Value']
> array(['b', 'd'],
>         dtype='|S4')
>
> and this should be the same as returned by loadtxt for
>
>   >>>  dt=np.dtype([('label', 'S2'), ('Value', 'S4')])
>   >>>  d = np.loadtxt(StringIO("a\tb\nc\td"), delimiter="\t", dtype=dt)
>
> right? Maybe I should modify my test in #1071 to compare to the full
> array, like
>
> a = np.array([(1,2,3, asbytes('start ')),
>            (4,5,6, asbytes('  ')), (7,8,9.5, asbytes(''))])
> assert_array_equal(x, a)
>
>
> HTH,
> 					Derek
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
Thanks,
Yes I was not thinking carefully enough.

I think it is good to compare the full array because it ensures all 
elements should be the same in value and dtype.

Bruce


From friedrichromstedt at gmail.com  Tue Apr  5 09:36:48 2011
From: friedrichromstedt at gmail.com (Friedrich Romstedt)
Date: Tue, 5 Apr 2011 15:36:48 +0200
Subject: [Numpy-discussion] Strange behavior of operator *=
In-Reply-To: <4D9B148F.2060008@gmail.com>
References: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>
	<4D9B148F.2060008@gmail.com>
Message-ID: <BANLkTi=-+Nn9wQMYXNWTp8hC8t5qyXZw6w@mail.gmail.com>

2011/4/5 Alan G Isaac <alan.isaac at gmail.com>:
> On 4/5/2011 5:49 AM, Fran?ois Steinmetz wrote:
>> ?>>> a = eye(2, dtype='int')
>> ?>>> a *= 1.0
>> ?>>> a ; a.dtype
>> array([[1, 0],
>> ? ? ? ? [0, 1]])
>> dtype('int64')
>
> This in-place (!) multiplication should not change
> the dtype of a. ?I suspect you did not exactly cut
> and paste...

Guess he's on a 64 bit system.

$ # 32 bit
$ python2.6
Python 2.6.5 (r265:79063, Jul 18 2010, 12:14:53)
[GCC 4.2.1 (Apple Inc. build 5659)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> a = numpy.ones(1, dtype='int')
>>> a.dtype
dtype('int32')
>>> ^D

$ python2.6-64
Python 2.6.5 (r265:79063, Jul 18 2010, 12:14:53)
[GCC 4.2.1 (Apple Inc. build 5659)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> a = numpy.ones(1, dtype='int')
>>> a.dtype
dtype('int64')

This is with 1.4.1 (sorry for the missing update).

Same behaviour with numpy-1.5.1 on py2.7.

Friedrich


From alan.isaac at gmail.com  Tue Apr  5 09:39:01 2011
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Tue, 05 Apr 2011 09:39:01 -0400
Subject: [Numpy-discussion] Strange behavior of operator *=
In-Reply-To: <BANLkTin5CevKJq1e63YZaYs0=0vcCvaH8A@mail.gmail.com>
References: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>
	<4D9B148F.2060008@gmail.com>
	<BANLkTin5CevKJq1e63YZaYs0=0vcCvaH8A@mail.gmail.com>
Message-ID: <4D9B1B75.6000204@gmail.com>

On 4/5/2011 9:26 AM, Fran?ois Steinmetz wrote:
> It does not change the dtype, 'int' is just interpreted as 'int64' :


So the meaning of 'int' is system specific?

     >>> import numpy as np; a=np.eye(2,dtype='int'); a.dtype
     dtype('int32')

Alan Isaac



From paul.anton.letnes at gmail.com  Tue Apr  5 09:48:08 2011
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Tue, 5 Apr 2011 15:48:08 +0200
Subject: [Numpy-discussion] Strange behavior of operator *=
In-Reply-To: <4D9B1B75.6000204@gmail.com>
References: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>
	<4D9B148F.2060008@gmail.com>
	<BANLkTin5CevKJq1e63YZaYs0=0vcCvaH8A@mail.gmail.com>
	<4D9B1B75.6000204@gmail.com>
Message-ID: <9E3AB9FA-03A8-46EA-985B-222F0672572B@gmail.com>


On 5. apr. 2011, at 15.39, Alan G Isaac wrote:

> On 4/5/2011 9:26 AM, Fran?ois Steinmetz wrote:
>> It does not change the dtype, 'int' is just interpreted as 'int64' :
> 
> 
> So the meaning of 'int' is system specific?
> 
>>>> import numpy as np; a=np.eye(2,dtype='int'); a.dtype
>     dtype('int32')
> 
> Alan Isaac

'int' is system specific, 'int32' and 'int64' is not. At least, as far as I can tell.

Paul.

From infinity.probability at gmail.com  Tue Apr  5 10:11:20 2011
From: infinity.probability at gmail.com (M.)
Date: Tue, 5 Apr 2011 11:11:20 -0300
Subject: [Numpy-discussion] Problem with np.power in numpy 1.5.1-2
Message-ID: <BANLkTimWgU44AOEFLNA01q=uCXdKjTvihA@mail.gmail.com>

Hi all,

I've been using numpy without problems - version .1.4.1-5 - until this
morning, when I upgraded my Debian sid system to numpy 1.5.1-2. A
function that was working that uses np.power

def get_far_field(self, chi, n):
        return np.sum(self.an[n,1:self.j] * np.power(chi,-np.arange(1,self.j)))

outputs "Warning: overflow encountered in power". I'm using python
2.6.6-12. Is this a bug in the numpy library or (more probably) my
code? If it's in my code, what can be happening? How to debug it?

Thanks


From charlesr.harris at gmail.com  Tue Apr  5 10:12:00 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 5 Apr 2011 08:12:00 -0600
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <4D9AA6FC.1010904@noaa.gov>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
Message-ID: <BANLkTindJr8LAQjxrZYKzJ-Z1=JwCJVfHA@mail.gmail.com>

On Mon, Apr 4, 2011 at 11:22 PM, Chris Barker <Chris.Barker at noaa.gov> wrote:

> On 4/4/11 9:03 PM, josef.pktd at gmail.com wrote:
> > On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris
>
> >>>    File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
> >>> line 477, in open
> >>>      return _file_openers[ext](found, mode=mode)
> >>> IOError: invalid mode: Ub
> >>>
> >>
> >> Guess that wasn't tested before ;) I thought that was strange when I saw
> it.
> >> The source of the problem is line 2035 in npyio.py. Additionally, Since
> >> genloadtxt needs to have byte strings the 'rb" mode should probably be
> used.
> >> That works on linux, both for python 2 and python 3, but doing that
> might
> >> uncover genfromtxt problems on other platforms.
>
> > "rb" is fine on Windows with python 3.2, (that's what I tested
> > initially for this bug)
>
> IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
> sense when used with "b" for binary. I looked at the code a ways back,
> and I can't remember the resolution order, but there isn't any checking
> for incompatible flags.
>
> I'd expect that genfromtxt, being txt, and line oriented, should use
> 'rU'. but if it wants the raw line endings (why would it?) then rb
> should be fine.
>
> Note that if you only test with unix (\n) and dos (\r\n) line endings,
> it may work with 'b', if it's splitting on '\n', but not if you try it
> with Mac endings (\r). Of course with OS-X mac endings are getting
> pretty uncommon.
>
>
The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
python, as it works just fine on python 2.7. It may indeed be desirable to
read the files as text, but that would require more work on both loadtxt and
genfromtxt.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/c7ddc99c/attachment.html>

From njs at pobox.com  Tue Apr  5 10:37:56 2011
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 5 Apr 2011 07:37:56 -0700
Subject: [Numpy-discussion] Strange behavior of operator *=
In-Reply-To: <4D9B1B75.6000204@gmail.com>
References: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>
	<4D9B148F.2060008@gmail.com>
	<BANLkTin5CevKJq1e63YZaYs0=0vcCvaH8A@mail.gmail.com>
	<4D9B1B75.6000204@gmail.com>
Message-ID: <BANLkTinp0c1SRfpGFAU2MJnYMh=mpC8PMA@mail.gmail.com>

On Tue, Apr 5, 2011 at 6:39 AM, Alan G Isaac <alan.isaac at gmail.com> wrote:
> On 4/5/2011 9:26 AM, Fran?ois Steinmetz wrote:
>> It does not change the dtype, 'int' is just interpreted as 'int64' :
>
> So the meaning of 'int' is system specific?
>
> ? ? >>> import numpy as np; a=np.eye(2,dtype='int'); a.dtype
> ? ? dtype('int32')

Yes, this is a fact about Python 'int', not a fact about numpy -- a
Python 'int' is defined to hold a C 'long', which will be either 32 or
64 bits depending on platform.

# 32-bit Linux:
$ python -c 'import sys; print sys.maxint'
2147483647

# 64-bit Linux:
$ python -c 'import sys; print sys.maxint'
9223372036854775807

-- Nathaniel


From alan.isaac at gmail.com  Tue Apr  5 10:44:45 2011
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Tue, 05 Apr 2011 10:44:45 -0400
Subject: [Numpy-discussion] Strange behavior of operator *=
In-Reply-To: <BANLkTinp0c1SRfpGFAU2MJnYMh=mpC8PMA@mail.gmail.com>
References: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>	<4D9B148F.2060008@gmail.com>	<BANLkTin5CevKJq1e63YZaYs0=0vcCvaH8A@mail.gmail.com>	<4D9B1B75.6000204@gmail.com>
	<BANLkTinp0c1SRfpGFAU2MJnYMh=mpC8PMA@mail.gmail.com>
Message-ID: <4D9B2ADD.1000208@gmail.com>

On 4/5/2011 10:37 AM, Nathaniel Smith wrote:
> Yes, this is a fact about Python 'int', not a fact about numpy -- a
> Python 'int' is defined to hold a C 'long', which will be either 32 or
> 64 bits depending on platform.


So what is the rule for Python 3, where iiuc it can no longer be "a fact about Python"?

Thanks,
Alan Isaac



From charlesr.harris at gmail.com  Tue Apr  5 10:57:48 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 5 Apr 2011 08:57:48 -0600
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTindJr8LAQjxrZYKzJ-Z1=JwCJVfHA@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTindJr8LAQjxrZYKzJ-Z1=JwCJVfHA@mail.gmail.com>
Message-ID: <BANLkTimsY+djo4LFWk9gTgZCZq-Vs10Kbg@mail.gmail.com>

On Tue, Apr 5, 2011 at 8:12 AM, Charles R Harris
<charlesr.harris at gmail.com>wrote:

>
>
> On Mon, Apr 4, 2011 at 11:22 PM, Chris Barker <Chris.Barker at noaa.gov>wrote:
>
>> On 4/4/11 9:03 PM, josef.pktd at gmail.com wrote:
>> > On Mon, Apr 4, 2011 at 11:42 PM, Charles R Harris
>>
>> >>>    File "/sw/lib/python2.4/site-packages/numpy/lib/_datasource.py",
>> >>> line 477, in open
>> >>>      return _file_openers[ext](found, mode=mode)
>> >>> IOError: invalid mode: Ub
>> >>>
>> >>
>> >> Guess that wasn't tested before ;) I thought that was strange when I
>> saw it.
>> >> The source of the problem is line 2035 in npyio.py. Additionally, Since
>> >> genloadtxt needs to have byte strings the 'rb" mode should probably be
>> used.
>> >> That works on linux, both for python 2 and python 3, but doing that
>> might
>> >> uncover genfromtxt problems on other platforms.
>>
>> > "rb" is fine on Windows with python 3.2, (that's what I tested
>> > initially for this bug)
>>
>> IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
>> sense when used with "b" for binary. I looked at the code a ways back,
>> and I can't remember the resolution order, but there isn't any checking
>> for incompatible flags.
>>
>> I'd expect that genfromtxt, being txt, and line oriented, should use
>> 'rU'. but if it wants the raw line endings (why would it?) then rb
>> should be fine.
>>
>> Note that if you only test with unix (\n) and dos (\r\n) line endings,
>> it may work with 'b', if it's splitting on '\n', but not if you try it
>> with Mac endings (\r). Of course with OS-X mac endings are getting
>> pretty uncommon.
>>
>>
> The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
> python, as it works just fine on python 2.7. It may indeed be desirable to
> read the files as text, but that would require more work on both loadtxt and
> genfromtxt.
>
>
Curiously, 'rbU' and 'rU' do work on 2.4, but not 'Urb',  'Ub', 'Ur', or
'bU'. Due to Python 3 not hadling '\r' in binary mode, the shortest path
forward might be to open files as text and call asbytes on the lines as they
are read.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/0b112aee/attachment.html>

From charlesr.harris at gmail.com  Tue Apr  5 12:01:42 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 5 Apr 2011 10:01:42 -0600
Subject: [Numpy-discussion] Strange behavior of operator *=
In-Reply-To: <4D9B2ADD.1000208@gmail.com>
References: <BANLkTi=DgPvFQukWQp-D0wu71FVRjsS6EQ@mail.gmail.com>
	<4D9B148F.2060008@gmail.com>
	<BANLkTin5CevKJq1e63YZaYs0=0vcCvaH8A@mail.gmail.com>
	<4D9B1B75.6000204@gmail.com>
	<BANLkTinp0c1SRfpGFAU2MJnYMh=mpC8PMA@mail.gmail.com>
	<4D9B2ADD.1000208@gmail.com>
Message-ID: <BANLkTikwiXKqY_bJyHRRj7xNjduSSnu88g@mail.gmail.com>

On Tue, Apr 5, 2011 at 8:44 AM, Alan G Isaac <alan.isaac at gmail.com> wrote:

> On 4/5/2011 10:37 AM, Nathaniel Smith wrote:
> > Yes, this is a fact about Python 'int', not a fact about numpy -- a
> > Python 'int' is defined to hold a C 'long', which will be either 32 or
> > 64 bits depending on platform.
>
>
> So what is the rule for Python 3, where iiuc it can no longer be "a fact
> about Python"?
>
>
At the c-api level I believe we still use the routines that convert python
ints to longs. If we drop support for python < 2.6 we could change to using
Py_ssize_t instead since there are python functions for that and we might
get larger ints on 64 bit windows since windows longs are always 32 bits.
That might also be compiler dependent however...

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/ac80d121/attachment.html>

From gnurser at gmail.com  Tue Apr  5 12:31:44 2011
From: gnurser at gmail.com (George Nurser)
Date: Tue, 5 Apr 2011 17:31:44 +0100
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
Message-ID: <BANLkTikMkbnhSNOLiBLZ-8hOJNr6G5bXzw@mail.gmail.com>

SUSE 11.3, python 2.7, gcc 4.3.4, gfortran from gcc 4.6.0, I get two failures on
commit 1439a8ddcb2eda20fa102aa44e846783f29c0af3 (head of 1.6.x
maintenance branch).

--George.


======================================================================
FAIL: Test basic arithmetic function errors
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/testing/decorators.py",
line 215, in knownfailer
    return f(*args, **kwargs)
  File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
line 321, in test_floating_exceptions
    lambda a,b:a/b, ft_tiny, ft_max)
  File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py",
line 271, in assert_raises_fpe
    "Type %s did not raise fpe error '%s'." % (ftype, fpeerr))
  File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/testing/utils.py",
line 34, in assert_
    raise AssertionError(msg)
AssertionError: Type <type 'numpy.float32'> did not raise fpe error 'underflow'.

======================================================================
FAIL: test_kind.TestKind.test_all
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/nose/case.py",
line 187, in runTest
    self.test(*self.arg)
  File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/f2py/tests/test_kind.py",
line 30, in test_all
    'selectedrealkind(%s): expected %r but got %r' %  (i,
selected_real_kind(i), selectedrealkind(i)))
  File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/testing/utils.py",
line 34, in assert_
    raise AssertionError(msg)
AssertionError: selectedrealkind(19): expected -1 but got 16

----------------------------------------------------------------------
Ran 3544 tests in 48.039s

FAILED (KNOWNFAIL=4, failures=2)
<nose.result.TextTestResult run=3544 errors=0 failures=2>


On 4 April 2011 21:04, Ralf Gommers <ralf.gommers at googlemail.com> wrote:
> Hi,
>
> I am pleased to announce the availability of the second beta of NumPy
> 1.6.0. Due to the extensive changes in the Numpy core for this
> release, the beta testing phase will last at least one month. Please
> test this beta and report any problems on the Numpy mailing list.
>
> Many bug fixes went in since beta 1, among which:
> - fix installation of source release with python 3
> - f2py fixes for assumed shape arrays
> - several loadtxt bug fixes and enhancements
> - change default floating point error handling from "print" to "warn"
> - much more
>
> I quickly counted in the timeline, and in the last few weeks the
> number of open tickets has been decreased by over 100. Thanks to
> everyone who contributed to this spring cleaning!
>
> Sources and binaries can be found at
> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b2/
> For (preliminary) release notes see below.
>
> Enjoy,
> Ralf
>
>
>
> Note: NumPy 1.6.0 is not yet released.
>
>
> =========================
> NumPy 1.6.0 Release Notes
> =========================
>
> This release includes several new features as well as numerous bug fixes and
> improved documentation. ?It is backward compatible with the 1.5.0 release, and
> supports Python 2.4 - 2.7 and 3.1 - 3.2.
>
>
> Highlights
> ==========
>
> * Re-introduction of datetime dtype support to deal with dates in arrays.
>
> * A new 16-bit floating point type.
>
> * A new iterator, which improves performance of many functions.
>
>
> New features
> ============
>
> New 16-bit floating point type
> ------------------------------
>
> This release adds support for the IEEE 754-2008 binary16 format, available as
> the data type ``numpy.half``. ?Within Python, the type behaves similarly to
> `float` or `double`, and C extensions can add support for it with the exposed
> half-float API.
>
>
> New iterator
> ------------
>
> A new iterator has been added, replacing the functionality of the
> existing iterator and multi-iterator with a single object and API.
> This iterator works well with general memory layouts different from
> C or Fortran contiguous, and handles both standard NumPy and
> customized broadcasting. The buffering, automatic data type
> conversion, and optional output parameters, offered by
> ufuncs but difficult to replicate elsewhere, are now exposed by this
> iterator.
>
>
> Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial``
> -------------------------------------------------------------------------
>
> Extend the number of polynomials available in the polynomial package. In
> addition, a new ``window`` attribute has been added to the classes in
> order to specify the range the ``domain`` maps to. This is mostly useful
> for the Laguerre, Hermite, and HermiteE polynomials whose natural domains
> are infinite and provides a more intuitive way to get the correct mapping
> of values without playing unnatural tricks with the domain.
>
>
> Fortran assumed shape array and size function support in ``numpy.f2py``
> -----------------------------------------------------------------------
>
> F2py now supports wrapping Fortran 90 routines that use assumed shape
> arrays. ?Before such routines could be called from Python but the
> corresponding Fortran routines received assumed shape arrays as zero
> length arrays which caused unpredicted results. Thanks to Lorenz
> H?depohl for pointing out the correct way to interface routines with
> assumed shape arrays.
>
> In addition, f2py interprets Fortran expression ``size(array, dim)``
> as ``shape(array, dim-1)`` which makes it possible to automatically
> wrap Fortran routines that use two argument ``size`` function in
> dimension specifications. Before users were forced to apply this
> mapping manually.
>
>
> Other new functions
> -------------------
>
> ``numpy.ravel_multi_index`` : Converts a multi-index tuple into
> an array of flat indices, applying boundary modes to the indices.
>
> ``numpy.einsum`` : Evaluate the Einstein summation convention. ?Using the
> Einstein summation convention, many common multi-dimensional array operations
> can be represented in a simple fashion. ?This function provides a way compute
> such summations.
>
> ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array.
>
> ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose
> the underlying type promotion used by the ufuncs and other operations to
> determine the types of outputs. These improve upon the ``numpy.common_type``
> and ``numpy.mintypecode`` which provide similar functionality but do
> not match the ufunc implementation.
>
>
> Changes
> =======
>
> Changes and improvements in the numpy core
> ------------------------------------------
>
> ``default error handling``
> --------------------------
>
> The default error handling has been change from ``print`` to ``warn`` for
> all except for ``underflow``, which remains as ``ignore``.
>
>
> ``numpy.distutils``
> -------------------
>
> Several new compilers are supported for building Numpy: the Portland Group
> Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C
> compiler on Linux.
>
>
> ``numpy.testing``
> -----------------
>
> The testing framework gained ``numpy.testing.assert_allclose``, which provides
> a more convenient way to compare floating point arrays than
> `assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`.
>
>
> ``C API``
> ---------
>
> In addition to the APIs for the new iterator and half data type, a number
> of other additions have been made to the C API. The type promotion
> mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``,
> ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration
> ``NPY_CASTING`` has been added which controls what types of casts are
> permitted. This is used by the new functions ``PyArray_CanCastArrayTo``
> and ``PyArray_CanCastTypeTo``. ?A more flexible way to handle
> conversion of arbitrary python objects into arrays is exposed by
> ``PyArray_GetArrayParamsFromObject``.
>
>
> Deprecated features
> ===================
>
> The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality
> will be replaced by the new "density" keyword.
>
>
> Removed features
> ================
>
> ``numpy.fft``
> -------------
>
> The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`,
> which were aliases for the same functions without the 'e' in the name, were
> removed.
>
>
> ``numpy.memmap``
> ----------------
>
> The `sync()` and `close()` methods of memmap were removed. ?Use `flush()` and
> "del memmap" instead.
>
>
> ``numpy.lib``
> -------------
>
> The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``,
> ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed.
>
>
> ``numpy.ma``
> ------------
>
> Several deprecated items were removed from the ``numpy.ma`` module::
>
> ?* ``numpy.ma.MaskedArray`` "raw_data" method
> ?* ``numpy.ma.MaskedArray`` constructor "flag" keyword
> ?* ``numpy.ma.make_mask`` "flag" keyword
> ?* ``numpy.ma.allclose`` "fill_value" keyword
>
>
> ``numpy.distutils``
> -------------------
>
> The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include``
> instead.
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From Chris.Barker at noaa.gov  Tue Apr  5 12:46:32 2011
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 05 Apr 2011 09:46:32 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
Message-ID: <4D9B4768.1060105@noaa.gov>

On 4/4/11 10:35 PM, Charles R Harris wrote:
>     IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
>     sense when used with "b" for binary. I looked at the code a ways back,
>     and I can't remember the resolution order, but there isn't any checking
>     for incompatible flags.
>
>     I'd expect that genfromtxt, being txt, and line oriented, should use
>     'rU'. but if it wants the raw line endings (why would it?) then rb
>     should be fine.
>
>
> "U" has been kept around for backwards compatibility, the python
> documentation recommends that it not be used for new code.

That is for  3.*  -- the 2.7.* docs say:

"""
In addition to the standard fopen() values mode may be 'U' or 'rU'. 
Python is usually built with universal newline support; supplying 'U' 
opens the file as a text file, but lines may be terminated by any of the 
following: the Unix end-of-line convention '\n', the Macintosh 
convention '\r', or the Windows convention '\r\n'. All of these external 
representations are seen as '\n' by the Python program. If Python is 
built without universal newline support a mode with 'U' is the same as 
normal text mode. Note that file objects so opened also have an 
attribute called newlines which has a value of None (if no newlines have 
yet been seen), '\n', '\r', '\r\n', or a tuple containing all the 
newline types seen.

Python enforces that the mode, after stripping 'U', begins with 'r', 'w' 
or 'a'.
""

which does, in fact indicate that 'Ub' is NOT allowed. We should be 
using 'Ur', I think. Maybe the "python enforces" is what we saw the 
error from -- it didn't used to enforce anything.


On 4/5/11 7:12 AM, Charles R Harris wrote:

> The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
> python, as it works just fine on python 2.7.

"Ub" never made any sense anywhere -- "U" means universal newline text 
file. "b" means binary -- combining them makes no sense. On older 
pythons, the behaviour of 'Ub' was undefined -- now, it looks like it is 
supposed to raise an error.

does 'Ur' work with \r line endings on Python 3?

According to my read of the docs, 'U' does nothing -- "universal" 
newline support is supposed to be the default:

"""
On input, if newline is None, universal newlines mode is enabled. Lines 
in the input can end in '\n', '\r', or '\r\n', and these are translated 
into '\n' before being returned to the caller.
"""

> It may indeed be desirable
> to read the files as text, but that would require more work on both
> loadtxt and genfromtxt.

Why can't we just open the file with mode 'Ur'? text is text, messing 
with line endings shouldn't hurt anything, and it might help.

If we stick with binary, then it comes down to:
- will having an extra \r with Windows files hurt anything? -- probably not.
- Are there many mac-style text files out there anymore? not many.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Chris.Barker at noaa.gov  Tue Apr  5 13:02:05 2011
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 05 Apr 2011 10:02:05 -0700
Subject: [Numpy-discussion] convert array to tuples?
Message-ID: <4D9B4B0D.2050506@noaa.gov>

HI folks,

What's the fastest way to convert a numpy array to tuples?

Unfortunately, not all packages take arbitrary sequences when they are 
expecting a list of tuples. In this case, I'm using PIL's ImageDraw, and 
want to do:

draw.line(points)

(points is an Nx2 numpy array of ints)

but if I pass in an array, I get, oddly enough, nothing. NO error, just 
nothing drawn.

if I do:

draw.line(points.tolist())

SystemError: new style getargs format but argument is not a tuple

So I seem to need tuples. This works:

draw.line([tuple(p) for p in points.tolist()])

but that seems like it would be slower than it should be (to be honest, 
not profiled).

Is there a faster way?

Maybe numpy should have a ndarray.totuple() method.

-Chris






-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From charlesr.harris at gmail.com  Tue Apr  5 13:20:45 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 5 Apr 2011 11:20:45 -0600
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <4D9B4768.1060105@noaa.gov>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
	<4D9B4768.1060105@noaa.gov>
Message-ID: <BANLkTik3yXumfwuvXA8vM57=VO-BwBVuGg@mail.gmail.com>

On Tue, Apr 5, 2011 at 10:46 AM, Christopher Barker
<Chris.Barker at noaa.gov>wrote:

> On 4/4/11 10:35 PM, Charles R Harris wrote:
> >     IIUC, "Ub" is undefined -- "U" means universal newlines, which makes
> no
> >     sense when used with "b" for binary. I looked at the code a ways
> back,
> >     and I can't remember the resolution order, but there isn't any
> checking
> >     for incompatible flags.
> >
> >     I'd expect that genfromtxt, being txt, and line oriented, should use
> >     'rU'. but if it wants the raw line endings (why would it?) then rb
> >     should be fine.
> >
> >
> > "U" has been kept around for backwards compatibility, the python
> > documentation recommends that it not be used for new code.
>
> That is for  3.*  -- the 2.7.* docs say:
>
> """
> In addition to the standard fopen() values mode may be 'U' or 'rU'.
> Python is usually built with universal newline support; supplying 'U'
> opens the file as a text file, but lines may be terminated by any of the
> following: the Unix end-of-line convention '\n', the Macintosh
> convention '\r', or the Windows convention '\r\n'. All of these external
> representations are seen as '\n' by the Python program. If Python is
> built without universal newline support a mode with 'U' is the same as
> normal text mode. Note that file objects so opened also have an
> attribute called newlines which has a value of None (if no newlines have
> yet been seen), '\n', '\r', '\r\n', or a tuple containing all the
> newline types seen.
>
> Python enforces that the mode, after stripping 'U', begins with 'r', 'w'
> or 'a'.
> ""
>
> which does, in fact indicate that 'Ub' is NOT allowed. We should be
> using 'Ur', I think. Maybe the "python enforces" is what we saw the
> error from -- it didn't used to enforce anything.
>
>
'rbU' works and I put that in as a quick fix.

>
> On 4/5/11 7:12 AM, Charles R Harris wrote:
>
> > The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
> > python, as it works just fine on python 2.7.
>
> "Ub" never made any sense anywhere -- "U" means universal newline text
> file. "b" means binary -- combining them makes no sense. On older
> pythons, the behaviour of 'Ub' was undefined -- now, it looks like it is
> supposed to raise an error.
>
> does 'Ur' work with \r line endings on Python 3?
>

Yes.


> According to my read of the docs, 'U' does nothing -- "universal"
> newline support is supposed to be the default:
>
> """
> On input, if newline is None, universal newlines mode is enabled. Lines
> in the input can end in '\n', '\r', or '\r\n', and these are translated
> into '\n' before being returned to the caller.
> """
>
> > It may indeed be desirable
> > to read the files as text, but that would require more work on both
> > loadtxt and genfromtxt.
>
> Why can't we just open the file with mode 'Ur'? text is text, messing
> with line endings shouldn't hurt anything, and it might help.
>
>
Well, text in the files then gets the numpy 'U' type instead of 'S', and
there are places where byte streams are assumed for stripping and such.
Which is to say that changing to text mode requires some work. Another
possibility is to use a generator:

def usetext(fname):
    f = open(fname, 'rt')
    for l in f:
       yield asbytes(f.next())

I think genfromtxt could use a refactoring and cleanup, but probably not for
1.6.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/7b0fde69/attachment.html>

From josef.pktd at gmail.com  Tue Apr  5 13:45:34 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 5 Apr 2011 13:45:34 -0400
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTik3yXumfwuvXA8vM57=VO-BwBVuGg@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
	<4D9B4768.1060105@noaa.gov>
	<BANLkTik3yXumfwuvXA8vM57=VO-BwBVuGg@mail.gmail.com>
Message-ID: <BANLkTimxVgZ7mcyttdjB35s3d7DJwHhafw@mail.gmail.com>

On Tue, Apr 5, 2011 at 1:20 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Tue, Apr 5, 2011 at 10:46 AM, Christopher Barker <Chris.Barker at noaa.gov>
> wrote:
>>
>> On 4/4/11 10:35 PM, Charles R Harris wrote:
>> > ? ? IIUC, "Ub" is undefined -- "U" means universal newlines, which makes
>> > no
>> > ? ? sense when used with "b" for binary. I looked at the code a ways
>> > back,
>> > ? ? and I can't remember the resolution order, but there isn't any
>> > checking
>> > ? ? for incompatible flags.
>> >
>> > ? ? I'd expect that genfromtxt, being txt, and line oriented, should use
>> > ? ? 'rU'. but if it wants the raw line endings (why would it?) then rb
>> > ? ? should be fine.
>> >
>> >
>> > "U" has been kept around for backwards compatibility, the python
>> > documentation recommends that it not be used for new code.
>>
>> That is for ?3.* ?-- the 2.7.* docs say:
>>
>> """
>> In addition to the standard fopen() values mode may be 'U' or 'rU'.
>> Python is usually built with universal newline support; supplying 'U'
>> opens the file as a text file, but lines may be terminated by any of the
>> following: the Unix end-of-line convention '\n', the Macintosh
>> convention '\r', or the Windows convention '\r\n'. All of these external
>> representations are seen as '\n' by the Python program. If Python is
>> built without universal newline support a mode with 'U' is the same as
>> normal text mode. Note that file objects so opened also have an
>> attribute called newlines which has a value of None (if no newlines have
>> yet been seen), '\n', '\r', '\r\n', or a tuple containing all the
>> newline types seen.
>>
>> Python enforces that the mode, after stripping 'U', begins with 'r', 'w'
>> or 'a'.
>> ""
>>
>> which does, in fact indicate that 'Ub' is NOT allowed. We should be
>> using 'Ur', I think. Maybe the "python enforces" is what we saw the
>> error from -- it didn't used to enforce anything.
>>
>
> 'rbU' works and I put that in as a quick fix.
>>
>> On 4/5/11 7:12 AM, Charles R Harris wrote:
>>
>> > The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
>> > python, as it works just fine on python 2.7.
>>
>> "Ub" never made any sense anywhere -- "U" means universal newline text
>> file. "b" means binary -- combining them makes no sense. On older
>> pythons, the behaviour of 'Ub' was undefined -- now, it looks like it is
>> supposed to raise an error.
>>
>> does 'Ur' work with \r line endings on Python 3?
>
> Yes.
>
>>
>> According to my read of the docs, 'U' does nothing -- "universal"
>> newline support is supposed to be the default:
>>
>> """
>> On input, if newline is None, universal newlines mode is enabled. Lines
>> in the input can end in '\n', '\r', or '\r\n', and these are translated
>> into '\n' before being returned to the caller.
>> """
>>
>> > It may indeed be desirable
>> > to read the files as text, but that would require more work on both
>> > loadtxt and genfromtxt.
>>
>> Why can't we just open the file with mode 'Ur'? text is text, messing
>> with line endings shouldn't hurt anything, and it might help.
>>
>
> Well, text in the files then gets the numpy 'U' type instead of 'S', and
> there are places where byte streams are assumed for stripping and such.
> Which is to say that changing to text mode requires some work. Another
> possibility is to use a generator:
>
> def usetext(fname):
> ??? f = open(fname, 'rt')
> ??? for l in f:
> ? ? ?? yield asbytes(f.next())
>
> I think genfromtxt could use a refactoring and cleanup, but probably not for
> 1.6.

I think it should also be possible to read "rb" and strip any \r, \r\n
in _iotools.py,
that's were the bytes are used, from my reading and the initial error message.

Josef

>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


From seb.haase at gmail.com  Tue Apr  5 13:52:05 2011
From: seb.haase at gmail.com (Sebastian Haase)
Date: Tue, 5 Apr 2011 19:52:05 +0200
Subject: [Numpy-discussion] convert array to tuples?
In-Reply-To: <4D9B4B0D.2050506@noaa.gov>
References: <4D9B4B0D.2050506@noaa.gov>
Message-ID: <BANLkTimri04bWRyN0T5BHQEz0UKYhCvUcQ@mail.gmail.com>

On Tue, Apr 5, 2011 at 7:02 PM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
> HI folks,
>
> What's the fastest way to convert a numpy array to tuples?
>
> Unfortunately, not all packages take arbitrary sequences when they are
> expecting a list of tuples. In this case, I'm using PIL's ImageDraw, and
> want to do:
>
> draw.line(points)
>
> (points is an Nx2 numpy array of ints)
>
> but if I pass in an array, I get, oddly enough, nothing. NO error, just
> nothing drawn.
>
> if I do:
>
> draw.line(points.tolist())
>
> SystemError: new style getargs format but argument is not a tuple
>
> So I seem to need tuples. This works:
>
> draw.line([tuple(p) for p in points.tolist()])
>
> but that seems like it would be slower than it should be (to be honest,
> not profiled).
>
> Is there a faster way?
>
> Maybe numpy should have a ndarray.totuple() method.
>
> -Chris
>
How about fixing PIL... I know that there is not a good track record
of getting patches into PIL ,
but did you get to the bottom of it and find how draw.line is implemented?
BTW, is it drawing anti-aliased lines ?

- Sebastian Haase


From josef.pktd at gmail.com  Tue Apr  5 13:53:58 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 5 Apr 2011 13:53:58 -0400
Subject: [Numpy-discussion] convert array to tuples?
In-Reply-To: <4D9B4B0D.2050506@noaa.gov>
References: <4D9B4B0D.2050506@noaa.gov>
Message-ID: <BANLkTikyvfT3xXm+EWOJASNEkf1m3D25GA@mail.gmail.com>

On Tue, Apr 5, 2011 at 1:02 PM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
> HI folks,
>
> What's the fastest way to convert a numpy array to tuples?
>
> Unfortunately, not all packages take arbitrary sequences when they are
> expecting a list of tuples. In this case, I'm using PIL's ImageDraw, and
> want to do:
>
> draw.line(points)
>
> (points is an Nx2 numpy array of ints)
>
> but if I pass in an array, I get, oddly enough, nothing. NO error, just
> nothing drawn.
>
> if I do:
>
> draw.line(points.tolist())
>
> SystemError: new style getargs format but argument is not a tuple
>
> So I seem to need tuples. This works:
>
> draw.line([tuple(p) for p in points.tolist()])
>
> but that seems like it would be slower than it should be (to be honest,
> not profiled).
>
> Is there a faster way?
>
> Maybe numpy should have a ndarray.totuple() method.

does a structured dtype work?

Josef

>
> -Chris
>
>
>
>
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice
> 7600 Sand Point Way NE ? (206) 526-6329 ? fax
> Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From charlesr.harris at gmail.com  Tue Apr  5 13:56:17 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 5 Apr 2011 11:56:17 -0600
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTimxVgZ7mcyttdjB35s3d7DJwHhafw@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
	<4D9B4768.1060105@noaa.gov>
	<BANLkTik3yXumfwuvXA8vM57=VO-BwBVuGg@mail.gmail.com>
	<BANLkTimxVgZ7mcyttdjB35s3d7DJwHhafw@mail.gmail.com>
Message-ID: <BANLkTi=ac_fAmw2BM_GPnFOJNu2jPoX88g@mail.gmail.com>

On Tue, Apr 5, 2011 at 11:45 AM, <josef.pktd at gmail.com> wrote:

> On Tue, Apr 5, 2011 at 1:20 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> >
> > On Tue, Apr 5, 2011 at 10:46 AM, Christopher Barker <
> Chris.Barker at noaa.gov>
> > wrote:
> >>
> >> On 4/4/11 10:35 PM, Charles R Harris wrote:
> >> >     IIUC, "Ub" is undefined -- "U" means universal newlines, which
> makes
> >> > no
> >> >     sense when used with "b" for binary. I looked at the code a ways
> >> > back,
> >> >     and I can't remember the resolution order, but there isn't any
> >> > checking
> >> >     for incompatible flags.
> >> >
> >> >     I'd expect that genfromtxt, being txt, and line oriented, should
> use
> >> >     'rU'. but if it wants the raw line endings (why would it?) then rb
> >> >     should be fine.
> >> >
> >> >
> >> > "U" has been kept around for backwards compatibility, the python
> >> > documentation recommends that it not be used for new code.
> >>
> >> That is for  3.*  -- the 2.7.* docs say:
> >>
> >> """
> >> In addition to the standard fopen() values mode may be 'U' or 'rU'.
> >> Python is usually built with universal newline support; supplying 'U'
> >> opens the file as a text file, but lines may be terminated by any of the
> >> following: the Unix end-of-line convention '\n', the Macintosh
> >> convention '\r', or the Windows convention '\r\n'. All of these external
> >> representations are seen as '\n' by the Python program. If Python is
> >> built without universal newline support a mode with 'U' is the same as
> >> normal text mode. Note that file objects so opened also have an
> >> attribute called newlines which has a value of None (if no newlines have
> >> yet been seen), '\n', '\r', '\r\n', or a tuple containing all the
> >> newline types seen.
> >>
> >> Python enforces that the mode, after stripping 'U', begins with 'r', 'w'
> >> or 'a'.
> >> ""
> >>
> >> which does, in fact indicate that 'Ub' is NOT allowed. We should be
> >> using 'Ur', I think. Maybe the "python enforces" is what we saw the
> >> error from -- it didn't used to enforce anything.
> >>
> >
> > 'rbU' works and I put that in as a quick fix.
> >>
> >> On 4/5/11 7:12 AM, Charles R Harris wrote:
> >>
> >> > The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
> >> > python, as it works just fine on python 2.7.
> >>
> >> "Ub" never made any sense anywhere -- "U" means universal newline text
> >> file. "b" means binary -- combining them makes no sense. On older
> >> pythons, the behaviour of 'Ub' was undefined -- now, it looks like it is
> >> supposed to raise an error.
> >>
> >> does 'Ur' work with \r line endings on Python 3?
> >
> > Yes.
> >
> >>
> >> According to my read of the docs, 'U' does nothing -- "universal"
> >> newline support is supposed to be the default:
> >>
> >> """
> >> On input, if newline is None, universal newlines mode is enabled. Lines
> >> in the input can end in '\n', '\r', or '\r\n', and these are translated
> >> into '\n' before being returned to the caller.
> >> """
> >>
> >> > It may indeed be desirable
> >> > to read the files as text, but that would require more work on both
> >> > loadtxt and genfromtxt.
> >>
> >> Why can't we just open the file with mode 'Ur'? text is text, messing
> >> with line endings shouldn't hurt anything, and it might help.
> >>
> >
> > Well, text in the files then gets the numpy 'U' type instead of 'S', and
> > there are places where byte streams are assumed for stripping and such.
> > Which is to say that changing to text mode requires some work. Another
> > possibility is to use a generator:
> >
> > def usetext(fname):
> >     f = open(fname, 'rt')
> >     for l in f:
> >        yield asbytes(f.next())
> >
> > I think genfromtxt could use a refactoring and cleanup, but probably not
> for
> > 1.6.
>
> I think it should also be possible to read "rb" and strip any \r, \r\n
> in _iotools.py,
> that's were the bytes are used, from my reading and the initial error
> message.
>
>
Doesn't work for \r, you get the whole file at once instead of line by line.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/53d3f47b/attachment.html>

From mniemczyk at mac.com  Tue Apr  5 14:49:53 2011
From: mniemczyk at mac.com (Mark S Niemczyk)
Date: Tue, 05 Apr 2011 13:49:53 -0500
Subject: [Numpy-discussion] convert array to tuples?
In-Reply-To: <4D9B4B0D.2050506@noaa.gov>
References: <4D9B4B0D.2050506@noaa.gov>
Message-ID: <AF7BE1F6-DF96-4B61-A88B-73E1E4A19A1D@mac.com>

Just a suggestion (I am very new to Numpy), but wouldn't

	draw.line(tuple(points.tolist()))

accomplish what you want.

Based on my reading, the ndarray.tolist method would return a Python list of elements; then Python's built-in tuple function would complete the conversion.


Regards,

	Mark


On Apr 5, 2011, at 12:02 PM, Christopher Barker wrote:

> HI folks,
> 
> What's the fastest way to convert a numpy array to tuples?
> 
> Unfortunately, not all packages take arbitrary sequences when they are 
> expecting a list of tuples. In this case, I'm using PIL's ImageDraw, and 
> want to do:
> 
> draw.line(points)
> 
> (points is an Nx2 numpy array of ints)
> 
> but if I pass in an array, I get, oddly enough, nothing. NO error, just 
> nothing drawn.
> 
> if I do:
> 
> draw.line(points.tolist())
> 
> SystemError: new style getargs format but argument is not a tuple
> 
> So I seem to need tuples. This works:
> 
> draw.line([tuple(p) for p in points.tolist()])
> 
> but that seems like it would be slower than it should be (to be honest, 
> not profiled).
> 
> Is there a faster way?
> 
> Maybe numpy should have a ndarray.totuple() method.
> 
> -Chris
> 
> 
> 
> 
> 
> 
> -- 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> 
> Chris.Barker at noaa.gov
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From charlesr.harris at gmail.com  Tue Apr  5 14:59:39 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 5 Apr 2011 12:59:39 -0600
Subject: [Numpy-discussion] Problem with np.power in numpy 1.5.1-2
In-Reply-To: <BANLkTimWgU44AOEFLNA01q=uCXdKjTvihA@mail.gmail.com>
References: <BANLkTimWgU44AOEFLNA01q=uCXdKjTvihA@mail.gmail.com>
Message-ID: <BANLkTinsPvUrAb1aztMcLfmGAkMONatnoQ@mail.gmail.com>

On Tue, Apr 5, 2011 at 8:11 AM, M. <infinity.probability at gmail.com> wrote:

> Hi all,
>
> I've been using numpy without problems - version .1.4.1-5 - until this
> morning, when I upgraded my Debian sid system to numpy 1.5.1-2. A
> function that was working that uses np.power
>
> def get_far_field(self, chi, n):
>        return np.sum(self.an[n,1:self.j] *
> np.power(chi,-np.arange(1,self.j)))
>
> outputs "Warning: overflow encountered in power". I'm using python
> 2.6.6-12. Is this a bug in the numpy library or (more probably) my
> code? If it's in my code, what can be happening? How to debug it?
>
>
Hard to say. Some warnings that were disabled by accident in earlier
versions of numpy were enabled in numpy 1.5, so it is possible the problem
has always been there. You might retry with the earlier version and use
seterr to enable the overflow warning just to check. A simple example with
actual numbers that reproduces the warning might also be helpful.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/cab6b01c/attachment.html>

From bsouthey at gmail.com  Tue Apr  5 15:06:08 2011
From: bsouthey at gmail.com (Bruce Southey)
Date: Tue, 05 Apr 2011 14:06:08 -0500
Subject: [Numpy-discussion] loadtxt/savetxt tickets
In-Reply-To: <BANLkTinHe8jpNcrTP1MhKCf_yO5knhCKoA@mail.gmail.com>
References: <AANLkTi=gJtN2E2+5YqzYSSH_2XnjOdX8AKudv0An43M7@mail.gmail.com>	<2F4F21FC-6F70-45B3-8417-B256F3A167AB@gmail.com>	<5D10525C-839D-4AFC-8F48-72A2C01D7217@astro.physik.uni-goettingen.de>	<F5B15BD5-88D9-4AEF-BD02-2516338B037B@gmail.com>	<AANLkTimFaYbWqUMFkc0GqE80O86ehFCyqvxMed=iy5SC@mail.gmail.com>	<AANLkTinezEqay7zg1n3FiLRjVfNNmUMsTVfRqp6-W61=@mail.gmail.com>	<CB9C54E6-4124-49EB-965A-6C9BAA2C7D1E@astro.physik.uni-goettingen.de>	<4D99EACB.6080000@gmail.com>	<BANLkTimjT1kAjUSoWuFs84Pq6F9biTCZDQ@mail.gmail.com>	<4D99F95F.7090200@gmail.com>
	<BANLkTinHe8jpNcrTP1MhKCf_yO5knhCKoA@mail.gmail.com>
Message-ID: <4D9B6820.20200@gmail.com>

On 04/04/2011 12:38 PM, Charles R Harris wrote:
>
>
> On Mon, Apr 4, 2011 at 11:01 AM, Bruce Southey <bsouthey at gmail.com 
> <mailto:bsouthey at gmail.com>> wrote:
>
>     On 04/04/2011 11:20 AM, Charles R Harris wrote:
>>
>>
>>     On Mon, Apr 4, 2011 at 9:59 AM, Bruce Southey <bsouthey at gmail.com
>>     <mailto:bsouthey at gmail.com>> wrote:
>>
>>         On 03/31/2011 12:02 PM, Derek Homeier wrote:
>>         > On 31 Mar 2011, at 17:03, Bruce Southey wrote:
>>         >
>>         >> This is an invalid ticket because the docstring clearly
>>         states that in
>>         >> 3 different, yet critical places, that missing values are
>>         not handled
>>         >> here:
>>         >>
>>         >> "Each row in the text file must have the same number of
>>         values."
>>         >> "genfromtxt : Load data with missing values handled as
>>         specified."
>>         >> "   This function aims to be a fast reader for simply
>>         formatted
>>         >> files.  The
>>         >>     `genfromtxt` function provides more sophisticated
>>         handling of,
>>         >> e.g.,
>>         >>     lines with missing values."
>>         >>
>>         >> Really I am trying to separate the usage of loadtxt and
>>         genfromtxt to
>>         >> avoid unnecessary duplication and confusion. Part of this is
>>         >> historical because loadtxt was added in 2007 and
>>         genfromtxt was added
>>         >> in 2009. So really certain features of loadtxt have been
>>          'kept' for
>>         >> backwards compatibility purposes yet these features can be
>>         'abused' to
>>         >> handle missing data. But I really consider that any
>>         missing values
>>         >> should cause loadtxt to fail.
>>         >>
>>         > OK, I was not aware of the design issues of loadtxt vs.
>>         genfromtxt -
>>         > you could probably say also for historical reasons since I
>>         have not
>>         > used genfromtxt much so far.
>>         > Anyway the docstring statement "Converters can also be used to
>>         >           provide a default value for missing data:"
>>         > then appears quite misleading, or an invitation to abuse,
>>         if you will.
>>         > This should better be removed from the documentation then,
>>         or users
>>         > explicitly discouraged from using converters instead of
>>         genfromtxt
>>         > (I don't see how you could completely prevent using
>>         converters in
>>         > this way).
>>         >
>>         >> The patch is incorrect because it should not include a
>>         space in the
>>         >> split() as indicated in the comment by the original
>>         reporter. Of
>>         > The split('\r\n') alone caused test_dtype_with_object(self)
>>         to fail,
>>         > probably
>>         > because it relies on stripping the blanks. But maybe the
>>         test is ill-
>>         > formed?
>>         >
>>         >> course a corrected patch alone still is not sufficient to
>>         address the
>>         >> problem without the user providing the correct converter.
>>         Also you
>>         >> start to run into problems with multiple delimiters (such
>>         as one space
>>         >> versus two spaces) so you start down the path to add all
>>         the features
>>         >> that duplicate genfromtxt.
>>         > Given that genfromtxt provides that functionality more
>>         conveniently,
>>         > I agree again users should be encouraged to use this instead of
>>         > converters.
>>         > But the actual tab-problem causes in fact an issue not
>>         related to
>>         > missing
>>         > values at all (well, depending on what you call a missing
>>         value).
>>         > I am describing an example on the ticket.
>>         >
>>         > Cheers,
>>         >                                       Derek
>>         >
>>         > _______________________________________________
>>         > NumPy-Discussion mailing list
>>         > NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>>         > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>         Okay I see that 1071 got closed which I am fine with.
>>
>>         I think that your following example should be a test because
>>         the two
>>         spaces should not be removed with a tab delimiter:
>>         np.loadtxt(StringIO("aa\tbb\n \t \ncc\t"), delimiter='\t',
>>         dtype=np.dtype([('label', 'S4'), ('comment', 'S4')]))
>>
>>
>>     Make a test and we'll put it in.
>>
>>     Chuck
>>
>>
>     I know!
>     Trying to write one made me realize that loadtxt is not handling
>     string arrays correctly. So I have to check more on this as I
>     think loadtxt is giving a 1-d array instead of a 2-d array.
>
>
> Tests often have that side effect.
>
> <snip>
>
> Chuck
>
>
Okay,
My confusion aside (sorry for that), I added ticket 1784 with a possible 
test that should work with ticket 1071:
http://projects.scipy.org/numpy/ticket/1794

Bruce


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/9cfc9630/attachment.html>

From Chris.Barker at noaa.gov  Tue Apr  5 15:35:24 2011
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 05 Apr 2011 12:35:24 -0700
Subject: [Numpy-discussion] convert array to tuples?
In-Reply-To: <BANLkTimri04bWRyN0T5BHQEz0UKYhCvUcQ@mail.gmail.com>
References: <4D9B4B0D.2050506@noaa.gov>
	<BANLkTimri04bWRyN0T5BHQEz0UKYhCvUcQ@mail.gmail.com>
Message-ID: <4D9B6EFC.8030505@noaa.gov>

On 4/5/11 10:52 AM, Sebastian Haase wrote:
> How about fixing PIL... I know that there is not a good track record
> of getting patches into PIL ,
> but did you get to the bottom of it and find how draw.line is implemented?

no. I haven't. And yes adapting PIL to use the buffer interface for this 
kind of thing would be great. I also have wanted to do that for 
wxPython, where it would be welcome, but still haven't gotten around to it.

> BTW, is it drawing anti-aliased lines ?

Not with the ImageDraw module, but there is a new AggDraw module that 
should be more pretty.

On 4/5/11 11:49 AM, Mark S Niemczyk wrote:
> Just a suggestion (I am very new to Numpy), but wouldn't
>
> 	draw.line(tuple(points.tolist()))
>
> accomplish what you want.

nope, tolist() creates a list of lists -- tuple() only converts the 
outer list to a tuple, which I why I did the list comprehension.


On 4/5/11 10:53 AM, josef.pktd at gmail.com wrote:
> does a structured dtype work?

Good idea -- I'll give it a try -- indeed it does:

dt = np.dtype([('x', np.int32),('y',np.int32)])

draw.line(points.view(dtype=dt).reshape((-1,)).tolist(), 
fill="rgb(0,255,0)", width=4)

but it's not any faster, alas.

The fastest I've come up with is putting the elements in a single 1-d 
list. PIL can take either a sequence of tuples:

[(x,y),(x,y),...]

or the flattened version:

[x,y,x,y,x,y,x,y,...]

So this is the fastest:

draw.line(points.flatten().tolist(), fill="rgb(0,255,0)", width=4)


Thanks all,

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From uvemas at gmail.com  Tue Apr  5 17:17:47 2011
From: uvemas at gmail.com (Vicent Mas)
Date: Tue, 5 Apr 2011 23:17:47 +0200
Subject: [Numpy-discussion] Installation problem: numpy1.5.1 + python2.7 +
	virtualenv-1, 5, 1
Message-ID: <201104052318.16262.uvemas@gmail.com>

Hi,

I'm trying to install numpy1.5.1 in a virtual environment (virtualenv-1.5.1) 
on my debian testing box. I'm using python 2.7 installed from a debian 
package.

This is what I get:

(venv2.7)vmas at rachael$ python setup.py install
Running from numpy source directory.F2PY Version 1
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in /usr/lib
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib/atlas-base']
    language = c
    include_dirs = ['/usr/include/atlas']

  FOUND:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib/atlas-base']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]
    include_dirs = ['/usr/include/atlas']

lapack_opt_info:
lapack_mkl_info:
mkl_info:
  libraries mkl,vml,guide not found in /usr/lib
  NOT AVAILABLE

  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack not found in /usr/lib/atlas-base
numpy.distutils.system_info.atlas_threads_info
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
    language = f77
    include_dirs = ['/usr/include/atlas']

  FOUND:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
    language = f77
    define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]
    include_dirs = ['/usr/include/atlas']

running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler 
options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler 
options
running build_src
build_src
building py_modules sources
creating build
creating build/src.linux-i686-2.7
creating build/src.linux-i686-2.7/numpy
creating build/src.linux-i686-2.7/numpy/distutils
building library "npymath" sources
Traceback (most recent call last):
  File "setup.py", line 211, in <module>
    setup_package()
  File "setup.py", line 204, in setup_package
    configuration=configuration )
  File 
"/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/core.py", 
line 186, in setup
    return old_setup(**new_attr)
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File 
"/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/install.py", 
line 55, in run
    r = old_install.run(self)
  File "/usr/lib/python2.7/distutils/command/install.py", line 601, in run
    self.run_command('build')
  File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File 
"/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build.py", 
line 37, in run
    old_build.run(self)
  File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
    self.run_command(cmd_name)
  File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File 
"/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py", 
line 152, in run
    self.build_sources()
  File 
"/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py", 
line 163, in build_sources
    self.build_library_sources(*libname_info)
  File 
"/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py", 
line 298, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File 
"/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py", 
line 385, in generate_sources
    source = func(extension, build_dir)
  File "numpy/core/setup.py", line 681, in get_mathlib_info
    st = config_cmd.try_link('int main(void) { return 0;}')
  File "/usr/lib/python2.7/distutils/command/config.py", line 248, in try_link
    self._check_compiler()
  File 
"/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/config.py", 
line 45, in _check_compiler
    old_config._check_compiler(self)
  File "/usr/lib/python2.7/distutils/command/config.py", line 103, in 
_check_compiler
    customize_compiler(self.compiler)
  File "/usr/lib/python2.7/distutils/ccompiler.py", line 44, in 
customize_compiler
    cpp = cc + " -E"           # not always
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

Doing the same with python2.6 (also from debian package) works just fine. Could 
somebody tell me what am I doing wrong? Thanks.

Vicent
::

	Share what you know, learn what you don't

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/397c6718/attachment.sig>

From ralf.gommers at googlemail.com  Tue Apr  5 17:43:36 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Tue, 5 Apr 2011 23:43:36 +0200
Subject: [Numpy-discussion] Remove deprecated skiprows
In-Reply-To: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>
References: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>
Message-ID: <BANLkTik=Gz7oMnAFDvrGH8qtHQKeJ9RYdw@mail.gmail.com>

On Mon, Apr 4, 2011 at 2:20 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> Should skiprows be removed?
>
> ??? if skiprows:
> ??????? warnings.warn(\
> ??????????? "The use of `skiprows` is deprecated, it will be removed in
> numpy 2.0.\n" \
> ??????????? "Please use `skip_header` instead.",
> ??????????? DeprecationWarning)
> ??????? skip_header = skiprows
>
> Its been deprecated since 1.4. Personally, I like skiprows better than
> skip_header ;)

It's a bit late now for 1.6, could you remove it only in trunk?

Ralf


From jsseabold at gmail.com  Tue Apr  5 17:45:46 2011
From: jsseabold at gmail.com (Skipper Seabold)
Date: Tue, 5 Apr 2011 17:45:46 -0400
Subject: [Numpy-discussion] Remove deprecated skiprows
In-Reply-To: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>
References: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>
Message-ID: <BANLkTin_PpZbMrs8mPr_z2fW0DB93qDOqg@mail.gmail.com>

On Sun, Apr 3, 2011 at 8:20 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> Should skiprows be removed?
>
> ??? if skiprows:
> ??????? warnings.warn(\
> ??????????? "The use of `skiprows` is deprecated, it will be removed in
> numpy 2.0.\n" \
> ??????????? "Please use `skip_header` instead.",
> ??????????? DeprecationWarning)
> ??????? skip_header = skiprows
>
> Its been deprecated since 1.4. Personally, I like skiprows better than
> skip_header ;)
>

+1 for skiprows. I always have to look it up.

Skipper


From matthew.brett at gmail.com  Tue Apr  5 17:46:18 2011
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 5 Apr 2011 14:46:18 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTi=ac_fAmw2BM_GPnFOJNu2jPoX88g@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
	<4D9B4768.1060105@noaa.gov>
	<BANLkTik3yXumfwuvXA8vM57=VO-BwBVuGg@mail.gmail.com>
	<BANLkTimxVgZ7mcyttdjB35s3d7DJwHhafw@mail.gmail.com>
	<BANLkTi=ac_fAmw2BM_GPnFOJNu2jPoX88g@mail.gmail.com>
Message-ID: <BANLkTimQ-OW5wKwHR=FLARvBCbxESjK5tg@mail.gmail.com>

Hi,

On Tue, Apr 5, 2011 at 10:56 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Tue, Apr 5, 2011 at 11:45 AM, <josef.pktd at gmail.com> wrote:
>>
>> On Tue, Apr 5, 2011 at 1:20 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> >
>> > On Tue, Apr 5, 2011 at 10:46 AM, Christopher Barker
>> > <Chris.Barker at noaa.gov>
>> > wrote:
>> >>
>> >> On 4/4/11 10:35 PM, Charles R Harris wrote:
>> >> > ? ? IIUC, "Ub" is undefined -- "U" means universal newlines, which
>> >> > makes
>> >> > no
>> >> > ? ? sense when used with "b" for binary. I looked at the code a ways
>> >> > back,
>> >> > ? ? and I can't remember the resolution order, but there isn't any
>> >> > checking
>> >> > ? ? for incompatible flags.
>> >> >
>> >> > ? ? I'd expect that genfromtxt, being txt, and line oriented, should
>> >> > use
>> >> > ? ? 'rU'. but if it wants the raw line endings (why would it?) then
>> >> > rb
>> >> > ? ? should be fine.
>> >> >
>> >> >
>> >> > "U" has been kept around for backwards compatibility, the python
>> >> > documentation recommends that it not be used for new code.
>> >>
>> >> That is for ?3.* ?-- the 2.7.* docs say:
>> >>
>> >> """
>> >> In addition to the standard fopen() values mode may be 'U' or 'rU'.
>> >> Python is usually built with universal newline support; supplying 'U'
>> >> opens the file as a text file, but lines may be terminated by any of
>> >> the
>> >> following: the Unix end-of-line convention '\n', the Macintosh
>> >> convention '\r', or the Windows convention '\r\n'. All of these
>> >> external
>> >> representations are seen as '\n' by the Python program. If Python is
>> >> built without universal newline support a mode with 'U' is the same as
>> >> normal text mode. Note that file objects so opened also have an
>> >> attribute called newlines which has a value of None (if no newlines
>> >> have
>> >> yet been seen), '\n', '\r', '\r\n', or a tuple containing all the
>> >> newline types seen.
>> >>
>> >> Python enforces that the mode, after stripping 'U', begins with 'r',
>> >> 'w'
>> >> or 'a'.
>> >> ""
>> >>
>> >> which does, in fact indicate that 'Ub' is NOT allowed. We should be
>> >> using 'Ur', I think. Maybe the "python enforces" is what we saw the
>> >> error from -- it didn't used to enforce anything.
>> >>
>> >
>> > 'rbU' works and I put that in as a quick fix.
>> >>
>> >> On 4/5/11 7:12 AM, Charles R Harris wrote:
>> >>
>> >> > The 'Ub' mode doesn't work for '\r' on python 3. This may be a bug in
>> >> > python, as it works just fine on python 2.7.
>> >>
>> >> "Ub" never made any sense anywhere -- "U" means universal newline text
>> >> file. "b" means binary -- combining them makes no sense. On older
>> >> pythons, the behaviour of 'Ub' was undefined -- now, it looks like it
>> >> is
>> >> supposed to raise an error.
>> >>
>> >> does 'Ur' work with \r line endings on Python 3?
>> >
>> > Yes.
>> >
>> >>
>> >> According to my read of the docs, 'U' does nothing -- "universal"
>> >> newline support is supposed to be the default:
>> >>
>> >> """
>> >> On input, if newline is None, universal newlines mode is enabled. Lines
>> >> in the input can end in '\n', '\r', or '\r\n', and these are translated
>> >> into '\n' before being returned to the caller.
>> >> """
>> >>
>> >> > It may indeed be desirable
>> >> > to read the files as text, but that would require more work on both
>> >> > loadtxt and genfromtxt.
>> >>
>> >> Why can't we just open the file with mode 'Ur'? text is text, messing
>> >> with line endings shouldn't hurt anything, and it might help.
>> >>
>> >
>> > Well, text in the files then gets the numpy 'U' type instead of 'S', and
>> > there are places where byte streams are assumed for stripping and such.
>> > Which is to say that changing to text mode requires some work. Another
>> > possibility is to use a generator:
>> >
>> > def usetext(fname):
>> > ??? f = open(fname, 'rt')
>> > ??? for l in f:
>> > ? ? ?? yield asbytes(f.next())
>> >
>> > I think genfromtxt could use a refactoring and cleanup, but probably not
>> > for
>> > 1.6.
>>
>> I think it should also be possible to read "rb" and strip any \r, \r\n
>> in _iotools.py,
>> that's were the bytes are used, from my reading and the initial error
>> message.
>>
>
> Doesn't work for \r, you get the whole file at once instead of line by line.

Thanks for trying to sort out this ugliness.  I've added another pull request:

https://github.com/numpy/numpy/pull/71

 - tests for \n \r\n and \r files, raising skiptest for currently
failing 3.2 \r mode.

Matthew


From ralf.gommers at googlemail.com  Tue Apr  5 17:52:33 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Tue, 5 Apr 2011 23:52:33 +0200
Subject: [Numpy-discussion] Remove deprecated skiprows
In-Reply-To: <BANLkTin_PpZbMrs8mPr_z2fW0DB93qDOqg@mail.gmail.com>
References: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>
	<BANLkTin_PpZbMrs8mPr_z2fW0DB93qDOqg@mail.gmail.com>
Message-ID: <BANLkTikxVpi2RrPrpw0jfYZPcf76xU-0Cw@mail.gmail.com>

On Tue, Apr 5, 2011 at 11:45 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Sun, Apr 3, 2011 at 8:20 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>> Should skiprows be removed?
>>
>> ??? if skiprows:
>> ??????? warnings.warn(\
>> ??????????? "The use of `skiprows` is deprecated, it will be removed in
>> numpy 2.0.\n" \
>> ??????????? "Please use `skip_header` instead.",
>> ??????????? DeprecationWarning)
>> ??????? skip_header = skiprows
>>
>> Its been deprecated since 1.4. Personally, I like skiprows better than
>> skip_header ;)
>>
>
> +1 for skiprows. I always have to look it up.

To me one is not much better than the other, but -1 for skiprows
because un-deprecating it and deprecating skip_header is inconsistent
and annoying for users.

Ralf


From matthew.brett at gmail.com  Tue Apr  5 17:56:59 2011
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 5 Apr 2011 14:56:59 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <4D9B4768.1060105@noaa.gov>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
	<4D9B4768.1060105@noaa.gov>
Message-ID: <BANLkTimtAD9_wDY6fc0Rq2_kunKKp-G0GQ@mail.gmail.com>

Hi,

On Tue, Apr 5, 2011 at 9:46 AM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
> On 4/4/11 10:35 PM, Charles R Harris wrote:
>> ? ? IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
>> ? ? sense when used with "b" for binary. I looked at the code a ways back,
>> ? ? and I can't remember the resolution order, but there isn't any checking
>> ? ? for incompatible flags.

I disagree that U makes no sense for binary file reading.

In python 3:

'b' means, "return byte objects"
't' means "return decoded strings"

'U' means two things:

1) When iterating by line, split lines at any of '\r', '\r\n', '\n'
2) When returning lines split this way, convert '\r' and '\r\n' to '\n'

If you support returning lines from a binary file (which python 3
does), then I think 'U' is a sensible thing to allow - as in this
case.

Best,

Matthew


From pgmdevlist at gmail.com  Tue Apr  5 18:05:54 2011
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 6 Apr 2011 00:05:54 +0200
Subject: [Numpy-discussion] Remove deprecated skiprows
In-Reply-To: <BANLkTikxVpi2RrPrpw0jfYZPcf76xU-0Cw@mail.gmail.com>
References: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>
	<BANLkTin_PpZbMrs8mPr_z2fW0DB93qDOqg@mail.gmail.com>
	<BANLkTikxVpi2RrPrpw0jfYZPcf76xU-0Cw@mail.gmail.com>
Message-ID: <643550D8-8FED-4316-A9D0-CC90981F92AD@gmail.com>


On Apr 5, 2011, at 11:52 PM, Ralf Gommers wrote:

> On Tue, Apr 5, 2011 at 11:45 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>> On Sun, Apr 3, 2011 at 8:20 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>> Should skiprows be removed?
>>> 
>>>     if skiprows:
>>>         warnings.warn(\
>>>             "The use of `skiprows` is deprecated, it will be removed in
>>> numpy 2.0.\n" \
>>>             "Please use `skip_header` instead.",
>>>             DeprecationWarning)
>>>         skip_header = skiprows
>>> 
>>> Its been deprecated since 1.4. Personally, I like skiprows better than
>>> skip_header ;)
>>> 
>> 
>> +1 for skiprows. I always have to look it up.
> 
> To me one is not much better than the other, but -1 for skiprows
> because un-deprecating it and deprecating skip_header is inconsistent
> and annoying for users.

-1 for skiprows. When  I introduced skip_footer, it looked more consistent to have a skip_header and skip_footer than a skiprows and a skip_footer.
Now, we could probably find more meaningful names, like, skip_header_rows... OK, just joking.

From jsseabold at gmail.com  Tue Apr  5 18:11:12 2011
From: jsseabold at gmail.com (Skipper Seabold)
Date: Tue, 5 Apr 2011 18:11:12 -0400
Subject: [Numpy-discussion] Remove deprecated skiprows
In-Reply-To: <643550D8-8FED-4316-A9D0-CC90981F92AD@gmail.com>
References: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>
	<BANLkTin_PpZbMrs8mPr_z2fW0DB93qDOqg@mail.gmail.com>
	<BANLkTikxVpi2RrPrpw0jfYZPcf76xU-0Cw@mail.gmail.com>
	<643550D8-8FED-4316-A9D0-CC90981F92AD@gmail.com>
Message-ID: <BANLkTim4+1+nHYugYHkTDvMcpMqff2F1Sg@mail.gmail.com>

On Tue, Apr 5, 2011 at 6:05 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
> On Apr 5, 2011, at 11:52 PM, Ralf Gommers wrote:
>
>> On Tue, Apr 5, 2011 at 11:45 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>>> On Sun, Apr 3, 2011 at 8:20 PM, Charles R Harris
>>> <charlesr.harris at gmail.com> wrote:
>>>> Should skiprows be removed?
>>>>
>>>> ? ? if skiprows:
>>>> ? ? ? ? warnings.warn(\
>>>> ? ? ? ? ? ? "The use of `skiprows` is deprecated, it will be removed in
>>>> numpy 2.0.\n" \
>>>> ? ? ? ? ? ? "Please use `skip_header` instead.",
>>>> ? ? ? ? ? ? DeprecationWarning)
>>>> ? ? ? ? skip_header = skiprows
>>>>
>>>> Its been deprecated since 1.4. Personally, I like skiprows better than
>>>> skip_header ;)
>>>>
>>>
>>> +1 for skiprows. I always have to look it up.
>>
>> To me one is not much better than the other, but -1 for skiprows
>> because un-deprecating it and deprecating skip_header is inconsistent
>> and annoying for users.
>
> -1 for skiprows. When ?I introduced skip_footer, it looked more consistent to have a skip_header and skip_footer than a skiprows and a skip_footer.
> Now, we could probably find more meaningful names, like, skip_header_rows... OK, just joking.
>

I'm only slightly serious here, but inconsistent keywords are also
annoying for users. e.g. vs. numpy.loadtxt, though I know that the use
cases are slightly different. To my mind, skip_headers is a bool and
skiprows is (more general and) an int. Using R too much maybe.

Skipper


From alan.isaac at gmail.com  Tue Apr  5 18:16:37 2011
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Tue, 05 Apr 2011 18:16:37 -0400
Subject: [Numpy-discussion] Remove deprecated skiprows
In-Reply-To: <BANLkTim4+1+nHYugYHkTDvMcpMqff2F1Sg@mail.gmail.com>
References: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>	<BANLkTin_PpZbMrs8mPr_z2fW0DB93qDOqg@mail.gmail.com>	<BANLkTikxVpi2RrPrpw0jfYZPcf76xU-0Cw@mail.gmail.com>	<643550D8-8FED-4316-A9D0-CC90981F92AD@gmail.com>
	<BANLkTim4+1+nHYugYHkTDvMcpMqff2F1Sg@mail.gmail.com>
Message-ID: <4D9B94C5.8050706@gmail.com>

On 4/5/2011 6:11 PM, Skipper Seabold wrote:
> To my mind, skip_headers is a bool


I agree.

Alan Isaac



From pgmdevlist at gmail.com  Tue Apr  5 18:31:34 2011
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 6 Apr 2011 00:31:34 +0200
Subject: [Numpy-discussion] Remove deprecated skiprows
In-Reply-To: <4D9B94C5.8050706@gmail.com>
References: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>	<BANLkTin_PpZbMrs8mPr_z2fW0DB93qDOqg@mail.gmail.com>	<BANLkTikxVpi2RrPrpw0jfYZPcf76xU-0Cw@mail.gmail.com>	<643550D8-8FED-4316-A9D0-CC90981F92AD@gmail.com>
	<BANLkTim4+1+nHYugYHkTDvMcpMqff2F1Sg@mail.gmail.com>
	<4D9B94C5.8050706@gmail.com>
Message-ID: <865B69E2-917E-457E-86E5-500BBB1E1C09@gmail.com>


On Apr 6, 2011, at 12:16 AM, Alan G Isaac wrote:

> On 4/5/2011 6:11 PM, Skipper Seabold wrote:
>> To my mind, skip_headers is a bool

Well, that's understandable... So, yet another change of name to skip_header_lines, skip_footer_lines, with the deprecation that goes with it ? More seriously, as it's been noted before, the change has been around for a while. 
Anyhow, it's up to y'all, as you see fit.

From josef.pktd at gmail.com  Tue Apr  5 18:36:30 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 5 Apr 2011 18:36:30 -0400
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTimtAD9_wDY6fc0Rq2_kunKKp-G0GQ@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
	<4D9B4768.1060105@noaa.gov>
	<BANLkTimtAD9_wDY6fc0Rq2_kunKKp-G0GQ@mail.gmail.com>
Message-ID: <BANLkTi=ErZT3Nz=ovhqB35CY7ERRF1a+uQ@mail.gmail.com>

On Tue, Apr 5, 2011 at 5:56 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Tue, Apr 5, 2011 at 9:46 AM, Christopher Barker
> <Chris.Barker at noaa.gov> wrote:
>> On 4/4/11 10:35 PM, Charles R Harris wrote:
>>> ? ? IIUC, "Ub" is undefined -- "U" means universal newlines, which makes no
>>> ? ? sense when used with "b" for binary. I looked at the code a ways back,
>>> ? ? and I can't remember the resolution order, but there isn't any checking
>>> ? ? for incompatible flags.
>
> I disagree that U makes no sense for binary file reading.
>
> In python 3:
>
> 'b' means, "return byte objects"
> 't' means "return decoded strings"
>
> 'U' means two things:
>
> 1) When iterating by line, split lines at any of '\r', '\r\n', '\n'
> 2) When returning lines split this way, convert '\r' and '\r\n' to '\n'
>
> If you support returning lines from a binary file (which python 3
> does), then I think 'U' is a sensible thing to allow - as in this
> case.

U looks appropriate in this case, better than the workarounds.
However, to me the python 3.2 docs seem to say that U only works for
text mode and readline only takes \n as line separator

readline(limit=-1)
Read and return one line from the stream. If limit is specified, at
most limit bytes will be read.

The line terminator is always b'\n' for binary files; for text files,
the newlines argument to open() can be used to select the line
terminator(s) recognized.

Josef


>
> Best,
>
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From mwwiebe at gmail.com  Tue Apr  5 19:05:27 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Tue, 5 Apr 2011 16:05:27 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <4D9A9623.7080601@uci.edu>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<4D9A9623.7080601@uci.edu>
Message-ID: <BANLkTimtORdE-dXnw1b7h0zfN=kb=KeNKw@mail.gmail.com>

On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:

>
> <snip>
>
> A few numpy tests fail on win-amd64:
>
> <snip>
>
> ======================================================================
> FAIL: test_iterator.test_iter_broadcasting_errors
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187, in
> runTest
>     self.test(*self.arg)
>   File
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> line 639, in test_iter_broadcasting_errors
>     'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
>   File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> 34, in assert_
>     raise AssertionError(msg)
> AssertionError: Message "non-broadcastable output operand with shape
> (%lld,%lld)
>  doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain
> operand shape (2,3)
>

I've pushed a fix for this to the 1.6.x branch, can you confirm that it
works on win-amd64?

Thanks,
Mark

<snip>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/aa357426/attachment.html>

From Chris.Barker at noaa.gov  Tue Apr  5 19:12:20 2011
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 05 Apr 2011 16:12:20 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTi=ErZT3Nz=ovhqB35CY7ERRF1a+uQ@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
	<4D9B4768.1060105@noaa.gov>
	<BANLkTimtAD9_wDY6fc0Rq2_kunKKp-G0GQ@mail.gmail.com>
	<BANLkTi=ErZT3Nz=ovhqB35CY7ERRF1a+uQ@mail.gmail.com>
Message-ID: <4D9BA1D4.2000408@noaa.gov>

On 4/5/11 3:36 PM, josef.pktd at gmail.com wrote:
>> I disagree that U makes no sense for binary file reading.

I wasn't saying that it made no sense to have a "U" mode for binary file 
reading, what I meant is that by the python2 definition, it made no 
sense. In Python 2, the ONLY difference between binary and text mode is 
line-feed translation.

As for Python 3:

>> In python 3:
>>
>> 'b' means, "return byte objects"
>> 't' means "return decoded strings"
>>
>> 'U' means two things:
>>
>> 1) When iterating by line, split lines at any of '\r', '\r\n', '\n'
>> 2) When returning lines split this way, convert '\r' and '\r\n' to '\n'

a) 'U' is default -- it's essentially the same as 't' (in PY3), so 't' 
means "return decoded and line-feed translated unicode objects"

b) I think the line-feed conversion is done regardless of if you are 
iterating by lines, i.e. with a full-on .read(). At least that's how it 
works in py2 -- not running py3 here to test.

>> If you support returning lines from a binary file (which python 3
>> does), then I think 'U' is a sensible thing to allow - as in this
>> case.

but what is a "binary file"?

I THINK what you are proposing is that we'd want to be able to have both 
linefeed translation and no decoding done. But I think that's impossible 
-- aren't the linefeeds themselves encoded differently with different 
encodings?

> U looks appropriate in this case, better than the workarounds.
> However, to me the python 3.2 docs seem to say that U only works for
> text mode

Agreed -- but I don't see the problem -- your files are either encoded 
in something that might treat newlines differently (UCS32, maybe?), in 
which case you'd want it decoded, or you are working with ascii or ansi 
or utf-8, in which case you can specify the encoding anyway.

I don't understand why we'd want a binary blob for text parsing -- the 
parsing code is going to have to know something about the encoding to 
work -- it might as well get passed in to the file open call, and work 
with unicode. I suppose if we still want to assume ascii for parsing, 
then we could use 't' and then re-encode to ascii to work with it. Which 
I agree does seem heavy handed just for fixing newlines.

Also, one problem I've often had with encodings is what happens if I 
think I have ascii, but really have a couple characters above 127 -- 
then the default is to get an error in decoding. I'd like to be able to 
pass in a flag that either skips the un-decodable characters or replaces 
them with something, but it doesn't look like you can do that with the 
file open function in py3.

> The line terminator is always b'\n' for binary files;

Once you really make the distiction between text and binary, the concept 
of a "line terminator" doesn't really make sense anyway.

In the ansi world, everyone should always have used 'U' for text. It 
probably would have been the default if it had been there from the 
beginning. People got away without it because:
  1) dos line feeds have a "\n" in them anyway
  2) most if the time it doesn't matter that there is an extra 
whitespace charater inther
  3) darn few of us ever had to deal with the mac "\r"

Now that we are in a unicode world (at least a little) there is simply 
no way around the fact that you can't reliably read a file without 
knowing how it is encoded.

My thought at this point is to say that the numpy text file reading 
stuff only works on 1byte, ansi encoding (nad maybe only ascii), and be 
done with it. utf-8 might be OK -- I don't know if there are any valid 
files in, say latin-1 that utf-8 will barf on -- you may not get the 
non-ascii symbols right, but that's better than barfing.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From tgrav at mac.com  Tue Apr  5 19:15:11 2011
From: tgrav at mac.com (Tommy Grav)
Date: Tue, 05 Apr 2011 19:15:11 -0400
Subject: [Numpy-discussion] atpy and readable buffer error
Message-ID: <DAC18BA0-1DE1-467D-BCDD-7EA6C5280554@mac.com>

I get this error that I do not understand. Does anyone know what is happening
and how to get around it

>>> import atpy
>>> t = atpy.Table()
>>> name = ["aa","bb","cc"]
>>> t.add_column("name",name)
>>> k = [1,2,3]
>>> t.add_column("num",k)
>>> t
<Table name='None' rows=3 fields=2>
>>> name = ["aaa","bbb","ccc"]
>>> p = atpy.Table()
>>> p.add_column("name",name)
>>> p.add_column("num",k)
>>> t.append(p)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/atpy/basetable.py", line 301, in append
   raise Exception("Columns do not match")
Exception: Columns do not match
>>> nn = t.name.astype("|S3")
>>> t.remove_columns("name")
>>> t.add_column("name",nn)
>>> t.append(p)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/atpy/basetable.py", line 302, in append
   self.data = np.hstack((self.data, table.data))
 File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/shape_base.py", line 258, in hstack
   return _nx.concatenate(map(atleast_1d,tup),1)
TypeError: expected a readable buffer object
>>> 

Cheers
  Tommy


From wesmckinn at gmail.com  Tue Apr  5 19:20:11 2011
From: wesmckinn at gmail.com (Wes McKinney)
Date: Tue, 5 Apr 2011 19:20:11 -0400
Subject: [Numpy-discussion] Remove deprecated skiprows
In-Reply-To: <BANLkTikxVpi2RrPrpw0jfYZPcf76xU-0Cw@mail.gmail.com>
References: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>
	<BANLkTin_PpZbMrs8mPr_z2fW0DB93qDOqg@mail.gmail.com>
	<BANLkTikxVpi2RrPrpw0jfYZPcf76xU-0Cw@mail.gmail.com>
Message-ID: <BANLkTik=APUBM-6Me6-saUab_xWgnMiZ=w@mail.gmail.com>

On Tue, Apr 5, 2011 at 5:52 PM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
> On Tue, Apr 5, 2011 at 11:45 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>> On Sun, Apr 3, 2011 at 8:20 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>> Should skiprows be removed?
>>>
>>> ??? if skiprows:
>>> ??????? warnings.warn(\
>>> ??????????? "The use of `skiprows` is deprecated, it will be removed in
>>> numpy 2.0.\n" \
>>> ??????????? "Please use `skip_header` instead.",
>>> ??????????? DeprecationWarning)
>>> ??????? skip_header = skiprows
>>>
>>> Its been deprecated since 1.4. Personally, I like skiprows better than
>>> skip_header ;)
>>>
>>
>> +1 for skiprows. I always have to look it up.
>
> To me one is not much better than the other, but -1 for skiprows
> because un-deprecating it and deprecating skip_header is inconsistent
> and annoying for users.
>
> Ralf
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

FYI:

http://docs.python.org/library/warnings.html

"DeprecationWarning	Base category for warnings about deprecated
features (ignored by default)."

Maybe DeprecationWarnings should be changed to something that isn't
ignored by default so users see it?


From robert.kern at gmail.com  Tue Apr  5 19:32:39 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 5 Apr 2011 18:32:39 -0500
Subject: [Numpy-discussion] Remove deprecated skiprows
In-Reply-To: <BANLkTik=APUBM-6Me6-saUab_xWgnMiZ=w@mail.gmail.com>
References: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>
	<BANLkTin_PpZbMrs8mPr_z2fW0DB93qDOqg@mail.gmail.com>
	<BANLkTikxVpi2RrPrpw0jfYZPcf76xU-0Cw@mail.gmail.com>
	<BANLkTik=APUBM-6Me6-saUab_xWgnMiZ=w@mail.gmail.com>
Message-ID: <BANLkTimbO1ydcMReTz8NX6KZ0mM9FvFBMA@mail.gmail.com>

On Tue, Apr 5, 2011 at 18:20, Wes McKinney <wesmckinn at gmail.com> wrote:

> FYI:
>
> http://docs.python.org/library/warnings.html
>
> "DeprecationWarning ? ? Base category for warnings about deprecated
> features (ignored by default)."
>
> Maybe DeprecationWarnings should be changed to something that isn't
> ignored by default so users see it?

The Python devs made DeprecationWarnings silent by default for a
reason. They unnecessarily annoy and frighten application end users
who don't know what they are, where they are coming from, or how to
silence them. The recommended practice is for library users (who do or
should know those things) to run their test suites with the warnings
turned on.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
? -- Umberto Eco


From matthew.brett at gmail.com  Tue Apr  5 19:51:55 2011
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 5 Apr 2011 16:51:55 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <4D9BA1D4.2000408@noaa.gov>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
	<4D9B4768.1060105@noaa.gov>
	<BANLkTimtAD9_wDY6fc0Rq2_kunKKp-G0GQ@mail.gmail.com>
	<BANLkTi=ErZT3Nz=ovhqB35CY7ERRF1a+uQ@mail.gmail.com>
	<4D9BA1D4.2000408@noaa.gov>
Message-ID: <BANLkTinZfFQDV-eX0bKoUwtMbpDQ_EckPw@mail.gmail.com>

Hi,

On Tue, Apr 5, 2011 at 4:12 PM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
> On 4/5/11 3:36 PM, josef.pktd at gmail.com wrote:
>>> I disagree that U makes no sense for binary file reading.
>
> I wasn't saying that it made no sense to have a "U" mode for binary file
> reading, what I meant is that by the python2 definition, it made no
> sense. In Python 2, the ONLY difference between binary and text mode is
> line-feed translation.

I think it's right to say that the difference between a text and a
binary file in python 2 is - none for unix, and '\r\n' -> '\n'
translation in windows.

The difference between 'rt' and 'U' is (this is for my own benefit):

For 'rt', a '\r' does not cause a line break - with 'U' - it does.
For 'rt' _not_ on Windows, '\r\n' stays the same - it is stripped to
'\n' with 'U'.

> As for Python 3:
>
>>> In python 3:
>>>
>>> 'b' means, "return byte objects"
>>> 't' means "return decoded strings"
>>>
>>> 'U' means two things:
>>>
>>> 1) When iterating by line, split lines at any of '\r', '\r\n', '\n'
>>> 2) When returning lines split this way, convert '\r' and '\r\n' to '\n'
>
> a) 'U' is default -- it's essentially the same as 't' (in PY3), so 't'
> means "return decoded and line-feed translated unicode objects"

Right - my argument is that the behavior implied by 'U' and 't' is
conceptually separable.   'U' is for how to do line-breaks, and
line-termination translations, 't' is for whether to decode the text
or not.  In python 3.

> b) I think the line-feed conversion is done regardless of if you are
> iterating by lines, i.e. with a full-on .read(). At least that's how it
> works in py2 -- not running py3 here to test.

Yes, that looks right.

>>> If you support returning lines from a binary file (which python 3
>>> does), then I think 'U' is a sensible thing to allow - as in this
>>> case.
>
> but what is a "binary file"?

In python 3 a binary file is a file which is not decoded, and returns
bytes.  It still has a concept of a 'line', as defined by line
terminators - you can iterate over one, or do .readlines().  In python
2, as you say, a binary file is essentially the same as a text file,
with the single exception of the windows \r\n -> \n translation.

> I THINK what you are proposing is that we'd want to be able to have both
> linefeed translation and no decoding done. But I think that's impossible
> -- aren't the linefeeds themselves encoded differently with different
> encodings?

Right - so obviously if you open a utf-16 file as binary, terrible
things may happen - this was what Pauli was pointing out before.  His
point was that utf-8 is the standard, and that we probably would not
hit many other encodings.    I agree with you if you are saying that
it would be good to be able to deal with them if we can - presumably
by allowing 'rt' file objects, producing python 3 strings.

>> U looks appropriate in this case, better than the workarounds.
>> However, to me the python 3.2 docs seem to say that U only works for
>> text mode
>
> Agreed -- but I don't see the problem -- your files are either encoded
> in something that might treat newlines differently (UCS32, maybe?), in
> which case you'd want it decoded, or you are working with ascii or ansi
> or utf-8, in which case you can specify the encoding anyway.
>
> I don't understand why we'd want a binary blob for text parsing -- the
> parsing code is going to have to know something about the encoding to
> work -- it might as well get passed in to the file open call, and work
> with unicode. I suppose if we still want to assume ascii for parsing,
> then we could use 't' and then re-encode to ascii to work with it. Which
> I agree does seem heavy handed just for fixing newlines.
>
> Also, one problem I've often had with encodings is what happens if I
> think I have ascii, but really have a couple characters above 127 --
> then the default is to get an error in decoding. I'd like to be able to
> pass in a flag that either skips the un-decodable characters or replaces
> them with something, but it doesn't look like you can do that with the
> file open function in py3.
>
>> The line terminator is always b'\n' for binary files;
>
> Once you really make the distiction between text and binary, the concept
> of a "line terminator" doesn't really make sense anyway.

Well - I was arguing that, given we can iterate over lines in binary
files, then there must be the concept of what a line is, in a binary
file, and that means that we need the concept of a line terminator.

I realize this is a discussion that would have to happen on the
python-dev list...

See you,

Matthew


From cgohlke at uci.edu  Tue Apr  5 20:07:54 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Tue, 05 Apr 2011 17:07:54 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTimtORdE-dXnw1b7h0zfN=kb=KeNKw@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>	<4D9A9623.7080601@uci.edu>
	<BANLkTimtORdE-dXnw1b7h0zfN=kb=KeNKw@mail.gmail.com>
Message-ID: <4D9BAEDA.8060205@uci.edu>



On 4/5/2011 4:05 PM, Mark Wiebe wrote:
> On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke <cgohlke at uci.edu
> <mailto:cgohlke at uci.edu>> wrote:
>
>
>     <snip>
>
>     A few numpy tests fail on win-amd64:
>
>     <snip>
>
>     ======================================================================
>     FAIL: test_iterator.test_iter_broadcasting_errors
>     ----------------------------------------------------------------------
>     Traceback (most recent call last):
>        File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187, in
>     runTest
>          self.test(*self.arg)
>        File
>     "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
>     line 639, in test_iter_broadcasting_errors
>     'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
>        File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
>     34, in assert_
>          raise AssertionError(msg)
>     AssertionError: Message "non-broadcastable output operand with shape
>     (%lld,%lld)
>       doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain
>     operand shape (2,3)
>
>
> I've pushed a fix for this to the 1.6.x branch, can you confirm that it
> works on win-amd64?
>
> Thanks,
> Mark
>
>     <snip>
>

Sorry, I forgot to mention that this test failed on 64-bit Python 2.6 
only. I now recognize it is due to a known issue with Python's 
PyErr_Format function <http://bugs.python.org/issue7228>. Unfortunately 
the fix will not be backported to Python 2.6. Maybe this test could be 
marked as known failure on win-amd64-py2.6?

Could you please revert your changes or set the format specifier to 
"%lld"? "%I64d" is not supported by PyErr_Format.

Thanks,

Christoph


From wesmckinn at gmail.com  Tue Apr  5 20:46:46 2011
From: wesmckinn at gmail.com (Wes McKinney)
Date: Tue, 5 Apr 2011 20:46:46 -0400
Subject: [Numpy-discussion] Remove deprecated skiprows
In-Reply-To: <BANLkTimbO1ydcMReTz8NX6KZ0mM9FvFBMA@mail.gmail.com>
References: <BANLkTimomOV-irxmPh26qyCD8FCdTuRaVQ@mail.gmail.com>
	<BANLkTin_PpZbMrs8mPr_z2fW0DB93qDOqg@mail.gmail.com>
	<BANLkTikxVpi2RrPrpw0jfYZPcf76xU-0Cw@mail.gmail.com>
	<BANLkTik=APUBM-6Me6-saUab_xWgnMiZ=w@mail.gmail.com>
	<BANLkTimbO1ydcMReTz8NX6KZ0mM9FvFBMA@mail.gmail.com>
Message-ID: <BANLkTikjDoqCuH+K+jvC4y11C+SY3wBe1w@mail.gmail.com>

On Tue, Apr 5, 2011 at 7:32 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, Apr 5, 2011 at 18:20, Wes McKinney <wesmckinn at gmail.com> wrote:
>
>> FYI:
>>
>> http://docs.python.org/library/warnings.html
>>
>> "DeprecationWarning ? ? Base category for warnings about deprecated
>> features (ignored by default)."
>>
>> Maybe DeprecationWarnings should be changed to something that isn't
>> ignored by default so users see it?
>
> The Python devs made DeprecationWarnings silent by default for a
> reason. They unnecessarily annoy and frighten application end users
> who don't know what they are, where they are coming from, or how to
> silence them. The recommended practice is for library users (who do or
> should know those things) to run their test suites with the warnings
> turned on.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ? -- Umberto Eco
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

You make a very convincing point-- makes complete sense.


From mwwiebe at gmail.com  Tue Apr  5 21:08:13 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Tue, 5 Apr 2011 18:08:13 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <4D9BAEDA.8060205@uci.edu>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<4D9A9623.7080601@uci.edu>
	<BANLkTimtORdE-dXnw1b7h0zfN=kb=KeNKw@mail.gmail.com>
	<4D9BAEDA.8060205@uci.edu>
Message-ID: <BANLkTikxdPzDcm9ejbE0Otoc007DtRQ+Cw@mail.gmail.com>

On Tue, Apr 5, 2011 at 5:07 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:

>
>
> On 4/5/2011 4:05 PM, Mark Wiebe wrote:
> > On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke <cgohlke at uci.edu
> > <mailto:cgohlke at uci.edu>> wrote:
> >
> >
> >     <snip>
> >
> >     A few numpy tests fail on win-amd64:
> >
> >     <snip>
> >
> >
> ======================================================================
> >     FAIL: test_iterator.test_iter_broadcasting_errors
> >
> ----------------------------------------------------------------------
> >     Traceback (most recent call last):
> >        File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187,
> in
> >     runTest
> >          self.test(*self.arg)
> >        File
> >
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> >     line 639, in test_iter_broadcasting_errors
> >     'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
> >        File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py",
> line
> >     34, in assert_
> >          raise AssertionError(msg)
> >     AssertionError: Message "non-broadcastable output operand with shape
> >     (%lld,%lld)
> >       doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain
> >     operand shape (2,3)
> >
> >
> > I've pushed a fix for this to the 1.6.x branch, can you confirm that it
> > works on win-amd64?
> >
> > Thanks,
> > Mark
> >
> >     <snip>
> >
>
> Sorry, I forgot to mention that this test failed on 64-bit Python 2.6
> only. I now recognize it is due to a known issue with Python's
> PyErr_Format function <http://bugs.python.org/issue7228>. Unfortunately
> the fix will not be backported to Python 2.6. Maybe this test could be
> marked as known failure on win-amd64-py2.6?
>
> Could you please revert your changes or set the format specifier to
> "%lld"? "%I64d" is not supported by PyErr_Format.
>

This means PyString_Format and PyOS_snprintf support different sets of
formatting characters, which begs the question of which set of functions
should the NPY_*_FMT macros be intended for? Currently it looks like just
the NPY_INTP_FMT is used with PyString_Format/PyErr_Format, and the rest are
used with PyOS_snprintf. Also, in other places npy_intp variables are cast
to long and "%ld" is used instead of NPY_INTP_FMT. I suppose I'll just
change NPY_INTP_FMT unconditionally to "%lld" when it's long long, since
that's what the PyString_Format 2.7 documentation says is the correct
portable formatter.

-Mark


>
> Thanks,
>
> Christoph
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/dfe71400/attachment.html>

From mwwiebe at gmail.com  Tue Apr  5 21:46:54 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Tue, 5 Apr 2011 18:46:54 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <4D9BAEDA.8060205@uci.edu>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<4D9A9623.7080601@uci.edu>
	<BANLkTimtORdE-dXnw1b7h0zfN=kb=KeNKw@mail.gmail.com>
	<4D9BAEDA.8060205@uci.edu>
Message-ID: <BANLkTin29q5Wg9m-88uVwR6VwxAS9toJ8g@mail.gmail.com>

On Tue, Apr 5, 2011 at 5:07 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:

>
>
> On 4/5/2011 4:05 PM, Mark Wiebe wrote:
> > On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke <cgohlke at uci.edu
> > <mailto:cgohlke at uci.edu>> wrote:
> >
> >
> >     <snip>
> >
> >     A few numpy tests fail on win-amd64:
> >
> >     <snip>
> >
> >
> ======================================================================
> >     FAIL: test_iterator.test_iter_broadcasting_errors
> >
> ----------------------------------------------------------------------
> >     Traceback (most recent call last):
> >        File "X:\Python26-x64\lib\site-packages\nose\case.py", line 187,
> in
> >     runTest
> >          self.test(*self.arg)
> >        File
> >
> "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> >     line 639, in test_iter_broadcasting_errors
> >     'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
> >        File "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py",
> line
> >     34, in assert_
> >          raise AssertionError(msg)
> >     AssertionError: Message "non-broadcastable output operand with shape
> >     (%lld,%lld)
> >       doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't contain
> >     operand shape (2,3)
> >
> >
> > I've pushed a fix for this to the 1.6.x branch, can you confirm that it
> > works on win-amd64?
> >
> > Thanks,
> > Mark
> >
> >     <snip>
> >
>
> Sorry, I forgot to mention that this test failed on 64-bit Python 2.6
> only. I now recognize it is due to a known issue with Python's
> PyErr_Format function <http://bugs.python.org/issue7228>. Unfortunately
> the fix will not be backported to Python 2.6. Maybe this test could be
> marked as known failure on win-amd64-py2.6?
>
> Could you please revert your changes or set the format specifier to
> "%lld"? "%I64d" is not supported by PyErr_Format.
>

I've committed an attempted workaround using the %zd formatter, can you
check whether it works on 64-bit Windows Python 2.6 now?

Thanks,
Mark


>
> Thanks,
>
> Christoph
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/72ff6226/attachment.html>

From cgohlke at uci.edu  Tue Apr  5 22:21:18 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Tue, 05 Apr 2011 19:21:18 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTin29q5Wg9m-88uVwR6VwxAS9toJ8g@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>	<4D9A9623.7080601@uci.edu>	<BANLkTimtORdE-dXnw1b7h0zfN=kb=KeNKw@mail.gmail.com>	<4D9BAEDA.8060205@uci.edu>
	<BANLkTin29q5Wg9m-88uVwR6VwxAS9toJ8g@mail.gmail.com>
Message-ID: <4D9BCE1E.5040405@uci.edu>



On 4/5/2011 6:46 PM, Mark Wiebe wrote:
> On Tue, Apr 5, 2011 at 5:07 PM, Christoph Gohlke <cgohlke at uci.edu
> <mailto:cgohlke at uci.edu>> wrote:
>
>
>
>     On 4/5/2011 4:05 PM, Mark Wiebe wrote:
>     >  On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke <cgohlke at uci.edu
>     <mailto:cgohlke at uci.edu>
>     >  <mailto:cgohlke at uci.edu <mailto:cgohlke at uci.edu>>> wrote:
>     >
>     >
>     >  <snip>
>     >
>     >      A few numpy tests fail on win-amd64:
>     >
>     >  <snip>
>     >
>     >
>     ======================================================================
>     >      FAIL: test_iterator.test_iter_broadcasting_errors
>     >
>     ----------------------------------------------------------------------
>     >      Traceback (most recent call last):
>     >         File "X:\Python26-x64\lib\site-packages\nose\case.py", line
>     187, in
>     >      runTest
>     >           self.test(*self.arg)
>     >         File
>     >  "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
>     >      line 639, in test_iter_broadcasting_errors
>     >  'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
>     >         File
>     "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
>     >      34, in assert_
>     >           raise AssertionError(msg)
>     >      AssertionError: Message "non-broadcastable output operand with
>     shape
>     >      (%lld,%lld)
>     >        doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't
>     contain
>     >      operand shape (2,3)
>     >
>     >
>     >  I've pushed a fix for this to the 1.6.x branch, can you confirm
>     that it
>     >  works on win-amd64?
>     >
>     >  Thanks,
>     >  Mark
>     >
>     >  <snip>
>     >
>
>     Sorry, I forgot to mention that this test failed on 64-bit Python 2.6
>     only. I now recognize it is due to a known issue with Python's
>     PyErr_Format function <http://bugs.python.org/issue7228>. Unfortunately
>     the fix will not be backported to Python 2.6. Maybe this test could be
>     marked as known failure on win-amd64-py2.6?
>
>     Could you please revert your changes or set the format specifier to
>     "%lld"? "%I64d" is not supported by PyErr_Format.
>
>
> I've committed an attempted workaround using the %zd formatter, can you
> check whether it works on 64-bit Windows Python 2.6 now?
>
> Thanks,
> Mark
>

I have not tried but I do not expect it to work. According to 
<http://docs.python.org/c-api/string.html> the "%zd" format is 'exactly 
equivalent to printf("%zd")'. The 'z' prefix character is C99 AFAIK and 
not supported by Visual Studio compilers 
<http://msdn.microsoft.com/en-us/library/xdb9w69d%28v=VS.90%29.aspx>.

Christoph


From mwwiebe at gmail.com  Tue Apr  5 22:44:56 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Tue, 5 Apr 2011 19:44:56 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <4D9BCE1E.5040405@uci.edu>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<4D9A9623.7080601@uci.edu>
	<BANLkTimtORdE-dXnw1b7h0zfN=kb=KeNKw@mail.gmail.com>
	<4D9BAEDA.8060205@uci.edu>
	<BANLkTin29q5Wg9m-88uVwR6VwxAS9toJ8g@mail.gmail.com>
	<4D9BCE1E.5040405@uci.edu>
Message-ID: <BANLkTi=BVij7UCGbjv0AzMmyeySzW=WnyQ@mail.gmail.com>

On Tue, Apr 5, 2011 at 7:21 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:

>
>
> On 4/5/2011 6:46 PM, Mark Wiebe wrote:
> > On Tue, Apr 5, 2011 at 5:07 PM, Christoph Gohlke <cgohlke at uci.edu
> > <mailto:cgohlke at uci.edu>> wrote:
> >
> >
> >
> >     On 4/5/2011 4:05 PM, Mark Wiebe wrote:
> >     >  On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke <cgohlke at uci.edu
> >     <mailto:cgohlke at uci.edu>
> >     >  <mailto:cgohlke at uci.edu <mailto:cgohlke at uci.edu>>> wrote:
> >     >
> >     >
> >     >  <snip>
> >     >
> >     >      A few numpy tests fail on win-amd64:
> >     >
> >     >  <snip>
> >     >
> >     >
> >
> ======================================================================
> >     >      FAIL: test_iterator.test_iter_broadcasting_errors
> >     >
> >
> ----------------------------------------------------------------------
> >     >      Traceback (most recent call last):
> >     >         File "X:\Python26-x64\lib\site-packages\nose\case.py", line
> >     187, in
> >     >      runTest
> >     >           self.test(*self.arg)
> >     >         File
> >     >
>  "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
> >     >      line 639, in test_iter_broadcasting_errors
> >     >  'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
> >     >         File
> >     "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
> >     >      34, in assert_
> >     >           raise AssertionError(msg)
> >     >      AssertionError: Message "non-broadcastable output operand with
> >     shape
> >     >      (%lld,%lld)
> >     >        doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't
> >     contain
> >     >      operand shape (2,3)
> >     >
> >     >
> >     >  I've pushed a fix for this to the 1.6.x branch, can you confirm
> >     that it
> >     >  works on win-amd64?
> >     >
> >     >  Thanks,
> >     >  Mark
> >     >
> >     >  <snip>
> >     >
> >
> >     Sorry, I forgot to mention that this test failed on 64-bit Python 2.6
> >     only. I now recognize it is due to a known issue with Python's
> >     PyErr_Format function <http://bugs.python.org/issue7228>.
> Unfortunately
> >     the fix will not be backported to Python 2.6. Maybe this test could
> be
> >     marked as known failure on win-amd64-py2.6?
> >
> >     Could you please revert your changes or set the format specifier to
> >     "%lld"? "%I64d" is not supported by PyErr_Format.
> >
> >
> > I've committed an attempted workaround using the %zd formatter, can you
> > check whether it works on 64-bit Windows Python 2.6 now?
> >
> > Thanks,
> > Mark
> >
>
> I have not tried but I do not expect it to work. According to
> <http://docs.python.org/c-api/string.html> the "%zd" format is 'exactly
> equivalent to printf("%zd")'. The 'z' prefix character is C99 AFAIK and
> not supported by Visual Studio compilers
> <http://msdn.microsoft.com/en-us/library/xdb9w69d%28v=VS.90%29.aspx>.


I think there's a good chance it will work, since the CPython code is
re-interpreting the format codes itself. In the Python 2.6 code, it looks
like it will become %Id on Windows, which would be correct.

-Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110405/6fbd95fc/attachment.html>

From cgohlke at uci.edu  Wed Apr  6 01:09:25 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Tue, 05 Apr 2011 22:09:25 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTi=BVij7UCGbjv0AzMmyeySzW=WnyQ@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>	<4D9A9623.7080601@uci.edu>	<BANLkTimtORdE-dXnw1b7h0zfN=kb=KeNKw@mail.gmail.com>	<4D9BAEDA.8060205@uci.edu>	<BANLkTin29q5Wg9m-88uVwR6VwxAS9toJ8g@mail.gmail.com>	<4D9BCE1E.5040405@uci.edu>
	<BANLkTi=BVij7UCGbjv0AzMmyeySzW=WnyQ@mail.gmail.com>
Message-ID: <4D9BF585.4080009@uci.edu>



On 4/5/2011 7:44 PM, Mark Wiebe wrote:
> On Tue, Apr 5, 2011 at 7:21 PM, Christoph Gohlke <cgohlke at uci.edu
> <mailto:cgohlke at uci.edu>> wrote:
>
>
>
>     On 4/5/2011 6:46 PM, Mark Wiebe wrote:
>     >  On Tue, Apr 5, 2011 at 5:07 PM, Christoph Gohlke <cgohlke at uci.edu
>     <mailto:cgohlke at uci.edu>
>     >  <mailto:cgohlke at uci.edu <mailto:cgohlke at uci.edu>>> wrote:
>     >
>     >
>     >
>     >      On 4/5/2011 4:05 PM, Mark Wiebe wrote:
>     >  >  On Mon, Apr 4, 2011 at 9:10 PM, Christoph Gohlke
>     <cgohlke at uci.edu <mailto:cgohlke at uci.edu>
>     >  <mailto:cgohlke at uci.edu <mailto:cgohlke at uci.edu>>
>     >  > <mailto:cgohlke at uci.edu <mailto:cgohlke at uci.edu>
>     <mailto:cgohlke at uci.edu <mailto:cgohlke at uci.edu>>>> wrote:
>     >  >
>     >  >
>     >  > <snip>
>     >  >
>     >  >      A few numpy tests fail on win-amd64:
>     >  >
>     >  > <snip>
>     >  >
>     >  >
>     >
>     ======================================================================
>     >  >      FAIL: test_iterator.test_iter_broadcasting_errors
>     >  >
>     >
>     ----------------------------------------------------------------------
>     >  >      Traceback (most recent call last):
>     >  >         File "X:\Python26-x64\lib\site-packages\nose\case.py", line
>     >      187, in
>     >  >      runTest
>     >  >           self.test(*self.arg)
>     >  >         File
>     >  >
>     "X:\Python26-x64\lib\site-packages\numpy\core\tests\test_iterator.py",
>     >  >      line 639, in test_iter_broadcasting_errors
>     >  > 'Message "%s" doesn\'t contain operand shape (2,3)' % msg)
>     >  >         File
>     >  "X:\Python26-x64\lib\site-packages\numpy\testing\utils.py", line
>     >  >      34, in assert_
>     >  >           raise AssertionError(msg)
>     >  >      AssertionError: Message "non-broadcastable output operand with
>     >      shape
>     >  >      (%lld,%lld)
>     >  >        doesn't match the broadcast shape (%lld,%lld,%lld)" doesn't
>     >      contain
>     >  >      operand shape (2,3)
>     >  >
>     >  >
>     >  >  I've pushed a fix for this to the 1.6.x branch, can you confirm
>     >      that it
>     >  >  works on win-amd64?
>     >  >
>     >  >  Thanks,
>     >  >  Mark
>     >  >
>     >  > <snip>
>     >  >
>     >
>     >      Sorry, I forgot to mention that this test failed on 64-bit
>     Python 2.6
>     >      only. I now recognize it is due to a known issue with Python's
>     >      PyErr_Format function <http://bugs.python.org/issue7228>.
>     Unfortunately
>     >      the fix will not be backported to Python 2.6. Maybe this test
>     could be
>     >      marked as known failure on win-amd64-py2.6?
>     >
>     >      Could you please revert your changes or set the format
>     specifier to
>     >  "%lld"? "%I64d" is not supported by PyErr_Format.
>     >
>     >
>     >  I've committed an attempted workaround using the %zd formatter,
>     can you
>     >  check whether it works on 64-bit Windows Python 2.6 now?
>     >
>     >  Thanks,
>     >  Mark
>     >
>
>     I have not tried but I do not expect it to work. According to
>     <http://docs.python.org/c-api/string.html> the "%zd" format is 'exactly
>     equivalent to printf("%zd")'. The 'z' prefix character is C99 AFAIK and
>     not supported by Visual Studio compilers
>     <http://msdn.microsoft.com/en-us/library/xdb9w69d%28v=VS.90%29.aspx>.
>
>
> I think there's a good chance it will work, since the CPython code is
> re-interpreting the format codes itself. In the Python 2.6 code, it
> looks like it will become %Id on Windows, which would be correct.
>
> -Mark
>

Agreed, it'll work if NPY_INTP_FMT is only used with certain Python 
functions that use %Id instead of %zd on Windows. Then why not simply 
define NPY_INTP_FMT as %zd?

-    #if (PY_VERSION_HEX > 0x02060000)
-        #define NPY_INTP_FMT "lld"
-    #else
          #define NPY_INTP_FMT "zd"
-    #endif

That should work unless 64 bit numpy is run on a LLP64 platform with 
Python < 2.5.

Christoph


From fperez.net at gmail.com  Wed Apr  6 03:26:07 2011
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 6 Apr 2011 00:26:07 -0700
Subject: [Numpy-discussion] Scientific Python at SIAM CSE 2011 conference
Message-ID: <BANLkTikuAs=zUXSoD6Ra5PigK8W9-buedA@mail.gmail.com>

Hi all,

sorry for the massive cross-post, but since all these projects were
highlighted with talks at this event, I figured there would be
interest...  Hans-Petter Langtangen, Randy LeVeque and I organized a
set of Python-focused sessions at the recent SIAM Computational
Science and Engineering conference, with talks on numpy/scipy, cython,
matplotlib, ipython, sympy, as well as application-oriented talks on
astronomy and femhub.  For those interested:

- The slides: http://fperez.org/events/2011_siam_cse/

- A blog post:
http://blog.fperez.org/2011/04/python-goes-to-reno-siam-cse-2011.html

- Some pictures: https://picasaweb.google.com/fdo.perez/SIAMCSE2011InReno#

Back to being quiet...

f


From emmanuelle.gouillart at normalesup.org  Wed Apr  6 07:33:24 2011
From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart)
Date: Wed, 6 Apr 2011 13:33:24 +0200
Subject: [Numpy-discussion] strange behavior of np.minimum and np.maximum
Message-ID: <20110406113324.GE30027@phare.normalesup.org>

	Hello,

>>> a, b, c = np.array([10]), np.array([2]), np.array([7])
>>> min_val = np.minimum(a, b, c)
>>> min_val
array([2])
>>> max_val = np.maximum(a, b, c)
>>> max_val
array([10])
>>> min_val
array([10])

(I'm using numpy 1.4, and I observed the same behavior with numpy
2.0.0.dev8600 on another machine). I'm quite surprised by this behavior
(It took me quite a long time to figure out what was happening in a
script of mine that wasn't giving what I expected, because of
np.maximum changing the output of np.minimum). Is it a bug, or am I
missing something?

	Cheers,
	Emmanuelle



From zachary.pincus at yale.edu  Wed Apr  6 07:59:00 2011
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Wed, 6 Apr 2011 07:59:00 -0400
Subject: [Numpy-discussion] strange behavior of np.minimum and np.maximum
In-Reply-To: <20110406113324.GE30027@phare.normalesup.org>
References: <20110406113324.GE30027@phare.normalesup.org>
Message-ID: <2F816E00-A66D-4716-AA0E-8CA40E8B7005@yale.edu>

>
>>>> a, b, c = np.array([10]), np.array([2]), np.array([7])
>>>> min_val = np.minimum(a, b, c)
>>>> min_val
> array([2])
>>>> max_val = np.maximum(a, b, c)
>>>> max_val
> array([10])
>>>> min_val
> array([10])
>
> (I'm using numpy 1.4, and I observed the same behavior with numpy
> 2.0.0.dev8600 on another machine). I'm quite surprised by this  
> behavior
> (It took me quite a long time to figure out what was happening in a
> script of mine that wasn't giving what I expected, because of
> np.maximum changing the output of np.minimum). Is it a bug, or am I
> missing something?

Read the documentation for numpy.minimum and numpy.maximum: they give  
you element-wise minimum values from two arrays passed as arguments.  
E.g.:

 >>> numpy.minimum([1,2,3],[3,2,1])
array([1, 2, 1])

The optional third parameter to numpy.minimum is an "out" array - an  
array to place the results into instead of making a new array for that  
purpose. (This can save time / memory in various cases.)

This should therefore be enough to explain the above behavior. (That  
is, min_val and max_val wind up being just other names for the array  
'c', which gets modified in-place by the numpy.minimum and  
numpy.maximum.)

If you want the minimum value of a sequence of arbitrary length, use  
the python min() function. If you have a numpy array already and you  
want the minimum (global, or along a particular axis), use the min()  
method of the array, or numpy.min(arr).

Zach



From derek at astro.physik.uni-goettingen.de  Wed Apr  6 08:00:26 2011
From: derek at astro.physik.uni-goettingen.de (Derek Homeier)
Date: Wed, 6 Apr 2011 14:00:26 +0200
Subject: [Numpy-discussion] strange behavior of np.minimum and np.maximum
In-Reply-To: <20110406113324.GE30027@phare.normalesup.org>
References: <20110406113324.GE30027@phare.normalesup.org>
Message-ID: <3D8B27BE-C763-4105-A08A-066E7DCF7703@astro.physik.uni-goettingen.de>

Hi Emmanuelle,

>>>> a, b, c = np.array([10]), np.array([2]), np.array([7])
>>>> min_val = np.minimum(a, b, c)
>>>> min_val
> array([2])
>>>> max_val = np.maximum(a, b, c)
>>>> max_val
> array([10])
>>>> min_val
> array([10])
>
> (I'm using numpy 1.4, and I observed the same behavior with numpy
> 2.0.0.dev8600 on another machine). I'm quite surprised by this  
> behavior
> (It took me quite a long time to figure out what was happening in a
> script of mine that wasn't giving what I expected, because of
> np.maximum changing the output of np.minimum). Is it a bug, or am I
> missing something?


you're just missing that np.minimum/np.maximum are _binary ufuncs_
with syntax
np.minimum(X, Y, out=None)
i.e. you were telling np.minimum to store it's output in array c and
then return min_val, obviously as a reference, not a copy of it.
Thus when storing the output of np.maximum in c as well, the contents
of c also changed again.
Being binary ufuncs, I think you'll have to apply them consecutively  
if you
need the min/max of several arrays.
See also np.info(np.minimum)

HTH,
							Derek



From emmanuelle.gouillart at normalesup.org  Wed Apr  6 08:13:09 2011
From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart)
Date: Wed, 6 Apr 2011 14:13:09 +0200
Subject: [Numpy-discussion] strange behavior of np.minimum and np.maximum
In-Reply-To: <2F816E00-A66D-4716-AA0E-8CA40E8B7005@yale.edu>
References: <20110406113324.GE30027@phare.normalesup.org>
	<2F816E00-A66D-4716-AA0E-8CA40E8B7005@yale.edu>
Message-ID: <20110406121309.GF30027@phare.normalesup.org>

Hi Zach and Derek,

thank you very much for your quick and clear answers. Of course the third
parameter is the out array, I was just being very stupid! (I had read the
documentation though, but somehow it didn't make it to my brain :-) Sorry...

> Read the documentation for numpy.minimum and numpy.maximum: they give  
> you element-wise minimum values from two arrays passed as arguments.  
> E.g.:

>  >>> numpy.minimum([1,2,3],[3,2,1])
> array([1, 2, 1])

> The optional third parameter to numpy.minimum is an "out" array - an  
> array to place the results into instead of making a new array for that  
> purpose. (This can save time / memory in various cases.)

Thanks again,
Emmanuelle


From dileepkunjaai at gmail.com  Wed Apr  6 09:14:18 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Wed, 6 Apr 2011 18:44:18 +0530
Subject: [Numpy-discussion] Rounding the decimal part of a real number
Message-ID: <BANLkTikE0uDzMZD7dUb8d1ZZmi8jpGfk3g@mail.gmail.com>

Dear sir,
 Is there any function for rounding the real number, for "n" (say) decimal
places:
Example:
 Let X= 6.9867349234888211237767867863478728314...
but i need only 4 decimal position.
That is the answer should be..

>>>answer=6.9867


-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110406/3f1a694a/attachment.html>

From alan.isaac at gmail.com  Wed Apr  6 09:24:26 2011
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Wed, 06 Apr 2011 09:24:26 -0400
Subject: [Numpy-discussion] Rounding the decimal part of a real number
In-Reply-To: <BANLkTikE0uDzMZD7dUb8d1ZZmi8jpGfk3g@mail.gmail.com>
References: <BANLkTikE0uDzMZD7dUb8d1ZZmi8jpGfk3g@mail.gmail.com>
Message-ID: <4D9C698A.8010504@gmail.com>

On 4/6/2011 9:14 AM, dileep kunjaai wrote:
>   Is there any function for rounding the real number, for "n" (say) decimal places:

http://www.google.com/search?q=numpy+round
produces
http://docs.scipy.org/doc/numpy/reference/generated/numpy.round_.html

Cheers,
Alan Isaac


From Chris.Barker at noaa.gov  Wed Apr  6 11:46:39 2011
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 06 Apr 2011 08:46:39 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTinZfFQDV-eX0bKoUwtMbpDQ_EckPw@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
	<4D9B4768.1060105@noaa.gov>
	<BANLkTimtAD9_wDY6fc0Rq2_kunKKp-G0GQ@mail.gmail.com>
	<BANLkTi=ErZT3Nz=ovhqB35CY7ERRF1a+uQ@mail.gmail.com>
	<4D9BA1D4.2000408@noaa.gov>
	<BANLkTinZfFQDV-eX0bKoUwtMbpDQ_EckPw@mail.gmail.com>
Message-ID: <4D9C8ADF.6060902@noaa.gov>

Sorry to keep harping on this, but for history's sake, I was one of the 
folks that got 'U' introduced in the first place. I was dealing with a 
nightmare of unix, mac and dos test files, 'U' was a  godsend.


On 4/5/11 4:51 PM, Matthew Brett wrote:
> The difference between 'rt' and 'U' is (this is for my own benefit):
>
> For 'rt', a '\r' does not cause a line break - with 'U' - it does.

Perhaps semantics, but what 'U' does is actually change any of the line 
breaks to '\n' -- any line breaking happens after the fact. In Py2, the 
difference between 'U' and 't' is that 't' assumes that any file read 
uses the native line endings -- a bad idea, IMHO. Back in the day, Guido 
argued that text file line ending conversion was the job of file 
transfer tools. The reality, however, is that users don't always use 
file transfer tools correctly, nor even understand the implications of 
line endings.

All that being said,  mac-style files are pretty rare these days. 
(though I bet I've got a few still kicking around)

> Right - my argument is that the behavior implied by 'U' and 't' is
> conceptually separable.   'U' is for how to do line-breaks, and
> line-termination translations, 't' is for whether to decode the text
> or not.  In python 3.

but 't' and 'U' are the same in python 3 -- there is no distinction. It 
seems you are arguing that there could/should be a way to translate line 
termination without decoding the text, but ...

> In python 3 a binary file is a file which is not decoded, and returns
> bytes.  It still has a concept of a 'line', as defined by line
> terminators - you can iterate over one, or do .readlines().

I'll take your word for it that it does, but that's not really a binary 
file then, it's a file that you are assuming is encoded in an 
ascii-compatible way.

While I know that "practicality beats purity", we really should be 
opening the file as a text file (it is text, after all), and specifying 
utf-8 or latin-1 or something as the encoding.

However, IIUC, then the issue here is later on down the line, numpy uses 
regular old C code, which expects ascii strings. In that case, we could 
encode the text as ascii, into a bytes object.

That's a lot of overhead for line ending translation, so probably not 
worth it. But if nothing else, we should be clear in the docs that numpy 
text file reading code is expecting ascii-compatible data.

(and it would be nice to get the line-ending translation)

> Right - so obviously if you open a utf-16 file as binary, terrible
> things may happen - this was what Pauli was pointing out before.  His
> point was that utf-8 is the standard,

but it's not the standard -- it's a common use, but not a standard -- 
ideally numpy wouldn't enforce any particular encoding (though it could 
default to one, and utf-8 would be a good choice for that)

>> Once you really make the distiction between text and binary, the concept
>> of a "line terminator" doesn't really make sense anyway.
>
> Well - I was arguing that, given we can iterate over lines in binary
> files, then there must be the concept of what a line is, in a binary
> file, and that means that we need the concept of a line terminator.

maybe, but that concept is built on a assumption that your file is 
ascii-compatible (for \n anyway), and you know what they say about 
assumptions...

> I realize this is a discussion that would have to happen on the
> python-dev list...

I'm not sure -- I was thinking that python missed something here, but I 
don't think so anymore. In the unicode world, there is not choice but to 
be explicit about encodings, and if you do that, then python's "text or 
binary" distinction makes sense. .readline() for binary file doesn't, 
but so be it.

Honestly, I've never been sure in this discussion what code actually 
needs fixing, so I'm done now -- we've talked enough that the issues 
MUST have been covered by now!

-Chris



From Chris.Barker at noaa.gov  Wed Apr  6 11:56:27 2011
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 06 Apr 2011 08:56:27 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 beta 2
In-Reply-To: <BANLkTinGdHXDeSGc7DB=gCOhvXk_1fzqTw@mail.gmail.com>
References: <BANLkTim8_KR+4x5cz_QCcb+VktQE0wQPGg@mail.gmail.com>
	<8C930E9F-61A0-4425-80BF-8451F348C647@astro.physik.uni-goettingen.de>
	<BANLkTimeKG6UwaOhgVjWdvttxvutLoM6rQ@mail.gmail.com>
	<BANLkTi=DcLH1m_v-PqGZ2v9xrXf0Kb36DQ@mail.gmail.com>
	<4D9AA6FC.1010904@noaa.gov>
	<BANLkTi=8k94uWhp_Ymehm+_YjevekZ+A3w@mail.gmail.com>
	<4D9B4768.1060105@noaa.gov>
	<BANLkTimtAD9_wDY6fc0Rq2_kunKKp-G0GQ@mail.gmail.com>
	<BANLkTi=ErZT3Nz=ovhqB35CY7ERRF1a+uQ@mail.gmail.com>
	<4D9BA1D4.2000408@noaa.gov>
	<BANLkTinZfFQDV-eX0bKoUwtMbpDQ_EckPw@mail.gmail.com>
	<4D9BE4E1.4090607@noaa.gov>
	<BANLkTinGdHXDeSGc7DB=gCOhvXk_1fzqTw@mail.gmail.com>
Message-ID: <4D9C8D2B.2000604@noaa.gov>

On 4/5/11 10:33 PM, Matthew Brett wrote:
> Did you mean to send this just to me?  It seems like the whole is
> generally interesting and helpful, at least to me...

I did mean to send to the list -- I've done that now.

> Well, the current code doesn't split on \r in py3k, admittedly that
> must be a rare case.

I guess that's a key question here -- It really *should* split on /r, 
but maybe it's rare enough to be unimportant.

> The general point about whether to read binary or text is also in
> play.   I agree with you, reading text sounds like a better idea to
> me, but I don't know the performance hit.   Pauli had it as reading
> binary in the original conversion and was defending that in an earlier
> email...

The thing is -- we're talking about parsing text here, so we really are 
reading text files, NOT binary files.

So the question really is -- do we want py3's file reading code to 
handle encoding issues for us, or do we want to handle them ourselves. 
If we only want to support ansi encodings, then handling ourselves may 
well be easier and faster performing. If we go that way we need to 
handle line-endings, too. The easy way is to only support line endings 
with a '\n' in them -- that works out of the box. But it's not that hard 
to support 'r' also, depending on how you want to do it. Before 'U' 
existed, I did that all the time, something like:

some_text = file.read(some_size_buffer)
some_text.replace('\r\n', '\n')
some_text.replace('\r', '\n')
lines = some_text.split('\n')

(by the way, if you're going to support this, it's really nice to 
support mixed line-endings (like this approach does) -- there are a lot 
of editors that can make a mess of line endings.

If you can read the entire file into memory at once, this is almost 
trivial, if you can't -- there is a bit more bookeeping code to be written.

DARN -- I think I said my last note was the last on this topic!

-Chris






-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Chris.Barker at noaa.gov  Wed Apr  6 12:06:00 2011
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 06 Apr 2011 09:06:00 -0700
Subject: [Numpy-discussion] Rounding the decimal part of a real number
In-Reply-To: <4D9C698A.8010504@gmail.com>
References: <BANLkTikE0uDzMZD7dUb8d1ZZmi8jpGfk3g@mail.gmail.com>
	<4D9C698A.8010504@gmail.com>
Message-ID: <4D9C8F68.5080101@noaa.gov>

On 4/6/11 6:24 AM, Alan G Isaac wrote:
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.round_.html

simple enough, of course, but just to be clear:

In [108]: np.round(1.23456789, 3)
Out[108]: 1.2350000000000001

so the number is rounded to the requested number of decimal places, but 
then stored in a binary floating point format, which may not be able to 
exactly represent that rounded number -- hence the "1" at the end. This 
is simply how floating point works.

and that slight difference _probably_ doesn't matter, but it's worth 
being aware of, because is does make a difference occasionally.

python has a decimal type that can work with exact decimal numbers -- 
numpy doesn't support that, as there is no hardware support for it (at 
least on most platforms).

If you want to display it differently, you can use the string formatters:

In [110]: "%.3f"%np.round(1.23456789, 3)
Out[110]: '1.235'

HTH, Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From ralf.gommers at googlemail.com  Wed Apr  6 14:08:48 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Wed, 6 Apr 2011 20:08:48 +0200
Subject: [Numpy-discussion] Installation problem: numpy1.5.1 + python2.7
 + virtualenv-1, 5, 1
In-Reply-To: <201104052318.16262.uvemas@gmail.com>
References: <201104052318.16262.uvemas@gmail.com>
Message-ID: <BANLkTikJB_Vv=JS73rDJj6iWq-1SQA6TAQ@mail.gmail.com>

On Tue, Apr 5, 2011 at 11:17 PM, Vicent Mas <uvemas at gmail.com> wrote:
> Hi,
>
> I'm trying to install numpy1.5.1 in a virtual environment (virtualenv-1.5.1)
> on my debian testing box. I'm using python 2.7 installed from a debian
> package.
>
> This is what I get:
>
> (venv2.7)vmas at rachael$ python setup.py install
> Running from numpy source directory.F2PY Version 1
> blas_opt_info:
> blas_mkl_info:
> ?libraries mkl,vml,guide not found in /usr/lib
> ?NOT AVAILABLE
>
> atlas_blas_threads_info:
> Setting PTATLAS=ATLAS
> Setting PTATLAS=ATLAS
> Setting PTATLAS=ATLAS
> ?FOUND:
> ? ?libraries = ['ptf77blas', 'ptcblas', 'atlas']
> ? ?library_dirs = ['/usr/lib/atlas-base']
> ? ?language = c
> ? ?include_dirs = ['/usr/include/atlas']
>
> ?FOUND:
> ? ?libraries = ['ptf77blas', 'ptcblas', 'atlas']
> ? ?library_dirs = ['/usr/lib/atlas-base']
> ? ?language = c
> ? ?define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]
> ? ?include_dirs = ['/usr/include/atlas']
>
> lapack_opt_info:
> lapack_mkl_info:
> mkl_info:
> ?libraries mkl,vml,guide not found in /usr/lib
> ?NOT AVAILABLE
>
> ?NOT AVAILABLE
>
> atlas_threads_info:
> Setting PTATLAS=ATLAS
> ?libraries lapack not found in /usr/lib/atlas-base
> numpy.distutils.system_info.atlas_threads_info
> Setting PTATLAS=ATLAS
> Setting PTATLAS=ATLAS
> ?FOUND:
> ? ?libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
> ? ?library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
> ? ?language = f77
> ? ?include_dirs = ['/usr/include/atlas']
>
> ?FOUND:
> ? ?libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
> ? ?library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
> ? ?language = f77
> ? ?define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]
> ? ?include_dirs = ['/usr/include/atlas']
>
> running install
> running build
> running config_cc
> unifing config_cc, config, build_clib, build_ext, build commands --compiler
> options
> running config_fc
> unifing config_fc, config, build_clib, build_ext, build commands --fcompiler
> options
> running build_src
> build_src
> building py_modules sources
> creating build
> creating build/src.linux-i686-2.7
> creating build/src.linux-i686-2.7/numpy
> creating build/src.linux-i686-2.7/numpy/distutils
> building library "npymath" sources
> Traceback (most recent call last):
> ?File "setup.py", line 211, in <module>
> ? ?setup_package()
> ?File "setup.py", line 204, in setup_package
> ? ?configuration=configuration )
> ?File
> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/core.py",
> line 186, in setup
> ? ?return old_setup(**new_attr)
> ?File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
> ? ?dist.run_commands()
> ?File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
> ? ?self.run_command(cmd)
> ?File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> ? ?cmd_obj.run()
> ?File
> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/install.py",
> line 55, in run
> ? ?r = old_install.run(self)
> ?File "/usr/lib/python2.7/distutils/command/install.py", line 601, in run
> ? ?self.run_command('build')
> ?File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
> ? ?self.distribution.run_command(command)
> ?File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> ? ?cmd_obj.run()
> ?File
> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build.py",
> line 37, in run
> ? ?old_build.run(self)
> ?File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
> ? ?self.run_command(cmd_name)
> ?File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
> ? ?self.distribution.run_command(command)
> ?File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> ? ?cmd_obj.run()
> ?File
> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py",
> line 152, in run
> ? ?self.build_sources()
> ?File
> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py",
> line 163, in build_sources
> ? ?self.build_library_sources(*libname_info)
> ?File
> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py",
> line 298, in build_library_sources
> ? ?sources = self.generate_sources(sources, (lib_name, build_info))
> ?File
> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py",
> line 385, in generate_sources
> ? ?source = func(extension, build_dir)
> ?File "numpy/core/setup.py", line 681, in get_mathlib_info
> ? ?st = config_cmd.try_link('int main(void) { return 0;}')
> ?File "/usr/lib/python2.7/distutils/command/config.py", line 248, in try_link
> ? ?self._check_compiler()
> ?File
> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/config.py",
> line 45, in _check_compiler
> ? ?old_config._check_compiler(self)
> ?File "/usr/lib/python2.7/distutils/command/config.py", line 103, in
> _check_compiler
> ? ?customize_compiler(self.compiler)
> ?File "/usr/lib/python2.7/distutils/ccompiler.py", line 44, in
> customize_compiler
> ? ?cpp = cc + " -E" ? ? ? ? ? # not always
> TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
>
> Doing the same with python2.6 (also from debian package) works just fine. Could
> somebody tell me what am I doing wrong? Thanks.

Looks like there is something going wrong with your virtualenv setup,
if you look at the backtrace it uses the python in /usr/lib/python2.7/
instead of the one in your virtualenv.

Ralf


From ralf.gommers at googlemail.com  Wed Apr  6 14:19:27 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Wed, 6 Apr 2011 20:19:27 +0200
Subject: [Numpy-discussion] Installation problem: numpy1.5.1 + python2.7
 + virtualenv-1, 5, 1
In-Reply-To: <BANLkTikJB_Vv=JS73rDJj6iWq-1SQA6TAQ@mail.gmail.com>
References: <201104052318.16262.uvemas@gmail.com>
	<BANLkTikJB_Vv=JS73rDJj6iWq-1SQA6TAQ@mail.gmail.com>
Message-ID: <BANLkTi=zy5N=8QfYP2CLb6g3zrCvR-RuRw@mail.gmail.com>

On Wed, Apr 6, 2011 at 8:08 PM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
> On Tue, Apr 5, 2011 at 11:17 PM, Vicent Mas <uvemas at gmail.com> wrote:
>> Hi,
>>
>> I'm trying to install numpy1.5.1 in a virtual environment (virtualenv-1.5.1)
>> on my debian testing box. I'm using python 2.7 installed from a debian
>> package.
>>
>> This is what I get:
>>
>> (venv2.7)vmas at rachael$ python setup.py install
>> Running from numpy source directory.F2PY Version 1
>> blas_opt_info:
>> blas_mkl_info:
>> ?libraries mkl,vml,guide not found in /usr/lib
>> ?NOT AVAILABLE
>>
>> atlas_blas_threads_info:
>> Setting PTATLAS=ATLAS
>> Setting PTATLAS=ATLAS
>> Setting PTATLAS=ATLAS
>> ?FOUND:
>> ? ?libraries = ['ptf77blas', 'ptcblas', 'atlas']
>> ? ?library_dirs = ['/usr/lib/atlas-base']
>> ? ?language = c
>> ? ?include_dirs = ['/usr/include/atlas']
>>
>> ?FOUND:
>> ? ?libraries = ['ptf77blas', 'ptcblas', 'atlas']
>> ? ?library_dirs = ['/usr/lib/atlas-base']
>> ? ?language = c
>> ? ?define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]
>> ? ?include_dirs = ['/usr/include/atlas']
>>
>> lapack_opt_info:
>> lapack_mkl_info:
>> mkl_info:
>> ?libraries mkl,vml,guide not found in /usr/lib
>> ?NOT AVAILABLE
>>
>> ?NOT AVAILABLE
>>
>> atlas_threads_info:
>> Setting PTATLAS=ATLAS
>> ?libraries lapack not found in /usr/lib/atlas-base
>> numpy.distutils.system_info.atlas_threads_info
>> Setting PTATLAS=ATLAS
>> Setting PTATLAS=ATLAS
>> ?FOUND:
>> ? ?libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>> ? ?library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
>> ? ?language = f77
>> ? ?include_dirs = ['/usr/include/atlas']
>>
>> ?FOUND:
>> ? ?libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>> ? ?library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
>> ? ?language = f77
>> ? ?define_macros = [('ATLAS_INFO', '"\\"?.?.?\\""')]
>> ? ?include_dirs = ['/usr/include/atlas']
>>
>> running install
>> running build
>> running config_cc
>> unifing config_cc, config, build_clib, build_ext, build commands --compiler
>> options
>> running config_fc
>> unifing config_fc, config, build_clib, build_ext, build commands --fcompiler
>> options
>> running build_src
>> build_src
>> building py_modules sources
>> creating build
>> creating build/src.linux-i686-2.7
>> creating build/src.linux-i686-2.7/numpy
>> creating build/src.linux-i686-2.7/numpy/distutils
>> building library "npymath" sources
>> Traceback (most recent call last):
>> ?File "setup.py", line 211, in <module>
>> ? ?setup_package()
>> ?File "setup.py", line 204, in setup_package
>> ? ?configuration=configuration )
>> ?File
>> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/core.py",
>> line 186, in setup
>> ? ?return old_setup(**new_attr)
>> ?File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
>> ? ?dist.run_commands()
>> ?File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
>> ? ?self.run_command(cmd)
>> ?File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
>> ? ?cmd_obj.run()
>> ?File
>> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/install.py",
>> line 55, in run
>> ? ?r = old_install.run(self)
>> ?File "/usr/lib/python2.7/distutils/command/install.py", line 601, in run
>> ? ?self.run_command('build')
>> ?File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
>> ? ?self.distribution.run_command(command)
>> ?File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
>> ? ?cmd_obj.run()
>> ?File
>> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build.py",
>> line 37, in run
>> ? ?old_build.run(self)
>> ?File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
>> ? ?self.run_command(cmd_name)
>> ?File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
>> ? ?self.distribution.run_command(command)
>> ?File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
>> ? ?cmd_obj.run()
>> ?File
>> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py",
>> line 152, in run
>> ? ?self.build_sources()
>> ?File
>> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py",
>> line 163, in build_sources
>> ? ?self.build_library_sources(*libname_info)
>> ?File
>> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py",
>> line 298, in build_library_sources
>> ? ?sources = self.generate_sources(sources, (lib_name, build_info))
>> ?File
>> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/build_src.py",
>> line 385, in generate_sources
>> ? ?source = func(extension, build_dir)
>> ?File "numpy/core/setup.py", line 681, in get_mathlib_info
>> ? ?st = config_cmd.try_link('int main(void) { return 0;}')
>> ?File "/usr/lib/python2.7/distutils/command/config.py", line 248, in try_link
>> ? ?self._check_compiler()
>> ?File
>> "/usr/local/src/ViTables_environ/General/numpy-1.5.1/numpy/distutils/command/config.py",
>> line 45, in _check_compiler
>> ? ?old_config._check_compiler(self)
>> ?File "/usr/lib/python2.7/distutils/command/config.py", line 103, in
>> _check_compiler
>> ? ?customize_compiler(self.compiler)
>> ?File "/usr/lib/python2.7/distutils/ccompiler.py", line 44, in
>> customize_compiler
>> ? ?cpp = cc + " -E" ? ? ? ? ? # not always
>> TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
>>
>> Doing the same with python2.6 (also from debian package) works just fine. Could
>> somebody tell me what am I doing wrong? Thanks.
>
> Looks like there is something going wrong with your virtualenv setup,
> if you look at the backtrace it uses the python in /usr/lib/python2.7/
> instead of the one in your virtualenv.

Please ignore that, I was typing before thinking.

Never seen that error before, sorry.
Ralf


From uvemas at gmail.com  Wed Apr  6 15:14:01 2011
From: uvemas at gmail.com (Vicent Mas)
Date: Wed, 6 Apr 2011 21:14:01 +0200
Subject: [Numpy-discussion] Installation problem: numpy1.5.1 + python2.7
	+ virtualenv-1, 5, 1
In-Reply-To: <201104052318.16262.uvemas@gmail.com>
References: <201104052318.16262.uvemas@gmail.com>
Message-ID: <201104062114.02364.uvemas@gmail.com>

On 2011-04-05 Vicent Mas <uvemas at gmail.com> said:

> Hi,
> 
> I'm trying to install numpy1.5.1 in a virtual environment
> (virtualenv-1.5.1) on my debian testing box. I'm using python 2.7
> installed from a debian package.
> 
> [...]
>
> Doing the same with python2.6 (also from debian package) works just fine.
> Could somebody tell me what am I doing wrong? Thanks.
> 

FYI,

it seems that python2.7 packaged by debian is not fully compatible with 
virtualenv-1.5.1. The following changes in /usr/lib/python2.7/sysconfig.py
fix the problem:

vmas at rachael$ diff -u /usr/lib/python2.7/sysconfig.py /tmp/sysconfig.py
--- /usr/lib/python2.7/sysconfig.py     2011-04-06 20:51:00.000000000 +0200
+++ /tmp/sysconfig.py   2011-04-06 20:47:12.000000000 +0200
@@ -294,7 +294,7 @@
 def _get_makefile_filename():
     if _PYTHON_BUILD:
         return os.path.join(_PROJECT_BASE, "Makefile")
-    return os.path.join(get_path('platstdlib').replace("/local","",1), "config" + (sys.pydebug and "_d" or ""), "Makefile")
+    return os.path.join(get_path('platstdlib').replace("/usr/local","/usr",1), "config" + (sys.pydebug and "_d" or ""), "Makefile")
 
 
 def _init_posix(vars):
@@ -394,7 +394,7 @@
         else:
             inc_dir = _PROJECT_BASE
     else:
-        inc_dir = get_path('platinclude').replace("/local","",1)+(sys.pydebug and "_d" or "")
+        inc_dir = get_path('platinclude').replace("/usr/local","/usr",1)+(sys.pydebug and "_d" or "")
     return os.path.join(inc_dir, 'pyconfig.h')
 
 def get_scheme_names():


I suppose these changes can create other problems sooner or later so they are
not a fully satisfactory solution, just a workaround to this specific problem.
If someone has a better solution please, tell me.

Vicent

::

	Share what you know, learn what you don't

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110406/ec1ee22c/attachment.sig>

From bsouthey at gmail.com  Wed Apr  6 22:01:25 2011
From: bsouthey at gmail.com (Bruce Southey)
Date: Wed, 6 Apr 2011 21:01:25 -0500
Subject: [Numpy-discussion] Uninstallation of 1.6.0b2 leaves files behind
 with Python3.x under windows 7
Message-ID: <BANLkTi=4eNw4kVODhA4CBqY=Pu4fOpAnfQ@mail.gmail.com>

Hi,
 I tend to get files left behind for 32-bit versions of Python 3.1 and
Python3.2 usually after I have ran the tests and then immediately try
to uninstall it using the Windows uninstaller via control panel.

With  Python 3.1 files are left behind have the '.pyd' suffix:
C:\Python31\Lib\site-packages\numpy\linalg\lapack_lite.pyd
C:\Python31\Lib\site-packages\numpy\fft\fftpack_lite.pyd

With Python 3.2, most are in the  __pycache__ directories. I do not
see these  __pycache__ directories with Python3.1 binary installer.

Also can people with Windows check that ticket 144 (Uninstall in
Windows does not remove some directories) is still valid?
http://projects.scipy.org/numpy/ticket/1466

I do not see this with the Python 2.x - I think...

Bruce


From cournape at gmail.com  Thu Apr  7 00:33:25 2011
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 7 Apr 2011 13:33:25 +0900
Subject: [Numpy-discussion] Uninstallation of 1.6.0b2 leaves files
 behind with Python3.x under windows 7
In-Reply-To: <BANLkTi=4eNw4kVODhA4CBqY=Pu4fOpAnfQ@mail.gmail.com>
References: <BANLkTi=4eNw4kVODhA4CBqY=Pu4fOpAnfQ@mail.gmail.com>
Message-ID: <BANLkTinfyq1wn1gcMstrmnU0=NMA--h3sQ@mail.gmail.com>

On Thu, Apr 7, 2011 at 11:01 AM, Bruce Southey <bsouthey at gmail.com> wrote:
> Hi,
> ?I tend to get files left behind for 32-bit versions of Python 3.1 and
> Python3.2 usually after I have ran the tests and then immediately try
> to uninstall it using the Windows uninstaller via control panel.
>
> With ?Python 3.1 files are left behind have the '.pyd' suffix:
> C:\Python31\Lib\site-packages\numpy\linalg\lapack_lite.pyd
> C:\Python31\Lib\site-packages\numpy\fft\fftpack_lite.pyd
>
> With Python 3.2, most are in the ?__pycache__ directories. I do not
> see these ?__pycache__ directories with Python3.1 binary installer.

__pycache__ is a feature added in python 3.2 to my knowledge:
http://www.python.org/dev/peps/pep-3147

cheers,

David


From dileepkunjaai at gmail.com  Thu Apr  7 02:36:02 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Thu, 7 Apr 2011 12:06:02 +0530
Subject: [Numpy-discussion] Rounding the decimal part of a real number
In-Reply-To: <4D9C8F68.5080101@noaa.gov>
References: <BANLkTikE0uDzMZD7dUb8d1ZZmi8jpGfk3g@mail.gmail.com>
	<4D9C698A.8010504@gmail.com> <4D9C8F68.5080101@noaa.gov>
Message-ID: <BANLkTinRTKzw-QA67SLpSQQPsNOTjK32vQ@mail.gmail.com>

Thank you sir...........  thank you very much.............





On Wed, Apr 6, 2011 at 9:36 PM, Christopher Barker <Chris.Barker at noaa.gov>wrote:

> On 4/6/11 6:24 AM, Alan G Isaac wrote:
> > http://docs.scipy.org/doc/numpy/reference/generated/numpy.round_.html
>
> simple enough, of course, but just to be clear:
>
> In [108]: np.round(1.23456789, 3)
> Out[108]: 1.2350000000000001
>
> so the number is rounded to the requested number of decimal places, but
> then stored in a binary floating point format, which may not be able to
> exactly represent that rounded number -- hence the "1" at the end. This
> is simply how floating point works.
>
> and that slight difference _probably_ doesn't matter, but it's worth
> being aware of, because is does make a difference occasionally.
>
> python has a decimal type that can work with exact decimal numbers --
> numpy doesn't support that, as there is no hardware support for it (at
> least on most platforms).
>
> If you want to display it differently, you can use the string formatters:
>
> In [110]: "%.3f"%np.round(1.23456789, 3)
> Out[110]: '1.235'
>
> HTH, Chris
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110407/d81b4f79/attachment.html>

From lpc at cmu.edu  Thu Apr  7 16:02:29 2011
From: lpc at cmu.edu (Luis Pedro Coelho)
Date: Thu, 07 Apr 2011 16:02:29 -0400
Subject: [Numpy-discussion] ANN: Mahotas 0.6.4
Message-ID: <1302206549.5358.11.camel@oakeshott>

Hello all,

I'm happy to announce a new release of mahotas, my computer vision
library for Python.

WHAT'S MAHOTAS
--------------

Mahotas is a library which includes several image processing algorithm.
It works over numpy arrays. All its computation heavy functions are
implemented in C++ (using templates so that they work for every data
type without conversions).

The result is a fast, optimised image processing and computer vision
library.

It is under heavy development and has no known bugs. Reported bugs
almost always get fixed in a day.

WHAT'S NEW
----------

Major change is that Mac OS compilation now works! Thanks to K-Michael
Aye for the patch that saved the day.

Minor changes include fixes to ``cwatershed`` and adding the tests to
the source distribution (use `python setup.py test` to run them). Some
of these.

INFO
----
*API Docs*: http://packages.python.org/mahotas/

*Mailing List*: Use the pythonvision mailing list
http://groups.google.com/group/pythonvision?pli=1 for questions, bug
submissions, etc.

*Author*: Luis Pedro Coelho (with code by Zachary Pincus [from
scikits.image], Peter J. Verveer [from scipy.ndimage], and Davis King
[from dlib]

*License*: GPLv2

Thank you,
-- 
Luis Pedro Coelho | Carnegie Mellon University | http://luispedro.org


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110407/9dc739b5/attachment.sig>

From dileepkunjaai at gmail.com  Fri Apr  8 06:54:58 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Fri, 8 Apr 2011 16:24:58 +0530
Subject: [Numpy-discussion] How to use a function into "import function" in
	python
Message-ID: <BANLkTinnSYcPBykXh53nf7qXomd2byUpyg@mail.gmail.com>

Dear sir,
 I defined a function  " hit_rate( )"  i want to use this into "import
function(name of function)".
For example:
   I defined a function as

   * def* hit_function(a,b,c):
     .........................
,...............................
............................
        * return*  value

 Now I want to use this in another function as
*
def *test_hr( ):
    import hit_function(a, b, c)
.......................
,,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,
      *return* value1


Please help me to solve this problem.
                                      thanking  you






-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110408/6ee1a4f4/attachment.html>

From pav at iki.fi  Fri Apr  8 07:29:28 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 8 Apr 2011 11:29:28 +0000 (UTC)
Subject: [Numpy-discussion] How to use a function into "import function"
	in	python
References: <BANLkTinnSYcPBykXh53nf7qXomd2byUpyg@mail.gmail.com>
Message-ID: <inmrin$ju3$1@dough.gmane.org>

Fri, 08 Apr 2011 16:24:58 +0530, dileep kunjaai wrote:
>  I defined a function  " hit_rate( )"  i want to use this into "import
> function(name of function)".

Please read the Python tutorial first:

http://docs.python.org/tutorial/

http://docs.python.org/tutorial/modules.html



From dileepkunjaai at gmail.com  Fri Apr  8 08:21:38 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Fri, 8 Apr 2011 17:51:38 +0530
Subject: [Numpy-discussion] netcdf to grib format
Message-ID: <BANLkTimhpqMbnR_2fW=Dpy21p25cUzeVOw@mail.gmail.com>

Dear sir,
 Is there any tool for changing the  'netcdf' file format to 'grib' format
in python or cdat

-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110408/147b2c13/attachment.html>

From alan.isaac at gmail.com  Fri Apr  8 08:22:19 2011
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 08 Apr 2011 08:22:19 -0400
Subject: [Numpy-discussion] How to use a function into "import function"
 in python
In-Reply-To: <BANLkTinnSYcPBykXh53nf7qXomd2byUpyg@mail.gmail.com>
References: <BANLkTinnSYcPBykXh53nf7qXomd2byUpyg@mail.gmail.com>
Message-ID: <4D9EFDFB.4090704@gmail.com>

On 4/8/2011 6:54 AM, dileep kunjaai wrote:
>   I defined a function  " hit_rate( )"  i want to use this into "import function(name of function)".

http://docs.python.org/tutorial/modules.html

Alan Isaac



From dave.hirschfeld at gmail.com  Fri Apr  8 08:43:56 2011
From: dave.hirschfeld at gmail.com (Dave)
Date: Fri, 8 Apr 2011 12:43:56 +0000 (UTC)
Subject: [Numpy-discussion] Unable to clone git repository
Message-ID: <loom.20110408T142852-311@post.gmane.org>

Disclaimer: I'm on windows (Win7 x64)

Following the instructions at:

http://docs.scipy.org/doc/numpy/dev/gitwash/following_latest.html

I got the following (rather unhelpful) error message:

C:\dev\src>git clone git://github.com/numpy/numpy.git
Cloning into numpy...
github.com[0: 207.97.227.239]: errno=No error
fatal: unable to connect a socket (No error)


After a bit of googling I got the vague impression that this may possibly
be to do with work blocking access? Fortunately I was able clone the read
only https url - https://github.com/numpy/numpy

C:\dev\src>git clone https://github.com/numpy/numpy.git
Cloning into numpy...
remote: Counting objects: 60461, done.
remote: Compressing objects: 100% (13476/13476), done.
remote: Total 60461 (delta 47928), reused 58235 (delta 46157)
Receiving objects: 100% (60461/60461), 19.41 MiB | 2.66 MiB/s, done.
Resolving deltas: 100% (47928/47928), done.

I would suggest that since they are "the instructions if you just want to
follow the latest NumPy source, but you don?t need to do any development"
that providing the https url as an alternative would be helpful. Perhaps
something like:


>From the command line:

    git clone git://github.com/numpy/numpy.git

You should now have a copy of the code tree in the new `numpy` directory.
If this doesn't work you can try the alternative read-only url:

    git clone https://github.com/numpy/numpy.git






From dileepkunjaai at gmail.com  Fri Apr  8 08:53:35 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Fri, 8 Apr 2011 18:23:35 +0530
Subject: [Numpy-discussion] How to use a function into "import function"
 in python
In-Reply-To: <4D9EFDFB.4090704@gmail.com>
References: <BANLkTinnSYcPBykXh53nf7qXomd2byUpyg@mail.gmail.com>
	<4D9EFDFB.4090704@gmail.com>
Message-ID: <BANLkTimjW9LQqNZVkEQA6w_4PNUZ4XHfqg@mail.gmail.com>

Thank you sir ........thank you for your quick and correct answer

On Fri, Apr 8, 2011 at 5:52 PM, Alan G Isaac <alan.isaac at gmail.com> wrote:

> On 4/8/2011 6:54 AM, dileep kunjaai wrote:
> >   I defined a function  " hit_rate( )"  i want to use this into "import
> function(name of function)".
>
> http://docs.python.org/tutorial/modules.html
>
> Alan Isaac
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110408/3854c8ca/attachment.html>

From nadavh at visionsense.com  Fri Apr  8 11:33:32 2011
From: nadavh at visionsense.com (Nadav Horesh)
Date: Fri, 8 Apr 2011 08:33:32 -0700
Subject: [Numpy-discussion] netcdf to grib format
In-Reply-To: <BANLkTimhpqMbnR_2fW=Dpy21p25cUzeVOw@mail.gmail.com>
References: <BANLkTimhpqMbnR_2fW=Dpy21p25cUzeVOw@mail.gmail.com>
Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D5289@VA3DIAXVS361.RED001.local>

Wikipedia has this link

http://www.pyngl.ucar.edu/Nio.shtml

 Does it do the job?

  Nadav

________________________________
From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of dileep kunjaai [dileepkunjaai at gmail.com]
Sent: 08 April 2011 15:21
To: Discussion of Numerical Python
Subject: [Numpy-discussion] netcdf to grib format

Dear sir,
 Is there any tool for changing the  'netcdf' file format to 'grib' format in python or cdat

--
DILEEPKUMAR. R
J R F, IIT DELHI

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110408/f07942de/attachment.html>

From dileepkunjaai at gmail.com  Fri Apr  8 12:31:10 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Fri, 8 Apr 2011 22:01:10 +0530
Subject: [Numpy-discussion] netcdf to grib format
In-Reply-To: <26FC23E7C398A64083C980D16001012D1CE03D5289@VA3DIAXVS361.RED001.local>
References: <BANLkTimhpqMbnR_2fW=Dpy21p25cUzeVOw@mail.gmail.com>
	<26FC23E7C398A64083C980D16001012D1CE03D5289@VA3DIAXVS361.RED001.local>
Message-ID: <BANLkTinGKpUkNL5ued4mMjiX=X-8GpZ+Pw@mail.gmail.com>

Thank you sir ,,,,,thank uuu....very much..................

On Fri, Apr 8, 2011 at 9:03 PM, Nadav Horesh <nadavh at visionsense.com> wrote:

>  Wikipedia has this link
>
> http://www.pyngl.ucar.edu/Nio.shtml
>
>  Does it do the job?
>
>   Nadav
>
>  ------------------------------
> *From:* numpy-discussion-bounces at scipy.org [
> numpy-discussion-bounces at scipy.org] On Behalf Of dileep kunjaai [
> dileepkunjaai at gmail.com]
> *Sent:* 08 April 2011 15:21
> *To:* Discussion of Numerical Python
> *Subject:* [Numpy-discussion] netcdf to grib format
>
>  Dear sir,
>  Is there any tool for changing the  'netcdf' file format to 'grib' format
> in python or cdat
>
> --
> DILEEPKUMAR. R
> J R F, IIT DELHI
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110408/c6d5d55f/attachment.html>

From dbrown at ucar.edu  Fri Apr  8 12:34:36 2011
From: dbrown at ucar.edu (David Brown)
Date: Fri, 8 Apr 2011 10:34:36 -0600
Subject: [Numpy-discussion] netcdf to grib format
In-Reply-To: <BANLkTinGKpUkNL5ued4mMjiX=X-8GpZ+Pw@mail.gmail.com>
References: <BANLkTimhpqMbnR_2fW=Dpy21p25cUzeVOw@mail.gmail.com>
	<26FC23E7C398A64083C980D16001012D1CE03D5289@VA3DIAXVS361.RED001.local>
	<BANLkTinGKpUkNL5ued4mMjiX=X-8GpZ+Pw@mail.gmail.com>
Message-ID: <65C94EAB-610B-412F-B1B5-CFF2FF7797E2@ucar.edu>

Hi Nadav,

I'm sorry to disappoint you but PyNIO has read-only support for GRIB. It can be used to convert GRIB to NetCDF but not the other way around.
 -Dave Brown
 PyNIO developer

On Apr 8, 2011, at 10:31 AM, dileep kunjaai wrote:

> Thank you sir ,,,,,thank uuu....very much..................
> 
> On Fri, Apr 8, 2011 at 9:03 PM, Nadav Horesh <nadavh at visionsense.com> wrote:
> Wikipedia has this link
>  
> http://www.pyngl.ucar.edu/Nio.shtml
>  
>  Does it do the job?
>  
>   Nadav
>  
> From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of dileep kunjaai [dileepkunjaai at gmail.com]
> Sent: 08 April 2011 15:21
> To: Discussion of Numerical Python
> Subject: [Numpy-discussion] netcdf to grib format
> 
> Dear sir,
>  Is there any tool for changing the  'netcdf' file format to 'grib' format in python or cdat
> 
> -- 
> DILEEPKUMAR. R
> J R F, IIT DELHI
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> 
> 
> -- 
> DILEEPKUMAR. R
> J R F, IIT DELHI
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110408/19d1adf3/attachment.html>

From dileepkunjaai at gmail.com  Fri Apr  8 12:44:53 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Fri, 8 Apr 2011 22:14:53 +0530
Subject: [Numpy-discussion] netcdf to grib format
In-Reply-To: <65C94EAB-610B-412F-B1B5-CFF2FF7797E2@ucar.edu>
References: <BANLkTimhpqMbnR_2fW=Dpy21p25cUzeVOw@mail.gmail.com>
	<26FC23E7C398A64083C980D16001012D1CE03D5289@VA3DIAXVS361.RED001.local>
	<BANLkTinGKpUkNL5ued4mMjiX=X-8GpZ+Pw@mail.gmail.com>
	<65C94EAB-610B-412F-B1B5-CFF2FF7797E2@ucar.edu>
Message-ID: <BANLkTikrMRf_n35CsJgk3WFTi7mC=WAoMg@mail.gmail.com>

No sir ..... I got another method for converting this,,, we can use
CDO(climate data operators) this tool is help ful

On Fri, Apr 8, 2011 at 10:04 PM, David Brown <dbrown at ucar.edu> wrote:

> Hi Nadav,
>
> I'm sorry to disappoint you but PyNIO has read-only support for GRIB. It
> can be used to convert GRIB to NetCDF but not the other way around.
>  -Dave Brown
>  PyNIO developer
>
> On Apr 8, 2011, at 10:31 AM, dileep kunjaai wrote:
>
> Thank you sir ,,,,,thank uuu....very much..................
>
> On Fri, Apr 8, 2011 at 9:03 PM, Nadav Horesh <nadavh at visionsense.com>wrote:
>
>>  Wikipedia has this link
>>
>> http://www.pyngl.ucar.edu/Nio.shtml
>>
>>  Does it do the job?
>>
>>   Nadav
>>
>>  ------------------------------
>> *From:* numpy-discussion-bounces at scipy.org [
>> numpy-discussion-bounces at scipy.org] On Behalf Of dileep kunjaai [
>> dileepkunjaai at gmail.com]
>> *Sent:* 08 April 2011 15:21
>> *To:* Discussion of Numerical Python
>> *Subject:* [Numpy-discussion] netcdf to grib format
>>
>>  Dear sir,
>>  Is there any tool for changing the  'netcdf' file format to 'grib' format
>> in python or cdat
>>
>> --
>> DILEEPKUMAR. R
>> J R F, IIT DELHI
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
>
> --
> DILEEPKUMAR. R
> J R F, IIT DELHI
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110408/e34cada9/attachment.html>

From rblove_lists at comcast.net  Fri Apr  8 22:23:16 2011
From: rblove_lists at comcast.net (Robert Love)
Date: Fri, 8 Apr 2011 21:23:16 -0500
Subject: [Numpy-discussion] Arrays as nested dtype
Message-ID: <B3128A29-CF8F-4A12-900F-D8EE0287F082@comcast.net>


Using np.loadtxt I can easily read my file that has columns of time, mode, 3 float64 for position and 3 for velocity like this.

dt = dtype([('time', '|S12'), ('mode','|S3'),('rx','f8'),('ry','f8'),('rz','f8'),('vx','f8'),('vy','f8'),('vz','f8')])

data = np.loadtxt('file', dtype=dt)


I can then put the two pairs of 3 components into np.arrays and start performing the vector operations I need.

How can I read them directly into np.arrays?


dt = dtype([('time', '|S12'), ('mode','|S3'),np.array('r'), np.array('v')])

I've seen examples for nested data that create a tuple but not an array.  Any tips  appreciated.

From warren.weckesser at enthought.com  Fri Apr  8 22:52:37 2011
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Fri, 8 Apr 2011 21:52:37 -0500
Subject: [Numpy-discussion] Arrays as nested dtype
In-Reply-To: <B3128A29-CF8F-4A12-900F-D8EE0287F082@comcast.net>
References: <B3128A29-CF8F-4A12-900F-D8EE0287F082@comcast.net>
Message-ID: <BANLkTiku_xPvxn2bBA62yYC7-YfQSXYUMQ@mail.gmail.com>

On Fri, Apr 8, 2011 at 9:23 PM, Robert Love <rblove_lists at comcast.net>wrote:

>
> Using np.loadtxt I can easily read my file that has columns of time, mode,
> 3 float64 for position and 3 for velocity like this.
>
> dt = dtype([('time', '|S12'),
> ('mode','|S3'),('rx','f8'),('ry','f8'),('rz','f8'),('vx','f8'),('vy','f8'),('vz','f8')])
>
> data = np.loadtxt('file', dtype=dt)
>
>
> I can then put the two pairs of 3 components into np.arrays and start
> performing the vector operations I need.
>
> How can I read them directly into np.arrays?
>
>
> dt = dtype([('time', '|S12'), ('mode','|S3'),np.array('r'), np.array('v')])
>
> I've seen examples for nested data that create a tuple but not an array.
>  Any tips  appreciated.
>


If you do this:

>>> dt = dtype([('time', '|S12'), ('mode','|S3'), ('r','f8', 3),
('v','f8',3)])
>>> data = np.loadtxt('file', dtype=dt)

then data['r'] and data['v'] are the arrays of positions and velocities.
You can then give them more convenient names:

>>> r = data['r']
>>> v = data['v']


Warren


> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110408/9727ab2e/attachment.html>

From fperez.net at gmail.com  Sat Apr  9 04:05:26 2011
From: fperez.net at gmail.com (Fernando Perez)
Date: Sat, 9 Apr 2011 01:05:26 -0700
Subject: [Numpy-discussion] [ANN] IPython 0.10.2 is out
Message-ID: <BANLkTikP5-fzP+vG_p8QfhAWG7dPgF47qA@mail.gmail.com>

Hi all,

we've just released IPython 0.10.2, full release notes are below.

Downloads in source and windows binary form are available in the usual location:
http://ipython.scipy.org/dist/

as well as on github:
http://github.com/ipython/ipython/archives/rel-0.10.2

and at the Python Package Index (which easy_install will use by default):
http://pypi.python.org/pypi/ipython

so at any time you should find a location with good download speeds.

You can find the full documentation at:
http://ipython.github.com/ipython-doc/rel-0.10.2/html/
http://ipython.github.com/ipython-doc/rel-0.10.2/ipython.pdf


Enjoy!

Fernando (on behalf of the whole IPython team)


Release 0.10.2
==============

IPython 0.10.2 was released April 9, 2011.  This is a minor bugfix release that
preserves backward compatibility.  At this point, all IPython development
resources are focused on the 0.11 series that includes a complete architectural
restructuring of the project as well as many new capabilities, so this is
likely to be the last release of the 0.10.x series.  We have tried to fix all
major bugs in this series so that it remains a viable platform for those not
ready yet to transition to the 0.11 and newer codebase (since that will require
some porting effort, as a number of APIs have changed).

Thus, we are not opening a 0.10.3 active development branch yet, but if the
user community requires new patches and is willing to maintain/release such a
branch, we'll be happy to host it on the IPython github repositories.

Highlights of this release:

- The main one is the closing of github ticket #185, a major regression we had
  in 0.10.1 where pylab mode with GTK (or gthread) was not working correctly,
  hence plots were blocking with GTK.  Since this is the default matplotlib
  backend on Unix systems, this was a major annoyance for many users.  Many
  thanks to Paul Ivanov for helping resolve this issue.

- Fix IOError bug on Windows when used with -gthread.
- Work robustly if $HOME is missing from environment.
- Better POSIX support in ssh scripts (remove bash-specific idioms).
- Improved support for non-ascii characters in log files.
- Work correctly in environments where GTK can be imported but not started
  (such as a linux text console without X11).

For this release we merged 24 commits, contributed by the following people
(please let us know if we ommitted your name and we'll gladly fix this in the
notes for the future):

* Fernando Perez
* MinRK
* Paul Ivanov
* Pieter Cristiaan de Groot
* TvrtkoM


From ralf.gommers at googlemail.com  Sat Apr  9 17:52:53 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sat, 9 Apr 2011 23:52:53 +0200
Subject: [Numpy-discussion] Uninstallation of 1.6.0b2 leaves files
 behind with Python3.x under windows 7
In-Reply-To: <BANLkTinfyq1wn1gcMstrmnU0=NMA--h3sQ@mail.gmail.com>
References: <BANLkTi=4eNw4kVODhA4CBqY=Pu4fOpAnfQ@mail.gmail.com>
	<BANLkTinfyq1wn1gcMstrmnU0=NMA--h3sQ@mail.gmail.com>
Message-ID: <BANLkTi=Vhrywye4DjNkU9uxn0=pQUtgjRQ@mail.gmail.com>

On Thu, Apr 7, 2011 at 6:33 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Thu, Apr 7, 2011 at 11:01 AM, Bruce Southey <bsouthey at gmail.com> wrote:
>> Hi,
>> ?I tend to get files left behind for 32-bit versions of Python 3.1 and
>> Python3.2 usually after I have ran the tests and then immediately try
>> to uninstall it using the Windows uninstaller via control panel.
>>
>> With ?Python 3.1 files are left behind have the '.pyd' suffix:
>> C:\Python31\Lib\site-packages\numpy\linalg\lapack_lite.pyd
>> C:\Python31\Lib\site-packages\numpy\fft\fftpack_lite.pyd
>>
>> With Python 3.2, most are in the ?__pycache__ directories. I do not
>> see these ?__pycache__ directories with Python3.1 binary installer.
>
> __pycache__ is a feature added in python 3.2 to my knowledge:
> http://www.python.org/dev/peps/pep-3147

This looks like a bug in distribute. I find the same .pyc files right
beside the source .py files as in __pycache__/filename.cpython-32.pyc.
I'm guessing the former are created during the byte-compiling step at
install time while the latter are created on the first import. Only
the latter should be present according to PEP 3147.

>> Also can people with Windows check that ticket 144 (Uninstall in
>> Windows does not remove some directories) is still valid?
>> http://projects.scipy.org/numpy/ticket/1466

The numpy NSIS script does not contain any uninstall info. The default
uninstall mode seems to be to create a log of files that was
installed, and deleting those when the uninstaller is run. This causes
the above __pycache__ issue as well as #1466 (the user ran coverage.py
it seems, has left-over .cover files that are not cleaned up). I don't
have a Windows machine to test, but #1466 must still be valid.

The alternative would be to tell the uninstaller to delete the
complete site-packages/numpy/ dir no matter what's in it. That's
probably the right way to do it.

Ralf


From sturla at molden.no  Sun Apr 10 19:20:39 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 11 Apr 2011 01:20:39 +0200
Subject: [Numpy-discussion] Shared memory ndarrays (update)
Message-ID: <4DA23B47.10301@molden.no>

Here is an update for the shared memory arrays that Ga?l and I wrote two 
years ago. They are NumPy arrays referencing shared memory, and IPC 
using multiprocessing.Queue is possible by monkey patching how ndarrays 
are pickled.

Usage:

     import numpy as np
     import sharedmem as sm

     shared_array = sm.zeros(n)

I.e. the only difference from ndarrays is that pickle.dumps and 
multiprocessing.Queue do not make a copy of the buffer, and that 
allocated memory is shared between professes (e.g. created with os.fork, 
subprocess or multiprocessing.)

A named memory map of the paging file is used on Windows. Unix System V 
IPC is used on Linux/Unix (thanks to Philip Semanchuk for assistance).

Changes:

- 64-bit support.
- Memory leak on Linux/Unix should be gone (monkey patch for os._exit).
- Added a global lock as there are callbacks to Python (the GIL is not 
sufficient serialization).

I need help with testing, particularly on Linux / Apple and with the 
most recent NumPy.

I'm an idiot with build tools, hence no setup.py. Invoke Cython and then 
cc. Compile sharedmemory_sysv.pyx for Linux/Unix or 
sharedmemory_sysv.pyx and ntqueryobject.c for Windows.


Regards,
Sturla
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sharedmem.zip
Type: application/x-zip-compressed
Size: 8794 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/a6b16acb/attachment.bin>

From sturla at molden.no  Sun Apr 10 19:24:16 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 11 Apr 2011 01:24:16 +0200
Subject: [Numpy-discussion] Shared memory ndarrays (update)
In-Reply-To: <4DA23B47.10301@molden.no>
References: <4DA23B47.10301@molden.no>
Message-ID: <4DA23C20.6060605@molden.no>

Den 11.04.2011 01:20, skrev Sturla Molden:
>
> I'm an idiot with build tools, hence no setup.py. Invoke Cython and 
> then cc. Compile sharedmemory_sysv.pyx for Linux/Unix or 
> sharedmemory_sysv.pyx and ntqueryobject.c for Windows.

Eh, that is sharedmemory_win.pyx and ntqueryobject.c for Windows :-)

Sturla


From david.crisp at gmail.com  Sun Apr 10 20:01:49 2011
From: david.crisp at gmail.com (David Crisp)
Date: Mon, 11 Apr 2011 10:01:49 +1000
Subject: [Numpy-discussion] Classes as records within an numpy array
Message-ID: <BANLkTikUTK1J=BYvgCYu+x2Mm1RNZs++7w@mail.gmail.com>

I am trying to make a class a record within a numpy array.

The piece of code I am working on has the following class within it:

class Record:
       def __init__(self):
               self.count = 0
               self.level = 0
               self.list = []
               self.max = None
               self.min = None
       def add(self, value):
               self.count += 1
               self.level += value
               if self.max is None:
                       self.max = value
               else:
                       self.max = max(value, self.max)

               if self.min is None:
                       self.min = value
               else:
                       self.min = min(value, self.min)
               self.list.append(value)

when I use this class manually it works as I would expect and does
what I want  eg:

x = Record()

x.add(42)
x.add(22)
x.add(342)
x.add(44)
x.add(12)
x.add(2)

for eachitem in x.list:
       print eachitem
42
22
342
44
12
2

It's Okay to here.

When I create the following code:

vegetation_matrix = numpy.empty(shape=(x_bins+1,y_bins+1))

for each_line in point_scandata:
       matrix_x = round_to_value((each_line[0]-x_offset),horizontal_precision)
       matrix_y = round_to_value((each_line[1]-y_offset),horizontal_precision)
       vegetation_matrix[matrix_x][matrix_y] = Record()

I get an error:

Traceback (most recent call last):
 File "MatrixTo3bynArray.py", line 159, in <module>
   vegetation_matrix[matrix_x][matrix_y] = Record()
SystemError: error return without exception set

I would ask the universal what am I doing wrong,  but how about I ask
in this particular situation, what am I doing wrong?

Can anybody guide me through this problem?

Regards,
David


From sturla at molden.no  Sun Apr 10 21:00:48 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 11 Apr 2011 03:00:48 +0200
Subject: [Numpy-discussion] Classes as records within an numpy array
In-Reply-To: <BANLkTikUTK1J=BYvgCYu+x2Mm1RNZs++7w@mail.gmail.com>
References: <BANLkTikUTK1J=BYvgCYu+x2Mm1RNZs++7w@mail.gmail.com>
Message-ID: <4DA252C0.4010907@molden.no>

Den 11.04.2011 02:01, skrev David Crisp:
> Can anybody guide me through this problem?
>

You must use dtype=object for the array vegetation_matrix.

You also index vegetation_matrix wrong, use x[i,j] not x[i][j].

Sturla



From david.crisp at gmail.com  Sun Apr 10 21:59:00 2011
From: david.crisp at gmail.com (David Crisp)
Date: Mon, 11 Apr 2011 11:59:00 +1000
Subject: [Numpy-discussion] Classes as records within an numpy array
In-Reply-To: <4DA252C0.4010907@molden.no>
References: <BANLkTikUTK1J=BYvgCYu+x2Mm1RNZs++7w@mail.gmail.com>
	<4DA252C0.4010907@molden.no>
Message-ID: <BANLkTin6N=YEVOWhihaUansw_07H_PgcXQ@mail.gmail.com>

On Mon, Apr 11, 2011 at 11:00 AM, Sturla Molden <sturla at molden.no> wrote:
> Den 11.04.2011 02:01, skrev David Crisp:
>> Can anybody guide me through this problem?
>>
>
> You must use dtype=object for the array vegetation_matrix.

Thank you.  This seemed to remove the error I was having.

> You also index vegetation_matrix wrong, use x[i,j] not x[i][j].

Is it a significant problem if I use the x[i][j] method and not the
x[i,j] method?  Or are we talking about trying to stick with a
convention?

Regards,
David


From sturla at molden.no  Sun Apr 10 22:17:04 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 11 Apr 2011 04:17:04 +0200
Subject: [Numpy-discussion] Classes as records within an numpy array
In-Reply-To: <BANLkTin6N=YEVOWhihaUansw_07H_PgcXQ@mail.gmail.com>
References: <BANLkTikUTK1J=BYvgCYu+x2Mm1RNZs++7w@mail.gmail.com>	<4DA252C0.4010907@molden.no>
	<BANLkTin6N=YEVOWhihaUansw_07H_PgcXQ@mail.gmail.com>
Message-ID: <4DA264A0.2030708@molden.no>

Den 11.04.2011 03:59, skrev David Crisp:
>
> Is it a significant problem if I use the x[i][j] method and not the
> x[i,j] method?  Or are we talking about trying to stick with a
> convention?

It is a significant problem.

Consider that x[i][j] means ( x[i] )[j]. Then consider that x[i] in your 
case returns an instance of Record, not ndarray.

If you want 2D indexing on an ndarray, it should be written as x[i,j]. A 
2D NumPY array is not an array of arrays.

If you work with nested Python lists, you should index x[i][j], because 
x[i] will return a list.

Sturla





From sturla at molden.no  Sun Apr 10 23:02:33 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 11 Apr 2011 05:02:33 +0200
Subject: [Numpy-discussion] Classes as records within an numpy array
In-Reply-To: <4DA264A0.2030708@molden.no>
References: <BANLkTikUTK1J=BYvgCYu+x2Mm1RNZs++7w@mail.gmail.com>	<4DA252C0.4010907@molden.no>	<BANLkTin6N=YEVOWhihaUansw_07H_PgcXQ@mail.gmail.com>
	<4DA264A0.2030708@molden.no>
Message-ID: <4DA26F49.1050301@molden.no>

Den 11.04.2011 04:17, skrev Sturla Molden:
>
> Consider that x[i][j] means ( x[i] )[j]. Then consider that x[i] in your
> case returns an instance of Record, not ndarray.

Sorry, disregard this.

a[i] is the same as a[i,:], which means that a[i,j] and a[i][j] will 
return the same.

But a[i,j] will be faster as you avoid a function call and avoid having 
to construct a view array for row i.

It's getting late :(


Sturla


From david.crisp at gmail.com  Sun Apr 10 23:17:25 2011
From: david.crisp at gmail.com (David Crisp)
Date: Mon, 11 Apr 2011 13:17:25 +1000
Subject: [Numpy-discussion] Classes as records within an numpy array
In-Reply-To: <4DA252C0.4010907@molden.no>
References: <BANLkTikUTK1J=BYvgCYu+x2Mm1RNZs++7w@mail.gmail.com>
	<4DA252C0.4010907@molden.no>
Message-ID: <BANLkTinMsDHsjcXCWXFasPi_WXWueX-q-g@mail.gmail.com>

On Mon, Apr 11, 2011 at 11:00 AM, Sturla Molden <sturla at molden.no> wrote:
> Den 11.04.2011 02:01, skrev David Crisp:
>> Can anybody guide me through this problem?
>>
>
> You must use dtype=object for the array vegetation_matrix.

I changed the line which set up the vegetation_matrix to be the following:

vegetation_matrix = numpy.empty(shape=(x_bins+1,y_bins+1,),dtype=object)

Then I spotted the next problem.

In my loop which goes through the raw data and populates the matrix
with the required values,  I am constantly resetting the Record() to a
new record.

for each_line in point_scandata:
	matrix_x = round_to_value((each_line[0]-x_offset),horizontal_precision)
	matrix_y = round_to_value((each_line[1]-y_offset),horizontal_precision)
--> 	vegetation_matrix[matrix_x,matrix_y] = Record() # This is wrong:
Everytime its called it resets the record back to default new..
	vegetation_matrix[matrix_x,matrix_y].add(each_line[2]

This means that there is always only 1 entry in the record as its
always being reset after each iteration.

How do I go about setting up the numpy array such that its default
record is Record()

(Im sure this IS a brain fade as It seems really familliar)

Regards,
David


From david.crisp at gmail.com  Sun Apr 10 23:40:27 2011
From: david.crisp at gmail.com (David Crisp)
Date: Mon, 11 Apr 2011 13:40:27 +1000
Subject: [Numpy-discussion] Classes as records within an numpy array
In-Reply-To: <BANLkTinMsDHsjcXCWXFasPi_WXWueX-q-g@mail.gmail.com>
References: <BANLkTikUTK1J=BYvgCYu+x2Mm1RNZs++7w@mail.gmail.com>
	<4DA252C0.4010907@molden.no>
	<BANLkTinMsDHsjcXCWXFasPi_WXWueX-q-g@mail.gmail.com>
Message-ID: <BANLkTik9iyjmnGs+=G--e=Dzno2H1YxAdQ@mail.gmail.com>

On Mon, Apr 11, 2011 at 1:17 PM, David Crisp <david.crisp at gmail.com> wrote:
> On Mon, Apr 11, 2011 at 11:00 AM, Sturla Molden <sturla at molden.no> wrote:
>> Den 11.04.2011 02:01, skrev David Crisp:
>>> Can anybody guide me through this problem?

I dont know how acceptable it is to answer your own question :P   here goes:

TO some extent I have solved my problem,  But I am unsure if it is the
correct way of solving it:

for each_line in point_scandata:
	matrix_x = round_to_value((each_line[0]-x_offset),horizontal_precision)
	matrix_y = round_to_value((each_line[1]-y_offset),horizontal_precision)
	if vegetation_matrix[matrix_x,matrix_y] == None:
		vegetation_matrix[matrix_x,matrix_y] = Record()
	vegetation_matrix[matrix_x,matrix_y].add(each_line[2])

I added the IF check.  If vegetation_matrix[matrix_x,matrix_y] is
still equal to None then that means the record for that grid cell has
not been created yet.
So if it IS none then simply create a record.

If vegetation_matrix[matrix_x,matrix_y] is NOT None then the only
other value it could be is Record (Major Assumption that nobody else
has fiddled with the code)  and so simply add a new value to the
record with the appropriate value.

It works but is it the correct way of doing things...

Regards,
David


From gael.varoquaux at normalesup.org  Mon Apr 11 01:05:23 2011
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 11 Apr 2011 07:05:23 +0200
Subject: [Numpy-discussion] Shared memory ndarrays (update)
In-Reply-To: <4DA23B47.10301@molden.no>
References: <4DA23B47.10301@molden.no>
Message-ID: <20110411050523.GA30988@phare.normalesup.org>

Hey Sturla,

It's really great that you are still working on that. I'll test the code
under Linux.

The scipy community has moved to github. If I create a repository under
github and put the code on it, would you use it? If I find time, I'll add
a setup.py.

Ga?l



From seb.haase at gmail.com  Mon Apr 11 03:21:23 2011
From: seb.haase at gmail.com (Sebastian Haase)
Date: Mon, 11 Apr 2011 09:21:23 +0200
Subject: [Numpy-discussion] Shared memory ndarrays (update)
In-Reply-To: <20110411050523.GA30988@phare.normalesup.org>
References: <4DA23B47.10301@molden.no>
	<20110411050523.GA30988@phare.normalesup.org>
Message-ID: <BANLkTinUr3GQLW680p=+_eZVr2k5xitNfg@mail.gmail.com>

On Mon, Apr 11, 2011 at 7:05 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> Hey Sturla,
>
> It's really great that you are still working on that. I'll test the code
> under Linux.
>
> The scipy community has moved to github. If I create a repository under
> github and put the code on it, would you use it? If I find time, I'll add
> a setup.py.
>
> Ga?l
>
Hi,

just wanted to say that I find this module, and shared memory in
general very interesting, because I work with very large image data
sets.

I have a non python question:
for Java there seems to exist a module/package/class called nio

http://download.oracle.com/javase/1.4.2/docs/api/java/nio/MappedByteBuffer.html
   public abstract class MappedByteBuffer    extends ByteBuffer
   A direct byte buffer whose content is a memory-mapped region of a file.

Could this be used to share memory between   Java applications  and  numpy ?

Just hoping that someone here knows Java much better than me.
- Sebastian Haase


From zbyszek at in.waw.pl  Mon Apr 11 08:58:42 2011
From: zbyszek at in.waw.pl (=?UTF-8?B?WmJpZ25pZXcgSsSZZHJ6ZWpld3NraS1Tem1law==?=)
Date: Mon, 11 Apr 2011 14:58:42 +0200
Subject: [Numpy-discussion] Shared memory ndarrays (update)
In-Reply-To: <BANLkTinUr3GQLW680p=+_eZVr2k5xitNfg@mail.gmail.com>
References: <4DA23B47.10301@molden.no>	<20110411050523.GA30988@phare.normalesup.org>
	<BANLkTinUr3GQLW680p=+_eZVr2k5xitNfg@mail.gmail.com>
Message-ID: <4DA2FB02.7050308@in.waw.pl>

On 04/11/2011 09:21 AM, Sebastian Haase wrote:

> I have a non python question:
> for Java there seems to exist a module/package/class called nio
> 
> http://download.oracle.com/javase/1.4.2/docs/api/java/nio/MappedByteBuffer.html
>    public abstract class MappedByteBuffer    extends ByteBuffer
>    A direct byte buffer whose content is a memory-mapped region of a file.
> 
> Could this be used to share memory between   Java applications  and  numpy ?
Hi,
it could, but you'd have to do the parsing of data yourself. So nothing
fancy unless you want to reimplement numpy in Java :). But if you
use a mmaped file as a backing for a numpy array of one of the types
also available in Java (signed byte, short, int, long, float, double)
and then map it on the Java side with DoubleBuffer or FloatBuffer or
..., then it seems straightforward enough.

Zbyszek


From zbyszek at in.waw.pl  Mon Apr 11 09:17:28 2011
From: zbyszek at in.waw.pl (=?UTF-8?B?WmJpZ25pZXcgSsSZZHJ6ZWpld3NraS1Tem1law==?=)
Date: Mon, 11 Apr 2011 15:17:28 +0200
Subject: [Numpy-discussion] Classes as records within an numpy array
In-Reply-To: <BANLkTik9iyjmnGs+=G--e=Dzno2H1YxAdQ@mail.gmail.com>
References: <BANLkTikUTK1J=BYvgCYu+x2Mm1RNZs++7w@mail.gmail.com>	<4DA252C0.4010907@molden.no>	<BANLkTinMsDHsjcXCWXFasPi_WXWueX-q-g@mail.gmail.com>
	<BANLkTik9iyjmnGs+=G--e=Dzno2H1YxAdQ@mail.gmail.com>
Message-ID: <4DA2FF68.70001@in.waw.pl>

On 04/11/2011 05:40 AM, David Crisp wrote:
> On Mon, Apr 11, 2011 at 1:17 PM, David Crisp <david.crisp at gmail.com> wrote:
>> On Mon, Apr 11, 2011 at 11:00 AM, Sturla Molden <sturla at molden.no> wrote:
>>> Den 11.04.2011 02:01, skrev David Crisp:
>>>> Can anybody guide me through this problem?
> 
> I dont know how acceptable it is to answer your own question :P   here goes:
> 
> TO some extent I have solved my problem,  But I am unsure if it is the
> correct way of solving it:
> 
> for each_line in point_scandata:
> 	matrix_x = round_to_value((each_line[0]-x_offset),horizontal_precision)
> 	matrix_y = round_to_value((each_line[1]-y_offset),horizontal_precision)   <---- vertical_precision?
> 	if vegetation_matrix[matrix_x,matrix_y] == None:
> 		vegetation_matrix[matrix_x,matrix_y] = Record()
> 	vegetation_matrix[matrix_x,matrix_y].add(each_line[2])
Hi,
you should initialize the matrix when creating it. What about
  vegeration_matrix = numpy.array([[Record() for _ in xrange(x_bins+1)] for _ in xrange(y_bins+1)])
Then you could avoid the ugly if.

A more general question: why do you want to use numpy object arrays at all?
You are not using any array operatations with your Record objects. The numpy array
of objects is equivalent to a two-dimensional list. You could use a normal Python
list just as well:
vegetation_matrix = [[Record() for _ in xrange(x_bins+1)] for _ in xrange(y_bins+1)]
vegetation_matrix[4][3].add(11)

Best,
Zbyszek


From srean.list at gmail.com  Mon Apr 11 13:42:10 2011
From: srean.list at gmail.com (srean)
Date: Mon, 11 Apr 2011 12:42:10 -0500
Subject: [Numpy-discussion] Shared memory ndarrays (update)
In-Reply-To: <4DA2FB02.7050308@in.waw.pl>
References: <4DA23B47.10301@molden.no>
	<20110411050523.GA30988@phare.normalesup.org>
	<BANLkTinUr3GQLW680p=+_eZVr2k5xitNfg@mail.gmail.com>
	<4DA2FB02.7050308@in.waw.pl>
Message-ID: <BANLkTimY9GxsgXsaZXP+XisSpRHWR99piw@mail.gmail.com>

Hi everyone,

  I was looking up the options that are available for shared memory arrays
and this thread came up at the right time. The doc says that multiprocessing
.Array(...) gives a shared memory array. But from the code it seems to me
that it is actually using a mmap. Is that correct a correct assessment, and
if so, is there any advantage in using multiprocessing.Array(...) over
simple numpy mmaped arrays.

Regards
  srean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/fa534227/attachment.html>

From srean.list at gmail.com  Mon Apr 11 14:11:39 2011
From: srean.list at gmail.com (srean)
Date: Mon, 11 Apr 2011 13:11:39 -0500
Subject: [Numpy-discussion] Shared memory ndarrays (update)
In-Reply-To: <BANLkTimY9GxsgXsaZXP+XisSpRHWR99piw@mail.gmail.com>
References: <4DA23B47.10301@molden.no>
	<20110411050523.GA30988@phare.normalesup.org>
	<BANLkTinUr3GQLW680p=+_eZVr2k5xitNfg@mail.gmail.com>
	<4DA2FB02.7050308@in.waw.pl>
	<BANLkTimY9GxsgXsaZXP+XisSpRHWR99piw@mail.gmail.com>
Message-ID: <BANLkTimZ6j7r8BgxWHxoNa5jeUx6SP1jGw@mail.gmail.com>

Apologies for adding to my own post. multiprocessing.Array(...) uses an
anonymous mmapped file. I am not sure if that means it is resident on RAM or
the swap device. But my original question remains, what are the pros and
cons of using it versus numpy mmapped arrays. If  multiprocessing.Array is
indeed resident in memory (subject to swapping of course) that would still
be advatageous compared to a file mapped from a on-disk filesystem.

On Mon, Apr 11, 2011 at 12:42 PM, srean <srean.list at gmail.com> wrote:

> Hi everyone,
>
>   I was looking up the options that are available for shared memory arrays
> and this thread came up at the right time. The doc says thatmultiprocessing
> .Array(...) gives a shared memory array. But from the code it seems to me
> that it is actually using a mmap. Is that correct a correct assessment, and
> if so, is there any advantage in using multiprocessing.Array(...) over
> simple numpy mmaped arrays.
>
> Regards
>   srean
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/d1cba104/attachment.html>

From sturla at molden.no  Mon Apr 11 14:36:43 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 11 Apr 2011 20:36:43 +0200
Subject: [Numpy-discussion] Shared memory ndarrays (update)
In-Reply-To: <BANLkTimZ6j7r8BgxWHxoNa5jeUx6SP1jGw@mail.gmail.com>
References: <4DA23B47.10301@molden.no>	<20110411050523.GA30988@phare.normalesup.org>	<BANLkTinUr3GQLW680p=+_eZVr2k5xitNfg@mail.gmail.com>	<4DA2FB02.7050308@in.waw.pl>	<BANLkTimY9GxsgXsaZXP+XisSpRHWR99piw@mail.gmail.com>
	<BANLkTimZ6j7r8BgxWHxoNa5jeUx6SP1jGw@mail.gmail.com>
Message-ID: <4DA34A3B.9080105@molden.no>

"Shared memory" is memory mapping from the paging file (i.e. RAM), not a 
file on disk. They can have a name or be anonymous. I have explained why 
we need named shared memory before. If you didn't understand it, try to 
pass an instance of |multiprocessing.Array over | |multiprocessing.Queue.

|Sturla


Den 11.04.2011 20:11, skrev srean:
> Apologies for adding to my own post. |multiprocessing.Array(...) uses 
> an anonymous mmapped file. I am not sure if that means it is resident 
> on RAM or the swap device. But my original question remains, what are 
> the pros and cons of using it versus numpy mmapped arrays. If 
> ||multiprocessing.Arrayis indeed resident in memory (subject to 
> swapping of course) that would still be advatageous compared to a file 
> mapped from a on-disk filesystem.|
>
> On Mon, Apr 11, 2011 at 12:42 PM, srean <srean.list at gmail.com 
> <mailto:srean.list at gmail.com>> wrote:
>
>     Hi everyone,
>
>       I was looking up the options that are available for shared
>     memory arrays and this thread came up at the right time. The doc
>     says that|multiprocessing.Array(...) gives a shared memory array.
>     But from the code it seems to me that it is actually using a mmap.
>     Is that correct a correct assessment, and if so, is there any
>     advantage in using ||multiprocessing.Array(...)over simple numpy
>     mmaped arrays.
>
>     Regards
>       srean
>     ||| 
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/2635f75d/attachment.html>

From sturla at molden.no  Mon Apr 11 14:47:28 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 11 Apr 2011 20:47:28 +0200
Subject: [Numpy-discussion] Shared memory ndarrays (update)
In-Reply-To: <4DA2FB02.7050308@in.waw.pl>
References: <4DA23B47.10301@molden.no>	<20110411050523.GA30988@phare.normalesup.org>	<BANLkTinUr3GQLW680p=+_eZVr2k5xitNfg@mail.gmail.com>
	<4DA2FB02.7050308@in.waw.pl>
Message-ID: <4DA34CC0.8000705@molden.no>

Den 11.04.2011 14:58, skrev Zbigniew J?drzejewski-Szmek:
> Hi,
> it could, but you'd have to do the parsing of data yourself. So nothing
> fancy unless you want to reimplement numpy in Java :)

Not really. Only the data buffer is stored in shared memory. If you can 
pass the required fields to Java (shape, strides, sagment name, offset 
to first element), the memory mappred ndarray can be used by Java. 
(Doing it with JNI is trivial, I am not sure what Java's standard 
library supports.) Interop with C++, Fortran, C#, Matalab, whatever, can 
be done similarly.


> . But if you
> use a mmaped file as a backing for a numpy array of one of the types
> also available in Java (signed byte, short, int, long, float, double)
> and then map it on the Java side with DoubleBuffer or FloatBuffer or
> ..., then it seems straightforward enough.

Which is what we are doing, except that we are not memory mapping a 
physical file but a RAM segment with a filename.


Sturla


From jsseabold at gmail.com  Mon Apr 11 14:54:03 2011
From: jsseabold at gmail.com (Skipper Seabold)
Date: Mon, 11 Apr 2011 14:54:03 -0400
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1 and
	Numpy > 1.5.1
Message-ID: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>

All,

We noticed some failing tests for statsmodels between numpy 1.5.1 and
numpy >= 1.6.0. These are the versions where I noticed the change. It
seems that when you divide a float array and multiply by a boolean
array the answers are different (unless the others are also off by
some floating point). Can anyone replicate the following using this
script or point out what I'm missing?

import numpy as np
print np.__version__
np.random.seed(12345)

test = np.random.randint(0,2,size=10).astype(bool)
t = 1.345
arr = np.random.random(size=10)

arr # okay
t/arr # okay
(1-test)*arr # okay
(1-test)*t/arr # huh?

Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> print np.__version__
2.0.0.dev-fe3852f
>>> np.random.seed(12345)
>>>
>>> test = np.random.randint(0,2,size=10).astype(bool)
>>> t = 1.345
>>> arr = np.random.random(size=10)
>>>
>>> arr # okay
array([ 0.5955447 ,  0.96451452,  0.6531771 ,  0.74890664,  0.65356987,
        0.74771481,  0.96130674,  0.0083883 ,  0.10644438,  0.29870371])
>>> t/arr # okay
array([   2.25843668,    1.39448393,    2.05916589,    1.7959515 ,
          2.0579284 ,    1.79881418,    1.39913718,  160.342421  ,
         12.63570742,    4.50278968])
>>> (1-test)*arr # okay
array([ 0.5955447 ,  0.        ,  0.        ,  0.        ,  0.65356987,
        0.        ,  0.96130674,  0.0083883 ,  0.        ,  0.29870371])
>>> (1-test)*t/arr # huh?
array([   2.25797754,    0.        ,    0.        ,    0.        ,
          2.05751003,    0.        ,    1.39885274,  160.3098235 ,
          0.        ,    4.50187427])


Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> print np.__version__
1.5.1
>>> np.random.seed(12345)
>>>
>>> test = np.random.randint(0,2,size=10).astype(bool)
>>> t = 1.345
>>> arr = np.random.random(size=10)
>>>
>>> arr # okay
array([ 0.5955447 ,  0.96451452,  0.6531771 ,  0.74890664,  0.65356987,
        0.74771481,  0.96130674,  0.0083883 ,  0.10644438,  0.29870371])
>>> t/arr # okay
array([   2.25843668,    1.39448393,    2.05916589,    1.7959515 ,
          2.0579284 ,    1.79881418,    1.39913718,  160.342421  ,
         12.63570742,    4.50278968])
>>> (1-test)*arr # okay
array([ 0.5955447 ,  0.        ,  0.        ,  0.        ,  0.65356987,
        0.        ,  0.96130674,  0.0083883 ,  0.        ,  0.29870371])
>>> (1-test)*t/arr # huh?
array([   2.25843668,    0.        ,    0.        ,    0.        ,
          2.0579284 ,    0.        ,    1.39913718,  160.342421  ,
          0.        ,    4.50278968])

Skipper


From srean.list at gmail.com  Mon Apr 11 15:15:01 2011
From: srean.list at gmail.com (srean)
Date: Mon, 11 Apr 2011 14:15:01 -0500
Subject: [Numpy-discussion] Shared memory ndarrays (update)
In-Reply-To: <4DA34A3B.9080105@molden.no>
References: <4DA23B47.10301@molden.no>
	<20110411050523.GA30988@phare.normalesup.org>
	<BANLkTinUr3GQLW680p=+_eZVr2k5xitNfg@mail.gmail.com>
	<4DA2FB02.7050308@in.waw.pl>
	<BANLkTimY9GxsgXsaZXP+XisSpRHWR99piw@mail.gmail.com>
	<BANLkTimZ6j7r8BgxWHxoNa5jeUx6SP1jGw@mail.gmail.com>
	<4DA34A3B.9080105@molden.no>
Message-ID: <BANLkTimzZJz=fPbr_Ffxxi1zi1N6LM8w5g@mail.gmail.com>

Got you and thanks a lot for the explanation. I am not using Queues so I
think I am safe for the time being.  Given that you have worked a lot on
these issues, would you recommend plain mmapped numpy arrays over
multiprocessing.Array

Thanks again

-- srean

On Mon, Apr 11, 2011 at 1:36 PM, Sturla Molden <sturla at molden.no> wrote:

>  "Shared memory" is memory mapping from the paging file (i.e. RAM), not a
> file on disk. They can have a name or be anonymous. I have explained why we
> need named shared memory before. If you didn't understand it, try to pass an
> instance of multiprocessing.Array over multiprocessing.Queue.
>
> Sturla
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/3dfe6818/attachment.html>

From sturla at molden.no  Mon Apr 11 15:29:35 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 11 Apr 2011 21:29:35 +0200
Subject: [Numpy-discussion] Shared memory ndarrays (update)
In-Reply-To: <4DA23B47.10301@molden.no>
References: <4DA23B47.10301@molden.no>
Message-ID: <4DA3569F.2070606@molden.no>

Den 11.04.2011 01:20, skrev Sturla Molden:
>
> Changes:
>
> - 64-bit support.
> - Memory leak on Linux/Unix should be gone (monkey patch for os._exit).
> - Added a global lock as there are callbacks to Python (the GIL is not 
> sufficient serialization).

I will also add a barrier synchronization primitive to this (as it is 
very useful for numerical computing, more so than mutexes or semaphores) 
and a work scheduler, so it will be easy to distribute work between the 
processes.

The barrier is implemented with atomic read/writes on top of shared 
memory, so it can be sent over multiprocessing.Queue.

It is also possible to implement locks, semaphores, events, etc., the 
same way.

Observe that the synchronization primitives in multiprocessing, e.g. 
multiprocessing.RLock, cannot be sent over a Queue. Named shared memory 
takes that restriction away. Thus, objects that are pickled for 
multiprocessing.Queue can contain locks, events, barriers, etc.

Sturla


From sturla at molden.no  Mon Apr 11 15:34:27 2011
From: sturla at molden.no (Sturla Molden)
Date: Mon, 11 Apr 2011 21:34:27 +0200
Subject: [Numpy-discussion] Shared memory ndarrays (update)
In-Reply-To: <BANLkTimzZJz=fPbr_Ffxxi1zi1N6LM8w5g@mail.gmail.com>
References: <4DA23B47.10301@molden.no>	<20110411050523.GA30988@phare.normalesup.org>	<BANLkTinUr3GQLW680p=+_eZVr2k5xitNfg@mail.gmail.com>	<4DA2FB02.7050308@in.waw.pl>	<BANLkTimY9GxsgXsaZXP+XisSpRHWR99piw@mail.gmail.com>	<BANLkTimZ6j7r8BgxWHxoNa5jeUx6SP1jGw@mail.gmail.com>	<4DA34A3B.9080105@molden.no>
	<BANLkTimzZJz=fPbr_Ffxxi1zi1N6LM8w5g@mail.gmail.com>
Message-ID: <4DA357C3.60507@molden.no>

Den 11.04.2011 21:15, skrev srean:
> Got you and thanks a lot for the explanation. I am not using Queues so 
> I think I am safe for the time being.  Given that you have worked a 
> lot on these issues, would you recommend plain mmapped numpy arrays 
> over |multiprocessing.Array|
|
With multiprocessing you must use multiprocessing.Array.

With os.fork you can use either.

Sturla
|
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/08e30cda/attachment.html>

From robert.kern at gmail.com  Mon Apr 11 15:37:28 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 11 Apr 2011 14:37:28 -0500
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
Message-ID: <BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>

On Mon, Apr 11, 2011 at 13:54, Skipper Seabold <jsseabold at gmail.com> wrote:
> All,
>
> We noticed some failing tests for statsmodels between numpy 1.5.1 and
> numpy >= 1.6.0. These are the versions where I noticed the change. It
> seems that when you divide a float array and multiply by a boolean
> array the answers are different (unless the others are also off by
> some floating point). Can anyone replicate the following using this
> script or point out what I'm missing?
>
> import numpy as np
> print np.__version__
> np.random.seed(12345)
>
> test = np.random.randint(0,2,size=10).astype(bool)
> t = 1.345
> arr = np.random.random(size=10)
>
> arr # okay
> t/arr # okay
> (1-test)*arr # okay
> (1-test)*t/arr # huh?

[~]
|12> 1-test
array([1, 0, 0, 0, 1, 0, 1, 1, 0, 1], dtype=int8)

[~]
|13> (1-test)*t
array([ 1.34472656,  0.        ,  0.        ,  0.        ,
1.34472656,  0.        ,
        1.34472656,  1.34472656,  0.        ,  1.34472656], dtype=float16)

Some of the recent ufunc changes or the introduction of the float16
type must have changed the way the dtype is chosen for the
"int8-array*float64-scalar" case. Previously, the result was a float64
array, as expected.

Mark, I expect this is a result of one of your changes. Can you take a
look at this?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
? -- Umberto Eco


From charlesr.harris at gmail.com  Mon Apr 11 15:39:00 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 11 Apr 2011 13:39:00 -0600
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
Message-ID: <BANLkTinOQQAMCuGCwFTm1B8WwdZ=GCFTAw@mail.gmail.com>

On Mon, Apr 11, 2011 at 12:54 PM, Skipper Seabold <jsseabold at gmail.com>wrote:

> All,
>
> We noticed some failing tests for statsmodels between numpy 1.5.1 and
> numpy >= 1.6.0. These are the versions where I noticed the change. It
> seems that when you divide a float array and multiply by a boolean
> array the answers are different (unless the others are also off by
> some floating point). Can anyone replicate the following using this
> script or point out what I'm missing?
>
> import numpy as np
> print np.__version__
> np.random.seed(12345)
>
> test = np.random.randint(0,2,size=10).astype(bool)
> t = 1.345
> arr = np.random.random(size=10)
>
> arr # okay
> t/arr # okay
> (1-test)*arr # okay
> (1-test)*t/arr # huh?
>
> Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
> [GCC 4.4.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import numpy as np
> >>> print np.__version__
> 2.0.0.dev-fe3852f
> >>> np.random.seed(12345)
> >>>
> >>> test = np.random.randint(0,2,size=10).astype(bool)
> >>> t = 1.345
> >>> arr = np.random.random(size=10)
> >>>
> >>> arr # okay
> array([ 0.5955447 ,  0.96451452,  0.6531771 ,  0.74890664,  0.65356987,
>        0.74771481,  0.96130674,  0.0083883 ,  0.10644438,  0.29870371])
> >>> t/arr # okay
> array([   2.25843668,    1.39448393,    2.05916589,    1.7959515 ,
>          2.0579284 ,    1.79881418,    1.39913718,  160.342421  ,
>         12.63570742,    4.50278968])
> >>> (1-test)*arr # okay
> array([ 0.5955447 ,  0.        ,  0.        ,  0.        ,  0.65356987,
>        0.        ,  0.96130674,  0.0083883 ,  0.        ,  0.29870371])
> >>> (1-test)*t/arr # huh?
> array([   2.25797754,    0.        ,    0.        ,    0.        ,
>          2.05751003,    0.        ,    1.39885274,  160.3098235 ,
>          0.        ,    4.50187427])
>
>
Looks like it is going through float16:

In [2]: float16(1.345)/0.5955447
Out[2]: 2.25797754979601

The scalar 1.345 should be converted to double and that isn't happening.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/ea2c3592/attachment.html>

From mwwiebe at gmail.com  Mon Apr 11 16:31:15 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Mon, 11 Apr 2011 13:31:15 -0700
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
Message-ID: <BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>

On Mon, Apr 11, 2011 at 12:37 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Mon, Apr 11, 2011 at 13:54, Skipper Seabold <jsseabold at gmail.com>
> wrote:
> > All,
> >
> > We noticed some failing tests for statsmodels between numpy 1.5.1 and
> > numpy >= 1.6.0. These are the versions where I noticed the change. It
> > seems that when you divide a float array and multiply by a boolean
> > array the answers are different (unless the others are also off by
> > some floating point). Can anyone replicate the following using this
> > script or point out what I'm missing?
> >
> > import numpy as np
> > print np.__version__
> > np.random.seed(12345)
> >
> > test = np.random.randint(0,2,size=10).astype(bool)
> > t = 1.345
> > arr = np.random.random(size=10)
> >
> > arr # okay
> > t/arr # okay
> > (1-test)*arr # okay
> > (1-test)*t/arr # huh?
>
> [~]
> |12> 1-test
> array([1, 0, 0, 0, 1, 0, 1, 1, 0, 1], dtype=int8)
>
> [~]
> |13> (1-test)*t
> array([ 1.34472656,  0.        ,  0.        ,  0.        ,
> 1.34472656,  0.        ,
>        1.34472656,  1.34472656,  0.        ,  1.34472656], dtype=float16)
>
> Some of the recent ufunc changes or the introduction of the float16
> type must have changed the way the dtype is chosen for the
> "int8-array*float64-scalar" case. Previously, the result was a float64
> array, as expected.
>
> Mark, I expect this is a result of one of your changes. Can you take a
> look at this?
>

It's the type promotion rules change that is causing this, and it's
definitely a 1.6.0 release blocker. Good catch!

I can think of an easy, reasonable way to fix it in the result_type
function, but since the ufuncs themselves use a poor method of resolving the
types, it will take some thought to apply the same idea there. Maybe
detecting that the ufunc is a binary op at its creation time, setting a
flag, and using the result_type function in that case. This would have the
added bonus of being faster, too.

Going back to the 1.5 code isn't an option, since adding the new data types
while maintaining ABI compatibility required major surgery to this part of
the system.

-Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/4b2b2e59/attachment.html>

From charlesr.harris at gmail.com  Mon Apr 11 16:55:22 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 11 Apr 2011 14:55:22 -0600
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
Message-ID: <BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>

On Mon, Apr 11, 2011 at 2:31 PM, Mark Wiebe <mwwiebe at gmail.com> wrote:

> On Mon, Apr 11, 2011 at 12:37 PM, Robert Kern <robert.kern at gmail.com>wrote:
>
>> On Mon, Apr 11, 2011 at 13:54, Skipper Seabold <jsseabold at gmail.com>
>> wrote:
>> > All,
>> >
>> > We noticed some failing tests for statsmodels between numpy 1.5.1 and
>> > numpy >= 1.6.0. These are the versions where I noticed the change. It
>> > seems that when you divide a float array and multiply by a boolean
>> > array the answers are different (unless the others are also off by
>> > some floating point). Can anyone replicate the following using this
>> > script or point out what I'm missing?
>> >
>> > import numpy as np
>> > print np.__version__
>> > np.random.seed(12345)
>> >
>> > test = np.random.randint(0,2,size=10).astype(bool)
>> > t = 1.345
>> > arr = np.random.random(size=10)
>> >
>> > arr # okay
>> > t/arr # okay
>> > (1-test)*arr # okay
>> > (1-test)*t/arr # huh?
>>
>> [~]
>> |12> 1-test
>> array([1, 0, 0, 0, 1, 0, 1, 1, 0, 1], dtype=int8)
>>
>> [~]
>> |13> (1-test)*t
>> array([ 1.34472656,  0.        ,  0.        ,  0.        ,
>> 1.34472656,  0.        ,
>>        1.34472656,  1.34472656,  0.        ,  1.34472656], dtype=float16)
>>
>> Some of the recent ufunc changes or the introduction of the float16
>> type must have changed the way the dtype is chosen for the
>> "int8-array*float64-scalar" case. Previously, the result was a float64
>> array, as expected.
>>
>> Mark, I expect this is a result of one of your changes. Can you take a
>> look at this?
>>
>
> It's the type promotion rules change that is causing this, and it's
> definitely a 1.6.0 release blocker. Good catch!
>
> I can think of an easy, reasonable way to fix it in the result_type
> function, but since the ufuncs themselves use a poor method of resolving the
> types, it will take some thought to apply the same idea there. Maybe
> detecting that the ufunc is a binary op at its creation time, setting a
> flag, and using the result_type function in that case. This would have the
> added bonus of being faster, too.
>
> Going back to the 1.5 code isn't an option, since adding the new data types
> while maintaining ABI compatibility required major surgery to this part of
> the system.
>
>
There isn't a big rush here, so take the time work it through.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/4dfab0cc/attachment.html>

From sergio.pasra at gmail.com  Mon Apr 11 17:36:52 2011
From: sergio.pasra at gmail.com (Sergio Pascual)
Date: Mon, 11 Apr 2011 23:36:52 +0200
Subject: [Numpy-discussion] Extending numpy statistics functions (like mean)
Message-ID: <BANLkTinu8PF+AKPtt1gDd7b9tw7G-4+NPg@mail.gmail.com>

Hi list.

For mi application, I would like to implement some new statistics
functions over numpy arrays, such as truncated mean. Ideally this new
function should have the same arguments
than numpy.mean: axis, dtype and out. Is there a way of writing this
function that doesn't imply writing it in C from scratch?

I have read the documentation, but as far a I see ufuncs convert a N
dimensional array into another and generalized ufuncs require fixed
dimensions. numpy mean converts a N dimensional array either in a
number or a N - 1 dimensional array.


Regards, Sergio


From kwgoodman at gmail.com  Mon Apr 11 18:03:16 2011
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 11 Apr 2011 15:03:16 -0700
Subject: [Numpy-discussion] Extending numpy statistics functions (like
	mean)
In-Reply-To: <BANLkTinu8PF+AKPtt1gDd7b9tw7G-4+NPg@mail.gmail.com>
References: <BANLkTinu8PF+AKPtt1gDd7b9tw7G-4+NPg@mail.gmail.com>
Message-ID: <BANLkTikw=AoUVyh3PcJgrCAzVgAeKDvqGg@mail.gmail.com>

On Mon, Apr 11, 2011 at 2:36 PM, Sergio Pascual <sergio.pasra at gmail.com> wrote:
> Hi list.
>
> For mi application, I would like to implement some new statistics
> functions over numpy arrays, such as truncated mean. Ideally this new
> function should have the same arguments
> than numpy.mean: axis, dtype and out. Is there a way of writing this
> function that doesn't imply writing it in C from scratch?
>
> I have read the documentation, but as far a I see ufuncs convert a N
> dimensional array into another and generalized ufuncs require fixed
> dimensions. numpy mean converts a N dimensional array either in a
> number or a N - 1 dimensional array.

Here's a slow, brute force method:

>> a = np.arange(9).reshape(3,3)
>> a
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])
>> idx = a > 6
>> b = a. copy()
>> b[idx] = 0
>> b
array([[0, 1, 2],
       [3, 4, 5],
       [6, 0, 0]])
>> 1.0 * b.sum(axis=0) / (~idx).sum(axis=0)
   array([ 3. ,  2.5,  3.5])


From sergiopr at fis.ucm.es  Mon Apr 11 18:31:58 2011
From: sergiopr at fis.ucm.es (Sergio Pascual)
Date: Tue, 12 Apr 2011 00:31:58 +0200
Subject: [Numpy-discussion] Extending numpy statistics functions (like
	mean)
In-Reply-To: <BANLkTikw=AoUVyh3PcJgrCAzVgAeKDvqGg@mail.gmail.com>
References: <BANLkTinu8PF+AKPtt1gDd7b9tw7G-4+NPg@mail.gmail.com>
	<BANLkTikw=AoUVyh3PcJgrCAzVgAeKDvqGg@mail.gmail.com>
Message-ID: <BANLkTimkkQ-7b29C1xnQWYEWPzuoGwdkRw@mail.gmail.com>

What I have is some C++ functions that implement statistic functions.
What I need is some kind of ufunc where I can "plug" my functions. But
I doesn't seem to exist an ufunc that operates on a N-d array and
turns it into a number.

2011/4/12 Keith Goodman <kwgoodman at gmail.com>:
> On Mon, Apr 11, 2011 at 2:36 PM, Sergio Pascual <sergio.pasra at gmail.com> wrote:
>> Hi list.
>>
>> For mi application, I would like to implement some new statistics
>> functions over numpy arrays, such as truncated mean. Ideally this new
>> function should have the same arguments
>> than numpy.mean: axis, dtype and out. Is there a way of writing this
>> function that doesn't imply writing it in C from scratch?
>>
>> I have read the documentation, but as far a I see ufuncs convert a N
>> dimensional array into another and generalized ufuncs require fixed
>> dimensions. numpy mean converts a N dimensional array either in a
>> number or a N - 1 dimensional array.
>
> Here's a slow, brute force method:
>
>>> a = np.arange(9).reshape(3,3)
>>> a
> array([[0, 1, 2],
> ? ? ? [3, 4, 5],
> ? ? ? [6, 7, 8]])
>>> idx = a > 6
>>> b = a. copy()
>>> b[idx] = 0
>>> b
> array([[0, 1, 2],
> ? ? ? [3, 4, 5],
> ? ? ? [6, 0, 0]])
>>> 1.0 * b.sum(axis=0) / (~idx).sum(axis=0)
> ? array([ 3. , ?2.5, ?3.5])
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Sergio Pascual ? ? ? ? ? ? ? ? ? ? ? ?http://guaix.fis.ucm.es/~spr
gpg fingerprint: 5203 B42D 86A0 5649 410A F4AC A35F D465 F263 BCCC
Departamento de Astrof?sica -- Universidad Complutense de Madrid (Spain)


From g.turek at niwa.co.nz  Mon Apr 11 21:25:17 2011
From: g.turek at niwa.co.nz (Gabriella Turek)
Date: Tue, 12 Apr 2011 13:25:17 +1200
Subject: [Numpy-discussion] Error importing numpy in cygwin
Message-ID: <4DA452BD0200008C0000F715@gwia.niwa.co.nz>


Hello I'm working with cygwin 1.7.9. I've installed python 2.6 from the cygwin distro. I've also installed nympy from the distro (v. 1.4.1), and when that failed, I tried to installed directly form source (v. 1.5.1) 
In both cases when I try to run a script that imports numpy (including running the numpy tests) I get the following
error message:

-- 


>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/site-packages/numpy/__init__.py", line 154, in <modul
e>
    import ctypeslib
  File "/usr/lib/python2.6/site-packages/numpy/ctypeslib.py", line 60, in <modul
e>
    import ctypes
  File "/usr/lib/python2.6/ctypes/__init__.py", line 439, in <module>
    pythonapi = PyDLL("libpython%d.%d.dll" % _sys.version_info[:2])
  File "/usr/lib/python2.6/ctypes/__init__.py", line 353, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: No such file or directory

Any ideas?
Thanx
Gaby


Please consider the environment before printing this email.

NIWA is the trading name of the National Institute of Water & Atmospheric Research Ltd.


From oliphant at enthought.com  Mon Apr 11 23:48:49 2011
From: oliphant at enthought.com (Travis Oliphant)
Date: Mon, 11 Apr 2011 22:48:49 -0500
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
	and Numpy > 1.5.1
In-Reply-To: <BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
Message-ID: <63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>


On Apr 11, 2011, at 3:55 PM, Charles R Harris wrote:

> 
> 
> On Mon, Apr 11, 2011 at 2:31 PM, Mark Wiebe <mwwiebe at gmail.com> wrote:
> On Mon, Apr 11, 2011 at 12:37 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Mon, Apr 11, 2011 at 13:54, Skipper Seabold <jsseabold at gmail.com> wrote:
> > All,
> >
> > We noticed some failing tests for statsmodels between numpy 1.5.1 and
> > numpy >= 1.6.0. These are the versions where I noticed the change. It
> > seems that when you divide a float array and multiply by a boolean
> > array the answers are different (unless the others are also off by
> > some floating point). Can anyone replicate the following using this
> > script or point out what I'm missing?
> >
> > import numpy as np
> > print np.__version__
> > np.random.seed(12345)
> >
> > test = np.random.randint(0,2,size=10).astype(bool)
> > t = 1.345
> > arr = np.random.random(size=10)
> >
> > arr # okay
> > t/arr # okay
> > (1-test)*arr # okay
> > (1-test)*t/arr # huh?
> 
> [~]
> |12> 1-test
> array([1, 0, 0, 0, 1, 0, 1, 1, 0, 1], dtype=int8)
> 
> [~]
> |13> (1-test)*t
> array([ 1.34472656,  0.        ,  0.        ,  0.        ,
> 1.34472656,  0.        ,
>        1.34472656,  1.34472656,  0.        ,  1.34472656], dtype=float16)
> 
> Some of the recent ufunc changes or the introduction of the float16
> type must have changed the way the dtype is chosen for the
> "int8-array*float64-scalar" case. Previously, the result was a float64
> array, as expected.
> 
> Mark, I expect this is a result of one of your changes. Can you take a
> look at this?
> 
> It's the type promotion rules change that is causing this, and it's definitely a 1.6.0 release blocker. Good catch!
> 
> I can think of an easy, reasonable way to fix it in the result_type function, but since the ufuncs themselves use a poor method of resolving the types, it will take some thought to apply the same idea there. Maybe detecting that the ufunc is a binary op at its creation time, setting a flag, and using the result_type function in that case. This would have the added bonus of being faster, too.
> 
> Going back to the 1.5 code isn't an option, since adding the new data types while maintaining ABI compatibility required major surgery to this part of the system.
> 
> 
> There isn't a big rush here, so take the time work it through.

I agree with Charles.   Let's take the needed time and work this through.   This is the sort of thing I was a bit nervous about with the changes made to the casting rules.    Right now, I'm not confident that the scalar-array casting rules are being handled correctly. 

From your explanation, I don't understand how you intend to solve the problem.   Surely there is a better approach than special casing the binary op and this work-around flag (and why is the result_type even part of the discussion)? 

It would be good to see a simple test case and understand why the boolean multiplied by the scalar double is becoming a float16.     In other words,  why does

(1-test)*t 

return a float16 array

This does not sound right at all and it would be good to understand why this occurs, now.   How are you handling scalars multiplied by arrays in general?  


-Travis


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/a3a7fda5/attachment.html>

From mwwiebe at gmail.com  Tue Apr 12 00:43:34 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Mon, 11 Apr 2011 21:43:34 -0700
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
Message-ID: <BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>

On Mon, Apr 11, 2011 at 8:48 PM, Travis Oliphant <oliphant at enthought.com>wrote:

>
> On Apr 11, 2011, at 3:55 PM, Charles R Harris wrote:
> <snip>
> I agree with Charles.   Let's take the needed time and work this through.
> This is the sort of thing I was a bit nervous about with the changes made to
> the casting rules.    Right now, I'm not confident that the scalar-array
> casting rules are being handled correctly.
>

I would suggest viewing this as having exposed a problem in the NumPy test
suite. All necessary type promotion behaviors should have tests for them,
and I added tests since before 1.6 there were none. This case, where the
scalar type has a greater "kind" than the array type, was missed and had no
preexisting tests. If the test suite is thorough, you can be confident that
the scalar-array casting rules are being handled right.

Here's where to add more tests. Both np.add and np.result_type are validated
with the same set of tests.

https://github.com/numpy/numpy/blob/master/numpy/core/tests/test_numeric.py#L345


> From your explanation, I don't understand how you intend to solve the
> problem.   Surely there is a better approach than special casing the binary
> op and this work-around flag (and why is the result_type even part of the
> discussion)?
>

Yes, there's definitely a better approach, but I don't see one without
changing the ufunc API. In other libraries and programming languages, type
promotion is defined with a set of explicit promotion rules, but since the
current ufuncs have a list of functions and do linear searches in those
lists to find a matching loop, defining and implementing such an explicit
rule set is more difficult.

The reason to bring the numpy.result_type function into it is that it
cleanly encapsulates the NumPy type-promotion rules. A good design needs a
single, consistent way to handle type promotion, so that it can be validated
to be correct in a single place. I created the numpy.promote_types,
numpy.min_scalar_type, and numpy.result_type functions for this purpose, and
the numpy.nditer uses this API to determine the output type when it
allocates a requested array for output. By fixing result_type, then having
the ufunc binary operations use it, we can be confident that the type
promotion will be consistent.

It would be good to see a simple test case and understand why the boolean
> multiplied by the scalar double is becoming a float16.     In other words,
>  why does
>
> (1-test)*t
>
> return a float16 array
>
> This does not sound right at all and it would be good to understand why
> this occurs, now.   How are you handling scalars multiplied by arrays in
> general?
>

The reason it's float16 is that the first function in the multiply function
list for which both types can be safely cast to the output type, after
applying the min_scalar_type function to the scalars, is float16. The change
I'm suggesting is to remove the min_scalar_type part if the "kind" of the
scalar type is higher than the "kind" of the array type. Maybe it can be
done without switching the ufunc to use result_type in 1.6, but result_type
needs to be fixed as well to make the nditer and anything else that wants to
follow the ufunc rules be consistent.

I created a ticket for the issue here:
http://projects.scipy.org/numpy/ticket/1798

Cheers,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110411/6ffc3ac6/attachment.html>

From cournape at gmail.com  Tue Apr 12 03:17:10 2011
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 12 Apr 2011 16:17:10 +0900
Subject: [Numpy-discussion] Error importing numpy in cygwin
In-Reply-To: <4DA452BD0200008C0000F715@gwia.niwa.co.nz>
References: <4DA452BD0200008C0000F715@gwia.niwa.co.nz>
Message-ID: <BANLkTi=42DCqJb_W+Fxyvo9Dx1sCN6MmOA@mail.gmail.com>

On Tue, Apr 12, 2011 at 10:25 AM, Gabriella Turek <g.turek at niwa.co.nz> wrote:
>
> Hello I'm working with cygwin 1.7.9. I've installed python 2.6 from the cygwin distro. I've also installed nympy from the distro (v. 1.4.1), and when that failed, I tried to installed directly form source (v. 1.5.1)
> In both cases when I try to run a script that imports numpy (including running the numpy tests) I get the following
> error message:

It seems ctypes import fails. Are you sure that your python is
correctly installed ? What does the following does:

python -c "import ctypes"

If that does not work, the problem is how python was installed, not with numpy.

cheers,

David


From bsouthey at gmail.com  Tue Apr 12 09:35:04 2011
From: bsouthey at gmail.com (Bruce Southey)
Date: Tue, 12 Apr 2011 08:35:04 -0500
Subject: [Numpy-discussion] Extending numpy statistics functions (like
 mean)
In-Reply-To: <BANLkTikw=AoUVyh3PcJgrCAzVgAeKDvqGg@mail.gmail.com>
References: <BANLkTinu8PF+AKPtt1gDd7b9tw7G-4+NPg@mail.gmail.com>
	<BANLkTikw=AoUVyh3PcJgrCAzVgAeKDvqGg@mail.gmail.com>
Message-ID: <4DA45508.60704@gmail.com>

On 04/11/2011 05:03 PM, Keith Goodman wrote:
> On Mon, Apr 11, 2011 at 2:36 PM, Sergio Pascual<sergio.pasra at gmail.com>  wrote:
>> Hi list.
>>
>> For mi application, I would like to implement some new statistics
>> functions over numpy arrays, such as truncated mean. Ideally this new
>> function should have the same arguments
>> than numpy.mean: axis, dtype and out. Is there a way of writing this
>> function that doesn't imply writing it in C from scratch?
>>
>> I have read the documentation, but as far a I see ufuncs convert a N
>> dimensional array into another and generalized ufuncs require fixed
>> dimensions. numpy mean converts a N dimensional array either in a
>> number or a N - 1 dimensional array.
> Here's a slow, brute force method:
>
>>> a = np.arange(9).reshape(3,3)
>>> a
> array([[0, 1, 2],
>         [3, 4, 5],
>         [6, 7, 8]])
>>> idx = a>  6
>>> b = a. copy()
>>> b[idx] = 0
>>> b
> array([[0, 1, 2],
>         [3, 4, 5],
>         [6, 0, 0]])
>>> 1.0 * b.sum(axis=0) / (~idx).sum(axis=0)
>     array([ 3. ,  2.5,  3.5])
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
The truncated functions are easily handled by masked arrays and somewhat 
harder by using indexing (as seen below). There is limited functionality 
in scipy.stats as well. So first check scipy.stats to see if the 
functions you need are there. Otherwise please post a list of possible 
functions to the scipy-dev list because that is the most likely home.


 >>> import numpy as np
 >>> from numpy import ma
 >>> y = np.arange(35).reshape(5,7)
 >>> b=y>20
 >>> z=ma.masked_where(y <= 20, y)
 >>> z.mean()
27.5
 >>> z.mean(axis=0)
masked_array(data = [24.5 25.5 26.5 27.5 28.5 29.5 30.5],
              mask = [False False False False False False False],
        fill_value = 1e+20)

 >>> z.mean(axis=1)
masked_array(data = [-- -- -- 24.0 31.0],
              mask = [ True  True  True False False],
        fill_value = 1e+20)

 >>> y[b].mean()
27.5
 >>> y[b[:,5]].mean(axis=0)
array([ 24.5,  25.5,  26.5,  27.5,  28.5,  29.5,  30.5])
 >>> y[b[:,5]].mean(axis=1)
array([ 24.,  31.])


Bruce


From robert.kern at gmail.com  Tue Apr 12 11:24:56 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 12 Apr 2011 10:24:56 -0500
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
Message-ID: <BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>

On Mon, Apr 11, 2011 at 23:43, Mark Wiebe <mwwiebe at gmail.com> wrote:
> On Mon, Apr 11, 2011 at 8:48 PM, Travis Oliphant <oliphant at enthought.com>
> wrote:

>> It would be good to see a simple test case and understand why the boolean
>> multiplied by the scalar double is becoming a float16. ? ? In other words,
>> ?why does
>> (1-test)*t
>> return a float16 array
>> This does not sound right at all and it would be good to understand why
>> this occurs, now. ? How are you handling scalars multiplied by arrays in
>> general?
>
> The reason it's float16 is that the first function in the multiply function
> list for which both types can be safely cast to the output type,

Except that float64 cannot be safely cast to float16.

> after
> applying the min_scalar_type function to the scalars, is float16.

This is implemented incorrectly, then. It makes no sense for floats,
for which the limiting attribute is precision, not range. For floats,
the result of min_scalar_type should be the type of the object itself,
nothing else. E.g. min_scalar_type(x)==float64 if type(x) is float no
matter what value it has.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
? -- Umberto Eco


From mwwiebe at gmail.com  Tue Apr 12 12:20:56 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Tue, 12 Apr 2011 09:20:56 -0700
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
Message-ID: <BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>

On Tue, Apr 12, 2011 at 8:24 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Mon, Apr 11, 2011 at 23:43, Mark Wiebe <mwwiebe at gmail.com> wrote:
> > On Mon, Apr 11, 2011 at 8:48 PM, Travis Oliphant <oliphant at enthought.com
> >
> > wrote:
>
> >> It would be good to see a simple test case and understand why the
> boolean
> >> multiplied by the scalar double is becoming a float16.     In other
> words,
> >>  why does
> >> (1-test)*t
> >> return a float16 array
> >> This does not sound right at all and it would be good to understand why
> >> this occurs, now.   How are you handling scalars multiplied by arrays in
> >> general?
> >
> > The reason it's float16 is that the first function in the multiply
> function
> > list for which both types can be safely cast to the output type,
>
> Except that float64 cannot be safely cast to float16.
>

That's true, but it was already being done this way with respect to float32.
Rereading the documentation for min_scalar_type, I see the explanation could
elaborate on the purpose of the function further. Float64 cannot be safely
cast to float32, but this is what NumPy does:

>>> import numpy as np
>>> np.__version__
'1.4.1'
>>> np.float64(3.5) * np.ones(2,dtype=np.float32)
array([ 3.5,  3.5], dtype=float32)
>>>

> after
> > applying the min_scalar_type function to the scalars, is float16.
>
> This is implemented incorrectly, then. It makes no sense for floats,
> for which the limiting attribute is precision, not range. For floats,
> the result of min_scalar_type should be the type of the object itself,
> nothing else. E.g. min_scalar_type(x)==float64 if type(x) is float no
> matter what value it has.
>

I believe this behavior is necessary to avoid undesirable promotion of
arrays to float64. If you are working with extremely large float32 arrays,
and multiply or add a Python float, you would always have to say
np.float32(value), something rather tedious. Looking at the archives, I see
you've explained this before. ;)

http://mail.scipy.org/pipermail/numpy-discussion/2007-April/027079.html

The re-implementation of type casting, other than for the case at hand which
I consider to be a wrong behavior, follows the existing pattern as closely
as I could understand it from the previous implementation. I tightened the
rules just slightly to avoid some problematic downcasts which previously
were occurring:

>>> np.__version__
'1.4.1'
>>> 100000*np.ones(2,dtype=np.int8)
array([-31072, -31072], dtype=int16)
>>> 1e60*np.ones(2,dtype=np.float32)
array([ Inf,  Inf], dtype=float32)
>>>

>>> np.__version__
'2.0.0.dev-4cb2eb4'
>>> 100000*np.ones(2,dtype=np.int8)
array([100000, 100000], dtype=int32)
>>> 1e60*np.ones(2,dtype=np.float32)
array([  1.00000000e+60,   1.00000000e+60])
>>>

-Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110412/cadbc78c/attachment.html>

From robert.kern at gmail.com  Tue Apr 12 12:30:29 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 12 Apr 2011 11:30:29 -0500
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
	<BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
Message-ID: <BANLkTinSfx34_U9HUw50BbGDbQEHSZdNWg@mail.gmail.com>

On Tue, Apr 12, 2011 at 11:20, Mark Wiebe <mwwiebe at gmail.com> wrote:
> On Tue, Apr 12, 2011 at 8:24 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Mon, Apr 11, 2011 at 23:43, Mark Wiebe <mwwiebe at gmail.com> wrote:
>> > On Mon, Apr 11, 2011 at 8:48 PM, Travis Oliphant
>> > <oliphant at enthought.com>
>> > wrote:
>>
>> >> It would be good to see a simple test case and understand why the
>> >> boolean
>> >> multiplied by the scalar double is becoming a float16. ? ? In other
>> >> words,
>> >> ?why does
>> >> (1-test)*t
>> >> return a float16 array
>> >> This does not sound right at all and it would be good to understand why
>> >> this occurs, now. ? How are you handling scalars multiplied by arrays
>> >> in
>> >> general?
>> >
>> > The reason it's float16 is that the first function in the multiply
>> > function
>> > list for which both types can be safely cast to the output type,
>>
>> Except that float64 cannot be safely cast to float16.
>
> That's true, but it was already being done this way with respect to float32.
> Rereading the documentation for min_scalar_type, I see the explanation could
> elaborate on the purpose of the function further. Float64 cannot be safely
> cast to float32, but this is what NumPy does:
>>>> import numpy as np
>>>> np.__version__
> '1.4.1'
>>>> np.float64(3.5) * np.ones(2,dtype=np.float32)
> array([ 3.5, ?3.5], dtype=float32)
>>>>

You're missing the key part of the rule that numpy uses: for
array*scalar cases, when both array and scalar are the same kind (both
floating point or both integers), then the array dtype always wins.
Only when they are different kinds do you try to negotiate a common
safe type between the scalar and the array.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
? -- Umberto Eco


From mwwiebe at gmail.com  Tue Apr 12 12:49:39 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Tue, 12 Apr 2011 09:49:39 -0700
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTinSfx34_U9HUw50BbGDbQEHSZdNWg@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
	<BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
	<BANLkTinSfx34_U9HUw50BbGDbQEHSZdNWg@mail.gmail.com>
Message-ID: <BANLkTik-Q+7U5dWWHWcmQWsn1XkiWqNOkA@mail.gmail.com>

On Tue, Apr 12, 2011 at 9:30 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Tue, Apr 12, 2011 at 11:20, Mark Wiebe <mwwiebe at gmail.com> wrote:
> > On Tue, Apr 12, 2011 at 8:24 AM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >>
> >> On Mon, Apr 11, 2011 at 23:43, Mark Wiebe <mwwiebe at gmail.com> wrote:
> >> > On Mon, Apr 11, 2011 at 8:48 PM, Travis Oliphant
> >> > <oliphant at enthought.com>
> >> > wrote:
> >>
> >> >> It would be good to see a simple test case and understand why the
> >> >> boolean
> >> >> multiplied by the scalar double is becoming a float16.     In other
> >> >> words,
> >> >>  why does
> >> >> (1-test)*t
> >> >> return a float16 array
> >> >> This does not sound right at all and it would be good to understand
> why
> >> >> this occurs, now.   How are you handling scalars multiplied by arrays
> >> >> in
> >> >> general?
> >> >
> >> > The reason it's float16 is that the first function in the multiply
> >> > function
> >> > list for which both types can be safely cast to the output type,
> >>
> >> Except that float64 cannot be safely cast to float16.
> >
> > That's true, but it was already being done this way with respect to
> float32.
> > Rereading the documentation for min_scalar_type, I see the explanation
> could
> > elaborate on the purpose of the function further. Float64 cannot be
> safely
> > cast to float32, but this is what NumPy does:
> >>>> import numpy as np
> >>>> np.__version__
> > '1.4.1'
> >>>> np.float64(3.5) * np.ones(2,dtype=np.float32)
> > array([ 3.5,  3.5], dtype=float32)
> >>>>
>
> You're missing the key part of the rule that numpy uses: for
> array*scalar cases, when both array and scalar are the same kind (both
> floating point or both integers), then the array dtype always wins.
> Only when they are different kinds do you try to negotiate a common
> safe type between the scalar and the array.


I'm afraid I'm not seeing the point you're driving at, can you provide some
examples which tease apart these issues? Here's the same example but with
different kinds, and to me it seems to have the same character as the case
with float32/float64:

>>> np.__version__
'1.4.1'
>>> 1e60*np.ones(2,dtype=np.complex64)
array([ Inf NaNj,  Inf NaNj], dtype=complex64)

>>> np.__version__
'2.0.0.dev-4cb2eb4'
>>> 1e60*np.ones(2,dtype=np.complex64)
array([  1.00000000e+60+0.j,   1.00000000e+60+0.j])

 -Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110412/2e549ff1/attachment.html>

From ralf.gommers at googlemail.com  Tue Apr 12 12:51:57 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Tue, 12 Apr 2011 18:51:57 +0200
Subject: [Numpy-discussion] Error importing numpy in cygwin
In-Reply-To: <BANLkTi=42DCqJb_W+Fxyvo9Dx1sCN6MmOA@mail.gmail.com>
References: <4DA452BD0200008C0000F715@gwia.niwa.co.nz>
	<BANLkTi=42DCqJb_W+Fxyvo9Dx1sCN6MmOA@mail.gmail.com>
Message-ID: <BANLkTimBL0Fz2pLOEaUep+tcd_U4-gq_+w@mail.gmail.com>

On Tue, Apr 12, 2011 at 9:17 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Tue, Apr 12, 2011 at 10:25 AM, Gabriella Turek <g.turek at niwa.co.nz> wrote:
>>
>> Hello I'm working with cygwin 1.7.9. I've installed python 2.6 from the cygwin distro. I've also installed nympy from the distro (v. 1.4.1), and when that failed, I tried to installed directly form source (v. 1.5.1)
>> In both cases when I try to run a script that imports numpy (including running the numpy tests) I get the following
>> error message:
>
> It seems ctypes import fails. Are you sure that your python is
> correctly installed ? What does the following does:
>
> python -c "import ctypes"
>
> If that does not work, the problem is how python was installed, not with numpy.

It's a known problem that ctypes is not always built for Python, see for example
http://projects.scipy.org/numpy/ticket/1475
http://bugs.python.org/issue1516
http://bugs.python.org/issue2401

Also this ticket (ctypes + Cygwin) may be related:
http://projects.scipy.org/numpy/ticket/904

The submitter of #1475 suggests that it's worth considering if numpy
should depend on ctypes. I agree that it would be better not to.
Ctypes is not in the Python 2.4 stdlib, and even after that can give
problems on anything that's not Linux. Would it be possible to use
Cython instead with a reasonable amount of effort?

Ralf


From charlesr.harris at gmail.com  Tue Apr 12 13:27:19 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 12 Apr 2011 11:27:19 -0600
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTik-Q+7U5dWWHWcmQWsn1XkiWqNOkA@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
	<BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
	<BANLkTinSfx34_U9HUw50BbGDbQEHSZdNWg@mail.gmail.com>
	<BANLkTik-Q+7U5dWWHWcmQWsn1XkiWqNOkA@mail.gmail.com>
Message-ID: <BANLkTik60yEaFfsynmZD5mCbxu3CjbiJYg@mail.gmail.com>

On Tue, Apr 12, 2011 at 10:49 AM, Mark Wiebe <mwwiebe at gmail.com> wrote:

> On Tue, Apr 12, 2011 at 9:30 AM, Robert Kern <robert.kern at gmail.com>wrote:
>
>> On Tue, Apr 12, 2011 at 11:20, Mark Wiebe <mwwiebe at gmail.com> wrote:
>> > On Tue, Apr 12, 2011 at 8:24 AM, Robert Kern <robert.kern at gmail.com>
>> wrote:
>> >>
>> >> On Mon, Apr 11, 2011 at 23:43, Mark Wiebe <mwwiebe at gmail.com> wrote:
>> >> > On Mon, Apr 11, 2011 at 8:48 PM, Travis Oliphant
>> >> > <oliphant at enthought.com>
>> >> > wrote:
>> >>
>> >> >> It would be good to see a simple test case and understand why the
>> >> >> boolean
>> >> >> multiplied by the scalar double is becoming a float16.     In other
>> >> >> words,
>> >> >>  why does
>> >> >> (1-test)*t
>> >> >> return a float16 array
>> >> >> This does not sound right at all and it would be good to understand
>> why
>> >> >> this occurs, now.   How are you handling scalars multiplied by
>> arrays
>> >> >> in
>> >> >> general?
>> >> >
>> >> > The reason it's float16 is that the first function in the multiply
>> >> > function
>> >> > list for which both types can be safely cast to the output type,
>> >>
>> >> Except that float64 cannot be safely cast to float16.
>> >
>> > That's true, but it was already being done this way with respect to
>> float32.
>> > Rereading the documentation for min_scalar_type, I see the explanation
>> could
>> > elaborate on the purpose of the function further. Float64 cannot be
>> safely
>> > cast to float32, but this is what NumPy does:
>> >>>> import numpy as np
>> >>>> np.__version__
>> > '1.4.1'
>> >>>> np.float64(3.5) * np.ones(2,dtype=np.float32)
>> > array([ 3.5,  3.5], dtype=float32)
>> >>>>
>>
>> You're missing the key part of the rule that numpy uses: for
>> array*scalar cases, when both array and scalar are the same kind (both
>> floating point or both integers), then the array dtype always wins.
>> Only when they are different kinds do you try to negotiate a common
>> safe type between the scalar and the array.
>
>
> I'm afraid I'm not seeing the point you're driving at, can you provide some
> examples which tease apart these issues? Here's the same example but with
> different kinds, and to me it seems to have the same character as the case
> with float32/float64:
>
> >>> np.__version__
> '1.4.1'
> >>> 1e60*np.ones(2,dtype=np.complex64)
> array([ Inf NaNj,  Inf NaNj], dtype=complex64)
>
> >>> np.__version__
> '2.0.0.dev-4cb2eb4'
> >>> 1e60*np.ones(2,dtype=np.complex64)
> array([  1.00000000e+60+0.j,   1.00000000e+60+0.j])
>
>
IIRC, the behavior with respect to scalars sort of happened in the code on
the fly, so this is a good discussion to have. We should end up with
documented rules and tests to enforce them. I agree with Mark that the tests
have been deficient up to this point.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110412/a6d800e8/attachment.html>

From charlesr.harris at gmail.com  Tue Apr 12 13:42:32 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 12 Apr 2011 11:42:32 -0600
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
	<BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
Message-ID: <BANLkTikerXhtqSX5eS+6C7qCmiS-afz=tw@mail.gmail.com>

On Tue, Apr 12, 2011 at 10:20 AM, Mark Wiebe <mwwiebe at gmail.com> wrote:

> On Tue, Apr 12, 2011 at 8:24 AM, Robert Kern <robert.kern at gmail.com>wrote:
>
>> On Mon, Apr 11, 2011 at 23:43, Mark Wiebe <mwwiebe at gmail.com> wrote:
>> > On Mon, Apr 11, 2011 at 8:48 PM, Travis Oliphant <
>> oliphant at enthought.com>
>> > wrote:
>>
>> >> It would be good to see a simple test case and understand why the
>> boolean
>> >> multiplied by the scalar double is becoming a float16.     In other
>> words,
>> >>  why does
>> >> (1-test)*t
>> >> return a float16 array
>> >> This does not sound right at all and it would be good to understand why
>> >> this occurs, now.   How are you handling scalars multiplied by arrays
>> in
>> >> general?
>> >
>> > The reason it's float16 is that the first function in the multiply
>> function
>> > list for which both types can be safely cast to the output type,
>>
>> Except that float64 cannot be safely cast to float16.
>>
>
> That's true, but it was already being done this way with respect to
> float32. Rereading the documentation for min_scalar_type, I see the
> explanation could elaborate on the purpose of the function further. Float64
> cannot be safely cast to float32, but this is what NumPy does:
>
>
Yep, I remember noticing that on occasion. I didn't think it was really the
right thing to do...

<snip>

Chuck

>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110412/13bf5f6f/attachment.html>

From robert.kern at gmail.com  Tue Apr 12 13:56:11 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 12 Apr 2011 12:56:11 -0500
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTik60yEaFfsynmZD5mCbxu3CjbiJYg@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
	<BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
	<BANLkTinSfx34_U9HUw50BbGDbQEHSZdNWg@mail.gmail.com>
	<BANLkTik-Q+7U5dWWHWcmQWsn1XkiWqNOkA@mail.gmail.com>
	<BANLkTik60yEaFfsynmZD5mCbxu3CjbiJYg@mail.gmail.com>
Message-ID: <BANLkTik1EEmi3APVWNmT5MXsQwN6DBwxUw@mail.gmail.com>

On Tue, Apr 12, 2011 at 12:27, Charles R Harris
<charlesr.harris at gmail.com> wrote:

> IIRC, the behavior with respect to scalars sort of happened in the code on
> the fly, so this is a good discussion to have. We should end up with
> documented rules and tests to enforce them. I agree with Mark that the tests
> have been deficient up to this point.

It's been documented for a long time now.

http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
? -- Umberto Eco


From mat.yeates at gmail.com  Tue Apr 12 14:06:14 2011
From: mat.yeates at gmail.com (Mathew Yeates)
Date: Tue, 12 Apr 2011 11:06:14 -0700
Subject: [Numpy-discussion] f2py pass by reference
Message-ID: <BANLkTi=i_n=54OiifWw1XU0A7FMkwjAG5g@mail.gmail.com>

I have
subroutine foo (a)
      integer a
      print*, "Hello from Fortran!"
      print*, "a=",a
      a=2
      end

and from python I want to do
>>> a=1
>>> foo(a)

and I want a's value to now be 2.
How do I do this?

Mathew


From charlesr.harris at gmail.com  Tue Apr 12 14:17:37 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 12 Apr 2011 12:17:37 -0600
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTik1EEmi3APVWNmT5MXsQwN6DBwxUw@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
	<BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
	<BANLkTinSfx34_U9HUw50BbGDbQEHSZdNWg@mail.gmail.com>
	<BANLkTik-Q+7U5dWWHWcmQWsn1XkiWqNOkA@mail.gmail.com>
	<BANLkTik60yEaFfsynmZD5mCbxu3CjbiJYg@mail.gmail.com>
	<BANLkTik1EEmi3APVWNmT5MXsQwN6DBwxUw@mail.gmail.com>
Message-ID: <BANLkTikpS=bwSACzMqz9RvVH_YB72utSsg@mail.gmail.com>

On Tue, Apr 12, 2011 at 11:56 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Tue, Apr 12, 2011 at 12:27, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>
> > IIRC, the behavior with respect to scalars sort of happened in the code
> on
> > the fly, so this is a good discussion to have. We should end up with
> > documented rules and tests to enforce them. I agree with Mark that the
> tests
> > have been deficient up to this point.
>
> It's been documented for a long time now.
>
> http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
>
>
Nope, the kind stuff is missing. Note the cast to float32 that Mark pointed
out. Also that the casting of python integers depends on their sign and
magnitude.

In [1]: ones(3, '?') + 0
Out[1]: array([1, 1, 1], dtype=int8)

In [2]: ones(3, '?') + 1000
Out[2]: array([1001, 1001, 1001], dtype=int16)


Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110412/f3adf293/attachment.html>

From sameer.grover.1 at gmail.com  Tue Apr 12 14:41:09 2011
From: sameer.grover.1 at gmail.com (Sameer Grover)
Date: Wed, 13 Apr 2011 00:11:09 +0530
Subject: [Numpy-discussion] f2py pass by reference
In-Reply-To: <BANLkTi=i_n=54OiifWw1XU0A7FMkwjAG5g@mail.gmail.com>
References: <BANLkTi=i_n=54OiifWw1XU0A7FMkwjAG5g@mail.gmail.com>
Message-ID: <4DA49CC5.7040807@gmail.com>

On 12/04/11 23:36, Mathew Yeates wrote:
> I have
> subroutine foo (a)
>        integer a
>        print*, "Hello from Fortran!"
>        print*, "a=",a
>        a=2
>        end
>
> and from python I want to do
>>>> a=1
>>>> foo(a)
> and I want a's value to now be 2.
> How do I do this?
>
> Mathew
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

Someone can correct me if I'm wrong but I don't think this is possible 
with integers because they are treated as immuatable types in python. 
You can, however, do this with numpy arrays with "intent(inplace)". For 
example,

subroutine foo(a)
       integer::a(2)
!f2py intent(inplace)::a
       write(*,*) "Hello from Fortran!"
       write(*,*) "a=",a
       a(1)=2
end subroutine foo

a = np.array([1,2])
foo(a)
#a is now [2,2]



From pearu.peterson at gmail.com  Tue Apr 12 14:45:18 2011
From: pearu.peterson at gmail.com (Pearu Peterson)
Date: Tue, 12 Apr 2011 21:45:18 +0300
Subject: [Numpy-discussion] f2py pass by reference
In-Reply-To: <BANLkTi=i_n=54OiifWw1XU0A7FMkwjAG5g@mail.gmail.com>
References: <BANLkTi=i_n=54OiifWw1XU0A7FMkwjAG5g@mail.gmail.com>
Message-ID: <BANLkTikT_gggRZaWm4vbbf9TdsXdyA9GXg@mail.gmail.com>

On Tue, Apr 12, 2011 at 9:06 PM, Mathew Yeates <mat.yeates at gmail.com> wrote:

> I have
> subroutine foo (a)
>      integer a
>      print*, "Hello from Fortran!"
>      print*, "a=",a
>      a=2
>      end
>
> and from python I want to do
> >>> a=1
> >>> foo(a)
>
> and I want a's value to now be 2.
> How do I do this?
>

With

     subroutine foo (a)
     integer a
!f2py intent(in, out) a
     print*, "Hello from Fortran!"
     print*, "a=",a
     a=2
     end

you will have desired effect:

>>> a=1
>>> a = foo(a)
>>> print a
2

HTH,
Pearu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110412/4c722234/attachment.html>

From climateforu at gmail.com  Tue Apr 12 14:48:54 2011
From: climateforu at gmail.com (Climate Research)
Date: Wed, 13 Apr 2011 00:18:54 +0530
Subject: [Numpy-discussion] Data standardizing
Message-ID: <BANLkTikXbp9kV4b8-Pjdw5W+L4yJ6TvgqA@mail.gmail.com>

Hi
I am purely new to python and numpy..  I am using python for doing
statistical calculations to Climate data..

I  have  a  data set in the following format..

Year  Jan      feb       Mar    Apr.................   Dec
1900  1000    1001       ,        ,                         ,
1901  1011    1012       ,        ,                         ,
1902  1009    1007       ,                                  ,
,,,,        ,           '          ,        ,                         ,
,,,,        ,           ,
2010  1008    1002       ,        ,                         ,

I actually want to standardize each of these values with corresponding
standard deviations for  each monthly data column..
I have found out the standard deviations for each column..  but now i need
to  find the standared deviation  only for a prescribed mean value
ie,  when i am finding the standared deviation for the January data
column..  the mean should be calculated only for the january data, say from
1950-1970. With this mean  i  want to calculate the SD  for entire column.
Any help will be appreciated..
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110413/06d8050e/attachment.html>

From mwwiebe at gmail.com  Tue Apr 12 14:51:55 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Tue, 12 Apr 2011 11:51:55 -0700
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTikpS=bwSACzMqz9RvVH_YB72utSsg@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
	<BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
	<BANLkTinSfx34_U9HUw50BbGDbQEHSZdNWg@mail.gmail.com>
	<BANLkTik-Q+7U5dWWHWcmQWsn1XkiWqNOkA@mail.gmail.com>
	<BANLkTik60yEaFfsynmZD5mCbxu3CjbiJYg@mail.gmail.com>
	<BANLkTik1EEmi3APVWNmT5MXsQwN6DBwxUw@mail.gmail.com>
	<BANLkTikpS=bwSACzMqz9RvVH_YB72utSsg@mail.gmail.com>
Message-ID: <BANLkTi=ffexSYdQvzq1iVwPoEEsOSwsP4g@mail.gmail.com>

On Tue, Apr 12, 2011 at 11:17 AM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

>
>
> On Tue, Apr 12, 2011 at 11:56 AM, Robert Kern <robert.kern at gmail.com>wrote:
>
>> On Tue, Apr 12, 2011 at 12:27, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>
>> > IIRC, the behavior with respect to scalars sort of happened in the code
>> on
>> > the fly, so this is a good discussion to have. We should end up with
>> > documented rules and tests to enforce them. I agree with Mark that the
>> tests
>> > have been deficient up to this point.
>>
>> It's been documented for a long time now.
>>
>> http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
>>
>>
> Nope, the kind stuff is missing. Note the cast to float32 that Mark pointed
> out. Also that the casting of python integers depends on their sign and
> magnitude.
>
> In [1]: ones(3, '?') + 0
> Out[1]: array([1, 1, 1], dtype=int8)
>
> In [2]: ones(3, '?') + 1000
> Out[2]: array([1001, 1001, 1001], dtype=int16)
>

This is the behaviour with master - it's a good idea to cross-check with an
older NumPy. I think we're discussing 3 things here, what NumPy 1.5 and
earlier did, what NumPy 1.6 beta currently does, and what people think NumPy
did. The old implementation had a bit of a spaghetti-factor to it, and had
problems like asymmetry and silent overflows. The new implementation is in
my opinion cleaner and follows well-defined semantics while trying to stay
true to the old implementation. I admit the documentation I wrote doesn't
fully explain them, but here's the rule for a set of arbitrary arrays (not
necessarily just 2):

- if all the arrays are scalars, do type promotion on the types as is
- otherwise, do type promotion on min_scalar_type(a) of each array a

The function min_scalar_type returns the array type if a has >= 1
dimensions, or the smallest type of the same kind (allowing int->uint in the
case of positive-valued signed integers) to which the value can be cast
without overflow if a has 0 dimensions.

The promote_types function used for the type promotion is symmetric and
associative, so the result won't change when shuffling the inputs. There's a
bit of a wrinkle in the implementation to handle the fact that the uint type
values aren't a strict subset of the same-sized int type values, but
otherwise this is what happens.

https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/convert_datatype.c#L1075

The change I'm proposing is to modify this as follows:

- if all the arrays are scalars, do type promotion on the types as is
- if the maximum kind of all the scalars is > the maximum kind of all the
arrays, do type promotion on the types as is
- otherwise, do type promotion on min_scalar_type(a) of each array a

One case where this may not capture a possible desired semantics is
[complex128 scalar] * [float32 array] -> [complex128]. In this case
[complex64] may be desired. This is directly analogous to the original
[float64 scalar] * [int8 array], however, and in the latter case it's clear
a float64 should result.

-Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110412/59e2dc7f/attachment.html>

From mat.yeates at gmail.com  Tue Apr 12 14:52:20 2011
From: mat.yeates at gmail.com (Mathew Yeates)
Date: Tue, 12 Apr 2011 11:52:20 -0700
Subject: [Numpy-discussion] f2py pass by reference
In-Reply-To: <BANLkTikT_gggRZaWm4vbbf9TdsXdyA9GXg@mail.gmail.com>
References: <BANLkTi=i_n=54OiifWw1XU0A7FMkwjAG5g@mail.gmail.com>
	<BANLkTikT_gggRZaWm4vbbf9TdsXdyA9GXg@mail.gmail.com>
Message-ID: <BANLkTimGjObjm_za9K3J=viw8MxYpTW92A@mail.gmail.com>

bizarre
I get
=================
>>> hello.foo(a)
 Hello from Fortran!
 a= 1
2
>>> a
1
>>> hello.foo(a)
 Hello from Fortran!
 a= 1
2
>>> print a
1
>>>
=================================

i.e. The value of 2 gets printed! This is numpy 1.3.0

-Mathew


On Tue, Apr 12, 2011 at 11:45 AM, Pearu Peterson
<pearu.peterson at gmail.com> wrote:
>
>
> On Tue, Apr 12, 2011 at 9:06 PM, Mathew Yeates <mat.yeates at gmail.com> wrote:
>>
>> I have
>> subroutine foo (a)
>> ? ? ?integer a
>> ? ? ?print*, "Hello from Fortran!"
>> ? ? ?print*, "a=",a
>> ? ? ?a=2
>> ? ? ?end
>>
>> and from python I want to do
>> >>> a=1
>> >>> foo(a)
>>
>> and I want a's value to now be 2.
>> How do I do this?
>
> With
>
> ???? subroutine foo (a)
> ? ? ?integer a
> !f2py intent(in, out) a
> ? ? ?print*, "Hello from Fortran!"
> ? ? ?print*, "a=",a
> ? ? ?a=2
> ? ? ?end
>
> you will have desired effect:
>
>>>> a=1
>>>> a = foo(a)
>>>> print a
> 2
>
> HTH,
> Pearu
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


From pearu.peterson at gmail.com  Tue Apr 12 15:11:28 2011
From: pearu.peterson at gmail.com (Pearu Peterson)
Date: Tue, 12 Apr 2011 22:11:28 +0300
Subject: [Numpy-discussion] f2py pass by reference
In-Reply-To: <BANLkTimGjObjm_za9K3J=viw8MxYpTW92A@mail.gmail.com>
References: <BANLkTi=i_n=54OiifWw1XU0A7FMkwjAG5g@mail.gmail.com>
	<BANLkTikT_gggRZaWm4vbbf9TdsXdyA9GXg@mail.gmail.com>
	<BANLkTimGjObjm_za9K3J=viw8MxYpTW92A@mail.gmail.com>
Message-ID: <BANLkTikC7nwU=M1o4aLDde08MHyt+Hd-LA@mail.gmail.com>

Note that hello.foo(a) returns the value of Fortran `a` value. This explains
the printed 2 value.
So, use

>>> a = hello.foo(a)

and not

>>> hello.foo(a)

As Sameer noted in previous mail, passing Python scalar values to Fortran by
reference is not
possible because Python scalars are immutable. Hence the need to use `a =
foo(a)`.

HTH,
Pearu

On Tue, Apr 12, 2011 at 9:52 PM, Mathew Yeates <mat.yeates at gmail.com> wrote:

> bizarre
> I get
> =================
> >>> hello.foo(a)
>  Hello from Fortran!
>  a= 1
> 2
> >>> a
> 1
> >>> hello.foo(a)
>  Hello from Fortran!
>  a= 1
> 2
> >>> print a
> 1
> >>>
> =================================
>
> i.e. The value of 2 gets printed! This is numpy 1.3.0
>
> -Mathew
>
>
> On Tue, Apr 12, 2011 at 11:45 AM, Pearu Peterson
> <pearu.peterson at gmail.com> wrote:
> >
> >
> > On Tue, Apr 12, 2011 at 9:06 PM, Mathew Yeates <mat.yeates at gmail.com>
> wrote:
> >>
> >> I have
> >> subroutine foo (a)
> >>      integer a
> >>      print*, "Hello from Fortran!"
> >>      print*, "a=",a
> >>      a=2
> >>      end
> >>
> >> and from python I want to do
> >> >>> a=1
> >> >>> foo(a)
> >>
> >> and I want a's value to now be 2.
> >> How do I do this?
> >
> > With
> >
> >      subroutine foo (a)
> >      integer a
> > !f2py intent(in, out) a
> >      print*, "Hello from Fortran!"
> >      print*, "a=",a
> >      a=2
> >      end
> >
> > you will have desired effect:
> >
> >>>> a=1
> >>>> a = foo(a)
> >>>> print a
> > 2
> >
> > HTH,
> > Pearu
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110412/98dd4abf/attachment.html>

From robert.kern at gmail.com  Tue Apr 12 15:17:39 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 12 Apr 2011 14:17:39 -0500
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTik-Q+7U5dWWHWcmQWsn1XkiWqNOkA@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
	<BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
	<BANLkTinSfx34_U9HUw50BbGDbQEHSZdNWg@mail.gmail.com>
	<BANLkTik-Q+7U5dWWHWcmQWsn1XkiWqNOkA@mail.gmail.com>
Message-ID: <BANLkTi=X+5twEyKycE4c8hbXpQFwramgww@mail.gmail.com>

On Tue, Apr 12, 2011 at 11:49, Mark Wiebe <mwwiebe at gmail.com> wrote:
> On Tue, Apr 12, 2011 at 9:30 AM, Robert Kern <robert.kern at gmail.com> wrote:

>> You're missing the key part of the rule that numpy uses: for
>> array*scalar cases, when both array and scalar are the same kind (both
>> floating point or both integers), then the array dtype always wins.
>> Only when they are different kinds do you try to negotiate a common
>> safe type between the scalar and the array.
>
> I'm afraid I'm not seeing the point you're driving at, can you provide some
> examples which tease apart these issues? Here's the same example but with
> different kinds, and to me it seems to have the same character as the case
> with float32/float64:
>>>> np.__version__
> '1.4.1'
>>>> 1e60*np.ones(2,dtype=np.complex64)
> array([ Inf NaNj, ?Inf NaNj], dtype=complex64)
>>>> np.__version__
> '2.0.0.dev-4cb2eb4'
>>>> 1e60*np.ones(2,dtype=np.complex64)
> array([ ?1.00000000e+60+0.j, ? 1.00000000e+60+0.j])

The point is that when you multiply an array by a scalar, and the
array-dtype is the same kind as the scalar-dtype, the output dtype is
the array-dtype. That's what gets you the behavior of the
float32-array staying the same when you multiply it with a Python
float(64). min_scalar_type should never be consulted in this case, so
you don't need to try to account for this case in its rules. This
cross-kind example is irrelevant to the point I'm trying to make.

For cross-kind operations, then you do need to find a common output
type that is safe for both array and scalar. However, please keep in
mind that for floating point types, keeping precision is more
important than range!

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
? -- Umberto Eco


From robert.kern at gmail.com  Tue Apr 12 15:21:37 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 12 Apr 2011 14:21:37 -0500
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTikpS=bwSACzMqz9RvVH_YB72utSsg@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
	<BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
	<BANLkTinSfx34_U9HUw50BbGDbQEHSZdNWg@mail.gmail.com>
	<BANLkTik-Q+7U5dWWHWcmQWsn1XkiWqNOkA@mail.gmail.com>
	<BANLkTik60yEaFfsynmZD5mCbxu3CjbiJYg@mail.gmail.com>
	<BANLkTik1EEmi3APVWNmT5MXsQwN6DBwxUw@mail.gmail.com>
	<BANLkTikpS=bwSACzMqz9RvVH_YB72utSsg@mail.gmail.com>
Message-ID: <BANLkTimmCwE9OQwHf=RHpOedr87e8b8d+g@mail.gmail.com>

On Tue, Apr 12, 2011 at 13:17, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Tue, Apr 12, 2011 at 11:56 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Tue, Apr 12, 2011 at 12:27, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>
>> > IIRC, the behavior with respect to scalars sort of happened in the code
>> > on
>> > the fly, so this is a good discussion to have. We should end up with
>> > documented rules and tests to enforce them. I agree with Mark that the
>> > tests
>> > have been deficient up to this point.
>>
>> It's been documented for a long time now.
>>
>> http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules
>>
>
> Nope, the kind stuff is missing. Note the cast to float32 that Mark pointed
> out.

The float32-array*float64-scalar case? That's covered in the last
paragraph; they are the same kind so array dtype wins.

> Also that the casting of python integers depends on their sign and
> magnitude.
>
> In [1]: ones(3, '?') + 0
> Out[1]: array([1, 1, 1], dtype=int8)
>
> In [2]: ones(3, '?') + 1000
> Out[2]: array([1001, 1001, 1001], dtype=int16)

bool and int cross kinds. Not a counterexample. I'm not saying that
the size of the values should be ignored for cross-kind upcasting. I'm
saying that you don't need value-size calculations to preserve the
float32-array*float64-scalar behavior.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
? -- Umberto Eco


From csecook at gmail.com  Wed Apr 13 09:30:35 2011
From: csecook at gmail.com (c cook)
Date: Wed, 13 Apr 2011 09:30:35 -0400
Subject: [Numpy-discussion] problem installing numpy on Fedora
Message-ID: <BANLkTi=3OhW2=fSmAaO0NM+Gw3cRWJZq9g@mail.gmail.com>

Hello,

I am trying to install numpy 1.6.0b2 version on Fedora and I am getting the
following error:

In file included from numpy/core/src/private/npy_config.h:4,
                 from numpy/core/src/multiarray/common.c:8,
                 from
numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
build/src.linux-x86_64-2.4/numpy/core/include/numpy/config.h:34:1: warning:
"HAVE_HYPOT" redefined
In file included from /home/cse/Apps/Python-2.4/Include/Python.h:8,
                 from numpy/core/src/multiarray/common.c:2,
                 from
numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
/home/cse/Apps/Python-2.4/pyconfig.h:232:1: warning: this is the location of
the previous definition
In file included from
numpy/core/src/multiarray/multiarraymodule_onefile.c:14:
numpy/core/src/multiarray/arraytypes.c.src: In function `VOID_getitem':
numpy/core/src/multiarray/arraytypes.c.src:615: warning: label `finish'
defined but not used
In file included from
numpy/core/src/multiarray/multiarraymodule_onefile.c:19:
numpy/core/src/multiarray/descriptor.c: In function `descr_repeat':
numpy/core/src/multiarray/descriptor.c:2840: warning: long int format,
different type arg (arg 3)
In file included from
numpy/core/src/multiarray/multiarraymodule_onefile.c:27:
numpy/core/src/multiarray/methods.c: In function `NpyArg_ParseKeywords':
numpy/core/src/multiarray/methods.c:41: warning: passing arg 3 of
`PyArg_VaParseTupleAndKeywords' discards qualifiers from pointer target type
In file included from
numpy/core/src/multiarray/multiarraymodule_onefile.c:28:
numpy/core/src/multiarray/convert_datatype.c: In function
`min_scalar_type_num':
numpy/core/src/multiarray/convert_datatype.c:774: warning: comparison is
always true due to limited range of data type
numpy/core/src/multiarray/descriptor.c: In function
`_convert_divisor_to_multiple':
numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used
uninitialized in this function
numpy/core/src/multiarray/einsum.c.src: In function
`float_sum_of_products_contig_outstride0_one':
numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn:
(insn:HI 440 228 481 14
/usr/lib/gcc/x86_64-redhat-linux/3.4.4/include/xmmintrin.h:915 (set (reg:SF
148)
        (vec_select:SF (reg/v:V4SF 67 [ accum_sse ])
            (parallel [
                    (const_int 0 [0x0])
                ]))) -1 (insn_list 213 (nil))
    (nil))
numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: in
extract_insn, at recog.c:2083
Please submit a full bug report,
with preprocessed source if appropriate.
See <URL:http://bugzilla.redhat.com/bugzilla> for instructions.
Preprocessed source stored into /tmp/cck42rVN.out file, please attach this
to your bugreport.
In file included from numpy/core/src/private/npy_config.h:4,
                 from numpy/core/src/multiarray/common.c:8,
                 from
numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
build/src.linux-x86_64-2.4/numpy/core/include/numpy/config.h:34:1: warning:
"HAVE_HYPOT" redefined
In file included from /home/cse/Apps/Python-2.4/Include/Python.h:8,
                 from numpy/core/src/multiarray/common.c:2,
                 from
numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
/home/cse/Apps/Python-2.4/pyconfig.h:232:1: warning: this is the location of
the previous definition
In file included from
numpy/core/src/multiarray/multiarraymodule_onefile.c:14:
numpy/core/src/multiarray/arraytypes.c.src: In function `VOID_getitem':
numpy/core/src/multiarray/arraytypes.c.src:615: warning: label `finish'
defined but not used
In file included from
numpy/core/src/multiarray/multiarraymodule_onefile.c:19:
numpy/core/src/multiarray/descriptor.c: In function `descr_repeat':
numpy/core/src/multiarray/descriptor.c:2840: warning: long int format,
different type arg (arg 3)
In file included from
numpy/core/src/multiarray/multiarraymodule_onefile.c:27:
numpy/core/src/multiarray/methods.c: In function `NpyArg_ParseKeywords':
numpy/core/src/multiarray/methods.c:41: warning: passing arg 3 of
`PyArg_VaParseTupleAndKeywords' discards qualifiers from pointer target type
In file included from
numpy/core/src/multiarray/multiarraymodule_onefile.c:28:
numpy/core/src/multiarray/convert_datatype.c: In function
`min_scalar_type_num':
numpy/core/src/multiarray/convert_datatype.c:774: warning: comparison is
always true due to limited range of data type
numpy/core/src/multiarray/descriptor.c: In function
`_convert_divisor_to_multiple':
numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used
uninitialized in this function
numpy/core/src/multiarray/einsum.c.src: In function
`float_sum_of_products_contig_outstride0_one':
numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn:
(insn:HI 440 228 481 14
/usr/lib/gcc/x86_64-redhat-linux/3.4.4/include/xmmintrin.h:915 (set (reg:SF
148)
        (vec_select:SF (reg/v:V4SF 67 [ accum_sse ])
            (parallel [
                    (const_int 0 [0x0])
                ]))) -1 (insn_list 213 (nil))
    (nil))
numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: in
extract_insn, at recog.c:2083
Please submit a full bug report,
with preprocessed source if appropriate.
See <URL:http://bugzilla.redhat.com/bugzilla> for instructions.
Preprocessed source stored into /tmp/cck42rVN.out file, please attach this
to your bugreport.
error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
-Wstrict-prototypes -fPIC -Inumpy/core/include
-Ibuild/src.linux-x86_64-2.4/numpy/core/include/numpy
-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core
-Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath
-Inumpy/core/include -I/home/cse/Apps/Python-2.4/Include
-I/home/cse/Apps/Python-2.4
-Ibuild/src.linux-x86_64-2.4/numpy/core/src/multiarray
-Ibuild/src.linux-x86_64-2.4/numpy/core/src/umath -c
numpy/core/src/multiarray/multiarraymodule_onefile.c -o
build/temp.linux-x86_64-2.4/multiarraymodule_onefile.o" failed with exit
status 1


Any idea what is the problem?

Thank you,
Csaba
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110413/590f1c7f/attachment.html>

From jrocher at enthought.com  Wed Apr 13 09:50:13 2011
From: jrocher at enthought.com (Jonathan Rocher)
Date: Wed, 13 Apr 2011 08:50:13 -0500
Subject: [Numpy-discussion] Data standardizing
In-Reply-To: <BANLkTikXbp9kV4b8-Pjdw5W+L4yJ6TvgqA@mail.gmail.com>
References: <BANLkTikXbp9kV4b8-Pjdw5W+L4yJ6TvgqA@mail.gmail.com>
Message-ID: <BANLkTikH--ZOrLz2P8SWjpS0JmpOU=cyLQ@mail.gmail.com>

Hi,

I assume you have this data in a txt file, correct? You can load up all of
it in a numpy array using
import numpy as np
data = np.loadtxt("climat_file.txt", skiprows = 1)

Then you can compute the mean you want by taking it on a slice of the data
array. For example, if you want to compute the mean of your data in Jan for
1950-1970 (say including 1970)
mean1950_1970 = data[1950:1971,1].mean()

Then the std deviation you want could be computed using
my_std = np.sqrt(np.mean((data[:,1]-mean1950_1970)**2))

Hope this helps,
Jonathan

On Tue, Apr 12, 2011 at 1:48 PM, Climate Research <climateforu at gmail.com>wrote:

> Hi
> I am purely new to python and numpy..  I am using python for doing
> statistical calculations to Climate data..
>
> I  have  a  data set in the following format..
>
> Year  Jan      feb       Mar    Apr.................   Dec
> 1900  1000    1001       ,        ,                         ,
> 1901  1011    1012       ,        ,                         ,
> 1902  1009    1007       ,                                  ,
> ,,,,        ,           '          ,        ,                         ,
> ,,,,        ,           ,
> 2010  1008    1002       ,        ,                         ,
>
> I actually want to standardize each of these values with corresponding
> standard deviations for  each monthly data column..
> I have found out the standard deviations for each column..  but now i need
> to  find the standared deviation  only for a prescribed mean value
> ie,  when i am finding the standared deviation for the January data
> column..  the mean should be calculated only for the january data, say from
> 1950-1970. With this mean  i  want to calculate the SD  for entire column.
> Any help will be appreciated..
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
Jonathan Rocher, PhD
Scientific software developer
Enthought, Inc.
jrocher at enthought.com
1-512-536-1057
http://www.enthought.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110413/c61101a1/attachment.html>

From jdh2358 at gmail.com  Wed Apr 13 09:50:47 2011
From: jdh2358 at gmail.com (John Hunter)
Date: Wed, 13 Apr 2011 08:50:47 -0500
Subject: [Numpy-discussion] segfault on complex array on solaris x86
In-Reply-To: <AANLkTiki4Za-J7ZmTjYX4U__toaOicHNmJFy9BGJDLW1@mail.gmail.com>
References: <AANLkTi=W3DU2pD7ODPCoCr-5UkPaiKtnpctr2nM9ON5T@mail.gmail.com>
	<AANLkTiki4Za-J7ZmTjYX4U__toaOicHNmJFy9BGJDLW1@mail.gmail.com>
Message-ID: <BANLkTim09dYeVN3eOAvLO5FKmVvLmSSTyQ@mail.gmail.com>

On Sat, Jan 15, 2011 at 7:28 AM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
> I've opened http://projects.scipy.org/numpy/ticket/1713 so this doesn't get
> lost.

Just wanted to bump this -- bug still exists in numpy HEAD 2.0.0.dev-fe3852f


From bsouthey at gmail.com  Wed Apr 13 10:14:02 2011
From: bsouthey at gmail.com (Bruce Southey)
Date: Wed, 13 Apr 2011 09:14:02 -0500
Subject: [Numpy-discussion] problem installing numpy on Fedora
In-Reply-To: <BANLkTi=3OhW2=fSmAaO0NM+Gw3cRWJZq9g@mail.gmail.com>
References: <BANLkTi=3OhW2=fSmAaO0NM+Gw3cRWJZq9g@mail.gmail.com>
Message-ID: <4DA5AFAA.8000704@gmail.com>

On 04/13/2011 08:30 AM, c cook wrote:
> Hello,
>
> I am trying to install numpy 1.6.0b2 version on Fedora and I am 
> getting the following error:
>
> In file included from numpy/core/src/private/npy_config.h:4,
>                  from numpy/core/src/multiarray/common.c:8,
>                  from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
> build/src.linux-x86_64-2.4/numpy/core/include/numpy/config.h:34:1: 
> warning: "HAVE_HYPOT" redefined
> In file included from /home/cse/Apps/Python-2.4/Include/Python.h:8,
>                  from numpy/core/src/multiarray/common.c:2,
>                  from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
> /home/cse/Apps/Python-2.4/pyconfig.h:232:1: warning: this is the 
> location of the previous definition
> In file included from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:14:
> numpy/core/src/multiarray/arraytypes.c.src: In function `VOID_getitem':
> numpy/core/src/multiarray/arraytypes.c.src:615: warning: label 
> `finish' defined but not used
> In file included from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:19:
> numpy/core/src/multiarray/descriptor.c: In function `descr_repeat':
> numpy/core/src/multiarray/descriptor.c:2840: warning: long int format, 
> different type arg (arg 3)
> In file included from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:27:
> numpy/core/src/multiarray/methods.c: In function `NpyArg_ParseKeywords':
> numpy/core/src/multiarray/methods.c:41: warning: passing arg 3 of 
> `PyArg_VaParseTupleAndKeywords' discards qualifiers from pointer 
> target type
> In file included from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:28:
> numpy/core/src/multiarray/convert_datatype.c: In function 
> `min_scalar_type_num':
> numpy/core/src/multiarray/convert_datatype.c:774: warning: comparison 
> is always true due to limited range of data type
> numpy/core/src/multiarray/descriptor.c: In function 
> `_convert_divisor_to_multiple':
> numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used 
> uninitialized in this function
> numpy/core/src/multiarray/einsum.c.src: In function 
> `float_sum_of_products_contig_outstride0_one':
> numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn:
> (insn:HI 440 228 481 14 
> /usr/lib/gcc/x86_64-redhat-linux/3.4.4/include/xmmintrin.h:915 (set 
> (reg:SF 148)
>         (vec_select:SF (reg/v:V4SF 67 [ accum_sse ])
>             (parallel [
>                     (const_int 0 [0x0])
>                 ]))) -1 (insn_list 213 (nil))
>     (nil))
> numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: 
> in extract_insn, at recog.c:2083
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <URL:http://bugzilla.redhat.com/bugzilla> for instructions.
> Preprocessed source stored into /tmp/cck42rVN.out file, please attach 
> this to your bugreport.
> In file included from numpy/core/src/private/npy_config.h:4,
>                  from numpy/core/src/multiarray/common.c:8,
>                  from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
> build/src.linux-x86_64-2.4/numpy/core/include/numpy/config.h:34:1: 
> warning: "HAVE_HYPOT" redefined
> In file included from /home/cse/Apps/Python-2.4/Include/Python.h:8,
>                  from numpy/core/src/multiarray/common.c:2,
>                  from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
> /home/cse/Apps/Python-2.4/pyconfig.h:232:1: warning: this is the 
> location of the previous definition
> In file included from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:14:
> numpy/core/src/multiarray/arraytypes.c.src: In function `VOID_getitem':
> numpy/core/src/multiarray/arraytypes.c.src:615: warning: label 
> `finish' defined but not used
> In file included from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:19:
> numpy/core/src/multiarray/descriptor.c: In function `descr_repeat':
> numpy/core/src/multiarray/descriptor.c:2840: warning: long int format, 
> different type arg (arg 3)
> In file included from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:27:
> numpy/core/src/multiarray/methods.c: In function `NpyArg_ParseKeywords':
> numpy/core/src/multiarray/methods.c:41: warning: passing arg 3 of 
> `PyArg_VaParseTupleAndKeywords' discards qualifiers from pointer 
> target type
> In file included from 
> numpy/core/src/multiarray/multiarraymodule_onefile.c:28:
> numpy/core/src/multiarray/convert_datatype.c: In function 
> `min_scalar_type_num':
> numpy/core/src/multiarray/convert_datatype.c:774: warning: comparison 
> is always true due to limited range of data type
> numpy/core/src/multiarray/descriptor.c: In function 
> `_convert_divisor_to_multiple':
> numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used 
> uninitialized in this function
> numpy/core/src/multiarray/einsum.c.src: In function 
> `float_sum_of_products_contig_outstride0_one':
> numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn:
> (insn:HI 440 228 481 14 
> /usr/lib/gcc/x86_64-redhat-linux/3.4.4/include/xmmintrin.h:915 (set 
> (reg:SF 148)
>         (vec_select:SF (reg/v:V4SF 67 [ accum_sse ])
>             (parallel [
>                     (const_int 0 [0x0])
>                 ]))) -1 (insn_list 213 (nil))
>     (nil))
> numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: 
> in extract_insn, at recog.c:2083
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <URL:http://bugzilla.redhat.com/bugzilla> for instructions.
> Preprocessed source stored into /tmp/cck42rVN.out file, please attach 
> this to your bugreport.
> error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 
> -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include 
> -Ibuild/src.linux-x86_64-2.4/numpy/core/include/numpy 
> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core 
> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray 
> -Inumpy/core/src/umath -Inumpy/core/include 
> -I/home/cse/Apps/Python-2.4/Include -I/home/cse/Apps/Python-2.4 
> -Ibuild/src.linux-x86_64-2.4/numpy/core/src/multiarray 
> -Ibuild/src.linux-x86_64-2.4/numpy/core/src/umath -c 
> numpy/core/src/multiarray/multiarraymodule_onefile.c -o 
> build/temp.linux-x86_64-2.4/multiarraymodule_onefile.o" failed with 
> exit status 1
>
>
> Any idea what is the problem?
>
> Thank you,
> Csaba
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

I have no problem with Python2.4 and gcc version 4.5 on Fedora 14.
Which version of Fedora are you using?
That gcc version 3.4.4 is rather old as in Fedora Core 3 (cf Fedora 15 
in alpha release now).

I think that numpy requires at least gcc version 4+ but it actual 
version needs to be more specific than that. Hopefully someone with 
sufficient gcc knowledge will correct me and the documentation.

Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110413/1e04eab5/attachment.html>

From charlesr.harris at gmail.com  Wed Apr 13 10:17:15 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 13 Apr 2011 08:17:15 -0600
Subject: [Numpy-discussion] problem installing numpy on Fedora
In-Reply-To: <BANLkTi=3OhW2=fSmAaO0NM+Gw3cRWJZq9g@mail.gmail.com>
References: <BANLkTi=3OhW2=fSmAaO0NM+Gw3cRWJZq9g@mail.gmail.com>
Message-ID: <BANLkTi=JmjOEckN-AE4ue2XvH6-hjcgqtg@mail.gmail.com>

On Wed, Apr 13, 2011 at 7:30 AM, c cook <csecook at gmail.com> wrote:

> Hello,
>
> I am trying to install numpy 1.6.0b2 version on Fedora and I am getting the
> following error:
>
> In file included from numpy/core/src/private/npy_config.h:4,
>                  from numpy/core/src/multiarray/common.c:8,
>                  from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
> build/src.linux-x86_64-2.4/numpy/core/include/numpy/config.h:34:1: warning:
> "HAVE_HYPOT" redefined
> In file included from /home/cse/Apps/Python-2.4/Include/Python.h:8,
>                  from numpy/core/src/multiarray/common.c:2,
>                  from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
> /home/cse/Apps/Python-2.4/pyconfig.h:232:1: warning: this is the location
> of the previous definition
> In file included from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:14:
> numpy/core/src/multiarray/arraytypes.c.src: In function `VOID_getitem':
> numpy/core/src/multiarray/arraytypes.c.src:615: warning: label `finish'
> defined but not used
> In file included from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:19:
> numpy/core/src/multiarray/descriptor.c: In function `descr_repeat':
> numpy/core/src/multiarray/descriptor.c:2840: warning: long int format,
> different type arg (arg 3)
> In file included from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:27:
> numpy/core/src/multiarray/methods.c: In function `NpyArg_ParseKeywords':
> numpy/core/src/multiarray/methods.c:41: warning: passing arg 3 of
> `PyArg_VaParseTupleAndKeywords' discards qualifiers from pointer target type
> In file included from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:28:
> numpy/core/src/multiarray/convert_datatype.c: In function
> `min_scalar_type_num':
> numpy/core/src/multiarray/convert_datatype.c:774: warning: comparison is
> always true due to limited range of data type
> numpy/core/src/multiarray/descriptor.c: In function
> `_convert_divisor_to_multiple':
> numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used
> uninitialized in this function
> numpy/core/src/multiarray/einsum.c.src: In function
> `float_sum_of_products_contig_outstride0_one':
> numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn:
> (insn:HI 440 228 481 14
> /usr/lib/gcc/x86_64-redhat-linux/3.4.4/include/xmmintrin.h:915 (set (reg:SF
> 148)
>         (vec_select:SF (reg/v:V4SF 67 [ accum_sse ])
>             (parallel [
>                     (const_int 0 [0x0])
>                 ]))) -1 (insn_list 213 (nil))
>     (nil))
> numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: in
> extract_insn, at recog.c:2083
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <URL:http://bugzilla.redhat.com/bugzilla> for instructions.
> Preprocessed source stored into /tmp/cck42rVN.out file, please attach this
> to your bugreport.
> In file included from numpy/core/src/private/npy_config.h:4,
>                  from numpy/core/src/multiarray/common.c:8,
>                  from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
> build/src.linux-x86_64-2.4/numpy/core/include/numpy/config.h:34:1: warning:
> "HAVE_HYPOT" redefined
> In file included from /home/cse/Apps/Python-2.4/Include/Python.h:8,
>                  from numpy/core/src/multiarray/common.c:2,
>                  from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
> /home/cse/Apps/Python-2.4/pyconfig.h:232:1: warning: this is the location
> of the previous definition
> In file included from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:14:
> numpy/core/src/multiarray/arraytypes.c.src: In function `VOID_getitem':
> numpy/core/src/multiarray/arraytypes.c.src:615: warning: label `finish'
> defined but not used
> In file included from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:19:
> numpy/core/src/multiarray/descriptor.c: In function `descr_repeat':
> numpy/core/src/multiarray/descriptor.c:2840: warning: long int format,
> different type arg (arg 3)
> In file included from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:27:
> numpy/core/src/multiarray/methods.c: In function `NpyArg_ParseKeywords':
> numpy/core/src/multiarray/methods.c:41: warning: passing arg 3 of
> `PyArg_VaParseTupleAndKeywords' discards qualifiers from pointer target type
> In file included from
> numpy/core/src/multiarray/multiarraymodule_onefile.c:28:
> numpy/core/src/multiarray/convert_datatype.c: In function
> `min_scalar_type_num':
> numpy/core/src/multiarray/convert_datatype.c:774: warning: comparison is
> always true due to limited range of data type
> numpy/core/src/multiarray/descriptor.c: In function
> `_convert_divisor_to_multiple':
> numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used
> uninitialized in this function
> numpy/core/src/multiarray/einsum.c.src: In function
> `float_sum_of_products_contig_outstride0_one':
> numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn:
> (insn:HI 440 228 481 14
> /usr/lib/gcc/x86_64-redhat-linux/3.4.4/include/xmmintrin.h:915 (set (reg:SF
> 148)
>         (vec_select:SF (reg/v:V4SF 67 [ accum_sse ])
>             (parallel [
>                     (const_int 0 [0x0])
>                 ]))) -1 (insn_list 213 (nil))
>     (nil))
> numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: in
> extract_insn, at recog.c:2083
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <URL:http://bugzilla.redhat.com/bugzilla> for instructions.
> Preprocessed source stored into /tmp/cck42rVN.out file, please attach this
> to your bugreport.
> error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
> -Wstrict-prototypes -fPIC -Inumpy/core/include
> -Ibuild/src.linux-x86_64-2.4/numpy/core/include/numpy
> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core
> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath
> -Inumpy/core/include -I/home/cse/Apps/Python-2.4/Include
> -I/home/cse/Apps/Python-2.4
> -Ibuild/src.linux-x86_64-2.4/numpy/core/src/multiarray
> -Ibuild/src.linux-x86_64-2.4/numpy/core/src/umath -c
> numpy/core/src/multiarray/multiarraymodule_onefile.c -o
> build/temp.linux-x86_64-2.4/multiarraymodule_onefile.o" failed with exit
> status 1
>
>
> Any idea what is the problem?
>
>
I'll guess the combination of gcc-3.4.4, which is six years old, and the sse
instructions. It might be worth trying gcc-3.4.6 if you can't upgrade the
compiler outright.  Out of curiousity, what architecture and fedora version
are you using?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110413/287faf05/attachment.html>

From csecook at gmail.com  Wed Apr 13 15:39:21 2011
From: csecook at gmail.com (c cook)
Date: Wed, 13 Apr 2011 15:39:21 -0400
Subject: [Numpy-discussion] problem installing numpy on Fedora
In-Reply-To: <BANLkTi=JmjOEckN-AE4ue2XvH6-hjcgqtg@mail.gmail.com>
References: <BANLkTi=3OhW2=fSmAaO0NM+Gw3cRWJZq9g@mail.gmail.com>
	<BANLkTi=JmjOEckN-AE4ue2XvH6-hjcgqtg@mail.gmail.com>
Message-ID: <BANLkTi=x9=y+CRaujiQPv2iGFtG4fUWndQ@mail.gmail.com>

I have Fedora Core 3 and x86_64 architecture. But I don't understand why it
appears to look for /usr/lib/gcc/x86_64-redhat-linux/3.4.4/

instead of /usr/bin/gcc

Csaba

On Wed, Apr 13, 2011 at 10:17 AM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

>
>
> On Wed, Apr 13, 2011 at 7:30 AM, c cook <csecook at gmail.com> wrote:
>
>> Hello,
>>
>> I am trying to install numpy 1.6.0b2 version on Fedora and I am getting
>> the following error:
>>
>> In file included from numpy/core/src/private/npy_config.h:4,
>>                  from numpy/core/src/multiarray/common.c:8,
>>                  from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
>> build/src.linux-x86_64-2.4/numpy/core/include/numpy/config.h:34:1:
>> warning: "HAVE_HYPOT" redefined
>> In file included from /home/cse/Apps/Python-2.4/Include/Python.h:8,
>>                  from numpy/core/src/multiarray/common.c:2,
>>                  from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
>> /home/cse/Apps/Python-2.4/pyconfig.h:232:1: warning: this is the location
>> of the previous definition
>> In file included from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:14:
>> numpy/core/src/multiarray/arraytypes.c.src: In function `VOID_getitem':
>> numpy/core/src/multiarray/arraytypes.c.src:615: warning: label `finish'
>> defined but not used
>> In file included from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:19:
>> numpy/core/src/multiarray/descriptor.c: In function `descr_repeat':
>> numpy/core/src/multiarray/descriptor.c:2840: warning: long int format,
>> different type arg (arg 3)
>> In file included from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:27:
>> numpy/core/src/multiarray/methods.c: In function `NpyArg_ParseKeywords':
>> numpy/core/src/multiarray/methods.c:41: warning: passing arg 3 of
>> `PyArg_VaParseTupleAndKeywords' discards qualifiers from pointer target type
>> In file included from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:28:
>> numpy/core/src/multiarray/convert_datatype.c: In function
>> `min_scalar_type_num':
>> numpy/core/src/multiarray/convert_datatype.c:774: warning: comparison is
>> always true due to limited range of data type
>> numpy/core/src/multiarray/descriptor.c: In function
>> `_convert_divisor_to_multiple':
>> numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used
>> uninitialized in this function
>> numpy/core/src/multiarray/einsum.c.src: In function
>> `float_sum_of_products_contig_outstride0_one':
>> numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn:
>> (insn:HI 440 228 481 14
>> /usr/lib/gcc/x86_64-redhat-linux/3.4.4/include/xmmintrin.h:915 (set (reg:SF
>> 148)
>>         (vec_select:SF (reg/v:V4SF 67 [ accum_sse ])
>>             (parallel [
>>                     (const_int 0 [0x0])
>>                 ]))) -1 (insn_list 213 (nil))
>>     (nil))
>> numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: in
>> extract_insn, at recog.c:2083
>> Please submit a full bug report,
>> with preprocessed source if appropriate.
>> See <URL:http://bugzilla.redhat.com/bugzilla> for instructions.
>> Preprocessed source stored into /tmp/cck42rVN.out file, please attach this
>> to your bugreport.
>> In file included from numpy/core/src/private/npy_config.h:4,
>>                  from numpy/core/src/multiarray/common.c:8,
>>                  from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
>> build/src.linux-x86_64-2.4/numpy/core/include/numpy/config.h:34:1:
>> warning: "HAVE_HYPOT" redefined
>> In file included from /home/cse/Apps/Python-2.4/Include/Python.h:8,
>>                  from numpy/core/src/multiarray/common.c:2,
>>                  from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
>> /home/cse/Apps/Python-2.4/pyconfig.h:232:1: warning: this is the location
>> of the previous definition
>> In file included from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:14:
>> numpy/core/src/multiarray/arraytypes.c.src: In function `VOID_getitem':
>> numpy/core/src/multiarray/arraytypes.c.src:615: warning: label `finish'
>> defined but not used
>> In file included from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:19:
>> numpy/core/src/multiarray/descriptor.c: In function `descr_repeat':
>> numpy/core/src/multiarray/descriptor.c:2840: warning: long int format,
>> different type arg (arg 3)
>> In file included from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:27:
>> numpy/core/src/multiarray/methods.c: In function `NpyArg_ParseKeywords':
>> numpy/core/src/multiarray/methods.c:41: warning: passing arg 3 of
>> `PyArg_VaParseTupleAndKeywords' discards qualifiers from pointer target type
>> In file included from
>> numpy/core/src/multiarray/multiarraymodule_onefile.c:28:
>> numpy/core/src/multiarray/convert_datatype.c: In function
>> `min_scalar_type_num':
>> numpy/core/src/multiarray/convert_datatype.c:774: warning: comparison is
>> always true due to limited range of data type
>> numpy/core/src/multiarray/descriptor.c: In function
>> `_convert_divisor_to_multiple':
>> numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used
>> uninitialized in this function
>> numpy/core/src/multiarray/einsum.c.src: In function
>> `float_sum_of_products_contig_outstride0_one':
>> numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn:
>> (insn:HI 440 228 481 14
>> /usr/lib/gcc/x86_64-redhat-linux/3.4.4/include/xmmintrin.h:915 (set (reg:SF
>> 148)
>>         (vec_select:SF (reg/v:V4SF 67 [ accum_sse ])
>>             (parallel [
>>                     (const_int 0 [0x0])
>>                 ]))) -1 (insn_list 213 (nil))
>>     (nil))
>> numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: in
>> extract_insn, at recog.c:2083
>> Please submit a full bug report,
>> with preprocessed source if appropriate.
>> See <URL:http://bugzilla.redhat.com/bugzilla> for instructions.
>> Preprocessed source stored into /tmp/cck42rVN.out file, please attach this
>> to your bugreport.
>> error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
>> -Wstrict-prototypes -fPIC -Inumpy/core/include
>> -Ibuild/src.linux-x86_64-2.4/numpy/core/include/numpy
>> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core
>> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath
>> -Inumpy/core/include -I/home/cse/Apps/Python-2.4/Include
>> -I/home/cse/Apps/Python-2.4
>> -Ibuild/src.linux-x86_64-2.4/numpy/core/src/multiarray
>> -Ibuild/src.linux-x86_64-2.4/numpy/core/src/umath -c
>> numpy/core/src/multiarray/multiarraymodule_onefile.c -o
>> build/temp.linux-x86_64-2.4/multiarraymodule_onefile.o" failed with exit
>> status 1
>>
>>
>> Any idea what is the problem?
>>
>>
> I'll guess the combination of gcc-3.4.4, which is six years old, and the
> sse instructions. It might be worth trying gcc-3.4.6 if you can't upgrade
> the compiler outright.  Out of curiousity, what architecture and fedora
> version are you using?
>
> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110413/1333d0c9/attachment.html>

From sergio.pasra at gmail.com  Wed Apr 13 17:26:22 2011
From: sergio.pasra at gmail.com (Sergio Pascual)
Date: Wed, 13 Apr 2011 23:26:22 +0200
Subject: [Numpy-discussion] problem installing numpy on Fedora
In-Reply-To: <BANLkTi=x9=y+CRaujiQPv2iGFtG4fUWndQ@mail.gmail.com>
References: <BANLkTi=3OhW2=fSmAaO0NM+Gw3cRWJZq9g@mail.gmail.com>
	<BANLkTi=JmjOEckN-AE4ue2XvH6-hjcgqtg@mail.gmail.com>
	<BANLkTi=x9=y+CRaujiQPv2iGFtG4fUWndQ@mail.gmail.com>
Message-ID: <BANLkTinOVMeL5zja=_DMESxVgbuhdEW0og@mail.gmail.com>

You should check if you have two different compilers installed

ls /usr/bin/gcc* ?

2011/4/13 c cook <csecook at gmail.com>:
> I have Fedora Core 3 and?x86_64 architecture. But I don't understand why it
> appears to look for?/usr/lib/gcc/x86_64-redhat-linux/3.4.4/
> instead of /usr/bin/gcc
> Csaba
>
> On Wed, Apr 13, 2011 at 10:17 AM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>>
>> On Wed, Apr 13, 2011 at 7:30 AM, c cook <csecook at gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> I am trying to install numpy 1.6.0b2 version on Fedora and I am getting
>>> the following error:
>>>
>>> In file included from numpy/core/src/private/npy_config.h:4,
>>> ???????????????? from numpy/core/src/multiarray/common.c:8,
>>> ???????????????? from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
>>> build/src.linux-x86_64-2.4/numpy/core/include/numpy/config.h:34:1:
>>> warning: "HAVE_HYPOT" redefined
>>> In file included from /home/cse/Apps/Python-2.4/Include/Python.h:8,
>>> ???????????????? from numpy/core/src/multiarray/common.c:2,
>>> ???????????????? from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
>>> /home/cse/Apps/Python-2.4/pyconfig.h:232:1: warning: this is the location
>>> of the previous definition
>>> In file included from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:14:
>>> numpy/core/src/multiarray/arraytypes.c.src: In function `VOID_getitem':
>>> numpy/core/src/multiarray/arraytypes.c.src:615: warning: label `finish'
>>> defined but not used
>>> In file included from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:19:
>>> numpy/core/src/multiarray/descriptor.c: In function `descr_repeat':
>>> numpy/core/src/multiarray/descriptor.c:2840: warning: long int format,
>>> different type arg (arg 3)
>>> In file included from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:27:
>>> numpy/core/src/multiarray/methods.c: In function `NpyArg_ParseKeywords':
>>> numpy/core/src/multiarray/methods.c:41: warning: passing arg 3 of
>>> `PyArg_VaParseTupleAndKeywords' discards qualifiers from pointer target type
>>> In file included from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:28:
>>> numpy/core/src/multiarray/convert_datatype.c: In function
>>> `min_scalar_type_num':
>>> numpy/core/src/multiarray/convert_datatype.c:774: warning: comparison is
>>> always true due to limited range of data type
>>> numpy/core/src/multiarray/descriptor.c: In function
>>> `_convert_divisor_to_multiple':
>>> numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used
>>> uninitialized in this function
>>> numpy/core/src/multiarray/einsum.c.src: In function
>>> `float_sum_of_products_contig_outstride0_one':
>>> numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn:
>>> (insn:HI 440 228 481 14
>>> /usr/lib/gcc/x86_64-redhat-linux/3.4.4/include/xmmintrin.h:915 (set (reg:SF
>>> 148)
>>> ??????? (vec_select:SF (reg/v:V4SF 67 [ accum_sse ])
>>> ??????????? (parallel [
>>> ??????????????????? (const_int 0 [0x0])
>>> ??????????????? ]))) -1 (insn_list 213 (nil))
>>> ??? (nil))
>>> numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: in
>>> extract_insn, at recog.c:2083
>>> Please submit a full bug report,
>>> with preprocessed source if appropriate.
>>> See <URL:http://bugzilla.redhat.com/bugzilla> for instructions.
>>> Preprocessed source stored into /tmp/cck42rVN.out file, please attach
>>> this to your bugreport.
>>> In file included from numpy/core/src/private/npy_config.h:4,
>>> ???????????????? from numpy/core/src/multiarray/common.c:8,
>>> ???????????????? from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
>>> build/src.linux-x86_64-2.4/numpy/core/include/numpy/config.h:34:1:
>>> warning: "HAVE_HYPOT" redefined
>>> In file included from /home/cse/Apps/Python-2.4/Include/Python.h:8,
>>> ???????????????? from numpy/core/src/multiarray/common.c:2,
>>> ???????????????? from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:8:
>>> /home/cse/Apps/Python-2.4/pyconfig.h:232:1: warning: this is the location
>>> of the previous definition
>>> In file included from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:14:
>>> numpy/core/src/multiarray/arraytypes.c.src: In function `VOID_getitem':
>>> numpy/core/src/multiarray/arraytypes.c.src:615: warning: label `finish'
>>> defined but not used
>>> In file included from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:19:
>>> numpy/core/src/multiarray/descriptor.c: In function `descr_repeat':
>>> numpy/core/src/multiarray/descriptor.c:2840: warning: long int format,
>>> different type arg (arg 3)
>>> In file included from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:27:
>>> numpy/core/src/multiarray/methods.c: In function `NpyArg_ParseKeywords':
>>> numpy/core/src/multiarray/methods.c:41: warning: passing arg 3 of
>>> `PyArg_VaParseTupleAndKeywords' discards qualifiers from pointer target type
>>> In file included from
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c:28:
>>> numpy/core/src/multiarray/convert_datatype.c: In function
>>> `min_scalar_type_num':
>>> numpy/core/src/multiarray/convert_datatype.c:774: warning: comparison is
>>> always true due to limited range of data type
>>> numpy/core/src/multiarray/descriptor.c: In function
>>> `_convert_divisor_to_multiple':
>>> numpy/core/src/multiarray/descriptor.c:606: warning: 'q' might be used
>>> uninitialized in this function
>>> numpy/core/src/multiarray/einsum.c.src: In function
>>> `float_sum_of_products_contig_outstride0_one':
>>> numpy/core/src/multiarray/einsum.c.src:852: error: unrecognizable insn:
>>> (insn:HI 440 228 481 14
>>> /usr/lib/gcc/x86_64-redhat-linux/3.4.4/include/xmmintrin.h:915 (set (reg:SF
>>> 148)
>>> ??????? (vec_select:SF (reg/v:V4SF 67 [ accum_sse ])
>>> ??????????? (parallel [
>>> ??????????????????? (const_int 0 [0x0])
>>> ??????????????? ]))) -1 (insn_list 213 (nil))
>>> ??? (nil))
>>> numpy/core/src/multiarray/einsum.c.src:852: internal compiler error: in
>>> extract_insn, at recog.c:2083
>>> Please submit a full bug report,
>>> with preprocessed source if appropriate.
>>> See <URL:http://bugzilla.redhat.com/bugzilla> for instructions.
>>> Preprocessed source stored into /tmp/cck42rVN.out file, please attach
>>> this to your bugreport.
>>> error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
>>> -Wstrict-prototypes -fPIC -Inumpy/core/include
>>> -Ibuild/src.linux-x86_64-2.4/numpy/core/include/numpy
>>> -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core
>>> -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath
>>> -Inumpy/core/include -I/home/cse/Apps/Python-2.4/Include
>>> -I/home/cse/Apps/Python-2.4
>>> -Ibuild/src.linux-x86_64-2.4/numpy/core/src/multiarray
>>> -Ibuild/src.linux-x86_64-2.4/numpy/core/src/umath -c
>>> numpy/core/src/multiarray/multiarraymodule_onefile.c -o
>>> build/temp.linux-x86_64-2.4/multiarraymodule_onefile.o" failed with exit
>>> status 1
>>>
>>>
>>> Any idea what is the problem?
>>>
>>
>> I'll guess the combination of gcc-3.4.4, which is six years old, and the
>> sse instructions. It might be worth trying gcc-3.4.6 if you can't upgrade
>> the compiler outright.? Out of curiousity, what architecture and fedora
>> version are you using?
>>
>> Chuck
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


From mwwiebe at gmail.com  Wed Apr 13 18:34:06 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Wed, 13 Apr 2011 15:34:06 -0700
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTi=ffexSYdQvzq1iVwPoEEsOSwsP4g@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
	<BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
	<BANLkTinSfx34_U9HUw50BbGDbQEHSZdNWg@mail.gmail.com>
	<BANLkTik-Q+7U5dWWHWcmQWsn1XkiWqNOkA@mail.gmail.com>
	<BANLkTik60yEaFfsynmZD5mCbxu3CjbiJYg@mail.gmail.com>
	<BANLkTik1EEmi3APVWNmT5MXsQwN6DBwxUw@mail.gmail.com>
	<BANLkTikpS=bwSACzMqz9RvVH_YB72utSsg@mail.gmail.com>
	<BANLkTi=ffexSYdQvzq1iVwPoEEsOSwsP4g@mail.gmail.com>
Message-ID: <BANLkTimsdZT9bTRBOB-7XtXsaSv2++xW9Q@mail.gmail.com>

On Tue, Apr 12, 2011 at 11:51 AM, Mark Wiebe <mwwiebe at gmail.com> wrote:

> <snip>
>
here's the rule for a set of arbitrary arrays (not necessarily just 2):
>
> - if all the arrays are scalars, do type promotion on the types as is
> - otherwise, do type promotion on min_scalar_type(a) of each array a
>
> The function min_scalar_type returns the array type if a has >= 1
> dimensions, or the smallest type of the same kind (allowing int->uint in the
> case of positive-valued signed integers) to which the value can be cast
> without overflow if a has 0 dimensions.
>
> The promote_types function used for the type promotion is symmetric and
> associative, so the result won't change when shuffling the inputs. There's a
> bit of a wrinkle in the implementation to handle the fact that the uint type
> values aren't a strict subset of the same-sized int type values, but
> otherwise this is what happens.
>
>
> https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/convert_datatype.c#L1075
>
> The change I'm proposing is to modify this as follows:
>
> - if all the arrays are scalars, do type promotion on the types as is
> - if the maximum kind of all the scalars is > the maximum kind of all the
> arrays, do type promotion on the types as is
> - otherwise, do type promotion on min_scalar_type(a) of each array a
>
> One case where this may not capture a possible desired semantics is
> [complex128 scalar] * [float32 array] -> [complex128]. In this case
> [complex64] may be desired. This is directly analogous to the original
> [float64 scalar] * [int8 array], however, and in the latter case it's clear
> a float64 should result.
>

I've implemented what I suggested, and improved the documentation to better
explain what's going on. One thing I adjusted slightly is instead of using
the existing kinds, I used three categories: boolean, integer (int/uint),
and floating point (float/complex). This way, we get [float32 array] + 0j
producing a [complex64 array] instead of a [complex128 array], while still
fixing the original regression.

Please review my patch, thanks in advance!

https://github.com/numpy/numpy/pull/73

Cheers,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110413/5d5838c0/attachment.html>

From mwwiebe at gmail.com  Wed Apr 13 18:48:04 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Wed, 13 Apr 2011 15:48:04 -0700
Subject: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1
 and Numpy > 1.5.1
In-Reply-To: <BANLkTimsdZT9bTRBOB-7XtXsaSv2++xW9Q@mail.gmail.com>
References: <BANLkTi=q1YecRk5rYcpmRm9U=d+dR61V9A@mail.gmail.com>
	<BANLkTinHZ7baFpKZCD4TpDigzsdfTf8y7Q@mail.gmail.com>
	<BANLkTimPGCcnZ=wVBoyL2v2pRR6PbjDA3w@mail.gmail.com>
	<BANLkTinr7h_5_ShuoKXghyhSLw-GqYU-1Q@mail.gmail.com>
	<63E21335-D501-4B68-80FE-762E9EC77BF7@enthought.com>
	<BANLkTi=nZAwvXmgUbTNPLDQmBhJLAhVUaA@mail.gmail.com>
	<BANLkTi=qwDrLz0tEnTbBkSrFa60mU_y73g@mail.gmail.com>
	<BANLkTimOqPZz_FgR-Z1tUS9oAAGuNgzgiQ@mail.gmail.com>
	<BANLkTinSfx34_U9HUw50BbGDbQEHSZdNWg@mail.gmail.com>
	<BANLkTik-Q+7U5dWWHWcmQWsn1XkiWqNOkA@mail.gmail.com>
	<BANLkTik60yEaFfsynmZD5mCbxu3CjbiJYg@mail.gmail.com>
	<BANLkTik1EEmi3APVWNmT5MXsQwN6DBwxUw@mail.gmail.com>
	<BANLkTikpS=bwSACzMqz9RvVH_YB72utSsg@mail.gmail.com>
	<BANLkTi=ffexSYdQvzq1iVwPoEEsOSwsP4g@mail.gmail.com>
	<BANLkTimsdZT9bTRBOB-7XtXsaSv2++xW9Q@mail.gmail.com>
Message-ID: <BANLkTika8O0fcn-F29C3ntnCGUc7-s1RdQ@mail.gmail.com>

On Wed, Apr 13, 2011 at 3:34 PM, Mark Wiebe <mwwiebe at gmail.com> wrote:

> On Tue, Apr 12, 2011 at 11:51 AM, Mark Wiebe <mwwiebe at gmail.com> wrote:
>
>> <snip>
>>
>  here's the rule for a set of arbitrary arrays (not necessarily just 2):
>>
>> - if all the arrays are scalars, do type promotion on the types as is
>> - otherwise, do type promotion on min_scalar_type(a) of each array a
>>
>> The function min_scalar_type returns the array type if a has >= 1
>> dimensions, or the smallest type of the same kind (allowing int->uint in the
>> case of positive-valued signed integers) to which the value can be cast
>> without overflow if a has 0 dimensions.
>>
>> The promote_types function used for the type promotion is symmetric and
>> associative, so the result won't change when shuffling the inputs. There's a
>> bit of a wrinkle in the implementation to handle the fact that the uint type
>> values aren't a strict subset of the same-sized int type values, but
>> otherwise this is what happens.
>>
>>
>> https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/convert_datatype.c#L1075
>>
>> The change I'm proposing is to modify this as follows:
>>
>> - if all the arrays are scalars, do type promotion on the types as is
>> - if the maximum kind of all the scalars is > the maximum kind of all the
>> arrays, do type promotion on the types as is
>> - otherwise, do type promotion on min_scalar_type(a) of each array a
>>
>>  One case where this may not capture a possible desired semantics is
>> [complex128 scalar] * [float32 array] -> [complex128]. In this case
>> [complex64] may be desired. This is directly analogous to the original
>> [float64 scalar] * [int8 array], however, and in the latter case it's clear
>> a float64 should result.
>>
>
> I've implemented what I suggested, and improved the documentation to better
> explain what's going on. One thing I adjusted slightly is instead of using
> the existing kinds, I used three categories: boolean, integer (int/uint),
> and floating point (float/complex). This way, we get [float32 array] + 0j
> producing a [complex64 array] instead of a [complex128 array], while still
> fixing the original regression.
>
> Please review my patch, thanks in advance!
>
> https://github.com/numpy/numpy/pull/73
>
> Cheers,
> Mark
>

I forgot to mention that Travis was right, and it wasn't necessary to detect
that the ufunc is a binary operator. I think at some point splitting out
binary operators as a special case of ufuncs is desirable for performance
reasons, but for now this patch is much simpler.

-Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110413/ab815fe1/attachment.html>

From wesmckinn at gmail.com  Wed Apr 13 21:39:49 2011
From: wesmckinn at gmail.com (Wes McKinney)
Date: Wed, 13 Apr 2011 21:39:49 -0400
Subject: [Numpy-discussion] Data standardizing
In-Reply-To: <BANLkTikH--ZOrLz2P8SWjpS0JmpOU=cyLQ@mail.gmail.com>
References: <BANLkTikXbp9kV4b8-Pjdw5W+L4yJ6TvgqA@mail.gmail.com>
	<BANLkTikH--ZOrLz2P8SWjpS0JmpOU=cyLQ@mail.gmail.com>
Message-ID: <BANLkTimwsuhyCfZ3HHZqyHZzsnOjzz2pgQ@mail.gmail.com>

On Wed, Apr 13, 2011 at 9:50 AM, Jonathan Rocher <jrocher at enthought.com> wrote:
> Hi,
>
> I assume you have this data in a txt file, correct? You can load up all of
> it in a numpy array using
> import numpy as np
> data = np.loadtxt("climat_file.txt", skiprows = 1)
>
> Then you can compute the mean you want by taking it on a slice of the data
> array. For example, if you want to compute the mean of your data in Jan for
> 1950-1970 (say including 1970)
> mean1950_1970 = data[1950:1971,1].mean()
>
> Then the std deviation you want could be computed using
> my_std = np.sqrt(np.mean((data[:,1]-mean1950_1970)**2))
>
> Hope this helps,
> Jonathan
>
> On Tue, Apr 12, 2011 at 1:48 PM, Climate Research <climateforu at gmail.com>
> wrote:
>>
>> Hi
>> I am purely new to python and numpy..? I am using python for doing
>> statistical calculations to Climate data..
>>
>> I? have? a? data set in the following format..
>>
>> Year? Jan??? ? feb?????? Mar??? Apr................. ? Dec
>> 1900? 1000??? 1001?????? ,??????? ,???????????????????????? ,
>> 1901? 1011??? 1012?????? ,??????? ,???????????????????????? ,
>> 1902? 1009??? 1007?????? ,????????????????????????????????? ,
>> ,,,,??????? ,?????????? '??????? ? ,??????? ,???????????????????????? ,
>> ,,,,??????? ,?????????? ,
>> 2010? 1008??? 1002?????? ,??????? ,???????????????????????? ,
>>
>> I actually want to standardize each of these values with corresponding
>> standard deviations for? each monthly data column..
>> I have found out the standard deviations for each column..? but now i need
>> to? find the standared deviation? only for a prescribed mean value
>> ie,? when i am finding the standared deviation for the January data
>> column..? the mean should be calculated only for the january data, say from
>> 1950-1970. With this mean? i? want to calculate the SD? for entire column.
>> Any help will be appreciated..
>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
> Jonathan Rocher, PhD
> Scientific software developer
> Enthought, Inc.
> jrocher at enthought.com
> 1-512-536-1057
> http://www.enthought.com
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>

To standardize the data over each column you'll want to do:

(data - data.mean(axis=0)) / data.std(axis=0, ddof=1)

Note the broadcasting behavior of the (matrix - vector) operation--see
NumPy documentation for more details. The ddof=1 is there to give you
the (unbiased) sample standard deviation.

<shameless plug>

If you're looking for data structures to carry around your metadata
(dates and month labels), look to pandas (my project:
http://pandas.sourceforge.net/) or larry
(http://larry.sourceforge.net/).

</shameless plug>

- Wes


From pgmdevlist at gmail.com  Thu Apr 14 05:39:47 2011
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 14 Apr 2011 11:39:47 +0200
Subject: [Numpy-discussion] Data standardizing
In-Reply-To: <BANLkTikXbp9kV4b8-Pjdw5W+L4yJ6TvgqA@mail.gmail.com>
References: <BANLkTikXbp9kV4b8-Pjdw5W+L4yJ6TvgqA@mail.gmail.com>
Message-ID: <217427D9-B101-4372-AE40-D57DDC9FE04A@gmail.com>


On Apr 12, 2011, at 8:48 PM, Climate Research wrote:

> Hi 
> I am purely new to python and numpy..  I am using python for doing statistical calculations to Climate data..

Check the scikits.timeseries and scikits.hydroclimpy as well, they have routines for that very purpose.

From russel at appliedminds.com  Thu Apr 14 21:32:27 2011
From: russel at appliedminds.com (russel)
Date: Thu, 14 Apr 2011 18:32:27 -0700
Subject: [Numpy-discussion] dtype itemsize stored in 32 bit value?
Message-ID: <4DA7A02B.7000903@appliedminds.com>

Is this a bug?

In [81]: a = np.dtype([('field', 'u1', (2**31,))])

In [82]: a.itemsize
Out[82]: -2147483648



From arrigo.benedetti at gmail.com  Fri Apr 15 20:59:01 2011
From: arrigo.benedetti at gmail.com (arrigo)
Date: Fri, 15 Apr 2011 17:59:01 -0700 (PDT)
Subject: [Numpy-discussion] Cython error with complex numpy array
Message-ID: <f4da60d7-3e07-4ccf-83b9-fa84dfacb5cf@f11g2000vbx.googlegroups.com>

I am trying to use Cython to speed up some calculation with complex
numpy arrays and I am getting this error:

C:>python setup_post.py build_ext --inplace
running build_ext
cythoning SVDc_post.pyx to SVDc_post.c

Error converting Pyrex file to C:
------------------------------------------------------------
...

# static inline Real csq(cComplex x) { return creal(x*conj(x)); }
cdef inline double csq(x): return real(x*x.conjugate())


def SVDc2(int m, int n,
^
------------------------------------------------------------

C:\SVDc_post.pyx:11:0: Compiler crash in IntroduceBufferAuxiliaryVars

ModuleNode.body = StatListNode(SVDc_post.pyx:2:0)
StatListNode.stats[4] = StatListNode(SVDc_post.pyx:11:0)
StatListNode.stats[0] = DefNode(SVDc_post.pyx:11:0,
   modifiers = [...]/0,
   name = u'SVDc2',
   num_required_args = 7,
   reqd_kw_flags_cname = '0')



The code is below (I removed all the code not relevant to this error):



import numpy as np
cimport numpy as np

ctypedef np.float64_t dtype_t

cdef inline double csq(x): return real(x*x.conjugate())


def SVDc2(int m, int n,
              np.ndarray[np.complex128_t, ndim = 2] Ao,
              np.ndarray[np.float64_t, ndim = 1] d,
              np.ndarray[np.complex128_t, ndim = 2] Vo,
              np.ndarray[np.complex128_t, ndim = 2] Wo,
              int sort):

   cdef dtype_t EPS = 1.5e-31
   cdef dtype_t DBL_EPS = 2.2204460492503131e-16

   cdef int maxsweeps = 50
   cdef int err = True

   cdef int     rev = ((m - n) >> 15) & 1
   cdef int     nm = min(m, n)
   cdef int     nx = max(m, n)
   cdef dtype_t red = .01 / (nx * nx * nx * nx)
   cdef int     ldV = nx
   cdef int     ldW = nx
   cdef int     ldA = nx
   cdef dtype_t thresh = 0.0

   cdef np.ndarray[np.complex128_t, ndim=3] VW = zeros((3, nx, nx),
dtype=complex128_t)

   cdef np.ndarray[np.int32, ndim=1] ppi = empty(nm, dtype=int32)

   for p in range(0, nx): VW[0, p, p] = 1
   for p in range(0, nx): VW[1, p, p] = 1

   if rev:
       for q in range(0, m):
           for p in range(0, n):
               VW[2, p, q] = Ao[q, p]
   else:
       for q in range(0,m):
           for p in range(0,n):
               VW[2, q, p] = Ao[q, p]


   return


I was wondering if I need a special setup.py when using numpy arrays.
Any ideas?

Thanks,

-Arrigo


From gandalf at shopzeus.com  Sat Apr 16 15:08:55 2011
From: gandalf at shopzeus.com (Laszlo Nagy)
Date: Sat, 16 Apr 2011 21:08:55 +0200
Subject: [Numpy-discussion] Beginner's question
Message-ID: <4DA9E947.7060209@shopzeus.com>


   Hi All,

I have this example program:

import numpy as np
import numpy.random as rnd

def dim_weight(X):
     weights = X[0]
     volumes = X[1]*X[2]*X[3]
     res = np.empty(len(volumes), dtype=np.double)
     for i,v in enumerate(volumes):
         if v>5184:
             res[i] = v/194.0
         else:
             res[i] = weights[i]
     return res


# TEST
N = 10
X = rnd.randint( 1,25, (4,N))
print dim_weight(X)

I want to implement the dim_weight() function effectively. But I'm not 
sure how to construct a numpy expression that does the same think for me.

I got to this point:

def dim_weight(X):
     weights = X[0]
     volumes = X[1]*X[2]*X[3]
     dweights = volumes/194.0
     return ( weights[weights>5184] , dweights[weights<=5184] ) # ???

I don't know how to preserve the order of the elements and return the 
result in one array. Maybe I need to do create a matrix multiply matrix? 
But don't know how.

Thanks,

    Laszlo


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.



From ben.root at ou.edu  Sat Apr 16 15:12:29 2011
From: ben.root at ou.edu (Benjamin Root)
Date: Sat, 16 Apr 2011 14:12:29 -0500
Subject: [Numpy-discussion] Beginner's question
In-Reply-To: <4DA9E947.7060209@shopzeus.com>
References: <4DA9E947.7060209@shopzeus.com>
Message-ID: <BANLkTi=cgcBvUfrF4McJMgHX+Te1AZkHNw@mail.gmail.com>

On Saturday, April 16, 2011, Laszlo Nagy <gandalf at shopzeus.com> wrote:
>
>  ? Hi All,
>
> I have this example program:
>
> import numpy as np
> import numpy.random as rnd
>
> def dim_weight(X):
>  ? ? weights = X[0]
>  ? ? volumes = X[1]*X[2]*X[3]
>  ? ? res = np.empty(len(volumes), dtype=np.double)
>  ? ? for i,v in enumerate(volumes):
>  ? ? ? ? if v>5184:
>  ? ? ? ? ? ? res[i] = v/194.0
>  ? ? ? ? else:
>  ? ? ? ? ? ? res[i] = weights[i]
>  ? ? return res
>
>
> # TEST
> N = 10
> X = rnd.randint( 1,25, (4,N))
> print dim_weight(X)
>
> I want to implement the dim_weight() function effectively. But I'm not
> sure how to construct a numpy expression that does the same think for me.
>
> I got to this point:
>
> def dim_weight(X):
>  ? ? weights = X[0]
>  ? ? volumes = X[1]*X[2]*X[3]
>  ? ? dweights = volumes/194.0
>  ? ? return ( weights[weights>5184] , dweights[weights<=5184] ) # ???
>
> I don't know how to preserve the order of the elements and return the
> result in one array. Maybe I need to do create a matrix multiply matrix?
> But don't know how.
>
> Thanks,
>
>  ? ?Laszlo
>
>

I think you want np.where().

Ben Root


From forrest.bao at gmail.com  Sat Apr 16 23:39:06 2011
From: forrest.bao at gmail.com (Forrest Sheng Bao)
Date: Sat, 16 Apr 2011 22:39:06 -0500
Subject: [Numpy-discussion] an array become a matrix after an element
	assignment
Message-ID: <BANLkTikx8bsGFh1mvJr-b6a0+SRZ8f9TNw@mail.gmail.com>

hi,

I encountered a very weird problem tonight. First I created an array and did
element-wise multiplication on it:

>>> a=zeros((3,3)) + eye(3)
>>> a*a
array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  0.,  1.]])

Then i assigned a value to one of the element:

>>> a[1,2]=4

Finally, the '*' operator was interpreted as a matrix multiplication

>>> a*a
array([[  1.,   0.,   0.],
       [  0.,   1.,  16.],
       [  0.,   0.,   1.]])

The return types of zeros() and eye() are both arrays.

The even weird thing happened later. When I do the same thing in another
program (obtaining an adjacency matrix and computing graph powers), my
"matrix" was considered as an ''array."

So, why could an element assignment turn an array into a matrix? And
sometimes not?

Cheers, Forrest
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110416/1400fe7b/attachment.html>

From njs at pobox.com  Sat Apr 16 23:47:59 2011
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 16 Apr 2011 20:47:59 -0700
Subject: [Numpy-discussion] an array become a matrix after an element
	assignment
In-Reply-To: <BANLkTikx8bsGFh1mvJr-b6a0+SRZ8f9TNw@mail.gmail.com>
References: <BANLkTikx8bsGFh1mvJr-b6a0+SRZ8f9TNw@mail.gmail.com>
Message-ID: <BANLkTikrvMS_yZoibdfpWpLDaaMg6X4avQ@mail.gmail.com>

On Sat, Apr 16, 2011 at 8:39 PM, Forrest Sheng Bao
<forrest.bao at gmail.com> wrote:
>>>> a=zeros((3,3)) + eye(3)
>>>> a*a
> array([[ 1.,? 0.,? 0.],
> ?????? [ 0.,? 1.,? 0.],
> ?????? [ 0.,? 0.,? 1.]])
>
> Then i assigned a value to one of the element:
>
>>>> a[1,2]=4
>
> Finally, the '*' operator was interpreted as a matrix multiplication
>
>>>> a*a
> array([[? 1.,?? 0.,?? 0.],
> ?????? [? 0.,?? 1.,? 16.],
> ?????? [? 0.,?? 0.,?? 1.]])

That looks like element-wise multiplication to me? (Matrix
multiplication would have given you an 8 where you have a 16.)

-- Nathaniel


From forrest.bao at gmail.com  Sat Apr 16 23:51:52 2011
From: forrest.bao at gmail.com (Forrest Sheng Bao)
Date: Sat, 16 Apr 2011 22:51:52 -0500
Subject: [Numpy-discussion] an array become a matrix after an element
	assignment
In-Reply-To: <BANLkTikrvMS_yZoibdfpWpLDaaMg6X4avQ@mail.gmail.com>
References: <BANLkTikx8bsGFh1mvJr-b6a0+SRZ8f9TNw@mail.gmail.com>
	<BANLkTikrvMS_yZoibdfpWpLDaaMg6X4avQ@mail.gmail.com>
Message-ID: <BANLkTim-M5-4ftYzL17G3Bayb9vMFV4MNg@mail.gmail.com>

Oh yeah, my bad. Forget about it. I guess I was too tired.

Cheers, Forrest

On Sat, Apr 16, 2011 at 10:47 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Sat, Apr 16, 2011 at 8:39 PM, Forrest Sheng Bao
> <forrest.bao at gmail.com> wrote:
> >>>> a=zeros((3,3)) + eye(3)
> >>>> a*a
> > array([[ 1.,  0.,  0.],
> >        [ 0.,  1.,  0.],
> >        [ 0.,  0.,  1.]])
> >
> > Then i assigned a value to one of the element:
> >
> >>>> a[1,2]=4
> >
> > Finally, the '*' operator was interpreted as a matrix multiplication
> >
> >>>> a*a
> > array([[  1.,   0.,   0.],
> >        [  0.,   1.,  16.],
> >        [  0.,   0.,   1.]])
>
> That looks like element-wise multiplication to me? (Matrix
> multiplication would have given you an 8 where you have a 16.)
>
> -- Nathaniel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110416/0115e243/attachment.html>

From njs at pobox.com  Sun Apr 17 00:04:21 2011
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 16 Apr 2011 21:04:21 -0700
Subject: [Numpy-discussion] an array become a matrix after an element
	assignment
In-Reply-To: <BANLkTim-M5-4ftYzL17G3Bayb9vMFV4MNg@mail.gmail.com>
References: <BANLkTikx8bsGFh1mvJr-b6a0+SRZ8f9TNw@mail.gmail.com>
	<BANLkTikrvMS_yZoibdfpWpLDaaMg6X4avQ@mail.gmail.com>
	<BANLkTim-M5-4ftYzL17G3Bayb9vMFV4MNg@mail.gmail.com>
Message-ID: <BANLkTimvPgB1C3OPCj4d3Bu0a2ce3Q_D9A@mail.gmail.com>

Happens to me all the time...

On Sat, Apr 16, 2011 at 8:51 PM, Forrest Sheng Bao
<forrest.bao at gmail.com> wrote:
> Oh yeah, my bad. Forget about it. I guess I was too tired.
>
> Cheers, Forrest
>
> On Sat, Apr 16, 2011 at 10:47 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Sat, Apr 16, 2011 at 8:39 PM, Forrest Sheng Bao
>> <forrest.bao at gmail.com> wrote:
>> >>>> a=zeros((3,3)) + eye(3)
>> >>>> a*a
>> > array([[ 1.,? 0.,? 0.],
>> > ?????? [ 0.,? 1.,? 0.],
>> > ?????? [ 0.,? 0.,? 1.]])
>> >
>> > Then i assigned a value to one of the element:
>> >
>> >>>> a[1,2]=4
>> >
>> > Finally, the '*' operator was interpreted as a matrix multiplication
>> >
>> >>>> a*a
>> > array([[? 1.,?? 0.,?? 0.],
>> > ?????? [? 0.,?? 1.,? 16.],
>> > ?????? [? 0.,?? 0.,?? 1.]])
>>
>> That looks like element-wise multiplication to me? (Matrix
>> multiplication would have given you an 8 where you have a 16.)
>>
>> -- Nathaniel
>
>


From ralf.gommers at googlemail.com  Sun Apr 17 09:53:43 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sun, 17 Apr 2011 15:53:43 +0200
Subject: [Numpy-discussion] Unable to clone git repository
In-Reply-To: <loom.20110408T142852-311@post.gmane.org>
References: <loom.20110408T142852-311@post.gmane.org>
Message-ID: <BANLkTi=PUvb05ML+q4u5myEDwJGUdL-O7A@mail.gmail.com>

On Fri, Apr 8, 2011 at 2:43 PM, Dave <dave.hirschfeld at gmail.com> wrote:
> Disclaimer: I'm on windows (Win7 x64)
>
> Following the instructions at:
>
> http://docs.scipy.org/doc/numpy/dev/gitwash/following_latest.html
>
> I got the following (rather unhelpful) error message:
>
> C:\dev\src>git clone git://github.com/numpy/numpy.git
> Cloning into numpy...
> github.com[0: 207.97.227.239]: errno=No error
> fatal: unable to connect a socket (No error)
>
>
> After a bit of googling I got the vague impression that this may possibly
> be to do with work blocking access? Fortunately I was able clone the read
> only https url - https://github.com/numpy/numpy
>
> C:\dev\src>git clone https://github.com/numpy/numpy.git
> Cloning into numpy...
> remote: Counting objects: 60461, done.
> remote: Compressing objects: 100% (13476/13476), done.
> remote: Total 60461 (delta 47928), reused 58235 (delta 46157)
> Receiving objects: 100% (60461/60461), 19.41 MiB | 2.66 MiB/s, done.
> Resolving deltas: 100% (47928/47928), done.
>
> I would suggest that since they are "the instructions if you just want to
> follow the latest NumPy source, but you don?t need to do any development"
> that providing the https url as an alternative would be helpful. Perhaps
> something like:
>
>
> From the command line:
>
> ? ?git clone git://github.com/numpy/numpy.git
>
> You should now have a copy of the code tree in the new `numpy` directory.
> If this doesn't work you can try the alternative read-only url:
>
> ? ?git clone https://github.com/numpy/numpy.git

Done in the doc wiki, thanks for the suggestion.

Ralf


From ralf.gommers at googlemail.com  Sun Apr 17 11:55:37 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sun, 17 Apr 2011 17:55:37 +0200
Subject: [Numpy-discussion] status for 1.6.0
Message-ID: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>

Hi,

The list of open issues for 1.6.0 is down to a handful:

- f2py segfault on Ubuntu reported by David (David, did you get any
further with this?)
- #1801: test_noncentral_f failure on win64
- #1798: type promotion regression (needs review,
https://github.com/numpy/numpy/pull/73)
- API docs for new parameters to ufuncs (full list: 'casting', 'out',
'order', 'subok', 'dtype'),
- API docs, some docstrings and update of NEP for datetime

I'm planning to release the first release candidate next weekend if
these are solved.  As far as I know no one is working on the doc
issues right now. It would be great if someone could step up and do
this.

Thanks,
Ralf


From kbasye1 at jhu.edu  Sun Apr 17 14:01:11 2011
From: kbasye1 at jhu.edu (Ken Basye)
Date: Sun, 17 Apr 2011 14:01:11 -0400
Subject: [Numpy-discussion] NumPy-Discussion Digest, Vol 55, Issue 42
In-Reply-To: <mailman.13.1303059603.32290.numpy-discussion@scipy.org>
References: <mailman.13.1303059603.32290.numpy-discussion@scipy.org>
Message-ID: <4DAB2AE7.60906@jhu.edu>

I think this does what you want:

def dim_weight(x):
     weights = x[0]
     volumes = x[1]*x[2]*x[3]
     return np.where(volumes>5184, volumes / 194.0, weights)

Best,
     Ken


On 4/17/11 1:00 PM, Laszlo Nagy <gandalf at shopzeus.com> wrote:
> Message: 1
> Date: Sat, 16 Apr 2011 21:08:55 +0200
> From: Laszlo Nagy<gandalf at shopzeus.com>
> Subject: [Numpy-discussion] Beginner's question
> To: Discussion of Numerical Python<numpy-discussion at scipy.org>
> Message-ID:<4DA9E947.7060209 at shopzeus.com>
> Content-Type: text/plain; charset=ISO-8859-2; format=flowed
>
>
>     Hi All,
>
> I have this example program:
>
> import numpy as np
> import numpy.random as rnd
>
> def dim_weight(X):
>       weights = X[0]
>       volumes = X[1]*X[2]*X[3]
>       res = np.empty(len(volumes), dtype=np.double)
>       for i,v in enumerate(volumes):
>           if v>5184:
>               res[i] = v/194.0
>           else:
>               res[i] = weights[i]
>       return res
>
>
> # TEST
> N = 10
> X = rnd.randint( 1,25, (4,N))
> print dim_weight(X)
>
> I want to implement the dim_weight() function effectively. But I'm not
> sure how to construct a numpy expression that does the same think for me.
>
> I got to this point:
>
> def dim_weight(X):
>       weights = X[0]
>       volumes = X[1]*X[2]*X[3]
>       dweights = volumes/194.0
>       return ( weights[weights>5184] , dweights[weights<=5184] ) # ???
>
> I don't know how to preserve the order of the elements and return the
> result in one array. Maybe I need to do create a matrix multiply matrix?
> But don't know how.
>
> Thanks,
>
>      Laszlo
>


From cgohlke at uci.edu  Sun Apr 17 20:36:15 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Sun, 17 Apr 2011 17:36:15 -0700
Subject: [Numpy-discussion] status for 1.6.0
In-Reply-To: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
References: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
Message-ID: <4DAB877F.3010109@uci.edu>



On 4/17/2011 8:55 AM, Ralf Gommers wrote:
> Hi,
>
> The list of open issues for 1.6.0 is down to a handful:
>
> - f2py segfault on Ubuntu reported by David (David, did you get any
> further with this?)
> - #1801: test_noncentral_f failure on win64
> - #1798: type promotion regression (needs review,
> https://github.com/numpy/numpy/pull/73)
> - API docs for new parameters to ufuncs (full list: 'casting', 'out',
> 'order', 'subok', 'dtype'),
> - API docs, some docstrings and update of NEP for datetime
>
> I'm planning to release the first release candidate next weekend if
> these are solved.  As far as I know no one is working on the doc
> issues right now. It would be great if someone could step up and do
> this.
>
> Thanks,
> Ralf


Hi Ralf,

I submitted a patch for issue #1801. All tests now pass on 
win-amd64-py3.2 with numpy 1.6.x from git.

Would it be possible to get the attached patch for scipy ticket #678 
<http://projects.scipy.org/scipy/ticket/678> into numpy 1.6? The 
fix/workaround for the scipy ODR failures is to change the ifort 
optimization flag from /O3 to /O1.

Christoph
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fcompiler_intel.diff
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110417/04063475/attachment.ksh>

From josef.pktd at gmail.com  Sun Apr 17 23:59:05 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 17 Apr 2011 23:59:05 -0400
Subject: [Numpy-discussion] assert and allclose
Message-ID: <BANLkTin=DHJCcJNDhu09eEugEcgJCVzedg@mail.gmail.com>

Some observations about allclose

I have a test case where I would like to impose absolute
(logical_)and relative errors

(Pdb) pdf_r
array([  2.97341655e-90,   3.68756271e-24,   2.01840159e-07,
         3.98942280e-01,   4.83941449e-01,   1.07981933e-01])
(Pdb) pdf_st
array([  3.09721604e-90,   3.69697466e-24,   2.01872148e-07,
         3.98942181e-01,   4.83941207e-01,   1.07982122e-01])
(Pdb) pdf_st - pdf_r
array([  1.23799491e-91,   9.41194691e-27,   3.19891525e-11,
        -9.95821123e-08,  -2.41764579e-07,   1.89015140e-07])
(Pdb) (pdf_st - pdf_r) / pdf_r
array([  4.16354347e-02,   2.55234897e-03,   1.58487551e-04,
        -2.49615338e-07,  -4.99574028e-07,   1.75043301e-06])

allclose just uses the sum of abolute and relative errors, which I
think is binding in the wrong way, for what I want.

(Pdb) np.allclose(pdf_st, pdf_r, rtol=1e-4, atol=1e-8)
True

the relative and absolute errors are to tight on one side and too
loose on the other side

(Pdb) np.allclose(pdf_st, pdf_r, rtol=1e-2, atol=0)
False
(Pdb) np.allclose(pdf_st, pdf_r, rtol=0, atol=1e-8)
False

tighter bounds for both relative and absolute errors if I impose both separately

(Pdb) np.allclose(pdf_st, pdf_r, rtol=0, atol=1e-6)
True
(Pdb) np.allclose(pdf_st, pdf_r, rtol=1e-1, atol=0)
True

Are there any alternatives to this? (or am I too tired)

(test cases are for statistical distributions, where all values are
smaller than one, and some values can be very small)

Josef


From cournape at gmail.com  Mon Apr 18 00:47:14 2011
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 18 Apr 2011 13:47:14 +0900
Subject: [Numpy-discussion] status for 1.6.0
In-Reply-To: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
References: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
Message-ID: <BANLkTinVTZ4S=SpSmavRQtmVWOcu+ctEyQ@mail.gmail.com>

On Mon, Apr 18, 2011 at 12:55 AM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
> Hi,
>
> The list of open issues for 1.6.0 is down to a handful:
>
> - f2py segfault on Ubuntu reported by David (David, did you get any
> further with this?)

I could reproduce on another machine (MAC OS X, different hardware), I
will take a look at it tonight.

cheers,

David


From mmueller at python-academy.de  Mon Apr 18 18:53:56 2011
From: mmueller at python-academy.de (=?ISO-8859-15?Q?Mike_M=FCller?=)
Date: Tue, 19 Apr 2011 00:53:56 +0200
Subject: [Numpy-discussion] [ANN] Courses in Colorado: "Introduction to
 Python and Python for Scientists and Engineers"
Message-ID: <4DACC104.1030802@python-academy.de>

Python Course in Golden, CO, USA
================================

Introduction to Python and Python for Scientists and Engineers
--------------------------------------------------------------

June 3 - 4, 2011      Introduction to Python
June 5, 2011          Python for Scientists and Engineers

Both courses can be booked individually or together.

Venue: Colorado School of Mines, Golden, CO (20 minutes west of Denver)
Trainer: Mike M?ller


Target Audience
---------------

The introductory course is designed for people with basic programming
background. Since it is a general introduction to Python it is suitable
for everybody interested in Python.

The scientist's course assumes a working knowledge of Python. You will be fine
if you take the two-day introduction before hand. The topics are of general
interest for scientists and engineers. Even though some examples come from the
groundwater modeling domain, they are easy to understand for people without
prior knowledge in this field.


About the Trainer
-----------------

Mike M?ller, has been teaching Python since 2004. He is the
founder of Python Academy and regularly gives open and in-house
Python courses as well as tutorials at PyCon US, OSCON, EuroSciPy
and PyCon Asia-Pacific.


More Information and Course Registration
----------------------------------------

http://igwmc.mines.edu/short-course/intro_python.html

--
Mike
mmueller at python-academy.de


From rowen at uw.edu  Mon Apr 18 18:08:22 2011
From: rowen at uw.edu (Russell E. Owen)
Date: Mon, 18 Apr 2011 15:08:22 -0700
Subject: [Numpy-discussion] Question about numpy.random, especially .poisson
Message-ID: <rowen-323487.15082218042011@news.gmane.org>

I stumbled across code that looks like this:

imageArr = # a 2-d array of floats
noiseArr = numpy.random.poisson(imageArr)

This works fine in numpy 1.5.1 and seems to do what I would hope: return 
an array of random ints whose "expectation of interval" is set by the 
corresponding element of the input array. Very nice!

However, I can't find any documentation supporting this usage. The 
standard help says:

   poisson(lam=1.0, size=None)
    
    Draw samples from a Poisson distribution.
    
    The Poisson distribution is the limit of the Binomial
    distribution for large N.
    
    Parameters
    ----------
    lam : float
        Expectation of interval, should be >= 0.
    size : int or tuple of ints, optional
        Output shape. If the given shape is, e.g., ``(m, n, k)``, then
        ``m * n * k`` samples are drawn.

Which suggest that Iam must be a scalar.

So... is the usage of passing in an array for Iam actually 
supported/safe to use?

And is there some general rule I could have used to predict that?

I'm not complaining -- quite the opposite. But I'd hate to code up 
something that uses an unsafe API, and I'd also like to be able to 
predict nifty features like this to get the most out of numpy.

-- Russell



From robert.kern at gmail.com  Mon Apr 18 22:58:06 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 18 Apr 2011 21:58:06 -0500
Subject: [Numpy-discussion] Question about numpy.random,
	especially .poisson
In-Reply-To: <rowen-323487.15082218042011@news.gmane.org>
References: <rowen-323487.15082218042011@news.gmane.org>
Message-ID: <BANLkTinQXvCe8p58LwJVuiHcr0hcjomSbg@mail.gmail.com>

On Mon, Apr 18, 2011 at 17:08, Russell E. Owen <rowen at uw.edu> wrote:
> I stumbled across code that looks like this:
>
> imageArr = # a 2-d array of floats
> noiseArr = numpy.random.poisson(imageArr)
>
> This works fine in numpy 1.5.1 and seems to do what I would hope: return
> an array of random ints whose "expectation of interval" is set by the
> corresponding element of the input array. Very nice!
>
> However, I can't find any documentation supporting this usage. The
> standard help says:
>
> ? poisson(lam=1.0, size=None)
>
> ? ?Draw samples from a Poisson distribution.
>
> ? ?The Poisson distribution is the limit of the Binomial
> ? ?distribution for large N.
>
> ? ?Parameters
> ? ?----------
> ? ?lam : float
> ? ? ? ?Expectation of interval, should be >= 0.
> ? ?size : int or tuple of ints, optional
> ? ? ? ?Output shape. If the given shape is, e.g., ``(m, n, k)``, then
> ? ? ? ?``m * n * k`` samples are drawn.
>
> Which suggest that Iam must be a scalar.
>
> So... is the usage of passing in an array for Iam actually
> supported/safe to use?

Yes, it is safe. All of the univariate distribution parameters will
broadcast against each other as they would with ufuncs. I don't think
we do it for multinomial and a couple of the other ones where the
semantics would have made my head hurt.

> And is there some general rule I could have used to predict that?

Just the one I stated, which is limited to numpy.random. It might even
be documented somewhere. Unfortunately, most of the individual methods
had their parameters documented before this capability was added.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
? -- Umberto Eco


From cournape at gmail.com  Tue Apr 19 02:54:18 2011
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 19 Apr 2011 15:54:18 +0900
Subject: [Numpy-discussion] status for 1.6.0
In-Reply-To: <BANLkTinVTZ4S=SpSmavRQtmVWOcu+ctEyQ@mail.gmail.com>
References: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
	<BANLkTinVTZ4S=SpSmavRQtmVWOcu+ctEyQ@mail.gmail.com>
Message-ID: <BANLkTimMSdGWBz6WCNRE5i0UAJhPM+o_hQ@mail.gmail.com>

On Mon, Apr 18, 2011 at 1:47 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Mon, Apr 18, 2011 at 12:55 AM, Ralf Gommers
> <ralf.gommers at googlemail.com> wrote:
>> Hi,
>>
>> The list of open issues for 1.6.0 is down to a handful:
>>
>> - f2py segfault on Ubuntu reported by David (David, did you get any
>> further with this?)
>
> I could reproduce on another machine (MAC OS X, different hardware), I
> will take a look at it tonight.

Ok, so the problem appears only when run through nosetests for a
in-place build of numpy. I think it is fair to say it is a bug
somewhere in nosetests or some weird import-related corner cases that
we don't care about.

If I run the same failing test for a "normal" install, I don't get the issue.

cheers,

David


From totonixsame at gmail.com  Tue Apr 19 20:01:06 2011
From: totonixsame at gmail.com (Thiago Franco de Moraes)
Date: Tue, 19 Apr 2011 21:01:06 -0300
Subject: [Numpy-discussion] Memmap with multiprocessing
Message-ID: <4DAE2242.9030506@gmail.com>

Hi all,

I'm having a error using memmap objects shared among processes created
by the multprocessing module. This error only happen in Windows with
numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac
OS X it doesn't happen. This error is demonstrated by this little
example script here https://gist.github.com/929168 , and the traceback
is bellow (between <traceback> tags):

<traceback>
Process Process-1:
Traceback (most recent call last):
   File "C:\Python26\Lib\multiprocessing\process.py", line 232, in 
_bootstrap
     self.run()
   File "C:\Python26\Lib\multiprocessing\process.py", line 88, in run
     self._target(*self._args, **self._kwargs)
   File "C:\Documents and Settings\phamorim\Desktop\test.py", line 7, in 
print_ma
trix
     print matrix
   File "C:\Python26\Lib\site-packages\numpy\core\numeric.py", line 
1379, in arra
y_str
     return array2string(a, max_line_width, precision, suppress_small, ' 
', "", s
tr)
   File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 
309, in ar
ray2string
     separator, prefix)
   File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 
189, in _a
rray2string
     data = _leading_trailing(a)
   File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 
162, in _l
eading_trailing
     min(len(a), _summaryEdgeItems))]
   File "C:\Python26\Lib\site-packages\numpy\core\memmap.py", line 257, 
in __arra
y_finalize__
     self.filename = obj.filename
AttributeError: 'memmap' object has no attribute 'filename'
Exception AttributeError: AttributeError("'NoneType' object has no 
attribute 'te
ll'",) in <bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 
0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0], dtype=int16)> ignored
</traceback>

I don't know if it's a bug, but I thought it's is import to report
because the version 1.4.1 was working and 1.5.0 and above was not.

Thanks!



From y.copin at ipnl.in2p3.fr  Wed Apr 20 06:27:09 2011
From: y.copin at ipnl.in2p3.fr (Yannick Copin)
Date: Wed, 20 Apr 2011 12:27:09 +0200
Subject: [Numpy-discussion] Suggestion: Add the unsqueeze function to numpy
Message-ID: <4DAEB4FD.6060506@ipnl.in2p3.fr>

Hi,

I'm a very frequent user of the following "unsqueeze" function, which I 
initially copied from scipy/stats/models/robust/scale.py and which seems to be 
also present in scikits.statsmodels.tools.unsqueeze. Would it be possible to 
include it natively to numpy?

def unsqueeze(data, axis, oldshape):
     """
     Unsqueeze a collapsed array

     >>> from numpy import mean
     >>> from numpy.random import standard_normal
     >>> x = standard_normal((3,4,5))
     >>> m = mean(x, axis=1)
     >>> m.shape
     (3, 5)
     >>> m = unsqueeze(m, 1, x.shape)
     >>> m.shape
     (3, 1, 5)
     """

     if axis is None:
         return data

     newshape = list(oldshape)
     newshape[axis] = 1
     return data.reshape(newshape)

Cheers,
-- 
    .~.   Yannick COPIN  (o:>*  Doctus cum libro
    /V\   Institut de physique nucl?aire de Lyon
   // \\  Universit? de Lyon - CNRS-IN2P3
  /(   )\ AIM: YnCopin ICQ: 236931013
   ^`~'^  http://snovae.in2p3.fr/ycopin/




From josef.pktd at gmail.com  Wed Apr 20 06:56:26 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 20 Apr 2011 06:56:26 -0400
Subject: [Numpy-discussion] Suggestion: Add the unsqueeze function to
	numpy
In-Reply-To: <4DAEB4FD.6060506@ipnl.in2p3.fr>
References: <4DAEB4FD.6060506@ipnl.in2p3.fr>
Message-ID: <BANLkTinu1gZSKr0z71Ro9YGo0cSgtMnKMg@mail.gmail.com>

On Wed, Apr 20, 2011 at 6:27 AM, Yannick Copin <y.copin at ipnl.in2p3.fr> wrote:
> Hi,
>
> I'm a very frequent user of the following "unsqueeze" function, which I
> initially copied from scipy/stats/models/robust/scale.py and which seems to be
> also present in scikits.statsmodels.tools.unsqueeze. Would it be possible to
> include it natively to numpy?
>
> def unsqueeze(data, axis, oldshape):
> ? ? """
> ? ? Unsqueeze a collapsed array
>
> ? ? >>> from numpy import mean
> ? ? >>> from numpy.random import standard_normal
> ? ? >>> x = standard_normal((3,4,5))
> ? ? >>> m = mean(x, axis=1)
> ? ? >>> m.shape
> ? ? (3, 5)
> ? ? >>> m = unsqueeze(m, 1, x.shape)
> ? ? >>> m.shape
> ? ? (3, 1, 5)
> ? ? """
>
> ? ? if axis is None:
> ? ? ? ? return data
>
> ? ? newshape = list(oldshape)
> ? ? newshape[axis] = 1
> ? ? return data.reshape(newshape)

I also proposed this already once.

However there is already function in numpy  (where I have often
problems remembering the name):

numpy.expand_dims(a, axis)
Expand the shape of an array.
Insert a new axis, corresponding to a given position in the array shape.


Josef


>
> Cheers,
> --
> ? ?.~. ? Yannick COPIN ?(o:>* ?Doctus cum libro
> ? ?/V\ ? Institut de physique nucl?aire de Lyon
> ? // \\ ?Universit? de Lyon - CNRS-IN2P3
> ?/( ? )\ AIM: YnCopin ICQ: 236931013
> ? ^`~'^ ?http://snovae.in2p3.fr/ycopin/
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From brett.olsen at gmail.com  Wed Apr 20 11:03:36 2011
From: brett.olsen at gmail.com (Brett Olsen)
Date: Wed, 20 Apr 2011 10:03:36 -0500
Subject: [Numpy-discussion] Beginner's question
In-Reply-To: <4DA9E947.7060209@shopzeus.com>
References: <4DA9E947.7060209@shopzeus.com>
Message-ID: <BANLkTikxxdvEwLypYumxdgaiQVSWNUFTNw@mail.gmail.com>

On Sat, Apr 16, 2011 at 2:08 PM, Laszlo Nagy <gandalf at shopzeus.com> wrote:
> import numpy as np
> import numpy.random as rnd
>
> def dim_weight(X):
> ? ? weights = X[0]
> ? ? volumes = X[1]*X[2]*X[3]
> ? ? res = np.empty(len(volumes), dtype=np.double)
> ? ? for i,v in enumerate(volumes):
> ? ? ? ? if v>5184:
> ? ? ? ? ? ? res[i] = v/194.0
> ? ? ? ? else:
> ? ? ? ? ? ? res[i] = weights[i]
> ? ? return res
> N = 10
> X = rnd.randint( 1,25, (4,N))
> print dim_weight(X)
> ? ?Laszlo

This works:

def dim_weight2(X):
    w = X[0]
    v = X[1]*X[2]*X[3]
    res = np.empty(len(volumes), dtype=np.double)
    res[:] = w[:]
    res[v>5184] = v[v>5184]/194.0
    return res

~Brett


From matthew.brett at gmail.com  Wed Apr 20 13:27:40 2011
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 20 Apr 2011 10:27:40 -0700
Subject: [Numpy-discussion] ATLAS - support letter
Message-ID: <BANLkTi=gFdM2uiDtZz79SaVAwy9Bfv_PTg@mail.gmail.com>

Hi,

I'm on the ATLAS mailing list, maybe some of y'all are too.  Clint
Whaley, the author of ATLAS, was asking for letters to support his
tenure case.  That is, letters saying that lots of us benefit greatly
from his work - which is obviously true.

Can we the numpy community produce such a letter?  Who would it best come from?

Cheers,

Matthew


From yannick.copin at laposte.net  Wed Apr 20 13:24:25 2011
From: yannick.copin at laposte.net (Yannick Copin)
Date: Wed, 20 Apr 2011 17:24:25 +0000 (UTC)
Subject: [Numpy-discussion]
	=?utf-8?q?Suggestion=3A_Add_the_unsqueeze_func?=
	=?utf-8?q?tion_to=09numpy?=
References: <4DAEB4FD.6060506@ipnl.in2p3.fr>
	<BANLkTinu1gZSKr0z71Ro9YGo0cSgtMnKMg@mail.gmail.com>
Message-ID: <loom.20110420T192244-79@post.gmane.org>

 <josef.pktd <at> gmail.com> writes:
> I also proposed this already once.
> 
> However there is already function in numpy  (where I have often
> problems remembering the name):
> 
> numpy.expand_dims(a, axis)

Ah, thanks for the tip, I didn't know this one. The name is unfortunate indeed...

Cheers,

Yannick





From dplepage at gmail.com  Wed Apr 20 14:19:31 2011
From: dplepage at gmail.com (Daniel Lepage)
Date: Wed, 20 Apr 2011 14:19:31 -0400
Subject: [Numpy-discussion] Suggestion: Add the unsqueeze function to
	numpy
In-Reply-To: <loom.20110420T192244-79@post.gmane.org>
References: <4DAEB4FD.6060506@ipnl.in2p3.fr>
	<BANLkTinu1gZSKr0z71Ro9YGo0cSgtMnKMg@mail.gmail.com>
	<loom.20110420T192244-79@post.gmane.org>
Message-ID: <BANLkTi=FYPcfT1J-OW=ZtVtdJUySndEaEQ@mail.gmail.com>

You can also insert new axes when you slice an array via np.newaxis, fwiw:

>>> import numpy as np
>>> x = np.random.random((3,4,5))
>>> y = x.mean(axis=1)
>>> y.shape
(3, 5)
>>> y[:,np.newaxis,:].shape
(3, 1, 5)

-- 
Dan Lepage

On Wed, Apr 20, 2011 at 1:24 PM, Yannick Copin
<yannick.copin at laposte.net> wrote:
> ?<josef.pktd <at> gmail.com> writes:
>> I also proposed this already once.
>>
>> However there is already function in numpy ?(where I have often
>> problems remembering the name):
>>
>> numpy.expand_dims(a, axis)
>
> Ah, thanks for the tip, I didn't know this one. The name is unfortunate indeed...
>
> Cheers,
>
> Yannick
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From bsouthey at gmail.com  Wed Apr 20 14:20:07 2011
From: bsouthey at gmail.com (Bruce Southey)
Date: Wed, 20 Apr 2011 13:20:07 -0500
Subject: [Numpy-discussion] Suggestion: Add the unsqueeze function to
 numpy
In-Reply-To: <loom.20110420T192244-79@post.gmane.org>
References: <4DAEB4FD.6060506@ipnl.in2p3.fr>	<BANLkTinu1gZSKr0z71Ro9YGo0cSgtMnKMg@mail.gmail.com>
	<loom.20110420T192244-79@post.gmane.org>
Message-ID: <4DAF23D7.9000801@gmail.com>

On 04/20/2011 12:24 PM, Yannick Copin wrote:
>   <josef.pktd<at>  gmail.com>  writes:
>> I also proposed this already once.
>>
>> However there is already function in numpy  (where I have often
>> problems remembering the name):
>>
>> numpy.expand_dims(a, axis)
> Ah, thanks for the tip, I didn't know this one. The name is unfortunate indeed...
>
> Cheers,
>
> Yannick
>
>
>
If you don't like the name, you can propose a different name  via ticket 
(probably less likely but numpy 2.0 is around the corner). In the 
immediate term, please find time to update the documentation so both 
squeeze and expand_dim are added to the respective pages so that it 
easier to find - an joint example of squeeze/expand_dim would help here.

Bruce


From josef.pktd at gmail.com  Wed Apr 20 14:27:10 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 20 Apr 2011 14:27:10 -0400
Subject: [Numpy-discussion] Suggestion: Add the unsqueeze function to
	numpy
In-Reply-To: <BANLkTi=FYPcfT1J-OW=ZtVtdJUySndEaEQ@mail.gmail.com>
References: <4DAEB4FD.6060506@ipnl.in2p3.fr>
	<BANLkTinu1gZSKr0z71Ro9YGo0cSgtMnKMg@mail.gmail.com>
	<loom.20110420T192244-79@post.gmane.org>
	<BANLkTi=FYPcfT1J-OW=ZtVtdJUySndEaEQ@mail.gmail.com>
Message-ID: <BANLkTimFvy433eHyDDj6-8tGUO_Me=Wh5Q@mail.gmail.com>

On Wed, Apr 20, 2011 at 2:19 PM, Daniel Lepage <dplepage at gmail.com> wrote:
> You can also insert new axes when you slice an array via np.newaxis, fwiw:
>
>>>> import numpy as np
>>>> x = np.random.random((3,4,5))
>>>> y = x.mean(axis=1)
>>>> y.shape
> (3, 5)
>>>> y[:,np.newaxis,:].shape
> (3, 1, 5)

That's convenient if you only have a specific axis to take care of,
but I like expand_dims when I write functions that are supposed to
work with any axis.

def demean(x, axis=0):
    return x - np.expand_dims(x.mean(axis), axis)

or something like this

Josef

> --
> Dan Lepage
>
> On Wed, Apr 20, 2011 at 1:24 PM, Yannick Copin
> <yannick.copin at laposte.net> wrote:
>> ?<josef.pktd <at> gmail.com> writes:
>>> I also proposed this already once.
>>>
>>> However there is already function in numpy ?(where I have often
>>> problems remembering the name):
>>>
>>> numpy.expand_dims(a, axis)
>>
>> Ah, thanks for the tip, I didn't know this one. The name is unfortunate indeed...
>>
>> Cheers,
>>
>> Yannick
>>
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From ralf.gommers at googlemail.com  Wed Apr 20 16:37:39 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Wed, 20 Apr 2011 22:37:39 +0200
Subject: [Numpy-discussion] status for 1.6.0
In-Reply-To: <BANLkTimMSdGWBz6WCNRE5i0UAJhPM+o_hQ@mail.gmail.com>
References: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
	<BANLkTinVTZ4S=SpSmavRQtmVWOcu+ctEyQ@mail.gmail.com>
	<BANLkTimMSdGWBz6WCNRE5i0UAJhPM+o_hQ@mail.gmail.com>
Message-ID: <BANLkTinmiKb6U4vi0s3wYrNBRZdg5Pki=w@mail.gmail.com>

On Tue, Apr 19, 2011 at 8:54 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Mon, Apr 18, 2011 at 1:47 PM, David Cournapeau <cournape at gmail.com> wrote:
>> On Mon, Apr 18, 2011 at 12:55 AM, Ralf Gommers
>> <ralf.gommers at googlemail.com> wrote:
>>> Hi,
>>>
>>> The list of open issues for 1.6.0 is down to a handful:
>>>
>>> - f2py segfault on Ubuntu reported by David (David, did you get any
>>> further with this?)
>>
>> I could reproduce on another machine (MAC OS X, different hardware), I
>> will take a look at it tonight.
>
> Ok, so the problem appears only when run through nosetests for a
> in-place build of numpy. I think it is fair to say it is a bug
> somewhere in nosetests or some weird import-related corner cases that
> we don't care about.
>
> If I run the same failing test for a "normal" install, I don't get the issue.

Okay, that for checking. That can be ignored then.

Cheers,
Ralf


From ralf.gommers at googlemail.com  Wed Apr 20 16:55:07 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Wed, 20 Apr 2011 22:55:07 +0200
Subject: [Numpy-discussion] status for 1.6.0
In-Reply-To: <4DAB877F.3010109@uci.edu>
References: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
	<4DAB877F.3010109@uci.edu>
Message-ID: <BANLkTikYs+ZTQTAGvipAj15Y9HpRzwnuxQ@mail.gmail.com>

On Mon, Apr 18, 2011 at 2:36 AM, Christoph Gohlke <cgohlke at uci.edu> wrote:
>
>
> On 4/17/2011 8:55 AM, Ralf Gommers wrote:
>>
>> Hi,
>>
>> The list of open issues for 1.6.0 is down to a handful:
>>
>> - f2py segfault on Ubuntu reported by David (David, did you get any
>> further with this?)
>> - #1801: test_noncentral_f failure on win64
>> - #1798: type promotion regression (needs review,
>> https://github.com/numpy/numpy/pull/73)
>> - API docs for new parameters to ufuncs (full list: 'casting', 'out',
>> 'order', 'subok', 'dtype'),
>> - API docs, some docstrings and update of NEP for datetime
>>
>> I'm planning to release the first release candidate next weekend if
>> these are solved. ?As far as I know no one is working on the doc
>> issues right now. It would be great if someone could step up and do
>> this.
>>
>> Thanks,
>> Ralf
>
>
> Hi Ralf,
>
> I submitted a patch for issue #1801. All tests now pass on win-amd64-py3.2
> with numpy 1.6.x from git.

Great, thanks.

> Would it be possible to get the attached patch for scipy ticket #678
> <http://projects.scipy.org/scipy/ticket/678> into numpy 1.6? The
> fix/workaround for the scipy ODR failures is to change the ifort
> optimization flag from /O3 to /O1.

Several other people already ran into this, so it would probably still
good to change this for 1.6.0. This change is part of a proposed
patch: http://projects.scipy.org/numpy/ticket/1378. Do any of the
other changes there look relevant too?

Ralf


From cgohlke at uci.edu  Wed Apr 20 21:37:33 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Wed, 20 Apr 2011 18:37:33 -0700
Subject: [Numpy-discussion] status for 1.6.0
In-Reply-To: <BANLkTikYs+ZTQTAGvipAj15Y9HpRzwnuxQ@mail.gmail.com>
References: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>	<4DAB877F.3010109@uci.edu>
	<BANLkTikYs+ZTQTAGvipAj15Y9HpRzwnuxQ@mail.gmail.com>
Message-ID: <4DAF8A5D.6020603@uci.edu>



On 4/20/2011 1:55 PM, Ralf Gommers wrote:
> On Mon, Apr 18, 2011 at 2:36 AM, Christoph Gohlke<cgohlke at uci.edu>  wrote:
>>
>>
>> On 4/17/2011 8:55 AM, Ralf Gommers wrote:
>>>
>>> Hi,
>>>
>>> The list of open issues for 1.6.0 is down to a handful:
>>>
>>> - f2py segfault on Ubuntu reported by David (David, did you get any
>>> further with this?)
>>> - #1801: test_noncentral_f failure on win64
>>> - #1798: type promotion regression (needs review,
>>> https://github.com/numpy/numpy/pull/73)
>>> - API docs for new parameters to ufuncs (full list: 'casting', 'out',
>>> 'order', 'subok', 'dtype'),
>>> - API docs, some docstrings and update of NEP for datetime
>>>
>>> I'm planning to release the first release candidate next weekend if
>>> these are solved.  As far as I know no one is working on the doc
>>> issues right now. It would be great if someone could step up and do
>>> this.
>>>
>>> Thanks,
>>> Ralf
>>
>>
>> Hi Ralf,
>>
>> I submitted a patch for issue #1801. All tests now pass on win-amd64-py3.2
>> with numpy 1.6.x from git.
>
> Great, thanks.
>
>> Would it be possible to get the attached patch for scipy ticket #678
>> <http://projects.scipy.org/scipy/ticket/678>  into numpy 1.6? The
>> fix/workaround for the scipy ODR failures is to change the ifort
>> optimization flag from /O3 to /O1.
>
> Several other people already ran into this, so it would probably still
> good to change this for 1.6.0. This change is part of a proposed
> patch: http://projects.scipy.org/numpy/ticket/1378. Do any of the
> other changes there look relevant too?
>
> Ralf

The changes for the Intel Visual Fortran Compiler looks OK. '/Qip' is 
the default option, and '/Qipo','/Qipo_obj' have already been removed 
some time ago.

I have not used older Intel Fortran compilers but the '-xHOST' switch 
does not seem apt.

Christoph


From pratik.mallya at gmail.com  Wed Apr 20 21:55:18 2011
From: pratik.mallya at gmail.com (pratik)
Date: Thu, 21 Apr 2011 07:25:18 +0530
Subject: [Numpy-discussion] ATLAS - support letter
In-Reply-To: <BANLkTi=gFdM2uiDtZz79SaVAwy9Bfv_PTg@mail.gmail.com>
References: <BANLkTi=gFdM2uiDtZz79SaVAwy9Bfv_PTg@mail.gmail.com>
Message-ID: <4DAF8E86.1050304@gmail.com>

On Wednesday 20 April 2011 10:57 PM, Matthew Brett wrote:
> Hi,
>
> I'm on the ATLAS mailing list, maybe some of y'all are too.  Clint
> Whaley, the author of ATLAS, was asking for letters to support his
> tenure case.  That is, letters saying that lots of us benefit greatly
> from his work - which is obviously true.
>
> Can we the numpy community produce such a letter?  Who would it best come from?
>
> Cheers,
>
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>    
If the place where he is seeking tenure does not know his name (i.e 
hasn't heard of ATLAS) then it is not a good place to seek tenure in :) .



From alan.isaac at gmail.com  Wed Apr 20 22:25:46 2011
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Wed, 20 Apr 2011 22:25:46 -0400
Subject: [Numpy-discussion] ATLAS - support letter
In-Reply-To: <4DAF8E86.1050304@gmail.com>
References: <BANLkTi=gFdM2uiDtZz79SaVAwy9Bfv_PTg@mail.gmail.com>
	<4DAF8E86.1050304@gmail.com>
Message-ID: <4DAF95AA.9010900@gmail.com>

On 4/20/2011 9:55 PM, pratik wrote:
> If the place where he is seeking tenure does not know his name (i.e
> hasn't heard of ATLAS)

Letters are often more for administrators, who can be from any field,
than for the department faculty.

fwiw,
Alan Isaac



From gael.varoquaux at normalesup.org  Thu Apr 21 01:45:22 2011
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 21 Apr 2011 07:45:22 +0200
Subject: [Numpy-discussion] ATLAS - support letter
In-Reply-To: <4DAF8E86.1050304@gmail.com>
References: <BANLkTi=gFdM2uiDtZz79SaVAwy9Bfv_PTg@mail.gmail.com>
	<4DAF8E86.1050304@gmail.com>
Message-ID: <20110421054522.GA11208@phare.normalesup.org>

On Thu, Apr 21, 2011 at 07:25:18AM +0530, pratik wrote:
> If the place where he is seeking tenure does not know his name (i.e 
> hasn't heard of ATLAS) then it is not a good place to seek tenure in :) .

Scholars undervalue code and don't realise the difficulty and the amount
of work it takes to produce. More than once I have had colleagues tell me
that they valued a paper more than a software. It is important to show
that such software gives a large benefit to the scientific community.

Matthew, I didn't feel that I could do much to answer your call, but if
you feel different, please let me know.

Gael


From matthew.brett at gmail.com  Thu Apr 21 01:49:10 2011
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 20 Apr 2011 22:49:10 -0700
Subject: [Numpy-discussion] ATLAS - support letter
In-Reply-To: <20110421054522.GA11208@phare.normalesup.org>
References: <BANLkTi=gFdM2uiDtZz79SaVAwy9Bfv_PTg@mail.gmail.com>
	<4DAF8E86.1050304@gmail.com>
	<20110421054522.GA11208@phare.normalesup.org>
Message-ID: <BANLkTikk114bSji7fSKnuBsc4-PbyZSk1Q@mail.gmail.com>

Hi,

On Wed, Apr 20, 2011 at 10:45 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Thu, Apr 21, 2011 at 07:25:18AM +0530, pratik wrote:
>> If the place where he is seeking tenure does not know his name (i.e
>> hasn't heard of ATLAS) then it is not a good place to seek tenure in :) .
>
> Scholars undervalue code and don't realise the difficulty and the amount
> of work it takes to produce. More than once I have had colleagues tell me
> that they valued a paper more than a software. It is important to show
> that such software gives a large benefit to the scientific community.
>
> Matthew, I didn't feel that I could do much to answer your call, but if
> you feel different, please let me know.

Well - thanks for the offer - Clint was asking for individual letters
too, you could email and ask him?

Are you on the math-atlas list?  If not I'll forward you his request...

See you,

Matthew


From gael.varoquaux at normalesup.org  Thu Apr 21 01:51:25 2011
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 21 Apr 2011 07:51:25 +0200
Subject: [Numpy-discussion] ATLAS - support letter
In-Reply-To: <BANLkTikk114bSji7fSKnuBsc4-PbyZSk1Q@mail.gmail.com>
References: <BANLkTi=gFdM2uiDtZz79SaVAwy9Bfv_PTg@mail.gmail.com>
	<4DAF8E86.1050304@gmail.com>
	<20110421054522.GA11208@phare.normalesup.org>
	<BANLkTikk114bSji7fSKnuBsc4-PbyZSk1Q@mail.gmail.com>
Message-ID: <20110421055125.GB11208@phare.normalesup.org>

On Wed, Apr 20, 2011 at 10:49:10PM -0700, Matthew Brett wrote:
> Well - thanks for the offer - Clint was asking for individual letters
> too, you could email and ask him?

I can do that, and ask around me.

> Are you on the math-atlas list? 

No I am not.

> If not I'll forward you his request...

Please do that.

See you,

Gael


From matthew.brett at gmail.com  Thu Apr 21 01:54:45 2011
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 20 Apr 2011 22:54:45 -0700
Subject: [Numpy-discussion] ATLAS - support letter
In-Reply-To: <4DAF8E86.1050304@gmail.com>
References: <BANLkTi=gFdM2uiDtZz79SaVAwy9Bfv_PTg@mail.gmail.com>
	<4DAF8E86.1050304@gmail.com>
Message-ID: <BANLkTimX5wudSotsB4s0N6AWiu5WGsg-gg@mail.gmail.com>

Hi,

On Wed, Apr 20, 2011 at 6:55 PM, pratik <pratik.mallya at gmail.com> wrote:
> On Wednesday 20 April 2011 10:57 PM, Matthew Brett wrote:
>> Hi,
>>
>> I'm on the ATLAS mailing list, maybe some of y'all are too. ?Clint
>> Whaley, the author of ATLAS, was asking for letters to support his
>> tenure case. ?That is, letters saying that lots of us benefit greatly
>> from his work - which is obviously true.
>>
>> Can we the numpy community produce such a letter? ?Who would it best come from?
>>
>> Cheers,
>>
>> Matthew
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> If the place where he is seeking tenure does not know his name (i.e
> hasn't heard of ATLAS) then it is not a good place to seek tenure in :) .

It seems to me that we are so used to depending on ATLAS we forget
just how much we rely on Clint to support and improve it.  This seems
like one of those rare times when it's fairly easy to give something
back...

Best,

Matthew


From gokhansever at gmail.com  Thu Apr 21 16:06:56 2011
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Thu, 21 Apr 2011 14:06:56 -0600
Subject: [Numpy-discussion] Segfault using "fromstring" and reading variable
	length string
Message-ID: <BANLkTi=oKoYtzRX9QELOkJBd0OBz3rUQ9Q@mail.gmail.com>

Hello,

Given this piece of code (I can provide the meg file off-the list for those
who wants to reproduce the error)


import numpy as np

f = open("a08A0122.341071.meg", "rb")

dt = np.dtype([('line1', '|S80'), ('line2', np.object_), ('line3', '|S80'),
('line4', '|S80'),
               ('line5', '|S80'), ('line6', '|S2'), ('line7', np.int32,
2000), ('line8', '|S2'),
               ('line9', np.int32, 2000), ('line10', '|S2')])

k = np.fromstring(f.read(dt.itemsize), dt)[0]

Accessing k causes a "Segmentation fault (core dumped)" and kills my python
and IPython sessions immediately.  I actually know that the culprit is
"np.object_" in this case.  The original was as ('line2', '|S81') however
those meg files (mix of text and binary content) have a funny habit of
switching from 80 characters to 81 (including "/r/n" chars). I was testing
if I could create a variable length string dtype, which seems not possible.

Little more info: that line2 has time stamps, one of which is in the form
of 22:34:59.999. I have seen in the file that 22:34:59.999 was originally
written as 22:34:59.1000 which causes that extra character flow.
(Interestingly, millisecond should cycle from 0-999 and overflow at 999
instead of 1000 which to me indicates a slight bug) Because of this reason,
I can't read the whole content of those meg files since somewhere in the
middle fromstring attempts reading a shifted (erroneous) content. Should I
go fix that millisecond overflow first or is there an alternative way to
approach this problem?

Thanks

================================================================================
Platform     : Linux-2.6.35.12-88.fc14.x86_64-x86_64-with-fedora-14-Laughlin
Python       : ('CPython', 'tags/r27', '82500')
IPython      : 0.10
NumPy       : 2.0.0.dev-2e96d91
================================================================================

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110421/a41fc1fe/attachment.html>

From ralf.gommers at googlemail.com  Fri Apr 22 13:50:44 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Fri, 22 Apr 2011 19:50:44 +0200
Subject: [Numpy-discussion] status for 1.6.0
In-Reply-To: <4DAF8A5D.6020603@uci.edu>
References: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
	<4DAB877F.3010109@uci.edu>
	<BANLkTikYs+ZTQTAGvipAj15Y9HpRzwnuxQ@mail.gmail.com>
	<4DAF8A5D.6020603@uci.edu>
Message-ID: <BANLkTinyONJ57or8uD92TpKLW+y9GTu6hw@mail.gmail.com>

On Thu, Apr 21, 2011 at 3:37 AM, Christoph Gohlke <cgohlke at uci.edu> wrote:
>
>
> On 4/20/2011 1:55 PM, Ralf Gommers wrote:
>> On Mon, Apr 18, 2011 at 2:36 AM, Christoph Gohlke<cgohlke at uci.edu> ?wrote:
>>>
>>>
>>> On 4/17/2011 8:55 AM, Ralf Gommers wrote:
>>>>
>>>> Hi,
>>>>
>>>> The list of open issues for 1.6.0 is down to a handful:
>>>>
>>>> - f2py segfault on Ubuntu reported by David (David, did you get any
>>>> further with this?)
>>>> - #1801: test_noncentral_f failure on win64
>>>> - #1798: type promotion regression (needs review,
>>>> https://github.com/numpy/numpy/pull/73)
>>>> - API docs for new parameters to ufuncs (full list: 'casting', 'out',
>>>> 'order', 'subok', 'dtype'),
>>>> - API docs, some docstrings and update of NEP for datetime
>>>>
>>>> I'm planning to release the first release candidate next weekend if
>>>> these are solved. ?As far as I know no one is working on the doc
>>>> issues right now. It would be great if someone could step up and do
>>>> this.
>>>>
>>>> Thanks,
>>>> Ralf
>>>
>>>
>>> Hi Ralf,
>>>
>>> I submitted a patch for issue #1801. All tests now pass on win-amd64-py3.2
>>> with numpy 1.6.x from git.
>>
>> Great, thanks.
>>
>>> Would it be possible to get the attached patch for scipy ticket #678
>>> <http://projects.scipy.org/scipy/ticket/678> ?into numpy 1.6? The
>>> fix/workaround for the scipy ODR failures is to change the ifort
>>> optimization flag from /O3 to /O1.
>>
>> Several other people already ran into this, so it would probably still
>> good to change this for 1.6.0. This change is part of a proposed
>> patch: http://projects.scipy.org/numpy/ticket/1378. Do any of the
>> other changes there look relevant too?
>>
>> Ralf
>
> The changes for the Intel Visual Fortran Compiler looks OK. '/Qip' is
> the default option, and '/Qipo','/Qipo_obj' have already been removed
> some time ago.
>
> I have not used older Intel Fortran compilers but the '-xHOST' switch
> does not seem apt.

Thanks for checking, should be fixed now.

Cheers,
Ralf


From ralf.gommers at googlemail.com  Fri Apr 22 14:37:46 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Fri, 22 Apr 2011 20:37:46 +0200
Subject: [Numpy-discussion] Segfault using "fromstring" and reading
 variable length string
In-Reply-To: <BANLkTi=oKoYtzRX9QELOkJBd0OBz3rUQ9Q@mail.gmail.com>
References: <BANLkTi=oKoYtzRX9QELOkJBd0OBz3rUQ9Q@mail.gmail.com>
Message-ID: <BANLkTin7xdG2ZZ5UzpDfmp=Un68J1eHdRA@mail.gmail.com>

On Thu, Apr 21, 2011 at 10:06 PM, G?khan Sever <gokhansever at gmail.com> wrote:
> Hello,
> Given this piece of code (I can provide the meg file off-the list for those
> who wants to reproduce the error)

Can you instead construct a test as simple as possible for this? It
sounds like you need only a two line string to reproduce this. The bug
sounds similar to http://projects.scipy.org/numpy/ticket/1689.

Ralf


> import numpy as np
> f = open("a08A0122.341071.meg", "rb")
> dt = np.dtype([('line1', '|S80'), ('line2', np.object_), ('line3', '|S80'),
> ('line4', '|S80'),
> ?? ? ? ? ? ? ? ('line5', '|S80'), ('line6', '|S2'), ('line7', np.int32,
> 2000), ('line8', '|S2'),
> ?? ? ? ? ? ? ? ('line9', np.int32, 2000), ('line10', '|S2')])
> k = np.fromstring(f.read(dt.itemsize), dt)[0]
> Accessing k causes a "Segmentation fault (core dumped)" and kills my python
> and IPython sessions?immediately. ?I actually know that the culprit is
> "np.object_" in this case. ?The original was as?('line2', '|S81') however
> those meg files (mix of text and binary content) have a funny habit of
> switching from 80 characters to 81 (including "/r/n" chars). I was testing
> if I could create a variable length string dtype, which seems not possible.
> Little more info: that line2 has time stamps, one of which is in the form
> of?22:34:59.999. I have seen in the file that 22:34:59.999 was originally
> written as 22:34:59.1000 which causes that extra character flow.
> (Interestingly, millisecond should cycle from 0-999 and overflow at 999
> instead of 1000 which to me indicates a slight bug) Because of this reason,
> I can't read the whole content of those meg files since somewhere in the
> middle fromstring attempts reading a shifted (erroneous) content. Should I
> go fix that millisecond overflow first or is there an alternative way to
> approach this problem?


From gokhansever at gmail.com  Fri Apr 22 17:52:05 2011
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Fri, 22 Apr 2011 15:52:05 -0600
Subject: [Numpy-discussion] Segfault using "fromstring" and reading
 variable length string
In-Reply-To: <BANLkTin7xdG2ZZ5UzpDfmp=Un68J1eHdRA@mail.gmail.com>
References: <BANLkTi=oKoYtzRX9QELOkJBd0OBz3rUQ9Q@mail.gmail.com>
	<BANLkTin7xdG2ZZ5UzpDfmp=Un68J1eHdRA@mail.gmail.com>
Message-ID: <BANLkTim1Y+bXXZbnu9rs3sds8sit9C=Bpw@mail.gmail.com>

On Fri, Apr 22, 2011 at 12:37 PM, Ralf Gommers
<ralf.gommers at googlemail.com>wrote:

> On Thu, Apr 21, 2011 at 10:06 PM, G?khan Sever <gokhansever at gmail.com>
> wrote:
> > Hello,
> > Given this piece of code (I can provide the meg file off-the list for
> those
> > who wants to reproduce the error)
>
> Can you instead construct a test as simple as possible for this? It
> sounds like you need only a two line string to reproduce this. The bug
> sounds similar to http://projects.scipy.org/numpy/ticket/1689.
>
> Ralf


This simple case segfaults as well (The commented line works correctly):

import numpy as np
from StringIO import StringIO

c = StringIO(" hello \r\n world \r\n")

dt = np.dtype([('line1', '|S6'), ('line2', np.object_)])
#dt = np.dtype([('line1', '|S9'), ('line2', '|S9')])
k = np.fromstring(c.read(dt.itemsize), dt)[0]

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110422/0e2949b2/attachment.html>

From mwwiebe at gmail.com  Fri Apr 22 17:54:42 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Fri, 22 Apr 2011 14:54:42 -0700
Subject: [Numpy-discussion] status for 1.6.0
In-Reply-To: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
References: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
Message-ID: <BANLkTimbdV9SKxQobbSN06jKC8imRkRPOg@mail.gmail.com>

On Sun, Apr 17, 2011 at 8:55 AM, Ralf Gommers
<ralf.gommers at googlemail.com>wrote:

> Hi,
>
> The list of open issues for 1.6.0 is down to a handful:
>
> - f2py segfault on Ubuntu reported by David (David, did you get any
> further with this?)
> - #1801: test_noncentral_f failure on win64
> - #1798: type promotion regression (needs review,
> https://github.com/numpy/numpy/pull/73)
>

I've pushed this change, anyone interested should review the tests for this
to verify that it accords with their intuition:

https://github.com/numpy/numpy/blob/maintenance%2F1.6.x/numpy/core/tests/test_numeric.py#L372

- API docs for new parameters to ufuncs (full list: 'casting', 'out',
> 'order', 'subok', 'dtype'),
>

Done.


> - API docs, some docstrings and update of NEP for datetime
>
> I'm planning to release the first release candidate next weekend if
> these are solved.  As far as I know no one is working on the doc
> issues right now. It would be great if someone could step up and do
> this.
>

There's a new ticket #1805 which should get fixed for the RC too.

-Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110422/c8bbabbf/attachment.html>

From cgohlke at uci.edu  Fri Apr 22 20:08:19 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Fri, 22 Apr 2011 17:08:19 -0700
Subject: [Numpy-discussion] Segfault using "fromstring" and reading
 variable length string
In-Reply-To: <BANLkTim1Y+bXXZbnu9rs3sds8sit9C=Bpw@mail.gmail.com>
References: <BANLkTi=oKoYtzRX9QELOkJBd0OBz3rUQ9Q@mail.gmail.com>	<BANLkTin7xdG2ZZ5UzpDfmp=Un68J1eHdRA@mail.gmail.com>
	<BANLkTim1Y+bXXZbnu9rs3sds8sit9C=Bpw@mail.gmail.com>
Message-ID: <4DB21873.2060601@uci.edu>



On 4/22/2011 2:52 PM, G?khan Sever wrote:
>
>
> On Fri, Apr 22, 2011 at 12:37 PM, Ralf Gommers
> <ralf.gommers at googlemail.com <mailto:ralf.gommers at googlemail.com>> wrote:
>
>     On Thu, Apr 21, 2011 at 10:06 PM, G?khan Sever
>     <gokhansever at gmail.com <mailto:gokhansever at gmail.com>> wrote:
>     >  Hello,
>     >  Given this piece of code (I can provide the meg file off-the list
>     for those
>     >  who wants to reproduce the error)
>
>     Can you instead construct a test as simple as possible for this? It
>     sounds like you need only a two line string to reproduce this. The bug
>     sounds similar to http://projects.scipy.org/numpy/ticket/1689.
>
>     Ralf
>
>
> This simple case segfaults as well (The commented line works correctly):
>
> import numpy as np
> from StringIO import StringIO
>
> c = StringIO(" hello \r\n world \r\n")
>
> dt = np.dtype([('line1', '|S6'), ('line2', np.object_)])
> #dt = np.dtype([('line1', '|S9'), ('line2', '|S9')])
> k = np.fromstring(c.read(dt.itemsize), dt)[0]
>

I can reproduce the crash with a recent build of numpy 1.6. It is in 
arraytypes.c.src, line 521:

static PyObject *
OBJECT_getitem(char *ip, PyArrayObject *ap)
{
     PyObject *obj;
     NPY_COPY_PYOBJECT_PTR(&obj, ip);
     if (obj == NULL) {
         Py_INCREF(Py_None);
         return Py_None;
     }
     else {
         Py_INCREF(obj);   /* <== crash */
         return obj;
     }
}


There's no check whether obj is a valid PyObject (it is not in this case).

Christoph


From mwwiebe at gmail.com  Fri Apr 22 20:32:37 2011
From: mwwiebe at gmail.com (Mark Wiebe)
Date: Fri, 22 Apr 2011 17:32:37 -0700
Subject: [Numpy-discussion] Segfault using "fromstring" and reading
 variable length string
In-Reply-To: <4DB21873.2060601@uci.edu>
References: <BANLkTi=oKoYtzRX9QELOkJBd0OBz3rUQ9Q@mail.gmail.com>
	<BANLkTin7xdG2ZZ5UzpDfmp=Un68J1eHdRA@mail.gmail.com>
	<BANLkTim1Y+bXXZbnu9rs3sds8sit9C=Bpw@mail.gmail.com>
	<4DB21873.2060601@uci.edu>
Message-ID: <BANLkTiktB2UPjqw2NUggyObgNWo8xyaqjA@mail.gmail.com>

On Fri, Apr 22, 2011 at 5:08 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:

>
>
> On 4/22/2011 2:52 PM, G?khan Sever wrote:
> >
> >
> > On Fri, Apr 22, 2011 at 12:37 PM, Ralf Gommers
> > <ralf.gommers at googlemail.com <mailto:ralf.gommers at googlemail.com>>
> wrote:
> >
> >     On Thu, Apr 21, 2011 at 10:06 PM, G?khan Sever
> >     <gokhansever at gmail.com <mailto:gokhansever at gmail.com>> wrote:
> >     >  Hello,
> >     >  Given this piece of code (I can provide the meg file off-the list
> >     for those
> >     >  who wants to reproduce the error)
> >
> >     Can you instead construct a test as simple as possible for this? It
> >     sounds like you need only a two line string to reproduce this. The
> bug
> >     sounds similar to http://projects.scipy.org/numpy/ticket/1689.
> >
> >     Ralf
> >
> >
> > This simple case segfaults as well (The commented line works correctly):
> >
> > import numpy as np
> > from StringIO import StringIO
> >
> > c = StringIO(" hello \r\n world \r\n")
> >
> > dt = np.dtype([('line1', '|S6'), ('line2', np.object_)])
> > #dt = np.dtype([('line1', '|S9'), ('line2', '|S9')])
> > k = np.fromstring(c.read(dt.itemsize), dt)[0]
> >
>
> I can reproduce the crash with a recent build of numpy 1.6. It is in
> arraytypes.c.src, line 521:
>
> static PyObject *
> OBJECT_getitem(char *ip, PyArrayObject *ap)
> {
>     PyObject *obj;
>     NPY_COPY_PYOBJECT_PTR(&obj, ip);
>     if (obj == NULL) {
>         Py_INCREF(Py_None);
>         return Py_None;
>     }
>     else {
>         Py_INCREF(obj);   /* <== crash */
>         return obj;
>     }
> }
>
>
> There's no check whether obj is a valid PyObject (it is not in this case).
>
>
I took a quick look at this issue and committed a fix. PyArray_FromString
was doing a check to exclude object arrays, but that check was incorrect.
Now it should appropriately raise an exception instead of creating an
invalid array.

https://github.com/numpy/numpy/commit/f75bfab3a2ab74ac82047f153a36c71c58fe3732

-Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110422/e0f352d9/attachment.html>

From mail.till at gmx.de  Sat Apr 23 09:33:36 2011
From: mail.till at gmx.de (Till Stensitzki)
Date: Sat, 23 Apr 2011 13:33:36 +0000 (UTC)
Subject: [Numpy-discussion] Python crashes while printing a huge numpy array
	(Windows 1.6b2)
Message-ID: <loom.20110423T153046-829@post.gmane.org>

Hello,
due a mistake i found the following bug:


import numpy as np
x=np.zeros((262144, 262144))
print x.shape
print x


The last line crashes the python interpreter without any feedback.
Windows 7, Python 2.6.5, Numpy 1.6b2



From cournape at gmail.com  Sat Apr 23 09:41:13 2011
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 23 Apr 2011 22:41:13 +0900
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <loom.20110423T153046-829@post.gmane.org>
References: <loom.20110423T153046-829@post.gmane.org>
Message-ID: <BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>

On Sat, Apr 23, 2011 at 10:33 PM, Till Stensitzki <mail.till at gmx.de> wrote:
> Hello,
> due a mistake i found the following bug:
>
>
> import numpy as np
> x=np.zeros((262144, 262144))
> print x.shape

Hm, this is a bit weird. The line x = np.zeros... should fail (you
most likely do not have 512 Gb Ram available on your machine :) ).
Maybe a check is wrong due to some wrong configuration on windows. Are
you on windows 32 or 64 bits ?

cheers,

David


From josef.pktd at gmail.com  Sat Apr 23 10:01:23 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 23 Apr 2011 10:01:23 -0400
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>
References: <loom.20110423T153046-829@post.gmane.org>
	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>
Message-ID: <BANLkTinRVLG_vsxsyV2wz6n=oXgxXZOZ_Q@mail.gmail.com>

On Sat, Apr 23, 2011 at 9:41 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Sat, Apr 23, 2011 at 10:33 PM, Till Stensitzki <mail.till at gmx.de> wrote:
>> Hello,
>> due a mistake i found the following bug:
>>
>>
>> import numpy as np
>> x=np.zeros((262144, 262144))
>> print x.shape
>
> Hm, this is a bit weird. The line x = np.zeros... should fail (you
> most likely do not have 512 Gb Ram available on your machine :) ).
> Maybe a check is wrong due to some wrong configuration on windows. Are
> you on windows 32 or 64 bits ?

32 bit python on Windows 7 (64)

I get the correct value error with python 2.6.5 and numpy 1.5.1

but on python 2.5.4 with numpy 1.3.0 I get the crash instead of the ValueError.


Josef

>
> cheers,
>
> David
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From josef.pktd at gmail.com  Sat Apr 23 10:09:21 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 23 Apr 2011 10:09:21 -0400
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <BANLkTinRVLG_vsxsyV2wz6n=oXgxXZOZ_Q@mail.gmail.com>
References: <loom.20110423T153046-829@post.gmane.org>
	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>
	<BANLkTinRVLG_vsxsyV2wz6n=oXgxXZOZ_Q@mail.gmail.com>
Message-ID: <BANLkTi=Rwp3UfAR3mZ0L6B3c0tW_f1LjHQ@mail.gmail.com>

On Sat, Apr 23, 2011 at 10:01 AM,  <josef.pktd at gmail.com> wrote:
> On Sat, Apr 23, 2011 at 9:41 AM, David Cournapeau <cournape at gmail.com> wrote:
>> On Sat, Apr 23, 2011 at 10:33 PM, Till Stensitzki <mail.till at gmx.de> wrote:
>>> Hello,
>>> due a mistake i found the following bug:
>>>
>>>
>>> import numpy as np
>>> x=np.zeros((262144, 262144))
>>> print x.shape
>>
>> Hm, this is a bit weird. The line x = np.zeros... should fail (you
>> most likely do not have 512 Gb Ram available on your machine :) ).
>> Maybe a check is wrong due to some wrong configuration on windows. Are
>> you on windows 32 or 64 bits ?
>
> 32 bit python on Windows 7 (64)
>
> I get the correct value error with python 2.6.5 and numpy 1.5.1
>
> but on python 2.5.4 with numpy 1.3.0 I get the crash instead of the ValueError.

to rule out two more cases

numpy 1.5.1 on python 2.7.1 32 bit : ValueError
numpy 1.5.1 on python 3.2    64 bit: MemoryError

Josef

>
>
> Josef
>
>>
>> cheers,
>>
>> David
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>


From mail.till at gmx.de  Sat Apr 23 10:11:42 2011
From: mail.till at gmx.de (Till Stensitzki)
Date: Sat, 23 Apr 2011 14:11:42 +0000 (UTC)
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
	array (Windows 1.6b2)
References: <loom.20110423T153046-829@post.gmane.org>
	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>
Message-ID: <loom.20110423T160723-665@post.gmane.org>

 

> Hm, this is a bit weird. The line x = np.zeros... should fail (you
> most likely do not have 512 Gb Ram available on your machine :) ).
> Maybe a check is wrong due to some wrong configuration on windows. Are
> you on windows 32 or 64 bits ?
> 

I took the outer product of two (512,512)-matrices instead of two 515-vectors :)

32 bit Windows here. 






From charlesr.harris at gmail.com  Sat Apr 23 10:21:28 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 23 Apr 2011 08:21:28 -0600
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <loom.20110423T160723-665@post.gmane.org>
References: <loom.20110423T153046-829@post.gmane.org>
	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>
	<loom.20110423T160723-665@post.gmane.org>
Message-ID: <BANLkTimcK7K=8wSKmDxKSjGHL=vAsXQ01A@mail.gmail.com>

On Sat, Apr 23, 2011 at 8:11 AM, Till Stensitzki <mail.till at gmx.de> wrote:

>
>
> > Hm, this is a bit weird. The line x = np.zeros... should fail (you
> > most likely do not have 512 Gb Ram available on your machine :) ).
> > Maybe a check is wrong due to some wrong configuration on windows. Are
> > you on windows 32 or 64 bits ?
> >
>
> I took the outer product of two (512,512)-matrices instead of two
> 515-vectors :)
>
> 32 bit Windows here.
>
>
Do you also have an earlier version of numpy installed? As David says, this
should raise an error for recent numpy and I'm wondering if you are
inadvertently running an earlier version.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110423/837929da/attachment.html>

From mail.till at gmx.de  Sat Apr 23 10:58:00 2011
From: mail.till at gmx.de (Till Stensitzki)
Date: Sat, 23 Apr 2011 14:58:00 +0000 (UTC)
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
	array (Windows 1.6b2)
References: <loom.20110423T153046-829@post.gmane.org>
	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>
	<loom.20110423T160723-665@post.gmane.org>
	<BANLkTimcK7K=8wSKmDxKSjGHL=vAsXQ01A@mail.gmail.com>
Message-ID: <loom.20110423T165540-168@post.gmane.org>

 
> Do you also have an earlier version of numpy installed? As David says, this 
>should raise an error for recent numpy and 
>I'm wondering if you are inadvertently 
>running an earlier version.Chuck 


I only have one python installation and 
numpy.__version__ shows 1.6b. 
I could reinstall numpy, if it would help.



From bsouthey at gmail.com  Sat Apr 23 13:09:46 2011
From: bsouthey at gmail.com (Bruce Southey)
Date: Sat, 23 Apr 2011 12:09:46 -0500
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <loom.20110423T165540-168@post.gmane.org>
References: <loom.20110423T153046-829@post.gmane.org>
	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>
	<loom.20110423T160723-665@post.gmane.org>
	<BANLkTimcK7K=8wSKmDxKSjGHL=vAsXQ01A@mail.gmail.com>
	<loom.20110423T165540-168@post.gmane.org>
Message-ID: <BANLkTimsmoyMqv+m+SxMGeHcJzbMzKBW0Q@mail.gmail.com>

On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till at gmx.de> wrote:
>
>> Do you also have an earlier version of numpy installed? As David says, this
>>should raise an error for recent numpy and
>>I'm wondering if you are inadvertently
>>running an earlier version.Chuck
>
>
> I only have one python installation and
> numpy.__version__ shows 1.6b.
> I could reinstall numpy, if it would help.
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

Hi,
I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and
3.1 and numpy 1.6b (fresh install)  IDLE and the command line. I can
also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the
same system.

Actually this is 'weird' when printing and crashed with the range -
accessing unassigned memory?
A smaller array gives an numpy error or memory error in idle.

Bruce


>>> import numpy as np
>>> x=np.zeros((262144, 262144))
>>> x
array([], shape=(262144, 262144), dtype=float64)
>>> x[0,0]
2.1453735050108555e-314
>>> x[1:10,1:10]

>>> ================================ RESTART ================================
>>> import numpy as np

>>> x=np.zeros((26214, 26214))

Traceback (most recent call last):
  File "<pyshell#7>", line 1, in <module>
    x=np.zeros((26214, 26214))
ValueError: array is too big.
>>>
>>> x=np.zeros((262144, 26214))

Traceback (most recent call last):
  File "<pyshell#8>", line 1, in <module>
    x=np.zeros((262144, 26214))
MemoryError


From charlesr.harris at gmail.com  Sat Apr 23 13:41:00 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 23 Apr 2011 11:41:00 -0600
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <BANLkTimsmoyMqv+m+SxMGeHcJzbMzKBW0Q@mail.gmail.com>
References: <loom.20110423T153046-829@post.gmane.org>
	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>
	<loom.20110423T160723-665@post.gmane.org>
	<BANLkTimcK7K=8wSKmDxKSjGHL=vAsXQ01A@mail.gmail.com>
	<loom.20110423T165540-168@post.gmane.org>
	<BANLkTimsmoyMqv+m+SxMGeHcJzbMzKBW0Q@mail.gmail.com>
Message-ID: <BANLkTi=3aMctNUtOFfpRLBwnQg=L4cMp=A@mail.gmail.com>

On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey <bsouthey at gmail.com> wrote:

> On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till at gmx.de> wrote:
> >
> >> Do you also have an earlier version of numpy installed? As David says,
> this
> >>should raise an error for recent numpy and
> >>I'm wondering if you are inadvertently
> >>running an earlier version.Chuck
> >
> >
> > I only have one python installation and
> > numpy.__version__ shows 1.6b.
> > I could reinstall numpy, if it would help.
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> Hi,
> I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and
> 3.1 and numpy 1.6b (fresh install)  IDLE and the command line. I can
> also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the
> same system.
>
> Actually this is 'weird' when printing and crashed with the range -
> accessing unassigned memory?
> A smaller array gives an numpy error or memory error in idle.
>
> Bruce
>
>
> >>> import numpy as np
> >>> x=np.zeros((262144, 262144))
> >>> x
> array([], shape=(262144, 262144), dtype=float64)
> >>> x[0,0]
> 2.1453735050108555e-314
> >>> x[1:10,1:10]
>
> >>> ================================ RESTART
> ================================
> >>> import numpy as np
>
> >>> x=np.zeros((26214, 26214))
>
> Traceback (most recent call last):
>  File "<pyshell#7>", line 1, in <module>
>    x=np.zeros((26214, 26214))
> ValueError: array is too big.
> >>>
> >>> x=np.zeros((262144, 26214))
>
> Traceback (most recent call last):
>  File "<pyshell#8>", line 1, in <module>
>    x=np.zeros((262144, 26214))
> MemoryError
> _____
>

This was fixed before, maybe it got broken again. Since this looks windows
specific, I'm guessing it has something to do with the size of long being 32
bits.

The previous problem was integer overflow when multiplying the dimensions
together to get the array size when repeated divisions of the maximum size
should have be used instead.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110423/b3b3ed00/attachment.html>

From ralf.gommers at googlemail.com  Sat Apr 23 14:45:49 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sat, 23 Apr 2011 20:45:49 +0200
Subject: [Numpy-discussion] status for 1.6.0
In-Reply-To: <BANLkTimbdV9SKxQobbSN06jKC8imRkRPOg@mail.gmail.com>
References: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
	<BANLkTimbdV9SKxQobbSN06jKC8imRkRPOg@mail.gmail.com>
Message-ID: <BANLkTi=PQh2M78PJSqfq37oeuT_a5dCFig@mail.gmail.com>

On Fri, Apr 22, 2011 at 11:54 PM, Mark Wiebe <mwwiebe at gmail.com> wrote:
> On Sun, Apr 17, 2011 at 8:55 AM, Ralf Gommers <ralf.gommers at googlemail.com>
> wrote:
>>
>> Hi,
>>
>> The list of open issues for 1.6.0 is down to a handful:
>>
>> - f2py segfault on Ubuntu reported by David (David, did you get any
>> further with this?)
>> - #1801: test_noncentral_f failure on win64
>> - #1798: type promotion regression (needs review,
>> https://github.com/numpy/numpy/pull/73)
>
> I've pushed this change, anyone interested should review the tests for this
> to verify that it accords with their intuition:
> https://github.com/numpy/numpy/blob/maintenance%2F1.6.x/numpy/core/tests/test_numeric.py#L372
>>
>> - API docs for new parameters to ufuncs (full list: 'casting', 'out',
>> 'order', 'subok', 'dtype'),
>
> Done.

Great, thanks.
>
>>
>> - API docs, some docstrings and update of NEP for datetime
>>
>> I'm planning to release the first release candidate next weekend if
>> these are solved. ?As far as I know no one is working on the doc
>> issues right now. It would be great if someone could step up and do
>> this.
>
> There's a new ticket #1805 which should get fixed for the RC too.

Charles already attached a patch to #1805, it would be great if you
could review that. If the fix is correct we're good to go.

Ralf


From charlesr.harris at gmail.com  Sat Apr 23 15:09:18 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 23 Apr 2011 13:09:18 -0600
Subject: [Numpy-discussion] status for 1.6.0
In-Reply-To: <BANLkTi=PQh2M78PJSqfq37oeuT_a5dCFig@mail.gmail.com>
References: <BANLkTing3mw7bGbEUN65krrVMfH-PnJxkw@mail.gmail.com>
	<BANLkTimbdV9SKxQobbSN06jKC8imRkRPOg@mail.gmail.com>
	<BANLkTi=PQh2M78PJSqfq37oeuT_a5dCFig@mail.gmail.com>
Message-ID: <BANLkTikJ+X75X1CZqnErhhsGts-pkK=89w@mail.gmail.com>

On Sat, Apr 23, 2011 at 12:45 PM, Ralf Gommers
<ralf.gommers at googlemail.com>wrote:

> On Fri, Apr 22, 2011 at 11:54 PM, Mark Wiebe <mwwiebe at gmail.com> wrote:
> > On Sun, Apr 17, 2011 at 8:55 AM, Ralf Gommers <
> ralf.gommers at googlemail.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> The list of open issues for 1.6.0 is down to a handful:
> >>
> >> - f2py segfault on Ubuntu reported by David (David, did you get any
> >> further with this?)
> >> - #1801: test_noncentral_f failure on win64
> >> - #1798: type promotion regression (needs review,
> >> https://github.com/numpy/numpy/pull/73)
> >
> > I've pushed this change, anyone interested should review the tests for
> this
> > to verify that it accords with their intuition:
> >
> https://github.com/numpy/numpy/blob/maintenance%2F1.6.x/numpy/core/tests/test_numeric.py#L372
> >>
> >> - API docs for new parameters to ufuncs (full list: 'casting', 'out',
> >> 'order', 'subok', 'dtype'),
> >
> > Done.
>
> Great, thanks.
> >
> >>
> >> - API docs, some docstrings and update of NEP for datetime
> >>
> >> I'm planning to release the first release candidate next weekend if
> >> these are solved.  As far as I know no one is working on the doc
> >> issues right now. It would be great if someone could step up and do
> >> this.
> >
> > There's a new ticket #1805 which should get fixed for the RC too.
>
> Charles already attached a patch to #1805, it would be great if you
> could review that. If the fix is correct we're good to go.
>
>
Oh, that just makes numpy run. It doesn't run correctly, though, there are
lots of failing tests. The patch is a work in progress and is up for review
and tinkering.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110423/ae20c974/attachment.html>

From cgohlke at uci.edu  Sat Apr 23 17:38:21 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Sat, 23 Apr 2011 14:38:21 -0700
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <BANLkTi=3aMctNUtOFfpRLBwnQg=L4cMp=A@mail.gmail.com>
References: <loom.20110423T153046-829@post.gmane.org>	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>	<loom.20110423T160723-665@post.gmane.org>	<BANLkTimcK7K=8wSKmDxKSjGHL=vAsXQ01A@mail.gmail.com>	<loom.20110423T165540-168@post.gmane.org>	<BANLkTimsmoyMqv+m+SxMGeHcJzbMzKBW0Q@mail.gmail.com>
	<BANLkTi=3aMctNUtOFfpRLBwnQg=L4cMp=A@mail.gmail.com>
Message-ID: <4DB346CD.8050002@uci.edu>



On 4/23/2011 10:41 AM, Charles R Harris wrote:
>
>
> On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey <bsouthey at gmail.com
> <mailto:bsouthey at gmail.com>> wrote:
>
>     On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till at gmx.de
>     <mailto:mail.till at gmx.de>> wrote:
>     >
>     > > Do you also have an earlier version of numpy installed? As David
>     says, this
>     > >should raise an error for recent numpy and
>     > >I'm wondering if you are inadvertently
>     > >running an earlier version.Chuck
>     >
>     >
>     >  I only have one python installation and
>     >  numpy.__version__ shows 1.6b.
>     >  I could reinstall numpy, if it would help.
>     >
>     >  _______________________________________________
>     >  NumPy-Discussion mailing list
>     >  NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     >  http://mail.scipy.org/mailman/listinfo/numpy-discussion
>     >
>
>     Hi,
>     I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and
>     3.1 and numpy 1.6b (fresh install)  IDLE and the command line. I can
>     also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the
>     same system.
>
>     Actually this is 'weird' when printing and crashed with the range -
>     accessing unassigned memory?
>     A smaller array gives an numpy error or memory error in idle.
>
>     Bruce
>
>
>     > >> import numpy as np
>     > >> x=np.zeros((262144, 262144))
>     > >> x
>     array([], shape=(262144, 262144), dtype=float64)
>     > >> x[0,0]
>     2.1453735050108555e-314
>     > >> x[1:10,1:10]
>
>     > >> ================================ RESTART
>     ================================
>     > >> import numpy as np
>
>     > >> x=np.zeros((26214, 26214))
>
>     Traceback (most recent call last):
>       File "<pyshell#7>", line 1, in <module>
>         x=np.zeros((26214, 26214))
>     ValueError: array is too big.
>     > >>
>     > >> x=np.zeros((262144, 26214))
>
>     Traceback (most recent call last):
>       File "<pyshell#8>", line 1, in <module>
>         x=np.zeros((262144, 26214))
>     MemoryError
>     _____
>
>
> This was fixed before, maybe it got broken again. Since this looks
> windows specific, I'm guessing it has something to do with the size of
> long being 32 bits.
>
> The previous problem was integer overflow when multiplying the
> dimensions together to get the array size when repeated divisions of the
> maximum size should have be used instead.
>
> Chuck

Could be related to this change:

<https://github.com/numpy/numpy/commit/fcc6cc73ddcb1fc85446ba9256ac24ecdda6c6d8#L1L1121>

Christoph


From charlesr.harris at gmail.com  Sat Apr 23 17:47:27 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 23 Apr 2011 15:47:27 -0600
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <4DB346CD.8050002@uci.edu>
References: <loom.20110423T153046-829@post.gmane.org>
	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>
	<loom.20110423T160723-665@post.gmane.org>
	<BANLkTimcK7K=8wSKmDxKSjGHL=vAsXQ01A@mail.gmail.com>
	<loom.20110423T165540-168@post.gmane.org>
	<BANLkTimsmoyMqv+m+SxMGeHcJzbMzKBW0Q@mail.gmail.com>
	<BANLkTi=3aMctNUtOFfpRLBwnQg=L4cMp=A@mail.gmail.com>
	<4DB346CD.8050002@uci.edu>
Message-ID: <BANLkTi=v7rCXnnQFQcfOGHZ36ajprgrByg@mail.gmail.com>

On Sat, Apr 23, 2011 at 3:38 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:

>
>
> On 4/23/2011 10:41 AM, Charles R Harris wrote:
> >
> >
> > On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey <bsouthey at gmail.com
> > <mailto:bsouthey at gmail.com>> wrote:
> >
> >     On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till at gmx.de
> >     <mailto:mail.till at gmx.de>> wrote:
> >     >
> >     > > Do you also have an earlier version of numpy installed? As David
> >     says, this
> >     > >should raise an error for recent numpy and
> >     > >I'm wondering if you are inadvertently
> >     > >running an earlier version.Chuck
> >     >
> >     >
> >     >  I only have one python installation and
> >     >  numpy.__version__ shows 1.6b.
> >     >  I could reinstall numpy, if it would help.
> >     >
> >     >  _______________________________________________
> >     >  NumPy-Discussion mailing list
> >     >  NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
> >     >  http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >     >
> >
> >     Hi,
> >     I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and
> >     3.1 and numpy 1.6b (fresh install)  IDLE and the command line. I can
> >     also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the
> >     same system.
> >
> >     Actually this is 'weird' when printing and crashed with the range -
> >     accessing unassigned memory?
> >     A smaller array gives an numpy error or memory error in idle.
> >
> >     Bruce
> >
> >
> >     > >> import numpy as np
> >     > >> x=np.zeros((262144, 262144))
> >     > >> x
> >     array([], shape=(262144, 262144), dtype=float64)
> >     > >> x[0,0]
> >     2.1453735050108555e-314
> >     > >> x[1:10,1:10]
> >
> >     > >> ================================ RESTART
> >     ================================
> >     > >> import numpy as np
> >
> >     > >> x=np.zeros((26214, 26214))
> >
> >     Traceback (most recent call last):
> >       File "<pyshell#7>", line 1, in <module>
> >         x=np.zeros((26214, 26214))
> >     ValueError: array is too big.
> >     > >>
> >     > >> x=np.zeros((262144, 26214))
> >
> >     Traceback (most recent call last):
> >       File "<pyshell#8>", line 1, in <module>
> >         x=np.zeros((262144, 26214))
> >     MemoryError
> >     _____
> >
> >
> > This was fixed before, maybe it got broken again. Since this looks
> > windows specific, I'm guessing it has something to do with the size of
> > long being 32 bits.
> >
> > The previous problem was integer overflow when multiplying the
> > dimensions together to get the array size when repeated divisions of the
> > maximum size should have be used instead.
> >
> > Chuck
>
> Could be related to this change:
>
> <
> https://github.com/numpy/numpy/commit/fcc6cc73ddcb1fc85446ba9256ac24ecdda6c6d8#L1L1121
> >
>
>
My, that does look suspicious ;) Could you revert that loop and test it out?

There was also a function for doing that check, I don't recall which, and it
should probably be checked to make sure it remains as was.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110423/1443b3d8/attachment.html>

From charlesr.harris at gmail.com  Sat Apr 23 17:52:47 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 23 Apr 2011 15:52:47 -0600
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <BANLkTi=v7rCXnnQFQcfOGHZ36ajprgrByg@mail.gmail.com>
References: <loom.20110423T153046-829@post.gmane.org>
	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>
	<loom.20110423T160723-665@post.gmane.org>
	<BANLkTimcK7K=8wSKmDxKSjGHL=vAsXQ01A@mail.gmail.com>
	<loom.20110423T165540-168@post.gmane.org>
	<BANLkTimsmoyMqv+m+SxMGeHcJzbMzKBW0Q@mail.gmail.com>
	<BANLkTi=3aMctNUtOFfpRLBwnQg=L4cMp=A@mail.gmail.com>
	<4DB346CD.8050002@uci.edu>
	<BANLkTi=v7rCXnnQFQcfOGHZ36ajprgrByg@mail.gmail.com>
Message-ID: <BANLkTim2w1reyKU2-HqDUBPv_U1NkCaCdQ@mail.gmail.com>

On Sat, Apr 23, 2011 at 3:47 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Sat, Apr 23, 2011 at 3:38 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:
>
>>
>>
>> On 4/23/2011 10:41 AM, Charles R Harris wrote:
>> >
>> >
>> > On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey <bsouthey at gmail.com
>> > <mailto:bsouthey at gmail.com>> wrote:
>> >
>> >     On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till at gmx.de
>> >     <mailto:mail.till at gmx.de>> wrote:
>> >     >
>> >     > > Do you also have an earlier version of numpy installed? As David
>> >     says, this
>> >     > >should raise an error for recent numpy and
>> >     > >I'm wondering if you are inadvertently
>> >     > >running an earlier version.Chuck
>> >     >
>> >     >
>> >     >  I only have one python installation and
>> >     >  numpy.__version__ shows 1.6b.
>> >     >  I could reinstall numpy, if it would help.
>> >     >
>> >     >  _______________________________________________
>> >     >  NumPy-Discussion mailing list
>> >     >  NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>> >     >  http://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >     >
>> >
>> >     Hi,
>> >     I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and
>> >     3.1 and numpy 1.6b (fresh install)  IDLE and the command line. I can
>> >     also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the
>> >     same system.
>> >
>> >     Actually this is 'weird' when printing and crashed with the range -
>> >     accessing unassigned memory?
>> >     A smaller array gives an numpy error or memory error in idle.
>> >
>> >     Bruce
>> >
>> >
>> >     > >> import numpy as np
>> >     > >> x=np.zeros((262144, 262144))
>> >     > >> x
>> >     array([], shape=(262144, 262144), dtype=float64)
>> >     > >> x[0,0]
>> >     2.1453735050108555e-314
>> >     > >> x[1:10,1:10]
>> >
>> >     > >> ================================ RESTART
>> >     ================================
>> >     > >> import numpy as np
>> >
>> >     > >> x=np.zeros((26214, 26214))
>> >
>> >     Traceback (most recent call last):
>> >       File "<pyshell#7>", line 1, in <module>
>> >         x=np.zeros((26214, 26214))
>> >     ValueError: array is too big.
>> >     > >>
>> >     > >> x=np.zeros((262144, 26214))
>> >
>> >     Traceback (most recent call last):
>> >       File "<pyshell#8>", line 1, in <module>
>> >         x=np.zeros((262144, 26214))
>> >     MemoryError
>> >     _____
>> >
>> >
>> > This was fixed before, maybe it got broken again. Since this looks
>> > windows specific, I'm guessing it has something to do with the size of
>> > long being 32 bits.
>> >
>> > The previous problem was integer overflow when multiplying the
>> > dimensions together to get the array size when repeated divisions of the
>> > maximum size should have be used instead.
>> >
>> > Chuck
>>
>> Could be related to this change:
>>
>> <
>> https://github.com/numpy/numpy/commit/fcc6cc73ddcb1fc85446ba9256ac24ecdda6c6d8#L1L1121
>> >
>>
>>
> My, that does look suspicious ;) Could you revert that loop and test it
> out?
>
> There was also a function for doing that check, I don't recall which, and
> it should probably be checked to make sure it remains as was.
>
>
There was also a test that should have caught this. Evidently it wasn't
quite up to snuff.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110423/e3c781d8/attachment.html>

From cgohlke at uci.edu  Sat Apr 23 18:17:50 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Sat, 23 Apr 2011 15:17:50 -0700
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <BANLkTi=v7rCXnnQFQcfOGHZ36ajprgrByg@mail.gmail.com>
References: <loom.20110423T153046-829@post.gmane.org>	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>	<loom.20110423T160723-665@post.gmane.org>	<BANLkTimcK7K=8wSKmDxKSjGHL=vAsXQ01A@mail.gmail.com>	<loom.20110423T165540-168@post.gmane.org>	<BANLkTimsmoyMqv+m+SxMGeHcJzbMzKBW0Q@mail.gmail.com>	<BANLkTi=3aMctNUtOFfpRLBwnQg=L4cMp=A@mail.gmail.com>	<4DB346CD.8050002@uci.edu>
	<BANLkTi=v7rCXnnQFQcfOGHZ36ajprgrByg@mail.gmail.com>
Message-ID: <4DB3500E.2030002@uci.edu>



On 4/23/2011 2:47 PM, Charles R Harris wrote:
>
>
> On Sat, Apr 23, 2011 at 3:38 PM, Christoph Gohlke <cgohlke at uci.edu
> <mailto:cgohlke at uci.edu>> wrote:
>
>
>
>     On 4/23/2011 10:41 AM, Charles R Harris wrote:
>     >
>     >
>     >  On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey
>     <bsouthey at gmail.com <mailto:bsouthey at gmail.com>
>     >  <mailto:bsouthey at gmail.com <mailto:bsouthey at gmail.com>>> wrote:
>     >
>     >      On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki
>     <mail.till at gmx.de <mailto:mail.till at gmx.de>
>     >  <mailto:mail.till at gmx.de <mailto:mail.till at gmx.de>>> wrote:
>     >  >
>     >  > > Do you also have an earlier version of numpy installed? As David
>     >      says, this
>     >  > >should raise an error for recent numpy and
>     >  > >I'm wondering if you are inadvertently
>     >  > >running an earlier version.Chuck
>     >  >
>     >  >
>     >  >  I only have one python installation and
>     >  >  numpy.__version__ shows 1.6b.
>     >  >  I could reinstall numpy, if it would help.
>     >  >
>     >  >  _______________________________________________
>     >  >  NumPy-Discussion mailing list
>     >  > NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     <mailto:NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>>
>     >  > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>     >  >
>     >
>     >      Hi,
>     >      I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7
>     (below) and
>     >      3.1 and numpy 1.6b (fresh install)  IDLE and the command line.
>     I can
>     >      also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the
>     >      same system.
>     >
>     >      Actually this is 'weird' when printing and crashed with the
>     range -
>     >      accessing unassigned memory?
>     >      A smaller array gives an numpy error or memory error in idle.
>     >
>     >      Bruce
>     >
>     >
>     >  > >> import numpy as np
>     >  > >> x=np.zeros((262144, 262144))
>     >  > >> x
>     >      array([], shape=(262144, 262144), dtype=float64)
>     >  > >> x[0,0]
>     >      2.1453735050108555e-314
>     >  > >> x[1:10,1:10]
>     >
>     >  > >> ================================ RESTART
>     >      ================================
>     >  > >> import numpy as np
>     >
>     >  > >> x=np.zeros((26214, 26214))
>     >
>     >      Traceback (most recent call last):
>     >        File "<pyshell#7>", line 1, in <module>
>     >          x=np.zeros((26214, 26214))
>     >      ValueError: array is too big.
>     >  > >>
>     >  > >> x=np.zeros((262144, 26214))
>     >
>     >      Traceback (most recent call last):
>     >        File "<pyshell#8>", line 1, in <module>
>     >          x=np.zeros((262144, 26214))
>     >      MemoryError
>     >      _____
>     >
>     >
>     >  This was fixed before, maybe it got broken again. Since this looks
>     >  windows specific, I'm guessing it has something to do with the size of
>     >  long being 32 bits.
>     >
>     >  The previous problem was integer overflow when multiplying the
>     >  dimensions together to get the array size when repeated divisions
>     of the
>     >  maximum size should have be used instead.
>     >
>     >  Chuck
>
>     Could be related to this change:
>
>     <https://github.com/numpy/numpy/commit/fcc6cc73ddcb1fc85446ba9256ac24ecdda6c6d8#L1L1121>
>
>
> My, that does look suspicious ;) Could you revert that loop and test it out?
>
> There was also a function for doing that check, I don't recall which,
> and it should probably be checked to make sure it remains as was.
>
> Chuck
>
>


Reverting the change worked. A patch is attached.

>>> import numpy as np
>>> np.__version__
'1.6.0b3'
>>> x=np.zeros((262144, 262144))
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ValueError: array is too big.


Christoph
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: array_is_too_big.diff
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110423/83cac68f/attachment.ksh>

From charlesr.harris at gmail.com  Sat Apr 23 18:59:00 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 23 Apr 2011 16:59:00 -0600
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <4DB3500E.2030002@uci.edu>
References: <loom.20110423T153046-829@post.gmane.org>
	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>
	<loom.20110423T160723-665@post.gmane.org>
	<BANLkTimcK7K=8wSKmDxKSjGHL=vAsXQ01A@mail.gmail.com>
	<loom.20110423T165540-168@post.gmane.org>
	<BANLkTimsmoyMqv+m+SxMGeHcJzbMzKBW0Q@mail.gmail.com>
	<BANLkTi=3aMctNUtOFfpRLBwnQg=L4cMp=A@mail.gmail.com>
	<4DB346CD.8050002@uci.edu>
	<BANLkTi=v7rCXnnQFQcfOGHZ36ajprgrByg@mail.gmail.com>
	<4DB3500E.2030002@uci.edu>
Message-ID: <BANLkTikbP9f=B0U_fnfr8JvDEZm+qxU2Ew@mail.gmail.com>

On Sat, Apr 23, 2011 at 4:17 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:

>
>
> On 4/23/2011 2:47 PM, Charles R Harris wrote:
>
>>
>>
>> On Sat, Apr 23, 2011 at 3:38 PM, Christoph Gohlke <cgohlke at uci.edu
>> <mailto:cgohlke at uci.edu>> wrote:
>>
>>
>>
>>    On 4/23/2011 10:41 AM, Charles R Harris wrote:
>>    >
>>    >
>>    >  On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey
>>    <bsouthey at gmail.com <mailto:bsouthey at gmail.com>
>>    >  <mailto:bsouthey at gmail.com <mailto:bsouthey at gmail.com>>> wrote:
>>    >
>>    >      On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki
>>    <mail.till at gmx.de <mailto:mail.till at gmx.de>
>>    >  <mailto:mail.till at gmx.de <mailto:mail.till at gmx.de>>> wrote:
>>    >  >
>>    >  > > Do you also have an earlier version of numpy installed? As David
>>    >      says, this
>>    >  > >should raise an error for recent numpy and
>>    >  > >I'm wondering if you are inadvertently
>>    >  > >running an earlier version.Chuck
>>    >  >
>>    >  >
>>    >  >  I only have one python installation and
>>    >  >  numpy.__version__ shows 1.6b.
>>    >  >  I could reinstall numpy, if it would help.
>>    >  >
>>    >  >  _______________________________________________
>>    >  >  NumPy-Discussion mailing list
>>    >  > NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>>    <mailto:NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org
>> >>
>>
>>    >  > http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>    >  >
>>    >
>>    >      Hi,
>>    >      I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7
>>    (below) and
>>    >      3.1 and numpy 1.6b (fresh install)  IDLE and the command line.
>>    I can
>>    >      also confirm the 'ValueError' with Python2.6 and numpy 1.51 on
>> the
>>    >      same system.
>>    >
>>    >      Actually this is 'weird' when printing and crashed with the
>>    range -
>>    >      accessing unassigned memory?
>>    >      A smaller array gives an numpy error or memory error in idle.
>>    >
>>    >      Bruce
>>    >
>>    >
>>    >  > >> import numpy as np
>>    >  > >> x=np.zeros((262144, 262144))
>>    >  > >> x
>>    >      array([], shape=(262144, 262144), dtype=float64)
>>    >  > >> x[0,0]
>>    >      2.1453735050108555e-314
>>    >  > >> x[1:10,1:10]
>>    >
>>    >  > >> ================================ RESTART
>>    >      ================================
>>    >  > >> import numpy as np
>>    >
>>    >  > >> x=np.zeros((26214, 26214))
>>    >
>>    >      Traceback (most recent call last):
>>    >        File "<pyshell#7>", line 1, in <module>
>>    >          x=np.zeros((26214, 26214))
>>    >      ValueError: array is too big.
>>    >  > >>
>>    >  > >> x=np.zeros((262144, 26214))
>>    >
>>    >      Traceback (most recent call last):
>>    >        File "<pyshell#8>", line 1, in <module>
>>    >          x=np.zeros((262144, 26214))
>>    >      MemoryError
>>    >      _____
>>    >
>>    >
>>    >  This was fixed before, maybe it got broken again. Since this looks
>>    >  windows specific, I'm guessing it has something to do with the size
>> of
>>    >  long being 32 bits.
>>    >
>>    >  The previous problem was integer overflow when multiplying the
>>    >  dimensions together to get the array size when repeated divisions
>>    of the
>>    >  maximum size should have be used instead.
>>    >
>>    >  Chuck
>>
>>    Could be related to this change:
>>
>>    <
>> https://github.com/numpy/numpy/commit/fcc6cc73ddcb1fc85446ba9256ac24ecdda6c6d8#L1L1121
>> >
>>
>>
>> My, that does look suspicious ;) Could you revert that loop and test it
>> out?
>>
>> There was also a function for doing that check, I don't recall which,
>> and it should probably be checked to make sure it remains as was.
>>
>> Chuck
>>
>>
>>
>
> Reverting the change worked. A patch is attached.
>
>  import numpy as np
>>>> np.__version__
>>>>
>>> '1.6.0b3'
>
>  x=np.zeros((262144, 262144))
>>>>
>>> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>
> ValueError: array is too big.
>
>
Thanks, applied.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110423/0af8a7a8/attachment.html>

From sturla at molden.no  Sat Apr 23 21:04:32 2011
From: sturla at molden.no (Sturla Molden)
Date: Sun, 24 Apr 2011 03:04:32 +0200
Subject: [Numpy-discussion] Python crashes while printing a huge numpy
 array (Windows 1.6b2)
In-Reply-To: <BANLkTi=Rwp3UfAR3mZ0L6B3c0tW_f1LjHQ@mail.gmail.com>
References: <loom.20110423T153046-829@post.gmane.org>	<BANLkTinw+uzMxEdDmm4vHyO5TYQ9s0M2TA@mail.gmail.com>	<BANLkTinRVLG_vsxsyV2wz6n=oXgxXZOZ_Q@mail.gmail.com>
	<BANLkTi=Rwp3UfAR3mZ0L6B3c0tW_f1LjHQ@mail.gmail.com>
Message-ID: <4DB37720.5010902@molden.no>

Den 23.04.2011 16:09, skrev josef.pktd at gmail.com:
> to rule out two more cases
>
> numpy 1.5.1 on python 2.7.1 32 bit : ValueError
> numpy 1.5.1 on python 3.2    64 bit: MemoryError
>

numpy 1.5.1 on Python 2.7.1 64 bit: MemoryError


Sturla





From davide.lasagna at polito.it  Sun Apr 24 10:53:16 2011
From: davide.lasagna at polito.it (Davide)
Date: Sun, 24 Apr 2011 16:53:16 +0200
Subject: [Numpy-discussion] [ANN]: OpenPiv. A python package for PIV image
	analysis
Message-ID: <4DB4395C.3060605@polito.it>

Hi all,

I am pleased to announce the re-birth of OpenPiv, a python package for 
the analysis of PIV images. PIV is a well established optical 
measurement technique for fluid flows. The package has the goal of 
provide an implementation of state-of-the-art processing algorithms, and 
compete with commercial closed-source software packages.

OpenPiv is in alpha state and several thing has to be done. However, the 
code is already usable and a pure python implementation of the standard 
cross-correlation algorithm is already available.

OpenPiv needs developers and contributors, helping in with code and 
documentation. If you can provide some, and if you are interested in 
becoming a core developer, please drop us an email at openpiv-develop at 
lists dot sourceforge dot net. A draft website can be found at 
www.openpiv.net/python.

Thanks for your attention,

Davide Lasagna


From ralf.gommers at googlemail.com  Sun Apr 24 12:48:33 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sun, 24 Apr 2011 18:48:33 +0200
Subject: [Numpy-discussion] assert and allclose
In-Reply-To: <BANLkTin=DHJCcJNDhu09eEugEcgJCVzedg@mail.gmail.com>
References: <BANLkTin=DHJCcJNDhu09eEugEcgJCVzedg@mail.gmail.com>
Message-ID: <BANLkTinQjkJz6Fo5ns1NHg3aaGBfgG2BSg@mail.gmail.com>

On Mon, Apr 18, 2011 at 5:59 AM,  <josef.pktd at gmail.com> wrote:
> Some observations about allclose
>
> I have a test case where I would like to impose absolute
> (logical_)and relative errors
>
> (Pdb) pdf_r
> array([ ?2.97341655e-90, ? 3.68756271e-24, ? 2.01840159e-07,
> ? ? ? ? 3.98942280e-01, ? 4.83941449e-01, ? 1.07981933e-01])
> (Pdb) pdf_st
> array([ ?3.09721604e-90, ? 3.69697466e-24, ? 2.01872148e-07,
> ? ? ? ? 3.98942181e-01, ? 4.83941207e-01, ? 1.07982122e-01])
> (Pdb) pdf_st - pdf_r
> array([ ?1.23799491e-91, ? 9.41194691e-27, ? 3.19891525e-11,
> ? ? ? ?-9.95821123e-08, ?-2.41764579e-07, ? 1.89015140e-07])
> (Pdb) (pdf_st - pdf_r) / pdf_r
> array([ ?4.16354347e-02, ? 2.55234897e-03, ? 1.58487551e-04,
> ? ? ? ?-2.49615338e-07, ?-4.99574028e-07, ? 1.75043301e-06])
>
> allclose just uses the sum of abolute and relative errors, which I
> think is binding in the wrong way, for what I want.
>
> (Pdb) np.allclose(pdf_st, pdf_r, rtol=1e-4, atol=1e-8)
> True
>
> the relative and absolute errors are to tight on one side and too
> loose on the other side
>
> (Pdb) np.allclose(pdf_st, pdf_r, rtol=1e-2, atol=0)
> False
> (Pdb) np.allclose(pdf_st, pdf_r, rtol=0, atol=1e-8)
> False
>
> tighter bounds for both relative and absolute errors if I impose both separately
>
> (Pdb) np.allclose(pdf_st, pdf_r, rtol=0, atol=1e-6)
> True
> (Pdb) np.allclose(pdf_st, pdf_r, rtol=1e-1, atol=0)
> True
>
> Are there any alternatives to this? (or am I too tired)

This indeed doesn't seem to work well for your purpose, and I can't
think of anything better than doing the logical_and by hand. Perhaps
we can build this behavior (separate atol and rtol comparisons, then
"and" those) into allclose with a keyword to select it?

It would also be cleaner to do the rtol comparison with 2 * abs(a - b)
/ (abs(a) + abs(b), to remove the asymmetric response to input
arguments.

Ralf

> (test cases are for statistical distributions, where all values are
> smaller than one, and some values can be very small)


From gokhansever at gmail.com  Sun Apr 24 18:32:19 2011
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Sun, 24 Apr 2011 16:32:19 -0600
Subject: [Numpy-discussion] Segfault using "fromstring" and reading
 variable length string
In-Reply-To: <BANLkTiktB2UPjqw2NUggyObgNWo8xyaqjA@mail.gmail.com>
References: <BANLkTi=oKoYtzRX9QELOkJBd0OBz3rUQ9Q@mail.gmail.com>
	<BANLkTin7xdG2ZZ5UzpDfmp=Un68J1eHdRA@mail.gmail.com>
	<BANLkTim1Y+bXXZbnu9rs3sds8sit9C=Bpw@mail.gmail.com>
	<4DB21873.2060601@uci.edu>
	<BANLkTiktB2UPjqw2NUggyObgNWo8xyaqjA@mail.gmail.com>
Message-ID: <BANLkTim9xo_wyv=xaRJ1NOTpFQENOP8KrQ@mail.gmail.com>

On Fri, Apr 22, 2011 at 6:32 PM, Mark Wiebe <mwwiebe at gmail.com> wrote:

> I took a quick look at this issue and committed a fix. PyArray_FromString
> was doing a check to exclude object arrays, but that check was incorrect.
> Now it should appropriately raise an exception instead of creating an
> invalid array.
>
>
> https://github.com/numpy/numpy/commit/f75bfab3a2ab74ac82047f153a36c71c58fe3732
>
> -Mark
>

Thanks for the fix.

Now that line yields:

I[6]: k = np.fromstring(c.read(dt.itemsize), dt)[0]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

/home/gsever/Desktop/python-repo/numpy/<ipython console> in <module>()

ValueError: Cannot create an object array from a string


-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110424/ed162070/attachment.html>

From dileepkunjaai at gmail.com  Mon Apr 25 02:50:53 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Mon, 25 Apr 2011 12:20:53 +0530
Subject: [Numpy-discussion] Array as Variable using "from cdms2 import MV2
	as MV"
Message-ID: <BANLkTim-6kJJK3_Wo-YUh9F2YCJ__Dr8Cw@mail.gmail.com>

Dear sir,

     I am have 2  mxn numpy array say "obs" & "fcst". I have to
calculate    * sum of squre of (obs[i, j]-fcst[i, j])* using     from cdms2
import MV2 as MV   in CDAT without using "for"  loop.

For example:
obs=
[0.6    1.1    0.02    0.2   0.2
0.8    0.    0.    0.4   0.8
0.5    5.5    1.5    0.5   1.5
3.5    0.5    1.5    5.0   2.6
5.1    4.1    3.2    2.3   1.5
4.4    0.9    1.5    2.    2.3
1.1    1.1    1.5    12.6  1.3
2.2    12    1.7    1.6   15
1.9    1.5    0.9    2.5   5.5 ]



fcst=

[0.7    0.1    0.2    0.2   0.2
0.3    0.8    0.    0.    0.
0.5    0.5    0.5    0.5   0.5
0.7    1.     1.5    2.    2.6
5.1    4.1    3.2    2.3   1.5
0.7    1.    1.5    2.    2.3
1.1    1.1    1.1    12.7  1.3
2.2    2.    1.7    1.6   1.5
1.9    1.5    0.9    0.5   7.5]

here "obs" and "fcst" are numpy array
I give

obs=MV.array(obs)
fcst=MV.array(fcst)

Then it become


sbst=obs-fcst

>> subst=
[[ -0.1    1.    -0.18   0.     0.  ]
 [  0.5   -0.8    0.     0.4    0.8 ]
 [  0.     5.     1.     0.     1.  ]
 [  2.8   -0.5    0.     3.     0.  ]
 [  0.     0.     0.     0.     0.  ]
 [  3.7   -0.1    0.     0.     0.  ]
 [  0.     0.     0.4   -0.1    0.  ]
 [  0.    10.     0.     0.    13.5 ]
 [  0.     0.     0.     2.    -2.  ]]

But i dont know how to find sum of squre of each term....(Actually my aim is
to finding MEAN SQUARED ERROR)




Thanking you...................

-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110425/2be25f4c/attachment.html>

From josef.pktd at gmail.com  Mon Apr 25 03:17:42 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 25 Apr 2011 03:17:42 -0400
Subject: [Numpy-discussion] Array as Variable using "from cdms2 import
 MV2 as MV"
In-Reply-To: <BANLkTim-6kJJK3_Wo-YUh9F2YCJ__Dr8Cw@mail.gmail.com>
References: <BANLkTim-6kJJK3_Wo-YUh9F2YCJ__Dr8Cw@mail.gmail.com>
Message-ID: <BANLkTi=oW97r5OwwHLEZzfbTAPSr1-X6+w@mail.gmail.com>

On Mon, Apr 25, 2011 at 2:50 AM, dileep kunjaai <dileepkunjaai at gmail.com> wrote:
> Dear sir,
>
> ???? I am have 2? mxn numpy array say "obs" & "fcst". I have to
> calculate???? sum of squre of (obs[i, j]-fcst[i, j]) using???? from cdms2
> import MV2 as MV ? in CDAT without using "for"? loop.
>
> For example:
> obs=
> [0.6?? ?1.1?? ?0.02?? ?0.2?? 0.2
> 0.8?? ?0.?? ?0.?? ?0.4?? 0.8
> 0.5?? ?5.5?? ?1.5?? ?0.5?? 1.5
> 3.5?? ?0.5?? ?1.5?? ?5.0?? 2.6
> 5.1?? ?4.1?? ?3.2?? ?2.3?? 1.5
> 4.4?? ?0.9?? ?1.5?? ?2.??? 2.3
> 1.1?? ?1.1?? ?1.5?? ?12.6? 1.3
> 2.2?? ?12?? ?1.7?? ?1.6?? 15
> 1.9?? ?1.5?? ?0.9?? ?2.5?? 5.5 ]
>
>
>
> fcst=
>
> [0.7??? 0.1??? 0.2??? 0.2?? 0.2
> 0.3??? 0.8??? 0.??? 0.??? 0.
> 0.5??? 0.5??? 0.5??? 0.5?? 0.5
> 0.7??? 1. ??? 1.5??? 2.??? 2.6
> 5.1??? 4.1??? 3.2??? 2.3?? 1.5
> 0.7??? 1.??? 1.5??? 2.??? 2.3
> 1.1??? 1.1??? 1.1??? 12.7? 1.3
> 2.2??? 2.??? 1.7??? 1.6?? 1.5
> 1.9??? 1.5??? 0.9??? 0.5?? 7.5]
>
> here "obs" and "fcst" are numpy array
> I give
>
> obs=MV.array(obs)
> fcst=MV.array(fcst)
>
> Then it become
>
>
> sbst=obs-fcst
>
>>> subst=
> [[ -0.1??? 1.??? -0.18?? 0.???? 0.? ]
> ?[? 0.5?? -0.8??? 0.???? 0.4??? 0.8 ]
> ?[? 0.???? 5.???? 1.???? 0.???? 1.? ]
> ?[? 2.8?? -0.5??? 0.???? 3.???? 0.? ]
> ?[? 0.???? 0.???? 0.???? 0.???? 0.? ]
> ?[? 3.7?? -0.1??? 0.???? 0.???? 0.? ]
> ?[? 0.???? 0.???? 0.4?? -0.1??? 0.? ]
> ?[? 0.??? 10.???? 0.???? 0.??? 13.5 ]
> ?[? 0.???? 0.???? 0.???? 2.??? -2.? ]]
>
> But i dont know how to find sum of squre of each term....(Actually my aim is
> to finding MEAN SQUARED ERROR)

(sbst**2).sum()

or with sum along columns
(sbst**2).sum(0)

explanation is in the documentation

Josef


>
>
>
>
> Thanking you...................
>
> --
> DILEEPKUMAR. R
> J R F, IIT DELHI
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>

From dileepkunjaai at gmail.com  Mon Apr 25 05:36:45 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Mon, 25 Apr 2011 15:06:45 +0530
Subject: [Numpy-discussion] Array as Variable using "from cdms2 import
 MV2 as MV"
In-Reply-To: <BANLkTi=oW97r5OwwHLEZzfbTAPSr1-X6+w@mail.gmail.com>
References: <BANLkTim-6kJJK3_Wo-YUh9F2YCJ__Dr8Cw@mail.gmail.com>
	<BANLkTi=oW97r5OwwHLEZzfbTAPSr1-X6+w@mail.gmail.com>
Message-ID: <BANLkTi=OxQvkBwMYO0rqepyvQ1qQLnxsDQ@mail.gmail.com>

Thank you sir .......thanks a lot
                                                      Keep in touch




On Mon, Apr 25, 2011 at 12:47 PM, <josef.pktd at gmail.com> wrote:

> On Mon, Apr 25, 2011 at 2:50 AM, dileep kunjaai <dileepkunjaai at gmail.com>
> wrote:
> > Dear sir,
> >
> >      I am have 2  mxn numpy array say "obs" & "fcst". I have to
> > calculate     sum of squre of (obs[i, j]-fcst[i, j]) using     from cdms2
> > import MV2 as MV   in CDAT without using "for"  loop.
> >
> > For example:
> > obs=
> > [0.6    1.1    0.02    0.2   0.2
> > 0.8    0.    0.    0.4   0.8
> > 0.5    5.5    1.5    0.5   1.5
> > 3.5    0.5    1.5    5.0   2.6
> > 5.1    4.1    3.2    2.3   1.5
> > 4.4    0.9    1.5    2.    2.3
> > 1.1    1.1    1.5    12.6  1.3
> > 2.2    12    1.7    1.6   15
> > 1.9    1.5    0.9    2.5   5.5 ]
> >
> >
> >
> > fcst=
> >
> > [0.7    0.1    0.2    0.2   0.2
> > 0.3    0.8    0.    0.    0.
> > 0.5    0.5    0.5    0.5   0.5
> > 0.7    1.     1.5    2.    2.6
> > 5.1    4.1    3.2    2.3   1.5
> > 0.7    1.    1.5    2.    2.3
> > 1.1    1.1    1.1    12.7  1.3
> > 2.2    2.    1.7    1.6   1.5
> > 1.9    1.5    0.9    0.5   7.5]
> >
> > here "obs" and "fcst" are numpy array
> > I give
> >
> > obs=MV.array(obs)
> > fcst=MV.array(fcst)
> >
> > Then it become
> >
> >
> > sbst=obs-fcst
> >
> >>> subst=
> > [[ -0.1    1.    -0.18   0.     0.  ]
> >  [  0.5   -0.8    0.     0.4    0.8 ]
> >  [  0.     5.     1.     0.     1.  ]
> >  [  2.8   -0.5    0.     3.     0.  ]
> >  [  0.     0.     0.     0.     0.  ]
> >  [  3.7   -0.1    0.     0.     0.  ]
> >  [  0.     0.     0.4   -0.1    0.  ]
> >  [  0.    10.     0.     0.    13.5 ]
> >  [  0.     0.     0.     2.    -2.  ]]
> >
> > But i dont know how to find sum of squre of each term....(Actually my aim
> is
> > to finding MEAN SQUARED ERROR)
>
> (sbst**2).sum()
>
> or with sum along columns
> (sbst**2).sum(0)
>
> explanation is in the documentation
>
> Josef
>
>
> >
> >
> >
> >
> > Thanking you...................
> >
> > --
> > DILEEPKUMAR. R
> > J R F, IIT DELHI
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110425/37d8efb0/attachment.html>

From dileepkunjaai at gmail.com  Mon Apr 25 05:50:33 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Mon, 25 Apr 2011 15:20:33 +0530
Subject: [Numpy-discussion] cdms2.tvariable.TransientVariable to float
	variable
Message-ID: <BANLkTinF1HtG2DugpCjTDjUPg1AoXD_qCw@mail.gmail.com>

Dear sir,
    How to change  cdms2.tvariable.TransientVariable to float. Actually this
"cdms2.tvariable.TransientVariable" obtained from using  from cdms2 import
MV2

Thanking you



-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110425/7712a5ab/attachment.html>

From dileepkunjaai at gmail.com  Mon Apr 25 05:56:02 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Mon, 25 Apr 2011 15:26:02 +0530
Subject: [Numpy-discussion] cdms2.tvariable.TransientVariable to float
	variable
In-Reply-To: <BANLkTinF1HtG2DugpCjTDjUPg1AoXD_qCw@mail.gmail.com>
References: <BANLkTinF1HtG2DugpCjTDjUPg1AoXD_qCw@mail.gmail.com>
Message-ID: <BANLkTikcKuZmGTrtqUcd96rUyLEwacNcKQ@mail.gmail.com>

SORRRY FRIENDS THAT I WRONGLY SENT............................

On Mon, Apr 25, 2011 at 3:20 PM, dileep kunjaai <dileepkunjaai at gmail.com>wrote:

> Dear sir,
>     How to change  cdms2.tvariable.TransientVariable to float. Actually
> this "cdms2.tvariable.TransientVariable" obtained from using  from cdms2
> import MV2
>
> Thanking you
>
>
>
> --
> DILEEPKUMAR. R
> J R F, IIT DELHI
>
>


-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110425/ca51c40d/attachment.html>

From totonixsame at gmail.com  Mon Apr 25 07:16:12 2011
From: totonixsame at gmail.com (Thiago Franco Moraes)
Date: Mon, 25 Apr 2011 08:16:12 -0300
Subject: [Numpy-discussion] Memmap with multiprocessing
In-Reply-To: <4DAE2242.9030506@gmail.com>
References: <4DAE2242.9030506@gmail.com>
Message-ID: <BANLkTi=SsoQ-mbTokYBO_+rL099myYNTkA@mail.gmail.com>

Hi,

Has anyone confirmed if this is a bug? Should I post this in the bug tracker?

Thanks!

On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes
<totonixsame at gmail.com> wrote:
> Hi all,
>
> I'm having a error using memmap objects shared among processes created
> by the multprocessing module. This error only happen in Windows with
> numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac
> OS X it doesn't happen. This error is demonstrated by this little
> example script here https://gist.github.com/929168 , and the traceback
> is bellow (between <traceback> tags):
>
> <traceback>
> Process Process-1:
> Traceback (most recent call last):
> ?File "C:\Python26\Lib\multiprocessing\process.py", line 232, in _bootstrap
> ? ?self.run()
> ?File "C:\Python26\Lib\multiprocessing\process.py", line 88, in run
> ? ?self._target(*self._args, **self._kwargs)
> ?File "C:\Documents and Settings\phamorim\Desktop\test.py", line 7, in
> print_ma
> trix
> ? ?print matrix
> ?File "C:\Python26\Lib\site-packages\numpy\core\numeric.py", line 1379, in
> arra
> y_str
> ? ?return array2string(a, max_line_width, precision, suppress_small, ' ',
> "", s
> tr)
> ?File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 309, in
> ar
> ray2string
> ? ?separator, prefix)
> ?File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 189, in
> _a
> rray2string
> ? ?data = _leading_trailing(a)
> ?File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 162, in
> _l
> eading_trailing
> ? ?min(len(a), _summaryEdgeItems))]
> ?File "C:\Python26\Lib\site-packages\numpy\core\memmap.py", line 257, in
> __arra
> y_finalize__
> ? ?self.filename = obj.filename
> AttributeError: 'memmap' object has no attribute 'filename'
> Exception AttributeError: AttributeError("'NoneType' object has no attribute
> 'te
> ll'",) in <bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0], dtype=int16)> ignored
> </traceback>
>
> I don't know if it's a bug, but I thought it's is import to report
> because the version 1.4.1 was working and 1.5.0 and above was not.
>
> Thanks!
>
>


From silva at lma.cnrs-mrs.fr  Mon Apr 25 08:38:07 2011
From: silva at lma.cnrs-mrs.fr (Fabrice Silva)
Date: Mon, 25 Apr 2011 14:38:07 +0200
Subject: [Numpy-discussion] [ANN]: OpenPiv. A python package for PIV
 image analysis
In-Reply-To: <4DB4395C.3060605@polito.it>
References: <4DB4395C.3060605@polito.it>
Message-ID: <1303735087.2506.15.camel@Portable-s2m.cnrs-mrs.fr>

Le dimanche 24 avril 2011 ? 16:53 +0200, Davide a ?crit :
> Hi all,
> I am pleased to announce the re-birth of OpenPiv, a python package for 
> the analysis of PIV images. PIV is a well established optical 
> measurement technique for fluid flows. The package has the goal of 
> provide an implementation of state-of-the-art processing algorithms, and 
> compete with commercial closed-source software packages.
> OpenPiv is in alpha state and several thing has to be done. However, the 
> code is already usable and a pure python implementation of the standard 
> cross-correlation algorithm is already available.
> OpenPiv needs developers and contributors, helping in with code and 
> documentation. If you can provide some, and if you are interested in 
> becoming a core developer, please drop us an email at openpiv-develop at 
> lists dot sourceforge dot net. A draft website can be found at 
> www.openpiv.net/python.
> 
> Thanks for your attention,
> 
> Davide Lasagna

Hi Davide
Investigations in fluid mechanics lead me to build a few months a
wrapper on the I/O functions from Davis (LaVision), a commercial
solution used to handle the complete procedure from camera calibration
to postprocessing velocity fields. 
I had to postprocess previous measurements stored in Davis format to a
deeper analysis than their software allows, so that I needed a tool able
to read data from their (almost documented) format.
The solution I found googling were not satisfying at the moment. For
example, PIVMAT [2] only worked for 2D fields and we were dealing with
3D flows and Stereoscopic PIV. After speaking with it author, it appears
that it would require too much work to support the third component in a
decent way.

The ctypes-based wrappers are available here
https://launchpad.net/libim7 
https://github.com/FabricioS/libim7 
It is a quite dumb piece of code, but it allows to extract images or
velocity fields from binary files (extensions im7, imx and vc7),
avoiding some of the traps in the extraction of the data.

I would be happy if you consider useful this tool within OpenPIV. It
would also be possible to change the ctype dependance to cython...

Best regards


[1] http://www.lavision.de/en/products/davis.php
[2] http://www.fast.u-psud.fr/pivmat/
-- 
Fabrice Silva



From kwgoodman at gmail.com  Mon Apr 25 10:32:08 2011
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 25 Apr 2011 07:32:08 -0700
Subject: [Numpy-discussion] Array as Variable using "from cdms2 import
 MV2 as MV"
In-Reply-To: <BANLkTi=oW97r5OwwHLEZzfbTAPSr1-X6+w@mail.gmail.com>
References: <BANLkTim-6kJJK3_Wo-YUh9F2YCJ__Dr8Cw@mail.gmail.com>
	<BANLkTi=oW97r5OwwHLEZzfbTAPSr1-X6+w@mail.gmail.com>
Message-ID: <BANLkTi=UPtxzDg1m=3pAQSKmW0nGH0epJQ@mail.gmail.com>

On Mon, Apr 25, 2011 at 12:17 AM,  <josef.pktd at gmail.com> wrote:
> On Mon, Apr 25, 2011 at 2:50 AM, dileep kunjaai <dileepkunjaai at gmail.com> wrote:
>> Dear sir,
>>
>> ???? I am have 2? mxn numpy array say "obs" & "fcst". I have to
>> calculate???? sum of squre of (obs[i, j]-fcst[i, j]) using???? from cdms2
>> import MV2 as MV ? in CDAT without using "for"? loop.
>>
>> For example:
>> obs=
>> [0.6?? ?1.1?? ?0.02?? ?0.2?? 0.2
>> 0.8?? ?0.?? ?0.?? ?0.4?? 0.8
>> 0.5?? ?5.5?? ?1.5?? ?0.5?? 1.5
>> 3.5?? ?0.5?? ?1.5?? ?5.0?? 2.6
>> 5.1?? ?4.1?? ?3.2?? ?2.3?? 1.5
>> 4.4?? ?0.9?? ?1.5?? ?2.??? 2.3
>> 1.1?? ?1.1?? ?1.5?? ?12.6? 1.3
>> 2.2?? ?12?? ?1.7?? ?1.6?? 15
>> 1.9?? ?1.5?? ?0.9?? ?2.5?? 5.5 ]
>>
>>
>>
>> fcst=
>>
>> [0.7??? 0.1??? 0.2??? 0.2?? 0.2
>> 0.3??? 0.8??? 0.??? 0.??? 0.
>> 0.5??? 0.5??? 0.5??? 0.5?? 0.5
>> 0.7??? 1. ??? 1.5??? 2.??? 2.6
>> 5.1??? 4.1??? 3.2??? 2.3?? 1.5
>> 0.7??? 1.??? 1.5??? 2.??? 2.3
>> 1.1??? 1.1??? 1.1??? 12.7? 1.3
>> 2.2??? 2.??? 1.7??? 1.6?? 1.5
>> 1.9??? 1.5??? 0.9??? 0.5?? 7.5]
>>
>> here "obs" and "fcst" are numpy array
>> I give
>>
>> obs=MV.array(obs)
>> fcst=MV.array(fcst)
>>
>> Then it become
>>
>>
>> sbst=obs-fcst
>>
>>>> subst=
>> [[ -0.1??? 1.??? -0.18?? 0.???? 0.? ]
>> ?[? 0.5?? -0.8??? 0.???? 0.4??? 0.8 ]
>> ?[? 0.???? 5.???? 1.???? 0.???? 1.? ]
>> ?[? 2.8?? -0.5??? 0.???? 3.???? 0.? ]
>> ?[? 0.???? 0.???? 0.???? 0.???? 0.? ]
>> ?[? 3.7?? -0.1??? 0.???? 0.???? 0.? ]
>> ?[? 0.???? 0.???? 0.4?? -0.1??? 0.? ]
>> ?[? 0.??? 10.???? 0.???? 0.??? 13.5 ]
>> ?[? 0.???? 0.???? 0.???? 2.??? -2.? ]]
>>
>> But i dont know how to find sum of squre of each term....(Actually my aim is
>> to finding MEAN SQUARED ERROR)
>
> (sbst**2).sum()
>
> or with sum along columns
> (sbst**2).sum(0)
>
> explanation is in the documentation

If speed is an issue then the development version of bottleneck
(0.5.dev) has a fast sum of squares function:

    >> a = np.random.rand(1000, 10)
    >> timeit (a * a).sum(1)
    10000 loops, best of 3: 143 us per loop
    >> timeit bn.ss(a, 1)
    100000 loops, best of 3: 14.3 us per loop

It is a faster replacement for scipy.stats.ss:

    >> from scipy import stats
    >> timeit stats.ss(a, 1)
    10000 loops, best of 3: 159 us per loop

Speed up factor depends on the shape of the input array and the axis:

    >> timeit stats.ss(a, 0)
    10000 loops, best of 3: 42.8 us per loop
    >> timeit bn.ss(a, 0)
    100000 loops, best of 3: 17.3 us per loop


From zhaoruzhang at gmail.com  Mon Apr 25 10:32:34 2011
From: zhaoruzhang at gmail.com (Zhaoru Zhang)
Date: Mon, 25 Apr 2011 09:32:34 -0500
Subject: [Numpy-discussion] using numpy to save arrays in a loop
Message-ID: <3D80CE80-71AD-4F7C-BA15-AA4B0F0517CC@gmail.com>

Hi guys,

I have a time loop and at each time step I get a 1-d array and want to save it in the data file 'data.dat '(or whatever suffix), so finally i got . I made this by using the python 'write' function, but I' thinking about if there is any numpy function that can realize this? Thanks.

Zhaoru



From isaiah.norton at gmail.com  Mon Apr 25 10:40:12 2011
From: isaiah.norton at gmail.com (Isaiah Norton)
Date: Mon, 25 Apr 2011 10:40:12 -0400
Subject: [Numpy-discussion] using numpy to save arrays in a loop
In-Reply-To: <3D80CE80-71AD-4F7C-BA15-AA4B0F0517CC@gmail.com>
References: <3D80CE80-71AD-4F7C-BA15-AA4B0F0517CC@gmail.com>
Message-ID: <BANLkTimkH7By13qDLpM4wkxeFkKpyzh=KQ@mail.gmail.com>

There are various numpy and scipy input/output functions described here:

http://www.scipy.org/Cookbook/InputOutput
http://docs.scipy.org/doc/scipy/reference/io.html

On Mon, Apr 25, 2011 at 10:32 AM, Zhaoru Zhang <zhaoruzhang at gmail.com>wrote:

> Hi guys,
>
> I have a time loop and at each time step I get a 1-d array and want to save
> it in the data file 'data.dat '(or whatever suffix), so finally i got . I
> made this by using the python 'write' function, but I' thinking about if
> there is any numpy function that can realize this? Thanks.
>
> Zhaoru
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110425/19ac3761/attachment.html>

From gael.varoquaux at normalesup.org  Mon Apr 25 12:57:00 2011
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 25 Apr 2011 18:57:00 +0200
Subject: [Numpy-discussion] random number generator, entropy and pickling
Message-ID: <20110425165700.GB1600@phare.normalesup.org>

Hi there,

We are courrently having a discussion on the scikits learn mailing list
about which patterns to adopt for random number generation. One thing
that is absolutely clear is that making the stream of random numbers
reproducible is critical. We have several objects that can serve as random
variate generators. So far, we instanciate these objects with a optional
seed or PRNG argument, as in:

    def __init__(self, prng=None):
	if prng is None:
	    prng = np.random
	self.prng = prng

The problem with this pattern is that np.random doesn't pickle, and
therefore the objects do not pickle by default. A bit of pickling magic
would solve this, but we'd rather avoid it.

We thought that we could simply have a PRNG per object, as in:

    def __init__(self, prng=None):
	if prng is None:
	    prng = np.random.RandomState()
	self.prng = prng

I don't like this option, because it means that with a given pieve of
code, setting the seed of numpy's PRNG isn't enough to make it
reproducible.

I couldn't retrieve a handle on a picklable instance for the global PRNG.

The only option I can see would be to use the global numpy PRNG to seed
an instance specific RandomState, as in:

    def __init__(self, prng=None):
	if prng is None:
	    prng = np.random.RandomState(np.random.random())
	self.prng = prng

That way seeding the global PRNG really does control the full random
number generation. I am wondering if it would have an adverse consequence
on the entropy of the stream of random numbers. Does anybody have
suggestions? Advices?

Cheers,

Gael


From beezer at ups.edu  Mon Apr 25 13:16:13 2011
From: beezer at ups.edu (Rob Beezer)
Date: Mon, 25 Apr 2011 10:16:13 -0700
Subject: [Numpy-discussion] Zero row in SVD's unitary matrix on some Mac's
Message-ID: <4DB5AC5D.7020000@ups.edu>

I am working to make many of NumPy's matrix decomposition routines available in 
Sage.  As part of testing a new routine, we have found some odd behavior with 
the singular value decomposition.

On certain Mac's, the numpy built in Sage will return the second of the unitary 
matrices with a row of all zeros, so of course, it can't be unitary.  For the 
configurations in question, this erroneous output happens only for certain sized 
matrices and for those sizes it always occurs.  The smallest sizes are 3 x 4, 4 
x 5, 5 x 6, 5 x 7, 6 x 7, 6 x 8, 6 x 9, 7 x 8, 7 x 9, 8 x 9.  The fault is not 
in Sage code per se, as it can be reproduced by running Sage's python and using 
numpy directly.  It could be possible Sage is not building numpy correctly, we 
have not tested a standalone version of numpy since this problem seems to be 
limited to very few configurations.

The initial report, and a confirmation, are both on Macs where Sage is built 
using gcc 4.0.1 and gcc 4.2.1.  The test that uncovered this situation was 
introduced two alpha releases back, and has not failed for testers on Linux or 
newer Macs.  The svd routine itself has been in Sage for about three years 
without exhibiting any problems, but maybe the cases above were not tested.  I 
do not own a Mac, so testing out scenarios involves sending suggestions to the 
two folks who have reported failures.

Many more details and complete transcripts are at:
http://trac.sagemath.org/sage_trac/ticket/11248

Any thoughts or advice to help us understand this would be greatly appreciated. 
  Thanks in advance.

Rob
-- 
Robert A. Beezer
Professor
Department of Mathematics and Computer Science
University of Puget Sound
1500 N Warner
Tacoma, WA  98416-1043

beezer at ups.edu
http://buzzard.ups.edu

Voice:  253.879.3564
Fax: 253.879.3522


From alex.liberzon at gmail.com  Mon Apr 25 13:15:32 2011
From: alex.liberzon at gmail.com (Alex Liberzon)
Date: Mon, 25 Apr 2011 20:15:32 +0300
Subject: [Numpy-discussion] [ANN]: OpenPiv. A python package for PIV
	image analysis
Message-ID: <4CAE9E47-9D6D-4187-86CD-A6C2A6224B5E@gmail.com>

Dear Fabrice, 

Let's take this discussion off-line, contact us directly through http://github.com/alexlib or to e-mail. It's a coincidence but I just finished wrapping the same library using SWIG :). Such a pity I didn't know about your work. I also believe that we talk about the same thing as I did it also to work with PIVMAT and was in touch with its author. We would be happy to use your stuff and include it as a part of OpenPIV. 

I'm not sure how much can we go into the 3D stuff (although my second big thing is managing the 3D-PTV http://ptv.origo.ethz.ch) and we do have tools for 3D analysis. I'd be very happy to release everything we have in our backyards, Matlab, C, Python, to bring it all together to a serious skikit or similar project based on Numpy/Scipy and Cython for the good of all fluid mechanics experimentalists. Such packages exist actually for the CFD data, I am not aware of the open source and free ones. 

Would you join our team? 

Best regards,
Alex Liberzon

Turbulence Structure Laboratory [http://www.eng.tau.ac.il/turbulencelab]
School of Mechanical Engineering
Tel Aviv University


Le dimanche 24 avril 2011 ? 16:53 +0200, Davide a ?crit :
> Hi all,
> I am pleased to announce the re-birth of OpenPiv, a python package for 
> the analysis of PIV images. PIV is a well established optical 
> measurement technique for fluid flows. The package has the goal of 
> provide an implementation of state-of-the-art processing algorithms, and 
> compete with commercial closed-source software packages.
> OpenPiv is in alpha state and several thing has to be done. However, the 
> code is already usable and a pure python implementation of the standard 
> cross-correlation algorithm is already available.
> OpenPiv needs developers and contributors, helping in with code and 
> documentation. If you can provide some, and if you are interested in 
> becoming a core developer, please drop us an email at openpiv-develop at 
> lists dot sourceforge dot net. A draft website can be found at 
> www.openpiv.net/python.
> 
> Thanks for your attention,
> 
> Davide Lasagna

Hi Davide
Investigations in fluid mechanics lead me to build a few months a
wrapper on the I/O functions from Davis (LaVision), a commercial
solution used to handle the complete procedure from camera calibration
to postprocessing velocity fields. 
I had to postprocess previous measurements stored in Davis format to a
deeper analysis than their software allows, so that I needed a tool able
to read data from their (almost documented) format.
The solution I found googling were not satisfying at the moment. For
example, PIVMAT [2] only worked for 2D fields and we were dealing with
3D flows and Stereoscopic PIV. After speaking with it author, it appears
that it would require too much work to support the third component in a
decent way.

The ctypes-based wrappers are available here
https://launchpad.net/libim7 
https://github.com/FabricioS/libim7 
It is a quite dumb piece of code, but it allows to extract images or
velocity fields from binary files (extensions im7, imx and vc7),
avoiding some of the traps in the extraction of the data.

I would be happy if you consider useful this tool within OpenPIV. It
would also be possible to change the ctype dependance to cython...

Best regards


[1] http://www.lavision.de/en/products/davis.php
[2] http://www.fast.u-psud.fr/pivmat/
-- 
Fabrice Silva








From robert.kern at gmail.com  Mon Apr 25 13:49:50 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 25 Apr 2011 12:49:50 -0500
Subject: [Numpy-discussion] [Scikit-learn-general] random number
	generator, entropy and pickling
In-Reply-To: <20110425165700.GB1600@phare.normalesup.org>
References: <20110425165700.GB1600@phare.normalesup.org>
Message-ID: <BANLkTinoHQiM3ZR=HM_HT0xdc94ADev5+g@mail.gmail.com>

On Mon, Apr 25, 2011 at 11:57, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> Hi there,
>
> We are courrently having a discussion on the scikits learn mailing list
> about which patterns to adopt for random number generation. One thing
> that is absolutely clear is that making the stream of random numbers
> reproducible is critical. We have several objects that can serve as random
> variate generators. So far, we instanciate these objects with a optional
> seed or PRNG argument, as in:
>
> ? ?def __init__(self, prng=None):
> ? ? ? ?if prng is None:
> ? ? ? ? ? ?prng = np.random
> ? ? ? ?self.prng = prng
>
> The problem with this pattern is that np.random doesn't pickle, and
> therefore the objects do not pickle by default. A bit of pickling magic
> would solve this, but we'd rather avoid it.
>
> We thought that we could simply have a PRNG per object, as in:
>
> ? ?def __init__(self, prng=None):
> ? ? ? ?if prng is None:
> ? ? ? ? ? ?prng = np.random.RandomState()
> ? ? ? ?self.prng = prng
>
> I don't like this option, because it means that with a given pieve of
> code, setting the seed of numpy's PRNG isn't enough to make it
> reproducible.
>
> I couldn't retrieve a handle on a picklable instance for the global PRNG.

It's accessible as np.random.mtrand._rand, though we have kept it
"private" for a reason. The Option (a) from the original thread on
scikits-learn-general, "use your own default global RandomState
instance in scikits.learn", would be preferable.

> The only option I can see would be to use the global numpy PRNG to seed
> an instance specific RandomState, as in:
>
> ? ?def __init__(self, prng=None):
> ? ? ? ?if prng is None:
> ? ? ? ? ? ?prng = np.random.RandomState(np.random.random())
> ? ? ? ?self.prng = prng
>
> That way seeding the global PRNG really does control the full random
> number generation. I am wondering if it would have an adverse consequence
> on the entropy of the stream of random numbers. Does anybody have
> suggestions? Advices?

Use a single, common default PRNG, either np.random.mtrand._rand or
your own. Don't use multiple seeds from a PRNG. Use a utility function
to avoid repeating yourself, even if it's just a one-liner. In this
case, it's important that everyone do exactly the same thing for
consistency, both inside scikits.learn and in code that uses or
extends scikits.learn. The best way to ensure that is to provide a
utility function as the One, Obvious Way To Do It. Note that if you do
hide the details behind a utility function, I would remove my
objection to using np.random.mtrand._rand. ;-)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
? -- Umberto Eco


From pav at iki.fi  Mon Apr 25 13:57:33 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 25 Apr 2011 17:57:33 +0000 (UTC)
Subject: [Numpy-discussion] Zero row in SVD's unitary matrix on some
	Mac's
References: <4DB5AC5D.7020000@ups.edu>
Message-ID: <ip4cmd$7or$1@dough.gmane.org>

On Mon, 25 Apr 2011 10:16:13 -0700, Rob Beezer wrote:
[clip]
> Many more details and complete transcripts are at:
> http://trac.sagemath.org/sage_trac/ticket/11248
> 
> Any thoughts or advice to help us understand this would be greatly
> appreciated.

The Numpy routine is a very thin wrapper of LAPACK's ZGESDD, and probably 
cannot have any bugs of this kind, so the problem is most likely with the 
LAPACK and BLAS libraries you use. You will probably be able to reproduce 
the problem also with an equivalent Fortran/C snippet calling LAPACK 
directly.

Problems like this in BLAS/LAPACK are somewhat difficult to track. You 
could try switching to another BLAS library (or, if you use ATLAS, 
compile it differently) and checking if the problem disappears.

-- 
Pauli Virtanen



From tjhnson at gmail.com  Mon Apr 25 14:05:05 2011
From: tjhnson at gmail.com (T J)
Date: Mon, 25 Apr 2011 11:05:05 -0700
Subject: [Numpy-discussion] random number generator, entropy and pickling
In-Reply-To: <20110425165700.GB1600@phare.normalesup.org>
References: <20110425165700.GB1600@phare.normalesup.org>
Message-ID: <BANLkTik9Z+B_=-Zo=Phr8HyVuj=_8MvghQ@mail.gmail.com>

On Mon, Apr 25, 2011 at 9:57 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
>
> We thought that we could simply have a PRNG per object, as in:
>
> ? ?def __init__(self, prng=None):
> ? ? ? ?if prng is None:
> ? ? ? ? ? ?prng = np.random.RandomState()
> ? ? ? ?self.prng = prng
>
> I don't like this option, because it means that with a given pieve of
> code, setting the seed of numpy's PRNG isn't enough to make it
> reproducible.
>
> I couldn't retrieve a handle on a picklable instance for the global PRNG.
>
> The only option I can see would be to use the global numpy PRNG to seed
> an instance specific RandomState, as in:
>
> ? ?def __init__(self, prng=None):
> ? ? ? ?if prng is None:
> ? ? ? ? ? ?prng = np.random.RandomState(np.random.random())
> ? ? ? ?self.prng = prng
>
> That way seeding the global PRNG really does control the full random
> number generation. I am wondering if it would have an adverse consequence
> on the entropy of the stream of random numbers. Does anybody have
> suggestions? Advices?
>

If code A relies on code B (eg, some numpy function) and code B
changes, then the stream of random numbers will no longer be the same.
 The point here is that the user wrote code A but depended on code B,
and even though code A was unchanged, their random numbers were not
the same.

The situation is improved if scikits.learn used its own global
RandomState instance.  Then code A will at least give the same stream
of random numbers for a fixed version of scikits.learn.  It should be
made very clear though that the data stream cannot be expected to be
the same across versions.

As to each object having its own RandomState instance, I definitely
see that it makes restoring the overall state of a piece of code
harder, but perhaps utility functions could make this easier.  I can
imagine that users might want to arbitrarily set the seed for a
particular object in the midsts of a larger piece of code.  Perhaps
the user is testing known failure cases of a algorithm A interacting
with algorithm B.  If the user wants to loop over known seeds which
cause algorithm A to fail but needs algorithm B to keep its seed
fixed, then it seems like having a global seed makes this more
difficult.  In that sense, it might be desirable to have "independent"
prngs. On the other hand, maybe that is an uncommon use case that
could be handled through manually setting the seed.

Post back on what you guys decide.


From gael.varoquaux at normalesup.org  Mon Apr 25 14:15:24 2011
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 25 Apr 2011 20:15:24 +0200
Subject: [Numpy-discussion] random number generator, entropy and pickling
In-Reply-To: <BANLkTik9Z+B_=-Zo=Phr8HyVuj=_8MvghQ@mail.gmail.com>
References: <20110425165700.GB1600@phare.normalesup.org>
	<BANLkTik9Z+B_=-Zo=Phr8HyVuj=_8MvghQ@mail.gmail.com>
Message-ID: <20110425181524.GD1600@phare.normalesup.org>

On Mon, Apr 25, 2011 at 11:05:05AM -0700, T J wrote:
> If code A relies on code B (eg, some numpy function) and code B
> changes, then the stream of random numbers will no longer be the same.
>  The point here is that the user wrote code A but depended on code B,
> and even though code A was unchanged, their random numbers were not
> the same.

Yes, that's exactly why we want the different objects to able to recieve
their own PRNG.

> The situation is improved if scikits.learn used its own global
> RandomState instance.  Then code A will at least give the same stream
> of random numbers for a fixed version of scikits.learn.  It should be
> made very clear though that the data stream cannot be expected to be
> the same across versions.

The use case that we are trying to catter for, with the global PRNG, is
for mister Joe average, who is used to setting the numpy PRNG to control
what is going on. In my experience, the less you need to teach Mr Joe A.,
the better (I am not dumbing down Joe A., just acknowledging the fact
that he probably has many other things to worry about).

> As to each object having its own RandomState instance, I definitely
> see that it makes restoring the overall state of a piece of code
> harder, but perhaps utility functions could make this easier. 

That's what we are leaning toward: a utility function, that by default
returns the numpy PRNG object, but enables the use of specific PNRGs or
seeds. In other words, we are thinking of following Robert's suggestion
(option 'a' in the original mail, but enriched with Robert's input on
mtrand.rand). We'll probably wait a bit for feedback before making a
decision.

Thanks for all your input,

G


From robert.kern at gmail.com  Mon Apr 25 14:23:12 2011
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 25 Apr 2011 13:23:12 -0500
Subject: [Numpy-discussion] [Scikit-learn-general] random number
 generator, entropy and pickling
In-Reply-To: <20110425181524.GD1600@phare.normalesup.org>
References: <20110425165700.GB1600@phare.normalesup.org>
	<BANLkTik9Z+B_=-Zo=Phr8HyVuj=_8MvghQ@mail.gmail.com>
	<20110425181524.GD1600@phare.normalesup.org>
Message-ID: <BANLkTin16RjH-igeEu=B2pmjOK8YNGuTww@mail.gmail.com>

On Mon, Apr 25, 2011 at 13:15, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Mon, Apr 25, 2011 at 11:05:05AM -0700, T J wrote:
>> If code A relies on code B (eg, some numpy function) and code B
>> changes, then the stream of random numbers will no longer be the same.
>> ?The point here is that the user wrote code A but depended on code B,
>> and even though code A was unchanged, their random numbers were not
>> the same.
>
> Yes, that's exactly why we want the different objects to able to recieve
> their own PRNG.

But seriously, they are running A+B, the combination of A and B. If
A+B changes to A+B', then the results may be different. That's to be
expected.

>> The situation is improved if scikits.learn used its own global
>> RandomState instance. ?Then code A will at least give the same stream
>> of random numbers for a fixed version of scikits.learn. ?It should be
>> made very clear though that the data stream cannot be expected to be
>> the same across versions.
>
> The use case that we are trying to catter for, with the global PRNG, is
> for mister Joe average, who is used to setting the numpy PRNG to control
> what is going on.

Honestly, they really shouldn't be, except as a workaround to
poorly-written functions that don't let you pass in your own PRNG.
Someone snuck in the module-level alias to the global PRNG's seed()
method when I wasn't paying attention. :-)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
? -- Umberto Eco


From gael.varoquaux at normalesup.org  Mon Apr 25 14:28:13 2011
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 25 Apr 2011 20:28:13 +0200
Subject: [Numpy-discussion] [Scikit-learn-general] random number
 generator, entropy and pickling
In-Reply-To: <BANLkTin16RjH-igeEu=B2pmjOK8YNGuTww@mail.gmail.com>
References: <20110425165700.GB1600@phare.normalesup.org>
	<BANLkTik9Z+B_=-Zo=Phr8HyVuj=_8MvghQ@mail.gmail.com>
	<20110425181524.GD1600@phare.normalesup.org>
	<BANLkTin16RjH-igeEu=B2pmjOK8YNGuTww@mail.gmail.com>
Message-ID: <20110425182813.GE1600@phare.normalesup.org>

On Mon, Apr 25, 2011 at 01:23:12PM -0500, Robert Kern wrote:
> > Yes, that's exactly why we want the different objects to able to recieve
> > their own PRNG.

> But seriously, they are running A+B, the combination of A and B. If
> A+B changes to A+B', then the results may be different. That's to be
> expected.

Fair enough. Let's say the example was not ideal, but we still want to be
able to control the random number generation of an algorithm
independently of what's going on elsewhere. That's why we are happy to be
able to have a PNRG dedicated to that processing pipeline. I think that
everybody agrees with that.

> > The use case that we are trying to catter for, with the global PRNG, is
> > for mister Joe average, who is used to setting the numpy PRNG to control
> > what is going on.

> Honestly, they really shouldn't be, except as a workaround to
> poorly-written functions that don't let you pass in your own PRNG.

Right, but many users transit from Matlab, where they learn this pattern.
I am not interested in fighting against user's behavior unless I have a
very good reason. What they do in their code is their problem.

G


From paul.anton.letnes at gmail.com  Mon Apr 25 16:13:33 2011
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Mon, 25 Apr 2011 22:13:33 +0200
Subject: [Numpy-discussion] Zero row in SVD's unitary matrix on some
	Mac's
In-Reply-To: <ip4cmd$7or$1@dough.gmane.org>
References: <4DB5AC5D.7020000@ups.edu> <ip4cmd$7or$1@dough.gmane.org>
Message-ID: <25B927E6-1B29-464B-A7EB-07A7BA60D485@gmail.com>

On 25. apr. 2011, at 19.57, Pauli Virtanen wrote:

> On Mon, 25 Apr 2011 10:16:13 -0700, Rob Beezer wrote:
> [clip]
>> Many more details and complete transcripts are at:
>> http://trac.sagemath.org/sage_trac/ticket/11248
>> 
>> Any thoughts or advice to help us understand this would be greatly
>> appreciated.
> 
> The Numpy routine is a very thin wrapper of LAPACK's ZGESDD, and probably 
> cannot have any bugs of this kind, so the problem is most likely with the 
> LAPACK and BLAS libraries you use. You will probably be able to reproduce 
> the problem also with an equivalent Fortran/C snippet calling LAPACK 
> directly.
> 
> Problems like this in BLAS/LAPACK are somewhat difficult to track. You 
> could try switching to another BLAS library (or, if you use ATLAS, 
> compile it differently) and checking if the problem disappears.
> 
> -- 
> Pauli Virtanen

I cannot claim anything concrete, but I did notice something seemingly really odd about the blas+lapack that ships with macosx. In our light scattering simulation code we use energy conservation as a test of consistency. When running small test runs on my macbook pro, the energy would be significantly less well conserved compared to when the exact same simulation was run on a linux box. No harm done, we don't care about the exact results on the laptop, but it did seem odd. We do not rely on SVD, however, we only use LU+backsubstitution (cgesv).

I have a vague feeling that there is something odd about lapack+blas on macosx, but I do not have any hard evidence.

Paul

From beezer at ups.edu  Mon Apr 25 16:34:42 2011
From: beezer at ups.edu (Rob Beezer)
Date: Mon, 25 Apr 2011 13:34:42 -0700
Subject: [Numpy-discussion] Zero row in SVD's unitary matrix on
	some	Mac's
In-Reply-To: <ip4cmd$7or$1@dough.gmane.org>
References: <4DB5AC5D.7020000@ups.edu> <ip4cmd$7or$1@dough.gmane.org>
Message-ID: <4DB5DAE2.5060603@ups.edu>

Thanks for the reply, Pauli.  I suspected this might be the case, but was hoping 
maybe this was something that had been seen before.  I've included your 
suggestions on the bug report for Sage.

Rob

On 04/25/2011 10:57 AM, Pauli Virtanen wrote:
> The Numpy routine is a very thin wrapper of LAPACK's ZGESDD, and probably
> cannot have any bugs of this kind, so the problem is most likely with the
> LAPACK and BLAS libraries you use. You will probably be able to reproduce
> the problem also with an equivalent Fortran/C snippet calling LAPACK
> directly.
>
> Problems like this in BLAS/LAPACK are somewhat difficult to track. You
> could try switching to another BLAS library (or, if you use ATLAS,
> compile it differently) and checking if the problem disappears.
>


From jason-sage at creativetrax.com  Tue Apr 26 11:33:07 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 26 Apr 2011 10:33:07 -0500
Subject: [Numpy-discussion] Zero row in SVD's unitary matrix on
	some	Mac's
In-Reply-To: <ip4cmd$7or$1@dough.gmane.org>
References: <4DB5AC5D.7020000@ups.edu> <ip4cmd$7or$1@dough.gmane.org>
Message-ID: <4DB6E5B3.1000606@creativetrax.com>

On 4/25/11 12:57 PM, Pauli Virtanen wrote:
> On Mon, 25 Apr 2011 10:16:13 -0700, Rob Beezer wrote:
> [clip]
>> Many more details and complete transcripts are at:
>> http://trac.sagemath.org/sage_trac/ticket/11248
>>
>> Any thoughts or advice to help us understand this would be greatly
>> appreciated.
>
> The Numpy routine is a very thin wrapper of LAPACK's ZGESDD, and probably
> cannot have any bugs of this kind, so the problem is most likely with the
> LAPACK and BLAS libraries you use. You will probably be able to reproduce
> the problem also with an equivalent Fortran/C snippet calling LAPACK
> directly.
>
> Problems like this in BLAS/LAPACK are somewhat difficult to track. You
> could try switching to another BLAS library (or, if you use ATLAS,
> compile it differently) and checking if the problem disappears.


I was just looking up the documentation for ZGESDD and noticed that the 
value we have for rwork in the numpy call [1] does not match the Lapack 
docs.  This was changed in Lapack 3.2.2 [2].  I've submitted a pull request:

https://github.com/numpy/numpy/pull/78

but I have not tested the change.

I doubt this fix will fix the problem on this thread, but it makes sense 
to make the change anyway.

Thanks,

Jason

[1] https://github.com/numpy/numpy/blob/master/numpy/linalg/linalg.py#L1300

[2] http://icl.cs.utk.edu/lapack-forum/viewtopic.php?f=2&t=1779, or 
http://www.netlib.org/lapack/lapack-3.2.2.html (bug 0046)



From jason-sage at creativetrax.com  Tue Apr 26 12:07:56 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 26 Apr 2011 11:07:56 -0500
Subject: [Numpy-discussion] Zero row in SVD's unitary matrix on
	some	Mac's
In-Reply-To: <ip4cmd$7or$1@dough.gmane.org>
References: <4DB5AC5D.7020000@ups.edu> <ip4cmd$7or$1@dough.gmane.org>
Message-ID: <4DB6EDDC.2050508@creativetrax.com>

On 4/25/11 12:57 PM, Pauli Virtanen wrote:
> The Numpy routine is a very thin wrapper of LAPACK's ZGESDD, and probably
> cannot have any bugs of this kind,

As noted in my other message, I've been digging through the ZGESDD docs 
to understand it better.  Here is the doc string for what becomes the 
V^T matrix:

*  VT      (output) COMPLEX*16 array, dimension (LDVT,N)
*          If JOBZ = 'A' or JOBZ = 'O' and M >= N, VT contains the
*          N-by-N unitary matrix V**H;
*          if JOBZ = 'S', VT contains the first min(M,N) rows of
*          V**H (the right singular vectors, stored rowwise);
*          if JOBZ = 'O' and M < N, or JOBZ = 'N', VT is not referenced.

Notice that VT is supposed to contain the N-by-N V**H if M>=N (i.e., 
more rows than columns).  In our problem cases, we have M<N.  I looked 
at the numpy linalg code and it doesn't seem to check to see if M<N 
before returning the V matrix.  Is this a problem?

When I run an example C program that links to the OSX Accelerate 
framework, I get a V matrix that has a zero row at the bottom.  I got 
these results by:

1. Download the example found at 
http://software.intel.com/sites/products/documentation/hpc/mkl/lapack/mkl_lapack_examples/zgesdd_ex.c.htm

2. Change the zgesdd call to have first argument "A" (so it mimics the 
default numpy call)

3. Change

print_matrix( "Right singular vectors (stored rowwise)", m, n, vt, ldvt );

to

print_matrix( "Right singular vectors (stored rowwise)", n, n, vt, ldvt );

(this is to print out the full n-by-n matrix, instead of just the first 
m rows)

4. Compile the program with "gcc zgesdd_ex.c -framework Accelerate"

5. Run the program: "./a.out"

And indeed, I get a 0 row as the last row of the V**H matrix

However, when I do the SVD of this same matrix using numpy (which I 
*think* uses the Accelerate framework the way we compile it in Sage), I 
get a full V matrix:

In [13]: I=1j
In [14]: a=numpy.asarray([( -5.40+ I*  7.40), (  6.00+ I*  6.38), ( 
9.91+ I*  0.16), ( -5.28+ I* -4.16),   (  1.09+ I*  1.55), (  2.60+ I* 
0.07), (  3.98+ I* -5.26), (  2.03+ I*  1.11),   (  9.88+ I*  1.91), ( 
4.92+ I*  6.31), ( -2.11+ I*  7.39), ( -9.81+ I* 
-8.98)],dtype=complex).reshape(3,4)

In [15]: numpy.linalg.svd(a)Out[15]:
(array([[ 0.54742764+0.j        ,  0.76302168+0.j        , 
-0.34368721+0.j        ],
        [-0.03507684-0.15148438j,  0.27097680-0.22637514j,
          0.54572628-0.74386208j],
        [ 0.81299016+0.12325614j, -0.52311095-0.13956616j,
          0.13357577-0.1135282j ]]),
  array([ 21.75519279,  16.59545017,   3.97327576]),
  array([[ 0.23160531+0.20669796j,  0.36590896+0.3864613j ,
          0.24259328+0.32833854j, -0.56133932-0.37233547j],
        [-0.57911906+0.40329699j,  0.10921398+0.17242422j,
          0.59673801-0.27492812j,  0.15998810+0.05510835j],
        [ 0.60420072+0.12337134j, -0.18988750+0.29722068j,
          0.39210635+0.19697635j,  0.45451433+0.31015037j],
        [-0.08335334+0.13557583j,  0.69622499-0.25685378j,
         -0.15350399+0.43075518j,  0.46295826+0.02290113j]]))

That seems odd.  I must be missing something.  It also seems odd that 
numpy returns V as a full 4x4 matrix even when M<N.

Thanks,

Jason


From jason-sage at creativetrax.com  Tue Apr 26 12:12:06 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 26 Apr 2011 11:12:06 -0500
Subject: [Numpy-discussion] Zero row in SVD's unitary matrix
	on	some	Mac's
In-Reply-To: <4DB6EDDC.2050508@creativetrax.com>
References: <4DB5AC5D.7020000@ups.edu> <ip4cmd$7or$1@dough.gmane.org>
	<4DB6EDDC.2050508@creativetrax.com>
Message-ID: <4DB6EED6.4030207@creativetrax.com>

On 4/26/11 11:07 AM, Jason Grout wrote:
> And indeed, I get a 0 row as the last row of the V**H matrix

I just double-checked things one last time and saw that I actually 
hadn't changed the first argument of zgesdd to "A" in the program that I 
actually ran.  So with this change, I get a nonzero last row of the V**H 
matrix from the C call to zgesdd.  So everything is consistent between 
the C call to zgesdd and the numpy svd call.

So now my remaining question is: if the Lapack docs only say that V**H 
is the full n-by-n matrix if M>=N, why is numpy returning it even if M<N?

Thanks,

Jason



From jason-sage at creativetrax.com  Tue Apr 26 12:22:46 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 26 Apr 2011 11:22:46 -0500
Subject: [Numpy-discussion] Zero row in SVD's unitary
	matrix	on	some	Mac's
In-Reply-To: <4DB6EED6.4030207@creativetrax.com>
References: <4DB5AC5D.7020000@ups.edu>
	<ip4cmd$7or$1@dough.gmane.org>	<4DB6EDDC.2050508@creativetrax.com>
	<4DB6EED6.4030207@creativetrax.com>
Message-ID: <4DB6F156.5070209@creativetrax.com>

On 4/26/11 11:12 AM, Jason Grout wrote:
> On 4/26/11 11:07 AM, Jason Grout wrote:
>> And indeed, I get a 0 row as the last row of the V**H matrix
>
> I just double-checked things one last time and saw that I actually
> hadn't changed the first argument of zgesdd to "A" in the program that I
> actually ran.  So with this change, I get a nonzero last row of the V**H
> matrix from the C call to zgesdd.  So everything is consistent between
> the C call to zgesdd and the numpy svd call.
>
> So now my remaining question is: if the Lapack docs only say that V**H
> is the full n-by-n matrix if M>=N, why is numpy returning it even if M<N?

One more post talking to myself...

I notice that the zgesvd routine docs guarantee that the V returned is 
unitary, regardless of the size of A.  So this might be another argument 
for calling zgesvd instead of zgesdd.

Thanks,

Jason



From jason-sage at creativetrax.com  Tue Apr 26 12:36:19 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 26 Apr 2011 11:36:19 -0500
Subject: [Numpy-discussion] Zero row in SVD's
	unitary	matrix	on	some	Mac's
In-Reply-To: <4DB6F156.5070209@creativetrax.com>
References: <4DB5AC5D.7020000@ups.edu>	<ip4cmd$7or$1@dough.gmane.org>	<4DB6EDDC.2050508@creativetrax.com>	<4DB6EED6.4030207@creativetrax.com>
	<4DB6F156.5070209@creativetrax.com>
Message-ID: <4DB6F483.30403@creativetrax.com>

On 4/26/11 11:22 AM, Jason Grout wrote:
> On 4/26/11 11:12 AM, Jason Grout wrote:
>> On 4/26/11 11:07 AM, Jason Grout wrote:
>>> And indeed, I get a 0 row as the last row of the V**H matrix
>>
>> I just double-checked things one last time and saw that I actually
>> hadn't changed the first argument of zgesdd to "A" in the program that I
>> actually ran.  So with this change, I get a nonzero last row of the V**H
>> matrix from the C call to zgesdd.  So everything is consistent between
>> the C call to zgesdd and the numpy svd call.
>>
>> So now my remaining question is: if the Lapack docs only say that V**H
>> is the full n-by-n matrix if M>=N, why is numpy returning it even if M<N?
>
> One more post talking to myself...
>
> I notice that the zgesvd routine docs guarantee that the V returned is
> unitary, regardless of the size of A.  So this might be another argument
> for calling zgesvd instead of zgesdd.


Okay, just one more data point.  Our people that are seeing the problem 
with numpy returning a non-unitary V also see a non-unitary V being 
returned by the test C call to zgesdd.  In other words, it really 
appears that zgesdd follows the Lapack docs, and if rows<columns, the 
returned V is not necessarily unitary, but may contain a zero row.  This 
makes numpy's assumptions in using zgesdd false.

You can see this report at 
http://trac.sagemath.org/sage_trac/ticket/11248#comment:25

Thanks,

Jason



From pav at iki.fi  Tue Apr 26 12:49:11 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 26 Apr 2011 16:49:11 +0000 (UTC)
Subject: [Numpy-discussion] Zero row in
	SVD's	unitary	matrix	on	some	Mac's
References: <4DB5AC5D.7020000@ups.edu> <ip4cmd$7or$1@dough.gmane.org>
	<4DB6EDDC.2050508@creativetrax.com> <4DB6EED6.4030207@creativetrax.com>
	<4DB6F156.5070209@creativetrax.com> <4DB6F483.30403@creativetrax.com>
Message-ID: <ip6t27$jpi$1@dough.gmane.org>

Tue, 26 Apr 2011 11:36:19 -0500, Jason Grout wrote:
[clip]
> Okay, just one more data point.  Our people that are seeing the problem
> with numpy returning a non-unitary V also see a non-unitary V being
> returned by the test C call to zgesdd.  In other words, it really
> appears that zgesdd follows the Lapack docs, and if rows<columns, the
> returned V is not necessarily unitary, but may contain a zero row.  This
> makes numpy's assumptions in using zgesdd false.
> 
> You can see this report at
> http://trac.sagemath.org/sage_trac/ticket/11248#comment:25

What LAPACK promises is not very clear. Earlier on the the man page:

       JOBZ    (input) CHARACTER*1
               Specifies options for computing all or part of the matrix U:
               = 'A':  all M columns of U and all N rows of V**H are  returned
               in the arrays U and VT; = 'S':  the first min(M,N) columns of U
               and the first min(M,N) rows of V**H are returned in the  arrays
               U and VT; = 'O':  If M >= N, the first N columns of U are over?
               written in the array A and all rows of V**H are returned in the
               array VT; otherwise, all columns of U are returned in the array
               U and the first M rows of V**H are overwritten in the array  A;
               = 'N':  no columns of U or rows of V**H are computed.

Looks to me the statement should be parsed as:

	return_n_rows = (jobz == 'A') or (jobz == 'O' and m >= n)

So the current usage should be OK. (At least, as long as jobz == 'A',
in the other cases, I don't think the wrapper does the right thing.)

But apparently either there's a bug, or the LAPACK man page needs to
be understood as you say.

	Pauli



From ralf.gommers at googlemail.com  Tue Apr 26 13:19:08 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Tue, 26 Apr 2011 19:19:08 +0200
Subject: [Numpy-discussion] Zero row in SVD's unitary matrix on some
	Mac's
In-Reply-To: <4DB6F483.30403@creativetrax.com>
References: <4DB5AC5D.7020000@ups.edu> <ip4cmd$7or$1@dough.gmane.org>
	<4DB6EDDC.2050508@creativetrax.com>
	<4DB6EED6.4030207@creativetrax.com>
	<4DB6F156.5070209@creativetrax.com>
	<4DB6F483.30403@creativetrax.com>
Message-ID: <BANLkTimvMSAsR4+QacjNHUvaNLn1V6XFPA@mail.gmail.com>

On Tue, Apr 26, 2011 at 6:36 PM, Jason Grout
<jason-sage at creativetrax.com> wrote:
> On 4/26/11 11:22 AM, Jason Grout wrote:
>> On 4/26/11 11:12 AM, Jason Grout wrote:
>>> On 4/26/11 11:07 AM, Jason Grout wrote:
>>>> And indeed, I get a 0 row as the last row of the V**H matrix
>>>
>>> I just double-checked things one last time and saw that I actually
>>> hadn't changed the first argument of zgesdd to "A" in the program that I
>>> actually ran. ?So with this change, I get a nonzero last row of the V**H
>>> matrix from the C call to zgesdd. ?So everything is consistent between
>>> the C call to zgesdd and the numpy svd call.
>>>
>>> So now my remaining question is: if the Lapack docs only say that V**H
>>> is the full n-by-n matrix if M>=N, why is numpy returning it even if M<N?
>>
>> One more post talking to myself...
>>
>> I notice that the zgesvd routine docs guarantee that the V returned is
>> unitary, regardless of the size of A. ?So this might be another argument
>> for calling zgesvd instead of zgesdd.
>
>
> Okay, just one more data point. ?Our people that are seeing the problem
> with numpy returning a non-unitary V also see a non-unitary V being
> returned by the test C call to zgesdd. ?In other words, it really
> appears that zgesdd follows the Lapack docs, and if rows<columns, the
> returned V is not necessarily unitary, but may contain a zero row. ?This
> makes numpy's assumptions in using zgesdd false.
>
> You can see this report at
> http://trac.sagemath.org/sage_trac/ticket/11248#comment:25

If it is an Accelerate bug it got fixed on OS X 10.6. Below is the
output with both current master and 1.5.1. It may be worth checking
this on 10.5 with the binary Numpy installer from Sourceforge to make
sure it's not the Sage build process causing this somehow.

Ralf


>>> import numpy
>>> import scipy.linalg
>>> A = numpy.array( [[1 - 1j,     -3j, -2 + 1j,      1, -2 + 3j],
...                  [ 1 - 1j, -2 + 1j,  1 + 4j,      0,  2 + 1j],
...                  [     -1, -5 + 1j, -2 + 1j, 1 + 1j, -5 - 4j],
...                  [-2 + 4j,  2 - 1j,  8 - 4j, 1 - 8j,  3 - 2j]])
>>> U, S, VH = scipy.linalg.svd(A)
>>> VH
array([[-0.10378056+0.25534799j,  0.22371070-0.07765616j,
         0.55736639-0.40276461j, -0.04874824-0.50177439j,
         0.32025785-0.19584279j],
       [-0.28067375-0.0116824j , -0.49586800+0.1305711j ,
         0.14432786+0.08803575j,  0.21076793+0.04575091j,
        -0.19220732-0.73899332j],
       [ 0.10136055-0.1681669j ,  0.13465050+0.69687595j,
         0.19709231-0.03497225j, -0.42056091+0.37188099j,
         0.28760069-0.14046187j],
       [-0.27289262+0.66077235j,  0.06177902+0.25464336j,
        -0.52050549+0.06307174j,  0.11847581-0.00883692j,
         0.35771384-0.05719981j],
       [ 0.49800436+0.21785287j, -0.32014745+0.07789755j,
         0.15924098-0.39775924j,  0.50925102+0.33273359j,
         0.14247809+0.14849585j]])
>>> numpy.dot(VH, VH.conj().T)
array([[  1.00000000e+00 +1.38777878e-17j,
         -8.32667268e-17 +2.84494650e-16j,
          1.38777878e-16 -1.45716772e-16j,
         -1.94289029e-16 +1.38777878e-17j,
         -3.46944695e-17 +2.08166817e-17j],
       [ -8.32667268e-17 -2.77555756e-16j,
          1.00000000e+00 +0.00000000e+00j,
         -7.56339436e-16 +1.38777878e-16j,
          5.55111512e-17 -1.66533454e-16j,
          2.11636264e-16 +1.11022302e-16j],
       [  1.38777878e-16 +1.17961196e-16j,
         -7.56339436e-16 -1.28369537e-16j,
          1.00000000e+00 +6.93889390e-18j,
         -3.88578059e-16 +4.16333634e-17j,
         -1.04083409e-16 +3.12250226e-17j],
       [ -1.94289029e-16 -1.04083409e-17j,
          5.55111512e-17 +1.99493200e-16j,
         -3.88578059e-16 -2.08166817e-17j,
          1.00000000e+00 -6.93889390e-18j,
         -9.71445147e-17 -8.15320034e-17j],
       [ -3.46944695e-17 +2.08166817e-17j,
          2.11636264e-16 -1.00613962e-16j,
         -1.04083409e-16 -4.16333634e-17j,
         -9.71445147e-17 +6.93889390e-17j,
          1.00000000e+00 +0.00000000e+00j]])


From beezer at ups.edu  Tue Apr 26 13:45:08 2011
From: beezer at ups.edu (Rob Beezer)
Date: Tue, 26 Apr 2011 10:45:08 -0700
Subject: [Numpy-discussion] Zero row in SVD's unitary matrix on
	some	Mac's
In-Reply-To: <BANLkTimvMSAsR4+QacjNHUvaNLn1V6XFPA@mail.gmail.com>
References: <4DB5AC5D.7020000@ups.edu>
	<ip4cmd$7or$1@dough.gmane.org>	<4DB6EDDC.2050508@creativetrax.com>	<4DB6EED6.4030207@creativetrax.com>	<4DB6F156.5070209@creativetrax.com>	<4DB6F483.30403@creativetrax.com>
	<BANLkTimvMSAsR4+QacjNHUvaNLn1V6XFPA@mail.gmail.com>
Message-ID: <4DB704A4.6060707@ups.edu>

On 04/26/2011 10:19 AM, Ralf Gommers wrote:
> If it is an Accelerate bug it got fixed on OS X 10.6. Below is the
> output with both current master and 1.5.1. It may be worth checking
> this on 10.5 with the binary Numpy installer from Sourceforge to make
> sure it's not the Sage build process causing this somehow.

Thanks, Ralf.  Indeed, we have no reports of problems on OSX 10.6, and it is 
likely there have been many tests on that platform.  So a test of numpy on 10.5, 
independent of Sage's build process, would be useful.

I don't have access to a Mac, but I posted your suggestion to the Sage ticket 
for this, where hopefully somebody will take it up.


From mat.yeates at gmail.com  Tue Apr 26 14:00:30 2011
From: mat.yeates at gmail.com (Mathew Yeates)
Date: Tue, 26 Apr 2011 11:00:30 -0700
Subject: [Numpy-discussion] reading tiff images
Message-ID: <BANLkTi=pzbNE3KCHisZwmN+iqmp6TAgEBg@mail.gmail.com>

Hi
What is current method of using ndiimage on a Tiff file? I've seen
different methods using ndimage itself, scipy.misc and Pil.

Mathew


From pav at iki.fi  Tue Apr 26 14:25:22 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 26 Apr 2011 18:25:22 +0000 (UTC)
Subject: [Numpy-discussion] Zero row in SVD's unitary matrix
	on	some	Mac's
References: <4DB5AC5D.7020000@ups.edu> <ip4cmd$7or$1@dough.gmane.org>
	<4DB6E5B3.1000606@creativetrax.com>
Message-ID: <ip72mi$ffm$1@dough.gmane.org>

On Tue, 26 Apr 2011 10:33:07 -0500, Jason Grout wrote:
[clip]
> I was just looking up the documentation for ZGESDD and noticed that the
> value we have for rwork in the numpy call [1] does not match the Lapack
> docs.  This was changed in Lapack 3.2.2 [2].  I've submitted a pull
> request:
> 
> https://github.com/numpy/numpy/pull/78
> 
> but I have not tested the change.
> 
> I doubt this fix will fix the problem on this thread, but it makes sense
> to make the change anyway.

I think it is worth testing if this change fixes it. Note that current 
LAPACK docs have the number 7 there instead of 5, so there may really be 
a bug here.

	Pauli



From dplepage at gmail.com  Tue Apr 26 14:31:34 2011
From: dplepage at gmail.com (Daniel Lepage)
Date: Tue, 26 Apr 2011 14:31:34 -0400
Subject: [Numpy-discussion] reading tiff images
In-Reply-To: <BANLkTi=pzbNE3KCHisZwmN+iqmp6TAgEBg@mail.gmail.com>
References: <BANLkTi=pzbNE3KCHisZwmN+iqmp6TAgEBg@mail.gmail.com>
Message-ID: <BANLkTimegc8inoKFQpNCJUks4csheEJL8Q@mail.gmail.com>

You need PIL no matter what; scipy.misc.imread, scipy.ndimage.imread,
and scikits.image.io.imread all call PIL.

Theoretically there's no difference between any of them, although in
actuality some use "import Image" and others use "from PIL import
Image"; one of these may fail depending on how you installed PIL. (I'm
not sure which is supposed to be standard - the PIL docs use both
interchangeably, and I think the latest version of PIL on pypi sets it
up so that both will work).

I'd use whichever tool you're already importing - if you're using
ndimage anyway, just use ndimage.imread rather than adding more
imports.

Note that using PIL directly is easy, but does require adding an extra
step; OTOH, if you're familiar with PIL, you can use some of its
transformations from the start, e.g.

def imread(fname, mode='RGBA'):
    return np.asarray(Image.open(fname).convert(mode))

to ensure that you always get 4-channel images, even for images that
were initially RGB or grayscale.

HTH,
Dan

On Tue, Apr 26, 2011 at 2:00 PM, Mathew Yeates <mat.yeates at gmail.com> wrote:
> Hi
> What is current method of using ndiimage on a Tiff file? I've seen
> different methods using ndimage itself, scipy.misc and Pil.
>
> Mathew
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From ralf.gommers at googlemail.com  Tue Apr 26 14:43:31 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Tue, 26 Apr 2011 20:43:31 +0200
Subject: [Numpy-discussion] reading tiff images
In-Reply-To: <BANLkTimegc8inoKFQpNCJUks4csheEJL8Q@mail.gmail.com>
References: <BANLkTi=pzbNE3KCHisZwmN+iqmp6TAgEBg@mail.gmail.com>
	<BANLkTimegc8inoKFQpNCJUks4csheEJL8Q@mail.gmail.com>
Message-ID: <BANLkTinR27N7tZzm74v3hsoKLJMzbF_fvw@mail.gmail.com>

On Tue, Apr 26, 2011 at 8:31 PM, Daniel Lepage <dplepage at gmail.com> wrote:
> You need PIL no matter what; scipy.misc.imread, scipy.ndimage.imread,
> and scikits.image.io.imread all call PIL.

Scikits.image has a plugin system for IO and can use FreeImage to load
images. PIL's Tiff image handling is pretty buggy (especially
multi-page and 16-bit tiffs), so that may be a good option too.

Ralf


From zachary.pincus at yale.edu  Tue Apr 26 14:45:35 2011
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Tue, 26 Apr 2011 14:45:35 -0400
Subject: [Numpy-discussion] reading tiff images
In-Reply-To: <BANLkTimegc8inoKFQpNCJUks4csheEJL8Q@mail.gmail.com>
References: <BANLkTi=pzbNE3KCHisZwmN+iqmp6TAgEBg@mail.gmail.com>
	<BANLkTimegc8inoKFQpNCJUks4csheEJL8Q@mail.gmail.com>
Message-ID: <4B2F4523-B909-441C-BC3D-D3501093690A@yale.edu>


On Apr 26, 2011, at 2:31 PM, Daniel Lepage wrote:

> You need PIL no matter what; scipy.misc.imread, scipy.ndimage.imread,
> and scikits.image.io.imread all call PIL.

scikits.image.io also has a ctypes wrapper for the freeimage library.  
I prefer these (well, I wrote them), though apparently there are some  
64-bit issues (crashes?). I haven't been working on a 64-bit system so  
I haven't been able to address them, but I will be soon. It's a very  
thin wrapper around a simple image IO library, so there's lots of room  
to add and extend as need be...

All of the PIL wrappers are kluges around serious flaws in how PIL  
reads images, particularly non-8-bit images and in particular non- 
native-endian 16-bit images.

Zach


> Theoretically there's no difference between any of them, although in
> actuality some use "import Image" and others use "from PIL import
> Image"; one of these may fail depending on how you installed PIL. (I'm
> not sure which is supposed to be standard - the PIL docs use both
> interchangeably, and I think the latest version of PIL on pypi sets it
> up so that both will work).
>
> I'd use whichever tool you're already importing - if you're using
> ndimage anyway, just use ndimage.imread rather than adding more
> imports.
>
> Note that using PIL directly is easy, but does require adding an extra
> step; OTOH, if you're familiar with PIL, you can use some of its
> transformations from the start, e.g.
>
> def imread(fname, mode='RGBA'):
>    return np.asarray(Image.open(fname).convert(mode))
>
> to ensure that you always get 4-channel images, even for images that
> were initially RGB or grayscale.
>
> HTH,
> Dan
>
> On Tue, Apr 26, 2011 at 2:00 PM, Mathew Yeates  
> <mat.yeates at gmail.com> wrote:
>> Hi
>> What is current method of using ndiimage on a Tiff file? I've seen
>> different methods using ndimage itself, scipy.misc and Pil.
>>
>> Mathew
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



From beezer at ups.edu  Tue Apr 26 14:58:07 2011
From: beezer at ups.edu (Rob Beezer)
Date: Tue, 26 Apr 2011 11:58:07 -0700
Subject: [Numpy-discussion] Zero row in SVD's unitary
	matrix	on	some	Mac's
In-Reply-To: <ip72mi$ffm$1@dough.gmane.org>
References: <4DB5AC5D.7020000@ups.edu>
	<ip4cmd$7or$1@dough.gmane.org>	<4DB6E5B3.1000606@creativetrax.com>
	<ip72mi$ffm$1@dough.gmane.org>
Message-ID: <4DB715BF.80603@ups.edu>

On 04/26/2011 11:25 AM, Pauli Virtanen wrote:
> I think it is worth testing if this change fixes it. Note that current
> LAPACK docs have the number 7 there instead of 5, so there may really be
> a bug here.

It appears to me the problem is more subtle than just a strict comparison 
between the number of rows and the number of columns.

http://trac.sagemath.org/sage_trac/attachment/ticket/11248/broken.png

shows a plot of matrix sizes (number of rows on the horizontal, number of 
columns on vertical).  This is on an OSX 10.5 machine and whereever there is a 
dot then the SVD seems to *always* produce a non-unitary V, and the absence of a 
dot indicates no failures were found.  So maybe a workspace size could be the 
culprit?


From mat.yeates at gmail.com  Tue Apr 26 15:09:53 2011
From: mat.yeates at gmail.com (Mathew Yeates)
Date: Tue, 26 Apr 2011 12:09:53 -0700
Subject: [Numpy-discussion] reading tiff images
In-Reply-To: <4B2F4523-B909-441C-BC3D-D3501093690A@yale.edu>
References: <BANLkTi=pzbNE3KCHisZwmN+iqmp6TAgEBg@mail.gmail.com>
	<BANLkTimegc8inoKFQpNCJUks4csheEJL8Q@mail.gmail.com>
	<4B2F4523-B909-441C-BC3D-D3501093690A@yale.edu>
Message-ID: <BANLkTim833PM06vx-jt6vC4SWQ3ZRdE=Zg@mail.gmail.com>

is scikits.image.io documented anywhere?

On Tue, Apr 26, 2011 at 11:45 AM, Zachary Pincus
<zachary.pincus at yale.edu> wrote:
>
> On Apr 26, 2011, at 2:31 PM, Daniel Lepage wrote:
>
>> You need PIL no matter what; scipy.misc.imread, scipy.ndimage.imread,
>> and scikits.image.io.imread all call PIL.
>
> scikits.image.io also has a ctypes wrapper for the freeimage library.
> I prefer these (well, I wrote them), though apparently there are some
> 64-bit issues (crashes?). I haven't been working on a 64-bit system so
> I haven't been able to address them, but I will be soon. It's a very
> thin wrapper around a simple image IO library, so there's lots of room
> to add and extend as need be...
>
> All of the PIL wrappers are kluges around serious flaws in how PIL
> reads images, particularly non-8-bit images and in particular non-
> native-endian 16-bit images.
>
> Zach
>
>
>> Theoretically there's no difference between any of them, although in
>> actuality some use "import Image" and others use "from PIL import
>> Image"; one of these may fail depending on how you installed PIL. (I'm
>> not sure which is supposed to be standard - the PIL docs use both
>> interchangeably, and I think the latest version of PIL on pypi sets it
>> up so that both will work).
>>
>> I'd use whichever tool you're already importing - if you're using
>> ndimage anyway, just use ndimage.imread rather than adding more
>> imports.
>>
>> Note that using PIL directly is easy, but does require adding an extra
>> step; OTOH, if you're familiar with PIL, you can use some of its
>> transformations from the start, e.g.
>>
>> def imread(fname, mode='RGBA'):
>> ? ?return np.asarray(Image.open(fname).convert(mode))
>>
>> to ensure that you always get 4-channel images, even for images that
>> were initially RGB or grayscale.
>>
>> HTH,
>> Dan
>>
>> On Tue, Apr 26, 2011 at 2:00 PM, Mathew Yeates
>> <mat.yeates at gmail.com> wrote:
>>> Hi
>>> What is current method of using ndiimage on a Tiff file? I've seen
>>> different methods using ndimage itself, scipy.misc and Pil.
>>>
>>> Mathew
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From ralf.gommers at googlemail.com  Tue Apr 26 15:18:57 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Tue, 26 Apr 2011 21:18:57 +0200
Subject: [Numpy-discussion] reading tiff images
In-Reply-To: <BANLkTim833PM06vx-jt6vC4SWQ3ZRdE=Zg@mail.gmail.com>
References: <BANLkTi=pzbNE3KCHisZwmN+iqmp6TAgEBg@mail.gmail.com>
	<BANLkTimegc8inoKFQpNCJUks4csheEJL8Q@mail.gmail.com>
	<4B2F4523-B909-441C-BC3D-D3501093690A@yale.edu>
	<BANLkTim833PM06vx-jt6vC4SWQ3ZRdE=Zg@mail.gmail.com>
Message-ID: <BANLkTim6WxhzdVHJHC91DKYFMNCtDJFckQ@mail.gmail.com>

On Tue, Apr 26, 2011 at 9:09 PM, Mathew Yeates <mat.yeates at gmail.com> wrote:
> is scikits.image.io documented anywhere?

http://stefanv.github.com/scikits.image/api/scikits.image.io.html


From jason-sage at creativetrax.com  Tue Apr 26 15:52:52 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 26 Apr 2011 14:52:52 -0500
Subject: [Numpy-discussion] Zero row
	in	SVD's	unitary	matrix	on	some	Mac's
In-Reply-To: <ip6t27$jpi$1@dough.gmane.org>
References: <4DB5AC5D.7020000@ups.edu>
	<ip4cmd$7or$1@dough.gmane.org>	<4DB6EDDC.2050508@creativetrax.com>
	<4DB6EED6.4030207@creativetrax.com>	<4DB6F156.5070209@creativetrax.com>
	<4DB6F483.30403@creativetrax.com> <ip6t27$jpi$1@dough.gmane.org>
Message-ID: <4DB72294.5070709@creativetrax.com>

On 4/26/11 11:49 AM, Pauli Virtanen wrote:
> Tue, 26 Apr 2011 11:36:19 -0500, Jason Grout wrote:
> [clip]
>> Okay, just one more data point.  Our people that are seeing the problem
>> with numpy returning a non-unitary V also see a non-unitary V being
>> returned by the test C call to zgesdd.  In other words, it really
>> appears that zgesdd follows the Lapack docs, and if rows<columns, the
>> returned V is not necessarily unitary, but may contain a zero row.  This
>> makes numpy's assumptions in using zgesdd false.
>>
>> You can see this report at
>> http://trac.sagemath.org/sage_trac/ticket/11248#comment:25
>
> What LAPACK promises is not very clear. Earlier on the the man page:
>
>         JOBZ    (input) CHARACTER*1
>                 Specifies options for computing all or part of the matrix U:
>                 = 'A':  all M columns of U and all N rows of V**H are  returned
>                 in the arrays U and VT; = 'S':  the first min(M,N) columns of U
>                 and the first min(M,N) rows of V**H are returned in the  arrays
>                 U and VT; = 'O':  If M>= N, the first N columns of U are over?
>                 written in the array A and all rows of V**H are returned in the
>                 array VT; otherwise, all columns of U are returned in the array
>                 U and the first M rows of V**H are overwritten in the array  A;
>                 = 'N':  no columns of U or rows of V**H are computed.
>
> Looks to me the statement should be parsed as:
>
> 	return_n_rows = (jobz == 'A') or (jobz == 'O' and m>= n)
>
> So the current usage should be OK. (At least, as long as jobz == 'A',
> in the other cases, I don't think the wrapper does the right thing.)
>
> But apparently either there's a bug, or the LAPACK man page needs to
> be understood as you say.


Ah, you're right that it makes sense to parse their statement that way 
too, so I'm not so sure what Lapack really is saying anymore either.  If 
it's parsed the way you propose (which makes sense given the JOBZ 
description), I think it points to a bug in the Accelerate Lapack on the 
affected platforms, as we get the same zero row when we call the 
function directly from C, without numpy, python, or Sage in the middle.

The updated rwork calculation makes no difference with a 3x4 matrix 
(both the old calculation and the new calculation give 66 in the 3x4 
case), so I don't think that is affecting anything.

Jason


From jason-sage at creativetrax.com  Tue Apr 26 16:18:02 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 26 Apr 2011 15:18:02 -0500
Subject: [Numpy-discussion] Zero row
	in	SVD's	unitary	matrix	on	some	Mac's
In-Reply-To: <ip6t27$jpi$1@dough.gmane.org>
References: <4DB5AC5D.7020000@ups.edu>
	<ip4cmd$7or$1@dough.gmane.org>	<4DB6EDDC.2050508@creativetrax.com>
	<4DB6EED6.4030207@creativetrax.com>	<4DB6F156.5070209@creativetrax.com>
	<4DB6F483.30403@creativetrax.com> <ip6t27$jpi$1@dough.gmane.org>
Message-ID: <4DB7287A.8080606@creativetrax.com>

On 4/26/11 11:49 AM, Pauli Virtanen wrote:
> But apparently either there's a bug, or the LAPACK man page needs to
> be understood as you say.

I've posted a question to the Lapack forum:

http://icl.cs.utk.edu/lapack-forum/viewtopic.php?f=2&t=2402

Thanks,

Jason



From thouis at gmail.com  Tue Apr 26 16:23:18 2011
From: thouis at gmail.com (Thouis (Ray) Jones)
Date: Tue, 26 Apr 2011 22:23:18 +0200
Subject: [Numpy-discussion] reading tiff images
In-Reply-To: <BANLkTimegc8inoKFQpNCJUks4csheEJL8Q@mail.gmail.com>
References: <BANLkTi=pzbNE3KCHisZwmN+iqmp6TAgEBg@mail.gmail.com>
	<BANLkTimegc8inoKFQpNCJUks4csheEJL8Q@mail.gmail.com>
Message-ID: <BANLkTimCabrmjtZ0YusMK9vY8ybXwH+mxQ@mail.gmail.com>

On Tue, Apr 26, 2011 at 20:31, Daniel Lepage <dplepage at gmail.com> wrote:
> You need PIL no matter what; scipy.misc.imread, scipy.ndimage.imread,
> and scikits.image.io.imread all call PIL.

I believe there are two pure python readers:
http://code.google.com/p/pylibtiff/
http://www.lfd.uci.edu/~gohlke/code/tifffile.py.html

The first can also write TIFF, but does not handle tiled TIFFs.


From pav at iki.fi  Tue Apr 26 16:29:40 2011
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 26 Apr 2011 20:29:40 +0000 (UTC)
Subject: [Numpy-discussion] Zero
	row	in	SVD's	unitary	matrix	on	some	Mac's
References: <4DB5AC5D.7020000@ups.edu> <ip4cmd$7or$1@dough.gmane.org>
	<4DB6EDDC.2050508@creativetrax.com> <4DB6EED6.4030207@creativetrax.com>
	<4DB6F156.5070209@creativetrax.com> <4DB6F483.30403@creativetrax.com>
	<ip6t27$jpi$1@dough.gmane.org> <4DB72294.5070709@creativetrax.com>
Message-ID: <ip79vk$34n$1@dough.gmane.org>

On Tue, 26 Apr 2011 14:52:52 -0500, Jason Grout wrote:
[clip]
> The updated rwork calculation makes no difference with a 3x4 matrix
> (both the old calculation and the new calculation give 66 in the 3x4
> case), so I don't think that is affecting anything.

Actually, there *is* a difference for the 3*4 matrix:

	old = 3*3*5 + 5*3 = 60
	new = 3*3*5 + 7*3 = 66

The old calculation had 5 instead of 7 in the formula --- I don't know if 
it was written according to an older version of the documentation, or if 
was is simply a bug.

	Pauli



From jason-sage at creativetrax.com  Tue Apr 26 16:37:59 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 26 Apr 2011 15:37:59 -0500
Subject: [Numpy-discussion]
	Zero	row	in	SVD's	unitary	matrix	on	some	Mac's
In-Reply-To: <ip79vk$34n$1@dough.gmane.org>
References: <4DB5AC5D.7020000@ups.edu>
	<ip4cmd$7or$1@dough.gmane.org>	<4DB6EDDC.2050508@creativetrax.com>
	<4DB6EED6.4030207@creativetrax.com>	<4DB6F156.5070209@creativetrax.com>
	<4DB6F483.30403@creativetrax.com>	<ip6t27$jpi$1@dough.gmane.org>
	<4DB72294.5070709@creativetrax.com> <ip79vk$34n$1@dough.gmane.org>
Message-ID: <4DB72D27.30502@creativetrax.com>

On 4/26/11 3:29 PM, Pauli Virtanen wrote:
> On Tue, 26 Apr 2011 14:52:52 -0500, Jason Grout wrote:
> [clip]
>> The updated rwork calculation makes no difference with a 3x4 matrix
>> (both the old calculation and the new calculation give 66 in the 3x4
>> case), so I don't think that is affecting anything.
>
> Actually, there *is* a difference for the 3*4 matrix:
>
> 	old = 3*3*5 + 5*3 = 60
> 	new = 3*3*5 + 7*3 = 66
>
> The old calculation had 5 instead of 7 in the formula --- I don't know if
> it was written according to an older version of the documentation, or if
> was is simply a bug.


I was talking about our C example program, based on the example from 
Intel [1].  Intel already had the 3*3*5+7*3 calculation; that's what I 
called the "old" calculation.  The "new" calculation I was referring to 
was the "min(M,N)*max(5*min(M,N)+7,2*max(M,N)+2*min(M,N)+1)" calculation 
from Lapack 3.3.2.

Thanks,

Jason


[1] 
http://software.intel.com/sites/products/documentation/hpc/mkl/lapack/mkl_lapack_examples/zgesdd_ex.c.htm


From ralf.gommers at googlemail.com  Tue Apr 26 17:21:03 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Tue, 26 Apr 2011 23:21:03 +0200
Subject: [Numpy-discussion] Memmap with multiprocessing
In-Reply-To: <BANLkTi=SsoQ-mbTokYBO_+rL099myYNTkA@mail.gmail.com>
References: <4DAE2242.9030506@gmail.com>
	<BANLkTi=SsoQ-mbTokYBO_+rL099myYNTkA@mail.gmail.com>
Message-ID: <BANLkTin9jrD=fOurTgFSDUWs3L0Vp1Ms4w@mail.gmail.com>

On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes
<totonixsame at gmail.com> wrote:
> Hi,
>
> Has anyone confirmed if this is a bug? Should I post this in the bug tracker?

I see the same thing with recent master. Something very strange is
going on in the memmap.__array_finalize__ method under Windows. Can
you file a bug?

Ralf


>
> Thanks!
>
> On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes
> <totonixsame at gmail.com> wrote:
>> Hi all,
>>
>> I'm having a error using memmap objects shared among processes created
>> by the multprocessing module. This error only happen in Windows with
>> numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac
>> OS X it doesn't happen. This error is demonstrated by this little
>> example script here https://gist.github.com/929168 , and the traceback
>> is bellow (between <traceback> tags):
>>
>> <traceback>
>> Process Process-1:
>> Traceback (most recent call last):
>> ?File "C:\Python26\Lib\multiprocessing\process.py", line 232, in _bootstrap
>> ? ?self.run()
>> ?File "C:\Python26\Lib\multiprocessing\process.py", line 88, in run
>> ? ?self._target(*self._args, **self._kwargs)
>> ?File "C:\Documents and Settings\phamorim\Desktop\test.py", line 7, in
>> print_ma
>> trix
>> ? ?print matrix
>> ?File "C:\Python26\Lib\site-packages\numpy\core\numeric.py", line 1379, in
>> arra
>> y_str
>> ? ?return array2string(a, max_line_width, precision, suppress_small, ' ',
>> "", s
>> tr)
>> ?File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 309, in
>> ar
>> ray2string
>> ? ?separator, prefix)
>> ?File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 189, in
>> _a
>> rray2string
>> ? ?data = _leading_trailing(a)
>> ?File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 162, in
>> _l
>> eading_trailing
>> ? ?min(len(a), _summaryEdgeItems))]
>> ?File "C:\Python26\Lib\site-packages\numpy\core\memmap.py", line 257, in
>> __arra
>> y_finalize__
>> ? ?self.filename = obj.filename
>> AttributeError: 'memmap' object has no attribute 'filename'
>> Exception AttributeError: AttributeError("'NoneType' object has no attribute
>> 'te
>> ll'",) in <bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0,
>> 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0], dtype=int16)> ignored
>> </traceback>
>>
>> I don't know if it's a bug, but I thought it's is import to report
>> because the version 1.4.1 was working and 1.5.0 and above was not.
>>
>> Thanks!
>>
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From warren.weckesser at enthought.com  Tue Apr 26 17:50:48 2011
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Tue, 26 Apr 2011 16:50:48 -0500
Subject: [Numpy-discussion] Memmap with multiprocessing
In-Reply-To: <BANLkTin9jrD=fOurTgFSDUWs3L0Vp1Ms4w@mail.gmail.com>
References: <4DAE2242.9030506@gmail.com>
	<BANLkTi=SsoQ-mbTokYBO_+rL099myYNTkA@mail.gmail.com>
	<BANLkTin9jrD=fOurTgFSDUWs3L0Vp1Ms4w@mail.gmail.com>
Message-ID: <BANLkTinTVtub0GMzesF9a3ZEARj8t17fjA@mail.gmail.com>

On Tue, Apr 26, 2011 at 4:21 PM, Ralf Gommers
<ralf.gommers at googlemail.com>wrote:

> On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes
> <totonixsame at gmail.com> wrote:
> > Hi,
> >
> > Has anyone confirmed if this is a bug? Should I post this in the bug
> tracker?
>
> I see the same thing with recent master. Something very strange is
> going on in the memmap.__array_finalize__ method under Windows. Can
> you file a bug?
>
>

Just a confirmation: I get the error with numpy 1.5.1, Python 2.7.1 in
Windows XP, but not in Mac OSX or Linux (Ubuntu).

Warren


 Ralf
>
>
> >
> > Thanks!
> >
> > On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes
> > <totonixsame at gmail.com> wrote:
> >> Hi all,
> >>
> >> I'm having a error using memmap objects shared among processes created
> >> by the multprocessing module. This error only happen in Windows with
> >> numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac
> >> OS X it doesn't happen. This error is demonstrated by this little
> >> example script here https://gist.github.com/929168 , and the traceback
> >> is bellow (between <traceback> tags):
> >>
> >> <traceback>
> >> Process Process-1:
> >> Traceback (most recent call last):
> >>  File "C:\Python26\Lib\multiprocessing\process.py", line 232, in
> _bootstrap
> >>    self.run()
> >>  File "C:\Python26\Lib\multiprocessing\process.py", line 88, in run
> >>    self._target(*self._args, **self._kwargs)
> >>  File "C:\Documents and Settings\phamorim\Desktop\test.py", line 7, in
> >> print_ma
> >> trix
> >>    print matrix
> >>  File "C:\Python26\Lib\site-packages\numpy\core\numeric.py", line 1379,
> in
> >> arra
> >> y_str
> >>    return array2string(a, max_line_width, precision, suppress_small, '
> ',
> >> "", s
> >> tr)
> >>  File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line
> 309, in
> >> ar
> >> ray2string
> >>    separator, prefix)
> >>  File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line
> 189, in
> >> _a
> >> rray2string
> >>    data = _leading_trailing(a)
> >>  File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line
> 162, in
> >> _l
> >> eading_trailing
> >>    min(len(a), _summaryEdgeItems))]
> >>  File "C:\Python26\Lib\site-packages\numpy\core\memmap.py", line 257, in
> >> __arra
> >> y_finalize__
> >>    self.filename = obj.filename
> >> AttributeError: 'memmap' object has no attribute 'filename'
> >> Exception AttributeError: AttributeError("'NoneType' object has no
> attribute
> >> 'te
> >> ll'",) in <bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0,
> 0, 0,
> >> 0,
> >> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> >>       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0,
> >>       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0,
> >>       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0,
> >>       0, 0, 0, 0, 0, 0, 0, 0], dtype=int16)> ignored
> >> </traceback>
> >>
> >> I don't know if it's a bug, but I thought it's is import to report
> >> because the version 1.4.1 was working and 1.5.0 and above was not.
> >>
> >> Thanks!
> >>
> >>
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110426/f08a62ca/attachment.html>

From totonixsame at gmail.com  Wed Apr 27 08:26:44 2011
From: totonixsame at gmail.com (Thiago Franco Moraes)
Date: Wed, 27 Apr 2011 09:26:44 -0300
Subject: [Numpy-discussion] Memmap with multiprocessing
In-Reply-To: <BANLkTin9jrD=fOurTgFSDUWs3L0Vp1Ms4w@mail.gmail.com>
References: <4DAE2242.9030506@gmail.com>
	<BANLkTi=SsoQ-mbTokYBO_+rL099myYNTkA@mail.gmail.com>
	<BANLkTin9jrD=fOurTgFSDUWs3L0Vp1Ms4w@mail.gmail.com>
Message-ID: <BANLkTimdCJpUc1RxAOiNHo54aBPPWk2gMA@mail.gmail.com>

On Tue, Apr 26, 2011 at 6:21 PM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
> On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes
> <totonixsame at gmail.com> wrote:
>> Hi,
>>
>> Has anyone confirmed if this is a bug? Should I post this in the bug tracker?
>
> I see the same thing with recent master. Something very strange is
> going on in the memmap.__array_finalize__ method under Windows. Can
> you file a bug?
>
> Ralf

Hi Ralf,

Done http://projects.scipy.org/numpy/ticket/1809

Thanks!

>> Thanks!
>>
>> On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes
>> <totonixsame at gmail.com> wrote:
>>> Hi all,
>>>
>>> I'm having a error using memmap objects shared among processes created
>>> by the multprocessing module. This error only happen in Windows with
>>> numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac
>>> OS X it doesn't happen. This error is demonstrated by this little
>>> example script here https://gist.github.com/929168 , and the traceback
>>> is bellow (between <traceback> tags):
>>>
>>> <traceback>
>>> Process Process-1:
>>> Traceback (most recent call last):
>>> ?File "C:\Python26\Lib\multiprocessing\process.py", line 232, in _bootstrap
>>> ? ?self.run()
>>> ?File "C:\Python26\Lib\multiprocessing\process.py", line 88, in run
>>> ? ?self._target(*self._args, **self._kwargs)
>>> ?File "C:\Documents and Settings\phamorim\Desktop\test.py", line 7, in
>>> print_ma
>>> trix
>>> ? ?print matrix
>>> ?File "C:\Python26\Lib\site-packages\numpy\core\numeric.py", line 1379, in
>>> arra
>>> y_str
>>> ? ?return array2string(a, max_line_width, precision, suppress_small, ' ',
>>> "", s
>>> tr)
>>> ?File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 309, in
>>> ar
>>> ray2string
>>> ? ?separator, prefix)
>>> ?File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 189, in
>>> _a
>>> rray2string
>>> ? ?data = _leading_trailing(a)
>>> ?File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 162, in
>>> _l
>>> eading_trailing
>>> ? ?min(len(a), _summaryEdgeItems))]
>>> ?File "C:\Python26\Lib\site-packages\numpy\core\memmap.py", line 257, in
>>> __arra
>>> y_finalize__
>>> ? ?self.filename = obj.filename
>>> AttributeError: 'memmap' object has no attribute 'filename'
>>> Exception AttributeError: AttributeError("'NoneType' object has no attribute
>>> 'te
>>> ll'",) in <bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> ? ? ? 0, 0, 0, 0, 0, 0, 0, 0], dtype=int16)> ignored
>>> </traceback>
>>>
>>> I don't know if it's a bug, but I thought it's is import to report
>>> because the version 1.4.1 was working and 1.5.0 and above was not.
>>>
>>> Thanks!
>>>
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From matthew.brett at gmail.com  Wed Apr 27 14:52:15 2011
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 27 Apr 2011 14:52:15 -0400
Subject: [Numpy-discussion] Numpy steering group?
Message-ID: <BANLkTin1ERp4bxnok6GsT6eH6VzM6r-NDA@mail.gmail.com>

Hi,

This is just to follow up on a dead thread of mine a little while back.

I was asking about letters for Clint Whaley's tenure case, from numpy,
but I realized that I don't know who 'numpy' is :)

Is there in fact a numpy steering group?    Who is best to write
letters representing the 'numpy community'?

Thanks a lot,

Matthew


From cgohlke at uci.edu  Wed Apr 27 18:07:43 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Wed, 27 Apr 2011 15:07:43 -0700
Subject: [Numpy-discussion] Memmap with multiprocessing
In-Reply-To: <BANLkTin9jrD=fOurTgFSDUWs3L0Vp1Ms4w@mail.gmail.com>
References: <4DAE2242.9030506@gmail.com>	<BANLkTi=SsoQ-mbTokYBO_+rL099myYNTkA@mail.gmail.com>
	<BANLkTin9jrD=fOurTgFSDUWs3L0Vp1Ms4w@mail.gmail.com>
Message-ID: <4DB893AF.8060402@uci.edu>

I don't think this was working correctly in numpy 1.4 either. The 
underlying problem seems to be that instance attributes of ndarray 
subtypes get lost during pickling:

import pickle
import numpy as np
class aarray(np.ndarray):
     def __new__(subtype):
         self = np.ndarray.__new__(subtype, (1,))
         self.attr = 'attr'
         return self
     def __array_finalize__(self, obj):
         self.attr = getattr(obj, 'attr', None)
a = aarray()
b = pickle.loads(a.dumps())
assert a.attr == b.attr, (a.attr, b.attr)

AssertionError: ('attr', None)

Christoph


On 4/26/2011 2:21 PM, Ralf Gommers wrote:
> On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes
> <totonixsame at gmail.com>  wrote:
>> Hi,
>>
>> Has anyone confirmed if this is a bug? Should I post this in the bug tracker?
>
> I see the same thing with recent master. Something very strange is
> going on in the memmap.__array_finalize__ method under Windows. Can
> you file a bug?
>
> Ralf
>
>
>>
>> Thanks!
>>
>> On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes
>> <totonixsame at gmail.com>  wrote:
>>> Hi all,
>>>
>>> I'm having a error using memmap objects shared among processes created
>>> by the multprocessing module. This error only happen in Windows with
>>> numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac
>>> OS X it doesn't happen. This error is demonstrated by this little
>>> example script here https://gist.github.com/929168 , and the traceback
>>> is bellow (between<traceback>  tags):
>>>
>>> <traceback>
>>> Process Process-1:
>>> Traceback (most recent call last):
>>>   File "C:\Python26\Lib\multiprocessing\process.py", line 232, in _bootstrap
>>>     self.run()
>>>   File "C:\Python26\Lib\multiprocessing\process.py", line 88, in run
>>>     self._target(*self._args, **self._kwargs)
>>>   File "C:\Documents and Settings\phamorim\Desktop\test.py", line 7, in
>>> print_ma
>>> trix
>>>     print matrix
>>>   File "C:\Python26\Lib\site-packages\numpy\core\numeric.py", line 1379, in
>>> arra
>>> y_str
>>>     return array2string(a, max_line_width, precision, suppress_small, ' ',
>>> "", s
>>> tr)
>>>   File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 309, in
>>> ar
>>> ray2string
>>>     separator, prefix)
>>>   File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 189, in
>>> _a
>>> rray2string
>>>     data = _leading_trailing(a)
>>>   File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 162, in
>>> _l
>>> eading_trailing
>>>     min(len(a), _summaryEdgeItems))]
>>>   File "C:\Python26\Lib\site-packages\numpy\core\memmap.py", line 257, in
>>> __arra
>>> y_finalize__
>>>     self.filename = obj.filename
>>> AttributeError: 'memmap' object has no attribute 'filename'
>>> Exception AttributeError: AttributeError("'NoneType' object has no attribute
>>> 'te
>>> ll'",) in<bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>        0, 0, 0, 0, 0, 0, 0, 0], dtype=int16)>  ignored
>>> </traceback>
>>>
>>> I don't know if it's a bug, but I thought it's is import to report
>>> because the version 1.4.1 was working and 1.5.0 and above was not.
>>>
>>> Thanks!
>>>
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>


From bpederse at gmail.com  Wed Apr 27 18:19:50 2011
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 27 Apr 2011 16:19:50 -0600
Subject: [Numpy-discussion] Memmap with multiprocessing
In-Reply-To: <4DB893AF.8060402@uci.edu>
References: <4DAE2242.9030506@gmail.com>
	<BANLkTi=SsoQ-mbTokYBO_+rL099myYNTkA@mail.gmail.com>
	<BANLkTin9jrD=fOurTgFSDUWs3L0Vp1Ms4w@mail.gmail.com>
	<4DB893AF.8060402@uci.edu>
Message-ID: <BANLkTi=dihduh2S=Uu9-cM8rib5a2KmhSQ@mail.gmail.com>

On Wed, Apr 27, 2011 at 4:07 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:
> I don't think this was working correctly in numpy 1.4 either. The
> underlying problem seems to be that instance attributes of ndarray
> subtypes get lost during pickling:
>
> import pickle
> import numpy as np
> class aarray(np.ndarray):
> ? ? def __new__(subtype):
> ? ? ? ? self = np.ndarray.__new__(subtype, (1,))
> ? ? ? ? self.attr = 'attr'
> ? ? ? ? return self
> ? ? def __array_finalize__(self, obj):
> ? ? ? ? self.attr = getattr(obj, 'attr', None)
> a = aarray()
> b = pickle.loads(a.dumps())
> assert a.attr == b.attr, (a.attr, b.attr)
>
> AssertionError: ('attr', None)
>
> Christoph
>

possibly related to this ticket:
http://projects.scipy.org/numpy/ticket/1452


>
> On 4/26/2011 2:21 PM, Ralf Gommers wrote:
>> On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes
>> <totonixsame at gmail.com> ?wrote:
>>> Hi,
>>>
>>> Has anyone confirmed if this is a bug? Should I post this in the bug tracker?
>>
>> I see the same thing with recent master. Something very strange is
>> going on in the memmap.__array_finalize__ method under Windows. Can
>> you file a bug?
>>
>> Ralf
>>
>>
>>>
>>> Thanks!
>>>
>>> On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes
>>> <totonixsame at gmail.com> ?wrote:
>>>> Hi all,
>>>>
>>>> I'm having a error using memmap objects shared among processes created
>>>> by the multprocessing module. This error only happen in Windows with
>>>> numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac
>>>> OS X it doesn't happen. This error is demonstrated by this little
>>>> example script here https://gist.github.com/929168 , and the traceback
>>>> is bellow (between<traceback> ?tags):
>>>>
>>>> <traceback>
>>>> Process Process-1:
>>>> Traceback (most recent call last):
>>>> ? File "C:\Python26\Lib\multiprocessing\process.py", line 232, in _bootstrap
>>>> ? ? self.run()
>>>> ? File "C:\Python26\Lib\multiprocessing\process.py", line 88, in run
>>>> ? ? self._target(*self._args, **self._kwargs)
>>>> ? File "C:\Documents and Settings\phamorim\Desktop\test.py", line 7, in
>>>> print_ma
>>>> trix
>>>> ? ? print matrix
>>>> ? File "C:\Python26\Lib\site-packages\numpy\core\numeric.py", line 1379, in
>>>> arra
>>>> y_str
>>>> ? ? return array2string(a, max_line_width, precision, suppress_small, ' ',
>>>> "", s
>>>> tr)
>>>> ? File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 309, in
>>>> ar
>>>> ray2string
>>>> ? ? separator, prefix)
>>>> ? File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 189, in
>>>> _a
>>>> rray2string
>>>> ? ? data = _leading_trailing(a)
>>>> ? File "C:\Python26\Lib\site-packages\numpy\core\arrayprint.py", line 162, in
>>>> _l
>>>> eading_trailing
>>>> ? ? min(len(a), _summaryEdgeItems))]
>>>> ? File "C:\Python26\Lib\site-packages\numpy\core\memmap.py", line 257, in
>>>> __arra
>>>> y_finalize__
>>>> ? ? self.filename = obj.filename
>>>> AttributeError: 'memmap' object has no attribute 'filename'
>>>> Exception AttributeError: AttributeError("'NoneType' object has no attribute
>>>> 'te
>>>> ll'",) in<bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> 0,
>>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> ? ? ? ?0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> ? ? ? ?0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> ? ? ? ?0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> ? ? ? ?0, 0, 0, 0, 0, 0, 0, 0], dtype=int16)> ?ignored
>>>> </traceback>
>>>>
>>>> I don't know if it's a bug, but I thought it's is import to report
>>>> because the version 1.4.1 was working and 1.5.0 and above was not.
>>>>
>>>> Thanks!
>>>>
>>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From jason-sage at creativetrax.com  Thu Apr 28 07:04:52 2011
From: jason-sage at creativetrax.com (Jason Grout)
Date: Thu, 28 Apr 2011 06:04:52 -0500
Subject: [Numpy-discussion] Zero
	row	in	SVD's	unitary	matrix	on	some	Mac's
In-Reply-To: <4DB7287A.8080606@creativetrax.com>
References: <4DB5AC5D.7020000@ups.edu>	<ip4cmd$7or$1@dough.gmane.org>	<4DB6EDDC.2050508@creativetrax.com>	<4DB6EED6.4030207@creativetrax.com>	<4DB6F156.5070209@creativetrax.com>	<4DB6F483.30403@creativetrax.com>
	<ip6t27$jpi$1@dough.gmane.org> <4DB7287A.8080606@creativetrax.com>
Message-ID: <4DB949D4.1070002@creativetrax.com>

On 4/26/11 3:18 PM, Jason Grout wrote:
> On 4/26/11 11:49 AM, Pauli Virtanen wrote:
>> But apparently either there's a bug, or the LAPACK man page needs to
>> be understood as you say.
>
> I've posted a question to the Lapack forum:
>
> http://icl.cs.utk.edu/lapack-forum/viewtopic.php?f=2&t=2402

We've got a reply now:

http://icl.cs.utk.edu/lapack-forum/viewtopic.php?f=2&t=2402

Pauli, you're right that the statement should be parsed as:

"If (JOBZ = 'A') or (JOBZ = 'O' and M >= N), VT contains the N-by-N 
unitary matrix V**H;"

so you're right that the function is written correctly in the case we 
are bringing up (but still has an error in assuming too much when JOBZ 
is not 'A').

We'll work on providing an example with the Accelerate framework and the 
zero row.

Thanks,

Jason



From halbert at halwitz.org  Thu Apr 28 17:22:53 2011
From: halbert at halwitz.org (Dan Halbert)
Date: Thu, 28 Apr 2011 17:22:53 -0400 (EDT)
Subject: [Numpy-discussion] numpy.load truncates read from network file on XP
Message-ID: <1304025773.276522670@beta.apps.rackspace.com>

I'm having trouble loading a large remote .npy file on Windows XP. This is on numpy-1.3.0 on Windows XP SP3:

    numpy.load(r'\\myserver\mydir\big.npy')

will fail with this sort of error being printed:
    "14328000 items requested but only 54 read"
and then I get this with a backtrace:
    "ValueError: total size of new array must be unchanged" (due to the truncated array)

The file big.npy is a big 2d array, about 112MB.

The same file when stored locally gives no error when read. I can also read it into an editor, or copy it, and I get the whole thing.

More strangely, the same file when read from the same UNC path on Windows 7 64-bit (with the same 32-bit versions of all Python-related software) does not give an error either.

The fileserver in question is a NetApp box. Any clues? My websearching hasn't netted any leads.

Thanks a lot,
Dan





From bsouthey at gmail.com  Thu Apr 28 21:56:42 2011
From: bsouthey at gmail.com (Bruce Southey)
Date: Thu, 28 Apr 2011 20:56:42 -0500
Subject: [Numpy-discussion] numpy.load truncates read from network file
	on XP
In-Reply-To: <1304025773.276522670@beta.apps.rackspace.com>
References: <1304025773.276522670@beta.apps.rackspace.com>
Message-ID: <BANLkTik7dwLULRQuEcUVggROST5xrk+YCQ@mail.gmail.com>

On Thu, Apr 28, 2011 at 4:22 PM, Dan Halbert <halbert at halwitz.org> wrote:
> I'm having trouble loading a large remote .npy file on Windows XP. This is on numpy-1.3.0 on Windows XP SP3:
>
> ? ?numpy.load(r'\\myserver\mydir\big.npy')
>
> will fail with this sort of error being printed:
> ? ?"14328000 items requested but only 54 read"
> and then I get this with a backtrace:
> ? ?"ValueError: total size of new array must be unchanged" (due to the truncated array)
>
> The file big.npy is a big 2d array, about 112MB.
>
> The same file when stored locally gives no error when read. I can also read it into an editor, or copy it, and I get the whole thing.
>
> More strangely, the same file when read from the same UNC path on Windows 7 64-bit (with the same 32-bit versions of all Python-related software) does not give an error either.
>
> The fileserver in question is a NetApp box. Any clues? My websearching hasn't netted any leads.
>
> Thanks a lot,
> Dan
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
Don't know what a 'NetApp box' is, what OS it runs or what are the
filesystem types being used. Really we need more information
especially what the file is and how it was created etc., as well as
the different OSes involved with the exact Python and numpy versions
on each system - numpy 1.3 is getting a little old now with 1.6 due
soon.

A previous error has been the different line endings between OS's when
transferring between Linux and windows. That should show up with a
smaller version of the file. So at least try finding the smallest file
that you gives an error.

Other other issue may be connection related in that Python is not
getting the complete file so you might want to read the file directly
from Python first or change some of XP's virtual memory settings.
There have been significant changes between XP and Win 7 in that
regard:
http://en.wikipedia.org/wiki/Comparison_of_Windows_Vista_and_Windows_XP
"File copy operations proved to be one area where Vista performs
better than XP. A 1.25 GB file was copied from a network share to each
desktop. For XP, it took 2 minutes and 54 seconds, for Vista with SP1
it took 2 minutes and 29 seconds. This test was done by CRN Test
Center, but it omitted the fact that a machine running Vista takes
circa one extra minute to boot, if compared to a similar one operating
XP. However, the Vista implementation of the file copy is arguably
more complete and correct as the file does not register as being
transferred until it has completely transferred. In Windows XP, the
file completed dialogue box is displayed prior to the file actually
finishing its copy or transfer, with the file completing after the
dialogue is displayed. This can cause an issue if the storage device
is ejected prior to the file being successfully transferred or copied
in windows XP due to the dialogue box's premature prompt."


Bruce


From nadavh at visionsense.com  Fri Apr 29 01:18:06 2011
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu, 28 Apr 2011 22:18:06 -0700
Subject: [Numpy-discussion] numpy.load truncates read from network
	file	on XP
In-Reply-To: <BANLkTik7dwLULRQuEcUVggROST5xrk+YCQ@mail.gmail.com>
References: <1304025773.276522670@beta.apps.rackspace.com>,
	<BANLkTik7dwLULRQuEcUVggROST5xrk+YCQ@mail.gmail.com>
Message-ID: <26FC23E7C398A64083C980D16001012D1CE03D52DE@VA3DIAXVS361.RED001.local>

Several time I encountered problems in transfering large files between XP stations on a wireless network. Could be a result of the unsafe UDP prtocol used by microsoft network protocol (do not have a vista/7 machines to test it).

  Nadav

________________________________________
From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Bruce Southey [bsouthey at gmail.com]
Sent: 29 April 2011 04:56
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] numpy.load truncates read from network file     on XP

On Thu, Apr 28, 2011 at 4:22 PM, Dan Halbert <halbert at halwitz.org> wrote:
> I'm having trouble loading a large remote .npy file on Windows XP. This is on numpy-1.3.0 on Windows XP SP3:
>
>    numpy.load(r'\\myserver\mydir\big.npy')
>
> will fail with this sort of error being printed:
>    "14328000 items requested but only 54 read"
> and then I get this with a backtrace:
>    "ValueError: total size of new array must be unchanged" (due to the truncated array)
>
> The file big.npy is a big 2d array, about 112MB.
>
> The same file when stored locally gives no error when read. I can also read it into an editor, or copy it, and I get the whole thing.
>
> More strangely, the same file when read from the same UNC path on Windows 7 64-bit (with the same 32-bit versions of all Python-related software) does not give an error either.
>
> The fileserver in question is a NetApp box. Any clues? My websearching hasn't netted any leads.
>
> Thanks a lot,
> Dan
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
Don't know what a 'NetApp box' is, what OS it runs or what are the
filesystem types being used. Really we need more information
especially what the file is and how it was created etc., as well as
the different OSes involved with the exact Python and numpy versions
on each system - numpy 1.3 is getting a little old now with 1.6 due
soon.

A previous error has been the different line endings between OS's when
transferring between Linux and windows. That should show up with a
smaller version of the file. So at least try finding the smallest file
that you gives an error.

Other other issue may be connection related in that Python is not
getting the complete file so you might want to read the file directly
from Python first or change some of XP's virtual memory settings.
There have been significant changes between XP and Win 7 in that
regard:
http://en.wikipedia.org/wiki/Comparison_of_Windows_Vista_and_Windows_XP
"File copy operations proved to be one area where Vista performs
better than XP. A 1.25 GB file was copied from a network share to each
desktop. For XP, it took 2 minutes and 54 seconds, for Vista with SP1
it took 2 minutes and 29 seconds. This test was done by CRN Test
Center, but it omitted the fact that a machine running Vista takes
circa one extra minute to boot, if compared to a similar one operating
XP. However, the Vista implementation of the file copy is arguably
more complete and correct as the file does not register as being
transferred until it has completely transferred. In Windows XP, the
file completed dialogue box is displayed prior to the file actually
finishing its copy or transfer, with the file completing after the
dialogue is displayed. This can cause an issue if the storage device
is ejected prior to the file being successfully transferred or copied
in windows XP due to the dialogue box's premature prompt."


Bruce
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion at scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

From dileepkunjaai at gmail.com  Fri Apr 29 03:26:04 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Fri, 29 Apr 2011 12:56:04 +0530
Subject: [Numpy-discussion] Array of size 'n' with common difference < 1
Message-ID: <BANLkTinw=qYoQ-4iOHuqZ9AhGCw1P5X8yw@mail.gmail.com>

Dear sir,
 I am trying to make an array of varies from  -60 to 90 with difference
0.25. I tried the following command ...

>>import numpy as N
    lat=N.array(xrange(-6000, 9000, 25), dtype=float)
    print lat/100


I know that there is another easy method......
Please give me a replay ........
                                                                 Thanking
you..............


-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110429/05d073ea/attachment.html>

From youngung.jeong at gmail.com  Fri Apr 29 03:30:23 2011
From: youngung.jeong at gmail.com (Youngung Jeong)
Date: Fri, 29 Apr 2011 16:30:23 +0900
Subject: [Numpy-discussion] Array of size 'n' with common difference < 1
In-Reply-To: <BANLkTinw=qYoQ-4iOHuqZ9AhGCw1P5X8yw@mail.gmail.com>
References: <BANLkTinw=qYoQ-4iOHuqZ9AhGCw1P5X8yw@mail.gmail.com>
Message-ID: <BANLkTimMfuXApiK_UFHdFXbt5TPZHHL-sw@mail.gmail.com>

Use  " >> np.arange(-60, 90.0001, 0.25)"

Youngung Jeong

Graduate student
Materials Mechanics Laboratory
Graduate Institute of Ferrous Technology
Pohang University of Science and Technology




On Fri, Apr 29, 2011 at 4:26 PM, dileep kunjaai <dileepkunjaai at gmail.com>wrote:

> Dear sir,
>  I am trying to make an array of varies from  -60 to 90 with difference
> 0.25. I tried the following command ...
>
> >>import numpy as N
>     lat=N.array(xrange(-6000, 9000, 25), dtype=float)
>     print lat/100
>
>
> I know that there is another easy method......
> Please give me a replay ........
>                                                                  Thanking
> you..............
>
>
> --
> DILEEPKUMAR. R
> J R F, IIT DELHI
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110429/dda4e4b0/attachment.html>

From pratik.mallya at gmail.com  Fri Apr 29 03:31:53 2011
From: pratik.mallya at gmail.com (pratik)
Date: Fri, 29 Apr 2011 13:01:53 +0530
Subject: [Numpy-discussion] Array of size 'n' with common difference < 1
In-Reply-To: <BANLkTinw=qYoQ-4iOHuqZ9AhGCw1P5X8yw@mail.gmail.com>
References: <BANLkTinw=qYoQ-4iOHuqZ9AhGCw1P5X8yw@mail.gmail.com>
Message-ID: <4DBA6969.1030107@gmail.com>

On Friday 29 April 2011 12:56 PM, dileep kunjaai wrote:
> Dear sir,
>  I am trying to make an array of varies from  -60 to 90 with 
> difference 0.25. I tried the following command ...
>
> >>import numpy as N
>     lat=N.array(xrange(-6000, 9000, 25), dtype=float)
>     print lat/100
>
>
> I know that there is another easy method......
> Please give me a replay ........
>                                                                  
> Thanking you..............
>
>
> -- 
> DILEEPKUMAR. R
> J R F, IIT DELHI
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>    
use the numpy.ogrid command:
lat =numpy.mgrid[-60:90:.25]

pratik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110429/bce33eb8/attachment.html>

From pratik.mallya at gmail.com  Fri Apr 29 03:33:02 2011
From: pratik.mallya at gmail.com (pratik)
Date: Fri, 29 Apr 2011 13:03:02 +0530
Subject: [Numpy-discussion] Array of size 'n' with common difference < 1
In-Reply-To: <4DBA6969.1030107@gmail.com>
References: <BANLkTinw=qYoQ-4iOHuqZ9AhGCw1P5X8yw@mail.gmail.com>
	<4DBA6969.1030107@gmail.com>
Message-ID: <4DBA69AE.5050304@gmail.com>

On Friday 29 April 2011 01:01 PM, pratik wrote:
> On Friday 29 April 2011 12:56 PM, dileep kunjaai wrote:
>> Dear sir,
>>  I am trying to make an array of varies from  -60 to 90 with 
>> difference 0.25. I tried the following command ...
>>
>> >>import numpy as N
>>     lat=N.array(xrange(-6000, 9000, 25), dtype=float)
>>     print lat/100
>>
>>
>> I know that there is another easy method......
>> Please give me a replay ........
>>                                                                  
>> Thanking you..............
>>
>>
>> -- 
>> DILEEPKUMAR. R
>> J R F, IIT DELHI
>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>    
> use the numpy.ogrid command:
> lat =numpy.mgrid[-60:90:.25]
>
> pratik
i meant numpy.mgrid (not ogrid) :)

pratik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110429/cefa7d30/attachment.html>

From thouis at gmail.com  Fri Apr 29 09:45:19 2011
From: thouis at gmail.com (Thouis (Ray) Jones)
Date: Fri, 29 Apr 2011 15:45:19 +0200
Subject: [Numpy-discussion] Possible funding opportunity for numpy and scipy
	development
Message-ID: <BANLkTiniEh_zO6A_S-kT8RGpOA=jHA+3VQ@mail.gmail.com>

(Posted for a friend)

This NSF opportunity just renewed last week and might interest the
NumPy/SciPy community:
NSF Software Infrastructure for Sustained Innovation  (SI2)
http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=503489

It sounds like it's not suited for end-user software in a particular
domain but is intended to be more software infrastructure oriented.


From Chris.Barker at noaa.gov  Fri Apr 29 11:41:14 2011
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 29 Apr 2011 08:41:14 -0700
Subject: [Numpy-discussion] Array of size 'n' with common difference < 1
In-Reply-To: <4DBA6969.1030107@gmail.com>
References: <BANLkTinw=qYoQ-4iOHuqZ9AhGCw1P5X8yw@mail.gmail.com>
	<4DBA6969.1030107@gmail.com>
Message-ID: <4DBADC1A.4060502@noaa.gov>

On 4/29/11 12:31 AM, pratik wrote:
> On Friday 29 April 2011 12:56 PM, dileep kunjaai wrote:
>> Dear sir,
>> I am trying to make an array of varies from -60 to 90 with difference
>> 0.25. I tried the following command ...

>> >>import numpy as N
>> lat=N.array(xrange(-6000, 9000, 25), dtype=float)
>> print lat/100

xrange() (or range(), or np.arange()) is almost never the right solution 
for floating point ranges, due to the intricacies of floating point 
precision.

> lat =numpy.mgrid[-60:90:.25]

or np.linspace:

np.linspace(-60,90,((60.+90.)*4. + 1))

((60.+90.)*4. + 1) is the number of points you want -- the +1 because 
you want both end points.

mgrid is usually used for 2-d (or higher) grids, though it looks like it 
makes sense for this use, too, though note that it doesn't give you both 
endpoints in this case. From the docs:

"""If the step length is not a
     complex number, then the stop is not inclusive.
"""

and an example:

In [15]: np.mgrid[-1:3:.25]
Out[15]:
array([-1.  , -0.75, -0.5 , -0.25,  0.  ,  0.25,  0.5 ,  0.75,  1.  ,
         1.25,  1.5 ,  1.75,  2.  ,  2.25,  2.5 ,  2.75])

I think this is too bad, actually, because we're back to range()-type 
tricks to get the end point:

In [20]: np.mgrid[-1:3.25:.25]
Out[20]:
array([-1.  , -0.75, -0.5 , -0.25,  0.  ,  0.25,  0.5 ,  0.75,  1.  ,
         1.25,  1.5 ,  1.75,  2.  ,  2.25,  2.5 ,  2.75,  3.  ])


-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From dileepkunjaai at gmail.com  Fri Apr 29 11:55:47 2011
From: dileepkunjaai at gmail.com (dileep kunjaai)
Date: Fri, 29 Apr 2011 21:25:47 +0530
Subject: [Numpy-discussion] Array of size 'n' with common difference < 1
In-Reply-To: <4DBADC1A.4060502@noaa.gov>
References: <BANLkTinw=qYoQ-4iOHuqZ9AhGCw1P5X8yw@mail.gmail.com>
	<4DBA6969.1030107@gmail.com> <4DBADC1A.4060502@noaa.gov>
Message-ID: <BANLkTikZ7uH4Ba1YKFx=3JAdwuXMDxz-XQ@mail.gmail.com>

Thank you sir: thank you....very much for ur time and
consideration..................

On Fri, Apr 29, 2011 at 9:11 PM, Christopher Barker
<Chris.Barker at noaa.gov>wrote:

> On 4/29/11 12:31 AM, pratik wrote:
> > On Friday 29 April 2011 12:56 PM, dileep kunjaai wrote:
> >> Dear sir,
> >> I am trying to make an array of varies from -60 to 90 with difference
> >> 0.25. I tried the following command ...
>
> >> >>import numpy as N
> >> lat=N.array(xrange(-6000, 9000, 25), dtype=float)
> >> print lat/100
>
> xrange() (or range(), or np.arange()) is almost never the right solution
> for floating point ranges, due to the intricacies of floating point
> precision.
>
> > lat =numpy.mgrid[-60:90:.25]
>
> or np.linspace:
>
> np.linspace(-60,90,((60.+90.)*4. + 1))
>
> ((60.+90.)*4. + 1) is the number of points you want -- the +1 because
> you want both end points.
>
> mgrid is usually used for 2-d (or higher) grids, though it looks like it
> makes sense for this use, too, though note that it doesn't give you both
> endpoints in this case. From the docs:
>
> """If the step length is not a
>     complex number, then the stop is not inclusive.
> """
>
> and an example:
>
> In [15]: np.mgrid[-1:3:.25]
> Out[15]:
> array([-1.  , -0.75, -0.5 , -0.25,  0.  ,  0.25,  0.5 ,  0.75,  1.  ,
>         1.25,  1.5 ,  1.75,  2.  ,  2.25,  2.5 ,  2.75])
>
> I think this is too bad, actually, because we're back to range()-type
> tricks to get the end point:
>
> In [20]: np.mgrid[-1:3.25:.25]
> Out[20]:
> array([-1.  , -0.75, -0.5 , -0.25,  0.  ,  0.25,  0.5 ,  0.75,  1.  ,
>         1.25,  1.5 ,  1.75,  2.  ,  2.25,  2.5 ,  2.75,  3.  ])
>
>
> -Chris
>
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
DILEEPKUMAR. R
J R F, IIT DELHI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110429/36fc0a95/attachment.html>

From halbert at halwitz.org  Fri Apr 29 13:06:43 2011
From: halbert at halwitz.org (Dan Halbert)
Date: Fri, 29 Apr 2011 13:06:43 -0400 (EDT)
Subject: [Numpy-discussion] numpy.load truncates read from network file
	on XP
In-Reply-To: <1304025773.276522670@beta.apps.rackspace.com>
References: <1304025773.276522670@beta.apps.rackspace.com>
Message-ID: <1304096803.556420400@beta.apps.rackspace.com>

On Thursday, April 28, 2011 5:22pm, "Dan Halbert" <halbert at halwitz.org> said:
> I'm having trouble loading a large remote .npy file on Windows XP. This is on
> numpy-1.3.0 on Windows XP SP3:
> 
>     numpy.load(r'\\myserver\mydir\big.npy')
> 
> will fail with this sort of error being printed:
>     "14328000 items requested but only 54 read"

I have tracked this down: it's not specific to numpy, or even Python, though
there's a Python bug report about it:
"size limit exceeded for read() from network drive"
http://bugs.python.org/issue1478529

The problem stems from a buffer size limit of not quite 64MB for the Windows API
ReadFile()function. ReadFile() is used (maybe indirectly) by the Microsoft
fread() implementation, which is in turn used by Python file.read().

Apparently most of the time the underlying Windows code is cognizant of this limit,
but under certain circumstances, a read from a large networked file is not handled
correctly (I think when there is also an entry in "My Network Places" for the path).
It's quite obscure, and there's nothing that can be fixed at the numpy level.

Dan




From pierre.raybaut at gmail.com  Fri Apr 29 13:07:23 2011
From: pierre.raybaut at gmail.com (Pierre Raybaut)
Date: Fri, 29 Apr 2011 19:07:23 +0200
Subject: [Numpy-discussion] ANN: Spyder v2.0.11
Message-ID: <09317DC5AFAD46FE8AA719F390F4C918@asus>

Hi all,

I am pleased to announced that Spyder v2.0.11 has just been released.
As this is mostly a maintenance release, a lot of bugs were fixed and some
minor features were added (see below).

Embedding an interactive Python console into your GUI-based application
(Qt):
This version includes an example of application using the Spyder's internal
shell as a debugging console which also demonstrates the py2exe deployment
procedure.

Spyder (previously known as Pydee) is a free open-source Python development
environment providing MATLAB-like features in a simple and light-weighted
software, available for Windows XP/Vista/7, GNU/Linux and MacOS X:
http://spyderlib.googlecode.com/
Spyder is part of spyderlib, a Python module based on PyQt4, pyflakes and
rope (QScintilla's dependency has been removed in version 2.0 and
rope features have been integrated since this version as well).

Some of Spyder basic features:
    * Python, C/C++, Fortran source editor with class/function browser, code
completion and calltips
    * consoles:
          o open as many Python interpreters, IPython consoles or command
windows as you need
          o code completion and calltips
          o variable explorer with GUI-based editors for a lot of data types
(numbers, strings, lists, arrays, dictionaries, ...)
    * object inspector: provide documentation or source code on any Python
object (class, function, module, ...)
    * online documentation: automatically generated html documentation on
installed Python modules
    * find in files
    * file explorer
    * project manager
    * MATLAB-like PYTHONPATH management dialog box (works with all consoles)
    * Windows only: current user environment variables editor
    * direct links to documentation (Python, Qt, Matplotlib, NumPy, Scipy,
etc.)
    * direct link to Python(x,y) launcher
    * direct links to QtDesigner, QtLinguist and QtAssistant (Qt
documentation)


Bug fixes (since v2.0.9):
(Fixes Issue 616) Pylint plugin: tree widget header text was not updated
when analyizing a new script (the last analyzed script name was still shown)
Editor/completion widget/bugfix: pressing shift was hiding the completion
combo box
(Fixes Issue 630) Added missing default settings for "Spyder light" (only
necessary when installing from scratch and without any remaining .spyder.ini
file)
Editor/Console-bugfix: info tooltips (calltips) were hidden right after
being shown (i.e. when typing any character after the left parenthesis)
(Fixes Issue 631) Drag and drop of files into editor on Linux was pasting
path instead of opening the file
(Fixes Issue 640) Editor: block comment was not working correctly at end of
file
Code completion widget (Editor/Console) - bugfix: parenting to the ancestor
widget was necessary on Linux
(Fixes Issue 546) [Contributor: Alex Fargus] C/Cpp syntax highlighting
bugfix
(Fixes Issue 646) IPython integration: fixed pyreadline monkey-patch for
pyreadline v1.7

Enhancements (since v2.0.9):
File explorer widget/plugin: improved performances (widget is now populated
in a separate thread)
Spyder crash dialog: warning the user about the '--reset' option (this will
remove all configuration files)

Cheers,
Pierre




From seb.haase at gmail.com  Fri Apr 29 16:27:06 2011
From: seb.haase at gmail.com (Sebastian Haase)
Date: Fri, 29 Apr 2011 22:27:06 +0200
Subject: [Numpy-discussion] Array of size 'n' with common difference < 1
In-Reply-To: <4DBADC1A.4060502@noaa.gov>
References: <BANLkTinw=qYoQ-4iOHuqZ9AhGCw1P5X8yw@mail.gmail.com>
	<4DBA6969.1030107@gmail.com> <4DBADC1A.4060502@noaa.gov>
Message-ID: <BANLkTikJdLH-Bqr=ra4rKnO8WuBXMaHY0Q@mail.gmail.com>

On Fri, Apr 29, 2011 at 5:41 PM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
> On 4/29/11 12:31 AM, pratik wrote:
>> On Friday 29 April 2011 12:56 PM, dileep kunjaai wrote:
>>> Dear sir,
>>> I am trying to make an array of varies from -60 to 90 with difference
>>> 0.25. I tried the following command ...
>
>>> >>import numpy as N
>>> lat=N.array(xrange(-6000, 9000, 25), dtype=float)
>>> print lat/100
>
> xrange() (or range(), or np.arange()) is almost never the right solution
> for floating point ranges, due to the intricacies of floating point
> precision.
>
>> lat =numpy.mgrid[-60:90:.25]
>
> or np.linspace:
>
> np.linspace(-60,90,((60.+90.)*4. + 1))
>
> ((60.+90.)*4. + 1) is the number of points you want -- the +1 because
> you want both end points.
>
> mgrid is usually used for 2-d (or higher) grids, though it looks like it
> makes sense for this use, too, though note that it doesn't give you both
> endpoints in this case. From the docs:
>
> """If the step length is not a
> ? ? complex number, then the stop is not inclusive.
> """
>
> and an example:
>
> In [15]: np.mgrid[-1:3:.25]
> Out[15]:
> array([-1. ?, -0.75, -0.5 , -0.25, ?0. ?, ?0.25, ?0.5 , ?0.75, ?1. ?,
> ? ? ? ? 1.25, ?1.5 , ?1.75, ?2. ?, ?2.25, ?2.5 , ?2.75])
>
> I think this is too bad, actually, because we're back to range()-type
> tricks to get the end point:
>
> In [20]: np.mgrid[-1:3.25:.25]
> Out[20]:
> array([-1. ?, -0.75, -0.5 , -0.25, ?0. ?, ?0.25, ?0.5 , ?0.75, ?1. ?,
> ? ? ? ? 1.25, ?1.5 , ?1.75, ?2. ?, ?2.25, ?2.5 , ?2.75, ?3. ?])
>
>
Just for completeness, note this paragraph from the mgrid docs:

However, if the step length is a *complex number* (e.g. 5j), then the
integer part of its magnitude is interpreted as specifying the number
of points to create between the start and stop values, where the stop
value *is inclusive*.

-Sebastian Haase


From Chris.Barker at noaa.gov  Fri Apr 29 17:15:34 2011
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 29 Apr 2011 14:15:34 -0700
Subject: [Numpy-discussion] Array of size 'n' with common difference < 1
In-Reply-To: <BANLkTikJdLH-Bqr=ra4rKnO8WuBXMaHY0Q@mail.gmail.com>
References: <BANLkTinw=qYoQ-4iOHuqZ9AhGCw1P5X8yw@mail.gmail.com>
	<4DBA6969.1030107@gmail.com> <4DBADC1A.4060502@noaa.gov>
	<BANLkTikJdLH-Bqr=ra4rKnO8WuBXMaHY0Q@mail.gmail.com>
Message-ID: <4DBB2A76.8070306@noaa.gov>

On 4/29/11 1:27 PM, Sebastian Haase wrote:

> Just for completeness, note this paragraph from the mgrid docs:
>
> However, if the step length is a *complex number* (e.g. 5j), then the
> integer part of its magnitude is interpreted as specifying the number
> of points to create between the start and stop values, where the stop
> value *is inclusive*.

OK -- for a kluge, I figure you could do complex, then take the real 
part, but I can't seem to get a complex grid to work at all:

In [37]: np.mgrid[(0+0j):(1+0j):(0.1+0j)]
Out[37]: array([], dtype=complex128)

What am I doing wrong?

Go it -- the docs could use some clarification here -- Actually, passing 
in a complex nubmer is a FLAG, indicating that the index means somethign 
different, but it's still doing everything with real numbers (floats). 
Wow, klunky API!

but here is what the OP would want:

np.mgrid[-60:90:((60.j+90.j)*4. + 1j)]

which is klunkier that linspace.

I'd have used two different function names for the different mgrid 
functionality, rather that than the complex kludge

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From dyssident at gmail.com  Fri Apr 29 22:56:22 2011
From: dyssident at gmail.com (Alan Gibson)
Date: Fri, 29 Apr 2011 21:56:22 -0500
Subject: [Numpy-discussion] Less dimensions than expected with record array
Message-ID: <BANLkTinrs7pJMu0E4oo_7bZNqq3J5FXstw@mail.gmail.com>

Hello all,

This question may seem elementary (mostly because it is), but I can't
find documentation anywhere as to why the following are true:

>>> import numpy as np
>>> data = [(1,2,3),(4,5,6),(7,8,9)]
>>> dt = [('a',int),('b',int),('c',int)]
>>> normal_array = np.array(data)
>>> record_array = np.array(data, dtype=dt)
>>> print "ndarray has shape %s but record array has shape %s" % \
...     (normal_array.shape, record_array.shape)
ndarray has shape (3, 3) but record array has shape (3,)
>>> print "ndarray has %s dimensions but record array has %s dimensions" % \
...     (normal_array.ndim, record_array.ndim)
ndarray has 2 dimensions but record array has 1 dimensions

This makes seemingly reasonable things, like using apply_along_axis()
over a table of data with named columns, impossible:

>>> np.apply_along_axis(record_array, 1, lambda x: x)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/local/lib/python2.6/dist-packages/numpy/lib/shape_base.py",
line 72, in apply_along_axis
   % (axis,nd))
ValueError: axis must be less than arr.ndim; axis=1, rank=0.

What's the reason for this behavior? Is there a way to make such
operations work with record arrays?

Thanks,

Alan


From josef.pktd at gmail.com  Fri Apr 29 23:12:19 2011
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 29 Apr 2011 23:12:19 -0400
Subject: [Numpy-discussion] Less dimensions than expected with record
	array
In-Reply-To: <BANLkTinrs7pJMu0E4oo_7bZNqq3J5FXstw@mail.gmail.com>
References: <BANLkTinrs7pJMu0E4oo_7bZNqq3J5FXstw@mail.gmail.com>
Message-ID: <BANLkTimRhLt6p_EZWAOeRbBmgswTtMMW3w@mail.gmail.com>

On Fri, Apr 29, 2011 at 10:56 PM, Alan Gibson <dyssident at gmail.com> wrote:
> Hello all,
>
> This question may seem elementary (mostly because it is), but I can't
> find documentation anywhere as to why the following are true:
>
>>>> import numpy as np
>>>> data = [(1,2,3),(4,5,6),(7,8,9)]
>>>> dt = [('a',int),('b',int),('c',int)]
>>>> normal_array = np.array(data)
>>>> record_array = np.array(data, dtype=dt)
>>>> print "ndarray has shape %s but record array has shape %s" % \
> ... ? ? (normal_array.shape, record_array.shape)
> ndarray has shape (3, 3) but record array has shape (3,)
>>>> print "ndarray has %s dimensions but record array has %s dimensions" % \
> ... ? ? (normal_array.ndim, record_array.ndim)
> ndarray has 2 dimensions but record array has 1 dimensions
>
> This makes seemingly reasonable things, like using apply_along_axis()
> over a table of data with named columns, impossible:
>
>>>> np.apply_along_axis(record_array, 1, lambda x: x)
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> ?File "/usr/local/lib/python2.6/dist-packages/numpy/lib/shape_base.py",
> line 72, in apply_along_axis
> ? % (axis,nd))
> ValueError: axis must be less than arr.ndim; axis=1, rank=0.
>
> What's the reason for this behavior? Is there a way to make such
> operations work with record arrays?

each row (record) is treated as one array element, so the structured
array is only 1d.

If you have rows/records with content that is not homogenous, then
working along axis=1 (across elements of a record) doesn't make sense.
for example I just struggle with 2 datetime columns and the rest are
integers.

If you want an array with homogenous elements (all floats or all ints)
with operations along axis, then larry (la) is, I think, still the
best bet. I don't know what the status with the dataarray for numpy
is.

Josef

>
> Thanks,
>
> Alan
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>


From tonthon21 at gmail.com  Sat Apr 30 02:54:01 2011
From: tonthon21 at gmail.com (tonthon)
Date: Sat, 30 Apr 2011 08:54:01 +0200
Subject: [Numpy-discussion] numpy histogram : Maybe a little error in the
	tutorial
Message-ID: <4DBBB209.8040806@gmail.com>

Hi,

I was followong the tutorial about numpy's basics and found a little 
error here : 
http://www.scipy.org/Tentative_NumPy_Tutorial#head-aa75ec76530ff51a2e98071adb7224a4b793519e

(n, bins) = numpy.histogram(v, bins=50, normed=1)  # NumPy version (no plot)
should be :
(n, bins) = numpy.histogram(v, bins=50, normed=True)

since in the 'new' version of histogram, the normed var is explicitly 
tested as a boolean (in numpy/lib/function_base.py line 380):
if normed is False:
...
elif normed is True:
...

Cheers

Gaston


From ralf.gommers at googlemail.com  Sat Apr 30 05:21:22 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sat, 30 Apr 2011 11:21:22 +0200
Subject: [Numpy-discussion] Numpy steering group?
In-Reply-To: <BANLkTin1ERp4bxnok6GsT6eH6VzM6r-NDA@mail.gmail.com>
References: <BANLkTin1ERp4bxnok6GsT6eH6VzM6r-NDA@mail.gmail.com>
Message-ID: <BANLkTikP2r1ahrM4LCC+ePQBsjPSoihPgQ@mail.gmail.com>

On Wed, Apr 27, 2011 at 8:52 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> This is just to follow up on a dead thread of mine a little while back.
>
> I was asking about letters for Clint Whaley's tenure case, from numpy,
> but I realized that I don't know who 'numpy' is :)
>
> Is there in fact a numpy steering group? ? ?Who is best to write
> letters representing the 'numpy community'?

At http://scipy.org/Developer_Zone there's a list of people under a
big header "steering committee". It seems to me that writing such a
letter representing the community is one of the purposes that
committee could serve.

Ralf


From ralf.gommers at googlemail.com  Sat Apr 30 12:52:34 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sat, 30 Apr 2011 18:52:34 +0200
Subject: [Numpy-discussion] numpy histogram : Maybe a little error in
	the tutorial
In-Reply-To: <4DBBB209.8040806@gmail.com>
References: <4DBBB209.8040806@gmail.com>
Message-ID: <BANLkTi=DhVupPOPjECcWWVvguzcjQraJ3g@mail.gmail.com>

On Sat, Apr 30, 2011 at 8:54 AM, tonthon <tonthon21 at gmail.com> wrote:
> Hi,
>
> I was followong the tutorial about numpy's basics and found a little
> error here :
> http://www.scipy.org/Tentative_NumPy_Tutorial#head-aa75ec76530ff51a2e98071adb7224a4b793519e
>
> (n, bins) = numpy.histogram(v, bins=50, normed=1) ?# NumPy version (no plot)
> should be :
> (n, bins) = numpy.histogram(v, bins=50, normed=True)

Thanks for reporting, this is changed now.
>
> since in the 'new' version of histogram, the normed var is explicitly
> tested as a boolean (in numpy/lib/function_base.py line 380):
> if normed is False:
> ...
> elif normed is True:

This is bad style, but it was already changed in master.

Cheers,
Ralf


From ralf.gommers at googlemail.com  Sat Apr 30 16:19:50 2011
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sat, 30 Apr 2011 22:19:50 +0200
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1
Message-ID: <BANLkTik8XL_nsvRfegAWJvG1Sq4tGY8ioQ@mail.gmail.com>

Hi,

I am pleased to announce the availability of the first release
candidate of NumPy 1.6.0. If no new problems are reported, the final
release will be in one week.

Sources and binaries can be found at
http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc1/
For (preliminary) release notes see below.

Enjoy,
Ralf


=========================
NumPy 1.6.0 Release Notes
=========================

This release includes several new features as well as numerous bug fixes and
improved documentation.  It is backward compatible with the 1.5.0 release, and
supports Python 2.4 - 2.7 and 3.1 - 3.2.


Highlights
==========

* Re-introduction of datetime dtype support to deal with dates in arrays.

* A new 16-bit floating point type.

* A new iterator, which improves performance of many functions.


New features
============

New 16-bit floating point type
------------------------------

This release adds support for the IEEE 754-2008 binary16 format, available as
the data type ``numpy.half``.  Within Python, the type behaves similarly to
`float` or `double`, and C extensions can add support for it with the exposed
half-float API.


New iterator
------------

A new iterator has been added, replacing the functionality of the
existing iterator and multi-iterator with a single object and API.
This iterator works well with general memory layouts different from
C or Fortran contiguous, and handles both standard NumPy and
customized broadcasting. The buffering, automatic data type
conversion, and optional output parameters, offered by
ufuncs but difficult to replicate elsewhere, are now exposed by this
iterator.


Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial``
-------------------------------------------------------------------------

Extend the number of polynomials available in the polynomial package. In
addition, a new ``window`` attribute has been added to the classes in
order to specify the range the ``domain`` maps to. This is mostly useful
for the Laguerre, Hermite, and HermiteE polynomials whose natural domains
are infinite and provides a more intuitive way to get the correct mapping
of values without playing unnatural tricks with the domain.


Fortran assumed shape array and size function support in ``numpy.f2py``
-----------------------------------------------------------------------

F2py now supports wrapping Fortran 90 routines that use assumed shape
arrays.  Before such routines could be called from Python but the
corresponding Fortran routines received assumed shape arrays as zero
length arrays which caused unpredicted results. Thanks to Lorenz
H?depohl for pointing out the correct way to interface routines with
assumed shape arrays.

In addition, f2py interprets Fortran expression ``size(array, dim)``
as ``shape(array, dim-1)`` which makes it possible to automatically
wrap Fortran routines that use two argument ``size`` function in
dimension specifications. Before users were forced to apply this
mapping manually.


Other new functions
-------------------

``numpy.ravel_multi_index`` : Converts a multi-index tuple into
an array of flat indices, applying boundary modes to the indices.

``numpy.einsum`` : Evaluate the Einstein summation convention.  Using the
Einstein summation convention, many common multi-dimensional array operations
can be represented in a simple fashion.  This function provides a way compute
such summations.

``numpy.count_nonzero`` : Counts the number of non-zero elements in an array.

``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose
the underlying type promotion used by the ufuncs and other operations to
determine the types of outputs. These improve upon the ``numpy.common_type``
and ``numpy.mintypecode`` which provide similar functionality but do
not match the ufunc implementation.


Changes
=======

Changes and improvements in the numpy core
------------------------------------------

``default error handling``
--------------------------

The default error handling has been change from ``print`` to ``warn`` for
all except for ``underflow``, which remains as ``ignore``.


``numpy.distutils``
-------------------

Several new compilers are supported for building Numpy: the Portland Group
Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C
compiler on Linux.


``numpy.testing``
-----------------

The testing framework gained ``numpy.testing.assert_allclose``, which provides
a more convenient way to compare floating point arrays than
`assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`.


``C API``
---------

In addition to the APIs for the new iterator and half data type, a number
of other additions have been made to the C API. The type promotion
mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``,
``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration
``NPY_CASTING`` has been added which controls what types of casts are
permitted. This is used by the new functions ``PyArray_CanCastArrayTo``
and ``PyArray_CanCastTypeTo``.  A more flexible way to handle
conversion of arbitrary python objects into arrays is exposed by
``PyArray_GetArrayParamsFromObject``.


Deprecated features
===================

The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality
will be replaced by the new "density" keyword.


Removed features
================

``numpy.fft``
-------------

The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`,
which were aliases for the same functions without the 'e' in the name, were
removed.


``numpy.memmap``
----------------

The `sync()` and `close()` methods of memmap were removed.  Use `flush()` and
"del memmap" instead.


``numpy.lib``
-------------

The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``,
``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed.


``numpy.ma``
------------

Several deprecated items were removed from the ``numpy.ma`` module::

  * ``numpy.ma.MaskedArray`` "raw_data" method
  * ``numpy.ma.MaskedArray`` constructor "flag" keyword
  * ``numpy.ma.make_mask`` "flag" keyword
  * ``numpy.ma.allclose`` "fill_value" keyword


``numpy.distutils``
-------------------

The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include``
instead.


From cgohlke at uci.edu  Sat Apr 30 19:12:33 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Sat, 30 Apr 2011 16:12:33 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1
In-Reply-To: <BANLkTik8XL_nsvRfegAWJvG1Sq4tGY8ioQ@mail.gmail.com>
References: <BANLkTik8XL_nsvRfegAWJvG1Sq4tGY8ioQ@mail.gmail.com>
Message-ID: <4DBC9761.9070908@uci.edu>



On 4/30/2011 1:19 PM, Ralf Gommers wrote:
> Hi,
>
> I am pleased to announce the availability of the first release
> candidate of NumPy 1.6.0. If no new problems are reported, the final
> release will be in one week.
>
> Sources and binaries can be found at
> http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc1/
> For (preliminary) release notes see below.
>
> Enjoy,
> Ralf
>
>
> =========================
> NumPy 1.6.0 Release Notes
> =========================
>
> This release includes several new features as well as numerous bug fixes and
> improved documentation.  It is backward compatible with the 1.5.0 release, and
> supports Python 2.4 - 2.7 and 3.1 - 3.2.
>
>
> Highlights
> ==========
>
> * Re-introduction of datetime dtype support to deal with dates in arrays.
>
> * A new 16-bit floating point type.
>
> * A new iterator, which improves performance of many functions.
>
>
> New features
> ============
>
> New 16-bit floating point type
> ------------------------------
>
> This release adds support for the IEEE 754-2008 binary16 format, available as
> the data type ``numpy.half``.  Within Python, the type behaves similarly to
> `float` or `double`, and C extensions can add support for it with the exposed
> half-float API.
>
>
> New iterator
> ------------
>
> A new iterator has been added, replacing the functionality of the
> existing iterator and multi-iterator with a single object and API.
> This iterator works well with general memory layouts different from
> C or Fortran contiguous, and handles both standard NumPy and
> customized broadcasting. The buffering, automatic data type
> conversion, and optional output parameters, offered by
> ufuncs but difficult to replicate elsewhere, are now exposed by this
> iterator.
>
>
> Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial``
> -------------------------------------------------------------------------
>
> Extend the number of polynomials available in the polynomial package. In
> addition, a new ``window`` attribute has been added to the classes in
> order to specify the range the ``domain`` maps to. This is mostly useful
> for the Laguerre, Hermite, and HermiteE polynomials whose natural domains
> are infinite and provides a more intuitive way to get the correct mapping
> of values without playing unnatural tricks with the domain.
>
>
> Fortran assumed shape array and size function support in ``numpy.f2py``
> -----------------------------------------------------------------------
>
> F2py now supports wrapping Fortran 90 routines that use assumed shape
> arrays.  Before such routines could be called from Python but the
> corresponding Fortran routines received assumed shape arrays as zero
> length arrays which caused unpredicted results. Thanks to Lorenz
> H?depohl for pointing out the correct way to interface routines with
> assumed shape arrays.
>
> In addition, f2py interprets Fortran expression ``size(array, dim)``
> as ``shape(array, dim-1)`` which makes it possible to automatically
> wrap Fortran routines that use two argument ``size`` function in
> dimension specifications. Before users were forced to apply this
> mapping manually.
>
>
> Other new functions
> -------------------
>
> ``numpy.ravel_multi_index`` : Converts a multi-index tuple into
> an array of flat indices, applying boundary modes to the indices.
>
> ``numpy.einsum`` : Evaluate the Einstein summation convention.  Using the
> Einstein summation convention, many common multi-dimensional array operations
> can be represented in a simple fashion.  This function provides a way compute
> such summations.
>
> ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array.
>
> ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose
> the underlying type promotion used by the ufuncs and other operations to
> determine the types of outputs. These improve upon the ``numpy.common_type``
> and ``numpy.mintypecode`` which provide similar functionality but do
> not match the ufunc implementation.
>
>
> Changes
> =======
>
> Changes and improvements in the numpy core
> ------------------------------------------
>
> ``default error handling``
> --------------------------
>
> The default error handling has been change from ``print`` to ``warn`` for
> all except for ``underflow``, which remains as ``ignore``.
>
>
> ``numpy.distutils``
> -------------------
>
> Several new compilers are supported for building Numpy: the Portland Group
> Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C
> compiler on Linux.
>
>
> ``numpy.testing``
> -----------------
>
> The testing framework gained ``numpy.testing.assert_allclose``, which provides
> a more convenient way to compare floating point arrays than
> `assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`.
>
>
> ``C API``
> ---------
>
> In addition to the APIs for the new iterator and half data type, a number
> of other additions have been made to the C API. The type promotion
> mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``,
> ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration
> ``NPY_CASTING`` has been added which controls what types of casts are
> permitted. This is used by the new functions ``PyArray_CanCastArrayTo``
> and ``PyArray_CanCastTypeTo``.  A more flexible way to handle
> conversion of arbitrary python objects into arrays is exposed by
> ``PyArray_GetArrayParamsFromObject``.
>
>
> Deprecated features
> ===================
>
> The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality
> will be replaced by the new "density" keyword.
>
>
> Removed features
> ================
>
> ``numpy.fft``
> -------------
>
> The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`,
> which were aliases for the same functions without the 'e' in the name, were
> removed.
>
>
> ``numpy.memmap``
> ----------------
>
> The `sync()` and `close()` methods of memmap were removed.  Use `flush()` and
> "del memmap" instead.
>
>
> ``numpy.lib``
> -------------
>
> The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``,
> ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed.
>
>
> ``numpy.ma``
> ------------
>
> Several deprecated items were removed from the ``numpy.ma`` module::
>
>    * ``numpy.ma.MaskedArray`` "raw_data" method
>    * ``numpy.ma.MaskedArray`` constructor "flag" keyword
>    * ``numpy.ma.make_mask`` "flag" keyword
>    * ``numpy.ma.allclose`` "fill_value" keyword
>
>
> ``numpy.distutils``
> -------------------
>
> The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include``
> instead.
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>

I get a null pointer access violation during numpy.test() with all 
msvc9/MKL builds for win32 (not win-amd64). The crash occurs during 
test_result_type() in test_numeric.py and can be reduced to the 
following code:

>>> import numpy as np
>>> np.result_type(np.array([np.float32(0)]), np.complex128(0))

np.float64(0) and np.float16(0) also crash. Unfortunately the debug 
builds do not crash.

Christoph




From charlesr.harris at gmail.com  Sat Apr 30 19:53:14 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 30 Apr 2011 17:53:14 -0600
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1
In-Reply-To: <4DBC9761.9070908@uci.edu>
References: <BANLkTik8XL_nsvRfegAWJvG1Sq4tGY8ioQ@mail.gmail.com>
	<4DBC9761.9070908@uci.edu>
Message-ID: <BANLkTinKSxeUhFEDbMayb1+UNjHaNJP3tg@mail.gmail.com>

<snip>


> I get a null pointer access violation during numpy.test() with all
> msvc9/MKL builds for win32 (not win-amd64). The crash occurs during
> test_result_type() in test_numeric.py and can be reduced to the
> following code:
>
> >>> import numpy as np
> >>> np.result_type(np.array([np.float32(0)]), np.complex128(0))
>
> np.float64(0) and np.float16(0) also crash. Unfortunately the debug
> builds do not crash.
>
>
This is new, right?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110430/86a55ae9/attachment.html>

From charlesr.harris at gmail.com  Sat Apr 30 19:58:48 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 30 Apr 2011 17:58:48 -0600
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1
In-Reply-To: <BANLkTinKSxeUhFEDbMayb1+UNjHaNJP3tg@mail.gmail.com>
References: <BANLkTik8XL_nsvRfegAWJvG1Sq4tGY8ioQ@mail.gmail.com>
	<4DBC9761.9070908@uci.edu>
	<BANLkTinKSxeUhFEDbMayb1+UNjHaNJP3tg@mail.gmail.com>
Message-ID: <BANLkTi=cukm8v4qqA1r27zow76aHw9kGcg@mail.gmail.com>

On Sat, Apr 30, 2011 at 5:53 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

> <snip>
>
>
>> I get a null pointer access violation during numpy.test() with all
>> msvc9/MKL builds for win32 (not win-amd64). The crash occurs during
>> test_result_type() in test_numeric.py and can be reduced to the
>> following code:
>>
>> >>> import numpy as np
>> >>> np.result_type(np.array([np.float32(0)]), np.complex128(0))
>>
>> np.float64(0) and np.float16(0) also crash. Unfortunately the debug
>> builds do not crash.
>>
>>
> This is new, right?
>
>
Does it depend on the optimization level?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110430/5027d958/attachment.html>

From cgohlke at uci.edu  Sat Apr 30 20:50:13 2011
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Sat, 30 Apr 2011 17:50:13 -0700
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1
In-Reply-To: <BANLkTi=cukm8v4qqA1r27zow76aHw9kGcg@mail.gmail.com>
References: <BANLkTik8XL_nsvRfegAWJvG1Sq4tGY8ioQ@mail.gmail.com>	<4DBC9761.9070908@uci.edu>	<BANLkTinKSxeUhFEDbMayb1+UNjHaNJP3tg@mail.gmail.com>
	<BANLkTi=cukm8v4qqA1r27zow76aHw9kGcg@mail.gmail.com>
Message-ID: <4DBCAE45.8080402@uci.edu>



On 4/30/2011 4:58 PM, Charles R Harris wrote:
>
>
> On Sat, Apr 30, 2011 at 5:53 PM, Charles R Harris
> <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
>
>     <snip>
>
>
>         I get a null pointer access violation during numpy.test() with all
>         msvc9/MKL builds for win32 (not win-amd64). The crash occurs during
>         test_result_type() in test_numeric.py and can be reduced to the
>         following code:
>
>         > >> import numpy as np
>         > >> np.result_type(np.array([np.float32(0)]), np.complex128(0))
>
>         np.float64(0) and np.float16(0) also crash. Unfortunately the debug
>         builds do not crash.
>
>     This is new, right?
>
>
> Does it depend on the optimization level?
>
> Chuck
>

Yes it's new. The pure msvc9 builds without MKL also crash. The crash 
disapperars When compiling with /Od (disable optimization) instead of 
/Ox (maximum optimization; the default for distutils).

Christoph


From charlesr.harris at gmail.com  Sat Apr 30 21:37:34 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 30 Apr 2011 19:37:34 -0600
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1
In-Reply-To: <4DBCAE45.8080402@uci.edu>
References: <BANLkTik8XL_nsvRfegAWJvG1Sq4tGY8ioQ@mail.gmail.com>
	<4DBC9761.9070908@uci.edu>
	<BANLkTinKSxeUhFEDbMayb1+UNjHaNJP3tg@mail.gmail.com>
	<BANLkTi=cukm8v4qqA1r27zow76aHw9kGcg@mail.gmail.com>
	<4DBCAE45.8080402@uci.edu>
Message-ID: <BANLkTinLqr4R=kJvjpHDhQf-mMySc74k+A@mail.gmail.com>

On Sat, Apr 30, 2011 at 6:50 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:

>
>
> On 4/30/2011 4:58 PM, Charles R Harris wrote:
> >
> >
> > On Sat, Apr 30, 2011 at 5:53 PM, Charles R Harris
> > <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
> >
> >     <snip>
> >
> >
> >         I get a null pointer access violation during numpy.test() with
> all
> >         msvc9/MKL builds for win32 (not win-amd64). The crash occurs
> during
> >         test_result_type() in test_numeric.py and can be reduced to the
> >         following code:
> >
> >         > >> import numpy as np
> >         > >> np.result_type(np.array([np.float32(0)]), np.complex128(0))
> >
> >         np.float64(0) and np.float16(0) also crash. Unfortunately the
> debug
> >         builds do not crash.
> >
> >     This is new, right?
> >
> >
> > Does it depend on the optimization level?
> >
> > Chuck
> >
>
> Yes it's new. The pure msvc9 builds without MKL also crash. The crash
> disapperars When compiling with /Od (disable optimization) instead of
> /Ox (maximum optimization; the default for distutils).
>
>
So all of np.float16(0), np.float32(0), np.float64(0), etc crash? Does it
depend at all on 0 as the argument, or is it the same for 1, 0.0, etc. What
about string arguments like np.float64("0"). I want to pin the location down
a bit more. Too bad it doesn't crash in the debugger.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110430/a0291b4c/attachment.html>

From charlesr.harris at gmail.com  Sat Apr 30 21:55:44 2011
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 30 Apr 2011 19:55:44 -0600
Subject: [Numpy-discussion] ANN: Numpy 1.6.0 release candidate 1
In-Reply-To: <BANLkTinLqr4R=kJvjpHDhQf-mMySc74k+A@mail.gmail.com>
References: <BANLkTik8XL_nsvRfegAWJvG1Sq4tGY8ioQ@mail.gmail.com>
	<4DBC9761.9070908@uci.edu>
	<BANLkTinKSxeUhFEDbMayb1+UNjHaNJP3tg@mail.gmail.com>
	<BANLkTi=cukm8v4qqA1r27zow76aHw9kGcg@mail.gmail.com>
	<4DBCAE45.8080402@uci.edu>
	<BANLkTinLqr4R=kJvjpHDhQf-mMySc74k+A@mail.gmail.com>
Message-ID: <BANLkTim5Tq4rRPxFzwfqoQBD-M+H-e5hZQ@mail.gmail.com>

On Sat, Apr 30, 2011 at 7:37 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Sat, Apr 30, 2011 at 6:50 PM, Christoph Gohlke <cgohlke at uci.edu> wrote:
>
>>
>>
>> On 4/30/2011 4:58 PM, Charles R Harris wrote:
>> >
>> >
>> > On Sat, Apr 30, 2011 at 5:53 PM, Charles R Harris
>> > <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
>> >
>> >     <snip>
>> >
>> >
>> >         I get a null pointer access violation during numpy.test() with
>> all
>> >         msvc9/MKL builds for win32 (not win-amd64). The crash occurs
>> during
>> >         test_result_type() in test_numeric.py and can be reduced to the
>> >         following code:
>> >
>> >         > >> import numpy as np
>> >         > >> np.result_type(np.array([np.float32(0)]), np.complex128(0))
>> >
>> >         np.float64(0) and np.float16(0) also crash. Unfortunately the
>> debug
>> >         builds do not crash.
>> >
>> >     This is new, right?
>> >
>> >
>> > Does it depend on the optimization level?
>> >
>> > Chuck
>> >
>>
>> Yes it's new. The pure msvc9 builds without MKL also crash. The crash
>> disapperars When compiling with /Od (disable optimization) instead of
>> /Ox (maximum optimization; the default for distutils).
>>
>>
> So all of np.float16(0), np.float32(0), np.float64(0), etc crash? Does it
> depend at all on 0 as the argument, or is it the same for 1, 0.0, etc. What
> about string arguments like np.float64("0"). I want to pin the location down
> a bit more. Too bad it doesn't crash in the debugger.
>
>
I'm suspecting 566ace25f63985e0739bcc600c35336d3c66508c in master, just
because it looks like the last commit to touch the right files.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110430/178aec2e/attachment.html>