From travis at enthought.com  Sun Apr  1 10:22:47 2007
From: travis at enthought.com (Travis Vaught)
Date: Sun, 1 Apr 2007 09:22:47 -0500
Subject: [Numpy-discussion] organizational question
In-Reply-To: <Mahogany-0.67.0-656-20070330-152914.00@american.edu>
References: <Mahogany-0.67.0-656-20070330-152914.00@american.edu>
Message-ID: <DB5745F1-CBDE-40FD-AB0E-17D851860186@enthought.com>

I was just about to respond that _substantial_ support by a non- 
profit best describes Travis Oliphant's many contributions through  
BYU.  I see now that BYU/SciPy/NumPy has been nominated ... excellent!

It might be appropriate for people from various constituencies to  
comment...

"Preference will be given to projects that benefit more and larger  
constituencies, demonstrate exceptional promise or performance, and  
meet particularly urgent needs."

http://matc.mellon.org/2007_nominations/brigham-young-university/ 
scipy-numpy

Best,

Travis (Vaught)

On Mar 30, 2007, at 2:29 PM, Alan G Isaac wrote:

> Is either NumPy or SciPy substantially supported
> by an identifiable and actual non-profit organization?
>
> I ask because I think both fit under
> http://www.mellon.org/grant_programs/programs/copy_of_research
> item 4.
>
> Here is the announcement:
> http://matc.mellon.org/
>
> Note that universities are among the nominees:
> http://matc.mellon.org/2007_nominations
>
> Cheers,
> Alan Isaac
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070401/99a07463/attachment.html>

From chanley at stsci.edu  Sun Apr  1 13:58:41 2007
From: chanley at stsci.edu (Christopher Hanley)
Date: Sun, 01 Apr 2007 13:58:41 -0400
Subject: [Numpy-discussion] test_multiarray.test_clip fails on Solaris 8
	system
Message-ID: <460FF2D1.9080007@stsci.edu>

The following test fails on a Solaris 8 system:

======================================================================
FAIL: check_basic (numpy.core.tests.test_multiarray.test_clip)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/basil5/site-packages/lib/python/numpy/core/tests/test_multiarray.py", line 388, in check_basic
    self._clip_type('float',1024,-12.8,100.2, inplace=inplace)
  File "/data/basil5/site-packages/lib/python/numpy/core/tests/test_multiarray.py", line 382, in _clip_type
    assert_equal(byteorder,x.dtype.byteorder)
  File "/data/basil5/site-packages/lib/python/numpy/testing/utils.py", line 143, in assert_equal
    assert desired == actual, msg
AssertionError: 
Items are not equal:
 ACTUAL: '<'
 DESIRED: '='

----------------------------------------------------------------------
Ran 562 tests in 19.376s

FAILED (failures=1)




From charlesr.harris at gmail.com  Sun Apr  1 14:14:41 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 1 Apr 2007 12:14:41 -0600
Subject: [Numpy-discussion] test_multiarray.test_clip fails on Solaris 8
	system
In-Reply-To: <460FF2D1.9080007@stsci.edu>
References: <460FF2D1.9080007@stsci.edu>
Message-ID: <e06186140704011114h6ea1b319n98df340bbf7e8120@mail.gmail.com>

On 4/1/07, Christopher Hanley <chanley at stsci.edu> wrote:
>
> The following test fails on a Solaris 8 system:
>
> ======================================================================
> FAIL: check_basic (numpy.core.tests.test_multiarray.test_clip)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/data/basil5/site-packages/lib/python/numpy/core/tests/test_multiarray.py",
> line 388, in check_basic
>     self._clip_type('float',1024,-12.8,100.2, inplace=inplace)
>   File
> "/data/basil5/site-packages/lib/python/numpy/core/tests/test_multiarray.py",
> line 382, in _clip_type
>     assert_equal(byteorder,x.dtype.byteorder)
>   File "/data/basil5/site-packages/lib/python/numpy/testing/utils.py",
> line 143, in assert_equal
>     assert desired == actual, msg
> AssertionError:
> Items are not equal:
> ACTUAL: '<'
> DESIRED: '='
>
> ----------------------------------------------------------------------
> Ran 562 tests in 19.376s
>
> FAILED (failures=1)
>
>
> _


Hmm, Sun hardware is big endian, no? I wonder what happens on PPC? I don't
see any problems here on Athlon64.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070401/45d9d0c6/attachment.html>

From chanley at stsci.edu  Sun Apr  1 14:28:53 2007
From: chanley at stsci.edu (Christopher Hanley)
Date: Sun, 01 Apr 2007 14:28:53 -0400
Subject: [Numpy-discussion] test_multiarray.test_clip fails on Solaris
 8	system
In-Reply-To: <e06186140704011114h6ea1b319n98df340bbf7e8120@mail.gmail.com>
References: <460FF2D1.9080007@stsci.edu>
	<e06186140704011114h6ea1b319n98df340bbf7e8120@mail.gmail.com>
Message-ID: <460FF9E5.8090203@stsci.edu>

Sun hardware is big endian.  To be specific, this test was done on a Sun 
Ultra 10.  I don't have access to a PPC right now.  I can check tomorrow 
once I am in the office.

Chris

>
> Hmm, Sun hardware is big endian, no? I wonder what happens on PPC? I 
> don't see any problems here on Athlon64.
>
> Chuck
>



From faltet at carabos.com  Sun Apr  1 15:35:28 2007
From: faltet at carabos.com (Francesc Altet)
Date: Sun, 01 Apr 2007 21:35:28 +0200
Subject: [Numpy-discussion] Release of NumPy 1.0.2 for Monday
In-Reply-To: <460F2D09.40506@ee.byu.edu>
References: <460F2D09.40506@ee.byu.edu>
Message-ID: <1175456128.2556.7.camel@localhost.localdomain>

El ds 31 de 03 del 2007 a les 21:54 -0600, en/na Travis Oliphant va
escriure:
> I'm going to be tagging the tree for the NumPy 1.0.2 release tomorrow 
> evening in preparation for the release on Monday.  I've closed several 
> bugs.  Are there any show-stoppers remaining to be fixed?

Mmm... PyTables test suite segfaults with latest SVN. The problem is
described in #487 and I'm afraid that it is a clear stopper for us.
Sorry for running the PyTables test suite so late before the new NumPy
release.

Also, I've found a patch for #483. Although it is not critical for us,
it would be nice if it can be added to NumPy 1.0.2 final.

Good night!

-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth



From charlesr.harris at gmail.com  Sun Apr  1 16:54:41 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 1 Apr 2007 14:54:41 -0600
Subject: [Numpy-discussion] Violation of array scalar multiplication rules?
Message-ID: <e06186140704011354j69c50416sbe657eec5ec413aa@mail.gmail.com>

Just asking.

In [35]: type(array(1.0)*2)
Out[35]: <type 'numpy.float64'>

In [36]: type(array(1.0))
Out[36]: <type 'numpy.ndarray'>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070401/9c0ffa32/attachment.html>

From oliphant at ee.byu.edu  Sun Apr  1 17:16:58 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sun, 01 Apr 2007 15:16:58 -0600
Subject: [Numpy-discussion] Violation of array scalar multiplication
	rules?
In-Reply-To: <e06186140704011354j69c50416sbe657eec5ec413aa@mail.gmail.com>
References: <e06186140704011354j69c50416sbe657eec5ec413aa@mail.gmail.com>
Message-ID: <4610214A.4030307@ee.byu.edu>

Charles R Harris wrote:
> Just asking.
>
> In [35]: type(array(1.0)*2)
> Out[35]: <type 'numpy.float64'>
>
> In [36]: type(array(1.0))
> Out[36]: <type 'numpy.ndarray'>

No,  in ufuncs 0-d arrays are considered scalars, as are Python scalars 
and array scalars.

Also, ufuncs that result in scalars return NumPy scalars.

-Travis



From stefan at sun.ac.za  Sun Apr  1 17:22:46 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Sun, 1 Apr 2007 23:22:46 +0200
Subject: [Numpy-discussion] test_multiarray.test_clip fails on Solaris
	8	system
In-Reply-To: <460FF9E5.8090203@stsci.edu>
References: <460FF2D1.9080007@stsci.edu>
	<e06186140704011114h6ea1b319n98df340bbf7e8120@mail.gmail.com>
	<460FF9E5.8090203@stsci.edu>
Message-ID: <20070401212246.GI18196@mentat.za.net>

Hi Chris

Would you please run the following commands and show their output?

import sys
print sys.byteorder

import numpy as N
print N.array([1,2,3],N.dtype(N.int16).newbyteorder('<')).dtype.byteorder
print N.array([1,2,3],N.dtype(N.int16).newbyteorder('>')).dtype.byteorder
print N.array([1,2,3],N.dtype(N.int16).newbyteorder('=')).dtype.byteorder

Output on my little-endian system is

little
<
>
=

and I'd be curious to see if the output on a big-endian system follows
the same pattern.

I'd expect

big
<
>
=

Cheers
St?fan


From chanley at stsci.edu  Sun Apr  1 19:03:37 2007
From: chanley at stsci.edu (Christopher Hanley)
Date: Sun, 01 Apr 2007 19:03:37 -0400
Subject: [Numpy-discussion] test_multiarray.test_clip fails on Solaris
 8	system
In-Reply-To: <20070401212246.GI18196@mentat.za.net>
References: <460FF2D1.9080007@stsci.edu>	<e06186140704011114h6ea1b319n98df340bbf7e8120@mail.gmail.com>	<460FF9E5.8090203@stsci.edu>
	<20070401212246.GI18196@mentat.za.net>
Message-ID: <46103A49.5030705@stsci.edu>

Hi Stefan,

This is what I get:
 >>> import sys
 >>> print sys.byteorder
big
 >>> import numpy as N
 >>> print 
N.array([1,2,3],N.dtype(N.int16).newbyteorder('<')).dtype.byteorder
<
 >>> print 
N.array([1,2,3],N.dtype(N.int16).newbyteorder('>')).dtype.byteorder
 >
 >>> print 
N.array([1,2,3],N.dtype(N.int16).newbyteorder('=')).dtype.byteorder
=
 >>>

Stefan van der Walt wrote:
> Hi Chris
>
> Would you please run the following commands and show their output?
>
> import sys
> print sys.byteorder
>
> import numpy as N
> print N.array([1,2,3],N.dtype(N.int16).newbyteorder('<')).dtype.byteorder
> print N.array([1,2,3],N.dtype(N.int16).newbyteorder('>')).dtype.byteorder
> print N.array([1,2,3],N.dtype(N.int16).newbyteorder('=')).dtype.byteorder
>
> Output on my little-endian system is
>
> little
> <
>   
> =
>
> and I'd be curious to see if the output on a big-endian system follows
> the same pattern.
>
> I'd expect
>
> big
> <
>   
> =
>
> Cheers
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From ted.horst at earthlink.net  Sun Apr  1 19:06:34 2007
From: ted.horst at earthlink.net (Ted Horst)
Date: Sun, 1 Apr 2007 18:06:34 -0500
Subject: [Numpy-discussion] test_multiarray.test_clip fails on Solaris
	8	system
In-Reply-To: <20070401212246.GI18196@mentat.za.net>
References: <460FF2D1.9080007@stsci.edu>
	<e06186140704011114h6ea1b319n98df340bbf7e8120@mail.gmail.com>
	<460FF9E5.8090203@stsci.edu> <20070401212246.GI18196@mentat.za.net>
Message-ID: <78D0A115-93D6-4370-9E0E-9C8710CFCDEF@earthlink.net>

I get the same failure on ppc.  Here is the result of your commands:

big
<
 >
=

On Apr 1, 2007, at 16:22, Stefan van der Walt wrote:

> Hi Chris
>
> Would you please run the following commands and show their output?
>
> import sys
> print sys.byteorder
>
> import numpy as N
> print N.array([1,2,3],N.dtype(N.int16).newbyteorder 
> ('<')).dtype.byteorder
> print N.array([1,2,3],N.dtype(N.int16).newbyteorder 
> ('>')).dtype.byteorder
> print N.array([1,2,3],N.dtype(N.int16).newbyteorder 
> ('=')).dtype.byteorder
>
> Output on my little-endian system is
>
> little
> <
>>
> =
>
> and I'd be curious to see if the output on a big-endian system follows
> the same pattern.
>
> I'd expect
>
> big
> <
>>
> =
>
> Cheers
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From wbaxter at gmail.com  Sun Apr  1 22:41:31 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Mon, 2 Apr 2007 11:41:31 +0900
Subject: [Numpy-discussion] Assembling an array from parts
Message-ID: <e86a5fd00704011941u45f91f58y5012402d59d5a132@mail.gmail.com>

What's the best way of assembling a big matrix from parts?
I'm using lagrange multipliers to enforce constraints and this kind of
matrix comes up a lot:

[[ K,   G],
 [ G.T  , 0]]

In matlab you can use the syntax
   [K G; G' zeros(nc)]

In numpy I'm using
   vstack([ hstack([ K,G ]), hstack([ G.T, zeros((nc,nc)) ]) ])

Which has a lot of excess verbiage and parentheses that make it hard
to type and hard to parse what's going on.  It would be a little nicer
if there were some kind of function like 'arraystack':

  arraystack( [ [K, G], [G.T, zeros((nc,nc)) ]] )

Is there anything like this already?  I haven't found anything in the
example list like that.  But maybe concatenate() is flexible enough

--bb


From bioinformed at gmail.com  Mon Apr  2 00:05:12 2007
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Mon, 2 Apr 2007 00:05:12 -0400
Subject: [Numpy-discussion] Assembling an array from parts
In-Reply-To: <e86a5fd00704011941u45f91f58y5012402d59d5a132@mail.gmail.com>
References: <e86a5fd00704011941u45f91f58y5012402d59d5a132@mail.gmail.com>
Message-ID: <2e1434c10704012105p2a716bfh540af33fb6dd8a@mail.gmail.com>

I had to poke around before finding it too:

  bmat( [[K,G],[G.T, zeros(nc)]] )

On 4/1/07, Bill Baxter <wbaxter at gmail.com> wrote:
>
> What's the best way of assembling a big matrix from parts?
> I'm using lagrange multipliers to enforce constraints and this kind of
> matrix comes up a lot:
>
> [[ K,   G],
> [ G.T  , 0]]
>
> In matlab you can use the syntax
>    [K G; G' zeros(nc)]
>
> In numpy I'm using
>    vstack([ hstack([ K,G ]), hstack([ G.T, zeros((nc,nc)) ]) ])
>
> Which has a lot of excess verbiage and parentheses that make it hard
> to type and hard to parse what's going on.  It would be a little nicer
> if there were some kind of function like 'arraystack':
>
>   arraystack( [ [K, G], [G.T, zeros((nc,nc)) ]] )
>
> Is there anything like this already?  I haven't found anything in the
> example list like that.  But maybe concatenate() is flexible enough
>
> --bb
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070402/fb6f2fbf/attachment.html>

From faltet at carabos.com  Mon Apr  2 08:31:58 2007
From: faltet at carabos.com (Francesc Altet)
Date: Mon, 02 Apr 2007 14:31:58 +0200
Subject: [Numpy-discussion] Release of NumPy 1.0.2 for Monday
In-Reply-To: <1175456128.2556.7.camel@localhost.localdomain>
References: <460F2D09.40506@ee.byu.edu>
	<1175456128.2556.7.camel@localhost.localdomain>
Message-ID: <1175517118.6549.1.camel@localhost.localdomain>

El dg 01 de 04 del 2007 a les 21:35 +0200, en/na Francesc Altet va
escriure:
> El ds 31 de 03 del 2007 a les 21:54 -0600, en/na Travis Oliphant va
> escriure:
> > I'm going to be tagging the tree for the NumPy 1.0.2 release tomorrow 
> > evening in preparation for the release on Monday.  I've closed several 
> > bugs.  Are there any show-stoppers remaining to be fixed?
> 
> Mmm... PyTables test suite segfaults with latest SVN. The problem is
> described in #487 and I'm afraid that it is a clear stopper for us.
> Sorry for running the PyTables test suite so late before the new NumPy
> release.
> 
> Also, I've found a patch for #483. Although it is not critical for us,
> it would be nice if it can be added to NumPy 1.0.2 final.

I've checked out r3640 and everything seems to go smoothly with Linux
(32-bit and 64-bit platforms).

Many thanks!

-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth



From gsteele at qualcomm.com  Mon Apr  2 12:07:22 2007
From: gsteele at qualcomm.com (Steele, Greg)
Date: Mon, 2 Apr 2007 09:07:22 -0700
Subject: [Numpy-discussion] Dynamic module not initialized properly
In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EC023@hardwire.esri.com>
References: <627102C921CD9745B070C3B10CB8199B010EC023@hardwire.esri.com>
Message-ID: <4D7539987C6B3340ADA6BC7AE609F73615EDC0@NAEX14.na.qualcomm.com>

Mark,

 

It is hard to comment since you have not provided much information. Your
link to a previous thread brought up a post that I had sent. The issue
that I encountered had to do with the "multiarraymodule.c" extension
module. When "numpy" is imported, it imports this module and the static
variable "_multiarray_module_loaded" gets set. When Python is finalized
is does not unload the multiarraymodule.c DLL. When Python is
initialized again and "numpy" is imported again, the static variable is
already set and multiarraymodule does not import correctly. Hence the
error.

 

The way I dealt with this is a 'hack', but it worked for us. This was on
a windows platform. After I finalize Python, I forcibly unload the
multiarraymodule DLL using the "FreeLibrary" call. The C code looks like

 

    if (multiarray_loaded) {

      HINSTANCE hDLL = NULL;

      hDLL = LoadLibraryEx(buf, NULL,LOAD_WITH_ALTERED_SEARCH_PATH);

      FreeLibrary(hDLL);

      FreeLibrary(hDLL);

    }

 

The two calls of FreeLibrary are needed since each call to LoadLibraryEx
increments the DLL reference count. The call to LoadLibraryEx here gets
a handle to the DLL. 

 

What needs to be done long term is the removal of the static variable in
multiarraymodule. I don't understand the code well enough to know why it
is needed, but that appears to be the crux of the issue. Another
solution would be for Python to call FreeLibrary on all the DLLs during
Py_Finalize.

 

Greg

________________________________

From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Mark Janikas
Sent: Friday, March 30, 2007 4:55 PM
To: Discussion of Numerical Python
Subject: [Numpy-discussion] Dynamic module not initialized properly

 

Hello all,

 

I am having an issue importing numpy on subsequent (I.e. not on first
load) attempts in our software.  The majority of the code is written in
C, C++ and I am a python developer and do not have direct access to a
lot of it.  This is a bit of a difficult question to ask all of you
because I cant provide you a direct example.  All I can do is point to a
numpy thread that discusses the issue:

 

http://groups.google.com/group/Numpy-discussion/browse_thread/thread/321
77a82deab05ae/d8eecaf494ba5ad5?lnk=st&q=dynamic+module+not+initialized+p
roperly+numpy&rnum=1&hl=en#d8eecaf494ba5ad5

 

ERROR:

exceptions.SystemError: dynamic module not initialized properly

 

 What is really odd about my specific issue is that if I don't change
anything in the source code.... Then the error doesn't pop up.
Furthermore, the error doesn't show on some attempts even after I make a
change!!!!  Not sure whether there is anything I can do from the
scripting side (some alternative form of reload?)... or if I have to
forward it along to the C developers.  You have my appreciation ahead of
time.  

 

Mark Janikas

Product Engineer

ESRI, Geoprocessing

380 New York St.

Redlands, CA 92373

909-793-2853 (2563)

mjanikas at esri.com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070402/79ca53c7/attachment.html>

From mjanikas at esri.com  Mon Apr  2 12:46:22 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Mon, 2 Apr 2007 09:46:22 -0700
Subject: [Numpy-discussion] Dynamic module not initialized properly
In-Reply-To: <4D7539987C6B3340ADA6BC7AE609F73615EDC0@NAEX14.na.qualcomm.com>
Message-ID: <627102C921CD9745B070C3B10CB8199B010EC025@hardwire.esri.com>

Thanks for the info Greg.   Yup.  I am sorry that I had to post a thread
without code to back it up.... unfortunately, there just isn't a way for
me to roll it into an example without the entire package being
installed.  This is all very good info you have provided.  Ill let you
know how things work out.  Thanks again,

 

MJ

 

________________________________

From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Steele, Greg
Sent: Monday, April 02, 2007 9:07 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Dynamic module not initialized properly

 

Mark,

 

It is hard to comment since you have not provided much information. Your
link to a previous thread brought up a post that I had sent. The issue
that I encountered had to do with the "multiarraymodule.c" extension
module. When "numpy" is imported, it imports this module and the static
variable "_multiarray_module_loaded" gets set. When Python is finalized
is does not unload the multiarraymodule.c DLL. When Python is
initialized again and "numpy" is imported again, the static variable is
already set and multiarraymodule does not import correctly. Hence the
error.

 

The way I dealt with this is a 'hack', but it worked for us. This was on
a windows platform. After I finalize Python, I forcibly unload the
multiarraymodule DLL using the "FreeLibrary" call. The C code looks like

 

    if (multiarray_loaded) {

      HINSTANCE hDLL = NULL;

      hDLL = LoadLibraryEx(buf, NULL,LOAD_WITH_ALTERED_SEARCH_PATH);

      FreeLibrary(hDLL);

      FreeLibrary(hDLL);

    }

 

The two calls of FreeLibrary are needed since each call to LoadLibraryEx
increments the DLL reference count. The call to LoadLibraryEx here gets
a handle to the DLL. 

 

What needs to be done long term is the removal of the static variable in
multiarraymodule. I don't understand the code well enough to know why it
is needed, but that appears to be the crux of the issue. Another
solution would be for Python to call FreeLibrary on all the DLLs during
Py_Finalize.

 

Greg

________________________________

From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Mark Janikas
Sent: Friday, March 30, 2007 4:55 PM
To: Discussion of Numerical Python
Subject: [Numpy-discussion] Dynamic module not initialized properly

 

Hello all,

 

I am having an issue importing numpy on subsequent (I.e. not on first
load) attempts in our software.  The majority of the code is written in
C, C++ and I am a python developer and do not have direct access to a
lot of it.  This is a bit of a difficult question to ask all of you
because I cant provide you a direct example.  All I can do is point to a
numpy thread that discusses the issue:

 

http://groups.google.com/group/Numpy-discussion/browse_thread/thread/321
77a82deab05ae/d8eecaf494ba5ad5?lnk=st&q=dynamic+module+not+initialized+p
roperly+numpy&rnum=1&hl=en#d8eecaf494ba5ad5

 

ERROR:

exceptions.SystemError: dynamic module not initialized properly

 

 What is really odd about my specific issue is that if I don't change
anything in the source code.... Then the error doesn't pop up.
Furthermore, the error doesn't show on some attempts even after I make a
change!!!!  Not sure whether there is anything I can do from the
scripting side (some alternative form of reload?)... or if I have to
forward it along to the C developers.  You have my appreciation ahead of
time.  

 

Mark Janikas

Product Engineer

ESRI, Geoprocessing

380 New York St.

Redlands, CA 92373

909-793-2853 (2563)

mjanikas at esri.com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070402/d0b819cb/attachment.html>

From a.h.jaffe at gmail.com  Mon Apr  2 14:19:35 2007
From: a.h.jaffe at gmail.com (Andrew Jaffe)
Date: Mon, 02 Apr 2007 19:19:35 +0100
Subject: [Numpy-discussion] test_multiarray.test_clip fails on Solaris 8
	system
In-Reply-To: <e06186140704011114h6ea1b319n98df340bbf7e8120@mail.gmail.com>
References: <460FF2D1.9080007@stsci.edu>
	<e06186140704011114h6ea1b319n98df340bbf7e8120@mail.gmail.com>
Message-ID: <eurhfn$aj6$1@sea.gmane.org>


>     The following test fails on a Solaris 8 system:
>     ======================================================================
>     FAIL: check_basic (numpy.core.tests.test_multiarray.test_clip)
>     ----------------------------------------------------------------------
>     Traceback (most recent call last):
>  ....[removed]....
>     "/data/basil5/site-packages/lib/python/numpy/testing/utils.py", line
>     143, in assert_equal
>         assert desired == actual, msg
>     AssertionError:
>     Items are not equal:
>     ACTUAL: '<'
>     DESIRED: '='
> 
> Hmm, Sun hardware is big endian, no? I wonder what happens on PPC? I 
> don't see any problems here on Athlon64.

Indeed, I get the same problem on a PPC running OSX 10.4

Andrew



From giorgio.luciano at chimica.unige.it  Tue Apr  3 08:25:27 2007
From: giorgio.luciano at chimica.unige.it (Giorgio Luciano)
Date: Tue, 03 Apr 2007 14:25:27 +0200
Subject: [Numpy-discussion] question about standalone small software and
	teaching
Message-ID: <461247B7.6020507@chimica.unige.it>

Hello Dear All,
I just have a question for all that  uses python/numpy/scipy/matplotlib 
for making science.
I use with no problem in my computer  python+numpy+scipy+matplotlib and 
I'm very satisfied with them.
I was a matlab user. I still not have unearthed the power ot python but 
I'm happy to use a programming language and not a metalanguage. When I 
gave people my software (in matlab) the all ask me if I could compile 
and create some interface.
I tried to use matlab GUIs, succeded in creating, but then I had a lot 
of problems. Compiling not always worked. after compiling you have not a 
workspace and so I had to make all output as txt files... and so on.
Now that I use python I'm again with the same problem. I create easy 
routines (for chemometrics) and then people ask me if I can make a 
standalone program with interface.
I used orange and for NN it's surely one of the best, but I'm not good 
at programming widgets. Then I think about it, searched the web and 
didn't find anything.
What I'm searching is something similar to labview :)
At first I thought ... hey why people wat an interface, just use the 
console, and then after listening  to their reason I have to agree.
What do I generally do ? I have a matrix in txt, I apply my routines (a 
SVD, a PCA, a filter etc etc  written in python), plot them (using 
maplotlib) and then I want an output. that's it.
I started looking at various Qt etc. etc. but for me it's overhelming, 
because I think that the most important part should be dedicate to the 
routines creation and not to making a gui, compiling, etc. etc. I need 
simple command like people wants. grids, paste and copy, small working 
plots :)
I mean I can get crazy with setting my program, importing etc. etc. but 
I also have to say that needs and claim about writing simple guis, 
common paste and copy etc should be considered from someone there (we 
wait for the help of some guru that makes things easier ;)

thanks for reading the mail
Giorgio



From gael.varoquaux at normalesup.org  Tue Apr  3 09:21:59 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 3 Apr 2007 15:21:59 +0200
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <461247B7.6020507@chimica.unige.it>
References: <461247B7.6020507@chimica.unige.it>
Message-ID: <20070403132159.GC13747@clipper.ens.fr>

You can do a script with a GUI front end, as described in the first
chapter of my tutorial
http://gael-varoquaux.info/computers/traits_tutorial/traits_tutorial.html
. You can also build a complete interactive application, as described in
the rest of the tutorial, but this is more work.

If you have more questions about this approach feal free to ask.

Ga?l



From faltet at carabos.com  Tue Apr  3 11:47:22 2007
From: faltet at carabos.com (Francesc Altet)
Date: Tue, 3 Apr 2007 17:47:22 +0200
Subject: [Numpy-discussion] About NumPy description
Message-ID: <200704031747.23088.faltet@carabos.com>

Hi,

I've seen the description of NumPy in the recent announcement (btw, good 
work!) and I think it misses something important. To put this in context, let 
me paste the description used for the latest announce:

"""
NumPy is a Python extension that provides a multi-dimensional array and 
basic mathematical processing to Python.  NumPy also provides foundation 
for basic image and signal processing, linear algebra and Fourier 
transforms, and random number generation.  NumPy also includes easy 
integration with ctypes and Fortran allowing powerful extensions to be 
written.
"""

This is ok but I think it stresses a bit too much the mathematical side of 
NumPy and doesn't mention the important field of application for the flexible 
multi-dimensional data containers that it provides. Namely, these containers 
does offer support for completely general datatypes (not only numerical 
types, but also ascii strings and unicode strings)  that can be combined in a 
general way (even forming heterogeneous and/or nested datatypes) in one 
single container in a very efficient manner (both in terms of speed and 
memory).

I think that mentioning this concept can bring the attention of more people, 
and most specially, the developers of database wrappers. One should not 
forget that one of the limitations of the database interfaces in Python is 
that each element retrieved from the database has to be wrapped with a Python 
container (normally string or int/float objects), and this is very ineficient 
when you have to retrieve large datasets from databases. IMHO, database 
developers should learn about this capability of NumPy for the sake of 
achieving efficient databases interfaces for Python.

Cheers,

-- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"


From charlesr.harris at gmail.com  Tue Apr  3 13:53:38 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 3 Apr 2007 11:53:38 -0600
Subject: [Numpy-discussion] LocalBadContent
Message-ID: <e06186140704031053w532339f0l53ef97f3d3299b6@mail.gmail.com>

Scipy Admin,

We might want to delete the contents of http://www.scipy.org/LocalBadContent
.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070403/1462d9ed/attachment.html>

From robert.kern at gmail.com  Tue Apr  3 13:58:12 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 03 Apr 2007 12:58:12 -0500
Subject: [Numpy-discussion] LocalBadContent
In-Reply-To: <e06186140704031053w532339f0l53ef97f3d3299b6@mail.gmail.com>
References: <e06186140704031053w532339f0l53ef97f3d3299b6@mail.gmail.com>
Message-ID: <461295B4.8080105@gmail.com>

Charles R Harris wrote:
> Scipy Admin,
> 
> We might want to delete the contents of
> http://www.scipy.org/LocalBadContent.

Done.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From oliphant at ee.byu.edu  Tue Apr  3 14:23:29 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Tue, 03 Apr 2007 12:23:29 -0600
Subject: [Numpy-discussion] About NumPy description
In-Reply-To: <200704031747.23088.faltet@carabos.com>
References: <200704031747.23088.faltet@carabos.com>
Message-ID: <46129BA1.2000100@ee.byu.edu>

Francesc Altet wrote:
> Hi,
>
> I've seen the description of NumPy in the recent announcement (btw, good 
> work!) and I think it misses something important. To put this in context, let 
> me paste the description used for the latest announce:
>
> """
> NumPy is a Python extension that provides a multi-dimensional array and 
> basic mathematical processing to Python.  NumPy also provides foundation 
> for basic image and signal processing, linear algebra and Fourier 
> transforms, and random number generation.  NumPy also includes easy 
> integration with ctypes and Fortran allowing powerful extensions to be 
> written.
> """
>
> This is ok but I think it stresses a bit too much the mathematical side of 
> NumPy and doesn't mention the important field of application for the flexible 
> multi-dimensional data containers that it provides.

I agree wholeheartedly.   I would love to see a better description.   
Perhaps we can put one up that I can use whenever an announcement is 
made.    Anybody want to put one together?

-Travis



From wbaxter at gmail.com  Tue Apr  3 15:28:19 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Wed, 4 Apr 2007 04:28:19 +0900
Subject: [Numpy-discussion] List of Numpy users anywhere?
Message-ID: <e86a5fd00704031228x5fddfa9fw638a5bfcd88c8f61@mail.gmail.com>

Is there any place on the Wiki that lists all the known software that
uses Numpy in some way?

I just started playing with the Inkscape vector drawing progam.  It
implements extensions using python, and I noticed that one of the
extensions in the latest release uses numpy (and not Numeric or
Numarray).  So I thought I would add Inkscape to "the list" of users,
however, I couldn't find "the list" on the wiki anywhere.

It would be nice to start collecting such a list if there isn't one
already.  Screenshots would be nice too.

--bb


From robert.kern at gmail.com  Tue Apr  3 15:30:16 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 03 Apr 2007 14:30:16 -0500
Subject: [Numpy-discussion] List of Numpy users anywhere?
In-Reply-To: <e86a5fd00704031228x5fddfa9fw638a5bfcd88c8f61@mail.gmail.com>
References: <e86a5fd00704031228x5fddfa9fw638a5bfcd88c8f61@mail.gmail.com>
Message-ID: <4612AB48.6040102@gmail.com>

Bill Baxter wrote:
> Is there any place on the Wiki that lists all the known software that
> uses Numpy in some way?
> 
> I just started playing with the Inkscape vector drawing progam.  It
> implements extensions using python, and I noticed that one of the
> extensions in the latest release uses numpy (and not Numeric or
> Numarray).  So I thought I would add Inkscape to "the list" of users,
> however, I couldn't find "the list" on the wiki anywhere.
> 
> It would be nice to start collecting such a list if there isn't one
> already.  Screenshots would be nice too.

There is no such list that I know of, but you may start one on the wiki if you like.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From oliphant at ee.byu.edu  Tue Apr  3 18:43:29 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Tue, 03 Apr 2007 16:43:29 -0600
Subject: [Numpy-discussion] NumPy 1.0.2 released
Message-ID: <4612D891.8040200@ee.byu.edu>


To all SciPy / NumPy users:

NumPy 1.0.2 was released yesterday (4-02-07).   Get it by following the 
download link at

http://numpy.scipy.org

This is a bug-fix release with a couple of additional features.   Thanks 
to everybody who helped track down and fix bugs.


-Travis



From charlesr.harris at gmail.com  Tue Apr  3 18:42:31 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 3 Apr 2007 16:42:31 -0600
Subject: [Numpy-discussion] About NumPy description
In-Reply-To: <46129BA1.2000100@ee.byu.edu>
References: <200704031747.23088.faltet@carabos.com>
	<46129BA1.2000100@ee.byu.edu>
Message-ID: <e06186140704031542r78f6fbcbgd80a89d220699e8b@mail.gmail.com>

OT, but...

Francesc, could you say whether tickets 373 and 394, reporting possible
memory leaks, are still valid?
Travis, is there something to be done about ticket 399 or should it be
marked an enhancement or some such.

Just trying to clean stuff up ;)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070403/839dd172/attachment.html>

From charlesr.harris at gmail.com  Tue Apr  3 19:30:49 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 3 Apr 2007 17:30:49 -0600
Subject: [Numpy-discussion] NumPy 1.0.2 released
In-Reply-To: <4612D891.8040200@ee.byu.edu>
References: <4612D891.8040200@ee.byu.edu>
Message-ID: <e06186140704031630i73b84d62v651a970a7bb467b9@mail.gmail.com>

On 4/3/07, Travis Oliphant <oliphant at ee.byu.edu> wrote:
>
>
> To all SciPy / NumPy users:
>
> NumPy 1.0.2 was released yesterday (4-02-07).   Get it by following the
> download link at
>
> http://numpy.scipy.org
>
> This is a bug-fix release with a couple of additional features.   Thanks
> to everybody who helped track down and fix bugs.


And thanks for getting it out.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070403/06346b6b/attachment.html>

From steve at shrogers.com  Tue Apr  3 22:13:39 2007
From: steve at shrogers.com (Steven H. Rogers)
Date: Tue, 03 Apr 2007 20:13:39 -0600
Subject: [Numpy-discussion] About NumPy description
In-Reply-To: <46129BA1.2000100@ee.byu.edu>
References: <200704031747.23088.faltet@carabos.com>
	<46129BA1.2000100@ee.byu.edu>
Message-ID: <461309D3.20005@shrogers.com>

Travis Oliphant wrote:
> Francesc Altet wrote:
>> Hi,
>>
>> I've seen the description of NumPy in the recent announcement (btw, good 
>> work!) and I think it misses something important. To put this in context, let 
>> me paste the description used for the latest announce:
>>
>> """
>> NumPy is a Python extension that provides a multi-dimensional array and 
>> basic mathematical processing to Python.  NumPy also provides foundation 
>> for basic image and signal processing, linear algebra and Fourier 
>> transforms, and random number generation.  NumPy also includes easy 
>> integration with ctypes and Fortran allowing powerful extensions to be 
>> written.
>> """
>>
>> This is ok but I think it stresses a bit too much the mathematical side of 
>> NumPy and doesn't mention the important field of application for the flexible 
>> multi-dimensional data containers that it provides.
> 

How about:
"""
NumPy extends Python with a multi-dimensional array type (class) and 
related mathematical functions.  This provides the Python user with 
useful abstractions for managing and computing with multi-dimensional 
bulk data.  This provides a strong foundation for for such domains as 
statistics, image and signal processing, finance, and general systems 
modeling.  Easy integration with ctypes and Fortran allow efficient 
leveraging of high performance C and Fortran code.
"""


From charlesr.harris at gmail.com  Tue Apr  3 22:34:07 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 3 Apr 2007 20:34:07 -0600
Subject: [Numpy-discussion] Ticket cleanup: build related
Message-ID: <e06186140704031934m18c1a1fbw7f495eff43c4e18@mail.gmail.com>

Hi All,

I am trying to go through the active tickets and determine which are still
valid. Today's list : build problems/compiler warnings.

#113 <http://projects.scipy.org/scipy/numpy/ticket/113> Compiler
warnings<http://projects.scipy.org/scipy/numpy/ticket/113>
#114 <http://projects.scipy.org/scipy/numpy/ticket/114> Problems with
building with MSVC and GCC under
Cygwin<http://projects.scipy.org/scipy/numpy/ticket/114>
#117 <http://projects.scipy.org/scipy/numpy/ticket/117> bdist_rpm and
config_fc don't work together<http://projects.scipy.org/scipy/numpy/ticket/117>
#147 <http://projects.scipy.org/scipy/numpy/ticket/147> ValueError or
segfault wrapping ALLOCATABLE f90 arrays using ifort on
AMD64/EM64T<http://projects.scipy.org/scipy/numpy/ticket/147>
#164 <http://projects.scipy.org/scipy/numpy/ticket/164> Patch to build
numpy-0.9.8 on Windows X64 (AMD64) with
MSVS2005<http://projects.scipy.org/scipy/numpy/ticket/164>
#213 <http://projects.scipy.org/scipy/numpy/ticket/213> SharedLibrary
builder for numpy.distutils<http://projects.scipy.org/scipy/numpy/ticket/213>
#220 <http://projects.scipy.org/scipy/numpy/ticket/220> Build fails on Alpha
Linux <http://projects.scipy.org/scipy/numpy/ticket/220>
#244 <http://projects.scipy.org/scipy/numpy/ticket/244> Build fails with
Intel Visual Fortran compiler<http://projects.scipy.org/scipy/numpy/ticket/244>
#311 <http://projects.scipy.org/scipy/numpy/ticket/311> --f77exec= and
--f90exec= are not honoured in f2py
compile<http://projects.scipy.org/scipy/numpy/ticket/311>
#385 <http://projects.scipy.org/scipy/numpy/ticket/385> distutils has
invalid flag for nagware fortran compiler version
5<http://projects.scipy.org/scipy/numpy/ticket/385>
#395 <http://projects.scipy.org/scipy/numpy/ticket/395> GCC compiler
warning<http://projects.scipy.org/scipy/numpy/ticket/395>
#417 <http://projects.scipy.org/scipy/numpy/ticket/417> Numpy 1.0.1
compilation fails on IRIX 6.5<http://projects.scipy.org/scipy/numpy/ticket/417>
#430 <http://projects.scipy.org/scipy/numpy/ticket/430> numpy fails with
PyWin build 210
(PyWin32-210.win32-py2.5.exe)<http://projects.scipy.org/scipy/numpy/ticket/430>

Anyone who knows where things stand for these tickets please reply.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070403/15c14709/attachment.html>

From giorgio.luciano at chimica.unige.it  Wed Apr  4 03:08:09 2007
From: giorgio.luciano at chimica.unige.it (Giorgio Luciano)
Date: Wed, 04 Apr 2007 09:08:09 +0200
Subject: [Numpy-discussion] question about standalone small software and
 teaching
In-Reply-To: <20070403132159.GC13747@clipper.ens.fr>
References: <461247B7.6020507@chimica.unige.it>
	<20070403132159.GC13747@clipper.ens.fr>
Message-ID: <46134ED9.9050301@chimica.unige.it>

Thanks for the reply I will sure try to use it and so some small software.
Giorgio


From faltet at carabos.com  Wed Apr  4 05:13:57 2007
From: faltet at carabos.com (Francesc Altet)
Date: Wed, 4 Apr 2007 11:13:57 +0200
Subject: [Numpy-discussion] About NumPy description
In-Reply-To: <e06186140704031542r78f6fbcbgd80a89d220699e8b@mail.gmail.com>
References: <200704031747.23088.faltet@carabos.com>
	<46129BA1.2000100@ee.byu.edu>
	<e06186140704031542r78f6fbcbgd80a89d220699e8b@mail.gmail.com>
Message-ID: <200704041114.00657.faltet@carabos.com>

A Dimecres 04 Abril 2007 00:42, Charles R Harris escrigu?:
> OT, but...
>
> Francesc, could you say whether tickets 373 and 394, reporting possible
> memory leaks, are still valid?

394 was fixed by Travis long time ago (he simply forgot to close the ticket, 
but now he have done it). Regarding 373, well, while I'm definitely not an 
expert at interpreting valgrind output, I don't really think this would 
represent a serious problem, so I'd close it.

> Travis, is there something to be done about ticket 399 or should it be
> marked an enhancement or some such.

Travis has fixed this tonight :)

> Just trying to clean stuff up ;)

Of course, good idea!

-- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"


From svetosch at gmx.net  Wed Apr  4 11:54:54 2007
From: svetosch at gmx.net (Sven Schreiber)
Date: Wed, 04 Apr 2007 17:54:54 +0200
Subject: [Numpy-discussion] NumPy 1.0.2 released
In-Reply-To: <e06186140704031630i73b84d62v651a970a7bb467b9@mail.gmail.com>
References: <4612D891.8040200@ee.byu.edu>
	<e06186140704031630i73b84d62v651a970a7bb467b9@mail.gmail.com>
Message-ID: <4613CA4E.1010909@gmx.net>

Charles R Harris wrote:
> 
> 
> On 4/3/07, *Travis Oliphant* <oliphant at ee.byu.edu

> 
>     NumPy 1.0.2 was released yesterday (4-02-07).   Get it by following the

> And thanks for getting it out.
> 

>From me too!
-sven



From mjanikas at esri.com  Wed Apr  4 13:31:01 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Wed, 4 Apr 2007 10:31:01 -0700
Subject: [Numpy-discussion] Silent install of .exe
Message-ID: <627102C921CD9745B070C3B10CB8199B010EC03A@hardwire.esri.com>

Is there a way to silently install the numpy.exe from a Microsoft DOS
prompt?

 

Something like: numpy-1.0.2.win32-py2.4.exe -silent

 

Thanks ahead of time...

 

MJ

 

Mark Janikas

Product Engineer

ESRI, Geoprocessing

380 New York St.

Redlands, CA 92373

909-793-2853 (2563)

mjanikas at esri.com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070404/338907cf/attachment.html>

From faltet at carabos.com  Wed Apr  4 15:01:19 2007
From: faltet at carabos.com (Francesc Altet)
Date: Wed, 4 Apr 2007 21:01:19 +0200
Subject: [Numpy-discussion] About NumPy description
In-Reply-To: <461309D3.20005@shrogers.com>
References: <200704031747.23088.faltet@carabos.com>
	<46129BA1.2000100@ee.byu.edu> <461309D3.20005@shrogers.com>
Message-ID: <200704042101.20777.faltet@carabos.com>

A Dimecres 04 Abril 2007 04:13, Steven H. Rogers escrigu?:
> How about:
> """
> NumPy extends Python with a multi-dimensional array type (class) and
> related mathematical functions.  This provides the Python user with
> useful abstractions for managing and computing with multi-dimensional
> bulk data.  This provides a strong foundation for for such domains as
> statistics, image and signal processing, finance, and general systems
> modeling.  Easy integration with ctypes and Fortran allow efficient
> leveraging of high performance C and Fortran code.
> """

Yeah. That's better because it stresses both parts, the data containers and 
the mathematical functionality. However, I think that the word 
multi-dimensional (which is cited twice) may mislead some potential users 
that might be more interested in the heterogeneous type support for use in 
the database field.

At any rate, it's very difficult to expose all the nice things that is 
offering NumPy to the Python users in just a few lines.  Because of this, 
perhaps it is worth to expand these possibilities in the home page of NumPy 
(numpy.scipy.org). BTW, I see this better explained now:

"""
Besides it's obvious scientific uses, NumPy can also be used as an efficient 
multi-dimensional container of generic data. Arbitrary data-types can be 
defined. This allows NumPy to seamlessly and speedily integrate with a 
wide-variety of databases.
"""

Perhaps Travis has modifyied this recently? In that case, I find the above 
paragraph quite informative for the database field.

Lately, I tend to think (as someone has already suggested here) that NumPy is 
embracing many different fields at once and that a split in a couple of 
packages could adapt better to the users need.  IMO, a core with the 
containers and very few functions on top of them (sorting, lookup, 
selection,...), and another layer on top of the core with more mathematical 
(random, linalg, FFT...) functionality would make more sense than the  
current organisation (all in one single package).

Of course, such a split could introduce more difficulties to manage (more 
SVNs, more distribution lists, more releases, more binary packages and so on 
and so forth), but also could lead to a small core that do few things but 
very well done (and hence, easy to integrate in many different projects) and 
the next layer that do many more things but that would be used by a more 
specific user base (more scientific/technical biased). Incidentally, at the 
top of this stack of layers should be scipy, of course.

Finally, I suppose that the NumPy core  that I'm referring to would somehow 
overlap with the PEP for the new buffer interface that Travis is lately 
pushing forward (and that is meant to be integrated with Python 3000), 
although as I see the things, adding basic functionality to this buffer (like 
as I already said, fast sorting, lookup, data selection...) and packaging 
this in a single, compact package could be a really great contribution to the 
Python community in general.

Just some thoughts,

-- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"


From faltet at carabos.com  Wed Apr  4 15:11:18 2007
From: faltet at carabos.com (Francesc Altet)
Date: Wed, 4 Apr 2007 21:11:18 +0200
Subject: [Numpy-discussion] [ANN] PyTables 2.0b2 relased
Message-ID: <200704042111.19939.faltet@carabos.com>

===========================
 Announcing PyTables 2.0b2
===========================

PyTables is a library for managing hierarchical datasets and designed to
efficiently cope with extremely large amounts of data with support for
full 64-bit file addressing.  PyTables runs on top of the HDF5 library
and NumPy package for achieving maximum throughput and convenient use.

The PyTables development team is happy to announce the public
availability of the second *beta* version of PyTables 2.0. This will
hopefully be the last beta version of 2.0 series, so we need your
feedback if you want your issues to be solved before 2.0 final would be
out.

You can download a source package of the version 2.0b2 with
generated PDF and HTML docs and binaries for Windows from
http://www.pytables.org/download/preliminary/

For an on-line version of the manual, visit:
http://www.pytables.org/docs/manual-2.0b2
Please have in mind that some sections in the manual can be obsolete
(specially the "Optimization tips" chapter).  Other chapters should be
fairly up-to-date though (although still a bit in state of flux).

In case you want to know more in detail what has changed in this
version, have a look at ``RELEASE_NOTES.txt``.  Find the HTML version
for this document at:
http://www.pytables.org/moin/ReleaseNotes/Release_2.0b2

If you are a user of PyTables 1.x, probably it is worth for you to look
at ``MIGRATING_TO_2.x.txt`` file where you will find directions on how
to migrate your existing PyTables 1.x apps to the 2.0 version.  You can
find an HTML version of this document at
http://www.pytables.org/moin/ReleaseNotes/Migrating_To_2.x

Keep reading for an overview of the most prominent improvements in
PyTables 2.0 series.


New features of PyTables 2.0
============================

- NumPy is finally at the core!  That means that PyTables no longer
  needs numarray in order to operate, although it continues to be
  supported (as well as Numeric).  This also means that you should be
  able to run PyTables in scenarios combining Python 2.5 and 64-bit
  platforms (these are a source of problems with numarray/Numeric
  because they don't support this combination as of this writing).

- Most of the operations in PyTables have experimented noticeable
  speed-ups (sometimes up to 2x, like in regular Python table
  selections).  This is a consequence of both using NumPy internally and
  a considerable effort in terms of refactorization and optimization of
  the new code.

- Combined conditions are finally supported for in-kernel selections.
  So, now it is possible to perform complex selections like::

      result = [ row['var3'] for row in
                 table.where('(var2 < 20) | (var1 == "sas")') ]

  or::

      complex_cond = '((%s <= col5) & (col2 <= %s)) ' \
                     '| (sqrt(col1 + 3.1*col2 + col3*col4) > 3)'
      result = [ row['var3'] for row in
                 table.where(complex_cond % (inf, sup)) ]

  and run them at full C-speed (or perhaps more, due to the cache-tuned
  computing kernel of Numexpr, which has been integrated into PyTables).

- Now, it is possible to get fields of the ``Row`` iterator by
  specifying their position, or even ranges of positions (extended
  slicing is supported).  For example, you can do::

      result = [ row[4] for row in table    # fetch field #4
                 if row[1] < 20 ]
      result = [ row[:] for row in table    # fetch all fields
                 if row['var2'] < 20 ]
      result = [ row[1::2] for row in       # fetch odd fields
                 table.iterrows(2, 3000, 3) ]

  in addition to the classical::

      result = [row['var3'] for row in table.where('var2 < 20')]

- ``Row`` has received a new method called ``fetch_all_fields()`` in
  order to easily retrieve all the fields of a row in situations like::

      [row.fetch_all_fields() for row in table.where('column1 < 0.3')]

  The difference between ``row[:]`` and ``row.fetch_all_fields()`` is
  that the former will return all the fields as a tuple, while the
  latter will return the fields in a NumPy void type and should be
  faster.  Choose whatever fits better to your needs.

- Now, all data that is read from disk is converted, if necessary, to
  the native byteorder of the hosting machine (before, this only
  happened with ``Table`` objects).  This should help to accelerate
  applications that have to do computations with data generated in
  platforms with a byteorder different than the user machine.

- The modification of values in ``*Array`` objects (through __setitem__)
  now doesn't make a copy of the value in the case that the shape of the
  value passed is the same as the slice to be overwritten. This results
  in considerable memory savings when you are modifying disk objects
  with big array values.

- All the leaf constructors (except Array) have received a new
  ``chunkshape`` argument that lets the user to explicitly select the
  chunksizes for the underlying HDF5 datasets (only for advanced users).

- All the leaf constructors have received a new parameter called
  ``byteorder`` that lets the user specify the byteorder of their data
  *on disk*.  This effectively allows to create datasets in other
  byteorders than the native platform.

- Native HDF5 datasets with ``H5T_ARRAY`` datatypes are fully supported
  for reading now.

- The test suites for the different packages are installed now, so you
  don't need a copy of the PyTables sources to run the tests.  Besides,
  you can run the test suite from the Python console by using::

  >>> tables.tests()


Resources
=========

Go to the PyTables web site for more details:

http://www.pytables.org

About the HDF5 library:

http://hdf.ncsa.uiuc.edu/HDF5/

About NumPy:

http://numpy.scipy.org/

To know more about the company behind the development of PyTables, see:

http://www.carabos.com/


Acknowledgments
===============

Thanks to many users who provided feature improvements, patches, bug
reports, support and suggestions.  See the ``THANKS`` file in the
distribution package for a (incomplete) list of contributors.  Many
thanks also to SourceForge who have helped to make and distribute this
package!  And last, but not least thanks a lot to the HDF5 and NumPy
(and numarray!) makers. Without them PyTables simply would exists.


Share your experience
=====================

Let us know of any bugs, suggestions, gripes, kudos, etc. you may
have.


----

  **Enjoy data!**

  -- The PyTables Team

- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"


From vallis.35530053 at bloglines.com  Wed Apr  4 17:24:14 2007
From: vallis.35530053 at bloglines.com (vallis.35530053 at bloglines.com)
Date: 4 Apr 2007 21:24:14 -0000
Subject: [Numpy-discussion] question about standalone small software and
	teaching
Message-ID: <1175721854.346316249.9616.sendItem@bloglines.com>

Hello Gael (numpy friends),

I'd love to use Traits and TraitsUI. It looks
like a very promising approach. But why is it so difficult to install? If
I download the source from http://code.enthought.com/traits/, and follow the
instructions in enthought.traits-1.1.0/README, and then run the "code snippet
#1" in your tutorial, I get

--- begin error message ---

/Users/vallis/Desktop/enthought.traits-1.1.0/enthought/pyface/util/python_stc.py:14:
DeprecationWarning: The wxPython compatibility package is no longer automatically
generated or activly maintained.  Please switch to the wx package as soon
as possible.
  from wxPython.wx import *
Traceback (most recent call last):

  File "prova.py", line 23, in ?
    camera.configure_traits()
  File "enthought/traits/has_traits.py",
line 1871, in configure_traits
  File "/Users/vallis/Desktop/enthought.traits-1.1.0/enthought/traits/ui/wx/toolkit.py",
line 134, in view_application
    import view_application
  File "/Users/vallis/Desktop/enthought.traits-1.1.0/enthought/traits/ui/wx/view_application.py",
line 29, in ?
    from enthought.debug.fbi \
  File "/Users/vallis/Desktop/enthought.traits-1.1.0/enthought/debug/fbi.py",
line 257, in ?
    auto_size          = False
  File "/Users/vallis/Desktop/enthought.traits-1.1.0/enthought/traits/ui/editors.py",
line 196, in TableEditor
    return toolkit().table_editor( *args, **traits
)
  File "/Users/vallis/Desktop/enthought.traits-1.1.0/enthought/traits/ui/wx/toolkit.py",
line 514, in table_editor
    return te.ToolkitEditorFactory( *args, **traits
)
  File "/Users/vallis/Desktop/enthought.traits-1.1.0/enthought/traits/ui/editor_factory.py",
line 55, in __init__
    HasPrivateTraits.__init__( self, **traits )
  File
"enthought/traits/trait_handlers.py", line 172, in error
enthought.traits.trait_errors.TraitError:
The 'selection_color' trait of a ToolkitEditorFactory instance must be a wx.Colour
instance, an integer which in hex is of the form 0xRRGGBB, where RR is red,
GG is green, and BB is blue, but a value of black was specified.

--- end
error message ---

BTW, I'm using Python 2.4.4 on Macintel, with wxPython-2.8.0.


If I get the latest SVN of the enthought tool suite, go to enthought/src/lib/enthought/traits,
and build with

python setup.py build_src build_clib build_ext --inplace


as suggested in the enthought wiki, and then add enthought/src/lib to my
PYTHONPATH, then your snippet fails with

--- begin error message ---


Traceback (most recent call last):
  File "prova.py", line 5, in ?
   
class Camera(HasTraits):
NameError: name 'HasTraits' is not defined

---
end error message ---

Last, I see that matplotlib includes some enthought/traits
code, but not the ui frontends. Why is that? Is the matplotlib traits usable?


As you can see, I'm very confused... if only there was a traits Python
egg...

Thanks!

Michele


--- Discussion of Numerical Python <numpy-discussion at scipy.org
wrote:
You can do a script with a GUI front end, as described in the first

> chapter of my tutorial
> http://gael-varoquaux.info/computers/traits_tutorial/traits_tutorial.html

> . You can also build a complete interactive application, as described in

> the rest of the tutorial, but this is more work.
> 
> If you have more
questions about this approach feal free to ask.
> 
> Ga?l
> 
> _______________________________________________

> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

> 


From robert.kern at gmail.com  Wed Apr  4 17:36:19 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 04 Apr 2007 16:36:19 -0500
Subject: [Numpy-discussion] question about standalone small software and
 teaching
In-Reply-To: <1175721854.346316249.9616.sendItem@bloglines.com>
References: <1175721854.346316249.9616.sendItem@bloglines.com>
Message-ID: <46141A53.7060902@gmail.com>

vallis.35530053 at bloglines.com wrote:
> Hello Gael (numpy friends),
> 
> I'd love to use Traits and TraitsUI. It looks
> like a very promising approach. But why is it so difficult to install? If
> I download the source from http://code.enthought.com/traits/, and follow the
> instructions in enthought.traits-1.1.0/README, and then run the "code snippet
> #1" in your tutorial, I get

> BTW, I'm using Python 2.4.4 on Macintel, with wxPython-2.8.0.

We require wxPython 2.6 at the moment.

> If I get the latest SVN of the enthought tool suite, go to enthought/src/lib/enthought/traits,
> and build with
> 
> python setup.py build_src build_clib build_ext --inplace
> 
> 
> as suggested in the enthought wiki, and then add enthought/src/lib to my
> PYTHONPATH, then your snippet fails with
> 
> --- begin error message ---
> 
> Traceback (most recent call last):
>   File "prova.py", line 5, in ?
>    
> class Camera(HasTraits):
> NameError: name 'HasTraits' is not defined

Hmm, it works for me. Are you sure that your build is being correctly picked up?
Import enthought, then print enthought.__file__.

> Last, I see that matplotlib includes some enthought/traits
> code, but not the ui frontends. Why is that? Is the matplotlib traits usable?

No, in fact I don't believe it was used at all. It was going to be experimented
with, but I don't think that ever progressed anywhere.

> As you can see, I'm very confused... if only there was a traits Python
> egg...

There are, but only binaries for win32 at the moment. Building from source on OS
X should be straightforward, though.

  https://svn.enthought.com/enthought/wiki/IntelMacPython25

Python 2.4 from www.python.org should work similarly.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Wed Apr  4 17:44:33 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 4 Apr 2007 23:44:33 +0200
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <46141A53.7060902@gmail.com>
References: <1175721854.346316249.9616.sendItem@bloglines.com>
	<46141A53.7060902@gmail.com>
Message-ID: <20070404214433.GB15011@clipper.ens.fr>

On Wed, Apr 04, 2007 at 04:36:19PM -0500, Robert Kern wrote:
> > As you can see, I'm very confused... if only there was a traits Python
> > egg...

> There are, but only binaries for win32 at the moment. Building from
> source on OS X should be straightforward, though.

How about linux eggs ? I had the feeling they had made a lot of progress.

Michele, indeed I would say that the weak point of traitsUI is the
packaging. It is a great module, but it lacks packaging. I personnaly
compile it from svn, but I know this is not an option for everybody.
People are working on this issue, and it is making progress, but
unfortunatly it seems to me that packaging things on OS X is a bit
challenge currently.

Regards,

Ga?l



From vallis.35530053 at bloglines.com  Wed Apr  4 17:51:38 2007
From: vallis.35530053 at bloglines.com (vallis.35530053 at bloglines.com)
Date: 4 Apr 2007 21:51:38 -0000
Subject: [Numpy-discussion] question about standalone small software and
	teaching
Message-ID: <1175723498.4214336904.5601.sendItem@bloglines.com>

--- Discussion of Numerical Python <numpy-discussion at scipy.org wrote:
vallis.35530053 at bloglines.com
wrote:

> > BTW, I'm using Python 2.4.4 on Macintel, with wxPython-2.8.0.

> 
> We require wxPython 2.6 at the moment.

Ah, good to know. This could
explain the errors I get when compiling in place.

> > If I get the latest
SVN of the enthought tool suite, go to enthought/src/lib/enthought/traits,

> > and build with
> > 
> > python setup.py build_src build_clib build_ext
--inplace
> > 
> > 
> > as suggested in the enthought wiki, and then add
enthought/src/lib to my
> > PYTHONPATH, then your snippet fails with
> >

> > --- begin error message ---
> > 
> > Traceback (most recent call last):

> >   File "prova.py", line 5, in ?
> >    
> > class Camera(HasTraits):

> > NameError: name 'HasTraits' is not defined
> 
> Hmm, it works for me.
Are you sure that your build is being correctly picked up?
> Import enthought,
then print enthought.__file__.

Yes, it is picking up the right one. I assume
I can run the setup.py in enthought/src/lib/enthought/traits to get only traits,
right? I'm not installing scipy, or anything else.

> > As you can see,
I'm very confused... if only there was a traits Python
> > egg...
> 
>
There are, but only binaries for win32 at the moment. Building from source
on OS
> X should be straightforward, though.
> 
>   https://svn.enthought.com/enthought/wiki/IntelMacPython25


Ok, I'll try tomorrow and let you know.

Michele 



From robert.kern at gmail.com  Wed Apr  4 18:04:29 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 04 Apr 2007 17:04:29 -0500
Subject: [Numpy-discussion] question about standalone small software and
 teaching
In-Reply-To: <20070404214433.GB15011@clipper.ens.fr>
References: <1175721854.346316249.9616.sendItem@bloglines.com>	<46141A53.7060902@gmail.com>
	<20070404214433.GB15011@clipper.ens.fr>
Message-ID: <461420ED.101@gmail.com>

Gael Varoquaux wrote:
> On Wed, Apr 04, 2007 at 04:36:19PM -0500, Robert Kern wrote:
>>> As you can see, I'm very confused... if only there was a traits Python
>>> egg...
> 
>> There are, but only binaries for win32 at the moment. Building from
>> source on OS X should be straightforward, though.
> 
> How about linux eggs ? I had the feeling they had made a lot of progress.

There's certainly been progress on making the subpackages independently
buildable. I don't think you'll see eggs for Linux, though. Frankly, eggs for
Linux are nearly useless except for specific deployment goals. A Fedora Core 4
egg won't work on a Debian box, etc.

Building enthought from source is not hard. Certainly easier than scipy or
matplotlib.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Wed Apr  4 18:07:38 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 04 Apr 2007 17:07:38 -0500
Subject: [Numpy-discussion] question about standalone small software and
 teaching
In-Reply-To: <1175723498.4214336904.5601.sendItem@bloglines.com>
References: <1175723498.4214336904.5601.sendItem@bloglines.com>
Message-ID: <461421AA.2000304@gmail.com>

vallis.35530053 at bloglines.com wrote:
> --- Discussion of Numerical Python <numpy-discussion at scipy.org wrote:
> vallis.35530053 at bloglines.com
> wrote:

>>> If I get the latest
> SVN of the enthought tool suite, go to enthought/src/lib/enthought/traits,
> 
>>> and build with
>>>
>>> python setup.py build_src build_clib build_ext
> --inplace
>>>
>>> as suggested in the enthought wiki, and then add
> enthought/src/lib to my
>>> PYTHONPATH, then your snippet fails with
>>>
> 
>>> --- begin error message ---
>>>
>>> Traceback (most recent call last):
> 
>>>   File "prova.py", line 5, in ?
>>>    
>>> class Camera(HasTraits):
> 
>>> NameError: name 'HasTraits' is not defined
>> Hmm, it works for me.
> Are you sure that your build is being correctly picked up?
>> Import enthought,
> then print enthought.__file__.
> 
> Yes, it is picking up the right one. I assume
> I can run the setup.py in enthought/src/lib/enthought/traits to get only traits,
> right? I'm not installing scipy, or anything else.

Ah, sorry, I missed the bit where you said you only built inside
enthought/traits/. I'd build the whole suite. It'll take a bit, building the
extension modules for Kiva, but nothing too bad. I don't know why you'd get the
error, though. enthought.traits.api should have HasTraits.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From haase at msg.ucsf.edu  Wed Apr  4 19:41:29 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed, 4 Apr 2007 16:41:29 -0700
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <461421AA.2000304@gmail.com>
References: <1175723498.4214336904.5601.sendItem@bloglines.com>
	<461421AA.2000304@gmail.com>
Message-ID: <bc657ead0704041641y4428e922n316e6a6cfae8f1c8@mail.gmail.com>

Is enthought now defaulting to numpy ?

-Sebastian


On 4/4/07, Robert Kern <robert.kern at gmail.com> wrote:
> vallis.35530053 at bloglines.com wrote:
> > --- Discussion of Numerical Python <numpy-discussion at scipy.org wrote:
> > vallis.35530053 at bloglines.com
> > wrote:
>
> >>> If I get the latest
> > SVN of the enthought tool suite, go to enthought/src/lib/enthought/traits,
> >
> >>> and build with
> >>>
> >>> python setup.py build_src build_clib build_ext
> > --inplace
> >>>
> >>> as suggested in the enthought wiki, and then add
> > enthought/src/lib to my
> >>> PYTHONPATH, then your snippet fails with
> >>>
> >
> >>> --- begin error message ---
> >>>
> >>> Traceback (most recent call last):
> >
> >>>   File "prova.py", line 5, in ?
> >>>
> >>> class Camera(HasTraits):
> >
> >>> NameError: name 'HasTraits' is not defined
> >> Hmm, it works for me.
> > Are you sure that your build is being correctly picked up?
> >> Import enthought,
> > then print enthought.__file__.
> >
> > Yes, it is picking up the right one. I assume
> > I can run the setup.py in enthought/src/lib/enthought/traits to get only traits,
> > right? I'm not installing scipy, or anything else.
>
> Ah, sorry, I missed the bit where you said you only built inside
> enthought/traits/. I'd build the whole suite. It'll take a bit, building the
> extension modules for Kiva, but nothing too bad. I don't know why you'd get the
> error, though. enthought.traits.api should have HasTraits.
>
> --
> Robert Kern


From haase at msg.ucsf.edu  Wed Apr  4 19:46:59 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed, 4 Apr 2007 16:46:59 -0700
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <20070403132159.GC13747@clipper.ens.fr>
References: <461247B7.6020507@chimica.unige.it>
	<20070403132159.GC13747@clipper.ens.fr>
Message-ID: <bc657ead0704041646j7be8abeeo232a2657cf064cb2@mail.gmail.com>

Hello Gael,

Short question regarding your tutorial -- I'm very intrigued by traits
and would like to use them too ....
Why do you define e.g. the Point class like this:
class Point(object):
    """ 3D Point objects """
    x = 0.
    y = 0.
    z = 0.

and not like this:
class Point(object):
    """ 3D Point objects """
    def __init__(self):
       self.x = 0.
       self.y = 0.
       self.z = 0.

I thought in the first case, if one did "a = Point(); a.x = 6"  that
from then on ANY new point ( "b = Point()" ) would be created with b.x
being 6 -- because 'x' is a class attribute   and nor a instance
attribute  !?

This is obviously a beginners question - and I'm hopefully missing something.

Thanks,
Sebastian Haase




On 4/3/07, Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:
> You can do a script with a GUI front end, as described in the first
> chapter of my tutorial
> http://gael-varoquaux.info/computers/traits_tutorial/traits_tutorial.html
> . You can also build a complete interactive application, as described in
> the rest of the tutorial, but this is more work.
>
> If you have more questions about this approach feal free to ask.
>
> Ga?l
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Wed Apr  4 19:48:52 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 04 Apr 2007 18:48:52 -0500
Subject: [Numpy-discussion] question about standalone small software and
 teaching
In-Reply-To: <bc657ead0704041641y4428e922n316e6a6cfae8f1c8@mail.gmail.com>
References: <1175723498.4214336904.5601.sendItem@bloglines.com>	<461421AA.2000304@gmail.com>
	<bc657ead0704041641y4428e922n316e6a6cfae8f1c8@mail.gmail.com>
Message-ID: <46143964.7030300@gmail.com>

Sebastian Haase wrote:
> Is enthought now defaulting to numpy ?

Still set NUMERIX=numpy for now.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Wed Apr  4 19:50:27 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 04 Apr 2007 18:50:27 -0500
Subject: [Numpy-discussion] question about standalone small software and
 teaching
In-Reply-To: <bc657ead0704041646j7be8abeeo232a2657cf064cb2@mail.gmail.com>
References: <461247B7.6020507@chimica.unige.it>	<20070403132159.GC13747@clipper.ens.fr>
	<bc657ead0704041646j7be8abeeo232a2657cf064cb2@mail.gmail.com>
Message-ID: <461439C3.3090102@gmail.com>

Sebastian Haase wrote:
> Hello Gael,
> 
> Short question regarding your tutorial -- I'm very intrigued by traits
> and would like to use them too ....
> Why do you define e.g. the Point class like this:
> class Point(object):
>     """ 3D Point objects """
>     x = 0.
>     y = 0.
>     z = 0.
> 
> and not like this:
> class Point(object):
>     """ 3D Point objects """
>     def __init__(self):
>        self.x = 0.
>        self.y = 0.
>        self.z = 0.
> 
> I thought in the first case, if one did "a = Point(); a.x = 6"  that
> from then on ANY new point ( "b = Point()" ) would be created with b.x
> being 6 -- because 'x' is a class attribute   and nor a instance
> attribute  !?

No, setting "a.x = 6" will set it on the instance, not the class.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From jturner at gemini.edu  Wed Apr  4 20:04:46 2007
From: jturner at gemini.edu (James Turner)
Date: Wed, 04 Apr 2007 20:04:46 -0400
Subject: [Numpy-discussion] nd_image.affine_transform edge effects
In-Reply-To: <20070328172557.GU18196@mentat.za.net>
References: <20070328172557.GU18196@mentat.za.net>
Message-ID: <46143D1E.6020102@gemini.edu>

 > OK, that was a one-line patch.  Please test to see if there are any
 > subtle conditions on the border that I may have missed. I know of one
 > already, but I'd be glad if you can find any others :)

Thanks, Stefan! That looks much better.

Today I finally had time to figure out the basics of SVN, make a patch and
apply your changes to my numarray version of nd_image (I'll switch to numpy
as soon as STScI does a full numpy-based release...). Your integer clipping
changes wouldn't compile under numarray unmodified, but my data are floating
point anyway, so I just applied and tested the array indexing changes.

It looks like there may still be some edge effects due to the mirroring
properties of the spline algorithm for higher orders, but the gross problem
of extrapolating 1 pixel into the mirrored data has gone away :-). I think
that's good enough for nd_image to be useful for me, but if I can find time
later it would be good to look into improving the behaviour further.

For my real data, mode="constant" now seems to work well, but I also tested
some simple examples (like in this thread) using "reflect" and "wrap". I'm
not 100% sure from the numarray manual what their correct behaviour is
supposed to be, but I noticed some things that seem anomalous. For example:

-----

import numarray as N
import numarray.nd_image as ndi

I = N.zeros((2,4),N.Float32)
I[:,:] = N.arange(4.0, 0.0, -1.0)

trmatrix = N.array([[1,0],[0,1]])
troffset1 = (0.0, -0.1)

I_off1 = ndi.affine_transform(I, trmatrix, troffset1, order=1,
               mode='reflect', output_shape=(2,6))

print I
print I_off1

-----

produces

[[ 4.  3.  2.  1.]
  [ 4.  3.  2.  1.]]
[[ 3.0999999   3.0999999   2.0999999   1.10000002  1.89999998
    1.89999998]
  [ 3.0999999   3.0999999   2.0999999   1.10000002  1.89999998
    1.89999998]]

It looks like the last output value is produced by reflecting the
input and then interpolating, but presumably then the first value
should be 3.9, for consistency, not 3.1? Does that make sense?

Cheers,

James.


From jturner at gemini.edu  Wed Apr  4 20:29:50 2007
From: jturner at gemini.edu (James Turner)
Date: Wed, 04 Apr 2007 20:29:50 -0400
Subject: [Numpy-discussion] nd_image.affine_transform edge effects
In-Reply-To: <46143D1E.6020102@gemini.edu>
References: <46143D1E.6020102@gemini.edu>
Message-ID: <461442FE.2020301@gemini.edu>

 > It looks like the last output value is produced by reflecting the
 > input and then interpolating, but presumably then the first value
 > should be 3.9, for consistency, not 3.1? Does that make sense?

Aargh. I think I see what's happening now. The input is supposed to
be interpolated and then reflected like this:

  [4  3  2  1]  ->  [3.1  3.1  2.1  1.1  1.1]

The problem is that there is still one value too many being
"interpolated" here before the reflection takes place. Do the
sections beginning at lines 168 & 178 need changing in a similar way
to their counterparts at lines 129 & 139? I started looking into
this, but I don't understand the code well enough to be sure I'm
making the right changes...

Thanks,

James.



From seb.haase at gmx.net  Wed Apr  4 21:53:52 2007
From: seb.haase at gmx.net (Sebastian Haase)
Date: Wed, 4 Apr 2007 18:53:52 -0700
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <461439C3.3090102@gmail.com>
References: <461247B7.6020507@chimica.unige.it>
	<20070403132159.GC13747@clipper.ens.fr>
	<bc657ead0704041646j7be8abeeo232a2657cf064cb2@mail.gmail.com>
	<461439C3.3090102@gmail.com>
Message-ID: <bc657ead0704041853o56dc51b9x590d70477b4bb616@mail.gmail.com>

On 4/4/07, Robert Kern <robert.kern at gmail.com> wrote:
> Sebastian Haase wrote:
> > Hello Gael,
> >
> > Short question regarding your tutorial -- I'm very intrigued by traits
> > and would like to use them too ....
> > Why do you define e.g. the Point class like this:
> > class Point(object):
> >     """ 3D Point objects """
> >     x = 0.
> >     y = 0.
> >     z = 0.
> >
> > and not like this:
> > class Point(object):
> >     """ 3D Point objects """
> >     def __init__(self):
> >        self.x = 0.
> >        self.y = 0.
> >        self.z = 0.
> >
> > I thought in the first case, if one did "a = Point(); a.x = 6"  that
> > from then on ANY new point ( "b = Point()" ) would be created with b.x
> > being 6 -- because 'x' is a class attribute   and nor a instance
> > attribute  !?
>
> No, setting "a.x = 6" will set it on the instance, not the class.

OK, but what is "wrong" with the first way !?  I mean,  it somehow
seems not like "it's usually done" in Python ?  Normally there is
always a __init__(self) that sets up everything referring to self --
why is this tutorial doing it differently ?

-Sebastian


From robert.kern at gmail.com  Wed Apr  4 22:00:37 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 04 Apr 2007 21:00:37 -0500
Subject: [Numpy-discussion] question about standalone small software and
 teaching
In-Reply-To: <bc657ead0704041853o56dc51b9x590d70477b4bb616@mail.gmail.com>
References: <461247B7.6020507@chimica.unige.it>	<20070403132159.GC13747@clipper.ens.fr>	<bc657ead0704041646j7be8abeeo232a2657cf064cb2@mail.gmail.com>	<461439C3.3090102@gmail.com>
	<bc657ead0704041853o56dc51b9x590d70477b4bb616@mail.gmail.com>
Message-ID: <46145845.4080600@gmail.com>

Sebastian Haase wrote:

> OK, but what is "wrong" with the first way !?  I mean,  it somehow
> seems not like "it's usually done" in Python ?  Normally there is
> always a __init__(self) that sets up everything referring to self --
> why is this tutorial doing it differently ?

Because it makes the code more readable for the point it's trying to get across.
The __init__ style code would be irrelevant detail detracting from the main point.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From wbaxter at gmail.com  Wed Apr  4 22:10:28 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Thu, 5 Apr 2007 11:10:28 +0900
Subject: [Numpy-discussion] Big list of Numpy & Scipy users
Message-ID: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>

On 4/4/07, Robert Kern <robert.kern at gmail.com> wrote:
> Bill Baxter wrote:
> > Is there any place on the Wiki that lists all the known software that
> > uses Numpy in some way?
> >
>> > It would be nice to start collecting such a list if there isn't one
> > already.  Screenshots would be nice too.
>
> There is no such list that I know of, but you may start one on the wiki if you like.

Ok, I made a start:  http://www.scipy.org/Scipy_Projects
Anyone who has a project that depends on Numpy or Scipy, please go add
your info there!

I haven't linked it from anywhere, because it looks pretty pathetic
right now with only three or four entries.  But hopefully everyone
will jumps in and add their project to the list.

Part of the idea is that this should be a good place to point
nay-sayers to when they say "meh - numpy... that's a niche project for
a handful of scientists."

So ... hopefully a good portion of the links will be things other than
science projects.  There will hopefully be a lot of things that
"ordinary users" would care about.  :-)

I couldn't figure out how to add an image, but if someone knows how to
do that, please do.

--bb


From wbaxter at gmail.com  Wed Apr  4 22:12:52 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Thu, 5 Apr 2007 11:12:52 +0900
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <46145845.4080600@gmail.com>
References: <461247B7.6020507@chimica.unige.it>
	<20070403132159.GC13747@clipper.ens.fr>
	<bc657ead0704041646j7be8abeeo232a2657cf064cb2@mail.gmail.com>
	<461439C3.3090102@gmail.com>
	<bc657ead0704041853o56dc51b9x590d70477b4bb616@mail.gmail.com>
	<46145845.4080600@gmail.com>
Message-ID: <e86a5fd00704041912x32d75215ia6eed5cf3e5bf1ab@mail.gmail.com>

Ok, I got another hopefully easy question:

Why this:
    class Point(object):
          ...

Instead of the style that's used in the Python tutorial in the
'classes' chapter:
    class Point:
        ...

--bb


On 4/5/07, Robert Kern <robert.kern at gmail.com> wrote:
> Sebastian Haase wrote:
>
> > OK, but what is "wrong" with the first way !?  I mean,  it somehow
> > seems not like "it's usually done" in Python ?  Normally there is
> > always a __init__(self) that sets up everything referring to self --
> > why is this tutorial doing it differently ?
>
> Because it makes the code more readable for the point it's trying to get across.
> The __init__ style code would be irrelevant detail detracting from the main point.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless enigma
>  that is made terrible by our own mad attempt to interpret it as though it had
>  an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Wed Apr  4 22:16:30 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 04 Apr 2007 21:16:30 -0500
Subject: [Numpy-discussion] question about standalone small software and
 teaching
In-Reply-To: <e86a5fd00704041912x32d75215ia6eed5cf3e5bf1ab@mail.gmail.com>
References: <461247B7.6020507@chimica.unige.it>	<20070403132159.GC13747@clipper.ens.fr>	<bc657ead0704041646j7be8abeeo232a2657cf064cb2@mail.gmail.com>	<461439C3.3090102@gmail.com>	<bc657ead0704041853o56dc51b9x590d70477b4bb616@mail.gmail.com>	<46145845.4080600@gmail.com>
	<e86a5fd00704041912x32d75215ia6eed5cf3e5bf1ab@mail.gmail.com>
Message-ID: <46145BFE.1070809@gmail.com>

Bill Baxter wrote:
> Ok, I got another hopefully easy question:
> 
> Why this:
>     class Point(object):
>           ...
> 
> Instead of the style that's used in the Python tutorial in the
> 'classes' chapter:
>     class Point:
>         ...

Because the former make new-style classes and the latter make old-style classes.
It's not an issue of personal preference: they are somewhat different object
models and there are things that old-style classes can't do. As HasTraits is
also a new-style class, there's no point in using old-style classes in this
tutorial.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From wbaxter at gmail.com  Wed Apr  4 22:23:02 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Thu, 5 Apr 2007 11:23:02 +0900
Subject: [Numpy-discussion] basic python questions
Message-ID: <e86a5fd00704041923y4c0a58dfje2cfaedf003ec7cd@mail.gmail.com>

On 4/5/07, Robert Kern <robert.kern at gmail.com> wrote:
> Bill Baxter wrote:
> > Ok, I got another hopefully easy question:
> >
> > Why this:
> >     class Point(object):
> >           ...
> >
> > Instead of the style that's used in the Python tutorial in the
> > 'classes' chapter:
> >     class Point:
> >         ...
>
> Because the former make new-style classes and the latter make old-style classes.
> It's not an issue of personal preference: they are somewhat different object
> models and there are things that old-style classes can't do. As HasTraits is
> also a new-style class, there's no point in using old-style classes in this
> tutorial.

What's the difference in the object models?  I'm surprised that the
Python tutorial seems to be completely silent on this issue.
(http://docs.python.org/tut/node11.html)

--bb


From robert.kern at gmail.com  Wed Apr  4 22:35:17 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 04 Apr 2007 21:35:17 -0500
Subject: [Numpy-discussion] basic python questions
In-Reply-To: <e86a5fd00704041923y4c0a58dfje2cfaedf003ec7cd@mail.gmail.com>
References: <e86a5fd00704041923y4c0a58dfje2cfaedf003ec7cd@mail.gmail.com>
Message-ID: <46146065.6070407@gmail.com>

Bill Baxter wrote:
> On 4/5/07, Robert Kern <robert.kern at gmail.com> wrote:
>> Bill Baxter wrote:
>>> Ok, I got another hopefully easy question:
>>>
>>> Why this:
>>>     class Point(object):
>>>           ...
>>>
>>> Instead of the style that's used in the Python tutorial in the
>>> 'classes' chapter:
>>>     class Point:
>>>         ...
>> Because the former make new-style classes and the latter make old-style classes.
>> It's not an issue of personal preference: they are somewhat different object
>> models and there are things that old-style classes can't do. As HasTraits is
>> also a new-style class, there's no point in using old-style classes in this
>> tutorial.
> 
> What's the difference in the object models?  I'm surprised that the
> Python tutorial seems to be completely silent on this issue.
> (http://docs.python.org/tut/node11.html)

http://www.python.org/doc/newstyle.html

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From wbaxter at gmail.com  Wed Apr  4 22:57:04 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Thu, 5 Apr 2007 11:57:04 +0900
Subject: [Numpy-discussion] basic python questions
In-Reply-To: <46146065.6070407@gmail.com>
References: <e86a5fd00704041923y4c0a58dfje2cfaedf003ec7cd@mail.gmail.com>
	<46146065.6070407@gmail.com>
Message-ID: <e86a5fd00704041957g5a6b0d1bnd14ef8d2f2cdcce6@mail.gmail.com>

On 4/5/07, Robert Kern <robert.kern at gmail.com> wrote:
> Bill Baxter wrote:
> > On 4/5/07, Robert Kern <robert.kern at gmail.com> wrote:
> >> Bill Baxter wrote:
> >>> Ok, I got another hopefully easy question:
> >>>
> >>> Why this:
> >>>     class Point(object):
> >>>           ...
> >>>
> >>> Instead of the style that's used in the Python tutorial in the
> >>> 'classes' chapter:
> >>>     class Point:
> >>>         ...
> >> Because the former make new-style classes and the latter make old-style classes.
> >> It's not an issue of personal preference: they are somewhat different object
> >> models and there are things that old-style classes can't do. As HasTraits is
> >> also a new-style class, there's no point in using old-style classes in this
> >> tutorial.
> >
> > What's the difference in the object models?  I'm surprised that the
> > Python tutorial seems to be completely silent on this issue.
> > (http://docs.python.org/tut/node11.html)
>
> http://www.python.org/doc/newstyle.html
>

Good link.  Thanks!

--bb


From efiring at hawaii.edu  Wed Apr  4 23:10:58 2007
From: efiring at hawaii.edu (Eric Firing)
Date: Wed, 04 Apr 2007 17:10:58 -1000
Subject: [Numpy-discussion] basic python questions
In-Reply-To: <46146065.6070407@gmail.com>
References: <e86a5fd00704041923y4c0a58dfje2cfaedf003ec7cd@mail.gmail.com>
	<46146065.6070407@gmail.com>
Message-ID: <461468C2.10302@hawaii.edu>

Robert Kern wrote:
> Bill Baxter wrote:
>> On 4/5/07, Robert Kern <robert.kern at gmail.com> wrote:
>>> Bill Baxter wrote:
>>>> Ok, I got another hopefully easy question:
>>>>
>>>> Why this:
>>>>     class Point(object):
>>>>           ...
>>>>
>>>> Instead of the style that's used in the Python tutorial in the
>>>> 'classes' chapter:
>>>>     class Point:
>>>>         ...
>>> Because the former make new-style classes and the latter make old-style classes.
>>> It's not an issue of personal preference: they are somewhat different object
>>> models and there are things that old-style classes can't do. As HasTraits is
>>> also a new-style class, there's no point in using old-style classes in this
>>> tutorial.
>> What's the difference in the object models?  I'm surprised that the
>> Python tutorial seems to be completely silent on this issue.
>> (http://docs.python.org/tut/node11.html)
> 
> http://www.python.org/doc/newstyle.html
> 

Key point: properties work with new-style classes but fail silently and 
mysteriously with classic classes.

Eric



From aisaac at american.edu  Wed Apr  4 23:26:30 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 4 Apr 2007 23:26:30 -0400
Subject: [Numpy-discussion] basic python questions
In-Reply-To: <461468C2.10302@hawaii.edu>
References: <e86a5fd00704041923y4c0a58dfje2cfaedf003ec7cd@mail.gmail.com><46146065.6070407@gmail.com><461468C2.10302@hawaii.edu>
Message-ID: <Mahogany-0.67.0-792-20070404-232630.00@american.edu>

On Wed, 04 Apr 2007, Eric Firing apparently wrote: 
> Key point: properties work with new-style classes but fail 
> silently and mysteriously with classic classes. 

Or making the same point a little more generally,
descriptors only work for new-style classes:
http://users.rcn.com/python/download/Descriptor.htm

I am nevertheless uncertain why you need new-style
classes if you create a iterable class that you
want numpy to be able to convert to an array.

Cheers,
Alan Isaac




From haase at msg.ucsf.edu  Thu Apr  5 00:49:47 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed, 4 Apr 2007 21:49:47 -0700
Subject: [Numpy-discussion] Big list of Numpy & Scipy users
In-Reply-To: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>
References: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>
Message-ID: <bc657ead0704042149u5c0d7d1es5a4cca1c6e860f17@mail.gmail.com>

Hi,
Why do you call it
Scipy_Projects
if it also lists people/project who use (only) numpy.

I wish I could suggest a better name ...
I just checked the swig.org web site;  the call it just
"projects"   ( http://www.swig.org/projects.html )
[ Open source projects using SWIG ]
so maybe just leaving out the "Scipy_" part.

BTW,  do peer review papers count !?  I have two of them, using numpy
(originally numarray, but now it's numpy)

Maybe the projects should be in categories:
- open source
- commercial   (?)
- papers
- ??

-Sebastian




On 4/4/07, Bill Baxter <wbaxter at gmail.com> wrote:
> On 4/4/07, Robert Kern <robert.kern at gmail.com> wrote:
> > Bill Baxter wrote:
> > > Is there any place on the Wiki that lists all the known software that
> > > uses Numpy in some way?
> > >
> >> > It would be nice to start collecting such a list if there isn't one
> > > already.  Screenshots would be nice too.
> >
> > There is no such list that I know of, but you may start one on the wiki if you like.
>
> Ok, I made a start:  http://www.scipy.org/Scipy_Projects
> Anyone who has a project that depends on Numpy or Scipy, please go add
> your info there!
>
> I haven't linked it from anywhere, because it looks pretty pathetic
> right now with only three or four entries.  But hopefully everyone
> will jumps in and add their project to the list.
>
> Part of the idea is that this should be a good place to point
> nay-sayers to when they say "meh - numpy... that's a niche project for
> a handful of scientists."
>
> So ... hopefully a good portion of the links will be things other than
> science projects.  There will hopefully be a lot of things that
> "ordinary users" would care about.  :-)
>
> I couldn't figure out how to add an image, but if someone knows how to
> do that, please do.
>
> --bb
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From haase at msg.ucsf.edu  Thu Apr  5 01:07:09 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed, 4 Apr 2007 22:07:09 -0700
Subject: [Numpy-discussion] basic python questions
In-Reply-To: <e86a5fd00704041923y4c0a58dfje2cfaedf003ec7cd@mail.gmail.com>
References: <e86a5fd00704041923y4c0a58dfje2cfaedf003ec7cd@mail.gmail.com>
Message-ID: <bc657ead0704042207s1958d838od5cf83dba5cc531f@mail.gmail.com>

On 4/4/07, Bill Baxter <wbaxter at gmail.com> wrote:
> On 4/5/07, Robert Kern <robert.kern at gmail.com> wrote:
> > Bill Baxter wrote:
> > > Ok, I got another hopefully easy question:
> > >
> > > Why this:
> > >     class Point(object):
> > >           ...
> > >
> > > Instead of the style that's used in the Python tutorial in the
> > > 'classes' chapter:
> > >     class Point:
> > >         ...
> >
> > Because the former make new-style classes and the latter make old-style classes.
> > It's not an issue of personal preference: they are somewhat different object
> > models and there are things that old-style classes can't do. As HasTraits is
> > also a new-style class, there's no point in using old-style classes in this
> > tutorial.
>
> What's the difference in the object models?  I'm surprised that the
> Python tutorial seems to be completely silent on this issue.
> (http://docs.python.org/tut/node11.html)
>
Not really answering your question -- but I have complained about that
tutorial before, with regards to new language features ... it does not
mention
from __future__ import division
In my mind this should be put at the very front - because it's going
to be a very big thing once Python 3000 comes around.
The Python-list people did not like my arguing because apparently the
tutorial is supposed to "look nice" ....  [[ don't get me wrong,  I
really recommend the tutorial, I like it, I think it's good ]]
But some (even if) ugly things should be said up front, if they clear
up the way.
Python 3000 will also default to new-style classes -- so that
"(object)" thing would go away again if I'm not mistaken.

-Sebastian

PS:
Maybe this list could officially endorse the
from __future__ import division
I would be very interested in this !
Math becomes clearer, and  things like  arr[5/2]  won't only suddenly
fail in the future,  they should be NOW written as arr[5//2] (if you
need integer division)
Thanks. [[ please start a new thread, and put up a page on the wiki ;-) ]]


From wbaxter at gmail.com  Thu Apr  5 01:52:13 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Thu, 5 Apr 2007 14:52:13 +0900
Subject: [Numpy-discussion] Big list of Numpy & Scipy users
In-Reply-To: <bc657ead0704042149u5c0d7d1es5a4cca1c6e860f17@mail.gmail.com>
References: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>
	<bc657ead0704042149u5c0d7d1es5a4cca1c6e860f17@mail.gmail.com>
Message-ID: <e86a5fd00704042252i3e4a47ddt7c52b9bbf2bc929b@mail.gmail.com>

On 4/5/07, Sebastian Haase <haase at msg.ucsf.edu> wrote:
> Hi,
> Why do you call it
> Scipy_Projects
> if it also lists people/project who use (only) numpy.
>
> I wish I could suggest a better name ...
> I just checked the swig.org web site;  the call it just
> "projects"   ( http://www.swig.org/projects.html )
> [ Open source projects using SWIG ]
> so maybe just leaving out the "Scipy_" part.
>
> BTW,  do peer review papers count !?  I have two of them, using numpy
> (originally numarray, but now it's numpy)
>
> Maybe the projects should be in categories:
> - open source
> - commercial   (?)
> - papers
> - ??
>
> -Sebastian
>
>
>
>
> On 4/4/07, Bill Baxter <wbaxter at gmail.com> wrote:
> > On 4/4/07, Robert Kern <robert.kern at gmail.com> wrote:
> > > Bill Baxter wrote:
> > > > Is there any place on the Wiki that lists all the known software that
> > > > uses Numpy in some way?
> > > >
> > >> > It would be nice to start collecting such a list if there isn't one
> > > > already.  Screenshots would be nice too.
> > >
> > > There is no such list that I know of, but you may start one on the wiki if you like.
> >
> > Ok, I made a start:  http://www.scipy.org/Scipy_Projects
> > Anyone who has a project that depends on Numpy or Scipy, please go add
> > your info there!
> >
> > I haven't linked it from anywhere, because it looks pretty pathetic
> > right now with only three or four entries.  But hopefully everyone
> > will jumps in and add their project to the list.
> >
> > Part of the idea is that this should be a good place to point
> > nay-sayers to when they say "meh - numpy... that's a niche project for
> > a handful of scientists."
> >
> > So ... hopefully a good portion of the links will be things other than
> > science projects.  There will hopefully be a lot of things that
> > "ordinary users" would care about.  :-)
> >
> > I couldn't figure out how to add an image, but if someone knows how to
> > do that, please do.
> >
> > --bb

Good points.  I don't really have answers.  So just go to town
reorganizing as you feel fit.
But open source / commercial is not a real dichotomy.  And a paper may
be written about a system that is available as open source.  I tried
to think of some such categories, but applications vs libs was about
the best I could come up with.  Probably a tagging system would be the
most appropriate, but that's not so easy to make work on a Wiki.

--bb


From gael.varoquaux at normalesup.org  Thu Apr  5 02:04:12 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 5 Apr 2007 08:04:12 +0200
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <461421AA.2000304@gmail.com>
References: <1175723498.4214336904.5601.sendItem@bloglines.com>
	<461421AA.2000304@gmail.com>
Message-ID: <20070405060412.GB1370@clipper.ens.fr>

On Wed, Apr 04, 2007 at 05:07:38PM -0500, Robert Kern wrote:
> Ah, sorry, I missed the bit where you said you only built inside
> enthought/traits/. I'd build the whole suite. It'll take a bit,
> building the extension modules for Kiva, but nothing too bad. I don't
> know why you'd get the error, though. enthought.traits.api should have
> HasTraits.

Actually if you have problem compiling you can try leaving out kiva, it
is not terribly useful for what you are currently doing. To do this
comment the "config.add_subpackage('kiva') in "setup.py". I have found
out that kiva is the package that is the hardest to compile.

The way I compile the ETS is to use the old "./build_inplace.sh numpy"
method. IWOMB (it works on my box).

Ga?l


From gael.varoquaux at normalesup.org  Thu Apr  5 02:24:17 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 5 Apr 2007 08:24:17 +0200
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <bc657ead0704041646j7be8abeeo232a2657cf064cb2@mail.gmail.com>
References: <461247B7.6020507@chimica.unige.it>
	<20070403132159.GC13747@clipper.ens.fr>
	<bc657ead0704041646j7be8abeeo232a2657cf064cb2@mail.gmail.com>
Message-ID: <20070405062417.GC1370@clipper.ens.fr>

On Wed, Apr 04, 2007 at 04:46:59PM -0700, Sebastian Haase wrote:
> Why do you define e.g. the Point class like this:
> class Point(object):
>     """ 3D Point objects """
>     x = 0.
>     y = 0.
>     z = 0.

> and not like this:
> class Point(object):
>     """ 3D Point objects """
>     def __init__(self):
>        self.x = 0.
>        self.y = 0.
>        self.z = 0.

> I thought in the first case, if one did "a = Point(); a.x = 6"  that
> from then on ANY new point ( "b = Point()" ) would be created with b.x
> being 6 -- because 'x' is a class attribute   and nor a instance
> attribute  !?

> This is obviously a beginners question - and I'm hopefully missing something.

Actually this raises quite subtle problems. 

The first syntax defines class attributes, and the second instance
attributes. Now in the given example there is no diference, because "x"
is a float, wich is immutable, so x get replaced each time it is
modified. I tend to use class attributes because I find that the class
definition is more readable, and they also survive to a forgotten
"super().__init__" in the init. Another reason is that traits are always
class attributes, so as I use traits a lot I tend to have the habit of
class attributes.

Now the subtle problems are illustrated by this code snippet:

++++++++++++++++++++++++
class Foo(object):
    a = [1, ]

f = Foo()
f.a.append(1)

g = Foo()

print g.a
++++++++++++++++++++++++

This will print out "[1, 1]". The reason is that the class attribute "a"
is a mutable that has been modified in place by the "append" operation.

I have seen beginners get tangled in these problems and I have recently
started avoided using class attributes when not necessary, so as to
avoid having to explain what are mutables, and class attributes... to
beginners, who often do not find all this easy to understand.

Ga?l


From stefan at sun.ac.za  Thu Apr  5 06:15:30 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 5 Apr 2007 12:15:30 +0200
Subject: [Numpy-discussion] nd_image.affine_transform edge effects
In-Reply-To: <461442FE.2020301@gemini.edu>
References: <46143D1E.6020102@gemini.edu> <461442FE.2020301@gemini.edu>
Message-ID: <20070405101530.GU18196@mentat.za.net>

Hi James

On Wed, Apr 04, 2007 at 08:29:50PM -0400, James Turner wrote:
>  > It looks like the last output value is produced by reflecting the
>  > input and then interpolating, but presumably then the first value
>  > should be 3.9, for consistency, not 3.1? Does that make sense?
> 
> Aargh. I think I see what's happening now. The input is supposed to
> be interpolated and then reflected like this:
> 
>   [4  3  2  1]  ->  [3.1  3.1  2.1  1.1  1.1]
> 
> The problem is that there is still one value too many being
> "interpolated" here before the reflection takes place. Do the
> sections beginning at lines 168 & 178 need changing in a similar way
> to their counterparts at lines 129 & 139? I started looking into
> this, but I don't understand the code well enough to be sure I'm
> making the right changes...

Thanks for spotting that.  When I fix those lines, I see:

[[ 3.9000001   3.0999999   2.0999999   1.10000002  1.89999998  2.9000001 ]
 [ 3.9000001   3.0999999   2.0999999   1.10000002  1.89999998  2.9000001 ]]

I'll submit to SVN later today.  Note that I also enabled 'mirror'
mode, which works almost the same way as reflect:

Reflect: 1 2 3 4 -> 1 2 3 4 4 3 2 1
Mirror:  1 2 3 4 -> 1 2 3 4 3 2 1

Cheers
St?fan


From hurak at fel.cvut.cz  Thu Apr  5 08:22:17 2007
From: hurak at fel.cvut.cz (=?UTF-8?B?WmRlbsSbayBIdXLDoWs=?=)
Date: Thu, 05 Apr 2007 14:22:17 +0200
Subject: [Numpy-discussion] F2PY and underscore problem
Message-ID: <ev2plp$vff$1@sea.gmane.org>

I am sorry to come up with a question that probably has been asked and
answered many times but I honestly searched for an answer but failed to
find one.

I want to create Python modules from some Fortran numerical functions that
rely on LAPACK. I start with a function SB02MD.f that provides a solver for
algebraic Riccati equation (ARE) that can be obtained at
http://www.slicot.de/. I proceeded the standard f2py way:

f2py SB02MD.f -m are -h are1.pyf

modified are1.pyf (saved as are.pyf) to tell which arguments are inputs and
outputs and compiled the module:

f2py -c are.pyf SB02MD.f --link-lapack_opt

However, when trying to import the module, Python cries that there is and
undefined symbol: sb02mw_. The same situation if I omit --link-lapack_opt.

The famous underscore problem. Perhaps important information is that I am on
Linux and experience this problem both with lapack-atlas and
lapack-reference libraries. Why I am so focused on Lapack? Some other
examples for f2py that do not need Lapack work well here.  

Even though Google gives miriad of links for "f2py underscore" problem, I
failed to find the one that is relevant for me. I played with the options
like -DUNDERSCORE_G77 and -DNO_APPEND_FORTRAN but no help and I actually do
not understand their use at all (not explained in man for f2py).

Thanks for whatever hint/advice.


-- 
------------------------------------------------
Zdenek Hurak
Czech Technical University in Prague
------------------------------------------------



From nicolas.chauvat at logilab.fr  Mon Apr  2 11:43:22 2007
From: nicolas.chauvat at logilab.fr (Nicolas Chauvat)
Date: Mon, 2 Apr 2007 17:43:22 +0200
Subject: [Numpy-discussion] [ANN] EuroPython 2007: Call for Proposals
Message-ID: <20070402154322.GB24884@crater.logilab.fr>

Book Monday 9th July to Wednesday 11th July 2007 in your calendar!
EuroPython 2007, the European Python and Zope Conference, will be held in
Vilnius, Lithuania.  Last year's conference was a great success, featuring
a variety of tracks, amazing lightning talks and inspiring keynotes.  With
your participation, we want to make EuroPython 2007, the sixth EuroPython,
even more successful than the previous five.

Talks, Papers and Themes
------------------------

This year we have decided to borrow a few good ideas from PyCon, one of
which is to move away from the 'track' structure.  Instead, speakers are
invited to submit presentations about anything they have done that they
think would be of interest to the Python community.  We will then arrange
them into related groups and schedule them in the space available.  In the
past, EuroPython participants have found the following themes to be of
interest:

 * Science
 * Python Language and Libraries
 * Web Related Technologies
 * Education
 * Games
 * Agile Methodologies and Testing
 * Social Skills

In addition to talks, we will also accept full paper submissions about any
of the above themes.  The Call for Refereed Papers will be posted shortly.

The deadline for talk proposals is Friday 18th May at midnight (24:00
CEST, Central European Summer Time, UTC+2).

Other ways to participate
-------------------------

Apart from giving talks, there are plenty of other ways to participate in
the conference.  Just attending and talking to people you find here can be
satisfying enough, but there are three other kinds of activity you may wish
to plan for: Lightning Talks, Open Space and Sprints.  Lightning Talks are
very short talks that give you just enough time to introduce a topic or
project, Open Space is an area reserved for informal discussions, and
Sprints are focused gatherings for developers interested in particular
projects.  For more information please see the following pages:

 * Lightning Talks: http://www.europython.org/sections/events/lightning_talks
 * Open Space: http://www.europython.org/sections/events/open_space
 * Sprints: http://www.europython.org/sections/sprints_and_wiki

Your Contribution
-----------------

To propose a talk or a paper, go to...

 * http://www.europython.org/submit

For more general information on the conference, please visit...

 * http://www.europython.org/

Looking forward to seeing what you fine folk have been up to,

The EuroPython Team



From Andy.cheesman at bristol.ac.uk  Wed Apr  4 06:57:19 2007
From: Andy.cheesman at bristol.ac.uk (Andy Cheesman)
Date: Wed, 04 Apr 2007 11:57:19 +0100
Subject: [Numpy-discussion] Identification of neighbouring sites for a
	periodic numpy array
Message-ID: <4613848F.5090100@bris.ac.uk>

Hi people,

I was wondering if people could give me a pointer or two upon the
efficient identification of neighbouring sites for a given point upon a
numpy array which has periodic conditions.
Suggestions upon the web I've seen seem to use lots of loops which does
not seem to be the most economical method.

Sorry if this is muppet-ish post!

Andy


From Myhuong.Nguyen at eng.monash.edu.au  Wed Apr  4 21:33:38 2007
From: Myhuong.Nguyen at eng.monash.edu.au (Myhuong Nguyen)
Date: Thu, 05 Apr 2007 11:33:38 +1000
Subject: [Numpy-discussion] numpy-1.0.2 installation error
Message-ID: <f5955ea25f7b.5f7bf5955ea2@monash.edu.au>

Hi,
 
I have tried to install numpy-1.0.2 on my SGI (irix6.5) using command
"python setup.py install" and received the following error message:

File "numpy/core/setup.py", line 48, in generate_config_h
  raise "ERROR: Failed to test configuration"
ERROR: Failed to test configuration

What does the message mean? and how to fix it?

Many thanks in advance.

Myhuong Nguyen



From oliver.tomic at matforsk.no  Thu Apr  5 04:44:41 2007
From: oliver.tomic at matforsk.no (oliver.tomic at matforsk.no)
Date: Thu, 5 Apr 2007 10:44:41 +0200
Subject: [Numpy-discussion] [SciPy-user] Big list of Numpy & Scipy users
In-Reply-To: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>
Message-ID: <OFBA967CB9.8002F6ED-ONC12572B4.002FEC0C-C12572B4.002FE4B9@matforsk.no>

Hi list,

I've put out our project on the scipy project site.

Hopefully, this will become a very long list.

Oliver

scipy-user-bounces at scipy.org wrote on 05.04.2007 04:10:28:

> On 4/4/07, Robert Kern <robert.kern at gmail.com> wrote:
> > Bill Baxter wrote:
> > > Is there any place on the Wiki that lists all the known software that
> > > uses Numpy in some way?
> > >
> >> > It would be nice to start collecting such a list if there isn't one
> > > already.  Screenshots would be nice too.
> >
> > There is no such list that I know of, but you may start one on the
> wiki if you like.
>
> Ok, I made a start:  http://www.scipy.org/Scipy_Projects
> Anyone who has a project that depends on Numpy or Scipy, please go add
> your info there!
>
> I haven't linked it from anywhere, because it looks pretty pathetic
> right now with only three or four entries.  But hopefully everyone
> will jumps in and add their project to the list.
>
> Part of the idea is that this should be a good place to point
> nay-sayers to when they say "meh - numpy... that's a niche project for
> a handful of scientists."
>
> So ... hopefully a good portion of the links will be things other than
> science projects.  There will hopefully be a lot of things that
> "ordinary users" would care about.  :-)
>
> I couldn't figure out how to add an image, but if someone knows how to
> do that, please do.
>
> --bb
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user



From Chris.Barker at noaa.gov  Thu Apr  5 12:18:40 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 05 Apr 2007 09:18:40 -0700
Subject: [Numpy-discussion] question about standalone small software and
 teaching
In-Reply-To: <20070405062417.GC1370@clipper.ens.fr>
References: <461247B7.6020507@chimica.unige.it>
	<20070403132159.GC13747@clipper.ens.fr>
	<bc657ead0704041646j7be8abeeo232a2657cf064cb2@mail.gmail.com>
	<20070405062417.GC1370@clipper.ens.fr>
Message-ID: <46152160.8050007@noaa.gov>

Gael Varoquaux wrote:
> I have recently
> started avoided using class attributes when not necessary,

I agree. I use class attributes when I need, well, class attributes. 
That is an attribute that is shared by all the instances of the class.

In fact, in the example:

class A:
    x = 4

A_instance = A()

A_instance.x = 10

A.x is NOT the class attribute, it is now an instance attribute, which 
is found before the still existing class attribute A.x. Yes, the class 
attribute can serve as a default, but, I think, in a situation when you 
are intending the class attribute to be over-ridden by an instance 
attribute, then it's clearer to define it as an instance attribute in 
the first place:

class A:
     def __init___(self, ...)
         self.x = 4

Even though it's more typing.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From charlesr.harris at gmail.com  Thu Apr  5 12:22:22 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 5 Apr 2007 10:22:22 -0600
Subject: [Numpy-discussion] numpy-1.0.2 installation error
In-Reply-To: <f5955ea25f7b.5f7bf5955ea2@monash.edu.au>
References: <f5955ea25f7b.5f7bf5955ea2@monash.edu.au>
Message-ID: <e06186140704050922m663148c9l42a3b53d09aa2034@mail.gmail.com>

On 4/4/07, Myhuong Nguyen <Myhuong.Nguyen at eng.monash.edu.au> wrote:
>
> Hi,
>
> I have tried to install numpy-1.0.2 on my SGI (irix6.5) using command
> "python setup.py install" and received the following error message:
>
> File "numpy/core/setup.py", line 48, in generate_config_h
>   raise "ERROR: Failed to test configuration"
> ERROR: Failed to test configuration
>
> What does the message mean? and how to fix it?


This isn't an answer to your question, but another build error on
irix6.5was reported as ticket 417, so maybe you can help us get numpy
building on
those machines. What compiler/hardware are you using?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070405/e95520c1/attachment.html>

From oliphant at ee.byu.edu  Thu Apr  5 12:26:11 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 05 Apr 2007 10:26:11 -0600
Subject: [Numpy-discussion] numpy-1.0.2 installation error
In-Reply-To: <f5955ea25f7b.5f7bf5955ea2@monash.edu.au>
References: <f5955ea25f7b.5f7bf5955ea2@monash.edu.au>
Message-ID: <46152323.7060806@ee.byu.edu>

Myhuong Nguyen wrote:

>Hi,
> 
>I have tried to install numpy-1.0.2 on my SGI (irix6.5) using command
>"python setup.py install" and received the following error message:
>
>File "numpy/core/setup.py", line 48, in generate_config_h
>  raise "ERROR: Failed to test configuration"
>ERROR: Failed to test configuration
>
>What does the message mean? and how to fix it?
>  
>
It means that something is going wrong during the configuration stage.   
We need help getting numpy to build on your hardware. 

This means getting the right config.h file generated.  We can start by 
manually creating a config.h file that works for your platform and then 
figuring out how to get it written.

This configuration file just defines various variables depending on 
whether or not your platform (combination of hardware and compiler) has 
various functions, has various types defined, and what the sizes of 
different types are on your platform.

It usually takes somebody with that hardware to get it fixed.  Are you 
willing to help?

-Travis



From David.L.Goldsmith at noaa.gov  Thu Apr  5 12:43:47 2007
From: David.L.Goldsmith at noaa.gov (David L Goldsmith)
Date: Thu, 05 Apr 2007 09:43:47 -0700
Subject: [Numpy-discussion] Big list of Numpy & Scipy users
In-Reply-To: <bc657ead0704042149u5c0d7d1es5a4cca1c6e860f17@mail.gmail.com>
References: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>
	<bc657ead0704042149u5c0d7d1es5a4cca1c6e860f17@mail.gmail.com>
Message-ID: <46152743.9050400@noaa.gov>

Sebastian Haase wrote:
> Maybe the projects should be in categories:
> - open source
> - commercial   (?)
> - papers
> - ??
>   
I agree: I'm using numpy in two of my projects, but these are likely to 
have a small, specialized and self-contained user base indefinitely, so 
I'm a little "shy" about throwing them in to a general mix which 
includes large many-user projects and thereby making them seem, perhaps, 
"grander" than they are.  So at least one additional category: "small" 
projects.

DG
> -Sebastian
>
>
>
>
> On 4/4/07, Bill Baxter <wbaxter at gmail.com> wrote:
>   
>> On 4/4/07, Robert Kern <robert.kern at gmail.com> wrote:
>>     
>>> Bill Baxter wrote:
>>>       
>>>> Is there any place on the Wiki that lists all the known software that
>>>> uses Numpy in some way?
>>>>
>>>>         
>>>>> It would be nice to start collecting such a list if there isn't one
>>>>>           
>>>> already.  Screenshots would be nice too.
>>>>         
>>> There is no such list that I know of, but you may start one on the wiki if you like.
>>>       
>> Ok, I made a start:  http://www.scipy.org/Scipy_Projects
>> Anyone who has a project that depends on Numpy or Scipy, please go add
>> your info there!
>>
>> I haven't linked it from anywhere, because it looks pretty pathetic
>> right now with only three or four entries.  But hopefully everyone
>> will jumps in and add their project to the list.
>>
>> Part of the idea is that this should be a good place to point
>> nay-sayers to when they say "meh - numpy... that's a niche project for
>> a handful of scientists."
>>
>> So ... hopefully a good portion of the links will be things other than
>> science projects.  There will hopefully be a lot of things that
>> "ordinary users" would care about.  :-)
>>
>> I couldn't figure out how to add an image, but if someone knows how to
>> do that, please do.
>>
>> --bb
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>>     
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   


-- 
ERD/ORR/NOS/NOAA <http://response.restoration.noaa.gov/emergencyresponse/>


From robert.kern at gmail.com  Thu Apr  5 12:50:09 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 05 Apr 2007 11:50:09 -0500
Subject: [Numpy-discussion] F2PY and underscore problem
In-Reply-To: <ev2plp$vff$1@sea.gmane.org>
References: <ev2plp$vff$1@sea.gmane.org>
Message-ID: <461528C1.8070302@gmail.com>

Zden?k Hur?k wrote:
> I am sorry to come up with a question that probably has been asked and
> answered many times but I honestly searched for an answer but failed to
> find one.
> 
> I want to create Python modules from some Fortran numerical functions that
> rely on LAPACK. I start with a function SB02MD.f that provides a solver for
> algebraic Riccati equation (ARE) that can be obtained at
> http://www.slicot.de/. I proceeded the standard f2py way:
> 
> f2py SB02MD.f -m are -h are1.pyf
> 
> modified are1.pyf (saved as are.pyf) to tell which arguments are inputs and
> outputs and compiled the module:
> 
> f2py -c are.pyf SB02MD.f --link-lapack_opt
> 
> However, when trying to import the module, Python cries that there is and
> undefined symbol: sb02mw_. The same situation if I omit --link-lapack_opt.

What FORTRAN compiler are you using to build the module? What FORTRAN compiler
was used to build LAPACK?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From subscriber100 at rjs.org  Thu Apr  5 12:27:53 2007
From: subscriber100 at rjs.org (Ray S)
Date: Thu, 05 Apr 2007 09:27:53 -0700
Subject: [Numpy-discussion] Big list of Numpy & Scipy users
Message-ID: <6.2.3.4.2.20070405091600.059a7a58@blue-cove.com>

How's this, for under Applications?

CV-mini
The CV-mini is a signal analyzer written in Python, NumPy, and 
wxPython.
It is a high-speed, general-purpose, multi-channel FFT instrument 
available in 3 versions (so far).

Link: http://cognitivevision.com/CVmini.htm

Which reminds me, we should add Python & NumPy credits on the page!

The newest products do not have their pages up, yet...

I don't have a user account on the wiki, so
if: someone else wants to add it, please do.
else: I could sign up.



From aisaac at american.edu  Thu Apr  5 13:36:28 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 5 Apr 2007 13:36:28 -0400
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <46152160.8050007@noaa.gov>
References: <461247B7.6020507@chimica.unige.it><20070403132159.GC13747@clipper.ens.fr><bc657ead0704041646j7be8abeeo232a2657cf064cb2@mail.gmail.com><20070405062417.GC1370@clipper.ens.fr><46152160.8050007@noaa.gov>
Message-ID: <Mahogany-0.67.0-1068-20070405-133628.00@american.edu>

On Thu, 05 Apr 2007, Christopher Barker apparently wrote: 
> I think, in a situation when you are intending the class 
> attribute to be over-ridden by an instance attribute, then 
> it's clearer to define it as an instance attribute in the 
> first place: 

This is true of course.
BUT I am with Gael on this one:
when **introducing** newbies to their first class(es),
it can be useful not to introduce with the __init__ function.

Remember, for many perfectly intelligent people,
the phrases 'class object' and 'instance object'
are no more than 'n o i s e  object'.

Of course, shortly thereafter you should clear all this up.
But one step at a time when tutoring/teaching ...

Cheers,
Alan Isaac






From gael.varoquaux at normalesup.org  Thu Apr  5 13:39:18 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 5 Apr 2007 19:39:18 +0200
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <Mahogany-0.67.0-1068-20070405-133628.00@american.edu>
References: <46152160.8050007@noaa.gov>
	<Mahogany-0.67.0-1068-20070405-133628.00@american.edu>
Message-ID: <20070405173918.GC25579@clipper.ens.fr>

On Thu, Apr 05, 2007 at 01:36:28PM -0400, Alan G Isaac wrote:
> On Thu, 05 Apr 2007, Christopher Barker apparently wrote: 
> > I think, in a situation when you are intending the class 
> > attribute to be over-ridden by an instance attribute, then 
> > it's clearer to define it as an instance attribute in the 
> > first place: 

> This is true of course.
> BUT I am with Gael on this one:
> when **introducing** newbies to their first class(es),
> it can be useful not to introduce with the __init__ function.

> Remember, for many perfectly intelligent people,
> the phrases 'class object' and 'instance object'
> are no more than 'n o i s e  object'.

Actually I do it the other way around nowadays. I don't talk about class
object, instance object... I simply tell them to create attributes by
putting them in the __init__ . That way I simply avoid the explaination
until they actually need class attributes. By the time they need these
they are flying with their own wings.

When I would do it the other way around they would trip on the issue,
eventually, and than I would have to give the explaination, and this has
always been a difficult moment.

Ga?l


From hurak at fel.cvut.cz  Thu Apr  5 13:43:25 2007
From: hurak at fel.cvut.cz (=?UTF-8?B?WmRlbsSbayBIdXLDoWs=?=)
Date: Thu, 05 Apr 2007 19:43:25 +0200
Subject: [Numpy-discussion] F2PY and underscore problem
References: <ev2plp$vff$1@sea.gmane.org> <461528C1.8070302@gmail.com>
Message-ID: <ev3cft$d9d$1@sea.gmane.org>

Robert Kern wrote:

> Zden?k Hur?k wrote:
[...]
>> I want to create Python modules from some Fortran numerical functions
>> that rely on LAPACK. 
[...]
>> However, when trying to import the module, Python cries that there is and
>> undefined symbol: sb02mw_. 
[...]

> What FORTRAN compiler are you using to build the module? What FORTRAN
> compiler was used to build LAPACK?

Sorry for not giving this important detail. I use gcc-4.1.1 compiler suite,
which means that gfortran is used as a Fortran compiler. The same compiler
was used to build Lapack.

Zdenek Hurak



From strawman at astraw.com  Thu Apr  5 14:12:25 2007
From: strawman at astraw.com (Andrew Straw)
Date: Thu, 05 Apr 2007 11:12:25 -0700
Subject: [Numpy-discussion] Big list of Numpy & Scipy users
In-Reply-To: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>
References: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>
Message-ID: <46153C09.5020909@astraw.com>

Bill Baxter wrote:
> On 4/4/07, Robert Kern <robert.kern at gmail.com> wrote:
>   
>> Bill Baxter wrote:
>>     
>>> Is there any place on the Wiki that lists all the known software that
>>> uses Numpy in some way?
>>>
>>>       
>>>> It would be nice to start collecting such a list if there isn't one
>>>>         
>>> already.  Screenshots would be nice too.
>>>       
>> There is no such list that I know of, but you may start one on the wiki if you like.
>>     
>
> Ok, I made a start:  http://www.scipy.org/Scipy_Projects
>   
Great idea. I renamed the page to http://www.scipy.org/Projects so
Numpy-only users wouldn't feel excluded.


From Chris.Barker at noaa.gov  Thu Apr  5 14:32:25 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 05 Apr 2007 11:32:25 -0700
Subject: [Numpy-discussion] Big list of Numpy & Scipy users
In-Reply-To: <46152743.9050400@noaa.gov>
References: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>
	<bc657ead0704042149u5c0d7d1es5a4cca1c6e860f17@mail.gmail.com>
	<46152743.9050400@noaa.gov>
Message-ID: <461540B9.3080900@noaa.gov>

David L Goldsmith wrote:
> Sebastian Haase wrote:
>> Maybe the projects should be in categories:
>> - open source
>> - commercial   (?)
>> - papers

Maybe we need a section called "organizations" or something. We use 
numpy a lot around here, but, so far, only for small in-house tools that 
we can't point anyone to a web page or anything for.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Chris.Barker at noaa.gov  Thu Apr  5 15:04:43 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 05 Apr 2007 12:04:43 -0700
Subject: [Numpy-discussion] how to run the tests.
Message-ID: <4615484B.6000700@noaa.gov>

Just a quick comment.

I just built 1.0.2 on my OS-X box, and it took me entirely too long to 
figure out how to run the tests. I suggest something like:

""
after installing, to run the numpy unit tests, you can run:

import numpy
numpy.test()
"""

be added to the readme after the instructions on how to run setup.py 
install.

A question:
lapack_lite.so is linked against:
 
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate

Does that mean the Apple-supplied BLAS/LAPACK is being used?

-Chris

PS: it seems to be working fine with the Universal Framework build of 
Python2.5. I'll submit a package to be put on the pythonmac repository.


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From cookedm at physics.mcmaster.ca  Thu Apr  5 15:25:02 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu, 05 Apr 2007 15:25:02 -0400
Subject: [Numpy-discussion] how to run the tests.
In-Reply-To: <4615484B.6000700@noaa.gov>
References: <4615484B.6000700@noaa.gov>
Message-ID: <46154D0E.6050906@physics.mcmaster.ca>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christopher Barker wrote:
> Just a quick comment.
> 
> I just built 1.0.2 on my OS-X box, and it took me entirely too long to 
> figure out how to run the tests. I suggest something like:
> 
> ""
> after installing, to run the numpy unit tests, you can run:
> 
> import numpy
> numpy.test()
> """
> 
> be added to the readme after the instructions on how to run setup.py 
> install.

I use python -c 'import numpy; numpy.test()' for that. I've added a note
 to the README.

> A question:
> lapack_lite.so is linked against:
>  
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
> 
> Does that mean the Apple-supplied BLAS/LAPACK is being used?

Yes; the Accelerate framework exists on all installations of OS X, so we
can use it with no problems.

- --
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGFU0NN9ixZKFWjRQRAtcKAJ0QOu+A3MdjJfAbcmjyLJ3C2yI7SACeLQXa
dZOkiYWSa2kWZRoUJ/MEFxM=
=EoOZ
-----END PGP SIGNATURE-----


From robert.kern at gmail.com  Thu Apr  5 15:31:16 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 05 Apr 2007 14:31:16 -0500
Subject: [Numpy-discussion] Big list of Numpy & Scipy users
In-Reply-To: <461540B9.3080900@noaa.gov>
References: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>	<bc657ead0704042149u5c0d7d1es5a4cca1c6e860f17@mail.gmail.com>	<46152743.9050400@noaa.gov>
	<461540B9.3080900@noaa.gov>
Message-ID: <46154E84.2070402@gmail.com>

Christopher Barker wrote:
> David L Goldsmith wrote:
>> Sebastian Haase wrote:
>>> Maybe the projects should be in categories:
>>> - open source
>>> - commercial   (?)
>>> - papers
> 
> Maybe we need a section called "organizations" or something.

If you don't think you fit into any of the categories that are there, please,
everyone, feel free to make a new one. You don't need anyone's approval to put
content that you think is accurate onto the Wiki.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Thu Apr  5 15:32:55 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 05 Apr 2007 14:32:55 -0500
Subject: [Numpy-discussion] how to run the tests.
In-Reply-To: <4615484B.6000700@noaa.gov>
References: <4615484B.6000700@noaa.gov>
Message-ID: <46154EE7.5060402@gmail.com>

Christopher Barker wrote:
> A question:
> lapack_lite.so is linked against:
>  
> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
> 
> Does that mean the Apple-supplied BLAS/LAPACK is being used?

Yup. I haven't yet been able to build a Universal ATLAS library that works, so
we're stuck with Accelerate.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Thu Apr  5 15:59:30 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 05 Apr 2007 12:59:30 -0700
Subject: [Numpy-discussion] how to run the tests.
In-Reply-To: <46154EE7.5060402@gmail.com>
References: <4615484B.6000700@noaa.gov> <46154EE7.5060402@gmail.com>
Message-ID: <46155522.7000307@noaa.gov>

Robert Kern wrote:
> Yup. I haven't yet been able to build a Universal ATLAS library that works, so
> we're stuck with Accelerate.

Is that a bad thing? It seems like Apple is in the position to do it 
right, and have a differently tuned version for each of their hardware. 
If only MS would supply BLAS/LAPACK.

And aren't they using atlas anyway?

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From kwgoodman at gmail.com  Thu Apr  5 16:07:27 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 5 Apr 2007 13:07:27 -0700
Subject: [Numpy-discussion] how to run the tests.
In-Reply-To: <46155522.7000307@noaa.gov>
References: <4615484B.6000700@noaa.gov> <46154EE7.5060402@gmail.com>
	<46155522.7000307@noaa.gov>
Message-ID: <f4f93d420704051307s4d45c73asa1afdfc93d93a3f7@mail.gmail.com>

On 4/5/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> If only MS would supply BLAS/LAPACK.

Yeah, too bad more people don't use atlas. Then MS would embrace
atlas, extend it, and...I forget the last step.


From robert.kern at gmail.com  Thu Apr  5 16:12:42 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 05 Apr 2007 15:12:42 -0500
Subject: [Numpy-discussion] how to run the tests.
In-Reply-To: <46155522.7000307@noaa.gov>
References: <4615484B.6000700@noaa.gov> <46154EE7.5060402@gmail.com>
	<46155522.7000307@noaa.gov>
Message-ID: <4615583A.8000608@gmail.com>

Christopher Barker wrote:
> Robert Kern wrote:
>> Yup. I haven't yet been able to build a Universal ATLAS library that works, so
>> we're stuck with Accelerate.
> 
> Is that a bad thing? It seems like Apple is in the position to do it 
> right, and have a differently tuned version for each of their hardware. 
> If only MS would supply BLAS/LAPACK.
> 
> And aren't they using atlas anyway?

They are, but an older one. There have been significant improvements to the Core
Duo 2 code in ATLAS releases, but no updates to Accelerate.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From kwgoodman at gmail.com  Thu Apr  5 16:19:07 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 5 Apr 2007 13:19:07 -0700
Subject: [Numpy-discussion] how to run the tests.
In-Reply-To: <4615583A.8000608@gmail.com>
References: <4615484B.6000700@noaa.gov> <46154EE7.5060402@gmail.com>
	<46155522.7000307@noaa.gov> <4615583A.8000608@gmail.com>
Message-ID: <f4f93d420704051319v5302a7c3g1ba1160002a55eb5@mail.gmail.com>

On 4/5/07, Robert Kern <robert.kern at gmail.com> wrote:
> There have been significant improvements to the Core
> Duo 2 code in ATLAS releases [snip]

What kind of speed up are people seeing with Core 2 Duo aware ATLAS?


From aisaac at american.edu  Thu Apr  5 16:24:01 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 5 Apr 2007 16:24:01 -0400
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <20070405173918.GC25579@clipper.ens.fr>
References: <46152160.8050007@noaa.gov><Mahogany-0.67.0-1068-20070405-133628.00@american.edu><20070405173918.GC25579@clipper.ens.fr>
Message-ID: <Mahogany-0.67.0-1068-20070405-162401.00@american.edu>

On Thu, 5 Apr 2007, Gael Varoquaux apparently wrote: 
> Actually I do it the other way around nowadays. 

Except in the tutorial?
But anyway, I'm willing to try anything that gets them moving.
It is true that avoiding the appearance of complexity can
sometimes add complexity.

Cheers,
Alan Isaac






From robert.kern at gmail.com  Thu Apr  5 16:28:42 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 05 Apr 2007 15:28:42 -0500
Subject: [Numpy-discussion] how to run the tests.
In-Reply-To: <f4f93d420704051319v5302a7c3g1ba1160002a55eb5@mail.gmail.com>
References: <4615484B.6000700@noaa.gov>
	<46154EE7.5060402@gmail.com>	<46155522.7000307@noaa.gov>
	<4615583A.8000608@gmail.com>
	<f4f93d420704051319v5302a7c3g1ba1160002a55eb5@mail.gmail.com>
Message-ID: <46155BFA.80103@gmail.com>

Keith Goodman wrote:
> On 4/5/07, Robert Kern <robert.kern at gmail.com> wrote:
>> There have been significant improvements to the Core
>> Duo 2 code in ATLAS releases [snip]
> 
> What kind of speed up are people seeing with Core 2 Duo aware ATLAS?

>From the announcements on math-atlas-devel:

"""
3.7.17 is out. It's mainly for Core2Duo. I've got a new double precision
gemm kernel that bumps up performance to roughly 77% of peak for large
GEMM. This kernel also seems faster on Pentium-4 chips, but I haven't
updated any other architecture defaults. I've got a few more experiments
that might speed up the Core2Duo kernel, and I'll wait until the new kernel
is more set before updateing other defaults.

Anyway, if you have a Core2Duo, grab this release for a noticable double
precision speedup.
"""

3.7.19: """
Well, there's a couple of days I'll never see again. However, the new config
can finally (at least for me) successfully build 32/64 bit libs when you
override the compiler default on the configure line (using -b 32 or -b 64).
I think the last of the assembly needed to build the 64-bit Core2 defaults
under OS X have been adapted as well.
"""

"""
I've just released 3.7.29. The big deal is that it has new algorithms
(for real precisions only) that essentially doubles the performance (at least
on the Core2Duo, but should work many places) for long skinny matrices.
I.e., shapes like M=56, N=56, K=200000.
"""

"""
OK, it's been a long time coming, but 3.7.30 is out. In it, I have finished
adding the just-in-time copy techniqe to complex GEMM. Some long-K speedup
is possible on some platforms. On some architectures, asymptotic complex
GEMM performance goes up as well. Essentially, if you are on a platform where
real GEMM is faster than complex GEMM, there's a good chance you'll see a
speedup here, though you may have to recompute [c,z]Xover.h to make it
happen (I've updated only the Core2Duo [6% ZGEMM speedup] and Athlon-64
[no speedup] arch defs).
"""


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Thu Apr  5 16:32:20 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 05 Apr 2007 15:32:20 -0500
Subject: [Numpy-discussion] how to run the tests.
In-Reply-To: <f4f93d420704051319v5302a7c3g1ba1160002a55eb5@mail.gmail.com>
References: <4615484B.6000700@noaa.gov>
	<46154EE7.5060402@gmail.com>	<46155522.7000307@noaa.gov>
	<4615583A.8000608@gmail.com>
	<f4f93d420704051319v5302a7c3g1ba1160002a55eb5@mail.gmail.com>
Message-ID: <46155CD4.7080107@gmail.com>

Keith Goodman wrote:
> On 4/5/07, Robert Kern <robert.kern at gmail.com> wrote:
>> There have been significant improvements to the Core
>> Duo 2 code in ATLAS releases [snip]
> 
> What kind of speed up are people seeing with Core 2 Duo aware ATLAS?

Oh, and from the first release with Core 2 support:

"""
I just released 3.7.16. There's some misc stuff, but the big news is
that I have added architectural support for the Core2 family (though have
not tuned any assembly for it yet), and this guy is a beast! Looks like
after years of getting schooled by AMD, Intel finally got tired of it.

...

This means Core2 is the fastest floating point x86 ever produced. It ties
the G5, which was previously the best peak performer of any desktop system.
"""

So yeah, we're missing out.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Thu Apr  5 16:40:16 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 5 Apr 2007 22:40:16 +0200
Subject: [Numpy-discussion] question about standalone small software and
	teaching
In-Reply-To: <Mahogany-0.67.0-1068-20070405-162401.00@american.edu>
References: <20070405173918.GC25579@clipper.ens.fr>
	<Mahogany-0.67.0-1068-20070405-162401.00@american.edu>
Message-ID: <20070405204016.GA16867@clipper.ens.fr>

On Thu, Apr 05, 2007 at 04:24:01PM -0400, Alan G Isaac wrote:
> On Thu, 5 Apr 2007, Gael Varoquaux apparently wrote: 
> > Actually I do it the other way around nowadays. 

> Except in the tutorial?

Yes, shame on me, I changed policy after writing it. I guess I should
correct it. I'll add this to my TODO list.

Ga?l


From wbaxter at gmail.com  Thu Apr  5 17:06:17 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Fri, 6 Apr 2007 06:06:17 +0900
Subject: [Numpy-discussion] Identification of neighbouring sites for a
	periodic numpy array
In-Reply-To: <4613848F.5090100@bris.ac.uk>
References: <4613848F.5090100@bris.ac.uk>
Message-ID: <e86a5fd00704051406n239f2c1bj2014462c61250574@mail.gmail.com>

On 4/4/07, Andy Cheesman <Andy.cheesman at bristol.ac.uk> wrote:
> Hi people,
>
> I was wondering if people could give me a pointer or two upon the
> efficient identification of neighbouring sites for a given point upon a
> numpy array which has periodic conditions.
> Suggestions upon the web I've seen seem to use lots of loops which does
> not seem to be the most economical method.

Neighbors of x[i] in array x with length N are

    x[[i-1,i,(i+1)%N]]

The -1 is ok without a guard because x[-1] is the last element of a
list in python.

If that's not what you were after then you might need to be more
specific about the context you're talking about.

> Sorry if this is muppet-ish post!

I do not know what you mean.  You think you sound like Kermit the Frog
or Fozzie Bear?  You do not.  And I think we all know that it really
isn't easy being green.

:-)

--bb


From mkg at cs.nyu.edu  Thu Apr  5 18:00:24 2007
From: mkg at cs.nyu.edu (Matthew Koichi Grimes)
Date: Thu, 05 Apr 2007 18:00:24 -0400
Subject: [Numpy-discussion] inconsistent dtype promotion in dimensionless
	arrays
Message-ID: <46157178.8020309@cs.nyu.edu>

I've noticed two dtype promotion behaviors that are surprising to me. 
I'm hoping to get people's input as to whether I should file bug tickets 
on them.

First weirdness:
When you do an operation between a float32 Numpy array and a python or 
Numpy float64, the result is a float32 array, not a float64 array:

 >>> import numpy as N
 >>> vec32 = N.ones(2, N.float32)
 >>> vec32
array([ 1.,  1.], dtype=float32)
 >>> result = vec32 - 1.0
 >>> result.dtype
dtype('float32')
 >>> result = vec32 - N.float64(1.0)
 >>> result.dtype
dtype('float32')


This is of course not the case if the float64 is replaced with a 
1-dimensional numpy array:
 >>> result = vec32 - N.array([1.0], N.float64)
 >>> result.dtype
dtype('float64')


Second weirdness:
Type promotion doesn't happen if you replace the 1-d, 1-element array 
with a 0-d array:

 >>> result = vec32 - N.array(1.0, N.float64)
 >>> result.dtype
dtype('float32')


The second weirdness in particular strikes me as inconsistent when 
compared to operations between vec32 and the 1-d array.

I didn't see these listed on the trac site as bugs, but then again a 
bunch of tickets seem to have been removed/fixed recently. Sorry if 
these have been mentioned, filed, and fixed already. Otherwise, should I 
file a ticket on the above behaviors?

-- Matt


From robert.kern at gmail.com  Thu Apr  5 18:07:29 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 05 Apr 2007 17:07:29 -0500
Subject: [Numpy-discussion] inconsistent dtype promotion in
 dimensionless arrays
In-Reply-To: <46157178.8020309@cs.nyu.edu>
References: <46157178.8020309@cs.nyu.edu>
Message-ID: <46157321.7040006@gmail.com>

Matthew Koichi Grimes wrote:
> I've noticed two dtype promotion behaviors that are surprising to me. 
> I'm hoping to get people's input as to whether I should file bug tickets 
> on them.
> 
> First weirdness:
> When you do an operation between a float32 Numpy array and a python or 
> Numpy float64, the result is a float32 array, not a float64 array:
> 
>  >>> import numpy as N
>  >>> vec32 = N.ones(2, N.float32)
>  >>> vec32
> array([ 1.,  1.], dtype=float32)
>  >>> result = vec32 - 1.0
>  >>> result.dtype
> dtype('float32')
>  >>> result = vec32 - N.float64(1.0)
>  >>> result.dtype
> dtype('float32')
> 
> 
> This is of course not the case if the float64 is replaced with a 
> 1-dimensional numpy array:
>  >>> result = vec32 - N.array([1.0], N.float64)
>  >>> result.dtype
> dtype('float64')

This is actually expected behavior. The rule is that when an array is operated
with a scalar and the dtypes are of the same kind (floating point, integer,
complex are all different "kinds"; float32 and float64 are of the same kind,
float32 and int32 are not), then the array's dtype takes precedence. This helps
resolve long-standing issues of using float32 arrays and still multiplying them
by 2.0, for example. When the two are of different kinds (float32 and int32, for
example), the common "safe" dtype is negotiated between the two.

> Second weirdness:
> Type promotion doesn't happen if you replace the 1-d, 1-element array 
> with a 0-d array:
> 
>  >>> result = vec32 - N.array(1.0, N.float64)
>  >>> result.dtype
> dtype('float32')
> 
> 
> The second weirdness in particular strikes me as inconsistent when 
> compared to operations between vec32 and the 1-d array.

Rank-0 arrays are treated as scalars in ufuncs, too, so these are consistent.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From oliphant at ee.byu.edu  Thu Apr  5 18:11:06 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu, 05 Apr 2007 16:11:06 -0600
Subject: [Numpy-discussion] inconsistent dtype promotion in
 dimensionless arrays
In-Reply-To: <46157178.8020309@cs.nyu.edu>
References: <46157178.8020309@cs.nyu.edu>
Message-ID: <461573FA.9050607@ee.byu.edu>

Matthew Koichi Grimes wrote:

>I've noticed two dtype promotion behaviors that are surprising to me. 
>I'm hoping to get people's input as to whether I should file bug tickets 
>on them.
>  
>

Short answer:  No, they are not bugs.

The rule is:

In any mixed-type operation between two objects of the same fundamental 
"kind" (i.e. integer, float, complex) arrays always have precedence over 
"scalars"  (where a 0-d array is considered a scalar in this context).

It's a rule whose reason has a long history.


-Travis



From topengineer at gmail.com  Fri Apr  6 03:15:32 2007
From: topengineer at gmail.com (Hui Chang Moon)
Date: Fri, 6 Apr 2007 16:15:32 +0900
Subject: [Numpy-discussion] I can't know where I update the code...
Message-ID: <296323b50704060015k6d941c62xe4095cf3961b2ca6@mail.gmail.com>

Warning (from warnings module):
  File "C:\Python25\lib\site-packages\numpy\testing\numpytest.py", line 634
    DeprecationWarning)
DeprecationWarning: ScipyTest is now called NumpyTest; please update your
code

I have written the simulation code with numpy 1.0.1 & scipy 0.52.
I installed the numpy 1.0.2, and the run the code,
above message is seen.

I don't use ScipyTest in my code, so I can't update my code.

What will I do?

Please help, ~

Moon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070406/8282b30e/attachment.html>

From ceball at gmail.com  Fri Apr  6 06:40:27 2007
From: ceball at gmail.com (Chris)
Date: Fri, 6 Apr 2007 10:40:27 +0000 (UTC)
Subject: [Numpy-discussion] I can't know where I update the code...
References: <296323b50704060015k6d941c62xe4095cf3961b2ca6@mail.gmail.com>
Message-ID: <loom.20070406T123759-57@post.gmane.org>


> Warning (from warnings module):? File "C:\Python25\lib\site-
packages\numpy\testing\numpytest.py", line 634??? DeprecationWarning)
DeprecationWarning: ScipyTest is now called NumpyTest; please update your code


Sorry, this is not an answer, but it might help someone else to answer your 
question: importing weave (using numpy-1.0.2) is one way to get this error 
message.

Chris



From ceball at gmail.com  Fri Apr  6 08:56:01 2007
From: ceball at gmail.com (Chris)
Date: Fri, 6 Apr 2007 12:56:01 +0000 (UTC)
Subject: [Numpy-discussion] I can't know where I update the code...
References: <296323b50704060015k6d941c62xe4095cf3961b2ca6@mail.gmail.com>
	<loom.20070406T123759-57@post.gmane.org>
Message-ID: <loom.20070406T144245-811@post.gmane.org>


> Sorry, this is not an answer, but it might help someone else to answer your 
> question: importing weave (using numpy-1.0.2) is one way to get this error 
> message.

Is this something we should post on the scipy-dev mailing list instead? (Maybe 
there are other parts of scipy that need to be updated? Or maybe this has 
already been updated on the current svn version of scipy?)

Thanks,
Chris



From sdave at ufl.edu  Fri Apr  6 10:10:29 2007
From: sdave at ufl.edu (David Shepherd)
Date: Fri, 06 Apr 2007 10:10:29 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
Message-ID: <461654D5.401@ufl.edu>

Hey all,

We started to try and compile the numpy module on an embedded PowerPC 
Xilinx board with no luck.  This is one of the errors I get when I try 
to build the module on-board.  It is due to the fact that the compiler 
is located in a different location than the original compiler for python 
(i think).  I made symbolic links to fix the first error.  The second 
more critical error that I cannot recover from is the "Error: 
Unrecognized opcode: `fldenv'".  There are many more duplicates of this 
error, but it occurs when the "umathmodule.c" is being compiled.

I have tried changing the CFLAGS multiple times with no luck.  Python 
was compiled with uclibc libraries, so could this be the problem?  Is 
there any way we could compile Python and numpy together in one step?

I am using Binutils 2.17, GCC 3.4.6, and Python 2.5.  Again, python was 
compiled with uclibc, not the standard libraries.  We require the FFT 
module for a project that is due in about a week.  Any help would be 
appreciated.

-David Shepherd


C compiler: 
/opt/automated-build/buildroot/build_powerpc/staging_dir/bin/powerpc
-linux-uclibc-gcc -DNDEBUG -Os -pipe -fPIC

compile options: '-I/usr/include/python2.5 -Inumpy/core/src 
-Inumpy/core/include
  -I/usr/include/python2.5 -c'
powerpc-linux-uclibc-gcc: _configtest.c
sh: 
/opt/automated-build/buildroot/build_powerpc/staging_dir/bin/powerpc-linux-u
clibc-gcc: not found
sh: 
/opt/automated-build/buildroot/build_powerpc/staging_dir/bin/powerpc-linux-u
clibc-gcc: not found
failure.

I made symbolic links to fix the above errors, but then I also get 
multiple assembler opcode errors when it tries to run the entry build:

powerpc-linux-gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC 
-Ibuild/src.linux-ppc-2.5/numpy/core/src -Inumpy/core/include 
-Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
-Inumpy/core/include -I/root/python/include/python2.5 -c 
build/src.linux-ppc-2.5/numpy/core/src/umathmodule.c -o 
build/temp.linux-ppc-2.5/build/src.linux-ppc-2.5/numpy/core/src/umathmodule.o
.
.
.
.
Error: Unrecognized opcode: `fldenv'
.
.
.


From cookedm at physics.mcmaster.ca  Fri Apr  6 11:23:04 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Fri, 06 Apr 2007 11:23:04 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
In-Reply-To: <461654D5.401@ufl.edu>
References: <461654D5.401@ufl.edu>
Message-ID: <461665D8.8010408@physics.mcmaster.ca>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Shepherd wrote:
> Hey all,
> 
> We started to try and compile the numpy module on an embedded PowerPC 
> Xilinx board with no luck.  This is one of the errors I get when I try 
> to build the module on-board.  It is due to the fact that the compiler 
> is located in a different location than the original compiler for python 
> (i think).  I made symbolic links to fix the first error.  The second 
> more critical error that I cannot recover from is the "Error: 
> Unrecognized opcode: `fldenv'".  There are many more duplicates of this 
> error, but it occurs when the "umathmodule.c" is being compiled.

Setting CC to your C compiler should work

> I am using Binutils 2.17, GCC 3.4.6, and Python 2.5.  Again, python was 
> compiled with uclibc, not the standard libraries.  We require the FFT 
> module for a project that is due in about a week.  Any help would be 
> appreciated.

> multiple assembler opcode errors when it tries to run the entry build:
> 
> powerpc-linux-gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC 
> -Ibuild/src.linux-ppc-2.5/numpy/core/src -Inumpy/core/include 
> -Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
> -Inumpy/core/include -I/root/python/include/python2.5 -c 
> build/src.linux-ppc-2.5/numpy/core/src/umathmodule.c -o 
> build/temp.linux-ppc-2.5/build/src.linux-ppc-2.5/numpy/core/src/umathmodule.o
> .
> .
> .
> .
> Error: Unrecognized opcode: `fldenv'

The culprit looks like numpy/core/include/numpy/fenv/fenv.h, which is
odd, as I don't see how it would be included -- it's only used for
cygwin. I would think there would be no floating point environment
support included, as the system <fenv.h> is only included when one of
__GLIBC__, __APPLE__, or __MINGW32__ is defined.

See if the attached patch helps.

- --
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGFmXYN9ixZKFWjRQRAu23AJ44AslsO5HxqDiVuLjYTzI59Dpt4wCgnBOY
2zEHO9XAQVDVTLtWS7xhWpA=
=Hag4
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: uclibc-fenv.patch
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070406/df1e531f/attachment.ksh>

From Chris.Barker at noaa.gov  Fri Apr  6 12:11:08 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 06 Apr 2007 09:11:08 -0700
Subject: [Numpy-discussion] how to run the tests.
In-Reply-To: <4615583A.8000608@gmail.com>
References: <4615484B.6000700@noaa.gov> <46154EE7.5060402@gmail.com>
	<46155522.7000307@noaa.gov> <4615583A.8000608@gmail.com>
Message-ID: <4616711C.9080800@noaa.gov>

Robert Kern wrote:
> They are, but an older one. There have been significant improvements to the Core
> Duo 2 code in ATLAS releases, but no updates to Accelerate.

Let's hope they'll be forthcoming.

Thanks for the update. This may be important to me soon.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From giorgio at gilestro.tk  Fri Apr  6 12:14:20 2007
From: giorgio at gilestro.tk (Giorgio F. Gilestro)
Date: Fri, 6 Apr 2007 11:14:20 -0500
Subject: [Numpy-discussion] problem reading binary data from file
Message-ID: <212906f40704060914q319dcba4sbb6532e496b1806a@mail.gmail.com>

Hello everyone.

Here I go with my first problem for this ml!

I am reading a long sequence of binary data from a file using a call
similar to the following

numpy.core.records.fromfile (filename, formats='i2', byteorder='big')

My problem is that this function returns an array of tuples rather
than an array of actual values.

Basically I get something like this: [(45,) (29,) (23,)....(21,) (20,) (15,)]
While I would obviously prefer something like this: [45, 29, 23,
....., 21, 20, 15]
I can pass from one to the other through an iteration but this is
definitely time expensive.
I also know that if i specify the formats to be ['i2']*N I will get a
tuple of length N but that is not really feasible when my N is 5 *10^5

Is there any way I could get a 1D array (no tuples please!) directly
from the file?
(BTW numpy.core.records.fromstring gives the same output)

DOCS here:
http://www.scipy.org/doc/numpy_api_docs/numpy.core.records.html#fromfile


From pgmdevlist at gmail.com  Fri Apr  6 12:41:01 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 6 Apr 2007 12:41:01 -0400
Subject: [Numpy-discussion] problem reading binary data from file
In-Reply-To: <212906f40704060914q319dcba4sbb6532e496b1806a@mail.gmail.com>
References: <212906f40704060914q319dcba4sbb6532e496b1806a@mail.gmail.com>
Message-ID: <200704061241.02228.pgmdevlist@gmail.com>

On Friday 06 April 2007 12:14:20 Giorgio F. Gilestro wrote:

> Is there any way I could get a 1D array (no tuples please!) directly
> from the file?
> (BTW numpy.core.records.fromstring gives the same output)

numpy.core.records.fromfile (filename, formats='i2', byteorder='big')['f0']

With numpy.core.records, you create a record array. Just get the field ('f0' 
by default) to get a ndarray.

(I'm sure there's a simpler way, but that should do it for now).


From oliphant at ee.byu.edu  Fri Apr  6 12:46:08 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri, 06 Apr 2007 10:46:08 -0600
Subject: [Numpy-discussion] problem reading binary data from file
In-Reply-To: <212906f40704060914q319dcba4sbb6532e496b1806a@mail.gmail.com>
References: <212906f40704060914q319dcba4sbb6532e496b1806a@mail.gmail.com>
Message-ID: <46167950.5040505@ee.byu.edu>

Giorgio F. Gilestro wrote:

>Hello everyone.
>
>Here I go with my first problem for this ml!
>
>I am reading a long sequence of binary data from a file using a call
>similar to the following
>
>numpy.core.records.fromfile (filename, formats='i2', byteorder='big')
>
>My problem is that this function returns an array of tuples rather
>  
>
>than an array of actual values.
>  
>

You aren't getting an array of "tuples", but an array of "records."   
Each element of a record-array just displays as a tuple.

It sounds like you just want a regular array.  If that is the case, I'm 
not sure why you are using the records sub-class at all.   I'm also not 
clear on your use of the byteorder argument.  Are you reading big-endian 
data on a little-endian machine?  If you are always reading an writing 
in native format, then you don't need to specify the byte-order.

Once you have an array of records, you can easily view this as another 
kind of array.  Just use the .view method.  This does not copy any data 
but just gives you a different way of "looking" at the data.

In your case, a.view('i2') should do it.


Also, numpy.rec 

is short for numpy.core.records

-Travis



From Chris.Barker at noaa.gov  Fri Apr  6 12:54:09 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 06 Apr 2007 09:54:09 -0700
Subject: [Numpy-discussion] problem reading binary data from file
In-Reply-To: <200704061241.02228.pgmdevlist@gmail.com>
References: <212906f40704060914q319dcba4sbb6532e496b1806a@mail.gmail.com>
	<200704061241.02228.pgmdevlist@gmail.com>
Message-ID: <46167B31.8000207@noaa.gov>

Pierre GM wrote:
> With numpy.core.records, you create a record array. Just get the field ('f0' 
> by default) to get a ndarray.

there is. Don't use numpy.core.records. Use numpy.fromfile:

import numpy

a = numpy.fromfile(filename, dtype=numpy.int16)

(filename can be a filename, or an open file object)

You may want to get fancier with the dtype to force the correct byte order.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From sdave at ufl.edu  Fri Apr  6 13:54:58 2007
From: sdave at ufl.edu (David Shepherd)
Date: Fri, 06 Apr 2007 13:54:58 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
In-Reply-To: <461665D8.8010408@physics.mcmaster.ca>
References: <461654D5.401@ufl.edu> <461665D8.8010408@physics.mcmaster.ca>
Message-ID: <46168972.6080208@ufl.edu>

Well, I get further into the compile process, but now it fails on 
another gcc compile operations.  This is essentially the same error 
message I was getting before.  It tries to #include <fenv.h>, but does 
not find it (it shouldn't be including it anyway, as you said).  This 
time it fails when trying to compile "_capi.c".  Thanks for all your 
help so far!

Dave Shepherd

Here is the error output:

creating build/temp.linux-ppc-2.5/numpy/lib
creating build/temp.linux-ppc-2.5/numpy/lib/src
compile options: '-Inumpy/core/include 
-Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
-Inumpy/core/include -I/usr/local/include/python2.5 -c'
gcc: numpy/lib/src/_compiled_base.c
gcc -pthread -shared 
build/temp.linux-ppc-2.5/numpy/lib/src/_compiled_base.o -o 
build/lib.linux-ppc-2.5/numpy/lib/_compiled_base.so
building 'numpy.numarray._capi' extension
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -mhard-float -DNDEBUG -g 
-O3 -Wall -Wstrict-prototypes -fPIC

creating build/temp.linux-ppc-2.5/numpy/numarray
compile options: '-Inumpy/core/include 
-Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
-Inumpy/core/include -I/usr/local/include/python2.5 -c'
gcc: numpy/numarray/_capi.c
numpy/numarray/_capi.c:228:18: fenv.h: No such file or directory
numpy/numarray/_capi.c: In function `int_overflow_error':
numpy/numarray/_capi.c:230: warning: implicit declaration of function 
`feraiseexcept'
numpy/numarray/_capi.c:230: error: `FE_OVERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:230: error: (Each undeclared identifier is 
reported only once
numpy/numarray/_capi.c:230: error: for each function it appears in.)
numpy/numarray/_capi.c: In function `NA_checkFPErrors':
numpy/numarray/_capi.c:2947: warning: implicit declaration of function 
`fetestexcept'
numpy/numarray/_capi.c:2948: error: `FE_DIVBYZERO' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2948: error: `FE_OVERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2948: error: `FE_UNDERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2948: error: `FE_INVALID' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2954: warning: implicit declaration of function 
`feclearexcept'
numpy/numarray/_capi.c:228:18: fenv.h: No such file or directory
numpy/numarray/_capi.c: In function `int_overflow_error':
numpy/numarray/_capi.c:230: warning: implicit declaration of function 
`feraiseexcept'
numpy/numarray/_capi.c:230: error: `FE_OVERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:230: error: (Each undeclared identifier is 
reported only once
numpy/numarray/_capi.c:230: error: for each function it appears in.)
numpy/numarray/_capi.c: In function `NA_checkFPErrors':
numpy/numarray/_capi.c:2947: warning: implicit declaration of function 
`fetestexcept'
numpy/numarray/_capi.c:2948: error: `FE_DIVBYZERO' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2948: error: `FE_OVERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2948: error: `FE_UNDERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2948: error: `FE_INVALID' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2954: warning: implicit declaration of function 
`feclearexcept'
error: Command "gcc -pthread -fno-strict-aliasing -mhard-float -DNDEBUG 
-g -O3 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include 
-Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
-Inumpy/core/include -I/usr/local/include/python2.5 -c 
numpy/numarray/_capi.c -o 
build/temp.linux-ppc-2.5/numpy/numarray/_capi.o" failed with exit status 1
#

David M. Cooke wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> David Shepherd wrote:
>> Hey all,
>>
>> We started to try and compile the numpy module on an embedded PowerPC 
>> Xilinx board with no luck.  This is one of the errors I get when I try 
>> to build the module on-board.  It is due to the fact that the compiler 
>> is located in a different location than the original compiler for python 
>> (i think).  I made symbolic links to fix the first error.  The second 
>> more critical error that I cannot recover from is the "Error: 
>> Unrecognized opcode: `fldenv'".  There are many more duplicates of this 
>> error, but it occurs when the "umathmodule.c" is being compiled.
> 
> Setting CC to your C compiler should work
> 
>> I am using Binutils 2.17, GCC 3.4.6, and Python 2.5.  Again, python was 
>> compiled with uclibc, not the standard libraries.  We require the FFT 
>> module for a project that is due in about a week.  Any help would be 
>> appreciated.
> 
>> multiple assembler opcode errors when it tries to run the entry build:
>>
>> powerpc-linux-gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC 
>> -Ibuild/src.linux-ppc-2.5/numpy/core/src -Inumpy/core/include 
>> -Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
>> -Inumpy/core/include -I/root/python/include/python2.5 -c 
>> build/src.linux-ppc-2.5/numpy/core/src/umathmodule.c -o 
>> build/temp.linux-ppc-2.5/build/src.linux-ppc-2.5/numpy/core/src/umathmodule.o
>> .
>> .
>> .
>> .
>> Error: Unrecognized opcode: `fldenv'
> 
> The culprit looks like numpy/core/include/numpy/fenv/fenv.h, which is
> odd, as I don't see how it would be included -- it's only used for
> cygwin. I would think there would be no floating point environment
> support included, as the system <fenv.h> is only included when one of
> __GLIBC__, __APPLE__, or __MINGW32__ is defined.
> 
> See if the attached patch helps.
> 
> - --
> |>|\/|<
> /------------------------------------------------------------------\
> |David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
> |cookedm at physics.mcmaster.ca
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iD8DBQFGFmXYN9ixZKFWjRQRAu23AJ44AslsO5HxqDiVuLjYTzI59Dpt4wCgnBOY
> 2zEHO9XAQVDVTLtWS7xhWpA=
> =Hag4
> -----END PGP SIGNATURE-----
> 
> 
> ------------------------------------------------------------------------
> 
> Index: numpy/core/include/numpy/ufuncobject.h
> ===================================================================
> --- numpy/core/include/numpy/ufuncobject.h	(revision 3673)
> +++ numpy/core/include/numpy/ufuncobject.h	(working copy)
> @@ -276,6 +276,13 @@
>  	(void) fpsetsticky(0);						\
>  	}
>  
> +#elif defined(__UCLIBC__)
> +
> +#define NO_FLOATING_POINT_SUPPORT
> +#define UFUNC_CHECK_STATUS(ret) { \
> +    ret = 0;                      \
> +  }
> +
>  #elif defined(linux) || defined(__APPLE__) || defined(__CYGWIN__) || defined(__MINGW32__)
>  
>  #if defined(__GLIBC__) || defined(__APPLE__) || defined(__MINGW32__)
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


From sdave at ufl.edu  Fri Apr  6 14:10:03 2007
From: sdave at ufl.edu (David Shepherd)
Date: Fri, 06 Apr 2007 14:10:03 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
In-Reply-To: <46168972.6080208@ufl.edu>
References: <461654D5.401@ufl.edu> <461665D8.8010408@physics.mcmaster.ca>
	<46168972.6080208@ufl.edu>
Message-ID: <46168CFB.1050102@ufl.edu>

You know, looking at the core it looks like it has something to do with 
the "define(linux)" statement.  I am running linux, but its a buildroot 
(uclibc) rootfs.  Let me know if you need anymore information.  Thanks.

Dave

David Shepherd wrote:
> Well, I get further into the compile process, but now it fails on 
> another gcc compile operations.  This is essentially the same error 
> message I was getting before.  It tries to #include <fenv.h>, but does 
> not find it (it shouldn't be including it anyway, as you said).  This 
> time it fails when trying to compile "_capi.c".  Thanks for all your 
> help so far!
> 
> Dave Shepherd
> 
> Here is the error output:
> 
> creating build/temp.linux-ppc-2.5/numpy/lib
> creating build/temp.linux-ppc-2.5/numpy/lib/src
> compile options: '-Inumpy/core/include 
> -Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
> -Inumpy/core/include -I/usr/local/include/python2.5 -c'
> gcc: numpy/lib/src/_compiled_base.c
> gcc -pthread -shared 
> build/temp.linux-ppc-2.5/numpy/lib/src/_compiled_base.o -o 
> build/lib.linux-ppc-2.5/numpy/lib/_compiled_base.so
> building 'numpy.numarray._capi' extension
> compiling C sources
> C compiler: gcc -pthread -fno-strict-aliasing -mhard-float -DNDEBUG -g 
> -O3 -Wall -Wstrict-prototypes -fPIC
> 
> creating build/temp.linux-ppc-2.5/numpy/numarray
> compile options: '-Inumpy/core/include 
> -Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
> -Inumpy/core/include -I/usr/local/include/python2.5 -c'
> gcc: numpy/numarray/_capi.c
> numpy/numarray/_capi.c:228:18: fenv.h: No such file or directory
> numpy/numarray/_capi.c: In function `int_overflow_error':
> numpy/numarray/_capi.c:230: warning: implicit declaration of function 
> `feraiseexcept'
> numpy/numarray/_capi.c:230: error: `FE_OVERFLOW' undeclared (first use 
> in this function)
> numpy/numarray/_capi.c:230: error: (Each undeclared identifier is 
> reported only once
> numpy/numarray/_capi.c:230: error: for each function it appears in.)
> numpy/numarray/_capi.c: In function `NA_checkFPErrors':
> numpy/numarray/_capi.c:2947: warning: implicit declaration of function 
> `fetestexcept'
> numpy/numarray/_capi.c:2948: error: `FE_DIVBYZERO' undeclared (first use 
> in this function)
> numpy/numarray/_capi.c:2948: error: `FE_OVERFLOW' undeclared (first use 
> in this function)
> numpy/numarray/_capi.c:2948: error: `FE_UNDERFLOW' undeclared (first use 
> in this function)
> numpy/numarray/_capi.c:2948: error: `FE_INVALID' undeclared (first use 
> in this function)
> numpy/numarray/_capi.c:2954: warning: implicit declaration of function 
> `feclearexcept'
> numpy/numarray/_capi.c:228:18: fenv.h: No such file or directory
> numpy/numarray/_capi.c: In function `int_overflow_error':
> numpy/numarray/_capi.c:230: warning: implicit declaration of function 
> `feraiseexcept'
> numpy/numarray/_capi.c:230: error: `FE_OVERFLOW' undeclared (first use 
> in this function)
> numpy/numarray/_capi.c:230: error: (Each undeclared identifier is 
> reported only once
> numpy/numarray/_capi.c:230: error: for each function it appears in.)
> numpy/numarray/_capi.c: In function `NA_checkFPErrors':
> numpy/numarray/_capi.c:2947: warning: implicit declaration of function 
> `fetestexcept'
> numpy/numarray/_capi.c:2948: error: `FE_DIVBYZERO' undeclared (first use 
> in this function)
> numpy/numarray/_capi.c:2948: error: `FE_OVERFLOW' undeclared (first use 
> in this function)
> numpy/numarray/_capi.c:2948: error: `FE_UNDERFLOW' undeclared (first use 
> in this function)
> numpy/numarray/_capi.c:2948: error: `FE_INVALID' undeclared (first use 
> in this function)
> numpy/numarray/_capi.c:2954: warning: implicit declaration of function 
> `feclearexcept'
> error: Command "gcc -pthread -fno-strict-aliasing -mhard-float -DNDEBUG 
> -g -O3 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include 
> -Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
> -Inumpy/core/include -I/usr/local/include/python2.5 -c 
> numpy/numarray/_capi.c -o 
> build/temp.linux-ppc-2.5/numpy/numarray/_capi.o" failed with exit status 1
> #
> 
> David M. Cooke wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> David Shepherd wrote:
>>> Hey all,
>>>
>>> We started to try and compile the numpy module on an embedded PowerPC 
>>> Xilinx board with no luck.  This is one of the errors I get when I try 
>>> to build the module on-board.  It is due to the fact that the compiler 
>>> is located in a different location than the original compiler for python 
>>> (i think).  I made symbolic links to fix the first error.  The second 
>>> more critical error that I cannot recover from is the "Error: 
>>> Unrecognized opcode: `fldenv'".  There are many more duplicates of this 
>>> error, but it occurs when the "umathmodule.c" is being compiled.
>> Setting CC to your C compiler should work
>>
>>> I am using Binutils 2.17, GCC 3.4.6, and Python 2.5.  Again, python was 
>>> compiled with uclibc, not the standard libraries.  We require the FFT 
>>> module for a project that is due in about a week.  Any help would be 
>>> appreciated.
>>> multiple assembler opcode errors when it tries to run the entry build:
>>>
>>> powerpc-linux-gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC 
>>> -Ibuild/src.linux-ppc-2.5/numpy/core/src -Inumpy/core/include 
>>> -Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
>>> -Inumpy/core/include -I/root/python/include/python2.5 -c 
>>> build/src.linux-ppc-2.5/numpy/core/src/umathmodule.c -o 
>>> build/temp.linux-ppc-2.5/build/src.linux-ppc-2.5/numpy/core/src/umathmodule.o
>>> .
>>> .
>>> .
>>> .
>>> Error: Unrecognized opcode: `fldenv'
>> The culprit looks like numpy/core/include/numpy/fenv/fenv.h, which is
>> odd, as I don't see how it would be included -- it's only used for
>> cygwin. I would think there would be no floating point environment
>> support included, as the system <fenv.h> is only included when one of
>> __GLIBC__, __APPLE__, or __MINGW32__ is defined.
>>
>> See if the attached patch helps.
>>
>> - --
>> |>|\/|<
>> /------------------------------------------------------------------\
>> |David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
>> |cookedm at physics.mcmaster.ca
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.6 (Darwin)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>>
>> iD8DBQFGFmXYN9ixZKFWjRQRAu23AJ44AslsO5HxqDiVuLjYTzI59Dpt4wCgnBOY
>> 2zEHO9XAQVDVTLtWS7xhWpA=
>> =Hag4
>> -----END PGP SIGNATURE-----
>>
>>
>> ------------------------------------------------------------------------
>>
>> Index: numpy/core/include/numpy/ufuncobject.h
>> ===================================================================
>> --- numpy/core/include/numpy/ufuncobject.h	(revision 3673)
>> +++ numpy/core/include/numpy/ufuncobject.h	(working copy)
>> @@ -276,6 +276,13 @@
>>  	(void) fpsetsticky(0);						\
>>  	}
>>  
>> +#elif defined(__UCLIBC__)
>> +
>> +#define NO_FLOATING_POINT_SUPPORT
>> +#define UFUNC_CHECK_STATUS(ret) { \
>> +    ret = 0;                      \
>> +  }
>> +
>>  #elif defined(linux) || defined(__APPLE__) || defined(__CYGWIN__) || defined(__MINGW32__)
>>  
>>  #if defined(__GLIBC__) || defined(__APPLE__) || defined(__MINGW32__)
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 


From cookedm at physics.mcmaster.ca  Fri Apr  6 14:18:55 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Fri, 06 Apr 2007 14:18:55 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
In-Reply-To: <46168CFB.1050102@ufl.edu>
References: <461654D5.401@ufl.edu>
	<461665D8.8010408@physics.mcmaster.ca>	<46168972.6080208@ufl.edu>
	<46168CFB.1050102@ufl.edu>
Message-ID: <46168F0F.5070908@physics.mcmaster.ca>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Shepherd wrote:
> You know, looking at the core it looks like it has something to do with 
> the "define(linux)" statement.  I am running linux, but its a buildroot 
> (uclibc) rootfs.  Let me know if you need anymore information.  Thanks.
> 
> Dave
> 
> David Shepherd wrote:
>> Well, I get further into the compile process, but now it fails on 
>> another gcc compile operations.  This is essentially the same error 
>> message I was getting before.  It tries to #include <fenv.h>, but does 
>> not find it (it shouldn't be including it anyway, as you said).  This 
>> time it fails when trying to compile "_capi.c".  Thanks for all your 
>> help so far!

Try this updated patch. It replaces the defined(linux) tests with
defined(__GLIBC__).

- --
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGFo8PN9ixZKFWjRQRAsFRAJwKK4pOaxxTUCR71vF3P7R+QMY2dACgsnsY
4xssXvgP96hfEbiOvdSFUUM=
=AT85
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: uclibc-fenv.patch
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070406/8ed42aac/attachment.ksh>

From sdave at ufl.edu  Fri Apr  6 14:56:53 2007
From: sdave at ufl.edu (David Shepherd)
Date: Fri, 06 Apr 2007 14:56:53 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
In-Reply-To: <46168F0F.5070908@physics.mcmaster.ca>
References: <461654D5.401@ufl.edu>	<461665D8.8010408@physics.mcmaster.ca>	<46168972.6080208@ufl.edu>	<46168CFB.1050102@ufl.edu>
	<46168F0F.5070908@physics.mcmaster.ca>
Message-ID: <461697F5.2050706@ufl.edu>

The second part of the patch is failing:

# patch -p0 < ../uclibc-fenv.patch
patching file numpy/core/include/numpy/ufuncobject.h
patching file numpy/numarray/_capi.c
Hunk #1 FAILED at 224.
Hunk #2 FAILED at 2937.
2 out of 2 hunks FAILED -- saving rejects to file numpy/numarray/_capi.c.rej
#

I attached the .rej file.


-Dave


David M. Cooke wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> David Shepherd wrote:
>> You know, looking at the core it looks like it has something to do with 
>> the "define(linux)" statement.  I am running linux, but its a buildroot 
>> (uclibc) rootfs.  Let me know if you need anymore information.  Thanks.
>>
>> Dave
>>
>> David Shepherd wrote:
>>> Well, I get further into the compile process, but now it fails on 
>>> another gcc compile operations.  This is essentially the same error 
>>> message I was getting before.  It tries to #include <fenv.h>, but does 
>>> not find it (it shouldn't be including it anyway, as you said).  This 
>>> time it fails when trying to compile "_capi.c".  Thanks for all your 
>>> help so far!
> 
> Try this updated patch. It replaces the defined(linux) tests with
> defined(__GLIBC__).
> 
> - --
> |>|\/|<
> /------------------------------------------------------------------\
> |David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
> |cookedm at physics.mcmaster.ca
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iD8DBQFGFo8PN9ixZKFWjRQRAsFRAJwKK4pOaxxTUCR71vF3P7R+QMY2dACgsnsY
> 4xssXvgP96hfEbiOvdSFUUM=
> =AT85
> -----END PGP SIGNATURE-----
> 
> 
> ------------------------------------------------------------------------
> 
> Index: numpy/core/include/numpy/ufuncobject.h
> ===================================================================
> --- numpy/core/include/numpy/ufuncobject.h	(revision 3673)
> +++ numpy/core/include/numpy/ufuncobject.h	(working copy)
> @@ -276,7 +276,7 @@
>  	(void) fpsetsticky(0);						\
>  	}
>  
> -#elif defined(linux) || defined(__APPLE__) || defined(__CYGWIN__) || defined(__MINGW32__)
> +#elif defined(__GLIBC__) || defined(__APPLE__) || defined(__CYGWIN__) || defined(__MINGW32__)
>  
>  #if defined(__GLIBC__) || defined(__APPLE__) || defined(__MINGW32__)
>  #include <fenv.h>
> Index: numpy/numarray/_capi.c
> ===================================================================
> --- numpy/numarray/_capi.c	(revision 3673)
> +++ numpy/numarray/_capi.c	(working copy)
> @@ -224,7 +224,7 @@
>  }
>  
>  /* Likewise for Integer overflows */
> -#if defined(linux) || defined(__APPLE__) || defined(__CYGWIN__) || defined(__MINGW32__)
> +#if defined(__GLIBC__) || defined(__APPLE__) || defined(__CYGWIN__) || defined(__MINGW32__)
>  #if defined(__GLIBC__) || defined(__APPLE__) || defined(__MINGW32__)
>  #include <fenv.h>
>  #elif defined(__CYGWIN__)
> @@ -2937,7 +2937,7 @@
>  	return retstatus;
>  }
>  
> -#elif defined(linux) || defined(__APPLE__) || defined(__CYGWIN__) || defined(__MINGW32__)
> +#elif defined(__GLIBC__) || defined(__APPLE__) || defined(__CYGWIN__) || defined(__MINGW32__)
>  #if defined(__GLIBC__) || defined(darwin) || defined(__MINGW32__)
>  #include <fenv.h>
>  #elif defined(__CYGWIN__)
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: _capi.c.rej
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070406/e4b0e046/attachment.ksh>

From robert.kern at gmail.com  Fri Apr  6 15:06:35 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 06 Apr 2007 14:06:35 -0500
Subject: [Numpy-discussion] I can't know where I update the code...
In-Reply-To: <loom.20070406T144245-811@post.gmane.org>
References: <296323b50704060015k6d941c62xe4095cf3961b2ca6@mail.gmail.com>	<loom.20070406T123759-57@post.gmane.org>
	<loom.20070406T144245-811@post.gmane.org>
Message-ID: <46169A3B.9020903@gmail.com>

Chris wrote:
>> Sorry, this is not an answer, but it might help someone else to answer your 
>> question: importing weave (using numpy-1.0.2) is one way to get this error 
>> message.
> 
> Is this something we should post on the scipy-dev mailing list instead? (Maybe 
> there are other parts of scipy that need to be updated? Or maybe this has 
> already been updated on the current svn version of scipy?)

All references to ScipyTest have been removed from the SVN version of scipy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From giorgio at gilestro.tk  Fri Apr  6 15:12:37 2007
From: giorgio at gilestro.tk (Giorgio F. Gilestro)
Date: Fri, 6 Apr 2007 14:12:37 -0500
Subject: [Numpy-discussion] problem reading binary data from file
In-Reply-To: <46167B31.8000207@noaa.gov>
References: <212906f40704060914q319dcba4sbb6532e496b1806a@mail.gmail.com>
	<200704061241.02228.pgmdevlist@gmail.com> <46167B31.8000207@noaa.gov>
Message-ID: <212906f40704061212x7d8ab22fve39235b28cccbc2f@mail.gmail.com>

Thanks to everyone, really.
I already know this ml is going to be as useful as numpy itself!

In fact numpy.fromfile was the first function I tried to use but I
couldn't manage to get it working with the byteorder I needed
(big-endian).
The ['f0'] trick works very nicely though,  so I think I'll use that for now.
(Any hint on were to find docs about it?)


From cookedm at physics.mcmaster.ca  Fri Apr  6 15:13:44 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Fri, 06 Apr 2007 15:13:44 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
In-Reply-To: <461697F5.2050706@ufl.edu>
References: <461654D5.401@ufl.edu>	<461665D8.8010408@physics.mcmaster.ca>	<46168972.6080208@ufl.edu>	<46168CFB.1050102@ufl.edu>	<46168F0F.5070908@physics.mcmaster.ca>
	<461697F5.2050706@ufl.edu>
Message-ID: <46169BE8.1000303@physics.mcmaster.ca>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Shepherd wrote:
> The second part of the patch is failing:
> 
> # patch -p0 < ../uclibc-fenv.patch
> patching file numpy/core/include/numpy/ufuncobject.h
> patching file numpy/numarray/_capi.c
> Hunk #1 FAILED at 224.
> Hunk #2 FAILED at 2937.
> 2 out of 2 hunks FAILED -- saving rejects to file
> numpy/numarray/_capi.c.rej
> #

Ahh, you're not using a current subversion checkout :-) For your
purposes, you could just change the #if defined(linux) to #if
defined(__GLIBC__) (or #if 0 if that strikes your fancy).

- --
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGFpvoN9ixZKFWjRQRAuAeAJ4xaAxUUz828DeMRUd5vYPl0K6TfACgsSq7
2lvovjwFjEDECCJHKeQwhTQ=
=OGOK
-----END PGP SIGNATURE-----


From mkg at cs.nyu.edu  Fri Apr  6 15:23:46 2007
From: mkg at cs.nyu.edu (Matthew Koichi Grimes)
Date: Fri, 06 Apr 2007 15:23:46 -0400
Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 7, Issue 13
In-Reply-To: <mailman.8878.1175843741.2614.numpy-discussion@scipy.org>
References: <mailman.8878.1175843741.2614.numpy-discussion@scipy.org>
Message-ID: <46169E42.3000807@cs.nyu.edu>

Travis wrote:
>
> Short answer:  No, they are not bugs.
>
> The rule is:
>
> In any mixed-type operation between two objects of the same 
> fundamental "kind" (i.e. integer, float, complex) arrays always have 
> precedence over "scalars"  (where a 0-d array is considered a scalar 
> in this context).
I guess it's not hard to accept and remember that operations between 
arrays and scalars behave differently than operations between two 
arrays, since arrays and scalars are obviously different classes.

However, the fact that array-array operations behave differently 
depending on whether one of the arrays is 0-dimensional still strikes me 
as bad. Bad in the sense that it can lead to surprising and 
hard-to-identify bugs in vectorized code designed to work with tensors 
of various ranks, including 0. This equivalency between rank-0 arrays 
and scalars seems like another instance of the rank-0 special-casing 
that has been deemed bad in recarray and elsewhere.

A usage case my friend encountered is when finding the small difference 
between two tensors whose values are close to within 10e-8, precisely 
the case where one would want 64-bit precision. Say these tensors are of 
different rank, so this is a broadcasted subtraction:

def someVectorizedFunc(...):
    ...
    smallDifference = largeRank_float32 - smallRank_float64
    ...

This code works just fine as long as smallRank_float64 has a rank of at 
least one, but if it's a scalar, all of a sudden smallDifference is 
filled with 0.0s. If the ranks of these tensors are dependent on the 
input arguments (as is the case in vectorized code), this leads to 
pretty subtle bugs.
>
> It's a rule whose reason has a long history.
>
I'd be interested to read the discussion that led to making the type 
promotion rules different for array-array vs array-scalar operations. 
Does anybody remember the approximate date that this thread occurred in 
numpy-discussion, so I can look it up?

-- Matt


From Chris.Barker at noaa.gov  Fri Apr  6 15:31:33 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 06 Apr 2007 12:31:33 -0700
Subject: [Numpy-discussion] problem reading binary data from file
In-Reply-To: <212906f40704061212x7d8ab22fve39235b28cccbc2f@mail.gmail.com>
References: <212906f40704060914q319dcba4sbb6532e496b1806a@mail.gmail.com>
	<200704061241.02228.pgmdevlist@gmail.com> <46167B31.8000207@noaa.gov>
	<212906f40704061212x7d8ab22fve39235b28cccbc2f@mail.gmail.com>
Message-ID: <4616A015.7050100@noaa.gov>

> In fact numpy.fromfile was the first function I tried to use but I
> couldn't manage to get it working with the byteorder I needed
> (big-endian).

a = fromfile(...)
a.byteswap(True)


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From oliphant at ee.byu.edu  Fri Apr  6 15:55:47 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri, 06 Apr 2007 13:55:47 -0600
Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 7, Issue 13
In-Reply-To: <46169E42.3000807@cs.nyu.edu>
References: <mailman.8878.1175843741.2614.numpy-discussion@scipy.org>
	<46169E42.3000807@cs.nyu.edu>
Message-ID: <4616A5C3.7070806@ee.byu.edu>

Matthew Koichi Grimes wrote:

>Travis wrote:
>  
>
>>Short answer:  No, they are not bugs.
>>
>>The rule is:
>>
>>In any mixed-type operation between two objects of the same 
>>fundamental "kind" (i.e. integer, float, complex) arrays always have 
>>precedence over "scalars"  (where a 0-d array is considered a scalar 
>>in this context).
>>    
>>
>However, the fact that array-array operations behave differently 
>depending on whether one of the arrays is 0-dimensional still strikes me 
>as bad. Bad in the sense that it can lead to surprising and 
>hard-to-identify bugs in vectorized code designed to work with tensors 
>of various ranks, including 0. This equivalency between rank-0 arrays 
>and scalars seems like another instance of the rank-0 special-casing 
>that has been deemed bad in recarray and elsewhere.
>  
>
It all depends on your perspective.  In this case, there are situations 
where one can argue either way.  In this case, the simplicity of the 
definition of scalar is easier to remember:  All things that become 
rank-0 arrays when array(...) is called are considered "scalars".   
Viewed from that perspective, your drawing of the line is less compelling.

Changing rules at this point has a built-in negative bias.  Thus, the 
benefit of changing the rules must be substantial in order to get it to 
happen. 

Thanks for the feedback.


-Travis



From sdave at ufl.edu  Fri Apr  6 16:02:51 2007
From: sdave at ufl.edu (David Shepherd)
Date: Fri, 06 Apr 2007 16:02:51 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
In-Reply-To: <46169BE8.1000303@physics.mcmaster.ca>
References: <461654D5.401@ufl.edu>	<461665D8.8010408@physics.mcmaster.ca>	<46168972.6080208@ufl.edu>	<46168CFB.1050102@ufl.edu>	<46168F0F.5070908@physics.mcmaster.ca>	<461697F5.2050706@ufl.edu>
	<46169BE8.1000303@physics.mcmaster.ca>
Message-ID: <4616A76B.1050101@ufl.edu>

Okay, I manually changed defined(linux) to defined(__GLIBC__).  I still 
get the same error with "_capi.c".  It keeps trying to #include 
<fenv.h>.  Any other things I can change to try to prevent the problem?

Thanks,
David

-Dave

Error output:

creating build/temp.linux-ppc-2.5/numpy/lib
creating build/temp.linux-ppc-2.5/numpy/lib/src
compile options: '-Inumpy/core/include 
-Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
-Inumpy/core/include -I/usr/local/include/python2.5 -c'
gcc: numpy/lib/src/_compiled_base.c
gcc -pthread -shared 
build/temp.linux-ppc-2.5/numpy/lib/src/_compiled_base.o -o 
build/lib.linux-ppc-2.5/numpy/lib/_compiled_base.so
building 'numpy.numarray._capi' extension
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -mhard-float -DNDEBUG -g 
-O3 -Wall -Wstrict-prototypes -fPIC

creating build/temp.linux-ppc-2.5/numpy/numarray
compile options: '-Inumpy/core/include 
-Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
-Inumpy/core/include -I/usr/local/include/python2.5 -c'
gcc: numpy/numarray/_capi.c
numpy/numarray/_capi.c:229:18: fenv.h: No such file or directory
numpy/numarray/_capi.c: In function `int_overflow_error':
numpy/numarray/_capi.c:234: warning: implicit declaration of function 
`feraiseexcept'
numpy/numarray/_capi.c:234: error: `FE_OVERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:234: error: (Each undeclared identifier is 
reported only once
numpy/numarray/_capi.c:234: error: for each function it appears in.)
numpy/numarray/_capi.c: In function `NA_checkFPErrors':
numpy/numarray/_capi.c:2951: warning: implicit declaration of function 
`fetestexcept'
numpy/numarray/_capi.c:2952: error: `FE_DIVBYZERO' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2952: error: `FE_OVERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2952: error: `FE_UNDERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2952: error: `FE_INVALID' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2958: warning: implicit declaration of function 
`feclearexcept'
numpy/numarray/_capi.c:229:18: fenv.h: No such file or directory
numpy/numarray/_capi.c: In function `int_overflow_error':
numpy/numarray/_capi.c:234: warning: implicit declaration of function 
`feraiseexcept'
numpy/numarray/_capi.c:234: error: `FE_OVERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:234: error: (Each undeclared identifier is 
reported only once
numpy/numarray/_capi.c:234: error: for each function it appears in.)
numpy/numarray/_capi.c: In function `NA_checkFPErrors':
numpy/numarray/_capi.c:2951: warning: implicit declaration of function 
`fetestexcept'
numpy/numarray/_capi.c:2952: error: `FE_DIVBYZERO' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2952: error: `FE_OVERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2952: error: `FE_UNDERFLOW' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2952: error: `FE_INVALID' undeclared (first use 
in this function)
numpy/numarray/_capi.c:2958: warning: implicit declaration of function 
`feclearexcept'
error: Command "gcc -pthread -fno-strict-aliasing -mhard-float -DNDEBUG 
-g -O3 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include 
-Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
-Inumpy/core/include -I/usr/local/include/python2.5 -c 
numpy/numarray/_capi.c -o build/temp.linux-ppc-2.5/numpy/numarray/_capi.

David M. Cooke wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> David Shepherd wrote:
>> The second part of the patch is failing:
>>
>> # patch -p0 < ../uclibc-fenv.patch
>> patching file numpy/core/include/numpy/ufuncobject.h
>> patching file numpy/numarray/_capi.c
>> Hunk #1 FAILED at 224.
>> Hunk #2 FAILED at 2937.
>> 2 out of 2 hunks FAILED -- saving rejects to file
>> numpy/numarray/_capi.c.rej
>> #
> 
> Ahh, you're not using a current subversion checkout :-) For your
> purposes, you could just change the #if defined(linux) to #if
> defined(__GLIBC__) (or #if 0 if that strikes your fancy).
> 
> - --
> |>|\/|<
> /------------------------------------------------------------------\
> |David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
> |cookedm at physics.mcmaster.ca
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iD8DBQFGFpvoN9ixZKFWjRQRAuAeAJ4xaAxUUz828DeMRUd5vYPl0K6TfACgsSq7
> 2lvovjwFjEDECCJHKeQwhTQ=
> =OGOK
> -----END PGP SIGNATURE-----
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 


From hayes.tyler at gmail.com  Fri Apr  6 16:26:03 2007
From: hayes.tyler at gmail.com (Tyler J Hayes)
Date: Fri, 6 Apr 2007 20:26:03 +0000 (UTC)
Subject: [Numpy-discussion] F2PY and underscore problem
References: <ev2plp$vff$1@sea.gmane.org> <461528C1.8070302@gmail.com>
	<ev3cft$d9d$1@sea.gmane.org>
Message-ID: <loom.20070406T222258-369@post.gmane.org>

Zden?k Hur?k <hurak <at> fel.cvut.cz> writes:
> Sorry for not giving this important detail. I use gcc-4.1.1 compiler suite,
> which means that gfortran is used as a Fortran compiler. The same compiler
> was used to build Lapack.

I'm not sure if this will help, but have you tried to specify the gfortran
compiler via the option:

--fcompiler=gnu95 

when calling f2py in the second line after the ".pyf" file has been created?
Perhaps that will work.

Unfortunately, that's all I can offer as I'm somewhat new to Python and f2py
myself.

Let me know if it works.

Cheers,

Tyler 



From nadavh at visionsense.com  Sat Apr  7 03:13:18 2007
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sat, 7 Apr 2007 09:13:18 +0200
Subject: [Numpy-discussion] problem reading binary data from file
Message-ID: <07C6A61102C94148B8104D42DE95F7E8C8F223@exchange2k.envision.co.il>

How about:

numpy.fromfile(filename, dtype='>i2')

  Nadav

-----Original Message-----
From:	numpy-discussion-bounces at scipy.org on behalf of Pierre GM
Sent:	Fri 06-Apr-07 18:41
To:	Discussion of Numerical Python
Cc:	
Subject:	Re: [Numpy-discussion] problem reading binary data from file

On Friday 06 April 2007 12:14:20 Giorgio F. Gilestro wrote:

> Is there any way I could get a 1D array (no tuples please!) directly
> from the file?
> (BTW numpy.core.records.fromstring gives the same output)

numpy.core.records.fromfile (filename, formats='i2', byteorder='big')['f0']

With numpy.core.records, you create a record array. Just get the field ('f0' 
by default) to get a ndarray.

(I'm sure there's a simpler way, but that should do it for now).
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 3017 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070407/8c9bb03d/attachment.bin>

From staneff at constructiondatares.com  Sat Apr  7 12:31:37 2007
From: staneff at constructiondatares.com (Steve Staneff)
Date: Sat, 7 Apr 2007 10:31:37 -0600 (MDT)
Subject: [Numpy-discussion] newbie question - large dataset
Message-ID: <51343.72.42.76.106.1175963497.squirrel@webmail.idiom.com>

Hi,

I'm looking for a better solution to managing a very large calculation. 
Set A is composed of tuples a, each of the form a = [float, string]; set B
is composed of tuples of similar structure (b = [float, string]).  For
each possible combination of a and b I'm calculating c, of the form c =
f(a,b) = [g(a[0], b[0]), h(a[1], b[1])] where g() and h() are non-trivial
functions.

There are now 15,000 or more tuples in A, and 100,000 or more tuples in B.
 B is expected to grow with time as the source database grows.  In
addition, there are many more elements in a and b than I've stated (and
many more functions operating on them).  I'm currently using python to
loop through each a in A and each b in B, which takes days.

If anyone can point me to a better approach via numpy ( or anything
else!), I'd be very appreciative.

Thanks in advance,

Steve




From charlesr.harris at gmail.com  Sat Apr  7 13:04:01 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 7 Apr 2007 11:04:01 -0600
Subject: [Numpy-discussion] newbie question - large dataset
In-Reply-To: <51343.72.42.76.106.1175963497.squirrel@webmail.idiom.com>
References: <51343.72.42.76.106.1175963497.squirrel@webmail.idiom.com>
Message-ID: <e06186140704071004x6000b2ffmd386679053ed9445@mail.gmail.com>

On 4/7/07, Steve Staneff <staneff at constructiondatares.com> wrote:
>
> Hi,
>
> I'm looking for a better solution to managing a very large calculation.
> Set A is composed of tuples a, each of the form a = [float, string]; set B
> is composed of tuples of similar structure (b = [float, string]).  For
> each possible combination of a and b I'm calculating c, of the form c =
> f(a,b) = [g(a[0], b[0]), h(a[1], b[1])] where g() and h() are non-trivial
> functions.


Are you computing with *all* combinations of elements in a and b? I can't
tell from your description. What do the functions g and h look like? What do
the inputs and outputs actually look like? PyTables might be a good way to
manage the data sets, which look like they could be stored as record arrays.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070407/82083e59/attachment.html>

From peridot.faceted at gmail.com  Sat Apr  7 14:48:47 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 7 Apr 2007 14:48:47 -0400
Subject: [Numpy-discussion] newbie question - large dataset
In-Reply-To: <51343.72.42.76.106.1175963497.squirrel@webmail.idiom.com>
References: <51343.72.42.76.106.1175963497.squirrel@webmail.idiom.com>
Message-ID: <ce557a360704071148j53e376e4u3dcddac334996bfc@mail.gmail.com>

On 07/04/07, Steve Staneff <staneff at constructiondatares.com> wrote:
> Hi,
>
> I'm looking for a better solution to managing a very large calculation.
> Set A is composed of tuples a, each of the form a = [float, string]; set B
> is composed of tuples of similar structure (b = [float, string]).  For
> each possible combination of a and b I'm calculating c, of the form c =
> f(a,b) = [g(a[0], b[0]), h(a[1], b[1])] where g() and h() are non-trivial
> functions.
>
> There are now 15,000 or more tuples in A, and 100,000 or more tuples in B.
>  B is expected to grow with time as the source database grows.  In
> addition, there are many more elements in a and b than I've stated (and
> many more functions operating on them).  I'm currently using python to
> loop through each a in A and each b in B, which takes days.
>
> If anyone can point me to a better approach via numpy ( or anything
> else!), I'd be very appreciative.

It's pretty difficult to tell what's going on from this very abstract
description. But from what you're describing, it sounds like the
majority of your time is being spent in g and h (which are being
called a billion and a half times each). I'm fairly sure that a simple
nested python loop shouldn't take days to run a billion and a half
times. If that's the case, then using numpy only to optimize the
looping will do you no good. You will need to look at some way of
making g and h faster.

First priority, of course, should be to look and see if there is some
way to reduce the amount of code that gets run a billion and a half
times - if some part of g or h is doing some calculation on only A or
only B, taking that outside the loop will be a huge win. If many pairs
(a,b) are irrelevant, zero, or negligible, not doing any calculations
on them may accelerate things dramatically. If there are only a modest
number of different strings or floating-point numbers, you may be able
to use a dictionary to cache function values.

If none of those algorithmic improvements are possible, you can look
at other possibilities for speeding things up (though the speedups
will be modest). Parallelism is an obvious one - if you've got a
multicore machine you may be able to cut your processing time by a
factor of the number of cores you have available with minimal effort
(for example by replacing a for loop with a simple foreach,
implemented as in the attached file). You could also try psyco, a
runtime optimizer, though I've never found it accelerated any code of
mine. Moving up the difficulty scale, you could try numexpr (if your
function is numeric), pyrex, or weave (which allow you to write
compiled code for use in python with a minimum of pain) or writing the
functions in C directly.

Finally, I should point out that working with gigantic arrays (c has a
billion and a half elements) in numpy can be slower than working with
them using list comprehensions (say) in stock python, because you get
better data locality, and you end up copying less and allocating fewer
(giant) intermediate arrays. In many contexts, numpy should be viewed
not as a way to speed up the execution of your code but as a way to
speed up *writing* your code.

Anne M. Archibald
-------------- next part --------------
A non-text attachment was scrubbed...
Name: handythread.py
Type: text/x-python
Size: 1047 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070407/b8b45cc4/attachment.py>

From stefan at sun.ac.za  Sat Apr  7 16:12:11 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Sat, 7 Apr 2007 22:12:11 +0200
Subject: [Numpy-discussion] newbie question - large dataset
In-Reply-To: <ce557a360704071148j53e376e4u3dcddac334996bfc@mail.gmail.com>
References: <51343.72.42.76.106.1175963497.squirrel@webmail.idiom.com>
	<ce557a360704071148j53e376e4u3dcddac334996bfc@mail.gmail.com>
Message-ID: <20070407201211.GU18196@mentat.za.net>

On Sat, Apr 07, 2007 at 02:48:47PM -0400, Anne Archibald wrote:
> If none of those algorithmic improvements are possible, you can look
> at other possibilities for speeding things up (though the speedups
> will be modest). Parallelism is an obvious one - if you've got a
> multicore machine you may be able to cut your processing time by a
> factor of the number of cores you have available with minimal effort
> (for example by replacing a for loop with a simple foreach,
> implemented as in the attached file).

Would this code speed things up under Python?  I was under the
impression that there is only one process, irrespective of whether or
not "threads" are used, and that the global interpreter lock is used
when swapping between threads to make sure that only one executes at
any instance in time.

If my above understanding is correct, it would be better to use a
multi-process engine like IPython1.

Cheers
St?fan


From fperez.net at gmail.com  Sat Apr  7 16:22:11 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Sat, 7 Apr 2007 14:22:11 -0600
Subject: [Numpy-discussion] newbie question - large dataset
In-Reply-To: <20070407201211.GU18196@mentat.za.net>
References: <51343.72.42.76.106.1175963497.squirrel@webmail.idiom.com>
	<ce557a360704071148j53e376e4u3dcddac334996bfc@mail.gmail.com>
	<20070407201211.GU18196@mentat.za.net>
Message-ID: <db6b5ecc0704071322h6777932enb5fb1ffc52115461@mail.gmail.com>

On 4/7/07, Stefan van der Walt <stefan at sun.ac.za> wrote:
> On Sat, Apr 07, 2007 at 02:48:47PM -0400, Anne Archibald wrote:
> > If none of those algorithmic improvements are possible, you can look
> > at other possibilities for speeding things up (though the speedups
> > will be modest). Parallelism is an obvious one - if you've got a
> > multicore machine you may be able to cut your processing time by a
> > factor of the number of cores you have available with minimal effort
> > (for example by replacing a for loop with a simple foreach,
> > implemented as in the attached file).
>
> Would this code speed things up under Python?  I was under the
> impression that there is only one process, irrespective of whether or
> not "threads" are used, and that the global interpreter lock is used
> when swapping between threads to make sure that only one executes at
> any instance in time.

You are correct.  If g,h in the OP's description satisfy:

a) they are bloody expensive

b) they release the GIL internally via the proper C API calls, which
means they are promising not to modify any shared python objects

the pure python threads approach could help *somewhat*.

But yes, for this kind of distribution problem in python, a
multi-process approach is probably a better approach, if parallelism
is going to be used.

I suspect, however, that trying to lower the quadratic complexity of
the OP's formulation in the first place is probably a better idea.
Distribution lowers the constants, not the asymptotic behavior; as
Anne accurately pointed out, this is much more of an algorithmic
question.

Cheers,

f


From peridot.faceted at gmail.com  Sat Apr  7 16:54:58 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 7 Apr 2007 16:54:58 -0400
Subject: [Numpy-discussion] degree to which numpy releases threads
Message-ID: <ce557a360704071354q51697b13k1ba835692ca2a03d@mail.gmail.com>

On 07/04/07, Fernando Perez <fperez.net at gmail.com> wrote:

> You are correct.  If g,h in the OP's description satisfy:
>
> a) they are bloody expensive
>
> b) they release the GIL internally via the proper C API calls, which
> means they are promising not to modify any shared python objects
>
> the pure python threads approach could help *somewhat*.

I feel silly. Some (many?) numpy functions satisfy both criteria (dot,
for example, when using BLAS, definitely allows threads and can be
bloody expensive), but the OP is hardly likely to be using giant slow
numpy functions and asking this question.

How thorough is numpy's NPY_BEGIN_ALLOW_THREADS coverage? How about
f2py-generated code? It should be fairly easy for f2py to tell when a
hunk of FORTRAN might modify a shared python object (never?), and this
would cover not only much of scipy but much user code as well.

Anne


From bsouthey at gmail.com  Sat Apr  7 16:59:27 2007
From: bsouthey at gmail.com (Bruce Southey)
Date: Sat, 7 Apr 2007 15:59:27 -0500
Subject: [Numpy-discussion] newbie question - large dataset
In-Reply-To: <51343.72.42.76.106.1175963497.squirrel@webmail.idiom.com>
References: <51343.72.42.76.106.1175963497.squirrel@webmail.idiom.com>
Message-ID: <bbcd77d00704071359h3fbe357ftcaac04287980c84d@mail.gmail.com>

Hi,
Why tuples as your g() function is one floats and h() is on strings
with g() and h() independent?

You should be able vectorize this very easily since everything just
depends on the tuple value (you need to write g() and/or h() in
matrix/vector notation first). But you need to supply either the
functions or some description of what g() and h() are doing. Further
you probably want to look at the full algorithm that you are using as
the 'many more functions' may require refinement especially if
function f() is the final product and everything else is temporary.

Bruce

On 4/7/07, Steve Staneff <staneff at constructiondatares.com> wrote:
> Hi,
>
> I'm looking for a better solution to managing a very large calculation.
> Set A is composed of tuples a, each of the form a = [float, string]; set B
> is composed of tuples of similar structure (b = [float, string]).  For
> each possible combination of a and b I'm calculating c, of the form c =
> f(a,b) = [g(a[0], b[0]), h(a[1], b[1])] where g() and h() are non-trivial
> functions.
>
> There are now 15,000 or more tuples in A, and 100,000 or more tuples in B.
>  B is expected to grow with time as the source database grows.  In
> addition, there are many more elements in a and b than I've stated (and
> many more functions operating on them).  I'm currently using python to
> loop through each a in A and each b in B, which takes days.
>
> If anyone can point me to a better approach via numpy ( or anything
> else!), I'd be very appreciative.
>
> Thanks in advance,
>
> Steve
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From mattknox_ca at hotmail.com  Mon Apr  9 09:33:27 2007
From: mattknox_ca at hotmail.com (Matt Knox)
Date: Mon, 9 Apr 2007 13:33:27 +0000 (UTC)
Subject: [Numpy-discussion] Big list of Numpy & Scipy users
References: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>
	<46153C09.5020909@astraw.com>
Message-ID: <loom.20070409T151903-611@post.gmane.org>

Andrew Straw <strawman <at> astraw.com> writes:

> 
> Bill Baxter wrote:
> > On 4/4/07, Robert Kern <robert.kern <at> gmail.com> wrote:
> >   
> >> Bill Baxter wrote:
> >>     
> >>> Is there any place on the Wiki that lists all the known software that
> >>> uses Numpy in some way?
> >>>
> >>>       
> >>>> It would be nice to start collecting such a list if there isn't one
> >>>>         
> >>> already.  Screenshots would be nice too.
> >>>       
> >> There is no such list that I know of, but you may start one on the wiki if
you like.
> >>     
> >
> > Ok, I made a start:  http://www.scipy.org/Scipy_Projects
> >   
> Great idea. I renamed the page to http://www.scipy.org/Projects so
> Numpy-only users wouldn't feel excluded.
> 

Looking at the page, I think it might be nice if the table of contents included
links to the individual projects as well so you can get a sense of the variety
of applications using numpy/scipy at a quick glance without scrolling through
the whole page of dense text.

ie something like this:

   1. Middleware
         1. Python Imaging Library
         2. Matplotlib
         3. PyOpenGL
         ...
   2. Applications
         1. Inkscape
         2. PyPedal
         ...

This can be done automatically by making the headings level 2 headings instead
of the current level 4 (ie. == Heading ==  instead of ==== Heading ====). And
perhaps for applications where it isn't entirely obvious what it does based on
the name, a brief (less than one line) description in brackets beside the
heading? (eg. "Inkscape (vector graphics editor)")

I'm not a professional wiki designer by any means, so take this for what it's
worth. Feel free to use or ignore my suggestion as you see fit :)

- Matt



From david at ar.media.kyoto-u.ac.jp  Mon Apr  9 10:34:38 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 09 Apr 2007 23:34:38 +0900
Subject: [Numpy-discussion] *ALPHA* release of packaged numpy and scipy
 (RPM) for openSUSE, Fedora Core 5 and 6
Message-ID: <461A4EFE.5050408@ar.media.kyoto-u.ac.jp>

Hi,

   This is an annoucement for packaged binaries of numpy and scipy (eg 
rpm and deb, for easy installation).

   * SUMMARY:

   I have an initial version of numpy and scipy RPM for openSUSE and 
Fedora core ready for testing:
       * both numpy and scipy are available, WITH blas and lapack 
support dependencies (99% of the work was making shared libraries for 
blas/lapack work).
       * supported distributions: openSUSE 10.2, Fedora Core 5 and 6 are 
available (x86 only).

   ****** BIG FAT WARNING *******

   I am new to RPM packaging (and do not use any rpm based 
distribution), and a mistake may have the potential to break the system; 
also, because the packages do not use (yet) ATLAS, they are not really 
efficient. Finally, I did not attempt to follow the packaging policy of 
the various distributions (AFAIK, rpm does not make that easy). I can 
only say that the packages were installed on test machines (fedora core 
6 and open suse 10.2 on x86) successfully, and ran all scipy tests 
successfully. I *strongly* recommend that for now, only experienced 
users try those packages, and for testing purpose only.

   * Where ?

   You can find the repositories for the supported distributions here 
(please do not ask me how to add the repository to your system; if you 
do not know how to do it, you should not install those packages anyway):

   http://software.opensuse.org/download/home:/ashigabou/

   Packages' name are python-numpy and python-scipy.


   * Technical details:

   Those packages are done using the open suse build system: 
http://en.opensuse.org/Build_Service:

   "The openSUSE Build Service is open and complete distribution 
development platform that provides infrastructure for a development of 
the future openSUSE based distributions. It provides software developers 
with a tool to compile, release and publish their software for the broad 
user audience, including creation of their own Linux distribution based 
on openSUSE for various hardware architectures."

   The big point of this service compared to doing the packaging by hand 
is that everytime you modify one of the dependency, all packages are 
rebuilt for all distribution automatically. The system also provides a 
build farm, so that I don't have to build the rpm on each supported 
distribution. Note that the code to do the packaging is 100 % compatible 
with normal rpm tools, so that the building itself can be done the usual 
way (eg if for some reason the opensuse build system is not available 
anymore, the work is not lost).

   * TODO:

   I did those packages more for a proof of concept than for final 
releases. If other people find this solution useful, I hope that people 
more knowledgeable than me for each supported distribution will help 
polishing the packages:
       * For now, only minimal dependencies were added (netlib BLAS and 
LAPACK). Ideally, we would put fftw, umfpack and atlas.
       * Make the packages follow each distribution policy
       * Make x86_64 available (I know nothing about 64 bits convention, 
where to put libraries, and I do not have access to any 64 bits machines).
       * add Debian/Ubuntu packages (they will be much easier to 
package, because debian actually packages the BLAS and LAPACK correctly, 
contrary to fedora and suse which are kind of broken in that respect), 
and other useful packages such as matplotlib, ipython and pytables.

   cheers,

   David


From rmay at ou.edu  Mon Apr  9 12:03:59 2007
From: rmay at ou.edu (Ryan May)
Date: Mon, 09 Apr 2007 11:03:59 -0500
Subject: [Numpy-discussion] scipy.io.loadmat incompatible with Numpy 1.0.2
Message-ID: <461A63EF.8090203@ou.edu>

Hi,

As far as I can tell, the new Numpy 1.0.2 broke scipy.io.loadmat. 
Here's what I get when I try to open a file with using loadmat with 
numpy 1.0.2 (on gentoo AMD64):

In [2]: loadmat('tep_iqdata.mat')
---------------------------------------------------------------------------
exceptions.AttributeError                            Traceback (most 
recent call last)

/usr/lib64/python2.4/site-packages/scipy/io/mio.py in loadmat(file_name, 
mdict, appendmat, basename, **kwargs)
      94     '''
      95     MR = mat_reader_factory(file_name, appendmat, **kwargs)
---> 96     matfile_dict = MR.get_variables()
      97     if mdict is not None:
      98         mdict.update(matfile_dict)

/usr/lib64/python2.4/site-packages/scipy/io/miobase.py in 
get_variables(self, variable_names)
     267             variable_names = [variable_names]
     268         self.mat_stream.seek(0)
--> 269         mdict = self.file_header()
     270         mdict['__globals__'] = []
     271         while not self.end_of_stream():

/usr/lib64/python2.4/site-packages/scipy/io/mio5.py in file_header(self)
     508         hdict = {}
     509         hdr = self.read_dtype(self.dtypes['file_header'])
--> 510         hdict['__header__'] = hdr['description'].strip(' \t\n\000')
     511         v_major = hdr['version'] >> 8
     512         v_minor = hdr['version'] & 0xFF

AttributeError: 'numpy.ndarray' object has no attribute 'strip'


Reverting to numpy 1.0.1 works fine for the same code.  So the question 
is, does scipy need an update, or did something unintended creep into 
Numpy 1.0.2? (Hence the cross-post)

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma




From sdave at ufl.edu  Fri Apr  6 10:09:20 2007
From: sdave at ufl.edu (David Shepherd)
Date: Fri, 06 Apr 2007 10:09:20 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
Message-ID: <46165490.70702@ufl.edu>

Hey all,

We started to try and compile the numpy module on an embedded PowerPC 
Xilinx board with no luck.  This is one of the errors I get when I try 
to build the module on-board.  It is due to the fact that the compiler 
is located in a different location than the original compiler for python 
(i think).  I made symbolic links to fix the first error.  The second 
more critical error that I cannot recover from is the "Error: 
Unrecognized opcode: `fldenv'".  There are many more duplicates of this 
error, but it occurs when the "umathmodule.c" is being compiled.

I have tried changing the CFLAGS multiple times with no luck.  Python 
was compiled with uclibc libraries, so could this be the problem?  Is 
there any way we could compile Python and numpy together in one step?

I am using Binutils 2.17, GCC 3.4.6, and Python 2.5.  Again, python was 
compiled with uclibc, not the standard libraries.  We require the FFT 
module for a project that is due in about a week.  Any help would be 
appreciated.

-David Shepherd


C compiler: 
/opt/automated-build/buildroot/build_powerpc/staging_dir/bin/powerpc
-linux-uclibc-gcc -DNDEBUG -Os -pipe -fPIC

compile options: '-I/usr/include/python2.5 -Inumpy/core/src 
-Inumpy/core/include
  -I/usr/include/python2.5 -c'
powerpc-linux-uclibc-gcc: _configtest.c
sh: 
/opt/automated-build/buildroot/build_powerpc/staging_dir/bin/powerpc-linux-u
clibc-gcc: not found
sh: 
/opt/automated-build/buildroot/build_powerpc/staging_dir/bin/powerpc-linux-u
clibc-gcc: not found
failure.

I made symbolic links to fix the above errors, but then I also get 
multiple assembler opcode errors when it tries to run the entry build:

powerpc-linux-gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC 
-Ibuild/src.linux-ppc-2.5/numpy/core/src -Inumpy/core/include 
-Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src 
-Inumpy/core/include -I/root/python/include/python2.5 -c 
build/src.linux-ppc-2.5/numpy/core/src/umathmodule.c -o 
build/temp.linux-ppc-2.5/build/src.linux-ppc-2.5/numpy/core/src/umathmodule.o
.
.
.
.
Error: Unrecognized opcode: `fldenv'
.
.
.


From miquel.poch at gmail.com  Mon Apr  9 06:28:33 2007
From: miquel.poch at gmail.com (Miquel Poch)
Date: Mon, 9 Apr 2007 12:28:33 +0200
Subject: [Numpy-discussion] Translation of a Matlab expresion
Message-ID: <8b0882fc0704090328nea58b66l82333035a5e5835@mail.gmail.com>

Hi,

I've got a function write in Matlab, and I need to tranlate it into python.
I've found an expresion like this:

BF = b0 + (b1 + (b2 + (b3 + (b4 + (b5 + b6*T).*T).*T).*T).*T).*T
or
dBFdT = b1 + (2 * b2 + (3 * b3 + (4 * b4 + (5 * b5 + 6* b6*T).*T).*T).*T).*T

T is a matrix and the rest of the variables are floats. I don't know why are
this '.' in the expresion, and that's why I can't translate it.

I tried to do it without the '.' but then the expression it's incorrect. We
can't multiply two matrix wich number of rows and columns are the same.

''' exceptions.ValueError : objects are not aligned '''

There's some function in numpy to translate this expression?

Thanks in advance,
Miquel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070409/e6f561a1/attachment.html>

From oliphant at ee.byu.edu  Mon Apr  9 12:45:16 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Mon, 09 Apr 2007 10:45:16 -0600
Subject: [Numpy-discussion] scipy.io.loadmat incompatible with Numpy
	1.0.2
In-Reply-To: <461A63EF.8090203@ou.edu>
References: <461A63EF.8090203@ou.edu>
Message-ID: <461A6D9C.4020505@ee.byu.edu>

Ryan May wrote:

>Hi,
>
>As far as I can tell, the new Numpy 1.0.2 broke scipy.io.loadmat. 
>  
>
Yes, it was the one place that scipy used the fact that field selection 
of a 0-d array returned a scalar.  This has been changed in NumPy 1.0.2 
to return a 0-d array.  

The fix is in SciPy SVN.   Just get the mio.py file from SVN and drop it 
in to your distribution and things should work fine.   Or wait until a 
SciPy release is made.

-Travis



From pgmdevlist at gmail.com  Mon Apr  9 12:47:36 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 9 Apr 2007 12:47:36 -0400
Subject: [Numpy-discussion] Translation of a Matlab expresion
In-Reply-To: <8b0882fc0704090328nea58b66l82333035a5e5835@mail.gmail.com>
References: <8b0882fc0704090328nea58b66l82333035a5e5835@mail.gmail.com>
Message-ID: <200704091247.36562.pgmdevlist@gmail.com>

On Monday 09 April 2007 06:28:33 Miquel Poch wrote:

> T is a matrix and the rest of the variables are floats. I don't know why
> are this '.' in the expresion, and that's why I can't translate it.

http://www.scipy.org/NumPy_for_Matlab_Users

The .* means: element-wise multiplitcation.

> I tried to do it without the '.' but then the expression it's incorrect. We
> can't multiply two matrix wich number of rows and columns are the same.
>
> ''' exceptions.ValueError : objects are not aligned '''

Use numpy.multiply, or don't use matrices at all, just plain ndarrays.


From charlesr.harris at gmail.com  Mon Apr  9 13:12:14 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 9 Apr 2007 11:12:14 -0600
Subject: [Numpy-discussion] Translation of a Matlab expresion
In-Reply-To: <8b0882fc0704090328nea58b66l82333035a5e5835@mail.gmail.com>
References: <8b0882fc0704090328nea58b66l82333035a5e5835@mail.gmail.com>
Message-ID: <e06186140704091012o194b138aieed936a0eca2521e@mail.gmail.com>

On 4/9/07, Miquel Poch <miquel.poch at gmail.com> wrote:
>
> Hi,
>
> I've got a function write in Matlab, and I need to tranlate it into
> python. I've found an expresion like this:
> ?
> BF = b0 + (b1 + (b2 + (b3 + (b4 + (b5 + b6*T).*T).*T).*T).*T).*T
> or
> dBFdT = b1 + (2 * b2 + (3 * b3 + (4 * b4 + (5 * b5 + 6*
> b6*T).*T).*T).*T).*T


It's Horner's method for evaluating polynomials and their derivatives. You
can also do this in numpy as:

In [5]: p1,p2,p3,p4,p5,p6 = [1,2,3,4,5,6]

In [6]: p = poly1d([p6,p5,p4,p3,p2,p1])

In [7]: T = array([[1,2],[3,4]])

In [8]: p(T)
Out[8]:
array([[  21,  321],
       [2005, 7737]])

In [9]: polyder(p)(T)
Out[9]:
array([[  70,  702],
       [3098, 9178]])

Note the reversal of the coefficient order. Just make sure your arrays are
floats, the example I gave is all integers.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070409/427d71d7/attachment.html>

From rmay at ou.edu  Mon Apr  9 13:12:35 2007
From: rmay at ou.edu (Ryan May)
Date: Mon, 09 Apr 2007 12:12:35 -0500
Subject: [Numpy-discussion] scipy.io.loadmat incompatible with
	Numpy	1.0.2
In-Reply-To: <461A6D9C.4020505@ee.byu.edu>
References: <461A63EF.8090203@ou.edu> <461A6D9C.4020505@ee.byu.edu>
Message-ID: <461A7403.6090403@ou.edu>

Travis Oliphant wrote:
> Ryan May wrote:
> 
>> Hi,
>>
>> As far as I can tell, the new Numpy 1.0.2 broke scipy.io.loadmat. 
>>  
>>
> Yes, it was the one place that scipy used the fact that field selection 
> of a 0-d array returned a scalar.  This has been changed in NumPy 1.0.2 
> to return a 0-d array.  
> 
> The fix is in SciPy SVN.   Just get the mio.py file from SVN and drop it 
> in to your distribution and things should work fine.   Or wait until a 
> SciPy release is made.
> 
> -Travis
> 

It worked if I also got the new mio5.py (rev. 2893).

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From wbaxter at gmail.com  Mon Apr  9 15:39:34 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Tue, 10 Apr 2007 04:39:34 +0900
Subject: [Numpy-discussion] Big list of Numpy & Scipy user
In-Reply-To: <loom.20070409T151903-611@post.gmane.org>
References: <e86a5fd00704041910y190cc4c6m50ee97b1cd004530@mail.gmail.com>
	<46153C09.5020909@astraw.com>
	<loom.20070409T151903-611@post.gmane.org>
Message-ID: <e86a5fd00704091239p4b3fb794kdd7df8b710fd0e33@mail.gmail.com>

On 4/9/07, Matt Knox <mattknox_ca at hotmail.com> wrote:
> Andrew Straw <strawman <at> astraw.com> writes:
> > >>> Is there any place on the Wiki that lists all the known software that
> > >>> uses Numpy in some way?
> > >>>
> > >
> > Great idea. I renamed the page to http://www.scipy.org/Projects so
> > Numpy-only users wouldn't feel excluded.
> >
>
> Looking at the page, I think it might be nice if the table of contents included
> links to the individual projects as well so you can get a sense of the variety
> of applications using numpy/scipy at a quick glance without scrolling through
> the whole page of dense text.
> ...
>
> This can be done automatically by making the headings level 2 headings instead
> of the current level 4 (ie. == Heading ==  instead of ==== Heading ====).

I made the projects be 4th level headings because 1st and 2nd level
headings didn't look different enough to me.  And because I thought
there might be some desire to insert some intermediate level headings
as the list grows.  Like Applications / Science  or Applications /
Graphics.   But that doesn't seem likely to happen, really, since
there are too many overlaps between any potential categories.
Middleware and Applications isn't really a true dichotomy either as
PIL comes with handy image manipulation scripts and SFE comes with
some reusable libs.

But I went ahead and made the projects 2nd level headings.  I hate the
fact that you can hardly tell the difference between 1st level and 2nd
level headings, but that's really an issue that should be dealt with
at the Wiki stylesheet level, not the page level.


> I'm not a professional wiki designer

I don't think there is such a job category ;-).  Jimmy Wales from
Wikipedia fits the bill maybe.

--bb


From sdave at ufl.edu  Mon Apr  9 16:29:43 2007
From: sdave at ufl.edu (David Shepherd)
Date: Mon, 09 Apr 2007 16:29:43 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
Message-ID: <461AA237.3020700@ufl.edu>

My message may have gotten lost over the weekend, but I wanted to see if 
anyone had anything else I can try to fix the problem.  We have to demo 
this project for a ECE project on Thursday and I was hoping to get the 
FFT working.

-------- Original Message --------
Subject: Re: [Numpy-discussion] Numpy with uclibc compiled python
Date: Fri, 06 Apr 2007 16:02:51 -0400
From: David Shepherd <sdave at ufl.edu>
To: Discussion of Numerical Python <numpy-discussion at scipy.org>
References: <461654D5.401 at ufl.edu> 
<461665D8.8010408 at physics.mcmaster.ca>	<46168972.6080208 at ufl.edu> 
<46168CFB.1050102 at ufl.edu>	<46168F0F.5070908 at physics.mcmaster.ca> 
<461697F5.2050706 at ufl.edu> <46169BE8.1000303 at physics.mcmaster.ca>

Okay, I manually changed defined(linux) to defined(__GLIBC__).  I still
get the same error with "_capi.c".  It keeps trying to #include
<fenv.h>.  Any other things I can change to try to prevent the problem?

Thanks,
David

-Dave

Error output:

creating build/temp.linux-ppc-2.5/numpy/lib
creating build/temp.linux-ppc-2.5/numpy/lib/src
compile options: '-Inumpy/core/include
-Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src
-Inumpy/core/include -I/usr/local/include/python2.5 -c'
gcc: numpy/lib/src/_compiled_base.c
gcc -pthread -shared
build/temp.linux-ppc-2.5/numpy/lib/src/_compiled_base.o -o
build/lib.linux-ppc-2.5/numpy/lib/_compiled_base.so
building 'numpy.numarray._capi' extension
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -mhard-float -DNDEBUG -g
-O3 -Wall -Wstrict-prototypes -fPIC

creating build/temp.linux-ppc-2.5/numpy/numarray
compile options: '-Inumpy/core/include
-Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src
-Inumpy/core/include -I/usr/local/include/python2.5 -c'
gcc: numpy/numarray/_capi.c
numpy/numarray/_capi.c:229:18: fenv.h: No such file or directory
numpy/numarray/_capi.c: In function `int_overflow_error':
numpy/numarray/_capi.c:234: warning: implicit declaration of function
`feraiseexcept'
numpy/numarray/_capi.c:234: error: `FE_OVERFLOW' undeclared (first use
in this function)
numpy/numarray/_capi.c:234: error: (Each undeclared identifier is
reported only once
numpy/numarray/_capi.c:234: error: for each function it appears in.)
numpy/numarray/_capi.c: In function `NA_checkFPErrors':
numpy/numarray/_capi.c:2951: warning: implicit declaration of function
`fetestexcept'
numpy/numarray/_capi.c:2952: error: `FE_DIVBYZERO' undeclared (first use
in this function)
numpy/numarray/_capi.c:2952: error: `FE_OVERFLOW' undeclared (first use
in this function)
numpy/numarray/_capi.c:2952: error: `FE_UNDERFLOW' undeclared (first use
in this function)
numpy/numarray/_capi.c:2952: error: `FE_INVALID' undeclared (first use
in this function)
numpy/numarray/_capi.c:2958: warning: implicit declaration of function
`feclearexcept'
numpy/numarray/_capi.c:229:18: fenv.h: No such file or directory
numpy/numarray/_capi.c: In function `int_overflow_error':
numpy/numarray/_capi.c:234: warning: implicit declaration of function
`feraiseexcept'
numpy/numarray/_capi.c:234: error: `FE_OVERFLOW' undeclared (first use
in this function)
numpy/numarray/_capi.c:234: error: (Each undeclared identifier is
reported only once
numpy/numarray/_capi.c:234: error: for each function it appears in.)
numpy/numarray/_capi.c: In function `NA_checkFPErrors':
numpy/numarray/_capi.c:2951: warning: implicit declaration of function
`fetestexcept'
numpy/numarray/_capi.c:2952: error: `FE_DIVBYZERO' undeclared (first use
in this function)
numpy/numarray/_capi.c:2952: error: `FE_OVERFLOW' undeclared (first use
in this function)
numpy/numarray/_capi.c:2952: error: `FE_UNDERFLOW' undeclared (first use
in this function)
numpy/numarray/_capi.c:2952: error: `FE_INVALID' undeclared (first use
in this function)
numpy/numarray/_capi.c:2958: warning: implicit declaration of function
`feclearexcept'
error: Command "gcc -pthread -fno-strict-aliasing -mhard-float -DNDEBUG
-g -O3 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include
-Ibuild/src.linux-ppc-2.5/numpy/core -Inumpy/core/src
-Inumpy/core/include -I/usr/local/include/python2.5 -c
numpy/numarray/_capi.c -o build/temp.linux-ppc-2.5/numpy/numarray/_capi.

David M. Cooke wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> David Shepherd wrote:
>> The second part of the patch is failing:
>>
>> # patch -p0 < ../uclibc-fenv.patch
>> patching file numpy/core/include/numpy/ufuncobject.h
>> patching file numpy/numarray/_capi.c
>> Hunk #1 FAILED at 224.
>> Hunk #2 FAILED at 2937.
>> 2 out of 2 hunks FAILED -- saving rejects to file
>> numpy/numarray/_capi.c.rej
>> #
> 
> Ahh, you're not using a current subversion checkout :-) For your
> purposes, you could just change the #if defined(linux) to #if
> defined(__GLIBC__) (or #if 0 if that strikes your fancy).
> 
> - --
> |>|\/|<
> /------------------------------------------------------------------\
> |David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
> |cookedm at physics.mcmaster.ca
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iD8DBQFGFpvoN9ixZKFWjRQRAuAeAJ4xaAxUUz828DeMRUd5vYPl0K6TfACgsSq7
> 2lvovjwFjEDECCJHKeQwhTQ=
> =OGOK
> -----END PGP SIGNATURE-----
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



From oliphant.travis at ieee.org  Mon Apr  9 17:44:30 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Mon, 09 Apr 2007 15:44:30 -0600
Subject: [Numpy-discussion] Numpy with uclibc compiled python
In-Reply-To: <461AA237.3020700@ufl.edu>
References: <461AA237.3020700@ufl.edu>
Message-ID: <461AB3BE.7070603@ieee.org>

David Shepherd wrote:
> My message may have gotten lost over the weekend, but I wanted to see if 
> anyone had anything else I can try to fix the problem.  We have to demo 
> this project for a ECE project on Thursday and I was hoping to get the 
> FFT working.
>
>   

Ideally, you can help us get this working right on your hardware. 

But, in a pinch, you can just comment out the line in numpy/setup.py 
that states:

config.add_subpackage('numarray')

Do you need the numarray package?

-Travis



From oliphant.travis at ieee.org  Mon Apr  9 17:47:32 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Mon, 09 Apr 2007 15:47:32 -0600
Subject: [Numpy-discussion] Numpy with uclibc compiled python
In-Reply-To: <461AA237.3020700@ufl.edu>
References: <461AA237.3020700@ufl.edu>
Message-ID: <461AB474.5000405@ieee.org>

David Shepherd wrote:
> My message may have gotten lost over the weekend, but I wanted to see if 
> anyone had anything else I can try to fix the problem.  We have to demo 
> this project for a ECE project on Thursday and I was hoping to get the 
> FFT working.
>   

I know nothing about ulibc.   Do you have any of the floating-point 
routines that this section of code is trying to get?

It looks like you on your platform we probably shouldn't even be trying 
to include fenv.h at all.     I would replace the entire line  (#227)  to

#if 0

so that the "default" section of code gets included.  Do this for every 
case where fenv.h is trying to be included.

-Travis



From Chris.Barker at noaa.gov  Mon Apr  9 17:52:48 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 09 Apr 2007 14:52:48 -0700
Subject: [Numpy-discussion] numpy on WinCE / PocketPC...
Message-ID: <461AB5B0.4080304@noaa.gov>

Hi all,

Has anyone gotten numpy working on WinCE PocketPC Windows Mobile (or 
whatever the heck they are calling it now)?

We're about to dive in and figure out what it takes to compile it, but I 
thought I 'd see if anyone else had done it, or knew any reason we 
shouldn't expect it to work.

-thanks,

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From sdave at ufl.edu  Mon Apr  9 18:12:50 2007
From: sdave at ufl.edu (David Shepherd)
Date: Mon, 09 Apr 2007 18:12:50 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
In-Reply-To: <461AB3BE.7070603@ieee.org>
References: <461AA237.3020700@ufl.edu> <461AB3BE.7070603@ieee.org>
Message-ID: <461ABA62.6030707@ufl.edu>

Well, we only need to do an FFT.  So I will try all your suggestions 
tonight and let you know of the results.  Thanks for your help!

-Dave

Travis Oliphant wrote:
> David Shepherd wrote:
>> My message may have gotten lost over the weekend, but I wanted to see if 
>> anyone had anything else I can try to fix the problem.  We have to demo 
>> this project for a ECE project on Thursday and I was hoping to get the 
>> FFT working.
>>
>>   
> 
> Ideally, you can help us get this working right on your hardware. 
> 
> But, in a pinch, you can just comment out the line in numpy/setup.py 
> that states:
> 
> config.add_subpackage('numarray')
> 
> Do you need the numarray package?
> 
> -Travis
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 


From david.doukhan at gmail.com  Tue Apr 10 00:49:23 2007
From: david.doukhan at gmail.com (David Doukhan)
Date: Tue, 10 Apr 2007 00:49:23 -0400
Subject: [Numpy-discussion] ndarray allocation question
Message-ID: <b94d068b0704092149i84b9ad2w91b41433b8bb56b7@mail.gmail.com>

Hi!
I'm writing you this mail because I would like to do "advanced" use of
ndarray memory allocation.

So here is a short description of my problem:

I'm interfacing multidimensionnal Numpy array to C using the ctypes module.
For now, there is no problem.
The C program that will use the pointer to the data of the array runs on a
CellBE architecture.
The consequences are:
* the adress of the beginning of the data of the array must be a multiple a
16 bytes (that could be specified using the attribute 'align' of dtype)
* the memory allocated MUST be a multiple of 16 bytes, and that's my current
problem, because I didn't find a way to specify it using dtypes...
For now, the only arguments that I know that could be given to a dtype
constructor are "names", "format", "offsets", "titles",
"align"<numpy-discussion at scipy.org>
Does any other arguments exist?

I could do the assumption that try to access to the data outside the last
element of the array won't produce a segfault, but, i would like to avoid as
much as possible doing so dirty stuffs :-))

So, do the think there is a "clean" way to do what I want. For example,
asking for an array of 2 lines and 7 columns of float32, so that the adress
of the biginning of the data would be a multiple of 16 bytes (that's already
possible)  AND beeing sure that the allocated memory for the data would be
at least 16 * sizeof(float32), instead of 14*sizeof(float32).

Thanks for your help!

-- 
David Doukhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070410/7574115b/attachment.html>

From sdave at ufl.edu  Tue Apr 10 01:05:01 2007
From: sdave at ufl.edu (David Shepherd)
Date: Tue, 10 Apr 2007 01:05:01 -0400
Subject: [Numpy-discussion] Numpy with uclibc compiled python
In-Reply-To: <461AB474.5000405@ieee.org>
References: <461AA237.3020700@ufl.edu> <461AB474.5000405@ieee.org>
Message-ID: <461B1AFD.1010901@ufl.edu>

The default section of the code compiled just fine and the FFT is 
working great.  Thanks so much for your help!

-Dave

Travis Oliphant wrote:
> David Shepherd wrote:
>> My message may have gotten lost over the weekend, but I wanted to see if 
>> anyone had anything else I can try to fix the problem.  We have to demo 
>> this project for a ECE project on Thursday and I was hoping to get the 
>> FFT working.
>>   
> 
> I know nothing about ulibc.   Do you have any of the floating-point 
> routines that this section of code is trying to get?
> 
> It looks like you on your platform we probably shouldn't even be trying 
> to include fenv.h at all.     I would replace the entire line  (#227)  to
> 
> #if 0
> 
> so that the "default" section of code gets included.  Do this for every 
> case where fenv.h is trying to be included.
> 
> -Travis
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 


From oliphant.travis at ieee.org  Tue Apr 10 02:38:07 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Tue, 10 Apr 2007 00:38:07 -0600
Subject: [Numpy-discussion] ndarray allocation question
In-Reply-To: <b94d068b0704092149i84b9ad2w91b41433b8bb56b7@mail.gmail.com>
References: <b94d068b0704092149i84b9ad2w91b41433b8bb56b7@mail.gmail.com>
Message-ID: <461B30CF.70409@ieee.org>

David Doukhan wrote:
> Hi!
> I'm writing you this mail because I would like to do "advanced" use of 
> ndarray memory allocation.
>
> So here is a short description of my problem:
>
> I'm interfacing multidimensionnal Numpy array to C using the ctypes 
> module. For now, there is no problem.
> The C program that will use the pointer to the data of the array runs 
> on a CellBE architecture.

> The consequences are:
> * the adress of the beginning of the data of the array must be a 
> multiple a 16 bytes (that could be specified using the attribute 
> 'align' of dtype)
> * the memory allocated MUST be a multiple of 16 bytes, and that's my 
> current problem, because I didn't find a way to specify it using dtypes...

You can't specify the alignment for a data-type unless you write your 
own data-type (the alignment is a property of the data-type and is 
determined at compile time for built-in data-types).   I don't think you 
can set the alignment for a derived data-type at this point.

> So, do the think there is a "clean" way to do what I want. For 
> example, asking for an array of 2 lines and 7 columns of float32, so 
> that the adress of the biginning of the data would be a multiple of 16 
> bytes (that's already possible)
How are you doing that?

> AND beeing sure that the allocated memory for the data would be at 
> least 16 * sizeof(float32), instead of 14*sizeof(float32).
You will have to make sure that the array is large-enough yourself by 
over requesting the size.

-Travis



From markbak at gmail.com  Tue Apr 10 09:02:46 2007
From: markbak at gmail.com (mark)
Date: Tue, 10 Apr 2007 13:02:46 -0000
Subject: [Numpy-discussion] combining two arrays into one
Message-ID: <1176210166.594457.187020@b75g2000hsg.googlegroups.com>

Hello list -

I want to combine two arrays into one, and I cannot find a clean way
to do it.

I have the following two arrays:

>>> x = array([[1, 2, 3],
               [4, 5, 6]])
>>> y = array([[10, 20, 30],
               [40, 50, 60]])

Now I want to make a new array z, such that z[:,:,0] gives me x an d
z[:,:,1] gives me y.

But what do I do? This doesn't work the right way for me:

>>> z = array([x,y])
>>> z[:,:,0]
array([[ 1,  4],
       [10, 40]])

Thanks for your help,

Mark



From wbaxter at gmail.com  Tue Apr 10 09:38:27 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Tue, 10 Apr 2007 22:38:27 +0900
Subject: [Numpy-discussion] combining two arrays into one
In-Reply-To: <1176210166.594457.187020@b75g2000hsg.googlegroups.com>
References: <1176210166.594457.187020@b75g2000hsg.googlegroups.com>
Message-ID: <e86a5fd00704100638o4e1bb495he8439ba99f4e72ab@mail.gmail.com>

I'm pretty sure dstack([x,y]) is what you're after.

--bb

On 4/10/07, mark <markbak at gmail.com> wrote:
> Hello list -
>
> I want to combine two arrays into one, and I cannot find a clean way
> to do it.
>
> I have the following two arrays:
>
> >>> x = array([[1, 2, 3],
>                [4, 5, 6]])
> >>> y = array([[10, 20, 30],
>                [40, 50, 60]])
>
> Now I want to make a new array z, such that z[:,:,0] gives me x an d
> z[:,:,1] gives me y.
>
> But what do I do? This doesn't work the right way for me:
>
> >>> z = array([x,y])
> >>> z[:,:,0]
> array([[ 1,  4],
>        [10, 40]])
>
> Thanks for your help,
>
> Mark


From markbak at gmail.com  Tue Apr 10 09:48:24 2007
From: markbak at gmail.com (mark)
Date: Tue, 10 Apr 2007 13:48:24 -0000
Subject: [Numpy-discussion] combining two arrays into one
In-Reply-To: <e86a5fd00704100638o4e1bb495he8439ba99f4e72ab@mail.gmail.com>
References: <1176210166.594457.187020@b75g2000hsg.googlegroups.com>
	<e86a5fd00704100638o4e1bb495he8439ba99f4e72ab@mail.gmail.com>
Message-ID: <1176212904.709859.121390@e65g2000hsc.googlegroups.com>

Yes, dstack is what I need.
Thanks for the quick response,
Mark

On 10 apr, 15:38, "Bill Baxter" <wbax... at gmail.com> wrote:
> I'm pretty sure dstack([x,y]) is what you're after.
>
> --bb
>
> On 4/10/07, mark <mark... at gmail.com> wrote:
>
> > Hello list -
>
> > I want to combine two arrays into one, and I cannot find a clean way
> > to do it.
>
> > I have the following two arrays:
>
> > >>> x = array([[1, 2, 3],
> >                [4, 5, 6]])
> > >>> y = array([[10, 20, 30],
> >                [40, 50, 60]])
>
> > Now I want to make a new array z, such that z[:,:,0] gives me x an d
> > z[:,:,1] gives me y.
>
> > But what do I do? This doesn't work the right way for me:
>
> > >>> z = array([x,y])
> > >>> z[:,:,0]
> > array([[ 1,  4],
> >        [10, 40]])
>
> > Thanks for your help,
>
> > Mark
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion



From jasonbraswell314 at gmail.com  Tue Apr 10 13:55:10 2007
From: jasonbraswell314 at gmail.com (Jason Braswell)
Date: Tue, 10 Apr 2007 12:55:10 -0500
Subject: [Numpy-discussion] cannot import name ccompiler
Message-ID: <e016079c0704101055p18c6b99bn9b6f12c327bb01ec@mail.gmail.com>

Hi, folks.  I'm a newbie trying to get started with SciPy on Windows.  I
installed it without problem, but when I try to import the SciPy module or
run the 'numpy' test, I get the following output:
***********************************************
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "C:\Python24\Lib\site-packages\numpy\__init__.py", line 89, in test
    return NumpyTest().test(level, verbosity)
  File "C:\Python24\Lib\site-packages\numpy\testing\numpytest.py", line 288,
in
__init__
    from numpy.distutils.misc_util import get_frame
  File "C:\Python24\Lib\site-packages\numpy\distutils\__init__.py", line 5,
in ?

    import ccompiler
  File "C:\Python24\Lib\site-packages\numpy\distutils\ccompiler.py", line 6,
in
?
    from distutils.ccompiler import *
  File "C:\Python24\Lib\site-packages\numpy\distutils\__init__.py", line 5,
in ?

    import ccompiler
  File "C:\Python24\Lib\site-packages\numpy\distutils\ccompiler.py", line 7,
in
?
    from distutils import ccompiler #as ccompiler
ImportError: cannot import name ccompiler
*****************************************************************
I tried to fix it by adding damn near everything to my path, which now looks
like this:
*********************************************************************
>>> sys.path
['', 'C:\\Python24\\lib\\site-packages\\setuptools-0.6a11dev-py2.4.egg',
'C:\\Py
thon24', 'C:\\Python24\\Lib\\site-packages',
'C:\\Python24\\Lib\\site-packages\\
numarray', 'C:\\Documents and Settings\\xtrader\\Numeric',
'C:\\Python24\\Lib\\s
ite-packages\\PIL', 'C:\\Python24\\Lib\\site-packages\\scipy',
'C:\\Python24\\Li
b\\site-packages\\numpy', 'C:\\Python24\\Lib\\site-packages\\numarray\\fft',
'C:
\\Python24\\Lib\\site-packages\\Numeric\\FFT',
'C:\\Python24\\Lib\\site-packages
\\numpy\\dft', 'C:\\Python24\\Lib\\site-packages\\scipy\\fftpack',
'C:\\Python24
\\Lib\\site-package\\numpy\\distutils',
'C:\\WINDOWS\\system32\\python24.zip', '
C:\\Documents and Settings\\xtrader', 'C:\\Python24\\DLLs',
'C:\\Python24\\lib',
 'C:\\Python24\\lib\\plat-win', 'C:\\Python24\\lib\\lib-tk',
'C:\\Python24\\lib\
\site-packages\\Numeric', 'C:\\Python24\\lib\\site-packages\\win32',
'C:\\Python
24\\lib\\site-packages\\win32\\lib',
'C:\\Python24\\lib\\site-packages\\Pythonwi
n', 'C:\\Python24\\lib\\site-packages\\vtk_python',
'C:\\Python24\\lib\\site-pac
kages\\wx-2.6-msw-unicode-enthought', 'C:\\Documents and
Settings\\xtrader\\Nume
ric', 'C:\\Python24\\Lib\\site-package\\numpy\\distutils']
********************************************************************************

Any ideas?  Thanks in advance.
-- 
Jason Braswell
(352) 256-7581
www.ronpaul.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070410/726c8488/attachment.html>

From robert.kern at gmail.com  Tue Apr 10 14:52:20 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 10 Apr 2007 13:52:20 -0500
Subject: [Numpy-discussion] cannot import name ccompiler
In-Reply-To: <e016079c0704101055p18c6b99bn9b6f12c327bb01ec@mail.gmail.com>
References: <e016079c0704101055p18c6b99bn9b6f12c327bb01ec@mail.gmail.com>
Message-ID: <461BDCE4.4030906@gmail.com>

Jason Braswell wrote:
> Hi, folks.  I'm a newbie trying to get started with SciPy on Windows.  I
> installed it without problem, but when I try to import the SciPy module
> or run the 'numpy' test, I get the following output:
> ***********************************************
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "C:\Python24\Lib\site-packages\numpy\__init__.py", line 89, in test
>     return NumpyTest().test(level, verbosity)
>   File "C:\Python24\Lib\site-packages\numpy\testing\numpytest.py", line
> 288, in
> __init__
>     from numpy.distutils.misc_util import get_frame
>   File "C:\Python24\Lib\site-packages\numpy\distutils\__init__.py", line
> 5, in ?
> 
>     import ccompiler
>   File "C:\Python24\Lib\site-packages\numpy\distutils\ccompiler.py",
> line 6, in
> ?
>     from distutils.ccompiler import *
>   File "C:\Python24\Lib\site-packages\numpy\distutils\__init__.py", line
> 5, in ?
> 
>     import ccompiler
>   File "C:\Python24\Lib\site-packages\numpy\distutils\ccompiler.py",
> line 7, in
> ?
>     from distutils import ccompiler #as ccompiler
> ImportError: cannot import name ccompiler
> *****************************************************************
> I tried to fix it by adding damn near everything to my path, which now
> looks like this: 

That's the problem. You have the numpy package directory in your sys.path, so
when it tries to import the standard library's distutils package, it finds
numpy/distutils instead. You want site-packages/ on your sys.path, not
site-packages/numpy/.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From jasonbraswell314 at gmail.com  Tue Apr 10 15:55:44 2007
From: jasonbraswell314 at gmail.com (Jason Braswell)
Date: Tue, 10 Apr 2007 14:55:44 -0500
Subject: [Numpy-discussion] cannot import name ccompiler
In-Reply-To: <461BDCE4.4030906@gmail.com>
References: <e016079c0704101055p18c6b99bn9b6f12c327bb01ec@mail.gmail.com>
	<461BDCE4.4030906@gmail.com>
Message-ID: <e016079c0704101255y709e5a64tf5eaaee5bcbaf85e@mail.gmail.com>

Ah, thank you kindly...worked like a charm.

On 4/10/07, Robert Kern <robert.kern at gmail.com> wrote:
>
> Jason Braswell wrote:
> > Hi, folks.  I'm a newbie trying to get started with SciPy on Windows.  I
> > installed it without problem, but when I try to import the SciPy module
> > or run the 'numpy' test, I get the following output:
> > ***********************************************
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in ?
> >   File "C:\Python24\Lib\site-packages\numpy\__init__.py", line 89, in
> test
> >     return NumpyTest().test(level, verbosity)
> >   File "C:\Python24\Lib\site-packages\numpy\testing\numpytest.py", line
> > 288, in
> > __init__
> >     from numpy.distutils.misc_util import get_frame
> >   File "C:\Python24\Lib\site-packages\numpy\distutils\__init__.py", line
> > 5, in ?
> >
> >     import ccompiler
> >   File "C:\Python24\Lib\site-packages\numpy\distutils\ccompiler.py",
> > line 6, in
> > ?
> >     from distutils.ccompiler import *
> >   File "C:\Python24\Lib\site-packages\numpy\distutils\__init__.py", line
> > 5, in ?
> >
> >     import ccompiler
> >   File "C:\Python24\Lib\site-packages\numpy\distutils\ccompiler.py",
> > line 7, in
> > ?
> >     from distutils import ccompiler #as ccompiler
> > ImportError: cannot import name ccompiler
> > *****************************************************************
> > I tried to fix it by adding damn near everything to my path, which now
> > looks like this:
>
> That's the problem. You have the numpy package directory in your sys.path,
> so
> when it tries to import the standard library's distutils package, it finds
> numpy/distutils instead. You want site-packages/ on your sys.path, not
> site-packages/numpy/.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma
> that is made terrible by our own mad attempt to interpret it as though it
> had
> an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
Jason Braswell
(352) 256-7581
www.ronpaul.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070410/9877721d/attachment.html>

From markbak at gmail.com  Tue Apr 10 15:57:00 2007
From: markbak at gmail.com (mark)
Date: Tue, 10 Apr 2007 19:57:00 -0000
Subject: [Numpy-discussion] making two 2D real arrays into one complex array
Message-ID: <1176235020.247655.111040@h3g2000cwc.googlegroups.com>

Hello List -

Any reason why the following doesn't work?

>>> x,y = meshgrid(linspace(-1,1,10),linspace(-1,1,10))
>>> z = complex(x,y)
Traceback (most recent call last):
  File "<pyshell#225>", line 1, in ?
    z = complex(x,y)
TypeError: only length-1 arrays can be converted to Python scalars


I know I can make an empty complex z array with the size of x, then
set z.real equal to x and z.imag equal to y. But is there any reason
why the above shouldn't work? It would be easy to implement in numpy,
wouldn't it? If so, I would *really* like it.

Thanks for your help,

Mark



From david.doukhan at gmail.com  Tue Apr 10 15:58:25 2007
From: david.doukhan at gmail.com (David Doukhan)
Date: Tue, 10 Apr 2007 15:58:25 -0400
Subject: [Numpy-discussion] ndarray allocation question
In-Reply-To: <461B30CF.70409@ieee.org>
References: <b94d068b0704092149i84b9ad2w91b41433b8bb56b7@mail.gmail.com>
	<461B30CF.70409@ieee.org>
Message-ID: <b94d068b0704101258s6b35edd3t4cdaa1d364bd974f@mail.gmail.com>

2007/4/10, Travis Oliphant <oliphant.travis at ieee.org>:
> David Doukhan wrote:
> > Hi!
> > I'm writing you this mail because I would like to do "advanced" use of
> > ndarray memory allocation.
> >
> > So here is a short description of my problem:
> >
> > I'm interfacing multidimensionnal Numpy array to C using the ctypes
> > module. For now, there is no problem.
> > The C program that will use the pointer to the data of the array runs
> > on a CellBE architecture.
>
> > The consequences are:
> > * the adress of the beginning of the data of the array must be a
> > multiple a 16 bytes (that could be specified using the attribute
> > 'align' of dtype)
> > * the memory allocated MUST be a multiple of 16 bytes, and that's my
> > current problem, because I didn't find a way to specify it using dtypes...
>
> You can't specify the alignment for a data-type unless you write your
> own data-type (the alignment is a property of the data-type and is
> determined at compile time for built-in data-types).   I don't think you
> can set the alignment for a derived data-type at this point.
>
> > So, do the think there is a "clean" way to do what I want. For
> > example, asking for an array of 2 lines and 7 columns of float32, so
> > that the adress of the biginning of the data would be a multiple of 16
> > bytes (that's already possible)
> How are you doing that?
>

Sorry, i thought doing the following lines:

clib = ctypes.cdll.LoadLibrary("./libtest.so")
dt = numpy.dtype({'names' : ['x'], 'formats' : [N.float32]}, align=67)
myarray = numpy.empty((5,6,7),dtype=dt)

would create an array such than when doing :

my_pointer = myarray.ctypes.data_as(POINTER(c_float))
clib.print_pointer(my_pointer)

mypointer would be a multiple of 67.

So after checking it, I realised that it is not the case :-((

So I will ask you another question:
* What is the use of the argument align in :
numpy.dtype({'names' : ['x'], 'formats' : [N.float32]}, align=67) ?
* Is there other arguments than "names", "format", "offsets",
"titles", "align" that I could give to numpy.dtype ?

Actually, in a ideal world, I would like to be able to align the
address of the data contained in the array, as if I was doing a
posix_memalign in C, and to make the allocated memory size a multiple
of a given size, even if the real data size is smaller.

It seems that it won't be possible, unless I misunderstood something.

-- 
David Doukhan


From pgmdevlist at gmail.com  Tue Apr 10 16:03:47 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 10 Apr 2007 16:03:47 -0400
Subject: [Numpy-discussion] making two 2D real arrays into one complex
	array
In-Reply-To: <1176235020.247655.111040@h3g2000cwc.googlegroups.com>
References: <1176235020.247655.111040@h3g2000cwc.googlegroups.com>
Message-ID: <200704101603.47761.pgmdevlist@gmail.com>

On Tuesday 10 April 2007 15:57:00 mark wrote:
> Hello List -
>
> Any reason why the following doesn't work?
>
> >>> x,y = meshgrid(linspace(-1,1,10),linspace(-1,1,10))
> >>> z = complex(x,y)

Because complex is a python function that doesn't work well w/ ndarrays. 
int/float/bool would give the same result.

Please try:
z = x + 1j *y


From robert.kern at gmail.com  Tue Apr 10 16:09:00 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 10 Apr 2007 15:09:00 -0500
Subject: [Numpy-discussion] making two 2D real arrays into one complex
 array
In-Reply-To: <1176235020.247655.111040@h3g2000cwc.googlegroups.com>
References: <1176235020.247655.111040@h3g2000cwc.googlegroups.com>
Message-ID: <461BEEDC.8030902@gmail.com>

mark wrote:
> Hello List -
> 
> Any reason why the following doesn't work?
> 
>>>> x,y = meshgrid(linspace(-1,1,10),linspace(-1,1,10))
>>>> z = complex(x,y)
> Traceback (most recent call last):
>   File "<pyshell#225>", line 1, in ?
>     z = complex(x,y)
> TypeError: only length-1 arrays can be converted to Python scalars
> 
> 
> I know I can make an empty complex z array with the size of x, then
> set z.real equal to x and z.imag equal to y. But is there any reason
> why the above shouldn't work?

Yes. complex() is a builtin function for Python. It only takes Python ints and
floats. It knows nothing about numpy.

> It would be easy to implement in numpy,
> wouldn't it? If so, I would *really* like it.

I don't think there's much need for it to be provided in numpy. It's trivial
enough that you can define it yourself wherever you need it.

def complexarray(x, y):
    return x + 1j*y

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From gerard.vermeulen at grenoble.cnrs.fr  Tue Apr 10 16:13:44 2007
From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen)
Date: Tue, 10 Apr 2007 22:13:44 +0200
Subject: [Numpy-discussion] ANN: PyQwt-5.0.0 released
Message-ID: <20070410221344.54f11d6c@zombie.grenoble.cnrs.fr>

What is PyQwt ( http://pyqwt.sourceforge.net ) ?

- it is a set of Python bindings for the Qwt C++ class library which
  extends the Qt framework with widgets for scientific and engineering
  applications.   It provides a widget to plot 2-dimensional data and
  various widgets to display and control bounded or unbounded floating
  point values.

- it requires and extends PyQt, a set of Python bindings for Qt.

- it supports the use of PyQt, Qt, Qwt, and optionally NumPy or SciPy
  in a GUI Python application or in an interactive Python session.

- it runs on POSIX, Mac OS X and Windows platforms (practically any
  platform supported by Qt and Python).

- it plots fast: displaying data with 100,000 points takes about 0.1 s.

- it is licensed under the GPL with an exception to allow dynamic
linking with non-free releases of Qt and PyQt. 


PyQwt-5.0.0 is a major release with support for Qt-4.x, many API changes
compared to PyQwt-4.2.x, and a NSIS Windows installer.


PyQwt-5.0.0 supports:
1. Python-2.5, -2.4 or -2.3. 
2. PyQt-3.18 (to be released in April 2007), or PyQt-3.17.
3. PyQt-4.2 (to be released in April 2007), or PyQt-4.1.x.
3  SIP-4.6 (to be released in April 2007), or SIP-4.5.x.
4. Qt-3.3.x, or -3.2.x.
5. Qt-4.2.x, or -4.1.x.
6. Recent versions of NumPy, numarray, and/or Numeric.


Enjoy -- Gerard Vermeulen


From oliphant at ee.byu.edu  Tue Apr 10 16:59:17 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Tue, 10 Apr 2007 14:59:17 -0600
Subject: [Numpy-discussion] ndarray allocation question
In-Reply-To: <b94d068b0704101258s6b35edd3t4cdaa1d364bd974f@mail.gmail.com>
References: <b94d068b0704092149i84b9ad2w91b41433b8bb56b7@mail.gmail.com>	<461B30CF.70409@ieee.org>
	<b94d068b0704101258s6b35edd3t4cdaa1d364bd974f@mail.gmail.com>
Message-ID: <461BFAA5.2070608@ee.byu.edu>

David Doukhan wrote:

>Sorry, i thought doing the following lines:
>
>clib = ctypes.cdll.LoadLibrary("./libtest.so")
>dt = numpy.dtype({'names' : ['x'], 'formats' : [N.float32]}, align=67)
>myarray = numpy.empty((5,6,7),dtype=dt)
>
>would create an array such than when doing :
>
>my_pointer = myarray.ctypes.data_as(POINTER(c_float))
>clib.print_pointer(my_pointer)
>
>mypointer would be a multiple of 67.
>
>So after checking it, I realised that it is not the case :-((
>  
>

The align argument is a Boolean argument that states whether or not you 
would like the fields aligned as the compiler would align them if they 
were in a C-struct.

>* Is there other arguments than "names", "format", "offsets",
>"titles", "align" that I could give to numpy.dtype ?
>  
>
No.  However, you can have "unused" space in a data-type (i.e. skipped 
bytes).

>Actually, in a ideal world, I would like to be able to align the
>address of the data contained in the array, as if I was doing a
>posix_memalign in C, and to make the allocated memory size a multiple
>of a given size, even if the real data size is smaller.
>It seems that it won't be possible, unless I misunderstood something.
>  
>

Sure it's possible, you just may have to do more work.  The system is 
not set up to make this kind of special case trivial.

There are lots of ways to do it.  For example,  you can allocate your 
own memory (use posix_memalign called through c-types if you want).  You 
can then put an ndarray wrapper around it using a data-type that has the 
floats where-ever you want them in each record. 

It's all quite doable, just not "automatic"

Look at the ndarray constructor for how to build an ndarray from an 
arbitrary "buffer"


-Travis




From david.doukhan at gmail.com  Tue Apr 10 18:05:15 2007
From: david.doukhan at gmail.com (David Doukhan)
Date: Tue, 10 Apr 2007 18:05:15 -0400
Subject: [Numpy-discussion] ndarray allocation question
In-Reply-To: <461BFAA5.2070608@ee.byu.edu>
References: <b94d068b0704092149i84b9ad2w91b41433b8bb56b7@mail.gmail.com>
	<461B30CF.70409@ieee.org>
	<b94d068b0704101258s6b35edd3t4cdaa1d364bd974f@mail.gmail.com>
	<461BFAA5.2070608@ee.byu.edu>
Message-ID: <b94d068b0704101505n3c5f8db7ue377b1ad26271771@mail.gmail.com>

2007/4/10, Travis Oliphant <oliphant at ee.byu.edu>:
> David Doukhan wrote:
>
> >Sorry, i thought doing the following lines:
> >
> >clib = ctypes.cdll.LoadLibrary("./libtest.so")
> >dt = numpy.dtype({'names' : ['x'], 'formats' : [N.float32]}, align=67)
> >myarray = numpy.empty((5,6,7),dtype=dt)
> >
> >would create an array such than when doing :
> >
> >my_pointer = myarray.ctypes.data_as(POINTER(c_float))
> >clib.print_pointer(my_pointer)
> >
> >mypointer would be a multiple of 67.
> >
> >So after checking it, I realised that it is not the case :-((
> >
> >
>
> The align argument is a Boolean argument that states whether or not you
> would like the fields aligned as the compiler would align them if they
> were in a C-struct.
>

Oops!
Sorry for that misunderstanding.
Actually, I did that error because I've seen the following line :

dt = N.dtype({'names' : ['i', 'j'], 'formats' : [N.intc, N.float64]}, align=1)

on this website:  http://www.scipy.org/Cookbook/Ctypes

So I thought the align argument was numeric....
Now, with your explanations, it will be ok.

> >* Is there other arguments than "names", "format", "offsets",
> >"titles", "align" that I could give to numpy.dtype ?
> >
> >
> No.  However, you can have "unused" space in a data-type (i.e. skipped
> bytes).
>
> >Actually, in a ideal world, I would like to be able to align the
> >address of the data contained in the array, as if I was doing a
> >posix_memalign in C, and to make the allocated memory size a multiple
> >of a given size, even if the real data size is smaller.
> >It seems that it won't be possible, unless I misunderstood something.
> >
> >
>
> Sure it's possible, you just may have to do more work.  The system is
> not set up to make this kind of special case trivial.
>
> There are lots of ways to do it.  For example,  you can allocate your
> own memory (use posix_memalign called through c-types if you want).  You
> can then put an ndarray wrapper around it using a data-type that has the
> floats where-ever you want them in each record.
>
> It's all quite doable, just not "automatic"
>
> Look at the ndarray constructor for how to build an ndarray from an
> arbitrary "buffer"
>

I was already doing this kind of stuffs, but I was wondering if there
was a way to automatize it, in order to depend as less as possible of
a C code at this step. My misunderstaing of the use of the align
argument made me think this kind of things could be automatized by
Numpy...

>
> -Travis
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

So thanks for your help, for the speed at wich you managed to answer
my mails, and for your patience.


-- 
David Doukhan


From a.schmolck at gmx.net  Wed Apr 11 11:35:21 2007
From: a.schmolck at gmx.net (Alexander Schmolck)
Date: 11 Apr 2007 16:35:21 +0100
Subject: [Numpy-discussion] [ANN] mlabrap-1.0final: a high level python to
	matlab
Message-ID: <yfstzvmkjpi.fsf@oc.ex.ac.uk>

I'm pleased to finally announce mlabwrap-1.0:

Project website
---------------
<http://mlabwrap.sourceforge.net/>


Description
-----------

Mlabwrap-1.0 is a high-level python to matlab(tm) bridge that makes calling
matlab functions from python almost as convenient as using a normal python
library. It is available under a very liberal license (BSD/MIT) and should
work on all major platforms and (non-ancient) python and matlab versions and
either numpy or Numeric (Numeric support will be dropped in the future).


Examples
--------

Creating a simple line plot:

>>> from mlabwrap import mlab; mlab.plot([1,2,3],'-o')

Creating a surface plot:

>>> from mlabwrap import mlab; from numpy import *
>>> xx = arange(-2*pi, 2*pi, 0.2)
>>> mlab.surf(subtract.outer(sin(xx),cos(xx)))

Creating a neural network and training it on the xor problem (requires netlab)

>>> net = mlab.mlp(2,3,1,'logistic')
>>> net = mlab.mlptrain(net, [[1,1], [0,0], [1,0], [0,1]], [0,0,1,1], 1000)

What the future holds
---------------------

Please note that mlabwrap-1.0 will be the last non-bugfix release with Numeric
support. Future versions of mlabwrap will require numpy be a part of scipy's
scikits infrastructure (so the package name will be ``scikits.mlabwrap``) and
use setuptools rather than distutils so that it should be possible to
automatically download and install via EasyInstall.

The next major version of mlabwrap should also bring more powerful proxying
and marshalling facilities, but the default conversion behavior might be
changed to reflect the fact that matlab is becoming increasingly less
``double`` (-matrix) centric; although wrappers for old-style behavior will be
provided if backwards-incompatible interface changes are introduced, for
upwards compatibility it is recommended to explicitly pass in float64 arrays
rather than e.g. lists of ints if the desired input type that matlab should
see is a double array (i.e. use ``mlab.sin(array([1., 2., 3.])`` rather than
``mlab.sin([1,2,3])`` for production code in order to be on the safe side)).
Please have a look at <http://www.scipy.org/Mlabwrap> if you're interested in
the ongoing development of mlabwrap and planned features.


Feedback and support
--------------------

The preferred formum for users to request help and offer feedback and keep
informed about new releases is mlabwrap-user:

 <https://lists.sourceforge.net/lists/listinfo/mlabwrap-user>

the list is low-volume and subscription is recommended.

Discussion of mlabwrap development takes place on the scipy-dev (please
mention mlabwrap in the subject line):

 <http://projects.scipy.org/mailman/listinfo/scipy-dev> 

cheers,

Alexander Schmolck, mlabwrap author and maintainer


From mkg at cs.nyu.edu  Wed Apr 11 18:12:16 2007
From: mkg at cs.nyu.edu (Matthew Koichi Grimes)
Date: Wed, 11 Apr 2007 18:12:16 -0400
Subject: [Numpy-discussion] detecting shared data
Message-ID: <461D5D40.6070509@cs.nyu.edu>

Is there any way to detect whether one array is a view into another 
array? I'd like something like:

 >>> arr = N.arange(5)
 >>> subarr = arr[1:3]
 >>> sharesdata(arr, subarr)
True

-- Matt


From pgmdevlist at gmail.com  Wed Apr 11 18:18:26 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 11 Apr 2007 18:18:26 -0400
Subject: [Numpy-discussion] detecting shared data
In-Reply-To: <461D5D40.6070509@cs.nyu.edu>
References: <461D5D40.6070509@cs.nyu.edu>
Message-ID: <200704111818.26461.pgmdevlist@gmail.com>

On Wednesday 11 April 2007 18:12:16 Matthew Koichi Grimes wrote:
> Is there any way to detect whether one array is a view into another
>
> array? I'd like something like:
>  >>> arr = N.arange(5)
>  >>> subarr = arr[1:3]
>  >>> sharesdata(arr, subarr)

Mmh, would arr.flags['OWNDATA'] would do the trick ?
p.


From stefan at sun.ac.za  Wed Apr 11 18:28:24 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 12 Apr 2007 00:28:24 +0200
Subject: [Numpy-discussion] detecting shared data
In-Reply-To: <461D5D40.6070509@cs.nyu.edu>
References: <461D5D40.6070509@cs.nyu.edu>
Message-ID: <20070411222824.GC18196@mentat.za.net>

On Wed, Apr 11, 2007 at 06:12:16PM -0400, Matthew Koichi Grimes wrote:
> Is there any way to detect whether one array is a view into another 
> array? I'd like something like:
> 
>  >>> arr = N.arange(5)
>  >>> subarr = arr[1:3]
>  >>> sharesdata(arr, subarr)
> True

Your best bet is probably

N.may_share_memory(arr,subarr)

Cheers
St?fan


From mkg at cs.nyu.edu  Wed Apr 11 18:37:05 2007
From: mkg at cs.nyu.edu (Matthew Koichi Grimes)
Date: Wed, 11 Apr 2007 18:37:05 -0400
Subject: [Numpy-discussion] detecting shared data
In-Reply-To: <200704111818.26461.pgmdevlist@gmail.com>
References: <461D5D40.6070509@cs.nyu.edu>
	<200704111818.26461.pgmdevlist@gmail.com>
Message-ID: <461D6311.6090400@cs.nyu.edu>

It's better than nothing. I basically want some sanity-check assert code 
that can assert that some arrays are in fact sub-arrays of another 
array. Your OWNDATA suggestion meets me halfway by allowing me to check 
that these sub-arrays are at least sub-arrays of *someone*.

Thanks,
-- Matt

Pierre GM wrote:
> On Wednesday 11 April 2007 18:12:16 Matthew Koichi Grimes wrote:
>   
>> Is there any way to detect whether one array is a view into another
>>
>> array? I'd like something like:
>>  >>> arr = N.arange(5)
>>  >>> subarr = arr[1:3]
>>  >>> sharesdata(arr, subarr)
>>     
>
> Mmh, would arr.flags['OWNDATA'] would do the trick ?
> p.
>   



From wbaxter at gmail.com  Wed Apr 11 19:34:18 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Thu, 12 Apr 2007 08:34:18 +0900
Subject: [Numpy-discussion] detecting shared data
In-Reply-To: <461D6311.6090400@cs.nyu.edu>
References: <461D5D40.6070509@cs.nyu.edu>
	<200704111818.26461.pgmdevlist@gmail.com>
	<461D6311.6090400@cs.nyu.edu>
Message-ID: <e86a5fd00704111634j2883e742s5b07a592179aed32@mail.gmail.com>

On 4/12/07, Matthew Koichi Grimes <mkg at cs.nyu.edu> wrote:
> It's better than nothing. I basically want some sanity-check assert code
> that can assert that some arrays are in fact sub-arrays of another
> array. Your OWNDATA suggestion meets me halfway by allowing me to check
> that these sub-arrays are at least sub-arrays of *someone*.
>
> Thanks,
> -- Matt
>
> Pierre GM wrote:
> > On Wednesday 11 April 2007 18:12:16 Matthew Koichi Grimes wrote:
> >
> >> Is there any way to detect whether one array is a view into another
> >>
> >> array? I'd like something like:
> >>  >>> arr = N.arange(5)
> >>  >>> subarr = arr[1:3]
> >>  >>> sharesdata(arr, subarr)
> >>
> >
> > Mmh, would arr.flags['OWNDATA'] would do the trick ?
> > p.

I wrote this a while back that may do what you want:


def same_array(a, b):
    """Tries to figure out if a and b are sharing (some of) the same
memory or not.
    This is sometimes useful for determining if a copy was made as a
result of some
    operation, or for determining if modifying a might also modify b.
    A True result means that a and b both borrow memory from the same
array object,
    but it does not necessarily mean the memory regions used overlap.
For example:
    >>> x = rand(4,4)
    >>> a,b = x[:2], x[2:]
    >>> same_array(a,b)
    True
    A False result means the arrays definitely do not overlap in memory.
    same_array(a, b) -> Bool
    """
    ab = a
    while ab.base is not None:
        ab = ab.base
    bb = b
    while bb.base is not None:
        bb = bb.base
    return ab is bb

But maybe that's pretty much what may_share_memory does?
--bb


From Chris.Barker at noaa.gov  Wed Apr 11 19:43:07 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 11 Apr 2007 16:43:07 -0700
Subject: [Numpy-discussion] detecting shared data
In-Reply-To: <e86a5fd00704111634j2883e742s5b07a592179aed32@mail.gmail.com>
References: <461D5D40.6070509@cs.nyu.edu>
	<200704111818.26461.pgmdevlist@gmail.com> <461D6311.6090400@cs.nyu.edu>
	<e86a5fd00704111634j2883e742s5b07a592179aed32@mail.gmail.com>
Message-ID: <461D728B.9090104@noaa.gov>



Bill Baxter wrote:

> But maybe that's pretty much what may_share_memory does?

I think so. Travis added it after a discussion much like this one on the 
list.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From wbaxter at gmail.com  Wed Apr 11 19:53:49 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Thu, 12 Apr 2007 08:53:49 +0900
Subject: [Numpy-discussion] detecting shared data
In-Reply-To: <461D728B.9090104@noaa.gov>
References: <461D5D40.6070509@cs.nyu.edu>
	<200704111818.26461.pgmdevlist@gmail.com>
	<461D6311.6090400@cs.nyu.edu>
	<e86a5fd00704111634j2883e742s5b07a592179aed32@mail.gmail.com>
	<461D728B.9090104@noaa.gov>
Message-ID: <e86a5fd00704111653m3182259o4c6492e9576d8aa1@mail.gmail.com>

On 4/12/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
>
>
> Bill Baxter wrote:
>
> > But maybe that's pretty much what may_share_memory does?
>
> I think so. Travis added it after a discussion much like this one on the
> list.

Must be pretty recent.  I'm using 1.0.2.dev3520 (enthought egg) and
the function's not there.

--bb


From jturner at gemini.edu  Wed Apr 11 20:10:24 2007
From: jturner at gemini.edu (James Turner)
Date: Wed, 11 Apr 2007 20:10:24 -0400
Subject: [Numpy-discussion] nd_image.affine_transform edge effects
In-Reply-To: <20070405101530.GU18196@mentat.za.net>
References: <20070405101530.GU18196@mentat.za.net>
Message-ID: <461D78F0.40709@gemini.edu>

Hi Stefan,

Sorry for the slow reply to this.

 > Thanks for spotting that.  When I fix those lines, I see:
 >
 > [[ 3.9000001   3.0999999   2.0999999   1.10000002  1.89999998  2.9000001 ]
 >  [ 3.9000001   3.0999999   2.0999999   1.10000002  1.89999998  2.9000001 ]]

Actually, I think I made a mistake in my first post of 4 April, when I
suggested the first value should be 3.9. I was thinking that the "reflect"
mode just mirrors the input (like the "mirror" mode) and then interpolates
the resulting values, in which case the first value would be 3.9 and the
last would be 1.9 and 2.9. That seems to be what is now happening in your
example above. However, I later realized that it is probably supposed to
interpolate only within the bounds of the input data and *then* pad the
output by "reflecting" the *interpolated* values in the way you describe
below. What confused me was the extra 1.9 in the 2nd-last output column,
but I think that is just the same problem (calculating one point too many)
that I reported and you fixed for the "constant" mode, isn't it?

If this is true, I think the (rounded) result should in fact be

[[ 3.1  3.1  2.1  1.1  1.1  2.1]
  [ 3.1  3.1  2.1  1.1  1.1  2.1]]

Does that make sense? Sorry if I confused the issue previously.

So that's my impression of the intended behaviour, but whether it makes
the most sense or not is a different matter; I don't personally have a
use for padding with reflected values at the moment. I notice, however,
that what I'm describing here can preserve symmetry when doubling the
original array size (eg. [ 3.1  3.1  2.1  1.1  1.1  2.1  3.1  3.1 ]),
if shifted appropriately, which may be the idea of "reflect" as opposed
to "mirror" (otherwise the latter makes more sense to me). Unfortunately,
interpolating breaks this symmetry for more general multiples, by
unavoidably reducing the number of known values by one.

 > I'll submit to SVN later today.  Note that I also enabled 'mirror'
 > mode, which works almost the same way as reflect:
 >
 > Reflect: 1 2 3 4 -> 1 2 3 4 4 3 2 1
 > Mirror:  1 2 3 4 -> 1 2 3 4 3 2 1

Yes, I noticed the mirror mode. It does seem good to have both options.
I may have used the words "mirror" and "reflect" interchangeably in
previous emails though.

Cheers,

James.



From peridot.faceted at gmail.com  Wed Apr 11 23:06:13 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 11 Apr 2007 23:06:13 -0400
Subject: [Numpy-discussion] detecting shared data
In-Reply-To: <e86a5fd00704111653m3182259o4c6492e9576d8aa1@mail.gmail.com>
References: <461D5D40.6070509@cs.nyu.edu>
	<200704111818.26461.pgmdevlist@gmail.com>
	<461D6311.6090400@cs.nyu.edu>
	<e86a5fd00704111634j2883e742s5b07a592179aed32@mail.gmail.com>
	<461D728B.9090104@noaa.gov>
	<e86a5fd00704111653m3182259o4c6492e9576d8aa1@mail.gmail.com>
Message-ID: <ce557a360704112006w4f83352ey9ddef34de94a9c6c@mail.gmail.com>

On 11/04/07, Bill Baxter <wbaxter at gmail.com> wrote:

> Must be pretty recent.  I'm using 1.0.2.dev3520 (enthought egg) and
> the function's not there.

It is.

I've never been quite happy with it, though; I realize it's not very
feasible to write one that efficiently checks all possible overlaps,
but the current one claims (e.g.) a.real and a.imag may share memory,
which irks me. I put in a quick fix. Also, may_share_memory did not
have any tests (for shame!), so I put some of those in too. I wasn't
sure how to express "expected failure", and they're not the most
thorough, but they should help.

Anne M. Archibald
-------------- next part --------------
A non-text attachment was scrubbed...
Name: share-patch
Type: application/octet-stream
Size: 4065 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070411/a1e04583/attachment.obj>

From bronto at pobox.com  Thu Apr 12 02:19:31 2007
From: bronto at pobox.com (Anton Sherwood)
Date: Wed, 11 Apr 2007 23:19:31 -0700
Subject: [Numpy-discussion] MacNoob
Message-ID: <461DCF73.7090605@pobox.com>

When I try to build numpy (or PIL) on MacOS, I get this error:
"gcc: cannot specify -o with -c or -S and multiple compilations"

Assuming at least one of you is using numpy on MacOS,
how did you get around that?

Thanks.

-- 
Anton Sherwood, http://www.ogre.nu/
"How'd ya like to climb this high *without* no mountain?" --Porky Pine


From robert.kern at gmail.com  Thu Apr 12 02:26:59 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Apr 2007 01:26:59 -0500
Subject: [Numpy-discussion] MacNoob
In-Reply-To: <461DCF73.7090605@pobox.com>
References: <461DCF73.7090605@pobox.com>
Message-ID: <461DD133.7040102@gmail.com>

Anton Sherwood wrote:
> When I try to build numpy (or PIL) on MacOS, I get this error:
> "gcc: cannot specify -o with -c or -S and multiple compilations"
> 
> Assuming at least one of you is using numpy on MacOS,
> how did you get around that?

I don't get that error for either package.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Thu Apr 12 03:55:44 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 12 Apr 2007 09:55:44 +0200
Subject: [Numpy-discussion] detecting shared data
In-Reply-To: <ce557a360704112006w4f83352ey9ddef34de94a9c6c@mail.gmail.com>
References: <461D5D40.6070509@cs.nyu.edu>
	<200704111818.26461.pgmdevlist@gmail.com>
	<461D6311.6090400@cs.nyu.edu>
	<e86a5fd00704111634j2883e742s5b07a592179aed32@mail.gmail.com>
	<461D728B.9090104@noaa.gov>
	<e86a5fd00704111653m3182259o4c6492e9576d8aa1@mail.gmail.com>
	<ce557a360704112006w4f83352ey9ddef34de94a9c6c@mail.gmail.com>
Message-ID: <20070412075544.GE18196@mentat.za.net>

On Wed, Apr 11, 2007 at 11:06:13PM -0400, Anne Archibald wrote:
> On 11/04/07, Bill Baxter <wbaxter at gmail.com> wrote:
> 
> >Must be pretty recent.  I'm using 1.0.2.dev3520 (enthought egg) and
> >the function's not there.
> 
> It is.
> 
> I've never been quite happy with it, though; I realize it's not very
> feasible to write one that efficiently checks all possible overlaps,
> but the current one claims (e.g.) a.real and a.imag may share memory,
> which irks me. I put in a quick fix. Also, may_share_memory did not
> have any tests (for shame!), so I put some of those in too. I wasn't
> sure how to express "expected failure", and they're not the most
> thorough, but they should help.

Thank you for taking the time to write those tests!

Failures may be expressed using

NumpyTestCase.failIf(self, expr, msg=None)

Regards
St?fan


From bernhard.voigt at gmail.com  Thu Apr 12 04:35:01 2007
From: bernhard.voigt at gmail.com (Bernhard Voigt)
Date: Thu, 12 Apr 2007 10:35:01 +0200
Subject: [Numpy-discussion] Slicing with index lists
Message-ID: <21a270aa0704120135g6ff7f7adj77ec0dc1557893f4@mail.gmail.com>

Dear all!

in a 2-dim array I would like to select rows specified by a list of indexes
and from these rows I'd like to select columns specified by another list of
indexes. That's what I found as a solution:

In [90]: a = arange(15).reshape(5,3)
In [91]: a
Out[91]:
array([[ 0,  1,  2],
       [ 3,  4,  5],
       [ 6,  7,  8],
       [ 9, 10, 11],
       [12, 13, 14]])

# select rows 1,3,4 and columns 0,2

In[92]: a[[1,3,4],:][:,[0,2]]
Out[92]:
array([[ 3,  5],
       [ 9, 11],
       [12, 14]])

Is there a better way to express this selection? I thought there might be
single slicing expression for this, but I couldn't figure it out :-(

I tried:

In [97]: a[[1,3,4],[0,2]]
---------------------------------------------------------------------------
exceptions.ValueError                                Traceback (most recent
call last)

ValueError: shape mismatch: objects cannot be broadcast to a single shape

Thanks for help! Bernhard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070412/7a7742d5/attachment.html>

From robert.kern at gmail.com  Thu Apr 12 04:43:19 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Apr 2007 03:43:19 -0500
Subject: [Numpy-discussion] Slicing with index lists
In-Reply-To: <21a270aa0704120135g6ff7f7adj77ec0dc1557893f4@mail.gmail.com>
References: <21a270aa0704120135g6ff7f7adj77ec0dc1557893f4@mail.gmail.com>
Message-ID: <461DF127.6070106@gmail.com>

Bernhard Voigt wrote:
> Dear all!
> 
> in a 2-dim array I would like to select rows specified by a list of
> indexes and from these rows I'd like to select columns specified by
> another list of indexes. That's what I found as a solution:
> 
> In [90]: a = arange(15).reshape(5,3)
> In [91]: a
> Out[91]:
> array([[ 0,  1,  2],
>        [ 3,  4,  5],
>        [ 6,  7,  8],
>        [ 9, 10, 11],
>        [12, 13, 14]])
> 
> # select rows 1,3,4 and columns 0,2
> 
> In[92]: a[[1,3,4],:][:,[0,2]]
> Out[92]:
> array([[ 3,  5],
>        [ 9, 11],
>        [12, 14]])
> 
> Is there a better way to express this selection? I thought there might
> be single slicing expression for this, but I couldn't figure it out :-(

In [3]: from numpy import *

In [4]: a = arange(15).reshape(5,3)

In [5]: a[ix_([1,3,4], [0,2])]
Out[5]:
array([[ 3,  5],
       [ 9, 11],
       [12, 14]])

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From v-nijs at kellogg.northwestern.edu  Thu Apr 12 08:56:24 2007
From: v-nijs at kellogg.northwestern.edu (Vincent Nijs)
Date: Thu, 12 Apr 2007 07:56:24 -0500
Subject: [Numpy-discussion] MacNoob
In-Reply-To: <461DCF73.7090605@pobox.com>
Message-ID: <C24396A8.579E%v-nijs@kellogg.northwestern.edu>

I think that error went away when I used the latest developer tools from
Apple, and made sure I was using gcc4.

Take a look at:

http://www.scipy.org/Installing_SciPy/Mac_OS_X

Vincent


On 4/12/07 1:19 AM, "Anton Sherwood" <bronto at pobox.com> wrote:

> When I try to build numpy (or PIL) on MacOS, I get this error:
> "gcc: cannot specify -o with -c or -S and multiple compilations"
> 
> Assuming at least one of you is using numpy on MacOS,
> how did you get around that?
> 
> Thanks.




From peridot.faceted at gmail.com  Thu Apr 12 11:29:31 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 12 Apr 2007 11:29:31 -0400
Subject: [Numpy-discussion] detecting shared data
In-Reply-To: <20070412075544.GE18196@mentat.za.net>
References: <461D5D40.6070509@cs.nyu.edu>
	<200704111818.26461.pgmdevlist@gmail.com>
	<461D6311.6090400@cs.nyu.edu>
	<e86a5fd00704111634j2883e742s5b07a592179aed32@mail.gmail.com>
	<461D728B.9090104@noaa.gov>
	<e86a5fd00704111653m3182259o4c6492e9576d8aa1@mail.gmail.com>
	<ce557a360704112006w4f83352ey9ddef34de94a9c6c@mail.gmail.com>
	<20070412075544.GE18196@mentat.za.net>
Message-ID: <ce557a360704120829r3a7b960fub4acd0a91214d83f@mail.gmail.com>

On 12/04/07, Stefan van der Walt <stefan at sun.ac.za> wrote:

> Thank you for taking the time to write those tests!
>
> Failures may be expressed using
>
> NumpyTestCase.failIf(self, expr, msg=None)

That's not quite what I mean. There are situations, with the current
code, that it gets the answer wrong (i.e., claims arrays may share
memory when they don't). I know, and it's okay, and if it doesn't
there's a bug, but in view of possible future enhancements, I don't
want to signal an actual failure if it starts working. I do want to
test it though, so I was hoping there was a way to express "I expect
this test to fail, notify me if it doesn't, but don't call it a
failure if it starts working".

Anne


From stefan at sun.ac.za  Thu Apr 12 12:13:06 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 12 Apr 2007 18:13:06 +0200
Subject: [Numpy-discussion] detecting shared data
In-Reply-To: <ce557a360704120829r3a7b960fub4acd0a91214d83f@mail.gmail.com>
References: <461D5D40.6070509@cs.nyu.edu>
	<200704111818.26461.pgmdevlist@gmail.com>
	<461D6311.6090400@cs.nyu.edu>
	<e86a5fd00704111634j2883e742s5b07a592179aed32@mail.gmail.com>
	<461D728B.9090104@noaa.gov>
	<e86a5fd00704111653m3182259o4c6492e9576d8aa1@mail.gmail.com>
	<ce557a360704112006w4f83352ey9ddef34de94a9c6c@mail.gmail.com>
	<20070412075544.GE18196@mentat.za.net>
	<ce557a360704120829r3a7b960fub4acd0a91214d83f@mail.gmail.com>
Message-ID: <20070412161306.GK18196@mentat.za.net>

On Thu, Apr 12, 2007 at 11:29:31AM -0400, Anne Archibald wrote:
> > Failures may be expressed using
> >
> > NumpyTestCase.failIf(self, expr, msg=None)
> 
> That's not quite what I mean. There are situations, with the current
> code, that it gets the answer wrong (i.e., claims arrays may share
> memory when they don't). I know, and it's okay, and if it doesn't
> there's a bug, but in view of possible future enhancements, I don't
> want to signal an actual failure if it starts working. I do want to
> test it though, so I was hoping there was a way to express "I expect
> this test to fail, notify me if it doesn't, but don't call it a
> failure if it starts working".

If the test is supposed to pass but currently fails, we can always add
it using level=50 or so.  That way, most people who run the tests will
not see the failure, but the devs may still choose to run them.  We
could even display a warning when running the test suite with such a
high level.

Would such a scheme cause problems for anyone?  An alternative would
be to rework the test gatherer to filter tests based on some flag.

Cheers
St?fan


From Chris.Barker at noaa.gov  Thu Apr 12 12:17:45 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 12 Apr 2007 09:17:45 -0700
Subject: [Numpy-discussion] MacNoob
In-Reply-To: <461DCF73.7090605@pobox.com>
References: <461DCF73.7090605@pobox.com>
Message-ID: <461E5BA9.3020106@noaa.gov>

Anton Sherwood wrote:
> When I try to build numpy (or PIL) on MacOS, I get this error:
> "gcc: cannot specify -o with -c or -S and multiple compilations"
> 
> Assuming at least one of you is using numpy on MacOS,
> how did you get around that?
by installing a binary from:

pythonmac.org/packages

Otherwise, you'll need to tell us more aobut your build environment:

python version (what build, where did you get it?)

OS version

compiler version. (I'd use the latest Xcode distro.)

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From hurak at fel.cvut.cz  Thu Apr 12 14:35:39 2007
From: hurak at fel.cvut.cz (=?UTF-8?B?WmRlbsSbayBIdXLDoWs=?=)
Date: Thu, 12 Apr 2007 20:35:39 +0200
Subject: [Numpy-discussion] F2PY and underscore problem
References: <ev2plp$vff$1@sea.gmane.org> <461528C1.8070302@gmail.com>
	<ev3cft$d9d$1@sea.gmane.org>
	<loom.20070406T222258-369@post.gmane.org>
Message-ID: <evlu5r$io5$1@sea.gmane.org>

Tyler J Hayes wrote:

> I'm not sure if this will help, but have you tried to specify the gfortran
> compiler via the option:
> 
> --fcompiler=gnu95
> 
> when calling f2py in the second line after the ".pyf" file has been
> created? Perhaps that will work.

Sorry for late response, Tyler. But this suggestion does not bring
improvement. In fact, as I can see during the process of creating the
module, the system can find the right compiler (in fact, there is only one
fortran compiler on my machine GNU gfortran ---> gnu95).

Thanks anyway.

Zdenek



From oliver.siemoneit at web.de  Thu Apr 12 14:53:00 2007
From: oliver.siemoneit at web.de (Oliver Siemoneit)
Date: Thu, 12 Apr 2007 20:53:00 +0200
Subject: [Numpy-discussion] Syntax trouble in translating matlab code to
	numpy
Message-ID: <001101c77d33$cbcf8f10$fe7aa8c0@dusk>

Dear numpy experts!

I'm just trying to port some matlab algorithm to python which allows an 
image correction for color blind users. If I succeed this bit of code might 
be part of the MoinMoin wiki, the wikisoftware scipy.org uses (see for more 
on that http://moinmoin.wikiwikiweb.de/AccessibleMoin). Since I'm totally 
new to matlab and numpy it took me already hours to figure the basics out - 
without much success. So I decided to ask here on the mailing list for some 
help. I hope - as an exception - this is ok for you...

Here's the part of the matlab code which should be translated. The last 5 
lines actually cause the trouble (long version, see 
http://www.stanford.edu/~ofidaner/psych221_proj/colorblindness_project.htm):
    clear;
    file_name = 'grandparents' ;
    % transorm matrices
    rgb2lms = [17.8824 43.5161 4.11935; 3.45565 27.1554 3.86714; 0.0299566 
0.184309 1.46709] ;
    lms2rgb = inv(rgb2lms) ;

    %read picture into RGB value
    RGB = double(imread(file_name,'jpeg'));
    sizeRGB = size(RGB) ;

    %transform to LMS space
    for i = 1:sizeRGB(1)
       for j = 1:sizeRGB(2)
            rgb = RGB(i,j,:);
            rgb = rgb(:);
            LMS(i,j,:) = rgb2lms * rgb;
        end
    end

And that's what I did with numpy and PIL in Python. The last 2 lines seems 
to be wrong. But how to do it correct?

    im = Image.open(fpath) # fpath is the path to the image
    RGB = numpy.asarray(im) # checked: RGB is of shape[y,x,r/g/b]
    # Colorspace transformation matrices
    rgb2lms = 
numpy.array([[17.8824,43.5161,4.11935],[3.45565,27.1554,3.86714],[0.0299566,0.184309,1.46709]])
    lms2rgb = numpy.linalg.inv(rgb2lms)

    # Transform to LMS space
    LMS = numpy.zeros_like(RGB)
    for i in range(RGB.shape[0]):
        for j in range(RGB.shape[1]):
            rgb = RGB[i,j,:2]
            LMS[i,j,:2] = numpy.dot(rgb2lms, rgb)

This code fails in the last line with the error message "ValueError: 
matrices are not aligned". But how to align them?

Any help would be highly appreciated. Many thanks in advance!

With regards,
Oliver Siemoneit





From robert.kern at gmail.com  Thu Apr 12 15:09:12 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Apr 2007 14:09:12 -0500
Subject: [Numpy-discussion] Syntax trouble in translating matlab code
 to	numpy
In-Reply-To: <001101c77d33$cbcf8f10$fe7aa8c0@dusk>
References: <001101c77d33$cbcf8f10$fe7aa8c0@dusk>
Message-ID: <461E83D8.9070208@gmail.com>

Oliver Siemoneit wrote:

>     # Transform to LMS space
>     LMS = numpy.zeros_like(RGB)
>     for i in range(RGB.shape[0]):
>         for j in range(RGB.shape[1]):
>             rgb = RGB[i,j,:2]
>             LMS[i,j,:2] = numpy.dot(rgb2lms, rgb)
> 
> This code fails in the last line with the error message "ValueError: 
> matrices are not aligned". But how to align them?

You want :3, not :2.

By the way, I have some code that applies a more rigorous model for
colorblindness. The class LinearCorrespondingPairs in this file:

http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/colormap_explorer/file/9955ae141feb/colormap_explorer/vision_model.py

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Thu Apr 12 15:18:06 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Apr 2007 14:18:06 -0500
Subject: [Numpy-discussion] detecting shared data
In-Reply-To: <20070412161306.GK18196@mentat.za.net>
References: <461D5D40.6070509@cs.nyu.edu>	<200704111818.26461.pgmdevlist@gmail.com>	<461D6311.6090400@cs.nyu.edu>	<e86a5fd00704111634j2883e742s5b07a592179aed32@mail.gmail.com>	<461D728B.9090104@noaa.gov>	<e86a5fd00704111653m3182259o4c6492e9576d8aa1@mail.gmail.com>	<ce557a360704112006w4f83352ey9ddef34de94a9c6c@mail.gmail.com>	<20070412075544.GE18196@mentat.za.net>	<ce557a360704120829r3a7b960fub4acd0a91214d83f@mail.gmail.com>
	<20070412161306.GK18196@mentat.za.net>
Message-ID: <461E85EE.7060601@gmail.com>

Stefan van der Walt wrote:
> Would such a scheme cause problems for anyone?

Yes. Don't check in tests that fail. If you want to keep around a test file for
a feature that's in development, put it elsewhere. You can make a directory in
branches/ if you like (it doesn't actually have to be a full branch of the
trunk/, it's just a convenient place to put extra stuff).

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From numpy at hugo.doemaarwat.nl  Fri Apr 13 06:24:59 2007
From: numpy at hugo.doemaarwat.nl (Hugo)
Date: Fri, 13 Apr 2007 12:24:59 +0200
Subject: [Numpy-discussion] endianness ufuncs
Message-ID: <461F5A7B.9030104@hugo.doemaarwat.nl>

Hi,

As as astronomer I work with big endian arrays (pyfits 1.1rc1 with numpy 
1.0) on little endian machines. When I try to make a ufunc (e.g. to 
convert from degrees to radians), it doesn't recognize the big endianness:

In [3]: big = numpy.array([83.4, 83.5, 83.9], dtype='>f8')
In [4]: uradians = numpy.frompyfunc(math.radians, 1, 1)
In [6]: uradians(big)
Out[6]: array([-2.6919063283e-182, 1.26773292198e-318, 
-2.69190632908e-182], dtype=object)

I cannot figure out whether this is the intended behavior but it sure is 
counterintuitive, since most of the time numpy handles the endianness 
transparently.

For now I'm happy with just converting everything to native (little) 
endian at the start of my calculations and convert them back in the end. 
(Although I'm not sure what's the best way to do that yet.)

Hugo


From wbaxter at gmail.com  Fri Apr 13 06:43:44 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Fri, 13 Apr 2007 19:43:44 +0900
Subject: [Numpy-discussion] Fastest distance matrix calc
Message-ID: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>

I think someone posted some timings about this before but I don't recall.

The task is to compute the matrix D from two sets of vectors x (M,d)
and y (N,d).
The output should be  D where D[i,j] is norm(x[i]-y[j])

The Matlab NetLab toolkit uses something like this to compute it:

    d2 = (x*x).sum(1)[:,numpy.newaxis] + (y*y).sum(1) - 2*mult(x,y.T)

And then does
  d2[d2<0]=0
because roundoff in the above can sometimes create negative values.  And finally
  d = sqrt(d2)

But it just doesn't seem like the best way to do it.  Whatever was
posted before I don't remember anything about a subtract.outer
solution.  Seems like something along the lines of
     (subtract.outer(x,y)**2).sum(axis=0)
might be faster, and also avoid the need to check for negatives.

I'll do some timings if no one's already done it.  Just wanted to check first.

--bb


From bacmsantos at gmail.com  Fri Apr 13 06:50:30 2007
From: bacmsantos at gmail.com (Bruno Santos)
Date: Fri, 13 Apr 2007 11:50:30 +0100
Subject: [Numpy-discussion] Help using numPy to create a very large multi
	dimensional array
Message-ID: <699044520704130350j7b53e36br5c0bc068ec00643c@mail.gmail.com>

Dear Sirs,
I'm trying to use Numpy to solve a speed problem with Python, I need to
perform agglomerative clustering as a first step to k-means clustering.
My problem is that I'm using a very large list in Pyhton and the script is
taking more than 9minutes to process all the information, so I'm trying to
use Numpy to create a matrix.
I'm reading the vectors from a text file and I end up with an array of
115*2634 float elements, How can I create this structure with numpy?

Where is my code in python:
#Read each document vector to a matrix
    doclist = []
    matrix = []
    list = []
    for line in vecfile:
        list = line.split()
        for elem in range(1, len(list)):
            list[elem] = float(list[elem])
        matrix.append (list[1:])
    vecfile.close()

    #Read the desired number of final clusters
    numclust = input('Input the desired number of clusters: ')

#Clustering process
    clust = rows
    ind = [-1, -1]
    list_j=[]
    list_k=[]
    while (clust > numclust):
        min = 2147483647
        print('Number of Clusters %d \n' % clust)
        #Find the 2 most similares vectors in the file
        for j in range(0, clust):
            list_j=matrix[j]
            for k in range(j+1, clust):
                list_k=matrix[k]
                dist=0
                for e in range(0, columns):
                    result = list_j[e] - list_k[e]
                    dist += result * result
                if (dist < min):
                    ind[0] = j
                    ind[1] = k
                    min = dist

        #Combine the two most similaires vectores by median
        for e in range(0, columns): matrix[ind[0]][e] = (matrix[ind[0]][e] +
matrix[ind[1]][e]) / 2.0
        clust = clust -1

        #Move up all the remaining vectors
        for k in range(ind[1], (rows - 1)):
            for e in range(0, columns): matrix[k][e]=matrix[k+1][e]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070413/3f8fabcb/attachment.html>

From stefan at sun.ac.za  Fri Apr 13 07:19:53 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Fri, 13 Apr 2007 13:19:53 +0200
Subject: [Numpy-discussion] nd_image.affine_transform edge effects
In-Reply-To: <461D78F0.40709@gemini.edu>
References: <20070405101530.GU18196@mentat.za.net> <461D78F0.40709@gemini.edu>
Message-ID: <20070413111953.GU18196@mentat.za.net>

On Wed, Apr 11, 2007 at 08:10:24PM -0400, James Turner wrote:
> Hi Stefan,
> 
> Sorry for the slow reply to this.
> 
>  > Thanks for spotting that.  When I fix those lines, I see:
>  >
>  > [[ 3.9000001   3.0999999   2.0999999   1.10000002  1.89999998  2.9000001 ]
>  >  [ 3.9000001   3.0999999   2.0999999   1.10000002  1.89999998  2.9000001 ]]
> 
> Actually, I think I made a mistake in my first post of 4 April, when I
> suggested the first value should be 3.9. I was thinking that the "reflect"
> mode just mirrors the input (like the "mirror" mode) and then interpolates
> the resulting values, in which case the first value would be 3.9 and the
> last would be 1.9 and 2.9. That seems to be what is now happening in your
> example above. However, I later realized that it is probably supposed to
> interpolate only within the bounds of the input data and *then* pad the
> output by "reflecting" the *interpolated* values in the way you describe
> below. What confused me was the extra 1.9 in the 2nd-last output column,
> but I think that is just the same problem (calculating one point too many)
> that I reported and you fixed for the "constant" mode, isn't it?

Yes, there is a fundamental problem with using "reflect" mode for
interpolation.  You have data points

4 3 2 1
| | | |

so you can interpolate between the 4 and the 1.  Now, when you use
"reflect" mode, the data becomes:

1 2 3 4 4 3 2 1 1 2 3 4
        | | | |

This is where things become problematic.  If we try to interpolate a
point between the two 4's, we are going to get strange results because
of the spline-fitting routine (you remember the problems with
extrapolation we've had before).  So, the easiest way to fix this is
simply not to use reflect-mode, but to use mirror-mode instead.

1 2 3 4 3 2 1 2 3 4
      | | | |

This causes no problem, since, no matter where you interpolate, no
extrapolation is done.

I've enabled mirror-mode, and put in a warning whenever the user tries
to interpolate using reflect-mode.  There might still be a few minor
indexing issues, but fixing them unfortunately won't alleviate this
bigger design issue.

To address this problem would entail extending the input dataset by
the necessary number of elements (which depends on the order of
interpolation), using the applicable mode (i.e. mirror, reflect etc.).
Then, all indexing operations need to be adjusted accordingly.  Since
everything works at the moment when using mirror and constant mode,
I'm not sure all that effort is warranted.

Regards
St?fan


From victor at bit-man.com.ar  Fri Apr 13 08:42:14 2007
From: victor at bit-man.com.ar (=?ISO-8859-1?Q?=22V=EDctor_A=2E_Rodr=EDguez=22?=)
Date: Fri, 13 Apr 2007 09:42:14 -0300
Subject: [Numpy-discussion] Newbie Questio: Unused in wild import
Message-ID: <461F7AA6.9060206@bit-man.com.ar>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I'm really new to Python and SciPy and having a strange problem.
I've installed Python 2.5 on Windows, numpy (last version), Eclipse 3.2
and latest PyDev extensions.

When importing numpy ("from numpy import *") the next warning is issued :

Unused in wild import: from buffer, c_, FLOATING_POINT_SUPPORT, time,

... and a lot of symbol name ...

There's no problem in using numpy but i'm just curious about it.
BTW, sorry if this is not the correct list to post this.

Cheers.
- --
V?ctor A. Rodr?guez (http://www.Bit-Man.com.ar/)
Perl Mongers Capital Federal (http://cafe.pm.org/)
GNU/Linux User Group - FCEyN - UBA (http://glugcen.dc.uba.ar/)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGH3qm34Jsi77jNMIRAmB/AJ491M/qQS7hE2frVFeLFiLuR0BU0QCgpsxr
MqrVw2DhUxOjWpcOBi4qzWE=
=/oPO
-----END PGP SIGNATURE-----


From tim.hochberg at ieee.org  Fri Apr 13 09:49:00 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Fri, 13 Apr 2007 06:49:00 -0700
Subject: [Numpy-discussion] Fastest distance matrix calc
In-Reply-To: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>
References: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>
Message-ID: <e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>

On 4/13/07, Bill Baxter <wbaxter at gmail.com> wrote:
>
> I think someone posted some timings about this before but I don't recall.
>
> The task is to compute the matrix D from two sets of vectors x (M,d)
> and y (N,d).
> The output should be  D where D[i,j] is norm(x[i]-y[j])
>
> The Matlab NetLab toolkit uses something like this to compute it:
>
>     d2 = (x*x).sum(1)[:,numpy.newaxis] + (y*y).sum(1) - 2*mult(x,y.T)
>
> And then does
>   d2[d2<0]=0
> because roundoff in the above can sometimes create negative values.  And
> finally
>   d = sqrt(d2)
>
> But it just doesn't seem like the best way to do it.  Whatever was
> posted before I don't remember anything about a subtract.outer
> solution.  Seems like something along the lines of
>      (subtract.outer(x,y)**2).sum(axis=0)
> might be faster, and also avoid the need to check for negatives.
>
> I'll do some timings if no one's already done it.  Just wanted to check
> first.


I'm going to go out on a limb and contend, without running any timings, that
for large M and N, a solution using a for loop will beat either of those.
For example (untested):

results = empty([M, N], float)
# You could be fancy and swap axes depending on which array is larger, but
# I'll leave that for someone else
for i, v in enumerate(x):
    results[i] = sqrt(sum((v-y)**2, axis=-1))

Or something like that. The reason that I suspect this will be faster is
that it has better locality, completely finishing a computation on a
relatively small working set before moving onto the next one. The one liners
have to pull the potentially large MxN array into the processor repeatedly.

Or you could use numexpr.

-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070413/88ed0cd9/attachment.html>

From tim.hochberg at ieee.org  Fri Apr 13 10:08:43 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Fri, 13 Apr 2007 07:08:43 -0700
Subject: [Numpy-discussion] endianness ufuncs
In-Reply-To: <461F5A7B.9030104@hugo.doemaarwat.nl>
References: <461F5A7B.9030104@hugo.doemaarwat.nl>
Message-ID: <e4412d6b0704130708m33a177e6ja4d1ff93e48ee7fd@mail.gmail.com>

On 4/13/07, Hugo <numpy at hugo.doemaarwat.nl> wrote:
>
> Hi,
>
> As as astronomer I work with big endian arrays (pyfits 1.1rc1 with numpy
> 1.0) on little endian machines. When I try to make a ufunc (e.g. to
> convert from degrees to radians), it doesn't recognize the big endianness:
>
> In [3]: big = numpy.array([83.4, 83.5, 83.9], dtype='>f8')
> In [4]: uradians = numpy.frompyfunc(math.radians, 1, 1)
> In [6]: uradians(big)
> Out[6]: array([-2.6919063283e-182, 1.26773292198e-318,
> -2.69190632908e-182], dtype=object)
>
> I cannot figure out whether this is the intended behavior but it sure is
> counterintuitive, since most of the time numpy handles the endianness
> transparently.


I suspect that this is a bug, but I'm not familiar enough with frompyfunc to
be sure. However, I have a separate point I'd like to make about frompyfunc
/ vectorize.

This is the second time in the last week or so where someone has posted an
example using frompyfunc/vectorize that really has no business being
implemented that way. Now it's perfectly possible that these examples are
being used because they are simple and the authors are aware that they will
result in a speed penalty over a pure numpy solution, but just in case, I
felt I should sqawk.

I this case, implementing radians as a pure numpy function such as:
    def nrad(x):
        return x * (np.pi/180.0)
results in a function that's about 20x faster than uradians as defined
above.

Here's the actual timing code that I used:

    import numpy as np, math, timeit
    a= np.arange(100000, dtype=float)
    urad = np.frompyfunc(math.radians, 1, 1)
    def nrad(x):
        return x * (np.pi/180.0)
    assert np.allclose(urad(a), nrad(a))
    if __name__ == "__main__":
        print timeit.Timer("urad(a)", "from scratch import urad, nrad,
a").timeit(100)
        print timeit.Timer("nrad(a)", "from scratch import urad, nrad,
a").timeit(100)


For now I'm happy with just converting everything to native (little)
> endian at the start of my calculations and convert them back in the end.
> (Although I'm not sure what's the best way to do that yet.)
>
> Hugo
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070413/3df80fc3/attachment.html>

From lists.steve at arachnedesign.net  Fri Apr 13 10:25:55 2007
From: lists.steve at arachnedesign.net (Steve Lianoglou)
Date: Fri, 13 Apr 2007 10:25:55 -0400
Subject: [Numpy-discussion] Newbie Questio: Unused in wild import
In-Reply-To: <461F7AA6.9060206@bit-man.com.ar>
References: <461F7AA6.9060206@bit-man.com.ar>
Message-ID: <8F385352-93D5-419D-BE32-F981AC712FCB@arachnedesign.net>

Hi V?ctor,

> I'm really new to Python and SciPy and having a strange problem.
> I've installed Python 2.5 on Windows, numpy (last version), Eclipse  
> 3.2
> and latest PyDev extensions.
>
> When importing numpy ("from numpy import *") the next warning is  
> issued :
>
> Unused in wild import: from buffer, c_, FLOATING_POINT_SUPPORT, time,
>
> ... and a lot of symbol name ...

I don't get this problem myself. I'm using Eclipse 3.2, PyDev 1.3.1,  
Python 2.4.4, and on a Mac, but perhaps it's some change I've made in  
the:

Preferences > Pydev > PyDev Extension > Code Analysis > * preferences.

Perhaps this would be better addressed on the PyDev user forums  
(hosted on sourceforge: http://sourceforge.net/forum/forum.php? 
forum_id=293649)?

Also, as a "point of style," in general I think it's not the best to  
use the `from something import *` idiom, and a bit better to do  
something like `import numpy as N` (or some other (very short)  
namespace prefix (like np)).

-steve

From charlesr.harris at gmail.com  Fri Apr 13 13:36:59 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 13 Apr 2007 11:36:59 -0600
Subject: [Numpy-discussion] Help using numPy to create a very large
	multi dimensional array
In-Reply-To: <699044520704130350j7b53e36br5c0bc068ec00643c@mail.gmail.com>
References: <699044520704130350j7b53e36br5c0bc068ec00643c@mail.gmail.com>
Message-ID: <e06186140704131036t4174905y466a4c0a15a815f1@mail.gmail.com>

On 4/13/07, Bruno Santos <bacmsantos at gmail.com> wrote:
>
> Dear Sirs,
> I'm trying to use Numpy to solve a speed problem with Python, I need to
> perform agglomerative clustering as a first step to k-means clustering.
> My problem is that I'm using a very large list in Pyhton and the script is
> taking more than 9minutes to process all the information, so I'm trying to
> use Numpy to create a matrix.
> I'm reading the vectors from a text file and I end up with an array of
> 115*2634 float elements, How can I create this structure with numpy?
>
> Where is my code in python:
> #Read each document vector to a matrix
>     doclist = []
>     matrix = []
>     list = []
>     for line in vecfile:
>         list = line.split()
>         for elem in range(1, len(list)):
>             list[elem] = float(list[elem])
>         matrix.append (list[1:])
>     vecfile.close()


I don't know what your text file looks like or how many elements are in each
line, but assuming 115 entries/line and spaces, something like the following
will read in the data:

m = N.fromfile('name of text file', sep=' ').reshape(-1,115)

This assumes you have done import numpy as N and will result in a 2634x115
array, which isn't very large.

    #Read the desired number of final clusters
>     numclust = input('Input the desired number of clusters: ')
>
> #Clustering process
>     clust = rows
>     ind = [-1, -1]
>     list_j=[]
>     list_k=[]
>     while (clust > numclust):
>         min = 2147483647
>         print('Number of Clusters %d \n' % clust)
>         #Find the 2 most similares vectors in the file
>         for j in range(0, clust):
>             list_j=matrix[j]
>             for k in range(j+1, clust):
>                 list_k=matrix[k]
>                 dist=0
>                 for e in range(0, columns):
>                     result = list_j[e] - list_k[e]
>                     dist += result * result
>                 if (dist < min):
>                     ind[0] = j
>                     ind[1] = k
>                     min = dist

        #Combine the two most similaires vectores by median
>         for e in range(0, columns): matrix[ind[0]][e] = (matrix[ind[0]][e]
> + matrix[ind[1]][e]) / 2.0
>         clust = clust -1
>
>         #Move up all the remaining vectors
>         for k in range(ind[1], (rows - 1)):
>             for e in range(0, columns): matrix[k][e]=matrix[k+1][e]


This is the slow step, order N^3 in the number of vectors. It can be
vectorized, but perhaps there is a better implementation of this algorithm.
There may be an agglomerative clustering algorithm already available in
scipy, the documentation indicates that kmeans clustering software is
available. Perhaps someone closer to that library can help you there.

Chuck


>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070413/d876c93b/attachment.html>

From robert.kern at gmail.com  Fri Apr 13 14:09:55 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 13 Apr 2007 13:09:55 -0500
Subject: [Numpy-discussion] Newbie Questio: Unused in wild import
In-Reply-To: <461F7AA6.9060206@bit-man.com.ar>
References: <461F7AA6.9060206@bit-man.com.ar>
Message-ID: <461FC773.7080202@gmail.com>

V?ctor A. Rodr?guez wrote:
> Hi all,
> 
> I'm really new to Python and SciPy and having a strange problem.
> I've installed Python 2.5 on Windows, numpy (last version), Eclipse 3.2
> and latest PyDev extensions.
> 
> When importing numpy ("from numpy import *") the next warning is issued :
> 
> Unused in wild import: from buffer, c_, FLOATING_POINT_SUPPORT, time,
> 
> ... and a lot of symbol name ...
> 
> There's no problem in using numpy but i'm just curious about it.
> BTW, sorry if this is not the correct list to post this.

I'm not entirely familiar with the PyDev extensions, but it appears to me that
this is telling you that although you are importing all of those symbols because
of the *, you aren't using most of them. Everything is working exactly as it is
intended to.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From tgrav at mac.com  Fri Apr 13 14:42:45 2007
From: tgrav at mac.com (Tommy Grav)
Date: Fri, 13 Apr 2007 14:42:45 -0400
Subject: [Numpy-discussion] Import problem
Message-ID: <277FA00A-64D5-47DF-833C-F031EEA84D0D@mac.com>

I am trying to import scipy.optimize on my Mac OS X PowerPc and get  
this error

[tgrav at skathi] Python/Astronomy --> python
ActivePython 2.4.3 Build 11 (ActiveState Software Inc.) based on
Python 2.4.3 (#1, Apr  3 2006, 18:07:18)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import scipy.optimize
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/scipy/optimize/__init__.py", line 7, in ?
     from optimize import *
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/scipy/optimize/optimize.py", line 26, in ?
     import linesearch
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/site-packages/scipy/optimize/linesearch.py", line 3, in ?
     import minpack2
ImportError: Failure linking new module: /Library/Frameworks/ 
Python.framework/Versions/Current/lib/python2.4/site-packages/scipy/ 
optimize/minpack2.so: Library not loaded: /usr/local/lib/libg2c.0.dylib
   Referenced from: /Library/Frameworks/Python.framework/Versions/ 
Current/lib/python2.4/site-packages/scipy/optimize/minpack2.so
   Reason: image not found
 >>>

How do I get a hold of libg2c.0.dylib for my system?

Cheers
   Tommy


From robert.kern at gmail.com  Fri Apr 13 15:02:26 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 13 Apr 2007 14:02:26 -0500
Subject: [Numpy-discussion] Import problem
In-Reply-To: <277FA00A-64D5-47DF-833C-F031EEA84D0D@mac.com>
References: <277FA00A-64D5-47DF-833C-F031EEA84D0D@mac.com>
Message-ID: <461FD3C2.2050404@gmail.com>

Tommy Grav wrote:
> I am trying to import scipy.optimize on my Mac OS X PowerPc and get  
> this error
> 
> [tgrav at skathi] Python/Astronomy --> python
> ActivePython 2.4.3 Build 11 (ActiveState Software Inc.) based on
> Python 2.4.3 (#1, Apr  3 2006, 18:07:18)
> [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import scipy.optimize
> Traceback (most recent call last):
>    File "<stdin>", line 1, in ?
>    File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
> python2.4/site-packages/scipy/optimize/__init__.py", line 7, in ?
>      from optimize import *
>    File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
> python2.4/site-packages/scipy/optimize/optimize.py", line 26, in ?
>      import linesearch
>    File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
> python2.4/site-packages/scipy/optimize/linesearch.py", line 3, in ?
>      import minpack2
> ImportError: Failure linking new module: /Library/Frameworks/ 
> Python.framework/Versions/Current/lib/python2.4/site-packages/scipy/ 
> optimize/minpack2.so: Library not loaded: /usr/local/lib/libg2c.0.dylib
>    Referenced from: /Library/Frameworks/Python.framework/Versions/ 
> Current/lib/python2.4/site-packages/scipy/optimize/minpack2.so
>    Reason: image not found
>  >>>
> 
> How do I get a hold of libg2c.0.dylib for my system?

Where did you get the scipy binary from?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From tgrav at mac.com  Fri Apr 13 15:05:54 2007
From: tgrav at mac.com (Tommy Grav)
Date: Fri, 13 Apr 2007 15:05:54 -0400
Subject: [Numpy-discussion] Import problem
In-Reply-To: <461FD3C2.2050404@gmail.com>
References: <277FA00A-64D5-47DF-833C-F031EEA84D0D@mac.com>
	<461FD3C2.2050404@gmail.com>
Message-ID: <EC452091-39EA-4543-9B37-88BB2633BBA7@mac.com>

> Where did you get the scipy binary from?

I was using the superpack from the scipy.org downloads page. I  
however found
the library bundle up with g77 and downloaded that so now I get the  
import
to work. I expect that there might be other libraries I am missing  
but I will have
to deal with that as I go along.

Cheers
   Tommy


From tgrav at mac.com  Fri Apr 13 15:12:20 2007
From: tgrav at mac.com (Tommy Grav)
Date: Fri, 13 Apr 2007 15:12:20 -0400
Subject: [Numpy-discussion] index of minimum of array
Message-ID: <618112A8-4738-4750-A82B-5C1F51E5F1D6@mac.com>

Hi,

   how do I find the index of the minimum value of an numpy array?

Example
   a = array([1.,2.,0.4,3.])

I want the i=2 since a[i] = 0.4 is the smallest value in a.

Cheers
   Tommy 
  


From charlesr.harris at gmail.com  Fri Apr 13 15:13:04 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 13 Apr 2007 13:13:04 -0600
Subject: [Numpy-discussion] linking question
Message-ID: <e06186140704131213qe08cd16s7635256d65fc1e01@mail.gmail.com>

All,

ATLAS 3.7.30 produces two sets of libraries on my machine, the usual bunch
and a set prefixed by pt (posix threading). Am I correct in assuming that
numpy should be linked against the latter?

TIA,
Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070413/2b5162f8/attachment.html>

From tim.hochberg at ieee.org  Fri Apr 13 15:14:56 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Fri, 13 Apr 2007 12:14:56 -0700
Subject: [Numpy-discussion] index of minimum of array
In-Reply-To: <618112A8-4738-4750-A82B-5C1F51E5F1D6@mac.com>
References: <618112A8-4738-4750-A82B-5C1F51E5F1D6@mac.com>
Message-ID: <e4412d6b0704131214h73ab6092td8293cd309617634@mail.gmail.com>

On 4/13/07, Tommy Grav <tgrav at mac.com> wrote:
>
> Hi,
>
>    how do I find the index of the minimum value of an numpy array?
>
> Example
>    a = array([1.,2.,0.4,3.])
>
> I want the i=2 since a[i] = 0.4 is the smallest value in a.


argmin


Cheers
>    Tommy
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070413/6378fa9f/attachment.html>

From robert.kern at gmail.com  Fri Apr 13 15:31:42 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 13 Apr 2007 14:31:42 -0500
Subject: [Numpy-discussion] linking question
In-Reply-To: <e06186140704131213qe08cd16s7635256d65fc1e01@mail.gmail.com>
References: <e06186140704131213qe08cd16s7635256d65fc1e01@mail.gmail.com>
Message-ID: <461FDA9E.50900@gmail.com>

Charles R Harris wrote:
> All,
> 
> ATLAS 3.7.30 produces two sets of libraries on my machine, the usual
> bunch and a set prefixed by pt (posix threading). Am I correct in
> assuming that numpy should be linked against the latter?

Either should work, I think. If you have multiple processors, the threaded
versions may work faster. Or not.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From aisaac at american.edu  Fri Apr 13 16:34:48 2007
From: aisaac at american.edu (Alan Isaac)
Date: Fri, 13 Apr 2007 15:34:48 -0500
Subject: [Numpy-discussion] index of minimum of array
In-Reply-To: <e4412d6b0704131214h73ab6092td8293cd309617634@mail.gmail.com>
References: <618112A8-4738-4750-A82B-5C1F51E5F1D6@mac.com><e4412d6b0704131214h73ab6092td8293cd309617634@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-3536-20070413-153448.00@CASC64043305.american.edu>

> On 4/13/07, Tommy Grav <tgrav at mac.com> wrote:
>> how do I find the index of the minimum value of an numpy 
>> array?  Example a = array([1.,2.,0.4,3.]) I want the i=2 
>> since a[i] = 0.4 is the smallest value in a. 


On Fri, 13 Apr 2007, Timothy Hochberg wrote: 
> argmin 

Just a reminder that there exist a very useful example list
http://www.scipy.org/Numpy_Example_List#argmin

and a wonderful reference:
http://www.tramy.us/

Cheers,
Alan Isaac




From wbaxter at gmail.com  Fri Apr 13 15:57:29 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Sat, 14 Apr 2007 04:57:29 +0900
Subject: [Numpy-discussion] index of minimum of array
In-Reply-To: <Mahogany-0.67.0-3536-20070413-153448.00@CASC64043305.american.edu>
References: <618112A8-4738-4750-A82B-5C1F51E5F1D6@mac.com>
	<e4412d6b0704131214h73ab6092td8293cd309617634@mail.gmail.com>
	<Mahogany-0.67.0-3536-20070413-153448.00@CASC64043305.american.edu>
Message-ID: <e86a5fd00704131257p118e0d07p727195fbb83d546b@mail.gmail.com>

On 4/14/07, Alan Isaac <aisaac at american.edu> wrote:
> > On 4/13/07, Tommy Grav <tgrav at mac.com> wrote:
> >> how do I find the index of the minimum value of an numpy
> >> array?  Example a = array([1.,2.,0.4,3.]) I want the i=2
> >> since a[i] = 0.4 is the smallest value in a.
>
>
> On Fri, 13 Apr 2007, Timothy Hochberg wrote:
> > argmin
>
> Just a reminder that there exist a very useful example list
> http://www.scipy.org/Numpy_Example_List#argmin

And in addition to examples it has great "See Also" info.  I probably
use it more for the See Also than the actual examples.  Sure enough
See Also for min() and max() refer to argmin() and argmax().

--bb


From numpy at hugo.doemaarwat.nl  Fri Apr 13 16:56:46 2007
From: numpy at hugo.doemaarwat.nl (Hugo)
Date: Fri, 13 Apr 2007 22:56:46 +0200
Subject: [Numpy-discussion] index of minimum of array
In-Reply-To: <e86a5fd00704131257p118e0d07p727195fbb83d546b@mail.gmail.com>
References: <618112A8-4738-4750-A82B-5C1F51E5F1D6@mac.com>	<e4412d6b0704131214h73ab6092td8293cd309617634@mail.gmail.com>	<Mahogany-0.67.0-3536-20070413-153448.00@CASC64043305.american.edu>
	<e86a5fd00704131257p118e0d07p727195fbb83d546b@mail.gmail.com>
Message-ID: <461FEE8E.9080700@hugo.doemaarwat.nl>

Bill Baxter wrote:
> On 4/14/07, Alan Isaac <aisaac at american.edu> wrote:
>>> On 4/13/07, Tommy Grav <tgrav at mac.com> wrote:
>>>> how do I find the index of the minimum value of an numpy
>>>> array?  Example a = array([1.,2.,0.4,3.]) I want the i=2
>> 
>> Just a reminder that there exist a very useful example list
>> http://www.scipy.org/Numpy_Example_List#argmin
> 
> And in addition to examples it has great "See Also" info.  I probably
> use it more for the See Also than the actual examples.  Sure enough
> See Also for min() and max() refer to argmin() and argmax().

Perhaps in this particular case it would be nice to have a See Also for 'nan'
 at argmin() and argmax(), since there is info about nanargmin and nanargmax().

Hugo


From wbaxter at gmail.com  Fri Apr 13 18:49:40 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Sat, 14 Apr 2007 07:49:40 +0900
Subject: [Numpy-discussion] index of minimum of array
In-Reply-To: <461FEE8E.9080700@hugo.doemaarwat.nl>
References: <618112A8-4738-4750-A82B-5C1F51E5F1D6@mac.com>
	<e4412d6b0704131214h73ab6092td8293cd309617634@mail.gmail.com>
	<Mahogany-0.67.0-3536-20070413-153448.00@CASC64043305.american.edu>
	<e86a5fd00704131257p118e0d07p727195fbb83d546b@mail.gmail.com>
	<461FEE8E.9080700@hugo.doemaarwat.nl>
Message-ID: <e86a5fd00704131549j267b93a2se2680cf29b99a7e0@mail.gmail.com>

On 4/14/07, Hugo <numpy at hugo.doemaarwat.nl> wrote:
> Bill Baxter wrote:
> > On 4/14/07, Alan Isaac <aisaac at american.edu> wrote:
> >>> On 4/13/07, Tommy Grav <tgrav at mac.com> wrote:
> >>>> how do I find the index of the minimum value of an numpy
> >>>> array?  Example a = array([1.,2.,0.4,3.]) I want the i=2
> >>
> >> Just a reminder that there exist a very useful example list
> >> http://www.scipy.org/Numpy_Example_List#argmin
> >
> > And in addition to examples it has great "See Also" info.  I probably
> > use it more for the See Also than the actual examples.  Sure enough
> > See Also for min() and max() refer to argmin() and argmax().
>
> Perhaps in this particular case it would be nice to have a See Also for 'nan'
>  at argmin() and argmax(), since there is info about nanargmin and nanargmax().

Didn't know about those myself.  I think the nan* functions just
haven't been added to the example list yet (which includes adding
relevant SeeAlso's).   If you use 'em, feel free to go ahead and edit
the page.  It's a wiki after all.

--bb


From ckkart at hoc.net  Sat Apr 14 00:44:48 2007
From: ckkart at hoc.net (Christian K)
Date: Sat, 14 Apr 2007 13:44:48 +0900
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
Message-ID: <evpm71$q6c$1@sea.gmane.org>


Hi,
I'm trying to build numpy from svn on ubuntu edgy with atlas provided by ubuntu
package atlas3-sse2-dev which contains:

/usr
/usr/lib
/usr/lib/sse2
/usr/lib/sse2/libatlas.a
/usr/lib/sse2/libcblas.a
/usr/lib/sse2/libf77blas.a
/usr/lib/sse2/liblapack_atlas.a
/usr/lib/atlas
/usr/lib/atlas/sse2
/usr/lib/atlas/sse2/libblas.a
/usr/lib/atlas/sse2/liblapack.a
/usr/share
/usr/share/doc
/usr/share/doc/atlas3-sse2-dev
/usr/share/doc/atlas3-sse2-dev/copyright
/usr/share/doc/atlas3-sse2-dev/changelog.Debian.gz
/usr/lib/sse2/libatlas.so
/usr/lib/sse2/libcblas.so
/usr/lib/sse2/libf77blas.so
/usr/lib/sse2/liblapack_atlas.so
/usr/lib/atlas/sse2/libblas.so
/usr/lib/atlas/sse2/liblapack.so

I tried both with and without a site.cfg:


[DEFAULT]
library_dirs = /usr/lib/sse2
include_dirs = /usr/include

[blas_opt]
libraries = f77blas, cblas, atlas

[lapack_opt]
libraries = lapack, f77blas, cblas, atlas

and tested wether numpy is actually using the optimized libs as demonstrated in
a posting by Simon Burton
(http://article.gmane.org/gmane.comp.python.numeric.general/5849).

It apparently is linked to
/usr/lib/sse2/libatlas.so.3.0
/usr/lib/sse2/libcblas.so.3.0
/usr/lib/sse2/libf77blas.so.3.0
/usr/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so

The optimized lapack lib is not used. This is consistent with the output of the
build script:

ck at kiste:~/prog/scipy/numpy$ python setup.py build
Running from numpy source directory.
non-existing path in 'numpy/distutils': 'site.cfg'
F2PY Version 2_3714
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in /usr/local/lib
  libraries mkl,vml,guide not found in /usr/lib
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/atlas
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
  NOT AVAILABLE

atlas_blas_info:
  libraries f77blas,cblas,atlas not found in /usr/local/lib
  libraries f77blas,cblas,atlas not found in /usr/lib/atlas
  FOUND:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['/usr/lib/sse2']
    language = c

customize GnuFCompiler
customize GnuFCompiler
customize GnuFCompiler using config
compiling '_configtest.c':

/* This file is generated from numpy_distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -Wall
-Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
gcc -pthread _configtest.o -L/usr/lib/sse2 -lf77blas -lcblas -latlas -o _configtest
ATLAS version 3.6.0 built by root on Fri Jan  9 15:57:20 UTC 2004:
   UNAME    : Linux intech67 2.4.20 #1 SMP Fri Jan 10 18:29:51 EST 2003 i686
GNU/Linux
   INSTFLG  :
   MMDEF    : /fix/g/camm/atlas3-3.6.0/CONFIG/ARCHS/P4SSE2/gcc/gemm
   ARCHDEF  : /fix/g/camm/atlas3-3.6.0/CONFIG/ARCHS/P4SSE2/gcc/misc
   F2CDEFS  : -DAdd__ -DStringSunStyle
   CACHEEDGE: 1048576
   F77      : /usr/bin/g77, version GNU Fortran (GCC) 3.3.3 20031229
(prerelease) (Debian)
   F77FLAGS : -fomit-frame-pointer -O
   CC       : /usr/bin/gcc, version gcc (GCC) 3.3.3 20031229 (prerelease) (Debian)
   CC FLAGS : -fomit-frame-pointer -O3 -funroll-all-loops
   MCC      : /usr/bin/gcc, version gcc (GCC) 3.3.3 20031229 (prerelease) (Debian)
   MCCFLAGS : -fomit-frame-pointer -O
success!
removing: _configtest.c _configtest.o _configtest
  FOUND:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['/usr/lib/sse2']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"3.6.0\\""')]

lapack_opt_info:
lapack_mkl_info:
mkl_info:
  libraries mkl,vml,guide not found in /usr/local/lib
  libraries mkl,vml,guide not found in /usr/lib
  NOT AVAILABLE

  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/atlas
  libraries lapack_atlas not found in /usr/lib/atlas
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib
numpy.distutils.system_info.atlas_threads_info
  NOT AVAILABLE

atlas_info:
  libraries f77blas,cblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries f77blas,cblas,atlas not found in /usr/lib/atlas
  libraries lapack_atlas not found in /usr/lib/atlas
  libraries lapack not found in /usr/lib/sse2
  libraries f77blas,cblas,atlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib
numpy.distutils.system_info.atlas_info
/media/hda6/home/ck/prog/scipy/numpy/numpy/distutils/system_info.py:903:
UserWarning:
*********************************************************************
    Could not find lapack library within the ATLAS installation.
*********************************************************************

  warnings.warn(message)
  FOUND:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['/usr/lib/sse2']
    language = c
    define_macros = [('ATLAS_WITHOUT_LAPACK', None)]

lapack_info:
  libraries lapack not found in /usr/local/lib
  libraries lapack not found in /usr/lib
  NOT AVAILABLE


Confusingly lapack_atlas resides in /usr/lib but even though setup.py looks for
it in that place it reports 'not found'.

What should I try next?

Thanks, Christian



From ckkart at hoc.net  Sat Apr 14 00:57:32 2007
From: ckkart at hoc.net (Christian K)
Date: Sat, 14 Apr 2007 13:57:32 +0900
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <evpm71$q6c$1@sea.gmane.org>
References: <evpm71$q6c$1@sea.gmane.org>
Message-ID: <evpmut$q6c$2@sea.gmane.org>

Christian K wrote:
> Hi,
> I'm trying to build numpy from svn on ubuntu edgy with atlas provided by ubuntu
> package atlas3-sse2-dev which contains:

[...]

> 
> I tried both with and without a site.cfg:
> 
> 
> [DEFAULT]
> library_dirs = /usr/lib/sse2
> include_dirs = /usr/include
> 
> [blas_opt]
> libraries = f77blas, cblas, atlas
> 
> [lapack_opt]
> libraries = lapack, f77blas, cblas, atlas
> 
> and tested wether numpy is actually using the optimized libs as demonstrated in
> a posting by Simon Burton
> (http://article.gmane.org/gmane.comp.python.numeric.general/5849).
> 
> It apparently is linked to
> /usr/lib/sse2/libatlas.so.3.0
> /usr/lib/sse2/libcblas.so.3.0
> /usr/lib/sse2/libf77blas.so.3.0
> /usr/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so

I tried with a modified site.cfg

[DEFAULT]
library_dirs = /usr/lib/atlas/sse2:/usr/lib
include_dirs = /usr/include
[blas_opt]
libraries = f77blas, cblas, atlas
[lapack_opt]
libraries = lapack, f77blas, cblas, atlas

and now numpy seems to use both its own lapack_lite and the optimized one:

/usr/lib/atlas/sse2/liblapack.so.3.0
/usr/lib/atlas/sse2/liblapack.so.3.0
/usr/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so
/usr/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so

Is that ok?

Christian



From robert.kern at gmail.com  Sat Apr 14 02:24:38 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 14 Apr 2007 01:24:38 -0500
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <evpm71$q6c$1@sea.gmane.org>
References: <evpm71$q6c$1@sea.gmane.org>
Message-ID: <462073A6.4010602@gmail.com>

Christian K wrote:
> Hi,
> I'm trying to build numpy from svn on ubuntu edgy with atlas provided by ubuntu
> package atlas3-sse2-dev which contains:
> 
> /usr
> /usr/lib
> /usr/lib/sse2
> /usr/lib/sse2/libatlas.a
> /usr/lib/sse2/libcblas.a
> /usr/lib/sse2/libf77blas.a
> /usr/lib/sse2/liblapack_atlas.a
> /usr/lib/atlas
> /usr/lib/atlas/sse2
> /usr/lib/atlas/sse2/libblas.a
> /usr/lib/atlas/sse2/liblapack.a
> /usr/share
> /usr/share/doc
> /usr/share/doc/atlas3-sse2-dev
> /usr/share/doc/atlas3-sse2-dev/copyright
> /usr/share/doc/atlas3-sse2-dev/changelog.Debian.gz
> /usr/lib/sse2/libatlas.so
> /usr/lib/sse2/libcblas.so
> /usr/lib/sse2/libf77blas.so
> /usr/lib/sse2/liblapack_atlas.so
> /usr/lib/atlas/sse2/libblas.so
> /usr/lib/atlas/sse2/liblapack.so
> 
> I tried both with and without a site.cfg:
> 
> 
> [DEFAULT]
> library_dirs = /usr/lib/sse2
> include_dirs = /usr/include
> 
> [blas_opt]
> libraries = f77blas, cblas, atlas
> 
> [lapack_opt]
> libraries = lapack, f77blas, cblas, atlas

> Confusingly lapack_atlas resides in /usr/lib but even though setup.py looks for
> it in that place it reports 'not found'.
> 
> What should I try next?

Change this:

[lapack_opt]
libraries = lapack, f77blas, cblas, atlas

to this:

[lapack_opt]
libraries = lapack_atlas, f77blas, cblas, atlas

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From ckkart at hoc.net  Sat Apr 14 10:14:19 2007
From: ckkart at hoc.net (Christian K)
Date: Sat, 14 Apr 2007 23:14:19 +0900
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <462073A6.4010602@gmail.com>
References: <evpm71$q6c$1@sea.gmane.org> <462073A6.4010602@gmail.com>
Message-ID: <evqnir$44m$1@sea.gmane.org>

Robert Kern wrote:
> Christian K wrote:
>> Hi,
>> I'm trying to build numpy from svn on ubuntu edgy with atlas provided by ubuntu
>> package atlas3-sse2-dev which contains:
>>
>> /usr
>> /usr/lib
>> /usr/lib/sse2
>> /usr/lib/sse2/libatlas.a
>> /usr/lib/sse2/libcblas.a
>> /usr/lib/sse2/libf77blas.a
>> /usr/lib/sse2/liblapack_atlas.a
>> /usr/lib/atlas
>> /usr/lib/atlas/sse2
>> /usr/lib/atlas/sse2/libblas.a
>> /usr/lib/atlas/sse2/liblapack.a
>> /usr/share
>> /usr/share/doc
>> /usr/share/doc/atlas3-sse2-dev
>> /usr/share/doc/atlas3-sse2-dev/copyright
>> /usr/share/doc/atlas3-sse2-dev/changelog.Debian.gz
>> /usr/lib/sse2/libatlas.so
>> /usr/lib/sse2/libcblas.so
>> /usr/lib/sse2/libf77blas.so
>> /usr/lib/sse2/liblapack_atlas.so
>> /usr/lib/atlas/sse2/libblas.so
>> /usr/lib/atlas/sse2/liblapack.so
>>
>> I tried both with and without a site.cfg:
>>
>>
>> [DEFAULT]
>> library_dirs = /usr/lib/sse2
>> include_dirs = /usr/include
>>
>> [blas_opt]
>> libraries = f77blas, cblas, atlas
>>
>> [lapack_opt]
>> libraries = lapack, f77blas, cblas, atlas
> 
>> Confusingly lapack_atlas resides in /usr/lib but even though setup.py looks for
>> it in that place it reports 'not found'.
>>
>> What should I try next?
> 
> Change this:
> 
> [lapack_opt]
> libraries = lapack, f77blas, cblas, atlas
> 
> to this:
> 
> [lapack_opt]
> libraries = lapack_atlas, f77blas, cblas, atlas
> 

Thanks, but that didn't help:

atlas_info:
  libraries lapack not found in /usr/lib/sse2
  libraries f77blas,cblas,atlas not found in /usr/lib/atlas
  libraries lapack_atlas not found in /usr/lib/atlas
  libraries lapack not found in /usr/lib/sse2
  libraries f77blas,cblas,atlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib
numpy.distutils.system_info.atlas_info
/media/hda6/home/ck/prog/scipy/numpy/numpy/distutils/system_info.py:903:
UserWarning:
*********************************************************************
    Could not find lapack library within the ATLAS installation.
*********************************************************************

  warnings.warn(message)
  FOUND:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['/usr/lib/sse2']
    language = c
    define_macros = [('ATLAS_WITHOUT_LAPACK', None)]

lapack_info:
  libraries lapack not found in /usr/lib/sse2
  libraries lapack not found in /usr/lib
  NOT AVAILABLE


Christian



From subscriber100 at rjs.org  Sat Apr 14 11:27:09 2007
From: subscriber100 at rjs.org (Ray Schumacher)
Date: Sat, 14 Apr 2007 08:27:09 -0700
Subject: [Numpy-discussion] building numpy with Intel MS compiler and FFT
Message-ID: <6.2.3.4.2.20070414080540.02e2b900@pop-server.san.rr.com>

Has anyone  built Python/numpy with the Intel optimized compiler and 
FFT lib for Microsoft, and have any pointers?
We're counting on the extra speed, and will be getting the compiler 
and libraries next week.

Is there a consensus on distribution requirements for Python compiled 
with the Intel compiler? I was just recalling the MS 7 compiler debate.



From rex at nosyntax.com  Sat Apr 14 21:05:00 2007
From: rex at nosyntax.com (rex)
Date: Sat, 14 Apr 2007 18:05:00 -0700
Subject: [Numpy-discussion] building numpy with Intel MS compiler and	FFT
In-Reply-To: <6.2.3.4.2.20070414080540.02e2b900@pop-server.san.rr.com>
References: <6.2.3.4.2.20070414080540.02e2b900@pop-server.san.rr.com>
Message-ID: <20070415010500.GP9577@x2.nosyntax.com>

Ray Schumacher <subscriber100 at rjs.org> [2007-04-14 09:30]:
> Has anyone  built Python/numpy with the Intel optimized compiler and 
> FFT lib for Microsoft, and have any pointers?

I've compiled both Numpy and Python 2.5 with the Intel compiler. On a
Core 2 Duo, at least, the speed increase on Pybench was ~49%, even
before compiling Python with icc. My post about it was on 25 Jan, and
has subject: Compiling Python with icc

I haven't tried the FFT lib.

-rex
-- 
"Experience is a wonderful thing. It enables you to recognize a mistake
when you make it again."



From ckkart at hoc.net  Sun Apr 15 01:03:49 2007
From: ckkart at hoc.net (Christian K)
Date: Sun, 15 Apr 2007 14:03:49 +0900
Subject: [Numpy-discussion] pickable ndarray subclass
Message-ID: <evsbmj$qui$1@sea.gmane.org>

Hi,

could someone please provide example code for how to make a subclassed ndarray
pickable? I don't quite understand the docs of ndarray.__reduce__.
My subclassed ndarray has just one additional attribute.

Thanks, Christian



From stefan at sun.ac.za  Sun Apr 15 10:06:35 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Sun, 15 Apr 2007 16:06:35 +0200
Subject: [Numpy-discussion] pickable ndarray subclass
In-Reply-To: <evsbmj$qui$1@sea.gmane.org>
References: <evsbmj$qui$1@sea.gmane.org>
Message-ID: <20070415140635.GI18196@mentat.za.net>

Hi Christiaan

On Sun, Apr 15, 2007 at 02:03:49PM +0900, Christian K wrote:
> could someone please provide example code for how to make a subclassed ndarray
> pickable? I don't quite understand the docs of ndarray.__reduce__.
> My subclassed ndarray has just one additional attribute.
> 

__reduce__ is part of the pickle protocol, which is described at

http://docs.python.org/lib/node321.html

You need to modify __reduce__ to store the complete state of your
custom object, as well as __setstate__ which restores the state on
unpickling.

See the attached code as an example.  There, I have an InfoArray,
which is a subclassed numpy.ndarray.  The InfoArray contains an
additional member, info, which is a dictionary.

The __reduce__ method calls ndarray's __reduce__, then adds to the
result the state of the InfoArray.  Conversely, the __setstate__
method calls ndarray's __setstate__ as well as restoring the info
member.

Regards
St?fan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: infoarray.py
Type: text/x-python
Size: 1545 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070415/ebf0a380/attachment.py>

From gerard.vermeulen at grenoble.cnrs.fr  Sun Apr 15 10:56:34 2007
From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen)
Date: Sun, 15 Apr 2007 16:56:34 +0200
Subject: [Numpy-discussion] ANN: PyQwt3D-0.1.4 released
Message-ID: <20070415165634.133ad7c9@zombie.grenoble.cnrs.fr>

What is PyQwt3D?

- it is a set of Python bindings for the QwtPlot3D C++ class library
  which extends the Qt framework with widgets for 3D data visualization.
  PyQwt3D inherits the snappy feel from QwtPlot3D.
  The examples at http://pyqwt.sourceforge.net/pyqwt3d-examples.html
  show how easy it is to make a 3D plot and how to save a 3D plot to
  an image or an (E)PS/PDF file.

- it requires and extends PyQt, a set of Python bindings for Qt.

- it supports the use of PyQt, Qt, QwtPlot3D, and NumPy or SciPy in a
  GUI Python application or in an interactive Python session.

- it runs on POSIX, Mac OS X and Windows platforms (practically any
  platform supported by Qt and Python).

The home page of PyQwt3D is http://pyqwt.sourceforge.net.

PyQwt3D-0.1.4 is a minor release providing:
- support for SIP-4.6
- a binary installer for Windows.

PyQwt3D-0.1.4 supports:
1. Python-2.5.x, -2.4.x, or -2.3.x
2. PyQt-4.2 and -4.1.x, or -3.17.x
3. SIP-4.6, or -4.5.x
4. Qt-4.2.x, Qt-4.1.x, Qt-3.3.x, or -3.2.x
5. QwtPlot3D-0.2.6


Enjoy -- Gerard Vermeulen


From staneff at constructiondatares.com  Sun Apr 15 12:02:29 2007
From: staneff at constructiondatares.com (Steve Staneff)
Date: Sun, 15 Apr 2007 10:02:29 -0600
Subject: [Numpy-discussion] newbie question - large dataset
In-Reply-To: <mailman.11.1176051605.19801.numpy-discussion@scipy.org>
References: <mailman.11.1176051605.19801.numpy-discussion@scipy.org>
Message-ID: <46224C95.6050903@constructiondatares.com>

Many thanks to all those who responded to my question.  Not being a real programmer, sometimes it's good to be reminded 
of the basics - such as timing various sections of code.  It turns out that most of the time in the loop was spent on 
two of the functions.  I've rewritten the first to be pre-calculated (a real oversight on my part), and looked at 
improving the second as much as possible.

Some of the other time in the loop is being spent going back to the db server for data, and I'll look into using 
pytables as a way of speeding that up (possibly by initializing temporary tables on each of my client machines).

Thanks again,

Steve

> On 4/7/07, Steve Staneff <staneff at constructiondatares.com> wrote:
> 
>>Hi,
>>
>>I'm looking for a better solution to managing a very large calculation.
>>Set A is composed of tuples a, each of the form a = [float, string]; set B
>>is composed of tuples of similar structure (b = [float, string]).  For
>>each possible combination of a and b I'm calculating c, of the form c =
>>f(a,b) = [g(a[0], b[0]), h(a[1], b[1])] where g() and h() are non-trivial
>>functions.
> 


From subscriber100 at rjs.org  Sun Apr 15 14:19:22 2007
From: subscriber100 at rjs.org (Ray Schumacher)
Date: Sun, 15 Apr 2007 11:19:22 -0700
Subject: [Numpy-discussion] building numpy with Intel MS compiler and FFT
Message-ID: <6.2.3.4.2.20070415111706.02e41480@pop-server.san.rr.com>

Thanks Rex,
I'll give it a try next week.

 >I've compiled both Numpy and Python 2.5 with the Intel compiler. On a
 >Core 2 Duo, at least, the speed increase on Pybench was ~49%, even
 >before compiling Python with icc. My post about it was on 25 Jan, and
 >has subject: Compiling Python with icc



From david at ar.media.kyoto-u.ac.jp  Sun Apr 15 22:25:52 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 16 Apr 2007 11:25:52 +0900
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <evqnir$44m$1@sea.gmane.org>
References: <evpm71$q6c$1@sea.gmane.org> <462073A6.4010602@gmail.com>
	<evqnir$44m$1@sea.gmane.org>
Message-ID: <4622DEB0.6000206@ar.media.kyoto-u.ac.jp>

Christian K wrote:
>
> Thanks, but that didn't help:
>
> atlas_info:
>   libraries lapack not found in /usr/lib/sse2
>   libraries f77blas,cblas,atlas not found in /usr/lib/atlas
>   libraries lapack_atlas not found in /usr/lib/atlas
>   libraries lapack not found in /usr/lib/sse2
>   libraries f77blas,cblas,atlas not found in /usr/lib
>   libraries lapack_atlas not found in /usr/lib
> numpy.distutils.system_info.atlas_info
> /media/hda6/home/ck/prog/scipy/numpy/numpy/distutils/system_info.py:903:
> UserWarning:
> *********************************************************************
>     Could not find lapack library within the ATLAS installation.
> *********************************************************************
>
>   warnings.warn(message)
>   FOUND:
>     libraries = ['f77blas', 'cblas', 'atlas']
>     library_dirs = ['/usr/lib/sse2']
>     language = c
>     define_macros = [('ATLAS_WITHOUT_LAPACK', None)]
>
> lapack_info:
>   libraries lapack not found in /usr/lib/sse2
>   libraries lapack not found in /usr/lib
>   NOT AVAILABLE
>
>
> Christian
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
On Ubuntu and debian, you do NOT need any site.cfg to compile numpy with 
atlas support. Just install the package atlas3-base-dev, and you are 
done. The reason is that when *compiling* a software which needs atlas, 
the linker will try to find libblas.so in /usr/lib, not in 
/usr/lib/sse2. If you install atlas3-base-dev, the package will install 
those at the correct locations. I have updated the instructions for 
Ubuntu (also works for debian) on the wiki a few days ago:

http://www.scipy.org/Installing_SciPy/Linux#head-c5a062b2ecf76f746d78cfcde1dae00ae26109fe

Note that if you have also optimized version installed (sse, sse2), it 
will use those automatically when you launch numpy (you can check by 
doing ldd on the file numpy/core/_dotblas.so inside your numpy installation.

cheers,

David




From charlesr.harris at gmail.com  Sun Apr 15 23:28:16 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 15 Apr 2007 21:28:16 -0600
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
References: <evpm71$q6c$1@sea.gmane.org> <462073A6.4010602@gmail.com>
	<evqnir$44m$1@sea.gmane.org> <4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
Message-ID: <e06186140704152028k9c5cdc2m3f90d0a71028b48e@mail.gmail.com>

On 4/15/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>
> Christian K wrote:
> >
> > Thanks, but that didn't help:
> >
> > atlas_info:
> >   libraries lapack not found in /usr/lib/sse2
> >   libraries f77blas,cblas,atlas not found in /usr/lib/atlas
> >   libraries lapack_atlas not found in /usr/lib/atlas
> >   libraries lapack not found in /usr/lib/sse2
> >   libraries f77blas,cblas,atlas not found in /usr/lib
> >   libraries lapack_atlas not found in /usr/lib
> > numpy.distutils.system_info.atlas_info
> > /media/hda6/home/ck/prog/scipy/numpy/numpy/distutils/system_info.py:903:
> > UserWarning:
> > *********************************************************************
> >     Could not find lapack library within the ATLAS installation.
> > *********************************************************************
> >
> >   warnings.warn(message)
> >   FOUND:
> >     libraries = ['f77blas', 'cblas', 'atlas']
> >     library_dirs = ['/usr/lib/sse2']
> >     language = c
> >     define_macros = [('ATLAS_WITHOUT_LAPACK', None)]
> >
> > lapack_info:
> >   libraries lapack not found in /usr/lib/sse2
> >   libraries lapack not found in /usr/lib
> >   NOT AVAILABLE
> >
> >
> > Christian
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> On Ubuntu and debian, you do NOT need any site.cfg to compile numpy with
> atlas support. Just install the package atlas3-base-dev, and you are
> done. The reason is that when *compiling* a software which needs atlas,
> the linker will try to find libblas.so in /usr/lib, not in
> /usr/lib/sse2. If you install atlas3-base-dev, the package will install
> those at the correct locations. I have updated the instructions for
> Ubuntu (also works for debian) on the wiki a few days ago:
>
>
> http://www.scipy.org/Installing_SciPy/Linux#head-c5a062b2ecf76f746d78cfcde1dae00ae26109fe
>
> Note that if you have also optimized version installed (sse, sse2), it
> will use those automatically when you launch numpy (you can check by
> doing ldd on the file numpy/core/_dotblas.so inside your numpy
> installation.


Be aware that on recent 64 bit Intel processors running a 64 bit OS the base
Atlas package appears to be buggy. You might be better off building your own
version, on a fast machine it won't take long.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070415/a9eb0ba8/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Sun Apr 15 23:45:16 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 16 Apr 2007 12:45:16 +0900
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <e06186140704152028k9c5cdc2m3f90d0a71028b48e@mail.gmail.com>
References: <evpm71$q6c$1@sea.gmane.org>
	<462073A6.4010602@gmail.com>	<evqnir$44m$1@sea.gmane.org>
	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<e06186140704152028k9c5cdc2m3f90d0a71028b48e@mail.gmail.com>
Message-ID: <4622F14C.9080705@ar.media.kyoto-u.ac.jp>

Charles R Harris wrote:
>
>
> On 4/15/07, *David Cournapeau* <david at ar.media.kyoto-u.ac.jp 
> <mailto:david at ar.media.kyoto-u.ac.jp>> wrote:
>
>     Christian K wrote:
>     >
>     > Thanks, but that didn't help:
>     >
>     > atlas_info:
>     >   libraries lapack not found in /usr/lib/sse2
>     >   libraries f77blas,cblas,atlas not found in /usr/lib/atlas
>     >   libraries lapack_atlas not found in /usr/lib/atlas
>     >   libraries lapack not found in /usr/lib/sse2
>     >   libraries f77blas,cblas,atlas not found in /usr/lib
>     >   libraries lapack_atlas not found in /usr/lib
>     > numpy.distutils.system_info.atlas_info
>     >
>     /media/hda6/home/ck/prog/scipy/numpy/numpy/distutils/system_info.py:903:
>     > UserWarning:
>     >
>     *********************************************************************
>     >     Could not find lapack library within the ATLAS installation.
>     >
>     *********************************************************************
>     >
>     >   warnings.warn(message)
>     >   FOUND:
>     >     libraries = ['f77blas', 'cblas', 'atlas']
>     >     library_dirs = ['/usr/lib/sse2']
>     >     language = c
>     >     define_macros = [('ATLAS_WITHOUT_LAPACK', None)]
>     >
>     > lapack_info:
>     >   libraries lapack not found in /usr/lib/sse2
>     >   libraries lapack not found in /usr/lib
>     >   NOT AVAILABLE
>     >
>     >
>     > Christian
>     >
>     > _______________________________________________
>     > Numpy-discussion mailing list
>     > Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
>     > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>     >
>     >
>     On Ubuntu and debian, you do NOT need any site.cfg to compile
>     numpy with
>     atlas support. Just install the package atlas3-base-dev, and you are
>     done. The reason is that when *compiling* a software which needs
>     atlas,
>     the linker will try to find libblas.so in /usr/lib, not in
>     /usr/lib/sse2. If you install atlas3-base-dev, the package will
>     install
>     those at the correct locations. I have updated the instructions for
>     Ubuntu (also works for debian) on the wiki a few days ago:
>
>     http://www.scipy.org/Installing_SciPy/Linux#head-c5a062b2ecf76f746d78cfcde1dae00ae26109fe
>
>     Note that if you have also optimized version installed (sse, sse2), it
>     will use those automatically when you launch numpy (you can check by
>     doing ldd on the file numpy/core/_dotblas.so inside your numpy
>     installation.
>
>
> Be aware that on recent 64 bit Intel processors running a 64 bit OS 
> the base Atlas package appears to be buggy. You might be better off 
> building your own version, on a fast machine it won't take long.
I do not use any 64 bits machines myself, so I didn't know about this. 
Compiling atlas may take a long time on fast machine if you don't have 
arch defaults, though (it takes several hours for my workstation which 
has two xeon @ 3.2 ghz).

My impression is that using 64 bits OS is still a bit rough, anyway.

David


From Chris.Barker at noaa.gov  Mon Apr 16 17:13:00 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 16 Apr 2007 14:13:00 -0700
Subject: [Numpy-discussion] Fastest distance matrix calc
In-Reply-To: <e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>
References: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>
	<e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>
Message-ID: <4623E6DC.6000106@noaa.gov>

Timothy Hochberg wrote:
> results = empty([M, N], float)
> # You could be fancy and swap axes depending on which array is larger, but
> # I'll leave that for someone else
> for i, v in enumerate(x):
>     results[i] = sqrt(sum((v-y)**2, axis=-1))


you can probably use numpy.hypot(v-y) to speed this up more...


-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From rex at nosyntax.com  Mon Apr 16 18:53:34 2007
From: rex at nosyntax.com (rex)
Date: Mon, 16 Apr 2007 15:53:34 -0700
Subject: [Numpy-discussion] NumPy benchmark
Message-ID: <20070416225334.GT29927@x2.nosyntax.com>

I'm about to build numpy using Intel's MKL 9.1 beta and want to compare
it with the version I built using MKL 8.1. Is the LINPACK
benchmark the most appropriate?

Thanks,

-rex
-- 
Pollytheism: n., the belief that there are many gods, all of them parrots.


From mierle at gmail.com  Mon Apr 16 19:31:55 2007
From: mierle at gmail.com (Keir Mierle)
Date: Mon, 16 Apr 2007 19:31:55 -0400
Subject: [Numpy-discussion] Fastest distance matrix calc
In-Reply-To: <e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>
References: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>
	<e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>
Message-ID: <ef5675f30704161631r34e69314n981e3f615c9f2bf4@mail.gmail.com>

On 4/13/07, Timothy Hochberg <tim.hochberg at ieee.org> wrote:
> On 4/13/07, Bill Baxter <wbaxter at gmail.com> wrote:
> > I think someone posted some timings about this before but I don't recall.

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/498246

[snip]
> I'm going to go out on a limb and contend, without running any timings, that
> for large M and N, a solution using a for loop will beat either of those.
> For example (untested):
>
>  results = empty([M, N], float)
> # You could be fancy and swap axes depending on which array is larger, but
> # I'll leave that for someone else
> for i, v in enumerate(x):
>     results[i] = sqrt(sum((v-y)**2, axis=-1))
>  Or something like that. The reason that I suspect this will be faster is
> that it has better locality, completely finishing a computation on a
> relatively small working set before moving onto the next one. The one liners
> have to pull the potentially large MxN array into the processor repeatedly.

In my experience, it is indeed the case that the for loop version is
faster. The fastest of the three versions offered in the above url is
the last:

from numpy import mat, zeros, newaxis
def calcDistanceMatrixFastEuclidean2(nDimPoints):
    nDimPoints = array(nDimPoints)
    n,m = nDimPoints.shape
    delta = zeros((n,n),'d')
    for d in xrange(m):
        data = nDimPoints[:,d]
        delta += (data - data[:,newaxis])**2
    return sqrt(delta)

This is easily extended to two different nDimPoints matricies.

Cheers,
Keir


From wbaxter at gmail.com  Mon Apr 16 23:54:13 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Tue, 17 Apr 2007 12:54:13 +0900
Subject: [Numpy-discussion] Fastest distance matrix calc
In-Reply-To: <ef5675f30704161631r34e69314n981e3f615c9f2bf4@mail.gmail.com>
References: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>
	<e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>
	<ef5675f30704161631r34e69314n981e3f615c9f2bf4@mail.gmail.com>
Message-ID: <e86a5fd00704162054w418e642bpa4f12b998d736858@mail.gmail.com>

Here's a bunch of dist matrix implementations and their timings.
The upshot is that for most purposes this seems to be the best or at
least not too far off (basically the cookbook solution Kier posted)

def dist2hd(x,y):
    """Generate a 'coordinate' of the solution at a time"""
    d = npy.zeros((x.shape[0],y.shape[0]),dtype=x.dtype)
    for i in xrange(x.shape[1]):
        diff2 = x[:,i,None] - y[:,i]
        diff2 **= 2
        d += diff2
    npy.sqrt(d,d)
    return d

The only place where it's far from the best is for a small number of
points (~10) with high dimensionality (~100), which does come up in
machine learning contexts.  For those cases this does much better
(factor of :

def dist2b3(x,y):
    d = npy.dot(x,y.T)
    d *= -2.0
    d += (x*x).sum(1)[:,None]
    d += (y*y).sum(1)
    # Rounding errors occasionally cause negative entries in d
    d[d<0] = 0
    # in place sqrt
    npy.sqrt(d,d)
    return d

So given that, the obvious solution (if you don't want to delve into
non-numpy solutions) is  to use a hybrid that just switches between
the two.  Not sure what the proper switch is since it seems kind of
complicated, and probably depends some on cache specifics.  But just
switching based on the dimension of the points seems to be pretty
effective:

def dist2hy(x,y):
    if x.shape[1]<5:
        d = npy.zeros((x.shape[0],y.shape[0]),dtype=x.dtype)
        for i in xrange(x.shape[1]):
            diff2 = x[:,i,None] - y[:,i]
            diff2 **= 2
            d += diff2
        npy.sqrt(d,d)
        return d

    else:
        d = npy.dot(x,y.T)
        d *= -2.0
        d += (x*x).sum(1)[:,None]
        d += (y*y).sum(1)
        # Rounding errors occasionally cause negative entries in d
        d[d<0] = 0
        # in place sqrt
        npy.sqrt(d,d)
        return d

All of this assumes 'C' contiguous data.  All bets are off if you have
non-contiguous or 'F' ordered data.  And maybe if x and y have very
different numbers of points.


--bb



On 4/17/07, Keir Mierle <mierle at gmail.com> wrote:
> On 4/13/07, Timothy Hochberg <tim.hochberg at ieee.org> wrote:
> > On 4/13/07, Bill Baxter <wbaxter at gmail.com> wrote:
> > > I think someone posted some timings about this before but I don't recall.
>
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/498246
>
> [snip]
> > I'm going to go out on a limb and contend, without running any timings, that
> > for large M and N, a solution using a for loop will beat either of those.
> > For example (untested):
> >
> >  results = empty([M, N], float)
> > # You could be fancy and swap axes depending on which array is larger, but
> > # I'll leave that for someone else
> > for i, v in enumerate(x):
> >     results[i] = sqrt(sum((v-y)**2, axis=-1))
> >  Or something like that. The reason that I suspect this will be faster is
> > that it has better locality, completely finishing a computation on a
> > relatively small working set before moving onto the next one. The one liners
> > have to pull the potentially large MxN array into the processor repeatedly.
>
> In my experience, it is indeed the case that the for loop version is
> faster. The fastest of the three versions offered in the above url is
> the last:
>
> from numpy import mat, zeros, newaxis
> def calcDistanceMatrixFastEuclidean2(nDimPoints):
>     nDimPoints = array(nDimPoints)
>     n,m = nDimPoints.shape
>     delta = zeros((n,n),'d')
>     for d in xrange(m):
>         data = nDimPoints[:,d]
>         delta += (data - data[:,newaxis])**2
>     return sqrt(delta)
>
> This is easily extended to two different nDimPoints matricies.
>
> Cheers,
> Keir
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dist2perf.out
Type: application/octet-stream
Size: 6215 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070417/a7d1446d/attachment.obj>

From ckkart at hoc.net  Tue Apr 17 02:03:34 2007
From: ckkart at hoc.net (Christian K)
Date: Tue, 17 Apr 2007 15:03:34 +0900
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
References: <evpm71$q6c$1@sea.gmane.org>
	<462073A6.4010602@gmail.com>	<evqnir$44m$1@sea.gmane.org>
	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
Message-ID: <f01nug$nns$1@sea.gmane.org>

David Cournapeau wrote:
> On Ubuntu and debian, you do NOT need any site.cfg to compile numpy with 
> atlas support. Just install the package atlas3-base-dev, and you are 
> done. The reason is that when *compiling* a software which needs atlas, 
> the linker will try to find libblas.so in /usr/lib, not in 
> /usr/lib/sse2. If you install atlas3-base-dev, the package will install 
> those at the correct locations. I have updated the instructions for 
> Ubuntu (also works for debian) on the wiki a few days ago:

Indeed, installing atlas3-base-dev helps. I only had atlas3-base, atlas3-sse and
atlas3-sse2-dev installed. Sorry for the noise.

Christian



From ckkart at hoc.net  Tue Apr 17 02:06:18 2007
From: ckkart at hoc.net (Christian K)
Date: Tue, 17 Apr 2007 15:06:18 +0900
Subject: [Numpy-discussion] pickable ndarray subclass
In-Reply-To: <20070415140635.GI18196@mentat.za.net>
References: <evsbmj$qui$1@sea.gmane.org> <20070415140635.GI18196@mentat.za.net>
Message-ID: <f01o3k$o7s$1@sea.gmane.org>

Stefan van der Walt wrote:
> Hi Christiaan
> 
> On Sun, Apr 15, 2007 at 02:03:49PM +0900, Christian K wrote:
>> could someone please provide example code for how to make a subclassed ndarray
>> pickable? I don't quite understand the docs of ndarray.__reduce__.
>> My subclassed ndarray has just one additional attribute.
>>
> 
> __reduce__ is part of the pickle protocol, which is described at
> 
> http://docs.python.org/lib/node321.html
> 
> You need to modify __reduce__ to store the complete state of your
> custom object, as well as __setstate__ which restores the state on
> unpickling.
> 
> See the attached code as an example.  There, I have an InfoArray,
> which is a subclassed numpy.ndarray.  The InfoArray contains an
> additional member, info, which is a dictionary.
> 
> The __reduce__ method calls ndarray's __reduce__, then adds to the
> result the state of the InfoArray.  Conversely, the __setstate__
> method calls ndarray's __setstate__ as well as restoring the info
> member.

Thank you for the explanation and the example. Works very well.

Christian



From lbolla at gmail.com  Tue Apr 17 03:06:44 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Tue, 17 Apr 2007 09:06:44 +0200
Subject: [Numpy-discussion] NumPy benchmark
In-Reply-To: <20070416225334.GT29927@x2.nosyntax.com>
References: <20070416225334.GT29927@x2.nosyntax.com>
Message-ID: <80c99e790704170006o53b1a30ao940ccd0cdffb07fc@mail.gmail.com>

as soon as you do it, I'd like to compare them with the benchmarks I posted
here few days ago (compiled with gcc):
http://lbolla.wordpress.com/2007/04/11/numerical-computing-matlab-vs-pythonnumpyweave/

lorenzo.

On 4/17/07, rex <rex at nosyntax.com> wrote:
>
> I'm about to build numpy using Intel's MKL 9.1 beta and want to compare
> it with the version I built using MKL 8.1. Is the LINPACK
> benchmark the most appropriate?
>
> Thanks,
>
> -rex
> --
> Pollytheism: n., the belief that there are many gods, all of them parrots.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070417/5fe26130/attachment.html>

From rex at nosyntax.com  Tue Apr 17 03:32:53 2007
From: rex at nosyntax.com (rex)
Date: Tue, 17 Apr 2007 00:32:53 -0700
Subject: [Numpy-discussion] NumPy benchmark
In-Reply-To: <20070416225334.GT29927@x2.nosyntax.com>
References: <20070416225334.GT29927@x2.nosyntax.com>
Message-ID: <20070417073253.GZ29927@x2.nosyntax.com>

rex <rex at nosyntax.com> [2007-04-16 15:53]:
> I'm about to build numpy using Intel's MKL 9.1 beta and want to compare
> it with the version I built using MKL 8.1. Is the LINPACK
> benchmark the most appropriate?

I'm buried in responses. Not. 

A well-known benchmark (Scimark?) coded using NumPy/SciPy might
help people realize that they don't have to use a compiled language
for their problem. Alas, I can't find much in the way of benchmarks
coded using NumPy/SciPy. All I've found is LINPACK, coded using
Numarray.

import numarray, time
import numarray.random_array as naRA
import numarray.linear_algebra as naLA
n = 1000
a = naRA.random([n, n])
b = naRA.random([n, 1])
t = -time.time()
x = naLA.solve_linear_equations(a, b)
t += time.time()
r = numarray.dot(a, x) - b
r_n = numarray.maximum.reduce(abs(r))
print t, 2.0e-9 / 3.0 * n**3 / t
print r_n, r_n / (n * 1e-16)

Scimark is a broader test, but AFAIK it's only available in Java and
C. 

FWIW, one of my PCs was the first to break a gigaflop using Scimark.
Its score is 1043, which is 44% higher than the 2nd place score.

http://math.nist.gov/cgi-bin/ScimarkSummary

-rex
-- 
Neutrinos have bad breadth.


From matthieu.brucher at gmail.com  Tue Apr 17 09:57:38 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 17 Apr 2007 15:57:38 +0200
Subject: [Numpy-discussion] Fastest distance matrix calc
In-Reply-To: <4623E6DC.6000106@noaa.gov>
References: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>
	<e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>
	<4623E6DC.6000106@noaa.gov>
Message-ID: <e76aa17f0704170657v37664020w787fe2c32202c664@mail.gmail.com>

>
> you can probably use numpy.hypot(v-y) to speed this up more...
>

Tried it today, hypot takes two arguments :(
Is there a function that does the square root of the sum of squares ?

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070417/10495b0d/attachment.html>

From sturla at molden.no  Tue Apr 17 10:12:01 2007
From: sturla at molden.no (Sturla Molden)
Date: Tue, 17 Apr 2007 16:12:01 +0200
Subject: [Numpy-discussion] Fastest distance matrix calc
In-Reply-To: <e76aa17f0704170657v37664020w787fe2c32202c664@mail.gmail.com>
References: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>	<e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>	<4623E6DC.6000106@noaa.gov>
	<e76aa17f0704170657v37664020w787fe2c32202c664@mail.gmail.com>
Message-ID: <4624D5B1.1020307@molden.no>

On 4/17/2007 3:57 PM, Matthieu Brucher wrote:

> Is there a function that does the square root of the sum of squares ?

hm ...

f = lambda x : sqrt(sum(x**2))



S.M.


From subscriber100 at rjs.org  Tue Apr 17 10:56:49 2007
From: subscriber100 at rjs.org (Ray Schumacher)
Date: Tue, 17 Apr 2007 07:56:49 -0700
Subject: [Numpy-discussion] NumPy benchmark
Message-ID: <6.2.3.4.2.20070417075426.02e92340@pop-server.san.rr.com>

I'm still curious about the licensing aspects of using Intel's 
compiler and libs. Is the compiled Python/numpy result distributable, 
like any other compiled program?

Ray



From fullung at gmail.com  Tue Apr 17 11:19:42 2007
From: fullung at gmail.com (Albert Strasheim)
Date: Tue, 17 Apr 2007 17:19:42 +0200
Subject: [Numpy-discussion] NumPy benchmark
References: <6.2.3.4.2.20070417075426.02e92340@pop-server.san.rr.com>
Message-ID: <00bb01c78103$d2dcf710$0100a8c0@sun.ac.za>

Hello

----- Original Message ----- 
From: "Ray Schumacher" <subscriber100 at rjs.org>
To: <numpy-discussion at scipy.org>
Sent: Tuesday, April 17, 2007 4:56 PM
Subject: Re: [Numpy-discussion] NumPy benchmark


> I'm still curious about the licensing aspects of using Intel's 
> compiler and libs. Is the compiled Python/numpy result distributable, 
> like any other compiled program?

I'm not a lawyer, but I think the answer is yes.

The license agreements for the compiler and MKL:

http://www.intel.com/cd/software/products/asmo-na/eng/compilers/219625.htm
http://www.intel.com/cd/software/products/asmo-na/eng/219845.htm

and the FAQ for MKL:

http://www.intel.com/cd/software/products/asmo-na/eng/266854.htm

I put in a feature request with the Enthought folks to do exactly this:

https://svn.enthought.com/enthought/ticket/899

but it got postponed again recently.

Regards,

Albert


From strawman at astraw.com  Tue Apr 17 11:23:48 2007
From: strawman at astraw.com (Andrew Straw)
Date: Tue, 17 Apr 2007 08:23:48 -0700
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <f01nug$nns$1@sea.gmane.org>
References: <evpm71$q6c$1@sea.gmane.org>	<462073A6.4010602@gmail.com>	<evqnir$44m$1@sea.gmane.org>	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<f01nug$nns$1@sea.gmane.org>
Message-ID: <4624E684.9010606@astraw.com>

Christian K wrote:
> David Cournapeau wrote:
>   
>> On Ubuntu and debian, you do NOT need any site.cfg to compile numpy with 
>> atlas support. Just install the package atlas3-base-dev, and you are 
>> done. The reason is that when *compiling* a software which needs atlas, 
>> the linker will try to find libblas.so in /usr/lib, not in 
>> /usr/lib/sse2. If you install atlas3-base-dev, the package will install 
>> those at the correct locations. I have updated the instructions for 
>> Ubuntu (also works for debian) on the wiki a few days ago:
>>     
>
> Indeed, installing atlas3-base-dev helps. I only had atlas3-base, atlas3-sse and
> atlas3-sse2-dev installed. Sorry for the noise.
>   
Hmm, atlas3-sse2-dev doesn't depend on atlas3-base-dev? That sounds like 
a bug...

Off to investigate and possibly file a bug report with Debian or Ubuntu,
Andrew


From markusro at element.fkp.physik.tu-darmstadt.de  Tue Apr 17 11:36:19 2007
From: markusro at element.fkp.physik.tu-darmstadt.de (Markus Rosenstihl)
Date: Tue, 17 Apr 2007 17:36:19 +0200
Subject: [Numpy-discussion] Fastest distance matrix calc
In-Reply-To: <4624D5B1.1020307@molden.no>
References: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>	<e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>	<4623E6DC.6000106@noaa.gov>
	<e76aa17f0704170657v37664020w787fe2c32202c664@mail.gmail.com>
	<4624D5B1.1020307@molden.no>
Message-ID: <6503b4eb5af3777749527a83ebe78aea@element.fkp.physik.tu-darmstadt.de>

Or

f = sqrt(dot(x,x))


Am 17.04.2007 um 16:12 schrieb Sturla Molden:

> f = lambda x : sqrt(sum(x**2))
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: Signierter Teil der Nachricht
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070417/657e526c/attachment.sig>

From Chris.Barker at noaa.gov  Tue Apr 17 11:57:30 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 17 Apr 2007 08:57:30 -0700
Subject: [Numpy-discussion] Fastest distance matrix calc
In-Reply-To: <e76aa17f0704170657v37664020w787fe2c32202c664@mail.gmail.com>
References: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>
	<e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>
	<4623E6DC.6000106@noaa.gov>
	<e76aa17f0704170657v37664020w787fe2c32202c664@mail.gmail.com>
Message-ID: <4624EE6A.7010703@noaa.gov>

Matthieu Brucher wrote:
>     you can probably use numpy.hypot(v-y) to speed this up more...
> 
> 
> Tried it today, hypot takes two arguments :(
> Is there a function that does the square root of the sum of squares ?

then maybe you want:

numpy.hypot(v-y,v-y), though you should probably make a temporary v-y, 
so you dont' make two of them.

I've been assuming that hypot is written in C, rather than just a 
convenience function, but it it's the later, then it won't help here.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From bacmsantos at gmail.com  Tue Apr 17 12:03:29 2007
From: bacmsantos at gmail.com (Bruno Santos)
Date: Tue, 17 Apr 2007 17:03:29 +0100
Subject: [Numpy-discussion] Help using numPy to create a very large
	multi dimensional array
In-Reply-To: <e06186140704131036t4174905y466a4c0a15a815f1@mail.gmail.com>
References: <699044520704130350j7b53e36br5c0bc068ec00643c@mail.gmail.com>
	<e06186140704131036t4174905y466a4c0a15a815f1@mail.gmail.com>
Message-ID: <699044520704170903i26ea8686vcf0c63e0bd4d9cc4@mail.gmail.com>

I try to use the expression as you said, but I'm not getting the desired
result,
My text file look like this:

# num rows=115 num columns=2634
AbassiM.txt 0.033023 0.033023 0.033023 0.165115 0.462321....0.000000
AgricoleW.txt 0.038691 0.038691 0.038691 0.232147 0.541676....0.215300
AliR.txt 0.041885 0.041885 0.041885 0.125656 0.586395....0.633580
.....
....
....
ZhangJ.txt 0.047189 0.047189 0.047189 0.155048 0.613452....0.000000

using the code line you give I don't obtain a matrix with that shape,
instead I obtain the following  array([], shape=(0, 115), dtype=float64)

2007/4/13, Charles R Harris <charlesr.harris at gmail.com>:
>
>
>
> On 4/13/07, Bruno Santos <bacmsantos at gmail.com> wrote:
> >
> > Dear Sirs,
> > I'm trying to use Numpy to solve a speed problem with Python, I need to
> > perform agglomerative clustering as a first step to k-means clustering.
> > My problem is that I'm using a very large list in Pyhton and the script
> > is taking more than 9minutes to process all the information, so I'm trying
> > to use Numpy to create a matrix.
> > I'm reading the vectors from a text file and I end up with an array of
> > 115*2634 float elements, How can I create this structure with numpy?
> >
> > Where is my code in python:
> > #Read each document vector to a matrix
> >     doclist = []
> >     matrix = []
> >     list = []
> >     for line in vecfile:
> >         list = line.split()
> >         for elem in range(1, len(list)):
> >             list[elem] = float(list[elem])
> >         matrix.append (list[1:])
> >     vecfile.close()
>
>
> I don't know what your text file looks like or how many elements are in
> each line, but assuming 115 entries/line and spaces, something like the
> following will read in the data:
>
> m = N.fromfile('name of text file', sep=' ').reshape(-1,115)
>
> This assumes you have done import numpy as N and will result in a 2634x115
> array, which isn't very large.
>
>     #Read the desired number of final clusters
> >     numclust = input('Input the desired number of clusters: ')
> >
> > #Clustering process
> >     clust = rows
> >     ind = [-1, -1]
> >     list_j=[]
> >     list_k=[]
> >     while (clust > numclust):
> >         min = 2147483647
> >         print('Number of Clusters %d \n' % clust)
> >         #Find the 2 most similares vectors in the file
> >         for j in range(0, clust):
> >             list_j=matrix[j]
> >             for k in range(j+1, clust):
> >                 list_k=matrix[k]
> >                 dist=0
> >                 for e in range(0, columns):
> >                     result = list_j[e] - list_k[e]
> >                     dist += result * result
> >                 if (dist < min):
> >                     ind[0] = j
> >                     ind[1] = k
> >                     min = dist
>
>         #Combine the two most similaires vectores by median
> >         for e in range(0, columns): matrix[ind[0]][e] =
> > (matrix[ind[0]][e] + matrix[ind[1]][e]) / 2.0
> >         clust = clust -1
> >
> >         #Move up all the remaining vectors
> >         for k in range(ind[1], (rows - 1)):
> >             for e in range(0, columns): matrix[k][e]=matrix[k+1][e]
>
>
> This is the slow step, order N^3 in the number of vectors. It can be
> vectorized, but perhaps there is a better implementation of this algorithm.
> There may be an agglomerative clustering algorithm already available in
> scipy, the documentation indicates that kmeans clustering software is
> available. Perhaps someone closer to that library can help you there.
>
> Chuck
>
>
> >
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070417/b5ed1777/attachment.html>

From robert.kern at gmail.com  Tue Apr 17 12:08:10 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 Apr 2007 11:08:10 -0500
Subject: [Numpy-discussion] Fastest distance matrix calc
In-Reply-To: <4624EE6A.7010703@noaa.gov>
References: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>	<e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>	<4623E6DC.6000106@noaa.gov>	<e76aa17f0704170657v37664020w787fe2c32202c664@mail.gmail.com>
	<4624EE6A.7010703@noaa.gov>
Message-ID: <4624F0EA.3020409@gmail.com>

Christopher Barker wrote:
> Matthieu Brucher wrote:
>>     you can probably use numpy.hypot(v-y) to speed this up more...
>>
>> Tried it today, hypot takes two arguments :(
>> Is there a function that does the square root of the sum of squares ?
> 
> then maybe you want:
> 
> numpy.hypot(v-y,v-y), though you should probably make a temporary v-y, 
> so you dont' make two of them.
> 
> I've been assuming that hypot is written in C, rather than just a 
> convenience function, but it it's the later, then it won't help here.

It is written in C, but it doesn't do what you want.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From sberub at gmail.com  Tue Apr 17 12:43:37 2007
From: sberub at gmail.com (Simon Berube)
Date: Tue, 17 Apr 2007 16:43:37 -0000
Subject: [Numpy-discussion] Question about Optimization (Inline and Pyrex)
Message-ID: <1176828217.221367.205810@q75g2000hsh.googlegroups.com>

I recently made the switch from Matlab to Python and am very
interested in optimizing certain routines that I find too slow in
python/numpy (long loops).

I have looked and learned about the different methods used for such
problems such as blitz, weave and pyrex but had a question for more
experienced developpers.

It appears that pyrex is the fastest of the bunch with weave very
close behind but at the same time pyrex requires entirely different
modules while weave can be inserted almost painlessly into existing
code. Is the speed gain and usefulness of pyrex severely limited by
the extra maintenance required y having separate "fast" routines from
the rest of the code files?

I am greatly interested in finding out what more experienced
developers feel about these issues given that I may be completely off
track and missing on a useful tool(pyrex) thinking weave is better
than it actually is and I am quite frankly afraid of writing routines
in one format and realizing later that it creates problems that I need
to rewrite.

I have tried searching for previous similar posts but could not find
any. My apologies if this is a repeat or a severly dumb question.

Regards,

Simon Berube



From bsouthey at gmail.com  Tue Apr 17 14:11:20 2007
From: bsouthey at gmail.com (Bruce Southey)
Date: Tue, 17 Apr 2007 13:11:20 -0500
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <1176828217.221367.205810@q75g2000hsh.googlegroups.com>
References: <1176828217.221367.205810@q75g2000hsh.googlegroups.com>
Message-ID: <bbcd77d00704171111jcbe4b4bqfdf715a52adb7898@mail.gmail.com>

Hi,
You can find various suggestions to improve performance like Tim
Hochberg's list:

"""
0. Think about your algorithm.
1. Vectorize your inner loop.
2. Eliminate temporaries
3. Ask for help
4. Recode in C.
5. Accept that your code will never be fast.

Step zero should probably be repeated after every other step ;)
"""

The first item is very important because loop swapping and
factorization can really help. Item 1 is probably very important for
your 'long loops' . Also, Numpy may have a suitable function for some
of the calculations.

Bruce




On 4/17/07, Simon Berube <sberub at gmail.com> wrote:
> I recently made the switch from Matlab to Python and am very
> interested in optimizing certain routines that I find too slow in
> python/numpy (long loops).
>
> I have looked and learned about the different methods used for such
> problems such as blitz, weave and pyrex but had a question for more
> experienced developpers.
>
> It appears that pyrex is the fastest of the bunch with weave very
> close behind but at the same time pyrex requires entirely different
> modules while weave can be inserted almost painlessly into existing
> code. Is the speed gain and usefulness of pyrex severely limited by
> the extra maintenance required y having separate "fast" routines from
> the rest of the code files?
>
> I am greatly interested in finding out what more experienced
> developers feel about these issues given that I may be completely off
> track and missing on a useful tool(pyrex) thinking weave is better
> than it actually is and I am quite frankly afraid of writing routines
> in one format and realizing later that it creates problems that I need
> to rewrite.
>
> I have tried searching for previous similar posts but could not find
> any. My apologies if this is a repeat or a severly dumb question.
>
> Regards,
>
> Simon Berube
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From lou_boog2000 at yahoo.com  Tue Apr 17 14:48:00 2007
From: lou_boog2000 at yahoo.com (Lou Pecora)
Date: Tue, 17 Apr 2007 11:48:00 -0700 (PDT)
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <1176828217.221367.205810@q75g2000hsh.googlegroups.com>
Message-ID: <208795.83791.qm@web34408.mail.mud.yahoo.com>

You should probably look over your code and see if you
can eliminate loops by using the built in
vectorization of NumPy.  I've found this can really
speed things up.  E.g. given element by element
multiplication of two n-dimensional arrays x and y
replace, 

z=zeros(n)
for i in xrange(n):
  z[i]=x[i]*y[i]

with,

z=x*y   # NumPy will handle this in a vector fashion

Maybe you've already done that, but I thought I'd
offer it.
  
--- Simon Berube <sberub at gmail.com> wrote:

> I recently made the switch from Matlab to Python and
> am very
> interested in optimizing certain routines that I
> find too slow in
> python/numpy (long loops).
> 
> I have looked and learned about the different
> methods used for such
> problems such as blitz, weave and pyrex but had a
> question for more
> experienced developpers.
> 
> It appears that pyrex is the fastest of the bunch
> with weave very
> close behind but at the same time pyrex requires
> entirely different
> modules while weave can be inserted almost
> painlessly into existing
> code. Is the speed gain and usefulness of pyrex
> severely limited by
> the extra maintenance required y having separate
> "fast" routines from
> the rest of the code files?
> 
> I am greatly interested in finding out what more
> experienced
> developers feel about these issues given that I may
> be completely off
> track and missing on a useful tool(pyrex) thinking
> weave is better
> than it actually is and I am quite frankly afraid of
> writing routines
> in one format and realizing later that it creates
> problems that I need
> to rewrite.
> 
> I have tried searching for previous similar posts
> but could not find
> any. My apologies if this is a repeat or a severly
> dumb question.
> 
> Regards,
> 
> Simon Berube
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
>
http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 


-- Lou Pecora,   my views are my own.
---------------
"I knew I was going to take the wrong train, so I left early." 
--Yogi Berra

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


From peridot.faceted at gmail.com  Tue Apr 17 14:55:45 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 17 Apr 2007 14:55:45 -0400
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <208795.83791.qm@web34408.mail.mud.yahoo.com>
References: <1176828217.221367.205810@q75g2000hsh.googlegroups.com>
	<208795.83791.qm@web34408.mail.mud.yahoo.com>
Message-ID: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>

On 17/04/07, Lou Pecora <lou_boog2000 at yahoo.com> wrote:
> You should probably look over your code and see if you
> can eliminate loops by using the built in
> vectorization of NumPy.  I've found this can really
> speed things up.  E.g. given element by element
> multiplication of two n-dimensional arrays x and y
> replace,
>
> z=zeros(n)
> for i in xrange(n):
>   z[i]=x[i]*y[i]
>
> with,
>
> z=x*y   # NumPy will handle this in a vector fashion
>
> Maybe you've already done that, but I thought I'd
> offer it.

It's also worth mentioning that this sort of vectorization may allow
you to avoid python's global interpreter lock.

Normally, python's multithreading is effectively cooperative, because
the interpreter's data structures are all stored under the same lock,
so only one thread can be executing python bytecode at a time.
However, many of numpy's vectorized functions release the lock while
running, so on a multiprocessor or multicore machine you can have
several cores at once running vectorized code.

Anne M. Archibald


From miquel.poch at gmail.com  Tue Apr 17 14:58:43 2007
From: miquel.poch at gmail.com (Miquel Poch)
Date: Tue, 17 Apr 2007 20:58:43 +0200
Subject: [Numpy-discussion] Matlab Translation - sqrt elementwise
Message-ID: <8b0882fc0704171158w74d22b6bk3ba1d140d7fed97d@mail.gmail.com>

Hi,

I've found the next expression write it in Matlab,

Rtx   = sqrt(Rt);

Rtx is a matrix, and that's why I need sqrt() to operate elementwise. I've
read NumPy tutorial, and I know it's possible,

A set of this functions, has been provided wich optimize certain kinds of
calculations on arrays. Most of this functions, such as sin, cos and sqrt
are unary functions wich operate elementwise.  [Numerical Python, pg. 13]

but I don't know who to do it. The next error appear when I execute the
code,

''' exceptions.TypeError : only length-1 arrays can be converted to Python
scalars ''

Thanks in advance,
Mike <numpy-discussion at scipy.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070417/00f9509e/attachment.html>

From faltet at carabos.com  Tue Apr 17 15:00:49 2007
From: faltet at carabos.com (Francesc Altet)
Date: Tue, 17 Apr 2007 21:00:49 +0200
Subject: [Numpy-discussion] Question about Optimization (Inline
	and	Pyrex)
In-Reply-To: <1176828217.221367.205810@q75g2000hsh.googlegroups.com>
References: <1176828217.221367.205810@q75g2000hsh.googlegroups.com>
Message-ID: <1176836450.2559.115.camel@localhost.localdomain>

El dt 17 de 04 del 2007 a les 16:43 +0000, en/na Simon Berube va
escriure:
> I recently made the switch from Matlab to Python and am very
> interested in optimizing certain routines that I find too slow in
> python/numpy (long loops).
> 
> I have looked and learned about the different methods used for such
> problems such as blitz, weave and pyrex but had a question for more
> experienced developpers.
> 
> It appears that pyrex is the fastest of the bunch with weave very
> close behind but at the same time pyrex requires entirely different
> modules while weave can be inserted almost painlessly into existing
> code. Is the speed gain and usefulness of pyrex severely limited by
> the extra maintenance required y having separate "fast" routines from
> the rest of the code files?
> 
> I am greatly interested in finding out what more experienced
> developers feel about these issues given that I may be completely off
> track and missing on a useful tool(pyrex) thinking weave is better
> than it actually is and I am quite frankly afraid of writing routines
> in one format and realizing later that it creates problems that I need
> to rewrite.
> 
> I have tried searching for previous similar posts but could not find
> any. My apologies if this is a repeat or a severly dumb question.

Well, this is a delicate question. Let me put something clear before.
Pyrex code *might* be fast (as fast as C code can be in fact) if you
write good code, which is not an easy thing in most of situations mainly
because this does require mastery of not only the Pyrex language (which,
due to its similarity to Python, is relatively simple to learn), but
also (and specially) the internals of how your machine architecture
works (CPU bottlenecks, memory bottlenecks...).

When you compare weave (or whatever) against Pyrex in numerical
computations, you should have in mind other features as well and
specially the easy of use and the convenience to access the elements of
your numerical objects. I'm not a weave user, but I know that it allows
merging the weave code in your Python code and besides allows
multidimensional indexing (Pyrex don't). So, generally speaking, weave
is more high level (but still, fast!) than C, Fortran or Pyrex for doing
this kind of computations and depending on your needs, these factors
(and not only speed) can be worth considering.

Having said that, if you need to get all the performance that you
platform can offer to you, then Pyrex is an excellent option in that it
allows getting the maximum performance (if well coded, of course) from
the inside of the language. In addition, as it is heavily based on
Python syntax, it allows object oriented programming and excellent
interaction with Python code. These aforementioned factors are normally
very important ones when you have to develop relatively large modules
with high efficiency in mind. However, it must be stressed that Pyrex
*doesn't* allow to access multidimensional data in a convenient way (you
need to compute the indices your own for accessing the flat data array
in memory). It is true that this should'nt be a handicap for
undimensional or two-dimensional data, but it can be a pain if most of
your code has to deal with highly multidimensional objects.

Finally, don't let benchmarks fool you. If you can, it is always better
to run your own benchmarks made of your own problems. A tool that can be
killer for one application can be just mediocre for another (that's
somewhat extreme, but I hope you get the point).

Hope that helps,

-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth



From oliphant.travis at ieee.org  Tue Apr 17 15:09:14 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Tue, 17 Apr 2007 13:09:14 -0600
Subject: [Numpy-discussion] Matlab Translation - sqrt elementwise
In-Reply-To: <8b0882fc0704171158w74d22b6bk3ba1d140d7fed97d@mail.gmail.com>
References: <8b0882fc0704171158w74d22b6bk3ba1d140d7fed97d@mail.gmail.com>
Message-ID: <46251B5A.4050307@ieee.org>

Miquel Poch wrote:
> Hi,
>
> I've found the next expression write it in Matlab,
>
> Rtx   = sqrt(Rt);
>
> Rtx is a matrix, and that's why I need sqrt() to operate elementwise. 
> I've read NumPy tutorial, and I know it's possible,
>
> A set of this functions, has been provided wich optimize certain kinds 
> of calculations on arrays. Most of this functions, such as sin, cos 
> and sqrt are unary functions wich operate elementwise.  [Numerical 
> Python, pg. 13]
>
> but I don't know who to do it. The next error appear when I execute 
> the code,
>
> ''' exceptions.TypeError : only length-1 arrays can be converted to 
> Python scalars ''

It is a good idea to always show the code that leads to the error.  
Otherwise, we are guessing as to what you are really doing.

In this case, you are probably using the sqrt function from the math 
module rather than the sqrt function from the numpy module.

Try:

import numpy
numpy.sqrt(Rt)

-Travis



From rex at nosyntax.com  Tue Apr 17 15:06:07 2007
From: rex at nosyntax.com (rex)
Date: Tue, 17 Apr 2007 12:06:07 -0700
Subject: [Numpy-discussion] NumPy benchmark
In-Reply-To: <80c99e790704170006o53b1a30ao940ccd0cdffb07fc@mail.gmail.com>
References: <20070416225334.GT29927@x2.nosyntax.com>
	<80c99e790704170006o53b1a30ao940ccd0cdffb07fc@mail.gmail.com>
Message-ID: <20070417190607.GE8487@x2.nosyntax.com>

lorenzo bolla <lbolla at gmail.com> [2007-04-17 00:37]:
> as soon as you do it, I'd like to compare them with the benchmarks I posted
> here few days ago (compiled with gcc): 

http://lbolla.wordpress.com/2007/04/11/numerical-computing-matlab-vs-pythonnumpyweave/

Thanks for the link.

I haven't built numpy with MKL 9.1 yet, but here are some results
running laplace.py using MKL 8.1. The CPU is a Core 2 Duo (currently)
overclocked to 2.94 GHz (it will run at 3.52 GHz).

Using Python2.5 compiled with icc 9.1, numpy built with MKL 8.1
Doing 100 iterations on a 500x500 grid
numeric took 1.53 seconds
slow (100 iterations) took 130.02 seconds
slow with Psyco (100 iterations) took 107.91 seconds

Python compiled with icc takes 85 times longer to run this benchmark
than Python/NumPy does.

Using Python2.5 compiled with gcc, numpy built with MKL 8.1
Doing 100 iterations on a 500x500 grid
numeric took 1.57 seconds
slow (100 iterations) took 154.29 seconds
slow with Psyco (100 iterations) took 119.88 seconds

Python compiled with gcc takes 101 times longer to run this benchmark
than Python/NumPy/icc does.

The C++ version compiled with gcc 4.1.2 runs in .19 seconds.

-rex
-- 
I liked Occam's razor so much I bought the company.


From faltet at carabos.com  Tue Apr 17 15:06:05 2007
From: faltet at carabos.com (Francesc Altet)
Date: Tue, 17 Apr 2007 21:06:05 +0200
Subject: [Numpy-discussion] Matlab Translation - sqrt elementwise
In-Reply-To: <8b0882fc0704171158w74d22b6bk3ba1d140d7fed97d@mail.gmail.com>
References: <8b0882fc0704171158w74d22b6bk3ba1d140d7fed97d@mail.gmail.com>
Message-ID: <1176836765.2559.117.camel@localhost.localdomain>

El dt 17 de 04 del 2007 a les 20:58 +0200, en/na Miquel Poch va
escriure:
> Hi,
> 
> I've found the next expression write it in Matlab, 
> 
> Rtx   = sqrt(Rt);
> 
> Rtx is a matrix, and that's why I need sqrt() to operate elementwise.
> I've read NumPy tutorial, and I know it's possible, 
> 
> A set of this functions, has been provided wich optimize certain kinds
> of calculations on arrays. Most of this functions, such as sin, cos
> and sqrt are unary functions wich operate elementwise.  [Numerical
> Python, pg. 13]
> 
> but I don't know who to do it. The next error appear when I execute
> the code,
> 
> ''' exceptions.TypeError : only length-1 arrays can be converted to
> Python scalars ''

What are you doing? Operating element-wise is one of the most elementary
things that you can do with NumPy:

In [3]:a=numpy.array([[1,2],[3,4]])
In [4]:a
Out[4]:
array([[1, 2],
       [3, 4]])
In [5]:numpy.sqrt(a)
Out[5]:
array([[ 1.        ,  1.41421356],
       [ 1.73205081,  2.        ]])


Salut,

-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth



From lou_boog2000 at yahoo.com  Tue Apr 17 15:21:49 2007
From: lou_boog2000 at yahoo.com (Lou Pecora)
Date: Tue, 17 Apr 2007 12:21:49 -0700 (PDT)
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>
Message-ID: <439594.41469.qm@web34410.mail.mud.yahoo.com>

Now, I didn't know that.  That's cool because I have a
new dual core Intel Mac Pro.  I see I have some
learning to do with multithreading.  Thanks.

--- Anne Archibald <peridot.faceted at gmail.com> wrote:

> On 17/04/07, Lou Pecora <lou_boog2000 at yahoo.com>
> wrote:
> > You should probably look over your code and see if
> you
> > can eliminate loops by using the built in
> > vectorization of NumPy.  I've found this can
> really
> > speed things up.  E.g. given element by element
> > multiplication of two n-dimensional arrays x and y
> > replace,
> >
> > z=zeros(n)
> > for i in xrange(n):
> >   z[i]=x[i]*y[i]
> >
> > with,
> >
> > z=x*y   # NumPy will handle this in a vector
> fashion
> >
> > Maybe you've already done that, but I thought I'd
> > offer it.
> 
> It's also worth mentioning that this sort of
> vectorization may allow
> you to avoid python's global interpreter lock.
> 
> Normally, python's multithreading is effectively
> cooperative, because
> the interpreter's data structures are all stored
> under the same lock,
> so only one thread can be executing python bytecode
> at a time.
> However, many of numpy's vectorized functions
> release the lock while
> running, so on a multiprocessor or multicore machine
> you can have
> several cores at once running vectorized code.
> 
> Anne M. Archibald
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
>
http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 


-- Lou Pecora,   my views are my own.
---------------
"I knew I was going to take the wrong train, so I left early." 
--Yogi Berra

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


From peridot.faceted at gmail.com  Tue Apr 17 15:23:04 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 17 Apr 2007 15:23:04 -0400
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <1176836450.2559.115.camel@localhost.localdomain>
References: <1176828217.221367.205810@q75g2000hsh.googlegroups.com>
	<1176836450.2559.115.camel@localhost.localdomain>
Message-ID: <ce557a360704171223p1f64ebf4n275072efd900e2f8@mail.gmail.com>

On 17/04/07, Francesc Altet <faltet at carabos.com> wrote:

> Finally, don't let benchmarks fool you. If you can, it is always better
> to run your own benchmarks made of your own problems. A tool that can be
> killer for one application can be just mediocre for another (that's
> somewhat extreme, but I hope you get the point).

And, also important, don't forget that the time you usually care about
is the time until you obtain a correct solution of your problem -
which includes the time to write the code and the time to debug the
code. I find that it's extremely rare that the extra time it takes to
write highly-optimized code is worth the time it saves to run.

Anne M. Archibald


From peridot.faceted at gmail.com  Tue Apr 17 15:25:59 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 17 Apr 2007 15:25:59 -0400
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <439594.41469.qm@web34410.mail.mud.yahoo.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>
	<439594.41469.qm@web34410.mail.mud.yahoo.com>
Message-ID: <ce557a360704171225g66a9cf25t2b3da22ba5d21c0e@mail.gmail.com>

On 17/04/07, Lou Pecora <lou_boog2000 at yahoo.com> wrote:
> Now, I didn't know that.  That's cool because I have a
> new dual core Intel Mac Pro.  I see I have some
> learning to do with multithreading.  Thanks.

No problem. I had completely forgotten about the global interpreter
lock, wrote a little multithreading tool that ran my code in three
different threads, and got just about a 2x speedup on a dual-core
machine. Then someone reminded me about the GIL and I was puzzled...
your results will certainly depend on your code, but I found it useful
to have a little parallel-for-loop idiom for all those cases where
parallelism is stupidly easy.

Anne


From lou_boog2000 at yahoo.com  Tue Apr 17 15:29:47 2007
From: lou_boog2000 at yahoo.com (Lou Pecora)
Date: Tue, 17 Apr 2007 12:29:47 -0700 (PDT)
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <ce557a360704171225g66a9cf25t2b3da22ba5d21c0e@mail.gmail.com>
Message-ID: <931660.45117.qm@web34410.mail.mud.yahoo.com>

Ii get what you are saying, but I'm not even at the
Stupidly Easy Parallel level, yet.  Eventually. 
Thanks.

--- Anne Archibald <peridot.faceted at gmail.com> wrote:

> On 17/04/07, Lou Pecora <lou_boog2000 at yahoo.com>
> wrote:
> > Now, I didn't know that.  That's cool because I
> have a
> > new dual core Intel Mac Pro.  I see I have some
> > learning to do with multithreading.  Thanks.
> 
> No problem. I had completely forgotten about the
> global interpreter
> lock, wrote a little multithreading tool that ran my
> code in three
> different threads, and got just about a 2x speedup
> on a dual-core
> machine. Then someone reminded me about the GIL and
> I was puzzled...
> your results will certainly depend on your code, but
> I found it useful
> to have a little parallel-for-loop idiom for all
> those cases where
> parallelism is stupidly easy.
> 
> Anne
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
>
http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 


-- Lou Pecora,   my views are my own.
---------------
Great spirits have always encountered violent opposition from mediocre minds. 
-Albert Einstein

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


From jturner at gemini.edu  Tue Apr 17 15:55:09 2007
From: jturner at gemini.edu (James Turner)
Date: Tue, 17 Apr 2007 15:55:09 -0400
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>
Message-ID: <4625261D.2070303@gemini.edu>

Hi Anne,

Your reply to Lou raises a naive follow-up question of my own...

> Normally, python's multithreading is effectively cooperative, because
> the interpreter's data structures are all stored under the same lock,
> so only one thread can be executing python bytecode at a time.
> However, many of numpy's vectorized functions release the lock while
> running, so on a multiprocessor or multicore machine you can have
> several cores at once running vectorized code.

Are you saying that numpy's vectorized functions will perform a single
array operation in parallel on a multi-processor machine, or just that
the user can explicitly write threaded code to run *multiple* array
operations on different processors at the same time? I hope that's not
too stupid a question, but I haven't done any threaded programming yet
and the answer could be rather useful...

Thanks a lot,

James.



From peridot.faceted at gmail.com  Tue Apr 17 15:55:12 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 17 Apr 2007 15:55:12 -0400
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <931660.45117.qm@web34410.mail.mud.yahoo.com>
References: <ce557a360704171225g66a9cf25t2b3da22ba5d21c0e@mail.gmail.com>
	<931660.45117.qm@web34410.mail.mud.yahoo.com>
Message-ID: <ce557a360704171255s4c51a42brdf4c283df81b7df4@mail.gmail.com>

On 17/04/07, Lou Pecora <lou_boog2000 at yahoo.com> wrote:
> I get what you are saying, but I'm not even at the
> Stupidly Easy Parallel level, yet.  Eventually.

Well, it's hardly wonderful, but I wrote a little package to make idioms like:

d = {}
def work(f):
    d[f] = sum(exp(2.j*pi*f*times))
foreach(work,freqs,threads=3)

work fine.

Of course you need to make sure your threads don't accidentally
trample all over each other, but otherwise it's an easy way to get a
factor-of-two speedup.

Anne
-------------- next part --------------
A non-text attachment was scrubbed...
Name: handythread.py
Type: text/x-python
Size: 1047 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070417/9b159ac6/attachment.py>

From peridot.faceted at gmail.com  Tue Apr 17 16:02:24 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 17 Apr 2007 16:02:24 -0400
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <4625261D.2070303@gemini.edu>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>
	<4625261D.2070303@gemini.edu>
Message-ID: <ce557a360704171302s50e72f33r82fff6512ab656a8@mail.gmail.com>

On 17/04/07, James Turner <jturner at gemini.edu> wrote:
> Hi Anne,
>
> Your reply to Lou raises a naive follow-up question of my own...
>
> > Normally, python's multithreading is effectively cooperative, because
> > the interpreter's data structures are all stored under the same lock,
> > so only one thread can be executing python bytecode at a time.
> > However, many of numpy's vectorized functions release the lock while
> > running, so on a multiprocessor or multicore machine you can have
> > several cores at once running vectorized code.
>
> Are you saying that numpy's vectorized functions will perform a single
> array operation in parallel on a multi-processor machine, or just that
> the user can explicitly write threaded code to run *multiple* array
> operations on different processors at the same time? I hope that's not
> too stupid a question, but I haven't done any threaded programming yet
> and the answer could be rather useful...

For the most part, numpy's vectorized functions don't do anything
fancy in terms of computations; just giant for loops. What they do do
(and not necessarily all of them) is release the GIL so another thread
can be doing something else while they do that. That said, some of
them (dot for example) use BLAS in certain situations, and then all
bets are off. At the least a decent BLAS implementation will be smart
about cache behaviour; a fancy BLAS implementation might actually
vectorize the operation automatically. That would be using SSE3,
though, or some vector processor (Cray?), not likely SMP. Though I
can't say for sure. The scipy linear algebra functions use LAPACK,
which is more likely to be able to make such speedups (and in fact I'm
pretty sure there is an MPI-based LAPACK, though whether it's a
plug-in replacement I don't know).

Anne


From matthieu.brucher at gmail.com  Tue Apr 17 16:03:48 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 17 Apr 2007 22:03:48 +0200
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <4625261D.2070303@gemini.edu>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>
	<4625261D.2070303@gemini.edu>
Message-ID: <e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>

I would say that if the underlying atlas library is multithreaded, numpy
operations will be as well. Then, at the Python level, even if the
operations take a lot of time, the interpreter will be able to process
threads, as the lock is freed during the numpy operations - as I understood
for the last mails, only one thread can access the interpreter at a specific
time -

Matthieu

2007/4/17, James Turner <jturner at gemini.edu>:
>
> Hi Anne,
>
> Your reply to Lou raises a naive follow-up question of my own...
>
> > Normally, python's multithreading is effectively cooperative, because
> > the interpreter's data structures are all stored under the same lock,
> > so only one thread can be executing python bytecode at a time.
> > However, many of numpy's vectorized functions release the lock while
> > running, so on a multiprocessor or multicore machine you can have
> > several cores at once running vectorized code.
>
> Are you saying that numpy's vectorized functions will perform a single
> array operation in parallel on a multi-processor machine, or just that
> the user can explicitly write threaded code to run *multiple* array
> operations on different processors at the same time? I hope that's not
> too stupid a question, but I haven't done any threaded programming yet
> and the answer could be rather useful...
>
> Thanks a lot,
>
> James.
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070417/60f8e9f5/attachment.html>

From wbaxter at gmail.com  Tue Apr 17 16:24:37 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Wed, 18 Apr 2007 05:24:37 +0900
Subject: [Numpy-discussion] Fastest distance matrix calc
In-Reply-To: <e86a5fd00704162054w418e642bpa4f12b998d736858@mail.gmail.com>
References: <e86a5fd00704130343q48380b7fqeaf8c74cbcd6f341@mail.gmail.com>
	<e4412d6b0704130649s670dd00cybcf4efa553d0fb1c@mail.gmail.com>
	<ef5675f30704161631r34e69314n981e3f615c9f2bf4@mail.gmail.com>
	<e86a5fd00704162054w418e642bpa4f12b998d736858@mail.gmail.com>
Message-ID: <e86a5fd00704171324w2d560a2el3a40d8411d140657@mail.gmail.com>

Oops.  Looks like I forgot to attach the test program that generated
that output so you can tell what dist2g actually does.
Funny thing is -- despite being written in C, hypot doesn't actually
win any of the test cases for which it's applicable.

--bb

On 4/17/07, Bill Baxter <wbaxter at gmail.com> wrote:
> Here's a bunch of dist matrix implementations and their timings.
> The upshot is that for most purposes this seems to be the best or at
> least not too far off (basically the cookbook solution Kier posted)
>
> def dist2hd(x,y):
>     """Generate a 'coordinate' of the solution at a time"""
>     d = npy.zeros((x.shape[0],y.shape[0]),dtype=x.dtype)
>     for i in xrange(x.shape[1]):
>         diff2 = x[:,i,None] - y[:,i]
>         diff2 **= 2
>         d += diff2
>     npy.sqrt(d,d)
>     return d
>
> The only place where it's far from the best is for a small number of
> points (~10) with high dimensionality (~100), which does come up in
> machine learning contexts.  For those cases this does much better
> (factor of :
>
> def dist2b3(x,y):
>     d = npy.dot(x,y.T)
>     d *= -2.0
>     d += (x*x).sum(1)[:,None]
>     d += (y*y).sum(1)
>     # Rounding errors occasionally cause negative entries in d
>     d[d<0] = 0
>     # in place sqrt
>     npy.sqrt(d,d)
>     return d
>
> So given that, the obvious solution (if you don't want to delve into
> non-numpy solutions) is  to use a hybrid that just switches between
> the two.  Not sure what the proper switch is since it seems kind of
> complicated, and probably depends some on cache specifics.  But just
> switching based on the dimension of the points seems to be pretty
> effective:
>
> def dist2hy(x,y):
>     if x.shape[1]<5:
>         d = npy.zeros((x.shape[0],y.shape[0]),dtype=x.dtype)
>         for i in xrange(x.shape[1]):
>             diff2 = x[:,i,None] - y[:,i]
>             diff2 **= 2
>             d += diff2
>         npy.sqrt(d,d)
>         return d
>
>     else:
>         d = npy.dot(x,y.T)
>         d *= -2.0
>         d += (x*x).sum(1)[:,None]
>         d += (y*y).sum(1)
>         # Rounding errors occasionally cause negative entries in d
>         d[d<0] = 0
>         # in place sqrt
>         npy.sqrt(d,d)
>         return d
>
> All of this assumes 'C' contiguous data.  All bets are off if you have
> non-contiguous or 'F' ordered data.  And maybe if x and y have very
> different numbers of points.
>
>
> --bb
>
>
>
> On 4/17/07, Keir Mierle <mierle at gmail.com> wrote:
> > On 4/13/07, Timothy Hochberg <tim.hochberg at ieee.org> wrote:
> > > On 4/13/07, Bill Baxter <wbaxter at gmail.com> wrote:
> > > > I think someone posted some timings about this before but I don't recall.
> >
> > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/498246
> >
> > [snip]
> > > I'm going to go out on a limb and contend, without running any timings, that
> > > for large M and N, a solution using a for loop will beat either of those.
> > > For example (untested):
> > >
> > >  results = empty([M, N], float)
> > > # You could be fancy and swap axes depending on which array is larger, but
> > > # I'll leave that for someone else
> > > for i, v in enumerate(x):
> > >     results[i] = sqrt(sum((v-y)**2, axis=-1))
> > >  Or something like that. The reason that I suspect this will be faster is
> > > that it has better locality, completely finishing a computation on a
> > > relatively small working set before moving onto the next one. The one liners
> > > have to pull the potentially large MxN array into the processor repeatedly.
> >
> > In my experience, it is indeed the case that the for loop version is
> > faster. The fastest of the three versions offered in the above url is
> > the last:
> >
> > from numpy import mat, zeros, newaxis
> > def calcDistanceMatrixFastEuclidean2(nDimPoints):
> >     nDimPoints = array(nDimPoints)
> >     n,m = nDimPoints.shape
> >     delta = zeros((n,n),'d')
> >     for d in xrange(m):
> >         data = nDimPoints[:,d]
> >         delta += (data - data[:,newaxis])**2
> >     return sqrt(delta)
> >
> > This is easily extended to two different nDimPoints matricies.
> >
> > Cheers,
> > Keir
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dist2perf.py
Type: text/x-python
Size: 5359 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/f0a7b462/attachment.py>

From wbaxter at gmail.com  Tue Apr 17 16:31:56 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Wed, 18 Apr 2007 05:31:56 +0900
Subject: [Numpy-discussion] Matlab Translation - sqrt elementwise
In-Reply-To: <8b0882fc0704171158w74d22b6bk3ba1d140d7fed97d@mail.gmail.com>
References: <8b0882fc0704171158w74d22b6bk3ba1d140d7fed97d@mail.gmail.com>
Message-ID: <e86a5fd00704171331p24152db3p8afdb783bece2902@mail.gmail.com>

Be sure to check out the numpy examples page too.
    http://www.scipy.org/Numpy_Example_List
Always a good resource if you're not sure how to call a particular command.

--bb

On 4/18/07, Miquel Poch <miquel.poch at gmail.com> wrote:
> Hi,
>
> I've found the next expression write it in Matlab,
>
> Rtx   = sqrt(Rt);
>
> Rtx is a matrix, and that's why I need sqrt() to operate elementwise. I've
> read NumPy tutorial, and I know it's possible,
>
> A set of this functions, has been provided wich optimize certain kinds of
> calculations on arrays. Most of this functions, such as sin, cos and sqrt
> are unary functions wich operate elementwise.  [Numerical Python, pg. 13]
>
> but I don't know who to do it. The next error appear when I execute the
> code,
>
> ''' exceptions.TypeError : only length-1 arrays can be converted to Python
> scalars ''
>
> Thanks in advance,
> Mike
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From lou_boog2000 at yahoo.com  Tue Apr 17 16:42:07 2007
From: lou_boog2000 at yahoo.com (Lou Pecora)
Date: Tue, 17 Apr 2007 13:42:07 -0700 (PDT)
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <ce557a360704171255s4c51a42brdf4c283df81b7df4@mail.gmail.com>
Message-ID: <20070417204207.75756.qmail@web34407.mail.mud.yahoo.com>

Very nice.  Thanks.  Examples are welcome since they
are usually the best to get up to speed with
programming concepts.

--- Anne Archibald <peridot.faceted at gmail.com> wrote:

> On 17/04/07, Lou Pecora <lou_boog2000 at yahoo.com>
> wrote:
> > I get what you are saying, but I'm not even at the
> > Stupidly Easy Parallel level, yet.  Eventually.
> 
> Well, it's hardly wonderful, but I wrote a little
> package to make idioms like:

[cut]


-- Lou Pecora,   my views are my own.
---------------
Great spirits have always encountered violent opposition from mediocre minds. 
-Albert Einstein

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


From robert.kern at gmail.com  Tue Apr 17 16:43:38 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 Apr 2007 15:43:38 -0500
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>	<4625261D.2070303@gemini.edu>
	<e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>
Message-ID: <4625317A.10408@gmail.com>

Matthieu Brucher wrote:
> I would say that if the underlying atlas library is multithreaded, numpy
> operations will be as well. Then, at the Python level, even if the
> operations take a lot of time, the interpreter will be able to process
> threads, as the lock is freed during the numpy operations - as I
> understood for the last mails, only one thread can access the
> interpreter at a specific time -

ATLAS doesn't *underlie* much of numpy at all. Just dot() and the functions in
linalg, nothing else.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From rex at nosyntax.com  Tue Apr 17 20:14:36 2007
From: rex at nosyntax.com (rex)
Date: Tue, 17 Apr 2007 17:14:36 -0700
Subject: [Numpy-discussion] NumPy benchmark
In-Reply-To: <20070417190607.GE8487@x2.nosyntax.com>
References: <20070416225334.GT29927@x2.nosyntax.com>
	<80c99e790704170006o53b1a30ao940ccd0cdffb07fc@mail.gmail.com>
	<20070417190607.GE8487@x2.nosyntax.com>
Message-ID: <20070418001436.GH8487@x2.nosyntax.com>

Using MKL 9.1_beta made no difference in the prior benchmark, but it
does improve speed in an earlier benchmark I posted.

From:
http://projects.scipy.org/pipermail/numpy-discussion/2007-January/025673.html
================================================================================
''' A program that uses Monte Carlo to estimate how often the number of rare events
with a Poisson distribution will differ by a given amount.
'''
import numpy as n
from numpy.random import poisson
from time import time


lam = 4.0  # mu & var for Poisson distributed rands (they are equal in Poisson)
N = 10              #number of times to run the program
maxNumEvents = 20   #events larger than this are ignored
numPois = 100000    #number of pairs of outcomes to generate
freqA = 2           #number of times event A occurred
freqB = 6           #number of times event B occurred

print "#rands fraction [freqA,freqB]  fraction [lam,lam]  largest%  total[mean,mean]"
t0 = time()
for g in range(1):
    for h in range(N):
        suma = n.zeros((maxNumEvents+1,maxNumEvents+1), int)  #possible outcomes array
        count = poisson(lam, size =(numPois,2))  #generate array of pairs of Poissons
        for i in range(numPois):
            #if count[i,0] > maxNumEvents: continue
            #if count[i,1] > maxNumEvents: continue
            suma[count[i,0],count[i,1]] += 1
        d = n.sum(suma)
        print d, float(suma[freqA,freqB])/d, float(suma[lam,lam])/d , suma.max(), suma[lam,lam]
print 'time', time()-t0


Using the SUSE NumPy rpm:
python relative_risk.py
#rands fraction [2,6]  fraction [lam,lam]  largest%  total[mean,mean]
100000 0.01539 0.03869 3869 3869
100000 0.01534 0.03766 3907 3766
100000 0.01553 0.03841 3859 3841
100000 0.01496 0.03943 3943 3943
100000 0.01513 0.03829 3856 3829
100000 0.01485 0.03825 3993 3825
100000 0.01545 0.03716 3859 3716
100000 0.01526 0.03909 3919 3909
100000 0.01491 0.03826 3913 3826
100000 0.01478 0.03771 3782 3771
time 2.38847184181

Using the MKL [8.1] NumPy:
python relative_risk.py
#rands fraction [2,6]  fraction [lam,lam]  largest%  total[mean,mean]
100000 0.01502 0.03764 3895 3764
100000 0.01513 0.03841 3841 3841
100000 0.01511 0.03753 3810 3753
100000 0.01577 0.03766 3873 3766
100000 0.01541 0.0373 3963 3730
100000 0.01586 0.03862 3912 3862
100000 0.01552 0.03785 3870 3785
100000 0.01502 0.03854 3896 3854
100000 0.015 0.03803 3880 3803
100000 0.01515 0.03749 3855 3749
time 2.0455300808

So the rpm version only takes ~17% longer to run this program. I'm surprised
that there isn't a larger difference. Perhaps there will be in a
different type of program. BTW, the cpu is an Intel e6600 Core 2 Duo
overclocked to 3.06 GHz (it will run reliably at 3.24 GHz).
============================================================================

With NumPy built using MKL 9.1_beta the program runs in 1.66
seconds. Correcting for the slightly lower CPU speed used (2.93 GHz), this
corresponds to 1.59 seconds. The 8.1 version takes 29% longer (this may
be partially/all due to different compiler flags being used), and the
8.1 version used with Python compiled with gcc instead of icc takes 50%
longer to run. 

The icc flags used were: 
-fast  (enables -xP -O3 -ipo -no-prec-div -static)
-funroll-loops
-fno-alias
-parallel

I don't know if there are obviously better choices for the Core 2 Duo.

I'd like to run a more comprehensive benchmark, say Scimark translated
from C to Python/NumPy.

http://math.nist.gov/scimark2/download_c.html

-rex
-- 
Those who forget the pasta are condemed to reheat it.


From david at ar.media.kyoto-u.ac.jp  Tue Apr 17 22:51:08 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 18 Apr 2007 11:51:08 +0900
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <4624E684.9010606@astraw.com>
References: <evpm71$q6c$1@sea.gmane.org>	<462073A6.4010602@gmail.com>	<evqnir$44m$1@sea.gmane.org>	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>	<f01nug$nns$1@sea.gmane.org>
	<4624E684.9010606@astraw.com>
Message-ID: <4625879C.6000706@ar.media.kyoto-u.ac.jp>

Andrew Straw wrote:
> Christian K wrote:
>   
>> David Cournapeau wrote:
>>   
>>     
>>> On Ubuntu and debian, you do NOT need any site.cfg to compile numpy with 
>>> atlas support. Just install the package atlas3-base-dev, and you are 
>>> done. The reason is that when *compiling* a software which needs atlas, 
>>> the linker will try to find libblas.so in /usr/lib, not in 
>>> /usr/lib/sse2. If you install atlas3-base-dev, the package will install 
>>> those at the correct locations. I have updated the instructions for 
>>> Ubuntu (also works for debian) on the wiki a few days ago:
>>>     
>>>       
>> Indeed, installing atlas3-base-dev helps. I only had atlas3-base, atlas3-sse and
>> atlas3-sse2-dev installed. Sorry for the noise.
>>   
>>     
> Hmm, atlas3-sse2-dev doesn't depend on atlas3-base-dev? That sounds like 
> a bug...
>   
Now that you mention it, I am also puzzled by this one: I can see why 
you would use atlas3-sse2-dev without atlas3-base-dev (for the static 
library), but not having atlas3-base-dev makes it imposible to 
dynamically link to atlas libraries without customizing makefiles and/or 
configure options. I will report this as a bug, see what the maintainers 
say.

David


From haase at msg.ucsf.edu  Wed Apr 18 00:32:15 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 17 Apr 2007 21:32:15 -0700
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <ce557a360704171255s4c51a42brdf4c283df81b7df4@mail.gmail.com>
References: <ce557a360704171225g66a9cf25t2b3da22ba5d21c0e@mail.gmail.com>
	<931660.45117.qm@web34410.mail.mud.yahoo.com>
	<ce557a360704171255s4c51a42brdf4c283df81b7df4@mail.gmail.com>
Message-ID: <bc657ead0704172132k5f20a08crd4c3d5bcf59a7200@mail.gmail.com>

Hi Anne,
I'm just starting to look into your code (sound very interesting -
should probably be put onto the wiki)
 -- quick note:
you are mixing tabs and spaces  :-(
what editor are you using !?

-Sebastian



On 4/17/07, Anne Archibald <peridot.faceted at gmail.com> wrote:
> On 17/04/07, Lou Pecora <lou_boog2000 at yahoo.com> wrote:
> > I get what you are saying, but I'm not even at the
> > Stupidly Easy Parallel level, yet.  Eventually.
>
> Well, it's hardly wonderful, but I wrote a little package to make idioms like:
>
> d = {}
> def work(f):
>    d[f] = sum(exp(2.j*pi*f*times))
> foreach(work,freqs,threads=3)
>
> work fine.
>
> Of course you need to make sure your threads don't accidentally
> trample all over each other, but otherwise it's an easy way to get a
> factor-of-two speedup.
>
> Anne
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>


From haase at msg.ucsf.edu  Wed Apr 18 00:41:13 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 17 Apr 2007 21:41:13 -0700
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <4625317A.10408@gmail.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>
	<4625261D.2070303@gemini.edu>
	<e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>
	<4625317A.10408@gmail.com>
Message-ID: <bc657ead0704172141j1bfa6312s2c4f6df721b11d28@mail.gmail.com>

On 4/17/07, Robert Kern <robert.kern at gmail.com> wrote:
> Matthieu Brucher wrote:
> > I would say that if the underlying atlas library is multithreaded, numpy
> > operations will be as well. Then, at the Python level, even if the
> > operations take a lot of time, the interpreter will be able to process
> > threads, as the lock is freed during the numpy operations - as I
> > understood for the last mails, only one thread can access the
> > interpreter at a specific time -
>
> ATLAS doesn't *underlie* much of numpy at all. Just dot() and the functions in
> linalg, nothing else.
>
Hi,
I don't know much about ATLAS -- would there be other numpy functions
that *could*  or *should*  be implemented using ATLAS !?
Any ?

-Sebastian


From robert.kern at gmail.com  Wed Apr 18 00:43:41 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 17 Apr 2007 23:43:41 -0500
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <bc657ead0704172141j1bfa6312s2c4f6df721b11d28@mail.gmail.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>	<4625261D.2070303@gemini.edu>	<e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>	<4625317A.10408@gmail.com>
	<bc657ead0704172141j1bfa6312s2c4f6df721b11d28@mail.gmail.com>
Message-ID: <4625A1FD.1050001@gmail.com>

Sebastian Haase wrote:

> Hi,
> I don't know much about ATLAS -- would there be other numpy functions
> that *could*  or *should*  be implemented using ATLAS !?
> Any ?

Not really, no.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From peridot.faceted at gmail.com  Wed Apr 18 01:33:53 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 18 Apr 2007 01:33:53 -0400
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <4625A1FD.1050001@gmail.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>
	<4625261D.2070303@gemini.edu>
	<e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>
	<4625317A.10408@gmail.com>
	<bc657ead0704172141j1bfa6312s2c4f6df721b11d28@mail.gmail.com>
	<4625A1FD.1050001@gmail.com>
Message-ID: <ce557a360704172233v7790b6cfvac6ec8477ccbb4e1@mail.gmail.com>

On 18/04/07, Robert Kern <robert.kern at gmail.com> wrote:
> Sebastian Haase wrote:
>
> > Hi,
> > I don't know much about ATLAS -- would there be other numpy functions
> > that *could*  or *should*  be implemented using ATLAS !?
> > Any ?
>
> Not really, no.

ATLAS is a library designed to implement linear algebra functions
efficiently on many machines. It does things like reorder the
multiplications and additions in matrix multiplication to make the
best possible use of your cache, as measured by empirical testing.
(FFTW does something similar for the FFT.) But ATLAS is only designed
for linear algebra. If what you want to do is linear algebra, look at
scipy for a full selection of linear algebra routines that make fairly
good use of ATLAS where applicable.

It would be perfectly possible, in principle, to implement an
ATLAS-like library that handled a variety (perhaps all) of numpy's
basic operations in platform-optimized fashion. But implementing ATLAS
is not a simple process! And it's not clear how much gain would be
available - it would almost certainly be noticeably faster only for
very large numpy objects (where the python overhead is unimportant),
and those objects can be very inefficient because of excessive
copying. And the scope of improvement would be very limited; an
expression like A*B+C*D would be much more efficient, probably, if the
whole expression were evaluated at once for each element (due to
memory locality and temporary allocation). But it is impossible for
numpy, sitting inside python as it does, to do that.

Anne M. Archibald


From peridot.faceted at gmail.com  Wed Apr 18 01:39:22 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 18 Apr 2007 01:39:22 -0400
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <bc657ead0704172132k5f20a08crd4c3d5bcf59a7200@mail.gmail.com>
References: <ce557a360704171225g66a9cf25t2b3da22ba5d21c0e@mail.gmail.com>
	<931660.45117.qm@web34410.mail.mud.yahoo.com>
	<ce557a360704171255s4c51a42brdf4c283df81b7df4@mail.gmail.com>
	<bc657ead0704172132k5f20a08crd4c3d5bcf59a7200@mail.gmail.com>
Message-ID: <ce557a360704172239s73f09309j85951044c53468f3@mail.gmail.com>

On 18/04/07, Sebastian Haase <haase at msg.ucsf.edu> wrote:
> Hi Anne,
> I'm just starting to look into your code (sound very interesting -
> should probably be put onto the wiki)
>  -- quick note:
> you are mixing tabs and spaces  :-(
> what editor are you using !?

Agh. vim is misbehaving. Sorry about that. I just took another look at
that code and added a parallel_map I hadn't got around to writing
before, too. I'd be happy to stick it (and test file) on the wiki
under some open license or other ("do what thou wilt shall be the
whole of the law"?). It's certainly not competition for ipython1,
though, it's mostly to show an example of making threads easy to use.

Anne


From david at ar.media.kyoto-u.ac.jp  Wed Apr 18 01:38:49 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 18 Apr 2007 14:38:49 +0900
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <ce557a360704172233v7790b6cfvac6ec8477ccbb4e1@mail.gmail.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>	<4625261D.2070303@gemini.edu>	<e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>	<4625317A.10408@gmail.com>	<bc657ead0704172141j1bfa6312s2c4f6df721b11d28@mail.gmail.com>	<4625A1FD.1050001@gmail.com>
	<ce557a360704172233v7790b6cfvac6ec8477ccbb4e1@mail.gmail.com>
Message-ID: <4625AEE9.5070301@ar.media.kyoto-u.ac.jp>

Anne Archibald wrote:
>
>  And the scope of improvement would be very limited; an
> expression like A*B+C*D would be much more efficient, probably, if the
> whole expression were evaluated at once for each element (due to
> memory locality and temporary allocation). But it is impossible for
> numpy, sitting inside python as it does, to do that.
>   
My understanding is that alternative implementations of python such as 
pypy makes this kind of things possible (at least in theory). I asked a 
question about this a few months ago:

http://www.mail-archive.com/pypy-dev at codespeak.net/msg02243.html

David


From ckkart at hoc.net  Wed Apr 18 02:09:57 2007
From: ckkart at hoc.net (Christian K.)
Date: Wed, 18 Apr 2007 15:09:57 +0900
Subject: [Numpy-discussion] Help using numPy to create a very large
	multi dimensional array
In-Reply-To: <699044520704170903i26ea8686vcf0c63e0bd4d9cc4@mail.gmail.com>
References: <699044520704130350j7b53e36br5c0bc068ec00643c@mail.gmail.com>	<e06186140704131036t4174905y466a4c0a15a815f1@mail.gmail.com>
	<699044520704170903i26ea8686vcf0c63e0bd4d9cc4@mail.gmail.com>
Message-ID: <f04cme$k56$1@sea.gmane.org>

Bruno Santos wrote:
> I try to use the expression as you said, but I'm not getting the desired 
> result,
> My text file look like this:
> 
> # num rows=115 num columns=2634
> AbassiM.txt 0.033023 0.033023 0.033023 0.165115 0.462321....0.000000
> AgricoleW.txt 0.038691 0.038691 0.038691 0.232147 0.541676....0.215300
> AliR.txt 0.041885 0.041885 0.041885 0.125656 0.586395....0.633580
> .....
> ....
> ....
> ZhangJ.txt 0.047189 0.047189 0.047189 0.155048 0.613452....0.000000

I guess N.fromfile can't handle non numeric data. Use something like 
this instead (not tested):

import numpy as N

data = open('name of file').readlines()

data = N.array([[float(x) for x in row.split(' ')[1:]] for row in data[1:]])

(the above expression should be one line)

Christian



From cookedm at physics.mcmaster.ca  Wed Apr 18 02:33:55 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Wed, 18 Apr 2007 02:33:55 -0400
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <ce557a360704172233v7790b6cfvac6ec8477ccbb4e1@mail.gmail.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>	<4625261D.2070303@gemini.edu>	<e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>	<4625317A.10408@gmail.com>	<bc657ead0704172141j1bfa6312s2c4f6df721b11d28@mail.gmail.com>	<4625A1FD.1050001@gmail.com>
	<ce557a360704172233v7790b6cfvac6ec8477ccbb4e1@mail.gmail.com>
Message-ID: <4625BBD3.7050508@physics.mcmaster.ca>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Anne Archibald wrote:
> 
> It would be perfectly possible, in principle, to implement an
> ATLAS-like library that handled a variety (perhaps all) of numpy's
> basic operations in platform-optimized fashion. But implementing ATLAS
> is not a simple process! And it's not clear how much gain would be
> available - it would almost certainly be noticeably faster only for
> very large numpy objects (where the python overhead is unimportant),
> and those objects can be very inefficient because of excessive
> copying. And the scope of improvement would be very limited; an
> expression like A*B+C*D would be much more efficient, probably, if the
> whole expression were evaluated at once for each element (due to
> memory locality and temporary allocation). But it is impossible for
> numpy, sitting inside python as it does, to do that.

numexpr (in the scipy sandbox) does something like this: it takes an
expression like A*B+C*D and constructs a small bytecode program that
does that calculation in chunks, minimising temporary variables and
number of passes through memory. As it is, the speed is faster than the
python expression, and comparable to that of weave. I've been thinking
of making a JIT for it, but I haven't had the time :)

- --
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGJbvTN9ixZKFWjRQRAi+WAJ9HmeCTeB59Jso5vlVzbgHQ0TDj9ACfdKWy
jYEnsRYau8T5BVAKnZJWpLk=
=75Jc
-----END PGP SIGNATURE-----


From lbolla at gmail.com  Wed Apr 18 03:42:24 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Wed, 18 Apr 2007 09:42:24 +0200
Subject: [Numpy-discussion] NumPy benchmark
In-Reply-To: <20070417190607.GE8487@x2.nosyntax.com>
References: <20070416225334.GT29927@x2.nosyntax.com>
	<80c99e790704170006o53b1a30ao940ccd0cdffb07fc@mail.gmail.com>
	<20070417190607.GE8487@x2.nosyntax.com>
Message-ID: <80c99e790704180042g39eff280s40c4c13386f34241@mail.gmail.com>

the amazing performance of C++ code does not surprise me: a tenfold
improvement of the simple Python/Numpy code can be achieved with
weave.inline or Pyrex.

Hence your benchmarks seems to confirm that "weaved" or "pyrexed" code run
as fast as C++ compiled one.

Moreover, from your numbers, I can tell that compiling numpy with gcc or icc
makes no big difference.

Am I correct?

If yes, let me know if I can add this info to the scipy wiki: I'm preparing
an extention to this page http://www.scipy.org/PerformancePython.

cheers,
lorenzo




On 4/17/07, rex <rex at nosyntax.com> wrote:
>
> lorenzo bolla <lbolla at gmail.com> [2007-04-17 00:37]:
> > as soon as you do it, I'd like to compare them with the benchmarks I
> posted
> > here few days ago (compiled with gcc):
>
>
> http://lbolla.wordpress.com/2007/04/11/numerical-computing-matlab-vs-pythonnumpyweave/
>
> Thanks for the link.
>
> I haven't built numpy with MKL 9.1 yet, but here are some results
> running laplace.py using MKL 8.1. The CPU is a Core 2 Duo (currently)
> overclocked to 2.94 GHz (it will run at 3.52 GHz).
>
> Using Python2.5 compiled with icc 9.1, numpy built with MKL 8.1
> Doing 100 iterations on a 500x500 grid
> numeric took 1.53 seconds
> slow (100 iterations) took 130.02 seconds
> slow with Psyco (100 iterations) took 107.91 seconds
>
> Python compiled with icc takes 85 times longer to run this benchmark
> than Python/NumPy does.
>
> Using Python2.5 compiled with gcc, numpy built with MKL 8.1
> Doing 100 iterations on a 500x500 grid
> numeric took 1.57 seconds
> slow (100 iterations) took 154.29 seconds
> slow with Psyco (100 iterations) took 119.88 seconds
>
> Python compiled with gcc takes 101 times longer to run this benchmark
> than Python/NumPy/icc does.
>
> The C++ version compiled with gcc 4.1.2 runs in .19 seconds.
>
> -rex
> --
> I liked Occam's razor so much I bought the company.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/23819849/attachment.html>

From bacmsantos at gmail.com  Wed Apr 18 07:04:37 2007
From: bacmsantos at gmail.com (Bruno Santos)
Date: Wed, 18 Apr 2007 12:04:37 +0100
Subject: [Numpy-discussion] Help using numPy to create a very large
	multi dimensional array
In-Reply-To: <f04cme$k56$1@sea.gmane.org>
References: <699044520704130350j7b53e36br5c0bc068ec00643c@mail.gmail.com>
	<e06186140704131036t4174905y466a4c0a15a815f1@mail.gmail.com>
	<699044520704170903i26ea8686vcf0c63e0bd4d9cc4@mail.gmail.com>
	<f04cme$k56$1@sea.gmane.org>
Message-ID: <699044520704180404p5b27bef9xa0483f9093c47528@mail.gmail.com>

Finally I was able to read the data, by using the command you sair with some
small changes:
matrix = numpy.array([[float(x) for x in line.split()[1:]] for line in
vecfile])

But that doesn't solve my speed problem, now instead of taking 40seconds in
the slow step, takes 1min ant 10seconds :(
The slow step is this cycle:
for j in range(0, clust):
            list_j= numpy.asarray(matrix[j])
            for k in range(j+1, clust):
                list_k=numpy.asarray(matrix[k])
                dist=0
                for e in range(0, columns):
                    result = list_j[e] - list_k[e]
                    dist += result * result
                if (dist < min):
                    ind[0] = j
                    ind[1] = k
                    min = dist

I also try with list_j = numpy.array but it only slower even more the
calculation,
Does anyone have any ideia how I can speed up this step?


2007/4/18, Christian K. <ckkart at hoc.net>:
>
> Bruno Santos wrote:
> > I try to use the expression as you said, but I'm not getting the desired
> > result,
> > My text file look like this:
> >
> > # num rows=115 num columns=2634
> > AbassiM.txt 0.033023 0.033023 0.033023 0.165115 0.462321....0.000000
> > AgricoleW.txt 0.038691 0.038691 0.038691 0.232147 0.541676....0.215300
> > AliR.txt 0.041885 0.041885 0.041885 0.125656 0.586395....0.633580
> > .....
> > ....
> > ....
> > ZhangJ.txt 0.047189 0.047189 0.047189 0.155048 0.613452....0.000000
>
> I guess N.fromfile can't handle non numeric data. Use something like
> this instead (not tested):
>
> import numpy as N
>
> data = open('name of file').readlines()
>
> data = N.array([[float(x) for x in row.split(' ')[1:]] for row in
> data[1:]])
>
> (the above expression should be one line)
>
> Christian
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/a0b701bb/attachment.html>

From christian at marquardt.sc  Wed Apr 18 08:19:04 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Wed, 18 Apr 2007 14:19:04 +0200 (CEST)
Subject: [Numpy-discussion] Problems building numpy and scipy on AIX
Message-ID: <30420.193.17.11.23.1176898744.squirrel@webmail.marquardt.sc>

Hello,

I've run into a problem building numpy-1.0.2 on AIX (using gcc and native
Fortran compilers). The problem on that platform in general is that the
process for building shared libraries is different from what's normally
done (and it's a pain...)

Anyway. Core python has a dedicated script called ld_so_aix which is used
to create the shared objects (it can be found in the
.../lib/python2.5/config directory of any installed python version).
Everything compiled with the compiler that was used to build python
itself, and which is compiled through the standard distutils, is also
using this construction, so building C extensions usually works fine.

I ran into problems during the installation of numpy 1.0.2 using

   python setup.py config_fc --fcompiler=ibm install

because the numpy distutils try to use the xlf95 compiler to create a
shared image - what needs to be done instead is to wrap the call to the
compiler with the ld_so_aix script. I have attached a patch which solves
the problem for me (and python 2.5), but I don't know if it is the right
place and way to do that - and the python version is hardwired as well, so
it's not really a fix.

What I also find a bit suspicuous is that the Fortran linker is used in
that particular case at all - the problem occurs when building
_dotblas.so, and some targets after that one. However, the corresponding
code is written in C and has actually been compiled with the C compiler.
The problem did not appear with numpy 1.0.1, where the C compiler is
(correctly) used for linking. So maybe there's another hidden problem...

But anyway, I think that the missing wrapping with ld_so_aix is a bug on
its own.

When moving on to building scipy, I ran into similar problems with C++
compiled code - instead of wrapping the c++ compiler/linker with
ld_so_aix, the following command is executed

   g++ gcc -pthread
-bI:/homespace/grasppf/aix/lib/python2.5/config/python.exp
build/temp.aix-5.1-2.5/Lib/cluster/src/vq_wrap.o
-Lbuild/temp.aix-5.1-2.5 -o build/lib.aix-5.1-2.5/scipy/cluster/_vq.so


and predictably causes an error. What's happening is (I think) that the
numpy distutils partially overwrite the linker modifications from the core
python. (the -pthread -bI:/...python.exp is an argument to the ld_so_aix
script). My problem is that I do not know where in the numpy distutils
code this modification happens, so I've no idea where to try to fix it -
does anyone on this list know?

Many thanks,

  Christian.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: numpy.patch
Type: text/x-patch
Size: 757 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/eae273d5/attachment.bin>

From tim.hochberg at ieee.org  Wed Apr 18 09:59:48 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 18 Apr 2007 06:59:48 -0700
Subject: [Numpy-discussion] Help using numPy to create a very large
	multi dimensional array
In-Reply-To: <699044520704180404p5b27bef9xa0483f9093c47528@mail.gmail.com>
References: <699044520704130350j7b53e36br5c0bc068ec00643c@mail.gmail.com>
	<e06186140704131036t4174905y466a4c0a15a815f1@mail.gmail.com>
	<699044520704170903i26ea8686vcf0c63e0bd4d9cc4@mail.gmail.com>
	<f04cme$k56$1@sea.gmane.org>
	<699044520704180404p5b27bef9xa0483f9093c47528@mail.gmail.com>
Message-ID: <e4412d6b0704180659g7ee42331t58be3ce9f1d296dd@mail.gmail.com>

On 4/18/07, Bruno Santos <bacmsantos at gmail.com> wrote:
>
> Finally I was able to read the data, by using the command you sair with
> some small changes:
> matrix = numpy.array([[float(x) for x in line.split()[1:]] for line in
> vecfile])
>
> But that doesn't solve my speed problem, now instead of taking 40seconds
> in the slow step, takes 1min ant 10seconds :(
> The slow step is this cycle:
> for j in range(0, clust):
>             list_j= numpy.asarray(matrix[j])
>             for k in range(j+1, clust):
>                 list_k=numpy.asarray(matrix[k])
>                 dist=0
>                 for e in range(0, columns):
>                     result = list_j[e] - list_k[e]
>                     dist += result * result
>                 if (dist < min):
>                     ind[0] = j
>                     ind[1] = k
>                     min = dist
>
> I also try with list_j = numpy.array but it only slower even more the
> calculation,
> Does anyone have any ideia how I can speed up this step?


Two things:

Step 0: think about your alogrithm.


Depending on your data, their are probably faster approaches here. One way I
handled a similar problem at one time was to grid up my space into M chunks
and figure out which vector goes where. If your data is chunky, a dict of
lists can work for this. Then you only need to work your clustering magic on
elements in a given chunk and that's chunks neighbors. This reduces the
problem from N^2 to something like (N/M)^2. For more sophistication you
could also look to the fast multipole solver people for inspiration. I don't
know if they do clustering per se, but it seems likely that all their
hiearchial grouping stuff could be adapted for this.

Step 1: vectorize your innner loop.



Well, that's complicated, so you may want to try something simple first. It
looks like you could benefit from vectorizing at least your innermost loop
and maybe the innermost 2. The innermost could probably be rewritten:

dist = np.dot(list_j, list_l)

Also, all that converting back and forth from matrices is silly. Assuming
you need to convert at all (which you probably don't if you are using dot),
convert just once at the beginning and use the matrix version in the code.

Try that and see if it makes any difference. It would end up somethlng like:

array = numpy.asarray(matrix[j])
for j in range(0, clust):
            list_j= array[j]
            for k in range(j+1, clust):
                dist = numpy.dot(list_j - array[k])
                if (dist < min):
                    ind[0] = j
                    ind[1] = k
                    min = dist

Hope that's at least marginally useful. There's enough missing from the
original that it's hard to figure out exactly how this works.



-tim




2007/4/18, Christian K. <ckkart at hoc.net>:
> >
> > Bruno Santos wrote:
> > > I try to use the expression as you said, but I'm not getting the
> > desired
> > > result,
> > > My text file look like this:
> > >
> > > # num rows=115 num columns=2634
> > > AbassiM.txt 0.033023 0.033023 0.033023 0.165115 0.462321....0.000000
> > > AgricoleW.txt 0.038691 0.038691 0.038691 0.232147 0.541676....0.215300
> > > AliR.txt 0.041885 0.041885 0.041885 0.125656 0.586395....0.633580
> > > .....
> > > ....
> > > ....
> > > ZhangJ.txt 0.047189 0.047189 0.047189 0.155048 0.613452....0.000000
> >
> > I guess N.fromfile can't handle non numeric data. Use something like
> > this instead (not tested):
> >
> > import numpy as N
> >
> > data = open('name of file').readlines()
> >
> > data = N.array([[float(x) for x in row.split(' ')[1:]] for row in
> > data[1:]])
> >
> > (the above expression should be one line)
> >
> > Christian
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/4fccddcd/attachment.html>

From mpmusu at cc.usu.edu  Wed Apr 18 10:25:05 2007
From: mpmusu at cc.usu.edu (Mark.Miller)
Date: Wed, 18 Apr 2007 08:25:05 -0600
Subject: [Numpy-discussion] fumfunction question
Message-ID: <46262A41.5010301@cc.usu.edu>

Can additional function arguments (aside from the dimensions of an 
array) be used in conjunction with fromfunction?

Thanks,

-Mark


From lou_boog2000 at yahoo.com  Wed Apr 18 10:34:26 2007
From: lou_boog2000 at yahoo.com (Lou Pecora)
Date: Wed, 18 Apr 2007 07:34:26 -0700 (PDT)
Subject: [Numpy-discussion] Question about Optimization (Inline and
	Pyrex)
In-Reply-To: <ce557a360704172239s73f09309j85951044c53468f3@mail.gmail.com>
Message-ID: <20070418143427.32149.qmail@web34407.mail.mud.yahoo.com>


--- Anne Archibald <peridot.faceted at gmail.com> wrote:

> I just
> took another look at
> that code and added a parallel_map I hadn't got
> around to writing
> before, too. I'd be happy to stick it (and test
> file) on the wiki
> under some open license or other ("do what thou wilt
> shall be the
> whole of the law"?). It's certainly not competition
> for ipython1,
> though, it's mostly to show an example of making
> threads easy to use.
> 
> Anne

Please put the parallel map code on the Wiki.  I found
your first (obvious-parallel) example very helpful.



-- Lou Pecora,   my views are my own.
---------------
Great spirits have always encountered violent opposition from mediocre minds. 
-Albert Einstein

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


From cookedm at physics.mcmaster.ca  Wed Apr 18 11:20:33 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Wed, 18 Apr 2007 11:20:33 -0400
Subject: [Numpy-discussion] Problems building numpy and scipy on AIX
In-Reply-To: <30420.193.17.11.23.1176898744.squirrel@webmail.marquardt.sc>
References: <30420.193.17.11.23.1176898744.squirrel@webmail.marquardt.sc>
Message-ID: <46263741.7080403@physics.mcmaster.ca>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christian Marquardt wrote:
> Hello,
> 
> I've run into a problem building numpy-1.0.2 on AIX (using gcc and native
> Fortran compilers). The problem on that platform in general is that the
> process for building shared libraries is different from what's normally
> done (and it's a pain...)

Already fixed in svn :)

- --
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGJjctN9ixZKFWjRQRAoLZAJ4uz6L/dO1j47nz4o5BEFiFLlc6bwCfayha
tWZCkDzXjNR7lrJK7AVMyTc=
=9it9
-----END PGP SIGNATURE-----


From oliphant.travis at ieee.org  Wed Apr 18 12:03:39 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Wed, 18 Apr 2007 10:03:39 -0600
Subject: [Numpy-discussion] fumfunction question
In-Reply-To: <46262A41.5010301@cc.usu.edu>
References: <46262A41.5010301@cc.usu.edu>
Message-ID: <4626415B.8020505@ieee.org>

Mark.Miller wrote:
> Can additional function arguments (aside from the dimensions of an 
> array) be used in conjunction with fromfunction?
>   

Yes --- as long as they are keyword arguments.

Keyword arguments can be passed in and they will be handed over to the 
function as keyword arguments.

-Travis



From Chris.Barker at noaa.gov  Wed Apr 18 12:11:32 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 18 Apr 2007 09:11:32 -0700
Subject: [Numpy-discussion] Help using numPy to create a very
 large	multi dimensional array
In-Reply-To: <699044520704180404p5b27bef9xa0483f9093c47528@mail.gmail.com>
References: <699044520704130350j7b53e36br5c0bc068ec00643c@mail.gmail.com>
	<e06186140704131036t4174905y466a4c0a15a815f1@mail.gmail.com>
	<699044520704170903i26ea8686vcf0c63e0bd4d9cc4@mail.gmail.com>
	<f04cme$k56$1@sea.gmane.org>
	<699044520704180404p5b27bef9xa0483f9093c47528@mail.gmail.com>
Message-ID: <46264334.8080304@noaa.gov>

Bruno Santos wrote:
> Finally I was able to read the data, by using the command you sair with 
> some small changes:
> matrix = numpy.array([[float(x) for x in line.split()[1:]] for line in 
> vecfile])

it doesn't sound like you're concerned about the speed of reading the 
files, but you can still use fromfile() or maybe fromstring() to do 
this. You just need to read past the text part first, then process it.

using fromstring:

matrix = numpy.vstack([numpy.fromstring(line.split(" ", 1)[1], sep=" ") 
for line in vecfile])

or something like that.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From kwgoodman at gmail.com  Wed Apr 18 12:13:15 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 18 Apr 2007 09:13:15 -0700
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <4625879C.6000706@ar.media.kyoto-u.ac.jp>
References: <evpm71$q6c$1@sea.gmane.org> <462073A6.4010602@gmail.com>
	<evqnir$44m$1@sea.gmane.org> <4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<f01nug$nns$1@sea.gmane.org> <4624E684.9010606@astraw.com>
	<4625879C.6000706@ar.media.kyoto-u.ac.jp>
Message-ID: <f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>

On 4/17/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> Now that you mention it, I am also puzzled by this one: I can see why
> you would use atlas3-sse2-dev without atlas3-base-dev (for the static
> library), but not having atlas3-base-dev makes it imposible to
> dynamically link to atlas libraries without customizing makefiles and/or
> configure options.

I'd like to compile atlas so that I can take full advantage of my core
2 duo. Numpy dynamically links to the debian binary of atlas-sse that
I installed. But the atlas website says that they recommend static
linking.

Which do you recommend, static or dynamic? Are there good directions for either?


From haase at msg.ucsf.edu  Wed Apr 18 12:48:29 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed, 18 Apr 2007 09:48:29 -0700
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <ce557a360704172233v7790b6cfvac6ec8477ccbb4e1@mail.gmail.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>
	<4625261D.2070303@gemini.edu>
	<e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>
	<4625317A.10408@gmail.com>
	<bc657ead0704172141j1bfa6312s2c4f6df721b11d28@mail.gmail.com>
	<4625A1FD.1050001@gmail.com>
	<ce557a360704172233v7790b6cfvac6ec8477ccbb4e1@mail.gmail.com>
Message-ID: <bc657ead0704180948v452339avbf6e7ea07ba1e0bc@mail.gmail.com>

On 4/17/07, Anne Archibald <peridot.faceted at gmail.com> wrote:
> On 18/04/07, Robert Kern <robert.kern at gmail.com> wrote:
> > Sebastian Haase wrote:
> >
> > > Hi,
> > > I don't know much about ATLAS -- would there be other numpy functions
> > > that *could*  or *should*  be implemented using ATLAS !?
> > > Any ?
> >
> > Not really, no.
>
> ATLAS is a library designed to implement linear algebra functions
> efficiently on many machines. It does things like reorder the
> multiplications and additions in matrix multiplication to make the
> best possible use of your cache, as measured by empirical testing.

So, this means that 'matrixmultiply'  could / should be using ATLAS
for the same reason as 'dot' does - right ?

-Seb.

> (FFTW does something similar for the FFT.) But ATLAS is only designed
> for linear algebra. If what you want to do is linear algebra, look at
> scipy for a full selection of linear algebra routines that make fairly
> good use of ATLAS where applicable.
>
> It would be perfectly possible, in principle, to implement an
> ATLAS-like library that handled a variety (perhaps all) of numpy's
> basic operations in platform-optimized fashion. But implementing ATLAS
> is not a simple process! And it's not clear how much gain would be
> available - it would almost certainly be noticeably faster only for
> very large numpy objects (where the python overhead is unimportant),
> and those objects can be very inefficient because of excessive
> copying. And the scope of improvement would be very limited; an
> expression like A*B+C*D would be much more efficient, probably, if the
> whole expression were evaluated at once for each element (due to
> memory locality and temporary allocation). But it is impossible for
> numpy, sitting inside python as it does, to do that.


From spacey-numpy-discussion at lenin.net  Wed Apr 18 12:48:44 2007
From: spacey-numpy-discussion at lenin.net (Peter C. Norton)
Date: Wed, 18 Apr 2007 09:48:44 -0700
Subject: [Numpy-discussion] Building numpy on Solaris x86 with sun CC and
	libsunperf
Message-ID: <20070418164844.GG5812@lenin.net>

Hello all,

I'm trying to build numpy for some of my users, and I can't seem to
get the [blas_opt] or the [lapack_opt] settings to be honored in my
site.cfg:

$ CFLAGS="-L$STUDIODIR/lib/ -l=sunperf" CPPFLAGS='-DNO_APPEND_FORTRAN' \
/scratch/nortonp/python-2.5.1c1/bin/python setup.py config
Running from numpy source directory.
F2PY Version 2_3649
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in /usr/local/lib
  libraries mkl,vml,guide not found in /lang/SunOS.5.i386/studio-11.0/SUNWspro/lib
  NOT AVAILABLE
[etc, with nothing found]

And after all this, I get
/projects/python-2.5/numpy-1.0.2/numpy/distutils/system_info.py:1210:
UserWarning:
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)

and a similar thing happens with lapack. My site.cfg boils down to
this:

[DEFAULT]
library_dirs = /usr/local/lib:/lang/SunOS.5.i386/studio-11.0/SUNWspro/lib
include_dirs = /usr/local/include:/lang/SunOS.5.i386/studio-11.0/SUNWspro/include

[blas_opt]
libraries = sunperf

[lapack_opt]
libraries = sunperf

If I mess around with system_info.py I can get setup to acknowledge
the addition to the list, but it seems from the output that the
optimized libraries section in the site.cfg is ignored (eg. never
added to the classes _lib_names array).

Is there a known way to get around this?

Also, since the lapack and blas libraries are already essentially part
of libsunperf, built, do I still need a fortran compiler to link build
numpy or can I just bypass that (somehow) and link the .so and go on
my merry way?

Thanks,

-Peter

-- 
The 5 year plan:
In five years we'll make up another plan.
Or just re-use this one.



From charlesr.harris at gmail.com  Wed Apr 18 13:00:15 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 18 Apr 2007 11:00:15 -0600
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
References: <evpm71$q6c$1@sea.gmane.org> <462073A6.4010602@gmail.com>
	<evqnir$44m$1@sea.gmane.org> <4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<f01nug$nns$1@sea.gmane.org> <4624E684.9010606@astraw.com>
	<4625879C.6000706@ar.media.kyoto-u.ac.jp>
	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
Message-ID: <e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>

On 4/18/07, Keith Goodman <kwgoodman at gmail.com> wrote:
>
> On 4/17/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> > Now that you mention it, I am also puzzled by this one: I can see why
> > you would use atlas3-sse2-dev without atlas3-base-dev (for the static
> > library), but not having atlas3-base-dev makes it imposible to
> > dynamically link to atlas libraries without customizing makefiles and/or
> > configure options.
>
> I'd like to compile atlas so that I can take full advantage of my core
> 2 duo. Numpy dynamically links to the debian binary of atlas-sse that
> I installed. But the atlas website says that they recommend static
> linking.
>
> Which do you recommend, static or dynamic? Are there good directions for
> either?


I don't know which is best, although I suspect the statically linked version
will be larger. It might seem that just pulling in the gemm routines
wouldn't add much, but they pull in lots of supporting routines. To get
numpy to link statically you will also probably need to have a directory
that contains only the *.a versions because the linker will default to the
*.so if they are present; i don't think there is a way to specify the
-static flag to the gcc compiler. Maybe someone else knows how to do that.
For ATLAS, I believe the latest versions are also recommended because the
stable version is so old.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/268dfb85/attachment.html>

From kwgoodman at gmail.com  Wed Apr 18 13:17:34 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 18 Apr 2007 10:17:34 -0700
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>
References: <evpm71$q6c$1@sea.gmane.org> <462073A6.4010602@gmail.com>
	<evqnir$44m$1@sea.gmane.org> <4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<f01nug$nns$1@sea.gmane.org> <4624E684.9010606@astraw.com>
	<4625879C.6000706@ar.media.kyoto-u.ac.jp>
	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
	<e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>
Message-ID: <f4f93d420704181017n24670876laa32b1fd18858624@mail.gmail.com>

On 4/18/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
> On 4/18/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> > I'd like to compile atlas so that I can take full advantage of my core
> > 2 duo. Numpy dynamically links to the debian binary of atlas-sse that
> > I installed. But the atlas website says that they recommend static
> > linking.
> >
> > Which do you recommend, static or dynamic? Are there good directions for
> either?
>
> I don't know which is best, although I suspect the statically linked version
> will be larger. It might seem that just pulling in the gemm routines
> wouldn't add much, but they pull in lots of supporting routines. To get
> numpy to link statically you will also probably need to have a directory
> that contains only the *.a versions because the linker will default to the
> *.so if they are present; i don't think there is a way to specify the
> -static flag to the gcc compiler. Maybe someone else knows how to do that.
> For ATLAS, I believe the latest versions are also recommended because the
> stable version is so old.

At the moment best is equal to easiest since I have never compiled
atlas. Does anyone know of a howto on compiling atlas (dynamically
linked)?

Besides speed I'm also interested in seeing if I can get rid of the
repeatability problems I have with the debian atlas-sse2 binary.
(Repeated calulations, as discuss on this list, give give difference
results in numpy but not octave.)


From robert.kern at gmail.com  Wed Apr 18 13:19:32 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Apr 2007 12:19:32 -0500
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <bc657ead0704180948v452339avbf6e7ea07ba1e0bc@mail.gmail.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>	<4625261D.2070303@gemini.edu>	<e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>	<4625317A.10408@gmail.com>	<bc657ead0704172141j1bfa6312s2c4f6df721b11d28@mail.gmail.com>	<4625A1FD.1050001@gmail.com>	<ce557a360704172233v7790b6cfvac6ec8477ccbb4e1@mail.gmail.com>
	<bc657ead0704180948v452339avbf6e7ea07ba1e0bc@mail.gmail.com>
Message-ID: <46265324.3060506@gmail.com>

Sebastian Haase wrote:
> On 4/17/07, Anne Archibald <peridot.faceted at gmail.com> wrote:
>> On 18/04/07, Robert Kern <robert.kern at gmail.com> wrote:
>>> Sebastian Haase wrote:
>>>
>>>> Hi,
>>>> I don't know much about ATLAS -- would there be other numpy functions
>>>> that *could*  or *should*  be implemented using ATLAS !?
>>>> Any ?
>>> Not really, no.
>> ATLAS is a library designed to implement linear algebra functions
>> efficiently on many machines. It does things like reorder the
>> multiplications and additions in matrix multiplication to make the
>> best possible use of your cache, as measured by empirical testing.
> 
> So, this means that 'matrixmultiply'  could / should be using ATLAS
> for the same reason as 'dot' does - right ?

matrixmultiply() is just a long-deprecated alias to dot().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From christian at marquardt.sc  Wed Apr 18 13:26:20 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Wed, 18 Apr 2007 19:26:20 +0200 (CEST)
Subject: [Numpy-discussion] Problems building numpy and scipy on AIX
In-Reply-To: <46263741.7080403@physics.mcmaster.ca>
References: <30420.193.17.11.23.1176898744.squirrel@webmail.marquardt.sc>
	<46263741.7080403@physics.mcmaster.ca>
Message-ID: <34004.193.17.11.23.1176917180.squirrel@webmail.marquardt.sc>

Dear David,

the svn version of numpy does indeed build cleanly on AIX. Many thanks!

However, the wrapper problem still exists for the C++ compiler, and shows
up when compiling scipy. Now, I *assume* that SciPy is using the distutils
as installed by numpy. Do you know where the linker settings for the C++
compiler might be overwritten? There are two or three C compiler related
python modules in numpy/distutils... Or would you think that this problem
is entirely unrelated to the distutils in numpy?

Many thanks,

  Christian.



On Wed, April 18, 2007 17:20, David M. Cooke wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Christian Marquardt wrote:
>> Hello,
>>
>> I've run into a problem building numpy-1.0.2 on AIX (using gcc and
>> native
>> Fortran compilers). The problem on that platform in general is that the
>> process for building shared libraries is different from what's normally
>> done (and it's a pain...)
>
> Already fixed in svn :)
>
> - --
> |>|\/|<
> /------------------------------------------------------------------\
> |David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
> |cookedm at physics.mcmaster.ca
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.7 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFGJjctN9ixZKFWjRQRAoLZAJ4uz6L/dO1j47nz4o5BEFiFLlc6bwCfayha
> tWZCkDzXjNR7lrJK7AVMyTc=
> =9it9
> -----END PGP SIGNATURE-----
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>




From haase at msg.ucsf.edu  Wed Apr 18 13:40:47 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed, 18 Apr 2007 10:40:47 -0700
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <46265324.3060506@gmail.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>
	<4625261D.2070303@gemini.edu>
	<e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>
	<4625317A.10408@gmail.com>
	<bc657ead0704172141j1bfa6312s2c4f6df721b11d28@mail.gmail.com>
	<4625A1FD.1050001@gmail.com>
	<ce557a360704172233v7790b6cfvac6ec8477ccbb4e1@mail.gmail.com>
	<bc657ead0704180948v452339avbf6e7ea07ba1e0bc@mail.gmail.com>
	<46265324.3060506@gmail.com>
Message-ID: <bc657ead0704181040ya847fe5j2c98d9adfa46bc62@mail.gmail.com>

On 4/18/07, Robert Kern <robert.kern at gmail.com> wrote:
> Sebastian Haase wrote:
> > On 4/17/07, Anne Archibald <peridot.faceted at gmail.com> wrote:
> >> On 18/04/07, Robert Kern <robert.kern at gmail.com> wrote:
> >>> Sebastian Haase wrote:
> >>>
> >>>> Hi,
> >>>> I don't know much about ATLAS -- would there be other numpy functions
> >>>> that *could*  or *should*  be implemented using ATLAS !?
> >>>> Any ?
> >>> Not really, no.
> >> ATLAS is a library designed to implement linear algebra functions
> >> efficiently on many machines. It does things like reorder the
> >> multiplications and additions in matrix multiplication to make the
> >> best possible use of your cache, as measured by empirical testing.
> >
> > So, this means that 'matrixmultiply'  could / should be using ATLAS
> > for the same reason as 'dot' does - right ?
>
> matrixmultiply() is just a long-deprecated alias to dot().
Of course - I should have turn my brain on before hitting 'send'....
Does ATLAS/BLAS do anything special for element wise multiplication
and alike - if for example the data is not aligned or not contiguous?

-Seb.


From sturla at molden.no  Wed Apr 18 14:14:10 2007
From: sturla at molden.no (Sturla Molden)
Date: Wed, 18 Apr 2007 20:14:10 +0200
Subject: [Numpy-discussion] Efficient operator overloading
Message-ID: <46265FF2.2090004@molden.no>


On 4/18/2007 7:33 AM, Anne Archibald wrote:

 >copying. And the scope of improvement would be very limited; an
expression like A*B+C*D would be much more efficient, probably, if the
whole expression were evaluated at once for each element (due to
memory locality and temporary allocation). But it is impossible for
numpy, sitting inside python as it does, to do that.



Most numerical array/matrix libraries dependent on operator overloading 
generates temporaries. That is why Fortran is usually perceived as 
superior to C++ for scientific programming. The Fortran compiler knows 
about arrays and can avoid allocating three temporary arrays to evaluate 
and expression like

y = a * b + c * d

If this expression is evaluated bya Fortran 90/95 compiler, it will 
automatically generate code like

do i = 1,n
    y(i) = a(i) * b(i) + c(i) * d(i)
enddo

On the other hand, conventional use of overloaded operators would result 
in something like this:

allocate(tmp1,n)
do i = 1,n
    tmp1(i) = a(i) * b(i)
enddo
allocate(tmp2,n)
do i = 1,n
    tmp2(i) = c(i) * d(i)
enddo
allocate(tmp3,n)
do i = 1,n
    tmp3(i) = tmp1(i) + tmp2(i)
enddo
deallocate(tmp1)
deallocate(tmp2)
do i = 1,n
    y(i) = tmp3(i)
enddo
deallocate(tmp3)

Traversing memory is one of the most expensive thing a CPU can do. This 
approach is therefore extremely inefficient compared with what a Fortran 
compiler can do.

This does not mean that all use of operator overloading is inherently 
bad. Notably, there is a C++ numerical library called Blitz++ which can 
avoid  these tempraries for small fixed-size arrays. As it depends on 
template metaprogramming, the size must be known at compile time. But if 
this is the case, it can be just as efficient as Fortran if the 
optimizer is smart enough to remove the redundant operations. Most 
modern C++ compilers is smart enough to do this. Note that it only works 
for fixed size arrays. Fortran compilers can do this on a more general 
basis. It is therefore advisable to have array syntax built into the 
language syntax it self, as in Fortran 90/95 and Ada 95.

But if we implement the operator overloading a bit more intelligent, it 
should be possible to get rid of most of the temporary arrays. We could 
replace temporary arrays with an "unevaluated expression class" and let 
  the library pretend it is a compiler.

Let us assume again we have an expression like

y = a * b + c * d

where a,b,c and d are all arrays or matrices. In this case, the 
overloaded * and + operators woud not return a temporary array but an 
unevaluated expression of class Expr. Thus we would get

tmp1 = Expr('__mul__',a,b) # symbolic representation of 'a * b'
tmp2 = Expr('__mul__',c,d) # symbolic representation of 'c * d'
tmp3 = Expr('__add__',tmp1,tmp1) # symbolic "a * b + c * d"
del tmp1
del tmp2
y = tmp3 # y becomes a reference to an unevaluated expression

Finally, we need a mechanism to 'flush' the unevaluated expression. 
Python does not allow the assignment operator to be evaluated, so one 
could not depend on that. But one could a 'flush on read/write' 
mechanism, and let an Expr object exist in different finite states (e.g. 
unevaluated and evaluated). If anyone tries to read an element from y or 
change any of the objects it involves, the expression gets evaluated 
without temporaries. Before that, there is no need to evalute the 
expression at all! We can just keep a symbolic representation of it. 
Procrastination is good!

Thus later on...

x = y[i] # The expression 'a * b + c * d' gets evaluated. The
          # object referred to by y now holds an actual array.

or

a[i] = 2.0 # The expression 'a * b + c * d' gets evaluated. The
            # object referred to by y now holds an actual array.
            # Finally, 2.0 is written to a[i].

or

y[i] = 2.0 # The expression 'a * b + c * d' gets evaluated. The
            # object referred to by y now holds an actual array.
            # Finally, 2.0 is written to y[i].

When the expression 'a * b + c * d' is finally evaluated, we should 
through symbolic manipulation get something (at least close to) a single 
efficient loop:

do i = 1,n
    y(i) = a(i) * b(i) + c(i) * d(i)
enddo



I'd really like to see a Python extension library do this one day. It 
would be very cool and (almost) as efficient as plain Fortran - though 
not quite, we would still get some small temporary objects created. But 
that is a sacrifice I am willing to pay to use Python. We would gain 
some efficacy over Fortran by postponing indefinitely evaluation of 
computations that are not needed, when this is not known at compile 
time.


Any comments?


Sturla Molden
Ph.D.






From matthieu.brucher at gmail.com  Wed Apr 18 14:24:42 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 18 Apr 2007 20:24:42 +0200
Subject: [Numpy-discussion] Efficient operator overloading
In-Reply-To: <46265FF2.2090004@molden.no>
References: <46265FF2.2090004@molden.no>
Message-ID: <e76aa17f0704181124x4efc2adbp9aac562f98d20bf7@mail.gmail.com>

>
> This does not mean that all use of operator overloading is inherently
> bad. Notably, there is a C++ numerical library called Blitz++ which can
> avoid  these tempraries for small fixed-size arrays. As it depends on
> template metaprogramming, the size must be known at compile time. But if
> this is the case, it can be just as efficient as Fortran if the
> optimizer is smart enough to remove the redundant operations. Most
> modern C++ compilers is smart enough to do this. Note that it only works
> for fixed size arrays. Fortran compilers can do this on a more general
> basis. It is therefore advisable to have array syntax built into the
> language syntax it self, as in Fortran 90/95 and Ada 95.


Boost.ublas uses template expressions, I implemented them in my own matrix
lib, it's very simple to do. Even for big matrix, it can be interesting. If
the size is static, bothering with fixed-size arrays is useless, the
compilers do optimize the temporaries ; but using template expression with
iterators for general arrays allows to approach fixed-size arrays speed.

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/4f97a5a3/attachment.html>

From tim.hochberg at ieee.org  Wed Apr 18 14:31:06 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 18 Apr 2007 11:31:06 -0700
Subject: [Numpy-discussion] Efficient operator overloading
In-Reply-To: <46265FF2.2090004@molden.no>
References: <46265FF2.2090004@molden.no>
Message-ID: <e4412d6b0704181131v3b0897a5t56c6eafd15c76658@mail.gmail.com>

On 4/18/07, Sturla Molden <sturla at molden.no> wrote:
>
>
> On 4/18/2007 7:33 AM, Anne Archibald wrote:
>
> >copying. And the scope of improvement would be very limited; an
> expression like A*B+C*D would be much more efficient, probably, if the
> whole expression were evaluated at once for each element (due to
> memory locality and temporary allocation). But it is impossible for
> numpy, sitting inside python as it does, to do that.
>
>
>
> Most numerical array/matrix libraries dependent on operator overloading
> generates temporaries. That is why Fortran is usually perceived as
> superior to C++ for scientific programming. The Fortran compiler knows
> about arrays and can avoid allocating three temporary arrays to evaluate
> and expression like
>
> y = a * b + c * d
> [SNIP]
>
>
>
> I'd really like to see a Python extension library do this one day. It
> would be very cool and (almost) as efficient as plain Fortran - though
> not quite, we would still get some small temporary objects created. But
> that is a sacrifice I am willing to pay to use Python. We would gain
> some efficacy over Fortran by postponing indefinitely evaluation of
> computations that are not needed, when this is not known at compile
> time.
>
>
> Any comments?


I periodically wonder if something like this could be built on top of
numexpr without too much pain. Simply build up the expression on operations
and then evaluate when data is requested. You'd need some sort of Jekyl &
Hyde matrix that spent the first part of it's life unevaluated and then
converted itself to something very much like an array when it evaluated
itself. If might end up being tricky to make sure that everything got
transparently evaluated at the right time though.

I've never had time to try it though.



Sturla Molden
> Ph.D.
>
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/a93ed4d8/attachment.html>

From mpmusu at cc.usu.edu  Wed Apr 18 14:32:12 2007
From: mpmusu at cc.usu.edu (Mark.Miller)
Date: Wed, 18 Apr 2007 12:32:12 -0600
Subject: [Numpy-discussion] fumfunction question
In-Reply-To: <4626415B.8020505@ieee.org>
References: <46262A41.5010301@cc.usu.edu> <4626415B.8020505@ieee.org>
Message-ID: <4626642C.9020406@cc.usu.edu>

OK...I think I understand.  But just to clarify:

Suppose I have a function that can take 3 parameters

def f(x,y,constant):
    operations

Is it appropriate to use something like

a=numpy.fromfunction(f,(x,y),const)

where x and y give the array dimensions and const is another variable 
that is used to perform operations within f.

I know that it doesn't work, but I just wanted to check to see if there 
is possibly a syntax that can work.

Thanks,

-Mark

Travis Oliphant wrote:
> Mark.Miller wrote:
>> Can additional function arguments (aside from the dimensions of an 
>> array) be used in conjunction with fromfunction?
>>   
> 
> Yes --- as long as they are keyword arguments.
> 
> Keyword arguments can be passed in and they will be handed over to the 
> function as keyword arguments.
> 
> -Travis
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From charlesr.harris at gmail.com  Wed Apr 18 14:35:38 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 18 Apr 2007 12:35:38 -0600
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <f4f93d420704181017n24670876laa32b1fd18858624@mail.gmail.com>
References: <evpm71$q6c$1@sea.gmane.org> <462073A6.4010602@gmail.com>
	<evqnir$44m$1@sea.gmane.org> <4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<f01nug$nns$1@sea.gmane.org> <4624E684.9010606@astraw.com>
	<4625879C.6000706@ar.media.kyoto-u.ac.jp>
	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
	<e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>
	<f4f93d420704181017n24670876laa32b1fd18858624@mail.gmail.com>
Message-ID: <e06186140704181135w469cf329jaecbdc047d96fcec@mail.gmail.com>

On 4/18/07, Keith Goodman <kwgoodman at gmail.com> wrote:
>
> On 4/18/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
> > On 4/18/07, Keith Goodman <kwgoodman at gmail.com> wrote:
> > > I'd like to compile atlas so that I can take full advantage of my core
> > > 2 duo. Numpy dynamically links to the debian binary of atlas-sse that
> > > I installed. But the atlas website says that they recommend static
> > > linking.
> > >
> > > Which do you recommend, static or dynamic? Are there good directions
> for
> > either?
> >
> > I don't know which is best, although I suspect the statically linked
> version
> > will be larger. It might seem that just pulling in the gemm routines
> > wouldn't add much, but they pull in lots of supporting routines. To get
> > numpy to link statically you will also probably need to have a directory
> > that contains only the *.a versions because the linker will default to
> the
> > *.so if they are present; i don't think there is a way to specify the
> > -static flag to the gcc compiler. Maybe someone else knows how to do
> that.
> > For ATLAS, I believe the latest versions are also recommended because
> the
> > stable version is so old.
>
> At the moment best is equal to easiest since I have never compiled
> atlas. Does anyone know of a howto on compiling atlas (dynamically
> linked)?


The instructions that come with ATLAS are useable, and with the
3.7.30version the process is pretty easy. Just make sure you read the
part
concerning relocatable libraries first, as you will need to add some flags
to the ./configure command line, -fPIC  and a couple of others. After
compiling and install, IIRC, the libraries will be in /usr/local/lib/ATLAS,
which should be OK for debian. Then you need to add a file in /etc/ld.conf.d
with the name of the directory and run ldconfig in order to let the system
know where the libraries are and what they contain. Or you could save the
old /usr/lib/atlas directory and put the new libraries there and then run
ldconfig. I also found it necessary to delete the old numpy site-package
before reinstalling numpy.

I am not the expert here, so perhaps someone else will weigh in.

Besides speed I'm also interested in seeing if I can get rid of the
> repeatability problems I have with the debian atlas-sse2 binary.
> (Repeated calulations, as discuss on this list, give give difference
> results in numpy but not octave.)


I suspect this has something to do with resetting floating point flags, so
this might not get fixed.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/87cb2183/attachment.html>

From oliphant at ee.byu.edu  Wed Apr 18 14:52:03 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 18 Apr 2007 12:52:03 -0600
Subject: [Numpy-discussion] fumfunction question
In-Reply-To: <4626642C.9020406@cc.usu.edu>
References: <46262A41.5010301@cc.usu.edu> <4626415B.8020505@ieee.org>
	<4626642C.9020406@cc.usu.edu>
Message-ID: <462668D3.3040909@ee.byu.edu>

Mark.Miller wrote:

>OK...I think I understand.  But just to clarify:
>
>Suppose I have a function that can take 3 parameters
>
>def f(x,y,constant):
>    operations
>
>Is it appropriate to use something like
>
>a=numpy.fromfunction(f,(x,y),const)
>
>where x and y give the array dimensions and const is another variable 
>that is used to perform operations within f.
>
>I know that it doesn't work, but I just wanted to check to see if there 
>is possibly a syntax that can work.
>  
>

No, you can only pass extra keyword arguments on to the function.  Thus,

def f(x,y,myval=3):
      operations

could be called as

a = numpy.fromfunction(f, (10,20), myval=4)

-Travis



From mpmusu at cc.usu.edu  Wed Apr 18 14:54:15 2007
From: mpmusu at cc.usu.edu (Mark.Miller)
Date: Wed, 18 Apr 2007 12:54:15 -0600
Subject: [Numpy-discussion] fumfunction question
In-Reply-To: <462668D3.3040909@ee.byu.edu>
References: <46262A41.5010301@cc.usu.edu> <4626415B.8020505@ieee.org>
	<4626642C.9020406@cc.usu.edu> <462668D3.3040909@ee.byu.edu>
Message-ID: <46266957.50001@cc.usu.edu>

Got it...thanks!

Travis Oliphant wrote:
> Mark.Miller wrote:
> 
>> OK...I think I understand.  But just to clarify:
>>
>> Suppose I have a function that can take 3 parameters
>>
>> def f(x,y,constant):
>>    operations
>>
>> Is it appropriate to use something like
>>
>> a=numpy.fromfunction(f,(x,y),const)
>>
>> where x and y give the array dimensions and const is another variable 
>> that is used to perform operations within f.
>>
>> I know that it doesn't work, but I just wanted to check to see if there 
>> is possibly a syntax that can work.
>>  
>>
> 
> No, you can only pass extra keyword arguments on to the function.  Thus,
> 
> def f(x,y,myval=3):
>       operations
> 
> could be called as
> 
> a = numpy.fromfunction(f, (10,20), myval=4)
> 
> -Travis
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

-- 
Dr. Mark P. Miller
Department of Biology
5305 Old Main Hill
Utah State University
Logan, UT 84322-5305 USA
 ><><><><><><><><><><><><><><><><
http://www.biology.usu.edu/people/facultyinfo.asp?username=mpmbio
http://www.marksgeneticsoftware.net


From robert.kern at gmail.com  Wed Apr 18 14:57:23 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Apr 2007 13:57:23 -0500
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>
References: <evpm71$q6c$1@sea.gmane.org>
	<462073A6.4010602@gmail.com>	<evqnir$44m$1@sea.gmane.org>
	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>	<f01nug$nns$1@sea.gmane.org>
	<4624E684.9010606@astraw.com>	<4625879C.6000706@ar.media.kyoto-u.ac.jp>	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
	<e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>
Message-ID: <46266A13.5010101@gmail.com>

Charles R Harris wrote:

> I don't know which is best, although I suspect the statically linked
> version will be larger. It might seem that just pulling in the gemm
> routines wouldn't add much, but they pull in lots of supporting
> routines. To get numpy to link statically you will also probably need to
> have a directory that contains only the *.a versions because the linker
> will default to the *.so if they are present; i don't think there is a
> way to specify the -static flag to the gcc compiler. Maybe someone else
> knows how to do that.

Not really. Since you are *making* a shared library, the flags are already set
to *look up* shared libraries. I do not believe they are overridable, much to my
frequent vexation.

> For ATLAS, I believe the latest versions are also
> recommended because the stable version is so old.

And definitely if you have a Core 2 Duo. The optimizations for that processor
have only been incorporated in recent versions.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Wed Apr 18 14:59:44 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Apr 2007 13:59:44 -0500
Subject: [Numpy-discussion] Question about Optimization (Inline,
	and Pyrex)
In-Reply-To: <bc657ead0704181040ya847fe5j2c98d9adfa46bc62@mail.gmail.com>
References: <ce557a360704171155q43b9a45fp3cc77093dfe97475@mail.gmail.com>	<4625261D.2070303@gemini.edu>	<e76aa17f0704171303j687d75bbo40bf060711e1eaa7@mail.gmail.com>	<4625317A.10408@gmail.com>	<bc657ead0704172141j1bfa6312s2c4f6df721b11d28@mail.gmail.com>	<4625A1FD.1050001@gmail.com>	<ce557a360704172233v7790b6cfvac6ec8477ccbb4e1@mail.gmail.com>	<bc657ead0704180948v452339avbf6e7ea07ba1e0bc@mail.gmail.com>	<46265324.3060506@gmail.com>
	<bc657ead0704181040ya847fe5j2c98d9adfa46bc62@mail.gmail.com>
Message-ID: <46266AA0.2080209@gmail.com>

Sebastian Haase wrote:

> Does ATLAS/BLAS do anything special for element wise multiplication
> and alike - if for example the data is not aligned or not contiguous?

Nothing that ATLAS optimizes, no. They focus (rightly) on the more complicated
matrix operations (BLAS Level 3, if you are familiar with the BLAS levels).

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From rex at nosyntax.com  Wed Apr 18 15:36:32 2007
From: rex at nosyntax.com (rex)
Date: Wed, 18 Apr 2007 12:36:32 -0700
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
References: <evpm71$q6c$1@sea.gmane.org> <462073A6.4010602@gmail.com>
	<evqnir$44m$1@sea.gmane.org>
	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<f01nug$nns$1@sea.gmane.org> <4624E684.9010606@astraw.com>
	<4625879C.6000706@ar.media.kyoto-u.ac.jp>
	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
Message-ID: <20070418193632.GU8487@x2.nosyntax.com>

Keith Goodman <kwgoodman at gmail.com> [2007-04-18 10:49]:
> I'd like to compile atlas so that I can take full advantage of my core
> 2 duo. 

If your use is entirely non-commercial you can use Intel's MKL with
built-in optimized BLAS and LAPACK and avoid the need for ATLAS.

http://www.intel.com/cd/software/products/asmo-na/eng/266858.htm

The BLAS and LAPACK libraries are time-honored standards for solving a
large variety of linear algebra problems. The Intel? Math Kernel Library
(Intel? MKL) contains an implementation of BLAS and LAPACK that is
highly optimized for Intel? processors. Intel MKL can enable you to
achieve significant performance improvements over alternative
implementations of BLAS and LAPACK.

[...]

The charts immediately below show that, for Itanium? 2-based systems,
Intel MKL performs approximately 20 percent faster than ATLAS for large
matrices, and even faster for small matrices. On the new Dual-Core
Intel? Xeon? processors, Intel MKL provides similar performance
advantages.


I've compiled both Python (icc) and Numpy using icc 9.1 and MKL
9.1_beta. It's significantly faster than using gcc on my Core 2 Duo
system. I'm still looking for a broad performance test (something like
Scimark, say).

The best compiler flags I've found are: -fast -parallel 
In some cases -funroll-loops and -fno-alias helps.

-rex
-- 
Time flies like wind. Fruit flies like pears.


From kwgoodman at gmail.com  Wed Apr 18 15:45:19 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 18 Apr 2007 12:45:19 -0700
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <20070418193632.GU8487@x2.nosyntax.com>
References: <evpm71$q6c$1@sea.gmane.org> <462073A6.4010602@gmail.com>
	<evqnir$44m$1@sea.gmane.org> <4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<f01nug$nns$1@sea.gmane.org> <4624E684.9010606@astraw.com>
	<4625879C.6000706@ar.media.kyoto-u.ac.jp>
	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
	<20070418193632.GU8487@x2.nosyntax.com>
Message-ID: <f4f93d420704181245y51a4edc9i331f15f483fbf611@mail.gmail.com>

On 4/18/07, rex <rex at nosyntax.com> wrote:
> Keith Goodman <kwgoodman at gmail.com> [2007-04-18 10:49]:
> > I'd like to compile atlas so that I can take full advantage of my core
> > 2 duo.
>
> If your use is entirely non-commercial you can use Intel's MKL with
> built-in optimized BLAS and LAPACK and avoid the need for ATLAS.

Thanks for that. For a variety of reasons I'm sticking with atlas.
Does the parallel flag give you a big speed increase? I imagine it
speeds things up more for larger matrices.


From strawman at astraw.com  Wed Apr 18 16:03:01 2007
From: strawman at astraw.com (Andrew Straw)
Date: Wed, 18 Apr 2007 13:03:01 -0700
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <20070418193632.GU8487@x2.nosyntax.com>
References: <evpm71$q6c$1@sea.gmane.org>
	<462073A6.4010602@gmail.com>	<evqnir$44m$1@sea.gmane.org>	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>	<f01nug$nns$1@sea.gmane.org>
	<4624E684.9010606@astraw.com>	<4625879C.6000706@ar.media.kyoto-u.ac.jp>	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
	<20070418193632.GU8487@x2.nosyntax.com>
Message-ID: <46267975.9000904@astraw.com>

rex wrote:
> Keith Goodman <kwgoodman at gmail.com> [2007-04-18 10:49]:
>   
>> I'd like to compile atlas so that I can take full advantage of my core
>> 2 duo. 
>>     
>
> If your use is entirely non-commercial you can use Intel's MKL with
> built-in optimized BLAS and LAPACK and avoid the need for ATLAS.
>   
Just to clarify, my understanding is that if you buy a developer's
license, you can also use it for commercial use, including distributing
binaries. (Otherwise it would seem kind of silly for Intel to invest so
much in their performance libraries and compilers.)


From travis at enthought.com  Tue Apr 17 09:02:55 2007
From: travis at enthought.com (Travis Vaught)
Date: Tue, 17 Apr 2007 08:02:55 -0500
Subject: [Numpy-discussion] ANN: SciPy 2007 Conference
Message-ID: <39BD0132-B9E0-4C56-B543-D3B23134B5E9@enthought.com>

Greetings,

The *SciPy 2007 Conference* has been scheduled for mid-August at  
CalTech.

http://www.scipy.org/SciPy2007

Here's the rough schedule:

Tutorials:   August 14-15 (Tuesday and Wednesday)
Conference:  August 16-17 (Thursday and Friday)
Sprints:     August 18    (Saturday)

Exciting things are happening in the Python community, and the SciPy  
2007 Conference is an excellent opportunity to exchange ideas, learn  
techniques, contribute code and affect the direction of scientific  
computing (or just to learn what all the fuss is about).  Last year's  
conference saw a near-doubling of attendance to 138, and we're  
looking forward to continued gains in participation.  We'll be  
announcing the Keynote Speaker and providing a detailed schedule in  
the coming weeks.

Registration:
-------------
Registration is now open.
	
You may register online at https://www.enthought.com/scipy07.  Early  
registration for the conference is $150.00 and includes breakfast and  
lunch Thursday & Friday and a very nice dinner Thursday night.   
Tutorial registration is an additional $75.00.  After July 15, 2007,  
conference registration will increase to $200.00 (tutorial  
registration will remain the same at $75.00).

Call for Presenters
-------------------
If you are interested in presenting at the conference, you may submit  
an abstract in Plain Text, PDF or MS Word formats to  
abstracts at scipy.org -- the deadline for abstract submission is July  
6, 2007.  Papers and/or presentation slides are acceptable and are  
due by August 3, 2007.

Tutorial Sessions
-----------------
Last year's conference saw an overwhelming turnout for our first-ever  
tutorial sessions.  In order to better accommodate the community  
interest in tutorials, we've expanded them to 2 days and are  
providing food (requiring us to charge a modest fee for tutorials  
this year).

A tentative list of topics for tutorials includes:

- Wrapping Code with Python (extension module development)
- Building Rich Scientific Applications with Python
- Using SciPy for Statistical Analysis
- Using SciPy for Signal Processing and Image Processing
- Using Python as a Scientific IDE/Workbench
- Others...

This is a preliminary list; topics will change and be extended.

If you'd like to present a tutorial, or are interested in a  
particular topic for a tutorial, please email the SciPy users mailing  
list (link below).  A current list will be maintained here:

http://www.scipy.org/SciPy2007/Tutorials

Coding Sprints
--------------
We've dedicated the Saturday after the conference for a Coding  
Sprint.  Please include any ideas for Sprint topics on the Sprints  
wiki page here: http://www.scipy.org/SciPy2007/Sprints

We're looking forward to another great conference!

Best,

Travis

-------------
Links to various SciPy and NumPy mailing lists may be found here:   
http://www.scipy.org/Mailing_Lists


From gnata at obs.univ-lyon1.fr  Wed Apr 18 16:13:52 2007
From: gnata at obs.univ-lyon1.fr (Xavier Gnata)
Date: Wed, 18 Apr 2007 22:13:52 +0200
Subject: [Numpy-discussion] Efficient operator overloading
In-Reply-To: <e4412d6b0704181131v3b0897a5t56c6eafd15c76658@mail.gmail.com>
References: <46265FF2.2090004@molden.no>
	<e4412d6b0704181131v3b0897a5t56c6eafd15c76658@mail.gmail.com>
Message-ID: <46267C00.3050803@obs.univ-lyon1.fr>

Timothy Hochberg wrote:
>
>
> On 4/18/07, *Sturla Molden* <sturla at molden.no 
> <mailto:sturla at molden.no>> wrote:
>
>
>     On 4/18/2007 7:33 AM, Anne Archibald wrote:
>
>     >copying. And the scope of improvement would be very limited; an
>     expression like A*B+C*D would be much more efficient, probably, if the
>     whole expression were evaluated at once for each element (due to
>     memory locality and temporary allocation). But it is impossible for
>     numpy, sitting inside python as it does, to do that.
>
>
>
>     Most numerical array/matrix libraries dependent on operator
>     overloading
>     generates temporaries. That is why Fortran is usually perceived as
>     superior to C++ for scientific programming. The Fortran compiler knows
>     about arrays and can avoid allocating three temporary arrays to
>     evaluate
>     and expression like
>
>     y = a * b + c * d
>     [SNIP]
>
>
>
>     I'd really like to see a Python extension library do this one day. It
>     would be very cool and (almost) as efficient as plain Fortran - though
>     not quite, we would still get some small temporary objects
>     created. But
>     that is a sacrifice I am willing to pay to use Python. We would gain
>     some efficacy over Fortran by postponing indefinitely evaluation of
>     computations that are not needed, when this is not known at compile
>     time.
>
>
>     Any comments?
>
>
> I periodically wonder if something like this could be built on top of 
> numexpr without too much pain. Simply build up the expression on 
> operations and then evaluate when data is requested. You'd need some 
> sort of Jekyl & Hyde matrix that spent the first part of it's life 
> unevaluated and then converted itself to something very much like an 
> array when it evaluated itself. If might end up being tricky to make 
> sure that everything got transparently evaluated at the right time 
> though.
>
> I've never had time to try it though.
>
>  
>
>     Sturla Molden
>     Ph.D.
>
Hi,

Correct me if I'm wrong but it is not possible to code something working 
like boost++ in python.
It is due to the fact we cannot overload the = operator in python. As a 
result you cannot implement the way boost++ works : + * - / operators 
build a tree and all the loops are done in the = operator code.

I do not say that it is impossible to code an efficient way to deal with 
operators and matrix in python but only that it looks not possible to 
match the boost++ way of thinking.

Comments?

Xavier


-- 
############################################
Xavier Gnata
CRAL - Observatoire de Lyon
9, avenue Charles Andr?
69561 Saint Genis Laval cedex
Phone: +33 4 78 86 85 28
Fax: +33 4 78 86 83 86
E-mail: gnata at obs.univ-lyon1.fr
############################################ 



From rex at nosyntax.com  Wed Apr 18 16:27:29 2007
From: rex at nosyntax.com (rex)
Date: Wed, 18 Apr 2007 13:27:29 -0700
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <46267975.9000904@astraw.com>
References: <evpm71$q6c$1@sea.gmane.org> <462073A6.4010602@gmail.com>
	<evqnir$44m$1@sea.gmane.org>
	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<f01nug$nns$1@sea.gmane.org> <4624E684.9010606@astraw.com>
	<4625879C.6000706@ar.media.kyoto-u.ac.jp>
	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
	<20070418193632.GU8487@x2.nosyntax.com>
	<46267975.9000904@astraw.com>
Message-ID: <20070418202729.GV8487@x2.nosyntax.com>

Andrew Straw <strawman at astraw.com> [2007-04-18 13:22]:
> rex wrote:
> > If your use is entirely non-commercial you can use Intel's MKL with
> > built-in optimized BLAS and LAPACK and avoid the need for ATLAS.
> >   
> Just to clarify, my understanding is that if you buy a developer's
> license, you can also use it for commercial use, including distributing
> binaries. (Otherwise it would seem kind of silly for Intel to invest so
> much in their performance libraries and compilers.)

Yes, I should have included that. icc and MKL licenses for commercial
use are $399 each.

-rex
-- 
"It's a Singer, Captain Picard."   "Make it sew."



From sturla at molden.no  Wed Apr 18 16:52:09 2007
From: sturla at molden.no (Sturla Molden)
Date: Wed, 18 Apr 2007 22:52:09 +0200 (CEST)
Subject: [Numpy-discussion] Efficient operator overloading
In-Reply-To: <46267C00.3050803@obs.univ-lyon1.fr>
References: <46265FF2.2090004@molden.no>
	<e4412d6b0704181131v3b0897a5t56c6eafd15c76658@mail.gmail.com>
	<46267C00.3050803@obs.univ-lyon1.fr>
Message-ID: <1314.89.8.11.123.1176929529.squirrel@webmail.uio.no>

> Timothy Hochberg wrote:

> Correct me if I'm wrong but it is not possible to code something working
> like boost++ in python.
> It is due to the fact we cannot overload the = operator in python. As a
> result you cannot implement the way boost++ works : + * - / operators
> build a tree and all the loops are done in the = operator code.
>
> I do not say that it is impossible to code an efficient way to deal with
> operators and matrix in python but only that it looks not possible to
> match the boost++ way of thinking.

I have three comments:

1. Boost++ depends on template metaprogramming. For this to work, array
bounds must be known in advance. If you don't know the dimensions in
advance, the compiler cannot get rid of the temporary storage and
redundant loops. Basically, Boost++ and Blitz++ can be efficient for small
tight loops. But for small arrays, Boost++ and Blitz++ can do miracles.

2. Template metaprogramming and symbolic maths are very different
subjects. Evaluating symbols run-time is very different for evaluating
templates compile-time. If you know the eval() function in Python and Lisp
it is probably easier to understand what I suggested; eval() and Lisp
macros are not generally understood by the C++ community.

3. My suggestion did not require overloading the assignment operator (=).
I suggested to let the "Jekyll & Hyde array" evaluate itself when it's
content is requested (e.g. using the indexing operator [] as one such
signal). Actually, not overloading the assignment operator is important as
it allows us to procrastinate evaluation of expressions as long as we can.
It may seem strange, but it is an important optimization technique. The
things you don't do don't take any time.

















From eike.welk at gmx.net  Wed Apr 18 16:56:55 2007
From: eike.welk at gmx.net (Eike Welk)
Date: Wed, 18 Apr 2007 22:56:55 +0200
Subject: [Numpy-discussion] Efficient operator overloading
In-Reply-To: <46265FF2.2090004@molden.no>
References: <46265FF2.2090004@molden.no>
Message-ID: <200704182256.55751.eike.welk@gmx.net>

In the SciPy sandbox there is a module that goes into this direction. 
It was mentioned in the other optimization thread. numexpr:
http://projects.scipy.org/scipy/scipy/browser/trunk/Lib/sandbox/numexpr

This module needs to be combined with a derived array class where the 
operators return unevaluated expressions (lazy evaluation). All 
access methods and asarray must call the evaluation function. Like 
you proposed.
The unevaluated expressions and a compiler for them are in the numexpr 
module.
A very basic version of this would probably be not much work. 

I would like to replace:
import numpy as N
by:
import lazynumpy as N
:-)

Regards Eike.



From rex at nosyntax.com  Wed Apr 18 17:00:34 2007
From: rex at nosyntax.com (rex)
Date: Wed, 18 Apr 2007 14:00:34 -0700
Subject: [Numpy-discussion] Scimark, icc, & Core 2 Duo
Message-ID: <20070418210034.GW8487@x2.nosyntax.com>


Keith Goodman <kwgoodman at gmail.com> [2007-04-18 12:46]:
> Thanks for that. For a variety of reasons I'm sticking with atlas.
> Does the parallel flag give you a big speed increase? I imagine it
> speeds things up more for larger matrices.

Surprisingly little. Below are the results of running Scimark with
various icc and gcc compiler flags set. The maximum Scimark score is
55% larger with icc than with gcc, though there may be flags other than
-O3 that would help gcc.

The optimized (for Xeon, not for Core 2 Duo) LINPACK that ships with MKL
runs at about 7 gigaflops max on my Core 2 Duo overclocked to 2.93 GHz
(it's different from LINPACK 1000). There is a Core 2 Duo optimized
version for OSX.

icc with no flags set:
> icc *.c -o no_flags
> ./noflags -large
**                                                              **
** SciMark2 Numeric Benchmark, see http://math.nist.gov/scimark **
** for details. (Results can be submitted to pozo at nist.gov)     **
**                                                              **
Using       2.00 seconds min time per kenel.
Composite Score:          605.84
FFT             Mflops:   111.70    (N=1048576)
SOR             Mflops:   868.52    (1000 x 1000)
MonteCarlo:     Mflops:   120.37
Sparse matmult  Mflops:   853.33    (N=100000, nz=1000000)
LU              Mflops:  1075.27    (M=1000, N=1000)

> icc -fast  *.c -o fast
> ./fast -large
**                                                              **
** SciMark2 Numeric Benchmark, see http://math.nist.gov/scimark **
** for details. (Results can be submitted to pozo at nist.gov)     **
**                                                              **
Using       2.00 seconds min time per kenel.
Composite Score:          785.63
FFT             Mflops:   108.31    (N=1048576)
SOR             Mflops:   985.81    (1000 x 1000)
MonteCarlo:     Mflops:   848.81
Sparse matmult  Mflops:   825.81    (N=100000, nz=1000000)
LU              Mflops:  1159.42    (M=1000, N=1000)

> icc -fast  -parallel *.c -o fast_para
IPO: performing multi-file optimizations
IPO: generating object file /tmp/ipo_iccvHW42m.o
scimark2.c(63) : (col. 18) remark: LOOP WAS VECTORIZED.
kernel.c(157) : (col. 13) remark: LOOP WAS VECTORIZED.
kernel.c(212) : (col. 17) remark: LOOP WAS VECTORIZED.
> ./fast_para -large
**                                                              **
** SciMark2 Numeric Benchmark, see http://math.nist.gov/scimark **
** for details. (Results can be submitted to pozo at nist.gov)     **
**                                                              **
Using       2.00 seconds min time per kenel.
Composite Score:          796.33
FFT             Mflops:   111.70    (N=1048576)
SOR             Mflops:  1001.91    (1000 x 1000)
MonteCarlo:     Mflops:   855.57
Sparse matmult  Mflops:   832.52    (N=100000, nz=1000000)
LU              Mflops:  1179.94    (M=1000, N=1000)


> icc -fast -parallel -fno-alias *.c -o fast_para_noali
IPO: performing multi-file optimizations
IPO: generating object file /tmp/ipo_iccLUySDv.o
scimark2.c(63) : (col. 18) remark: LOOP WAS VECTORIZED.
kernel.c(157) : (col. 13) remark: LOOP WAS VECTORIZED.
kernel.c(212) : (col. 17) remark: LOOP WAS VECTORIZED.
> ./fast_para_noali -large
**                                                              **
** SciMark2 Numeric Benchmark, see http://math.nist.gov/scimark **
** for details. (Results can be submitted to pozo at nist.gov)     **
**                                                              **
Using       2.00 seconds min time per kenel.
Composite Score:          890.46
FFT             Mflops:   109.70    (N=1048576)
SOR             Mflops:  1488.28    (1000 x 1000)
MonteCarlo:     Mflops:   855.57

Sparse matmult  Mflops:   829.15    (N=100000, nz=1000000)
LU              Mflops:  1169.59    (M=1000, N=1000)
> icc -fast -parallel -fno-alias -funroll-loops *.c -o fast_para_noali_unr
IPO: performing multi-file optimizations
IPO: generating object file /tmp/ipo_icc2KA1ui.o
scimark2.c(63) : (col. 18) remark: LOOP WAS VECTORIZED.
kernel.c(157) : (col. 13) remark: LOOP WAS VECTORIZED.
kernel.c(212) : (col. 17) remark: LOOP WAS VECTORIZED.
> ./fast_para_noali_unr -large
**                                                              **
** SciMark2 Numeric Benchmark, see http://math.nist.gov/scimark **
** for details. (Results can be submitted to pozo at nist.gov)     **
**                                                              **
Using       2.00 seconds min time per kenel.
Composite Score:          901.11
FFT             Mflops:   113.48    (N=1048576)
SOR             Mflops:  1510.28    (1000 x 1000)
MonteCarlo:     Mflops:   865.92
Sparse matmult  Mflops:   835.92    (N=100000, nz=1000000)
LU              Mflops:  1179.94    (M=1000, N=1000)



> gcc -lm *.c -o ggc_none
> ./ggc_none -large
**                                                              **
** SciMark2 Numeric Benchmark, see http://math.nist.gov/scimark **
** for details. (Results can be submitted to pozo at nist.gov)     **
**                                                              **
Using       2.00 seconds min time per kenel.
Composite Score:          323.63
FFT             Mflops:    83.56    (N=1048576)
SOR             Mflops:   729.97    (1000 x 1000)
MonteCarlo:     Mflops:    73.75
Sparse matmult  Mflops:   329.26    (N=100000, nz=1000000)
LU              Mflops:   401.61    (M=1000, N=1000)

> gcc -lm -O3 *.c -o ggc_O3
> ./gcc_O3 -large
**                                                              **
** SciMark2 Numeric Benchmark, see http://math.nist.gov/scimark **
** for details. (Results can be submitted to pozo at nist.gov)     **
**                                                              **
Using       2.00 seconds min time per kenel.
Composite Score:          580.55
FFT             Mflops:   108.86    (N=1048576)
SOR             Mflops:   842.27    (1000 x 1000)
MonteCarlo:     Mflops:   115.70
Sparse matmult  Mflops:   825.81    (N=100000, nz=1000000)
LU              Mflops:  1010.10    (M=1000, N=1000)


-rex


From christian at marquardt.sc  Wed Apr 18 17:04:26 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Wed, 18 Apr 2007 23:04:26 +0200 (CEST)
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <20070418202729.GV8487@x2.nosyntax.com>
References: <evpm71$q6c$1@sea.gmane.org> <462073A6.4010602@gmail.com>
	<evqnir$44m$1@sea.gmane.org> <4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<f01nug$nns$1@sea.gmane.org> <4624E684.9010606@astraw.com>
	<4625879C.6000706@ar.media.kyoto-u.ac.jp>
	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
	<20070418193632.GU8487@x2.nosyntax.com> <46267975.9000904@astraw.com>
	<20070418202729.GV8487@x2.nosyntax.com>
Message-ID: <6658.84.167.123.76.1176930266.squirrel@webmail.marquardt.sc>

Sun has recently released their compilers under an opensource license for
Linux as well (Sun Studio Express or something), including their perflib -
which includes Blas and Lapack. Has somebody tried how that combination
performs, compared to Intel MKL or Atlas? I think they are free even for
commercial use.

Just a thought...

  Christian.


On Wed, April 18, 2007 22:27, rex wrote:
> Andrew Straw <strawman at astraw.com> [2007-04-18 13:22]:
>> rex wrote:
>> > If your use is entirely non-commercial you can use Intel's MKL with
>> > built-in optimized BLAS and LAPACK and avoid the need for ATLAS.
>> >
>> Just to clarify, my understanding is that if you buy a developer's
>> license, you can also use it for commercial use, including distributing
>> binaries. (Otherwise it would seem kind of silly for Intel to invest so
>> much in their performance libraries and compilers.)
>
> Yes, I should have included that. icc and MKL licenses for commercial
> use are $399 each.
>
> -rex
> --
> "It's a Singer, Captain Picard."   "Make it sew."
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>




From perry at stsci.edu  Wed Apr 18 17:15:04 2007
From: perry at stsci.edu (Perry Greenfield)
Date: Wed, 18 Apr 2007 17:15:04 -0400
Subject: [Numpy-discussion] Efficient operator overloading
In-Reply-To: <46265FF2.2090004@molden.no>
References: <46265FF2.2090004@molden.no>
Message-ID: <6DFF45E3-FEC5-4073-9B1E-6515FCCF70EA@stsci.edu>


On Apr 18, 2007, at 2:14 PM, Sturla Molden wrote:

[...]
>
> Let us assume again we have an expression like
>
> y = a * b + c * d
>
> where a,b,c and d are all arrays or matrices. In this case, the
> overloaded * and + operators woud not return a temporary array but an
> unevaluated expression of class Expr. Thus we would get
>
> tmp1 = Expr('__mul__',a,b) # symbolic representation of 'a * b'
> tmp2 = Expr('__mul__',c,d) # symbolic representation of 'c * d'
> tmp3 = Expr('__add__',tmp1,tmp1) # symbolic "a * b + c * d"
> del tmp1
> del tmp2
> y = tmp3 # y becomes a reference to an unevaluated expression
>
> Finally, we need a mechanism to 'flush' the unevaluated expression.
> Python does not allow the assignment operator to be evaluated, so one
> could not depend on that. But one could a 'flush on read/write'

At the risk of being remembered only for dredging up hacks, there is  
something similar that would at least serve for someone to try work  
along these lines. Nothing prevents you from overloading one of the  
augmented operators (e.g., |=) to trigger the evaluation. True, it  
uses up one of these that can be used for a numeric purpose, but it  
can at least be used to test the idea to see how workable it is (I  
forget who originally suggested this but I heard it suggested it at  
the Long Beach Python Conference; it might have been Tim Peters).

Perry


From eike.welk at gmx.net  Wed Apr 18 17:28:30 2007
From: eike.welk at gmx.net (Eike Welk)
Date: Wed, 18 Apr 2007 23:28:30 +0200
Subject: [Numpy-discussion] Efficient operator overloading
In-Reply-To: <46265FF2.2090004@molden.no>
References: <46265FF2.2090004@molden.no>
Message-ID: <200704182328.30759.eike.welk@gmx.net>

On Wednesday 18 April 2007 20:14, Sturla Molden wrote:
> a[i] = 2.0 # The expression 'a * b + c * d' gets evaluated. The
> ? ? ? ? ? ? # object referred to by y now holds an actual array.
> ? ? ? ? ? ? # Finally, 2.0 is written to a[i].

This case will require some extra work. The array needs to remember, 
that there is an unevaluated expression depending on it.

Regards Eike.



From sturla at molden.no  Wed Apr 18 18:16:47 2007
From: sturla at molden.no (Sturla Molden)
Date: Thu, 19 Apr 2007 00:16:47 +0200 (CEST)
Subject: [Numpy-discussion] Efficient operator overloading
In-Reply-To: <200704182328.30759.eike.welk@gmx.net>
References: <46265FF2.2090004@molden.no> <200704182328.30759.eike.welk@gmx.net>
Message-ID: <2845.89.8.11.123.1176934607.squirrel@webmail.uio.no>

> On Wednesday 18 April 2007 20:14, Sturla Molden wrote:

> This case will require some extra work. The array needs to remember,
> that there is an unevaluated expression depending on it.

This is certainly a complication that needs to be worked out. An array
could store a set of dependent expressions, but I don't know if it's
efficient enough.

The most na?ve solution would be to make arrays immutable (copy-on-write)
like strings. But that is not very appealing. Actually, Matlab's matrices
behaves exactly like this - and it isn't that bad - but I much rather
prefer mutable NumPy arrays.











From gnata at obs.univ-lyon1.fr  Wed Apr 18 18:19:11 2007
From: gnata at obs.univ-lyon1.fr (Xavier Gnata)
Date: Thu, 19 Apr 2007 00:19:11 +0200
Subject: [Numpy-discussion] Efficient operator overloading
In-Reply-To: <1314.89.8.11.123.1176929529.squirrel@webmail.uio.no>
References: <46265FF2.2090004@molden.no>	<e4412d6b0704181131v3b0897a5t56c6eafd15c76658@mail.gmail.com>	<46267C00.3050803@obs.univ-lyon1.fr>
	<1314.89.8.11.123.1176929529.squirrel@webmail.uio.no>
Message-ID: <4626995F.6020609@obs.univ-lyon1.fr>

Sturla Molden wrote:
>> Timothy Hochberg wrote:
>>     
>
>   
>> Correct me if I'm wrong but it is not possible to code something working
>> like boost++ in python.
>> It is due to the fact we cannot overload the = operator in python. As a
>> result you cannot implement the way boost++ works : + * - / operators
>> build a tree and all the loops are done in the = operator code.
>>
>> I do not say that it is impossible to code an efficient way to deal with
>> operators and matrix in python but only that it looks not possible to
>> match the boost++ way of thinking.
>>     
>
> I have three comments:
>
> 1. Boost++ depends on template metaprogramming. For this to work, array
> bounds must be known in advance. If you don't know the dimensions in
> advance, the compiler cannot get rid of the temporary storage and
> redundant loops. Basically, Boost++ and Blitz++ can be efficient for small
> tight loops. But for small arrays, Boost++ and Blitz++ can do miracles.
>
> 2. Template metaprogramming and symbolic maths are very different
> subjects. Evaluating symbols run-time is very different for evaluating
> templates compile-time. If you know the eval() function in Python and Lisp
> it is probably easier to understand what I suggested; eval() and Lisp
> macros are not generally understood by the C++ community.
>
> 3. My suggestion did not require overloading the assignment operator (=).
> I suggested to let the "Jekyll & Hyde array" evaluate itself when it's
> content is requested (e.g. using the indexing operator [] as one such
> signal). Actually, not overloading the assignment operator is important as
> it allows us to procrastinate evaluation of expressions as long as we can.
> It may seem strange, but it is an important optimization technique. The
> things you don't do don't take any time.
>
>   

yes yes and yes :).

I don't know if I'm a python or a c++ programmer but I fully understand 
what eval() is.
eval() is not evil in a python/lisp context. Not at all! I do like to 
have the capability to play with str and eval in python (I also do like 
template meta prog...)

My point was only to say that the way to provide the user with this 
lazzy operators can't be the same as in c++. Sorry if my message was not 
clear :(.
What you want is symbolics maths with matrix and a way to trigger the 
evaluation.
Well, as far as I have seen for now, it looks pretty hard to code that 
with a pythonic syntax and a way to report in case of errors. It is also 
pretty hard to avoid nasty infinite recursions playing with "to be 
evaluated objects".

I'm going to have a look to this scipy sandbox module.

Xavier

-- 
############################################
Xavier Gnata
CRAL - Observatoire de Lyon
9, avenue Charles Andr?
69561 Saint Genis Laval cedex
Phone: +33 4 78 86 85 28
Fax: +33 4 78 86 83 86
E-mail: gnata at obs.univ-lyon1.fr
############################################ 



From daniel.wheeler at nist.gov  Wed Apr 18 18:31:44 2007
From: daniel.wheeler at nist.gov (Daniel Wheeler)
Date: Wed, 18 Apr 2007 22:31:44 +0000
Subject: [Numpy-discussion] Efficient operator overloading
In-Reply-To: <e4412d6b0704181131v3b0897a5t56c6eafd15c76658@mail.gmail.com>
References: <46265FF2.2090004@molden.no>
	<e4412d6b0704181131v3b0897a5t56c6eafd15c76658@mail.gmail.com>
Message-ID: <B44E953C-E592-4844-AC12-1A535EC25220@nist.gov>

On Apr 18, 2007, at 6:31 PM, Timothy Hochberg wrote:

>
>
> On 4/18/07, Sturla Molden <sturla at molden.no> wrote:
>
>
>
> I'd really like to see a Python extension library do this one day. It
> would be very cool and (almost) as efficient as plain Fortran - though
> not quite, we would still get some small temporary objects created.  
> But
> that is a sacrifice I am willing to pay to use Python. We would gain
> some efficacy over Fortran by postponing indefinitely evaluation of
> computations that are not needed, when this is not known at compile
> time.
>
>
> Any comments?
>
> I periodically wonder if something like this could be built on top  
> of numexpr without too much pain. Simply build up the expression on  
> operations and then evaluate when data is requested. You'd need  
> some sort of Jekyl & Hyde matrix that spent the first part of it's  
> life unevaluated and then converted itself to something very much  
> like an array when it evaluated itself. If might end up being  
> tricky to make sure that everything got transparently evaluated at  
> the right time though.

It seems that we have developed something like this for the fipy code  
(http://www.ctcms.nist.gov/fipy). We  refer to
it as "lazy evaluation". Essentially, objects called "Variable"s hold  
numpy arrays. They behave
as ordinary numpy arrays, but are only evaluated when the information  
is requested. e.g.

    >>> from fipy import *
    >>> v0 = Variable(numerix.array((0,1,2)))
    >>> v1 = Variable(numerix.array((2,3,4)))
    >>> v2 = v0 * v1 + v0 * v1
    >>> v2
    ((Variable(value = array([0, 1, 2])) * Variable(value = array([3,  
4, 5]))) + (Variable(value = array([0, 1, 2])) * Variable(value =  
array([3, 4, 5]))))
    >>> print v2
    [ 0  8 20]

By passing an '--inline' flag at run time we can choose to evaluate  
this expression with weave. Variable's have a _getCstring() method
that's used to pass to weave when inlining along with other information.

    >>> print v2._getCstring()
    '((var00(i) * var01(i)) + (var10(i) * var11(i)))'

Actually, we could and should have been using numexpr to do this. I  
think numexpr would substantially clean up
the code we've developed. Anyway, you can browse variable.py at  
<http://matdl-osi.org/fipy/browser/trunk/fipy/variables/variable.py>.

Cheers

--
Daniel Wheeler


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/79a3b73a/attachment.html>

From nvf at uwm.edu  Wed Apr 18 19:09:38 2007
From: nvf at uwm.edu (Nick Fotopoulos)
Date: Wed, 18 Apr 2007 18:09:38 -0500
Subject: [Numpy-discussion] Help using numPy to create a very large
	multi dimensional array
In-Reply-To: <f687be130704181605t5ee447c4k52ca845b4a44ad41@mail.gmail.com>
References: <f687be130704181605t5ee447c4k52ca845b4a44ad41@mail.gmail.com>
Message-ID: <f687be130704181609h13091e6y9e47b8ec9a45a5eb@mail.gmail.com>

On 4/18/07, numpy-discussion-request at scipy.org
<numpy-discussion-request at scipy.org> wrote:

> ------------------------------
>
> Message: 5
> Date: Wed, 18 Apr 2007 09:11:32 -0700
> From: Christopher Barker <Chris.Barker at noaa.gov>
> Subject: Re: [Numpy-discussion] Help using numPy to create a very
>         large   multi dimensional array
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
> Message-ID: <46264334.8080304 at noaa.gov>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Bruno Santos wrote:
> > Finally I was able to read the data, by using the command you sair with
> > some small changes:
> > matrix = numpy.array([[float(x) for x in line.split()[1:]] for line in
> > vecfile])
>
> it doesn't sound like you're concerned about the speed of reading the
> files, but you can still use fromfile() or maybe fromstring() to do
> this. You just need to read past the text part first, then process it.
>
> using fromstring:
>
> matrix = numpy.vstack([numpy.fromstring(line.split(" ", 1)[1], sep=" ")
> for line in vecfile])
>
> or something like that.
>
> -Chris

I would strongly recommend pylab.load.  It handles comments, selects
columns, and is legible.

Examples from the docstring:

        t,y = load('test.dat', unpack=True) # for  two column data
        x,y,z = load('somefile.dat', usecols=(3,5,7), unpack=True)

A more advanced example from examples/load_converter.py:

dates, closes = load(
    'data/msft.csv', delimiter=',',
    converters={0:datestr2num}, skiprows=1, usecols=(0,2),
    unpack=True)

Devs, is there any possibility of moving/copying pylab.load to numpy?
I don't see anything in the source that requires the rest of
matplotlib.  Among convenience functions, I think that this function
ranks pretty highly in convenience.

Take care,
Nick


From spaceynumpy-discussion at lenin.net  Wed Apr 18 17:36:52 2007
From: spaceynumpy-discussion at lenin.net (Peter C. Norton)
Date: Wed, 18 Apr 2007 14:36:52 -0700
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <6658.84.167.123.76.1176930266.squirrel@webmail.marquardt.sc>
References: <evqnir$44m$1@sea.gmane.org>
	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<f01nug$nns$1@sea.gmane.org> <4624E684.9010606@astraw.com>
	<4625879C.6000706@ar.media.kyoto-u.ac.jp>
	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
	<20070418193632.GU8487@x2.nosyntax.com>
	<46267975.9000904@astraw.com>
	<20070418202729.GV8487@x2.nosyntax.com>
	<6658.84.167.123.76.1176930266.squirrel@webmail.marquardt.sc>
Message-ID: <20070418213652.GH5812@lenin.net>

I'm having great difficulty building numpy on solaris with the sun
compilers and libsunperf. I may try this on ubuntu x86_64 to see if
the setup is less painful.

Thanks for the idea,

-Peter

On Wed, Apr 18, 2007 at 11:04:26PM +0200, Christian Marquardt wrote:
> Sun has recently released their compilers under an opensource license for
> Linux as well (Sun Studio Express or something), including their perflib -
> which includes Blas and Lapack. Has somebody tried how that combination
> performs, compared to Intel MKL or Atlas? I think they are free even for
> commercial use.
> 
> Just a thought...
> 
>   Christian.
> 
> 
> On Wed, April 18, 2007 22:27, rex wrote:
> > Andrew Straw <strawman at astraw.com> [2007-04-18 13:22]:
> >> rex wrote:
> >> > If your use is entirely non-commercial you can use Intel's MKL with
> >> > built-in optimized BLAS and LAPACK and avoid the need for ATLAS.
> >> >
> >> Just to clarify, my understanding is that if you buy a developer's
> >> license, you can also use it for commercial use, including distributing
> >> binaries. (Otherwise it would seem kind of silly for Intel to invest so
> >> much in their performance libraries and compilers.)
> >
> > Yes, I should have included that. icc and MKL licenses for commercial
> > use are $399 each.
> >
> > -rex
> > --
> > "It's a Singer, Captain Picard."   "Make it sew."
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> 
> 
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

-- 
The 5 year plan:
In five years we'll make up another plan.
Or just re-use this one.



From david at ar.media.kyoto-u.ac.jp  Wed Apr 18 22:47:45 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 19 Apr 2007 11:47:45 +0900
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <46266A13.5010101@gmail.com>
References: <evpm71$q6c$1@sea.gmane.org>	<462073A6.4010602@gmail.com>	<evqnir$44m$1@sea.gmane.org>	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>	<f01nug$nns$1@sea.gmane.org>	<4624E684.9010606@astraw.com>	<4625879C.6000706@ar.media.kyoto-u.ac.jp>	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>	<e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>
	<46266A13.5010101@gmail.com>
Message-ID: <4626D851.9000602@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
> Charles R Harris wrote:
>
>   
>> I don't know which is best, although I suspect the statically linked
>> version will be larger. It might seem that just pulling in the gemm
>> routines wouldn't add much, but they pull in lots of supporting
>> routines. To get numpy to link statically you will also probably need to
>> have a directory that contains only the *.a versions because the linker
>> will default to the *.so if they are present; i don't think there is a
>> way to specify the -static flag to the gcc compiler. Maybe someone else
>> knows how to do that.
>>     
>
> Not really. Since you are *making* a shared library, the flags are already set
> to *look up* shared libraries. I do not believe they are overridable, much to my
> frequent vexation.
>   
I think there is a way, but not trivial. I had to do all kind of tricks 
to use fftw in mex (matlab extension written in C, practically 
dynamically loaded .so inside matlab), and linking statically some 
libraries was one of them. If you still need it (linking some libraries 
statically, other dynamically, to build a .so), I can try to dig in my 
matlab scripts (unused for some time thanks to scipy :) )

cheers,

David


From charlesr.harris at gmail.com  Wed Apr 18 22:57:14 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 18 Apr 2007 20:57:14 -0600
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <4626D851.9000602@ar.media.kyoto-u.ac.jp>
References: <evpm71$q6c$1@sea.gmane.org> <evqnir$44m$1@sea.gmane.org>
	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp> <f01nug$nns$1@sea.gmane.org>
	<4624E684.9010606@astraw.com>
	<4625879C.6000706@ar.media.kyoto-u.ac.jp>
	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
	<e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>
	<46266A13.5010101@gmail.com> <4626D851.9000602@ar.media.kyoto-u.ac.jp>
Message-ID: <e06186140704181957r4dc4ac9fiac69553740611f45@mail.gmail.com>

On 4/18/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>
> Robert Kern wrote:
> > Charles R Harris wrote:
> >
> >
> >> I don't know which is best, although I suspect the statically linked
> >> version will be larger. It might seem that just pulling in the gemm
> >> routines wouldn't add much, but they pull in lots of supporting
> >> routines. To get numpy to link statically you will also probably need
> to
> >> have a directory that contains only the *.a versions because the linker
> >> will default to the *.so if they are present; i don't think there is a
> >> way to specify the -static flag to the gcc compiler. Maybe someone else
> >> knows how to do that.
> >>
> >
> > Not really. Since you are *making* a shared library, the flags are
> already set
> > to *look up* shared libraries. I do not believe they are overridable,
> much to my
> > frequent vexation.
> >
> I think there is a way, but not trivial. I had to do all kind of tricks
> to use fftw in mex (matlab extension written in C, practically
> dynamically loaded .so inside matlab), and linking statically some
> libraries was one of them. If you still need it (linking some libraries
> statically, other dynamically, to build a .so), I can try to dig in my
> matlab scripts (unused for some time thanks to scipy :) )


I'm wondering if the static libraries could simply be compiled with the
-fPIC flag and linked with the program to produce the dynamic library. The
static libraries are just collections of *.o files, so I don't see why that
shouldn't work.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/2d3c5d06/attachment.html>

From robert.kern at gmail.com  Wed Apr 18 23:01:48 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Apr 2007 22:01:48 -0500
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <e06186140704181957r4dc4ac9fiac69553740611f45@mail.gmail.com>
References: <evpm71$q6c$1@sea.gmane.org>
	<evqnir$44m$1@sea.gmane.org>	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>
	<f01nug$nns$1@sea.gmane.org>	<4624E684.9010606@astraw.com>	<4625879C.6000706@ar.media.kyoto-u.ac.jp>	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>	<e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>	<46266A13.5010101@gmail.com>
	<4626D851.9000602@ar.media.kyoto-u.ac.jp>
	<e06186140704181957r4dc4ac9fiac69553740611f45@mail.gmail.com>
Message-ID: <4626DB9C.1030602@gmail.com>

Charles R Harris wrote:
> I'm wondering if the static libraries could simply be compiled with the
> -fPIC flag and linked with the program to produce the dynamic library.
> The static libraries are just collections of *.o files, so I don't see
> why that shouldn't work.

I don't think there is a reason why it *wouldn't* work; it's just that one needs
to do "gcc -shared" to make a .so and that flag also tells the linker to choose
.so's over .a's for the libraries that get linked into the target .so.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Wed Apr 18 23:03:44 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 19 Apr 2007 12:03:44 +0900
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <e06186140704181135w469cf329jaecbdc047d96fcec@mail.gmail.com>
References: <evpm71$q6c$1@sea.gmane.org>
	<462073A6.4010602@gmail.com>	<evqnir$44m$1@sea.gmane.org>
	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>	<f01nug$nns$1@sea.gmane.org>
	<4624E684.9010606@astraw.com>	<4625879C.6000706@ar.media.kyoto-u.ac.jp>	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>	<e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>	<f4f93d420704181017n24670876laa32b1fd18858624@mail.gmail.com>
	<e06186140704181135w469cf329jaecbdc047d96fcec@mail.gmail.com>
Message-ID: <4626DC10.30908@ar.media.kyoto-u.ac.jp>

Charles R Harris wrote:
>
>
> On 4/18/07, *Keith Goodman* <kwgoodman at gmail.com 
> <mailto:kwgoodman at gmail.com>> wrote:
>
>     On 4/18/07, Charles R Harris <charlesr.harris at gmail.com
>     <mailto:charlesr.harris at gmail.com>> wrote:
>     > On 4/18/07, Keith Goodman <kwgoodman at gmail.com
>     <mailto:kwgoodman at gmail.com>> wrote:
>     > > I'd like to compile atlas so that I can take full advantage of
>     my core
>     > > 2 duo. Numpy dynamically links to the debian binary of
>     atlas-sse that
>     > > I installed. But the atlas website says that they recommend
>     static
>     > > linking.
>     > >
>     > > Which do you recommend, static or dynamic? Are there good
>     directions for
>     > either?
>     >
>     > I don't know which is best, although I suspect the statically
>     linked version
>     > will be larger. It might seem that just pulling in the gemm routines
>     > wouldn't add much, but they pull in lots of supporting routines.
>     To get
>     > numpy to link statically you will also probably need to have a
>     directory
>     > that contains only the *.a versions because the linker will
>     default to the
>     > *.so if they are present; i don't think there is a way to
>     specify the
>     > -static flag to the gcc compiler. Maybe someone else knows how
>     to do that.
>     > For ATLAS, I believe the latest versions are also recommended
>     because the
>     > stable version is so old.
>
>     At the moment best is equal to easiest since I have never compiled
>     atlas. Does anyone know of a howto on compiling atlas (dynamically
>     linked)?
>
>
> The instructions that come with ATLAS are useable, and with the 3.7.30 
> version the process is pretty easy. Just make sure you read the part 
> concerning relocatable libraries first, as you will need to add some 
> flags to the ./configure command line, -fPIC  and a couple of others. 
> After compiling and install, IIRC, the libraries will be in 
> /usr/local/lib/ATLAS, which should be OK for debian. Then you need to 
> add a file in /etc/ld.conf.d with the name of the directory and run 
> ldconfig in order to let the system know where the libraries are and 
> what they contain. Or you could save the old /usr/lib/atlas directory 
> and put the new libraries there and then run ldconfig. I also found it 
> necessary to delete the old numpy site-package before reinstalling numpy.
Here is what I do to compile atlas with full lapack, dynamically, on 
debian systems (ubuntu is the same for that matter):

    - sudo apt-get install g77 lapack3-pic lapack3-test refblas3-test
    - untar atlas, go into ATLAS directory
    - mkdir MyObj, cd MyObj (this is where the lib will be built)
    - configure the build: ../configure -Si archdef 1 -Fa alg -fPIC -C 
if g77 -Ss flapack /usr/lib/liblapack_pic.a
    - build the lib: make
    - time and test the library: make time; make test
    - build the shared library: cd lib; make shared.

Explanations:
    - lapack3-test will be handy to test the compiled atlas, and 
lapack3-pic gives you a lapack library which can be used to build a full 
shared LAPACK library using atlas.
    - archdef 1 means uses arch default, much faster - up to 10 times - 
to compile the library
    - -Fa alg -fPIC says to add -fPIC to build all objects files (needed 
for shared lib)
    - -C if g77 says to use g77 and not gfortran as the fortran 
compiler. On debian, g77 is still the default, and gfortran has a 
different ABI, meaning all kind of problems if you use gfortran for now 
on debian if you do not know what you are doing.

I strongly advise you to use the officiel lapack and blas testers 
afterwards. On debian, as they are dynamically linked to blas/lapack, 
you can choose which library to test thanks to the LD_LIBRARY_PATH trick.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Wed Apr 18 23:12:24 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 19 Apr 2007 12:12:24 +0900
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <4626DB9C.1030602@gmail.com>
References: <evpm71$q6c$1@sea.gmane.org>	<evqnir$44m$1@sea.gmane.org>	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>	<f01nug$nns$1@sea.gmane.org>	<4624E684.9010606@astraw.com>	<4625879C.6000706@ar.media.kyoto-u.ac.jp>	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>	<e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>	<46266A13.5010101@gmail.com>	<4626D851.9000602@ar.media.kyoto-u.ac.jp>	<e06186140704181957r4dc4ac9fiac69553740611f45@mail.gmail.com>
	<4626DB9C.1030602@gmail.com>
Message-ID: <4626DE18.7050004@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
> Charles R Harris wrote:
>> I'm wondering if the static libraries could simply be compiled with the
>> -fPIC flag and linked with the program to produce the dynamic library.
>> The static libraries are just collections of *.o files, so I don't see
>> why that shouldn't work.
>
> I don't think there is a reason why it *wouldn't* work; it's just that one needs
> to do "gcc -shared" to make a .so and that flag also tells the linker to choose
> .so's over .a's for the libraries that get linked into the target .so.
>
Ok, I found it much faster than I expected: you should use the option 
-Bdynamic of the *linker*, that is, for example:

gcc -shared -o c_gaussd.mexglx c_gaussd.o mex_alloc.o 
-L/usr/local/matlab/bin/glnx86  -L/usr/media/boulot/local/lib 
-L/usr/lib/atlas -Wl,-Bstatic,-lem_full -llapack_atlas -latlas 
-Wl,-Bdynamic,-lmx -lmex

will build a shared "library" c_gaussd.mexglx from c_gaussd.o and 
mex_alloc.o, statically link libem_full, lapack_atlas and atlas, and 
dynamically everything else. This was working, as otherwise, matlab 
would have picked up atlas symbols from its existing, already linked own 
atlas, and not from the atlas installed on my system (which was crashing 
matlab and was the reason why I went through this pain in the first place).

cheers,

David



   



From robert.kern at gmail.com  Wed Apr 18 23:25:58 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Apr 2007 22:25:58 -0500
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <4626DE18.7050004@ar.media.kyoto-u.ac.jp>
References: <evpm71$q6c$1@sea.gmane.org>	<evqnir$44m$1@sea.gmane.org>	<4622DEB0.6000206@ar.media.kyoto-u.ac.jp>	<f01nug$nns$1@sea.gmane.org>	<4624E684.9010606@astraw.com>	<4625879C.6000706@ar.media.kyoto-u.ac.jp>	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>	<e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>	<46266A13.5010101@gmail.com>	<4626D851.9000602@ar.media.kyoto-u.ac.jp>	<e06186140704181957r4dc4ac9fiac69553740611f45@mail.gmail.com>	<4626DB9C.1030602@gmail.com>
	<4626DE18.7050004@ar.media.kyoto-u.ac.jp>
Message-ID: <4626E146.4080703@gmail.com>

David Cournapeau wrote:
> Robert Kern wrote:
>> Charles R Harris wrote:
>>> I'm wondering if the static libraries could simply be compiled with the
>>> -fPIC flag and linked with the program to produce the dynamic library.
>>> The static libraries are just collections of *.o files, so I don't see
>>> why that shouldn't work.
>> I don't think there is a reason why it *wouldn't* work; it's just that one needs
>> to do "gcc -shared" to make a .so and that flag also tells the linker to choose
>> .so's over .a's for the libraries that get linked into the target .so.
>>
> Ok, I found it much faster than I expected: you should use the option 
> -Bdynamic of the *linker*, that is, for example:
> 
> gcc -shared -o c_gaussd.mexglx c_gaussd.o mex_alloc.o 
> -L/usr/local/matlab/bin/glnx86  -L/usr/media/boulot/local/lib 
> -L/usr/lib/atlas -Wl,-Bstatic,-lem_full -llapack_atlas -latlas 
> -Wl,-Bdynamic,-lmx -lmex
> 
> will build a shared "library" c_gaussd.mexglx from c_gaussd.o and 
> mex_alloc.o, statically link libem_full, lapack_atlas and atlas, and 
> dynamically everything else. This was working, as otherwise, matlab 
> would have picked up atlas symbols from its existing, already linked own 
> atlas, and not from the atlas installed on my system (which was crashing 
> matlab and was the reason why I went through this pain in the first place).

Thank you! That's good information to know.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Thu Apr 19 00:20:20 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 18 Apr 2007 22:20:20 -0600
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <4626DB9C.1030602@gmail.com>
References: <evpm71$q6c$1@sea.gmane.org> <f01nug$nns$1@sea.gmane.org>
	<4624E684.9010606@astraw.com>
	<4625879C.6000706@ar.media.kyoto-u.ac.jp>
	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>
	<e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>
	<46266A13.5010101@gmail.com> <4626D851.9000602@ar.media.kyoto-u.ac.jp>
	<e06186140704181957r4dc4ac9fiac69553740611f45@mail.gmail.com>
	<4626DB9C.1030602@gmail.com>
Message-ID: <e06186140704182120p3506f1f0k2de6ee909940832c@mail.gmail.com>

On 4/18/07, Robert Kern <robert.kern at gmail.com> wrote:
>
> Charles R Harris wrote:
> > I'm wondering if the static libraries could simply be compiled with the
> > -fPIC flag and linked with the program to produce the dynamic library.
> > The static libraries are just collections of *.o files, so I don't see
> > why that shouldn't work.
>
> I don't think there is a reason why it *wouldn't* work; it's just that one
> needs
> to do "gcc -shared" to make a .so and that flag also tells the linker to
> choose
> .so's over .a's for the libraries that get linked into the target .so.


The linker picks dynamic over static even without the -shared flag. I think
that putting the static libraries in their own directory *without* the
dynamic libraries might do the trick. Hmm... something to try.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070418/2659d0f1/attachment.html>

From robert.kern at gmail.com  Thu Apr 19 00:26:32 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 18 Apr 2007 23:26:32 -0500
Subject: [Numpy-discussion] building numpy with atlas on ubuntu edgy
In-Reply-To: <e06186140704182120p3506f1f0k2de6ee909940832c@mail.gmail.com>
References: <evpm71$q6c$1@sea.gmane.org>
	<f01nug$nns$1@sea.gmane.org>	<4624E684.9010606@astraw.com>	<4625879C.6000706@ar.media.kyoto-u.ac.jp>	<f4f93d420704180913k56ee3925y2f380000e4169b06@mail.gmail.com>	<e06186140704181000t7d83280dn60059f0e8d68825c@mail.gmail.com>	<46266A13.5010101@gmail.com>
	<4626D851.9000602@ar.media.kyoto-u.ac.jp>	<e06186140704181957r4dc4ac9fiac69553740611f45@mail.gmail.com>	<4626DB9C.1030602@gmail.com>
	<e06186140704182120p3506f1f0k2de6ee909940832c@mail.gmail.com>
Message-ID: <4626EF78.8030906@gmail.com>

Charles R Harris wrote:
> 
> On 4/18/07, *Robert Kern* <robert.kern at gmail.com
> <mailto:robert.kern at gmail.com>> wrote:
> 
>     Charles R Harris wrote:
>     > I'm wondering if the static libraries could simply be compiled
>     with the
>     > -fPIC flag and linked with the program to produce the dynamic library.
>     > The static libraries are just collections of *.o files, so I don't
>     see
>     > why that shouldn't work.
> 
>     I don't think there is a reason why it *wouldn't* work; it's just
>     that one needs
>     to do "gcc -shared" to make a .so and that flag also tells the
>     linker to choose
>     .so's over .a's for the libraries that get linked into the target .so.
> 
> 
> The linker picks dynamic over static even without the -shared flag.

Oh sorry, using -static would normally make the linker choose static libraries
over shared libraries, but that only works when linking an executable. Of
course, David found the right way to bypass this and talk directly to the linker.

> I
> think that putting the static libraries in their own directory *without*
> the dynamic libraries might do the trick. Hmm... something to try.

That's usually the way I worked around it. Things may be difficult if your
shared libraries are installed to /usr/lib and you don't want to muck about as
root in there. IIRC (but I possibly don't), you'd have to remove the .so's from
the entire library search path in order to find the .a's.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Thu Apr 19 01:57:11 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 19 Apr 2007 07:57:11 +0200
Subject: [Numpy-discussion] Help using numPy to create a very large
	multi dimensional array
In-Reply-To: <f687be130704181609h13091e6y9e47b8ec9a45a5eb@mail.gmail.com>
References: <f687be130704181605t5ee447c4k52ca845b4a44ad41@mail.gmail.com>
	<f687be130704181609h13091e6y9e47b8ec9a45a5eb@mail.gmail.com>
Message-ID: <20070419055711.GA22826@clipper.ens.fr>

On Wed, Apr 18, 2007 at 06:09:38PM -0500, Nick Fotopoulos wrote:
> Devs, is there any possibility of moving/copying pylab.load to numpy?

+1

Ga?l



From wbaxter at gmail.com  Thu Apr 19 03:51:58 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Thu, 19 Apr 2007 16:51:58 +0900
Subject: [Numpy-discussion] C-API creating new copy of C data
Message-ID: <e86a5fd00704190051h4c583be4odd40075cd291194@mail.gmail.com>

What's the right way to make a new numpy array that's a copy of some C data?

There doesn't seem to be any API like PyArray_NewFromDescr that
/copies/ the void*data pointer for you.  Do I have to write my own
loops for this?  I can do that, it just seems like it should be a
library function already, so I'm guessing I'm just overlooking it.
There seem to be lots of APIs that will wrap pre-existing memory, but
the ones that allocate for you do not seem to copy.

A related question -- I'm only trying to copy in order to save myself
a little hassle regarding how to clean up the allocated chunks.  If
there's some simple way to trigger a particular deallocation function
to be called at the right time, then that would be the ideal, really.
Does that exist?

Thanks!
--bb


From matthieu.brucher at gmail.com  Thu Apr 19 03:52:15 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 19 Apr 2007 09:52:15 +0200
Subject: [Numpy-discussion] How to call methods from a class with custom
	matrices parameters with numpy arrays ?
Message-ID: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>

Hi,

I want to wrap some code I've done in the past with a custom array and pass
numpy arrays to it.
So I need to transform numpy arrays to my arrays at the construction of an
instance of my class, as well as each call to a method (pass by value).
Then, some method return by value an array I have to transform back in numpy
arrays. I imagine that such wrappers must be done in the .i file...
I'm still new to SWIG, even though I understand the easy wrapping, such
wrapping is a bit difficult for me at the moment. I do not understand for
instance the numpy.i wrappers, what they do, ... Is there a simpler doc
somewhere to explain this ? Or a simple SWIG tutorial and a Numpy interface
tutorial - Travis' book ? - ?

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070419/d3eb69e9/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Thu Apr 19 03:57:02 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 19 Apr 2007 16:57:02 +0900
Subject: [Numpy-discussion] How to call methods from a class with custom
 matrices parameters with numpy arrays ?
In-Reply-To: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>
References: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>
Message-ID: <462720CE.50801@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
> Hi,
>
> I want to wrap some code I've done in the past with a custom array and 
> pass numpy arrays to it.
> So I need to transform numpy arrays to my arrays at the construction 
> of an instance of my class, as well as each call to a method (pass by 
> value). Then, some method return by value an array I have to transform 
> back in numpy arrays. I imagine that such wrappers must be done in the 
> .i file...
> I'm still new to SWIG, even though I understand the easy wrapping, 
> such wrapping is a bit difficult for me at the moment. I do not 
> understand for instance the numpy.i wrappers, what they do, ... Is 
> there a simpler doc somewhere to explain this ? Or a simple SWIG 
> tutorial and a Numpy interface tutorial - Travis' book ? - ?
>
> Matthieu
>   
Hello Matthieu :)

    You should give ctypes a try, I find it much better than swig most 
of the time for wrapping. You can find some doc here:

    http://www.scipy.org/Cookbook/Ctypes2

    Basically, once you get your dll/so with a function foo(double *a, 
int n), you can call it directly in numpy by passing directly a numpy 
array. The catch is that the layout of the array should be exactly the 
same than your function expects (most of the time a contiguous array), 
but there are numpy functions which enforce that. As long as you are not 
calling thousand of function with small arrays, the wrapping is pretty 
efficient in my experience.

    cheers,

    David


From oliphant.travis at ieee.org  Thu Apr 19 04:17:09 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Thu, 19 Apr 2007 02:17:09 -0600
Subject: [Numpy-discussion] Help using numPy to create a very
 large	multi dimensional array
In-Reply-To: <f687be130704181609h13091e6y9e47b8ec9a45a5eb@mail.gmail.com>
References: <f687be130704181605t5ee447c4k52ca845b4a44ad41@mail.gmail.com>
	<f687be130704181609h13091e6y9e47b8ec9a45a5eb@mail.gmail.com>
Message-ID: <46272585.2090004@ieee.org>

Nick Fotopoulos wrote:
> Devs, is there any possibility of moving/copying pylab.load to numpy?
> I don't see anything in the source that requires the rest of
> matplotlib.  Among convenience functions, I think that this function
> ranks pretty highly in convenience.
>   

I'm supportive of this.  But, it can't be named numpy.load.

How about

numpy.loadtxt
numpy.savetxt

-Travis



From matthieu.brucher at gmail.com  Thu Apr 19 04:09:57 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 19 Apr 2007 10:09:57 +0200
Subject: [Numpy-discussion] How to call methods from a class with custom
	matrices parameters with numpy arrays ?
In-Reply-To: <462720CE.50801@ar.media.kyoto-u.ac.jp>
References: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>
	<462720CE.50801@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0704190109t1f66f35fv309e8a9f65415783@mail.gmail.com>

>
>     You should give ctypes a try, I find it much better than swig most
> of the time for wrapping. You can find some doc here:
>
>     http://www.scipy.org/Cookbook/Ctypes2
>
>     Basically, once you get your dll/so with a function foo(double *a,
> int n), you can call it directly in numpy by passing directly a numpy
> array. The catch is that the layout of the array should be exactly the
> same than your function expects (most of the time a contiguous array),
> but there are numpy functions which enforce that. As long as you are not
> calling thousand of function with small arrays, the wrapping is pretty
> efficient in my experience.
>
>     cheers,
>
>     David



Thanks for the fast answer ;)
I was wondering if ctypes would fit better, but in this case, I have to make
the wrapper myself, I suppose ?
Here is the basic prototype of what I have - it's a cost function I want to
optimize with optimizers I proposed on scipy recently - :

template<class MatrixType>
struct CostFunction
{
  CostFunction(const MatrixType& points, ...)
  {
  }

  const MatrixType gradient(const Matrixtype& parameters)
  {
    // ...
    return aNewGradient;
  }
};

So I have to create a function that takes a numpy array and that returns and
instance of my cost function, this is pretty simple to do, the matrix can be
constructed from its two dimensions and a pointer, but for the gradient
method, I have trouble seeing how to do wrap it :(

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070419/99bea961/attachment.html>

From faltet at carabos.com  Thu Apr 19 06:34:40 2007
From: faltet at carabos.com (Francesc Altet)
Date: Thu, 19 Apr 2007 12:34:40 +0200
Subject: [Numpy-discussion]
	=?iso-8859-1?q?Help_using_numPy_to_create_a_ve?=
	=?iso-8859-1?q?ry_large=09multi_dimensional_array?=
In-Reply-To: <46272585.2090004@ieee.org>
References: <f687be130704181605t5ee447c4k52ca845b4a44ad41@mail.gmail.com>
	<f687be130704181609h13091e6y9e47b8ec9a45a5eb@mail.gmail.com>
	<46272585.2090004@ieee.org>
Message-ID: <200704191234.40171.faltet@carabos.com>

A Dijous 19 Abril 2007 10:17, Travis Oliphant escrigu?:
> Nick Fotopoulos wrote:
> > Devs, is there any possibility of moving/copying pylab.load to numpy?
> > I don't see anything in the source that requires the rest of
> > matplotlib.  Among convenience functions, I think that this function
> > ranks pretty highly in convenience.
>
> I'm supportive of this.  But, it can't be named numpy.load.
>
> How about
>
> numpy.loadtxt
> numpy.savetxt

+1

-- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"


From david at ar.media.kyoto-u.ac.jp  Thu Apr 19 07:15:04 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 19 Apr 2007 20:15:04 +0900
Subject: [Numpy-discussion] How to call methods from a class with custom
 matrices parameters with numpy arrays ?
In-Reply-To: <e76aa17f0704190109t1f66f35fv309e8a9f65415783@mail.gmail.com>
References: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>	<462720CE.50801@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190109t1f66f35fv309e8a9f65415783@mail.gmail.com>
Message-ID: <46274F38.9060204@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
>         You should give ctypes a try, I find it much better than swig most
>     of the time for wrapping. You can find some doc here:
>
>         http://www.scipy.org/Cookbook/Ctypes2
>
>         Basically, once you get your dll/so with a function foo(double *a,
>     int n), you can call it directly in numpy by passing directly a numpy
>     array. The catch is that the layout of the array should be exactly the
>     same than your function expects (most of the time a contiguous array),
>     but there are numpy functions which enforce that. As long as you
>     are not
>     calling thousand of function with small arrays, the wrapping is pretty
>     efficient in my experience.
>
>         cheers,
>
>         David
>
>
>
> Thanks for the fast answer ;)
> I was wondering if ctypes would fit better, but in this case, I have 
> to make the wrapper myself, I suppose ?
> Here is the basic prototype of what I have - it's a cost function I 
> want to optimize with optimizers I proposed on scipy recently - :
>
> template<class MatrixType>
> struct CostFunction
> {
>   CostFunction(const MatrixType& points, ...)
>   {
>   }
>
>   const MatrixType gradient(const Matrixtype& parameters)
>   {
>     // ...
>     return aNewGradient;
>   }
> };
>
> So I have to create a function that takes a numpy array and that 
> returns and instance of my cost function, this is pretty simple to do, 
> the matrix can be constructed from its two dimensions and a pointer, 
> but for the gradient method, I have trouble seeing how to do wrap it :(
I forgot you are using C++. Using C++ with ctypes makes the wrapping 
more difficult, I think, specially since you are using templates: you 
would have to write a C wrapper around your classes, and then using 
ctypes on the wrapper (there may be better ways, but I don't see an 
obvious one). Basically, using ctypes is like using code from a 
dynamically loaded library (ala VST :) ). But writing some C code kills 
one of the strong point of ctypes (doing all the wrapping in python).

I have attached a simplistic example which show how to wrap some functions.

David
-------------- next part --------------
A non-text attachment was scrubbed...
Name: example.tbz2
Type: application/x-bzip-compressed-tar
Size: 4581 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070419/90314be3/attachment.bin>

From numpy at mspacek.mm.st  Thu Apr 19 07:32:08 2007
From: numpy at mspacek.mm.st (Martin Spacek)
Date: Thu, 19 Apr 2007 04:32:08 -0700
Subject: [Numpy-discussion] building with MKL in windows
Message-ID: <46275338.10803@mspacek.mm.st>

Does anyone know the right way to get numpy to build on windows using
Intel's MKL for LAPACK and BLAS libraries, under MSVC7.1?

I just did a whole lot of trial-and-error getting it to build. I
downloaded and installed MKL for windows from
http://www.intel.com/cd/software/products/asmo-na/eng/307757.htm

Here's my ~/.numpy-site.py file:

[mkl]
library_dirs = C:\bin\Intel\MKL\9.0\ia32\lib
mkl_libs = mkl_lapack, mkl_ia32, mkl_c_dll, libguide40
include_dirs = C:\bin\Intel\MKL\9.0\include

Most people would need to replace "C:\bin" with C:\Program Files". I
found that I could specify mkl_c instead of mkl_c_dll, and it would
still build. I found I could also specify libguide instead of libguide40.

Also, I found I had to remove the 2 lines in distutils/system_info.py
that mention pthread, mkl_lapack32, mkl_lapack64 (see attached patch)
since libraries with such names don't seem to exist in the MKL for
windows and were generating linking errors.

This obviously isn't the right thing to do, and I don't know enough to
know how to do it properly, but after running "python setup.py install"
it builds without errors, and seems to work:

  >>> numpy.version.version
'1.0.3.dev3719'
  >>> numpy.test()
    Found 10 tests for numpy.core.defmatrix
    Found 36 tests for numpy.core.ma
    Found 198 tests for numpy.core.multiarray
    Found 59 tests for numpy.core.numeric
    Found 31 tests for numpy.core.numerictypes
    Found 9 tests for numpy.core.records
    Found 5 tests for numpy.core.scalarmath
    Found 13 tests for numpy.core.umath
    Found 4 tests for numpy.ctypeslib
    Found 5 tests for numpy.distutils.misc_util
    Found 1 tests for numpy.fft.fftpack
    Found 3 tests for numpy.fft.helper
    Found 9 tests for numpy.lib.arraysetops
    Found 44 tests for numpy.lib.function_base
    Found 3 tests for numpy.lib.getlimits
    Found 4 tests for numpy.lib.index_tricks
    Found 2 tests for numpy.lib.polynomial
    Found 48 tests for numpy.lib.shape_base
    Found 13 tests for numpy.lib.twodim_base
    Found 42 tests for numpy.lib.type_check
    Found 1 tests for numpy.lib.ufunclike
    Found 32 tests for numpy.linalg
    Found 0 tests for __main__
........................................................................................................................
........................................................................................................................
........................................................................................................................
........................................................................................................................
............................................................................................
----------------------------------------------------------------------
Ran 572 tests in 0.641s

OK
<unittest.TextTestRunner object at 0x0131B8F0>
  >>> numpy.show_config()
lapack_opt_info:
      libraries = ['mkl_lapack', 'mkl_ia32', 'mkl_c_dll', 'libguide40']
      library_dirs = ['C:\\bin\\Intel\\MKL\\9.0\\ia32\\lib']
      define_macros = [('SCIPY_MKL_H', None)]
      include_dirs = ['C:\\bin\\Intel\\MKL\\9.0\\include',
'C:\\bin\\Python24\\include']

blas_opt_info:
      libraries = ['mkl_lapack', 'mkl_ia32', 'mkl_c_dll', 'libguide40']
      library_dirs = ['C:\\bin\\Intel\\MKL\\9.0\\ia32\\lib']
      define_macros = [('SCIPY_MKL_H', None)]
      include_dirs = ['C:\\bin\\Intel\\MKL\\9.0\\include',
'C:\\bin\\Python24\\include']

lapack_mkl_info:
      libraries = ['mkl_lapack', 'mkl_ia32', 'mkl_c_dll', 'libguide40']
      library_dirs = ['C:\\bin\\Intel\\MKL\\9.0\\ia32\\lib']
      define_macros = [('SCIPY_MKL_H', None)]
      include_dirs = ['C:\\bin\\Intel\\MKL\\9.0\\include',
'C:\\bin\\Python24\\include']

blas_mkl_info:
      libraries = ['mkl_lapack', 'mkl_ia32', 'mkl_c_dll', 'libguide40']
      library_dirs = ['C:\\bin\\Intel\\MKL\\9.0\\ia32\\lib']
      define_macros = [('SCIPY_MKL_H', None)]
      include_dirs = ['C:\\bin\\Intel\\MKL\\9.0\\include',
'C:\\bin\\Python24\\include']

mkl_info:
      libraries = ['mkl_lapack', 'mkl_ia32', 'mkl_c_dll', 'libguide40']
      library_dirs = ['C:\\bin\\Intel\\MKL\\9.0\\ia32\\lib']
      define_macros = [('SCIPY_MKL_H', None)]
      include_dirs = ['C:\\bin\\Intel\\MKL\\9.0\\include',
'C:\\bin\\Python24\\include']


Anyone have any comments? Should I add this to the wiki at
Installing_SciPy/Windows? Ideally, what should mkl_libs be set to in the
~/.numpy.site.cfg file? Here's Intel's description of all the files in
the ia32\lib directory:


mkl_c.lib    cdecl interface library
mkl_s.lib   CVF default interface library
mkl_c_dll.lib   cdecl interface library for dynamic library
mkl_s_dll.lib   CVF default interface library for dynamic library
mkl_lapack.lib  LAPACK routines and drivers
mkl_solver.lib  Sparse solver routines
mkl_ia32.lib    Optimized kernels (BLAS, cblas, Sparse BLAS, GMP, FFTs,
      DFTs, VML, VSL, interval arithmetic) for 32-bit applications
libguide40.lib  Interface library for dynamic threading library
libguide.lib    Static threading library


Cheers,

Martin

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mkl_windows.patch
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070419/97742162/attachment.ksh>

From matthieu.brucher at gmail.com  Thu Apr 19 08:29:46 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 19 Apr 2007 14:29:46 +0200
Subject: [Numpy-discussion] How to call methods from a class with custom
	matrices parameters with numpy arrays ?
In-Reply-To: <46274F38.9060204@ar.media.kyoto-u.ac.jp>
References: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>
	<462720CE.50801@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190109t1f66f35fv309e8a9f65415783@mail.gmail.com>
	<46274F38.9060204@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0704190529w4bee37c0tebb5b8fb46ff9bb3@mail.gmail.com>

>
> I forgot you are using C++.



Yes, all my code is in C++, and in fact the code I use in Python should be
object-oriented as well, that's why it is not that easy to define how I'm
going to do this...


Using C++ with ctypes makes the wrapping
> more difficult, I think, specially since you are using templates: you
> would have to write a C wrapper around your classes, and then using
> ctypes on the wrapper (there may be better ways, but I don't see an
> obvious one). Basically, using ctypes is like using code from a
> dynamically loaded library (ala VST :) ).



That works great for C then, not that well for C++...


But writing some C code kills
> one of the strong point of ctypes (doing all the wrapping in python).



Yes, but even with SWIG, I think I'll have to write wrapper in C or in SWIG
to transform one array in my matrix and vice versa. Well, that should be a
great example for my book ;)


I have attached a simplistic example which show how to wrap some functions.


Many thanks, it is simple, I only have to provide the wrappers :| and I have
to do something with the class instance :| I'll think a little bit about how
to do this properly and efficiently


Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070419/3d9b9d6f/attachment.html>

From emanuele at relativita.com  Thu Apr 19 09:23:24 2007
From: emanuele at relativita.com (Emanuele Olivetti)
Date: Thu, 19 Apr 2007 15:23:24 +0200
Subject: [Numpy-discussion] histogram2d bug?
Message-ID: <46276D4C.7020001@relativita.com>

While using histogram2d on simple examples I got these errors:

import numpy
x = numpy.array([0,0])
y = numpy.array([0,1])
numpy.histogram2d(x,y,bins=[2,2])
-----------------------------------------------------------------
Warning: divide by zero encountered in log10
---------------------------------------------------------------------------
exceptions.OverflowError                             Traceback (most
recent call last)

/home/ele/<ipython console>

/usr/lib/python2.4/site-packages/numpy/lib/twodim_base.py in
histogram2d(x, y, bins, range, normed, weights)
    180     if N != 1 and N != 2:
    181         xedges = yedges = asarray(bins, float)
    182         bins = [xedges, yedges]
--> 183     hist, edges = histogramdd([x,y], bins, range, normed, weights)
    184     return hist, edges[0], edges[1]

/usr/lib/python2.4/site-packages/numpy/lib/function_base.py in
histogramdd(sample, bins, range, normed, weights)
    206         decimal = int(-log10(dedges[i].min())) +6
    207         # Find which points are on the rightmost edge.
--> 208         on_edge = where(around(sample[:,i], decimal) ==
around(edges[i][-1], decimal))[0]
    209         # Shift these points one bin to the left.
    210         Ncount[i][on_edge] -= 1

/usr/lib/python2.4/site-packages/numpy/core/fromnumeric.py in round_(a,
decimals, out)
    687     except AttributeError:
    688         return _wrapit(a, 'round', decimals, out)
--> 689     return round(decimals, out)
    690
    691 around = round_

OverflowError: long int too large to convert to int
-----------------

numpy.__version__
'1.0.3.dev3719'

Hope this report helps,

Emanuele



From dalcinl at gmail.com  Thu Apr 19 09:50:27 2007
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 19 Apr 2007 10:50:27 -0300
Subject: [Numpy-discussion] Help using numPy to create a very large
	multi dimensional array
In-Reply-To: <46272585.2090004@ieee.org>
References: <f687be130704181605t5ee447c4k52ca845b4a44ad41@mail.gmail.com>
	<f687be130704181609h13091e6y9e47b8ec9a45a5eb@mail.gmail.com>
	<46272585.2090004@ieee.org>
Message-ID: <e7ba66e40704190650t640d7c33ve1abc0263b2922d8@mail.gmail.com>

On 4/19/07, Travis Oliphant <oliphant.travis at ieee.org> wrote:
> Nick Fotopoulos wrote:
> > Devs, is there any possibility of moving/copying pylab.load to numpy?
> > I don't see anything in the source that requires the rest of
> > matplotlib.  Among convenience functions, I think that this function
> > ranks pretty highly in convenience.
> >
>
> I'm supportive of this.  But, it can't be named numpy.load.
>

I am also +1 on this, but this functionality should be implemented in
C, I think. I've just tested numpy.fromfile('name.txt', sep=' ')
against pylab.load('name.txt') for a 35MB text file, the number are:

numpy.fromfile: 2.66 sec.
pylab.load:  16.64 sec.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594


From emanuele at relativita.com  Thu Apr 19 09:53:18 2007
From: emanuele at relativita.com (Emanuele Olivetti)
Date: Thu, 19 Apr 2007 15:53:18 +0200
Subject: [Numpy-discussion] histogram2d bug?
In-Reply-To: <46276D4C.7020001@relativita.com>
References: <46276D4C.7020001@relativita.com>
Message-ID: <4627744E.9000502@relativita.com>

An even simpler example generating the same error:

import numpy
x = numpy.array([0,0])
numpy.histogram2d(x,x)


HTH,

Emanuele

Emanuele Olivetti wrote:
> While using histogram2d on simple examples I got these errors:
>
> import numpy
> x = numpy.array([0,0])
> y = numpy.array([0,1])
> numpy.histogram2d(x,y,bins=[2,2])
> -----------------------------------------------------------------
> Warning: divide by zero encountered in log10
> ---------------------------------------------------------------------------
> exceptions.OverflowError                             Traceback (most
> recent call last)
>
> /home/ele/<ipython console>
>
> /usr/lib/python2.4/site-packages/numpy/lib/twodim_base.py in
> histogram2d(x, y, bins, range, normed, weights)
>     180     if N != 1 and N != 2:
>     181         xedges = yedges = asarray(bins, float)
>     182         bins = [xedges, yedges]
> --> 183     hist, edges = histogramdd([x,y], bins, range, normed, weights)
>     184     return hist, edges[0], edges[1]
>
> /usr/lib/python2.4/site-packages/numpy/lib/function_base.py in
> histogramdd(sample, bins, range, normed, weights)
>     206         decimal = int(-log10(dedges[i].min())) +6
>     207         # Find which points are on the rightmost edge.
> --> 208         on_edge = where(around(sample[:,i], decimal) ==
> around(edges[i][-1], decimal))[0]
>     209         # Shift these points one bin to the left.
>     210         Ncount[i][on_edge] -= 1
>
> /usr/lib/python2.4/site-packages/numpy/core/fromnumeric.py in round_(a,
> decimals, out)
>     687     except AttributeError:
>     688         return _wrapit(a, 'round', decimals, out)
> --> 689     return round(decimals, out)
>     690
>     691 around = round_
>
> OverflowError: long int too large to convert to int
> -----------------
>
> numpy.__version__
> '1.0.3.dev3719'
>
> Hope this report helps,
>
> Emanuele
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   



From lbolla at gmail.com  Thu Apr 19 11:34:53 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Thu, 19 Apr 2007 17:34:53 +0200
Subject: [Numpy-discussion] Problem with roots and complex coefficients
Message-ID: <80c99e790704190834q2b9e9233w5f331ed2fc13904e@mail.gmail.com>

dear all,
I've some problems with numpy.roots.
take a look at the following code:

========================================
import numpy

OK = numpy.roots([1, 1, 1])
OK = numpy.roots([1j, 1])
KO = numpy.roots([1, 1j, 1])
========================================

it fails with this error message, trying to execute the last line:

TypeError: can't convert complex to float; use
abs(z)/usr/lib/python2.4/site-packages/numpy/lib/polynomial.py in roots(p)
    119     if N > 1:
    120         # build companion matrix and find its eigenvalues (the
roots)
--> 121         A = diag(NX.ones((N-2,), p.dtype), -1)
    122         A[0, :] = -p[1:] / p[0]
    123         roots = _eigvals(A)

/usr/lib/python2.4/site-packages/numpy/lib/twodim_base.py in diag(v, k)
     66             i = arange(0,n+k)
     67             fi = i+(i-k)*n
---> 68         res.flat[fi] = v
     69         return res
     70     elif len(s)==2:

TypeError: can't convert complex to float; use abs(z)

any ideas?
thanks,
Lorenzo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070419/86ac5516/attachment.html>

From nwagner at iam.uni-stuttgart.de  Thu Apr 19 11:44:17 2007
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 19 Apr 2007 17:44:17 +0200
Subject: [Numpy-discussion] Problem with roots and complex coefficients
In-Reply-To: <80c99e790704190834q2b9e9233w5f331ed2fc13904e@mail.gmail.com>
References: <80c99e790704190834q2b9e9233w5f331ed2fc13904e@mail.gmail.com>
Message-ID: <46278E51.6090503@iam.uni-stuttgart.de>

lorenzo bolla wrote:
> dear all,
> I've some problems with numpy.roots.
> take a look at the following code:
>
> ========================================
> import numpy
>
> OK = numpy.roots([1, 1, 1])
> OK = numpy.roots([1j, 1])
> KO = numpy.roots([1, 1j, 1])
> ========================================
>
> it fails with this error message, trying to execute the last line:
>
> TypeError: can't convert complex to float; use
> abs(z)/usr/lib/python2.4/site-packages/numpy/lib/polynomial.py in
> roots(p)
>     119     if N > 1:
>     120         # build companion matrix and find its eigenvalues (the
> roots)
> --> 121         A = diag(NX.ones((N-2,), p.dtype), -1)
>     122         A[0, :] = -p[1:] / p[0]
>     123         roots = _eigvals(A)
>
> /usr/lib/python2.4/site-packages/numpy/lib/twodim_base.py in diag(v, k)
>      66             i = arange(0,n+k)
>      67             fi = i+(i-k)*n
> ---> 68         res.flat[fi] = v
>      69         return res
>      70     elif len(s)==2:
>
> TypeError: can't convert complex to float; use abs(z)
>
> any ideas?
> thanks,
> Lorenzo
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   
Works fine for me. Maybe you are using an old numpy version.

>>> from numpy import *
>>> roots([1, 1, 1])
array([-0.5+0.8660254j, -0.5-0.8660254j])
>>> roots([1j, 1])
array([-0.+1.j])
>>> roots([1, 1j, 1])
array([  0.00000000e+00-1.61803399j,   1.38777878e-17+0.61803399j])
>>> import numpy
>>> numpy.__version__
'1.0.3.dev3716'


Nils



From Chris.Barker at noaa.gov  Thu Apr 19 11:58:45 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 19 Apr 2007 08:58:45 -0700
Subject: [Numpy-discussion] Help using numPy to create a very
 large	multi dimensional array
In-Reply-To: <e7ba66e40704190650t640d7c33ve1abc0263b2922d8@mail.gmail.com>
References: <f687be130704181605t5ee447c4k52ca845b4a44ad41@mail.gmail.com>
	<f687be130704181609h13091e6y9e47b8ec9a45a5eb@mail.gmail.com>
	<46272585.2090004@ieee.org>
	<e7ba66e40704190650t640d7c33ve1abc0263b2922d8@mail.gmail.com>
Message-ID: <462791B5.6010101@noaa.gov>

Lisandro Dalcin wrote:
> I am also +1 on this, but this functionality should be implemented in
> C, I think.

well,  maybe.

> I've just tested numpy.fromfile('name.txt', sep=' ')
> against pylab.load('name.txt') for a 35MB text file, the number are:
> 
> numpy.fromfile: 2.66 sec.
> pylab.load:  16.64 sec.

exactly that's expected. fromfile is designed to do the easy cases as 
fast as possible, pylab.load is designed to be be flexible, I'm not user 
you need both the speed and flexibility at the same time.

By the way, I haven't looked at pylab.load() for a while, but it could 
perhaps be sped up by using fromfile() and or fromstring internally. 
There may be some opportunity to special case the easy ones too (i.e. 
all columns desired, etc.)

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From spacey-numpy-discussion at lenin.net  Thu Apr 19 11:59:05 2007
From: spacey-numpy-discussion at lenin.net (Peter C. Norton)
Date: Thu, 19 Apr 2007 08:59:05 -0700
Subject: [Numpy-discussion] further questions: numpy on Solaris x86 with sun
	CC and libsunperf
In-Reply-To: <20070418164844.GG5812@lenin.net>
References: <20070418164844.GG5812@lenin.net>
Message-ID: <20070419155905.GI5812@lenin.net>

I have found a way to build numpy on solaris x86 with
libsunperf. Basically, using the static library, and removing the
compiler flag for libf77compat (I think it's deprecated and has been
removed) and furthermore symlinking liblapack.a and libblas.a to the actual
libsunperf.a seems to result in a successful build.

However, running numpy.test() results in the following error:

....................................................................................F........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
======================================================================
FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "[long path removed]/python-2.5.1c1/lib/python2.5/site-packages/numpy/core/tests/test_scalarmath.py", line 46, in check_large_types
    assert b == 6765201, "error with %r: got %r" % (t,b)
AssertionError: error with <type 'numpy.float96'>: got 6765201.00000000000364

----------------------------------------------------------------------
Ran 573 tests in 1.462s

FAILED (failures=1)
<unittest.TextTestRunner object at 0x84a998c>


Can anyone offer an opinion as to the severity of this failure?

Thanks,

-Peter

-- 
The 5 year plan:
In five years we'll make up another plan.
Or just re-use this one.



From lbolla at gmail.com  Thu Apr 19 12:02:05 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Thu, 19 Apr 2007 18:02:05 +0200
Subject: [Numpy-discussion] Problem with roots and complex coefficients
In-Reply-To: <46278E51.6090503@iam.uni-stuttgart.de>
References: <80c99e790704190834q2b9e9233w5f331ed2fc13904e@mail.gmail.com>
	<46278E51.6090503@iam.uni-stuttgart.de>
Message-ID: <80c99e790704190902t5c882aban1bc18c619f5dcb44@mail.gmail.com>

updated.
now it works. many thanks.
L.

On 4/19/07, Nils Wagner <nwagner at iam.uni-stuttgart.de> wrote:
>
> lorenzo bolla wrote:
> > dear all,
> > I've some problems with numpy.roots.
> > take a look at the following code:
> >
> > ========================================
> > import numpy
> >
> > OK = numpy.roots([1, 1, 1])
> > OK = numpy.roots([1j, 1])
> > KO = numpy.roots([1, 1j, 1])
> > ========================================
> >
> > it fails with this error message, trying to execute the last line:
> >
> > TypeError: can't convert complex to float; use
> > abs(z)/usr/lib/python2.4/site-packages/numpy/lib/polynomial.py in
> > roots(p)
> >     119     if N > 1:
> >     120         # build companion matrix and find its eigenvalues (the
> > roots)
> > --> 121         A = diag(NX.ones((N-2,), p.dtype), -1)
> >     122         A[0, :] = -p[1:] / p[0]
> >     123         roots = _eigvals(A)
> >
> > /usr/lib/python2.4/site-packages/numpy/lib/twodim_base.py in diag(v, k)
> >      66             i = arange(0,n+k)
> >      67             fi = i+(i-k)*n
> > ---> 68         res.flat[fi] = v
> >      69         return res
> >      70     elif len(s)==2:
> >
> > TypeError: can't convert complex to float; use abs(z)
> >
> > any ideas?
> > thanks,
> > Lorenzo
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> Works fine for me. Maybe you are using an old numpy version.
>
> >>> from numpy import *
> >>> roots([1, 1, 1])
> array([-0.5+0.8660254j, -0.5-0.8660254j])
> >>> roots([1j, 1])
> array([-0.+1.j])
> >>> roots([1, 1j, 1])
> array([  0.00000000e+00-1.61803399j,   1.38777878e-17+0.61803399j])
> >>> import numpy
> >>> numpy.__version__
> '1.0.3.dev3716'
>
>
> Nils
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070419/7bbd5f9c/attachment.html>

From cournape at gmail.com  Thu Apr 19 12:03:00 2007
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 20 Apr 2007 01:03:00 +0900
Subject: [Numpy-discussion] How to call methods from a class with custom
	matrices parameters with numpy arrays ?
In-Reply-To: <e76aa17f0704190529w4bee37c0tebb5b8fb46ff9bb3@mail.gmail.com>
References: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>
	<462720CE.50801@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190109t1f66f35fv309e8a9f65415783@mail.gmail.com>
	<46274F38.9060204@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190529w4bee37c0tebb5b8fb46ff9bb3@mail.gmail.com>
Message-ID: <5b8d13220704190903gbb3ffd8wb29abfc32e67ff7b@mail.gmail.com>

On 4/19/07, Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
>
> > I forgot you are using C++.
>
>
> Yes, all my code is in C++, and in fact the code I use in Python should be
> object-oriented as well, that's why it is not that easy to define how I'm
> going to do this...
Doing the wrapping in an object oriented way is difficult, and maybe
not that useful. This does not prevent the API exposed to python to be
OO, of course.
>
> > Using C++ with ctypes makes the wrapping
> > more difficult, I think, specially since you are using templates: you
> > would have to write a C wrapper around your classes, and then using
> > ctypes on the wrapper (there may be better ways, but I don't see an
> > obvious one). Basically, using ctypes is like using code from a
> > dynamically loaded library (ala VST :) ).
>
>
> That works great for C then, not that well for C++...
Well, this is an inherent problem of C++ when you try to use it from
other languages, but let's not start this (again :) ),
>
> Yes, but even with SWIG, I think I'll have to write wrapper in C or in SWIG
> to transform one array in my matrix and vice versa. Well, that should be a
> great example for my book ;)
>
> > I have attached a simplistic example which show how to wrap some
> functions.
>
>
> Many thanks, it is simple, I only have to provide the wrappers :| and I have
> to do something with the class instance :| I'll think a little bit about how
> to do this properly and efficiently
>
The example shows basic wrapping, and some facilities provided by
numpy to help. Again, ctype is pretty efficient as long as you do not
need to do convertion. If you call it thousand of times, it will be
slow, but this is more or less inherent to python (function calls are
nowhere near as fast as in a compiled language, at least for now).

SWIG may be better in your case because it is aware of C++ classes,
and is *compiling* an extension, whereas ctypes does not compile
anything. This means that you do not have to care about binary
interfaces problemes between foreign object codes. SWIG parses a prett
good deal of C++, and is aware of classes (template is another matter,
obviously). numpy sources contain swig code to process automatically
numpy arrays (that is convert C representation of a numpy array to a
simple C array and vice et versa), if I remember correctly.

There is also boost.python, but I don't know if its situation towards
numpy array has changed (eg there was some code lying around to
represent numpy arrays in C++).

If I were you, and if there are only a few classes and a few member
functions, I would try the C wrapper called from python first.

David


From matthieu.brucher at gmail.com  Thu Apr 19 12:17:22 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 19 Apr 2007 18:17:22 +0200
Subject: [Numpy-discussion] How to call methods from a class with custom
	matrices parameters with numpy arrays ?
In-Reply-To: <5b8d13220704190903gbb3ffd8wb29abfc32e67ff7b@mail.gmail.com>
References: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>
	<462720CE.50801@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190109t1f66f35fv309e8a9f65415783@mail.gmail.com>
	<46274F38.9060204@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190529w4bee37c0tebb5b8fb46ff9bb3@mail.gmail.com>
	<5b8d13220704190903gbb3ffd8wb29abfc32e67ff7b@mail.gmail.com>
Message-ID: <e76aa17f0704190917l48af8b89l81adee237cebd2b9@mail.gmail.com>

>
> Doing the wrapping in an object oriented way is difficult, and maybe
> not that useful. This does not prevent the API exposed to python to be
> OO, of course.



I have some difficulties to do this in an automated way...
I'm trying now to make a derived object from my function, without templates
and, I'm hoping so, with a correct interface i.e. double + 2*int to save
make the conversions with numpy arrays.


> > That works great for C then, not that well for C++...
> Well, this is an inherent problem of C++ when you try to use it from
> other languages, but let's not start this (again :) ),



:D



> The example shows basic wrapping, and some facilities provided by
> numpy to help. Again, ctype is pretty efficient as long as you do not
> need to do convertion. If you call it thousand of times, it will be
> slow, but this is more or less inherent to python (function calls are
> nowhere near as fast as in a compiled language, at least for now).



It's for optimization, so the function will be called several hundreds of
times, I suppose, and I tried porting the whole function to Python, but I'm
not sure that the Python version behaves like the C++ version - the results
are not identic, so... -, thus the wrapping.


SWIG may be better in your case because it is aware of C++ classes,
> and is *compiling* an extension, whereas ctypes does not compile
> anything. This means that you do not have to care about binary
> interfaces problemes between foreign object codes. SWIG parses a prett
> good deal of C++, and is aware of classes (template is another matter,
> obviously). numpy sources contain swig code to process automatically
> numpy arrays (that is convert C representation of a numpy array to a
> simple C array and vice et versa), if I remember correctly.



Yes, I will try to use this solution, I have trouble figuring how passing
the output numpy array, Bill Baxter asked the same question today, at
exactly the same time ;) Well, I saw on the docs that such arrays must be
passed to the function, and already allocated, and that is a problem for
me...


There is also boost.python, but I don't know if its situation towards
> numpy array has changed (eg there was some code lying around to
> represent numpy arrays in C++).



That will be my next quest during the summer ;)


If I were you, and if there are only a few classes and a few member
> functions, I would try the C wrapper called from python first.



It's only one class and one method + the constructor. Not much but I'm a
real beginner in that domain. True, I could use the C API directly...


Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070419/3d41227a/attachment.html>

From ivilata at carabos.com  Thu Apr 19 12:50:04 2007
From: ivilata at carabos.com (Ivan Vilata i Balaguer)
Date: Thu, 19 Apr 2007 18:50:04 +0200
Subject: [Numpy-discussion] Four bugs in Numexpr and another repository
Message-ID: <20070419165004.GL12182@tardis.terramar.selidor.net>

Hi all,

Francesc and I have found four bugs in Numexpr while developing
PyTables.  To spare you from the gory details, I'll point you to the
PyTables Trac instance, where bugs are commented and patches are
available which should be more or less readily applicable to mainstream
Numexpr:

1. VariableNode error when numexpr is asked to evaluate some functions
   (arc*, pow): http://www.pytables.org/trac/ticket/54

   >>> numexpr.evaluate('arcsin(0.5)')
   Traceback (most recent call last):
     ...
   TypeError: 'VariableNode' object is not callable

2. 'int' object has no attribute '__gt__' error in tables.numexpr (also
   affects to original numexpr): http://www.pytables.org/trac/ticket/55

   >>> numexpr.evaluate('cos(1.3) < 0')
   Traceback (most recent call last):
     ...
   AttributeError: 'int' object has no attribute '__gt__'

3. Wrong behaviour of where function of numexpr (als affects to original
   numexpr): http://www.pytables.org/trac/ticket/56

   >>> a = numpy.arange(0,5)
   >>> numexpr.evaluate('where(a < 2, 1, 0)')
   Traceback (most recent call last):
     ...
   NotImplementedError: couldn't find matching opcode for 'where_bbbb'

4. Segmentation fault with string constant expression:
   http://www.pytables.org/trac/ticket/58

   >>> tables.numexpr.evaluate('"foo"')
   Segmentation fault (core dumped)

   This in fact only applies to the PyTables version of Numexpr, which
   includes the patches I've been sending here for a while.  I'm taking
   this opportunity to point out that you can check that version out
   from the PyTables Subversion repository:
   http://www.pytables.org/svn/pytables/trunk/tables/numexpr/

If you need more help on this, don't hesitate to ask.  Thanks a lot!

(Is it OK to report Numexpr bugs in the SciPy Trac instance?  And,
what's the right capitalisation of "Numexpr"? ;) )

::

	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
	       C?rabos Coop. V.  V  V   Enjoy Data
	                          ""
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 307 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070419/16c7b09c/attachment.sig>

From david.huard at gmail.com  Thu Apr 19 14:18:33 2007
From: david.huard at gmail.com (David Huard)
Date: Thu, 19 Apr 2007 14:18:33 -0400
Subject: [Numpy-discussion] histogram2d bug?
In-Reply-To: <4627744E.9000502@relativita.com>
References: <46276D4C.7020001@relativita.com> <4627744E.9000502@relativita.com>
Message-ID: <91cf711d0704191118nbd0217h19102f0a7c7cf1c@mail.gmail.com>

Hi Emanuele,

The bug is due to a part of the code that shifts the last bin's position to
make sure the array's maximum value is counted in the last bin, and not as
an outlier. To do so, the code computes an approximate precision used the
shift the bin edge by amount small compared to the array's value. In your
example, since all values in x are identical, the precision is ``infinite''.
So my question is, what kind of behaviour would you be expecting in this
case for the automatic placement of bin edges ?

That is, given
x : array of identical values, eg. [0, 0, 0, 0, 0, ..., 0]
smin, smax = x.min(), x.max()
How do you select the bin edges ?

One solution is to use the same scheme used by histogram:
if smin == smax:
    edges[i] = linspace(smin-.5, smax+.5, nbin[i]+1)

Would that be ok ?

David


 I'll submit a patch.

2007/4/19, Emanuele Olivetti <emanuele at relativita.com>:
>
> An even simpler example generating the same error:
>
> import numpy
> x = numpy.array([0,0])
> numpy.histogram2d(x,x)
>
>
> HTH,
>
> Emanuele
>
> Emanuele Olivetti wrote:
> > While using histogram2d on simple examples I got these errors:
> >
> > import numpy
> > x = numpy.array([0,0])
> > y = numpy.array([0,1])
> > numpy.histogram2d(x,y,bins=[2,2])
> > -----------------------------------------------------------------
> > Warning: divide by zero encountered in log10
> >
> ---------------------------------------------------------------------------
> > exceptions.OverflowError                             Traceback (most
> > recent call last)
> >
> > /home/ele/<ipython console>
> >
> > /usr/lib/python2.4/site-packages/numpy/lib/twodim_base.py in
> > histogram2d(x, y, bins, range, normed, weights)
> >     180     if N != 1 and N != 2:
> >     181         xedges = yedges = asarray(bins, float)
> >     182         bins = [xedges, yedges]
> > --> 183     hist, edges = histogramdd([x,y], bins, range, normed,
> weights)
> >     184     return hist, edges[0], edges[1]
> >
> > /usr/lib/python2.4/site-packages/numpy/lib/function_base.py in
> > histogramdd(sample, bins, range, normed, weights)
> >     206         decimal = int(-log10(dedges[i].min())) +6
> >     207         # Find which points are on the rightmost edge.
> > --> 208         on_edge = where(around(sample[:,i], decimal) ==
> > around(edges[i][-1], decimal))[0]
> >     209         # Shift these points one bin to the left.
> >     210         Ncount[i][on_edge] -= 1
> >
> > /usr/lib/python2.4/site-packages/numpy/core/fromnumeric.py in round_(a,
> > decimals, out)
> >     687     except AttributeError:
> >     688         return _wrapit(a, 'round', decimals, out)
> > --> 689     return round(decimals, out)
> >     690
> >     691 around = round_
> >
> > OverflowError: long int too large to convert to int
> > -----------------
> >
> > numpy.__version__
> > '1.0.3.dev3719'
> >
> > Hope this report helps,
> >
> > Emanuele
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070419/1e4ccd3b/attachment.html>

From cookedm at physics.mcmaster.ca  Thu Apr 19 16:03:45 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu, 19 Apr 2007 16:03:45 -0400
Subject: [Numpy-discussion] Four bugs in Numexpr and another repository
In-Reply-To: <20070419165004.GL12182@tardis.terramar.selidor.net>
References: <20070419165004.GL12182@tardis.terramar.selidor.net>
Message-ID: <4627CB21.80902@physics.mcmaster.ca>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ivan Vilata i Balaguer wrote:
> Hi all,
> 
> Francesc and I have found four bugs in Numexpr while developing
> PyTables.  To spare you from the gory details, I'll point you to the
> PyTables Trac instance, where bugs are commented and patches are
> available which should be more or less readily applicable to mainstream
> Numexpr:
> 
> 1. VariableNode error when numexpr is asked to evaluate some functions
>    (arc*, pow): http://www.pytables.org/trac/ticket/54
> 
>    >>> numexpr.evaluate('arcsin(0.5)')
>    Traceback (most recent call last):
>      ...
>    TypeError: 'VariableNode' object is not callable

Looks like an oversight. Fixed in svn (r2934).
> 
> 2. 'int' object has no attribute '__gt__' error in tables.numexpr (also
>    affects to original numexpr): http://www.pytables.org/trac/ticket/55
> 
>    >>> numexpr.evaluate('cos(1.3) < 0')
>    Traceback (most recent call last):
>      ...
>    AttributeError: 'int' object has no attribute '__gt__'

Fixed, r2935.

> 3. Wrong behaviour of where function of numexpr (als affects to original
>    numexpr): http://www.pytables.org/trac/ticket/56
> 
>    >>> a = numpy.arange(0,5)
>    >>> numexpr.evaluate('where(a < 2, 1, 0)')
>    Traceback (most recent call last):
>      ...
>    NotImplementedError: couldn't find matching opcode for 'where_bbbb'

Got to stare at this one. Support for True and False might be the best
thing here, so that the above is where_biii.

> 4. Segmentation fault with string constant expression:
>    http://www.pytables.org/trac/ticket/58
> 
>    >>> tables.numexpr.evaluate('"foo"')
>    Segmentation fault (core dumped)
> 
>    This in fact only applies to the PyTables version of Numexpr, which
>    includes the patches I've been sending here for a while.  I'm taking
>    this opportunity to point out that you can check that version out
>    from the PyTables Subversion repository:
>    http://www.pytables.org/svn/pytables/trunk/tables/numexpr/
> 
> If you need more help on this, don't hesitate to ask.  Thanks a lot!


> (Is it OK to report Numexpr bugs in the SciPy Trac instance?  And,
> what's the right capitalisation of "Numexpr"? ;) )

Yes; open a bug for #3 on the SciPy Trac, and assign them to me
(cookedm), so I don't forget about them. You can open a bug for adding
string expressions, so at least people can find it if they want. I'm
still not convinced of its usefulness (compared with the added complexity).

Looks like I use either numexpr or Numexpr for capitialisation (NumExpr
on the SciPy wiki, but that's just wikification).

- --
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGJ8shN9ixZKFWjRQRAvUpAJ9fjY434fuSokMXGZK8wHbv3O778gCgtl3q
Q/E8Lvmk/EyJNCwr66Vay2Y=
=muco
-----END PGP SIGNATURE-----


From cookedm at physics.mcmaster.ca  Thu Apr 19 16:14:26 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu, 19 Apr 2007 16:14:26 -0400
Subject: [Numpy-discussion] further questions: numpy on Solaris x86 with
 sun	CC and libsunperf
In-Reply-To: <20070419155905.GI5812@lenin.net>
References: <20070418164844.GG5812@lenin.net> <20070419155905.GI5812@lenin.net>
Message-ID: <4627CDA2.3030204@physics.mcmaster.ca>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter C. Norton wrote:
> I have found a way to build numpy on solaris x86 with
> libsunperf. Basically, using the static library, and removing the
> compiler flag for libf77compat (I think it's deprecated and has been
> removed) and furthermore symlinking liblapack.a and libblas.a to the actual
> libsunperf.a seems to result in a successful build.
> 
> However, running numpy.test() results in the following error:
> 
> ....................................................................................F........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
> ======================================================================
> FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "[long path removed]/python-2.5.1c1/lib/python2.5/site-packages/numpy/core/tests/test_scalarmath.py", line 46, in check_large_types
>     assert b == 6765201, "error with %r: got %r" % (t,b)
> AssertionError: error with <type 'numpy.float96'>: got 6765201.00000000000364
> 
> ----------------------------------------------------------------------
> Ran 573 tests in 1.462s
> 
> FAILED (failures=1)
> <unittest.TextTestRunner object at 0x84a998c>
> 
> 
> Can anyone offer an opinion as to the severity of this failure?

Only serious if you're using long doubles. I'm not sure where the extra
is coming from (it's about an error of 2^-60). Could be that Solaris's
support for long doubles isn't quite good enough.

- --
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGJ82hN9ixZKFWjRQRAhFtAJoDODg050+XppxgtfjV8iXeFEpXPQCeLuN4
fcfbssyxp0Mjry0KYnLt7p4=
=90If
-----END PGP SIGNATURE-----


From cookedm at physics.mcmaster.ca  Thu Apr 19 16:20:39 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu, 19 Apr 2007 16:20:39 -0400
Subject: [Numpy-discussion] Building numpy on Solaris x86 with sun CC
 and	libsunperf
In-Reply-To: <20070418164844.GG5812@lenin.net>
References: <20070418164844.GG5812@lenin.net>
Message-ID: <4627CF17.1080803@physics.mcmaster.ca>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter C. Norton wrote:
> Hello all,
> 
> I'm trying to build numpy for some of my users, and I can't seem to
> get the [blas_opt] or the [lapack_opt] settings to be honored in my
> site.cfg:
> 
> $ CFLAGS="-L$STUDIODIR/lib/ -l=sunperf" CPPFLAGS='-DNO_APPEND_FORTRAN' \
> /scratch/nortonp/python-2.5.1c1/bin/python setup.py config
> Running from numpy source directory.
> F2PY Version 2_3649
> blas_opt_info:
> blas_mkl_info:
>   libraries mkl,vml,guide not found in /usr/local/lib
>   libraries mkl,vml,guide not found in /lang/SunOS.5.i386/studio-11.0/SUNWspro/lib
>   NOT AVAILABLE
> [etc, with nothing found]
> 
> And after all this, I get
> /projects/python-2.5/numpy-1.0.2/numpy/distutils/system_info.py:1210:
> UserWarning:
>     Atlas (http://math-atlas.sourceforge.net/) libraries not found.
>     Directories to search for the libraries can be specified in the
>     numpy/distutils/site.cfg file (section [atlas]) or by setting
>     the ATLAS environment variable.
>   warnings.warn(AtlasNotFoundError.__doc__)
> 
> and a similar thing happens with lapack. My site.cfg boils down to
> this:
> 
> [DEFAULT]
> library_dirs = /usr/local/lib:/lang/SunOS.5.i386/studio-11.0/SUNWspro/lib
> include_dirs = /usr/local/include:/lang/SunOS.5.i386/studio-11.0/SUNWspro/include
> 
> [blas_opt]
> libraries = sunperf
> 
> [lapack_opt]
> libraries = sunperf
> 
> If I mess around with system_info.py I can get setup to acknowledge
> the addition to the list, but it seems from the output that the
> optimized libraries section in the site.cfg is ignored (eg. never
> added to the classes _lib_names array).
> 

Try this instead of blas_opt and lapack_opt:

[blas]
blas_libs = sunperf

[lapack]
lapack_libs = sunperf

> Also, since the lapack and blas libraries are already essentially part
> of libsunperf, built, do I still need a fortran compiler to link build
> numpy or can I just bypass that (somehow) and link the .so and go on
> my merry way?

I think you should be fine with a C compiler. You'd probably have to
fiddle with the definitions of the blas_info and lapack_info classes in
numpy.distutils.system_info -- they hardcode using Fortran.

At some point, system_info will get some more lovin' -- it should be
refactored to be more consistent.

- --
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGJ88XN9ixZKFWjRQRAlxtAJ41hBrT3XNLclomsnRV+v8pP+/CqACfXw4X
iAD00A86D+lj21Qo/p8p6Mk=
=/A7M
-----END PGP SIGNATURE-----


From cookedm at physics.mcmaster.ca  Thu Apr 19 16:38:12 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu, 19 Apr 2007 16:38:12 -0400
Subject: [Numpy-discussion] Problems building numpy and scipy on AIX
In-Reply-To: <34004.193.17.11.23.1176917180.squirrel@webmail.marquardt.sc>
References: <30420.193.17.11.23.1176898744.squirrel@webmail.marquardt.sc>	<46263741.7080403@physics.mcmaster.ca>
	<34004.193.17.11.23.1176917180.squirrel@webmail.marquardt.sc>
Message-ID: <4627D334.30800@physics.mcmaster.ca>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christian Marquardt wrote:
> Dear David,
> 
> the svn version of numpy does indeed build cleanly on AIX. Many thanks!
> 
> However, the wrapper problem still exists for the C++ compiler, and shows
> up when compiling scipy. Now, I *assume* that SciPy is using the distutils
> as installed by numpy. Do you know where the linker settings for the C++
> compiler might be overwritten? There are two or three C compiler related
> python modules in numpy/distutils... Or would you think that this problem
> is entirely unrelated to the distutils in numpy?

I'm working on a better solution, but the quick fix to your problem is
to look in numpy/distutils/command/build_ext.py. There are two lines
that reference self.compiler.linker_so[0]; change those 0s to a 1s, so
it keeps the ld_so_aix script there when switching the linker.

- --
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGJ9MzN9ixZKFWjRQRArNwAKC029wYORk9sm+FShcYKNd0UEcMdgCghHGC
rjYqtaESdt8zRgZHCDxYbDk=
=PS30
-----END PGP SIGNATURE-----


From wbaxter at gmail.com  Thu Apr 19 17:19:10 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Fri, 20 Apr 2007 06:19:10 +0900
Subject: [Numpy-discussion] How to call methods from a class with custom
	matrices parameters with numpy arrays ?
In-Reply-To: <e76aa17f0704190917l48af8b89l81adee237cebd2b9@mail.gmail.com>
References: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>
	<462720CE.50801@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190109t1f66f35fv309e8a9f65415783@mail.gmail.com>
	<46274F38.9060204@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190529w4bee37c0tebb5b8fb46ff9bb3@mail.gmail.com>
	<5b8d13220704190903gbb3ffd8wb29abfc32e67ff7b@mail.gmail.com>
	<e76aa17f0704190917l48af8b89l81adee237cebd2b9@mail.gmail.com>
Message-ID: <e86a5fd00704191419j3eb5792xa9955355f76be6ef@mail.gmail.com>

This may be of no help at all but I see mentions of C++/Python/OO/SWIG
and it triggers me to mention something I heard about recently called
"PyCXX":
    http://cxx.sourceforge.net/
I /think/ the idea behind it is to basically make a C++ version of the
Python C API.  You still do all the wrapping yourself, but it's much
less painful.

--bb

On 4/20/07, Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
>
> > Doing the wrapping in an object oriented way is difficult, and maybe
> > not that useful. This does not prevent the API exposed to python to be
> > OO, of course.
>
>
> I have some difficulties to do this in an automated way...
> I'm trying now to make a derived object from my function, without templates
> and, I'm hoping so, with a correct interface i.e. double + 2*int to save
> make the conversions with numpy arrays.
>
> > > That works great for C then, not that well for C++...
> > Well, this is an inherent problem of C++ when you try to use it from
> > other languages, but let's not start this (again :) ),
>
>
> :D
>
>
> > The example shows basic wrapping, and some facilities provided by
> > numpy to help. Again, ctype is pretty efficient as long as you do not
> > need to do convertion. If you call it thousand of times, it will be
> > slow, but this is more or less inherent to python (function calls are
> > nowhere near as fast as in a compiled language, at least for now).
>
>
> It's for optimization, so the function will be called several hundreds of
> times, I suppose, and I tried porting the whole function to Python, but I'm
> not sure that the Python version behaves like the C++ version - the results
> are not identic, so... -, thus the wrapping.
>
> > SWIG may be better in your case because it is aware of C++ classes,
> > and is *compiling* an extension, whereas ctypes does not compile
> > anything. This means that you do not have to care about binary
> > interfaces problemes between foreign object codes. SWIG parses a prett
> > good deal of C++, and is aware of classes (template is another matter,
> > obviously). numpy sources contain swig code to process automatically
> > numpy arrays (that is convert C representation of a numpy array to a
> > simple C array and vice et versa), if I remember correctly.
>
>
> Yes, I will try to use this solution, I have trouble figuring how passing
> the output numpy array, Bill Baxter asked the same question today, at
> exactly the same time ;) Well, I saw on the docs that such arrays must be
> passed to the function, and already allocated, and that is a problem for
> me...
>
> > There is also boost.python, but I don't know if its situation towards
> > numpy array has changed (eg there was some code lying around to
> > represent numpy arrays in C++).
>
>
> That will be my next quest during the summer ;)
>
> > If I were you, and if there are only a few classes and a few member
> > functions, I would try the C wrapper called from python first.
>
>
> It's only one class and one method + the constructor. Not much but I'm a
> real beginner in that domain. True, I could use the C API directly...
>
>
> Matthieu
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


From emanuele at relativita.com  Thu Apr 19 17:51:30 2007
From: emanuele at relativita.com (Emanuele Olivetti)
Date: Thu, 19 Apr 2007 23:51:30 +0200
Subject: [Numpy-discussion] histogram2d bug?
In-Reply-To: <91cf711d0704191118nbd0217h19102f0a7c7cf1c@mail.gmail.com>
References: <46276D4C.7020001@relativita.com> <4627744E.9000502@relativita.com>
	<91cf711d0704191118nbd0217h19102f0a7c7cf1c@mail.gmail.com>
Message-ID: <4627E462.9020208@relativita.com>

David Huard wrote:
> Hi Emanuele,
>
> The bug is due to a part of the code that shifts the last bin's
> position to make sure the array's maximum value is counted in the last
> bin, and not as an outlier. To do so, the code computes an approximate
> precision used the shift the bin edge by amount small compared to the
> array's value. In your example, since all values in x are identical,
> the precision is ``infinite''. So my question is, what kind of
> behaviour would you be expecting in this case for the automatic
> placement of bin edges ?
>
> That is, given
> x : array of identical values, eg. [0, 0, 0, 0, 0, ..., 0]
> smin, smax = x.min(), x.max()
> How do you select the bin edges ?
>
> One solution is to use the same scheme used by histogram:
> if smin == smax:
>     edges[i] = linspace(smin-.5, smax+.5, nbin[i]+1)
>
> Would that be ok ?
>
> David
>
>
>  I'll submit a patch.
>
The histogram solution seems ok. I can't see drawbacks.
My concerns are about not having exception in degenerate
cases, like the example I sent. I need to estimate many probability
distributions counting samples efficiently so histogram*
functions are really nice.

Please submit the patch. By the way the same issue affects
histogramdd.

Thanks a lot,

Emanuele



From v-nijs at kellogg.northwestern.edu  Thu Apr 19 17:59:59 2007
From: v-nijs at kellogg.northwestern.edu (Vincent Nijs)
Date: Thu, 19 Apr 2007 16:59:59 -0500
Subject: [Numpy-discussion] Help using numPy to create a very large
 multi dimensional array
In-Reply-To: <462791B5.6010101@noaa.gov>
Message-ID: <C24D508F.73FD%v-nijs@kellogg.northwestern.edu>

I think it would be a great idea to have pylab.load in numpy. It also seems
to be a lot faster than scipy.io.

One thing that is very nice about pylab.load is that it can read-in dates.
However, it can't, as far a I know, handle other non-float data.

I played around with python's csv module and pylab.load for a while
resulting in a database class I posted in the cookbook section:

http://www.scipy.org/Cookbook/dbase

This class can read any type of data in a csv file, including dates, into a
dictionary but is based on both pylab.load and the csv module. I use cPickle
for storing the data once it is read-in once. I haven't tried PyTables but
hear a lot of good things about it.

Vincent


On 4/19/07 10:58 AM, "Christopher Barker" <Chris.Barker at noaa.gov> wrote:

> Lisandro Dalcin wrote:
>> I am also +1 on this, but this functionality should be implemented in
>> C, I think.
> 
> well,  maybe.
> 
>> I've just tested numpy.fromfile('name.txt', sep=' ')
>> against pylab.load('name.txt') for a 35MB text file, the number are:
>> 
>> numpy.fromfile: 2.66 sec.
>> pylab.load:  16.64 sec.
> 
> exactly that's expected. fromfile is designed to do the easy cases as
> fast as possible, pylab.load is designed to be be flexible, I'm not user
> you need both the speed and flexibility at the same time.
> 
> By the way, I haven't looked at pylab.load() for a while, but it could
> perhaps be sped up by using fromfile() and or fromstring internally.
> There may be some opportunity to special case the easy ones too (i.e.
> all columns desired, etc.)
> 
> -Chris
> 
> 
> 
> 

-- 





From david at ar.media.kyoto-u.ac.jp  Thu Apr 19 22:20:08 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 20 Apr 2007 11:20:08 +0900
Subject: [Numpy-discussion] How to call methods from a class with custom
 matrices parameters with numpy arrays ?
In-Reply-To: <e76aa17f0704190917l48af8b89l81adee237cebd2b9@mail.gmail.com>
References: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>	<462720CE.50801@ar.media.kyoto-u.ac.jp>	<e76aa17f0704190109t1f66f35fv309e8a9f65415783@mail.gmail.com>	<46274F38.9060204@ar.media.kyoto-u.ac.jp>	<e76aa17f0704190529w4bee37c0tebb5b8fb46ff9bb3@mail.gmail.com>	<5b8d13220704190903gbb3ffd8wb29abfc32e67ff7b@mail.gmail.com>
	<e76aa17f0704190917l48af8b89l81adee237cebd2b9@mail.gmail.com>
Message-ID: <46282358.9040203@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
>     Doing the wrapping in an object oriented way is difficult, and maybe
>     not that useful. This does not prevent the API exposed to python
>     to be
>     OO, of course.
>
>
>
> I have some difficulties to do this in an automated way...
> I'm trying now to make a derived object from my function, without 
> templates and, I'm hoping so, with a correct interface i.e. double + 
> 2*int to save make the conversions with numpy arrays.
>  
>
>     > That works great for C then, not that well for C++...
>     Well, this is an inherent problem of C++ when you try to use it from
>     other languages, but let's not start this (again :) ),
>
>
>
> :D
>
>  
>
>     The example shows basic wrapping, and some facilities provided by
>     numpy to help. Again, ctype is pretty efficient as long as you do not
>     need to do convertion. If you call it thousand of times, it will be
>     slow, but this is more or less inherent to python (function calls are
>     nowhere near as fast as in a compiled language, at least for now).
>
>
>
> It's for optimization, so the function will be called several hundreds 
> of times, I suppose, and I tried porting the whole function to Python, 
> but I'm not sure that the Python version behaves like the C++ version 
> - the results are not identic, so... -, thus the wrapping.
A couple of hundred times is OK if the called function is doing 
something. I once benchmarked ctypes with respect to function calling, 
and I could do several hundred of thousand of calls/s, if I remember 
correctly.
>
>
> It's only one class and one method + the constructor. Not much but I'm 
> a real beginner in that domain. True, I could use the C API directly...
Ok, I have a simple working example. It is actually much easier than I 
thought, because no C compiler is involved at all (I was afraid about 
object layout, vtables and other horrors), only C++.

I attached the example: it shows how to mimic a C++ class in python 
through a C-like interface. The main difficulty is to be sure that your 
object get deleted by the python interpreter, which means having a 
__del__ function. The problem is that you cannot control how python will 
destroy its things: it may "destroy" ctypes module before your object, 
which is problematic since you need it to destroy your object. The idea 
is to "force" python to keep everything in the namespace through a fake 
second argument to the destructor.

http://docs.python.org/ref/customization.html (comments on __del__ ).

David
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hello.py
Type: text/x-python
Size: 410 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070420/4714dae7/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hellocpp.cpp
Type: text/x-c++src
Size: 334 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070420/4714dae7/attachment.cpp>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hellocpp.h
Type: text/x-chdr
Size: 232 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070420/4714dae7/attachment.h>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Makefile
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070420/4714dae7/attachment.ksh>

From matthieu.brucher at gmail.com  Fri Apr 20 03:38:12 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 20 Apr 2007 09:38:12 +0200
Subject: [Numpy-discussion] How to call methods from a class with custom
	matrices parameters with numpy arrays ?
In-Reply-To: <46282358.9040203@ar.media.kyoto-u.ac.jp>
References: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>
	<462720CE.50801@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190109t1f66f35fv309e8a9f65415783@mail.gmail.com>
	<46274F38.9060204@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190529w4bee37c0tebb5b8fb46ff9bb3@mail.gmail.com>
	<5b8d13220704190903gbb3ffd8wb29abfc32e67ff7b@mail.gmail.com>
	<e76aa17f0704190917l48af8b89l81adee237cebd2b9@mail.gmail.com>
	<46282358.9040203@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0704200038m14a70105t8f6f525c2bed5b30@mail.gmail.com>

Well, that's easy ;)
OK, I have to digg in for the transformations of numpy arrays, knowing that
I have other parameters. But for this, the Cookbook at scipy should help me
a lot.
Thanks for the help ;)

Matthieu

Ok, I have a simple working example. It is actually much easier than I
> thought, because no C compiler is involved at all (I was afraid about
> object layout, vtables and other horrors), only C++.
>
> I attached the example: it shows how to mimic a C++ class in python
> through a C-like interface. The main difficulty is to be sure that your
> object get deleted by the python interpreter, which means having a
> __del__ function. The problem is that you cannot control how python will
> destroy its things: it may "destroy" ctypes module before your object,
> which is problematic since you need it to destroy your object. The idea
> is to "force" python to keep everything in the namespace through a fake
> second argument to the destructor.
>
> http://docs.python.org/ref/customization.html (comments on __del__ ).
>
> David
>
> CC      = colorgcc
> CXX     = colorgcc
>
> LD      = g++
>
> PYTHONINC       = -I/usr/include/python2.5
> NUMPYINC        = -I/usr/lib/python2.5/site-packages/numpy/core/include
>
> WARN    = -W -Wall
>
> CFLAGS  = $(WARN) $(PYTHONINC) $(NUMPYINC)
>
> hellocpp.so: hellocpp.o
>         $(LD) -shared -o $@ $< -Wl,-soname,$@
>
> hellocpp.o: hellocpp.cpp
>         $(CXX) -c $(CFLAGS) -fPIC $<
>
> clean:
>         rm -f *.o
>         rm -f *.so
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070420/50ecb99a/attachment.html>

From persed at princeton.edu  Fri Apr 20 10:38:34 2007
From: persed at princeton.edu (Per B. Sederberg)
Date: Fri, 20 Apr 2007 14:38:34 +0000 (UTC)
Subject: [Numpy-discussion] Bus Error with string in ndarray with named
	fields
Message-ID: <loom.20070420T163126-240@post.gmane.org>

>>>

Hi Folks:

I'm getting a very strange bus error in the recent versions of numpy (almost
current svn).  Here's how you can (hopefully) replicate it:

On my MacBook:

Python 2.4.3 (#1, Apr  7 2006, 10:54:33)
[GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import numpy as N
 >>> N.version.version
'1.0.2.dev3569'
 >>> fields = [('x',(N.str,40))]
 >>> dat = N.zeros(1,fields)
 >>> dat
 Bus error


On my linux cluster:

Python 2.4.3 (#1, May  8 2006, 11:36:25)
[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-49)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as N
>>> N.version.version
'1.0.2.dev3567'
>>> fields = [('x',(N.str,40))]
>>> dat = N.zeros(1,fields)
>>> dat
Segmentation fault


But it works on my linux desktop with an older numpy:

Python 2.4.4c1 (#2, Oct 11 2006, 20:00:03)
[GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as N
>>> N.version.version
'1.0rc1'
>>> fields = [('x',(N.str,40))]
>>> dat = N.zeros(1,fields)
>>> dat
array([('',)],
     dtype=[('x', '|S40')])
>>>


Does anyone have any clue as to how to deal with this?  Is there a better way to
create an ndarray with named fields (recarray)?  My real code has a number of
fields of various types, but I was able to narrow it down to anytime I use a
string type.  For example, this works on the machines where using a string fails:

>>> import numpy as N
>>> fields = [('x',N.bool),('y',N.int32)]
>>> dat = N.zeros(1,fields)
>>> dat
array([(False, 0)],
     dtype=[('x', '|b1'), ('y', '<i4')])
>>>

Thanks for any help,
Per




From nwagner at iam.uni-stuttgart.de  Fri Apr 20 10:45:16 2007
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Fri, 20 Apr 2007 16:45:16 +0200
Subject: [Numpy-discussion] Bus Error with string in ndarray with named
 fields
In-Reply-To: <loom.20070420T163126-240@post.gmane.org>
References: <loom.20070420T163126-240@post.gmane.org>
Message-ID: <4628D1FC.1060005@iam.uni-stuttgart.de>

Per B. Sederberg wrote:
>
> Hi Folks:
>
> I'm getting a very strange bus error in the recent versions of numpy (almost
> current svn).  Here's how you can (hopefully) replicate it:
>
> On my MacBook:
>
> Python 2.4.3 (#1, Apr  7 2006, 10:54:33)
> [GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import numpy as N
>  >>> N.version.version
> '1.0.2.dev3569'
>  >>> fields = [('x',(N.str,40))]
>  >>> dat = N.zeros(1,fields)
>  >>> dat
>  Bus error
>
>
> On my linux cluster:
>
> Python 2.4.3 (#1, May  8 2006, 11:36:25)
> [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-49)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>   
>>>> import numpy as N
>>>> N.version.version
>>>>         
> '1.0.2.dev3567'
>   
>>>> fields = [('x',(N.str,40))]
>>>> dat = N.zeros(1,fields)
>>>> dat
>>>>         
> Segmentation fault
>
>
> But it works on my linux desktop with an older numpy:
>
> Python 2.4.4c1 (#2, Oct 11 2006, 20:00:03)
> [GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>   
>>>> import numpy as N
>>>> N.version.version
>>>>         
> '1.0rc1'
>   
>>>> fields = [('x',(N.str,40))]
>>>> dat = N.zeros(1,fields)
>>>> dat
>>>>         
> array([('',)],
>      dtype=[('x', '|S40')])
>   
>
>
> Does anyone have any clue as to how to deal with this?  Is there a better way to
> create an ndarray with named fields (recarray)?  My real code has a number of
> fields of various types, but I was able to narrow it down to anytime I use a
> string type.  For example, this works on the machines where using a string fails:
>
>   
>>>> import numpy as N
>>>> fields = [('x',N.bool),('y',N.int32)]
>>>> dat = N.zeros(1,fields)
>>>> dat
>>>>         
> array([(False, 0)],
>      dtype=[('x', '|b1'), ('y', '<i4')])
>   
>
> Thanks for any help,
> Per
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   
I cannot reproduce your problems with 1.0.3.dev3716.

Nils
 


From emanuele at relativita.com  Fri Apr 20 10:57:13 2007
From: emanuele at relativita.com (Emanuele Olivetti)
Date: Fri, 20 Apr 2007 16:57:13 +0200
Subject: [Numpy-discussion] why std() eats much memory in multidimensional
	case?
Message-ID: <4628D4C9.3020906@relativita.com>

Hi,
I'm working with 4D integer matrices and need to compute std() on a
given axis but I experience problems with excessive memory consumption.
Example:
---
import numpy
a = numpy.random.randint(100,size=(50,50,50,200)) # 4D randint matrix
b = a.std(3)
---
It seems that this code requires 100-200 Mb to allocate 'a'
as a matrix of integers, but requires >500Mb more just to
compute std(3). Is it possible to compute std(3) on integer
matrices without spending so much memory?

I manage 4D matrices that are not much bigger than the one in the example
and they require >1.2Gb of ram to compute std(3) only.
Note that quite all this memory is immediately released after
computing std() so it seems it's used just internally and not to
represent/store the result. Unfortunately I haven't all that RAM...

Could someone explain/correct this problem?

Thanks in advance,

Emanuele



From charlesr.harris at gmail.com  Fri Apr 20 12:06:11 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 20 Apr 2007 10:06:11 -0600
Subject: [Numpy-discussion] why std() eats much memory in
	multidimensional case?
In-Reply-To: <4628D4C9.3020906@relativita.com>
References: <4628D4C9.3020906@relativita.com>
Message-ID: <e06186140704200906k176d9aedq5958865284e668f7@mail.gmail.com>

On 4/20/07, Emanuele Olivetti <emanuele at relativita.com> wrote:
>
> Hi,
> I'm working with 4D integer matrices and need to compute std() on a
> given axis but I experience problems with excessive memory consumption.
> Example:
> ---
> import numpy
> a = numpy.random.randint(100,size=(50,50,50,200)) # 4D randint matrix
> b = a.std(3)
> ---
> It seems that this code requires 100-200 Mb to allocate 'a'
> as a matrix of integers, but requires >500Mb more just to
> compute std(3). Is it possible to compute std(3) on integer
> matrices without spending so much memory?
>
> I manage 4D matrices that are not much bigger than the one in the example
> and they require >1.2Gb of ram to compute std(3) only.
> Note that quite all this memory is immediately released after
> computing std() so it seems it's used just internally and not to
> represent/store the result. Unfortunately I haven't all that RAM...
>
> Could someone explain/correct this problem?


I suspect that a temporary double precision copy of the input array is made
before computing the std. If the original array is int32, then the temporary
would be twice the size. That would certainly be the easiest way to do the
computation.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070420/a0b8b467/attachment.html>

From mathewww at charter.net  Fri Apr 20 16:51:25 2007
From: mathewww at charter.net (Mathew Yeates)
Date: Fri, 20 Apr 2007 13:51:25 -0700
Subject: [Numpy-discussion] help with where
Message-ID: <462927CD.8030304@charter.net>

Hi
I have a list of objects that I want to be interpreted numpy.where. What 
class methods do I need to implement?

example:
class A:pass
a=A()
a.i=1
b=A()
b.i=0
numpy.where([a,b,a]) #desired result [0,2]

Thanks
Mathew



From robert.kern at gmail.com  Fri Apr 20 17:23:46 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 20 Apr 2007 16:23:46 -0500
Subject: [Numpy-discussion] help with where
In-Reply-To: <462927CD.8030304@charter.net>
References: <462927CD.8030304@charter.net>
Message-ID: <46292F62.7070306@gmail.com>

Mathew Yeates wrote:
> Hi
> I have a list of objects that I want to be interpreted numpy.where. What 
> class methods do I need to implement?

__nonzero__

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From matthieu.brucher at gmail.com  Fri Apr 20 17:30:06 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 20 Apr 2007 23:30:06 +0200
Subject: [Numpy-discussion] How to call methods from a class with custom
	matrices parameters with numpy arrays ?
In-Reply-To: <e76aa17f0704200038m14a70105t8f6f525c2bed5b30@mail.gmail.com>
References: <e76aa17f0704190052l2238a318i97c1d2a6131d7252@mail.gmail.com>
	<462720CE.50801@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190109t1f66f35fv309e8a9f65415783@mail.gmail.com>
	<46274F38.9060204@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704190529w4bee37c0tebb5b8fb46ff9bb3@mail.gmail.com>
	<5b8d13220704190903gbb3ffd8wb29abfc32e67ff7b@mail.gmail.com>
	<e76aa17f0704190917l48af8b89l81adee237cebd2b9@mail.gmail.com>
	<46282358.9040203@ar.media.kyoto-u.ac.jp>
	<e76aa17f0704200038m14a70105t8f6f525c2bed5b30@mail.gmail.com>
Message-ID: <e76aa17f0704201430w59e8e8cej11c015b71f862a4b@mail.gmail.com>

2007/4/20, Matthieu Brucher <matthieu.brucher at gmail.com>:
>
> Well, that's easy ;)
> OK, I have to digg in for the transformations of numpy arrays, knowing
> that I have other parameters. But for this, the Cookbook at scipy should
> help me a lot.
> Thanks for the help ;)
>
> Matthieu
>

Some news, I finished the wrappers, I had some troubles with the allocator,
it's not something you want to use every time, I'd say, it's not intuitive
to pass the function to the C wrapped function, calling it to get a pointer
to a new array and not returning it because it was saved in the allocator...
Now, I think I can say that the ported Python function does not function as
well as the wrapped function, I must have made an error somewhere, but now I
don't have to care about it, it works well.

Many thanks to David and Bill !

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070420/6f56c876/attachment.html>

From numpy at mspacek.mm.st  Fri Apr 20 20:38:52 2007
From: numpy at mspacek.mm.st (Martin Spacek)
Date: Fri, 20 Apr 2007 17:38:52 -0700
Subject: [Numpy-discussion] building with MKL in windows
In-Reply-To: <46275338.10803@mspacek.mm.st>
References: <46275338.10803@mspacek.mm.st>
Message-ID: <46295D1C.7090507@mspacek.mm.st>

> Also, I found I had to remove the 2 lines in distutils/system_info.py
> that mention pthread, mkl_lapack32, mkl_lapack64 (see attached patch)
> since libraries with such names don't seem to exist in the MKL for
> windows and were generating linking errors.
> 
> This obviously isn't the right thing to do, and I don't know enough to
> know how to do it properly

Here's a new patch that checks if sys.platform == 'win32' and does the 
appropriate thing. MKL on linux behaviour should be unaffected, but I 
don't have any way of testing that.

I've also included an improved .numpy-site.cfg file.

I've added a section on how to build with MKL and MSVC to 
http://www.scipy.org/Installing_SciPy/Windows

Cheers,

Martin
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mkl_windows.patch
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070420/9300f9b2/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: .numpy-site.cfg
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070420/9300f9b2/attachment-0001.ksh>

From christian at marquardt.sc  Sat Apr 21 09:03:57 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Sat, 21 Apr 2007 15:03:57 +0200 (CEST)
Subject: [Numpy-discussion] Problems building numpy and scipy on AIX
In-Reply-To: <4627D334.30800@physics.mcmaster.ca>
References: <30420.193.17.11.23.1176898744.squirrel@webmail.marquardt.sc>
	<46263741.7080403@physics.mcmaster.ca>
	<34004.193.17.11.23.1176917180.squirrel@webmail.marquardt.sc>
	<4627D334.30800@physics.mcmaster.ca>
Message-ID: <18030.84.167.95.237.1177160637.squirrel@webmail.marquardt.sc>

Yes,

that worked - many thanks!

  Christian.


On Thu, April 19, 2007 22:38, David M. Cooke wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Christian Marquardt wrote:
>> Dear David,
>>
>> the svn version of numpy does indeed build cleanly on AIX. Many thanks!
>>
>> However, the wrapper problem still exists for the C++ compiler, and
>> shows
>> up when compiling scipy. Now, I *assume* that SciPy is using the
>> distutils
>> as installed by numpy. Do you know where the linker settings for the C++
>> compiler might be overwritten? There are two or three C compiler related
>> python modules in numpy/distutils... Or would you think that this
>> problem
>> is entirely unrelated to the distutils in numpy?
>
> I'm working on a better solution, but the quick fix to your problem is
> to look in numpy/distutils/command/build_ext.py. There are two lines
> that reference self.compiler.linker_so[0]; change those 0s to a 1s, so
> it keeps the ld_so_aix script there when switching the linker.
>
> - --
> |>|\/|<
> /------------------------------------------------------------------\
> |David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
> |cookedm at physics.mcmaster.ca
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.7 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFGJ9MzN9ixZKFWjRQRArNwAKC029wYORk9sm+FShcYKNd0UEcMdgCghHGC
> rjYqtaESdt8zRgZHCDxYbDk=
> =PS30
> -----END PGP SIGNATURE-----
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>




From haase at msg.ucsf.edu  Sat Apr 21 10:38:43 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Sat, 21 Apr 2007 07:38:43 -0700
Subject: [Numpy-discussion] C-API creating new copy of C data
In-Reply-To: <e86a5fd00704190051h4c583be4odd40075cd291194@mail.gmail.com>
References: <e86a5fd00704190051h4c583be4odd40075cd291194@mail.gmail.com>
Message-ID: <bc657ead0704210738r2e0304b3x3c74dc2f1e45fbdf@mail.gmail.com>

On 4/19/07, Bill Baxter <wbaxter at gmail.com> wrote:
> What's the right way to make a new numpy array that's a copy of some C data?
>
> There doesn't seem to be any API like PyArray_NewFromDescr that
> /copies/ the void*data pointer for you.  Do I have to write my own
> loops for this?  I can do that, it just seems like it should be a
> library function already, so I'm guessing I'm just overlooking it.
> There seem to be lots of APIs that will wrap pre-existing memory, but
> the ones that allocate for you do not seem to copy.
>
> A related question -- I'm only trying to copy in order to save myself
> a little hassle regarding how to clean up the allocated chunks.  If
> there's some simple way to trigger a particular deallocation function
> to be called at the right time, then that would be the ideal, really.
> Does that exist?

This is a situation I have been waiting to address for a long time !
In our case the data size is generally considered to be to large to
accept the copy-solution.
A cookbook-wiki entry would be wonderful!
Generally we would have data memory that was allocated via alloc() and
data that was allocated via new[] -- so two different deallocation
functions (free()  and delete[], respectively) would be required for
this to be trigged, once the reference counter goes back to zero.

Thanks,
Sebastian Haase


From bronto at pobox.com  Sat Apr 21 12:56:38 2007
From: bronto at pobox.com (Anton Sherwood)
Date: Sat, 21 Apr 2007 09:56:38 -0700
Subject: [Numpy-discussion] MacNoob
In-Reply-To: <461DCF73.7090605@pobox.com>
References: <461DCF73.7090605@pobox.com>
Message-ID: <462A4246.3070005@pobox.com>

Anton Sherwood wrote (Apr 11):
 >> When I try to build numpy (or PIL) on MacOS, I get this error:
 >> "gcc: cannot specify -o with -c or -S and multiple compilations"
 >>
 >> Assuming at least one of you is using numpy on MacOS,
 >> how did you get around that?

Christopher Barker wrote:
 > by installing a binary from:
 > pythonmac.org/packages
 > [...]

Thanks, that did the trick!

Vincent Nijs wrote:
 > I think that error went away when I used the latest developer tools
 > from Apple, and made sure I was using gcc4.
 >
 > Take a look at:
 > http://www.scipy.org/Installing_SciPy/Mac_OS_X

Thanks for that too.

-- 
Anton Sherwood, http://www.ogre.nu/
"How'd ya like to climb this high *without* no mountain?" --Porky Pine


From oliphant.travis at ieee.org  Sat Apr 21 15:58:29 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Sat, 21 Apr 2007 13:58:29 -0600
Subject: [Numpy-discussion] C-API creating new copy of C data
In-Reply-To: <e86a5fd00704190051h4c583be4odd40075cd291194@mail.gmail.com>
References: <e86a5fd00704190051h4c583be4odd40075cd291194@mail.gmail.com>
Message-ID: <462A6CE5.9090206@ieee.org>

Bill Baxter wrote:
> What's the right way to make a new numpy array that's a copy of some C data?
>
> There doesn't seem to be any API like PyArray_NewFromDescr that
> /copies/ the void*data pointer for you.  Do I have to write my own
> loops for this?  I can do that, it just seems like it should be a
> library function already, so I'm guessing I'm just overlooking it.
> There seem to be lots of APIs that will wrap pre-existing memory, but
> the ones that allocate for you do not seem to copy.
>   

What do you mean by /copies/ the void * data pointer for you?   Do you 
mean the API would

1) Create new memory for the array
2) Copy the data from another void * pointer to the memory just created 
for the new array?

If that is what you mean, then you are right there is no such API.   I'm 
not sure that there needs to be one.  It is a two-liner using memcpy.

> A related question -- I'm only trying to copy in order to save myself
> a little hassle regarding how to clean up the allocated chunks.  If
> there's some simple way to trigger a particular deallocation function
> to be called at the right time, then that would be the ideal, really.
>   
No, there is no place to store that information in NumPy.  Either the 
ndarray dealloc function frees the memory it created or it doesn't free 
any memory.    I think the best thing to do in this case would be to 
create a memory object wrapping the pointer and then point the ndarray 
to it as the source of memory.

-Travis



From haase at msg.ucsf.edu  Sat Apr 21 17:35:41 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Sat, 21 Apr 2007 14:35:41 -0700
Subject: [Numpy-discussion] C-API creating new copy of C data
In-Reply-To: <462A6CE5.9090206@ieee.org>
References: <e86a5fd00704190051h4c583be4odd40075cd291194@mail.gmail.com>
	<462A6CE5.9090206@ieee.org>
Message-ID: <bc657ead0704211435r634e7a64hdf42f91fcacacb69@mail.gmail.com>

On 4/21/07, Travis Oliphant <oliphant.travis at ieee.org> wrote:
> Bill Baxter wrote:
> > What's the right way to make a new numpy array that's a copy of some C data?
> >
> > There doesn't seem to be any API like PyArray_NewFromDescr that
> > /copies/ the void*data pointer for you.  Do I have to write my own
> > loops for this?  I can do that, it just seems like it should be a
> > library function already, so I'm guessing I'm just overlooking it.
> > There seem to be lots of APIs that will wrap pre-existing memory, but
> > the ones that allocate for you do not seem to copy.
> >
>
> What do you mean by /copies/ the void * data pointer for you?   Do you
> mean the API would
>
> 1) Create new memory for the array
> 2) Copy the data from another void * pointer to the memory just created
> for the new array?
>
> If that is what you mean, then you are right there is no such API.   I'm
> not sure that there needs to be one.  It is a two-liner using memcpy.

Yes, I was thinking about memcpy() -- but how about non-contiguous
data ? or other non well behaved ndarrays (non-aligned, byte-swapped,
...?) ?

>
> > A related question -- I'm only trying to copy in order to save myself
> > a little hassle regarding how to clean up the allocated chunks.  If
> > there's some simple way to trigger a particular deallocation function
> > to be called at the right time, then that would be the ideal, really.
> >
> No, there is no place to store that information in NumPy.  Either the
> ndarray dealloc function frees the memory it created or it doesn't free
> any memory.    I think the best thing to do in this case would be to
> create a memory object wrapping the pointer and then point the ndarray
> to it as the source of memory.
>
Yes, I think one would probably want to create a custom-made python
class that provides the memory as buffer or so. This is what you meant
- right ?  And that class could then define /any/ function to be
called once the ref.count goes to zero - right?
Could someone with C-API knowledge put a sample together !?   This
would also be quite useful to be used with a SWIG output typemap.

-Sebastian


From wbaxter at gmail.com  Sat Apr 21 19:22:08 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Sun, 22 Apr 2007 08:22:08 +0900
Subject: [Numpy-discussion] C-API creating new copy of C data
In-Reply-To: <462A6CE5.9090206@ieee.org>
References: <e86a5fd00704190051h4c583be4odd40075cd291194@mail.gmail.com>
	<462A6CE5.9090206@ieee.org>
Message-ID: <e86a5fd00704211622h2e71174aw2baeba4fe2f6a5d0@mail.gmail.com>

On 4/22/07, Travis Oliphant <oliphant.travis at ieee.org> wrote:
> Bill Baxter wrote:
> > What's the right way to make a new numpy array that's a copy of some C data?
> >
> What do you mean by /copies/ the void * data pointer for you?   Do you
> mean the API would
>
> 1) Create new memory for the array
> 2) Copy the data from another void * pointer to the memory just created
> for the new array?

Yes.

> If that is what you mean, then you are right there is no such API.   I'm
> not sure that there needs to be one.  It is a two-liner using memcpy.

Ok, I see.  That's not too hard.  Though as Sebastian said below it
would be a little trickier if I needed to do a transpose or something
at the same time.  I guess for those cases it should be possible to
make a temporary PyArrayObject with PyArray_NewFromDescr that wraps
the memory and then do a PyArray_GETCONTIGUOUS?

> > A related question -- I'm only trying to copy in order to save myself
> > a little hassle regarding how to clean up the allocated chunks.  If
> > there's some simple way to trigger a particular deallocation function
> > to be called at the right time, then that would be the ideal, really.
> >
> No, there is no place to store that information in NumPy.  Either the
> ndarray dealloc function frees the memory it created or it doesn't free
> any memory.    I think the best thing to do in this case would be to
> create a memory object wrapping the pointer and then point the ndarray
> to it as the source of memory.

Hmm.  Sounds a little over my head.   I'll stick with copying for now.

--bb


From markusro at element.fkp.physik.tu-darmstadt.de  Sun Apr 22 11:33:18 2007
From: markusro at element.fkp.physik.tu-darmstadt.de (Markus Rosenstihl)
Date: Sun, 22 Apr 2007 17:33:18 +0200
Subject: [Numpy-discussion] Help using numPy to create a very large
	multi dimensional array
In-Reply-To: <e7ba66e40704190650t640d7c33ve1abc0263b2922d8@mail.gmail.com>
References: <f687be130704181605t5ee447c4k52ca845b4a44ad41@mail.gmail.com>
	<f687be130704181609h13091e6y9e47b8ec9a45a5eb@mail.gmail.com>
	<46272585.2090004@ieee.org>
	<e7ba66e40704190650t640d7c33ve1abc0263b2922d8@mail.gmail.com>
Message-ID: <5e842005decc0abf796e9e020fbe23bb@element.fkp.physik.tu-darmstadt.de>

Whats wrong with scipy.io.read_array?

Am 19.04.2007 um 15:50 schrieb Lisandro Dalcin:

> On 4/19/07, Travis Oliphant <oliphant.travis at ieee.org> wrote:
>> Nick Fotopoulos wrote:
>>> Devs, is there any possibility of moving/copying pylab.load to numpy?
>>> I don't see anything in the source that requires the rest of
>>> matplotlib.  Among convenience functions, I think that this function
>>> ranks pretty highly in convenience.
>>>
>>
>> I'm supportive of this.  But, it can't be named numpy.load.
>>
>
> I am also +1 on this, but this functionality should be implemented in
> C, I think. I've just tested numpy.fromfile('name.txt', sep=' ')
> against pylab.load('name.txt') for a 35MB text file, the number are:
>
> numpy.fromfile: 2.66 sec.
> pylab.load:  16.64 sec.
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: Signierter Teil der Nachricht
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070422/41240709/attachment.sig>

From v-nijs at kellogg.northwestern.edu  Sun Apr 22 11:43:04 2007
From: v-nijs at kellogg.northwestern.edu (Vincent Nijs)
Date: Sun, 22 Apr 2007 10:43:04 -0500
Subject: [Numpy-discussion] Help using numPy to create a very large
 multi dimensional array
In-Reply-To: <5e842005decc0abf796e9e020fbe23bb@element.fkp.physik.tu-darmstadt.de>
Message-ID: <C250ECB8.5A8A%v-nijs@kellogg.northwestern.edu>

It seems to be a lot slower than pylab.load for large arrays. Also, it
doesn't handle dates.

Vincent

On 4/22/07 10:33 AM, "Markus Rosenstihl"
<markusro at element.fkp.physik.tu-darmstadt.de> wrote:

> Whats wrong with scipy.io.read_array?
> 
> Am 19.04.2007 um 15:50 schrieb Lisandro Dalcin:
> 
>> On 4/19/07, Travis Oliphant <oliphant.travis at ieee.org> wrote:
>>> Nick Fotopoulos wrote:
>>>> Devs, is there any possibility of moving/copying pylab.load to numpy?
>>>> I don't see anything in the source that requires the rest of
>>>> matplotlib.  Among convenience functions, I think that this function
>>>> ranks pretty highly in convenience.
>>>> 
>>> 
>>> I'm supportive of this.  But, it can't be named numpy.load.
>>> 
>> 
>> I am also +1 on this, but this functionality should be implemented in
>> C, I think. I've just tested numpy.fromfile('name.txt', sep=' ')
>> against pylab.load('name.txt') for a 35MB text file, the number are:
>> 
>> numpy.fromfile: 2.66 sec.
>> pylab.load:  16.64 sec.
>> 
>> 
>> -- 
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion

-- 
Vincent R. Nijs
Assistant Professor of Marketing
Kellogg School of Management, Northwestern University
2001 Sheridan Road, Evanston, IL 60208-2001
Phone: +1-847-491-4574 Fax: +1-847-491-2498
E-mail: v-nijs at kellogg.northwestern.edu
Skype: vincentnijs





From cookedm at physics.mcmaster.ca  Sun Apr 22 17:19:19 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Sun, 22 Apr 2007 17:19:19 -0400
Subject: [Numpy-discussion] Problems building numpy and scipy on AIX
In-Reply-To: <18030.84.167.95.237.1177160637.squirrel@webmail.marquardt.sc>
References: <30420.193.17.11.23.1176898744.squirrel@webmail.marquardt.sc>
	<46263741.7080403@physics.mcmaster.ca>
	<34004.193.17.11.23.1176917180.squirrel@webmail.marquardt.sc>
	<4627D334.30800@physics.mcmaster.ca>
	<18030.84.167.95.237.1177160637.squirrel@webmail.marquardt.sc>
Message-ID: <BD8E97B4-6DB2-4F88-B3F1-E7919A010B41@physics.mcmaster.ca>

On Apr 21, 2007, at 09:03 , Christian Marquardt wrote:

> Yes,
>
> that worked - many thanks!

FWIW, svn should work out of the box now.

> On Thu, April 19, 2007 22:38, David M. Cooke wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Christian Marquardt wrote:
>>> Dear David,
>>>
>>> the svn version of numpy does indeed build cleanly on AIX. Many  
>>> thanks!
>>>
>>> However, the wrapper problem still exists for the C++ compiler, and
>>> shows
>>> up when compiling scipy. Now, I *assume* that SciPy is using the
>>> distutils
>>> as installed by numpy. Do you know where the linker settings for  
>>> the C++
>>> compiler might be overwritten? There are two or three C compiler  
>>> related
>>> python modules in numpy/distutils... Or would you think that this
>>> problem
>>> is entirely unrelated to the distutils in numpy?
>>
>> I'm working on a better solution, but the quick fix to your  
>> problem is
>> to look in numpy/distutils/command/build_ext.py. There are two lines
>> that reference self.compiler.linker_so[0]; change those 0s to a  
>> 1s, so
>> it keeps the ld_so_aix script there when switching the linker.
>>
>> - --
>> |>|\/|<
>> /------------------------------------------------------------------\
>> |David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
>> |cookedm at physics.mcmaster.ca
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.7 (Darwin)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>>
>> iD8DBQFGJ9MzN9ixZKFWjRQRArNwAKC029wYORk9sm+FShcYKNd0UEcMdgCghHGC
>> rjYqtaESdt8zRgZHCDxYbDk=
>> =PS30
>> -----END PGP SIGNATURE-----

-- 
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070422/7794af6f/attachment.sig>

From christian at marquardt.sc  Sun Apr 22 17:19:42 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Sun, 22 Apr 2007 23:19:42 +0200 (CEST)
Subject: [Numpy-discussion] uint64 typecasting with scalars broken (?)
Message-ID: <19721.84.167.104.46.1177276782.squirrel@webmail.marquardt.sc>

Hello,

The following is what I expected...

   >>> y = 1234
   >>> x = array([1], dtype = "uint64")
   >>> print x + y, (x + y).dtype.type
   [1235] <type 'numpy.uint64'>

but is this the way it should be? (numpy 1.0.2, Linux, Intel comilers)

   >>> print x[0] + y, type(x[0] + y)
   1235.0 <type 'numpy.float64'>

Thanks,

  Christian.



From charlesr.harris at gmail.com  Sun Apr 22 19:28:58 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 22 Apr 2007 17:28:58 -0600
Subject: [Numpy-discussion] uint64 typecasting with scalars broken (?)
In-Reply-To: <19721.84.167.104.46.1177276782.squirrel@webmail.marquardt.sc>
References: <19721.84.167.104.46.1177276782.squirrel@webmail.marquardt.sc>
Message-ID: <e06186140704221628n1a379986mf5c29d5cecdd9fbe@mail.gmail.com>

On 4/22/07, Christian Marquardt <christian at marquardt.sc> wrote:
>
> Hello,
>
> The following is what I expected...
>
>    >>> y = 1234
>    >>> x = array([1], dtype = "uint64")
>    >>> print x + y, (x + y).dtype.type
>    [1235] <type 'numpy.uint64'>
>
> but is this the way it should be? (numpy 1.0.2, Linux, Intel comilers)
>
>    >>> print x[0] + y, type(x[0] + y)
>    1235.0 <type 'numpy.float64'>


Looks like a bug to me:

In [5]: x = array([1],dtype=uint64)

In [6]: type(x[0])
Out[6]: <type 'numpy.uint64'>

In [7]: type(x[0]+1)
Out[7]: <type 'numpy.float64'>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070422/b8d2463b/attachment.html>

From charlesr.harris at gmail.com  Sun Apr 22 20:54:01 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 22 Apr 2007 18:54:01 -0600
Subject: [Numpy-discussion] uint64 typecasting with scalars broken (?)
In-Reply-To: <e06186140704221628n1a379986mf5c29d5cecdd9fbe@mail.gmail.com>
References: <19721.84.167.104.46.1177276782.squirrel@webmail.marquardt.sc>
	<e06186140704221628n1a379986mf5c29d5cecdd9fbe@mail.gmail.com>
Message-ID: <e06186140704221754h2d80b077j6b25131216b60224@mail.gmail.com>

On 4/22/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
>
>
>
> On 4/22/07, Christian Marquardt <christian at marquardt.sc> wrote:
> >
> > Hello,
> >
> > The following is what I expected...
> >
> >    >>> y = 1234
> >    >>> x = array([1], dtype = "uint64")
> >    >>> print x + y, (x + y).dtype.type
> >    [1235] <type ' numpy.uint64'>
> >
> > but is this the way it should be? (numpy 1.0.2, Linux, Intel comilers)
> >
> >    >>> print x[0] + y, type(x[0] + y)
> >    1235.0 <type 'numpy.float64'>
>
>
> Looks like a bug to me:
>
> In [5]: x = array([1],dtype=uint64)
>
> In [6]: type(x[0])
> Out[6]: <type 'numpy.uint64'>
>
> In [7]: type(x[0]+1)
> Out[7]: <type 'numpy.float64'>
>

Python integers should have enough precision to hold the result, although a
long integer would have to be used. I suspect this is an artifact of
considering regular python integers, which don't have the needed precision,
hence the double (which is a bit small itself).

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070422/8bdfc462/attachment.html>

From christian at marquardt.sc  Mon Apr 23 08:28:02 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Mon, 23 Apr 2007 14:28:02 +0200 (CEST)
Subject: [Numpy-discussion] uint64 typecasting with scalars broken (?)
In-Reply-To: <e06186140704221628n1a379986mf5c29d5cecdd9fbe@mail.gmail.com>
References: <19721.84.167.104.46.1177276782.squirrel@webmail.marquardt.sc>
	<e06186140704221628n1a379986mf5c29d5cecdd9fbe@mail.gmail.com>
Message-ID: <46151.193.17.11.23.1177331282.squirrel@webmail.marquardt.sc>

On Mon, April 23, 2007 01:28, Charles R Harris wrote:
> Looks like a bug to me:
>
> In [5]: x = array([1],dtype=uint64)
>
> In [6]: type(x[0])
> Out[6]: <type 'numpy.uint64'>
>
> In [7]: type(x[0]+1)
> Out[7]: <type 'numpy.float64'>
>
> Chuck

Yeah. Especially as it works apparently fine for uint32 and int64.

  Christian.





From mpmusu at cc.usu.edu  Mon Apr 23 10:37:57 2007
From: mpmusu at cc.usu.edu (Mark.Miller)
Date: Mon, 23 Apr 2007 08:37:57 -0600
Subject: [Numpy-discussion] efficient use of numpy.where() and .any()
Message-ID: <462CC4C5.8020106@cc.usu.edu>

Greetings:

In some of my code, I need to use large matrix of random numbers that 
meet specific criteria (i.e., some random numbers need to be removed and 
replaces with new ones).

I have been working with .any() and .where() to facilitate this process. 
  In the code below, .any() is used in a while loop to test for the 
presence of matrix elements that do not meet criteria.  After that, 
.where() is used to obtain the tuple of indices of elements that do not 
meet criteria.  I then use iteration over the tuple of indices and 
replace each 'bad' random number one at a time.

Here's an example:

 >>> a=numpy.random.normal(0,1,(3,3))
 >>> a
array([[ 0.79653228, -2.28751484,  1.15989261],
        [-0.7549573 ,  2.35133816,  0.22551564],
        [ 0.85666713,  1.17484243,  1.18306248]])
 >>> while (a<0).any() or (a>1).any():
	ind=numpy.where(a<0)
	for aa in xrange(len(ind[0])):
		a[ind[0][aa],ind[1][aa]]=numpy.random.normal(0,1)
	ind=numpy.where(a>1)
	for aa in xrange(len(ind[0])):
		a[ind[0][aa],ind[1][aa]]=numpy.random.normal(0,1)

		
 >>> a
array([[ 0.79653228,  0.99298488,  0.24903299],
        [ 0.10884186,  0.10139654,  0.22551564],
        [ 0.85666713,  0.76554597,  0.38383126]])
 >>>


My main question:  is there a more efficient approach that I could be 
using?  I would ideally like to be able to remove the two for loops.



An ancillary question:  I don't quite see why the test in the while loop 
above works fine, however, the following formation does not.

 >>> while (0<a<1).any():
	ind=numpy.where(a<0)
	for aa in xrange(len(ind[0])):
		a[ind[0][aa],ind[1][aa]]=numpy.random.normal(0,1)
	ind=numpy.where(a>1)
	for aa in xrange(len(ind[0])):
		a[ind[0][aa],ind[1][aa]]=numpy.random.normal(0,1)
	print type(ind)

	
Traceback (most recent call last):
   File "<pyshell#71>", line 1, in <module>
     while (0<a<1).any():
ValueError: The truth value of an array with more than one element is 
ambiguous. Use a.any() or a.all()
 >>>

Can someone clarify this?

Thanks,

-Mark


From tgrav at mac.com  Mon Apr 23 11:30:00 2007
From: tgrav at mac.com (Tommy Grav)
Date: Mon, 23 Apr 2007 11:30:00 -0400
Subject: [Numpy-discussion] Fwd: Getting subarray
References: <11ACD1D7-9CCE-4B5D-95A5-502C2BBD1FBB@cfa.harvard.edu>
Message-ID: <5E83E33A-9D22-4622-8D5D-6B02A07E33B6@mac.com>

I have two arrays:

a = numpy.array([0,1,2,3,4,5,6,7,8,9])
b = numpy.array([0,0,1,1,2,2,0,1,2,3])

I would like to get the part of a that corresponds
to where b is equal to i.

For example:

i = 0 => ([0,1,6])
i = 1 => ([2,3,7])

Cheers
    Tommy



From lists.steve at arachnedesign.net  Mon Apr 23 11:44:08 2007
From: lists.steve at arachnedesign.net (Steve Lianoglou)
Date: Mon, 23 Apr 2007 11:44:08 -0400
Subject: [Numpy-discussion] Fwd: Getting subarray
In-Reply-To: <5E83E33A-9D22-4622-8D5D-6B02A07E33B6@mac.com>
References: <11ACD1D7-9CCE-4B5D-95A5-502C2BBD1FBB@cfa.harvard.edu>
	<5E83E33A-9D22-4622-8D5D-6B02A07E33B6@mac.com>
Message-ID: <36996DD1-A169-49DD-95E6-C7A385C9CA08@arachnedesign.net>

Hi,

On Apr 23, 2007, at 11:30 AM, Tommy Grav wrote:

> I have two arrays:
>
> a = numpy.array([0,1,2,3,4,5,6,7,8,9])
> b = numpy.array([0,0,1,1,2,2,0,1,2,3])
>
> I would like to get the part of a that corresponds
> to where b is equal to i.
>
> For example:
>
> i = 0 => ([0,1,6])
> i = 1 => ([2,3,7])

a[numpy.where(b == 0)]

and

a[numpy.where(b == 1)]

respectively will get you what you're after.


-steve




From lists.steve at arachnedesign.net  Mon Apr 23 11:48:06 2007
From: lists.steve at arachnedesign.net (Steve Lianoglou)
Date: Mon, 23 Apr 2007 11:48:06 -0400
Subject: [Numpy-discussion] Fwd: Getting subarray
In-Reply-To: <5E83E33A-9D22-4622-8D5D-6B02A07E33B6@mac.com>
References: <11ACD1D7-9CCE-4B5D-95A5-502C2BBD1FBB@cfa.harvard.edu>
	<5E83E33A-9D22-4622-8D5D-6B02A07E33B6@mac.com>
Message-ID: <79123550-2667-4452-8121-2AECF91CDA92@arachnedesign.net>

> I have two arrays:
>
> a = numpy.array([0,1,2,3,4,5,6,7,8,9])
> b = numpy.array([0,0,1,1,2,2,0,1,2,3])
>
> I would like to get the part of a that corresponds
> to where b is equal to i.
>
> For example:
>
> i = 0 => ([0,1,6])
> i = 1 => ([2,3,7])

  a[b == 1]

and

a[b == 0]

work too, btw.

-steve


From pgmdevlist at gmail.com  Mon Apr 23 11:49:26 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 23 Apr 2007 11:49:26 -0400
Subject: [Numpy-discussion] efficient use of numpy.where() and .any()
In-Reply-To: <462CC4C5.8020106@cc.usu.edu>
References: <462CC4C5.8020106@cc.usu.edu>
Message-ID: <200704231149.27043.pgmdevlist@gmail.com>

On Monday 23 April 2007 10:37:57 Mark.Miller wrote:
> Greetings:
>
> In some of my code, I need to use large matrix of random numbers that
> meet specific criteria (i.e., some random numbers need to be removed and
> replaces with new ones).
>
> I have been working with .any() and .where() to facilitate this process.

Have you tried nonzero() ?

a[a<0] = numpy.random.normal(0,1)

will put a random number from the normal distribution where your initial a is 
negative. No Python loops needed, no Python temps.

> Traceback (most recent call last):
>    File "<pyshell#71>", line 1, in <module>
>      while (0<a<1).any():

The double condition (0<a<1) is not legit. You should try
logical.and(a>0,a<1)
or
(a>0) & (a<1)

Note the () around each condition in case #2.


From gael.varoquaux at normalesup.org  Mon Apr 23 11:49:51 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 23 Apr 2007 17:49:51 +0200
Subject: [Numpy-discussion] Fwd: Getting subarray
In-Reply-To: <36996DD1-A169-49DD-95E6-C7A385C9CA08@arachnedesign.net>
References: <11ACD1D7-9CCE-4B5D-95A5-502C2BBD1FBB@cfa.harvard.edu>
	<5E83E33A-9D22-4622-8D5D-6B02A07E33B6@mac.com>
	<36996DD1-A169-49DD-95E6-C7A385C9CA08@arachnedesign.net>
Message-ID: <20070423154933.GA14538@clipper.ens.fr>

On Mon, Apr 23, 2007 at 11:44:08AM -0400, Steve Lianoglou wrote:
> > I have two arrays:

> > a = numpy.array([0,1,2,3,4,5,6,7,8,9])
> > b = numpy.array([0,0,1,1,2,2,0,1,2,3])

> > I would like to get the part of a that corresponds
> > to where b is equal to i.

> > For example:

> > i = 0 => ([0,1,6])
> > i = 1 => ([2,3,7])

> a[numpy.where(b == 0)]

> and

> a[numpy.where(b == 1)]

How about "a[ b==0 ]" and "a[ b==1 ]" ?

Ga?l


From pgmdevlist at gmail.com  Mon Apr 23 11:53:57 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 23 Apr 2007 11:53:57 -0400
Subject: [Numpy-discussion] efficient use of numpy.where() and .any()
In-Reply-To: <462CC4C5.8020106@cc.usu.edu>
References: <462CC4C5.8020106@cc.usu.edu>
Message-ID: <200704231153.57610.pgmdevlist@gmail.com>

Oh, I pressed "send" too early.
Just an addition:
numpy.where creates a new array from some condition. If you only want to 
change elements of an existing array that satisfies a given condition, 
indexing is far more efficient: no temporary is created. Hence the suggestion 
of 
a[a<0]


From mpmusu at cc.usu.edu  Mon Apr 23 12:16:16 2007
From: mpmusu at cc.usu.edu (Mark.Miller)
Date: Mon, 23 Apr 2007 10:16:16 -0600
Subject: [Numpy-discussion] efficient use of numpy.where() and .any()
In-Reply-To: <200704231149.27043.pgmdevlist@gmail.com>
References: <462CC4C5.8020106@cc.usu.edu>
	<200704231149.27043.pgmdevlist@gmail.com>
Message-ID: <462CDBD0.1060105@cc.usu.edu>

Excellent suggestions...just a few comments:

Pierre GM wrote:
> On Monday 23 April 2007 10:37:57 Mark.Miller wrote:
>> Greetings:
>>
>> In some of my code, I need to use large matrix of random numbers that
>> meet specific criteria (i.e., some random numbers need to be removed and
>> replaces with new ones).
>>
>> I have been working with .any() and .where() to facilitate this process.
> 
> Have you tried nonzero() ?

Nonzero isn't quite what I'm after, as the tests are more complicated 
than what I illustrated in my example.

> 
> a[a<0] = numpy.random.normal(0,1)

This is a neat construct that I didn't realize was possible.  However, 
it has the undesirable (in my case) effect of placing a single new 
random number in each locations where a<0.  While this could work, I 
ideally need a different random number chosen for each replaced value. 
Does that make sense?

> will put a random number from the normal distribution where your initial a is 
> negative. No Python loops needed, no Python temps.
> 
>> Traceback (most recent call last):
>>    File "<pyshell#71>", line 1, in <module>
>>      while (0<a<1).any():
> 
> The double condition (0<a<1) is not legit. You should try
> logical.and(a>0,a<1)
> or
> (a>0) & (a<1)
> 
> Note the () around each condition in case #2.

This makes perfect sense.  Thanks for pointing it out to me.  It should 
easily do the trick.

Any and all additional suggestions are greatly appreciated,

-Mark



> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


From robert.kern at gmail.com  Mon Apr 23 12:23:49 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 Apr 2007 11:23:49 -0500
Subject: [Numpy-discussion] efficient use of numpy.where() and .any()
In-Reply-To: <462CDBD0.1060105@cc.usu.edu>
References: <462CC4C5.8020106@cc.usu.edu>	<200704231149.27043.pgmdevlist@gmail.com>
	<462CDBD0.1060105@cc.usu.edu>
Message-ID: <462CDD95.9090106@gmail.com>

Mark.Miller wrote:
> Pierre GM wrote:

>> a[a<0] = numpy.random.normal(0,1)
> 
> This is a neat construct that I didn't realize was possible.  However, 
> it has the undesirable (in my case) effect of placing a single new 
> random number in each locations where a<0.  While this could work, I 
> ideally need a different random number chosen for each replaced value. 
> Does that make sense?

Certainly. How about this?

mask = (a<0)
a[mask] = numpy.random.normal(0, 1, size=mask.sum())

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From peridot.faceted at gmail.com  Mon Apr 23 12:25:58 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 23 Apr 2007 12:25:58 -0400
Subject: [Numpy-discussion] efficient use of numpy.where() and .any()
In-Reply-To: <200704231149.27043.pgmdevlist@gmail.com>
References: <462CC4C5.8020106@cc.usu.edu>
	<200704231149.27043.pgmdevlist@gmail.com>
Message-ID: <ce557a360704230925m232e30q44a3ddc40d124ef4@mail.gmail.com>

On 23/04/07, Pierre GM <pgmdevlist at gmail.com> wrote:

> Have you tried nonzero() ?
>
> a[a<0] = numpy.random.normal(0,1)
>
> will put a random number from the normal distribution where your initial a is
> negative. No Python loops needed, no Python temps.

When you say "no python temps" I guess you mean, no temporary
*variables*? If I understand correctly, this allocates a temporary
boolean array to hold the result of "a<0".

> The double condition (0<a<1) is not legit. You should try
> logical.and(a>0,a<1)
> or
> (a>0) & (a<1)
>
> Note the () around each condition in case #2.

This is an unfortunate limitation that comes from the fact that we
can't override the behaviour of python's logical operations. a<b<c
does the right thing for python scalars, but it does it by being
expanded to (approximately) "a<b and b<c", and "and" doesn't do the
right thing for arrays. The best we can do is override the bitwise
operators for boolean arrays. This is a shame as I often want to
select array elements that fall into a given range, and creating three
temporary arrays instead of one is unpleasant.

Anne


From pgmdevlist at gmail.com  Mon Apr 23 12:32:18 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 23 Apr 2007 12:32:18 -0400
Subject: [Numpy-discussion] efficient use of numpy.where() and .any()
In-Reply-To: <462CDBD0.1060105@cc.usu.edu>
References: <462CC4C5.8020106@cc.usu.edu>
	<200704231149.27043.pgmdevlist@gmail.com>
	<462CDBD0.1060105@cc.usu.edu>
Message-ID: <200704231232.18645.pgmdevlist@gmail.com>

> > Have you tried nonzero() ?
>
> Nonzero isn't quite what I'm after, as the tests are more complicated
> than what I illustrated in my example.

Tests such as (a<0)&(b>1) will give you arrays of booleans. The nonzero give 
you where the two conditions are met (viz, where the results is True, or 1)

> > a[a<0] = numpy.random.normal(0,1)
>
> This is a neat construct that I didn't realize was possible.  However,
> it has the undesirable (in my case) effect of placing a single new
> random number in each locations where a<0.  While this could work, I
> ideally need a different random number chosen for each replaced value.
> Does that make sense?

Completely. So what about the slightly more complicated:
test = a<0
if test .any()
   a[test] = numpy.random.normal(0,1,size=len(test.nonzero()[0]))

test.nonzero() outputs a tuple of indices. test.nonzero()[0] gives you the 
indices along the first axis, len(test.nonzero()[0]) the number of elements 
where the condition is met. You could also get the same result with something 
like
test = a<0
if test .any()
   a[test] = numpy.random.normal(0,1,size=test.sum())

Actually, the following simpler solution  works as well:

a[a<0] = numpy.random.normal(0,1,size=a.size)


From pgmdevlist at gmail.com  Mon Apr 23 12:36:23 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 23 Apr 2007 12:36:23 -0400
Subject: [Numpy-discussion] efficient use of numpy.where() and .any()
In-Reply-To: <ce557a360704230925m232e30q44a3ddc40d124ef4@mail.gmail.com>
References: <462CC4C5.8020106@cc.usu.edu>
	<200704231149.27043.pgmdevlist@gmail.com>
	<ce557a360704230925m232e30q44a3ddc40d124ef4@mail.gmail.com>
Message-ID: <200704231236.23251.pgmdevlist@gmail.com>


> When you say "no python temps" I guess you mean, no temporary
> *variables*? If I understand correctly, this allocates a temporary
> boolean array to hold the result of "a<0".

Indeed, hence my precising "no *python* temps". There still be a tmp created 
at one point or another (if I'm not mistaken)

> This is an unfortunate limitation that comes from the fact that we
> can't override the behaviour of python's logical operations. a<b<c
> does the right thing for python scalars, but it does it by being
> expanded to (approximately) "a<b and b<c", and "and" doesn't do the
> right thing for arrays. 

Note that in addition of the bitwise operators, you can use the "logical_" 
functions. OK, you'll still end up w/ temporaries, but I wonder whether there 
couldn't be some tricks to bypass that...


From mpmusu at cc.usu.edu  Mon Apr 23 12:40:38 2007
From: mpmusu at cc.usu.edu (Mark.Miller)
Date: Mon, 23 Apr 2007 10:40:38 -0600
Subject: [Numpy-discussion] efficient use of numpy.where() and .any()
In-Reply-To: <462CDD95.9090106@gmail.com>
References: <462CC4C5.8020106@cc.usu.edu>
	<200704231149.27043.pgmdevlist@gmail.com> <462CDBD0.1060105@cc.usu.edu>
	<462CDD95.9090106@gmail.com>
Message-ID: <462CE186.1010908@cc.usu.edu>

Robert Kern wrote:
> Certainly. How about this?
> 
> mask = (a<0)
> a[mask] = numpy.random.normal(0, 1, size=mask.sum())
> 

That's slick.  I believe it's precisely what I'm after.

Appreciate it,

-Mark



From peridot.faceted at gmail.com  Mon Apr 23 12:47:07 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 23 Apr 2007 12:47:07 -0400
Subject: [Numpy-discussion] efficient use of numpy.where() and .any()
In-Reply-To: <200704231236.23251.pgmdevlist@gmail.com>
References: <462CC4C5.8020106@cc.usu.edu>
	<200704231149.27043.pgmdevlist@gmail.com>
	<ce557a360704230925m232e30q44a3ddc40d124ef4@mail.gmail.com>
	<200704231236.23251.pgmdevlist@gmail.com>
Message-ID: <ce557a360704230947ka59ff6bx2da76f6e9a077c89@mail.gmail.com>

On 23/04/07, Pierre GM <pgmdevlist at gmail.com> wrote:

> Note that in addition of the bitwise operators, you can use the "logical_"
> functions. OK, you'll still end up w/ temporaries, but I wonder whether there
> couldn't be some tricks to bypass that...

If you're really determined not to make many temps, you can use their
output arguments and do it all in-place on the first temporary. The
few times I've rewritten code that way it hasn't made an appreciable
positive difference in the speed, and it was sometimes significantly
slower, perhaps because of the increased memory footprint (the temps
were longer-lived than when I did it by hand). malloc() is really
really cheap, and garbage collection is also extremely cheap for huge
arrays.

Anne


From gael.varoquaux at normalesup.org  Mon Apr 23 13:15:08 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 23 Apr 2007 19:15:08 +0200
Subject: [Numpy-discussion] Inplace reshape
Message-ID: <20070423171508.GA8484@clipper.ens.fr>

Hi,

I thought I remembered there was a way to reshape in-place an array, but
neither google, nor greping my mailbox brings anything out.
Am I wrong, or is there indeed a way to reshape in-place an array ?

Cheers,

Ga?l


From peridot.faceted at gmail.com  Mon Apr 23 13:20:18 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 23 Apr 2007 13:20:18 -0400
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <20070423171508.GA8484@clipper.ens.fr>
References: <20070423171508.GA8484@clipper.ens.fr>
Message-ID: <ce557a360704231020p3b9c9389v6798ad181ecb2231@mail.gmail.com>

On 23/04/07, Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:
> Hi,
>
> I thought I remembered there was a way to reshape in-place an array, but
> neither google, nor greping my mailbox brings anything out.
> Am I wrong, or is there indeed a way to reshape in-place an array ?

Sometimes it's just impossible to reshape an array without copying.
But reshape()  (either the method or the function) copies only the
array descriptor if that's possible. So you'll have a new ndarray
object, but the data will not be copied. Assigning to the shape
attribute might do what you want (if it triggers a reshape()
internally, which I think it does).

I don't think it's possible to *rearrange* the data in memory in
place. I can't even think of an algorithm to do that, off the top of
my head.

Anne


From tim.hochberg at ieee.org  Mon Apr 23 13:20:43 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Mon, 23 Apr 2007 10:20:43 -0700
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <20070423171508.GA8484@clipper.ens.fr>
References: <20070423171508.GA8484@clipper.ens.fr>
Message-ID: <e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>

On 4/23/07, Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:
>
> Hi,
>
> I thought I remembered there was a way to reshape in-place an array, but
> neither google, nor greping my mailbox brings anything out.
> Am I wrong, or is there indeed a way to reshape in-place an array ?


Just set the shape of the array:

    somearray.shape = newshape

Should do the trick

regards,

-tim


Cheers,
>
> Ga?l
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070423/2593c419/attachment.html>

From robert.kern at gmail.com  Mon Apr 23 13:20:51 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 Apr 2007 12:20:51 -0500
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <20070423171508.GA8484@clipper.ens.fr>
References: <20070423171508.GA8484@clipper.ens.fr>
Message-ID: <462CEAF3.90004@gmail.com>

Gael Varoquaux wrote:
> Hi,
> 
> I thought I remembered there was a way to reshape in-place an array, but
> neither google, nor greping my mailbox brings anything out.
> Am I wrong, or is there indeed a way to reshape in-place an array ?

.reshape()

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Mon Apr 23 13:25:28 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 23 Apr 2007 19:25:28 +0200
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>
References: <20070423171508.GA8484@clipper.ens.fr>
	<e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>
Message-ID: <20070423172526.GB8484@clipper.ens.fr>

On Mon, Apr 23, 2007 at 10:20:43AM -0700, Timothy Hochberg wrote:
>    Just set the shape of the array:

>        somearray.shape = newshape

Of course ! Thanks

On Mon, Apr 23, 2007 at 12:20:51PM -0500, Robert Kern wrote:
> .reshape()

Unless I miss something obvious "a.reshape()" doesn't modify a, which is
somewhat missleading, IMHO.

Cheers,

Ga?l


From robert.kern at gmail.com  Mon Apr 23 13:33:25 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 Apr 2007 12:33:25 -0500
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <20070423172526.GB8484@clipper.ens.fr>
References: <20070423171508.GA8484@clipper.ens.fr>	<e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>
	<20070423172526.GB8484@clipper.ens.fr>
Message-ID: <462CEDE5.30104@gmail.com>

Gael Varoquaux wrote:
> On Mon, Apr 23, 2007 at 12:20:51PM -0500, Robert Kern wrote:
>> .reshape()
> 
> Unless I miss something obvious "a.reshape()" doesn't modify a, which is
> somewhat missleading, IMHO.

Nope, you caught me in a moment of stupidity.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Mon Apr 23 13:36:26 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 23 Apr 2007 10:36:26 -0700
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <20070423172526.GB8484@clipper.ens.fr>
References: <20070423171508.GA8484@clipper.ens.fr>
	<e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>
	<20070423172526.GB8484@clipper.ens.fr>
Message-ID: <462CEE9A.5030405@noaa.gov>

Gael Varoquaux wrote:
> Unless I miss something obvious "a.reshape()" doesn't modify a, which is
> somewhat missleading, IMHO.

quite correct. .reshape() creates a new array that shared data with the 
original:

 >>> import numpy
 >>> a = numpy.zeros((2,3))

 >>> help(a.reshape)
Help on built-in function reshape:

reshape(...)
     a.reshape(d1, d2, ..., dn, order='c')

     Return a new array from this one.  The new array must have the same 

     number of elements as self.  Also always returns a view or raises a
     ValueError if that is impossible.;

 >>> a
array([[ 0.,  0.,  0.],
        [ 0.,  0.,  0.]])

 >>> b = a.reshape((6,))
 >>> a
array([[ 0.,  0.,  0.],
        [ 0.,  0.,  0.]])

so a hasn't changed.

 >>> b
array([ 0.,  0.,  0.,  0.,  0.,  0.])

but b is a different shape.

 >>> b[1] = 5
 >>> a
array([[ 0.,  5.,  0.],
        [ 0.,  0.,  0.]])

b and a share data.

if you want to change the shape of a:

 >>> a.shape = (6,)
 >>> a
array([ 0.,  5.,  0.,  0.,  0.,  0.])

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From pgmdevlist at gmail.com  Mon Apr 23 13:44:56 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 23 Apr 2007 13:44:56 -0400
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <462CEE9A.5030405@noaa.gov>
References: <20070423171508.GA8484@clipper.ens.fr>
	<20070423172526.GB8484@clipper.ens.fr> <462CEE9A.5030405@noaa.gov>
Message-ID: <200704231344.56709.pgmdevlist@gmail.com>

On Monday 23 April 2007 13:36:26 Christopher Barker wrote:
> Gael Varoquaux wrote:
> > Unless I miss something obvious "a.reshape()" doesn't modify a, which is
> > somewhat missleading, IMHO.
>
> quite correct. .reshape() creates a new array that shared data with the
> original:

Mmh. My understanding is that .reshape() creates a *view* of the original 
array, so no data is actually copied. The only thing that changes is how the 
data is read. That means that modifying the reshaped version will modify the 
original.

So:

>>>import numpy as N
>>>a=N.zeros((3,2))
array([[ 0., ?0., ?0.],
? ? ? ? [ 0., ?0., ?0.]])
>>>b=a.reshape(6,)
>>>b
array([ 0., ?0., ?0., ?0., ?0., ?0.])

>>>b += 1
>>>b
array([ 1.,  1.,  1.,  1.,  1.,  1.])
>>>a
array([[ 1.,  1.],
       [ 1.,  1.],
       [ 1.,  1.]])


From charlesr.harris at gmail.com  Mon Apr 23 13:46:13 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 23 Apr 2007 11:46:13 -0600
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <462CEE9A.5030405@noaa.gov>
References: <20070423171508.GA8484@clipper.ens.fr>
	<e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>
	<20070423172526.GB8484@clipper.ens.fr> <462CEE9A.5030405@noaa.gov>
Message-ID: <e06186140704231046i67586476p1de73b8a75137a78@mail.gmail.com>

On 4/23/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
>
> Gael Varoquaux wrote:
> > Unless I miss something obvious "a.reshape()" doesn't modify a, which is
> > somewhat missleading, IMHO.
>
> quite correct. .reshape() creates a new array that shared data with the
> original:


Sometimes it do, sometimes it don't:

In [8]: x = ones((8,8))

In [9]: y = x[:3,:6]

In [10]: z = y.reshape(2,9)

In [11]: z[...] = 0

In [12]: x
Out[12]:
array([[ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.]])

In [13]: z
Out[13]:
array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]

As Ann pointed out, sometimes reusing the same array isn't possible. This
makes the use of reshaped arrays as l-values subject to subtle errors, so
caution is advised.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070423/ebcd14c2/attachment.html>

From kwgoodman at gmail.com  Mon Apr 23 13:55:19 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 23 Apr 2007 10:55:19 -0700
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <462CEE9A.5030405@noaa.gov>
References: <20070423171508.GA8484@clipper.ens.fr>
	<e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>
	<20070423172526.GB8484@clipper.ens.fr> <462CEE9A.5030405@noaa.gov>
Message-ID: <f4f93d420704231055w13822a99k835ca5bc01ce5a35@mail.gmail.com>

On 4/23/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> reshape(...)
>      a.reshape(d1, d2, ..., dn, order='c')
>
>      Return a new array from this one.  The new array must have the same
>
>      number of elements as self.  Also always returns a view or raises a
>      ValueError if that is impossible.;

Here's a better doc string that explains "This will be a new view
object if possible; otherwise, it will return a copy."

>> numpy.reshape?
Type:           function
Base Class:     <type 'function'>
String Form:    <function reshape at 0xb78894c4>
Namespace:      Interactive
File:           /usr/local/lib/python2.4/site-packages/numpy/core/fromnumeric.py
Definition:     numpy.reshape(a, newshape, order='C')
Docstring:
    Return an array that uses the data of the given array, but with a new
    shape.

    :Parameters:
      - `a` : array
      - `newshape` : shape tuple or int
        The new shape should be compatible with the original shape. If an
        integer, then the result will be a 1D array of that length.
      - `order` : 'C' or 'FORTRAN', optional (default='C')
        Whether the array data should be viewed as in C (row-major) order or
        FORTRAN (column-major) order.

    :Returns:
      - `reshaped_array` : array
        This will be a new view object if possible; otherwise, it will return
        a copy.

    :See also:
      numpy.ndarray.reshape() is the equivalent method.


From faltet at carabos.com  Mon Apr 23 13:57:04 2007
From: faltet at carabos.com (Francesc Altet)
Date: Mon, 23 Apr 2007 19:57:04 +0200
Subject: [Numpy-discussion] efficient use of numpy.where() and .any()
In-Reply-To: <ce557a360704230947ka59ff6bx2da76f6e9a077c89@mail.gmail.com>
References: <462CC4C5.8020106@cc.usu.edu>
	<200704231149.27043.pgmdevlist@gmail.com>
	<ce557a360704230925m232e30q44a3ddc40d124ef4@mail.gmail.com>
	<200704231236.23251.pgmdevlist@gmail.com>
	<ce557a360704230947ka59ff6bx2da76f6e9a077c89@mail.gmail.com>
Message-ID: <1177351024.2614.11.camel@localhost.localdomain>

El dl 23 de 04 del 2007 a les 12:47 -0400, en/na Anne Archibald va
escriure:
> On 23/04/07, Pierre GM <pgmdevlist at gmail.com> wrote:
> 
> > Note that in addition of the bitwise operators, you can use the "logical_"
> > functions. OK, you'll still end up w/ temporaries, but I wonder whether there
> > couldn't be some tricks to bypass that...
> 
> If you're really determined not to make many temps, you can use their
> output arguments and do it all in-place on the first temporary. The
> few times I've rewritten code that way it hasn't made an appreciable
> positive difference in the speed, and it was sometimes significantly
> slower, perhaps because of the increased memory footprint (the temps
> were longer-lived than when I did it by hand). malloc() is really
> really cheap, and garbage collection is also extremely cheap for huge
> arrays.

Or you can use numexpr. Numexpr doesn't need temporaries (provided that
all the arrays in the expresion are contiguous). It only uses a small
buffer for carrying out computations whose size is, when compared with
large matrices, negligible. It is also usally faster than the pure NumPy
approach:

# Pure NumPy
>>> Timer("c=(i>2) | ((f**2>3) & ~(i*f<2))", "import numpy;
i=numpy.arange(10**6); f=numpy.random.rand(10**6)").repeat(3, 10)
[0.55586099624633789, 0.55565214157104492, 0.556427001953125]

# Using Numexpr
>>> Timer("c=numexpr.evaluate('(i>2) | ((f**2>3) & ~(i*f<2))')", "import
tables; import tables.numexpr as numexpr; import numpy;
i=numpy.arange(10**6); f=numpy.random.rand(10**6)").repeat(3, 10)
[0.25569510459899902, 0.25547599792480469, 0.25554895401000977]

I've used here the numexpr instance that comes with PyTables, but you
can use also the one in the scipy's sandbox as it also supports booleans
since some months ago.

Cheers,

-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth



From charlesr.harris at gmail.com  Mon Apr 23 14:04:02 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 23 Apr 2007 12:04:02 -0600
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <f4f93d420704231055w13822a99k835ca5bc01ce5a35@mail.gmail.com>
References: <20070423171508.GA8484@clipper.ens.fr>
	<e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>
	<20070423172526.GB8484@clipper.ens.fr> <462CEE9A.5030405@noaa.gov>
	<f4f93d420704231055w13822a99k835ca5bc01ce5a35@mail.gmail.com>
Message-ID: <e06186140704231104h3d8e5ae7x6c4651f96b68bb09@mail.gmail.com>

On 4/23/07, Keith Goodman <kwgoodman at gmail.com> wrote:
>
> On 4/23/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> > reshape(...)
> >      a.reshape(d1, d2, ..., dn, order='c')
> >
> >      Return a new array from this one.  The new array must have the same
> >
> >      number of elements as self.  Also always returns a view or raises a
> >      ValueError if that is impossible.;
>
> Here's a better doc string that explains "This will be a new view
> object if possible; otherwise, it will return a copy."
>
> >> numpy.reshape?
> Type:           function
> Base Class:     <type 'function'>
> String Form:    <function reshape at 0xb78894c4>
> Namespace:      Interactive
> File:
> /usr/local/lib/python2.4/site-packages/numpy/core/fromnumeric.py
> Definition:     numpy.reshape(a, newshape, order='C')
> Docstring:
>     Return an array that uses the data of the given array, but with a new
>     shape.
>
>     :Parameters:
>       - `a` : array
>       - `newshape` : shape tuple or int
>         The new shape should be compatible with the original shape. If an
>         integer, then the result will be a 1D array of that length.
>       - `order` : 'C' or 'FORTRAN', optional (default='C')
>         Whether the array data should be viewed as in C (row-major) order
> or
>         FORTRAN (column-major) order.
>
>     :Returns:
>       - `reshaped_array` : array
>         This will be a new view object if possible; otherwise, it will
> return
>         a copy.
>
>     :See also:
>       numpy.ndarray.reshape() is the equivalent method.



I think that it should raise an error, or warn, if it needs to make a copy,
but that isn't the  tradition. Reshape does raise an error if the product of
the dimensions isn't the same for both the original and the reshaped array.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070423/e8a3ed23/attachment.html>

From Chris.Barker at noaa.gov  Mon Apr 23 14:29:42 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 23 Apr 2007 11:29:42 -0700
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <e06186140704231104h3d8e5ae7x6c4651f96b68bb09@mail.gmail.com>
References: <20070423171508.GA8484@clipper.ens.fr>
	<e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>
	<20070423172526.GB8484@clipper.ens.fr> <462CEE9A.5030405@noaa.gov>
	<f4f93d420704231055w13822a99k835ca5bc01ce5a35@mail.gmail.com>
	<e06186140704231104h3d8e5ae7x6c4651f96b68bb09@mail.gmail.com>
Message-ID: <462CFB16.4040806@noaa.gov>

Charles R Harris wrote:
>     Here's a better doc string that explains "This will be a new view
>     object if possible; otherwise, it will return a copy."

Does this exist somewhere, or are you contributing it now?

> I think that it should raise an error, or warn, if it needs to make a 
> copy,

I totally agree, and that behavior matches the current ('1.0.2') docstring.

> but that isn't the  tradition.

so sad -- could this be deprecated. I know we had a pretty length 
discussion about this issue in another context. In general, it's a bad 
idea for methods to sometimes return copies and sometimes views -- it's 
begging for subtle bugs!

oh well, either the code or the docstring should change, in any case.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From kwgoodman at gmail.com  Mon Apr 23 14:43:12 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 23 Apr 2007 11:43:12 -0700
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <462CFB16.4040806@noaa.gov>
References: <20070423171508.GA8484@clipper.ens.fr>
	<e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>
	<20070423172526.GB8484@clipper.ens.fr> <462CEE9A.5030405@noaa.gov>
	<f4f93d420704231055w13822a99k835ca5bc01ce5a35@mail.gmail.com>
	<e06186140704231104h3d8e5ae7x6c4651f96b68bb09@mail.gmail.com>
	<462CFB16.4040806@noaa.gov>
Message-ID: <f4f93d420704231143u77cd0fceqd2e69e327099cefe@mail.gmail.com>

On 4/23/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> Charles R Harris wrote:
> >     Here's a better doc string that explains "This will be a new view
> >     object if possible; otherwise, it will return a copy."
>
> Does this exist somewhere, or are you contributing it now?

At the moment numpy.reshape and array.reshape have different doc
strings (I'm using numpy 1.0.2.dev3546). The one I pasted is from
numpy.reshape.


From christian at marquardt.sc  Mon Apr 23 16:20:48 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Mon, 23 Apr 2007 22:20:48 +0200 (CEST)
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
Message-ID: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>

Dear all,

this is odd:

   >>> import numpy as np
   >>> fact = 28250000L * 86400L
   >>> nn = np.array([-20905000L])
   >>> nn
   array([-20905000], dtype=int64)
   >>> nn[0] // fact
   0

But:

   >>> long(nn[0]) // fact
   -1L

Is this a bug in numpy, or in python's implementation of longs? I would
think both should give the same, really... (Python 2.5, numpy 1.0.3dev3725,
Linux, Intel compilers...)

Many thanks for any ideas / advice,

  Christian



From christian at marquardt.sc  Mon Apr 23 16:29:00 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Mon, 23 Apr 2007 22:29:00 +0200 (CEST)
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>
Message-ID: <27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc>

Actually,

it happens for normal integers as well:

   >>> n = np.array([-5, -100, -150])
   >>> n // 100
   array([ 0, -1, -1])
   >>> -5//100, -100//100, -150//100
   (-1, -1, -2)

On Mon, April 23, 2007 22:20, Christian Marquardt wrote:
> Dear all,
>
> this is odd:
>
>    >>> import numpy as np
>    >>> fact = 28250000L * 86400L
>    >>> nn = np.array([-20905000L])
>    >>> nn
>    array([-20905000], dtype=int64)
>    >>> nn[0] // fact
>    0
>
> But:
>
>    >>> long(nn[0]) // fact
>    -1L
>
> Is this a bug in numpy, or in python's implementation of longs? I would
> think both should give the same, really... (Python 2.5, numpy
> 1.0.3dev3725,
> Linux, Intel compilers...)
>
> Many thanks for any ideas / advice,
>
>   Christian
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>




From christian at marquardt.sc  Mon Apr 23 16:41:56 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Mon, 23 Apr 2007 22:41:56 +0200 (CEST)
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>
	<27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc>
Message-ID: <9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc>

Hmmm,

On Mon, April 23, 2007 22:29, Christian Marquardt wrote:
> Actually,
>
> it happens for normal integers as well:
>
>    >>> n = np.array([-5, -100, -150])
>    >>> n // 100
>    array([ 0, -1, -1])
>    >>> -5//100, -100//100, -150//100
>    (-1, -1, -2)

and finally:

   >>> n % 100
   array([95,  0, 50])
   >>> -5 % 100, -100 % 100, -150 % 100
   (95, 0, 50)

So plain python / using long provides consistent results across //
and %, but numpy doesn't...

  Christian.

> On Mon, April 23, 2007 22:20, Christian Marquardt wrote:
>> Dear all,
>>
>> this is odd:
>>
>>    >>> import numpy as np
>>    >>> fact = 28250000L * 86400L
>>    >>> nn = np.array([-20905000L])
>>    >>> nn
>>    array([-20905000], dtype=int64)
>>    >>> nn[0] // fact
>>    0
>>
>> But:
>>
>>    >>> long(nn[0]) // fact
>>    -1L
>>
>> Is this a bug in numpy, or in python's implementation of longs? I would
>> think both should give the same, really... (Python 2.5, numpy
>> 1.0.3dev3725,
>> Linux, Intel compilers...)
>>
>> Many thanks for any ideas / advice,
>>
>>   Christian
>>
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>




From Chris.Barker at noaa.gov  Mon Apr 23 20:05:17 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 23 Apr 2007 17:05:17 -0700
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <f4f93d420704231143u77cd0fceqd2e69e327099cefe@mail.gmail.com>
References: <20070423171508.GA8484@clipper.ens.fr>
	<e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>
	<20070423172526.GB8484@clipper.ens.fr> <462CEE9A.5030405@noaa.gov>
	<f4f93d420704231055w13822a99k835ca5bc01ce5a35@mail.gmail.com>
	<e06186140704231104h3d8e5ae7x6c4651f96b68bb09@mail.gmail.com>
	<462CFB16.4040806@noaa.gov>
	<f4f93d420704231143u77cd0fceqd2e69e327099cefe@mail.gmail.com>
Message-ID: <462D49BD.3090809@noaa.gov>

Keith Goodman wrote:

> At the moment numpy.reshape and array.reshape have different doc
> strings (I'm using numpy 1.0.2.dev3546). The one I pasted is from
> numpy.reshape.

And I see from there:
     :See also:
       numpy.ndarray.reshape() is the equivalent method.

so it looks like they are the same thing. On the other hand, I *think* 
that some of the ndarray methods are slightly different, but the 
functions have been retained with the old functionality for backward 
compatibility reasons. However, it looks like ndarray.reshape() is 
behaving like the docstring for numpy.reshape(), so there is a bug of 
some sort (probably a doc bug).

if numpy.reshape() is delegating to ndarray.reshape() couldn't they 
share docstrings somehow?

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From robert.kern at gmail.com  Mon Apr 23 20:35:06 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 Apr 2007 19:35:06 -0500
Subject: [Numpy-discussion] Inplace reshape
In-Reply-To: <462D49BD.3090809@noaa.gov>
References: <20070423171508.GA8484@clipper.ens.fr>	<e4412d6b0704231020i3c999242lca6c721dd9b34baf@mail.gmail.com>	<20070423172526.GB8484@clipper.ens.fr>
	<462CEE9A.5030405@noaa.gov>	<f4f93d420704231055w13822a99k835ca5bc01ce5a35@mail.gmail.com>	<e06186140704231104h3d8e5ae7x6c4651f96b68bb09@mail.gmail.com>	<462CFB16.4040806@noaa.gov>	<f4f93d420704231143u77cd0fceqd2e69e327099cefe@mail.gmail.com>
	<462D49BD.3090809@noaa.gov>
Message-ID: <462D50BA.4070504@gmail.com>

Christopher Barker wrote:

> if numpy.reshape() is delegating to ndarray.reshape() couldn't they 
> share docstrings somehow?

Yes, of course. I got bored halfway through the conversions and didn't get to
doing the methods. If you want to speed up the process, please submit a patch.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From cookedm at physics.mcmaster.ca  Mon Apr 23 21:35:20 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Mon, 23 Apr 2007 21:35:20 -0400
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>
	<27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc>
	<9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc>
Message-ID: <0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca>

On Apr 23, 2007, at 16:41 , Christian Marquardt wrote:
> On Mon, April 23, 2007 22:29, Christian Marquardt wrote:
>> Actually,
>>
>> it happens for normal integers as well:
>>
>>>>> n = np.array([-5, -100, -150])
>>>>> n // 100
>>    array([ 0, -1, -1])
>>>>> -5//100, -100//100, -150//100
>>    (-1, -1, -2)
>
> and finally:
>
>>>> n % 100
>    array([95,  0, 50])
>>>> -5 % 100, -100 % 100, -150 % 100
>    (95, 0, 50)
>
> So plain python / using long provides consistent results across //
> and %, but numpy doesn't...

Python defines x // y as returning the floor of the division, and x %  
y has the same sign as y. However, in C89, it is implementation- 
defined (i.e., portability-pain-in-the-ass) whether the floor or ceil  
is used when the signs of x and y differ. In C99, the result should  
be truncated. From the C99 spec, sec. 6.5.5, #6:
        When integers are divided, the result of the / operator
        is   the   algebraic   quotient  with  any  fractional  part
        discarded.76)  If the quotient  a/b  is  representable,  the
        expression (a/b)*b + a%b shall equal a.

Numpy follows whatever the C compiler decides to use, because of  
speed-vs.-Python compatibility tradeoff.

>
>   Christian.
>
>> On Mon, April 23, 2007 22:20, Christian Marquardt wrote:
>>> Dear all,
>>>
>>> this is odd:
>>>
>>>>>> import numpy as np
>>>>>> fact = 28250000L * 86400L
>>>>>> nn = np.array([-20905000L])
>>>>>> nn
>>>    array([-20905000], dtype=int64)
>>>>>> nn[0] // fact
>>>    0
>>>
>>> But:
>>>
>>>>>> long(nn[0]) // fact
>>>    -1L
>>>
>>> Is this a bug in numpy, or in python's implementation of longs? I  
>>> would
>>> think both should give the same, really... (Python 2.5, numpy
>>> 1.0.3dev3725,
>>> Linux, Intel compilers...)
>>>
>>> Many thanks for any ideas / advice,
>>>
>>>   Christian

-- 
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070423/f166593d/attachment.sig>

From oliphant.travis at ieee.org  Mon Apr 23 21:44:20 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Mon, 23 Apr 2007 19:44:20 -0600
Subject: [Numpy-discussion] uint64 typecasting with scalars broken (?)
In-Reply-To: <19721.84.167.104.46.1177276782.squirrel@webmail.marquardt.sc>
References: <19721.84.167.104.46.1177276782.squirrel@webmail.marquardt.sc>
Message-ID: <462D60F4.1050803@ieee.org>

Christian Marquardt wrote:
> Hello,
>
> The following is what I expected...
>
>    >>> y = 1234
>    >>> x = array([1], dtype = "uint64")
>    >>> print x + y, (x + y).dtype.type
>    [1235] <type 'numpy.uint64'>
>
>   

This is "what you expect" only because y is a scalar and cannot 
determine the "kind" of the output.

> but is this the way it should be? (numpy 1.0.2, Linux, Intel comilers)
>
>    >>> print x[0] + y, type(x[0] + y)
>    1235.0 <type 'numpy.float64'>
>   

This is correct (sort of) because in a mixed operation between uint64 
and int32, because there is no int128, the sum must be placed in a 
float.  In reality it should be a long-double float but it was decided 
not to perpetuate long double floats like this because then on 64-bit 
platforms they would be showing up everywhere.

-Travis



From focke at slac.stanford.edu  Mon Apr 23 22:04:10 2007
From: focke at slac.stanford.edu (Warren Focke)
Date: Mon, 23 Apr 2007 19:04:10 -0700 (PDT)
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>
	<27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc>
	<9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc>
	<0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca>
Message-ID: <Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu>

But even C89 required that x == (x/y)*y + (x%y), and that's not the case
here.

w

On Mon, 23 Apr 2007, David M. Cooke wrote:

> On Apr 23, 2007, at 16:41 , Christian Marquardt wrote:
> > On Mon, April 23, 2007 22:29, Christian Marquardt wrote:
> >> Actually,
> >>
> >> it happens for normal integers as well:
> >>
> >>>>> n = np.array([-5, -100, -150])
> >>>>> n // 100
> >>    array([ 0, -1, -1])
> >>>>> -5//100, -100//100, -150//100
> >>    (-1, -1, -2)
> >
> > and finally:
> >
> >>>> n % 100
> >    array([95,  0, 50])
> >>>> -5 % 100, -100 % 100, -150 % 100
> >    (95, 0, 50)
> >
> > So plain python / using long provides consistent results across //
> > and %, but numpy doesn't...
>
> Python defines x // y as returning the floor of the division, and x %
> y has the same sign as y. However, in C89, it is implementation-
> defined (i.e., portability-pain-in-the-ass) whether the floor or ceil
> is used when the signs of x and y differ. In C99, the result should
> be truncated. From the C99 spec, sec. 6.5.5, #6:
>         When integers are divided, the result of the / operator
>         is   the   algebraic   quotient  with  any  fractional  part
>         discarded.76)  If the quotient  a/b  is  representable,  the
>         expression (a/b)*b + a%b shall equal a.
>
> Numpy follows whatever the C compiler decides to use, because of
> speed-vs.-Python compatibility tradeoff.
>
> >
> >   Christian.
> >
> >> On Mon, April 23, 2007 22:20, Christian Marquardt wrote:
> >>> Dear all,
> >>>
> >>> this is odd:
> >>>
> >>>>>> import numpy as np
> >>>>>> fact = 28250000L * 86400L
> >>>>>> nn = np.array([-20905000L])
> >>>>>> nn
> >>>    array([-20905000], dtype=int64)
> >>>>>> nn[0] // fact
> >>>    0
> >>>
> >>> But:
> >>>
> >>>>>> long(nn[0]) // fact
> >>>    -1L
> >>>
> >>> Is this a bug in numpy, or in python's implementation of longs? I
> >>> would
> >>> think both should give the same, really... (Python 2.5, numpy
> >>> 1.0.3dev3725,
> >>> Linux, Intel compilers...)
> >>>
> >>> Many thanks for any ideas / advice,
> >>>
> >>>   Christian
>
> --
> |>|\/|<
> /------------------------------------------------------------------\
> |David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
> |cookedm at physics.mcmaster.ca
>
>


From charlesr.harris at gmail.com  Mon Apr 23 23:27:39 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 23 Apr 2007 21:27:39 -0600
Subject: [Numpy-discussion] uint64 typecasting with scalars broken (?)
In-Reply-To: <462D60F4.1050803@ieee.org>
References: <19721.84.167.104.46.1177276782.squirrel@webmail.marquardt.sc>
	<462D60F4.1050803@ieee.org>
Message-ID: <e06186140704232027q70b4127tac9b42f0dcfd0639@mail.gmail.com>

On 4/23/07, Travis Oliphant <oliphant.travis at ieee.org> wrote:
>
> Christian Marquardt wrote:
> > Hello,
> >
> > The following is what I expected...
> >
> >    >>> y = 1234
> >    >>> x = array([1], dtype = "uint64")
> >    >>> print x + y, (x + y).dtype.type
> >    [1235] <type 'numpy.uint64'>
> >
> >
>
> This is "what you expect" only because y is a scalar and cannot
> determine the "kind" of the output.
>
> > but is this the way it should be? (numpy 1.0.2, Linux, Intel comilers)
> >
> >    >>> print x[0] + y, type(x[0] + y)
> >    1235.0 <type 'numpy.float64'>
> >
>
> This is correct (sort of) because in a mixed operation between uint64
> and int32, because there is no int128, the sum must be placed in a
> float.  In reality it should be a long-double float but it was decided
> not to perpetuate long double floats like this because then on 64-bit
> platforms they would be showing up everywhere.


I wonder if returning int64 wouldn't be better in this case. It has more
precision than a double and has the advantage of being an integer. True,
uint64s with the msb set would be wrongly interpreted, but... Or maybe throw
an error when mixing the two, since really the result can't be relied on. If
the latter, it would still be nice have an interpretation for python
integers, so just interpret them as uint64 (I believe a C type cast does
this) and just add using the usual modular arithmetic. This still allows
incrementing and decrementing. For instance, using uint2 as an example,

3 - 1 == 3 + (-1) == 0x11 + 0x11 == 0x10 == 2

I think this makes the most sense, after all, subtraction is defined for
uints and we already use modular arithmetic. I admit some strange results
can show up, but no stranger than treating the integers as floats having 51
bit precision. I suppose we could raise an error on integer overflow, but
that isn't how we have done things with the other integers.

Of course, all the other mixed integer types with the same number of bits
could be treated the same way, which would at least be consistent. The user
would then have to specify a larger type whenever it was needed and
explicitly deal with the case of uint64. Let's see what C does:

#include <stdio.h>

int main(int argc, char** args)
{
    unsigned long long x = 0;
    long long y = -1;

    printf("%Ld\n", x + y);

    return 1;
}

prints -1, as expected, and doesn't issue a compiler warning with -Wall.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070423/94ce13b2/attachment.html>

From charlesr.harris at gmail.com  Mon Apr 23 23:34:35 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 23 Apr 2007 21:34:35 -0600
Subject: [Numpy-discussion] uint64 typecasting with scalars broken (?)
In-Reply-To: <e06186140704232027q70b4127tac9b42f0dcfd0639@mail.gmail.com>
References: <19721.84.167.104.46.1177276782.squirrel@webmail.marquardt.sc>
	<462D60F4.1050803@ieee.org>
	<e06186140704232027q70b4127tac9b42f0dcfd0639@mail.gmail.com>
Message-ID: <e06186140704232034h3c677fe4nb1622b27ab510139@mail.gmail.com>

On 4/23/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
>
>
>
> On 4/23/07, Travis Oliphant <oliphant.travis at ieee.org> wrote:
> >
> > Christian Marquardt wrote:
> > > Hello,
> > >
> > > The following is what I expected...
> > >
> > >    >>> y = 1234
> > >    >>> x = array([1], dtype = "uint64")
> > >    >>> print x + y, (x + y).dtype.type
> > >    [1235] <type 'numpy.uint64'>
> > >
> > >
> >
> > This is "what you expect" only because y is a scalar and cannot
> > determine the "kind" of the output.
> >
> > > but is this the way it should be? (numpy 1.0.2, Linux, Intel comilers)
> > >
> > >    >>> print x[0] + y, type(x[0] + y)
> > >    1235.0 <type 'numpy.float64'>
> > >
> >
> > This is correct (sort of) because in a mixed operation between uint64
> > and int32, because there is no int128, the sum must be placed in a
> > float.  In reality it should be a long-double float but it was decided
> > not to perpetuate long double floats like this because then on 64-bit
> > platforms they would be showing up everywhere.
>
>
> I wonder if returning int64 wouldn't be better in this case. It has more
> precision than a double and has the advantage of being an integer. True,
> uint64s with the msb set would be wrongly interpreted, but... Or maybe throw
> an error when mixing the two, since really the result can't be relied on. If
> the latter, it would still be nice have an interpretation for python
> integers, so just interpret them as uint64 (I believe a C type cast does
> this) and just add using the usual modular arithmetic. This still allows
> incrementing and decrementing. For instance, using uint2 as an example,
>
> 3 - 1 == 3 + (-1) == 0x11 + 0x11 == 0x10 == 2
>
> I think this makes the most sense, after all, subtraction is defined for
> uints and we already use modular arithmetic. I admit some strange results
> can show up, but no stranger than treating the integers as floats having 51
> bit precision. I suppose we could raise an error on integer overflow, but
> that isn't how we have done things with the other integers.
>
> Of course, all the other mixed integer types with the same number of bits
> could be treated the same way, which would at least be consistent. The user
> would then have to specify a larger type whenever it was needed and
> explicitly deal with the case of uint64. Let's see what C does:
>
> #include <stdio.h>
>
> int main(int argc, char** args)
> {
>     unsigned long long x = 0;
>     long long y = -1;
>
>     printf("%Ld\n", x + y);
>
>     return 1;
> }
>
> prints -1, as expected, and doesn't issue a compiler warning with -Wall.
>

And using %Lu instead of %Ld prints 18446744073709551615, which is the same
bit pattern interpreted as unsigned.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070423/36b89b86/attachment.html>

From markbak at gmail.com  Tue Apr 24 03:28:35 2007
From: markbak at gmail.com (mark)
Date: Tue, 24 Apr 2007 07:28:35 -0000
Subject: [Numpy-discussion] ScipyTest Warning?
Message-ID: <1177399715.220081.61800@n15g2000prd.googlegroups.com>

Hello All -

I have a piece of code that works fine for me, but a friend tries to
run it and gets this warning.
He claims to have updated his Python (2.4), Scipy and numpy. A
Does anybody know what import triggers this Warning? I didn't think I
imported ScipyTest.

Thanks, Mark

Warning (from warnings module):
 File "C:\Python24\lib\site-packages\numpy\testing\numpytest.py", line
63
   DeprecationWarning)
DeprecationWarning: ScipyTest is now called NumpyTest; please update
your code



From stefan at sun.ac.za  Tue Apr 24 04:14:53 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Tue, 24 Apr 2007 10:14:53 +0200
Subject: [Numpy-discussion] ScipyTest Warning?
In-Reply-To: <1177399715.220081.61800@n15g2000prd.googlegroups.com>
References: <1177399715.220081.61800@n15g2000prd.googlegroups.com>
Message-ID: <20070424081453.GT6933@mentat.za.net>

Hi Mark

On Tue, Apr 24, 2007 at 07:28:35AM -0000, mark wrote:
> I have a piece of code that works fine for me, but a friend tries to
> run it and gets this warning.
> He claims to have updated his Python (2.4), Scipy and numpy. A
> Does anybody know what import triggers this Warning? I didn't think I
> imported ScipyTest.

This has been fixed in the latest versions of numpy/scipy.  There is
no need to be concerned, though, since it has no impact on the working
of your code.

Cheers
St?fan


From persed at princeton.edu  Fri Apr 20 09:38:45 2007
From: persed at princeton.edu (Per B. Sederberg)
Date: Fri, 20 Apr 2007 09:38:45 -0400
Subject: [Numpy-discussion] Bus Error with string in ndarray with named
	fields
Message-ID: <6b7179780704200638i44fa02b4nf347021fe102f054@mail.gmail.com>

Hi Folks:

I'm getting a very strange bus error in the recent versions of numpy
(almost current svn).  Here's how you can (hopefully) replicate it:

On my MacBook:

Python 2.4.3 (#1, Apr  7 2006, 10:54:33)
[GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as N
>>> N.version.version
'1.0.2.dev3569'
>>> fields = [('x',(N.str,40))]
>>> dat = N.zeros(1,fields)
>>> dat
Bus error


On my linux cluster:

Python 2.4.3 (#1, May  8 2006, 11:36:25)
[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-49)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as N
>>> N.version.version
'1.0.2.dev3567'
>>> fields = [('x',(N.str,40))]
>>> dat = N.zeros(1,fields)
>>> dat
Segmentation fault


But it works on my linux desktop with an older numpy:

Python 2.4.4c1 (#2, Oct 11 2006, 20:00:03)
[GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as N
>>> N.version.version
'1.0rc1'
>>> fields = [('x',(N.str,40))]
>>> dat = N.zeros(1,fields)
>>> dat
array([('',)],
      dtype=[('x', '|S40')])
>>>


Does anyone have any clue as to how to deal with this?  Is there a
better way to create an ndarray with named fields (recarray)?  My real
code has a number of fields of various types, but I was able to narrow
it down to anytime I use a string type.  For example, this works on
the machines where using a string fails:

>>> import numpy as N
>>> fields = [('x',N.bool),('y',N.int32)]
>>> dat = N.zeros(1,fields)
>>> dat
array([(False, 0)],
      dtype=[('x', '|b1'), ('y', '<i4')])
>>>

Thanks for any help,
Per


From dennis.a.cooke at gmail.com  Sat Apr 21 00:50:29 2007
From: dennis.a.cooke at gmail.com (Dennis Cooke)
Date: Sat, 21 Apr 2007 14:20:29 +0930
Subject: [Numpy-discussion] using the functions nonzero and where
Message-ID: <85970ffe0704202150o5eb5be1cl2b76e63869f7fb7e@mail.gmail.com>

I'm an ex-Matlab user trying to come up to speed with python and numpy.  I'm
confused on how to use the Numpy functions nonzero() and where().  In
matlab, the equivalent function (find(a>0) ) would return an array, whereas
in numpy, where() or nonzero() will return a single element tuple.  For
example:

In [35]: x = [ 0, 0, 0, 99, 0, 1, 5]
# and I wish to file the index of the first non-zero element of x - which is
3.

In [36]: from numpy import nonzero
In [37]: i=nonzero(x)
In [38]: i
Out[38]: (array([3, 5, 6]),)
In [39]: i[0]
Out[39]: array([3, 5, 6])

so nonzero() has output a tuple with a single element = the string
'array([3, 5, 6])'
How do I convert this string (or the original tuple)  into an array where
the first element = 3 (the index of the first non-zero element of my
original array x).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070421/f98998ab/attachment.html>

From tgrav at cfa.harvard.edu  Mon Apr 23 11:24:09 2007
From: tgrav at cfa.harvard.edu (Tommy Grav)
Date: Mon, 23 Apr 2007 11:24:09 -0400
Subject: [Numpy-discussion] Getting subarray
Message-ID: <11ACD1D7-9CCE-4B5D-95A5-502C2BBD1FBB@cfa.harvard.edu>

I have two arrays:

a = numpy.array([0,1,2,3,4,5,6,7,8,9])
b = numpy.array([0,0,1,1,2,2,0,1,2,3])

I would like to get the part of a that corresponds
to where b is equal to i.

For example:

i = 0 => ([0,1,6])
i = 1 => ([2,3,7])

Cheers
    Tommy


From buzzard at urubu.freeserve.co.uk  Mon Apr 23 11:55:29 2007
From: buzzard at urubu.freeserve.co.uk (Duncan Smith)
Date: Mon, 23 Apr 2007 16:55:29 +0100
Subject: [Numpy-discussion] numpy migration
Message-ID: <462CD6F1.70706@urubu.freeserve.co.uk>

Hello,
     Since moving to numpy I've had a few problems with my existing
code.  It basically revolves around the numpy scalar types. e.g.

------------------------------------------------

>>> import Numeric as N
>>> a = N.array([[0,1],[2,3]])
>>> a

array([[0, 1],
       [2, 3]])

>>> i = a[0,0]
>>> 1/i


Traceback (most recent call last):
  File "<pyshell#30>", line 1, in -toplevel-
    1/i
ZeroDivisionError: integer division or modulo by zero

>>> b = a * 1.5
>>> b

array([[ 0. ,  1.5],
       [ 3. ,  4.5]])

>>> N.floor(b)

array([[ 0.,  1.],
       [ 3.,  4.]])

>>> ================================ RESTART

================================

>>> import numpy as N
>>> a = N.array([[0,1],[2,3]])
>>> a

array([[0, 1],
       [2, 3]])

>>> i = a[0,0]
>>> 1/i

0

>>> b = a * 1.5
>>> b

array([[ 0. ,  1.5],
       [ 3. ,  4.5]])

>>> N.floor(b)

array([[ 0.,  1.],
       [ 3.,  4.]])

>>> a = N.array([[0,1],[2,3]], dtype='O')
>>> a

array([[0, 1],
       [2, 3]], dtype=object)

>>> i = a[0,0]
>>> 1/i


Traceback (most recent call last):
  File "<pyshell#45>", line 1, in -toplevel-
    1/i
ZeroDivisionError: integer division or modulo by zero

>>> b = a * 1.5
>>> b

array([[0.0, 1.5],
       [3.0, 4.5]], dtype=object)

>>> N.floor(b)


Traceback (most recent call last):
  File "<pyshell#48>", line 1, in -toplevel-
    N.floor(b)
AttributeError: 'float' object has no attribute 'floor'

>>>

----------------------------------------------

An additional problem involves classes that have e.g. __rmul__ methods
defined and are sufficiently similar to numpy arrays that my classes'
__rmul__ methods are not invoked when using numpy scalars.


Using the 'O' dtype gives me Python types that raise zero division
errors appropriately (for my code) and the desired calls to e.g.
__rmul__ methods, but reduced functionality in other repects.

I might (I hope) be missing something obvious; but it seems like, to be
safe, I'm going to have to do a lot of explicit conversions to Python
types (or abandon catching zero division errors, and documenting some of
my classes to highlight that whether scalar * a equals a * scalar
depends on whether a.__rmul__ is called, which depends on the type of
scalar).

I suppose I might get round both issues by subclassing existing numpy
dtypes.  Any ideas?  Cheers.  TIA.

Duncan



From wbaxter at gmail.com  Tue Apr 24 13:18:41 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Wed, 25 Apr 2007 02:18:41 +0900
Subject: [Numpy-discussion] Getting subarray
In-Reply-To: <11ACD1D7-9CCE-4B5D-95A5-502C2BBD1FBB@cfa.harvard.edu>
References: <11ACD1D7-9CCE-4B5D-95A5-502C2BBD1FBB@cfa.harvard.edu>
Message-ID: <e86a5fd00704241018m6815265bla0333205267d9ce4@mail.gmail.com>

Easy!
a[b==i]

--bb

On 4/24/07, Tommy Grav <tgrav at cfa.harvard.edu> wrote:
> I have two arrays:
>
> a = numpy.array([0,1,2,3,4,5,6,7,8,9])
> b = numpy.array([0,0,1,1,2,2,0,1,2,3])
>
> I would like to get the part of a that corresponds
> to where b is equal to i.
>
> For example:
>
> i = 0 => ([0,1,6])
> i = 1 => ([2,3,7])
>


From tim.hochberg at ieee.org  Tue Apr 24 13:22:35 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Tue, 24 Apr 2007 10:22:35 -0700
Subject: [Numpy-discussion] using the functions nonzero and where
In-Reply-To: <85970ffe0704202150o5eb5be1cl2b76e63869f7fb7e@mail.gmail.com>
References: <85970ffe0704202150o5eb5be1cl2b76e63869f7fb7e@mail.gmail.com>
Message-ID: <e4412d6b0704241022p5ffbffcbq3e913159439d1f32@mail.gmail.com>

On 4/20/07, Dennis Cooke <dennis.a.cooke at gmail.com> wrote:
>
> I'm an ex-Matlab user trying to come up to speed with python and numpy.
> I'm confused on how to use the Numpy functions nonzero() and where().  In
> matlab, the equivalent function (find(a>0) ) would return an array, whereas
> in numpy, where() or nonzero() will return a single element tuple.  For
> example:
>
> In [35]: x = [ 0, 0, 0, 99, 0, 1, 5]
> # and I wish to file the index of the first non-zero element of x - which
> is 3.
>
> In [36]: from numpy import nonzero
> In [37]: i=nonzero(x)
> In [38]: i
> Out[38]: (array([3, 5, 6]),)
> In [39]: i[0]
> Out[39]: array([3, 5, 6])
>
> so nonzero() has output a tuple with a single element = the string
> 'array([3, 5, 6])'


What makes you think that this is a string? It is in fact the array that you
seek:

>>> from numpy import nonzero
>>> x = [ 0, 0, 0, 99, 0, 1, 5]
>>> i = nonzero(x)
>>> i
(array([3, 5, 6]),)
>>> i[0]
array([3, 5, 6])
>>> type(i[0])
<type 'numpy.ndarray'>
>>> i[0][0]
3


How do I convert this string (or the original tuple)  into an array where
> the first element = 3 (the index of the first non-zero element of my
> original array x).


I fairly certain that you just want i[0].

-tim



_______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070424/ca1abad2/attachment.html>

From wbaxter at gmail.com  Tue Apr 24 13:24:46 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Wed, 25 Apr 2007 02:24:46 +0900
Subject: [Numpy-discussion] using the functions nonzero and where
In-Reply-To: <85970ffe0704202150o5eb5be1cl2b76e63869f7fb7e@mail.gmail.com>
References: <85970ffe0704202150o5eb5be1cl2b76e63869f7fb7e@mail.gmail.com>
Message-ID: <e86a5fd00704241024x2d6f690dj8e5b33f9304edb41@mail.gmail.com>

On 4/21/07, Dennis Cooke <dennis.a.cooke at gmail.com> wrote:
> I'm an ex-Matlab user trying to come up to speed with python and numpy.

Howdy.  First, I hope you've checked out the page:
   http://www.scipy.org/NumPy_for_Matlab_Users

> I'm
> confused on how to use the Numpy functions nonzero() and where().  In
> matlab, the equivalent function (find(a>0) ) would return an array, whereas
> in numpy, where() or nonzero() will return a single element tuple.  For
> example:
>
> In [35]: x = [ 0, 0, 0, 99, 0, 1, 5]
> # and I wish to file the index of the first non-zero element of x - which is
> 3.
>
> In [36]: from numpy import nonzero
> In [37]: i=nonzero(x)
> In [38]: i
> Out[38]: (array([3, 5, 6]),)
> In [39]: i[0]
> Out[39]: array([3, 5, 6])
>
> so nonzero() has output a tuple with a single element = the string
> 'array([3, 5, 6])'

The element is not a string.  It's the array, it shows up like a
string.  (To print things out the method __repr__() is called
automatically and the convention is that that's supposed to return
something which if you typed it in, would evaluate to the same thing.)

> How do I convert this string (or the original tuple)  into an array where
> the first element = 3 (the index of the first non-zero element of my
> original array x).

Just do i[0].  It's an array, not a string.  Try typing "type(i[0])"
and see what it tells you.

--bb


From Chris.Barker at noaa.gov  Tue Apr 24 13:34:52 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 24 Apr 2007 10:34:52 -0700
Subject: [Numpy-discussion] using the functions nonzero and where
In-Reply-To: <e86a5fd00704241024x2d6f690dj8e5b33f9304edb41@mail.gmail.com>
References: <85970ffe0704202150o5eb5be1cl2b76e63869f7fb7e@mail.gmail.com>
	<e86a5fd00704241024x2d6f690dj8e5b33f9304edb41@mail.gmail.com>
Message-ID: <462E3FBC.1080303@noaa.gov>

Bill Baxter wrote:
>> In [35]: x = [ 0, 0, 0, 99, 0, 1, 5]
>> In [37]: i=nonzero(x)
>> In [38]: i
>> Out[38]: (array([3, 5, 6]),)

> Just do i[0].  It's an array, not a string.  Try typing "type(i[0])"
> and see what it tells you.

Which still begs the question: why does nonzero() return a tuple with an 
array in it, rather than just the array?

Is it so you can so this?

 >>> a = numpy.array(((3,0,4),(5,21,0)))

 >>> numpy.nonzero(a)
(array([0, 0, 1, 1]), array([0, 2, 0, 1]))

 >>> a[numpy.nonzero(a)]
array([ 3,  4,  5, 21])

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From charlesr.harris at gmail.com  Tue Apr 24 13:34:04 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 24 Apr 2007 11:34:04 -0600
Subject: [Numpy-discussion] Bus Error with string in ndarray with named
	fields
In-Reply-To: <6b7179780704200638i44fa02b4nf347021fe102f054@mail.gmail.com>
References: <6b7179780704200638i44fa02b4nf347021fe102f054@mail.gmail.com>
Message-ID: <e06186140704241034x74365079xb3c6de1cdeae749c@mail.gmail.com>

On 4/20/07, Per B. Sederberg <persed at princeton.edu> wrote:
>
> Hi Folks:
>
> I'm getting a very strange bus error in the recent versions of numpy
> (almost current svn).  Here's how you can (hopefully) replicate it:
>
> On my MacBook:
>
> Python 2.4.3 (#1, Apr  7 2006, 10:54:33)
> [GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import numpy as N
> >>> N.version.version
> '1.0.2.dev3569'
> >>> fields = [('x',(N.str,40))]
> >>> dat = N.zeros(1,fields)
> >>> dat
> Bus error
>
>
> On my linux cluster:
>
> Python 2.4.3 (#1, May  8 2006, 11:36:25)
> [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-49)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import numpy as N
> >>> N.version.version
> '1.0.2.dev3567'
> >>> fields = [('x',(N.str,40))]
> >>> dat = N.zeros(1,fields)
> >>> dat
> Segmentation fault


Works here running 1.0.3.dev3679 on 32 bit linux. You might want to upgrade
to release 1.0.2, as your version is prior to that. Are you running a 64 bit
OS on the problem machines? That might be another source of the problem.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070424/7dbdf2e4/attachment.html>

From robert.kern at gmail.com  Tue Apr 24 13:34:25 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 24 Apr 2007 12:34:25 -0500
Subject: [Numpy-discussion] using the functions nonzero and where
In-Reply-To: <462E3FBC.1080303@noaa.gov>
References: <85970ffe0704202150o5eb5be1cl2b76e63869f7fb7e@mail.gmail.com>	<e86a5fd00704241024x2d6f690dj8e5b33f9304edb41@mail.gmail.com>
	<462E3FBC.1080303@noaa.gov>
Message-ID: <462E3FA1.1090703@gmail.com>

Christopher Barker wrote:
> Bill Baxter wrote:
>>> In [35]: x = [ 0, 0, 0, 99, 0, 1, 5]
>>> In [37]: i=nonzero(x)
>>> In [38]: i
>>> Out[38]: (array([3, 5, 6]),)
> 
>> Just do i[0].  It's an array, not a string.  Try typing "type(i[0])"
>> and see what it tells you.
> 
> Which still begs the question: why does nonzero() return a tuple with an 
> array in it, rather than just the array?
> 
> Is it so you can so this?
> 
>  >>> a = numpy.array(((3,0,4),(5,21,0)))
> 
>  >>> numpy.nonzero(a)
> (array([0, 0, 1, 1]), array([0, 2, 0, 1]))
> 
>  >>> a[numpy.nonzero(a)]
> array([ 3,  4,  5, 21])

Yes, consistency between the 1-D case and the N-D case.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From tim.hochberg at ieee.org  Tue Apr 24 13:38:24 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Tue, 24 Apr 2007 10:38:24 -0700
Subject: [Numpy-discussion] numpy migration
In-Reply-To: <462CD6F1.70706@urubu.freeserve.co.uk>
References: <462CD6F1.70706@urubu.freeserve.co.uk>
Message-ID: <e4412d6b0704241038x41dfa7adwd633ad4cb6c16487@mail.gmail.com>

On 4/23/07, Duncan Smith <buzzard at urubu.freeserve.co.uk> wrote:
>
> Hello,
>      Since moving to numpy I've had a few problems with my existing
> code.  It basically revolves around the numpy scalar types. e.g.
>
> ------------------------------------------------
>
> >>> import Numeric as N
> >>> a = N.array([[0,1],[2,3]])
> >>> a
>
> array([[0, 1],
>        [2, 3]])
>
> >>> i = a[0,0]
> >>> 1/i
>
>
> Traceback (most recent call last):
>   File "<pyshell#30>", line 1, in -toplevel-
>     1/i
> ZeroDivisionError: integer division or modulo by zero
>
> >>> b = a * 1.5
> >>> b
>
> array([[ 0. ,  1.5],
>        [ 3. ,  4.5]])
>
> >>> N.floor(b)
>
> array([[ 0.,  1.],
>        [ 3.,  4.]])
>
> >>> ================================ RESTART
>
> ================================
>
> >>> import numpy as N
> >>> a = N.array([[0,1],[2,3]])
> >>> a
>
> array([[0, 1],
>        [2, 3]])
>
> >>> i = a[0,0]
> >>> 1/i
>
> 0


You should be getting a warning here. Did one disappear in the cut and
paste? Or are you using a nonstandard shell that eats warnings? Or an old
version of numpy?

In any event if you want an error rather than a warning for zero division,
use:

>>> 1/i
Warning: divide by zero encountered in long_scalars
0
SyntaxError: invalid syntax
>>> N.seterr(divide='raise')
{'over': 'print', 'divide': 'print', 'invalid': 'print', 'under': 'ignore'}
>>> 1/i
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FloatingPointError: divide by zero encountered in long_scalars

Although I'm not sure why that's a floating point error.

>>> b = a * 1.5
> >>> b
>
> array([[ 0. ,  1.5],
>        [ 3. ,  4.5]])
>
> >>> N.floor(b)
>
> array([[ 0.,  1.],
>        [ 3.,  4.]])
>
> >>> a = N.array([[0,1],[2,3]], dtype='O')
> >>> a
>
> array([[0, 1],
>        [2, 3]], dtype=object)
>
> >>> i = a[0,0]
> >>> 1/i
>
>
> Traceback (most recent call last):
>   File "<pyshell#45>", line 1, in -toplevel-
>     1/i
> ZeroDivisionError: integer division or modulo by zero


Right -- for objects, all of the operations get delegated to the Python
object in question and this is the behaviour of Python ints.


>>> b = a * 1.5
> >>> b
>
> array([[0.0, 1.5],
>        [3.0, 4.5]], dtype=object)
>
> >>> N.floor(b)
>
>
> Traceback (most recent call last):
>   File "<pyshell#48>", line 1, in -toplevel-
>     N.floor(b)
> AttributeError: 'float' object has no attribute 'floor'
>
> >>>


I'm not sure what you expect here; for object arrays numpy simply delegats
everything to the corresponding object. If you want something that supports
floor, you should be using a numerical array, not an object array.


----------------------------------------------
>
> An additional problem involves classes that have e.g. __rmul__ methods
> defined and are sufficiently similar to numpy arrays that my classes'
> __rmul__ methods are not invoked when using numpy scalars.



Try adding "__array_priority__ = 10" or somesuch to your classes. This
should tell numpy to back off and let your methods go first when doing mixed
operations. Searching for "__array_priority__" will probably turn up some
examples.

Using the 'O' dtype gives me Python types that raise zero division
> errors appropriately (for my code) and the desired calls to e.g.
> __rmul__ methods, but reduced functionality in other repects.


Right, don't do that.

I might (I hope) be missing something obvious; but it seems like, to be
> safe, I'm going to have to do a lot of explicit conversions to Python
> types (or abandon catching zero division errors, and documenting some of
> my classes to highlight that whether scalar * a equals a * scalar
> depends on whether a.__rmul__ is called, which depends on the type of
> scalar).
>
> I suppose I might get round both issues by subclassing existing numpy
> dtypes.  Any ideas?  Cheers.  TIA.


I think you should be able to do everything you need, by using seterr at
startup to make numpy picky about division errors and then using array
priority to make your methods have precedence over numpy's.

 regards,

-tim

Duncan
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070424/d0f1465c/attachment.html>

From tim.hochberg at ieee.org  Tue Apr 24 13:45:24 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Tue, 24 Apr 2007 10:45:24 -0700
Subject: [Numpy-discussion] numpy migration
In-Reply-To: <e4412d6b0704241038x41dfa7adwd633ad4cb6c16487@mail.gmail.com>
References: <462CD6F1.70706@urubu.freeserve.co.uk>
	<e4412d6b0704241038x41dfa7adwd633ad4cb6c16487@mail.gmail.com>
Message-ID: <e4412d6b0704241045j75602318p8ae0d9816bd57920@mail.gmail.com>

On 4/24/07, Timothy Hochberg <tim.hochberg at ieee.org> wrote
[CHOP]

Sorry, cut and paste error, that should have  read:

:

>
> >
> > >>> i = a[0,0]
> > >>> 1/i
> >
> > 0
>
>
> You should be getting a warning here. Did one disappear in the cut and
> paste? Or are you using a nonstandard shell that eats warnings? Or an old
> version of numpy?
>
> In any event if you want an error rather than a warning for zero division,
> use seterr:
>
> >>> 1/i
> Warning: divide by zero encountered in long_scalars
> 0
> >>> N.seterr(divide='raise')
> {'over': 'print', 'divide': 'print', 'invalid': 'print', 'under':
> 'ignore'}
> >>> 1/i
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> FloatingPointError: divide by zero encountered in long_scalars
>
> Although I'm not sure why that's a floating point error.
>


[CHOP]

-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070424/bb3b73d3/attachment.html>

From cookedm at physics.mcmaster.ca  Tue Apr 24 16:23:14 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Tue, 24 Apr 2007 16:23:14 -0400
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>
	<27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc>
	<9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc>
	<0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca>
	<Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu>
Message-ID: <4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>

On Apr 23, 2007, at 22:04 , Warren Focke wrote:

> But even C89 required that x == (x/y)*y + (x%y), and that's not the  
> case
> here.

Missed that. You're right. We pull the same trick Python does with %  
so that the sign of x % y agrees with the sign of y, but we don't  
follow Python in guaranteeing floor division. To fix that means  
calculating x % y (as x - (x/y)*y) and checking if the sign is the  
same as y.

This would be a backward incompatible change, but it would restore  
the above invariant, which I think is important. Alternatively, we  
could change the modulo so that the invariant holds, but we don't  
agree with Python in either / or %. Comments?

> On Mon, 23 Apr 2007, David M. Cooke wrote:
>
>> On Apr 23, 2007, at 16:41 , Christian Marquardt wrote:
>>> On Mon, April 23, 2007 22:29, Christian Marquardt wrote:
>>>> Actually,
>>>>
>>>> it happens for normal integers as well:
>>>>
>>>>>>> n = np.array([-5, -100, -150])
>>>>>>> n // 100
>>>>    array([ 0, -1, -1])
>>>>>>> -5//100, -100//100, -150//100
>>>>    (-1, -1, -2)
>>>
>>> and finally:
>>>
>>>>>> n % 100
>>>    array([95,  0, 50])
>>>>>> -5 % 100, -100 % 100, -150 % 100
>>>    (95, 0, 50)
>>>
>>> So plain python / using long provides consistent results across //
>>> and %, but numpy doesn't...
>>
>> Python defines x // y as returning the floor of the division, and x %
>> y has the same sign as y. However, in C89, it is implementation-
>> defined (i.e., portability-pain-in-the-ass) whether the floor or ceil
>> is used when the signs of x and y differ. In C99, the result should
>> be truncated. From the C99 spec, sec. 6.5.5, #6:
>>         When integers are divided, the result of the / operator
>>         is   the   algebraic   quotient  with  any  fractional  part
>>         discarded.76)  If the quotient  a/b  is  representable,  the
>>         expression (a/b)*b + a%b shall equal a.
>>
>> Numpy follows whatever the C compiler decides to use, because of
>> speed-vs.-Python compatibility tradeoff.
>>
>>>
>>>   Christian.
>>>
>>>> On Mon, April 23, 2007 22:20, Christian Marquardt wrote:
>>>>> Dear all,
>>>>>
>>>>> this is odd:
>>>>>
>>>>>>>> import numpy as np
>>>>>>>> fact = 28250000L * 86400L
>>>>>>>> nn = np.array([-20905000L])
>>>>>>>> nn
>>>>>    array([-20905000], dtype=int64)
>>>>>>>> nn[0] // fact
>>>>>    0
>>>>>
>>>>> But:
>>>>>
>>>>>>>> long(nn[0]) // fact
>>>>>    -1L
>>>>>
>>>>> Is this a bug in numpy, or in python's implementation of longs? I
>>>>> would
>>>>> think both should give the same, really... (Python 2.5, numpy
>>>>> 1.0.3dev3725,
>>>>> Linux, Intel compilers...)
>>>>>
>>>>> Many thanks for any ideas / advice,
>>>>>
>>>>>   Christian
>>
>> --
>> |>|\/|<
>> /------------------------------------------------------------------\
>> |David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
>> |cookedm at physics.mcmaster.ca
>>
>>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

-- 
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070424/4b47bc35/attachment.sig>

From aisaac at american.edu  Tue Apr 24 16:38:42 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 24 Apr 2007 16:38:42 -0400
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc><27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc><9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc><0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca><Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu><4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>
Message-ID: <Mahogany-0.67.0-736-20070424-163842.00@american.edu>

Do restore the invariant.
Behave completely like Python if not too costly,
otherwise follow C89.

A user's view,
Alan Isaac





From christian at marquardt.sc  Tue Apr 24 17:04:00 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Tue, 24 Apr 2007 23:04:00 +0200 (CEST)
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <Mahogany-0.67.0-736-20070424-163842.00@american.edu>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc><27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc><9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc><0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca><Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu><4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>
	<Mahogany-0.67.0-736-20070424-163842.00@american.edu>
Message-ID: <26758.84.167.102.129.1177448640.squirrel@webmail.marquardt.sc>

Restore the invariant, and follow python.

This

   >>> -5 // 6
   -1

and

   >>> array([-5])[0] // 6
   0

simply doesn't make sense - in any language, you would expect that
all basic operators provide you with the same same answer when
applied to the same number, no?

  Christian.


On Tue, April 24, 2007 22:38, Alan G Isaac wrote:
> Do restore the invariant.
> Behave completely like Python if not too costly,
> otherwise follow C89.
>
> A user's view,
> Alan Isaac




From robert.kern at gmail.com  Tue Apr 24 17:08:11 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 24 Apr 2007 16:08:11 -0500
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <26758.84.167.102.129.1177448640.squirrel@webmail.marquardt.sc>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc><27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc><9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc><0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca><Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu><4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>	<Mahogany-0.67.0-736-20070424-163842.00@american.edu>
	<26758.84.167.102.129.1177448640.squirrel@webmail.marquardt.sc>
Message-ID: <462E71BB.2030204@gmail.com>

Christian Marquardt wrote:
> Restore the invariant, and follow python.
> 
> This
> 
>    >>> -5 // 6
>    -1
> 
> and
> 
>    >>> array([-5])[0] // 6
>    0
> 
> simply doesn't make sense - in any language, you would expect that
> all basic operators provide you with the same same answer when
> applied to the same number, no?

Not if they are different types, you don't, e.g. -5.0 / 6 .

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From christian at marquardt.sc  Tue Apr 24 17:31:05 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Tue, 24 Apr 2007 23:31:05 +0200 (CEST)
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <462E71BB.2030204@gmail.com>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc><27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc><9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc><0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca><Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu><4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>
	<Mahogany-0.67.0-736-20070424-163842.00@american.edu>
	<26758.84.167.102.129.1177448640.squirrel@webmail.marquardt.sc>
	<462E71BB.2030204@gmail.com>
Message-ID: <5741.84.167.102.129.1177450265.squirrel@webmail.marquardt.sc>

On Tue, April 24, 2007 23:08, Robert Kern wrote:
> Christian Marquardt wrote:
>> Restore the invariant, and follow python.
>>
>> This
>>
>>    >>> -5 // 6
>>    -1
>>
>> and
>>
>>    >>> array([-5])[0] // 6
>>    0
>>
>> simply doesn't make sense - in any language, you would expect that
>> all basic operators provide you with the same same answer when
>> applied to the same number, no?
>
> Not if they are different types, you don't, e.g. -5.0 / 6 .

But I would regard an integer and an array of integers as the same type.

  Christian.




>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma
>  that is made terrible by our own mad attempt to interpret it as though it
> had
>  an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>




From tim.hochberg at ieee.org  Tue Apr 24 17:33:52 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Tue, 24 Apr 2007 14:33:52 -0700
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <462E71BB.2030204@gmail.com>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>
	<27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc>
	<9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc>
	<0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca>
	<Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu>
	<4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>
	<Mahogany-0.67.0-736-20070424-163842.00@american.edu>
	<26758.84.167.102.129.1177448640.squirrel@webmail.marquardt.sc>
	<462E71BB.2030204@gmail.com>
Message-ID: <e4412d6b0704241433l526b4263td02a3a5d937e5afb@mail.gmail.com>

On 4/24/07, Robert Kern <robert.kern at gmail.com> wrote:
>
> Christian Marquardt wrote:



> > Restore the invariant, and follow python.


This seems to imply that once upon a time numpy/numeric/numarray followed
python here, but as far as I can recall that was never the case. Instead
they followed C completely, ostensibly for performance reasons. That
preserved the invariant in question, but not compatibility with Python.


>
> > This
> >
> >    >>> -5 // 6
> >    -1
> >
> > and
> >
> >    >>> array([-5])[0] // 6
> >    0
> >
> > simply doesn't make sense - in any language, you would expect that
> > all basic operators provide you with the same same answer when
> > applied to the same number, no?
>
> Not if they are different types, you don't, e.g. -5.0 / 6 .


That's perhaps not the best example since this difference is slated for
removal someday:

>>> from __future__ import division
>>> -5/6
-0.83333333333333337
>>> -5.0/6.0
-0.83333333333333337

Personally I'd opt for completely following Python here, with the C-like
integer division and mod operators available as appropriately named ufuncs
somewhere. It's a backwards incompatible change though, so it'd have to wait
till at least a minor realease.



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070424/403fbc4d/attachment.html>

From christian at marquardt.sc  Tue Apr 24 17:34:29 2007
From: christian at marquardt.sc (Christian Marquardt)
Date: Tue, 24 Apr 2007 23:34:29 +0200 (CEST)
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <5741.84.167.102.129.1177450265.squirrel@webmail.marquardt.sc>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc><27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc><9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc><0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca><Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu><4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>
	<Mahogany-0.67.0-736-20070424-163842.00@american.edu>
	<26758.84.167.102.129.1177448640.squirrel@webmail.marquardt.sc>
	<462E71BB.2030204@gmail.com>
	<5741.84.167.102.129.1177450265.squirrel@webmail.marquardt.sc>
Message-ID: <14243.84.167.102.129.1177450469.squirrel@webmail.marquardt.sc>

On Tue, April 24, 2007 23:31, Christian Marquardt wrote:
> On Tue, April 24, 2007 23:08, Robert Kern wrote:
>> Christian Marquardt wrote:
>>> Restore the invariant, and follow python.
>>>
>>> This
>>>
>>>    >>> -5 // 6
>>>    -1
>>>
>>> and
>>>
>>>    >>> array([-5])[0] // 6
>>>    0
>>>
>>> simply doesn't make sense - in any language, you would expect that
>>> all basic operators provide you with the same same answer when
>>> applied to the same number, no?
>>
>> Not if they are different types, you don't, e.g. -5.0 / 6 .
>
> But I would regard an integer and an array of integers as the same type.

By the way:

   >>> -5.0 // 6
   -1.0
   >>> array([-5.0]) // 6
   array([-1.])

  Christian.





From focke at slac.stanford.edu  Tue Apr 24 17:44:17 2007
From: focke at slac.stanford.edu (Warren Focke)
Date: Tue, 24 Apr 2007 14:44:17 -0700 (PDT)
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <e4412d6b0704241433l526b4263td02a3a5d937e5afb@mail.gmail.com>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>
	<27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc>
	<9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc>
	<0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca>
	<Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu>
	<4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>
	<Mahogany-0.67.0-736-20070424-163842.00@american.edu>
	<26758.84.167.102.129.1177448640.squirrel@webmail.marquardt.sc>
	<462E71BB.2030204@gmail.com>
	<e4412d6b0704241433l526b4263td02a3a5d937e5afb@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0704241439420.11417@circinus.slac.stanford.edu>



On Tue, 24 Apr 2007, Timothy Hochberg wrote:

> On 4/24/07, Robert Kern <robert.kern at gmail.com> wrote:
> >
> > Christian Marquardt wrote:
>
>
>
> > > Restore the invariant, and follow python.
>
>
> This seems to imply that once upon a time numpy/numeric/numarray followed
> python here, but as far as I can recall that was never the case. Instead
> they followed C completely, ostensibly for performance reasons. That
> preserved the invariant in question, but not compatibility with Python.

That's how I remember it, too.  And I think I'd prefer it be that way
again, unless it can be demonstrated that the hit from being consistent
with python is "small".

Note that C99 specifies behavior opposite that of python (but still
consistent with itself).

w



From oliphant at ee.byu.edu  Tue Apr 24 17:46:59 2007
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Tue, 24 Apr 2007 15:46:59 -0600
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <e4412d6b0704241433l526b4263td02a3a5d937e5afb@mail.gmail.com>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>	<27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc>	<9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc>	<0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca>	<Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu>	<4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>	<Mahogany-0.67.0-736-20070424-163842.00@american.edu>	<26758.84.167.102.129.1177448640.squirrel@webmail.marquardt.sc>	<462E71BB.2030204@gmail.com>
	<e4412d6b0704241433l526b4263td02a3a5d937e5afb@mail.gmail.com>
Message-ID: <462E7AD3.2030602@ee.byu.edu>


>  
> Personally I'd opt for completely following Python here, with the 
> C-like integer division and mod operators available as appropriately 
> named ufuncs somewhere. It's a backwards incompatible change though, 
> so it'd have to wait till at least a minor realease.


I'm supportive of following Python and retaining the invariant.

This could be regarded as a bug, because NumPy 1.0 changed the mod 
operators to be compatible with Python but did not change the division 
operators to perserve the invariant.

Given that this is the first time I've heard complaints, I suspect that 
changing this will have very few detractors, and while I'm hesistant 
because of potential backward incompatibilities, would be willing to 
make the change in Numpy 1.0.3

-Travis



From cookedm at physics.mcmaster.ca  Tue Apr 24 18:42:44 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Tue, 24 Apr 2007 18:42:44 -0400
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>
	<27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc>
	<9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc>
	<0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca>
Message-ID: <00FD0F12-F7FB-4E0F-8F58-EF81EE943EA8@physics.mcmaster.ca>

On Apr 23, 2007, at 21:35 , David M. Cooke wrote:
>
> Python defines x // y as returning the floor of the division, and x  
> % y has the same sign as y. However, in C89, it is implementation- 
> defined (i.e., portability-pain-in-the-ass) whether the floor or  
> ceil is used when the signs of x and y differ. In C99, the result  
> should be truncated. From the C99 spec, sec. 6.5.5, #6:
>        When integers are divided, the result of the / operator
>        is   the   algebraic   quotient  with  any  fractional  part
>        discarded.76)  If the quotient  a/b  is  representable,  the
>        expression (a/b)*b + a%b shall equal a.
>
> Numpy follows whatever the C compiler decides to use, because of  
> speed-vs.-Python compatibility tradeoff.

Just a followup note: gcc (4.0 and 4.2, at least) with -std=c99  
*doesn't* respect this; it still does the platform way (on my MacBook  
Core Duo, that's truncation towards zero). The assembly it generates  
is identical for gcc -std=c99 and gcc -std=c89

-- 
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070424/35c526cb/attachment.sig>

From aisaac at american.edu  Tue Apr 24 20:20:13 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 24 Apr 2007 20:20:13 -0400
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <e4412d6b0704241433l526b4263td02a3a5d937e5afb@mail.gmail.com>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc><27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc><9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc><0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca><Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu><4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca><Mahogany-0.67.0-736-20070424-163842.00@american.edu><26758.84.167.102.129.1177448640.squirrel@webmail.marquardt.sc><462E71BB.2030204@gmail.com><e4412d6b0704241433l526b4263td02a3a5d937e5afb@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-736-20070424-202013.00@american.edu>

On Tue, 24 Apr 2007, Timothy Hochberg apparently wrote: 
> Personally I'd opt for completely following Python here, 
> with the C-like integer division and mod operators 
> available as appropriately named ufuncs somewhere. It's 
> a backwards incompatible change though, so it'd have to 
> wait till at least a minor realease. 

This is one of those places where looking forward appears to 
produce a different suggestion than looking backward. 

That is, looking backward, we emphasize compatability with 
extant code.  Looking forward we put more weight on removing 
"gotcha" features from numpy, with an eye on an expanding 
user base. 

In my opinion, opting to maintain backward compatability 
here is opting to retain a "gotcha".  I hope Tim's 
suggestion is realized, and that backward compatability is 
again enabled through the compatability module.  I'm happy 
that Travis appears amenable to this.

A user's opinion, 
Alan Isaac 




From buzzard at contactbox.co.uk  Tue Apr 24 21:26:57 2007
From: buzzard at contactbox.co.uk (Duncan Smith)
Date: Wed, 25 Apr 2007 02:26:57 +0100
Subject: [Numpy-discussion] numpy migration
In-Reply-To: <e4412d6b0704241038x41dfa7adwd633ad4cb6c16487@mail.gmail.com>
References: <462CD6F1.70706@urubu.freeserve.co.uk>
	<e4412d6b0704241038x41dfa7adwd633ad4cb6c16487@mail.gmail.com>
Message-ID: <462EAE61.90500@contactbox.co.uk>



Timothy Hochberg wrote:
> On 4/23/07, Duncan Smith <buzzard at urubu.freeserve.co.uk> wrote:
> 
>>
>> Hello,
>>      Since moving to numpy I've had a few problems with my existing
>> code.  It basically revolves around the numpy scalar types. e.g.
>>
>> ------------------------------------------------
>>
>> >>> import Numeric as N
>> >>> a = N.array([[0,1],[2,3]])
>> >>> a
>>
>> array([[0, 1],
>>        [2, 3]])
>>
>> >>> i = a[0,0]
>> >>> 1/i
>>
>>
>> Traceback (most recent call last):
>>   File "<pyshell#30>", line 1, in -toplevel-
>>     1/i
>> ZeroDivisionError: integer division or modulo by zero
>>
>> >>> b = a * 1.5
>> >>> b
>>
>> array([[ 0. ,  1.5],
>>        [ 3. ,  4.5]])
>>
>> >>> N.floor(b)
>>
>> array([[ 0.,  1.],
>>        [ 3.,  4.]])
>>
>> >>> ================================ RESTART
>>
>> ================================
>>
>> >>> import numpy as N
>> >>> a = N.array([[0,1],[2,3]])
>> >>> a
>>
>> array([[0, 1],
>>        [2, 3]])
>>
>> >>> i = a[0,0]
>> >>> 1/i
>>
>> 0
> 
> 
> 
> You should be getting a warning here. Did one disappear in the cut and
> paste? Or are you using a nonstandard shell that eats warnings? Or an old
> version of numpy?
> 

Python 2.5.1 / numpy 1.0.1 / IDLE (under Windows 2000).  No warning.

> In any event if you want an error rather than a warning for zero division,
> use:
> 
>>>> 1/i
> 
> Warning: divide by zero encountered in long_scalars
> 0
> SyntaxError: invalid syntax
> 
>>>> N.seterr(divide='raise')
> 
> {'over': 'print', 'divide': 'print', 'invalid': 'print', 'under': 'ignore'}
> 
>>>> 1/i
> 
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> FloatingPointError: divide by zero encountered in long_scalars
> 
> Although I'm not sure why that's a floating point error.
> 

True, bug or feature?, I'm really looking for a ZeroDivisionError.

[snip]

> ----------------------------------------------
> 
>>
>> An additional problem involves classes that have e.g. __rmul__ methods
>> defined and are sufficiently similar to numpy arrays that my classes'
>> __rmul__ methods are not invoked when using numpy scalars.
> 
> 
> 
> 
> Try adding "__array_priority__ = 10" or somesuch to your classes. This
> should tell numpy to back off and let your methods go first when doing
> mixed
> operations. Searching for "__array_priority__" will probably turn up some
> examples.
> 

Yes, it did.  But for my classes it seems to have no effect.  Below is a
fairly minimal example.

-----------------example.py--------------------

from __future__ import division

import numpy

class MyClass(object):

    def __init__(self, arr, labels):
        self.arr = arr
        self.labels = labels

    def __repr__(self):
        return numpy.array2string(self.arr, separator=', ') +
repr(self.labels)

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, key):
        return self.arr[key]

    def __setitem__(self, key, item):
        self.arr[key] = item

    def __mul__(self, other):
        return self.__class__(self.arr * other, self.labels)

    __rmul__ = __mul__

----------------------------------------------------

>>> import example
>>> import numpy as N
>>> ex = example.MyClass(N.array([[6,7],[8,9]]), ['axis0', 'axis1'])
>>> i = ex.arr[0,0]
>>> ex
[[6, 7],
 [8, 9]]['axis0', 'axis1']
>>> ex * i
[[36, 42],
 [48, 54]]['axis0', 'axis1']
>>> i * ex
array([[36, 42],
       [48, 54]])
>>>


It seems that it requires having __len__, __setitem__ and __getitem__
defined to get the undesired behaviour.

[snip]

> 
> I think you should be able to do everything you need, by using seterr at
> startup to make numpy picky about division errors and then using array
> priority to make your methods have precedence over numpy's.
> 

Thanks Tim.  Unfortunately I really need it to raise a
ZeroDivisionError.  Homing in on the second issue, but I'm not sure I
can easily do away with the __len__ method (and I definitely need the
others).  Cheers.

Duncan


From strawman at astraw.com  Wed Apr 25 11:52:02 2007
From: strawman at astraw.com (Andrew Straw)
Date: Wed, 25 Apr 2007 08:52:02 -0700
Subject: [Numpy-discussion] Python issue of Computing in Science and
	Engineering available
Message-ID: <462F7922.7060704@astraw.com>

The May/June issue of Computing in Science and Engineering 
http://computer.org/cise: is out and has a Python theme. Many folks we 
know and love from the community and mailing lists contribute to the 
issue. Read articles by Paul Dubois and Travis Oliphant for free online.


From fperez.net at gmail.com  Wed Apr 25 12:11:03 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 25 Apr 2007 10:11:03 -0600
Subject: [Numpy-discussion] [Matplotlib-users] Python issue of Computing
	in Science and Engineering available
In-Reply-To: <462F7922.7060704@astraw.com>
References: <462F7922.7060704@astraw.com>
Message-ID: <db6b5ecc0704250911r1804068bp5f07211d88c32b80@mail.gmail.com>

On 4/25/07, Andrew Straw <strawman at astraw.com> wrote:
> The May/June issue of Computing in Science and Engineering
> http://computer.org/cise: is out and has a Python theme. Many folks we
> know and love from the community and mailing lists contribute to the
> issue. Read articles by Paul Dubois and Travis Oliphant for free online.

<plug>
Since authors are allowed by their publication policy to keep a
publicly available copy of their papers on their personal website,
here's the ipython one:

http://amath.colorado.edu/faculty/fperez/preprints/ipython-cise-final.pdf

</plug>

Cheers,

f


From jdh2358 at gmail.com  Wed Apr 25 12:18:56 2007
From: jdh2358 at gmail.com (John Hunter)
Date: Wed, 25 Apr 2007 11:18:56 -0500
Subject: [Numpy-discussion] [Matplotlib-users] Python issue of Computing
	in Science and Engineering available
In-Reply-To: <db6b5ecc0704250911r1804068bp5f07211d88c32b80@mail.gmail.com>
References: <462F7922.7060704@astraw.com>
	<db6b5ecc0704250911r1804068bp5f07211d88c32b80@mail.gmail.com>
Message-ID: <88e473830704250918n6c7631f4w514d8dc68b3dde94@mail.gmail.com>

On 4/25/07, Fernando Perez <fperez.net at gmail.com> wrote:
> Since authors are allowed by their publication policy to keep a
> publicly available copy of their papers on their personal website,
> here's the ipython one:

Didn't know that...  here's a link to my matplotlib article

http://nitace.bsd.uchicago.edu/misc/c3sci.pdf

It might be nice to create a scipy wiki page linking to these PDFs.

JDH


From kwgoodman at gmail.com  Wed Apr 25 12:26:35 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 25 Apr 2007 09:26:35 -0700
Subject: [Numpy-discussion] [Matplotlib-users] Python issue of Computing
	in Science and Engineering available
In-Reply-To: <db6b5ecc0704250911r1804068bp5f07211d88c32b80@mail.gmail.com>
References: <462F7922.7060704@astraw.com>
	<db6b5ecc0704250911r1804068bp5f07211d88c32b80@mail.gmail.com>
Message-ID: <f4f93d420704250926q9a86ed1ica5147e3106b2122@mail.gmail.com>

On 4/25/07, Fernando Perez <fperez.net at gmail.com> wrote:
> On 4/25/07, Andrew Straw <strawman at astraw.com> wrote:
> > The May/June issue of Computing in Science and Engineering
> > http://computer.org/cise: is out and has a Python theme. Many folks we
> > know and love from the community and mailing lists contribute to the
> > issue. Read articles by Paul Dubois and Travis Oliphant for free online.
>
> <plug>
> Since authors are allowed by their publication policy to keep a
> publicly available copy of their papers on their personal website,
> here's the ipython one:
>
> http://amath.colorado.edu/faculty/fperez/preprints/ipython-cise-final.pdf
>
> </plug>

I'm glad to see authors take control of their copyrights. It is a good trend.

Too bad the scipy web site can't host the papers.


From fperez.net at gmail.com  Wed Apr 25 12:29:01 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 25 Apr 2007 10:29:01 -0600
Subject: [Numpy-discussion] [Matplotlib-users] Python issue of Computing
	in Science and Engineering available
In-Reply-To: <88e473830704250918n6c7631f4w514d8dc68b3dde94@mail.gmail.com>
References: <462F7922.7060704@astraw.com>
	<db6b5ecc0704250911r1804068bp5f07211d88c32b80@mail.gmail.com>
	<88e473830704250918n6c7631f4w514d8dc68b3dde94@mail.gmail.com>
Message-ID: <db6b5ecc0704250929r2d4b0518ma7dc581ea76ee33f@mail.gmail.com>

On 4/25/07, John Hunter <jdh2358 at gmail.com> wrote:
> On 4/25/07, Fernando Perez <fperez.net at gmail.com> wrote:
> > Since authors are allowed by their publication policy to keep a
> > publicly available copy of their papers on their personal website,
> > here's the ipython one:
>
> Didn't know that...  here's a link to my matplotlib article

I'm going by the language here:

http://www.ieee.org/web/publications/rights/policies.html

Specifically:

 When IEEE publishes the work, the author must replace the previous
electronic version of the accepted paper with either (1) the full
citation to the IEEE work or (2) the IEEE-published version, including
the IEEE copyright notice and full citation. Prior or revised versions
of the paper must not be represented as the published version.


This explicitly mentions author website redistribution,  as long as
the official IEEE version is used.

Unless I'm misreading the above, I think it's OK for us to keep such
copies in our personal sites.  We can link to them from the scipy
wiki, though I don't think it would be OK to /copy/ the PDFs to the
scipy wiki.

As always, IANAL and all that.

Cheers,

f


From tim.hochberg at ieee.org  Wed Apr 25 12:31:01 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 25 Apr 2007 09:31:01 -0700
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <Mahogany-0.67.0-736-20070424-202013.00@american.edu>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>
	<9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc>
	<0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca>
	<Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu>
	<4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>
	<Mahogany-0.67.0-736-20070424-163842.00@american.edu>
	<26758.84.167.102.129.1177448640.squirrel@webmail.marquardt.sc>
	<462E71BB.2030204@gmail.com>
	<e4412d6b0704241433l526b4263td02a3a5d937e5afb@mail.gmail.com>
	<Mahogany-0.67.0-736-20070424-202013.00@american.edu>
Message-ID: <e4412d6b0704250931p7e1b5fcck3421c161448db2f0@mail.gmail.com>

On 4/24/07, Alan G Isaac <aisaac at american.edu> wrote:
>
> On Tue, 24 Apr 2007, Timothy Hochberg apparently wrote:
> > Personally I'd opt for completely following Python here,
> > with the C-like integer division and mod operators
> > available as appropriately named ufuncs somewhere. It's
> > a backwards incompatible change though, so it'd have to
> > wait till at least a minor realease.
>
> This is one of those places where looking forward appears to
> produce a different suggestion than looking backward.
>
> That is, looking backward, we emphasize compatability with
> extant code.  Looking forward we put more weight on removing
> "gotcha" features from numpy, with an eye on an expanding
> user base.
>
> In my opinion, opting to maintain backward compatability
> here is opting to retain a "gotcha".  I hope Tim's
> suggestion is realized, and that backward compatability is
> again enabled through the compatability module.  I'm happy
> that Travis appears amenable to this.


 Let me add that there are some factors that make this more compelling
looking forward than it did in the past. The root of these is the changing
meaning of "/" and the introduction of floor division "//". Eventually, the
integer division meaning of "/" will go away and the result of "a / b" is
always going to be a float or complex. You can get that behaviour now by
using "from __future__ import division". Therefore, the interesing behaviour
is what does '"//" do. SInce "//" is pronounced "floor division" it pretty
much has to round towards negative infinity if we are to avoid massive
confusion.  If we support this, as I believe we eventually must, then this
results in following python practice for '%' as well in order to maintain
our invariant.

I worry less about speed here than I used to since my experience is that
moving bits around is more often a bottleneck than the FPU and this doesn't
move any extra bits. Also, we should provide ufuncs (c_int_divide and
c_int_mod or some such) for those that are need the speed and know that
truncation to zero isn't going to bite them.



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070425/3a16dd94/attachment.html>

From kwgoodman at gmail.com  Wed Apr 25 12:56:51 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 25 Apr 2007 09:56:51 -0700
Subject: [Numpy-discussion] [Matplotlib-users] Python issue of Computing
	in Science and Engineering available
In-Reply-To: <db6b5ecc0704250929r2d4b0518ma7dc581ea76ee33f@mail.gmail.com>
References: <462F7922.7060704@astraw.com>
	<db6b5ecc0704250911r1804068bp5f07211d88c32b80@mail.gmail.com>
	<88e473830704250918n6c7631f4w514d8dc68b3dde94@mail.gmail.com>
	<db6b5ecc0704250929r2d4b0518ma7dc581ea76ee33f@mail.gmail.com>
Message-ID: <f4f93d420704250956w3c82793dh3060849b373c4444@mail.gmail.com>

On 4/25/07, Fernando Perez <fperez.net at gmail.com> wrote:
> On 4/25/07, John Hunter <jdh2358 at gmail.com> wrote:
> > On 4/25/07, Fernando Perez <fperez.net at gmail.com> wrote:
> > > Since authors are allowed by their publication policy to keep a
> > > publicly available copy of their papers on their personal website,
> > > here's the ipython one:
> >
> > Didn't know that...  here's a link to my matplotlib article
>
> I'm going by the language here:
>
> http://www.ieee.org/web/publications/rights/policies.html
>
> Specifically:
>
>  When IEEE publishes the work, the author must replace the previous
> electronic version of the accepted paper with either (1) the full
> citation to the IEEE work or (2) the IEEE-published version, including
> the IEEE copyright notice and full citation. Prior or revised versions
> of the paper must not be represented as the published version.
>
>
> This explicitly mentions author website redistribution,  as long as
> the official IEEE version is used.
>
> Unless I'm misreading the above, I think it's OK for us to keep such
> copies in our personal sites.  We can link to them from the scipy
> wiki, though I don't think it would be OK to /copy/ the PDFs to the
> scipy wiki.
>
> As always, IANAL and all that.

If any of the authors work for the government their papers are in the
public domain. Here is a reference:

http://cr.yp.to/writing/ieee.html


From tim.hochberg at ieee.org  Wed Apr 25 13:41:19 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 25 Apr 2007 10:41:19 -0700
Subject: [Numpy-discussion] numpy migration
In-Reply-To: <462EAE61.90500@contactbox.co.uk>
References: <462CD6F1.70706@urubu.freeserve.co.uk>
	<e4412d6b0704241038x41dfa7adwd633ad4cb6c16487@mail.gmail.com>
	<462EAE61.90500@contactbox.co.uk>
Message-ID: <e4412d6b0704251041r383ca65bu70eacf411de45de@mail.gmail.com>

On 4/24/07, Duncan Smith <buzzard at contactbox.co.uk> wrote:
>
>
>
> Timothy Hochberg wrote:


[SNIP]


>
> >
> > You should be getting a warning here. Did one disappear in the cut and
> > paste? Or are you using a nonstandard shell that eats warnings? Or an
> old
> > version of numpy?
> >
>
> Python 2.5.1 / numpy 1.0.1 / IDLE (under Windows 2000).  No warning.


I don't use Idle, but I expect that it is just eating everything that goes
to stderr. It'd be easy enough to check with something like:

>>> import sys
>>> print>>sys.stderr, "Does this get printed?"
Does this get printed?

 If this doesn't print anything, then that explains why you aren't seeing a
warning.

> In any event if you want an error rather than a warning for zero division,
>
> > use:
> >
> >>>> 1/i
> >
> > Warning: divide by zero encountered in long_scalars
> > 0
> > SyntaxError: invalid syntax
> >
> >>>> N.seterr(divide='raise')
> >
> > {'over': 'print', 'divide': 'print', 'invalid': 'print', 'under':
> 'ignore'}
> >
> >>>> 1/i
> >
> > Traceback (most recent call last):
> >  File "<stdin>", line 1, in <module>
> > FloatingPointError: divide by zero encountered in long_scalars
> >
> > Although I'm not sure why that's a floating point error.
> >
>
> True, bug or feature?, I'm really looking for a ZeroDivisionError.


I'm guessing that setting this to raise just blindly converts all of the
warnings to FloatingPointErrors. I imagine if someone with some spare cycles
submitted a patch to be more specific it would be accepted. I don't have
time to dive into that right now though.

You can hack around this for the time being by using seterr(divide='call')
and installing an appropriate handler (see code below).

[snip]
>
> > ----------------------------------------------
> >
> >>
> >> An additional problem involves classes that have e.g. __rmul__ methods
> >> defined and are sufficiently similar to numpy arrays that my classes'
> >> __rmul__ methods are not invoked when using numpy scalars.
> >
> >
> >
> >
> > Try adding "__array_priority__ = 10" or somesuch to your classes. This
> > should tell numpy to back off and let your methods go first when doing
> > mixed
> > operations. Searching for "__array_priority__" will probably turn up
> some
> > examples.
> >
>
> Yes, it did.  But for my classes it seems to have no effect.  Below is a
> fairly minimal example.


As it turns out, you need __array_wrap__ also (or maybe just __array_wrap__,
but __array_priority__ won't hurt). Try this:

 """
>>> import numpy as N
>>> ex = MyClass(N.array([[6,7],[8,9]]), ['axis0', 'axis1'])
>>> v = ex.arr[0]
>>> v * ex
[[36, 49],
 [48, 63]]['axis0', 'axis1']
>>> i = ex.arr[0,0]
>>> ex
[[6, 7],
 [8, 9]]['axis0', 'axis1']
>>> ex * i
[[36, 42],
 [48, 54]]['axis0', 'axis1']
>>> i * ex
[[36, 42],
 [48, 54]]['axis0', 'axis1']
>>> ex // 0
Traceback (most recent call last):
  ...
ZeroDivisionError: integer division or modulo by zero
"""
from __future__ import division
import numpy


def handler(message, _):
    if message == 'divide by zero':
        raise ZeroDivisionError('integer division or modulo by zero')
    raise FloatingPointError(message)

numpy.seterrcall(handler)
numpy.seterr(divide='call')

class MyClass(object):
    __array_priority__ = 10

    def __init__(self, arr, labels):
        self.arr = arr
        self.labels = labels

    def __repr__(self):
        return numpy.array2string(self.arr, separator=', ') + repr(
self.labels)

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, key):
        return self.arr[key]

    def __setitem__(self, key, item):
        self.arr[key] = item

    def __array_wrap__(self, arr, context=None):
        return  self.__class__(arr, self.labels)

    def __mul__(self, other):
        return self.__class__(self.arr * other, self.labels)

    __rmul__ = __mul__

    def __floordiv__(self, other):
        return self.__class__(self.arr // other, self.labels)

    def __rfloordiv__(self, other):
        return self.__class__(other // self.arr, self.labels)

if __name__ == "__main__":
    import doctest, scratch
    doctest.testmod(scratch)



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070425/fae3a94e/attachment.html>

From ndbecker2 at gmail.com  Wed Apr 25 13:52:41 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 25 Apr 2007 13:52:41 -0400
Subject: [Numpy-discussion] matlab vs. python question
Message-ID: <f0o4h9$k5m$1@sea.gmane.org>

I'm interested in this comparison (not in starting yet another flame fest). 
I actually know nothing about matlab, but almost all my peers use it.  One
of the things I recall reading on this subject is that matlab doesn't
support OO style programming.  I happened to look on the matlab vendor's
website, and found that it does have classes.  OTOH, I've seen at least
some matlab code, and never saw anyone use these features.

I'm perfectly happy with python myself, but I wonder if anyone has good
arguments for why to prefer python over matlab?



From Chris.Barker at noaa.gov  Wed Apr 25 14:24:47 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 25 Apr 2007 11:24:47 -0700
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <f0o4h9$k5m$1@sea.gmane.org>
References: <f0o4h9$k5m$1@sea.gmane.org>
Message-ID: <462F9CEF.6070203@noaa.gov>

Neal Becker wrote:
> I'm interested in this comparison

There have got to be comparison's on the web -- google away!

My few comments:
> I happened to look on the matlab vendor's
> website, and found that it does have classes.

Matlab added classes in a fairly recent version, so technically, yes, it 
does support OO. However, OO aside, Python is, in many ways, are far 
more sophisticated and capable language. It is better suited to larger 
projects, and well suited to wide variety of software development, 
rather than just numerical work. Indeed, python+numpy supports more 
sophisticated numerical work too (more data types, etc).

So, my reasons for using python+numpy (I did my entire dissertation with 
Matlab -- I loved it at the time):

* Better, more flexible language.

* I can use it for other things I do: web programming, sophisticated 
GUIs, etc.

* It integrates well with lots of other libraries

* It's free: both $$ and libre

What's better about Matlab:

* A wider selection of out-of-the-box numerical routines.

* Excellent integration of command-line and plotting.

In short, $$ and freedom aside, I think Matlab provides a slightly more 
productive environment for interactive experimentation and quickie 
prototypes and scripts, but much less productive for larger projects, or 
for people that need to do non-numerical work too.

just my $0.2

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From matthew.brett at gmail.com  Wed Apr 25 14:32:18 2007
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 25 Apr 2007 19:32:18 +0100
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <f0o4h9$k5m$1@sea.gmane.org>
References: <f0o4h9$k5m$1@sea.gmane.org>
Message-ID: <1e2af89e0704251132t185e582aw79a671f2a517474a@mail.gmail.com>

Hi,

On 4/25/07, Neal Becker <ndbecker2 at gmail.com> wrote:
> I'm interested in this comparison (not in starting yet another flame fest).
> I actually know nothing about matlab, but almost all my peers use it.  One
> of the things I recall reading on this subject is that matlab doesn't
> support OO style programming.  I happened to look on the matlab vendor's
> website, and found that it does have classes.  OTOH, I've seen at least
> some matlab code, and never saw anyone use these features.

Actually, I think I know why you've never seen anyone use matlab
objects, and this is because they are implemented in a rather strange
way that makes them painful to use.  I know because I've used them a
lot myself, and I still have the scars,

Matthew


From robert.kern at gmail.com  Wed Apr 25 14:34:57 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 25 Apr 2007 13:34:57 -0500
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <f0o4h9$k5m$1@sea.gmane.org>
References: <f0o4h9$k5m$1@sea.gmane.org>
Message-ID: <462F9F51.3050903@gmail.com>

Neal Becker wrote:
> I'm perfectly happy with python myself, but I wonder if anyone has good
> arguments for why to prefer python over matlab?

The things that I get annoyed with every time I have to read some Matlab code
are the lack of namespaces and first-class function objects.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From bryanv at enthought.com  Wed Apr 25 14:45:50 2007
From: bryanv at enthought.com (Bryan Van de Ven)
Date: Wed, 25 Apr 2007 13:45:50 -0500
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <f0o4h9$k5m$1@sea.gmane.org>
References: <f0o4h9$k5m$1@sea.gmane.org>
Message-ID: <462FA1DE.1050503@enthought.com>

Neal Becker wrote:
> I'm perfectly happy with python myself, but I wonder if anyone has good
> arguments for why to prefer python over matlab?
> 

 From my own experience, once you move past static plots and want to include 
some kind of interactive GUI (that is, build an actual application) then you are 
in for a world of pain. Also, there are far fewer tools and options for rich GUI 
development in Matlab than in python.



From buzzard at contactbox.co.uk  Wed Apr 25 15:22:54 2007
From: buzzard at contactbox.co.uk (Duncan Smith)
Date: Wed, 25 Apr 2007 20:22:54 +0100
Subject: [Numpy-discussion] numpy migration
In-Reply-To: <e4412d6b0704251041r383ca65bu70eacf411de45de@mail.gmail.com>
References: <462CD6F1.70706@urubu.freeserve.co.uk>	<e4412d6b0704241038x41dfa7adwd633ad4cb6c16487@mail.gmail.com>	<462EAE61.90500@contactbox.co.uk>
	<e4412d6b0704251041r383ca65bu70eacf411de45de@mail.gmail.com>
Message-ID: <462FAA8E.2030409@contactbox.co.uk>



Timothy Hochberg wrote:
> On 4/24/07, Duncan Smith <buzzard at contactbox.co.uk> wrote:
> 
>>
>>
>>
>> Timothy Hochberg wrote:
> 
> 
> 
> [SNIP]
> 
> 
>>
>> >
>> > You should be getting a warning here. Did one disappear in the cut and
>> > paste? Or are you using a nonstandard shell that eats warnings? Or an
>> old
>> > version of numpy?
>> >
>>
>> Python 2.5.1 / numpy 1.0.1 / IDLE (under Windows 2000).  No warning.
> 
> 
> 
> I don't use Idle, but I expect that it is just eating everything that goes
> to stderr. It'd be easy enough to check with something like:
> 
>>>> import sys
>>>> print>>sys.stderr, "Does this get printed?"
> 
> Does this get printed?
> 

Yes.  Maybe it just suppresses warnings by default; but I can't find
anything relevant to that.

> If this doesn't print anything, then that explains why you aren't seeing a
> warning.
> 
>> In any event if you want an error rather than a warning for zero
>> division,
>>
>> > use:
>> >
>> >>>> 1/i
>> >
>> > Warning: divide by zero encountered in long_scalars
>> > 0
>> > SyntaxError: invalid syntax
>> >
>> >>>> N.seterr(divide='raise')
>> >
>> > {'over': 'print', 'divide': 'print', 'invalid': 'print', 'under':
>> 'ignore'}
>> >
>> >>>> 1/i
>> >
>> > Traceback (most recent call last):
>> >  File "<stdin>", line 1, in <module>
>> > FloatingPointError: divide by zero encountered in long_scalars
>> >
>> > Although I'm not sure why that's a floating point error.
>> >
>>
>> True, bug or feature?, I'm really looking for a ZeroDivisionError.
> 
> 
> 
> I'm guessing that setting this to raise just blindly converts all of the
> warnings to FloatingPointErrors. I imagine if someone with some spare
> cycles
> submitted a patch to be more specific it would be accepted. I don't have
> time to dive into that right now though.
> 
> You can hack around this for the time being by using seterr(divide='call')
> and installing an appropriate handler (see code below).
> 

[snip]

Excellent, that does the trick.  It will keep me going for now.  I'll
have to come back to my classes and rethink the design when I have more
time.  Many thanks.

Duncan


From sturla at molden.no  Wed Apr 25 15:29:14 2007
From: sturla at molden.no (Sturla Molden)
Date: Wed, 25 Apr 2007 21:29:14 +0200
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <462F9F51.3050903@gmail.com>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
Message-ID: <462FAC0A.2040204@molden.no>

On 4/25/2007 8:34 PM, Robert Kern wrote:

> The things that I get annoyed with every time I have to read some Matlab code
> are the lack of namespaces and first-class function objects.

Matlab does have first-class function objects. You can get a handle to 
any function using the @ operator. Matlab has closures as well.


function f = addn(n)
    tmp = n
    function y = closure(x)
       y = x + tmp
    end
    f = @closure
end

The most recent version of Matlab also have classes and namespaces 
implemented properly - not like the old broken Matlab objects. But I've 
never used it as I have stopped paying for maintenance of my Matlab 
license - I use Python anyway. But if it weren't for NumPy, SciPy and 
Matplotlib, I would probably still be using Matlab.

As a programming language Matlab is very weak compared to Python. Matlab 
is to Fortran what Python is to Lisp. As an advanced desktop calculator 
Matlab has the edge. But that has more to do with the interactive 
environment and available toolboxes than the language it self. It is 
also easier to write C or Fortran extensions for Matlab than for Python.


Sturla Molden
















From robert.kern at gmail.com  Wed Apr 25 15:31:32 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 25 Apr 2007 14:31:32 -0500
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <462FAC0A.2040204@molden.no>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no>
Message-ID: <462FAC94.1040207@gmail.com>

Sturla Molden wrote:
> On 4/25/2007 8:34 PM, Robert Kern wrote:
> 
>> The things that I get annoyed with every time I have to read some Matlab code
>> are the lack of namespaces and first-class function objects.
> 
> Matlab does have first-class function objects. You can get a handle to 
> any function using the @ operator. Matlab has closures as well.

I wish people would use them, then.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Wed Apr 25 15:45:30 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 25 Apr 2007 12:45:30 -0700
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <462FAC0A.2040204@molden.no>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no>
Message-ID: <462FAFDA.80507@noaa.gov>

Sturla Molden wrote:
> It is 
> also easier to write C or Fortran extensions for Matlab than for Python.

Really? I"m not so sure about that -- I found mex file writing pretty 
painful.

With weave, boost, pyrex, swig, f2py, etc, the hardest thing about 
writing extensions for Python is choosing which tool to use!

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From focke at slac.stanford.edu  Wed Apr 25 15:44:18 2007
From: focke at slac.stanford.edu (Warren Focke)
Date: Wed, 25 Apr 2007 12:44:18 -0700 (PDT)
Subject: [Numpy-discussion] Oddity with numpy.int64 integer division
In-Reply-To: <Pine.LNX.4.58.0704241439420.11417@circinus.slac.stanford.edu>
References: <2948.84.167.127.8.1177359648.squirrel@webmail.marquardt.sc>
	<27297.84.167.127.8.1177360140.squirrel@webmail.marquardt.sc>
	<9437.84.167.127.8.1177360916.squirrel@webmail.marquardt.sc>
	<0A310220-A12A-4375-82BD-4DFFD3F74432@physics.mcmaster.ca>
	<Pine.LNX.4.58.0704231902470.6741@circinus.slac.stanford.edu>
	<4D1684DB-D01D-44B0-B8DF-C3D186443891@physics.mcmaster.ca>
	<Mahogany-0.67.0-736-20070424-163842.00@american.edu>
	<26758.84.167.102.129.1177448640.squirrel@webmail.marquardt.sc>
	<462E71BB.2030204@gmail.com>
	<e4412d6b0704241433l526b4263td02a3a5d937e5afb@mail.gmail.com>
	<Pine.LNX.4.58.0704241439420.11417@circinus.slac.stanford.edu>
Message-ID: <Pine.LNX.4.58.0704251243220.31346@circinus.slac.stanford.edu>

On futher contemplation, and hearing others' arguments, I'm changing my
vote.  Make it compatible with python.

w

On Tue, 24 Apr 2007, Warren Focke wrote:

>
>
> On Tue, 24 Apr 2007, Timothy Hochberg wrote:
>
> > On 4/24/07, Robert Kern <robert.kern at gmail.com> wrote:
> > >
> > > Christian Marquardt wrote:
> >
> >
> >
> > > > Restore the invariant, and follow python.
> >
> >
> > This seems to imply that once upon a time numpy/numeric/numarray followed
> > python here, but as far as I can recall that was never the case. Instead
> > they followed C completely, ostensibly for performance reasons. That
> > preserved the invariant in question, but not compatibility with Python.
>
> That's how I remember it, too.  And I think I'd prefer it be that way
> again, unless it can be demonstrated that the hit from being consistent
> with python is "small".
>
> Note that C99 specifies behavior opposite that of python (but still
> consistent with itself).
>
> w
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From wbaxter at gmail.com  Wed Apr 25 15:46:13 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Thu, 26 Apr 2007 04:46:13 +0900
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <462FAC94.1040207@gmail.com>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no> <462FAC94.1040207@gmail.com>
Message-ID: <e86a5fd00704251246h1fbe4003v4adf28376bd2ea54@mail.gmail.com>

On 4/26/07, Robert Kern <robert.kern at gmail.com> wrote:
> Sturla Molden wrote:
> > On 4/25/2007 8:34 PM, Robert Kern wrote:
> >
> >> The things that I get annoyed with every time I have to read some Matlab code
> >> are the lack of namespaces and first-class function objects.
> >
> > Matlab does have first-class function objects. You can get a handle to
> > any function using the @ operator. Matlab has closures as well.
>
> I wish people would use them, then.

I think part of the problem is that most people who learn Matlab learn
it in school strictly as a computing tool rather than a programming
language.  But there are many language features that people just don't
know about, because writing nice programs is not usually the emphasis
in the courses where Matlab is taught and used.

Python is better generally as a language, although the self. thing
makes me want to pull my hair out sometimes.  I find myself bending
over backwards to avoid creating classes just because converting a
function into a method generally results in an annoying amount of
self.this self.that self.theother junk.

--bb


From sturla at molden.no  Wed Apr 25 15:59:05 2007
From: sturla at molden.no (Sturla Molden)
Date: Wed, 25 Apr 2007 21:59:05 +0200
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <462FAC94.1040207@gmail.com>
References: <f0o4h9$k5m$1@sea.gmane.org>
	<462F9F51.3050903@gmail.com>	<462FAC0A.2040204@molden.no>
	<462FAC94.1040207@gmail.com>
Message-ID: <462FB309.1070707@molden.no>

On 4/25/2007 9:31 PM, Robert Kern wrote:

>>> The things that I get annoyed with every time I have to read some Matlab code
>>> are the lack of namespaces and first-class function objects.
>> Matlab does have first-class function objects. You can get a handle to 
>> any function using the @ operator. Matlab has closures as well.
> 
> I wish people would use them, then.


You have to realize that Matlab is mainly used by people who are not 
skilled programmers but scientists and engineers, the majority of which 
have never used anything else except perhaps Fortran 77. An array is the 
most advanced data structure they will ever need. I should know, I was 
among them for several years. Then I did something as silly as teaching 
myself C++. God knows why.

Matlab gives interactive access to Fortran arrays and libraries, e.g. 
LAPACK. That is what Matlab was designed to do and it does that very 
well. Even the name Matlab reflects that, it is an abbreviation for 
"matrix laboratory". I am not surprised that most Matlab users do not 
use facilities like function handles or closures. But Matlab does have 
that, nevertheless.


Sturla Molden






From rowen at cesmail.net  Wed Apr 25 16:01:53 2007
From: rowen at cesmail.net (Russell E. Owen)
Date: Wed, 25 Apr 2007 13:01:53 -0700
Subject: [Numpy-discussion] Questions about converting to numpy
Message-ID: <rowen-01812A.13015325042007@sea.gmane.org>

So I finally bit the bullet and converted most of my code from Numeric 
and numarray to numpy. (I haven't yet tried to convert one package that 
makes heavy use of nd_image and has C extensions).

But it left me with a few questions:

- What exception does numpy throw if it runs out of memory? (I can try 
harder to make it do that, but trying to chew up all memory tends to 
slow the machine down and my first tests weren't successful) -- the 
equivalent to numarray.memory.error. The numpy book is silent on the 
issue of what exceptions numpy can throw (at least the index was).

- Is there a list of the data types that we can expect to be available 
on all regular platforms (including 32-bit linux, MacOS X and Windows) 
and of usual speed for computations (instead of some kind of slow 
emulation)?

- Even after reading the book I'm not really clear on why one would use 
numpy.float_ instead of numpy.float or float for day-to-day programming 
(where the size doesn't matter enough to use float64 or whatever). Any 
hints?

Overall it went fine. I'm not entirely sure I caught everything yet, but 
the new code seems to be working. 

-- Russell



From robert.kern at gmail.com  Wed Apr 25 16:11:29 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 25 Apr 2007 15:11:29 -0500
Subject: [Numpy-discussion] Questions about converting to numpy
In-Reply-To: <rowen-01812A.13015325042007@sea.gmane.org>
References: <rowen-01812A.13015325042007@sea.gmane.org>
Message-ID: <462FB5F1.9010406@gmail.com>

Russell E. Owen wrote:
> So I finally bit the bullet and converted most of my code from Numeric 
> and numarray to numpy. (I haven't yet tried to convert one package that 
> makes heavy use of nd_image and has C extensions).
> 
> But it left me with a few questions:
> 
> - What exception does numpy throw if it runs out of memory? (I can try 
> harder to make it do that, but trying to chew up all memory tends to 
> slow the machine down and my first tests weren't successful) -- the 
> equivalent to numarray.memory.error. The numpy book is silent on the 
> issue of what exceptions numpy can throw (at least the index was).

The standard MemoryError exception.

> - Is there a list of the data types that we can expect to be available 
> on all regular platforms (including 32-bit linux, MacOS X and Windows) 
> and of usual speed for computations (instead of some kind of slow 
> emulation)?

Not anywhere particular, but these might not be available/useful on all
platforms: float96, float128, float256, complex182, complex256, complex512,
int64, uint64, int128, uint128, longlong, ulonglong, longfloat, clongfloat.

> - Even after reading the book I'm not really clear on why one would use 
> numpy.float_ instead of numpy.float or float for day-to-day programming 
> (where the size doesn't matter enough to use float64 or whatever). Any 
> hints?

If you wanted an array scalar of the "default" float dtype (whatever that
happened to be), you would have to use float_. Of course the "default" float
dtype is always (and will always be, AFAICT) float64, so really, you might as
well use that.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Wed Apr 25 16:23:40 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 25 Apr 2007 13:23:40 -0700
Subject: [Numpy-discussion] Questions about converting to numpy
In-Reply-To: <rowen-01812A.13015325042007@sea.gmane.org>
References: <rowen-01812A.13015325042007@sea.gmane.org>
Message-ID: <462FB8CC.5020709@noaa.gov>

Russell E. Owen wrote:
> So I finally bit the bullet and converted most of my code from Numeric 
> and numarray to numpy.

good for you!

I can only help with one:
> - Even after reading the book I'm not really clear on why one would use 
> numpy.float_ instead of numpy.float or float

They float and numpy.float are the same, and numpy.float_ is the same as 
numpy.float64:

 >>> import numpy
 >>> float is numpy.float
True
 >>> numpy.float64 is numpy.float64
True
 >>>

float was added to the numpy namespace so that we could write consistent 
code like:

a = array(object, numpy.float32)
b = array(object, numpy.float)

i.e. have it all in the same namespace.

I'm not sure why float_ is an alias for float64, though I'm guessing 
it's possible that on some platforms they are not the same.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From robert.kern at gmail.com  Wed Apr 25 17:31:50 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 25 Apr 2007 16:31:50 -0500
Subject: [Numpy-discussion] Questions about converting to numpy
In-Reply-To: <462FB8CC.5020709@noaa.gov>
References: <rowen-01812A.13015325042007@sea.gmane.org>
	<462FB8CC.5020709@noaa.gov>
Message-ID: <462FC8C6.3030200@gmail.com>

Christopher Barker wrote:

> I can only help with one:
>> - Even after reading the book I'm not really clear on why one would use 
>> numpy.float_ instead of numpy.float or float
> 
> They float and numpy.float are the same, and numpy.float_ is the same as 
> numpy.float64:
> 
>  >>> import numpy
>  >>> float is numpy.float
> True
>  >>> numpy.float64 is numpy.float64
> True
>  >>>
> 
> float was added to the numpy namespace so that we could write consistent 
> code like:
> 
> a = array(object, numpy.float32)
> b = array(object, numpy.float)
> 
> i.e. have it all in the same namespace.
> 
> I'm not sure why float_ is an alias for float64, though I'm guessing 
> it's possible that on some platforms they are not the same.

Rather, numpy.float used to be an alias for numpy.float64; however, it overrode
the builtin float() when "from numpy import *" was used at the interactive
prompt. Consequently, we renamed it numpy.float_ and specifically imported the
builtin float as numpy.float such that we didn't break code that had already
started using "numpy.float".

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From bthom at cs.hmc.edu  Wed Apr 25 17:36:16 2007
From: bthom at cs.hmc.edu (belinda thom)
Date: Wed, 25 Apr 2007 14:36:16 -0700
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <462FAC94.1040207@gmail.com>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no> <462FAC94.1040207@gmail.com>
Message-ID: <72D62C95-41DE-4013-B878-1A8E5BB538FF@cs.hmc.edu>

I used to use them frequently.

--b

On Apr 25, 2007, at 12:31 PM, Robert Kern wrote:

> Sturla Molden wrote:
>> On 4/25/2007 8:34 PM, Robert Kern wrote:
>>
>>> The things that I get annoyed with every time I have to read some  
>>> Matlab code
>>> are the lack of namespaces and first-class function objects.
>>
>> Matlab does have first-class function objects. You can get a  
>> handle to
>> any function using the @ operator. Matlab has closures as well.
>
> I wish people would use them, then.
>
> -- 
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a  
> harmless enigma
>  that is made terrible by our own mad attempt to interpret it as  
> though it had
>  an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From bthom at cs.hmc.edu  Wed Apr 25 17:53:31 2007
From: bthom at cs.hmc.edu (belinda thom)
Date: Wed, 25 Apr 2007 14:53:31 -0700
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <e86a5fd00704251246h1fbe4003v4adf28376bd2ea54@mail.gmail.com>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no> <462FAC94.1040207@gmail.com>
	<e86a5fd00704251246h1fbe4003v4adf28376bd2ea54@mail.gmail.com>
Message-ID: <B11B844B-3741-422C-B90C-9E43F7E9B279@cs.hmc.edu>


On Apr 25, 2007, at 12:46 PM, Bill Baxter wrote:

> On 4/26/07, Robert Kern <robert.kern at gmail.com> wrote:
>> Sturla Molden wrote:
>>> On 4/25/2007 8:34 PM, Robert Kern wrote:
>>>
>>>> The things that I get annoyed with every time I have to read  
>>>> some Matlab code
>>>> are the lack of namespaces and first-class function objects.
>>>
>>> Matlab does have first-class function objects. You can get a  
>>> handle to
>>> any function using the @ operator. Matlab has closures as well.
>>
>> I wish people would use them, then.
>
> I think part of the problem is that most people who learn Matlab learn
> it in school strictly as a computing tool rather than a programming
> language.  But there are many language features that people just don't
> know about, because writing nice programs is not usually the emphasis
> in the courses where Matlab is taught and used.
>
> Python is better generally as a language, although the self. thing
> makes me want to pull my hair out sometimes.  I find myself bending
> over backwards to avoid creating classes just because converting a
> function into a method generally results in an annoying amount of
> self.this self.that self.theother junk.

Agree w/most of what you've said, but will add one other thing that  
drives me nuts in python that hasn't been a problem in Matplotlib:

In Python, if interacting w/the interpreter as your primary IDE, and  
if you've got multiple files that depend on one another that you're  
modifying, then you need to restart the interpreter frequently b/c  
otherwise things in the interpreter can be stale; IOW, changes to  
several interdependent files aren't easy to import so that everything  
in your interpreted environment reflects the latest code. Yeah,  
there's reload tricks, but doing them in the right order and the  
right number of times can be a pain when dependencies are cyclic.

I realize in general that this issue of stale code is just difficult,  
that its not inherently a Python problem per se, for automatic  
percolation of code changes backwards in time is difficult in  
general, but I've never had the problem bite me when I was developing  
in Matlab. I just save whatever file, and it appears that whatever's  
latest on disk is what's executed. (Friends who know more about PL  
than I tell me I've been lucky.)

It was particularly a problem for me when I was using matplotlib  
interactively, which relates to this discussion b/c people often use  
Matlab for the graphing. W/that package, I couldn't use something  
like IDLE in such a way that it restarted its session every time I  
asked it to run a changed file. IDLE was nice, though, in that it had  
mouse-settable break points for debugging. That's the other big miss  
for me: Python's debugger is cumbersome to use.

Chris mentioned the IDE issue earlier. IMHO, IPython is a better IDE  
than IDLE, but its features (e.g. %psource prints code from disk, so  
most recent changes are displayed, but it doesn't reflect the code  
that an object actually executes) caused me even more confusion  
regarding stale code initially.

I've finally gotten used to developing w/IPython (using Francisco's  
debug_here() hack for better debugging integration), and I'm almost  
as productive as I was before, but it took several months to get there.

I've used Matlab's GUI stuff a bit and didn't find it as awful as  
others on this post claim. It also has the nice advantage of handling  
both GUI and figure-related threads in the same loop, so I could use  
it to write more sophisticated applications than one should do when  
using matplotlib+python alone. But I've yet to really use Python- 
based GUIs, which are probably much better than what Matlab has to  
offer.

HTH,
--b



From ryanlists at gmail.com  Wed Apr 25 18:01:36 2007
From: ryanlists at gmail.com (Ryan Krauss)
Date: Wed, 25 Apr 2007 17:01:36 -0500
Subject: [Numpy-discussion] [Matplotlib-users] Python issue of Computing
	in Science and Engineering available
In-Reply-To: <88e473830704250918n6c7631f4w514d8dc68b3dde94@mail.gmail.com>
References: <462F7922.7060704@astraw.com>
	<db6b5ecc0704250911r1804068bp5f07211d88c32b80@mail.gmail.com>
	<88e473830704250918n6c7631f4w514d8dc68b3dde94@mail.gmail.com>
Message-ID: <c5b438120704251501n522df7b8v59d7e6348e1c4de@mail.gmail.com>

My CiSE article can be downloaded from here:

http://www.siue.edu/~rkrauss/python_stuff.html

Ryan

On 4/25/07, John Hunter <jdh2358 at gmail.com> wrote:
> On 4/25/07, Fernando Perez <fperez.net at gmail.com> wrote:
> > Since authors are allowed by their publication policy to keep a
> > publicly available copy of their papers on their personal website,
> > here's the ipython one:
>
> Didn't know that...  here's a link to my matplotlib article
>
> http://nitace.bsd.uchicago.edu/misc/c3sci.pdf
>
> It might be nice to create a scipy wiki page linking to these PDFs.
>
> JDH
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Matplotlib-users mailing list
> Matplotlib-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>


From bblais at bryant.edu  Wed Apr 25 20:28:46 2007
From: bblais at bryant.edu (Brian Blais)
Date: Wed, 25 Apr 2007 20:28:46 -0400
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <462FAFDA.80507@noaa.gov>
References: <f0o4h9$k5m$1@sea.gmane.org>
	<462F9F51.3050903@gmail.com>	<462FAC0A.2040204@molden.no>
	<462FAFDA.80507@noaa.gov>
Message-ID: <462FF23E.7060901@bryant.edu>

Christopher Barker wrote:
> Sturla Molden wrote:
>> It is 
>> also easier to write C or Fortran extensions for Matlab than for Python.
> 
> Really? I"m not so sure about that -- I found mex file writing pretty 
> painful.
> 
> With weave, boost, pyrex, swig, f2py, etc, the hardest thing about 
> writing extensions for Python is choosing which tool to use!
> 

I agree with Sturla here, up until Pyrex.  Looking at the Python API, swig, f2py,
etc. I found mex files far more straightforward.  Mostly this is because of the
(mostly) single datatype in Matlab: the double matrix.  Having to do reference
counting in the Python API is a drag.

Now, with Pyrex, that's a different matter entirely.  Pyrex makes extension writing
so incredibly easy, its one of the major reasons I switched from Matlab to Python.

I'm trying to convince some people at work to explore Python as a Matlab replacement.
    If all you do is numerics, then Matlab is easier (by a little).  Once you go
beyond numerics (which you do for any real project), like reading datafiles,
interfacing with the Internet, accessing databases, etc... suddenly Python is a *lot*
nicer.

			bb

-- 
-----------------

              bblais at bryant.edu
              http://web.bryant.edu/~bblais



From david at ar.media.kyoto-u.ac.jp  Wed Apr 25 22:38:09 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 26 Apr 2007 11:38:09 +0900
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <462FF23E.7060901@bryant.edu>
References: <f0o4h9$k5m$1@sea.gmane.org>	<462F9F51.3050903@gmail.com>	<462FAC0A.2040204@molden.no>	<462FAFDA.80507@noaa.gov>
	<462FF23E.7060901@bryant.edu>
Message-ID: <46301091.9020409@ar.media.kyoto-u.ac.jp>

Brian Blais wrote:
> Christopher Barker wrote:
>> Sturla Molden wrote:
>>> It is 
>>> also easier to write C or Fortran extensions for Matlab than for Python.
>> Really? I"m not so sure about that -- I found mex file writing pretty 
>> painful.
>>
>> With weave, boost, pyrex, swig, f2py, etc, the hardest thing about 
>> writing extensions for Python is choosing which tool to use!
>>
>
> I agree with Sturla here, up until Pyrex.  Looking at the Python API, swig, f2py,
> etc. I found mex files far more straightforward.  Mostly this is because of the
> (mostly) single datatype in Matlab: the double matrix.  Having to do reference
> counting in the Python API is a drag.
It is a drag, but the matlab's way of doing things is totally broken, 
not simple. The Matlab C api is one of the reason why I looked at python 
in the first place. First, the memory management in mex files is 
unusable for anything non trivial (for example, what happens if you do 
Ctrl+C, but you are calling a function from a library which does malloc 
? You're screwed; many 3rd party mex files crash when you use ctrl+C in 
matlab). Also, wrapping with ctypes is really several order of magnitude 
easier AND more powerful than mex files. One thing I use to impress 
people used to matlab is to translate mex examples using ctypes: instead 
of many lines of boring glue *in C*, a few lines of python

The other big drawback of matlab is the lack of data structures which 
are not arrays. Cell is again, unusable for most things (most algorithms 
do not work on them): in python, you have list and dictionaries, and 
they certainly first class citizens in the language.

Finally, doing complex GUI is impossible in matlab. More generally, 
doing things not directly related to numerical computing is really 
difficult in matlab, because the language is not suitable for general 
programming, and because of the totally brain damaged C api.

I've done a fairly big amount of matlab programming, and consider myself 
quite familiar with it. I've done complex mex programming, even some 
java programming for interacting between matlab and java list/hashmap, 
etc... When you do those kind of things, the limitations of matlab are 
striking. Matlab is simpler to use for simple things: for me, there is 
no question about that fact. But python makes easy things just a bit 
more complicated, and all other things possible. The perfect example is 
namespace: not having namespace IS easier, at first. But this quickly 
makes things unmanageable. Functions are easier in matlab, but when you 
want to define function with many arguments and default values, this 
also becomes quickly unmanageable.

David


From peridot.faceted at gmail.com  Wed Apr 25 22:56:24 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 25 Apr 2007 22:56:24 -0400
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <462FB309.1070707@molden.no>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no> <462FAC94.1040207@gmail.com>
	<462FB309.1070707@molden.no>
Message-ID: <ce557a360704251956y10e6b239rab53cd4e38bcf4b0@mail.gmail.com>

On 25/04/07, Sturla Molden <sturla at molden.no> wrote:

> You have to realize that Matlab is mainly used by people who are not
> skilled programmers but scientists and engineers, the majority of which
> have never used anything else except perhaps Fortran 77. An array is the
> most advanced data structure they will ever need. I should know, I was
> among them for several years. Then I did something as silly as teaching
> myself C++. God knows why.

I have a quibble with this. I work with scientists too, and while an
array is usually the most sophisticated data structure they ever use,
it's not the most sophisticated data structure they ever *need*. Many
many times I've seen a project that is more limited, more error-prone,
slower, and more painful to use because the programmer didn't know
about or didn't use the right programming tools. Or was using a
language that didn't make them readily available.

Take, for example, optparse. Hardly a scientific package, indeed, but
it makes little command-line utilities vastly easier to write right.
And at least in our group, small command-line utilities do an awful
lot of the work.

Now, teaching us to actually write half-decent programs is another story...

Anne M. Archibald


From charlesr.harris at gmail.com  Wed Apr 25 23:46:48 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 25 Apr 2007 21:46:48 -0600
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <ce557a360704251956y10e6b239rab53cd4e38bcf4b0@mail.gmail.com>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no> <462FAC94.1040207@gmail.com>
	<462FB309.1070707@molden.no>
	<ce557a360704251956y10e6b239rab53cd4e38bcf4b0@mail.gmail.com>
Message-ID: <e06186140704252046l723914a8t501e4641d0f3fae3@mail.gmail.com>

On 4/25/07, Anne Archibald <peridot.faceted at gmail.com> wrote:
>
> On 25/04/07, Sturla Molden <sturla at molden.no> wrote:
>
> > You have to realize that Matlab is mainly used by people who are not
> > skilled programmers but scientists and engineers, the majority of which
> > have never used anything else except perhaps Fortran 77. An array is the
> > most advanced data structure they will ever need. I should know, I was
> > among them for several years. Then I did something as silly as teaching
> > myself C++. God knows why.
>
> I have a quibble with this. I work with scientists too, and while an
> array is usually the most sophisticated data structure they ever use,
> it's not the most sophisticated data structure they ever *need*. Many
> many times I've seen a project that is more limited, more error-prone,
> slower, and more painful to use because the programmer didn't know
> about or didn't use the right programming tools. Or was using a
> language that didn't make them readily available.


Yep. I remember a discussion with a guy whose only experience was FORTRAN. I
tried to point out the utility of such things as lists, stacks, and fifo's
in adaptive algorithms. He just didn't see the need, mostly because he was
unfamiliar with those data structures and just didn't think that way, so
missed all the possibilities.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070425/19a4c5f0/attachment.html>

From bronto at pobox.com  Thu Apr 26 02:34:07 2007
From: bronto at pobox.com (Anton Sherwood)
Date: Wed, 25 Apr 2007 23:34:07 -0700
Subject: [Numpy-discussion] sort bug
Message-ID: <463047DF.7070700@pobox.com>

This code --

	adj = [ [eval(y) for y in x.split()] for x in infile ]
	val,vec = numpy.linalg.eig(adj)
	master = zip( val, vec.transpose() )
	master.sort()

*sometimes* gives this error:

	Traceback (most recent call last):
	  File "3work.py", line 14, in <module>
	    master.sort()
	ValueError: The truth value of an array with more
	than one element is ambiguous. Use a.any() or a.all()

What does sort() care about truth values?!

numpy.linalg.eig() sometimes returns complex values, which may not have 
a natural ordering, but I get this error (sometimes) even when it 
returns reals.

-- 
Anton Sherwood, http://www.ogre.nu/
"How'd ya like to climb this high *without* no mountain?" --Porky Pine


From nadavh at visionsense.com  Thu Apr 26 02:52:27 2007
From: nadavh at visionsense.com (Nadav Horesh)
Date: Thu, 26 Apr 2007 09:52:27 +0300
Subject: [Numpy-discussion] matlab vs. python question
Message-ID: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>


Beside proper programing paradigm Python easily scales to large-scale number crunching: You can run large-matrices calculations with about 1/2 to 1/4 of memory consumption comparing to Matlab. It is not difficult to construct a program that run over several computers (independent  of the hardware and OS they have). If you are willing to invest a little time in software integration you'll have an access to powerful computing packages such as VTK, OpenCV, etc.

  Nadav.

-----Original Message-----
From:	numpy-discussion-bounces at scipy.org on behalf of Christopher Barker
Sent:	Wed 25-Apr-07 20:24
To:	Discussion of Numerical Python
Cc:	
Subject:	Re: [Numpy-discussion] matlab vs. python question

Neal Becker wrote:
> I'm interested in this comparison

There have got to be comparison's on the web -- google away!

My few comments:
> I happened to look on the matlab vendor's
> website, and found that it does have classes.

Matlab added classes in a fairly recent version, so technically, yes, it 
does support OO. However, OO aside, Python is, in many ways, are far 
more sophisticated and capable language. It is better suited to larger 
projects, and well suited to wide variety of software development, 
rather than just numerical work. Indeed, python+numpy supports more 
sophisticated numerical work too (more data types, etc).

So, my reasons for using python+numpy (I did my entire dissertation with 
Matlab -- I loved it at the time):

* Better, more flexible language.

* I can use it for other things I do: web programming, sophisticated 
GUIs, etc.

* It integrates well with lots of other libraries

* It's free: both $$ and libre

What's better about Matlab:

* A wider selection of out-of-the-box numerical routines.

* Excellent integration of command-line and plotting.

In short, $$ and freedom aside, I think Matlab provides a slightly more 
productive environment for interactive experimentation and quickie 
prototypes and scripts, but much less productive for larger projects, or 
for people that need to do non-numerical work too.

just my $0.2

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion





From oliphant.travis at ieee.org  Thu Apr 26 04:26:59 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Thu, 26 Apr 2007 02:26:59 -0600
Subject: [Numpy-discussion] sort bug
In-Reply-To: <463047DF.7070700@pobox.com>
References: <463047DF.7070700@pobox.com>
Message-ID: <46306253.20803@ieee.org>

Anton Sherwood wrote:
> This code --
>
> 	adj = [ [eval(y) for y in x.split()] for x in infile ]
> 	val,vec = numpy.linalg.eig(adj)
> 	master = zip( val, vec.transpose() )
> 	master.sort()
>
> *sometimes* gives this error:
>
> 	Traceback (most recent call last):
> 	  File "3work.py", line 14, in <module>
> 	    master.sort()
> 	ValueError: The truth value of an array with more
> 	than one element is ambiguous. Use a.any() or a.all()
>
> What does sort() care about truth values?!
>   

The sort method on lists basically uses cmp(x,y) to determine sort 
order.  In this case, I suspect you are getting errors whenever you have 
equal-valued eigenvalues so the comparison has to go to the second item 
in the tuple (which is the array).   

cmp(x,y) must return -1, 0, or 1 which doesn't work on arrays with more 
than 1 element because it is ambiguous.  Thus you get this error.

The operation is undefined.  What do you actually want to do when you 
have equal-valued eigenvalues?



-Travis





From matthew.brett at gmail.com  Thu Apr 26 05:44:43 2007
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 26 Apr 2007 10:44:43 +0100
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>
Message-ID: <1e2af89e0704260244j186ea867pd42c06689d4c6f33@mail.gmail.com>

Well - these threads always go on for a long time, but...

I've used matlab heavily for 10 years.  I found that I had to use perl
and C fairly heavily to get things done that matlab could not do well.
 Now I've switched to numpy, scipy, matplotlib, there is really
nothing I miss in matlab.  We would not attempt what we are doing now:

http://neuroimaging.scipy.org/

in matlab - it's just not the right tool for a large scale programming effort.

I agree that matlab has many attractions as a teaching tool and for
small numeric processing scripts, but if you are writing a large to
medium-sized application, I really don't think there is any
comparison...

Matthew


From hurak at fel.cvut.cz  Thu Apr 26 05:45:38 2007
From: hurak at fel.cvut.cz (=?UTF-8?B?WmRlbsSbayBIdXLDoWs=?=)
Date: Thu, 26 Apr 2007 11:45:38 +0200
Subject: [Numpy-discussion] matlab vs. python question
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>
Message-ID: <f0psc2$vfd$1@sea.gmane.org>

Nadav Horesh wrote:

> Matlab added classes in a fairly recent version, ...

Just wanted to correct this. Classes were introduced to Matlab something
like ten years ago. I guess that it was in version 5.

Zdenek



From hurak at fel.cvut.cz  Thu Apr 26 06:06:56 2007
From: hurak at fel.cvut.cz (=?UTF-8?B?WmRlbsSbayBIdXLDoWs=?=)
Date: Thu, 26 Apr 2007 12:06:56 +0200
Subject: [Numpy-discussion] matlab vs. python question
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>
	<1e2af89e0704260244j186ea867pd42c06689d4c6f33@mail.gmail.com>
Message-ID: <f0ptk0$5f3$1@sea.gmane.org>

Matthew Brett wrote:
[...]
> I agree that matlab has many attractions as a teaching tool and for
> small numeric processing scripts, but if you are writing a large to
> medium-sized application, I really don't think there is any
> comparison...
[...]

Matthew, there are also other (engineering) computational needs than just
neuroimaging. 

Coming from the field of control engineering, I don't think that at this
moment there is any replacement for their graphical interface to solvers
for nonlinear differential/difference equations called Simulink. Its
extension for discrete-event systems called Stateflow is also impressive. I
know on no other tool (commercial or free) that would offer this high
productivity. 

But what makes Matlab difficult to be replaced is that lots of other
projects (commercial: Mathematica, Maple, ... and free: octave, maxima,
scipy, ...) only offer computation and visualization, while engineers in my
field also need INTERACTION OF THE SYSTEM WITH EXTERNAL WORLD. That is,
compatibility with a real-time operating system and MOST available
input-output (AD-DA) cards. Being able to acquire measurement data from an
external industrial system, process them computationally (for instance,
solving some Riccati matrix differential equations), visualize the data and
put the computed results back to the real system, this is what we need.
Well, to make this complete, I should say that Scilab project is starting
to be a competitor. But it is not completely free and the user comfort is
not too high at the moment. (Perhaps joining efforts with Openmodelica
developers is the way to go for Python community?)

I am absolutely sure that Python (and Scipy and Numpy) project has a
potential to fulfill also these needs (and by starting using these and
sharing my own code I would like to contribute), but it is not the case at
the moment. Without being negative and discouraging, I think it is fair to
say that currently for some people it would be very difficult to switch
completely to Python libraries. 

But surely this is improving steadily.

Zdenek Hurak



From oliphant.travis at ieee.org  Thu Apr 26 06:36:59 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Thu, 26 Apr 2007 04:36:59 -0600
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <f0ptk0$5f3$1@sea.gmane.org>
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>	<1e2af89e0704260244j186ea867pd42c06689d4c6f33@mail.gmail.com>
	<f0ptk0$5f3$1@sea.gmane.org>
Message-ID: <463080CB.7090000@ieee.org>

Zden?k Hur?k wrote:
> Coming from the field of control engineering, I don't think that at this
> moment there is any replacement for their graphical interface to solvers
> for nonlinear differential/difference equations called Simulink. 

This is correct. This is the one thing that Python needs improvements 
in. Many of my colleagues use simulink to design and model a digital 
signal processing element which they can then download to an FPGA using 
a third-party tool from XiLINX. There is no way they could replace that 
with Python/SciPy at this point (although of course it "could" be done).

I would love to see some good contributions in the area of Simulink-like 
work. There are several things out there that are good starts.

> But what makes Matlab difficult to be replaced is that lots of other
> projects (commercial: Mathematica, Maple, ... and free: octave, maxima,
> scipy, ...) only offer computation and visualization, while engineers in my
> field also need INTERACTION OF THE SYSTEM WITH EXTERNAL WORLD. That is,
> compatibility with a real-time operating system and MOST available
> input-output (AD-DA) cards.
The only way to solve this is to get more users interested in making 
these kinds of things happen. Or to start a company that does this and 
charges for a special build of Python compatible with your favorite 
hardware.

>  Being able to acquire measurement data from an
> external industrial system, process them computationally (for instance,
> solving some Riccati matrix differential equations), visualize the data and
> put the computed results back to the real system, this is what we need.
>   
This is doable in many respects already (look at what Andrew Straw has 
done for example), but it of course could be made "easier" to do. But, 
I'm not sure it will happen without the work of a company. It would be 
great if hardware manufacturers used Python and made sure their stuff 
worked right with it, but this requires many more users.
> I am absolutely sure that Python (and Scipy and Numpy) project has a
> potential to fulfill also these needs (and by starting using these and
> sharing my own code I would like to contribute), but it is not the case at
> the moment. Without being negative and discouraging, I think it is fair to
> say that currently for some people it would be very difficult to switch
> completely to Python libraries. 
>   
Yes, this is fair to say. As you have indicated, the situation is 
improving.


-Travis



From hurak at fel.cvut.cz  Thu Apr 26 06:59:53 2007
From: hurak at fel.cvut.cz (=?UTF-8?B?WmRlbsSbayBIdXLDoWs=?=)
Date: Thu, 26 Apr 2007 12:59:53 +0200
Subject: [Numpy-discussion] matlab vs. python question
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>	<1e2af89e0704260244j186ea867pd42c06689d4c6f33@mail.gmail.com>
	<f0ptk0$5f3$1@sea.gmane.org> <463080CB.7090000@ieee.org>
Message-ID: <f0q0n9$fof$1@sea.gmane.org>

Travis Oliphant wrote:

[...]
> I would love to see some good contributions in the area of Simulink-like
> work. There are several things out there that are good starts.

Even though I praised Simulink highly in the previous contribution, I don't
think that it would be a good way to mimic it. That way (the way that, for
instance, Scilab/Scicos is going) you will only be SECOND.

What I like (still as a newcomer) about Scipy/Numpy is that it does not try
to be Matlab for poor. You (I mean the community) can take an inspiration
BY THE USE, but please, don't look too much on Matlab. Let's create
somethink better even at the cost of not being compatible and perhaps
requiring some learing effort from Matlab users. I keep my fingers crossed
for Octave project, which is kind of free (GNU GPL) Matlab clone, but at
the same moment I can recognize their limits in being JUST A CLONE. The
leaders of Octave project do realize this danger but user kind of dictate
compatibility with Matlab.

To be finally more specific, there is some discussion going on in the
systems&control community as to "block-diagram vs. equation-based
modelling". I am not a guru in dynamic systems modeling but in my humble
opinion it is worth considering something like Modelica language
http://www.modelica.org/ as the ground for modeling in Python. There is a
young free implementation called OpenModelica
http://www.ida.liu.se/~pelab/modelica/OpenModelica.html. A commercial
implementation is produced by Dynasim and is called Dymola
http://www.dynasim.com/ and can at least give an inspiration, showing that
from a user point of view, it is also just graphical blocks that are being
connected to build a model.

To make it clear, I am not a proponent of the above mentioned tools, I just
came across these a couple of days ago when searching for an alternative to
Simulink (commercial of free) and I found the whole Modelica movement
interesting and perhaps the only one with significant development effort
investment.

>> But what makes Matlab difficult to be replaced is that lots of other
>> projects (commercial: Mathematica, Maple, ... and free: octave, maxima,
>> scipy, ...) only offer computation and visualization, while engineers in
>> my field also need INTERACTION OF THE SYSTEM WITH EXTERNAL WORLD. That
>> is, compatibility with a real-time operating system and MOST available
>> input-output (AD-DA) cards.
> The only way to solve this is to get more users interested in making
> these kinds of things happen. Or to start a company that does this and
> charges for a special build of Python compatible with your favorite
> hardware.

I think that it would not be necessary to start from the scratch. There are
some free projects for interfacing PC to these cards like COMEDI
http://www.comedi.org/. There are also projects with real-time adaptations
of commong operating systems, for Linux, I know of two: RTAI
https://www.rtai.org/ and RTLinux-GPL  http://www.rtlinux-gpl.org/. I am 
not an expert in these domains but it appears that there should be no
problems to interface to these tools. The only think is the investment of
developers effort, of course...:-)

Best regards,

Zdenek



From faltet at carabos.com  Thu Apr 26 08:19:16 2007
From: faltet at carabos.com (Francesc Altet)
Date: Thu, 26 Apr 2007 14:19:16 +0200
Subject: [Numpy-discussion] numexpr efficency depends on the size of the
	computing kernel
In-Reply-To: <1173906351.2543.71.camel@localhost.localdomain>
References: <1173906351.2543.71.camel@localhost.localdomain>
Message-ID: <200704261419.18130.faltet@carabos.com>

Just a quick followup about this issue:

After a bit of investigation, I discovered that the responsible for the 
difference in performance between the original numexpr and their PyTables 
counterpart (see the message below) was due *only* to the different flags 
used for compiling (and not to a cache instruction overload in the CPU).

It turns out that the original numexpr always add the '-O2 -funroll-all-loops' 
flag (GCC compiler), while I compiled the PyTables instance with the python 
default (-O3).  After recompiling the latter using the same flags as original 
numexpr, then I get exactly the same results on either version of numexpr , 
even with a processor with as a small secondary cache as 64 KB (AMD Duron)
(i.e. the '-funroll-all-loops' flag seems to be *very* effective for 
optimizing the computing kernel of numexpr, at least with CPUs with small 
caches).

So, at least, this leads to the conclusion that the numexpr's virtual machine 
is still far away from getting overloaded, most specially with nowadays 
processors with 512KB of secondary cache or more.

Cheers,

A Dimecres 14 Mar? 2007 22:05, Francesc Altet escrigu?:
> Hi,
>
> Now that I'm commanding my old AMD Duron machine, I've made some
> benchmarks just to prove that the numexpr computing is not influenced by
> the size of the CPU cache, but I failed miserably (and Tim was right:
> there is a dependency of the numexpr efficency on CPU cache size).
>
> Provided that the pytables instance of the computing kernel of numexpr
> is quite larger (it supports more datatypes) than the original,
> comparing the performance of both versions can be a good way to check
> the influence of CPU cache on the computing efficency.
>
> The attached benchmark is a small modification of the timing.py that
> comes with the numexpr package (the modification was needed to allow the
> numexpr version of pytables to run all the cases). Basically, the
> expressions tested operations with arrays of 1 million of elements, with
> a mix of contiguous and strided arrays (no unaligned arrays are present
> here). See the code in benchmark for the details.
>
> The speed-ups of numexpr over plain numpy on a AMD Duron machine (64 +
> 64 KB L1 cache, 64 KB L2 cache) are:
>
> For the original numexpr package:
>
> 2.14, 2.21, 2.21  (these represent averages for 3 complete runs)
>
> For the modified pytables version (enlarged computing kernel):
>
> 1.32, 1.34, 1.37
>
> So, with a CPU with a very small cache, the original numexpr kernel is
> 1.6x faster than the pytables one.
>
> However, using a AMD Opteron which has a much bigger L2 cache (64 + 64
> KB L1 cache, 1 MB L2 cache), the speed-ups are quite similar:
>
> For the original numexpr package:
>
> 3.10, 3.35, 3.35
>
> For the modified pytables version (enlarged computing kernel):
>
> 3.37, 3.50, 3.45
>
> So, there is effectively a dependency on the CPU cache size. It would be
> nice to run the benchmark with other CPUs with a L2 cache in the range
> between 64 KB and 1 MB so as to find the point where the performance
> starts to be similar (this should be a good guess on the size of the
> computing kernel).
>
> Meanwhile, the lesson learned is that Tim worries were correct: one
> should be very careful on adding more opcodes (at least, until CPUs with
> a very small L2 cache are in use).  With this, perhaps we will have to
> reduce the opcodes in the numexpr version for pytables to a bare
> minimum :-/
>
> Cheers,

-- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"


From lists.steve at arachnedesign.net  Thu Apr 26 08:19:50 2007
From: lists.steve at arachnedesign.net (Steve Lianoglou)
Date: Thu, 26 Apr 2007 08:19:50 -0400
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>
Message-ID: <2B9BE3C1-EFCB-4BA3-9FB7-E38AA7B865BC@arachnedesign.net>

> Beside proper programing paradigm Python easily scales to large- 
> scale number crunching: You can run large-matrices calculations  
> with about 1/2 to 1/4 of memory consumption comparing to Matlab.

Is that really true? (The large-matrix number crunching, not the  
proper programming paradigm ;-)

By no scientific means of evaluation, I was under the impression that  
the opposite was true to a smaller degree.

Also, a lot of the times that I'm dealing with extremely large  
matrices, they're of the sparse variety, which matlab currently  
handles with a bit more ease (and transparency) to the user.

-steve




From sturla at molden.no  Thu Apr 26 08:35:02 2007
From: sturla at molden.no (Sturla Molden)
Date: Thu, 26 Apr 2007 14:35:02 +0200
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <2B9BE3C1-EFCB-4BA3-9FB7-E38AA7B865BC@arachnedesign.net>
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>
	<2B9BE3C1-EFCB-4BA3-9FB7-E38AA7B865BC@arachnedesign.net>
Message-ID: <46309C76.902@molden.no>

On 4/26/2007 2:19 PM, Steve Lianoglou wrote:

>> Beside proper programing paradigm Python easily scales to large- 
>> scale number crunching: You can run large-matrices calculations  
>> with about 1/2 to 1/4 of memory consumption comparing to Matlab.
> 
> Is that really true? (The large-matrix number crunching, not the  
> proper programming paradigm ;-)
> 
> By no scientific means of evaluation, I was under the impression that  
> the opposite was true to a smaller degree.

Matlab have pass-by-value semantics, so you have to copy your data in 
and copy your data out for every function call. You can achieve the same 
result in Python by pickling and unpickling arguments and return values, 
e.g. using this function decorator:


import cPickle as pickle

def Matlab_Semantics(f):

    '''
        Emulates Matlab's pass-by-value semantics,
        objects are serialized in and serialized out.

        Example:  @Matlab_Semantics
                  def foo(bar):
                      pass
    '''

    func = f
    return wrapper

    def wrapper(*args,**kwargs):
       args_in = pickle.loads(pickle.dumps(args))
       kwargs_in = {}
       for k in kwargs:
          kwargs_in[k] = pickle.loads(pickle.dumps(kwargs[k]))
       args_out = func(*args_in,**kwargs_in)
       args_out = pickle.loads(pickle.dumps(args_out))
       return args_out

Imagine using this horrible semantics in several layers of function 
calls. That is exactly what Matlab does. Granted, Matlab optimizes 
function calls by using copy-on-write, so it will be efficient in some 
cases, excessive cycles og copy-in and copy-out is usually what you get.


Sturla Molden


From hurak at fel.cvut.cz  Thu Apr 26 08:37:58 2007
From: hurak at fel.cvut.cz (=?UTF-8?B?WmRlbsSbayBIdXLDoWs=?=)
Date: Thu, 26 Apr 2007 14:37:58 +0200
Subject: [Numpy-discussion] matlab vs. python question
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>
	<2B9BE3C1-EFCB-4BA3-9FB7-E38AA7B865BC@arachnedesign.net>
Message-ID: <f0q6f6$957$1@sea.gmane.org>

Steve Lianoglou wrote:

>> Beside proper programing paradigm Python easily scales to large-
>> scale number crunching: You can run large-matrices calculations
>> with about 1/2 to 1/4 of memory consumption comparing to Matlab.
> 
> Is that really true? (The large-matrix number crunching, not the
> proper programming paradigm ;-)
> 
> By no scientific means of evaluation, I was under the impression that
> the opposite was true to a smaller degree.
> 
> Also, a lot of the times that I'm dealing with extremely large
> matrices, they're of the sparse variety, which matlab currently
> handles with a bit more ease (and transparency) to the user.
> 
> -steve

As Matlab is linking to Lapack, Atlas and similar libraries for (at least a
part of) linear algebra computations, if you link to these freely available
libraries from whatever other language, you should get about the same
results.

Zdenek



From david at ar.media.kyoto-u.ac.jp  Thu Apr 26 08:42:07 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 26 Apr 2007 21:42:07 +0900
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <46309C76.902@molden.no>
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>	<2B9BE3C1-EFCB-4BA3-9FB7-E38AA7B865BC@arachnedesign.net>
	<46309C76.902@molden.no>
Message-ID: <46309E1F.9000806@ar.media.kyoto-u.ac.jp>

Sturla Molden wrote:
> On 4/26/2007 2:19 PM, Steve Lianoglou wrote:
>
>>> Beside proper programing paradigm Python easily scales to large- 
>>> scale number crunching: You can run large-matrices calculations  
>>> with about 1/2 to 1/4 of memory consumption comparing to Matlab.
>> Is that really true? (The large-matrix number crunching, not the  
>> proper programming paradigm ;-)
>>
>> By no scientific means of evaluation, I was under the impression that  
>> the opposite was true to a smaller degree.
>
> Matlab have pass-by-value semantics, so you have to copy your data in 
> and copy your data out for every function call. 
You are true for the semantics, but wrong for the consequences on 
copying, as matlab is using COW, and this works well in matlab. I have 
never noticed a big difference between matlab and python + numpy for 
memory consumption; I fail to see any reason why it would be 
significantly different (except the fact that numpy does not have to use 
double, whereas matlab had to for a long time, and double is still the 
default on matlab).

David


From ndbecker2 at gmail.com  Thu Apr 26 08:47:43 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 26 Apr 2007 08:47:43 -0400
Subject: [Numpy-discussion] matlab vs. python question
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>
	<2B9BE3C1-EFCB-4BA3-9FB7-E38AA7B865BC@arachnedesign.net>
	<46309C76.902@molden.no>
Message-ID: <f0q71g$c0r$1@sea.gmane.org>

Sturla Molden wrote:

> On 4/26/2007 2:19 PM, Steve Lianoglou wrote:
> 
>>> Beside proper programing paradigm Python easily scales to large-
>>> scale number crunching: You can run large-matrices calculations
>>> with about 1/2 to 1/4 of memory consumption comparing to Matlab.
>> 
>> Is that really true? (The large-matrix number crunching, not the
>> proper programming paradigm ;-)
>> 
>> By no scientific means of evaluation, I was under the impression that
>> the opposite was true to a smaller degree.
> 
> Matlab have pass-by-value semantics, so you have to copy your data in
> and copy your data out for every function call. You can achieve the same
> result in Python by pickling and unpickling arguments and return values,
> e.g. using this function decorator:
> 
> 
> import cPickle as pickle
> 
> def Matlab_Semantics(f):
> 
>     '''
>         Emulates Matlab's pass-by-value semantics,
>         objects are serialized in and serialized out.
> 
>         Example:  @Matlab_Semantics
>                   def foo(bar):
>                       pass
>     '''
> 
>     func = f
>     return wrapper
> 
>     def wrapper(*args,**kwargs):
>        args_in = pickle.loads(pickle.dumps(args))
>        kwargs_in = {}
>        for k in kwargs:
>           kwargs_in[k] = pickle.loads(pickle.dumps(kwargs[k]))
>        args_out = func(*args_in,**kwargs_in)
>        args_out = pickle.loads(pickle.dumps(args_out))
>        return args_out
> 
> Imagine using this horrible semantics in several layers of function
> calls. That is exactly what Matlab does. Granted, Matlab optimizes
> function calls by using copy-on-write, so it will be efficient in some
> cases, excessive cycles og copy-in and copy-out is usually what you get.
> 
> 
That's interesting.  How did you find this information?



From sturla at molden.no  Thu Apr 26 09:01:06 2007
From: sturla at molden.no (Sturla Molden)
Date: Thu, 26 Apr 2007 15:01:06 +0200
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <f0q71g$c0r$1@sea.gmane.org>
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>	<2B9BE3C1-EFCB-4BA3-9FB7-E38AA7B865BC@arachnedesign.net>	<46309C76.902@molden.no>
	<f0q71g$c0r$1@sea.gmane.org>
Message-ID: <4630A292.1070908@molden.no>

On 4/26/2007 2:47 PM, Neal Becker wrote:

> That's interesting.  How did you find this information?

What information?

Matlab's pass-by-value semantics is well known to anyone who has ever 
used Matlab.

The Mathwork's employees have numerous times stated that Matlab uses 
copy-on-write to optimize function calls. I first learned of it in the 
usenet group comp.soft-sys.matlab.


Sturla Molden






From sturla at molden.no  Thu Apr 26 09:04:42 2007
From: sturla at molden.no (Sturla Molden)
Date: Thu, 26 Apr 2007 15:04:42 +0200
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <46309E1F.9000806@ar.media.kyoto-u.ac.jp>
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>	<2B9BE3C1-EFCB-4BA3-9FB7-E38AA7B865BC@arachnedesign.net>	<46309C76.902@molden.no>
	<46309E1F.9000806@ar.media.kyoto-u.ac.jp>
Message-ID: <4630A36A.80603@molden.no>

On 4/26/2007 2:42 PM, David Cournapeau wrote:

 > You are true for the semantics, but wrong for the consequences on 
copying, as matlab is using COW, and this works well in matlab.

It works well only if you don't change your input arguments. Never try 
to write to a matrix received as an argument in a function call. If you 
do, the memory expenditure may grow very rapidly.

But as long a s NumPy does not use lazy evaluation the difference is not 
as striking as it could be.


S.M.


From gael.varoquaux at normalesup.org  Thu Apr 26 09:16:33 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 26 Apr 2007 15:16:33 +0200
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <f0ptk0$5f3$1@sea.gmane.org>
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>
	<1e2af89e0704260244j186ea867pd42c06689d4c6f33@mail.gmail.com>
	<f0ptk0$5f3$1@sea.gmane.org>
Message-ID: <20070426131626.GD24024@clipper.ens.fr>

On Thu, Apr 26, 2007 at 12:06:56PM +0200, Zden?k Hur?k wrote:
> But what makes Matlab difficult to be replaced is that lots of other
> projects (commercial: Mathematica, Maple, ... and free: octave, maxima,
> scipy, ...) only offer computation and visualization, while engineers in my
> field also need INTERACTION OF THE SYSTEM WITH EXTERNAL WORLD. That is,
> compatibility with a real-time operating system and MOST available
> input-output (AD-DA) cards. Being able to acquire measurement data from an
> external industrial system, process them computationally (for instance,
> solving some Riccati matrix differential equations), visualize the data and
> put the computed results back to the real system, this is what we need.

I am very surprised that you think Matlab is more suited for such a task.
I have had the opposite experience. 

I work in an experimental lab where we do rather heavy experimental
physics (Bose Einstein condensation). I am building an experiment from
scratch and had to build a control framework for the experiment. It is
fully automated and with need some reasonably sophisticated logics to
scan parameters, do on the fly processing of the data, store it on the
disk, display all this in a useful interface for the user, and feed all
this back to the experiment.

Due to legacy reasons I built the software in Matlab. It was a bad
experience. 

First of all linking to C libraries to control hardware is really a pain.
Writing Mex files is not that much fun and the memory management of
MatLab is flawed. I have had segfaults for no obvious reasons, and the
problems where neither in my code nor in the drivers (I worked with the
engineer who wrote the driver to diagnose this). Matlab is single
threaded so all your calls to the hardware are blocking (unless you are
willing to add threads in you mex file, but this is very dangerous as you
are playing with a loop-hole in matlab's mex loader which keeps the mex
in the memory after it is executed). Finally the lack of proper object
oriented programming and separated namespaces make it hard to write good
code to control instruments.

The lack of object oriented programming, passing by reference, threads,
and proper control of the GUI event-loop also makes it very hard to write
a complex interactive GUI.

Python provides all this. What it does not provide are pre-linked
libraries to control hardware, but if you are trying to control exotic
hardware, all you have is a C or C++ SDK. It is even worse when the
hardware is homemade, as you have to write the library yourself, and
trust me, I'd much better write a hardware control library that speak to
my custom made electronic over a bus in Python than in Matlab (GPIB,
ethernet, serial, Python has bindings for all that).

I have had a second choice to build the computer control framework of a
similar experiment. I took my chance to build it in Python (it was a
different lab, and I was replacing an existing C software that had become
impossible to maintain). I never regretted it. Python has really been
very good at it due to all the different libraries available, the ease to
link to C, and the possibility to do proper OOP.

I have written an article about this, that I submitted a month ago to CiSE.
I have no news from them. The article is very poorly written (I had no
hindsight about these matters when I started writing it, submitted it
because I was tired to see it dragging along, and now I wish I had
reworked it a bit), but I think I raise some points about what you need
to build a control framework for an experiment, and how python can
address those needs.

Anyway, IANAL, and I am not to sure if releasing a preprint on a mailing
list renders the article ineligible for CiSE or not but I just put a
version on
http://gael-varoquaux.info/computers/agile_computer_control_of_an_experiment.pdf
. I hope it can help understanding what is needed to control an
experiment from Python, what can be improved, and what already exists and
is so incredibly useful.

I am interested in comments. Keep in mind that I am working in a field
where nobody is interested in computing and I sure could use some advice.
I have written quite a few softwares to control experiments in different
labs with different languages, and I think I have acquired some
experience, and made a lot of progress, but I also have had to reinvent
the wheel more than once. We all know that good software engineering
books or article readable by some one who nows little about computing are
hard to find. Let us address this issue !

Ga?l


From fperez.net at gmail.com  Thu Apr 26 10:10:14 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 26 Apr 2007 08:10:14 -0600
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <20070426131626.GD24024@clipper.ens.fr>
References: <07C6A61102C94148B8104D42DE95F7E8C8F24D@exchange2k.envision.co.il>
	<1e2af89e0704260244j186ea867pd42c06689d4c6f33@mail.gmail.com>
	<f0ptk0$5f3$1@sea.gmane.org> <20070426131626.GD24024@clipper.ens.fr>
Message-ID: <db6b5ecc0704260710u7a9f730eg24630d71f5f597b5@mail.gmail.com>

On 4/26/07, Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:

> Anyway, IANAL, and I am not to sure if releasing a preprint on a mailing
> list renders the article ineligible for CiSE or not but I just put a
> version on
> http://gael-varoquaux.info/computers/agile_computer_control_of_an_experiment.pdf

You're safe: there was a thread yesterday on the MPL list precisely on
this, the CiSE policy (IEEE/AIP in reality) allows you to do this, as
long as you replace it with their official PDF once/if it gets
published and you clearly indicate their copyright terms.  But hosting
a preprint on your own site is allowed by their policies.

I can hunt down the specific references if you need them.

Cheers,

f


From ivilata at carabos.com  Thu Apr 26 12:06:17 2007
From: ivilata at carabos.com (Ivan Vilata i Balaguer)
Date: Thu, 26 Apr 2007 18:06:17 +0200
Subject: [Numpy-discussion] [ANN] PyTables 2.0rc1 released
Message-ID: <20070426160617.GD7924@tardis.terramar.selidor.net>

Hi all,

The development of the second major release of PyTables is steadily
approaching its goal!  Today we want to announce the *first release
candidate version of PyTables 2.0*, i.e. PyTables 2.0rc1.  This version
settles the API and file format for the following PyTables 2.0 series.
No more features will be added until the final version of 2.0 is
released, so we do enter a time for exhaustive platform testing and
fixing the last remaining bugs, as well as updating any outdated
documentation.

Your collaboration is very important in this stage of development, so we
encourage you to download PyTables, test it, and report any problems you
find or any suggestions you have.  Thank you!

And now, the official announcement:


============================
 Announcing PyTables 2.0rc1
============================

PyTables is a library for managing hierarchical datasets and designed to
efficiently cope with extremely large amounts of data with support for
full 64-bit file addressing.  PyTables runs on top of the HDF5 library
and NumPy package for achieving maximum throughput and convenient use.

You can download a source package of the version 2.0rc1 with
generated PDF and HTML docs and binaries for Windows from
http://www.pytables.org/download/preliminary/

For an on-line version of the manual, visit:
http://www.pytables.org/docs/manual-2.0rc1
Please have in mind that some sections in the manual can be obsolete
(specially the "Optimization tips" chapter).  Other chapters should be
fairly up-to-date though (although still a bit in state of flux).

In case you want to know more in detail what has changed in this
version, have a look at ``RELEASE_NOTES.txt``.  Find the HTML version
for this document at:
http://www.pytables.org/moin/ReleaseNotes/Release_2.0rc1

If you are a user of PyTables 1.x, probably it is worth for you to look
at ``MIGRATING_TO_2.x.txt`` file where you will find directions on how
to migrate your existing PyTables 1.x apps to the 2.0 version.  You can
find an HTML version of this document at
http://www.pytables.org/moin/ReleaseNotes/Migrating_To_2.x

Keep reading for an overview of the most prominent improvements in
PyTables 2.0 series.


New features of PyTables 2.0
============================

- NumPy is finally at the core!  That means that PyTables no longer
  needs numarray in order to operate, although it continues to be
  supported (as well as Numeric).  This also means that you should be
  able to run PyTables in scenarios combining Python 2.5 and 64-bit
  platforms (these are a source of problems with numarray/Numeric
  because they don't support this combination as of this writing).

- Most of the operations in PyTables have experimented noticeable
  speed-ups (sometimes up to 2x, like in regular Python table
  selections).  This is a consequence of both using NumPy internally and
  a considerable effort in terms of refactorization and optimization of
  the new code.

- Combined conditions are finally supported for in-kernel selections.
  So, now it is possible to perform complex selections like::

      result = [ row['var3'] for row in
                 table.where('(var2 < 20) | (var1 == "sas")') ]

  or::

      complex_cond = '((%s <= col5) & (col2 <= %s)) ' \
                     '| (sqrt(col1 + 3.1*col2 + col3*col4) > 3)'
      result = [ row['var3'] for row in
                 table.where(complex_cond % (inf, sup)) ]

  and run them at full C-speed (or perhaps more, due to the cache-tuned
  computing kernel of Numexpr, which has been integrated into PyTables).

- Now, it is possible to get fields of the ``Row`` iterator by
  specifying their position, or even ranges of positions (extended
  slicing is supported).  For example, you can do::

      result = [ row[4] for row in table    # fetch field #4
                 if row[1] < 20 ]
      result = [ row[:] for row in table    # fetch all fields
                 if row['var2'] < 20 ]
      result = [ row[1::2] for row in       # fetch odd fields
                 table.iterrows(2, 3000, 3) ]

  in addition to the classical::

      result = [row['var3'] for row in table.where('var2 < 20')]

- ``Row`` has received a new method called ``fetch_all_fields()`` in
  order to easily retrieve all the fields of a row in situations like::

      [row.fetch_all_fields() for row in table.where('column1 < 0.3')]

  The difference between ``row[:]`` and ``row.fetch_all_fields()`` is
  that the former will return all the fields as a tuple, while the
  latter will return the fields in a NumPy void type and should be
  faster.  Choose whatever fits better to your needs.

- Now, all data that is read from disk is converted, if necessary, to
  the native byteorder of the hosting machine (before, this only
  happened with ``Table`` objects).  This should help to accelerate
  applications that have to do computations with data generated in
  platforms with a byteorder different than the user machine.

- The modification of values in ``*Array`` objects (through __setitem__)
  now doesn't make a copy of the value in the case that the shape of the
  value passed is the same as the slice to be overwritten. This results
  in considerable memory savings when you are modifying disk objects
  with big array values.

- All leaf constructors (except for ``Array``) have received a new
  ``chunkshape`` argument that lets the user explicitly select the
  chunksizes for the underlying HDF5 datasets (only for advanced users).

- All leaf constructors have received a new parameter called
  ``byteorder`` that lets the user specify the byteorder of their data
  *on disk*.  This effectively allows to create datasets in other
  byteorders than the native platform.

- Native HDF5 datasets with ``H5T_ARRAY`` datatypes are fully supported
  for reading now.

- The test suites for the different packages are installed now, so you
  don't need a copy of the PyTables sources to run the tests.  Besides,
  you can run the test suite from the Python console by using::

  >>> tables.tests()


Resources
=========

Go to the PyTables web site for more details:

http://www.pytables.org

About the HDF5 library:

http://hdf.ncsa.uiuc.edu/HDF5/

About NumPy:

http://numpy.scipy.org/

To know more about the company behind the development of PyTables, see:

http://www.carabos.com/


Acknowledgments
===============

Thanks to many users who provided feature improvements, patches, bug
reports, support and suggestions.  See the ``THANKS`` file in the
distribution package for a (incomplete) list of contributors.  Many
thanks also to SourceForge who have helped to make and distribute this
package!  And last, but not least thanks a lot to the HDF5 and NumPy
(and numarray!) makers. Without them PyTables simply would not exist.


Share your experience
=====================

Let us know of any bugs, suggestions, gripes, kudos, etc. you may
have.


----

  **Enjoy data!**

  -- The PyTables Team

::

	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
	       C?rabos Coop. V.  V  V   Enjoy Data
	                          ""
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 307 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070426/db8f41d3/attachment.sig>

From jdc at uwo.ca  Thu Apr 26 12:07:20 2007
From: jdc at uwo.ca (Dan Christensen)
Date: Thu, 26 Apr 2007 12:07:20 -0400
Subject: [Numpy-discussion] PEP 31XX: A Type Hierarchy for Numbers (and
	other algebraic entities)
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<f0pucb$86m$1@sea.gmane.org>
Message-ID: <873b2n5dc7.fsf@uwo.ca>

[sage list added]

"Travis E. Oliphant" <oliphant.travis at ieee.org> writes:

> The SAGE people may be interested in this, but I doubt there will more 
> than a handful of users of these algebraic base classes.

SAGE has quite a sophisticated type hierarchy, and a sophisticated set
of coercion methods.  What is done in SAGE should definitely be
consulted, since it is probably the most complex set of mathematical
types yet written in python.  The SAGE tutorial is at

  http://www.sagemath.org/doc/html/tut/tut.html

and Section 2.2 gives a brief introduction to numbers:

  http://www.sagemath.org/doc/html/tut/node9.html

The SAGE reference manual is at

  http://www.sagemath.org/doc/html/ref/index.html

Chapter 20:

  http://www.sagemath.org/doc/html/ref/node198.html

and nearby chapters are quite relevant for this discussion.

> For general purpose Python, I would do something like
>
> Complex
>    Rational_Complex
>      Integer_Complex
>    Floating_Complex  # uses hardware float
>    Decimal_Complex
>    Real
>      Rational
>        Integer
>      Floating        # uses hardware float
>      Decimal

Note also that double-precision reals are a subset of the rationals,
since each double precision real is exactly representable as a
rational number, but many rational numbers are not exactly
representable as double precision reals.  Not sure if this means
that reals should be a subclass of the rationals.

I believe that in SAGE these relationships aren't expressed using
subclassing.

Dan



From Chris.Barker at noaa.gov  Thu Apr 26 12:51:57 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 26 Apr 2007 09:51:57 -0700
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <462FF23E.7060901@bryant.edu>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no> <462FAFDA.80507@noaa.gov>
	<462FF23E.7060901@bryant.edu>
Message-ID: <4630D8AD.2080703@noaa.gov>

Brian Blais wrote:
> I agree with Sturla here, up until Pyrex.  Looking at the Python API, swig, f2py,
> etc. I found mex files far more straightforward.  Mostly this is because of the
> (mostly) single datatype in Matlab: the double matrix.

well, that's not really fair. I've written extensions to Numeric (but 
the same would be true for numpy) that only work on 2-d double arrays -- 
that was easy. Generalizing to other types and shapes is a pain, but you 
don't even have that option with Matlab.

> Having to do reference counting in the Python API is a drag.

Yes, it is. But every tool, other than writing to the C API directly, 
takes care of that for you.

> Now, with Pyrex, that's a different matter entirely.  Pyrex makes extension writing
> so incredibly easy,

Yes, very cool. I haven't used it for anything real, but f2py looks very 
easy too, if you're working with Fortran.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From jyasskin at gmail.com  Wed Apr 25 04:36:06 2007
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Wed, 25 Apr 2007 01:36:06 -0700
Subject: [Numpy-discussion] PEP 31XX: A Type Hierarchy for Numbers (and
	other algebraic entities)
Message-ID: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>

Here's a draft of the numbers ABCs PEP. The most up to date version
will live in the darcs repository at
http://jeffrey.yasskin.info/darcs/PEPs/pep-3141.txt (unless the number
changes) for now. Naming a PEP about numbers 3.141 seems cute, but of
course, I don't get to pick the number. :) This is my first PEP, so
apologies for any obvious mistakes.

I'd particularly like the numpy people's input on whether I've gotten
floating-point support right.

Thanks,
Jeffrey Yasskin

-------------------------------
PEP: 3141
Title: A Type Hierarchy for Numbers (and other algebraic entities)
Version: $Revision: 54928 $
Last-Modified: $Date: 2007-04-23 16:37:29 -0700 (Mon, 23 Apr 2007) $
Author: Jeffrey Yasskin <jyasskin at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 23-Apr-2007
Post-History: Not yet posted


Abstract
========

This proposal defines a hierarchy of Abstract Base Classes (ABCs)
[#pep3119] to represent numbers and other algebraic entities similar
to numbers.  It proposes:

* A hierarchy of algebraic concepts, including monoids, groups, rings,
  and fields with successively more operators and constraints on their
  operators. This will be added as a new library module named
  "algebra".

* A hierarchy of specifically numeric types, which can be converted to
  and from the native Python types. This will be added as a new
  library module named "numbers".

Rationale
=========

Functions that take numbers as arguments should be able to determine
the properties of those numbers, and if and when overloading based on
types is added to the language, should be overloadable based on the
types of the arguments. This PEP defines some abstract base classes
that are useful in numerical calculations. A function can check that
variable is an instance of one of these classes and then rely on the
properties specified for them. Of course, the language cannot check
these properties, so where I say something is "guaranteed", I really
just mean that it's one of those properties a user should be able to
rely on.

This PEP tries to find a balance between providing fine-grained
distinctions and specifying types that few people will ever use.

Specification
=============

Although this PEP uses terminology from PEP3119, the hierarchy is
meaningful for any systematic method of defining sets of
classes. **Todo:** link to the Interfaces PEP when it's ready. I'm
also using the extra notation from [#pep3107] (annotations) to specify
some types.

Object oriented systems have a general problem in constraining
functions that take two arguments. To take addition as an example,
``int(3) + int(4)`` is defined, and ``vector(1,2,3) + vector(3,4,5)``
is defined, but ``int(3) + vector(3,4,5)`` doesn't make much sense. So
``a + b`` is not guaranteed to be defined for any two instances of
``AdditiveGroup``, but it is guaranteed to be defined when ``type(a)
== type(b)``. On the other hand, ``+`` does make sense for any sorts
of numbers, so the ``Complex`` ABC refines the properties for plus so
that ``a + b`` is defined whenever ``isinstance(a,Complex) and
isinstance(b,Complex)``, even if ``type(a) != type(b)``.


Monoids (http://en.wikipedia.org/wiki/Monoid) consist of a set with an
associative operation, and an identity element under that
operation. **Open issue**: Is a @classmethod the best way to define
constants that depend only on the type?::

    class MonoidUnderPlus(Abstract):
	"""+ is associative but not necessarily commutative and has an
	identity given by plus_identity().

	Subclasses follow the laws:

	  a + (b + c) === (a + b) + c
	  a.plus_identity() + a === a === a + a.plus_identity()

	Sequences are monoids under plus (in Python) but are not
	AdditiveGroups.
	"""
	@abstractmethod
	def __add__(self, other):
	    raise NotImplementedError

	@classmethod
	@abstractmethod
	def plus_identity(cls):
	    raise NotImplementedError

I skip ordinary non-commutative groups here because I don't have any
common examples of groups that use ``+`` as their operator but aren't
commutative. If we find some, the class can be added later.::

    class AdditiveGroup(MonoidUnderPlus):
	"""Defines a commutative group whose operator is +, and whose inverses
	are produced by -x.
	See http://en.wikipedia.org/wiki/Abelian_group.

	Where a, b, and c are instances of the same subclass of
	AdditiveGroup, the operations should follow these laws, where
	'zero' is a.__class__.zero().

	      a + b === b + a
	(a + b) + c === a + (b + c)
	   zero + a === a
	   a + (-a) === zero
	      a - b === a + -b

	Some abstract subclasses, such as Complex, may extend the
	definition of + to heterogenous subclasses, but AdditiveGroup only
	guarantees it's defined on arguments of exactly the same types.

	Vectors are AdditiveGroups but are not Rings.
	"""
	@abstractmethod
	def __add__(self, other):
	    """Associative commutative operation, whose inverse is negation."""
	    raise NotImplementedError

**Open issue:** Do we want to give people a choice of which of the
following to define, or should we pick one arbitrarily?::

	def __neg__(self):
	    """Must define this or __sub__()."""
	    return self.zero() - self

	def __sub__(self, other):
	    """Must define this or __neg__()."""
	    return self + -other

	@classmethod
	@abstractmethod
	def zero(cls):
	    """A better name for +'s identity as we move into more mathematical
	    domains."""
	    raise NotImplementedError

	@classmethod
	def plus_identity(cls):
	    return cls.zero()

Including Semiring (http://en.wikipedia.org/wiki/Semiring) would help
a little with defining a type for the natural numbers. That can be
split out once someone needs it (see ``IntegralDomain`` for how).::

    class Ring(AdditiveGroup):
	"""A mathematical ring over the operations + and *.
	See http://en.wikipedia.org/wiki/Ring_%28mathematics%29.

	In addition to the requirements of the AdditiveGroup superclass, a
	Ring has an associative but not necessarily commutative
	multiplication operation with identity (one) that distributes over
	addition. A Ring can be constructed from any integer 'i' by adding
	'one' to itself 'i' times. When R is a subclass of Ring, the
	additive identity is R(0), and the multiplicative identity is
	R(1).

	Matrices are Rings but not Commutative Rings or Division
	Rings. The quaternions are a Division Ring but not a
	Field. The integers are a Commutative Ring but not a Field.
	"""
	@abstractmethod
	def __init__(self, i:int):
	    """An instance of a Ring may be constructed from an integer.

	    This may be a lossy conversion, as in the case of the integers
	    modulo N."""
	    pass

	@abstractmethod
	def __mul__(self, other):
	    """Satisfies:
	    a * (b * c) === (a * b) * c
		one * a === a
		a * one === a
	    a * (b + c) === a * b + a * c

	    where one == a.__class__(1)
	    """
	    raise NotImplementedError

	@classmethod
	def zero(cls):
	    return cls(0)

	@classmethod
	def one(cls):
	    return cls(1)

I'm skipping both CommutativeRing and DivisionRing here.

    class Field(Ring):
	"""The class Field adds to Ring the requirement that * be a
	commutative group operation except that zero does not have an
	inverse.
	See http://en.wikipedia.org/wiki/Field_%28mathematics%29.

	Practically, that means we can define division on a Field. The
	additional laws are:

	a * b === b * a
	a / a === a.__class_(1)  # when a != a.__class__(0)

	Division lets us construct a Field from any Python float,
	although the conversion is likely to be lossy. Some Fields
	include the real numbers, rationals, and integers mod a
	prime. Python's ``float`` resembles a Field closely.
	"""
	def __init__(self, f:float):
	    """A Field should be constructible from any rational number, which
	    includes Python floats."""
	    pass

	@abstractmethod
	def __div__(self, divisor):
	    raise NotImplementedError

Division is somewhat complicated in Python. You have both __floordiv__
and __div__, and ints produce floats when they're divided. For the
purposes of this hierarchy, ``__floordiv__(a, b)`` is defined by
``floor(__div__(a, b))``, and, since int is not a subclass of Field,
it's allowed to do whatever it wants with __div__.

There are four more reasonable classes that I'm skipping here in the
interest of keeping the initial library simple. They are:

``Algebraic``
    Rational powers of its elements are defined (and maybe a few other
    operations)
    (http://en.wikipedia.org/wiki/Algebraic_number). Complex numbers
    are the most well-known algebraic set. Real numbers are _not_
    algebraic, but Python does define these operations on floats,
    which makes defining this class somewhat difficult.

``Trancendental``
    The elementary functions
    (http://en.wikipedia.org/wiki/Elementary_function) are
    defined. These are basically arbitrary powers, trig functions, and
    logs, the contents of ``cmath``.

The following two classes can be reasonably combined with ``Integral``
for now.
``IntegralDomain``
    Defines __divmod__.
    (http://darcs.haskell.org/numericprelude/docs/html/Algebra-IntegralDomain.html#t%3AC)

``PrincipalIdealDomain``
    Defines gcd and lcm.
    (http://darcs.haskell.org/numericprelude/docs/html/Algebra-PrincipalIdealDomain.html#t%3AC)

If someone needs to split them later, they can use code like::
    import numbers
    class IntegralDomain(Ring): ...
    numbers.Integral.__bases__ = (IntegralDomain,) + numbers.Integral.__bases__


Finally, we get to numbers. This is where we switch from the "algebra"
module to the "numbers" module.::

    class Complex(Ring, Hashable):
        """The ``Complex`` ABC indicates that the value lies somewhere
        on the complex plane, not that it in fact has a complex
        component: ``int`` is a subclass of ``Complex``. Because these
        actually represent complex numbers, they can be converted to
        the ``complex`` type.

        ``Complex`` finally gets around to requiring its subtypes to
        be immutable so they can be hashed in a standard way.

        ``Complex`` also requires its operations to accept
        heterogenous arguments. Subclasses should override the
        operators to be more accurate when they can, but should fall
        back on the default definitions to handle arguments of
        different (Complex) types.

        **Open issue:** __abs__ doesn't fit here because it doesn't
        exist for the Gaussian integers
        (http://en.wikipedia.org/wiki/Gaussian_integer). In fact, it
        only exists for algebraic complex numbers and real numbers. We
        could define it in both places, or leave it out of the
        ``Complex`` classes entirely and let it be a custom extention
        of the ``complex`` type.

        The Gaussian integers are ``Complex`` but not a ``Field``.
	"""
	@abstractmethod
	def __complex__(self):
	    """Any Complex can be converted to a native complex object."""
	    raise NotImplementedError

	def __hash__(self):
	    return hash(complex(self))

	@abstractmethod
	def real(self) => Real:
	    raise NotImplementedError

	@abstractmethod
	def imag(self) => Real:
	    raise NotImplementedError

	@abstractmethod
	def __add__(self, other):
	    """The other Ring operations should be implemented similarly."""
	    if isinstance(other, Complex):
		return complex(self) + complex(other)
	    else:
		return NotImplemented

``FractionalComplex(Complex, Field)`` might fit here, except that it
wouldn't give us any new operations.

    class Real(Complex, TotallyOrdered):
	"""Numbers along the real line. Some subclasses of this class
	may contain NaNs that are not ordered with the rest of the
	instances of that type. Oh well. **Open issue:** what problems
	will that cause? Is it worth it in order to get a
	straightforward type hierarchy?
	"""
	@abstractmethod
	def __float__(self):
	    raise NotImplementedError
	def __complex__(self):
	    return complex(float(self))
	def real(self) => self.__class__:
	    return self
	def imag(self) => self.__class__:
	    return self.__class__(0)
        def __abs__(self) => self.__class__:
            if self < 0: return -self
            else: return self


    class FractionalReal(Real, Field):
	"""Rationals and floats. This class provides concrete
	definitions of the other four methods from properfraction and
	allows you to convert fractional reals to integers in a
	disciplined way.
	"""
	@abstractmethod
	def properfraction(self) => (int, self.__class__):
	    """Returns a pair (n,f) such that self == n+f, and:
	      * n is an integral number with the same sign as self; and
	      * f is a fraction with the same type and sign as self, and with
		absolute value less than 1.
	      """
	    raise NotImplementedError
	def floor(self) => int:
            n, r = self.properfraction()
            if r < 0 then n - 1 else n
	def ceiling(self) => int: ...
	def __trunc__(self) => int: ...
	def round(self) => int: ...


**Open issue:** What's the best name for this class? RealIntegral? Integer?::

    class Integral(Real):
	"""Integers!"""
	@abstractmethod
	def __int__(self):
	    raise NotImplementedError
	def __float__(self):
	    return float(int(self))

	@abstractmethod
	def __or__(self, other):
	    raise NotImplementedError
	@abstractmethod
	def __xor__(self, other):
	    raise NotImplementedError
	@abstractmethod
	def __and__(self, other):
	    raise NotImplementedError
	@abstractmethod
	def __lshift__(self, other):
	    raise NotImplementedError
	@abstractmethod
	def __rshift__(self, other):
	    raise NotImplementedError
	@abstractmethod
	def __invert__(self):
	    raise NotImplementedError


Floating point values may not exactly obey several of the properties
you would expect from their superclasses. For example, it is possible
for ``(large_val + -large_val) + 3 == 3``, but ``large_val +
(-large_val + 3) == 0``. On the values most functions deal with this
isn't a problem, but it is something to be aware of. Types like this
inherit from ``FloatingReal`` so that functions that care can know to
use a numerically stable algorithm on them. **Open issue:** Is this
the proper way to handle floating types?::

    class FloatingReal:
	"""A "floating" number is one that is represented as
	``mantissa * radix**exponent`` where mantissa, radix, and
	exponent are all integers. Subclasses of FloatingReal don't
	follow all the rules you'd expect numbers to follow. If you
	really care about the answer, you have to use numerically
	stable algorithms, whatever those are.

        **Open issue:** What other operations would be useful here?

	These include floats and Decimals.
	"""
	@classmethod
	@abstractmethod
	def radix(cls) => int:
	    raise NotImplementedError

	@classmethod
	@abstractmethod
	def digits(cls) => int:
	    """The number of significant digits of base cls.radix()."""
	    raise NotImplementedError

	@classmethod
	@abstractmethod
	def exponentRange(cls) => (int, int):
	    """A pair of the (lowest,highest) values possible in the exponent."""
	    raise NotImplementedError

	@abstractmethod
	def decode(self) => (int, int):
	    """Returns a pair (mantissa, exponent) such that
	    mantissa*self.radix()**exponent == self."""
	    raise NotImplementedError



Inspiration
===========
http://hackage.haskell.org/trac/haskell-prime/wiki/StandardClasses
http://repetae.net/john/recent/out/classalias.html

References
==========

.. [#pep3119] Introducing Abstract Base Classes
   (http://www.python.org/dev/peps/pep-3119/)

.. [#pep3107] Function Annotations
   (http://www.python.org/dev/peps/pep-3107/)

.. [3] Possible Python 3K Class Tree?, wiki page created by Bill Janssen
   (http://wiki.python.org/moin/AbstractBaseClasses)

.. [#numericprelude] NumericPrelude: An experimental alternative
hierarchy of numeric type classes
   (http://darcs.haskell.org/numericprelude/docs/html/index.html)


Acknowledgements
----------------

Thanks to Neil Norwitz for helping me through the PEP process.

The Haskell Numeric Prelude [#numericprelude] nicely condensed a lot
of experience with the Haskell numeric hierarchy into a form that was
relatively easily adaptable to Python.

Copyright
=========

This document has been placed in the public domain.




..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:

From jimjjewett at gmail.com  Wed Apr 25 17:23:25 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 25 Apr 2007 17:23:25 -0400
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
Message-ID: <fb6fbf560704251423qaac984cv6d81821f660b8940@mail.gmail.com>

On 4/25/07, Jeffrey Yasskin <jyasskin at gmail.com> wrote:
>     class MonoidUnderPlus(Abstract):

Is this useful?  Just because two things are both Monoid instances
doesn't mean I can add them -- they have to be part of the same
Monoid.  By the time you do

    assert isinstance(a, MonoidUnderPlus)
    assert isinstance(b, MonoidUnderPlus)
    assert isinstance(a, b.__class__) or isinstance(b, a.__class__)

I'm not sure how much you've really saved.

> **Open issue:** Do we want to give people a choice of which of the
> following to define, or should we pick one arbitrarily?::

>         def __neg__(self):
>             """Must define this or __sub__()."""
>             return self.zero() - self

>         def __sub__(self, other):
>             """Must define this or __neg__()."""
>             return self + -other

Probably better to pick one;  then it is clear that they have to
override something, and there won't be as many accidental infinite
loops.  If they really want to define the other, they can just do

    def __neg__(self):
        super(_this_class__, self).__neg__()

The decorator doesn't know it isn't a real override.

-jJ


From jcarlson at uci.edu  Wed Apr 25 12:03:14 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Wed, 25 Apr 2007 09:03:14 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
Message-ID: <20070425085350.6408.JCARLSON@uci.edu>


"Jeffrey Yasskin" <jyasskin at gmail.com> wrote:
> Here's a draft of the numbers ABCs PEP. The most up to date version
> will live in the darcs repository at
> http://jeffrey.yasskin.info/darcs/PEPs/pep-3141.txt (unless the number
> changes) for now. Naming a PEP about numbers 3.141 seems cute, but of
> course, I don't get to pick the number. :) This is my first PEP, so
> apologies for any obvious mistakes.
> 
> I'd particularly like the numpy people's input on whether I've gotten
> floating-point support right.
> 
> Thanks,
> Jeffrey Yasskin
> 
> -------------------------------
> PEP: 3141
> Title: A Type Hierarchy for Numbers (and other algebraic entities)
> Version: $Revision: 54928 $
> Last-Modified: $Date: 2007-04-23 16:37:29 -0700 (Mon, 23 Apr 2007) $
> Author: Jeffrey Yasskin <jyasskin at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 23-Apr-2007
> Post-History: Not yet posted
> 
> 
> Abstract
> ========
> 
> This proposal defines a hierarchy of Abstract Base Classes (ABCs)
> [#pep3119] to represent numbers and other algebraic entities similar
> to numbers.  It proposes:
[snip]

Are the base number operations in Python all that difficult to
understand?  Do we really need to add mathematical formalism into
Python's type system before people understand the meaning of X * Y? 
Because all I really see this doing is confusing the hell out of people
who aren't mathematicians; I'm a theoretical computer scientist and I
had to try to think for a moment to verify that all of those were really
necessary to separate the cases.

I really do understand wanting to be able to ask "is operation X
supported for values Y and Z", but is this really necessary for numbers?


 - Josiah



From jyasskin at gmail.com  Wed Apr 25 12:50:20 2007
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Wed, 25 Apr 2007 09:50:20 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <20070425085350.6408.JCARLSON@uci.edu>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<20070425085350.6408.JCARLSON@uci.edu>
Message-ID: <5d44f72f0704250950j2c2f6c77i61560a0d60ae7da4@mail.gmail.com>

[bcc'ing numpy-discussion. Comments should probably try to stay on the
python-3000 list.]

On 4/25/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> Are the base number operations in Python all that difficult to
> understand?  Do we really need to add mathematical formalism into
> Python's type system before people understand the meaning of X * Y?
> Because all I really see this doing is confusing the hell out of people
> who aren't mathematicians; I'm a theoretical computer scientist and I
> had to try to think for a moment to verify that all of those were really
> necessary to separate the cases.
>
> I really do understand wanting to be able to ask "is operation X
> supported for values Y and Z", but is this really necessary for numbers?

If you don't want to use the full power of the system, nothing forces
you to. If someone doesn't know what a Ring or a complex number is,
they can just check for Real and rely on the same operations, at the
small cost of overconstraining their inputs. One reason I'm suggesting
splitting the classes into the "algebra" and "numbers" modules is so
that people who don't know or care about the algebra can rely on just
"numbers" and still get nearly all of the benefits.

However, I think it's important to provide a framework for people who
want to do more powerful things. If you write a function that can take
an arbitrary ring, it should Just Work on a python int, or any other
Complex subclass, without you having to check for a lot of operations
separately. The Haskell98 numerical hierarchy actually does start at
Num, ignoring the algebraic structures underneath, and it draws a lot
of criticism for it.

That does mean that the names and documentation in the numbers module
need to be as clear as we can make them for non-mathematical
audiences. I'm certainly open to suggestions on that front.

-- 
Namast?,
Jeffrey Yasskin

From collinw at gmail.com  Wed Apr 25 12:54:20 2007
From: collinw at gmail.com (Collin Winter)
Date: Wed, 25 Apr 2007 09:54:20 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
Message-ID: <43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>

The following things mean absolutely nothing to me:

- Monoid
- MonoidUnderPlus
- Group
- Ring
- Semiring
- Field

So, most of the terminology in the PEP.

I can see use-cases for this level of formalism, but I'm a strong -1
on making any part of the stdlib effectively off-limits for people
without advanced math degrees. Why can't this be shipped as a
third-party module?

Collin Winter


From jyasskin at gmail.com  Wed Apr 25 13:09:45 2007
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Wed, 25 Apr 2007 10:09:45 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <20070425085350.6408.JCARLSON@uci.edu>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<20070425085350.6408.JCARLSON@uci.edu>
Message-ID: <5d44f72f0704251009w12eddd82l971f5d06b35b22fb@mail.gmail.com>

[bcc'ing numpy-discussion. Comments should probably try to stay on the
python-3000 list.]

On 4/25/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> Are the base number operations in Python all that difficult to
> understand?  Do we really need to add mathematical formalism into
> Python's type system before people understand the meaning of X * Y?
> Because all I really see this doing is confusing the hell out of people
> who aren't mathematicians; I'm a theoretical computer scientist and I
> had to try to think for a moment to verify that all of those were really
> necessary to separate the cases.
>
> I really do understand wanting to be able to ask "is operation X
> supported for values Y and Z", but is this really necessary for numbers?

If you don't want to use the full power of the system, nothing forces
you to. If someone doesn't know what a Ring or a complex number is,
they can just check for Real and rely on the same operations, at the
small cost of overconstraining their inputs. One reason I'm suggesting
splitting the classes into the "algebra" and "numbers" modules is so
that people who don't know or care about the algebra can rely on just
"numbers" and still get nearly all of the benefits.

However, I think it's important to provide a framework for people who
want to do more powerful things. If you write a function that can take
an arbitrary ring, it should Just Work on a python int, or any other
Complex subclass, without you having to check for a lot of operations
separately. The Haskell98 numerical hierarchy actually does start at
Num, ignoring the algebraic structures underneath, and it draws a lot
of criticism for it.

That does mean that the names and documentation in the numbers module
need to be as clear as we can make them for non-mathematical
audiences. I'm certainly open to suggestions on that front.

-- 
Namast?,
Jeffrey Yasskin

From guido at python.org  Wed Apr 25 14:19:56 2007
From: guido at python.org (Guido van Rossum)
Date: Wed, 25 Apr 2007 11:19:56 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
Message-ID: <ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>

On 4/25/07, Collin Winter <collinw at gmail.com> wrote:
> The following things mean absolutely nothing to me:
>
> - Monoid
> - MonoidUnderPlus
> - Group
> - Ring
> - Semiring
> - Field
>
> So, most of the terminology in the PEP.
>
> I can see use-cases for this level of formalism, but I'm a strong -1
> on making any part of the stdlib effectively off-limits for people
> without advanced math degrees. Why can't this be shipped as a
> third-party module?

As a math major I have no excuse, but I have to confess I'm also
really rusty on these things. (Nothing a quick look at wikipedia can't
refresh though.)

Jeffrey, is there any way you can drop the top of the tree and going
straight from Number to Complex -> Real -> Rational -> Integer? These
are the things that everyone with high school math will know.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From jyasskin at gmail.com  Wed Apr 25 16:04:18 2007
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Wed, 25 Apr 2007 13:04:18 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
Message-ID: <5d44f72f0704251304o15f2e080wcb9ade83be56403@mail.gmail.com>

On 4/25/07, Guido van Rossum <guido at python.org> wrote:
> Jeffrey, is there any way you can drop the top of the tree and going
> straight from Number to Complex -> Real -> Rational -> Integer? These
> are the things that everyone with high school math will know.

I think yes, if you can confirm that the
    import numbers
    class Ring(AdditiveGroup): ...
    numbers.Complex.__bases__ = (Ring,) + numbers.Complex.__bases__
hack I mention under IntegralDomain will make isinstance(int, Ring) return True.

Do you want "Number" to be equivalent to Ring+Hashable (or a
Haskell-like Ring+Hashable+__abs__+__str__)? I don't think a "Number"
class is strictly necessary since people can check for Complex or Real
directly, and one of those two is probably what they'll expect after
knowing that something's a number.

Also, I don't see much point in putting Rational between Real and
Integer. The current hierarchy is Real (int, float, Decimal, rational)
:> Integer (int) and Real :> FractionalReal (float, Decimal,
rational), but Integer and FractionalReal are just siblings.


I'm wondering how many people are writing new numeric types who don't
know the abstract algebra. I think we should be able to insulate
people who are just using the types from the complexities underneath,
and the people writing new types will benefit. Or will seeing "Ring"
in a list of superclasses be enough to confuse people?

-- 
Namast?,
Jeffrey Yasskin

From guido at python.org  Wed Apr 25 16:58:45 2007
From: guido at python.org (Guido van Rossum)
Date: Wed, 25 Apr 2007 13:58:45 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <5d44f72f0704251304o15f2e080wcb9ade83be56403@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<5d44f72f0704251304o15f2e080wcb9ade83be56403@mail.gmail.com>
Message-ID: <ca471dc20704251358m6c06dddfga9f300b2aa1b4550@mail.gmail.com>

On 4/25/07, Jeffrey Yasskin <jyasskin at gmail.com> wrote:
> On 4/25/07, Guido van Rossum <guido at python.org> wrote:
> > Jeffrey, is there any way you can drop the top of the tree and going
> > straight from Number to Complex -> Real -> Rational -> Integer? These
> > are the things that everyone with high school math will know.
>
> I think yes, if you can confirm that the
>     import numbers
>     class Ring(AdditiveGroup): ...
>     numbers.Complex.__bases__ = (Ring,) + numbers.Complex.__bases__
> hack I mention under IntegralDomain will make isinstance(int, Ring) return True.

Hm... That would require built-in types to be mutable (by mere
mortals) which would kill the idea of multiple interpreters. AFAIK the
only user of multiple interpreters is mod_python, which is a pretty
popular Apache module... (Google claims 1.8M hits, vs. 5.66M for
mod_perl, the market leader.)

I guess the alternative would be to keep the mathematical ones in but
make it so that mere mortals (among which I count myself :) never need
to know about them. Sort of like metaclasses, for most people.

But this still presents a bit of a problem. If we insist that built-in
types are immutable, whenever a mathematician needs a new kind of
algebraic category (you mention a few in the PEP), they have no way to
declare that int or float are members of that category. They could
write things like

class MyFloat(float, DivisionRing): pass

but this would be fairly painful. I would like to let users modify the
built-ins, but such changes ought to be isolated from other
interpreters running in the same address space. I don't want to be the
one to tell the mod_python community they won't be able to upgrade to
3.0...

> Do you want "Number" to be equivalent to Ring+Hashable (or a
> Haskell-like Ring+Hashable+__abs__+__str__)? I don't think a "Number"
> class is strictly necessary since people can check for Complex or Real
> directly, and one of those two is probably what they'll expect after
> knowing that something's a number.

Fair enough, you could start with Complex.

> Also, I don't see much point in putting Rational between Real and
> Integer. The current hierarchy is Real (int, float, Decimal, rational)
> :> Integer (int) and Real :> FractionalReal (float, Decimal,
> rational), but Integer and FractionalReal are just siblings.

Why? Why not have a properfraction() on Integer that always returns 0
for the fraction? The current math.modf() works just fine on ints
(returning a tuple of floats).

And why distinguish between Real and FractionalReal? What's the use
case for reals that don't know how to compute their fractional part?

> I'm wondering how many people are writing new numeric types who don't
> know the abstract algebra.

It's easy to know the algebra without recalling the terminologies.
Lots of programmers (myself included!) are largely self-taught.

> I think we should be able to insulate
> people who are just using the types from the complexities underneath,
> and the people writing new types will benefit. Or will seeing "Ring"
> in a list of superclasses be enough to confuse people?

Shouldn't only mathematicians be tempted to write classes deriving
from Ring? Everyone else is likely to subclass one of the more
familiar types, from Complex onwards.

PS. I've checked your PEP in, as PEP3141. We might as well keep a
fairly complete record in svn, even if we decide to cut out the
algebraic foundational classes.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From exarkun at divmod.com  Wed Apr 25 18:40:25 2007
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Wed, 25 Apr 2007 18:40:25 -0400
Subject: [Numpy-discussion] [Python-3000] ABC PEP isinstance issue Was:
 PEP 31XX: A Type Hierarchy for Numbers (and other algebraic entities)
In-Reply-To: <fb6fbf560704251510i2ba00c73if2aa5d1784df190d@mail.gmail.com>
Message-ID: <20070425224025.19381.1937422068.divmod.quotient.5365@ohm>

On Wed, 25 Apr 2007 18:10:23 -0400, Jim Jewett <jimjjewett at gmail.com> wrote:
>The current ABC proposal is to use isinstance as the test; Jeffrey
>Yaskin's numbers PEP highlighted the weakness there with a concrete
>example.
>
>If you need to an abstraction less powerful than an existing ABC,
>you're out of luck; you can't just assert that the existing class is
>already sufficient, nor can you expect everyone else to use multiple
>annotations.

I'm sure everyone is already aware of the behavior of the classImplements
and directlyProvides functions available in zope.interface, which exactly
satisfy this use-case in the interface world.

Jean-Paul


From pje at telecommunity.com  Wed Apr 25 20:31:10 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 25 Apr 2007 20:31:10 -0400
Subject: [Numpy-discussion] [Python-3000] ABC PEP isinstance issue Was:
 PEP 31XX: A Type Hierarchy for Numbers (and other algebraic entities)
In-Reply-To: <fb6fbf560704251510i2ba00c73if2aa5d1784df190d@mail.gmail.co
 m>
Message-ID: <5.1.1.6.0.20070425202858.04ee1ee8@sparrow.telecommunity.com>

At 06:10 PM 4/25/2007 -0400, Jim Jewett wrote:
>I suspect Phillip will say that we really need to make the ABCs
>generic functions...

Nope; they're *already* generic functions: hash(), len(), iter(), various 
operator.* functions, and I believe we're adding next().  I've got nothing 
to add to that list.  :)



From pje at telecommunity.com  Wed Apr 25 20:40:14 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 25 Apr 2007 20:40:14 -0400
Subject: [Numpy-discussion] [Python-3000] ABC PEP isinstance issue Was:
 PEP 31XX: A Type Hierarchy for Numbers (and other algebraic entities)
In-Reply-To: <20070425224025.19381.1937422068.divmod.quotient.5365@ohm>
References: <fb6fbf560704251510i2ba00c73if2aa5d1784df190d@mail.gmail.com>
Message-ID: <5.1.1.6.0.20070425203654.04ee17d8@sparrow.telecommunity.com>

At 06:40 PM 4/25/2007 -0400, Jean-Paul Calderone wrote:
>On Wed, 25 Apr 2007 18:10:23 -0400, Jim Jewett <jimjjewett at gmail.com> wrote:
> >The current ABC proposal is to use isinstance as the test; Jeffrey
> >Yaskin's numbers PEP highlighted the weakness there with a concrete
> >example.
> >
> >If you need to an abstraction less powerful than an existing ABC,
> >you're out of luck; you can't just assert that the existing class is
> >already sufficient, nor can you expect everyone else to use multiple
> >annotations.
>
>I'm sure everyone is already aware of the behavior of the classImplements
>and directlyProvides functions available in zope.interface, which exactly
>satisfy this use-case in the interface world.

I'm either misunderstanding Jim or you, because I don't see the 
relationship here.  If I understand Jim correctly, he's actually asking for 
something like PyProtocols' "protocolIsSubsetOf" declaration -- something 
that would be like dynamically adding a superclass to an existing interface 
in zope.interface.  Whereas the features you're talking about sound like 
declaring that a class object itself implements an interface -- something 
apparently unrelated to the question at hand.



From exarkun at divmod.com  Wed Apr 25 22:06:05 2007
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Wed, 25 Apr 2007 22:06:05 -0400
Subject: [Numpy-discussion] [Python-3000] ABC PEP isinstance issue Was:
 PEP 31XX: A Type Hierarchy for Numbers (and other algebraic entities)
In-Reply-To: <5.1.1.6.0.20070425203654.04ee17d8@sparrow.telecommunity.com>
Message-ID: <20070426020605.19381.476607338.divmod.quotient.5402@ohm>

On Wed, 25 Apr 2007 20:40:14 -0400, "Phillip J. Eby" <pje at telecommunity.com> wrote:
>At 06:40 PM 4/25/2007 -0400, Jean-Paul Calderone wrote:
>>On Wed, 25 Apr 2007 18:10:23 -0400, Jim Jewett <jimjjewett at gmail.com> 
>>wrote:
>> >The current ABC proposal is to use isinstance as the test; Jeffrey
>> >Yaskin's numbers PEP highlighted the weakness there with a concrete
>> >example.
>> >
>> >If you need to an abstraction less powerful than an existing ABC,
>> >you're out of luck; you can't just assert that the existing class is
>> >already sufficient, nor can you expect everyone else to use multiple
>> >annotations.
>>
>>I'm sure everyone is already aware of the behavior of the classImplements
>>and directlyProvides functions available in zope.interface, which exactly
>>satisfy this use-case in the interface world.
>
>I'm either misunderstanding Jim or you, because I don't see the relationship 
>here.  If I understand Jim correctly, he's actually asking for something 
>like PyProtocols' "protocolIsSubsetOf" declaration -- something that would 
>be like dynamically adding a superclass to an existing interface in 
>zope.interface.  Whereas the features you're talking about sound like 
>declaring that a class object itself implements an interface -- something 
>apparently unrelated to the question at hand.
>

Ugh, you're at least half right.  I didn't read the implementation of
ABC.meets carefully enough to notice that it's actually an adapter
registry, not just a predicate (meets seems like a funny name for that
method).

classImplements would have satisfied the adapterless use-case by letting
Jim declare that someone else's class definition actually implements the
feature set that some unit of code he wants to use that class with is
expecting.

With the adapter registry, I don't see why this isn't just PEP 246 adapt()
(if I am reading one of your later mails correctly though, you think it is
too).

I'm not particularly familiar with protocolIsSubsetOf, but it wouldn't
surprise me at all if it could also satisfy this use-case.  There are
lots of ways to do this, many of them fairly simple.  Using inheritance
is the part that makes it hard, since inheritance is controlled by the
wrong party in most cases and comes with unrelated features that are,
at best, irrelevant to the particular use case and at worst actively
detrimental.

I'm sure a way around this can be invented, I just don't see why it
should be.

Jean-Paul


From guido at python.org  Thu Apr 26 00:10:54 2007
From: guido at python.org (Guido van Rossum)
Date: Wed, 25 Apr 2007 21:10:54 -0700
Subject: [Numpy-discussion] [Python-3000] ABC PEP isinstance issue Was:
	PEP 31XX: A Type Hierarchy for Numbers (and other algebraic
	entities)
In-Reply-To: <20070426020605.19381.476607338.divmod.quotient.5402@ohm>
References: <5.1.1.6.0.20070425203654.04ee17d8@sparrow.telecommunity.com>
	<20070426020605.19381.476607338.divmod.quotient.5402@ohm>
Message-ID: <ca471dc20704252110t6c27543egdf626ce7ded51b4e@mail.gmail.com>

On 4/25/07, Jean-Paul Calderone <exarkun at divmod.com> wrote:
> On Wed, 25 Apr 2007 20:40:14 -0400, "Phillip J. Eby" <pje at telecommunity.com> wrote:
> >At 06:40 PM 4/25/2007 -0400, Jean-Paul Calderone wrote:
> >>On Wed, 25 Apr 2007 18:10:23 -0400, Jim Jewett <jimjjewett at gmail.com>
> >>wrote:
> >> >The current ABC proposal is to use isinstance as the test; Jeffrey
> >> >Yaskin's numbers PEP highlighted the weakness there with a concrete
> >> >example.
> >> >
> >> >If you need to an abstraction less powerful than an existing ABC,
> >> >you're out of luck; you can't just assert that the existing class is
> >> >already sufficient, nor can you expect everyone else to use multiple
> >> >annotations.
> >>
> >>I'm sure everyone is already aware of the behavior of the classImplements
> >>and directlyProvides functions available in zope.interface, which exactly
> >>satisfy this use-case in the interface world.
> >
> >I'm either misunderstanding Jim or you, because I don't see the relationship
> >here.  If I understand Jim correctly, he's actually asking for something
> >like PyProtocols' "protocolIsSubsetOf" declaration -- something that would
> >be like dynamically adding a superclass to an existing interface in
> >zope.interface.  Whereas the features you're talking about sound like
> >declaring that a class object itself implements an interface -- something
> >apparently unrelated to the question at hand.
> >
>
> Ugh, you're at least half right.  I didn't read the implementation of
> ABC.meets carefully enough to notice that it's actually an adapter
> registry, not just a predicate (meets seems like a funny name for that
> method).
>
> classImplements would have satisfied the adapterless use-case by letting
> Jim declare that someone else's class definition actually implements the
> feature set that some unit of code he wants to use that class with is
> expecting.
>
> With the adapter registry, I don't see why this isn't just PEP 246 adapt()
> (if I am reading one of your later mails correctly though, you think it is
> too).
>
> I'm not particularly familiar with protocolIsSubsetOf, but it wouldn't
> surprise me at all if it could also satisfy this use-case.  There are
> lots of ways to do this, many of them fairly simple.  Using inheritance
> is the part that makes it hard, since inheritance is controlled by the
> wrong party in most cases and comes with unrelated features that are,
> at best, irrelevant to the particular use case and at worst actively
> detrimental.
>
> I'm sure a way around this can be invented, I just don't see why it
> should be.

JP, would you mind dropping this and instead considering how we could
turn isinstance and issubclass into (poor-man's) GFs by allowing
classes to define class methods __isinstance__ and __issubclass__?
(Though the arguments are reversed so to avoid confusion perhaps the
names should be different, similar as with the 'in' operator and the
__contains__ magic method.)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From pav at iki.fi  Thu Apr 26 03:47:02 2007
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 26 Apr 2007 10:47:02 +0300
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <B11B844B-3741-422C-B90C-9E43F7E9B279@cs.hmc.edu>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no> <462FAC94.1040207@gmail.com>
	<e86a5fd00704251246h1fbe4003v4adf28376bd2ea54@mail.gmail.com>
	<B11B844B-3741-422C-B90C-9E43F7E9B279@cs.hmc.edu>
Message-ID: <463058F6.3030602@iki.fi>

belinda thom kirjoitti:
> On Apr 25, 2007, at 12:46 PM, Bill Baxter wrote:
> 
> Agree w/most of what you've said, but will add one other thing that  
> drives me nuts in python that hasn't been a problem in Matplotlib:
> 
> In Python, if interacting w/the interpreter as your primary IDE, and  
> if you've got multiple files that depend on one another that you're  
> modifying, then you need to restart the interpreter frequently b/c  
> otherwise things in the interpreter can be stale; IOW, changes to  
> several interdependent files aren't easy to import so that everything  
> in your interpreted environment reflects the latest code. Yeah,  
> there's reload tricks, but doing them in the right order and the  
> right number of times can be a pain when dependencies are cyclic.
> 
> I realize in general that this issue of stale code is just difficult,  
> that its not inherently a Python problem per se, for automatic  
> percolation of code changes backwards in time is difficult in  
> general, but I've never had the problem bite me when I was developing  
> in Matlab. I just save whatever file, and it appears that whatever's  
> latest on disk is what's executed. (Friends who know more about PL  
> than I tell me I've been lucky.)

I've been using the attached autoreload code (by Thomas Heller,
different versions are probably floating around the net) rather
successfully for a more Matlabish feel to IPython. It reloads modules
every time their files have been changed.

Of course, this doesn't solve all staleness problems you describe, but
only eliminates manual work involved in solving the most common ones.
Reloading modules does have pitfalls [1], but at least in my use these
haven't really mattered much in practice.

I think having autoreload functionality bundled with IPython (maybe
turned off per default) would be quite useful: although in some rare
cases autoreloading doesn't work in the ideal way, it's very convenient
not to have to type reload(foo) after every change, especially when
tweaking for example plotting scripts.

	Pauli

[1] For example, instances of classes created are not updated on reload,
but continue using the old code. Also, from foo import * imports are not
updated, so you'll have to manually touch files that do this if 'foo'
has been changed.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: autoreload.py
Type: text/x-python
Size: 6219 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070426/b206db3d/attachment.py>

From exarkun at divmod.com  Thu Apr 26 13:26:02 2007
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Thu, 26 Apr 2007 13:26:02 -0400
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
 Numbers (and other algebraic entities)
In-Reply-To: <07Apr26.101132pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <20070426172602.19381.679137565.divmod.quotient.5553@ohm>

On Thu, 26 Apr 2007 10:11:27 PDT, Bill Janssen <janssen at parc.com> wrote:
>> Are the base number operations in Python all that difficult to
>> understand?
>
>Well, they're a little tricky.
>
>But the basic problem with "number" support, in almost every
>programming language, is that they are too low-level an abstraction.
>A number in a program is never *just* an integer or a float.  It's
>always an integer or float (or complex, OK) deployed in support of a
>very specific purpose.  It's a loop index, or a weight in kilos, or
>the number of calendar entries for a particular day.  It often has
>units, and a fixed range of values or fixed set of values, neither of
>which are taken into account in most programming languages.
>
>Strings really suffer from the same lack of focus.  A string is never
>just a string.  It's a book title, or a filename, or a person's
>surname, or an email address.  It's usually got a particular human
>language (or computer OS) associated with it.  Each of these usages
>has particular limitations and patterns which limit both its range of
>values, and the operations which can be successfully applied to it.

I definitely agree with this, and resolving it would probably address
a large class of mistakes made when doing any programming involving
numbers and strings (anyone done much of that lately? :).

Does the introduction of a lot of base classes for `int' and `float'
do anything to address this, though?  What you really want is a bunch
of new subclasses  of them, the opposite.

Jean-Paul


From anton.sherwood at gmail.com  Thu Apr 26 14:30:15 2007
From: anton.sherwood at gmail.com (Anton Sherwood)
Date: Thu, 26 Apr 2007 11:30:15 -0700
Subject: [Numpy-discussion] sort bug
In-Reply-To: <46306253.20803@ieee.org>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
Message-ID: <fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>

Travis Oliphant wrote:
> cmp(x,y) must return -1, 0, or 1 which doesn't work on arrays with more
> than 1 element because it is ambiguous.  Thus you get this error.

Ah.  Since lists *can* be compared, I assumed arrays would inherit
that property.

> The operation is undefined.  What do you actually want to do when you
> have equal-valued eigenvalues?

Don't care.

All I really need is the four highest eigenvalues and their vectors.
I'll have a look in "Python Cookbook" to see if there's a more
efficient way to do that partial sort.

-- 
Anton Sherwood, http://www.ogre.nu/


From rowen at cesmail.net  Thu Apr 26 14:38:55 2007
From: rowen at cesmail.net (Russell E. Owen)
Date: Thu, 26 Apr 2007 11:38:55 -0700
Subject: [Numpy-discussion] numpy endian question
Message-ID: <rowen-084D61.11385526042007@sea.gmane.org>

In converting some code from numarray to numpy I had this:
isBigendian = (arr.isbyteswapped() != numarray.isBigEndian)

The only numpy version I've come up with is:
isBigEndian = (arr.dtype.descr[0][1][0] == '>')

which is short but very obscure. Has anyone got a suggestion for a 
clearer test? I found lots of *almost* useful flags and methods.

-- Russell



From David.L.Goldsmith at noaa.gov  Thu Apr 26 14:41:59 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Thu, 26 Apr 2007 11:41:59 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
 Numbers (and other algebraic entities)
In-Reply-To: <ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
Message-ID: <4630F277.1030008@noaa.gov>

Guido van Rossum wrote:
> Jeffrey, is there any way you can drop the top of the tree and going
> straight from Number to Complex -> Real -> Rational -> Integer? These
> are the things that everyone with high school math will know.
>   
Having taught mathematics at a community college for a little while 
(hell, having taught mathematics at a few Universities - I won't name 
names), I would suggest that this is not a valid premise.  I was going 
to qualify that by adding that hopefully it is a valid premise for 
would-be mathematics-package-using programmers, but I recall a fellow 
student in an Intro to C class I was once in (not teaching, however) 
who, even though the class was not supposed to be an Intro to 
Programming course (this was back in the day when those were still 
taught in Pascal and those taking the C class were supposed to have had 
successfully completed the Pascal course first), had to be handheld not 
simply through the implementation of algorithms in C, but in the very 
development of algorithms from a problem statement.  Even without having 
to resort to such an extreme anecdote, I can envision situations where 
knowledgeable, skilled, experienced, but "deficient"-in-math programmers 
are called upon (implicitly or explicitly) by management to use numpy/scipy.

All this said, I'm certainly of the opinion that we can't "dumb down" 
numpy *too* much (after all, by some definition, it is an "advanced" 
mathematics package, at least IMO), and to my mind the sequence above, 
i.e., X->C->R->Q->Z, is a more than reasonable "line in the sand".

DG


From rowen at cesmail.net  Thu Apr 26 14:50:39 2007
From: rowen at cesmail.net (Russell E. Owen)
Date: Thu, 26 Apr 2007 11:50:39 -0700
Subject: [Numpy-discussion] Questions about converting to numpy
References: <rowen-01812A.13015325042007@sea.gmane.org>
	<462FB8CC.5020709@noaa.gov> <462FC8C6.3030200@gmail.com>
Message-ID: <rowen-04F0E1.11503926042007@sea.gmane.org>

In article <462FC8C6.3030200 at gmail.com>,
 Robert Kern <robert.kern at gmail.com> wrote:

> Christopher Barker wrote:
> 
> > I can only help with one:
> >> - Even after reading the book I'm not really clear on why one would use 
> >> numpy.float_ instead of numpy.float or float
> > 
> > They float and numpy.float are the same, and numpy.float_ is the same as 
> > numpy.float64:
> > 
> >  >>> import numpy
> >  >>> float is numpy.float
> > True
> >  >>> numpy.float64 is numpy.float64
> > True
> >  >>>
> > 
> > float was added to the numpy namespace so that we could write consistent 
> > code like:
> > 
> > a = array(object, numpy.float32)
> > b = array(object, numpy.float)
> > 
> > i.e. have it all in the same namespace.
> > 
> > I'm not sure why float_ is an alias for float64, though I'm guessing 
> > it's possible that on some platforms they are not the same.
> 
> Rather, numpy.float used to be an alias for numpy.float64; however, it 
> overrode
> the builtin float() when "from numpy import *" was used at the interactive
> prompt. Consequently, we renamed it numpy.float_ and specifically imported 
> the
> builtin float as numpy.float such that we didn't break code that had already
> started using "numpy.float".

But I still don't understand why one shouldn't just use dtype=float or 
numpy.float. Does that result in an array with a different type of float 
than numpy.float_ (float64)? Or does it just somehow speed up numpy 
because it doesn't have to convert the python type into a numpy dtype.

Anyway, thank you all for the helpful answers! I'm glad numpy throws the 
standard MemoryError since I'm already testing for that.

-- Russell



From faltet at carabos.com  Thu Apr 26 14:57:04 2007
From: faltet at carabos.com (Francesc Altet)
Date: Thu, 26 Apr 2007 20:57:04 +0200
Subject: [Numpy-discussion] numpy endian question
In-Reply-To: <rowen-084D61.11385526042007@sea.gmane.org>
References: <rowen-084D61.11385526042007@sea.gmane.org>
Message-ID: <1177613824.2817.65.camel@localhost.localdomain>

El dj 26 de 04 del 2007 a les 11:38 -0700, en/na Russell E. Owen va
escriure:
> In converting some code from numarray to numpy I had this:
> isBigendian = (arr.isbyteswapped() != numarray.isBigEndian)
> 
> The only numpy version I've come up with is:
> isBigEndian = (arr.dtype.descr[0][1][0] == '>')

isBigEndian = (arr.dtype.str[0] == '>')

is a little bit shorter. A more elegant approach could be:

isBigEndian = arr.dtype.isnative ^ numpy.little_endian

Cheers,

-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth



From oliphant.travis at ieee.org  Thu Apr 26 15:08:15 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Thu, 26 Apr 2007 13:08:15 -0600
Subject: [Numpy-discussion] sort bug
In-Reply-To: <fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
Message-ID: <4630F89F.50003@ieee.org>

Anton Sherwood wrote:
> Travis Oliphant wrote:
>   
>> cmp(x,y) must return -1, 0, or 1 which doesn't work on arrays with more
>> than 1 element because it is ambiguous.  Thus you get this error.
>>     
>
> Ah.  Since lists *can* be compared, I assumed arrays would inherit
> that property.
>
>   
>> The operation is undefined.  What do you actually want to do when you
>> have equal-valued eigenvalues?
>>     
>
> Don't care.
>   

One approach is to use argsort to create an index list of sorted 
eigenvalues and then sort the eig and eigvector arrays before zipping 
them together in a list of tuples.


eig, val = numpy.linalg.eig(a)

indx = eig.argsort()
eig = eig.take(indx)
val = val.take(indx, axis=1)
master = zip(eig, val.T)

Will do what you want.


-Travis



From robert.kern at gmail.com  Thu Apr 26 15:02:47 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 26 Apr 2007 14:02:47 -0500
Subject: [Numpy-discussion] Questions about converting to numpy
In-Reply-To: <rowen-04F0E1.11503926042007@sea.gmane.org>
References: <rowen-01812A.13015325042007@sea.gmane.org>	<462FB8CC.5020709@noaa.gov>
	<462FC8C6.3030200@gmail.com>
	<rowen-04F0E1.11503926042007@sea.gmane.org>
Message-ID: <4630F757.2060306@gmail.com>

Russell E. Owen wrote:

> But I still don't understand why one shouldn't just use dtype=float or 
> numpy.float. Does that result in an array with a different type of float 
> than numpy.float_ (float64)? Or does it just somehow speed up numpy 
> because it doesn't have to convert the python type into a numpy dtype.

For specifying dtype=float in functions that take such an argument, go ahead and
use dtype=float. There isn't really a reason to use anything else if you don't
want to. However, the scalar type objects are used in other places, so float_
should exist.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Thu Apr 26 15:05:23 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 26 Apr 2007 13:05:23 -0600
Subject: [Numpy-discussion] sort bug
In-Reply-To: <fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
Message-ID: <e06186140704261205m2223eb03j2d843f7af496551c@mail.gmail.com>

On 4/26/07, Anton Sherwood <anton.sherwood at gmail.com> wrote:
>
> Travis Oliphant wrote:
> > cmp(x,y) must return -1, 0, or 1 which doesn't work on arrays with more
> > than 1 element because it is ambiguous.  Thus you get this error.
>
> Ah.  Since lists *can* be compared, I assumed arrays would inherit
> that property.
>
> > The operation is undefined.  What do you actually want to do when you
> > have equal-valued eigenvalues?
>
> Don't care.
>
> All I really need is the four highest eigenvalues and their vectors.
> I'll have a look in "Python Cookbook" to see if there's a more
> efficient way to do that partial sort.


I don't think there are any partial sorts available. I think the best
approach in this case an argsort on the eigenvalues, followed by a take
using the last four resulting indexes would be the way to go.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070426/81bfcf91/attachment.html>

From peridot.faceted at gmail.com  Thu Apr 26 15:05:47 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 26 Apr 2007 15:05:47 -0400
Subject: [Numpy-discussion] sort bug
In-Reply-To: <fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
Message-ID: <ce557a360704261205ic51cb8er2f3642b0c0c42318@mail.gmail.com>

On 26/04/07, Anton Sherwood <anton.sherwood at gmail.com> wrote:

> All I really need is the four highest eigenvalues and their vectors.

It's not really very efficient, but sorting time is going to be tiny
compared to however you get your eigenvalues, so you could argsort the
eigenvalue array and use the result to sort the eigenvector array.

You can also accelerate the process by quickly checking whether the
eigenvalue array is already sorted - usually it is.

Anne


From tan2tan2 at gmail.com  Thu Apr 26 15:16:14 2007
From: tan2tan2 at gmail.com (tan2)
Date: Thu, 26 Apr 2007 12:16:14 -0700
Subject: [Numpy-discussion] sort bug
In-Reply-To: <e06186140704261205m2223eb03j2d843f7af496551c@mail.gmail.com>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
	<e06186140704261205m2223eb03j2d843f7af496551c@mail.gmail.com>
Message-ID: <f05b24530704261216h646047bfga772774751de983@mail.gmail.com>

>
> On 4/26/07, Anton Sherwood <anton.sherwood at gmail.com> wrote:
> >
> >
> > All I really need is the four highest eigenvalues and their vectors.
> > I'll have a look in "Python Cookbook" to see if there's a more
> > efficient way to do that partial sort.
>
>
>

Maybe "heapq.nlargest()" is what you want?
http://www.python.org/doc/2.4/lib/module-heapq.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070426/8a06e1c9/attachment.html>

From oliphant.travis at ieee.org  Thu Apr 26 15:30:33 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Thu, 26 Apr 2007 13:30:33 -0600
Subject: [Numpy-discussion] numpy endian question
In-Reply-To: <rowen-084D61.11385526042007@sea.gmane.org>
References: <rowen-084D61.11385526042007@sea.gmane.org>
Message-ID: <4630FDD9.9030307@ieee.org>

Russell E. Owen wrote:
> In converting some code from numarray to numpy I had this:
> isBigendian = (arr.isbyteswapped() != numarray.isBigEndian)
>
> The only numpy version I've come up with is:
> isBigEndian = (arr.dtype.descr[0][1][0] == '>')
>
> which is short but very obscure. Has anyone got a suggestion for a 
> clearer test? I found lots of *almost* useful flags and methods.
>   

How about looking at  arr.dtype.byteorder?

What are you trying to test for (where a data-type is big-endian or not?).

sys.byteorder is either "little" or "big"
numpy.little_endian is True if you are on a little endian system.

arr.dtype.byteorder gives you either "=" for native or ">" for 
big-endian or "<" for little-endian or "|" for it doesn't matter.

To directly answer your question, however.

On page 42 of the sample pages (online for free) of my book, the 
recommended translation of numarray's arr.isbyteswapped() is  (not 
arr.dtype.isnative)

The translation of numarray.isBigEndian is (not numpy.little_endian)

Thus a direct translation of your code is:

isBigEndian = ((not arr.dtype.isnative) != (not numpy.little_endian))

which is equivalent to

isBigEndian = (arr.dtype.isnative != numpy.little_endian)

and because != on boolean items is equivalent to XOR this is the same as

isBigEndian = arr.dtype.isnative ^ numpy.little_endian


-Travis




From robert.kern at gmail.com  Thu Apr 26 15:25:47 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 26 Apr 2007 14:25:47 -0500
Subject: [Numpy-discussion] sort bug
In-Reply-To: <f05b24530704261216h646047bfga772774751de983@mail.gmail.com>
References: <463047DF.7070700@pobox.com>
	<46306253.20803@ieee.org>	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>	<e06186140704261205m2223eb03j2d843f7af496551c@mail.gmail.com>
	<f05b24530704261216h646047bfga772774751de983@mail.gmail.com>
Message-ID: <4630FCBB.9050103@gmail.com>

tan2 wrote:
> 
>     On 4/26/07, * Anton Sherwood* <anton.sherwood at gmail.com
>     <mailto:anton.sherwood at gmail.com>> wrote:
> 
>         All I really need is the four highest eigenvalues and their
>         vectors.
>         I'll have a look in "Python Cookbook" to see if there's a more
>         efficient way to do that partial sort.
> 
> Maybe "heapq.nlargest ()" is what you want?
> http://www.python.org/doc/2.4/lib/module-heapq.html

Just doing argsort() on the whole array is faster (up until about 1e6 elements)
because it does everything in C whereas heapq will create a lot of Python
objects because it is treating the array as a general Python container.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From tan2tan2 at gmail.com  Thu Apr 26 16:37:05 2007
From: tan2tan2 at gmail.com (tan2)
Date: Thu, 26 Apr 2007 13:37:05 -0700
Subject: [Numpy-discussion] sort bug
In-Reply-To: <4630FCBB.9050103@gmail.com>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
	<e06186140704261205m2223eb03j2d843f7af496551c@mail.gmail.com>
	<f05b24530704261216h646047bfga772774751de983@mail.gmail.com>
	<4630FCBB.9050103@gmail.com>
Message-ID: <f05b24530704261337l5d52d3dco3a302b97e468729c@mail.gmail.com>

> Just doing argsort() on the whole array is faster (up until about 1e6
> elements)
> because it does everything in C whereas heapq will create a lot of Python
> objects because it is treating the array as a general Python container.
>
>

That's a good point. I wasn't thinking about the efficiency issue.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070426/4f7845c7/attachment.html>

From rowen at cesmail.net  Thu Apr 26 17:07:42 2007
From: rowen at cesmail.net (Russell E. Owen)
Date: Thu, 26 Apr 2007 14:07:42 -0700
Subject: [Numpy-discussion] numpy endian question
References: <rowen-084D61.11385526042007@sea.gmane.org>
	<4630FDD9.9030307@ieee.org>
Message-ID: <rowen-1D3DBF.14074226042007@sea.gmane.org>

In article <4630FDD9.9030307 at ieee.org>,
 Travis Oliphant <oliphant.travis at ieee.org> wrote:

> Russell E. Owen asked how best to translate some numarray code that determines if an array is in big-endian order...
>
> What are you trying to test for (where a data-type is big-endian or not?).

I really want to know if the array is in big-endian order (I don't care 
whether it's native). This is for sending arrays to the ds9 image viewer 
via xpa (and communicating with ds9 is not easy). To do this reliably I 
need to indicate the byte order of the data.

I suppose there are alternatives such as making a "native byte order" 
copy of the data and sending that without any flag. But I think it's 
easier to just send the flag.

> To directly answer your question, however.
> 
> On page 42 of the sample pages (online for free) of my book, the 
> recommended translation of numarray's arr.isbyteswapped() is  (not 
> arr.dtype.isnative)

I found that (I own your book).

> The translation of numarray.isBigEndian is (not numpy.little_endian)

I missed that one (and I did look for it).

>...
> isBigEndian = arr.dtype.isnative ^ numpy.little_endian

That should be much clearer than my translation. Thanks!

-- Russell



From chanley at stsci.edu  Thu Apr 26 17:22:42 2007
From: chanley at stsci.edu (Christopher Hanley)
Date: Thu, 26 Apr 2007 17:22:42 -0400
Subject: [Numpy-discussion] numpy endian question
In-Reply-To: <rowen-1D3DBF.14074226042007@sea.gmane.org>
References: <rowen-084D61.11385526042007@sea.gmane.org>	<4630FDD9.9030307@ieee.org>
	<rowen-1D3DBF.14074226042007@sea.gmane.org>
Message-ID: <46311822.2010206@stsci.edu>

Russell,

This should work as a consistent test for bigendian:

-> isBigEndian = (obj.dtype.str[0] == '>')

Also, I have ported numarray's numdisplay to numpy if you would like to 
directly display an array in DS9.  We haven't done an official release 
yet (sometime soon) but I can forward you a copy if you are interested.

Chris



From oliphant.travis at ieee.org  Thu Apr 26 19:48:42 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Thu, 26 Apr 2007 17:48:42 -0600
Subject: [Numpy-discussion] numpy endian question
In-Reply-To: <rowen-1D3DBF.14074226042007@sea.gmane.org>
References: <rowen-084D61.11385526042007@sea.gmane.org>	<4630FDD9.9030307@ieee.org>
	<rowen-1D3DBF.14074226042007@sea.gmane.org>
Message-ID: <46313A5A.7030301@ieee.org>


> I really want to know if the array is in big-endian order (I don't care 
> whether it's native). This is for sending arrays to the ds9 image viewer 
> via xpa (and communicating with ds9 is not easy). To do this reliably I 
> need to indicate the byte order of the data.
>   

Thanks for the clarification and the original question.    By the way, 
are you aware of the  numpy.numarray module

numpy.numarray.isBigEndian  works as expected.


-Travis



From bthom at cs.hmc.edu  Fri Apr 27 01:23:33 2007
From: bthom at cs.hmc.edu (belinda thom)
Date: Thu, 26 Apr 2007 22:23:33 -0700
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <463058F6.3030602@iki.fi>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no> <462FAC94.1040207@gmail.com>
	<e86a5fd00704251246h1fbe4003v4adf28376bd2ea54@mail.gmail.com>
	<B11B844B-3741-422C-B90C-9E43F7E9B279@cs.hmc.edu>
	<463058F6.3030602@iki.fi>
Message-ID: <8A1A9308-ECB6-4035-9938-046CC9D2E0C4@cs.hmc.edu>

Thank you so much! When I finally get a moment to take a break, I'll  
look in more detail into using your suggestion.

--b


On Apr 26, 2007, at 12:47 AM, Pauli Virtanen wrote:

> belinda thom kirjoitti:
>> On Apr 25, 2007, at 12:46 PM, Bill Baxter wrote:
>>
>> Agree w/most of what you've said, but will add one other thing that
>> drives me nuts in python that hasn't been a problem in Matplotlib:
>>
>> In Python, if interacting w/the interpreter as your primary IDE, and
>> if you've got multiple files that depend on one another that you're
>> modifying, then you need to restart the interpreter frequently b/c
>> otherwise things in the interpreter can be stale; IOW, changes to
>> several interdependent files aren't easy to import so that everything
>> in your interpreted environment reflects the latest code. Yeah,
>> there's reload tricks, but doing them in the right order and the
>> right number of times can be a pain when dependencies are cyclic.
>>
>> I realize in general that this issue of stale code is just difficult,
>> that its not inherently a Python problem per se, for automatic
>> percolation of code changes backwards in time is difficult in
>> general, but I've never had the problem bite me when I was developing
>> in Matlab. I just save whatever file, and it appears that whatever's
>> latest on disk is what's executed. (Friends who know more about PL
>> than I tell me I've been lucky.)
>
> I've been using the attached autoreload code (by Thomas Heller,
> different versions are probably floating around the net) rather
> successfully for a more Matlabish feel to IPython. It reloads modules
> every time their files have been changed.
>
> Of course, this doesn't solve all staleness problems you describe, but
> only eliminates manual work involved in solving the most common ones.
> Reloading modules does have pitfalls [1], but at least in my use these
> haven't really mattered much in practice.
>
> I think having autoreload functionality bundled with IPython (maybe
> turned off per default) would be quite useful: although in some rare
> cases autoreloading doesn't work in the ideal way, it's very  
> convenient
> not to have to type reload(foo) after every change, especially when
> tweaking for example plotting scripts.
>
> 	Pauli
>
> [1] For example, instances of classes created are not updated on  
> reload,
> but continue using the old code. Also, from foo import * imports  
> are not
> updated, so you'll have to manually touch files that do this if 'foo'
> has been changed.
> """
>
> autoreload.py - automatically reload changed source
> code into a running program
>
> You might want to add the following to your ~/.ipython/ipythonrc
>
>     import_mod sys
>     execute sys.path.append('path/where/this/file/resides')
>     import_mod autoreload
>     execute autoreload.run()
>
> or adding the following
>
>     import sys
>     sys.path.append('path/where/this/file/resides')
>     import autoreload
>     autoreload.run()
>
> to some startup file.
>
>
> Created: Thomas Heller, 2000-04-17
> Modified: Pauli Virtanen, 2006
> """
>
> # $Id: autoreload.py 3117 2006-09-27 20:28:46Z pauli $
> #
> # $Log: autoreload.py,v $
> # Revision 1.9  2001/11/15 18:41:18  thomas
> # Cleaned up and made working again before posting to c.l.p.
> # Added code to update bound (or unbound) methods as suggested
> # by Just van Rossum. Thanks!
> #
> # ...
> #
> # Revision 1.1  2001/10/04 16:54:04  thomas
> # Discovered this old module on my machine, it didn't work too well,
> # but seems worth to continue with it...
> # Checked in as a first step.
>
>
> __version__ = "$Revision: 1.9 $".split()[1]
>
> # ToDo:
> #
> #  Cannot reload __main__ - explain why this cannot work,
> #  and explain a workaround.
> #
> #  Optimize - the number of watches objects (in old_objects)
> #  grows without limits. Think if this is really necessary...
>
>
> import time, os, threading, sys, types, imp, inspect, traceback
>
> def _get_compiled_ext():
>     for ext, mode, typ in imp.get_suffixes():
>         if typ == imp.PY_COMPILED:
>             return ext
>
> # the official way to get the extension of compiled files (.pyc  
> or .pyo)
> PY_COMPILED_EXT = _get_compiled_ext()
>
> class ModuleWatcher:
>     running = 0
>     def __init__(self):
>         # If we don't do this, there may be tracebacks
>         # when shutting down python.
>         import atexit
>         atexit.register(self.stop)
>
>     def run(self):
>         if self.running:
>             print "# autoreload already running"
>             return
>         print "# starting autoreload"
>         self.running = 1
>         self.thread = threading.Thread(target=self._check_modules)
>         self.thread.setDaemon(1)
>         self.thread.start()
>
>     def stop(self):
>         if not self.running:
>             print "# autoreload not running"
>             return
>         self.running = 0
>         self.thread.join()
>         #print "# autoreload stopped"
>
>     def _check_modules(self):
>         skipped = {}
>         while self.running:
>             time.sleep(0.01)
>             for m in sys.modules.values():
>                 if not hasattr(m, '__file__'):
>                     continue
>                 if m.__name__ == '__main__':
>
>                     # we cannot reload(__main__) First I thought we
>                     # could use mod = imp.load_module() and then
>                     # reload(mod) to simulate reload(main), but this
>                     # would execute the code in __main__ a second
>                     # time.
>
>                     continue
>                 file = m.__file__
>                 dirname = os.path.dirname(file)
>                 path, ext = os.path.splitext(file)
>
>                 if ext.lower() == '.py':
>                     ext = PY_COMPILED_EXT
>                     file = os.path.join(dirname, path +  
> PY_COMPILED_EXT)
>
>                 if ext != PY_COMPILED_EXT:
>                     continue
>
>                 try:
> 		    pymtime = os.stat(file[:-1])[8]
>                     if pymtime <= os.stat(file)[8]:
>                         continue
> 		    if skipped.get(file[:-1], None) == pymtime:
> 		        continue
>                 except OSError:
>                     continue
>
>                 try:
>                     superreload(m)
> 		    if file[:-1] in skipped:
> 		        del skipped[file[:-1]]
>                 except:
> 		    skipped[file[:-1]] = pymtime
>                     import traceback
>                     traceback.print_exc(0)
>
> def update_function(old, new, attrnames):
>     for name in attrnames:
>         setattr(old, name, getattr(new, name))
>
> def superreload(module,
>                 reload=reload,
>                 _old_objects = {}):
>     """superreload (module) -> module
>
>     Enhanced version of the builtin reload function.
>     superreload replaces the class dictionary of every top-level
>     class in the module with the new one automatically,
>     as well as every function's code object.
>     """
> ##    start = time.clock()
>     # retrieve the attributes from the module before the reload,
>     # and remember them in _old_objects.
>     for name, object in module.__dict__.items():
>         key = (module.__name__, name)
>         _old_objects.setdefault(key, []).append(object)
>         # print the refcount of old objects:
> ##        if type(object) in (types.FunctionType, types.ClassType):
> ##            print name, map(sys.getrefcount, _old_objects[key])
>
> ##    print "# reloading module %r" % module
>
>     module = reload(module)
>     # XXX We have a problem here if importing the module fails!
>
>     # iterate over all objects and update them
>     count = 0
>     # XXX Can we optimize here?
>     # It may be that no references to the objects are present
>     # except those from our _old_objects dictionary.
>     # We should remove those. I have to learn about weak-refs!
>     for name, new_obj in module.__dict__.items():
>         key = (module.__name__, name)
>         if _old_objects.has_key(key):
>             for old_obj in _old_objects[key]:
>                 if type(new_obj) == types.ClassType:
>                     old_obj.__dict__.update(new_obj.__dict__)
>                     count += 1
>                 elif type(new_obj) == types.FunctionType:
>                     update_function(old_obj,
>                            new_obj,
>                            "func_code func_defaults func_doc".split())
>                     count += 1
>                 elif type(new_obj) == types.MethodType:
>                     update_function(old_obj.im_func,
>                            new_obj.im_func,
>                            "func_code func_defaults func_doc".split())
>                     count += 1
> ##    stop = time.clock()
> ##    print "# updated %d objects from %s" % (count, module)
> ##    print "# This took %.3f seconds" % (stop - start)
>
>     return module
>
> _watcher = ModuleWatcher()
>
> run = _watcher.run
> stop = _watcher.stop
>
> __all__ = ['run', 'stop', 'superreload']
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From stefan at sun.ac.za  Fri Apr 27 06:16:40 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Fri, 27 Apr 2007 12:16:40 +0200
Subject: [Numpy-discussion] numpy endian question
In-Reply-To: <46311822.2010206@stsci.edu>
References: <rowen-084D61.11385526042007@sea.gmane.org>
	<4630FDD9.9030307@ieee.org>
	<rowen-1D3DBF.14074226042007@sea.gmane.org>
	<46311822.2010206@stsci.edu>
Message-ID: <20070427101640.GA3761@mentat.za.net>

On Thu, Apr 26, 2007 at 05:22:42PM -0400, Christopher Hanley wrote:
> This should work as a consistent test for bigendian:
> 
> -> isBigEndian = (obj.dtype.str[0] == '>')

Is this test always safe, even on big endian machines?  Couldn't the
dtype.str[0] sometimes be '='?

Regards
St?fan


From chanley at stsci.edu  Fri Apr 27 08:49:16 2007
From: chanley at stsci.edu (Christopher Hanley)
Date: Fri, 27 Apr 2007 08:49:16 -0400
Subject: [Numpy-discussion] numpy endian question
In-Reply-To: <20070427101640.GA3761@mentat.za.net>
References: <rowen-084D61.11385526042007@sea.gmane.org>	<4630FDD9.9030307@ieee.org>	<rowen-1D3DBF.14074226042007@sea.gmane.org>	<46311822.2010206@stsci.edu>
	<20070427101640.GA3761@mentat.za.net>
Message-ID: <4631F14C.9010108@stsci.edu>

Stefan van der Walt wrote:
> On Thu, Apr 26, 2007 at 05:22:42PM -0400, Christopher Hanley wrote:
>   
>> This should work as a consistent test for bigendian:
>>
>> -> isBigEndian = (obj.dtype.str[0] == '>')
>>     
>
> Is this test always safe, even on big endian machines?  Couldn't the
> dtype.str[0] sometimes be '='?
>
> Regards
> St?fan
> _______________________________________________
>   
The test does would on big endian machines.  What wouldn't work would be 
something like this:

-> isBigEndian = (obj.dtype.byteorder == '>')


On a big-endian machine you would get '=' in this case.

Cheers,
Chris



From stefan at sun.ac.za  Fri Apr 27 08:56:40 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Fri, 27 Apr 2007 14:56:40 +0200
Subject: [Numpy-discussion] numpy endian question
In-Reply-To: <4631F14C.9010108@stsci.edu>
References: <rowen-084D61.11385526042007@sea.gmane.org>
	<4630FDD9.9030307@ieee.org>
	<rowen-1D3DBF.14074226042007@sea.gmane.org>
	<46311822.2010206@stsci.edu> <20070427101640.GA3761@mentat.za.net>
	<4631F14C.9010108@stsci.edu>
Message-ID: <20070427125640.GC3761@mentat.za.net>

On Fri, Apr 27, 2007 at 08:49:16AM -0400, Christopher Hanley wrote:
> >> -> isBigEndian = (obj.dtype.str[0] == '>')
> >
> > Is this test always safe, even on big endian machines?  Couldn't the
> > dtype.str[0] sometimes be '='?
> >   
> The test does would on big endian machines.  What wouldn't work would be 
> something like this:
> 
> -> isBigEndian = (obj.dtype.byteorder == '>')
> 
> On a big-endian machine you would get '=' in this case.

Ah, yes, I was confused.  What I meant to ask was, couldn't
dtype.str[0] sometimes be '|'?

Cheers
St?fan


From chanley at stsci.edu  Fri Apr 27 09:05:28 2007
From: chanley at stsci.edu (Christopher Hanley)
Date: Fri, 27 Apr 2007 09:05:28 -0400
Subject: [Numpy-discussion] numpy endian question
In-Reply-To: <20070427125640.GC3761@mentat.za.net>
References: <rowen-084D61.11385526042007@sea.gmane.org>	<4630FDD9.9030307@ieee.org>	<rowen-1D3DBF.14074226042007@sea.gmane.org>	<46311822.2010206@stsci.edu>
	<20070427101640.GA3761@mentat.za.net>	<4631F14C.9010108@stsci.edu>
	<20070427125640.GC3761@mentat.za.net>
Message-ID: <4631F518.9000906@stsci.edu>

Stefan van der Walt wrote:
>
> Ah, yes, I was confused.  What I meant to ask was, couldn't
> dtype.str[0] sometimes be '|'?
>
>   

That is true.   It would happen for an array of strings. 

 >>> a = n.array(['1','2','3','4'])
 >>> a.dtype.str[0]
'|'

I haven't needed to worry about that case in terms of doing byte swapping.

Chris



From Glen.Mabey at swri.org  Fri Apr 27 09:38:28 2007
From: Glen.Mabey at swri.org (Glen W. Mabey)
Date: Fri, 27 Apr 2007 08:38:28 -0500
Subject: [Numpy-discussion] TypeError when writing .data with python 2.5.1
Message-ID: <20070427133828.GE25730@bams.ccf.swri.edu>

The following works fine with a python 2.4.3 installation, but with
2.5.1 I get:


In [1]:import numpy

In [2]:fid = file( '/tmp/tfile', 'w' )

In [3]:fid.write( numpy.zeros( 55 ).data )
---------------------------------------------------------------------------
<type 'exceptions.TypeError'>             Traceback (most recent call last)

/home/gmabey/src/R9619_dev_acqlibweb/Projects/R9619_NChannelDetection/NED/<ipython console> in <module>()

<type 'exceptions.TypeError'>: non-character (or 8-bit) array cannot be interpreted as character buffer
> <ipython console>(1)<module>()


I haven't tried it with python 2.5

Glen



From Glen.Mabey at swri.org  Fri Apr 27 09:39:18 2007
From: Glen.Mabey at swri.org (Glen W. Mabey)
Date: Fri, 27 Apr 2007 08:39:18 -0500
Subject: [Numpy-discussion] TypeError when writing .data with python
	2.5.1
In-Reply-To: <20070427133828.GE25730@bams.ccf.swri.edu>
References: <20070427133828.GE25730@bams.ccf.swri.edu>
Message-ID: <20070427133918.GF25730@bams.ccf.swri.edu>

On Fri, Apr 27, 2007 at 08:38:28AM -0500, Glen W. Mabey wrote:
> The following works fine with a python 2.4.3 installation, but with
> 2.5.1 I get:
> 
> 
> In [1]:import numpy
> 
> In [2]:fid = file( '/tmp/tfile', 'w' )
> 
> In [3]:fid.write( numpy.zeros( 55 ).data )
> ---------------------------------------------------------------------------
> <type 'exceptions.TypeError'>             Traceback (most recent call last)
> 
> /home/gmabey/src/R9619_dev_acqlibweb/Projects/R9619_NChannelDetection/NED/<ipython console> in <module>()
> 
> <type 'exceptions.TypeError'>: non-character (or 8-bit) array cannot be interpreted as character buffer
> > <ipython console>(1)<module>()
> 
> 
> I haven't tried it with python 2.5


Oh, and I'm using numpy from svn, updated this morning.


From charlesr.harris at gmail.com  Fri Apr 27 09:52:16 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 27 Apr 2007 07:52:16 -0600
Subject: [Numpy-discussion] TypeError when writing .data with python
	2.5.1
In-Reply-To: <20070427133828.GE25730@bams.ccf.swri.edu>
References: <20070427133828.GE25730@bams.ccf.swri.edu>
Message-ID: <e06186140704270652q182c5f46i2a4b63ed601a7174@mail.gmail.com>

On 4/27/07, Glen W. Mabey <Glen.Mabey at swri.org> wrote:
>
> The following works fine with a python 2.4.3 installation, but with
> 2.5.1 I get:
>
>
> In [1]:import numpy
>
> In [2]:fid = file( '/tmp/tfile', 'w' )
>
> In [3]:fid.write( numpy.zeros( 55 ).data )
>
> ---------------------------------------------------------------------------
> <type 'exceptions.TypeError'>             Traceback (most recent call
> last)
>
> /home/gmabey/src/R9619_dev_acqlibweb/Projects/R9619_NChannelDetection/NED/<ipython
> console> in <module>()
>
> <type 'exceptions.TypeError'>: non-character (or 8-bit) array cannot be
> interpreted as character buffer
> > <ipython console>(1)<module>()


Does

In [1]: numpy.zeros(55).tofile('/tmp/tfile')

work?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070427/0e88d327/attachment.html>

From Glen.Mabey at swri.org  Fri Apr 27 10:07:57 2007
From: Glen.Mabey at swri.org (Glen W. Mabey)
Date: Fri, 27 Apr 2007 09:07:57 -0500
Subject: [Numpy-discussion] TypeError when writing .data with
	python	2.5.1
In-Reply-To: <e06186140704270652q182c5f46i2a4b63ed601a7174@mail.gmail.com>
References: <20070427133828.GE25730@bams.ccf.swri.edu>
	<e06186140704270652q182c5f46i2a4b63ed601a7174@mail.gmail.com>
Message-ID: <20070427140757.GB27183@bams.ccf.swri.edu>

On Fri, Apr 27, 2007 at 07:52:16AM -0600, Charles R Harris wrote:
> On 4/27/07, Glen W. Mabey <Glen.Mabey at swri.org> wrote:
> >
> >The following works fine with a python 2.4.3 installation, but with
> >2.5.1 I get:
> >
> >
> >In [1]:import numpy
> >
> >In [2]:fid = file( '/tmp/tfile', 'w' )
> >
> >In [3]:fid.write( numpy.zeros( 55 ).data )
> >
> >---------------------------------------------------------------------------
> ><type 'exceptions.TypeError'>             Traceback (most recent call
> >last)
> >
> >/home/gmabey/src/R9619_dev_acqlibweb/Projects/R9619_NChannelDetection/NED/<ipython
> >console> in <module>()
> >
> ><type 'exceptions.TypeError'>: non-character (or 8-bit) array cannot be
> >interpreted as character buffer
> >> <ipython console>(1)<module>()
> 
> 
> Does
> 
> In [1]: numpy.zeros(55).tofile('/tmp/tfile')
> 
> work?

Yes!  I was unaware of that method.  Are the two equivalent?

Also, I have identical results with python 2.5.  I'm on AMD64-Linux, by
the way.

Thanks,
Glen


From guido at python.org  Fri Apr 27 10:14:16 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 27 Apr 2007 07:14:16 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <20070427144537.T1402@tribble.ilrt.bris.ac.uk>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<f0pucb$86m$1@sea.gmane.org> <873b2n5dc7.fsf@uwo.ca>
	<20070427144537.T1402@tribble.ilrt.bris.ac.uk>
Message-ID: <ca471dc20704270714t42e6bb67pffe935854fcaa889@mail.gmail.com>

On 4/27/07, Jan Grant <jan.grant at bristol.ac.uk> wrote:
> On Thu, 26 Apr 2007, Dan Christensen wrote:
>
> > Note also that double-precision reals are a subset of the rationals,
> > since each double precision real is exactly representable as a
> > rational number, but many rational numbers are not exactly
> > representable as double precision reals.  Not sure if this means
> > that reals should be a subclass of the rationals.
>
> Not quite all: the space of doubles include a small number of things
> that aren't representable by a rational (+/- inf, for instance).

This suddenly makes me think of a new idea -- perhaps we could changes
the type of Inf and NaNs to some *other* numeric type? We could then
reserve a place in the numeric hierarchy for its abstract base class.
Though I don't know if this extends to complex numbers with one or
both parts NaN/Inf or not.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From jyasskin at gmail.com  Fri Apr 27 10:58:43 2007
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Fri, 27 Apr 2007 07:58:43 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <ca471dc20704270714t42e6bb67pffe935854fcaa889@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<f0pucb$86m$1@sea.gmane.org> <873b2n5dc7.fsf@uwo.ca>
	<20070427144537.T1402@tribble.ilrt.bris.ac.uk>
	<ca471dc20704270714t42e6bb67pffe935854fcaa889@mail.gmail.com>
Message-ID: <5d44f72f0704270758n27f3c20y8e2a33c24d8128dd@mail.gmail.com>

On 4/27/07, Guido van Rossum <guido at python.org> wrote:
> On 4/27/07, Jan Grant <jan.grant at bristol.ac.uk> wrote:
> > On Thu, 26 Apr 2007, Dan Christensen wrote:
> >
> > > Note also that double-precision reals are a subset of the rationals,
> > > since each double precision real is exactly representable as a
> > > rational number, but many rational numbers are not exactly
> > > representable as double precision reals.  Not sure if this means
> > > that reals should be a subclass of the rationals.
> >
> > Not quite all: the space of doubles include a small number of things
> > that aren't representable by a rational (+/- inf, for instance).
>
> This suddenly makes me think of a new idea -- perhaps we could changes
> the type of Inf and NaNs to some *other* numeric type? We could then
> reserve a place in the numeric hierarchy for its abstract base class.
> Though I don't know if this extends to complex numbers with one or
> both parts NaN/Inf or not.

From the Fortress spec: "The trait Q ( QQ ) encompasses all finite
rational numbers, the results of dividing any integer by any nonzero
integer. The trait Q? ( QQ_star ) is Q with two extra elements, + ?
and ?? . The trait Q# ( QQ_splat ) is Q? with one additional element,
the indefinite rational (written 0/0 ), which is used as the result of
dividing zero by zero or of adding ?? to +?."

So separating Inf and NaN into other types has some precedent.

I'd also point out that A being a subset of B doesn't make it a
subtype also. If operations on A behave differently than the
specification of the operations on B, then As aren't substitutable for
Bs and A isn't a subtype. Because doubles have finite precision and
rationals don't, I don't think doubles are a subtype of the rationals,
even if you juggle Nan/Inf to make them a subset.

Then again, doubles aren't a group either because of this imprecision,
and I'm suggesting claiming they're a subclass of that, so maybe
there's room in a practical language to make them a subclass of the
rationals too.

-- 
Namast?,
Jeffrey Yasskin

From aisaac at american.edu  Fri Apr 27 13:10:31 2007
From: aisaac at american.edu (Alan Isaac)
Date: Fri, 27 Apr 2007 13:10:31 -0400
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <5d44f72f0704270758n27f3c20y8e2a33c24d8128dd@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com><f0pucb$86m$1@sea.gmane.org>
	<873b2n5dc7.fsf@uwo.ca><20070427144537.T1402@tribble.ilrt.bris.ac.uk><ca471dc20704270714t42e6bb67pffe935854fcaa889@mail.gmail.com><5d44f72f0704270758n27f3c20y8e2a33c24d8128dd@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-2680-20070427-131031.00@CASC64043305.american.edu>

On Fri, 27 Apr 2007, Jeffrey Yasskin wrote: 
> Then again, doubles aren't a group either because of this 
> imprecision, and I'm suggesting claiming they're 
> a subclass of that, so maybe there's room in a practical 
> language to make them a subclass of the rationals too. 

Would using language from the Scheme report be useful when 
discussing this?
http://www-swiss.ai.mit.edu/projects/scheme/documentation/scheme_5.html

Cheers,
Alan Isaac






From rowen at cesmail.net  Fri Apr 27 12:33:46 2007
From: rowen at cesmail.net (Russell E. Owen)
Date: Fri, 27 Apr 2007 09:33:46 -0700
Subject: [Numpy-discussion] numpy endian question
References: <rowen-084D61.11385526042007@sea.gmane.org>
	<4630FDD9.9030307@ieee.org>
	<rowen-1D3DBF.14074226042007@sea.gmane.org>
	<46311822.2010206@stsci.edu>
Message-ID: <rowen-440D30.09334627042007@sea.gmane.org>

In article <46311822.2010206 at stsci.edu>,
 Christopher Hanley <chanley at stsci.edu> wrote:

> Russell,
> 
> This should work as a consistent test for bigendian:
> 
> -> isBigEndian = (obj.dtype.str[0] == '>')
> 
> Also, I have ported numarray's numdisplay to numpy if you would like to 
> directly display an array in DS9.  We haven't done an official release 
> yet (sometime soon) but I can forward you a copy if you are interested.

I would very much like a copy. I've never heard of numdisplay before but 
am always interested in code that can talk to DS9.

I'm porting RO.DS9. it works but has some rather ugly bits of code in it 
to deal with the many vagaries of xpa and ds9.

-- Russell



From janssen at parc.com  Thu Apr 26 13:11:27 2007
From: janssen at parc.com (Bill Janssen)
Date: Thu, 26 Apr 2007 10:11:27 PDT
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <20070425085350.6408.JCARLSON@uci.edu> 
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<20070425085350.6408.JCARLSON@uci.edu>
Message-ID: <07Apr26.101132pdt."57996"@synergy1.parc.xerox.com>

> Are the base number operations in Python all that difficult to
> understand?

Well, they're a little tricky.

But the basic problem with "number" support, in almost every
programming language, is that they are too low-level an abstraction.
A number in a program is never *just* an integer or a float.  It's
always an integer or float (or complex, OK) deployed in support of a
very specific purpose.  It's a loop index, or a weight in kilos, or
the number of calendar entries for a particular day.  It often has
units, and a fixed range of values or fixed set of values, neither of
which are taken into account in most programming languages.

Strings really suffer from the same lack of focus.  A string is never
just a string.  It's a book title, or a filename, or a person's
surname, or an email address.  It's usually got a particular human
language (or computer OS) associated with it.  Each of these usages
has particular limitations and patterns which limit both its range of
values, and the operations which can be successfully applied to it.

Bill


From janssen at parc.com  Thu Apr 26 13:26:18 2007
From: janssen at parc.com (Bill Janssen)
Date: Thu, 26 Apr 2007 10:26:18 PDT
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com> 
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
Message-ID: <07Apr26.102627pdt."57996"@synergy1.parc.xerox.com>

> Jeffrey, is there any way you can drop the top of the tree and going
> straight from Number to Complex -> Real -> Rational -> Integer? These
> are the things that everyone with high school math will know.

I think knowledge of the concepts of group, ring, and field is
supposed to be standard knowledge for any high-school senior -- isn't
this what the "new math" was all about?.  But they're pretty
fundamental to computer science; anyone trying to do serious work in
the field (that is, anyone with a reason to read the PEP) should have
a nodding acquaintance with them.

Bill



From rowen at cesmail.net  Thu Apr 26 17:17:44 2007
From: rowen at cesmail.net (Russell E. Owen)
Date: Thu, 26 Apr 2007 14:17:44 -0700
Subject: [Numpy-discussion] numpy endian question
References: <rowen-084D61.11385526042007@sea.gmane.org>
	<1177613824.2817.65.camel@localhost.localdomain>
Message-ID: <rowen-02EC4E.14174426042007@sea.gmane.org>

In article <1177613824.2817.65.camel at localhost.localdomain>,
 Francesc Altet <faltet at carabos.com> wrote:

> El dj 26 de 04 del 2007 a les 11:38 -0700, en/na Russell E. Owen va
> escriure:
> > In converting some code from numarray to numpy I had this:
> > isBigendian = (arr.isbyteswapped() != numarray.isBigEndian)
> > 
> > The only numpy version I've come up with is:
> > isBigEndian = (arr.dtype.descr[0][1][0] == '>')
> 
> isBigEndian = (arr.dtype.str[0] == '>')
> 
> is a little bit shorter. A more elegant approach could be:
> 
> isBigEndian = arr.dtype.isnative ^ numpy.little_endian

Thank you. That's just what I wanted (I had looked for the numpy version 
of numarray.isBigEndian but somehow missed it). Your other suggestion is 
also much nicer than my version, but I'll use the latter.

-- Russell


From Graham.Dumpleton at gmail.com  Fri Apr 27 07:58:58 2007
From: Graham.Dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 27 Apr 2007 11:58:58 -0000
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <ca471dc20704251358m6c06dddfga9f300b2aa1b4550@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<5d44f72f0704251304o15f2e080wcb9ade83be56403@mail.gmail.com>
	<ca471dc20704251358m6c06dddfga9f300b2aa1b4550@mail.gmail.com>
Message-ID: <1177675138.377530.48520@u32g2000prd.googlegroups.com>

On Apr 26, 6:58 am, "Guido van Rossum" <g... at python.org> wrote:
> I would like to let users modify the
> built-ins, but such changes ought to be isolated from other
> interpreters running in the same address space. I don't want to be the
> one to tell the mod_python community they won't be able to upgrade to
> 3.0...

As someone from the mod_python community, the discussion has been
noted though. :-)

FWIW, in mod_wsgi, an alternate way of hosting Python in conjunction
with Apache which is being developed and already available for early
experimentation, it will also be optionally possible to isolate
applications into separate processes potentially being run as
different users. This will still though all be configured through and
managed by Apache with it being pretty transparent to an application
and a user as to whether a specific application is running in the
Apache child process or the distinct daemon process spawned by Apache.
Since the use of a separate daemon process will probably end up being
the preferred mode of operation, especially for commercial web
hosting, loosing the ability to also run parallel applications in
distinct interpreters within the same process would only be a minor
loss. Using multiple sub interpreters already causes lots of headaches
with the simple GIL state API, inability to have different versions of
extension modules loaded etc, so much so that separate processes may
have to be the norm.

Graham



From O.Doehring at cs.ucl.ac.uk  Fri Apr 27 08:43:15 2007
From: O.Doehring at cs.ucl.ac.uk (O.Doehring at cs.ucl.ac.uk)
Date: 27 Apr 2007 13:43:15 +0100
Subject: [Numpy-discussion] Numerics vs. numpy
Message-ID: <Prayer.1.0.16.0704271343150.6130@bells3.cs.ucl.ac.uk>

Dear community,

I have a question regarding the installation process of the Biopython
toolbox ( http://biopython.org/DIST/docs/install/Installation.html#htoc14 ):

I want to make sure that Numerical Python is correctly installed under
Windows XP which it is not in my case:

>>> from Numeric import *

Traceback (most recent call last):
 File "<pyshell#0>", line 1, in <module>
   from Numeric import *
ImportError: No module named Numeric

But if I am doing:

>>> from numpy import *
>>>

it works. This is not too surprised as it gets installed under numpy in the
folder: C:\Python25\Lib\site-packages

But the problem is that if I am making various library calls to Biopython I
get an error, e.g. for handling PDB-stuff:

 File "C:\Python25\Lib\site-packages\Bio\PDB\PDBParser.py", line 9, in
<module>
   from Numeric import array, Float0
ImportError: No module named Numeric

For help I would be very grateful. Thanks.


Yours,

Orlando



From jan.grant at bristol.ac.uk  Fri Apr 27 09:47:59 2007
From: jan.grant at bristol.ac.uk (Jan Grant)
Date: Fri, 27 Apr 2007 14:47:59 +0100 (BST)
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
 Numbers (and other algebraic entities)
In-Reply-To: <873b2n5dc7.fsf@uwo.ca>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<f0pucb$86m$1@sea.gmane.org> <873b2n5dc7.fsf@uwo.ca>
Message-ID: <20070427144537.T1402@tribble.ilrt.bris.ac.uk>

On Thu, 26 Apr 2007, Dan Christensen wrote:

> Note also that double-precision reals are a subset of the rationals,
> since each double precision real is exactly representable as a
> rational number, but many rational numbers are not exactly
> representable as double precision reals.  Not sure if this means
> that reals should be a subclass of the rationals.

Not quite all: the space of doubles include a small number of things 
that aren't representable by a rational (+/- inf, for instance).




-- 
jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/
Tel +44 (0)117 3317661   http://ioctl.org/jan/
Spreadsheet through network. Oh yeah.


From robert.kern at gmail.com  Fri Apr 27 14:21:21 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 27 Apr 2007 13:21:21 -0500
Subject: [Numpy-discussion] Numerics vs. numpy
In-Reply-To: <Prayer.1.0.16.0704271343150.6130@bells3.cs.ucl.ac.uk>
References: <Prayer.1.0.16.0704271343150.6130@bells3.cs.ucl.ac.uk>
Message-ID: <46323F21.9020705@gmail.com>

O.Doehring at cs.ucl.ac.uk wrote:
> Dear community,
> 
> I have a question regarding the installation process of the Biopython
> toolbox ( http://biopython.org/DIST/docs/install/Installation.html#htoc14 ):
> 
> I want to make sure that Numerical Python is correctly installed under
> Windows XP which it is not in my case:
> 
>>>> from Numeric import *
> 
> Traceback (most recent call last):
>  File "<pyshell#0>", line 1, in <module>
>    from Numeric import *
> ImportError: No module named Numeric

BioPython still requires Numeric, the predecessor to numpy. Here is a page
describing the history of these packages:

  http://www.scipy.org/History_of_SciPy

You can download Numeric from here:

  http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=1351

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From David.L.Goldsmith at noaa.gov  Fri Apr 27 14:25:15 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Fri, 27 Apr 2007 11:25:15 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
 Numbers (and other algebraic entities)
In-Reply-To: <07Apr26.102627pdt.57996@synergy1.parc.xerox.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<07Apr26.102627pdt.57996@synergy1.parc.xerox.com>
Message-ID: <4632400B.2030603@noaa.gov>

Bill Janssen wrote:
>> Jeffrey, is there any way you can drop the top of the tree and going
>> straight from Number to Complex -> Real -> Rational -> Integer? These
>> are the things that everyone with high school math will know.
>>     
>
> I think knowledge of the concepts of group, ring, and field is
> supposed to be standard knowledge for any high-school senior -- isn't
> this what the "new math" was all about?.  
Nah, AFAIK, not really.  Of course, for those who remember Tom Lehrer, 
New Math was about learning that one could base number representation on 
bases other than 10 (even more fundamental to CS, no?), but I think it 
was rather more about broadening the standard curriculum to include more 
than just arithmetic drill, i.e., to more prominently include things 
like problem solving skills, (somewhat) more abstraction, etc.  Along 
the latter lines, I do remember learning (and tutoring), by name, things 
like the symmetric, reflexive, transitive, associative, commutative, and 
distributive properties, but I didn't hear/see the words group, ring, or 
field (in a math context) 'til I got to college and was formally 
introduced to the subject of Linear Algebra (and lest you think my HS 
math curriculum was deficient and/or non-standard, I don't want to go in 
that direction, but trust me, it was more than sufficient and it was 
non-standard, but in the positive direction, not in the negative, and I 
graduated 1st in my class in math).  As far as the scientific 
disciplines which one might reasonably expect a college major therein to 
have at least a "nodding acquaintance" w/ the classes of Abstract 
Algebra, certainly Math (and hopefully Appl. Math and Stats), probably 
Physics, perhaps Chemistry, CS if you say so, but even Engineering I'd 
say you're beginning to tread in uncertain territory, and Biology, etc., 
unless taken with a large dose of formal mathematics, fuhget-about-it.

DG
> But they're pretty
> fundamental to computer science; anyone trying to do serious work in
> the field (that is, anyone with a reason to read the PEP) should have
> a nodding acquaintance with them.
>
> Bill
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From David.L.Goldsmith at noaa.gov  Fri Apr 27 14:53:52 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Fri, 27 Apr 2007 11:53:52 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
 Numbers (and other algebraic entities)
In-Reply-To: <ca471dc20704270714t42e6bb67pffe935854fcaa889@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<f0pucb$86m$1@sea.gmane.org> <873b2n5dc7.fsf@uwo.ca>
	<20070427144537.T1402@tribble.ilrt.bris.ac.uk>
	<ca471dc20704270714t42e6bb67pffe935854fcaa889@mail.gmail.com>
Message-ID: <463246C0.6060400@noaa.gov>

Guido van Rossum wrote:
> On 4/27/07, Jan Grant <jan.grant at bristol.ac.uk> wrote:
>   
>> On Thu, 26 Apr 2007, Dan Christensen wrote:
>>
>>     
>>> Note also that double-precision reals are a subset of the rationals,
>>> since each double precision real is exactly representable as a
>>> rational number, but many rational numbers are not exactly
>>> representable as double precision reals.  Not sure if this means
>>> that reals should be a subclass of the rationals.
>>>       
>> Not quite all: the space of doubles include a small number of things
>> that aren't representable by a rational (+/- inf, for instance).
>>     
>
> This suddenly makes me think of a new idea -- perhaps we could changes
> the type of Inf and NaNs to some *other* numeric type? We could then
> reserve a place in the numeric hierarchy for its abstract base class.
> Though I don't know if this extends to complex numbers with one or
> both parts NaN/Inf or not.
>   
As far as NaN is concerned, I'd assert that it's no different if we're 
talking about reals or complex - if it's not a number, it's not a number 
- IMO a complex number with one part finite and another part NaN makes 
no sense.  And as far as inf is concerned, *if* you're effectively doing 
your math on the Riemann sphere (or any Riemann surface with countably 
many cuts, if I remember correctly - someone please correct me if I'm 
wrong) then there is only one inf - if you need to know the direction 
from, or, equivalently, the curve along which you arrived at inf, then 
you need to keep track of that programmatically.  (Of course, there 
might be some instances where you want to know whether an operation made 
the real part or the imaginary part or both NaN or inf, but then I'd 
assert that you should be checking for that programmatically also.)

DG


From anton.sherwood at gmail.com  Fri Apr 27 14:59:07 2007
From: anton.sherwood at gmail.com (Anton Sherwood)
Date: Fri, 27 Apr 2007 11:59:07 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <20070427144537.T1402@tribble.ilrt.bris.ac.uk>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<f0pucb$86m$1@sea.gmane.org> <873b2n5dc7.fsf@uwo.ca>
	<20070427144537.T1402@tribble.ilrt.bris.ac.uk>
Message-ID: <fb420b880704271159l7d55a30fn8273b6b6b5cf70eb@mail.gmail.com>

Jan Grant wrote:
> . . .  the space of doubles include a small number of things
> that aren't representable by a rational (+/- inf, for instance).

+1/0, -1/0

-- 
Anton Sherwood, http://www.ogre.nu/


From David.L.Goldsmith at noaa.gov  Fri Apr 27 15:06:20 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Fri, 27 Apr 2007 12:06:20 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
 Numbers (and other algebraic entities)
In-Reply-To: <fb420b880704271159l7d55a30fn8273b6b6b5cf70eb@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<f0pucb$86m$1@sea.gmane.org> <873b2n5dc7.fsf@uwo.ca>
	<20070427144537.T1402@tribble.ilrt.bris.ac.uk>
	<fb420b880704271159l7d55a30fn8273b6b6b5cf70eb@mail.gmail.com>
Message-ID: <463249AC.8050009@noaa.gov>

Anton Sherwood wrote:
> Jan Grant wrote:
>   
>> . . .  the space of doubles include a small number of things
>> that aren't representable by a rational (+/- inf, for instance).
>>     
>
> +1/0, -1/0
>
>   
Are not unique representations for inf/-inf, (if that matters).

DG


From anton.sherwood at gmail.com  Fri Apr 27 15:17:34 2007
From: anton.sherwood at gmail.com (Anton Sherwood)
Date: Fri, 27 Apr 2007 12:17:34 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <463249AC.8050009@noaa.gov>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<f0pucb$86m$1@sea.gmane.org> <873b2n5dc7.fsf@uwo.ca>
	<20070427144537.T1402@tribble.ilrt.bris.ac.uk>
	<fb420b880704271159l7d55a30fn8273b6b6b5cf70eb@mail.gmail.com>
	<463249AC.8050009@noaa.gov>
Message-ID: <fb420b880704271217i72f716ach456b7a177dc442f1@mail.gmail.com>

> > Jan Grant wrote:
> >> . . .  the space of doubles include a small number of things
> >> that aren't representable by a rational (+/- inf, for instance).

> Anton Sherwood wrote:
> > +1/0, -1/0

David Goldsmith wrote:
> Are not unique representations for inf/-inf, (if that matters).

No rational representation is unique!

-- 
Anton Sherwood, http://www.ogre.nu/


From bsouthey at gmail.com  Fri Apr 27 15:45:02 2007
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 27 Apr 2007 14:45:02 -0500
Subject: [Numpy-discussion] Numerics vs. numpy
Message-ID: <bbcd77d00704271245tbf72937p8ec68c8f6d9352dd@mail.gmail.com>

Hi,
Ed Schofield has created a patch for NumPy to work with BioPython (see below).

Bruce

---------- Forwarded message ----------
From: bugzilla-daemon at portal.open-bio.org <bugzilla-daemon at portal.open-bio.org>
Date: Mar 27, 2007 8:08 AM
Subject: [Biopython-dev] [Bug 2251] New: [PATCH] NumPy support for BioPython
To: biopython-dev at biopython.org


http://bugzilla.open-bio.org/show_bug.cgi?id=2251

           Summary: [PATCH] NumPy support for BioPython
           Product: Biopython
           Version: Not Applicable
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Main Distribution
        AssignedTo: biopython-dev at biopython.org
        ReportedBy: edschofield at gmail.com


The following patch adds support for NumPy in addition to Numeric. It does so
with a thin wrapper layer at the C level and Python level, similar in purpose
to (but less ambitious than) the numerix wrapper layer used by matplotlib.

The patch is designed to prevent imports of incompatible compiled C modules in
the case one installs Numeric after installing BioPython with NumPy support.

It also changes the following:
- C #include statements
- Python import statements
- references to obsoleted function names
- the width of the dimension data types in cluster.c from int to intp (for
64-bit architectures)
- the distutils setup.py file to supply the correct NumPy header locations.
- the documentation (updating references to NumPy)

It also fixes array boolean operators in MarkovModel.py, which were silently
broken before.

It applies to BioPython 1.43, as follows:

$ tar xzvf biopython-1.43.tar.gz
$ cd biopython-1.43
$ patch -p1 < /path/to/biopython-1.43-numpy-support-v5.patch


--
Configure bugmail: http://bugzilla.open-bio.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
_______________________________________________
Biopython-dev mailing list
Biopython-dev at lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/biopython-dev


From Chris.Barker at noaa.gov  Fri Apr 27 16:12:43 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 27 Apr 2007 13:12:43 -0700
Subject: [Numpy-discussion] numpy endian question
In-Reply-To: <46313A5A.7030301@ieee.org>
References: <rowen-084D61.11385526042007@sea.gmane.org>
	<4630FDD9.9030307@ieee.org> <rowen-1D3DBF.14074226042007@sea.gmane.org>
	<46313A5A.7030301@ieee.org>
Message-ID: <4632593B.6000806@noaa.gov>


Is there really no single method to call on an ndarray that asks: "what 
endian are you"

I know not every two-liner should be made into a convenience method, but 
this seems like a good candidate to me.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From jks at iki.fi  Fri Apr 27 16:16:28 2007
From: jks at iki.fi (=?iso-8859-1?Q?Jouni_K=2E_Sepp=E4nen?=)
Date: Fri, 27 Apr 2007 23:16:28 +0300
Subject: [Numpy-discussion] take(..., out=something) does not seem to work
Message-ID: <m2vefho9nn.fsf@iki.fi>

Hi,

Either I have completely misunderstood what the out= arg to the take
function is supposed to do, or there is a bug in numpy 1.0.2, as
demonstrated below.

First, some initialization:

    In [1]: from numpy import *

    In [2]: a=arange(12).reshape((3,4))

    In [3]: a
    Out[3]: 
    array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11]])

Let's try take without an output argument:

    In [4]: take(a, [0,2], axis=1)
    Out[4]: 
    array([[ 0,  2],
           [ 4,  6],
           [ 8, 10]])

Good. Now let's create an array of the right size and dtype, and use
it for output:

    In [5]: b = zeros_like(_)

    In [6]: take(a, [0,2], axis=1, out=b)
    Out[6]: 
    array([[ 0,  2],
           [ 4,  6],
           [ 8, 10]])

    In [7]: b
    Out[7]: 
    array([[0, 0],
           [0, 0],
           [0, 0]])

It seems that b was not affected at all.

-- 
Jouni K. Sepp?nen
http://www.iki.fi/jks



From David.L.Goldsmith at noaa.gov  Fri Apr 27 16:29:45 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Fri, 27 Apr 2007 13:29:45 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
 Numbers (and other algebraic entities)
In-Reply-To: <fb420b880704271217i72f716ach456b7a177dc442f1@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<f0pucb$86m$1@sea.gmane.org> <873b2n5dc7.fsf@uwo.ca>
	<20070427144537.T1402@tribble.ilrt.bris.ac.uk>
	<fb420b880704271159l7d55a30fn8273b6b6b5cf70eb@mail.gmail.com>
	<463249AC.8050009@noaa.gov>
	<fb420b880704271217i72f716ach456b7a177dc442f1@mail.gmail.com>
Message-ID: <46325D39.4040506@noaa.gov>

Anton Sherwood wrote:
>>> Jan Grant wrote:
>>>       
>>>> . . .  the space of doubles include a small number of things
>>>> that aren't representable by a rational (+/- inf, for instance).
>>>>         
>
>   
>> Anton Sherwood wrote:
>>     
>>> +1/0, -1/0
>>>       
>
> David Goldsmith wrote:
>   
>> Are not unique representations for inf/-inf, (if that matters).
>>     
>
> No rational representation is unique!
>
>   
Of course - as soon as I hit send I knew someone was going to shoot that 
back at me! :-)  But for some reason (I'm speaking rhetorically, I know 
the reasons) they always drilled home non-zero/zero undef., not = inf.


DG


From oliphant.travis at ieee.org  Fri Apr 27 16:46:41 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Fri, 27 Apr 2007 14:46:41 -0600
Subject: [Numpy-discussion] take(...,
 out=something) does not seem to work
In-Reply-To: <m2vefho9nn.fsf@iki.fi>
References: <m2vefho9nn.fsf@iki.fi>
Message-ID: <46326131.6060406@ieee.org>

Jouni K. Sepp?nen wrote:
> Hi,
>
> Either I have completely misunderstood what the out= arg to the take
> function is supposed to do, or there is a bug in numpy 1.0.2, as
> demonstrated below.
>   

Thanks. Silly bug in C-code caused this.

-Travis



From sberub at gmail.com  Fri Apr 27 16:46:02 2007
From: sberub at gmail.com (Simon Berube)
Date: Fri, 27 Apr 2007 20:46:02 -0000
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <8A1A9308-ECB6-4035-9938-046CC9D2E0C4@cs.hmc.edu>
References: <f0o4h9$k5m$1@sea.gmane.org><462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no> <462FAC94.1040207@gmail.com>
	<e86a5fd00704251246h1fbe4003v4adf28376bd2ea54@mail.gmail.com>
	<B11B844B-3741-422C-B90C-9E43F7E9B279@cs.hmc.edu>
	<463058F6.3030602@iki.fi>
	<8A1A9308-ECB6-4035-9938-046CC9D2E0C4@cs.hmc.edu>
Message-ID: <1177706762.398429.93030@s33g2000prh.googlegroups.com>

Actually, all the points in this thread have been very good points but
I just want to add my 2 cents since I was in your boat just a month
ago and decided to try out python after seeing what it could do.

In short, I decided to look elsewhere as I was dealing with vector
data in database and maintaining so many different function and files
was getting hard and while Matlab has classes they are very unwieldy
to use and it is such an oddity in matlab that using them would
confuse whoever uses your code anyway. At least in my experience.

However after playing with Python, matplotlib and numpy I have found
that the code is much cleaner, the language is much more powerful and
clean, matplotlib is extremely capable for 2D plotting and iPython
integrates all those things very well.

The only thing I miss about matlab are the help tool which is quite
frankly the best I have seen, the many different toolboxes, the
compatibility with most hardware DAC and just the fact damn near
everyone in the scientific community uses either Matlab or plain
Fortran/C/C++.

However, I would disagree that Python with all its tools going to
replace Matlab well for everything. For large projects, for advanced
programmers and for non-standard things such as complex database
handling (in my case) it is definitly a clear winner. However, I would
be weary of getting Matlab completely out of the picture because I
find it is still a much better tool for algorithm testing, scripting
and other "use for a week" functions .

Hope this helps a bit too.

On Apr 27, 1:23 am, belinda thom <b... at cs.hmc.edu> wrote:
> Thank you so much! When I finally get a moment to take a break, I'll
> look in more detail into using your suggestion.
>
> --b
>
> On Apr 26, 2007, at 12:47 AM, Pauli Virtanen wrote:
>
>
>
> > belinda thom kirjoitti:
> >> On Apr 25, 2007, at 12:46 PM, Bill Baxter wrote:
>
> >> Agree w/most of what you've said, but will add one other thing that
> >> drives me nuts in python that hasn't been a problem in Matplotlib:
>
> >> In Python, if interacting w/the interpreter as your primary IDE, and
> >> if you've got multiple files that depend on one another that you're
> >> modifying, then you need to restart the interpreter frequently b/c
> >> otherwise things in the interpreter can be stale; IOW, changes to
> >> several interdependent files aren't easy to import so that everything
> >> in your interpreted environment reflects the latest code. Yeah,
> >> there's reload tricks, but doing them in the right order and the
> >> right number of times can be a pain when dependencies are cyclic.
>
> >> I realize in general that this issue of stale code is just difficult,
> >> that its not inherently a Python problem per se, for automatic
> >> percolation of code changes backwards in time is difficult in
> >> general, but I've never had the problem bite me when I was developing
> >> in Matlab. I just save whatever file, and it appears that whatever's
> >> latest on disk is what's executed. (Friends who know more about PL
> >> than I tell me I've been lucky.)
>
> > I've been using the attached autoreload code (by Thomas Heller,
> > different versions are probably floating around the net) rather
> > successfully for a more Matlabish feel to IPython. It reloads modules
> > every time their files have been changed.
>
> > Of course, this doesn't solve all staleness problems you describe, but
> > only eliminates manual work involved in solving the most common ones.
> > Reloading modules does have pitfalls [1], but at least in my use these
> > haven't really mattered much in practice.
>
> > I think having autoreload functionality bundled with IPython (maybe
> > turned off per default) would be quite useful: although in some rare
> > cases autoreloading doesn't work in the ideal way, it's very
> > convenient
> > not to have to type reload(foo) after every change, especially when
> > tweaking for example plotting scripts.
>
> >    Pauli
>
> > [1] For example, instances of classes created are not updated on
> > reload,
> > but continue using the old code. Also, from foo import * imports
> > are not
> > updated, so you'll have to manually touch files that do this if 'foo'
> > has been changed.
> > """
>
> > autoreload.py - automatically reload changed source
> > code into a running program
>
> > You might want to add the following to your ~/.ipython/ipythonrc
>
> >     import_mod sys
> >     execute sys.path.append('path/where/this/file/resides')
> >     import_mod autoreload
> >     execute autoreload.run()
>
> > or adding the following
>
> >     import sys
> >     sys.path.append('path/where/this/file/resides')
> >     import autoreload
> >     autoreload.run()
>
> > to some startup file.
>
> > Created: Thomas Heller, 2000-04-17
> > Modified: Pauli Virtanen, 2006
> > """
>
> > # $Id: autoreload.py 3117 2006-09-27 20:28:46Z pauli $
> > #
> > # $Log: autoreload.py,v $
> > # Revision 1.9  2001/11/15 18:41:18  thomas
> > # Cleaned up and made working again before posting to c.l.p.
> > # Added code to update bound (or unbound) methods as suggested
> > # by Just van Rossum. Thanks!
> > #
> > # ...
> > #
> > # Revision 1.1  2001/10/04 16:54:04  thomas
> > # Discovered this old module on my machine, it didn't work too well,
> > # but seems worth to continue with it...
> > # Checked in as a first step.
>
> > __version__ = "$Revision: 1.9 $".split()[1]
>
> > # ToDo:
> > #
> > #  Cannot reload __main__ - explain why this cannot work,
> > #  and explain a workaround.
> > #
> > #  Optimize - the number of watches objects (in old_objects)
> > #  grows without limits. Think if this is really necessary...
>
> > import time, os, threading, sys, types, imp, inspect, traceback
>
> > def _get_compiled_ext():
> >     for ext, mode, typ in imp.get_suffixes():
> >         if typ == imp.PY_COMPILED:
> >             return ext
>
> > # the official way to get the extension of compiled files (.pyc
> > or .pyo)
> > PY_COMPILED_EXT = _get_compiled_ext()
>
> > class ModuleWatcher:
> >     running = 0
> >     def __init__(self):
> >         # If we don't do this, there may be tracebacks
> >         # when shutting down python.
> >         import atexit
> >         atexit.register(self.stop)
>
> >     def run(self):
> >         if self.running:
> >             print "# autoreload already running"
> >             return
> >         print "# starting autoreload"
> >         self.running = 1
> >         self.thread = threading.Thread(target=self._check_modules)
> >         self.thread.setDaemon(1)
> >         self.thread.start()
>
> >     def stop(self):
> >         if not self.running:
> >             print "# autoreload not running"
> >             return
> >         self.running = 0
> >         self.thread.join()
> >         #print "# autoreload stopped"
>
> >     def _check_modules(self):
> >         skipped = {}
> >         while self.running:
> >             time.sleep(0.01)
> >             for m in sys.modules.values():
> >                 if not hasattr(m, '__file__'):
> >                     continue
> >                 if m.__name__ == '__main__':
>
> >                     # we cannot reload(__main__) First I thought we
> >                     # could use mod = imp.load_module() and then
> >                     # reload(mod) to simulate reload(main), but this
> >                     # would execute the code in __main__ a second
> >                     # time.
>
> >                     continue
> >                 file = m.__file__
> >                 dirname = os.path.dirname(file)
> >                 path, ext = os.path.splitext(file)
>
> >                 if ext.lower() == '.py':
> >                     ext = PY_COMPILED_EXT
> >                     file = os.path.join(dirname, path +
> > PY_COMPILED_EXT)
>
> >                 if ext != PY_COMPILED_EXT:
> >                     continue
>
> >                 try:
> >                pymtime = os.stat(file[:-1])[8]
> >                     if pymtime <= os.stat(file)[8]:
> >                         continue
> >                if skipped.get(file[:-1], None) == pymtime:
> >                    continue
> >                 except OSError:
> >                     continue
>
> >                 try:
> >                     superreload(m)
> >                if file[:-1] in skipped:
> >                    del skipped[file[:-1]]
> >                 except:
> >                skipped[file[:-1]] = pymtime
> >                     import traceback
> >                     traceback.print_exc(0)
>
> > def update_function(old, new, attrnames):
> >     for name in attrnames:
> >         setattr(old, name, getattr(new, name))
>
> > def superreload(module,
> >                 reload=reload,
> >                 _old_objects = {}):
> >     """superreload (module) -> module
>
> >     Enhanced version of the builtin reload function.
> >     superreload replaces the class dictionary of every top-level
> >     class in the module with the new one automatically,
> >     as well as every function's code object.
> >     """
> > ##    start = time.clock()
> >     # retrieve the attributes from the module before the reload,
> >     # and remember them in _old_objects.
> >     for name, object in module.__dict__.items():
> >         key = (module.__name__, name)
> >         _old_objects.setdefault(key, []).append(object)
> >         # print the refcount of old objects:
> > ##        if type(object) in (types.FunctionType, types.ClassType):
> > ##            print name, map(sys.getrefcount, _old_objects[key])
>
> > ##    print "# reloading module %r" % module
>
> >     module = reload(module)
> >     # XXX We have a problem here if importing the module fails!
>
> >     # iterate over all objects and update them
> >     count = 0
> >     # XXX Can we optimize here?
> >     # It may be that no references to the objects are present
> >     # except those from our _old_objects dictionary.
> >     # We should remove those. I have to learn about weak-refs!
> >     for name, new_obj in module.__dict__.items():
> >         key = (module.__name__, name)
> >         if _old_objects.has_key(key):
> >             for old_obj in _old_objects[key]:
> >                 if type(new_obj) == types.ClassType:
> >                     old_obj.__dict__.update(new_obj.__dict__)
> >                     count += 1
> >                 elif type(new_obj) == types.FunctionType:
> >                     update_function(old_obj,
> >                            new_obj,
> >                            "func_code func_defaults func_doc".split())
> >                     count += 1
> >                 elif type(new_obj) == types.MethodType:
> >                     update_function(old_obj.im_func,
> >                            new_obj.im_func,
> >                            "func_code func_defaults func_doc".split())
> >                     count += 1
> > ##    stop = time.clock()
> > ##    print "# updated %d objects from %s" % (count, module)
> > ##    print "# This took %.3f seconds" % (stop - start)
>
> >     return module
>
> > _watcher = ModuleWatcher()
>
> > run = _watcher.run
> > stop = _watcher.stop
>
> > __all__ = ['run', 'stop', 'superreload']
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discuss... at scipy.org
> >http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion



From rowen at cesmail.net  Fri Apr 27 17:17:34 2007
From: rowen at cesmail.net (Russell E. Owen)
Date: Fri, 27 Apr 2007 14:17:34 -0700
Subject: [Numpy-discussion] Compact way of performing array math with
	specified result type?
Message-ID: <rowen-23D294.14173327042007@sea.gmane.org>

I often find myself doing simple math on sequences of numbers (which 
might or might not be numpy arrays) where I want the result (and thus 
the inputs) coerced to a particular data type.

I'd like to be able to say:

  numpy.divide(seq1, seq2, dtype=float)

but ufuncs don't allow on to specify a result type. So I do this instead:

  numpy.array(seq1, dtype=float) / numpy.array(seq2, dtype=float)

Is there a more compact solution (without having to create the result 
array first and supply it as an argument)?

-- Russell



From robert.kern at gmail.com  Fri Apr 27 17:21:50 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 27 Apr 2007 16:21:50 -0500
Subject: [Numpy-discussion] Compact way of performing array math with
 specified result type?
In-Reply-To: <rowen-23D294.14173327042007@sea.gmane.org>
References: <rowen-23D294.14173327042007@sea.gmane.org>
Message-ID: <4632696E.6000701@gmail.com>

Russell E. Owen wrote:
> I often find myself doing simple math on sequences of numbers (which 
> might or might not be numpy arrays) where I want the result (and thus 
> the inputs) coerced to a particular data type.
> 
> I'd like to be able to say:
> 
>   numpy.divide(seq1, seq2, dtype=float)
> 
> but ufuncs don't allow on to specify a result type. So I do this instead:
> 
>   numpy.array(seq1, dtype=float) / numpy.array(seq2, dtype=float)
> 
> Is there a more compact solution (without having to create the result 
> array first and supply it as an argument)?

def fasarray(seq):
    return numpy.asarray(seq, dtype=float)

fasarray(seq1) / fasarray(seq2)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From spacey-numpy-discussion at lenin.net  Fri Apr 27 18:27:47 2007
From: spacey-numpy-discussion at lenin.net (Peter C. Norton)
Date: Fri, 27 Apr 2007 15:27:47 -0700
Subject: [Numpy-discussion] Building numpy - setting the run path
Message-ID: <20070427222747.GH22477@lenin.net>

Building numpy for my company's solaris distribution involves requring
a run_path for the lapack+blas libraries we're using (libsunperf,
though I'm considering swapping out for gsl since we may use that).

The situation we're in is that we need gcc to get the -R for paths
like /usr/local/gnu/lib /usr/local/python2.5.1/lib, etc. but the
default python distutils raise a not implemented exception on this.

Since it doesn't appear that the numpy distutils subclass the
ccompiler, my solution is this for now (in numpy/distutils/ccomiler):

def gen_lib_options(compiler, library_dirs, runtime_library_dirs, libraries):
    r = _distutils_gen_lib_options(compiler, library_dirs,
                                   [], libraries)
    if len(runtime_library_dirs) > 0:
        r.extend([ "-R%s" % lib for lib in runtime_library_dirs ])
    lib_opts = []
    for i in r:
        if is_sequence(i):
            lib_opts.extend(list(i))
        else:
            lib_opts.append(i)
    return lib_opts


This allows me to specify -R options on the command line with
build_ext without destroying the build. I'm pretty sure that "-R"
should be replaced with a lookup somewhere for the
platform-appropriate way to do this, but this does work.

Let me know if there's a better way to do this, but if not can this be
incorporated into numpy's distutils?

Thanks,

-Peter

-- 
The 5 year plan:
In five years we'll make up another plan.
Or just re-use this one.



From fullung at gmail.com  Fri Apr 27 19:45:54 2007
From: fullung at gmail.com (Albert Strasheim)
Date: Sat, 28 Apr 2007 01:45:54 +0200
Subject: [Numpy-discussion] Building numpy - setting the run path
References: <20070427222747.GH22477@lenin.net>
Message-ID: <001401c78926$32571980$0100a8c0@sun.ac.za>

Hello

----- Original Message ----- 
From: "Peter C. Norton" <spacey-numpy-discussion at lenin.net>
To: <numpy-discussion at scipy.org>
Sent: Saturday, April 28, 2007 12:27 AM
Subject: [Numpy-discussion] Building numpy - setting the run path


> Building numpy for my company's solaris distribution involves requring
> a run_path for the lapack+blas libraries we're using (libsunperf,
> though I'm considering swapping out for gsl since we may use that).
>
> The situation we're in is that we need gcc to get the -R for paths
> like /usr/local/gnu/lib /usr/local/python2.5.1/lib, etc. but the
> default python distutils raise a not implemented exception on this.

On Linux I've set the LD_RUN_PATH environment variable prior to compiling my 
programs. It seemed to do the trick. I don't know if this works on Solaris, 
though. A quick Google seems to indicate that LD_RUN_PATH is a GCC thing.

Cheers,

Albert 



From david at ar.media.kyoto-u.ac.jp  Sat Apr 28 03:20:30 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 28 Apr 2007 16:20:30 +0900
Subject: [Numpy-discussion] Building numpy - setting the run path
In-Reply-To: <20070427222747.GH22477@lenin.net>
References: <20070427222747.GH22477@lenin.net>
Message-ID: <4632F5BE.8070202@ar.media.kyoto-u.ac.jp>

Peter C. Norton wrote:
> Building numpy for my company's solaris distribution involves requring
> a run_path for the lapack+blas libraries we're using (libsunperf,
> though I'm considering swapping out for gsl since we may use that).
>
> The situation we're in is that we need gcc to get the -R for paths
> like /usr/local/gnu/lib /usr/local/python2.5.1/lib, etc. but the
> default python distutils raise a not implemented exception on this.
May I ask what the -R path option is doing ? I cannot find it in the doc 
of my gcc (on linux), and from your email, I don't see what it is doing 
? Does it say where to find library when linking ? When running ? Does 
it set the patch in the binary ?

David


From matthew.brett at gmail.com  Sat Apr 28 07:09:31 2007
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 28 Apr 2007 12:09:31 +0100
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <1177706762.398429.93030@s33g2000prh.googlegroups.com>
References: <f0o4h9$k5m$1@sea.gmane.org> <462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no> <462FAC94.1040207@gmail.com>
	<e86a5fd00704251246h1fbe4003v4adf28376bd2ea54@mail.gmail.com>
	<B11B844B-3741-422C-B90C-9E43F7E9B279@cs.hmc.edu>
	<463058F6.3030602@iki.fi>
	<8A1A9308-ECB6-4035-9938-046CC9D2E0C4@cs.hmc.edu>
	<1177706762.398429.93030@s33g2000prh.googlegroups.com>
Message-ID: <1e2af89e0704280409u3f6a7d99j2a7283da373dced3@mail.gmail.com>

Hi,

> However, I would disagree that Python with all its tools going to
> replace Matlab well for everything. For large projects, for advanced
> programmers and for non-standard things such as complex database
> handling (in my case) it is definitly a clear winner. However, I would
> be weary of getting Matlab completely out of the picture because I
> find it is still a much better tool for algorithm testing, scripting
> and other "use for a week" functions .

Thanks - that seems very fair.  I think you're right that the level of
help for matlab is far superior, and easier to get to, than numpy /
python.  Can you say anything more about the features of matlab that
you miss for the 'use a week' functions?  It might help guide our
thoughts on potential improvements...

Thanks a lot,

Matthew


From matthew.brett at gmail.com  Sat Apr 28 07:16:45 2007
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 28 Apr 2007 12:16:45 +0100
Subject: [Numpy-discussion] numpy endian question
In-Reply-To: <4632593B.6000806@noaa.gov>
References: <rowen-084D61.11385526042007@sea.gmane.org>
	<4630FDD9.9030307@ieee.org>
	<rowen-1D3DBF.14074226042007@sea.gmane.org>
	<46313A5A.7030301@ieee.org> <4632593B.6000806@noaa.gov>
Message-ID: <1e2af89e0704280416r34586db7ob9d6bd2af2336d51@mail.gmail.com>

On 4/27/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
>
> Is there really no single method to call on an ndarray that asks: "what
> endian are you"
>
> I know not every two-liner should be made into a convenience method, but
> this seems like a good candidate to me.

+1

I came across this source of minor confusion and excess code lines
when writing the matlab io module for scipy.

dtype.isbigendian?

Matthew


From brussel13 at yahoo.com  Sat Apr 28 12:43:27 2007
From: brussel13 at yahoo.com (Ross Harder)
Date: Sat, 28 Apr 2007 09:43:27 -0700 (PDT)
Subject: [Numpy-discussion] fft and roll
Message-ID: <759928.91618.qm@web55901.mail.re3.yahoo.com>

Hi,
I'm new to Numpy, just bought the guide last week.
 
I've been disappointed by the difficulty I'm having
finding functions that are documented in the guide. 
So
far I've had to spend a lot of time tracking down that
the fft2 and fftn functions from the fftpack library,
which are documented in the numpy guide, are actually
in scipy.  I've also not been able to find the roll
function documented on page 99 of the guide.  

Did I just buy the guide at a time when it's
particularly out of date and there will be an update
soon?

Ross

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


From robert.kern at gmail.com  Sat Apr 28 12:51:24 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 28 Apr 2007 11:51:24 -0500
Subject: [Numpy-discussion] fft and roll
In-Reply-To: <759928.91618.qm@web55901.mail.re3.yahoo.com>
References: <759928.91618.qm@web55901.mail.re3.yahoo.com>
Message-ID: <46337B8C.4080102@gmail.com>

Ross Harder wrote:
> Hi,
> I'm new to Numpy, just bought the guide last week.
>  
> I've been disappointed by the difficulty I'm having
> finding functions that are documented in the guide. 
> So
> far I've had to spend a lot of time tracking down that
> the fft2 and fftn functions from the fftpack library,
> which are documented in the numpy guide, are actually
> in scipy.  I've also not been able to find the roll
> function documented on page 99 of the guide.  
> 
> Did I just buy the guide at a time when it's
> particularly out of date and there will be an update
> soon?

Your numpy might be (very) out of date. All of those functions are in numpy 1.0
and later (at least).

In [5]: import numpy

In [6]: numpy.fft.fft2
Out[6]: <function fft2 at 0x17954f0>

In [7]: numpy.fft.fftn
Out[7]: <function fftn at 0x1795470>

In [8]: numpy.roll
Out[8]: <function roll at 0x1475170>


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From brussel13 at yahoo.com  Sat Apr 28 13:19:23 2007
From: brussel13 at yahoo.com (Ross Harder)
Date: Sat, 28 Apr 2007 10:19:23 -0700 (PDT)
Subject: [Numpy-discussion] fft and roll
In-Reply-To: <46337B8C.4080102@gmail.com>
Message-ID: <810157.5642.qm@web55914.mail.re3.yahoo.com>


Ahhh... that hadn't occured to me.  Just installed
Enthon and assumed it was up to date, but it's not.

Sorry for the misguided complaint.

Thanks,
Ross



--- Robert Kern <robert.kern at gmail.com> wrote:

> Ross Harder wrote:
> > Hi,
> > I'm new to Numpy, just bought the guide last week.
> >  
> > I've been disappointed by the difficulty I'm
> having
> > finding functions that are documented in the
> guide. 
> > So
> > far I've had to spend a lot of time tracking down
> that
> > the fft2 and fftn functions from the fftpack
> library,
> > which are documented in the numpy guide, are
> actually
> > in scipy.  I've also not been able to find the
> roll
> > function documented on page 99 of the guide.  
> > 
> > Did I just buy the guide at a time when it's
> > particularly out of date and there will be an
> update
> > soon?
> 
> Your numpy might be (very) out of date. All of those
> functions are in numpy 1.0
> and later (at least).
> 
> In [5]: import numpy
> 
> In [6]: numpy.fft.fft2
> Out[6]: <function fft2 at 0x17954f0>
> 
> In [7]: numpy.fft.fftn
> Out[7]: <function fftn at 0x1795470>
> 
> In [8]: numpy.roll
> Out[8]: <function roll at 0x1475170>
> 
> 
> -- 
> Robert Kern
> 
> "I have come to believe that the whole world is an
> enigma, a harmless enigma
>  that is made terrible by our own mad attempt to
> interpret it as though it had
>  an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
>
http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


From robert.kern at gmail.com  Sat Apr 28 13:33:52 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 28 Apr 2007 12:33:52 -0500
Subject: [Numpy-discussion] fft and roll
In-Reply-To: <810157.5642.qm@web55914.mail.re3.yahoo.com>
References: <810157.5642.qm@web55914.mail.re3.yahoo.com>
Message-ID: <46338580.2030701@gmail.com>

Ross Harder wrote:
> Ahhh... that hadn't occured to me.  Just installed
> Enthon and assumed it was up to date, but it's not.
> 
> Sorry for the misguided complaint.

No worries.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From sberub at gmail.com  Sat Apr 28 14:03:34 2007
From: sberub at gmail.com (Simon Berube)
Date: Sat, 28 Apr 2007 11:03:34 -0700
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <1e2af89e0704280409u3f6a7d99j2a7283da373dced3@mail.gmail.com>
References: <f0o4h9$k5m$1@sea.gmane.org><462F9F51.3050903@gmail.com>
	<462FAC0A.2040204@molden.no> <462FAC94.1040207@gmail.com>
	<e86a5fd00704251246h1fbe4003v4adf28376bd2ea54@mail.gmail.com>
	<B11B844B-3741-422C-B90C-9E43F7E9B279@cs.hmc.edu>
	<463058F6.3030602@iki.fi>
	<8A1A9308-ECB6-4035-9938-046CC9D2E0C4@cs.hmc.edu>
	<1177706762.398429.93030@s33g2000prh.googlegroups.com>
	<1e2af89e0704280409u3f6a7d99j2a7283da373dced3@mail.gmail.com>
Message-ID: <1177783414.369496.115660@y80g2000hsf.googlegroups.com>

Use a week functions are basically function that you use for a short
period of time where a full fledged well designed program is more of a
waste of time than anything else. Other then that, for what you miss
it really, really depends on your applications and goals. I work on
signal processing and analysis which is very well supported by numpy/
scipy/matplotlib but if you were working in neural networks, finance,
etc...  then python wouldn't be as productive right out of the box.

I suppose the general rule would be, if your project is large and to
be used by a lot of people then Python is probably much better and
more portable, not everyone will need a Matlab License with the proper
version. On the other hand, if you are more interested in small
projects where speed of development is more important than long term
sustainability of the code Matlab is probably better. This is usually
the case in research environment, at least until you figure out
exactly what you want to do, then switch to python.

I hope this help, but either way the decision will have to be made on
your end given that it's impossible for anyone else to have all the
information about your situation.  But keep in mind there is no clear
winner, one tool is good for one thing and another one is good for
another. Python and Matlab just have a large amount of overlap.

On Apr 28, 7:09 am, "Matthew Brett" <matthew.br... at gmail.com> wrote:
> Hi,
>
> > However, I would disagree that Python with all its tools going to
> > replace Matlab well for everything. For large projects, for advanced
> > programmers and for non-standard things such as complex database
> > handling (in my case) it is definitly a clear winner. However, I would
> > be weary of getting Matlab completely out of the picture because I
> > find it is still a much better tool for algorithm testing, scripting
> > and other "use for a week" functions .
>
> Thanks - that seems very fair.  I think you're right that the level of
> help for matlab is far superior, and easier to get to, than numpy /
> python.  Can you say anything more about the features of matlab that
> you miss for the 'use a week' functions?  It might help guide our
> thoughts on potential improvements...
>
> Thanks a lot,
>
> Matthew
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion



From oliphant.travis at ieee.org  Sat Apr 28 17:04:49 2007
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Sat, 28 Apr 2007 15:04:49 -0600
Subject: [Numpy-discussion] Compact way of performing array math with
 specified result type?
In-Reply-To: <rowen-23D294.14173327042007@sea.gmane.org>
References: <rowen-23D294.14173327042007@sea.gmane.org>
Message-ID: <4633B6F1.8050904@ieee.org>

Russell E. Owen wrote:
> I often find myself doing simple math on sequences of numbers (which 
> might or might not be numpy arrays) where I want the result (and thus 
> the inputs) coerced to a particular data type.
>
> I'd like to be able to say:
>
>   numpy.divide(seq1, seq2, dtype=float)
>
> but ufuncs don't allow on to specify a result type. So I do this instead:
>
>   numpy.array(seq1, dtype=float) / numpy.array(seq2, dtype=float)
>
> Is there a more compact solution (without having to create the result 
> array first and supply it as an argument)?
>   

Every ufunc has a little-documented keyword "sig" for (signature) which 
allows you to specify the signature of the inner loop.

Thus,

numpy.divide(seq1, seq1, sig=('d',)*3)

will do what you want.

-Travis


> -- Russell
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>   



From jyasskin at gmail.com  Sat Apr 28 18:19:47 2007
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Sat, 28 Apr 2007 15:19:47 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <Mahogany-0.67.0-2680-20070427-131031.00@CASC64043305.american.edu>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<f0pucb$86m$1@sea.gmane.org> <873b2n5dc7.fsf@uwo.ca>
	<20070427144537.T1402@tribble.ilrt.bris.ac.uk>
	<ca471dc20704270714t42e6bb67pffe935854fcaa889@mail.gmail.com>
	<5d44f72f0704270758n27f3c20y8e2a33c24d8128dd@mail.gmail.com>
	<Mahogany-0.67.0-2680-20070427-131031.00@CASC64043305.american.edu>
Message-ID: <5d44f72f0704281519g316a12abl87e53d42b935deb6@mail.gmail.com>

On 4/27/07, Alan Isaac <aisaac at american.edu> wrote:
> On Fri, 27 Apr 2007, Jeffrey Yasskin wrote:
> > Then again, doubles aren't a group either because of this
> > imprecision, and I'm suggesting claiming they're
> > a subclass of that, so maybe there's room in a practical
> > language to make them a subclass of the rationals too.
>
> Would using language from the Scheme report be useful when
> discussing this?
> http://www-swiss.ai.mit.edu/projects/scheme/documentation/scheme_5.html

Very much so. Thanks for sending the link. I'm not going to get a
chance to update the document for the next several days, so if you
want to put together a patch using such language, I'd be happy to see
it go in. Or I'll integrate those ideas once I have a bit more spare
time.

It looks like the primary ideas there are that it was worthwhile for
Scheme to have the full Number:>Complex:>Real:>Rational:>Integer
tower, and that it wasn't worthwhile to duplicate it entirely for the
exact/inexact distinction.

Thanks,
Jeffrey


From jyasskin at gmail.com  Sat Apr 28 21:38:41 2007
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Sat, 28 Apr 2007 18:38:41 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <f0pucb$86m$1@sea.gmane.org>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<f0pucb$86m$1@sea.gmane.org>
Message-ID: <5d44f72f0704281838j4e8244c4g152df6e3762aa6d4@mail.gmail.com>

Thanks for the comments!

On 4/26/07, Travis E. Oliphant <oliphant.travis at ieee.org> wrote:
>
> Forgive my ignorance, but I'm not really sure what this PEP is trying to
> do.  I don't want to sound negative, I really just don't understand the
> purpose.  I've just never encountered a problem this that I can see how
> this PEP will help me solve.

It's trying to do the same thing for numbers as Guido is doing for
collections in PEP 3119: let people make the useful distinctions
between them for whatever purposes people need to make distinctions.
It looks like the community is a lot more comfortable with overloading
than static typing, so some of the distinctions I proposed are
probably too fine.

> That doesn't mean it's not worthwhile, I just don't understand it.
>
> >
> > Abstract
> > ========
> >
> > This proposal defines a hierarchy of Abstract Base Classes (ABCs)
> > [#pep3119] to represent numbers and other algebraic entities similar
> > to numbers.  It proposes:
> >
> > * A hierarchy of algebraic concepts, including monoids, groups, rings,
> >   and fields with successively more operators and constraints on their
> >   operators. This will be added as a new library module named
> >   "algebra".
> >
>
> The SAGE people may be interested in this, but I doubt there will more
> than a handful of users of these algebraic base classes.

I expected that if someone overloads a function between integers and
rationals, the numeric people would be happier if the integer overload
triggered for square matrices as well, in which case we'd want to
encourage people to overload on Ring vs Field instead of Integral vs
FractionalReal. You'd know better whether that's actually true.
If not, I guess I'd better stop claiming that they'll be useful for
the numpy people. :)

> > * A hierarchy of specifically numeric types, which can be converted to
> >   and from the native Python types. This will be added as a new
> >   library module named "numbers".
> >
>
> I could see having a hierarchy of "numbers"  we have one in NumPy.  All
> the NumPy array scalars fall under a hierarchy of types so that it is
> easy to check whether or not you have signed or unsigned integers, or
> inexact numbers.
>
> > Object oriented systems have a general problem in constraining
> > functions that take two arguments. To take addition as an example,
> > ``int(3) + int(4)`` is defined, and ``vector(1,2,3) + vector(3,4,5)``
> > is defined, but ``int(3) + vector(3,4,5)`` doesn't make much sense.
>
> In NumPy, this kind of operation makes sense we just broadcast int(3) to
> the equivalent vector(3,3,3) and do the element-by-element operation.
>
> This is how most array packages outside of Python do it as well.

Oops; I found a bad example. Does vector(3,3,3) + matrix([4,5],[6,7])
work? Are there other good examples of the addition having to be from
the "same" group?

> I think adding the abstract base-classes for "algebras" is more
> complicated than has been discussed if you are going to start treating
> arrays as elements in their own algebra.  There is a lot of flexibility
> in how arrays can be viewed in an algebraic context.

Really? Since the operators work element-wise, I would think that a
numpy array would have exactly the same algebraic structure as its
elements. One could wrap an array to put it into a particular
algebraic context, but the array itself seems pretty simple.

> The author gives a valiant go at defining base classes for generic
> algebras, but I didn't see everything I've encountered in my travels
> through algebra, and I don't understand the motivation for trying to
> include these things in Python specifically.
>
> Therefore, I would stick with defining a hierarchy of numbers and leave
> it at that for now (if I even went that far).
>
> For numbers, several suggestions have been offered.  In NumPy we use
> abstract base classes that look like this
>
> Number
>     Integer
>        SignedInteger
>        UnsignedInteger
>     Inexact
>        Floating
>        ComplexFloating
>
> But, this is because the array scalars in NumPy are based on
> C-data-types and not on general-purpose integers or floats.  In NumPy it
> is useful to check if you have an integer-typed array and so the
> base-classes become useful.

Do you have an example of when you'd want to check for an integer-typed array?

At one point, Guido was asking for reasons to include Cardinal (aka
Natural, aka Unsigned) in the hierarchy. Is UnsignedInteger only
present in NumPy because it describes some native C types, or does it
have some other purpose?

> For general purpose Python, I would do something like
>
> Complex
>    Rational_Complex
>      Integer_Complex
>    Floating_Complex  # uses hardware float
>    Decimal_Complex
>    Real
>      Rational
>        Integer
>      Floating        # uses hardware float
>      Decimal

Are you specifying an abstract Floating type to incorporate both
floats and doubles? Glancing over the Decimal spec, and trying a few
things with the implementation, it looks like Decimal objects have a
specific precision for each thread, which can be changed for future
operations at any time. Is there still a reason to separate Floating
and Decimal, or will a modification of FloatingReal suffice?

Thanks again,
Jeffrey


From bronto at pobox.com  Sat Apr 28 23:47:36 2007
From: bronto at pobox.com (Anton Sherwood)
Date: Sat, 28 Apr 2007 20:47:36 -0700
Subject: [Numpy-discussion] sort bug
In-Reply-To: <4630F89F.50003@ieee.org>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
	<4630F89F.50003@ieee.org>
Message-ID: <46341558.6040403@pobox.com>

Travis Oliphant wrote:
> One approach is to use argsort to create an index list of sorted 
> eigenvalues and then sort the eig and eigvector arrays before zipping 
> them together in a list of tuples.
> 
> eig, val = numpy.linalg.eig(a)
> 
> indx = eig.argsort()
> eig = eig.take(indx)
> val = val.take(indx, axis=1)
> master = zip(eig, val.T)

Thank you, that worked.
http://www.ogre.nu/wp/?p=1978

I refined it slightly:

val,vec = numpy.linalg.eig(adj)
indx = val.argsort()[-4:-1]
val = val.take(indx)
vec = vec.take(indx, axis=1)
master = zip(val, vec.T)


-- 
Anton Sherwood, http://www.ogre.nu/
"How'd ya like to climb this high *without* no mountain?" --Porky Pine



From charlesr.harris at gmail.com  Sun Apr 29 01:11:29 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 28 Apr 2007 23:11:29 -0600
Subject: [Numpy-discussion] sort bug
In-Reply-To: <46341558.6040403@pobox.com>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
	<4630F89F.50003@ieee.org> <46341558.6040403@pobox.com>
Message-ID: <e06186140704282211x62252bb8o24c3bdde0ef6a43e@mail.gmail.com>

On 4/28/07, Anton Sherwood <bronto at pobox.com> wrote:
>
> Travis Oliphant wrote:
> > One approach is to use argsort to create an index list of sorted
> > eigenvalues and then sort the eig and eigvector arrays before zipping
> > them together in a list of tuples.
> >
> > eig, val = numpy.linalg.eig(a)
> >
> > indx = eig.argsort()
> > eig = eig.take(indx)
> > val = val.take(indx, axis=1)
> > master = zip(eig, val.T)
>
> Thank you, that worked.
> http://www.ogre.nu/wp/?p=1978
>
> I refined it slightly:
>
> val,vec = numpy.linalg.eig(adj)
> indx = val.argsort()[-4:-1]
> val = val.take(indx)
> vec = vec.take(indx, axis=1)
> master = zip(val, vec.T)


But that won't get the 4  largest, and will ignore the last eigenvalue,
whatever it is. If you want the four largest, do

val,vec =  numpy.linalg.eig(adj)
ind = val.argsort()
val = val.take(ind[-4:])
vec = vec.take(ind[-4:], axis=1)
master = zip(val, vec.T)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070428/94b2f3bb/attachment.html>

From bronto at pobox.com  Sun Apr 29 05:05:45 2007
From: bronto at pobox.com (Anton Sherwood)
Date: Sun, 29 Apr 2007 02:05:45 -0700
Subject: [Numpy-discussion] sort bug
In-Reply-To: <e06186140704282211x62252bb8o24c3bdde0ef6a43e@mail.gmail.com>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
	<4630F89F.50003@ieee.org> <46341558.6040403@pobox.com>
	<e06186140704282211x62252bb8o24c3bdde0ef6a43e@mail.gmail.com>
Message-ID: <46345FE9.6000704@pobox.com>

> Anton Sherwood wrote:
>     I refined it slightly:
> 
>     val,vec = numpy.linalg.eig(adj)
>     indx = val.argsort()[-4:-1]
>     val = val.take(indx)
>     vec = vec.take(indx, axis=1)
>     master = zip(val, vec.T)

Charles R Harris wrote:
> But that won't get the 4  largest, and will ignore the last eigenvalue, 
> whatever it is.  . . .

Are you making two points here or only one?

I'm using eigenvectors of a graph's adjacency matrix as "topological" 
coordinates of the graph's vertices as embedded in 3space (something I 
learned about just recently).  Whenever I've done this with a graph that 
*does* have a good 3d embedding, using the first eigenvector results in 
a flat model: apparently the first is not independent, at least in such 
cases.  So, yeah, I really did intend to throw away the largest and use 
the next three.  Are you saying this code doesn't give me that?  The 
output of my first few test cases looks good.

-- 
Anton Sherwood, http://www.ogre.nu/
"How'd ya like to climb this high *without* no mountain?" --Porky Pine


From openopt at ukr.net  Sun Apr 29 06:51:17 2007
From: openopt at ukr.net (dmitrey)
Date: Sun, 29 Apr 2007 13:51:17 +0300
Subject: [Numpy-discussion] simpliest way to check: array x is float,
	not integer
Message-ID: <463478A5.1050507@ukr.net>

hi all,
please inform me what is the simplest way to check, does the vector x 
that came to my func is float or integer. I.e. someone can pass to my 
func for example x0 = numpy.array([1, 0, 0]) and it can yield wrong 
unexpected results vs numpy.array([1.0, 0, 0]) .
Thx, D.




From lbolla at gmail.com  Sun Apr 29 07:18:24 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Sun, 29 Apr 2007 13:18:24 +0200
Subject: [Numpy-discussion] arctan2 with complex args
Message-ID: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>

Weird behaviour with arctan2(complex,complex).
Take  a look at this:

In [11]: numpy.arctan2(1.,1.)
Out[11]: 0.785398163397

In [12]: numpy.arctan2(1j,1j)
---------------------------------------------------------------------------
exceptions.AttributeError                            Traceback (most recent
call last)

AttributeError: 'complex' object has no attribute 'arctan2'

same error for:

In [13]: numpy.arctan2(1j,1.)
In [14]: numpy.arctan2(1.,1j)

But arctan2 is defined for complex arguments, as far as Octave knows :-) :

octave:7> atan2(1,1)
ans = 0.78540
octave:8> atan2(1j,1j)
ans = 0
octave:9> atan2(1j,1)
ans = 0
octave:10> atan2(1,1j)
ans = 1.5708

bug or wanted behaviour?
Lorenzo.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070429/be8e9011/attachment.html>

From robert.kern at gmail.com  Sun Apr 29 11:15:32 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 29 Apr 2007 10:15:32 -0500
Subject: [Numpy-discussion] sort bug
In-Reply-To: <46345FE9.6000704@pobox.com>
References: <463047DF.7070700@pobox.com>
	<46306253.20803@ieee.org>	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>	<4630F89F.50003@ieee.org>
	<46341558.6040403@pobox.com>	<e06186140704282211x62252bb8o24c3bdde0ef6a43e@mail.gmail.com>
	<46345FE9.6000704@pobox.com>
Message-ID: <4634B694.5000506@gmail.com>

Anton Sherwood wrote:
>> Anton Sherwood wrote:
>>     I refined it slightly:
>>
>>     val,vec = numpy.linalg.eig(adj)
>>     indx = val.argsort()[-4:-1]
>>     val = val.take(indx)
>>     vec = vec.take(indx, axis=1)
>>     master = zip(val, vec.T)
> 
> Charles R Harris wrote:
>> But that won't get the 4  largest, and will ignore the last eigenvalue, 
>> whatever it is.  . . .
> 
> Are you making two points here or only one?

Only one.

> I'm using eigenvectors of a graph's adjacency matrix as "topological" 
> coordinates of the graph's vertices as embedded in 3space (something I 
> learned about just recently).  Whenever I've done this with a graph that 
> *does* have a good 3d embedding, using the first eigenvector results in 
> a flat model: apparently the first is not independent, at least in such 
> cases.  So, yeah, I really did intend to throw away the largest and use 
> the next three.  Are you saying this code doesn't give me that?  The 
> output of my first few test cases looks good.

No, it's just that in previous emails you had stated that you wanted the four
largest eigenvalues and said nothing about actually wanting to throw away the
largest of the four. Sometimes people get confused with Python's (and thus
numpy's) indexing, so Charles wanted to make sure you knew what [-4:-1] actually
did.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Sun Apr 29 11:19:22 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 29 Apr 2007 09:19:22 -0600
Subject: [Numpy-discussion] sort bug
In-Reply-To: <46345FE9.6000704@pobox.com>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
	<4630F89F.50003@ieee.org> <46341558.6040403@pobox.com>
	<e06186140704282211x62252bb8o24c3bdde0ef6a43e@mail.gmail.com>
	<46345FE9.6000704@pobox.com>
Message-ID: <e06186140704290819w28627e3dlec3e005e964de440@mail.gmail.com>

On 4/29/07, Anton Sherwood <bronto at pobox.com> wrote:
>
> > Anton Sherwood wrote:
> >     I refined it slightly:
> >
> >     val,vec = numpy.linalg.eig(adj)
> >     indx = val.argsort()[-4:-1]
> >     val = val.take(indx)
> >     vec = vec.take(indx, axis=1)
> >     master = zip(val, vec.T)
>
> Charles R Harris wrote:
> > But that won't get the 4  largest, and will ignore the last eigenvalue,
> > whatever it is.  . . .
>
> Are you making two points here or only one?
>
> I'm using eigenvectors of a graph's adjacency matrix as "topological"
> coordinates of the graph's vertices as embedded in 3space (something I
> learned about just recently).  Whenever I've done this with a graph that
> *does* have a good 3d embedding, using the first eigenvector results in
> a flat model: apparently the first is not independent, at least in such
> cases.  So, yeah, I really did intend to throw away the largest and use
> the next three.  Are you saying this code doesn't give me that?  The
> output of my first few test cases looks good.


Looks good, then. In your original post you were talked about the four
largest eigenvalues. Anyway, the embedding part sounds interesting, I'll
have to think about why that works.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070429/bc5dcacf/attachment.html>

From openopt at ukr.net  Sun Apr 29 12:41:39 2007
From: openopt at ukr.net (dmitrey)
Date: Sun, 29 Apr 2007 19:41:39 +0300
Subject: [Numpy-discussion] bug in
	http://www.scipy.org/NumPy_for_Matlab_Users
Message-ID: <4634CAC3.2050409@ukr.net>

now there is
MATLAB            NDArray         Matrix
size(a,n)                 a.shape[n]       a.shape[n]

but it should be
size(a,n)                 a.shape[n-1]       a.shape[n-1]

WBR, D.


From David.L.Goldsmith at noaa.gov  Sun Apr 29 13:11:41 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Sun, 29 Apr 2007 10:11:41 -0700
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
Message-ID: <4634D1CD.8040608@noaa.gov>

I'll take a stab at this one; if I miss the mark, people, please chime in.

What's "strange" here is not numpy's behavior but octave's (IMO).  
Remember that, over R, arctan is used in two different ways: one is 
simply as a map from (-inf, inf) -> (-pi/2,pi/2) - here, let's call that 
invtan; the other is as a means to determine "the angle" (conventionally 
taken to be between -pi and pi) of a point in the plane - but since, for 
example, tan(pi/4) = tan(-3pi/4) (and in general tan(x) = tan(x-pi)) to 
uniquely determine said angle, we need to keep track of and take into 
account the quadrant in which the point lies; this is (the only reason) 
why arctan2 is a function of two arguments, one representing the 
abscissa, the other the ordinate of the point.  But when the argument is 
complex (arctan2, as the inverse of the tangent function, *is* a valid 
function on C), this geometric use no longer makes sense, so there's 
really no reason to implement arctan2(z,w), z, w complex.  If for some 
reason, e.g., uniformity of algorithmic expression - I don't see any 
(simple) way to preserve uniformity of code expression - as near as I 
can tell, you're going to have to implement an if/else if you need to 
allow for the invtan of two complex arguments - you need to handle 
arctan2(z,w), implement it as arctan(w/z):

 >>> import numpy
 >>> numpy.arctan(1j/1j)
(0.78539816339744828+0j)

DG

lorenzo bolla wrote:
> Weird behaviour with arctan2(complex,complex).
> Take  a look at this:
>
> In [11]: numpy.arctan2(1.,1.)
> Out[11]: 0.785398163397
>
> In [12]: numpy.arctan2(1j,1j)
> ---------------------------------------------------------------------------
> exceptions.AttributeError                            Traceback (most 
> recent call last)
>
> AttributeError: 'complex' object has no attribute 'arctan2'
>
> same error for:
>
> In [13]: numpy.arctan2(1j,1.)
> In [14]: numpy.arctan2(1.,1j)
>
> But arctan2 is defined for complex arguments, as far as Octave knows :-) :
>
> octave:7> atan2(1,1)
> ans = 0.78540
> octave:8> atan2(1j,1j)
> ans = 0
> octave:9> atan2(1j,1)
> ans = 0
> octave:10> atan2(1,1j)
> ans = 1.5708
>
> bug or wanted behaviour?
> Lorenzo.
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From kwgoodman at gmail.com  Sun Apr 29 13:16:31 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Sun, 29 Apr 2007 10:16:31 -0700
Subject: [Numpy-discussion] bug in
	http://www.scipy.org/NumPy_for_Matlab_Users
In-Reply-To: <4634CAC3.2050409@ukr.net>
References: <4634CAC3.2050409@ukr.net>
Message-ID: <f4f93d420704291016s51215d83q13867e56582d3b8f@mail.gmail.com>

On 4/29/07, dmitrey <openopt at ukr.net> wrote:
> now there is
> MATLAB            NDArray         Matrix
> size(a,n)                 a.shape[n]       a.shape[n]
>
> but it should be
> size(a,n)                 a.shape[n-1]       a.shape[n-1]

I made the change. But how should we change the comment?

"get the number of elements of the n'th dimension of array a"


From jyasskin at gmail.com  Sun Apr 29 13:25:35 2007
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Sun, 29 Apr 2007 10:25:35 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <f10g7k$fag$1@sea.gmane.org>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<f10g7k$fag$1@sea.gmane.org>
Message-ID: <5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>

On 4/28/07, Baptiste Carvello <baptiste13 at altern.org> wrote:
> 2) In the PEP, the concepts are used *inconsistently*. Complex derives from Ring
> because the set of complex numbers *is a* ring. Int derives from Complex because
> integer are complex numbers (or, alternatively, the set of integers *is included
> in* the set of complex numbers). The consistent way could be to make the Complex
> class an instance of Ring, not a subclass.

Good point. In this structure, isinstance(3, Ring) really means that 3
is a member of some (unspecified) ring, not that 3 isa Ring, but the
ambiguity there is probably the root cause of the problem with
mixed-mode operations. We should also say that isinstance(3, Complex)
means that 3 is a member of some subring of the complex numbers, which
preserves the claim that Complex is a subtype of Ring.

Up to here, things make sense, but because of how ABCs work, we need
issubclass(rational, Complex). I suppose that's true too, since
isinstance(3.4, rational) means "3.4 is a member of the rational
subring of the complex numbers", which implies that "3.4 is a member
of some subring of the complex numbers."

There may be better names for these concepts. Perhaps suffixing every
numeric ABC with "Element"? Do you have suggestions?


Jason Orendorff points out that Haskell typeclasses capture the fact
that complex is an instance of Ring. I think imitating them as much as
possible would indeed imply making the numeric ABCs into metaclasses
(in Haskell terminology, "kinds"). To tell if the arguments to a
function were in the same total order, you'd check if they had any
common superclasses that were themselves instances of TotallyOrdered.
I don't know enough about how metaclasses are typically used to know
how that would conflict.

-- 
Namast?,
Jeffrey Yasskin

From anton.sherwood at gmail.com  Sun Apr 29 14:15:59 2007
From: anton.sherwood at gmail.com (Anton Sherwood)
Date: Sun, 29 Apr 2007 11:15:59 -0700
Subject: [Numpy-discussion] sort bug
In-Reply-To: <e06186140704290819w28627e3dlec3e005e964de440@mail.gmail.com>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
	<4630F89F.50003@ieee.org> <46341558.6040403@pobox.com>
	<e06186140704282211x62252bb8o24c3bdde0ef6a43e@mail.gmail.com>
	<46345FE9.6000704@pobox.com>
	<e06186140704290819w28627e3dlec3e005e964de440@mail.gmail.com>
Message-ID: <fb420b880704291115p640650fbxe02d738718776e97@mail.gmail.com>

> Anton Sherwood wrote:
> > I'm using eigenvectors of a graph's adjacency matrix as "topological"
> > coordinates of the graph's vertices as embedded in 3space (something I
> > learned about just recently).  Whenever I've done this with a graph that
> > *does* have a good 3d embedding, using the first eigenvector results in
> > a flat model: apparently the first is not independent, at least in such
> > cases.  . . .

Charles R Harris wrote:
> . . . the embedding part sounds interesting,
> I'll have to think about why that works.

It's a mystery to me: I never did study enough matrix algebra to get a
feel for eigenvectors (indeed this is the first time I've had anything
to do with them).

I'll happily share my code with anyone who wants to experiment with it.

-- 
Anton Sherwood, http://www.ogre.nu/


From lbolla at gmail.com  Sun Apr 29 14:50:42 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Sun, 29 Apr 2007 20:50:42 +0200
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <4634D1CD.8040608@noaa.gov>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
	<4634D1CD.8040608@noaa.gov>
Message-ID: <80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>

You make your point, but I would expect a behaviour similar to Mathematica
or Matlab.

>From http://documents.wolfram.com/mathematica/functions/ArcTan
"If x or y is complex, then ArcTan[x, y] gives . When , ArcTan[x, y] gives
the number such that and ."

Lorenzo.

On 4/29/07, David Goldsmith <David.L.Goldsmith at noaa.gov> wrote:
>
> I'll take a stab at this one; if I miss the mark, people, please chime in.
>
> What's "strange" here is not numpy's behavior but octave's (IMO).
> Remember that, over R, arctan is used in two different ways: one is
> simply as a map from (-inf, inf) -> (-pi/2,pi/2) - here, let's call that
> invtan; the other is as a means to determine "the angle" (conventionally
> taken to be between -pi and pi) of a point in the plane - but since, for
> example, tan(pi/4) = tan(-3pi/4) (and in general tan(x) = tan(x-pi)) to
> uniquely determine said angle, we need to keep track of and take into
> account the quadrant in which the point lies; this is (the only reason)
> why arctan2 is a function of two arguments, one representing the
> abscissa, the other the ordinate of the point.  But when the argument is
> complex (arctan2, as the inverse of the tangent function, *is* a valid
> function on C), this geometric use no longer makes sense, so there's
> really no reason to implement arctan2(z,w), z, w complex.  If for some
> reason, e.g., uniformity of algorithmic expression - I don't see any
> (simple) way to preserve uniformity of code expression - as near as I
> can tell, you're going to have to implement an if/else if you need to
> allow for the invtan of two complex arguments - you need to handle
> arctan2(z,w), implement it as arctan(w/z):
>
> >>> import numpy
> >>> numpy.arctan(1j/1j)
> (0.78539816339744828+0j)
>
> DG
>
> lorenzo bolla wrote:
> > Weird behaviour with arctan2(complex,complex).
> > Take  a look at this:
> >
> > In [11]: numpy.arctan2(1.,1.)
> > Out[11]: 0.785398163397
> >
> > In [12]: numpy.arctan2(1j,1j)
> >
> ---------------------------------------------------------------------------
> > exceptions.AttributeError                            Traceback (most
> > recent call last)
> >
> > AttributeError: 'complex' object has no attribute 'arctan2'
> >
> > same error for:
> >
> > In [13]: numpy.arctan2(1j,1.)
> > In [14]: numpy.arctan2(1.,1j)
> >
> > But arctan2 is defined for complex arguments, as far as Octave knows :-)
> :
> >
> > octave:7> atan2(1,1)
> > ans = 0.78540
> > octave:8> atan2(1j,1j)
> > ans = 0
> > octave:9> atan2(1j,1)
> > ans = 0
> > octave:10> atan2(1,1j)
> > ans = 1.5708
> >
> > bug or wanted behaviour?
> > Lorenzo.
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070429/c8d9c62a/attachment.html>

From strawman at astraw.com  Sun Apr 29 15:39:22 2007
From: strawman at astraw.com (Andrew Straw)
Date: Sun, 29 Apr 2007 12:39:22 -0700
Subject: [Numpy-discussion] bug
	in	http://www.scipy.org/NumPy_for_Matlab_Users
In-Reply-To: <4634CAC3.2050409@ukr.net>
References: <4634CAC3.2050409@ukr.net>
Message-ID: <4634F46A.4060902@astraw.com>

No, the nth index of a Python sequence is a[n], where n starts from 
zero. Thus, if I want the nth dimension of array a, I want a.shape[n].

I reverted the page to its original form and added a couple explanatory 
comments about zero vs one based indexing.

dmitrey wrote:
> now there is
> MATLAB            NDArray         Matrix
> size(a,n)                 a.shape[n]       a.shape[n]
> 
> but it should be
> size(a,n)                 a.shape[n-1]       a.shape[n-1]
> 
> WBR, D.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From openopt at ukr.net  Sun Apr 29 15:45:49 2007
From: openopt at ukr.net (dmitrey)
Date: Sun, 29 Apr 2007 22:45:49 +0300
Subject: [Numpy-discussion]
	bug	in	http://www.scipy.org/NumPy_for_Matlab_Users
In-Reply-To: <4634F46A.4060902@astraw.com>
References: <4634CAC3.2050409@ukr.net> <4634F46A.4060902@astraw.com>
Message-ID: <4634F5ED.4060002@ukr.net>

I think it's better to add "see remark!" inside the cells
because not all people read the text from 4th column
and this can lead to serious mistakes and lot of time elapsed for bug 
hunting.
WBR, D.
Andrew Straw wrote:
> No, the nth index of a Python sequence is a[n], where n starts from 
> zero. Thus, if I want the nth dimension of array a, I want a.shape[n].
>
> I reverted the page to its original form and added a couple explanatory 
> comments about zero vs one based indexing.
>
> dmitrey wrote:
>   
>> now there is
>> MATLAB            NDArray         Matrix
>> size(a,n)                 a.shape[n]       a.shape[n]
>>
>> but it should be
>> size(a,n)                 a.shape[n-1]       a.shape[n-1]
>>
>> WBR, D.
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>     
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>   



From aisaac at american.edu  Sun Apr 29 16:02:41 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Sun, 29 Apr 2007 16:02:41 -0400
Subject: [Numpy-discussion]
	=?utf-8?b?YnVnCWluCWh0dHA6Ly93d3cuc2NpcHkub3Jn?=
	=?utf-8?q?/NumPy_for_Matlab_Users?=
In-Reply-To: <4634F46A.4060902@astraw.com>
References: <4634CAC3.2050409@ukr.net><4634F46A.4060902@astraw.com>
Message-ID: <Mahogany-0.67.0-1024-20070429-160241.00@american.edu>

On Sun, 29 Apr 2007, Andrew Straw apparently wrote: 
> No, the nth index of a Python sequence is a[n], where n starts from 
> zero. Thus, if I want the nth dimension of array a, I want a.shape[n]. 


So now we have no seventh day of the week?

Even the Python reference manual has Monday as the *first* 
(not zeroth) day of the week, with index 0.

Cheers,
Alan Isaac






From kwgoodman at gmail.com  Sun Apr 29 16:09:40 2007
From: kwgoodman at gmail.com (Keith Goodman)
Date: Sun, 29 Apr 2007 13:09:40 -0700
Subject: [Numpy-discussion] bug in
	http://www.scipy.org/NumPy_for_Matlab_Users
In-Reply-To: <4634F46A.4060902@astraw.com>
References: <4634CAC3.2050409@ukr.net> <4634F46A.4060902@astraw.com>
Message-ID: <f4f93d420704291309nfaf4408k4b38570fc77be563@mail.gmail.com>

On 4/29/07, Andrew Straw <strawman at astraw.com> wrote:
> No, the nth index of a Python sequence is a[n], where n starts from
> zero. Thus, if I want the nth dimension of array a, I want a.shape[n].
>
> I reverted the page to its original form and added a couple explanatory
> comments about zero vs one based indexing.

Would it be better to replace n with a number, say, 2?

size(a, 2)     a.shape[1]


From bioinformed at gmail.com  Sun Apr 29 16:19:55 2007
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Sun, 29 Apr 2007 16:19:55 -0400
Subject: [Numpy-discussion] bug in
	http://www.scipy.org/NumPy_for_Matlab_Users
In-Reply-To: <4634F46A.4060902@astraw.com>
References: <4634CAC3.2050409@ukr.net> <4634F46A.4060902@astraw.com>
Message-ID: <2e1434c10704291319o1566fe77k292c9b2894faf6c7@mail.gmail.com>

On 4/29/07, Andrew Straw <strawman at astraw.com> wrote:
>
> No, the nth index of a Python sequence is a[n], where n starts from
> zero. Thus, if I want the nth dimension of array a, I want a.shape[n].
>
> I reverted the page to its original form and added a couple explanatory
> comments about zero vs one based indexing.



<pedantic sounding rant -- apologies in advance>
Those among us who value correct English will continue to insist that
ordinal numbers begin with "first" and the concept of "zeroth" is an
unnatural technological bastardization.  (This is not to say that zero-based
indexing is bad -- just distinct from the ordinal.)

The first index of 'a' is 0, the first element is a[0], the second index is
1 and the second element is a[1], etc.  Thus, the n-th index or element, a
contraction of ordinal numbering, is correctly and canonically written as
a[n-1] in a zero-based index scheme.  The linguistics of "n-th" are that of
ordinality in both English and mathematics, requiring an explicitly mapping
to the technological concept of a given indexing syntax.

Glossing over that difference, especially when it contradicts the most
natural conventions of the target audience, is unfriendly and
counterintuitive.  The only reason why it makes sense to you only because of
your disadvantage of already understanding that which you are trying to
explain.
</pedantic sounding rant>

My recommendation: keep a[n-1] _and_ include a lucid discussion on
zero-based indexing.

Pedantically,
-Kevin



_______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070429/d918e9f7/attachment.html>

From charlesr.harris at gmail.com  Sun Apr 29 17:02:09 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 29 Apr 2007 15:02:09 -0600
Subject: [Numpy-discussion] bug in
	http://www.scipy.org/NumPy_for_Matlab_Users
In-Reply-To: <2e1434c10704291319o1566fe77k292c9b2894faf6c7@mail.gmail.com>
References: <4634CAC3.2050409@ukr.net> <4634F46A.4060902@astraw.com>
	<2e1434c10704291319o1566fe77k292c9b2894faf6c7@mail.gmail.com>
Message-ID: <e06186140704291402v5cadede4t5c13bd43ef523d5b@mail.gmail.com>

On 4/29/07, Kevin Jacobs <jacobs at bioinformed.com> <bioinformed at gmail.com>
wrote:
>
> On 4/29/07, Andrew Straw <strawman at astraw.com> wrote:
> >
> > No, the nth index of a Python sequence is a[n], where n starts from
> > zero. Thus, if I want the nth dimension of array a, I want a.shape[n].
> >
> > I reverted the page to its original form and added a couple explanatory
> > comments about zero vs one based indexing.
>
>
>
> <pedantic sounding rant -- apologies in advance>
> Those among us who value correct English will continue to insist that
> ordinal numbers begin with "first" and the concept of "zeroth" is an
> unnatural technological bastardization.  (This is not to say that zero-based
> indexing is bad -- just distinct from the ordinal.)
>

Ordinals begin with the empty set, 0, and continue 1 := {0}, 2 := {0, {0}},
3 := {0, {0}, {{0, {0}}}, ...

The first index of 'a' is 0, the first element is a[0], the second index is
> 1 and the second element is a[1], etc.  Thus, the n-th index or element, a
> contraction of ordinal numbering, is correctly and canonically written as
> a[n-1] in a zero-based index scheme.  The linguistics of "n-th" are that of
> ordinality in both English and mathematics, requiring an explicitly mapping
> to the technological concept of a given indexing syntax.
>

Fortran by default uses 1 based indexing, but uses a "magic" pointer so that
actual indexing is zero based, look at the internals of a Fortran compiler
sometime an see. I think this points out that zero based indexing is more
natural.

Glossing over that difference, especially when it contradicts the most
> natural conventions of the target audience, is unfriendly and
> counterintuitive.  The only reason why it makes sense to you only because of
> your disadvantage of already understanding that which you are trying to
> explain.
> </pedantic sounding rant>
>

I love these little flame wars, not that there aren't things I *really*
should be doing.

Chuck <http://projects.scipy.org/mailman/listinfo/numpy-discussion>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070429/e86e3012/attachment.html>

From matthieu.brucher at gmail.com  Sun Apr 29 17:04:12 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Sun, 29 Apr 2007 23:04:12 +0200
Subject: [Numpy-discussion] sort bug
In-Reply-To: <fb420b880704291115p640650fbxe02d738718776e97@mail.gmail.com>
References: <463047DF.7070700@pobox.com> <46306253.20803@ieee.org>
	<fb420b880704261130p5ff4abf8he365b49fffb3276f@mail.gmail.com>
	<4630F89F.50003@ieee.org> <46341558.6040403@pobox.com>
	<e06186140704282211x62252bb8o24c3bdde0ef6a43e@mail.gmail.com>
	<46345FE9.6000704@pobox.com>
	<e06186140704290819w28627e3dlec3e005e964de440@mail.gmail.com>
	<fb420b880704291115p640650fbxe02d738718776e97@mail.gmail.com>
Message-ID: <e76aa17f0704291404xa9f8a53y8cd652f43c0c9092@mail.gmail.com>

2007/4/29, Anton Sherwood <anton.sherwood at gmail.com>:
>
> > Anton Sherwood wrote:
> > > I'm using eigenvectors of a graph's adjacency matrix as "topological"
> > > coordinates of the graph's vertices as embedded in 3space (something I
> > > learned about just recently).  Whenever I've done this with a graph
> that
> > > *does* have a good 3d embedding, using the first eigenvector results
> in
> > > a flat model: apparently the first is not independent, at least in
> such
> > > cases.  . . .
>
> Charles R Harris wrote:
> > . . . the embedding part sounds interesting,
> > I'll have to think about why that works.
>
> It's a mystery to me: I never did study enough matrix algebra to get a
> feel for eigenvectors (indeed this is the first time I've had anything
> to do with them).
>
> I'll happily share my code with anyone who wants to experiment with it.
>

Seems to me that this is much like Isomap and class multidimensional
scaling, no ?

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070429/8ac48c05/attachment.html>

From guido at python.org  Sun Apr 29 18:46:58 2007
From: guido at python.org (Guido van Rossum)
Date: Sun, 29 Apr 2007 15:46:58 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<f10g7k$fag$1@sea.gmane.org>
	<5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
Message-ID: <ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>

On 4/29/07, Jeffrey Yasskin <jyasskin at gmail.com> wrote:
> On 4/28/07, Baptiste Carvello <baptiste13 at altern.org> wrote:
> > 2) In the PEP, the concepts are used *inconsistently*. Complex derives from Ring
> > because the set of complex numbers *is a* ring. Int derives from Complex because
> > integer are complex numbers (or, alternatively, the set of integers *is included
> > in* the set of complex numbers). The consistent way could be to make the Complex
> > class an instance of Ring, not a subclass.
>
> Good point. In this structure, isinstance(3, Ring) really means that 3
> is a member of some (unspecified) ring, not that 3 isa Ring,

To ask whether x is a Ring, you'd use issubclass(x, Ring). Now, in a
different context (still in Python) you might define a class Ring
whose *instances* are rings; but in the current draft of PEP 3141,
Ring is a class whose subclasses are rings.

[BTW "isa" is not an English word, nor used anywhere in Python. If
"isa" means "is a" you might as well write proper English; if it has
additional connotations, they're likely lost to this crowd so you
can't count on your readers knowing the distinction.]

>  but the ambiguity there is probably the root cause of the problem with
> mixed-mode operations.

(Where by mixed-mode operations you mean e.g. trying to add or
multiply members of two *different* rings, right?)

> We should also say that isinstance(3, Complex)
> means that 3 is a member of some subring of the complex numbers, which
> preserves the claim that Complex is a subtype of Ring.

Hm... it's beginning to look like binary operations in the
mathematical sense just don't have an exact equivalent in OO type
theory. (Or vice versa.) There really isn't a way to define an
operation binop(a: T, b: T) -> T in such a way that it is clear what
should happen if a and b are members of two different subtypes of T,
named T1 and T2. Classic OO seems to indicate that this must be
defined, and there are plenty of examples that seem to agree, e.g.
when T1 and T2 are trivial subtypes of T (maybe each adding a
different inessential method). OTOH we have the counter-example where
T==Ring and T1 and T2 are two different, unrelated Rings, and the
result may not be defined.

Hmm... Maybe the conclusion to draw from this is that we shouldn't
make Ring a class? Maybe it ought to be a metaclass, so we could ask
isinstance(Complex, Ring)?

Perhaps a similar line of reasoning migtht apply to PartiallyOrdered
and TotallyOrdered.

> Up to here, things make sense, but because of how ABCs work, we need
> issubclass(rational, Complex). I suppose that's true too, since
> isinstance(3.4, rational) means "3.4 is a member of the rational
> subring of the complex numbers", which implies that "3.4 is a member
> of some subring of the complex numbers."
>
> There may be better names for these concepts. Perhaps suffixing every
> numeric ABC with "Element"? Do you have suggestions?

Maybe we should stop trying to capture radically different
mathematical number systems using classes or types, and limit
ourselves to capturing the systems one learns in high school: C, R, Q,
Z, and (perhaps) N (really N0). The concrete types would be complex <:
C, float<:R, Decimal<:R, int<:Z. NumPy would have many more. One could
argue that float and Decimal are <:Q, but I'm not sure if that makes
things better pragmatically; I guess I'm coming from the old Algol
school where float was actually called real (and in retrospect I wish
I'd called it that in Python). I'd rather reserve membership of Q for
an infinite precision rational type (which many people have
independently implemented).

> Jason Orendorff points out that Haskell typeclasses capture the fact
> that complex is an instance of Ring. I think imitating them as much as
> possible would indeed imply making the numeric ABCs into metaclasses
> (in Haskell terminology, "kinds"). To tell if the arguments to a
> function were in the same total order, you'd check if they had any
> common superclasses that were themselves instances of TotallyOrdered.
> I don't know enough about how metaclasses are typically used to know
> how that would conflict.

The more I think about it, it sounds like the right thing to do. To
take PartiallyOrdered (let's say PO for brevity) as an example, the
Set class should specify PO as a metaclass. The PO metaclass could
require that the class implement __lt__ and __le__. If it found a
class that didn't implement them, it could make the class abstract by
adding the missing methods to its __abstractmethods__ attribute. Or,
if it found that the class implemented one but not the other, it could
inject a default implementation of the other in terms of the one and
__eq__.

This leaves us with the question of how to check whether an object is
partially orderable. Though that may really be the wrong question --
perhaps you should ask whether two objects are partially orderable
relative to each other. For that, you would first have to find the
most derived common base class (if that is even always a defined
operation(*)), and then check whether that class is an instance of PO.
It seems easier to just try the comparison -- duck typing isn't dead
yet! I don't think this is worth introducing a new inspection
primitive ('ismetainstance(x, PO)').

The PO class may still be useful for introspection: at the meta-level,
it may be useful occasionally to insist that or inquire whether a
given *class* is PO. (Or TO, or a Ring, etc.)

Now, you could argue that Complex should also be a metaclass. While
that may mathematically meaningful (for all I know there are people
doing complex number theory using Complex[Z/n]), for Python's numeric
classes I think it's better to make Complex a regular class
representing all the usual complex numbers (i.e. a pair of Real
numbers). I expect that the complex subclasses used in practice are
all happy under mixed arithmetic using the usual definition of mixed
arithmetic: convert both arguments to a common base class and compute
the operation in that domain.

(*) consider classes AB derived from (B, A) and BA derived from (A,
B). Would A or B be the most derived base class? Or would we have to
skip both and continue the search with A's and B's base classes?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido at python.org  Sun Apr 29 21:26:45 2007
From: guido at python.org (Guido van Rossum)
Date: Sun, 29 Apr 2007 18:26:45 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <46353440.5060402@acm.org>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<f10g7k$fag$1@sea.gmane.org>
	<5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
	<ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
	<46353440.5060402@acm.org>
Message-ID: <ca471dc20704291826p6cb8e001gf3df7a3dc6160667@mail.gmail.com>

On 4/29/07, Talin <talin at acm.org> wrote:
> Guido van Rossum wrote:
> > Maybe we should stop trying to capture radically different
> > mathematical number systems using classes or types, and limit
> > ourselves to capturing the systems one learns in high school: C, R, Q,
> > Z, and (perhaps) N (really N0). The concrete types would be complex <:
> > C, float<:R, Decimal<:R, int<:Z. NumPy would have many more. One could
> > argue that float and Decimal are <:Q, but I'm not sure if that makes
> > things better pragmatically; I guess I'm coming from the old Algol
> > school where float was actually called real (and in retrospect I wish
> > I'd called it that in Python). I'd rather reserve membership of Q for
> > an infinite precision rational type (which many people have
> > independently implemented).
>
> I haven't really been following this discussion, given my lack of
> understanding of the issues involved, but I want to make one observation
> about the discussion.
>
> Normally, when someone suggests an idea for a major addition to Python
> of this scope, the standard response is that they should go develop it
> as a 3rd-party package and see if becomes popular, and if it does it can
> be considered for inclusion in the standard library.
>
> Unfortunately, we are somewhat constrained in this case, because we're
> talking about altering the behavior of some fairly fundamental built-in
> types - which means that it's going to be hard to implement it as an
> add-on library.

Not entirely true in the latest incarnation -- the proposed
overloading of isinstance() and issubclass() will make it possible to
add Ring-ness to float and int as an afterthough from a user-defined
class.

> And yet, it seems to me that this particular set of features really does
> need a long gestation period compared to some others that we've
> discussed. Most of the 3000-series PEPs are really about a fairly small
> set of base decisions. Even the long PEPs are more about descriptions of
> the logical consequences of those decisions than about the decisions
> themselves. The ABC PEPs are different, in that they are standardizing a
> whole slew of things all at once. Moreover, I think there is a real
> danger here of a kind of ivory-tower decision making, isolated from
> day-to-day writing of applications to keep them grounded.
>
> The question of whether to limit to the "high school" number classes, or
> to go with the more mathematically abstract set of things is a decision
> which, it seems to me, ought to be made in the context of actual use
> cases, rather than abstract theorizing about use cases. (I'm also
> generally supportive about copying what other languages have done, on
> the theory that they too have been tested by real-world use.)

If that's the criterion (and I agree) it should be fairly obvious by
now that Ring and MonoidUnderPlus and everything in between should be
cast out, and we should stick with high school math. (Just note that
Travis Oliphant, Mr. NumPy Himself, pretty much confessed being
confused by the algebra stuff).

> If it were technically possible, I would recommend that this PEP have to
> run the same gauntlet that any other large library addition would, which
> is to go through a long period of community feedback and criticism,
> during which a large number of people actually attempt to use the
> feature for real work. I also think, in this case, that the special
> power of "core python developer fiat" should not be invoked unless it
> has to be, because I don't think that there is a firm basis for making
> such a judgment yet.
>
> I would also suggest that some thought be given to ways to allow for
> experimentation with different variations of this feature. If it is not
> possible to make these numeric classes definable in Python at runtime,
> then perhaps it is possible instead to allow for custom builds of the
> Python 3000 executable with different arrangements and configurations of
> these built-in classes.

So how about we reduce the scope of our (!) PEP (or perhaps of a new
one) to two items: (a) add @abstractmethod, and (b) overload
isinstance() and issubclass()? Library authors can do everything they
want with those, and we can always add a specific set of ABCs for
containers and/or numbers later in the 3.0 development cycle.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From jimjjewett at gmail.com  Sun Apr 29 21:51:52 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Sun, 29 Apr 2007 21:51:52 -0400
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<f10g7k$fag$1@sea.gmane.org>
	<5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
	<ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
Message-ID: <fb6fbf560704291851o7ca66a79g1cca54b13523cfbb@mail.gmail.com>

On 4/29/07, Guido van Rossum <guido at python.org> wrote:

> Hmm... Maybe the conclusion to draw from this is that we shouldn't
> make Ring a class? Maybe it ought to be a metaclass, so we could ask
> isinstance(Complex, Ring)?

Yes; all the ABCs are assertions about the class.  (Zope interfaces do
support instance-specific interfaces, which has been brought up as a
relative weakness of ABCs.)

The only thing two subclasses of an *Abstract* class need to have in
common is that they both (independently) meet the requirements of the
ABC.  If not for complexity of implementation, that would be better
described as a common metaclass.

Using a metaclass would also solve the "when to gripe" issue; the
metaclass would gripe if it couldn't make every method concrete.  If
this just used the standard metaclass machinery, then it would mean a
much deeper metaclass hierarchy than we're used to; MutableSet would a
have highly dervived metaclass.

> The more I think about it, it sounds like the right thing to do. To
> take PartiallyOrdered (let's say PO for brevity) as an example, the
> Set class should specify PO as a metaclass. The PO metaclass could
> require that the class implement __lt__ and __le__. If it found a
> class that didn't implement them, it could make the class abstract by
> adding the missing methods to its __abstractmethods__ attribute.

Or by making it a sub(meta)class, instead of a (regular instance) class.

> if it found that the class implemented one but not the other, it could
> inject a default implementation of the other in terms of the one and
> __eq__.

This also allows greater freedom in specifying which subsets of
methods must be defined.

> Now, you could argue that Complex should also be a metaclass. While
> that may mathematically meaningful (for all I know there are people
> doing complex number theory using Complex[Z/n]), for Python's numeric
> classes I think it's better to make Complex a regular class
> representing all the usual complex numbers (i.e. a pair of Real
> numbers).

complex already meets that need.  Complex would be the metaclass
representing the restrictions on the class, so that independing
implementations wouldn't have to fake-inherit from complex.

> I expect that the complex subclasses used in practice are
> all happy under mixed arithmetic using the usual definition of mixed
> arithmetic: convert both arguments to a common base class and compute
> the operation in that domain.

It is reasonable to insist that all Complex classes have a way to
tranform their instances into (builtin) complex instances, if only as
a final fallback.  There is no need for complex to be a base class.

-jJ


From jimjjewett at gmail.com  Sun Apr 29 22:01:57 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Sun, 29 Apr 2007 22:01:57 -0400
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <d11dcfba0704291719n27527c25w5f2ac07e14ea00e3@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<f10g7k$fag$1@sea.gmane.org>
	<5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
	<ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
	<46353440.5060402@acm.org>
	<d11dcfba0704291719n27527c25w5f2ac07e14ea00e3@mail.gmail.com>
Message-ID: <fb6fbf560704291901o2bad63d9x51142f4ed5cace6f@mail.gmail.com>

On 4/29/07, Steven Bethard <steven.bethard at gmail.com> wrote:
> On 4/29/07, Talin <talin at acm.org> wrote:
> > If it were technically possible, I would recommend that this PEP have to
> > run the same gauntlet that any other large library addition would, which
> > is to go through a long period of community feedback and criticism,
> > during which a large number of people actually attempt to use the
> > feature for real work.

> This sounds like a pretty good reason to add __isinstance__() and
> __issubclass__().  Then the various ABCs can be distributed as
> third-party modules but can still make sure that things like
> isinstance(42, Real) and issubclass(int, Complex) are True (or
> whatever other assertions people want to make).

Or isexample, so that we aren't locked into implementing ABCs as base classes.

    def isexample(val, ABC):
        return ABC.__example__(val)

    class ABC(object):
        def __example__(cls, val):
            if val in cls.__instance:  return True
            if val in cls.__non_instance:  return False
            for instclass in type(val).__mro__:
                if instclass in __class:  return True
                if instclass in __non_class: return False
            return False

        ... methods to register classes, unregister subclasses, etc

-jJ


From guido at python.org  Sun Apr 29 22:31:53 2007
From: guido at python.org (Guido van Rossum)
Date: Sun, 29 Apr 2007 19:31:53 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <fb6fbf560704291901o2bad63d9x51142f4ed5cace6f@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<f10g7k$fag$1@sea.gmane.org>
	<5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
	<ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
	<46353440.5060402@acm.org>
	<d11dcfba0704291719n27527c25w5f2ac07e14ea00e3@mail.gmail.com>
	<fb6fbf560704291901o2bad63d9x51142f4ed5cace6f@mail.gmail.com>
Message-ID: <ca471dc20704291931h24df36c8tb0869f48a4cce02b@mail.gmail.com>

On 4/29/07, Jim Jewett <jimjjewett at gmail.com> wrote:
> Or isexample, so that we aren't locked into implementing ABCs as base classes.

You don't have to use the feature even if it exists. :-)

I think there are good reasons to support overriding
isinstance/issubclass beyond ABCs.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido at python.org  Sun Apr 29 23:02:49 2007
From: guido at python.org (Guido van Rossum)
Date: Sun, 29 Apr 2007 20:02:49 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <fb6fbf560704291851o7ca66a79g1cca54b13523cfbb@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<f10g7k$fag$1@sea.gmane.org>
	<5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
	<ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
	<fb6fbf560704291851o7ca66a79g1cca54b13523cfbb@mail.gmail.com>
Message-ID: <ca471dc20704292002i4bb4696fw115b41210679610b@mail.gmail.com>

On 4/29/07, Jim Jewett <jimjjewett at gmail.com> wrote:
> On 4/29/07, Guido van Rossum <guido at python.org> wrote:
> > Hmm... Maybe the conclusion to draw from this is that we shouldn't
> > make Ring a class? Maybe it ought to be a metaclass, so we could ask
> > isinstance(Complex, Ring)?
>
> Yes; all the ABCs are assertions about the class.

I don't think so. Many are quite useful for introspection of instances
as well, e.g. Hashable/Iterable (the whole "One Trick Ponies" section)
as well as the distinction between Sequence and Mapping. It's the
binary operations where the class comes into play.

> (Zope interfaces do
> support instance-specific interfaces, which has been brought up as a
> relative weakness of ABCs.)
>
> The only thing two subclasses of an *Abstract* class need to have in
> common is that they both (independently) meet the requirements of the
> ABC.  If not for complexity of implementation, that would be better
> described as a common metaclass.

Again, not so fast; it depends. The way the Set section of the PEP is
currently written, all sets are comparable (in the subset/superset
sense) to all other sets, and for ComposableSet instances the union,
intersection and both types of differences are also computable across
class boundaries.

> Using a metaclass would also solve the "when to gripe" issue; the
> metaclass would gripe if it couldn't make every method concrete.  If
> this just used the standard metaclass machinery, then it would mean a
> much deeper metaclass hierarchy than we're used to; MutableSet would a
> have highly dervived metaclass.

I think you're going way too fast here.

> > The more I think about it, it sounds like the right thing to do. To
> > take PartiallyOrdered (let's say PO for brevity) as an example, the
> > Set class should specify PO as a metaclass. The PO metaclass could
> > require that the class implement __lt__ and __le__. If it found a
> > class that didn't implement them, it could make the class abstract by
> > adding the missing methods to its __abstractmethods__ attribute.
>
> Or by making it a sub(meta)class, instead of a (regular instance) class.

That makes no sense. Deciding on the fly whether something should be a
class or a metaclass sounds like a fine recipe for end-user confusion.

> > if it found that the class implemented one but not the other, it could
> > inject a default implementation of the other in terms of the one and
> > __eq__.
>
> This also allows greater freedom in specifying which subsets of
> methods must be defined.
>
> > Now, you could argue that Complex should also be a metaclass. While
> > that may mathematically meaningful (for all I know there are people
> > doing complex number theory using Complex[Z/n]), for Python's numeric
> > classes I think it's better to make Complex a regular class
> > representing all the usual complex numbers (i.e. a pair of Real
> > numbers).
>
> complex already meets that need.  Complex would be the metaclass
> representing the restrictions on the class, so that independing
> implementations wouldn't have to fake-inherit from complex.

I was thinking of other representations of complex numbers as found
e.g. in numpy. These vary mostly by using fewer (or more?) bits for
the real and imag parts. They can't realistically subclass complex, as
their implementation is independent; they should subclass Complex, to
indicate that they implement the Complex API. I really think you're
going too far with the metaclass idea.

Now, if we had parameterizable types (for which I've proposed a
notation, e.g. list[int] would be a list of integers, and
Mapping[String, Real] would be a mapping from strings to real numbers;
but I don't expect this to be in py3k, as it needs more
experimentation), Complex might be a parameterizable type, and e.g.
the current concrete complex type could be equated to Complex[float];
but without that, I think it's fine to see Complex as the Abstract
Base Class and complex as one concrete representation.

> > I expect that the complex subclasses used in practice are
> > all happy under mixed arithmetic using the usual definition of mixed
> > arithmetic: convert both arguments to a common base class and compute
> > the operation in that domain.
>
> It is reasonable to insist that all Complex classes have a way to
> tranform their instances into (builtin) complex instances, if only as
> a final fallback.  There is no need for complex to be a base class.

I agree complex shouldn't be a base class (apologies if I implied that
by using lowercase) but I still think Complex should be a base class.

To be honest, I'm not sure what should happen with mixed operations
between classes that only have an abstract common base class. The
normal approach for binary operations is that each side gets a try.
For pairs like int+float this is easy; int.__add__ returns
NotImplemented in this case, and then float.__radd__ is called which
converts the first argument to float and returns a float. For pairs
like numpy's complex 32-bit float and numpy's complex 64-bit float it
should also be easy (numpy is aware of both types and hence always
gets to choose); and for numpy's complex combined with the built-in
complex it's easy enough too (again, numpy always gets to choose, this
time because the built-in complex doesn't know about numpy).

But what if I wrote my own complex type based on decimal.Decimal, and
I encountered a numpy complex? numpy doesn't know about my type and
hence passes the ball to me; but perhaps I don't know about numpy
either, and then a TypeError will be raised. So, to be a good citizen,
I could check if the other arg was a Complex of unknown provenance,
and then I could convert to the built-in complex and pass the ball to
that type. But if all good citizens lived by that rule (and it
*appears* to be a reasonable rule), then numpy, being the ultimate
good citizen, would also convert *my* type to the built-in complex.
But that would mean that if my type *did* know about numpy (but numpy
didn't know about mine), I wouldn't get to choose what to do in half
the cases (if the numpy instance was on the left, it would get and
take the first opportunity). Perhaps we need to extend the built-in
operation processing so that if both sides return NotImplemented,
before raising TypeError, we look for some common base type
implementing the same operation. The abstract Complex type could
provide abstract implementations of the binary operators that would
convert their arguments to the concrete complex type and compute the
result that way. (Hah! Another use case for abstract methods with a
useful implementation! :-)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From David.L.Goldsmith at noaa.gov  Sun Apr 29 23:38:31 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Sun, 29 Apr 2007 20:38:31 -0700
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
	<4634D1CD.8040608@noaa.gov>
	<80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>
Message-ID: <463564B7.9040409@noaa.gov>

Far be it from me to challenge the mighty Wolfram, but I'm not sure that 
using the *formula* for calculating the arctan of a *single* complex 
argument from its real and imaginary parts makes any sense if x and/or y 
are themselves complex (in particular, does Lim(formula), as the 
imaginary part of complex x and/or y approaches zero, approach 
arctan2(realpart(x), realpart(y)?) - without going to the trouble to 
determine it one way or another, I'd be surprised if "their" 
continuation of the arctan2 function from RxR to CxC is (a. e.) 
continuous (not that I know for sure that "mine" is...).

DG

lorenzo bolla wrote:
> You make your point, but I would expect a behaviour similar to 
> Mathematica or Matlab.
>
> From http://documents.wolfram.com/mathematica/functions/ArcTan 
> <http://documents.wolfram.com/mathematica/functions/ArcTan>
> "If x or y is complex, then ArcTan[x , y] gives . When , ArcTan[x, y] 
> gives the number such that and ."
>
> Lorenzo.
>
> On 4/29/07, *David Goldsmith* < David.L.Goldsmith at noaa.gov 
> <mailto:David.L.Goldsmith at noaa.gov>> wrote:
>
>     I'll take a stab at this one; if I miss the mark, people, please
>     chime in.
>
>     What's "strange" here is not numpy's behavior but octave's (IMO).
>     Remember that, over R, arctan is used in two different ways: one is
>     simply as a map from (-inf, inf) -> (-pi/2,pi/2) - here, let's
>     call that
>     invtan; the other is as a means to determine "the angle"
>     (conventionally
>     taken to be between -pi and pi) of a point in the plane - but
>     since, for
>     example, tan(pi/4) = tan(-3pi/4) (and in general tan(x) =
>     tan(x-pi)) to
>     uniquely determine said angle, we need to keep track of and take into
>     account the quadrant in which the point lies; this is (the only
>     reason)
>     why arctan2 is a function of two arguments, one representing the
>     abscissa, the other the ordinate of the point.  But when the
>     argument is
>     complex (arctan2, as the inverse of the tangent function, *is* a valid
>     function on C), this geometric use no longer makes sense, so there's
>     really no reason to implement arctan2(z,w), z, w complex.  If for
>     some
>     reason, e.g., uniformity of algorithmic expression - I don't see any
>     (simple) way to preserve uniformity of code expression - as near as I
>     can tell, you're going to have to implement an if/else if you need to
>     allow for the invtan of two complex arguments - you need to handle
>     arctan2(z,w), implement it as arctan(w/z):
>
>     >>> import numpy
>     >>> numpy.arctan(1j/1j)
>     (0.78539816339744828+0j)
>
>     DG
>
>     lorenzo bolla wrote:
>     > Weird behaviour with arctan2(complex,complex).
>     > Take  a look at this:
>     >
>     > In [11]: numpy.arctan2(1.,1.)
>     > Out[11]: 0.785398163397
>     >
>     > In [12]: numpy.arctan2 (1j,1j)
>     >
>     ---------------------------------------------------------------------------
>     > exceptions.AttributeError                            Traceback (most
>     > recent call last)
>     >
>     > AttributeError: 'complex' object has no attribute 'arctan2'
>     >
>     > same error for:
>     >
>     > In [13]: numpy.arctan2(1j,1.)
>     > In [14]: numpy.arctan2(1.,1j)
>     >
>     > But arctan2 is defined for complex arguments, as far as Octave
>     knows :-) :
>     >
>     > octave:7> atan2(1,1)
>     > ans = 0.78540
>     > octave:8> atan2(1j,1j)
>     > ans = 0
>     > octave:9> atan2(1j,1)
>     > ans = 0
>     > octave:10> atan2(1,1j)
>     > ans = 1.5708
>     >
>     > bug or wanted behaviour?
>     > Lorenzo.
>     >
>     ------------------------------------------------------------------------
>     >
>     > _______________________________________________
>     > Numpy-discussion mailing list
>     > Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
>     > http://projects.scipy.org/mailman/listinfo/numpy-discussion
>     >
>
>     _______________________________________________
>     Numpy-discussion mailing list
>     Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
>     http://projects.scipy.org/mailman/listinfo/numpy-discussion
>     <http://projects.scipy.org/mailman/listinfo/numpy-discussion>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From jimjjewett at gmail.com  Sun Apr 29 23:52:32 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Sun, 29 Apr 2007 23:52:32 -0400
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <ca471dc20704292002i4bb4696fw115b41210679610b@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<f10g7k$fag$1@sea.gmane.org>
	<5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
	<ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
	<fb6fbf560704291851o7ca66a79g1cca54b13523cfbb@mail.gmail.com>
	<ca471dc20704292002i4bb4696fw115b41210679610b@mail.gmail.com>
Message-ID: <fb6fbf560704292052u3e599539m1b85f636040bcd0b@mail.gmail.com>

On 4/29/07, Guido van Rossum <guido at python.org> wrote:
> On 4/29/07, Jim Jewett <jimjjewett at gmail.com> wrote:
> > On 4/29/07, Guido van Rossum <guido at python.org> wrote:
> > > Hmm... Maybe the conclusion to draw from this is that we shouldn't
> > > make Ring a class? Maybe it ought to be a metaclass, so we could ask
> > > isinstance(Complex, Ring)?

> > Yes; all the ABCs are assertions about the class.

> I don't think so. Many are quite useful for introspection of instances
> as well, e.g. Hashable/Iterable (the whole "One Trick Ponies" section)
> as well as the distinction between Sequence and Mapping. It's the
> binary operations where the class comes into play.

I think those are one of the reasons it seemed like we should use
inheritance.  Treating them as assertions about each instance makes
sense -- but it makes just as much sense to treat them as assertions
about the class.  (Is it meaningful to ask about the size of this?
Well, for objects of this class, it is..)

> > The only thing two subclasses of an *Abstract* class need to have in
> > common is that they both (independently) meet the requirements of the
> > ABC.  If not for complexity of implementation, that would be better
> > described as a common metaclass.

> Again, not so fast; it depends. The way the Set section of the PEP is
> currently written, all sets are comparable (in the subset/superset
> sense) to all other sets, and for ComposableSet instances the union,
> intersection and both types of differences are also computable across
> class boundaries.

That is an additional constraint which the Set metaclass imposes --
and it does this by effectively coercing instances of both Set classes
to (buitin) frozenset instances to create the return value.  (It
doesn't actually create the intermediate frozenset instance, but
mySet() | mySet() will return a frozenset rather than a MySet.

> > Using a metaclass would also solve the "when to gripe" issue; the
> > metaclass would gripe if it couldn't make every method concrete.  If
> > this just used the standard metaclass machinery, then it would mean a
> > much deeper metaclass hierarchy than we're used to; MutableSet would a
> > have highly dervived metaclass.

> I think you're going way too fast here.

To be more precise, classes implementing MutabSet would have
MutableSet as a metaclass, which would mean (through inheritance) than
they would also have ComposableSet, Set, Sized, Iterable, and
PartiallyOrdered as metaclasses.

This is legal today, but *I* haven't seen code in the wild with a
metaclass hierarchy that deep.

> > > The more I think about it, it sounds like the right thing to do. To
> > > take PartiallyOrdered (let's say PO for brevity) as an example, the
> > > Set class should specify PO as a metaclass. The PO metaclass could
> > > require that the class implement __lt__ and __le__. If it found a
> > > class that didn't implement them, it could make the class abstract by
> > > adding the missing methods to its __abstractmethods__ attribute.

> > Or by making it a sub(meta)class, instead of a (regular instance) class.

> That makes no sense. Deciding on the fly whether something should be a
> class or a metaclass sounds like a fine recipe for end-user confusion.

Perhaps.  But how is that different from deciding on the fly whether
the class will be instantiable?  Either way, the user does need to
keep track of whether it is abstract; the difference is that when
inheriting, they would have to say

    (metaclass=ABC)   # I know it is an abstract class

instead of

    (ABC)  # Might be a regular class, but I can never instantiate it directly


> I was thinking of other representations of complex numbers as found
> e.g. in numpy. These vary mostly by using fewer (or more?) bits for
> the real and imag parts. They can't realistically subclass complex, as
> their implementation is independent; they should subclass Complex, to
> indicate that they implement the Complex API. I really think you're
> going too far with the metaclass idea.

Or they could have Compex as a metaclass, which would serve the same
introspective needs.

> > > I expect that the complex subclasses used in practice are
> > > all happy under mixed arithmetic using the usual definition of mixed
> > > arithmetic: convert both arguments to a common base class and compute
> > > the operation in that domain.
> >
> > It is reasonable to insist that all Complex classes have a way to
> > tranform their instances into (builtin) complex instances, if only as
> > a final fallback.  There is no need for complex to be a base class.

> I agree complex shouldn't be a base class (apologies if I implied that
> by using lowercase) but I still think Complex should be a base class.

Why?

What do you get by inheriting from it, that you couldn't get by
letting it inject any missing methods?

> take the first opportunity). Perhaps we need to extend the built-in
> operation processing so that if both sides return NotImplemented,
> before raising TypeError, we look for some common base type
> implementing the same operation. The abstract Complex type could
> provide abstract implementations of the binary operators that would
> convert their arguments to the concrete complex type and compute the
> result that way. (Hah! Another use case for abstract methods with a
> useful implementation! :-)

That seems sensible -- but you could also do just do it in the _r* method,
since the regular method already returned NotImplemented.

So Complex could inject a __radd__ method that tried

    self.__add__(complex(other))   # since addition is commutative for Complex

If the method weren't in either class, I would expect it to be a more
general fallback, like "if we don't have __lt__, try __cmp__"

-jJ


From peridot.faceted at gmail.com  Mon Apr 30 00:29:50 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 30 Apr 2007 00:29:50 -0400
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <463564B7.9040409@noaa.gov>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
	<4634D1CD.8040608@noaa.gov>
	<80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>
	<463564B7.9040409@noaa.gov>
Message-ID: <ce557a360704292129secbe4e3tc2a020d53f91d3b2@mail.gmail.com>

On 29/04/07, David Goldsmith <David.L.Goldsmith at noaa.gov> wrote:
> Far be it from me to challenge the mighty Wolfram, but I'm not sure that
> using the *formula* for calculating the arctan of a *single* complex
> argument from its real and imaginary parts makes any sense if x and/or y
> are themselves complex (in particular, does Lim(formula), as the
> imaginary part of complex x and/or y approaches zero, approach
> arctan2(realpart(x), realpart(y)?) - without going to the trouble to
> determine it one way or another, I'd be surprised if "their"
> continuation of the arctan2 function from RxR to CxC is (a. e.)
> continuous (not that I know for sure that "mine" is...).

Well, yes, in fact, theirs is continuous, and in fact analytic, except
along the branch cuts (which they describe). Formulas almost always
yield continuous functions apart from easy-to-recognize cases. (This
can be made into a specific theorem if you're determined.)

Their formula is a pretty reasonable choice, given that it's not at
all clear what arctan2 should mean for complex arguments. But for
numpy it's tempting to simply throw an exception (which would catch
quite a few programmer errors that would otherwise manifest as
nonsense numbers). Still, I suppose defining it on the complex numbers
in a way that is continuous close to the real plane allows people to
put in almost-real complex numbers and get out pretty much the answer
they expect. Does anyone have an application for which they need
arctan2 of, say, (1+i,1-i)?

Anne


From openopt at ukr.net  Mon Apr 30 02:52:00 2007
From: openopt at ukr.net (dmitrey)
Date: Mon, 30 Apr 2007 09:52:00 +0300
Subject: [Numpy-discussion] can't import repmat from numpy
Message-ID: <46359210.5060107@ukr.net>

What's wrong?

start python shell;

from numpy import sin   => all ok

from numpy import repmat =>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name repmat


D.


From lbolla at gmail.com  Mon Apr 30 03:06:17 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Mon, 30 Apr 2007 09:06:17 +0200
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <ce557a360704292129secbe4e3tc2a020d53f91d3b2@mail.gmail.com>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
	<4634D1CD.8040608@noaa.gov>
	<80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>
	<463564B7.9040409@noaa.gov>
	<ce557a360704292129secbe4e3tc2a020d53f91d3b2@mail.gmail.com>
Message-ID: <80c99e790704300006i43e66cay87580233cd2252a3@mail.gmail.com>

me!
I have two cases.

   1. I need that arctan2(1+0.00000001j,1-0.000001j) gives something
   close to arctan2(1,1): any decent analytic prolungation will do!
   2. if someone of you is familiar with electromagnetic problems, in
   particular with Snell's law, will recognize that in case of total
   internal reflection<http://en.wikipedia.org/wiki/Total_internal_reflection>the
wavevector tangential to the interface is real, while the normal one
is
   purely imaginary: hence the angle of diffraction is still given by
   arctan2(k_tangent, k_normal), that, as in Matlab or Octave, should give pi/2
   (that physically means no propagation -- total internal reflection, as
   said).

L.

On 4/30/07, Anne Archibald <peridot.faceted at gmail.com> wrote:
>
> On 29/04/07, David Goldsmith <David.L.Goldsmith at noaa.gov> wrote:
> > Far be it from me to challenge the mighty Wolfram, but I'm not sure that
> > using the *formula* for calculating the arctan of a *single* complex
> > argument from its real and imaginary parts makes any sense if x and/or y
> > are themselves complex (in particular, does Lim(formula), as the
> > imaginary part of complex x and/or y approaches zero, approach
> > arctan2(realpart(x), realpart(y)?) - without going to the trouble to
> > determine it one way or another, I'd be surprised if "their"
> > continuation of the arctan2 function from RxR to CxC is (a. e.)
> > continuous (not that I know for sure that "mine" is...).
>
> Well, yes, in fact, theirs is continuous, and in fact analytic, except
> along the branch cuts (which they describe). Formulas almost always
> yield continuous functions apart from easy-to-recognize cases. (This
> can be made into a specific theorem if you're determined.)
>
> Their formula is a pretty reasonable choice, given that it's not at
> all clear what arctan2 should mean for complex arguments. But for
> numpy it's tempting to simply throw an exception (which would catch
> quite a few programmer errors that would otherwise manifest as
> nonsense numbers). Still, I suppose defining it on the complex numbers
> in a way that is continuous close to the real plane allows people to
> put in almost-real complex numbers and get out pretty much the answer
> they expect. Does anyone have an application for which they need
> arctan2 of, say, (1+i,1-i)?
>
> Anne
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070430/d153d49e/attachment.html>

From lbolla at gmail.com  Mon Apr 30 03:17:08 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Mon, 30 Apr 2007 09:17:08 +0200
Subject: [Numpy-discussion] can't import repmat from numpy
In-Reply-To: <46359210.5060107@ukr.net>
References: <46359210.5060107@ukr.net>
Message-ID: <80c99e790704300017n29de4f5fyb1b66d2cfe114f49@mail.gmail.com>

it looks like repmat is not there anymore... why?
use numpy.repeat and numpy.tile, instead!
hth,
lorenzo.

On 4/30/07, dmitrey <openopt at ukr.net> wrote:
>
> What's wrong?
>
> start python shell;
>
> from numpy import sin   => all ok
>
> from numpy import repmat =>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ImportError: cannot import name repmat
>
>
> D.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070430/a71fc460/attachment.html>

From ptvirtan at cc.hut.fi  Mon Apr 30 03:22:01 2007
From: ptvirtan at cc.hut.fi (Pauli Virtanen)
Date: Mon, 30 Apr 2007 10:22:01 +0300
Subject: [Numpy-discussion] can't import repmat from numpy
In-Reply-To: <46359210.5060107@ukr.net>
References: <46359210.5060107@ukr.net>
Message-ID: <46359919.1000907@cc.hut.fi>

dmitrey kirjoitti:
> What's wrong?
> 
> start python shell;
> 
> from numpy import sin   => all ok
> 
> from numpy import repmat =>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ImportError: cannot import name repmat

In numpy, the equivalent function is called tile:

>>> from numpy import *
>>> a = array([[1,2],[3,4]])
>>> tile(a, (2,3))
array([[1, 2, 1, 2, 1, 2],
       [3, 4, 3, 4, 3, 4],
       [1, 2, 1, 2, 1, 2],
       [3, 4, 3, 4, 3, 4]])

I note that http://www.scipy.org/NumPy_for_Matlab_Users advertises
repmat instead of tile. (repmat is in numpy.matlib, a compatibility
wrapper library) Maybe this should be fixed?

-- 
Pauli Virtanen


From openopt at ukr.net  Mon Apr 30 03:27:34 2007
From: openopt at ukr.net (dmitrey)
Date: Mon, 30 Apr 2007 10:27:34 +0300
Subject: [Numpy-discussion] can't import repmat from numpy
In-Reply-To: <80c99e790704300017n29de4f5fyb1b66d2cfe114f49@mail.gmail.com>
References: <46359210.5060107@ukr.net>
	<80c99e790704300017n29de4f5fyb1b66d2cfe114f49@mail.gmail.com>
Message-ID: <46359A66.5070207@ukr.net>

if it was excluded for any reasons, corresponding changes in 
http://www.scipy.org/NumPy_for_Matlab_Users should be done
D.

lorenzo bolla wrote:
> it looks like repmat is not there anymore... why?
> use numpy.repeat and numpy.tile, instead!
> hth,
> lorenzo.
>
> On 4/30/07, *dmitrey* < openopt at ukr.net <mailto:openopt at ukr.net>> wrote:
>
>     What's wrong?
>
>     start python shell;
>
>     from numpy import sin   => all ok
>
>     from numpy import repmat =>
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in <module>
>     ImportError: cannot import name repmat
>
>
>     D.
>     _______________________________________________
>     Numpy-discussion mailing list
>     Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
>     http://projects.scipy.org/mailman/listinfo/numpy-discussion
>     <http://projects.scipy.org/mailman/listinfo/numpy-discussion>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From wbaxter at gmail.com  Mon Apr 30 04:51:31 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Mon, 30 Apr 2007 17:51:31 +0900
Subject: [Numpy-discussion] can't import repmat from numpy
In-Reply-To: <46359A66.5070207@ukr.net>
References: <46359210.5060107@ukr.net>
	<80c99e790704300017n29de4f5fyb1b66d2cfe114f49@mail.gmail.com>
	<46359A66.5070207@ukr.net>
Message-ID: <e86a5fd00704300151v3dd380dep141ede6ea2720f72@mail.gmail.com>

It's not on the matlab page simpy because numpy.tile didn't exist when
the page was created.  It should be fixed.  But repmat is still there
in numpy.matlib (I think that was what it was called.)
--bb

On 4/30/07, dmitrey <openopt at ukr.net> wrote:
> if it was excluded for any reasons, corresponding changes in
> http://www.scipy.org/NumPy_for_Matlab_Users should be done
> D.
>
> lorenzo bolla wrote:
> > it looks like repmat is not there anymore... why?
> > use numpy.repeat and numpy.tile, instead!
> > hth,
> > lorenzo.
> >
> > On 4/30/07, *dmitrey* < openopt at ukr.net <mailto:openopt at ukr.net>> wrote:
> >
> >     What's wrong?
> >
> >     start python shell;
> >
> >     from numpy import sin   => all ok
> >
> >     from numpy import repmat =>
> >     Traceback (most recent call last):
> >       File "<stdin>", line 1, in <module>
> >     ImportError: cannot import name repmat
> >
> >
> >     D.
> >     _______________________________________________
> >     Numpy-discussion mailing list
> >     Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
> >     http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >     <http://projects.scipy.org/mailman/listinfo/numpy-discussion>
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From David.L.Goldsmith at noaa.gov  Mon Apr 30 10:01:03 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Mon, 30 Apr 2007 07:01:03 -0700
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <80c99e790704300006i43e66cay87580233cd2252a3@mail.gmail.com>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
	<4634D1CD.8040608@noaa.gov>
	<80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>
	<463564B7.9040409@noaa.gov>
	<ce557a360704292129secbe4e3tc2a020d53f91d3b2@mail.gmail.com>
	<80c99e790704300006i43e66cay87580233cd2252a3@mail.gmail.com>
Message-ID: <4635F69F.40201@noaa.gov>

lorenzo bolla wrote:
> me!
> I have two cases.
>
>    1. I need that arctan2(1+0.00000001j,1-0.000001j) gives something
>       close to arctan2(1,1): any decent analytic prolungation will do!
>
This is the foreseeable use case described by Anne.

In any event, I stand not only corrected, but embarrassed (for numpy):

    Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
    [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
     >>> import numpy as N
     >>> complex(0,1)
    1j
     >>> N.arctan(1)
    0.78539816339744828
     >>> N.arctan(complex(0,1))
    Warning: invalid value encountered in arctan
    (nannanj)

I agree that arctan should be implemented for _at least_ *one* complex 
argument...

DG
>
>    1. if someone of you is familiar with electromagnetic problems, in
>       particular with Snell's law, will recognize that in case of
>       total internal reflection
>       <http://en.wikipedia.org/wiki/Total_internal_reflection> the
>       wavevector tangential to the interface is real, while the normal
>       one is purely imaginary: hence the angle of diffraction is still
>       given by arctan2(k_tangent, k_normal), that, as in Matlab or
>       Octave, should give pi/2 (that physically means no propagation
>       -- total internal reflection, as said).
>
> L.
>
> On 4/30/07, *Anne Archibald* <peridot.faceted at gmail.com 
> <mailto:peridot.faceted at gmail.com>> wrote:
>
>     On 29/04/07, David Goldsmith <David.L.Goldsmith at noaa.gov
>     <mailto:David.L.Goldsmith at noaa.gov>> wrote:
>     > Far be it from me to challenge the mighty Wolfram, but I'm not
>     sure that
>     > using the *formula* for calculating the arctan of a *single*
>     complex
>     > argument from its real and imaginary parts makes any sense if x
>     and/or y
>     > are themselves complex (in particular, does Lim(formula), as the
>     > imaginary part of complex x and/or y approaches zero, approach
>     > arctan2(realpart(x), realpart(y)?) - without going to the trouble to
>     > determine it one way or another, I'd be surprised if "their"
>     > continuation of the arctan2 function from RxR to CxC is (a. e.)
>     > continuous (not that I know for sure that "mine" is...).
>
>     Well, yes, in fact, theirs is continuous, and in fact analytic, except
>     along the branch cuts (which they describe). Formulas almost always
>     yield continuous functions apart from easy-to-recognize cases. (This
>     can be made into a specific theorem if you're determined.)
>
>     Their formula is a pretty reasonable choice, given that it's not at
>     all clear what arctan2 should mean for complex arguments. But for
>     numpy it's tempting to simply throw an exception (which would catch
>     quite a few programmer errors that would otherwise manifest as
>     nonsense numbers). Still, I suppose defining it on the complex
>     numbers
>     in a way that is continuous close to the real plane allows people to
>     put in almost-real complex numbers and get out pretty much the answer
>     they expect. Does anyone have an application for which they need
>     arctan2 of, say, (1+i,1-i)?
>
>     Anne
>     _______________________________________________
>     Numpy-discussion mailing list
>     Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
>     http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From tim.hochberg at ieee.org  Mon Apr 30 10:11:02 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Mon, 30 Apr 2007 07:11:02 -0700
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <4635F69F.40201@noaa.gov>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
	<4634D1CD.8040608@noaa.gov>
	<80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>
	<463564B7.9040409@noaa.gov>
	<ce557a360704292129secbe4e3tc2a020d53f91d3b2@mail.gmail.com>
	<80c99e790704300006i43e66cay87580233cd2252a3@mail.gmail.com>
	<4635F69F.40201@noaa.gov>
Message-ID: <e4412d6b0704300711u50947030q29ffb87bafeb3e5d@mail.gmail.com>

On 4/30/07, David Goldsmith <David.L.Goldsmith at noaa.gov> wrote:
>
> lorenzo bolla wrote:
> > me!
> > I have two cases.
> >
> >    1. I need that arctan2(1+0.00000001j,1-0.000001j) gives something
> >       close to arctan2(1,1): any decent analytic prolungation will do!
> >
> This is the foreseeable use case described by Anne.
>
> In any event, I stand not only corrected, but embarrassed (for numpy):
>
>     Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
>     [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
>     Type "help", "copyright", "credits" or "license" for more information.
>      >>> import numpy as N
>      >>> complex(0,1)
>     1j
>      >>> N.arctan(1)
>     0.78539816339744828
>      >>> N.arctan(complex(0,1))
>     Warning: invalid value encountered in arctan
>     (nannanj)
>
> I agree that arctan should be implemented for _at least_ *one* complex
> argument...


It is, you should look at that error a bit more carefully...

-tim

(hint what is arctan(0+1j)?)


DG
> >
> >    1. if someone of you is familiar with electromagnetic problems, in
> >       particular with Snell's law, will recognize that in case of
> >       total internal reflection
> >       <http://en.wikipedia.org/wiki/Total_internal_reflection> the
> >       wavevector tangential to the interface is real, while the normal
> >       one is purely imaginary: hence the angle of diffraction is still
> >       given by arctan2(k_tangent, k_normal), that, as in Matlab or
> >       Octave, should give pi/2 (that physically means no propagation
> >       -- total internal reflection, as said).
> >
> > L.
> >
> > On 4/30/07, *Anne Archibald* <peridot.faceted at gmail.com
> > <mailto:peridot.faceted at gmail.com>> wrote:
> >
> >     On 29/04/07, David Goldsmith <David.L.Goldsmith at noaa.gov
> >     <mailto:David.L.Goldsmith at noaa.gov>> wrote:
> >     > Far be it from me to challenge the mighty Wolfram, but I'm not
> >     sure that
> >     > using the *formula* for calculating the arctan of a *single*
> >     complex
> >     > argument from its real and imaginary parts makes any sense if x
> >     and/or y
> >     > are themselves complex (in particular, does Lim(formula), as the
> >     > imaginary part of complex x and/or y approaches zero, approach
> >     > arctan2(realpart(x), realpart(y)?) - without going to the trouble
> to
> >     > determine it one way or another, I'd be surprised if "their"
> >     > continuation of the arctan2 function from RxR to CxC is (a. e.)
> >     > continuous (not that I know for sure that "mine" is...).
> >
> >     Well, yes, in fact, theirs is continuous, and in fact analytic,
> except
> >     along the branch cuts (which they describe). Formulas almost always
> >     yield continuous functions apart from easy-to-recognize cases. (This
> >     can be made into a specific theorem if you're determined.)
> >
> >     Their formula is a pretty reasonable choice, given that it's not at
> >     all clear what arctan2 should mean for complex arguments. But for
> >     numpy it's tempting to simply throw an exception (which would catch
> >     quite a few programmer errors that would otherwise manifest as
> >     nonsense numbers). Still, I suppose defining it on the complex
> >     numbers
> >     in a way that is continuous close to the real plane allows people to
> >     put in almost-real complex numbers and get out pretty much the
> answer
> >     they expect. Does anyone have an application for which they need
> >     arctan2 of, say, (1+i,1-i)?
> >
> >     Anne
> >     _______________________________________________
> >     Numpy-discussion mailing list
> >     Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
> >     http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070430/0eaef35a/attachment.html>

From David.L.Goldsmith at noaa.gov  Mon Apr 30 10:39:16 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Mon, 30 Apr 2007 07:39:16 -0700
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <e4412d6b0704300711u50947030q29ffb87bafeb3e5d@mail.gmail.com>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
	<4634D1CD.8040608@noaa.gov>
	<80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>
	<463564B7.9040409@noaa.gov>
	<ce557a360704292129secbe4e3tc2a020d53f91d3b2@mail.gmail.com>
	<80c99e790704300006i43e66cay87580233cd2252a3@mail.gmail.com>
	<4635F69F.40201@noaa.gov>
	<e4412d6b0704300711u50947030q29ffb87bafeb3e5d@mail.gmail.com>
Message-ID: <4635FF94.5020205@noaa.gov>


> (hint what is arctan(0+1j)?)
>  
Well, at the risk of embarrassing myself, using arctan(x+iy) = I get:

arctan(0+1i) = -i*log((0+i*1)/sqrt(0^2 + 1^2)) = -i*log(i/1) = -i*log(i) 
= -i*log(exp(i*pi/2)) = -i*i*pi/2 = pi/2...

Is there some reason I'm forgetting (e.g., a branch cut convention or 
something) why this is wrong?

DG


From tim.hochberg at ieee.org  Mon Apr 30 10:49:57 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Mon, 30 Apr 2007 07:49:57 -0700
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <4635FF94.5020205@noaa.gov>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
	<4634D1CD.8040608@noaa.gov>
	<80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>
	<463564B7.9040409@noaa.gov>
	<ce557a360704292129secbe4e3tc2a020d53f91d3b2@mail.gmail.com>
	<80c99e790704300006i43e66cay87580233cd2252a3@mail.gmail.com>
	<4635F69F.40201@noaa.gov>
	<e4412d6b0704300711u50947030q29ffb87bafeb3e5d@mail.gmail.com>
	<4635FF94.5020205@noaa.gov>
Message-ID: <e4412d6b0704300749v3010b455y8812e1161012df5e@mail.gmail.com>

On 4/30/07, David Goldsmith <David.L.Goldsmith at noaa.gov> wrote:
>
>
> > (hint what is arctan(0+1j)?)
> >
> Well, at the risk of embarrassing myself, using arctan(x+iy) = I get:
>
> arctan(0+1i) = -i*log((0+i*1)/sqrt(0^2 + 1^2)) = -i*log(i/1) = -i*log(i)
> = -i*log(exp(i*pi/2)) = -i*i*pi/2 = pi/2...
>
> Is there some reason I'm forgetting (e.g., a branch cut convention or
> something) why this is wrong?


Oops. It may well be OK, I answered too fast (I'd explain my thinking, but
ti would be embarassing). However, I believe atan works in general for
complex numbers but it may well have a problem for this specific case.

I should look at this more closely though since the behaviour near zero does
seem a little odd. No time right now, but maybe later today.


DG
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 

//=][=\\

tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070430/62ec5bc6/attachment.html>

From lbolla at gmail.com  Mon Apr 30 10:50:33 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Mon, 30 Apr 2007 16:50:33 +0200
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <4635FF94.5020205@noaa.gov>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
	<4634D1CD.8040608@noaa.gov>
	<80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>
	<463564B7.9040409@noaa.gov>
	<ce557a360704292129secbe4e3tc2a020d53f91d3b2@mail.gmail.com>
	<80c99e790704300006i43e66cay87580233cd2252a3@mail.gmail.com>
	<4635F69F.40201@noaa.gov>
	<e4412d6b0704300711u50947030q29ffb87bafeb3e5d@mail.gmail.com>
	<4635FF94.5020205@noaa.gov>
Message-ID: <80c99e790704300750o115b1c89md04fbd4b23143b0@mail.gmail.com>

hold on, david. the formula I posted previously from wolfram is ArcTan[x,y]
with x or y complex: its the same of arctan2(x,y). arctan is another
function (even though arctan2(y,x) should be "a better" arctan(y/x)).

the correct formula for y = arctan(x), with any x (real or complex), should
be (if I still can play with sin and cos...):

y = arctan(x) = 1/(2j) * log((1j-x)/(1j+x))

[ you can get it doing: y = arctan(x) --> x = tan(y) = sin(x)/cos(x) = -1j *
(exp(1j*y)-exp(-1j*y))/(exp(1j*y)+exp(-1j*y); then let z = exp(1j*y) and
solve in z.]

I've tested the formula and it seems ok for different inputs (I've checked
that tan(arctan(x)) == x):
---------------------------
octave:56> x = 1; tan(1/2/1j*log((1j-x)/(1j+x)))
ans = 1.0000
octave:57> x = 1j; tan(1/2/1j*log((1j-x)/(1j+x)))
ans = -0 + 1i
octave:58> x = 2j; tan(1/2/1j*log((1j-x)/(1j+x)))
ans = 1.8369e-16 + 2.0000e+00i
octave:59> x = 1+2j; tan(1/2/1j*log((1j-x)/(1j+x)))
ans = 1.0000 + 2.0000i
---------------------------

hth,
L.

On 4/30/07, David Goldsmith <David.L.Goldsmith at noaa.gov> wrote:
>
>
> > (hint what is arctan(0+1j)?)
> >
> Well, at the risk of embarrassing myself, using arctan(x+iy) = I get:
>
> arctan(0+1i) = -i*log((0+i*1)/sqrt(0^2 + 1^2)) = -i*log(i/1) = -i*log(i)
> = -i*log(exp(i*pi/2)) = -i*i*pi/2 = pi/2...
>
> Is there some reason I'm forgetting (e.g., a branch cut convention or
> something) why this is wrong?
>
> DG
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070430/8d5b3f5e/attachment.html>

From David.L.Goldsmith at noaa.gov  Mon Apr 30 11:26:49 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Mon, 30 Apr 2007 08:26:49 -0700
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <80c99e790704300750o115b1c89md04fbd4b23143b0@mail.gmail.com>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
	<4634D1CD.8040608@noaa.gov>
	<80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>
	<463564B7.9040409@noaa.gov>
	<ce557a360704292129secbe4e3tc2a020d53f91d3b2@mail.gmail.com>
	<80c99e790704300006i43e66cay87580233cd2252a3@mail.gmail.com>
	<4635F69F.40201@noaa.gov>
	<e4412d6b0704300711u50947030q29ffb87bafeb3e5d@mail.gmail.com>
	<4635FF94.5020205@noaa.gov>
	<80c99e790704300750o115b1c89md04fbd4b23143b0@mail.gmail.com>
Message-ID: <46360AB9.5060205@noaa.gov>

lorenzo bolla wrote:
> hold on, david. the formula I posted previously from wolfram is 
> ArcTan[x,y] with x or y complex: its the same of arctan2(x,y). arctan 
> is another function (even though arctan2(y,x) should be "a better" 
> arctan(y/x)).
>
> the correct formula for y = arctan(x), with any x (real or complex), 
> should be (if I still can play with sin and cos...):
>
> y = arctan(x) = 1/(2j) * log((1j-x)/(1j+x))
After a little algebra, they're the same formula (as they must be; hint: 
"rationalize" the denominator inside the log and bring the 1/2 inside 
the log, making it a square root, and finally, rewrite 1/i as -i); I 
have to run my kid to school right now, but if you want me to flesh this 
out, reply and I'll type it out and send it when I get back..

DG
>
> [ you can get it doing: y = arctan(x) --> x = tan(y) = sin(x)/cos(x) = 
> -1j * (exp(1j*y)-exp(-1j*y))/(exp(1j*y)+exp(-1j*y); then let z = 
> exp(1j*y) and solve in z.]
>
> I've tested the formula and it seems ok for different inputs (I've 
> checked that tan(arctan(x)) == x):
> ---------------------------
> octave:56> x = 1; tan(1/2/1j*log((1j-x)/(1j+x)))
> ans = 1.0000
> octave:57> x = 1j; tan(1/2/1j*log((1j-x)/(1j+x)))
> ans = -0 + 1i
> octave:58> x = 2j; tan(1/2/1j*log((1j-x)/(1j+x)))
> ans = 1.8369e-16 + 2.0000e+00i
> octave:59> x = 1+2j; tan(1/2/1j*log((1j-x)/(1j+x)))
> ans = 1.0000 + 2.0000i
> ---------------------------
>
> hth,
> L.
>
> On 4/30/07, *David Goldsmith* <David.L.Goldsmith at noaa.gov 
> <mailto:David.L.Goldsmith at noaa.gov>> wrote:
>
>
>     > (hint what is arctan(0+1j)?)
>     >
>     Well, at the risk of embarrassing myself, using arctan(x+iy) = I get:
>
>     arctan(0+1i) = -i*log((0+i*1)/sqrt(0^2 + 1^2)) = -i*log(i/1) =
>     -i*log(i)
>     = -i*log(exp(i*pi/2)) = -i*i*pi/2 = pi/2...
>
>     Is there some reason I'm forgetting (e.g., a branch cut convention or
>     something) why this is wrong?
>
>     DG
>     _______________________________________________
>     Numpy-discussion mailing list
>     Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
>     http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From David.L.Goldsmith at noaa.gov  Mon Apr 30 11:33:41 2007
From: David.L.Goldsmith at noaa.gov (David Goldsmith)
Date: Mon, 30 Apr 2007 08:33:41 -0700
Subject: [Numpy-discussion] arctan2 with complex args
In-Reply-To: <e4412d6b0704300749v3010b455y8812e1161012df5e@mail.gmail.com>
References: <80c99e790704290418g389784cfq566db7e257d252fb@mail.gmail.com>
	<4634D1CD.8040608@noaa.gov>
	<80c99e790704291150j4716014cu310111b1aee9a7a6@mail.gmail.com>
	<463564B7.9040409@noaa.gov>
	<ce557a360704292129secbe4e3tc2a020d53f91d3b2@mail.gmail.com>
	<80c99e790704300006i43e66cay87580233cd2252a3@mail.gmail.com>
	<4635F69F.40201@noaa.gov>
	<e4412d6b0704300711u50947030q29ffb87bafeb3e5d@mail.gmail.com>
	<4635FF94.5020205@noaa.gov>
	<e4412d6b0704300749v3010b455y8812e1161012df5e@mail.gmail.com>
Message-ID: <46360C55.60001@noaa.gov>

Timothy Hochberg wrote:
>
>
> On 4/30/07, *David Goldsmith* <David.L.Goldsmith at noaa.gov 
> <mailto:David.L.Goldsmith at noaa.gov>> wrote:
>
>
>     > (hint what is arctan(0+1j)?)
>     >
>     Well, at the risk of embarrassing myself, using arctan(x+iy) = I get:
>
>     arctan(0+1i) = -i*log((0+i*1)/sqrt(0^2 + 1^2)) = -i*log(i/1) =
>     -i*log(i)
>     = -i*log(exp(i*pi/2)) = -i*i*pi/2 = pi/2...
>
>     Is there some reason I'm forgetting (e.g., a branch cut convention or
>     something) why this is wrong?
>
>
> Oops. It may well be OK, I answered too fast (I'd explain my thinking, 
> but ti would be embarassing).
Was it: "pi/2 is outside the range of arctan"? (No worries: I thought of 
this too when I saw what value fell out of the formula, but then, it's 
outside the range of the *real-valued* function, but clearly it's not 
outside the range of the *complex-valued* function.)

DG
> However, I believe atan works in general for complex numbers but it 
> may well have a problem for this specific case.
>
> I should look at this more closely though since the behaviour near 
> zero does seem a little odd. No time right now, but maybe later today.
>  
>
>     DG
>     _______________________________________________
>     Numpy-discussion mailing list
>     Numpy-discussion at scipy.org <mailto:Numpy-discussion at scipy.org>
>     http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>
> -- 
>
> //=][=\\
>
> tim.hochberg at ieee.org <mailto:tim.hochberg at ieee.org>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From Chris.Barker at noaa.gov  Mon Apr 30 12:31:13 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 30 Apr 2007 09:31:13 -0700
Subject: [Numpy-discussion] simpliest way to check: array x is float,
 not integer
In-Reply-To: <463478A5.1050507@ukr.net>
References: <463478A5.1050507@ukr.net>
Message-ID: <463619D1.5@noaa.gov>

dmitrey wrote:
> hi all,
> please inform me what is the simplest way to check, does the vector x 
> that came to my func is float or integer. I.e. someone can pass to my 
> func for example x0 = numpy.array([1, 0, 0]) and it can yield wrong 
> unexpected results vs numpy.array([1.0, 0, 0]) .

Usually, it's not so much that you need to know whether a parameter is a 
float or an integer, but rather that your function is designed to work 
with one or the other. In that case, you can force it with as array:

def test(InputArray):
     InputArray = numpy.asarray(InputArray, numpy.float)


asarray is a no-op if the input is already of the type you've specified

It's handy, as it will take *anything* that can be turned into an array 
of the specified type.

This won't work the function needs to change the input in place, however.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From blais at furius.ca  Mon Apr 30 17:59:57 2007
From: blais at furius.ca (Martin Blais)
Date: Mon, 30 Apr 2007 14:59:57 -0700
Subject: [Numpy-discussion] Documentation examples with doc
Message-ID: <1b151690704301459pa68a238u5cad5461dd929a13@mail.gmail.com>

Hi

Where is the script that generates the Numpy Examples with Doc on the
scipy website?
I cannot find it in the numpy svn.
Maybe I would like to create a version that generates LaTeX for
Python-style documentation.  I find searching through this big web
page a slightly painful experience.

thanks,


From robert.kern at gmail.com  Mon Apr 30 18:11:08 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 30 Apr 2007 17:11:08 -0500
Subject: [Numpy-discussion] Documentation examples with doc
In-Reply-To: <1b151690704301459pa68a238u5cad5461dd929a13@mail.gmail.com>
References: <1b151690704301459pa68a238u5cad5461dd929a13@mail.gmail.com>
Message-ID: <4636697C.5000108@gmail.com>

Martin Blais wrote:
> Hi
> 
> Where is the script that generates the Numpy Examples with Doc on the
> scipy website?
> I cannot find it in the numpy svn.
> Maybe I would like to create a version that generates LaTeX for
> Python-style documentation.  I find searching through this big web
> page a slightly painful experience.

The link "auto-generated" at the top of the page points to the script:

  http://www.scipy.org/Numpy_Example_List_With_Doc/script

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From tcickovs at nd.edu  Fri Apr 27 15:33:51 2007
From: tcickovs at nd.edu (Trevor M Cickovski)
Date: Fri, 27 Apr 2007 15:33:51 -0400
Subject: [Numpy-discussion] PyArray_FromDimsAndData weird seg fault
Message-ID: <4632501F.9020807@nd.edu>

Hi,

I'm using SWIG to wrap a function that calls the NumPy routine PyArray_FromDimsAndData.

PyObject* toArray() {
     int dims = 5;
     double* myH;
     myH = (double*)malloc(dims*sizeof(double));
     myH[0] = 0; myH[1] = 1; myH[2] = 2; myH[3] = 3; myH[4] = 4;
     return PyArray_FromDimsAndData(1,&dims,PyArray_FLOAT,(char*)myH);
}

However, for some reason when I call this from Python, the 
PyArray_FromDimsAndData()
seg faults everytime.  Does anybody see something wrong with my use of the function
itself?
Thanks!
-Trevor


-- 
*************************************************************
Trevor M. Cickovski
Laboratory For Computational Life Sciences
University of Notre Dame
325 Cushing Hall
Notre Dame, IN 46556
T: (574) 631-3906
F: (574) 631-9260
"Settle never, strive always."
*************************************************************



From benjamin at decideur.info  Sat Apr 28 14:37:39 2007
From: benjamin at decideur.info (Benjamin Thyreau)
Date: Sat, 28 Apr 2007 20:37:39 +0200
Subject: [Numpy-discussion] matlab vs. python question
In-Reply-To: <1177783414.369496.115660@y80g2000hsf.googlegroups.com>
References: <f0o4h9$k5m$1@sea.gmane.org>
	<1e2af89e0704280409u3f6a7d99j2a7283da373dced3@mail.gmail.com>
	<1177783414.369496.115660@y80g2000hsf.googlegroups.com>
Message-ID: <200704282037.39721.benjamin@decideur.info>

Le Samedi 28 Avril 2007 20:03, Simon Berube a ?crit?:
> (...)
>  On the other hand, if you are more interested in small
> projects where speed of development is more important than long term
> sustainability of the code Matlab is probably better. This is usually
> the case in research environment, at least until you figure out
> exactly what you want to do, then switch to python.
>
> I hope this help,  (...)

I guess the point of the question was, in what way should numpy/scipy be 
improved in order to get the same usability at those cases, according to 
you ? It's not as obvious as you make it sound, and there also are for sure 
some advantages in the ipython/scipy stack over matlab's prompt, 
usability-wise. Any experiences which helps in improving it is thus useful
Thanks,
Benjamin


From spacey at lenin.net  Sat Apr 28 15:27:39 2007
From: spacey at lenin.net (Peter C. Norton)
Date: Sat, 28 Apr 2007 12:27:39 -0700
Subject: [Numpy-discussion] Building numpy - setting the run path
In-Reply-To: <4632F5BE.8070202@ar.media.kyoto-u.ac.jp>
References: <20070427222747.GH22477@lenin.net>
	<4632F5BE.8070202@ar.media.kyoto-u.ac.jp>
Message-ID: <20070428192739.GI22477@lenin.net>

On Sat, Apr 28, 2007 at 04:20:30PM +0900, David Cournapeau wrote:
> Peter C. Norton wrote:
> > Building numpy for my company's solaris distribution involves requring
> > a run_path for the lapack+blas libraries we're using (libsunperf,
> > though I'm considering swapping out for gsl since we may use that).
> >
> > The situation we're in is that we need gcc to get the -R for paths
> > like /usr/local/gnu/lib /usr/local/python2.5.1/lib, etc. but the
> > default python distutils raise a not implemented exception on this.
> May I ask what the -R path option is doing ? I cannot find it in the doc 
> of my gcc (on linux), and from your email, I don't see what it is doing 
> ? Does it say where to find library when linking ? When running ? Does 
> it set the patch in the binary ?

It sets the run path, but it supercedes $LD_RUN_PATH.

-Peter

-- 
The 5 year plan:
In five years we'll make up another plan.
Or just re-use this one.



From talin at acm.org  Sun Apr 29 20:11:44 2007
From: talin at acm.org (Talin)
Date: Sun, 29 Apr 2007 17:11:44 -0700
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
 Numbers (and other algebraic entities)
In-Reply-To: <ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>	<f10g7k$fag$1@sea.gmane.org>	<5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
	<ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
Message-ID: <46353440.5060402@acm.org>

Guido van Rossum wrote:
> Maybe we should stop trying to capture radically different
> mathematical number systems using classes or types, and limit
> ourselves to capturing the systems one learns in high school: C, R, Q,
> Z, and (perhaps) N (really N0). The concrete types would be complex <:
> C, float<:R, Decimal<:R, int<:Z. NumPy would have many more. One could
> argue that float and Decimal are <:Q, but I'm not sure if that makes
> things better pragmatically; I guess I'm coming from the old Algol
> school where float was actually called real (and in retrospect I wish
> I'd called it that in Python). I'd rather reserve membership of Q for
> an infinite precision rational type (which many people have
> independently implemented).

I haven't really been following this discussion, given my lack of 
understanding of the issues involved, but I want to make one observation 
about the discussion.

Normally, when someone suggests an idea for a major addition to Python 
of this scope, the standard response is that they should go develop it 
as a 3rd-party package and see if becomes popular, and if it does it can 
be considered for inclusion in the standard library.

Unfortunately, we are somewhat constrained in this case, because we're 
talking about altering the behavior of some fairly fundamental built-in 
types - which means that it's going to be hard to implement it as an 
add-on library.

And yet, it seems to me that this particular set of features really does 
need a long gestation period compared to some others that we've 
discussed. Most of the 3000-series PEPs are really about a fairly small 
set of base decisions. Even the long PEPs are more about descriptions of 
the logical consequences of those decisions than about the decisions 
themselves. The ABC PEPs are different, in that they are standardizing a 
whole slew of things all at once. Moreover, I think there is a real 
danger here of a kind of ivory-tower decision making, isolated from 
day-to-day writing of applications to keep them grounded.

The question of whether to limit to the "high school" number classes, or 
to go with the more mathematically abstract set of things is a decision 
which, it seems to me, ought to be made in the context of actual use 
cases, rather than abstract theorizing about use cases. (I'm also 
generally supportive about copying what other languages have done, on 
the theory that they too have been tested by real-world use.)

If it were technically possible, I would recommend that this PEP have to 
run the same gauntlet that any other large library addition would, which 
is to go through a long period of community feedback and criticism, 
during which a large number of people actually attempt to use the 
feature for real work. I also think, in this case, that the special 
power of "core python developer fiat" should not be invoked unless it 
has to be, because I don't think that there is a firm basis for making 
such a judgment yet.

I would also suggest that some thought be given to ways to allow for 
experimentation with different variations of this feature. If it is not 
possible to make these numeric classes definable in Python at runtime, 
then perhaps it is possible instead to allow for custom builds of the 
Python 3000 executable with different arrangements and configurations of 
these built-in classes.

-- Talin


From steven.bethard at gmail.com  Sun Apr 29 20:19:14 2007
From: steven.bethard at gmail.com (Steven Bethard)
Date: Sun, 29 Apr 2007 18:19:14 -0600
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <46353440.5060402@acm.org>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<f10g7k$fag$1@sea.gmane.org>
	<5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
	<ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
	<46353440.5060402@acm.org>
Message-ID: <d11dcfba0704291719n27527c25w5f2ac07e14ea00e3@mail.gmail.com>

On 4/29/07, Talin <talin at acm.org> wrote:
> If it were technically possible, I would recommend that this PEP have to
> run the same gauntlet that any other large library addition would, which
> is to go through a long period of community feedback and criticism,
> during which a large number of people actually attempt to use the
> feature for real work.

This sounds like a pretty good reason to add __isinstance__() and
__issubclass__().  Then the various ABCs can be distributed as
third-party modules but can still make sure that things like
isinstance(42, Real) and issubclass(int, Complex) are True (or
whatever other assertions people want to make).

STeVe
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
        --- Bucky Katt, Get Fuzzy


From janssen at parc.com  Mon Apr 30 12:01:09 2007
From: janssen at parc.com (Bill Janssen)
Date: Mon, 30 Apr 2007 09:01:09 PDT
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <ca471dc20704291826p6cb8e001gf3df7a3dc6160667@mail.gmail.com> 
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<f10g7k$fag$1@sea.gmane.org>
	<5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
	<ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
	<46353440.5060402@acm.org>
	<ca471dc20704291826p6cb8e001gf3df7a3dc6160667@mail.gmail.com>
Message-ID: <07Apr30.090110pdt."57996"@synergy1.parc.xerox.com>

GvR wrote:
> So how about we reduce the scope of our (!) PEP (or perhaps of a new
> one) to two items: (a) add @abstractmethod, and (b) overload
> isinstance() and issubclass()? Library authors can do everything they
> want with those, and we can always add a specific set of ABCs for
> containers and/or numbers later in the 3.0 development cycle.

-1.  Adding mechanism without content seems less than ideal, despite
Talin's misgivings.  I'd recommend adding the base classes in, and see
how they work in the earliest 3.0 releases, then modify them as
necessary in subsequent releases.

Bill


From gnchen at cortechs.net  Mon Apr 30 12:12:59 2007
From: gnchen at cortechs.net (Gennan Chen)
Date: Mon, 30 Apr 2007 09:12:59 -0700
Subject: [Numpy-discussion] cannot compile under centos 5 after weekend svn
	update
Message-ID: <1471CA42-3B78-4BAE-A5BB-AB960DFE2AD2@cortechs.net>

Hi! All,


I cannot compile numpy anymore after last weekend's update. Anyone  
know what's wrong??

[gnchen at cortechs25:numpy]$ python setup.py config
Running from numpy source directory.
F2PY Version 2_3730
blas_opt_info:
blas_mkl_info:
Traceback (most recent call last):
   File "setup.py", line 89, in ?
     setup_package()
   File "setup.py", line 82, in setup_package
     configuration=configuration )
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/core.py",  
line 144, in setup
     config = configuration()
   File "setup.py", line 48, in configuration
     config.add_subpackage('numpy')
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
misc_util.py", line 765, in add_subpackage
     caller_level = 2)
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
misc_util.py", line 748, in get_subpackage
     caller_level = caller_level + 1)
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
misc_util.py", line 695, in _get_configuration_from_setup_py
     config = setup_module.configuration(*args)
   File "./numpy/setup.py", line 9, in configuration
     config.add_subpackage('core')
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
misc_util.py", line 765, in add_subpackage
     caller_level = 2)
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
misc_util.py", line 748, in get_subpackage
     caller_level = caller_level + 1)
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
misc_util.py", line 695, in _get_configuration_from_setup_py
     config = setup_module.configuration(*args)
   File "numpy/core/setup.py", line 228, in configuration
     blas_info = get_info('blas_opt',0)
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
system_info.py", line 256, in get_info
     return cl().get_info(notfound_action)
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
system_info.py", line 399, in get_info
     self.calc_info()
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
system_info.py", line 1287, in calc_info
     blas_mkl_info = get_info('blas_mkl')
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
system_info.py", line 256, in get_info
     return cl().get_info(notfound_action)
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
system_info.py", line 399, in get_info
     self.calc_info()
   File "/snap/c02_raid0/other_src/numpy/numpy/distutils/ 
system_info.py", line 819, in calc_info
     dict_append(libraries = ['pthread'])
TypeError: dict_append() takes exactly 1 non-keyword argument (0 given)


Gen-Nan Chen, PhD
Chief Scientific Officer
Research and Development Group
CorTechs Labs Inc (www.cortechs.net)
1020 Prospect St., #304, La Jolla, CA, 92037
Tel: 1-858-459-9703
Fax: 1-858-459-9705
Email: gnchen at cortechs.net


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070430/edec98d7/attachment.html>

From steven.bethard at gmail.com  Mon Apr 30 13:38:14 2007
From: steven.bethard at gmail.com (Steven Bethard)
Date: Mon, 30 Apr 2007 11:38:14 -0600
Subject: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for
	Numbers (and other algebraic entities)
In-Reply-To: <-5382129169035830252@unknownmsgid>
References: <5d44f72f0704250136l6683b7ffv3585fb00d86861aa@mail.gmail.com>
	<43aa6ff70704250954o6147a56br7f05a6af7ae6056e@mail.gmail.com>
	<ca471dc20704251119vaaccf3ao981fac7660b19cb5@mail.gmail.com>
	<f10g7k$fag$1@sea.gmane.org>
	<5d44f72f0704291025n2543cc47v6bb6d8f1f9e716b6@mail.gmail.com>
	<ca471dc20704291546t7e9db466ha50f713e1004de0e@mail.gmail.com>
	<46353440.5060402@acm.org>
	<ca471dc20704291826p6cb8e001gf3df7a3dc6160667@mail.gmail.com>
	<-5382129169035830252@unknownmsgid>
Message-ID: <d11dcfba0704301038h57a07a20g2a60cf7f336c63d4@mail.gmail.com>

On 4/30/07, Bill Janssen <janssen at parc.com> wrote:
> GvR wrote:
> > So how about we reduce the scope of our (!) PEP (or perhaps of a new
> > one) to two items: (a) add @abstractmethod, and (b) overload
> > isinstance() and issubclass()? Library authors can do everything they
> > want with those, and we can always add a specific set of ABCs for
> > containers and/or numbers later in the 3.0 development cycle.
>
> -1.  Adding mechanism without content seems less than ideal, despite
> Talin's misgivings.  I'd recommend adding the base classes in, and see
> how they work in the earliest 3.0 releases, then modify them as
> necessary in subsequent releases.

+1 for Guido's simplified PEP. There are clearly use cases for
@abstractmethod, __isinstance__() and __issubclass__(), so I see no
reason why the simplified PEP shouldn't be acceptable on its own.
Complaining about "adding mechanism without content" seems like
saying, for example, that we shouldn't have introduced
"functools.partial" without *using* it somewhere in the standard
library. Whether the use cases are in the standard library or
somewhere else should not determine whether a PEP is acceptable or
not.

I personally think the simplified PEP is a great compromise --
proponents of ABCs will now be able to introduce them (even for
existing types like 'int' and 'list') and folks that don't want ABCs
can simply never install the ABC module. Plus, it's much easier to
*add* a module to the standard library than it is to *remove* one
(even given Python 3.0's somewhat laxer first release plans).

STeVe
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
        --- Bucky Katt, Get Fuzzy