From jmiller at stsci.edu  Tue Feb  1 05:09:08 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Tue Feb  1 05:09:08 2005
Subject: [Numpy-discussion] Numeric array's actual data address?
In-Reply-To: <5.2.0.4.2.20050131165659.12d54640@blue-cove.com>
References: <5.2.0.4.2.20050131165659.12d54640@blue-cove.com>
Message-ID: <1107263206.5023.22.camel@jaytmiller.comcast.net>

On Mon, 2005-01-31 at 17:04 -0800, Ray S wrote:
> If I have an array N:
> 
>  >>> N = Numeric.zeros((1000,), Numeric.Float)
>  >>> repr(N.__copy__)
> '<built-in method __copy__ of array object at 0x00809F00>'
> 
> What is the actual address of the first element? 

In C,  look at a->data.

> Or, as an offset from the 
> object?

This doesn't really make sense given the layout of the objects and how
the memory is stored.  You should probably read over the section on the
C-API in the Numeric and numarray manuals.  The Numeric manual,  here,
is probably the best place to start because it is the simplest:

http://www.pfdubois.com/numpy/numpy.pdf

The numarray manual,  here:

http://prdownloads.sourceforge.net/numpy/numarray-1.1.pdf?download

has a section on a Numeric compatibility layer which is very similar.

> numarray gives us that:
>  >>> N = numarray.zeros((1000,), numarray.Float)
>  >>> N.info()
> class: <class 'numarray.numarraycore.NumArray'>
> shape: (1000,)
> strides: (8,)
> byteoffset: 0
> bytestride: 8
> itemsize: 8
> aligned: 1
> contiguous: 1
> data: <memory at 009d67b8 with size:8000 held by object 009d6798 aliasing 
> object  00000000>
> byteorder: little
> byteswap: 0
> type: Float64
> 
> In numarray, the offset appears to be 20.
> If I try to use memmove() to fill a Numeric array it faults when using an 
> offset of 20...

The offset of 0x20 is unique to the numarray.memory memory object.  It
has nothing to do with Numeric arrays so it could only work by accident.

Note that the "data" section of numarray's info() is referring to a
buffer object which is stored in the _data attribute.  I just changed
the info() to avoid this confusion in the future by renaming "data" to
"buffer".  From the overall perspective of the array object,  the
"working data pointer" (a->data in C) is the sum of the byteoffset and
_data base pointer.  

In response to this post,  I modified info() to the following:

>>> import numarray as na
>>> a = na.arange(10)
>>> a.info()
class: <class 'numarray.numarraycore.NumArray'>
shape: (10,)
strides: (4,)
byteoffset: 0
bytestride: 4
itemsize: 4
aligned: 1
contiguous: 1
buffer: <memory at 0xb7c07f68 with size:0x00000028 held by object 0xb7f8d5a0 aliasing object 0x00000000>
fragile data pointer: 0xb7c07f68 (DEBUG ONLY)
byteorder: 'little'
byteswap: 0
type: Int32

I think the "fragile data pointer" is generally useful information,  but
not completely dependable so I gave it the garish name it has.
Comments?

Regards,
Todd




From rays at san.rr.com  Tue Feb  1 08:24:53 2005
From: rays at san.rr.com (RJ)
Date: Tue Feb  1 08:24:53 2005
Subject: [Numpy-discussion] Numeric array's actual data address?
In-Reply-To: <1107263206.5023.22.camel@jaytmiller.comcast.net>
References: <5.2.0.4.2.20050131165659.12d54640@blue-cove.com>
 <5.2.0.4.2.20050131165659.12d54640@blue-cove.com>
Message-ID: <5.2.1.1.2.20050201070034.04bf2520@pop-server.san.rr.com>

Thanks Todd, Travis,

Yes, Travis, I _was_ using ctypes' memmove() with numarray to great benefit. I had also understood that the address of the numarray data was stable.

Originally, I was trying to move data from a hardware A/D driver DLL call to Python as quickly as possible and did some benchmarking, so as per Thomas Heller's suggestion on the ctypes list I used memmove():
http://sourceforge.net/mailarchive/forum.php?thread_id=6166311&forum_id=24606
 from ctypes import *
 import array 
 src = array.array("i", range(32))
 dst = (c_int * 32)()
 memmove(dst, *src.buffer_info())
memmove() is orders of magnitude faster than map() or assignments.
I then tested and found numarray faster for the rest of the tasks (FFT etc.) http://sourceforge.net/mailarchive/forum.php?thread_id=6205658&forum_id=24606
and so parsed the info() output and plugged the data[0] address into the call.

Thomas Heller (in the above) suggested he make a mod to ctypes to accept Python objects which implement the buffer interface as function parameters, which would allow Numeric use once implemented.
I like numarray's breadth of methods so I used it at the time, but in another try at speed-squeezing yesterday afternoon I found Numeric's FFT and subtraction to be >30% faster in this case, so I switched that code over (this increase includes the use of map() with Numeric to read the A/D in another thread). Using memmove() with Numeric would speed up the reader thread once again.

At 08:06 AM 2/1/2005 -0500, Todd Miller wrote:
>> 
>> What is the actual address of the first element? 
>
>In C,  look at a->data.

I had read the Numeric API and looked at the PyObject structure, as Travis then suggested, but my question then is: if the offset from the "array object at 0x..." (object address value) to the array[0] address is not fixed and must be read from the pointer in the PyObject structure, can we get that pointer's value directly from Python or ctypes? 
ctypes pointer() "allows" poking in and directly reading memory: 
"It is also possible to use indexes different from 0, but you must know what you're doing when you use this: You access or change arbitrary memory locations when you do this."
http://starship.python.net/crew/theller/ctypes/tutorial.html
So, could I use ctypes' pointer(offset) to read the structure's pointer-to-data[0]-value and then use it in memmove()? 
I'll cross-post this to ctypes-users...

>I think the "fragile data pointer" is generally useful information,  but
>not completely dependable so I gave it the garish name it has.
>Comments?

It's quite reasonable. 
Is the data pointer value really often changed for contiguous arrays in small stand-alone Python apps? If so, I may stick with map() to avoid having to re-read the pointer before each data buffer transfer.

If anyone is interested I could post some relevant code snips from the office...

Thanks for the help,
Ray
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050201/74d07089/attachment.html>

From oliphant at ee.byu.edu  Thu Feb  3 03:46:23 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb  3 03:46:23 2005
Subject: [Numpy-discussion] Numeric3 status
Message-ID: <42020E7D.3040109@ee.byu.edu>

I've gotten Numeric3 (equivalent to Numeric 30.0) into a state where 
multiarray compiles.

It is far from production ready.  Several sections of the code still 
need to be cleaned up to avoid segfaults.  But, the general outline is 
there.   As this is very alpha code (though based on a very mature 
code-base) it is named Numeric3 so it does not clobber your current 
Numeric installation just in case you accidentally do an install.    You 
will have to make sure the correct multiarray is being loaded through 
your sys.path list (or just point your sys.path to the build location of 
Numeric3) if you want to play with it. 

No array_number methods have been loaded yet (there is an interface to 
load your own if you like).   I expect to have ufuncs working with the 
new arrayobject by the end of the month.  Until then, arrayobjects are 
purely structural beasts.

If anybody would like to comment on the design or direction of Numeric 
30.0, your comments are welcome.  If you would like to checkout the code 
through CVS you may do that using

cvs -d:pserver:anonymous at cvs.sourceforge.net:/cvsroot/numpy login

press <Enter> when prompted for a password.
 
cvs -z3 -d:pserver:anonymous at cvs.sourceforge.net:/cvsroot/numpy co -P 
/Numeric3/

A rough design document is available by following the link at  
http://www.scipy.org/wikis/numdesign/num3news

The major goal is:

Numeric 30.0 will have all the important new features of numarray as 
well as maintain or improve on the speed of Numeric for all array types.

Besides this, Numeric 30.0 will not include any FFT's, LinearAlgebra,  
or Random Numbers (these are all provided much more proficiently by 
SciPy).  Masked Arrays, and Matrices will be provided as standard 
subclasses in Python.   Ideally, RecordArrays will be provided in the 
same way.

I will not re-invent code that I can borrow from past versions of 
Numeric and or numarray.

Best wishes,

-Travis Oliphant








From mdehoon at ims.u-tokyo.ac.jp  Thu Feb  3 04:22:18 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Thu Feb  3 04:22:18 2005
Subject: [Numpy-discussion] Numeric3 status
In-Reply-To: <42020E7D.3040109@ee.byu.edu>
References: <42020E7D.3040109@ee.byu.edu>
Message-ID: <420216CE.9080502@ims.u-tokyo.ac.jp>

Travis Oliphant wrote:
> Besides this, Numeric 30.0 will not include any FFT's, LinearAlgebra,  
> or Random Numbers (these are all provided much more proficiently by 
> SciPy).

I use Numeric's LinearAlgebra and RandomArray quite a lot and I will 
really miss these if they are absent in Numeric 30.0. While it is true 
that these can be provided by SciPy, the drawback is that SciPy is 
harder to install. Since linear algebra, random numbers, and possibly 
FFTs are such basic numerical routines, I think they should be included 
in a Numeric instead of requiring users to download and install both 
Numeric and SciPy. Especially for new users, that may be a bridge too far.

--Michiel.



From konrad.hinsen at laposte.net  Thu Feb  3 05:14:37 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Thu Feb  3 05:14:37 2005
Subject: [Numpy-discussion] Numeric3 status
In-Reply-To: <420216CE.9080502@ims.u-tokyo.ac.jp>
References: <42020E7D.3040109@ee.byu.edu> <420216CE.9080502@ims.u-tokyo.ac.jp>
Message-ID: <450914d39d01fcf682ee304547f081bb@laposte.net>

On 03.02.2005, at 13:19, Michiel Jan Laurens de Hoon wrote:

> I use Numeric's LinearAlgebra and RandomArray quite a lot and I will  
> really miss these if they are absent in Numeric 30.0. While it is true  
> that these can be provided by SciPy, the drawback is that SciPy is  
> harder to install. Since linear algebra, random numbers, and possibly  
> FFTs are such

I agree completely. In addition, there is compatibilty: lots of code  
out there uses the FFT and LinearAlgebra interfaces that have been part  
of Numeric for many years. numarray has them as well (it just requires  
a change in the import statements).

While I like many aspects of your Numeric 3.0 project, I fear that it  
will just create a third nearly-but-not-fully-compatible choice for an  
array module.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From hyclak at math.ohiou.edu  Thu Feb  3 06:47:38 2005
From: hyclak at math.ohiou.edu (Matt Hyclak)
Date: Thu Feb  3 06:47:38 2005
Subject: [Numpy-discussion] Problems compiling against ATLAS on x86_64
Message-ID: <20050203144231.GB24701@math.ohiou.edu>

I'm about ready to pull out my hair trying to get this stuff to compile for
a faculty member. I'm running RHEL 3 AS on a dual opteron machine. ATLAS
compiles fine as I go through xconfig, but I can't for the life of me get
Numeric to compile against it. I've tried ATLAS 3.6.0 and 3.7.8. I'm
trying to build Numeric 23.7. I keep getting the following error:

gcc -shared build/temp.linux-x86_64-2.2/lapack_litemodule.o
-L/usr/lib64/atlas -llapack -lcblas -lf77blas -latlas -lg2c -o
build/lib.linux-x86_64-2.2/lapack_lite.so
/usr/bin/ld: /usr/lib64/atlas/liblapack.a(dgesv.o): relocation R_X86_64_32
can not be used when making a shared object; recompile with -fPIC
/usr/lib64/atlas/liblapack.a: could not read symbols: Bad value
collect2: ld returned 1 exit status

I built ATLAS with -fPIC, but that didn't seem to help. Could someone
*please* point me in the right direction before I start hitting things?

Thanks!

Matt

-- 
Matt Hyclak
Department of Mathematics 
Department of Social Work
Ohio University
(740) 593-1263





From jdhunter at ace.bsd.uchicago.edu  Thu Feb  3 07:22:48 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Thu Feb  3 07:22:48 2005
Subject: [Numpy-discussion] Numeric3 status
In-Reply-To: <450914d39d01fcf682ee304547f081bb@laposte.net> (konrad.hinsen@laposte.net's
 message of "Thu, 3 Feb 2005 14:12:01 +0100")
References: <42020E7D.3040109@ee.byu.edu> <420216CE.9080502@ims.u-tokyo.ac.jp>
	<450914d39d01fcf682ee304547f081bb@laposte.net>
Message-ID: <m3vf99of39.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "konrad" == konrad hinsen <konrad.hinsen at laposte.net> writes:

    konrad> I agree completely. In addition, there is compatibilty:
    konrad> lots of code out there uses the FFT and LinearAlgebra
    konrad> interfaces that have been part of Numeric for many
    konrad> years. numarray has them as well (it just requires a
    konrad> change in the import statements).

I too would be *very disappointed* to see LinearAlgebra, RandomArray
and FFT leave Numeric.  matplotlib requires all three.  Numeric is
lightweight and easy to install, and although the scipy install is
significantly easier than it was in yesteryear, it still has higher
barrier to entry.  I am much happier requiring only Numeric than
Numeric and scipy.

As for the change in import statements, one thing that would be nice
to see in Numeric3 is a package structure for these extra modules that
mirrors numarray's.  This is what the matplotlib numerix module
currently does, and the in the various __init__ files imports either
the Numeric or numarray symbols depending on the configuration
settings.  This seems most sensible since the longer term goal is
integration.

    peds-pc311:~/python/projects/matplotlib/lib/matplotlib> find numerix/ | grep -v CVS
    numerix/
    numerix/mlab
    numerix/mlab/__init__.py
    numerix/linear_algebra
    numerix/linear_algebra/__init__.py
    numerix/__init__.py
    numerix/fft
    numerix/fft/__init__.py
    numerix/_na_imports.py
    numerix/random_array
    numerix/random_array/__init__.py
    numerix/_nc_imports.py


Finally, breaking both backward compatibility with Numeric, and
forward compatibility with numarray, seems ill-advised.

JDH



From faltet at carabos.com  Thu Feb  3 09:10:15 2005
From: faltet at carabos.com (Francesc Altet)
Date: Thu Feb  3 09:10:15 2005
Subject: [Numpy-discussion] pyrex numarray
In-Reply-To: <20050201121626.6150f9ff.simon@arrowtheory.com>
References: <20050201121626.6150f9ff.simon@arrowtheory.com>
Message-ID: <200502031805.23278.faltet@carabos.com>

A Dimarts 01 Febrer 2005 02:16, Simon Burton va escriure:
> Has anyone considered using pyrex to implement numarray ?

None that I'm aware of. But this issue has been already discussed in
this list at least a couple of times before. See, for example:

http://sourceforge.net/mailarchive/message.php?msg_id=8080203

> I don't know a lot of the details but it seems to me that pyrex could
> unify numarray's python/c source code mixture and smooth the transition
> from python (==untyped pyrex) code to c (==typed pyrex) code. It would also
> help clueless users like me understand, and perhaps contribute to, the
> codebase.

Well, after the past discussions, I've got the feeling that the main
problems to do that are:

1.- Construct general Ufuncs that can handle many different
  combinations of arrays and types. I think that this is currently
  done through a use of some C preprocesor than creates specific C
  files from kind of template (correct me if I'm wrong there). Do the
  same thing with Pyrex may potentially need such a preprocessor as
  well.
  
2.- Recoding the numarray beast in Pyrex could be *major* task

3.- The lack of experience with Pyrex (at least some time ago)

Perhaps I miss something still more important, but my guess is that
these reasons are the real stoppers. While problem 3 can be quite
easily surmounted, problem 1 probably is the big one (except if one
finds some easy way to deal with it). Although now that I think, one
can always take advantage of the files coming from the current
numarray C preprocessor and use Pyrex as the glue with Python. Problem
2 is also worrying but a combined effort could contribute to alleviate
it.

However, I do think that is probably worth the effort to concentrate
the efforts resolving first the remaining problems with numarray
performance (mainly array creation time), than go and switch to yet
another implementation that would take far more time to implement.

My 2 cents,

-- 
>qo<   Francesc Altet ? ? http://www.carabos.com/
V ?V   C?rabos Coop. V. ??Enjoy Data
 ""




From focke at slac.stanford.edu  Thu Feb  3 09:34:34 2005
From: focke at slac.stanford.edu (Warren Focke)
Date: Thu Feb  3 09:34:34 2005
Subject: [Numpy-discussion] pyrex numarray
In-Reply-To: <200502031805.23278.faltet@carabos.com>
References: <20050201121626.6150f9ff.simon@arrowtheory.com>
 <200502031805.23278.faltet@carabos.com>
Message-ID: <Pine.LNX.4.58.0502030924340.20901@auriga.slac.stanford.edu>


On Thu, 3 Feb 2005, Francesc Altet wrote:

> A Dimarts 01 Febrer 2005 02:16, Simon Burton va escriure:
> > Has anyone considered using pyrex to implement numarray ?
>
> None that I'm aware of. But this issue has been already discussed in
> this list at least a couple of times before. See, for example:
>
> http://sourceforge.net/mailarchive/message.php?msg_id=8080203

Tim Hochberg worked on a psyco version, look at:

http://sourceforge.net/mailarchive/message.php?msg_id=3845401

maybe that could be used as a start.

Warren Focke





From perry at stsci.edu  Thu Feb  3 09:55:47 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb  3 09:55:47 2005
Subject: [Numpy-discussion] pyrex numarray
In-Reply-To: <200502031805.23278.faltet@carabos.com>
References: <20050201121626.6150f9ff.simon@arrowtheory.com> <200502031805.23278.faltet@carabos.com>
Message-ID: <285db149e78a418f0bad7d50e3515ae9@stsci.edu>

On Feb 3, 2005, at 12:05 PM, Francesc Altet wrote:

> A Dimarts 01 Febrer 2005 02:16, Simon Burton va escriure:
>> Has anyone considered using pyrex to implement numarray ?
>
> None that I'm aware of. But this issue has been already discussed in
> this list at least a couple of times before. See, for example:
>
> http://sourceforge.net/mailarchive/message.php?msg_id=8080203
>
>> I don't know a lot of the details but it seems to me that pyrex could
>> unify numarray's python/c source code mixture and smooth the 
>> transition
>> from python (==untyped pyrex) code to c (==typed pyrex) code. It 
>> would also
>> help clueless users like me understand, and perhaps contribute to, the
>> codebase.
>
> Well, after the past discussions, I've got the feeling that the main
> problems to do that are:
>
> 1.- Construct general Ufuncs that can handle many different
>   combinations of arrays and types. I think that this is currently
>   done through a use of some C preprocesor than creates specific C
>   files from kind of template (correct me if I'm wrong there). Do the
>   same thing with Pyrex may potentially need such a preprocessor as
>   well.

I meant to respond this earlier but left it hanging due to other 
distractions. Francesc points out that this has been raised in the 
past, and he also correctly mentions the biggest issue. How do you 
handle all the possible numeric types? If handling all that results in 
large amounts of C or psuedo C code, what is pyrex buying you, really? 
I think there are various aspects of arrays that get confused sometime. 
One is thinking that pyrex would make it easy to code a C loop over an 
array in a fairly simple way. And it would--for a case where you have 
fixed the types of the arrays being operated on. But what about all 
possible types for arguments? That's a different story (at least I 
think so; if someone has a clever idea that solves these problems let's 
hear it).

Perry




From oliphant at ee.byu.edu  Thu Feb  3 11:53:07 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb  3 11:53:07 2005
Subject: [Numpy-discussion] Numeric3
Message-ID: <420280AD.8080309@ee.byu.edu>

John Hunter wrote:

>>>>>> "konrad" == konrad hinsen <konrad.hinsen at laposte.net> writes:
>>>>>>           
>>>>>
>
>    konrad> I agree completely. In addition, there is compatibilty:
>    konrad> lots of code out there uses the FFT and LinearAlgebra
>    konrad> interfaces that have been part of Numeric for many
>    konrad> years. numarray has them as well (it just requires a
>    konrad> change in the import statements).
>
> I too would be *very disappointed* to see LinearAlgebra, RandomArray
> and FFT leave Numeric.  matplotlib requires all three.  Numeric is
> lightweight and easy to install, and although the scipy install is
> significantly easier than it was in yesteryear, it still has higher
> barrier to entry.  I am much happier requiring only Numeric than
> Numeric and scipy.
>  
>

This is all a matter of semantics and packaging.

My point is that it is silly to have a really nice wrapping of fft, 
linear algebra, and random number routines in SciPy and then repeat them 
somewhat for Numeric (or vice-versa).

So to satisfy this complaint, the standard Numeric3 install is actually 
a "slimmed-down" scipy.

> As for the change in import statements, one thing that would be nice
> to see in Numeric3 is a package structure for these extra modules that
> mirrors numarray's. 
> This is what the matplotlib numerix module
> currently does, and the in the various __init__ files imports either
> the Numeric or numarray symbols depending on the configuration
> settings.  This seems most sensible since the longer term goal is
> integration.
>  
>
I'm not that concerned about what names we give things,  but why not the 
names that scipy has.

Or scipy should change it's names.

There was some discussion about this on the scipy list.  Nobody had a 
clear winner in terms of "naming" conventions.
SciPy's stated goals for some time have been to provide numerical 
operations for arrays.  If you want to call this super-structure 
numerix, then fine.

SciPy adjusted for those that don't want to install everything.  It is 
very easy to pick and choose the packages you want to install.  There 
appears to be a hesitancy on the part of some to adopt something like 
scipy.  I'm not sure whether it's scipy itself, or just the idea of 
scipy, that some people find difficult to adjust to.   

We talk about some version of the arrayobject getting into the core.  
Guido is not going to accept FFT's and LinearAlgebra (or maybe not even 
ufuncs).  Towards this end, I prefer  to separate the ideas of the 
arrayobject from the things you can do with it, which is why I believe 
scipy (with appropriate slimmed down versions) is a better place to 
discuss fft's, linear algebra, and the like.  

Old code can be handled with compatibility layers.

-Travis










From rays at blue-cove.com  Thu Feb  3 12:25:47 2005
From: rays at blue-cove.com (Ray S)
Date: Thu Feb  3 12:25:47 2005
Subject: [Numpy-discussion] Re: [ctypes-users] copying/slicing ctypes arrays, (c_ulong
 *n)() - to Numeric
In-Reply-To: <d5vhbijx.fsf@python.net>
References: <5.2.0.4.2.20050202134734.137b2930@blue-cove.com>
 <5.2.1.1.2.20050201081943.05264720@blue-cove.com>
 <5.2.1.1.2.20041222161345.09a30140@blue-cove.com>
 <5.2.1.1.2.20041216074849.090f3010@blue-cove.com>
 <5.2.0.4.2.20041215100117.0dff97b0@blue-cove.com>
 <5.2.1.1.2.20041214202542.05f02920@pop-server.san.rr.com>
 <5.2.1.1.2.20041214202542.05f02920@pop-server.san.rr.com>
 <5.2.0.4.2.20041215100117.0dff97b0@blue-cove.com>
 <5.2.1.1.2.20041216074849.090f3010@blue-cove.com>
 <5.2.1.1.2.20041222161345.09a30140@blue-cove.com>
 <5.2.1.1.2.20050201081943.05264720@blue-cove.com>
 <5.2.0.4.2.20050202134734.137b2930@blue-cove.com>
Message-ID: <5.2.0.4.2.20050203104517.1448f1a8@blue-cove.com>

At 07:32 PM 2/3/2005 +0100, you wrote:

>Don't be confused that the buffer() object says <read-only buffer ...>!
>The buffer call only asks for readable memory..., but ctypes doesn't
>care about the readonly attribute - it will happily write into this
>memory.

Hi Thomas,

Yes, I was thinking of what the shell error said upon assignment...

Upon adding to some working code all is well:

 >>> import Numeric, ctypes, string
 >>> N = Numeric.zeros((10,), Numeric.Float)
 >>> buf = buffer(N)
 >>> buf
<read-only buffer for 0x008F9C28, ptr 0x008D7780, size 80 at 0x008FE220>
 >>> int(string.split(repr(buf))[5][:-1], 16)
9271168

## numarray version
# nAddress = int(string.split(repr(N._data))[2], 16)

## Numeric version
NAddress = int(string.split(repr(buffer(N)))[5][:-1], 16)

## Load DLL here...
## do this to get data from the USB A/D's DLL
usb.GetData(usb.Sn, (bufferInsertPos * N.itemsize()) + NAddress,
                                     ctypes.byref( (types.c_long * 
buffersize)() ) )

Which is faster than getting data into a ctypes array (c_ulong *n)() and 
then doing memmove() to Numeric - one less step.

Maybe this snip would be of help to some others, although more so to numpy 
people.

Of course, the Python array works the same:
 >>> a = array.array('l',[1,2,3])
 >>> int(string.split(repr(buffer(a)))[5][:-1], 16)
8380408

>If this is too confusing, and this may well be, ctypes could expose a
>memory() function which would insist on read-write memory, but apart
>from that do the same that buffer does:

No, not confusing, just not clear to a non-expert C person that ctypes 
ignores where Python is read-only. A simple note in the tutorial would be 
fine.
Some over at numpy were also unaware of memmove()s' existence in the new 
releases, and seemed interested.

Thanks again,
Ray




From oliphant at ee.byu.edu  Thu Feb  3 13:31:26 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb  3 13:31:26 2005
Subject: [Numpy-discussion] Numeric3 and speed of new-style classes
Message-ID: <42029763.9090700@ee.byu.edu>


I fixed a major oversight in Numeric3 and now it works pretty well (it 
is basically old Numeric with a bunch of new types, no math (yet)), 
using new-style classes, and some checks for alignment when 
dereferencing data.

I did some simple array creation timing tests.  An example of the 
results is given below.  The bottom line is that new-style classes do 
nothing but speed up (a little bit) basic object creation. 


FYI:


First test (Numeric3):    .196081876755

import sys
sys.path.insert(0,"../build/lib.linux-i686-2.3/")

import time

import multiarray

start = time.time()
for k in range(100000):
    g = multiarray.arraytype((4,3),'d')
print time.time() - start


Second test (Numeric):  .203589916229

import Numeric
import time


start = time.time()
for k in range(100000):
    g = Numeric.empty((4,3),'d')
print time.time() - start













From Chris.Barker at noaa.gov  Thu Feb  3 15:13:26 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Thu Feb  3 15:13:26 2005
Subject: [Numpy-discussion] pyrex numarray
In-Reply-To: <285db149e78a418f0bad7d50e3515ae9@stsci.edu>
References: <20050201121626.6150f9ff.simon@arrowtheory.com> <200502031805.23278.faltet@carabos.com> <285db149e78a418f0bad7d50e3515ae9@stsci.edu>
Message-ID: <4202AF4E.8060207@noaa.gov>

Perry Greenfield wrote:
> How do you handle all 
> the possible numeric types? If handling all that results in large 
> amounts of C or psuedo C code, what is pyrex buying you, really?

There was a quote from Greg Ewing that went something along the lines of:

"I am not re-implementing C++ templates for Pyrex!"

Which I think summarized his take on this.

This begs the question for me:

Why not use C++ templates?

I know this was brought up very early in numarray development, 
specifically the idea of using Blitz++ (or something similar) as the 
basis for the new NumPy. I'm pretty sure it was rejected primarily on 
the grounds of limited good support for C++ templates on a wide variety 
of compilers. I think that's sad, as by now template support is much better.

The other idea bandied about is to make Psyco multi-array aware. That 
would be cool!

-Chris

---
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From mdehoon at ims.u-tokyo.ac.jp  Thu Feb  3 20:16:07 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Thu Feb  3 20:16:07 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420280AD.8080309@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu>
Message-ID: <4202F6A0.3@ims.u-tokyo.ac.jp>

Travis Oliphant wrote:
> John Hunter wrote:
>> I too would be *very disappointed* to see LinearAlgebra, RandomArray
>> and FFT leave Numeric.  matplotlib requires all three.  Numeric is
>> lightweight and easy to install, and although the scipy install is
>> significantly easier than it was in yesteryear, it still has higher
>> barrier to entry.  I am much happier requiring only Numeric than
>> Numeric and scipy.
>>  
>>
> 
> This is all a matter of semantics and packaging.
> 
Well it is mainly a matter of being able to install what you need, and 
not having to spend time figuring out how to install what you don't 
need. Let me give you an example from my own field (computational 
biology). I am one of the maintainers of Biopython, which uses 
LinearAlgebra and RandomArray. Many of our users are not very familiar 
with Python. Even installing Numerical Python sometimes causes problems, 
and I'm sure we have lost users in the past because of that. SciPy, in 
my experience, doesn't build out of the box. If Biopython required 
SciPy, many users will run into installation problems, most of them will 
give up and return to using Microsoft Excel for their numerical 
analysis, medical research will come to a standstill, and we'll all die.

But seriously, I think that there are advantages to Numerical Python and 
to SciPy, and they both have a reason to exist. Numerical Python has the 
advantage that it is easy to install and contains most (if not all) of 
the numerical routines users will need. SciPy, on the other hand, is 
harder to install (e.g. because it doesn't use ANSI-C only), but 
therefore can also offer pieces of software that are not available in 
Numerical Python.

Many users, however, will never need the additional capabilities that 
are in SciPy. Moving LinearAlgebra, RandomArray, and FFT to SciPy will 
make their life more difficult. I don't see an advantage for SciPy users 
either, because they need to install Numerical Python anyway.

So I would suggest the following demarcation line between Numerical 
Python and SciPy:

Stuff that is needed by lots of users (linear algebra, FFT, random 
numbers, special functions) and is available in ANSI-C (so its 
installation is straightforward and won't cause problems to users who 
don't need it) should go into Numerical Python.

Stuff that is needed by fewer users or is not available in ANSI-C should 
go into SciPy. SciPy, with its binary installers, can provide a real 
benefit to users who need extension modules written in Fortran/C++/whatnot.

To address the issue you raised about duplicated efforts in numpy and 
SciPy: What does SciPy do better in terms of linear algebra, FFT, or 
random numbers, than numpy? If so, is it possible to move the relevant 
SciPy code pieces to numpy (without affecting numpy's ease of installation)?

--Michiel.



From rkern at ucsd.edu  Thu Feb  3 20:46:48 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Thu Feb  3 20:46:48 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420280AD.8080309@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu>
Message-ID: <4202FE20.8040302@ucsd.edu>

Travis Oliphant wrote:

> We talk about some version of the arrayobject getting into the core.  
> Guido is not going to accept FFT's and LinearAlgebra (or maybe not even 
> ufuncs).  Towards this end, I prefer  to separate the ideas of the 
> arrayobject from the things you can do with it, which is why I believe 
> scipy (with appropriate slimmed down versions) is a better place to 
> discuss fft's, linear algebra, and the like. 

I think this is the point that some people are missing. Numeric3 is 
intended to be the multiarray object proposed to go into the standard 
library. FFTs will not go into the standard library. QR decompositions 
will not go into the standard library.

One can always support the extra packages externally.

So for people who need LinearAlgebra, et al., their current situation is 
this:

   ThirdPartyPackage depends on Numeric and a standard Python
   distribution.

Their new situation would be this:

   ThirdPartyPackage depends on NumericExtras and a standard Python
   distribution.

If you need to depend on these modules with their current interfaces, 
then someone needs to step up to support it. This support probably must 
come from the people who need the current interfaces. For far too long, 
the responsibility of maintaining Numeric and its extras has been 
saddled on people who don't have (current) interest in them.

That said, it might be a better use of energy to make the build process 
of a lite scipy easier and standardized.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From konrad.hinsen at laposte.net  Fri Feb  4 01:13:14 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb  4 01:13:14 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4202FE20.8040302@ucsd.edu>
References: <420280AD.8080309@ee.byu.edu> <4202FE20.8040302@ucsd.edu>
Message-ID: <35b6b589e88d3db534f8bf07f8c75dae@laposte.net>

On 04.02.2005, at 05:46, Robert Kern wrote:

> I think this is the point that some people are missing. Numeric3 is  
> intended to be the multiarray object proposed to go into the standard  
> library. FFTs

Wasn't numarray as well?


> So for people who need LinearAlgebra, et al., their current situation  
> is this:
>
>   ThirdPartyPackage depends on Numeric and a standard Python
>   distribution.
>
> Their new situation would be this:
>
>   ThirdPartyPackage depends on NumericExtras and a standard Python
>   distribution.

Fine. But before then, Numeric3 will get a lot less testing than it  
should if it can be used only by power-users who write all their  
scientific code themselves.

> If you need to depend on these modules with their current interfaces,  
> then someone needs to step up to support it. This support probably  
> must come from the people who need the current interfaces. For far too  
> long, the responsibility of maintaining Numeric and its extras has  
> been saddled on people who don't have (current) interest in them.

There is no responsibility at all in the OpenSource world. I am not  
criticizing the idea of making Numeric3 a subset of current Numeric, I  
am just pointing out that it will be much less used (and tested) as a  
result.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Fri Feb  4 01:19:12 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb  4 01:19:12 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4202F6A0.3@ims.u-tokyo.ac.jp>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
Message-ID: <1f7baacd417e55debdde0351e463cf10@laposte.net>

On 04.02.2005, at 05:14, Michiel Jan Laurens de Hoon wrote:

> give you an example from my own field (computational biology). I am  
> one of the maintainers of Biopython, which uses LinearAlgebra and  
> RandomArray. Many of our users are not very familiar with Python. Even  
> installing Numerical Python sometimes causes problems, and I'm sure we  
> have lost users in the past because of that. SciPy, in my experience,

My experience with users of MMTK, nMOLDYN, and DomainFinder is exactly  
the same. Installation problems are the #1 source for support  
questions, in particular under Windows and Irix.

There are a couple of nice things I could use and recommend to my  
users, e.g. SciPy or Atlas-accelerated LinearAlgebra. I don't because I  
know it will generate more support questions, which are already eating  
up too much of my time. I don't use features of very recent Python  
versions for the same reason.

> But seriously, I think that there are advantages to Numerical Python  
> and to SciPy, and they both have a reason to exist. Numerical Python  
> has the

Exactly.

> So I would suggest the following demarcation line between Numerical  
> Python and SciPy:
>
> Stuff that is needed by lots of users (linear algebra, FFT, random  
> numbers, special functions) and is available in ANSI-C (so its  
> installation is straightforward and won't cause problems to users who  
> don't need it) should go into Numerical Python.
>
> Stuff that is needed by fewer users or is not available in ANSI-C  
> should go into SciPy. SciPy, with its binary installers, can provide a  
> real benefit to users who need extension modules written in  
> Fortran/C++/whatnot.

That sounds like a good division line.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From Vincent.Picavet at inria.fr  Fri Feb  4 01:38:59 2005
From: Vincent.Picavet at inria.fr (Vincent Picavet)
Date: Fri Feb  4 01:38:59 2005
Subject: [Numpy-discussion] Load numarray with specific shape cause error
Message-ID: <20050204103751.20ee5979@hazard.inria.fr>

Hi,
Here is what I launch in the Python interpreter :
----
import numarray
simulation_file = "/path/to/my/file.bin"
a = numarray.fromfile(simulation_file, 'Float32',(3144, 33, 65))
a.sum() 
----

And here I get the following error :

----
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File
"/nfs/data2/picavet/workspace/softs/python/lib/python2.4/site-packages/
numarray/numarraycore.py", line 1133, in sum    return
ufunc.add.reduce(ufunc.add.areduce(self, type=type).flat, type=type)
libnumarray.error: add_Float64_reduce: access beyond buffer.
offset=25151 buffersize=17160
----

Just after that, I execute :
----
a.sum()
588108255.50844395
a.mean()
87.206215933326803
----

... and as you can see I got no error the second time.
The binary file I try to load is a raw file of 3144*33*64
4-Bytes float elements.

In fact I just saw that if I use the shape (65,33,3144) when calling
fromfile(), everything works just fine. And after having called a.sum()
the first time, the shape of the array has been changed.

----
>>> a.getshape()
(3144, 65, 33)
>>> a.sum()
Traceback (most recent call last):
[snip, same as above]
>>> a.getshape()
(33, 65, 3144)
----

Any idea why I can't load an array with such a shape ?

I use Python 2.4 and the latest version of numarray.
Python 2.4 (#1, Jan 27 2005, 15:21:15)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2

Thanks,
Vincent



-- 
Vincent Picavet,		Tel : +33 (0) 1 39 63 52 14	
INRIA Rocquencourt,		E-mail : vincent.picavet at inria.fr
Projet CLIME			http://www-rocq.inria.fr/clime	



From jmiller at stsci.edu  Fri Feb  4 03:01:50 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Fri Feb  4 03:01:50 2005
Subject: [Numpy-discussion] Load numarray with specific shape cause
	error
In-Reply-To: <20050204103751.20ee5979@hazard.inria.fr>
References: <20050204103751.20ee5979@hazard.inria.fr>
Message-ID: <1107514898.4962.18.camel@jaytmiller.comcast.net>

On Fri, 2005-02-04 at 10:37 +0100, Vincent Picavet wrote:
> Hi,
> Here is what I launch in the Python interpreter :
> ----
> import numarray
> simulation_file = "/path/to/my/file.bin"
> a = numarray.fromfile(simulation_file, 'Float32',(3144, 33, 65))
> a.sum() 
> ----
> 
> And here I get the following error :
> 
> ----
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File
> "/nfs/data2/picavet/workspace/softs/python/lib/python2.4/site-packages/
> numarray/numarraycore.py", line 1133, in sum    return
> ufunc.add.reduce(ufunc.add.areduce(self, type=type).flat, type=type)
> libnumarray.error: add_Float64_reduce: access beyond buffer.
> offset=25151 buffersize=17160
> ----
> 
> Just after that, I execute :
> ----
> a.sum()
> 588108255.50844395
> a.mean()
> 87.206215933326803
> ----
> 
> ... and as you can see I got no error the second time.
> The binary file I try to load is a raw file of 3144*33*64
> 4-Bytes float elements.
> 
> In fact I just saw that if I use the shape (65,33,3144) when calling
> fromfile(), everything works just fine. And after having called a.sum()
> the first time, the shape of the array has been changed.
> 
> ----
> >>> a.getshape()
> (3144, 65, 33)
> >>> a.sum()
> Traceback (most recent call last):
> [snip, same as above]
> >>> a.getshape()
> (33, 65, 3144)
> ----
> 
> Any idea why I can't load an array with such a shape ?

Yes.  You're hitting a known bug which I'm working on now in preparation
for numarray-1.2.  

In a nutshell,  sum() exposes surprisingly complex code in numarray in
order to use higher precision without making a temporary copy of the
array.  This should be fixed when numarray-1.2 is released,  hopefully
very soon.

For now,  there are two work arounds:

# use simpler "same type" summing code at lower precision
a.sum('Float32')  

# create a higher precision temporary and sum that
a.astype('Float64').sum()  

Regards,
Todd




From jmiller at stsci.edu  Fri Feb  4 04:24:59 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Fri Feb  4 04:24:59 2005
Subject: [Numpy-discussion] Numeric3 and speed of new-style classes
In-Reply-To: <42029763.9090700@ee.byu.edu>
References: <42029763.9090700@ee.byu.edu>
Message-ID: <1107519864.4962.74.camel@jaytmiller.comcast.net>

On Thu, 2005-02-03 at 14:28 -0700, Travis Oliphant wrote:
> 
> I fixed a major oversight in Numeric3 and now it works pretty well (it 
> is basically old Numeric with a bunch of new types, no math (yet)), 
> using new-style classes, and some checks for alignment when 
> dereferencing data.
> 
> I did some simple array creation timing tests.  An example of the 
> results is given below.  The bottom line is that new-style classes do 
> nothing but speed up (a little bit) basic object creation. 

I think you've done a real service here,  bounding the cost of new-style
classes,  but I'm not convinced we have a bottom line,  at least not in
the sense of my own FUD about new style classes.  I'm not contesting the
technical direction you're taking with Numeric3 (I hope it works,  but
the details are up to you),  but I am contesting the idea that new style
classes do nothing but speed up object creation.

I think among the promises of new style classes are the ability to
subclass in both C and Python,  and my thinking is that if you actually
do *that*,  you pay for it through the nose.  So,  I'm trying to dispell
the notion that new style classes are pure bliss;  I think they need to
be used either with insight (say a lot more than C++ OOP) or sidestepped
altogether (as, IMHO, you have done).  Using them,  you're up against
more than just faking OOP in C.

I think one of the flaws currently in numarray is that I tried to push a
Python class hierarchy down (or half-way down) into C.   IMO,  that's
where some of the cost of new style classes would be incurred.  The cost
of the object __dict__ I mostly expected to be optimized out,  perhaps
by freelist magic or some lazy construction technique.  If it's the
latter.

As far as state in the arrayobject goes, I *did* flatten the numarray
class heirarchy.  But I didn't flatten the numarray class hierarchy in
terms of constructors and methods,  and my sense is that there is a
significant cost there.   Calling up the class hierarchy uses Python
method calls (maybe an error) which themselves require the construction,
parsing, and destruction of tuple objects and at least some of the
objects the tuples contain.  

So,  I'd assert that unless you solve *that* problem,  using new style
classes and some form of inheritance,  saying that new style classes do
nothing but speed things up is stretching the truth.  To really disprove
the hidden costs in my FUD,  I'd want to see a C basetype and
inheritance by a Python subclass.

If you actually use them for anything,  I think you wind up paying
Python function call overheads.  That, anyway,  is a problem to solve
(or sidestep as you have done) for numarray.

AFICT, among the things you're doing with Numeric3 (which I applaud) is
basically throwing out inheritance from the fundamental array type;
squashing all the constructor functionality for NDArray and NumArray
into one type was more than I had the nerve to do.  So, I know you're
adding the ability to subclass,  but you're not actually using it,  so I
think it's stretching the truth to say it speeds things up.

In numarray,  we use it,  in both C and Python,  and that's maybe not a
good thing.  

If you can pull it off and still cleanly support the features of
numarray,  it will be a triumph of KISS.  But that's not a given.






From jmiller at stsci.edu  Fri Feb  4 06:07:05 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Fri Feb  4 06:07:05 2005
Subject: [Numpy-discussion] Just ignore me.  That was an accidental send.
In-Reply-To: <1107519864.4962.74.camel@jaytmiller.comcast.net>
References: <42029763.9090700@ee.byu.edu>
	 <1107519864.4962.74.camel@jaytmiller.comcast.net>
Message-ID: <1107525958.2505.23.camel@halloween.stsci.edu>

Please ignore that last post.   It consists of unfinished thoughts which
were sent accidentally.  Doing numarray isn't easy.

Regards,
Todd

On Fri, 2005-02-04 at 07:24, Todd Miller wrote:
> On Thu, 2005-02-03 at 14:28 -0700, Travis Oliphant wrote:
> > 
> > I fixed a major oversight in Numeric3 and now it works pretty well (it 
> > is basically old Numeric with a bunch of new types, no math (yet)), 
> > using new-style classes, and some checks for alignment when 
> > dereferencing data.
> > 
> > I did some simple array creation timing tests.  An example of the 
> > results is given below.  The bottom line is that new-style classes do 
> > nothing but speed up (a little bit) basic object creation. 
> 
> I think you've done a real service here,  bounding the cost of new-style
> classes,  but I'm not convinced we have a bottom line,  at least not in
> the sense of my own FUD about new style classes.  I'm not contesting the
> technical direction you're taking with Numeric3 (I hope it works,  but
> the details are up to you),  but I am contesting the idea that new style
> classes do nothing but speed up object creation.
> 
> I think among the promises of new style classes are the ability to
> subclass in both C and Python,  and my thinking is that if you actually
> do *that*,  you pay for it through the nose.  So,  I'm trying to dispell
> the notion that new style classes are pure bliss;  I think they need to
> be used either with insight (say a lot more than C++ OOP) or sidestepped
> altogether (as, IMHO, you have done).  Using them,  you're up against
> more than just faking OOP in C.
> 
> I think one of the flaws currently in numarray is that I tried to push a
> Python class hierarchy down (or half-way down) into C.   IMO,  that's
> where some of the cost of new style classes would be incurred.  The cost
> of the object __dict__ I mostly expected to be optimized out,  perhaps
> by freelist magic or some lazy construction technique.  If it's the
> latter.
> 
> As far as state in the arrayobject goes, I *did* flatten the numarray
> class heirarchy.  But I didn't flatten the numarray class hierarchy in
> terms of constructors and methods,  and my sense is that there is a
> significant cost there.   Calling up the class hierarchy uses Python
> method calls (maybe an error) which themselves require the construction,
> parsing, and destruction of tuple objects and at least some of the
> objects the tuples contain.  
> 
> So,  I'd assert that unless you solve *that* problem,  using new style
> classes and some form of inheritance,  saying that new style classes do
> nothing but speed things up is stretching the truth.  To really disprove
> the hidden costs in my FUD,  I'd want to see a C basetype and
> inheritance by a Python subclass.
> 
> If you actually use them for anything,  I think you wind up paying
> Python function call overheads.  That, anyway,  is a problem to solve
> (or sidestep as you have done) for numarray.
> 
> AFICT, among the things you're doing with Numeric3 (which I applaud) is
> basically throwing out inheritance from the fundamental array type;
> squashing all the constructor functionality for NDArray and NumArray
> into one type was more than I had the nerve to do.  So, I know you're
> adding the ability to subclass,  but you're not actually using it,  so I
> think it's stretching the truth to say it speeds things up.
> 
> In numarray,  we use it,  in both C and Python,  and that's maybe not a
> good thing.  
> 
> If you can pull it off and still cleanly support the features of
> numarray,  it will be a triumph of KISS.  But that's not a given.
> 
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 




From hyclak at math.ohiou.edu  Fri Feb  4 06:40:44 2005
From: hyclak at math.ohiou.edu (Matt Hyclak)
Date: Fri Feb  4 06:40:44 2005
Subject: [Numpy-discussion] Re: Problems compiling against ATLAS on x86_64
In-Reply-To: <20050203144231.GB24701@math.ohiou.edu>
References: <20050203144231.GB24701@math.ohiou.edu>
Message-ID: <20050204143935.GE24701@math.ohiou.edu>

On Thu, Feb 03, 2005 at 09:42:31AM -0500, Matt Hyclak enlightened us:
> I'm about ready to pull out my hair trying to get this stuff to compile for
> a faculty member. I'm running RHEL 3 AS on a dual opteron machine. ATLAS
> compiles fine as I go through xconfig, but I can't for the life of me get
> Numeric to compile against it. I've tried ATLAS 3.6.0 and 3.7.8. I'm
> trying to build Numeric 23.7. I keep getting the following error:
> 
> gcc -shared build/temp.linux-x86_64-2.2/lapack_litemodule.o
> -L/usr/lib64/atlas -llapack -lcblas -lf77blas -latlas -lg2c -o
> build/lib.linux-x86_64-2.2/lapack_lite.so
> /usr/bin/ld: /usr/lib64/atlas/liblapack.a(dgesv.o): relocation R_X86_64_32
> can not be used when making a shared object; recompile with -fPIC
> /usr/lib64/atlas/liblapack.a: could not read symbols: Bad value
> collect2: ld returned 1 exit status
> 
> I built ATLAS with -fPIC, but that didn't seem to help. Could someone
> *please* point me in the right direction before I start hitting things?
> 

Sorry for replying to myself. Just for the archives, the problem seems to be
the static lapack/blas libraries provided with RHEL3 are not compiled with
-fPIC. I ripped open the rpm and rebuild it with an -fPIC added in and all
at least compiles now. I'll leave it up to my faculty to tell me whether or
not it works :-)

Thanks,
Matt

-- 
Matt Hyclak
Department of Mathematics 
Department of Social Work
Ohio University
(740) 593-1263



From stephen.walton at csun.edu  Fri Feb  4 10:43:20 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Fri Feb  4 10:43:20 2005
Subject: [Numpy-discussion] Re: Problems compiling against ATLAS on x86_64
In-Reply-To: <20050204143935.GE24701@math.ohiou.edu>
References: <20050203144231.GB24701@math.ohiou.edu> <20050204143935.GE24701@math.ohiou.edu>
Message-ID: <4203C213.5020805@csun.edu>

Matt Hyclak wrote:

>Sorry for replying to myself. Just for the archives, the problem seems to be
>the static lapack/blas libraries provided with RHEL3...
>  
>
Common problem which I just posted to scipy-devel about.  In most 
variants of RH and Fedora, the RH-provided lapack RPM is only there to 
satisfy Octave's dependency.  If you're not using Octave, you'll be 
happiest uninstalling both it and the RedHat provided lapack (rpm -e 
lapack octave)





From oliphant at ee.byu.edu  Fri Feb  4 10:47:45 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb  4 10:47:45 2005
Subject: [Numpy-discussion] Just ignore me.  That was an accidental send.
In-Reply-To: <1107525958.2505.23.camel@halloween.stsci.edu>
References: <42029763.9090700@ee.byu.edu>	 <1107519864.4962.74.camel@jaytmiller.comcast.net> <1107525958.2505.23.camel@halloween.stsci.edu>
Message-ID: <4203C31A.8020607@ee.byu.edu>

Todd Miller wrote:

>Please ignore that last post.   It consists of unfinished thoughts which
>were sent accidentally.  Doing numarray isn't easy.
>  
>
Unfinished or not, there were some good comments here, worth hearing. 

>>I think you've done a real service here,  bounding the cost of new-style
>>classes,  but I'm not convinced we have a bottom line,  at least not in
>>the sense of my own FUD about new style classes.  I'm not contesting the
>>technical direction you're taking with Numeric3 (I hope it works,  but
>>the details are up to you),  but I am contesting the idea that new style
>>classes do nothing but speed up object creation.
>>    
>>
Yes, your point about subclassing new-types causing speed slow downs is 
a very valid one and that has not been tested at all.  But, for my 
purposes, I was just interested to know if allowing a record Array to 
subclass from Numeric3 would cause slow-downs to the rest of Numeric.   
It appears that this will not be the case.  Clearly any type that 
subclasses from Numeric3 might still have speed issues, but those were 
less of a concern for me, right now.

>>So,  I'd assert that unless you solve *that* problem,  using new style
>>classes and some form of inheritance,  saying that new style classes do
>>nothing but speed things up is stretching the truth.  To really disprove
>>the hidden costs in my FUD,  I'd want to see a C basetype and
>>inheritance by a Python subclass.
>>    
>>
Sure.  But, my hope is that only RecordArrays would need to subclass in 
Python (or potentially in C).   These might incur some creation 
"slowdown" but it would be inconsequential to the rest of Numeric.

>>If you can pull it off and still cleanly support the features of
>>numarray,  it will be a triumph of KISS.  But that's not a given.
>>
>>    
>>
Clearly it's not a given, but I have high hopes.  I really think it can 
be done. 

-Travis




From oliphant at ee.byu.edu  Fri Feb  4 11:20:44 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb  4 11:20:44 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1f7baacd417e55debdde0351e463cf10@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>
Message-ID: <4203CAD0.6050609@ee.byu.edu>

konrad.hinsen at laposte.net wrote:

> On 04.02.2005, at 05:14, Michiel Jan Laurens de Hoon wrote:
>
>> give you an example from my own field (computational biology). I am  
>> one of the maintainers of Biopython, which uses LinearAlgebra and  
>> RandomArray. Many of our users are not very familiar with Python. 
>> Even  installing Numerical Python sometimes causes problems, and I'm 
>> sure we  have lost users in the past because of that. SciPy, in my 
>> experience,
>
>
> My experience with users of MMTK, nMOLDYN, and DomainFinder is 
> exactly  the same. Installationproblems are the #1 source for support  
> questions, in particular under Windows and Irix.
>
O.K.   I can see that there are several out there who belive that SciPy 
is sufficiently hard to install that they are concerned about requiring 
it for their math-using packages (despite the provided binary 
distributions, and the work that continues on making it easier to 
install).   I'm very glad to hear these comments, as I constantly wonder 
why people seem to be duplicating SciPy's efforts instead of helping 
SciPy.   I really would like to help make SciPy into something that many 
people can use.  We have done a lot of work in trying to make SciPy 
modular, just for this reason.

While I can appreciate that installation issues are a major headache 
(one reason I support the idea of selling binary-only copies to 
customers), I think that this is an issue no matter what packages you 
decide to deliver, so I don't think it negates my comments that we 
should package numeric routines under the same framework (right now, 
since SciPy already exists, why not there?)

I'm glad to hear at least one specific regarding installation, and that 
is the use of FORTRAN code in SciPy.   I would be very interested to 
know which platforms people have installation difficulties caused by the 
use of FORTRAN.   The biggest problem, I see, is not using FORTRAN, but 
trying to support all the different FORTRAN compilers that might get 
used.  Pearu has done a magnificient job in this difficult area, 
already.   

I very much want to get at the root of why people are staying aloof from 
SciPy, and fix the problems if we can.  SciPy is very much an open 
project.  It is what it is because of the people involved.   Those of us 
developing it so far are pretty easy going and are willing to make 
changes to make others more satisfied with the result.

>> So I would suggest the following demarcation line between Numerical  
>> Python and SciPy:
>>
>> Stuff that is needed by lots of users (linear algebra, FFT, random  
>> numbers, special functions) and is available in ANSI-C (so its  
>> installation is straightforward and won't cause problems to users 
>> who  don't need it) should go into Numerical Python.
>>
If this is an important distinction, why not just place this division in 
SciPy itself?   SciPy's goal is not to be "hard to install", or "only 
for power users."    If these installation issues are real and cannot be 
fixed with better installers, then those of us at SciPy would be very 
glad to see an "easy-to-install" scipy sub-package, that fits into a 
single framework.   

In other words,  why aren't you helping make SciPy better, instead of 
just re-creating what it is doing.  Are you really saying that "scipy is 
beyond hope for me to help?"      Even if you can't get in and change 
SciPy, offering concrete suggestions such as  the one just offered 
"FORTRAN code is too hard to install and so there should be subset of 
SciPy in ANSI-C only" is great.   Then, we can discuss this issue, 
really get to the root of it, and fix the problem.

It would appear that so few actually want to do that.  I actually 
understand, because it sometimes means giving up a treasured notion that 
doesn't fit into the picture, and this happens to me over and over 
again.    Perhaps it is unavoidable.

In sum, I would prefer to see people rally behind SciPy and fix it to be 
a package (or collection of packages) that all can use.  If that doesn't 
happen, I certainly won't fight people who want to make Numeric Packages 
instead.   It just will not be a high priority for me and so won't 
appear in my workflow. 

-Travis






From Fernando.Perez at colorado.edu  Fri Feb  4 11:39:03 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri Feb  4 11:39:03 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203CAD0.6050609@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu>
Message-ID: <4203CF33.3090309@colorado.edu>

Travis Oliphant wrote:

> O.K.   I can see that there are several out there who belive that SciPy 
> is sufficiently hard to install that they are concerned about requiring 
> it for their math-using packages (despite the provided binary 
> distributions, and the work that continues on making it easier to 
> install).   I'm very glad to hear these comments, as I constantly wonder 
> why people seem to be duplicating SciPy's efforts instead of helping 
> SciPy.   I really would like to help make SciPy into something that many 
> people can use.  We have done a lot of work in trying to make SciPy 
> modular, just for this reason.

I can only speak for a linux (specifically Fedora3) environment.  But I'd 
venture that much of this fear is based on the old lore of epic battles fought 
by many (myself included) against scipy cvs.  It is also true that _very 
recently_ there have been the many -no-f2c problems, due to the complexities 
of handling this flag.

An aside for Fedora3 users, the blas/lapack supplied packages are broken:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=138683

Specifically, see my comment 15 there.   As of today (2005/02/04), redhat has 
completely ignored this.  If you want a functional default (non-atlas) lapack, 
you must either build it yourself or grab it from rawhide, at least the -28 
revision.

However, as we've discussed recently on the scipy list, with current CVS, and 
my scripts for rpm-building out of Pearu's binary ATLAS/lapack packages, 
building a fully installable, sub-architecture-specific RPM for scipy requires 
the following:

/path/to/scipy-cvs/ >  python setup.py bdist_rpm --release=$YOUR_ARCH_TAG

That's it.  Period.  One single command, and you get a properly tagged RPM 
which can be handled automatically on a network of machines with multiple 
architectures (you obviously will have to build one for each arch).

What I'm trying to say is that, thanks to all the recent work on this problem, 
its MUCH EASIER than most people think, at least under linux.

For Win32, Joe Cooper's Enthon releases basically solve the problem in one 
clean swoop, and they include not only scipy, but also a ton of other stuff, 
some of which is _a lot_ harder to build than scipy (ever built VTK?).

For OSX, I understand that R. Kern is working on a MacEnthon, which will 
hopefully solve the problem in a similar way.

So I fully agree with Travis that it would probably take a lot less work, and 
benefit our overall combined goals much more, if people spent a bit of their 
time in helping solve the install/distribution issues at the scipy core rather 
than reinventing their own side packages.  That's been my approach, and Pearu 
has been _extremely_ receptive to all my patches and requests, to the point 
where now scipy flies in autopilot for me, even keeping it up to date with raw 
CVS.

Just my $.02

Cheers,

f



From perry at stsci.edu  Fri Feb  4 12:01:38 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Fri Feb  4 12:01:38 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203CAD0.6050609@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu>
Message-ID: <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>

On Feb 4, 2005, at 2:19 PM, Travis Oliphant wrote:

> konrad.hinsen at laposte.net wrote:
>
>> On 04.02.2005, at 05:14, Michiel Jan Laurens de Hoon wrote:
>>
>>> give you an example from my own field (computational biology). I am  
>>> one of the maintainers of Biopython, which uses LinearAlgebra and  
>>> RandomArray. Many of our users are not very familiar with Python. 
>>> Even  installing Numerical Python sometimes causes problems, and I'm 
>>> sure we  have lost users in the past because of that. SciPy, in my 
>>> experience,
>>
>>
>> My experience with users of MMTK, nMOLDYN, and DomainFinder is 
>> exactly  the same. Installationproblems are the #1 source for support 
>>  questions, in particular under Windows and Irix.
>>
> O.K.   I can see that there are several out there who belive that 
> SciPy is sufficiently hard to install that they are concerned about 
> requiring it for their math-using packages (despite the provided 
> binary distributions, and the work that continues on making it easier 
> to install).   I'm
[...]
This sort of leads back to the original point, that is, putting 
something in the Python Standard Library. I suspect that for most 
scientists and engineers if they have to install anything extra at all 
to use arrays practially, they really don't care if it is in the core 
or not. The current Numeric (or numarray) is a fairly straightforward 
install now and if the issue is having users avoid this extra install, 
putting a reduced functionality array package in the core is not going 
to address this issue. On the other hand, it is likely to have the 
general Python community more likely to use arrays as an interchange 
format, which is a separate benefit (a point I stressed at the last 
scipy conference).

In other words, we should be clear about what should go into the core 
and what the sought-after benefit is. I don't think that has been done 
yet. If scipy is to be the solution for all scientists and engineers, 
why should Numeric (or numarray) go into the core? They still will have 
to install scipy (or some extra package) anyway.

Perry




From konrad.hinsen at laposte.net  Fri Feb  4 12:08:07 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb  4 12:08:07 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203CAD0.6050609@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu>
Message-ID: <55c095f595dfec3e16579094faea14f2@laposte.net>

On Feb 4, 2005, at 20:19, Travis Oliphant wrote:

> O.K.   I can see that there are several out there who belive that 
> SciPy is sufficiently hard to install that they are concerned about 
> requiring it for their math-using

It's not just belief, it's my own experience, admittedly not very 
recent (I last checked about a year ago).

> packages (despite the provided binary distributions, and the work that 
> continues on

For Windows and Debian - that covers none of the machines I have ever 
used in my scientific career. Moreover, my experience with binary 
installations for Linux in general is not so good. Hardly anyone uses 
an unmodified standard distribution, there's nearly always something 
that doesn't fit.

> While I can appreciate that installation issues are a major headache 
> (one reason I support the idea of selling binary-only copies to 
> customers), I think that this is an issue no matter what packages you 
> decide to deliver, so I don't think it negates my

In principle, yes, but there are significant differences in "mean 
expected trouble". Some libraries are moving targets (GTK comes to 
mind), and dependencies on them are a big source of fragility. With 
only C code and no external dependencies, there is rarely ever a 
problem, and that's why Numeric is a low-risk package.

There's also integration with other stuff to be considered. Take the 
installation instructions for MaxOS X for example. They start by 
recommending not to use the Fink package. Good advice, because I never 
got it to work. Fink has enormous package interdependencies, so usually 
you need to install or upgrade 10 or more packages, and the chance for 
one of them not installing correctly is quite high.

But then, following the instructions puts SciPy with the Apple-provided 
python. Fine if SciPy is all you want, but I have dozens of other 
Python packages installed for the Fink Python, which in general is the 
better choice because there is a very complete set of easy-to-install 
Python packages from Fink.

The net result is that I still don't have SciPy on my Mac, which is my 
main development machine.

> people have installation difficulties caused by the use of FORTRAN.   
> The biggest problem, I see, is not using FORTRAN, but trying to 
> support all the different FORTRAN compilers that might get used.  
> Pearu has done a magnificient job in

There's also the need to have a Fortran compiler. Installing g77 is not 
exactly an easy task for the average user.

> If this is an important distinction, why not just place this division 
> in SciPy itself?   SciPy's goal is not to be "hard to install", or 
> "only for power users."    If these installation issues are real and 
> cannot be fixed with better installers, then those of us at SciPy 
> would be very glad to see an "easy-to-install" scipy sub-package, that 
> fits into a single framework.

That sounds like a very good idea.

> In other words,  why aren't you helping make SciPy better, instead of 
> just re-creating what it is doing.  Are you really saying that "scipy 
> is beyond hope for me to

Because in the short run that's the path of least effort, or least 
pain: I actually like coding, but I strongly dislike sorting out 
installation problems. I dislike even more sorting out someone else's 
installation problems, which is what I would have to do if my published 
code depended on SciPy.

This is in fact not just a SciPy issue. Installation problems and the 
fragility of package interdependencies are the #1 problem in the 
OpenSource world in my opinion.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From oliphant at ee.byu.edu  Fri Feb  4 14:23:08 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb  4 14:23:08 2005
Subject: [Numpy-discussion] Optimization question for ufuncs
Message-ID: <4203F5A3.3020402@ee.byu.edu>

I've been thinking lately about ufuncs and I would love to hear the 
opinion of others.

I like what numarray has done with the temporary buffer ideas so that 
full copies are never made if they are just going to be thrown away.   
This has led to other thoughts about possible improvements to the ufunc 
object to support "ufunc chaining" so that array operations on 
expressions don't have to create any temporary copies (using buffers 
instead) --- I think I remember the numarray guys thinking along these 
lines as well.

Regardless, there is always an inner for loop (for each type) that 
performs the requested operation.   The question I have is whether to 
assume unit strides for the inner loop.  The current Numeric ufunc inner 
loops allow for discontiguous memory to be accessed during the loop 
(non-unit strides).   I'm not sure what numarray does, I think it only 
allows for unit strides and uses temporary buffers to support 
discontiguous arrays.

Is this requirement for unit-strides on the inner loop a good one?  Does 
it allow faster code to be compiled?   Is it part of the reason that 
numarray is a little faster on large arrays?

I am not an optimization expert, though I've read a bit as of late.  I'm 
just wondering what the experts on this list think about unit-strides 
versus non unit-strides on the inner loop?

Thanks,

Travis O.




From oliphant at ee.byu.edu  Fri Feb  4 14:32:06 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb  4 14:32:06 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
Message-ID: <4203F7CE.8070403@ee.byu.edu>

>> O.K.   I can see that there are several out there who belive that 
>> SciPy is sufficiently hard to install that they are concerned about 
>> requiring it for their math-using packages (despite the provided 
>> binary distributions, and the work that continues on making it easier 
>> to install).   I'm
>
> [...]
> This sort of leads back to the original point, that is, putting 
> something in the Python Standard Library. I suspect that for most 
> scientists and engineers if they have to install anything extra at all 
> to use arrays practially, they really don't care if it is in the core 
> or not. The current Numeric (or numarray) is a fairly straightforward 
> install now and if the issue is having users avoid this extra install, 
> putting a reduced functionality array package in the core is not going 
> to address this issue. On the other hand, it is likely to have the 
> general Python community more likely to use arrays as an interchange 
> format, which is a separate benefit (a point I stressed at the last 
> scipy conference).
>
As Perry suggested, I personally don't care whether it is in the Python 
core or not.  The only reason I ever wanted to put it into the core is 
because of the PIL.  When I asked the creator of the PIL, why he did not 
use Numeric Python, his response was that it wasn't in the core.    
Right now, I think we would benefit if the basic multidimensional array 
object were in the core.  It would be nicer if we could agree on how 
basic operations were to behave so math could be done with them, and put 
that in there as well.

Perry makes a great point about the benefit of using arrays as an 
interchange format...  There are people out there who use the array  
object in Python for just that.   It would be great if there was one 
multidimensional array object used by the entire Python community for 
applications requiring such a structure, but it wouldn't remove the need 
for "more stuff" to do science and so my interest in getting such an 
object into the Python core has waned considerably over the years. 

I'm more interested now in uniting a fractured community (or at least 
making interoperability easier). 

-Travis








From Chris.Barker at noaa.gov  Fri Feb  4 14:53:45 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Fri Feb  4 14:53:45 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <55c095f595dfec3e16579094faea14f2@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <55c095f595dfec3e16579094faea14f2@laposte.net>
Message-ID: <4203FCD8.8070204@noaa.gov>

konrad.hinsen at laposte.net wrote:

>> SciPy is sufficiently hard to install that they are concerned about 
>> requiring it for their math-using

> It's not just belief, it's my own experience,

And mine. I love the idea of SciPy, and have been following the project 
since the beginning, but have never really used it. I've tried a number 
of times, and it's been a pain.

> admittedly not very recent 

me neither.

> For Windows and Debian - that covers none of the machines

I use Gentoo and OS-X. I'm used to weird installs (or I wouldn't use 
Gentoo), but my colleagues all use OS-X, and It I can't give them an 
easy installer, forget it.

 > My experience with binary
> installations for Linux in general is not so good.

Mine neither, but source installs are far easier there than any other 
system! Probably because a lot of developers use Linux, but also because 
it's a very developer and open-source friendly system. Right now I'm 
working on making a binary for matplotlib on OS-X. No one would have a 
Linux box without libpng and libfreetype, but OS-X doesn't have those 
out of the box.

> In principle, yes, but there are significant differences in "mean 
> expected trouble". Some libraries are moving targets (GTK comes to 
> mind), and dependencies on them are a big source of fragility.

Exactly, and things get weird with overlapping dependencies. PyGTK 
requires X11, which has libfreetype. matplotlib requires freetype, but 
can be used with or without pyGTK....
> With only 
> C code and no external dependencies, there is rarely ever a problem, and 
> that's why Numeric is a low-risk package.

Except that it's setup.py has been broken for the last few releases. 
That's been a problem for a number of users.

By the way, here OS-X is much better than other systems, it comes with 
lapack built in! (Veclib)

 > Fine if SciPy is all you want, but I have dozens of other Python
> packages installed for the Fink Python, which in general is the better 
> choice because there is a very complete set of easy-to-install Python 
> packages from Fink.

Actually, I disagree. Apple's Python is a better choice (you can't run 
Aqua applications from a fink python, at least I don't think so), but we 
do need to do more to provide easy to install packages for it. (there 
are a lot)

Here Fink drives me crazy. It's a system-within-a-system. I want 
everything on my Mac to fit in well with my Mac. If there was a 
fink-like system (I haven't yet tried darwinports or Gentoo) that didn't 
duplicate Apples supplied stuff with incompatible versions (Python, 
X11), I'd be much happier.

> The net result is that I still don't have SciPy on my Mac

Me neither, though I don't think it's SciPy's fault. The core problem is 
  that there aren't all that many people using OS-X developing Python 
packages, and Apple has, of course, made OS-X just different enough from 
other unices to require some work (or take the easy way and use fink, 
but then it doesn't play with the rest of the Mac). Ironically, this 
should be even harder on Windows, but there are enough Windows users 
that someone usually gets around making a binary.

>> people have installation difficulties caused by the use of FORTRAN.   
>> The biggest problem, I see, is not using FORTRAN, but trying to 
>> support all the different FORTRAN compilers that might get used.

What's the distinction? Again, Apple doesn't provide f77. That's been a 
stopper for me.

> Because in the short run that's the path of least effort, or least pain: 

This is a key issue with all open-source projects (at least ones that 
aren't funder). We all do the easiest thing to get the functionality we 
need. If we combined all our duplicate efforts, we'd be in great shape, 
but an any given moment, I mostly need to get my job done, not make a 
better SciPy, or whatever.

> This is in fact not just a SciPy issue. Installation problems and the 
> fragility of package interdependencies are the #1 problem in the 
> OpenSource world in my opinion.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From Chris.Barker at noaa.gov  Fri Feb  4 14:58:56 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Fri Feb  4 14:58:56 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203F7CE.8070403@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu> <4203F7CE.8070403@ee.byu.edu>
Message-ID: <4203FDFF.4060904@noaa.gov>

Travis Oliphant wrote:
> The only reason I ever wanted to put it into the core is 
> because of the PIL.  When I asked the creator of the PIL, why he did not 
> use Numeric Python, his response was that it wasn't in the core.    

Same with wxPython. If you draw a line with wxPython, and pass in a 
(N,2) array for the point coordinates, it's slower than a list of 
tuples. With numarray, a LOT slower. If the core multiarray type were in 
the standard lib, wxPython would have code to use them and be much 
faster for this kind of thing.

> Perry makes a great point about the benefit of using arrays as an 
> interchange format...  There are people out there who use the array  
> object in Python for just that.   It would be great if there was one 
> multidimensional array object used by the entire Python community for 
> applications requiring such a structure, but it wouldn't remove the need 
> for "more stuff" to do science

No, but I'd also love to see at least the basic math operations in the 
core too. There are so many questions on c.l.p (and other places) that 
just cry out for a NumPy solution!

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From verveer at embl.de  Fri Feb  4 16:44:50 2005
From: verveer at embl.de (Peter Verveer)
Date: Fri Feb  4 16:44:50 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203CAD0.6050609@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu>
Message-ID: <3ba3b109f1f344e31b758074cbc7f64c@embl.de>

On Feb 4, 2005, at 8:19 PM, Travis Oliphant wrote:

> konrad.hinsen at laposte.net wrote:
>
>> On 04.02.2005, at 05:14, Michiel Jan Laurens de Hoon wrote:
>>
>>> give you an example from my own field (computational biology). I am  
>>> one of the maintainers of Biopython, which uses LinearAlgebra and  
>>> RandomArray. Many of our users are not very familiar with Python. 
>>> Even  installing Numerical Python sometimes causes problems, and I'm 
>>> sure we  have lost users in the past because of that. SciPy, in my 
>>> experience,
>>
>>
>> My experience with users of MMTK, nMOLDYN, and DomainFinder is 
>> exactly  the same. Installationproblems are the #1 source for support 
>>  questions, in particular under Windows and Irix.
>>
> O.K.   I can see that there are several out there who belive that 
> SciPy is sufficiently hard to install that they are concerned about 
> requiring it for their math-using packages (despite the provided 
> binary distributions, and the work that continues on making it easier 
> to install).

I have in the past tried to install SciPy on OSX and I had big problems 
with it. Motivated by the recent discussions on the list I tried again 
today, and found it much improved, but still quite involved. It would 
still be a pain to distribute software that has a dependency on SciPy 
to 'average users' that would have trouble installing SciPy. There does 
not seem to be a binary distribution on scipy.org for OS X, which would 
solve that problem. This is not unimportant, since I guess OS X is 
gaining popularity in scientific computing.

>   I'm very glad to hear these comments, as I constantly wonder why 
> people seem to be duplicating SciPy's efforts instead of helping 
> SciPy.

I agree with that, but fact is that the split between Numarray and 
Numeric prevented me from contributing to SciPy. I have developed for 
numarray (the image processing package nd_image, part of numarray, is 
done by me) because I felt numarray was much better then Numeric for 
that application, and that also ruled out SciPy. There seems now to be 
an effort going to either try to merge the two or to bring Numeric up 
to numarray standard. Thats great, and if that works out I will try to 
rewrite my package so it works with both. Would  be happy to contribute 
it to SciPy. But I would make sure it remains a package that can be 
installed separately from  SciPy if required.

> I really would like to help make SciPy into something that many people 
> can use.  We have done a lot of work in trying to make SciPy modular, 
> just for this reason.

Making SciPy modular is in my opinion a very good approach. Why do I 
need to install all of it, if I really  only  need a few parts? If I 
can install only the subset I need, the likelihood that I will have an 
easier job doing that will increase. That would make it also possible 
to have sub-sets that are free of dependencies such as FORTRAN.

> In sum, I would prefer to see people rally behind SciPy and fix it to 
> be a package (or collection of packages) that all can use.  If that 
> doesn't happen, I certainly won't fight people who want to make 
> Numeric Packages instead.   It just will not be a high priority for me 
> and so won't appear in my workflow.

On the other hand, I would think it would be a good idea to write 
software that does not  have dependencies on big packages such as SciPy 
when possible, but only on Numeric/Numarray. That does not exclude them 
from being included in SciPy. It seems to me that it would be a 
desirable property if a package can be installed just with Numeric (or 
numarray), with an appropriate sub-set of SciPy, or come with the big 
SciPy package. I would rather see a "collection of SciPy packages" then 
a "SciPy package".

Just my 2 cents, although currently I am using numarray exclusively, I 
am glad to see that there is some discussion going on to get the main 
python numerical packages to work together.

Peter





From oliphant at ee.byu.edu  Fri Feb  4 17:37:02 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb  4 17:37:02 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <3ba3b109f1f344e31b758074cbc7f64c@embl.de>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>
Message-ID: <42042318.2090704@ee.byu.edu>

>
> I have in the past tried to install SciPy on OSX and I had big 
> problems with it. Motivated by the recent discussions on the list I 
> tried again today, and found it much improved, but still quite 
> involved. It would still be a pain to distribute software that has a 
> dependency on SciPy to 'average users' that would have trouble 
> installing SciPy. There does not seem to be a binary distribution on 
> scipy.org for OS X, which would solve that problem. This is not 
> unimportant, since I guess OS X is gaining popularity in scientific 
> computing.

A binary for OS X is a desireable thing.  I think more of a stink should 
be made, so that one gets provided by default.  I know that Robert Kern 
has worked on such a thing and it is not out of the question, but is on 
the horizon.   More users asking for it would help.

>>   I'm very glad to hear these comments, as I constantly wonder why 
>> people seem to be duplicating SciPy's efforts instead of helping SciPy.
>
>
> I agree with that, but fact is that the split between Numarray and 
> Numeric prevented me from contributing to SciPy. I have developed for 
> numarray (the image processing package nd_image, part of numarray, is 
> done by me) because I felt numarray was much better then Numeric for 
> that application, and that also ruled out SciPy. There seems now to be 
> an effort going to either try to merge the two or to bring Numeric up 
> to numarray standard. Thats great, and if that works out I will try to 
> rewrite my package so it works with both. Would  be happy to 
> contribute it to SciPy. But I would make sure it remains a package 
> that can be installed separately from  SciPy if required.
>
"What scipy is" is quite open to interpretation.  The idea of scipy is 
just to have a super-package where different packages can live and have 
some kind of inter-operability.   It is made so that sub-packages can be 
easily added and built upon request.   We try hard to make sure that the 
only real dependency is scipy_base which should not be hard to install 
anywhere (does not require ATLAS or FORTRAN). 

It would be great to add nd_image as a sub-package to SciPy.    In fact, 
nd_image was one of the things that most alarmed me about the emerging 
split in the community, as it reproduces some of what is already in 
SciPy (N-D convolution)  while adding some great things that I've always 
wanted in SciPy (morphology).

I believe scipy can be easily installed as a collection of packages if 
that is desired.  Pearu has done an incredible job with scipy_disutils 
to make that process simple.   Making a single super package helps users 
see that they have only a part of what's available (but maybe all they 
need), and you get the added benefit of standard documentation 
efforts.    Nobody has taken up the ball to actually try and distribute 
individual packages of scipy, but it is a welcome effort.   
www.scipy.org has places where such packages could live.  Enthought has 
mainly been interested in distributing "batteries-included" packages for 
windows because it is what their customers need.  Other are welcome to 
pick up the slack.

Perhaps it would be good to get a list of which platforms actually need 
binaries for people to look more favorably at SciPy.   I can see that OS 
X is on the top of that list, surely somebody could distribute a copy of 
a binary for that platform (I know of people who have it installed).     
Also, scipy is still version 0.3 not because the code is unstable but 
because the layout of the packages is still open to change.  Come and 
discuss your visions of what it should look like with us on the 
scipy-dev at scipy.net  mailing list.

> Making SciPy modular is in my opinion a very good approach. Why do I 
> need to install all of it, if I really  only  need a few parts? If I 
> can install only the subset I need, the likelihood that I will have an 
> easier job doing that will increase. That would make it also possible 
> to have sub-sets that are free of dependencies such as FORTRAN.

>
> On the other hand, I would think it would be a good idea to write 
> software that does not  have dependencies on big packages such as 
> SciPy when possible, but only on Numeric/Numarray. That does not 
> exclude them from being included in SciPy. It seems to me that it 
> would be a desirable property if a package can be installed just with 
> Numeric (or numarray), with an appropriate sub-set of SciPy, or come 
> with the big SciPy package. I would rather see a "collection of SciPy 
> packages" then a "SciPy package".

I'd like to see more of the development work that goes on happen under a 
common standard, that is all.  SciPy has done a lot of work to provide 
an infrastructure within which to distribute packages.  The field is 
built, but people don't seem to want to come.    I understand that the 
Numeric / Numarray dichotomy is a big source of this problem.  That is 
the whole reason I'm devoting a big portion of my time to the problem 
right now.  

My very ambitious goal with Numeric3 is to replace both Numeric and 
Numarray (heavily borrowing code from each).  When I say replace them, 
I'm talking about the array object and umath module.  I'm not talking 
about the other packages that use them.  I want those to live under a 
common standard (i.e. scipy).   It would be fine with me if that common 
standard include a scipy_lite or something like that which was easy to 
install, if that is desired.   But, if the problem is really just 
availability of binary copies, then why don't we solve that instead of 
worrying about installation on platforms that nobody uses.   In 
addition, even though I've developed a lot of scipy, I'm willing to 
throw it all out or alter it considerably in order to satisfy the 
demands that exist for package developers to want to put their weight 
behind something like SciPy.   So, don't equate history with an 
unwillingness to cooperate. 

I think you'll find the people involved with scipy to be incredibly 
willing to cooperate to achieve mutual success.  If you have users of a 
package that demand a certain binary of scipy to be installed, then 
bring them over.  I can almost guarantee you that a copy of scipy will 
be available for them. 

While I do recognize the value of repeated work.  When it comes to 
interoperability, single standards seem to work best.  So, the longer 
this unnecessary division of resources goes on, the farther behind 
Python will get for doing anything but niche work. 

-Travis






From paul at pfdubois.com  Fri Feb  4 20:04:21 2005
From: paul at pfdubois.com (Paul F. Dubois)
Date: Fri Feb  4 20:04:21 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42042318.2090704@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu>
Message-ID: <42044565.1030902@pfdubois.com>

My experience with binary distributions has convinced me that it is a 
hopeless endeavor in the long run. I understand why people want them but 
until you've tried to provide them to the public and seen the number of 
things that go wrong ... Not to mention that it means you have to have 
someone build on all those systems and you don't have them all. And if 
the person is going to add their other extensions, compiled with a 
different compiler, blooie.


Travis Oliphant wrote:
> 
>>
>> I have in the past tried to install SciPy on OSX and I had big 
>> problems with it. Motivated by the recent discussions on the list I 
>> tried again today, and found it much improved, but still quite 
>> involved. It would still be a pain to distribute software that has a 
>> dependency on SciPy to 'average users' that would have trouble 
>> installing SciPy. There does not seem to be a binary distribution on 
>> scipy.org for OS X, which would solve that problem. This is not 
>> unimportant, since I guess OS X is gaining popularity in scientific 
>> computing.
> 
> 
> A binary for OS X is a desireable thing.  I think more of a stink should 
> be made, so that one gets provided by default.  I know that Robert Kern 
> has worked on such a thing and it is not out of the question, but is on 
> the horizon.   More users asking for it would help.
> 
>>>   I'm very glad to hear these comments, as I constantly wonder why 
>>> people seem to be duplicating SciPy's efforts instead of helping SciPy.
>>
>>
>>
>> I agree with that, but fact is that the split between Numarray and 
>> Numeric prevented me from contributing to SciPy. I have developed for 
>> numarray (the image processing package nd_image, part of numarray, is 
>> done by me) because I felt numarray was much better then Numeric for 
>> that application, and that also ruled out SciPy. There seems now to be 
>> an effort going to either try to merge the two or to bring Numeric up 
>> to numarray standard. Thats great, and if that works out I will try to 
>> rewrite my package so it works with both. Would  be happy to 
>> contribute it to SciPy. But I would make sure it remains a package 
>> that can be installed separately from  SciPy if required.
>>
> "What scipy is" is quite open to interpretation.  The idea of scipy is 
> just to have a super-package where different packages can live and have 
> some kind of inter-operability.   It is made so that sub-packages can be 
> easily added and built upon request.   We try hard to make sure that the 
> only real dependency is scipy_base which should not be hard to install 
> anywhere (does not require ATLAS or FORTRAN).
> It would be great to add nd_image as a sub-package to SciPy.    In fact, 
> nd_image was one of the things that most alarmed me about the emerging 
> split in the community, as it reproduces some of what is already in 
> SciPy (N-D convolution)  while adding some great things that I've always 
> wanted in SciPy (morphology).
> 
> I believe scipy can be easily installed as a collection of packages if 
> that is desired.  Pearu has done an incredible job with scipy_disutils 
> to make that process simple.   Making a single super package helps users 
> see that they have only a part of what's available (but maybe all they 
> need), and you get the added benefit of standard documentation 
> efforts.    Nobody has taken up the ball to actually try and distribute 
> individual packages of scipy, but it is a welcome effort.   
> www.scipy.org has places where such packages could live.  Enthought has 
> mainly been interested in distributing "batteries-included" packages for 
> windows because it is what their customers need.  Other are welcome to 
> pick up the slack.
> 
> Perhaps it would be good to get a list of which platforms actually need 
> binaries for people to look more favorably at SciPy.   I can see that OS 
> X is on the top of that list, surely somebody could distribute a copy of 
> a binary for that platform (I know of people who have it installed).     
> Also, scipy is still version 0.3 not because the code is unstable but 
> because the layout of the packages is still open to change.  Come and 
> discuss your visions of what it should look like with us on the 
> scipy-dev at scipy.net  mailing list.
> 
>> Making SciPy modular is in my opinion a very good approach. Why do I 
>> need to install all of it, if I really  only  need a few parts? If I 
>> can install only the subset I need, the likelihood that I will have an 
>> easier job doing that will increase. That would make it also possible 
>> to have sub-sets that are free of dependencies such as FORTRAN.
> 
> 
>>
>> On the other hand, I would think it would be a good idea to write 
>> software that does not  have dependencies on big packages such as 
>> SciPy when possible, but only on Numeric/Numarray. That does not 
>> exclude them from being included in SciPy. It seems to me that it 
>> would be a desirable property if a package can be installed just with 
>> Numeric (or numarray), with an appropriate sub-set of SciPy, or come 
>> with the big SciPy package. I would rather see a "collection of SciPy 
>> packages" then a "SciPy package".
> 
> 
> I'd like to see more of the development work that goes on happen under a 
> common standard, that is all.  SciPy has done a lot of work to provide 
> an infrastructure within which to distribute packages.  The field is 
> built, but people don't seem to want to come.    I understand that the 
> Numeric / Numarray dichotomy is a big source of this problem.  That is 
> the whole reason I'm devoting a big portion of my time to the problem 
> right now. 
> My very ambitious goal with Numeric3 is to replace both Numeric and 
> Numarray (heavily borrowing code from each).  When I say replace them, 
> I'm talking about the array object and umath module.  I'm not talking 
> about the other packages that use them.  I want those to live under a 
> common standard (i.e. scipy).   It would be fine with me if that common 
> standard include a scipy_lite or something like that which was easy to 
> install, if that is desired.   But, if the problem is really just 
> availability of binary copies, then why don't we solve that instead of 
> worrying about installation on platforms that nobody uses.   In 
> addition, even though I've developed a lot of scipy, I'm willing to 
> throw it all out or alter it considerably in order to satisfy the 
> demands that exist for package developers to want to put their weight 
> behind something like SciPy.   So, don't equate history with an 
> unwillingness to cooperate.
> I think you'll find the people involved with scipy to be incredibly 
> willing to cooperate to achieve mutual success.  If you have users of a 
> package that demand a certain binary of scipy to be installed, then 
> bring them over.  I can almost guarantee you that a copy of scipy will 
> be available for them.
> While I do recognize the value of repeated work.  When it comes to 
> interoperability, single standards seem to work best.  So, the longer 
> this unnecessary division of resources goes on, the farther behind 
> Python will get for doing anything but niche work.
> -Travis
> 
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 



From konrad.hinsen at laposte.net  Sat Feb  5 02:09:41 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb  5 02:09:41 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203FCD8.8070204@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <55c095f595dfec3e16579094faea14f2@laposte.net> <4203FCD8.8070204@noaa.gov>
Message-ID: <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net>

On 04.02.2005, at 23:53, Chris Barker wrote:

>> With only C code and no external dependencies, there is rarely ever a 
>> problem, and that's why Numeric is a low-risk package.
>
> Except that it's setup.py has been broken for the last few releases. 
> That's been a problem for a number of users.

Indeed.

> By the way, here OS-X is much better than other systems, it comes with 
> lapack built in! (Veclib)

I didn't know it has lapack, I thought it was only BLAS!

> Actually, I disagree. Apple's Python is a better choice (you can't run 
> Aqua applications from a fink python, at least I don't think so), but 
> we do need to do more to provide easy to install packages for it. 
> (there are a lot)

I don't care for Aqua as much as I care for all those nice Python 
packages I use every day. With Apple's Python, it's command line 
installation (which I don't mind, but many of my users do), or 
PackageManager for a select few packages. And I can't my own without 
proving my own Web server as well, so package contribution is 
effectively discouraged.

> Here Fink drives me crazy. It's a system-within-a-system. I want 
> everything

I certainly agree on that. I curse Fink about once a week. But in the 
absence of an alternative, Fink is better than no Fink.

>  on my Mac to fit in well with my Mac. If there was a fink-like system 
> (I haven't yet tried darwinports or Gentoo) that didn't duplicate 
> Apples supplied stuff with incompatible versions (Python, X11), I'd be 
> much happier.

Agreed. But there isn't.

>> The net result is that I still don't have SciPy on my Mac
>
> Me neither, though I don't think it's SciPy's fault. The core problem 
> is  that

Attributing fault makes little sense in the OpenSource world.

>  there aren't all that many people using OS-X developing Python 
> packages, and Apple has, of course, made OS-X just different enough 
> from other unices to require some work (or take the easy way and use 
> fink, but then it

Not really, you can do command-line installation ("python setup.py...") 
just like on any other Unix, usually without any modification. The 
problem is that Mac users expect installers.

It should be possible to extend distutils for making Mac installers 
just like it can make RPMs and Windows installers. I suppose nobody 
looked into this because the Apple Python community is divided into 
those who come from the Unix world and go for Fink, and those who come 
from pre-10 MacOS and go for MacPython plus PackageManager.

> What's the distinction? Again, Apple doesn't provide f77. That's been 
> a stopper for me.

I guess Apple's point of view is that you should buy a compiler. But 
they could easily add g77 to their developer CD, so there is indeed 
little excuse.

> This is a key issue with all open-source projects (at least ones that 
> aren't funder). We all do the easiest thing to get the functionality 
> we need. If we combined all our duplicate efforts, we'd be in great 
> shape, but an any given moment, I mostly need to get my job done, not 
> make a better SciPy, or whatever.

Right. But I still wonder why non-scientific projects (e.g. Python 
itself, or Linux) don't suffer so much from this problem. The larger 
potential developer base is probably just one part of the answer.

BTW, there is a scientific study of the Python development process that 
most Pythoneers are probably not aware of but which I think is quite 
interesting: "A Methodological Framework for Socio-Cognitive  Analyses 
of Collaborative Design of Open Source Software" by W. Sack, F. 
D?tienne, J-M. Burkhardt, F. Barcellini,  N. Ducheneaut, and D. 
Mahendran. Easy to find via Google.

Konrad.




From jochen at fhi-berlin.mpg.de  Sat Feb  5 04:15:46 2005
From: jochen at fhi-berlin.mpg.de (=?iso-8859-1?Q?Jochen_K=FCpper?=)
Date: Sat Feb  5 04:15:46 2005
Subject: The need for a unified numerical array object (was: [Numpy-discussion] Numeric3)
In-Reply-To: <4203F7CE.8070403@ee.byu.edu> (Travis Oliphant's message of "Fri,
	04 Feb 2005 15:31:42 -0700")
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	<1f7baacd417e55debdde0351e463cf10@laposte.net>
	<4203CAD0.6050609@ee.byu.edu>
	<f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
	<4203F7CE.8070403@ee.byu.edu>
Message-ID: <9ezmyjursi.fsf_-_@gowron.rz-berlin.mpg.de>

Travis Oliphant <oliphant at ee.byu.edu> writes:

> I'm more interested now in uniting a fractured community (or at
> least making interoperability easier).

All,

although I am a bit too busy these days I wanna add my 1.5ct:

Few years ago I had the hope I could do (almost) all my calculations
in python, maybe writing a few limited extension modules in C. 

That hope is cured for now. Generally, all the necessary pieces are
there, but they are scattered over many packages: Numeric/Numarray,
scipy, pygsl, plus stuff like pyQt/pyGTK/wxWindows/... But getting
everything up and running for myself and colleagues is just to
demanding, esp. if system administrators are busy as well and
reluctant to mess up their system or work schedule.

Therefore I do almost all calculations in C++ right now. Compilers are
there, BLAS, LAPACK and even GSL are typically provided already,
otherwise admins do not hesitate much to install the latest binary
packages, which are available. It takes me a little longer to develop
code, but I know I can run it on any system I find and I easily save
the time their.

I still do small stuff in Python and use Numarray for it, but
everything that requires anything else I simply write in C++.

What I would love to see is a twofold system for numerics in Python:

An arrayobject plus simple ufuncs like min/max/sin/exp/... in the
Python core.

A "single" scientific Python project that provides multiple packages
using that arrayobject. There should be packages, for example, with
  a) Python+C only to provide basic linear algebra, FFT, ... with
     everything included (no external dependencies)
  b) FFTW, GSL, ... wrappers that still are only Python + C, but
     require external libraries to be installed.
  c) An "expert" package which wraps the most advanced libraries for
     each field, i.e. LAPACK. Here is should be possible to install
     each wrapper separately.

If would be nice if the subset a) of b) would use an identical
interface as a), similar for c) <- b).

Moreover I would see it advantageous to have three independent
packages here, so I could tell a sysadmin: "Please install
SciencePython_simple" or "Please install SciencePython_wrappers"
instead of "Please install SciencePython with options a, b, c".
Somehow these sys-mins feel that is easier.


Anyway, I appreciate all of yours work and I do understand that first
of all you have to solve your own problems. However, if this scattered
community manages to unify and simplify (lower entrance barriers)
their approach to Scientific calculations with Python, that will allow
it to grow much further than ever possible with the current situation,
that somehow limits itself to double experts: scientists with *good*
computer science knowledge or vice versa.

Time will tell, but I wish you all the luck you need! For me new
projects will come up constantly, maybe even a larger one will be in
Python again sometime.

Greetings, 
Jochen
-- 
Einigkeit und Recht und Freiheit                http://www.Jochen-Kuepper.de
    Libert?, ?galit?, Fraternit?                GnuPG key: CC1B0B4D
        (Part 3 you find in my messages before fall 2003.)




From mdehoon at ims.u-tokyo.ac.jp  Sat Feb  5 06:24:17 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Sat Feb  5 06:24:17 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203CAD0.6050609@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu>
Message-ID: <4204D6F1.50106@ims.u-tokyo.ac.jp>

Travis Oliphant wrote:
> O.K.   I can see that there are several out there who belive that
> SciPy is sufficiently hard to install that they are concerned about
> requiring it for their math-using packages (despite the provided
> binary distributions, and the work that continues on making it easier
> to install).

Relying on binary distributions should be a method of last resort. You
can never keep up with all the possible operating systems x Python
versions x Numeric versions. I run Python on four computers here
(Windows, Mac OS X, Linux, and Unix), and only for Linux there is a
SciPy binary installer available (on Windows I am using Python 2.4, for
which there is no installer). Now SciPy binary installers can be made
for all these platforms, but I am sure that you have better ways to
spend your time.

We need to think carefully about what the goal of SciPy actually is. I
am worried that SciPy is turning into a goal in itself. To my mind, the
goal of Numerical Python and SciPy should be to help users perform
numerical calculations. Numerical Python does that by making commonly
needed numerical routines available as a convenient package. SciPy can
do that by making less-needed routines available either as a binary
installer or as source code. I don't see how moving well-functioning 
code from Numerical Python to SciPy will help any user.

--Michiel.




From verveer at embl.de  Sat Feb  5 08:01:44 2005
From: verveer at embl.de (Peter Verveer)
Date: Sat Feb  5 08:01:44 2005
Subject: The need for a unified numerical array object (was: [Numpy-discussion] Numeric3)
In-Reply-To: <9ezmyjursi.fsf_-_@gowron.rz-berlin.mpg.de>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu> <4203F7CE.8070403@ee.byu.edu> <9ezmyjursi.fsf_-_@gowron.rz-berlin.mpg.de>
Message-ID: <f8ad905b157df63154e45d42d9b58987@embl.de>

On Feb 5, 2005, at 1:15 PM, Jochen K?pper wrote:

> Travis Oliphant <oliphant at ee.byu.edu> writes:
>
>> I'm more interested now in uniting a fractured community (or at
>> least making interoperability easier).
> I still do small stuff in Python and use Numarray for it, but
> everything that requires anything else I simply write in C++.
>
> What I would love to see is a twofold system for numerics in Python:
>
> An arrayobject plus simple ufuncs like min/max/sin/exp/... in the
> Python core.

Yes, I am starting to think that this approach is the only one that 
would cure the split we currently have between Numarray and 
Numeric/SciPy. Provide this, and make sure that Numarray/SciPy can use 
its Python/C API's and we can take it from there.

> A "single" scientific Python project that provides multiple packages
> using that arrayobject. There should be packages, for example, with
>   a) Python+C only to provide basic linear algebra, FFT, ... with
>      everything included (no external dependencies)

That will be needed and should be easy to provide.  Basically these 
exist already.

>   b) FFTW, GSL, ... wrappers that still are only Python + C, but
>      require external libraries to be installed.

Don't forget that these packages tend to be using the GPL license. That 
should be fine, but means that such a package needs to be GPL-ed and 
separate from the rest.

>   c) An "expert" package which wraps the most advanced libraries for
>      each field, i.e. LAPACK. Here is should be possible to install
>      each wrapper separately.
>
> If would be nice if the subset a) of b) would use an identical
> interface as a), similar for c) <- b).
>
> Moreover I would see it advantageous to have three independent
> packages here, so I could tell a sysadmin: "Please install
> SciencePython_simple" or "Please install SciencePython_wrappers"
> instead of "Please install SciencePython with options a, b, c".
> Somehow these sys-mins feel that is easier.

Agreed they should be independent and I dont see why this could not be 
the case.

I would add another requirement to such packages: good documentation.

> Anyway, I appreciate all of yours work and I do understand that first
> of all you have to solve your own problems. However, if this scattered
> community manages to unify and simplify (lower entrance barriers)
> their approach to Scientific calculations with Python, that will allow
> it to grow much further than ever possible with the current situation,
> that somehow limits itself to double experts: scientists with *good*
> computer science knowledge or vice versa.
>
> Time will tell, but I wish you all the luck you need! For me new
> projects will come up constantly, maybe even a larger one will be in
> Python again sometime.

I can only agree with all what you wrote. My experience with Python has 
been the other way around: I have been leaving C/C++ behind for Python. 
But at the same time I have found myself limiting my code to a small 
sub-set of all the good stuff that is out there (basically I only use 
Numarray). I would like to see that change and what you outlined above 
seems to me the best way to go forward.

Peter




From verveer at embl.de  Sat Feb  5 08:28:48 2005
From: verveer at embl.de (Peter Verveer)
Date: Sat Feb  5 08:28:48 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42042318.2090704@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu>
Message-ID: <c533ee29143e3286b075927540f5844b@embl.de>

> "What scipy is" is quite open to interpretation.  The idea of scipy is 
> just to have a super-package where different packages can live and 
> have some kind of inter-operability.   It is made so that sub-packages 
> can be easily added and built upon request.   We try hard to make sure 
> that the only real dependency is scipy_base which should not be hard 
> to install anywhere (does not require ATLAS or FORTRAN).
> It would be great to add nd_image as a sub-package to SciPy.    In 
> fact, nd_image was one of the things that most alarmed me about the 
> emerging split in the community, as it reproduces some of what is 
> already in SciPy (N-D convolution)  while adding some great things 
> that I've always wanted in SciPy (morphology).

What I find unfortunate is that due to the differences between the 
packages, nd_image cannot be compiled for both Numeric and Numarray at 
the moment. I did not forsee that the split between the packages would 
exist for so long. I do however not agree that it is an example of 
duplication of work. The N-D convolutions are based on a 
filter-framework that is an essential part of nd_image, which would 
have to be implemented anyway (e.g. the morphology operators are also 
based on it.) So one should be too quick to judge if something is 
duplication and a wast of resources.

> I'd like to see more of the development work that goes on happen under 
> a common standard, that is all.  SciPy has done a lot of work to 
> provide an infrastructure within which to distribute packages.  The 
> field is built, but people don't seem to want to come.

A common standard is a very good idea. But right now I don't find SciPy 
attractive as a framework, because 1) it is too big and not easily 
installed. 2) it is not very well documented. Thus, I prefer to write 
my package against a smaller code-base, in this case Numarray, but it 
could have also been Numeric. That has the advantage that people can 
install it more easily, while it still can be included in things like 
SciPy if desired.

>  I understand that the Numeric / Numarray dichotomy is a big source of 
> this problem.  That is the whole reason I'm devoting a big portion of 
> my time to the problem right now.

I can only agree with that. Regardless if I want to use SciPy or not, 
or in whatever form, I would like to see this problem go away, so that 
my software can become available for everybody.

> My very ambitious goal with Numeric3 is to replace both Numeric and 
> Numarray (heavily borrowing code from each).  When I say replace them, 
> I'm talking about the array object and umath module.  I'm not talking 
> about the other packages that use them.

I have to admit that I was very sceptical when I read your announcement 
for Numeric3, since I thought it would not be a good idea to have yet 
another array package. However, now it is clearer to me what you mean 
to do, and I think this is in fact exactly how I would like to see it 
happen. To me it would be perfect if there is some array package that 
forms a minimal basis to build on. I would program the nd_image package 
to that and offer the result for inclusion in both Numarray and SciPy, 
and allow it to be installed standalone. But it seems to me that there 
is a danger for the "yet another package" effect to occur. I think I 
will remain sceptical unless you achieve three things: 1) It has the 
most important improvements that numarray  has. 2) It has a good API 
that can be used to write packages that work with Numeric3/SciPy and 
Numarray (the latter probably will not go away). 3) Inclusion in the 
main Python tree, so that it is guaranteed to be available.

> I want those to live under a common standard (i.e. scipy).   It would 
> be fine with me if that common standard include a scipy_lite or 
> something like that which was easy to install, if that is desired.

Jochem K?pper just outlined very well how it could look like: A small 
core, plus a common project with packages at different levels. I think 
it is a very good idea, and probably similar to what SciPy is trying to 
do now. But he suggests an explicit division between independent 
packages: basic packages, packages with external library dependencies 
like FFTW, and advanced packages. Maybe something like that should be 
set up if we get an arraybobject into the Python core.

BTW it was mentioned before that it would be a problem to remove 
packages like LinearAlgebra and FFT from the core Numeric. matplotlib 
was mentioned as an example of a package that depends on them. I think 
that points however to a fundamental problem with matplotlib: why does 
a plotting library need FFTs and linear algebra? So I don't think 
anybody can really argue that things like an FFT should be in a core 
array package.

> But, if the problem is really just availability of binary copies, then 
> why don't we solve that instead of worrying about installation on 
> platforms that nobody uses.   In addition, even though I've developed 
> a lot of scipy, I'm willing to throw it all out or alter it 
> considerably in order to satisfy the demands that exist for package 
> developers to want to put their weight behind something like SciPy.   
> So, don't equate history with an unwillingness to cooperate.

That sounds great. If you go the route of a separate Numeric3 that goes 
into the python core and keep a common (simple) API at both Python and 
C level, then I will support that by rewriting nd_image against it. The 
result should be able to replace the current nd_image package in 
numarray, and hopefully be included in SciPy.

> While I do recognize the value of repeated work.  When it comes to 
> interoperability, single standards seem to work best.  So, the longer 
> this unnecessary division of resources goes on, the farther behind 
> Python will get for doing anything but niche work.

Agreed about the single standard thing. But I am not willing to just 
'join' the SciPy project to achieve it (at least for nd_image). I am 
however very interested in the possibility of writing against a small 
high-quality array package that is included in the pyhton core. That 
would be all the standard I need. If you manage to make SciPy into a 
useful larger standard on top of that, great, more power to all of us!

Peter




From rays at san.rr.com  Sat Feb  5 08:28:49 2005
From: rays at san.rr.com (RJ)
Date: Sat Feb  5 08:28:49 2005
Subject: The need for a unified numerical array object (was:
  [Numpy-discussion] Numeric3)
In-Reply-To: <9ezmyjursi.fsf_-_@gowron.rz-berlin.mpg.de>
References: <4203F7CE.8070403@ee.byu.edu>
 <420280AD.8080309@ee.byu.edu>
 <4202F6A0.3@ims.u-tokyo.ac.jp>
 <1f7baacd417e55debdde0351e463cf10@laposte.net>
 <4203CAD0.6050609@ee.byu.edu>
 <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
 <4203F7CE.8070403@ee.byu.edu>
Message-ID: <5.2.1.1.2.20050205080759.026eb700@pop-server.san.rr.com>

At 01:15 PM 2/5/2005 +0100, Jochen K?pper wrote:
>Travis Oliphant <oliphant at ee.byu.edu> writes:
>
>Moreover I would see it advantageous to have three independent
>packages here, so I could tell a sysadmin: "Please install
>SciencePython_simple" or "Please install SciencePython_wrappers"
>instead of "Please install SciencePython with options a, b, c".
>Somehow these sys-mins feel that is easier.

I've used Python for a couple of years now, but spent 5 years before in Perl (and not interested in going back); but the one thing I miss from Perl is CPAN. When I needed a module, I typed CPAN <module> and the proper version magically installed, with dependances. And there are a lot of modules, all available in one place.
I would also like to see more of the PHP-style user-contribution online docs. I also maintain some PHP database code and find it very time-saving to have all the comments and examples right there, and it allows users to take some of the burden of docs off of the developers. They also fill in some of the knowledge assumptions the language developers skim past, which might not be so obvious to novices.

I find that there is a lot of Python resources available for most any task, including the numpy family, but finding and installing the right thing for the local install and task at hand does take some searching and reading.

No complaints! just my experience to date.
Thanks to all,
Ray Schumacher
http://rjs.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050205/e34f22b7/attachment.html>

From jdhunter at ace.bsd.uchicago.edu  Sat Feb  5 09:11:21 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Sat Feb  5 09:11:21 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <c533ee29143e3286b075927540f5844b@embl.de> (Peter Verveer's
 message of "Sat, 5 Feb 2005 17:28:09 +0100")
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	<1f7baacd417e55debdde0351e463cf10@laposte.net>
	<4203CAD0.6050609@ee.byu.edu>
	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>
	<42042318.2090704@ee.byu.edu>
	<c533ee29143e3286b075927540f5844b@embl.de>
Message-ID: <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Peter" == Peter Verveer <verveer at embl.de> writes:

    Peter> BTW it was mentioned before that it would be a problem to
    Peter> remove packages like LinearAlgebra and FFT from the core
    Peter> Numeric. matplotlib was mentioned as an example of a
    Peter> package that depends on them. I think that points however
    Peter> to a fundamental problem with matplotlib: why does a
    Peter> plotting library need FFTs and linear algebra? So I don't
    Peter> think anybody can really argue that things like an FFT
    Peter> should be in a core array package.

The dependence on fft arises from the original mission of matplotlib,
which was to provide a matlab compatible python plotting.  matlab has
functions like psd, csd and cohere, which in addition to computing the
spectral density and coherence functions, generate the plots.  So I
implemented Welch's Average Periodogram on top of fft to compute
spectral densities for these plotting commands.  

I use Matrix a lot for matrix multiplications to do text rotations and
layout.  If Matrix disappeared, it can be easily replace by wrapping
dot -- Todd did that already for the numarray equivalent in the
numerix module.

The matplotlib pylab module aspires to provide a matlab like
environment -- the focus is on plotting of course -- but as I've
needed things for my own work I've added commands to the
matplotlib.mlab module to provide matlab commands in python that do
require linear algebra, eg polyfit and polyval.  Basically
matplotlib.mlab and matplotlib.pylab extend Numeric's MLab.  This
section of the code could of course be folded into scipy.mlab,
Numeric.mlab, or wherever appropriate, but in my view it is already in
a pretty sensible place, in a module that attempts to provide a matlab
environment for python users.

If linear algebra, fft and random numbers disappeared from Numeric,
matplotlib would survive.  As you say, they don't go to the core of
plotting.  I could raise an exception if the user called psd and
didn't have scipy installed.  And everyone else who has written code
based on the functionality of Numeric's extra modules could rewrite
their code as well.  It would mean more work for me and less
functionality for some matplotlib users, but would be manageable.

We've put a fair amount of work in making Numeric and numarray work
transparently in matplotlib at the python and extension code level.
Having a third array package in the wild which is incompatible with
either is an unattractive option for me, but one which I can deal with
if need be.  And if the goal is to get an array package into the
standard library that has the core functionality of Numeric and
numarray, and Numeric3 is the most sensible means to that end, then I
am all for it.

JDH



From verveer at embl.de  Sat Feb  5 09:43:52 2005
From: verveer at embl.de (Peter Verveer)
Date: Sat Feb  5 09:43:52 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu>
Message-ID: <ce075fa4fe288fb3129ab3c4122c94b2@embl.de>

I did not realize fully what matplotlib intents to do, so it seems I 
picked a bad example, my apologies for that...

But I think your answers point out quite well that it may be worth to 
go this route. Sure, people will need to adapt their packages, but it 
will be worth it for all of us in the long run.

I do agree with your remarks about possible problems with having a 
third incompatible package. But I think the goal would be to get that 
package in the Python core, and to make actually sure that it is not 
incompatible with numarray and scipy.  I suppose when it does go into 
the core, everybody will have to be compatible to some level.

Peter

On Feb 5, 2005, at 6:00 PM, John Hunter wrote:

>>>>>> "Peter" == Peter Verveer <verveer at embl.de> writes:
>
>     Peter> BTW it was mentioned before that it would be a problem to
>     Peter> remove packages like LinearAlgebra and FFT from the core
>     Peter> Numeric. matplotlib was mentioned as an example of a
>     Peter> package that depends on them. I think that points however
>     Peter> to a fundamental problem with matplotlib: why does a
>     Peter> plotting library need FFTs and linear algebra? So I don't
>     Peter> think anybody can really argue that things like an FFT
>     Peter> should be in a core array package.
>
> The dependence on fft arises from the original mission of matplotlib,
> which was to provide a matlab compatible python plotting.  matlab has
> functions like psd, csd and cohere, which in addition to computing the
> spectral density and coherence functions, generate the plots.  So I
> implemented Welch's Average Periodogram on top of fft to compute
> spectral densities for these plotting commands.
>
> I use Matrix a lot for matrix multiplications to do text rotations and
> layout.  If Matrix disappeared, it can be easily replace by wrapping
> dot -- Todd did that already for the numarray equivalent in the
> numerix module.
>
> The matplotlib pylab module aspires to provide a matlab like
> environment -- the focus is on plotting of course -- but as I've
> needed things for my own work I've added commands to the
> matplotlib.mlab module to provide matlab commands in python that do
> require linear algebra, eg polyfit and polyval.  Basically
> matplotlib.mlab and matplotlib.pylab extend Numeric's MLab.  This
> section of the code could of course be folded into scipy.mlab,
> Numeric.mlab, or wherever appropriate, but in my view it is already in
> a pretty sensible place, in a module that attempts to provide a matlab
> environment for python users.
>
> If linear algebra, fft and random numbers disappeared from Numeric,
> matplotlib would survive.  As you say, they don't go to the core of
> plotting.  I could raise an exception if the user called psd and
> didn't have scipy installed.  And everyone else who has written code
> based on the functionality of Numeric's extra modules could rewrite
> their code as well.  It would mean more work for me and less
> functionality for some matplotlib users, but would be manageable.
>
> We've put a fair amount of work in making Numeric and numarray work
> transparently in matplotlib at the python and extension code level.
> Having a third array package in the wild which is incompatible with
> either is an unattractive option for me, but one which I can deal with
> if need be.  And if the goal is to get an array package into the
> standard library that has the core functionality of Numeric and
> numarray, and Numeric3 is the most sensible means to that end, then I
> am all for it.
>
> JDH
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion




From strawman at astraw.com  Sat Feb  5 13:30:14 2005
From: strawman at astraw.com (Andrew Straw)
Date: Sat Feb  5 13:30:14 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <55c095f595dfec3e16579094faea14f2@laposte.net> <4203FCD8.8070204@noaa.gov> <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net>
Message-ID: <04B12B1A-77BD-11D9-BC02-000D93476BAC@astraw.com>

Just a minor point, but I think the bdist_mpkg extension to distutils 
by Bob Ippolitto as part of py2app actually solves a lot of the 
problems you describe for Mac OS X.  
http://undefined.org/python/#py2app

On Feb 5, 2005, at 2:09 AM, konrad.hinsen at laposte.net wrote:

> On 04.02.2005, at 23:53, Chris Barker wrote:
>
>> Actually, I disagree. Apple's Python is a better choice (you can't 
>> run Aqua applications from a fink python, at least I don't think so), 
>> but we do need to do more to provide easy to install packages for it. 
>> (there are a lot)
>
> I don't care for Aqua as much as I care for all those nice Python 
> packages I use every day. With Apple's Python, it's command line 
> installation (which I don't mind, but many of my users do), or 
> PackageManager for a select few packages. And I can't my own without 
> proving my own Web server as well, so package contribution is 
> effectively discouraged.
>
>>  there aren't all that many people using OS-X developing Python 
>> packages, and Apple has, of course, made OS-X just different enough 
>> from other unices to require some work (or take the easy way and use 
>> fink, but then it
>
> Not really, you can do command-line installation ("python 
> setup.py...") just like on any other Unix, usually without any 
> modification. The problem is that Mac users expect installers.
>
> It should be possible to extend distutils for making Mac installers 
> just like it can make RPMs and Windows installers. I suppose nobody 
> looked into this because the Apple Python community is divided into 
> those who come from the Unix world and go for Fink, and those who come 
> from pre-10 MacOS and go for MacPython plus PackageManager.




From oliphant at ee.byu.edu  Sat Feb  5 16:00:09 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb  5 16:00:09 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <c533ee29143e3286b075927540f5844b@embl.de>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de>
Message-ID: <42055DAF.5020601@ee.byu.edu>

Peter Verveer wrote:

>
> What I find unfortunate is that due to the differences between the 
> packages, nd_image cannot be compiled for both Numeric and Numarray at 
> the moment. I did not forsee that the split between the packages would 
> exist for so long. I do however not agree that it is an example of 
> duplication of work. The N-D convolutions are based on a 
> filter-framework that is an essential part of nd_image, which would 
> have to be implemented anyway (e.g. the morphology operators are also 
> based on it.) So one should be too quick to judge if something is 
> duplication and a wast of resources.


This is exactly why I am dissatisfied with the way numarray has been 
advertised (I'm not blaming anyone here, I recognize the part that I 
played in it).  Third-party package providers starting to build on top 
of numarray before it was a clear replacement.   I don't see anything in 
nd_image that could not have sat on top of Numeric from the 
beginning.    I don't see why it needed to use numarray specific things 
at all.   By the way, nd_image it is a very nice piece of work that I 
have been admiring.  

> common standard is a very good idea. But right now I don't find SciPy 
> attractive as a framework, because 1) it is too big and not easily 
> installed. 2) it is not very well documented. Thus, I prefer to write 
> my package against a smaller code-base, in this case Numarray, but it 
> could have also been Numeric. That has the advantage that people can 
> install it more easily, while it still can be included in things like 
> SciPy if desired.

What is too big about it?  Which packages would you like to see not 
included?   Is it the dependence on Fortran that frightens, or the 
dependence on ATLAS (not a requirement by the way)?   How many realize 
that you can turn off installation of packages in the setup.py file by a 
simple switch?

The point is that nd_image should live perfectly well as a scipy package 
and be installed and distributed separately if that is desired as 
well.   But, it would present a uniform face to others looking to do 
scientific work with Python.  This appearance of multiple packages has 
only amplified the problem that scipy was trying to solve.  If scipy is 
solving it badly, then lets fix it.     I do appreciate the comments 
that are helping me understand at least what needs to be fixed.  But, 
everybody has always been welcome to come and help fix things --- it is 
quite easy to get CVS write access to the scipy core.

Matplotlib as well is reinventing stuff already available elsewhere.  
SciPy has always tried to bring a matlab-like environment to the user.  
That's always been my goal.  It is frustrating to see so many with the 
same goal work so independently of each other.    Sometimes I guess it 
is necessary so that new ideas can be tried out.   But, it sure seems to 
happen more often than I would like with Python and Science / Engineering.

Is it ownership that is a concern?  None of us involved with scipy have 
an alterior motive for trying to bring packages together except the 
creation of a single standard and infrastructure for scientific 
computing with Python.    I keep hearing that people like the idea of a 
small subset of packages, but that is exactly what scipy has always been 
trying to be.   Perhaps those involved with scipy have not voiced our 
intentions enough or loud enough. 

> I can only agree with that. Regardless if I want to use SciPy or not, 
> or in whatever form, I would like to see this problem go away, so that 
> my software can become available for everybody.

I'm glad to hear this. 

>> My very ambitious goal with Numeric3 is to replace both Numeric and 
>> Numarray (heavily borrowing code from each).  When I say replace 
>> them, I'm talking about the array object and umath module.  I'm not 
>> talking about the other packages that use them.
>
>
> I have to admit that I was very sceptical when I read your 
> announcement for Numeric3, since I thought it would not be a good idea 
> to have yet another array package. 

I've never seen it as yet another array package.  I've always seen it as 
a Numeric upgrade with the hope of getting closer to numarray (and even 
replace it if possible).    In fact, since I'm currently the maintainer 
of Numeric, those who use Numeric should perhaps be the most concerned 
because Numeric3 will replace Numeric (and will therefore try not to 
break old code too much --- though there may be a few changes).   I can 
guarantee that SciPy will work with Numeric3.   We've always told scipy 
users that if they use SciPy they won't have to worry about the 
arrayobject issue because scipy will shield them from it (at least on 
the Python level).

    Of course Numeric will still be there and if someone else wants to 
maintain it, they are welcome to.  I'm just serving notice that any 
resources I have will be devoted to Numeric3, and as soon as Numeric3 is 
out of alpha I would actually like to call it Numeric (verson 30.0).

> But it seems to me that there is a danger for the "yet another 
> package" effect to occur. I think I will remain sceptical unless you 
> achieve three things: 1) It has the most important improvements that 
> numarray  has. 2) It has a good API that can be used to write packages 
> that work with Numeric3/SciPy and Numarray (the latter probably will 
> not go away). 3) Inclusion in the main Python tree, so that it is 
> guaranteed to be available.

Thanks for your advice.  All encouragement is appreciated. 

1)  The design document is posted.  Please let me know what "the most 
important improvements" are.

2)  It will have an improved API and I would also like to support a lot 
of the numarray API as well (although I I don't understand the need for 
many of the API calls numarray allows and see much culling that needs to 
be done --- much input is appreciated here as to which are the most 
important API calls.  I will probably use nd_image as an example of what 
to support).

3) Inclusion in the main Python tree will rely on Guido feeling like 
there is a consensus.  I'm hopeful here and will try to pursue this -- 
but I need to finish code first.

> Jochem K?pper just outlined very well how it could look like: A small 
> core, plus a common project with packages at different levels. I think 
> it is a very good idea, and probably similar to what SciPy is trying 
> to do now. But he suggests an explicit division between independent 
> packages: basic packages, packages with external library dependencies 
> like FFTW, and advanced packages. Maybe something like that should be 
> set up if we get an arraybobject into the Python core.

Sounds great.  SciPy has been trying to do exactly this, but we need 
ideas --- especially from package developers who understand the issues 
--- as to how to set it up correctly.  We've already re-factored a 
couple of times.  We could do it again if we needed to, so that the 
infrastructure had the right feel.   A lot of this is already in place.  
I don't think many recognize some of the work that has already been 
done.  This is not an easy task. 

>
> BTW it was mentioned before that it would be a problem to remove 
> packages like LinearAlgebra and FFT from the core Numeric. matplotlib 
> was mentioned as an example of a package that depends on them. I think 
> that points however to a fundamental problem with matplotlib: why does 
> a plotting library need FFTs and linear algebra? So I don't think 
> anybody can really argue that things like an FFT should be in a core 
> array package.

My point exactly.  Having these things in Numeric/ Numarray actually 
encourages the creation of multiple separate packages as everybody tries 
to add just there little extra bit on top.

Plotting is potentially problematic because there are a lot of ways to 
plot.  I think we need to define interfaces in this regard and adapters 
so that commands that would throw up a plot could use several different 
plotting methods to do it.   I'm not favoring any plotting technique, so 
don't pre-guess me.  My ideas of plotting are probably very similiar to 
John's with matplotlib.  His work is another that I'm disappointed is 
not part of scipy and has led me to my current craziness with Numeric3 :-)

>
> Agreed about the single standard thing. But I am not willing to just 
> 'join' the SciPy project to achieve it (at least for nd_image). I am 
> however very interested in the possibility of writing against a small 
> high-quality array package that is included in the pyhton core. That 
> would be all the standard I need. If you manage to make SciPy into a 
> useful larger standard on top of that, great, more power to all of us!

Why not?  Your goals are not at odds with ours.   O.K. it may be that 
more work is required to re-write nd_image against the Numeric C-API 
than you'd like --- that's why Numeric3 is going to try and support the 
numarray C-API as well.

Thanks for your valuable feedback.   I appreciate the time it took you 
to provide it.   I hope we can work more together in the future.

Best regards,

-Travis





From oliphant at ee.byu.edu  Sat Feb  5 16:08:08 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb  5 16:08:08 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	<1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu>	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu>	<c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu>
Message-ID: <42055FA3.9050909@ee.byu.edu>

>The dependence on fft arises from the original mission of matplotlib,
>which was to provide a matlab compatible python plotting.  matlab has
>functions like psd, csd and cohere, which in addition to computing the
>spectral density and coherence functions, generate the plots.  So I
>implemented Welch's Average Periodogram on top of fft to compute
>spectral densities for these plotting commands.  
>  
>
Matlab-like environment is very much what scipy has always tried to be. 

>The matplotlib pylab module aspires to provide a matlab like
>environment -- the focus is on plotting of course -- but as I've
>needed things for my own work I've added commands to the
>matplotlib.mlab module to provide matlab commands in python that do
>require linear algebra, eg polyfit and polyval.  Basically
>matplotlib.mlab and matplotlib.pylab extend Numeric's MLab.  This
>section of the code could of course be folded into scipy.mlab,
>Numeric.mlab, or wherever appropriate, but in my view it is already in
>a pretty sensible place, in a module that attempts to provide a matlab
>environment for python users.
>  
>
But MATLAB is a huge package.  SciPy has been trying to do the same 
thing.  Plotting was needed and matplotlib did a great thing in 
providing excellent plotting.  I've never understood the reason for 
matplotlib to remain separate except as evidence of the great schism 
that was occuring in the community.

In fact, when Eric and I began scipy, I wanted to call it pylab (you'll 
notice I have the pylab.sourceforge.net site).  I have fond ties to 
MATLAB myself.    Eric chose the scipy name which is probably better.  
So, it is very concerning to me to see matplotlib try to become what 
scipy is already trying to become.   I think it is very confusing to 
newcomers.

>We've put a fair amount of work in making Numeric and numarray work
>transparently in matplotlib at the python and extension code level.
>Having a third array package in the wild which is incompatible with
>either is an unattractive option for me, but one which I can deal with
>if need be.  And if the goal is to get an array package into the
>standard library that has the core functionality of Numeric and
>numarray, and Numeric3 is the most sensible means to that end, then I
>am all for it.
>  
>
Again,  from my perspective once Numeric3 is working reasonably well, 
Numeric ceases to exist for me.  Yes others may continue to download it, 
but I won't be maintaining it anymore.

-Travis




From perry at stsci.edu  Sat Feb  5 17:49:10 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Sat Feb  5 17:49:10 2005
Subject: [Numpy-discussion] scipy, matplotlib, and installation dependencies.(previously:  Numeric3)
In-Reply-To: <42055FA3.9050909@ee.byu.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCGEAFFJAA.perry@stsci.edu>

Travis Oliphant wrote:
> >
> But MATLAB is a huge package.  SciPy has been trying to do the same 
> thing.  Plotting was needed and matplotlib did a great thing in 
> providing excellent plotting.  I've never understood the reason for 
> matplotlib to remain separate except as evidence of the great schism 
> that was occuring in the community.
> 
> In fact, when Eric and I began scipy, I wanted to call it pylab (you'll 
> notice I have the pylab.sourceforge.net site).  I have fond ties to 
> MATLAB myself.    Eric chose the scipy name which is probably better.  
> So, it is very concerning to me to see matplotlib try to become what 
> scipy is already trying to become.   I think it is very confusing to 
> newcomers.
> 
I think you are reading far too much into the overlap between
matplotlib and scipy. Yes, matplotlib duplicates some numerical
functionality, but the plotting and image display aspects of 
matplotlib are far and away the primary focus. Comparitively
little of it is devoted to numerical libraries. Likewise, 
although scipy has had plotting capability, we all recognize 
(at least I think we do) that it left a lot to be desired and 
was considered by most a stopgap until a better plotting solution
was found.

So this leads to the question: why wasn't matplotlib developed 
for scipy? You'll have to ask John for a definitive answer, but I 
think I can guess. As you may have noticed, many are fearful of 
building lots of dependencies into their library or package. That's
understandable. The more there are, the greater the chances that
installation will be too painful. Scipy has many such dependencies.
Until it has a reputation of being an easy install, it's unlikely 
lots of packages with a much narrower focus will depend on it.
Hence all the requests for something along the scale of what Numeric
packages rather than just all of scipy.

There are two competing forces at work here. One is to bundle
everything together and do the work of making sure it all plays
together nicely thus avoiding the pain of installing N different
packages independently (what scipy is trying to do). That's great
as far as it works easily, but with so many dependencies, more 
risk there is of one of them causing unsophiscated users (and 
even the sophisticated) stumbling on installation. The other 
force is for minimal dependencies. It makes for less headaches
for the package maintainer, but for users that have to install
several components, it can be more of a hassle.

Can scipy be made to be a painless install for most users? I'm 
not sure we know the answer yet. Until we do, I'd favor an approach
that shoots for a smaller, easily installable core, with a simple
way of asking for optional packages (having users edit setup.py
is probably asking too much). When packages prove that they are
reliably installed under all circumstances, then they can be added
to the core distribution, otherwise, it is understood that the 
optional packages may be tricker to install. The fewer you need,
the better. It seems to me that is what many would like to see,
but maybe I'm misrepresenting the most common point of view.

Perry



> >We've put a fair amount of work in making Numeric and numarray work
> >transparently in matplotlib at the python and extension code level.
> >Having a third array package in the wild which is incompatible with
> >either is an unattractive option for me, but one which I can deal with
> >if need be.  And if the goal is to get an array package into the
> >standard library that has the core functionality of Numeric and
> >numarray, and Numeric3 is the most sensible means to that end, then I
> >am all for it.
> >  
> >
> Again,  from my perspective once Numeric3 is working reasonably well, 
> Numeric ceases to exist for me.  Yes others may continue to download it, 
> but I won't be maintaining it anymore.
> 
> -Travis
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 




From southey at uiuc.edu  Sat Feb  5 19:28:10 2005
From: southey at uiuc.edu (Bruce Southey)
Date: Sat Feb  5 19:28:10 2005
Subject: [Numpy-discussion] Numeric3
Message-ID: <4760baf2.ca2b9b41.81db100@expms6.cites.uiuc.edu>

Hi,
I am not the developer of any of Numpy (in any form) so take my thoughts 'with a
grain of salt' because I am not doing the coding. I am really very thankful that
people have developed Numeric and numarray for Python!

Many of the comments I fully agree with especially on installation. Large
packages, like SciPy, require time and effort to ensure every thing works. Also,
various platforms lack compilers (i.e. MS Windows are not by default and free
Fortran 90 compilers for Linux). A user without 'system admin' experience or
access can not be expected get these compilers installed especially if the
dependencies don't compile or are incorrectly compiled.

With all the arguments that have been presented so far, I fail to see any need
for at least a third Python numerical option, especially one that may not
support many needs. This discussion shows that there doesn't seem to be the
'maturity' of the code base to justify a push towards the Python Standard
Library. By maturity, I mean that there are still too many differences between
Numeric and numarray that, while appear to be solvable, are not trivial to
address at the present time. (This is also one of the main reasons why many
different versions of the same thing exist.) 

I would like to see a single NumPy option that everyone can use. If numarray or
Numeric3 does that then great as I will support either. However, I do fall in
the numarray camp as I do tend to work with large matrices (plus features like
Masked Arrays are a big plus).

Regards
Bruce Southey



From oliphant at ee.byu.edu  Sat Feb  5 23:24:16 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb  5 23:24:16 2005
Subject: [Numpy-discussion] Proposal for making of Numarray a real Numeric
 'NG'
In-Reply-To: <200501211413.51663.faltet@carabos.com>
References: <200501211413.51663.faltet@carabos.com>
Message-ID: <4205C5DE.80504@ee.byu.edu>

Francesc Altet wrote:

>Hi List,
>
>I would like to make a formal proposal regarding with the subject of
>previous discussions in that list. This message is a bit long, but I've
>tried my best to expose my thoughts as clearly as possible.
>  
>
I did not have time to respond to this mail, but it is very good.  I  
will be placing some of its comments in the scipy site.

>It's worth remembering that Numeric has been a major breakthrough in
>introducing the capability to deal with large (homogeneous) datasets in
>Python in a very efficient mannner. In my opinion Numarray is, generally
>speaking, a very good package as well with many interesting new features
>that lack Numeric. Between the main advantages of Numarray vs Numeric I can
>list the next (although I can be a bit misleaded here because of my own user
>cases of both libraries):
>  
>
I think numarray has made some incrdedible strides in showing what the 
numeric object needs to be and in implementing some really neat 
functionality.   I just think its combination of Python and C code must 
be redone to overcome the speed issues that have arisen.  My opinion 
after perusing the numarray code is that it would be easier (for me 
anyway) to adjust Numeric to support the features of numarray, than to 
re-write and re-organize the relevant sections of numarray code.  One of 
the advantages of Numeric is it's tight implementation that added only 
two fundamental types, both written entirely in C.   I  was hoping that 
the Python dependencies for the fundamental types would fade as numarray 
matured, but it appears to me that this is not going to happen.

I did not have the time in the past to deal with this.  I wish I had 
looked at it more closely two years ago.   If I had done this I would 
have seen how to support the features that Perry wanted without 
completely re-writing everything.   But, then again, Python 2.2 changed 
what is possible on the C level and that has had an impact on the 
discussion.   

>- Memory-mapped objects: Allow working with on-disk numarray objects like if
>  they were in-memory.
>  
>
Numeric3 supports this cleanly and old Numeric did too (there was a 
memory-mapped module), it's just that byteswapping, and alignment had to 
be done manually.

>  
>- RecArrays: Objects that allow to deal with heterogeneous datasets
>  (tables) in an efficient manner. This ought to be very beneficial in many
>  fields.
>  
>
Heterogeneous arrays is the big one for old Numeric.  It is a good 
idea.  In Numeric3 it has required far fewer changes than I had at first 
imagined.

>  
>- CharArrays: Allow to work with large amounts of fixed and variable length
>  strings. I see this implementation much more powerful that Numeric.
>  
>
Also a good idea, and comees along for the ride with in Numeric3.   
Numeric had CHAR arrays but a vision was never specified  for how to 
make them more useful.  This change would have been a good step towards 
heterogeneous arrays.

>  
>- Index arrays within subscripts: e.g. if ind = array([4, 4, 0, 2])
>  and x = 2*arange(6), x[inx] results in array([8, 8, 0, 4])
>
>  
>
For scipy this was implemented on top of Numeric (so it is in Numeric3 
too),  the multidimensional version needs to be worked on, still.

>- New design interface: We should not forget that numarray has been designed
>  from the ground with Python Library integration in mind (or at least, this
>  is my impression). So, it should have more chances (if there is some hope)
>  to enter in the Standard Library than Numeric.
>  
>

Numeric has had this in mind for some time.  In fact the early Numeric 
developers were quite instrumental in getting significant changes into 
Python istelf, including Complex Objects, Ellipses, and Extended 
Slicing.   Guido was quite keen on the idea of including Numeric at one 
point.  Our infighting made him lose interest I think.   So claiming 
this as an advantage of numarray over Numeric is simply inaccurate.

>The real problem for Numarray: Object Creation Time
>===================================================
>
>On the other hand, the main drawback of Numarray vs Numeric is, in my
>opinion, its poor performance regarding object creation. This might look
>like a banal thing at first glance, but it is not in many cases. One example
>recently reported in this list is:
>  
>
Ah, and there's the rub.  I don't think this object creation time will 
go away until Numarray's infrastructure becomes essentially like that of 
Numeric.  One tight object all in C.   Getting it there seems harder 
than fixing Numeric, with the additional features of Numarray.

Thanks for these comments.   It is very good to hear what the most 
important features for users are.

-Travis





From verveer at embl.de  Sun Feb  6 05:28:45 2005
From: verveer at embl.de (Peter Verveer)
Date: Sun Feb  6 05:28:45 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42055DAF.5020601@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <42055DAF.5020601@ee.byu.edu>
Message-ID: <12fab106c08dbd04c949d9b7c5f96e3c@embl.de>

> This is exactly why I am dissatisfied with the way numarray has been 
> advertised (I'm not blaming anyone here, I recognize the part that I 
> played in it).  Third-party package providers starting to build on top 
> of numarray before it was a clear replacement.   I don't see anything 
> in nd_image that could not have sat on top of Numeric from the 
> beginning.    I don't see why it needed to use numarray specific 
> things at all.   By the way, nd_image it is a very nice piece of work 
> that I have been admiring.

Thanks. nd_image could sit easily on top of Numeric, it sits on top of 
numarray simply because I felt numarray was the better package. And 
this is still the case. There is however nothing inherent in nd_image 
that requires numarray. In fact, I am contemplating factoring out the 
image processing functionality into a separate C library, on which 
python packages then would be based.


>> common standard is a very good idea. But right now I don't find SciPy 
>> attractive as a framework, because 1) it is too big and not easily 
>> installed. 2) it is not very well documented. Thus, I prefer to write 
>> my package against a smaller code-base, in this case Numarray, but it 
>> could have also been Numeric. That has the advantage that people can 
>> install it more easily, while it still can be included in things like 
>> SciPy if desired.
>
> What is too big about it?  Which packages would you like to see not 
> included?

I think I was unfair to call scipy 'big'. There is nothing inherently 
bad about being big. The problem is rather installation, I suppose. 
Also I would like to see modularity, if I only want say, least-squares 
fitting, I should be able to just take that.

> Is it the dependence on Fortran that frightens, or the dependence on 
> ATLAS (not a requirement by the way)?   How many realize that you can 
> turn off installation of packages in the setup.py file by a simple 
> switch?

Fortran or Atlas do not scare me. But if they make it more difficult to 
have some average user to install scipy, just so they can use some of 
my software that depends on it, then there is a problem. Fear of such 
problems keeps me away from scipy. I can point somebody to numarray and 
say type 'python setup.py install' and be sure it works. I know, this a 
bit unfair, scipy aims to be more then numarray is now, but it would be 
nice to have functionality that is in scipy available with the same 
ease.

> The point is that nd_image should live perfectly well as a scipy 
> package and be installed and distributed separately if that is desired 
> as well.

It should live perfectly well as a scipy package because it should be 
dependent only on a basic array package. Then scipy would be welcome to 
include it. That is now not possible because of the split between 
numarray or numeric. If scipy and numarray would share the basic array 
package, nd_image would work perfectly fine with both.

> Is it ownership that is a concern?  None of us involved with scipy 
> have an alterior motive for trying to bring packages together except 
> the creation of a single standard and infrastructure for scientific 
> computing with Python.    I keep hearing that people like the idea of 
> a small subset of packages, but that is exactly what scipy has always 
> been trying to be.   Perhaps those involved with scipy have not voiced 
> our intentions enough or loud enough.

Maybe this is the problem. My impression of scipy has been that it 
tries to be a matlab-type of enviroment, i.e., including all in a 
single install. Nothing wrong with that, but also not what I really 
need, certainly not if it is difficult to install. I just need to be 
able get the packages that I need to write my programs in python, just 
like you would just link a library if you would do it in C...

So, although I do applaud the scipy effort to provide a complete 
enviroment, I would prefer to see it based on a set of packages that 
are as independent as possible, so that I can just stick to those if I 
want. I think you have been saying that SciPy has been designed with 
that goal in mind, so maybe I should investigate how easy it is to take 
out things I  need.

>> But it seems to me that there is a danger for the "yet another 
>> package" effect to occur. I think I will remain sceptical unless you 
>> achieve three things: 1) It has the most important improvements that 
>> numarray  has. 2) It has a good API that can be used to write 
>> packages that work with Numeric3/SciPy and Numarray (the latter 
>> probably will not go away). 3) Inclusion in the main Python tree, so 
>> that it is guaranteed to be available.
>
> Thanks for your advice.  All encouragement is appreciated.
> 1)  The design document is posted.  Please let me know what "the most 
> important improvements" are.

I will have a look and give my comments. On first reading it seems to 
fix all that I did not like in Numeric.

>
> 2)  It will have an improved API and I would also like to support a 
> lot of the numarray API as well (although I I don't understand the 
> need for many of the API calls numarray allows and see much culling 
> that needs to be done --- much input is appreciated here as to which 
> are the most important API calls.  I will probably use nd_image as an 
> example of what to support).

I think if Numeric3 is intended to be a basic package that might go 
into the Pyhton core, that its API should be as small as possible. Dont 
forget that numarray already includes a Numeric API, so writing 
packages that compile on both should be feasible. I personally prefer 
to switch to whatever the API in the core may be, over seeing multiple 
API's in such a core package. Having a set of multiple APIs will not 
help getting it accepted in the core, it should be kept as simple as 
possible.

>> Jochem K?pper just outlined very well how it could look like: A small 
>> core, plus a common project with packages at different levels. I 
>> think it is a very good idea, and probably similar to what SciPy is 
>> trying to do now. But he suggests an explicit division between 
>> independent packages: basic packages, packages with external library 
>> dependencies like FFTW, and advanced packages. Maybe something like 
>> that should be set up if we get an arraybobject into the Python core.
>
> Sounds great.  SciPy has been trying to do exactly this, but we need 
> ideas --- especially from package developers who understand the issues 
> --- as to how to set it up correctly.  We've already re-factored a 
> couple of times.  We could do it again if we needed to, so that the 
> infrastructure had the right feel.   A lot of this is already in 
> place.  I don't think many recognize some of the work that has already 
> been done.  This is not an easy task.

I think a crucial difference is that there would be different packages 
that need to be installed separately, which increasing levels of 
dependency and difficulties. Enviroments such as scipy could be build 
on top of it. At the same time it would make people like me that dont 
want to install enviroments that do everything, happier.
>
> Plotting is potentially problematic because there are a lot of ways to 
> plot.  I think we need to define interfaces in this regard and 
> adapters so that commands that would throw up a plot could use several 
> different plotting methods to do it.   I'm not favoring any plotting 
> technique, so don't pre-guess me.  My ideas of plotting are probably 
> very similiar to John's with matplotlib.  His work is another that I'm 
> disappointed is not part of scipy and has led me to my current 
> craziness with Numeric3 :-)

With things like plotting you get into the realm of user interfaces. In 
my opinion such things should not mix at all with packages that 
implement numerical functionality. Obviously an environment like scipy  
needs to have plotting, but again, if I don't want it, I should not 
have to download and install it. So within a structure that consists of 
different levels of packages, plotting and displaying should probably 
consist within their own set of packages.

>> Agreed about the single standard thing. But I am not willing to just 
>> 'join' the SciPy project to achieve it (at least for nd_image). I am 
>> however very interested in the possibility of writing against a small 
>> high-quality array package that is included in the pyhton core. That 
>> would be all the standard I need. If you manage to make SciPy into a 
>> useful larger standard on top of that, great, more power to all of 
>> us!
>
> Why not?  Your goals are not at odds with ours.   O.K. it may be that 
> more work is required to re-write nd_image against the Numeric C-API 
> than you'd like --- that's why Numeric3 is going to try and support 
> the numarray C-API as well.

Rewriting is not the problem, that can be done fairly easily, as long 
as it can be done such that the result compiles for both numarray and 
numeric. I am not sure if it makes sense to have multiple APIs into 
core package, keep it simple.

> Thanks for your valuable feedback.   I appreciate the time it took you 
> to provide it.   I hope we can work more together in the future.

For now I will watch and see what happens :-) nd_image remains a 
numarray package for now, but if it becomes feasible to make it work 
for both numarray and numeric I will give that a try. If some package 
makes into the python core, then I will support that and remove all 
dependencies that might exist on other packages.

Cheers, Peter




From konrad.hinsen at laposte.net  Sun Feb  6 06:46:25 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sun Feb  6 06:46:25 2005
Subject: The need for a unified numerical array object (was: [Numpy-discussion] Numeric3)
In-Reply-To: <5.2.1.1.2.20050205080759.026eb700@pop-server.san.rr.com>
References: <4203F7CE.8070403@ee.byu.edu> <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu> <4203F7CE.8070403@ee.byu.edu> <5.2.1.1.2.20050205080759.026eb700@pop-server.san.rr.com>
Message-ID: <55c61cbfde4feb91b4d8da721d1878a3@laposte.net>

On 05.02.2005, at 17:28, RJ wrote:

>  I've used Python for a couple of years now, but spent 5 years before 
> in Perl (and not interested in going back); but the one thing I miss 
> from Perl is CPAN. When I needed a module, I typed CPAN <module> and 
> the proper version magically installed, with dependances. And there 
> are a lot of modules, all available in one place.

How does CPAN handle external dependencies? For example, what does CPAN 
do with a GTK interface for Perl when it finds that GTK is not 
available on the machine? Or when the package requires a Fortran 
compiler and none is installed?

Konrad.




From dd55 at cornell.edu  Sun Feb  6 08:17:30 2005
From: dd55 at cornell.edu (Darren Dale)
Date: Sun Feb  6 08:17:30 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4760baf2.ca2b9b41.81db100@expms6.cites.uiuc.edu>
References: <4760baf2.ca2b9b41.81db100@expms6.cites.uiuc.edu>
Message-ID: <200502061116.35095.dd55@cornell.edu>

Hi,

As a relatively new user, I would like to contribute my experience to the 
discussion. Late in a graduate program, I caught the bug and wanted to try 
out python. Being established in Matlab, I was hoping to find a similar 
collection of capabilities for Python, something that I could experiment with 
and start learning fairly quickly. When I discovered Matplotlib, I felt home 
free. Without too much effort, I had a plot on screen and felt 'ok, this is 
feasible.' With the recent effort from Matplotlib and IPython, this should be 
even more true for newcomers.

On the downside, I have been somewhat confused by the state of things 
concerning Numeric and numarray. I think I can trace this confusion back to 
the pfdubios.com/numpy site (linked from numpy.sourceforge.net), which is the 
first hit I get doing a google search for either numpy or numerical python. 
"If you are new to Numerical Python, please use Numarray. The older module, 
Numeric, is unsupported." There is no date associated with the site, but it 
still advertises release 22.0. In the near future, I strongly suggest editing 
websites for misleading information.

So now I have both Numeric and numarray on my system, linked to the full, 
atlas-tuned, lapack and blas libraries. I started out using numarray, 
thinking it was in my long term interest, but I guess this is open to debate. 
I was also hesitant to use scipy, because it relied on Numeric. Being an 
uninformed newbie, I just wasnt able to make an informed decision (I am still 
struggling with it).

A lot of what I have read recently is very encouraging to me. I like the idea 
of bridging the API gap between numarray and Numeric. I like the idea of a 
multidimensional array getting into the core, something the community can 
agree on. I dont care what array package I am using, I just want it to be the 
accepted standard. I would like to hear what the numarray guys have to say 
about Numeric3.

Finally, I like the idea of scipy. I hope the community will come to an 
agreement and work within a framework that will result in the most efficient 
use of everyones time, the most logical interplay between packages, the least 
redundancy, and therefore the most gentle barrier to newcomers who want to 
experiment with scientific computing in Python.

Deepest regards,
Darren



From konrad.hinsen at laposte.net  Sun Feb  6 08:42:21 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sun Feb  6 08:42:21 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42055FA3.9050909@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu>
Message-ID: <ddf1e36c0b0166a093e3643bfb54e678@laposte.net>

On 06.02.2005, at 01:06, Travis Oliphant wrote:

> But MATLAB is a huge package.  SciPy has been trying to do the same  
> thing.  Plotting was needed and matplotlib did a great thing in  
> providing excellent plotting.  I've never understood the reason for  
> matplotlib to remain separate except as evidence of the great schism  
> that was occuring in the community.
>
> In fact, when Eric and I began scipy, I wanted to call it pylab  
> (you'll notice I have the pylab.sourceforge.net site).  I have fond  
> ties to MATLAB myself.

That's actually one of the non-technical issues I have with SciPy.  
SciPy "smells" like Matlab. Which is fine for those who know and like  
Matlab, but not for those (like me) who find Matlab rather incompatible  
with their brains. SciPy, like Matlab, is based on the principle that  
all operations should be expressed in terms of arrays. My personal  
approach is that operations should be expressed in terms of  
problem-specific classes in which the internal use of arrays is an  
implementation detail.

There are arguments for and against both approaches, and I think there  
is space (meme space) for both. I just mention this to point out that I  
don't expect SciPy to become the one and only scientific Python library  
that includes everything. I don't expect to contribute code to SciPy,  
for example. I wouldn't mind using it, of course, in the implementation  
of my classes, if and when the installation issues become less of a  
problem.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From rays at san.rr.com  Sun Feb  6 08:50:24 2005
From: rays at san.rr.com (RJ)
Date: Sun Feb  6 08:50:24 2005
Subject: The need for a unified numerical array object (was:
  [Numpy-discussion] Numeric3)
In-Reply-To: <55c61cbfde4feb91b4d8da721d1878a3@laposte.net>
References: <5.2.1.1.2.20050205080759.026eb700@pop-server.san.rr.com>
 <4203F7CE.8070403@ee.byu.edu>
 <420280AD.8080309@ee.byu.edu>
 <4202F6A0.3@ims.u-tokyo.ac.jp>
 <1f7baacd417e55debdde0351e463cf10@laposte.net>
 <4203CAD0.6050609@ee.byu.edu>
 <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
 <4203F7CE.8070403@ee.byu.edu>
 <5.2.1.1.2.20050205080759.026eb700@pop-server.san.rr.com>
Message-ID: <5.2.1.1.2.20050206081454.033838e0@pop-server.san.rr.com>

At 10:40 AM 2/6/2005 +0100, konrad.hinsen at laposte.net wrote:
>On 05.02.2005, at 17:28, RJ wrote:
>
>How does CPAN handle external dependencies? For example, what does CPAN do with a GTK interface for Perl when it finds that GTK is not available on the machine? Or when the package requires a Fortran compiler and none is installed?

It can fail for truly external requirements; module/bundle authors are largely responsible for declaring dependencies, and of course, they must be via CPAN, such as
http://cpan.org/authors/id/M/ML/MLEHMANN/Gtk-Perl-0.7009.tar.gz
among 7,582 others.

I mainly used ActivePerl's Perl Package Manager but apparently there is now a CPANPLUS.

http://aspn.activestate.com/ASPN/docs/ActivePerl/lib/CPAN.html
is the best resource for the guts of it.
In the above link, see:
The four CPAN::* Classes: Author, Bundle, Module, Distribution
and
POPULATE AN INSTALLATION WITH LOTS OF MODULES

A page search for "depend" points out what I think you're interested in, such as
" Please note, CPAN.pm does not know the dependency tree in advance and cannot sort the queue of things to install in a topologically correct order. It resolves perfectly well IFF all modules declare the prerequisites correctly with the PREREQ_PM attribute to MakeMaker. For bundles which fail and you need to install often, it is recommended sort the Bundle definition file manually. It is planned to improve the metadata situation for dependencies on CPAN in general, but this will still take some time."

At least here I didn't have to read any ($q*=2)+=$f=!fork;
As a second-place "winner" described:
"the winning program was not nearly as confusing as mine---it was actually very simple. If you fixed the indentation, it was quite straightforward. My program is difficult to understand even if you fix it up with good formatting and reasonable variable names. In the solution file I provided, I expanded it in this way, and it still took me about 2500 words to explain what was going on."

Ray Schumacher 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050206/33eda4f9/attachment.html>

From verveer at embl.de  Sun Feb  6 08:56:24 2005
From: verveer at embl.de (Peter Verveer)
Date: Sun Feb  6 08:56:24 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <ddf1e36c0b0166a093e3643bfb54e678@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <ddf1e36c0b0166a093e3643bfb54e678@laposte.net>
Message-ID: <8a8a1ec91042a7507db0129c203d816b@embl.de>

On Feb 6, 2005, at 5:41 PM, konrad.hinsen at laposte.net wrote:

> On 06.02.2005, at 01:06, Travis Oliphant wrote:
>
>> But MATLAB is a huge package.  SciPy has been trying to do the same 
>> thing.  Plotting was needed and matplotlib did a great thing in 
>> providing excellent plotting.  I've never understood the reason for 
>> matplotlib to remain separate except as evidence of the great schism 
>> that was occuring in the community.
>>
>> In fact, when Eric and I began scipy, I wanted to call it pylab 
>> (you'll notice I have the pylab.sourceforge.net site).  I have fond 
>> ties to MATLAB myself.
>
> That's actually one of the non-technical issues I have with SciPy. 
> SciPy "smells" like Matlab. Which is fine for those who know and like 
> Matlab, but not for those (like me) who find Matlab rather 
> incompatible with their brains. SciPy, like Matlab, is based on the 
> principle that all operations should be expressed in terms of arrays. 
> My personal approach is that operations should be expressed in terms 
> of problem-specific classes in which the internal use of arrays is an 
> implementation detail.
>
> There are arguments for and against both approaches, and I think there 
> is space (meme space) for both. I just mention this to point out that 
> I don't expect SciPy to become the one and only scientific Python 
> library that includes everything. I don't expect to contribute code to 
> SciPy, for example. I wouldn't mind using it, of course, in the 
> implementation of my classes, if and when the installation issues 
> become less of a problem.

That more or less sums up my approach too. I tend to program in a mix 
of these two approaches.

I also think there is space for both approaches. But when code goes 
into a matlab-like package it may become less accessible to people who 
prefer to work in the more programming oriented style. The other way 
around is better: if it is accessible to all in the form of a more or 
less independent package, it can be used for both approaches. This is 
why I am also not going to program specifically for SciPy, but I would 
program my packages to be 'scipy-ready' if that is made easy...

Peter




From jdhunter at ace.bsd.uchicago.edu  Sun Feb  6 09:00:22 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Sun Feb  6 09:00:22 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42055DAF.5020601@ee.byu.edu> (Travis Oliphant's message of
 "Sat, 05 Feb 2005 16:58:39 -0700")
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	<1f7baacd417e55debdde0351e463cf10@laposte.net>
	<4203CAD0.6050609@ee.byu.edu>
	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>
	<42042318.2090704@ee.byu.edu>
	<c533ee29143e3286b075927540f5844b@embl.de>
	<42055DAF.5020601@ee.byu.edu>
Message-ID: <m3zmyhr5us.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Travis" == Travis Oliphant <oliphant at ee.byu.edu> writes:

    Travis> Matplotlib as well is reinventing stuff already available
    Travis> elsewhere.  SciPy has always tried to bring a matlab-like
    Travis> environment to the user.  That's always been my goal.  It
    Travis> is frustrating to see so many with the same goal work so
    Travis> independently of each other.  Sometimes I guess it is
    Travis> necessary so that new ideas can be tried out.  But, it
    Travis> sure seems to happen more often than I would like with
    Travis> Python and Science / Engineering.

True there has been some reinvention, but as Perry pointed out 95% of
what we do on matplotlib is plotting.  The "matlab-like" environment
is created primarily by simply importing symbols from Numeric, MLab,
FFT and so on.  Along the way I've added *a few* analysis functions,
eg matplotlib.mlab's hist, psd, polyfit, but these were usually to
support some plotting feature or just to have some fun with linear
algebra. 

The other thing to emphasize vis-a-vis the matplotlib duplicating
scipy effort discussion is that the matlab-like environment is only
one thing matplotlib tries to do, and in terms of size of the code
base it is a small part of it.  We also provide an API to embed
matplotlib in every major python GUI (WX, GTK, Tk, FLTK, and QT)
including basic event handling.  We support dynamic images, eg for
building data monitoring applications (performance can be an issue
here, since matplotlib is too slow for some applications).  And we try
to appeal, with some success, to non-scientists -- financial plots,
simple pie and bar charts for business types, webapp developers...  I
would like to see it become a comprehensive 2D plotting toolkit, not
just a toolkit for scientists and engineers.

Thus, is scipy the best place for it?

    Travis> Is it ownership that is a concern?  None of us involved
    Travis> with scipy have an alterior motive for trying to bring
    Travis> packages together except the creation of a single standard
    Travis> and infrastructure for scientific computing with Python.

Ownership is a concern, but perhaps not in the sense you mean it.  I
think I pointed out before that when I started looking at plotting
solutions, I seriously considered putting my effort into
chaco/kiva/agg.  I choose not to, because it is a complex package that
was hard for me to get my head around.  I was something of a python
newbie at the time, though I had coded extensively in other languages.
I didn't really get how all the parts worked together, didn't know
much or anything about swig, agg, quartz and freetype, etc.  It's a
daunting code base to walk up to, and I don't say this pejoratively
because it's an amazing piece of work, it's just a fact.

By writing my own plotting library, I owned the code in the sense that
I fully understood it, knew how the pieces fit together, knew how to
extend and fix it.  A newbie walking up to matplotlib today might have
the same reaction I did to chaco -- they might rather roll their own
plotting solution tailored to their own needs.  I call it the curse of
python -- coding in python is so fun and easy that it facilitates if
not encourages you to roll your own solutions.

If that is right, then it really argues for making libraries as
modular as possible, so that people who want to reuse just some part
of the code, eg the chaco freetype engine, or the matplotlib mathtext
engine or W3C font manager, or the scipy constrained optimizer, can do
so.  In practice, as library developers, we want people to use (and
contribute) to our code, not to take out some juicy morsel and use it
as the basis to build a competing project.  So in addition to the
extra work it takes to make a library truly modular (matplotlib
certainly isn't) there may be a disincentive to do it.

But if it is true that new developers will shy away from largish and
complex projects, and I think scipy, chaco and matplotlib are all in
that category, then we might all benefit by making our packages more
modular.  To me, this basically means that there is an easy,
advertised and documented way with examples on how to take a piece of
the library out and plug it into your own project.  I don't know that
any of us really have the time to do it, since everyone I know is
maximally committed already to providing what they provide in
conjunction with their paying work.  Or maybe scipy already does this
and I'm not aware of it.

    Travis> I keep hearing that people like the idea of a small subset
    Travis> of packages, but that is exactly what scipy has always
    Travis> been trying to be.  Perhaps those involved with scipy have
    Travis> not voiced our intentions enough or loud enough.

I think there are two ideas about what scipy should be.  Many view it
as an awesome repository of high performance numerical algorithms that
are array aware.  You seem to view it as an integrated matlab-like
environment, eg with plotting.  I tend toward the former view in part
because I was unsatisfied with the plotting capabilities, but as you
note there are ways around that, and I'm open minded about integration
if that is the best solution.

An integrated matlab like environment, as you suggested earlier, is a
big job, and in my view should be a project that is separate from
scipy *and* matplotlib.  I don't view scipy and matplotlib as
competitors in providing a matlab-like environment, but as two
components that interoperate well and can be used together with other
software to build such an environment.  There is really very little
duplication of effort in scipy and matplotlib, and we both share a
goal of providing a matlab-like environment.  

In my view, building a matlab-like environment should be an effort
separate from scipy, matplotlib, and chaco, which incorporate scipy's
algorithms, matplotlib and/or chaco's 2D vis, mayavi's 3D vis, the
ipython shell, a good code editor, a notebook interface, and a bit
more.  Ie, we should all focus on our core competencies :-) .  A bit
like what envisage appears to be doing but envisage is a framework and
this is an application -- perhaps envisage is the framework that will
build this application.

JDH



From verveer at embl.de  Sun Feb  6 09:15:25 2005
From: verveer at embl.de (Peter Verveer)
Date: Sun Feb  6 09:15:25 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <m3zmyhr5us.fsf@peds-pc311.bsd.uchicago.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <42055DAF.5020601@ee.byu.edu> <m3zmyhr5us.fsf@peds-pc311.bsd.uchicago.edu>
Message-ID: <a4e68c9cb13cf98afba94ed7d6bd2bc4@embl.de>

> I think there are two ideas about what scipy should be.  Many view it
> as an awesome repository of high performance numerical algorithms that
> are array aware.  You seem to view it as an integrated matlab-like
> environment, eg with plotting.  I tend toward the former view in part
> because I was unsatisfied with the plotting capabilities, but as you
> note there are ways around that, and I'm open minded about integration
> if that is the best solution.

To me it seems that there are two things people want and they find more 
or less important depending on the way they work. On the one hand there 
is a need for a repository of numerical algorithms that anybody can use 
and everybody needs.  On the other hand there is a desire by a large 
group of people, but not all, to have a matlab-like environment. It 
seems to me these things should be kept separate. I am not going to 
tell anybody what their code should look like, but if I need to 
download and install a complete environment to get to a repository of 
code, I rather avoid that.




From paul at pfdubois.com  Sun Feb  6 09:52:10 2005
From: paul at pfdubois.com (Paul F. Dubois)
Date: Sun Feb  6 09:52:10 2005
Subject: [Numpy-discussion] Numerical site taken down
In-Reply-To: <200502061116.35095.dd55@cornell.edu>
References: <4760baf2.ca2b9b41.81db100@expms6.cites.uiuc.edu> <200502061116.35095.dd55@cornell.edu>
Message-ID: <4206562E.4000207@pfdubois.com>

My hosting of the Numerical Python website is now ended. Recent 
complaints that it was out of date and confusing are correct. I have 
long ago asked the current maintainers to change the page at 
numpy.sf.net which currently redirects to this page. I didn't want to 
just take it down and have nothing so I left it; and of course no good 
deed goes unpunished.

pfdubois.com/numpy now refers you to the sf.net/projects/python page.


Darren Dale wrote:
> Hi,
> 
> As a relatively new user, I would like to contribute my experience to the 
> discussion. Late in a graduate program, I caught the bug and wanted to try 
> out python. Being established in Matlab, I was hoping to find a similar 
> collection of capabilities for Python, something that I could experiment with 
> and start learning fairly quickly. When I discovered Matplotlib, I felt home 
> free. Without too much effort, I had a plot on screen and felt 'ok, this is 
> feasible.' With the recent effort from Matplotlib and IPython, this should be 
> even more true for newcomers.
> 
> On the downside, I have been somewhat confused by the state of things 
> concerning Numeric and numarray. I think I can trace this confusion back to 
> the pfdubios.com/numpy site (linked from numpy.sourceforge.net), which is the 
> first hit I get doing a google search for either numpy or numerical python. 
> "If you are new to Numerical Python, please use Numarray. The older module, 
> Numeric, is unsupported." There is no date associated with the site, but it 
> still advertises release 22.0. In the near future, I strongly suggest editing 
> websites for misleading information.
> 
> So now I have both Numeric and numarray on my system, linked to the full, 
> atlas-tuned, lapack and blas libraries. I started out using numarray, 
> thinking it was in my long term interest, but I guess this is open to debate. 
> I was also hesitant to use scipy, because it relied on Numeric. Being an 
> uninformed newbie, I just wasnt able to make an informed decision (I am still 
> struggling with it).
> 
> A lot of what I have read recently is very encouraging to me. I like the idea 
> of bridging the API gap between numarray and Numeric. I like the idea of a 
> multidimensional array getting into the core, something the community can 
> agree on. I dont care what array package I am using, I just want it to be the 
> accepted standard. I would like to hear what the numarray guys have to say 
> about Numeric3.
> 
> Finally, I like the idea of scipy. I hope the community will come to an 
> agreement and work within a framework that will result in the most efficient 
> use of everyones time, the most logical interplay between packages, the least 
> redundancy, and therefore the most gentle barrier to newcomers who want to 
> experiment with scientific computing in Python.
> 
> Deepest regards,
> Darren
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 



From strawman at astraw.com  Sun Feb  6 10:13:22 2005
From: strawman at astraw.com (Andrew Straw)
Date: Sun Feb  6 10:13:22 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4202F6A0.3@ims.u-tokyo.ac.jp>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
Message-ID: <42065E31.20906@astraw.com>

While energy is put into re-coding an array package, I'd like to bring 
up an interesting idea brought up on matplotlib-users by Norbert Nemec: 
http://sourceforge.net/mailarchive/message.php?msg_id=10528079 

Norbert's idea is a solution to the problem of a numeric package 
overriding builtins when doing "from Numeric import *". Currently 
Numeric (23.6) overrides sum() on my system, and numarray (1.2a) 
overrides round(), sum(), and abs(). The respective mlab modules bring 
in min() and max().

(I think we all agree that "from blah import *" is a naughty thing to 
do, but people are going to do it, especially for interactive work, 
especially those who have come from namespace-less languages such as 
matlab.)

Here's Norbert's original email, which I think summarizes the idea nicely.

Norbert Nemec wrote:

> Look at these one by one:
> * abs() already calls an __abs__() method. The clean way to extend it, would 
> therefore be to give arrays such a method. This should solve the problem 
> completely.
> 
> * round() does not seem to be extendable in Python. Maybe we should propose to 
> change Python itself to introduce a __round__ method? That would only be 
> straightforward.
> 
> * min, max and sum are all based on iterating over a sequence. Maybe, one 
> should again have __min__, __max__ and __sum__ which should then be checked 
> by the builtin before falling back to iterating over the sequence? I could 
> imagine many kinds of containers that could optimize these operations. So 
> this would again be a detail that should be changed in Python itself. If the 
> builtin min() function would then also pass on keyword arguments, that would 
> solve our problem completely and thoroughly.
> 
> Does anybody have experience with discussions in the Python forum to estimate 
> how realistic such a PEP would be?
>





From stephen.walton at csun.edu  Sun Feb  6 10:34:24 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Sun Feb  6 10:34:24 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <8a8a1ec91042a7507db0129c203d816b@embl.de>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <ddf1e36c0b0166a093e3643bfb54e678@laposte.net> <8a8a1ec91042a7507db0129c203d816b@embl.de>
Message-ID: <420662E7.9020003@csun.edu>

Peter Verveer wrote:

>
> On Feb 6, 2005, at 5:41 PM, konrad.hinsen at laposte.net wrote:
>
>> My personal approach is that operations should be expressed in terms 
>> of problem-specific classes in which the internal use of arrays is an 
>> implementation detail.
>>
>> There are arguments for and against both approaches, and I think 
>> there is space (meme space) for both.
>
>
> That more or less sums up my approach too. I tend to program in a mix 
> of these two approaches.

I'm both largely a lurker here and a Python naif, at least when it comes 
to the more object oriented aspects.  I'd be interested in seeing 
snippets of code illustrating the second approach here, as opposed to 
the array oriented approach with which I'm familiar.




From verveer at embl.de  Sun Feb  6 14:52:10 2005
From: verveer at embl.de (Peter Verveer)
Date: Sun Feb  6 14:52:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420662E7.9020003@csun.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <ddf1e36c0b0166a093e3643bfb54e678@laposte.net> <8a8a1ec91042a7507db0129c203d816b@embl.de> <420662E7.9020003@csun.edu>
Message-ID: <9b3f26bae489530a547c30d6afc183e0@embl.de>

On Feb 6, 2005, at 7:33 PM, Stephen Walton wrote:

> Peter Verveer wrote:
>
>>
>> On Feb 6, 2005, at 5:41 PM, konrad.hinsen at laposte.net wrote:
>>
>>> My personal approach is that operations should be expressed in terms 
>>> of problem-specific classes in which the internal use of arrays is 
>>> an implementation detail.
>>>
>>> There are arguments for and against both approaches, and I think 
>>> there is space (meme space) for both.
>>
>>
>> That more or less sums up my approach too. I tend to program in a mix 
>> of these two approaches.
>
> I'm both largely a lurker here and a Python naif, at least when it 
> comes to the more object oriented aspects.  I'd be interested in 
> seeing snippets of code illustrating the second approach here, as 
> opposed to the array oriented approach with which I'm familiar.

I guess it it a bit hard just to give a snippet, but maybe I can 
explain what it means for me. I use numarray mainly as part of 
implementing command-line programs that perform specific data analysis 
tasks. So python with numarray replaced for me what used to done in C 
or Fortran. To make the code reusable and flexible I use the 
programming constructs that Python offers, like classes and such. 
Obviously array orient numerics remains a part of that, you have to get 
the actual calculations done using the array functions. I usually have 
the array part still exposed at some level, but from what Konrad said, 
it appears he takes that a step further and he hides the actual 
mechanics of doing the calculations with arrays completely away.

I suppose most people that use python+numarray/numeric will do this to 
differents extents, but I guess many people like to have a more 
interactive enviroment, where you can type your commands and see the 
results straight away, which I guess matlab is very good at. I don't do 
that much, I prefer to design and write a program that does the task 
that needs to be done. Python+numarray provide a great programming 
language compared to C or Fortran in terms of programmer productivity. 
In this light, it may be understandable that I am not so much 
interested in having a python based numerical enviroment as much as in 
having a good repository of numeric algorithms that I can use in my 
programs.

Peter




From mdehoon at ims.u-tokyo.ac.jp  Sun Feb  6 19:58:22 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Sun Feb  6 19:58:22 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <m3zmyhr5us.fsf@peds-pc311.bsd.uchicago.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	<1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu>	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu>	<c533ee29143e3286b075927540f5844b@embl.de>	<42055DAF.5020601@ee.byu.edu> <m3zmyhr5us.fsf@peds-pc311.bsd.uchicago.edu>
Message-ID: <4206E720.60908@ims.u-tokyo.ac.jp>

>"Travis" == Travis Oliphant <oliphant at ee.byu.edu> writes:
> 
> 
>     Travis> Matplotlib as well is reinventing stuff already available
>     Travis> elsewhere.  SciPy has always tried to bring a matlab-like
>     Travis> environment to the user.  That's always been my goal.  It
>     Travis> is frustrating to see so many with the same goal work so
>     Travis> independently of each other.  Sometimes I guess it is
>     Travis> necessary so that new ideas can be tried out.  But, it
>     Travis> sure seems to happen more often than I would like with
>     Travis> Python and Science / Engineering.
> 
Speaking of code reinvention, it seems that SciPy contains some very 
useful code that overlaps with existing Python or Numerical Python code. 
Such code would reach more users if it were better integrated with 
existing code outside of SciPy. For example, I would love to see SciPy's 
additions to distutils integrated with Python's distutils, e.g. if that 
would allow users to easily compile fortran code.



From andrewm at object-craft.com.au  Mon Feb  7 00:24:28 2005
From: andrewm at object-craft.com.au (Andrew McNamara)
Date: Mon Feb  7 00:24:28 2005
Subject: [Numpy-discussion] MA and iterators
Message-ID: <20050207082300.5D00D3C889@coffee.object-craft.com.au>

I did some benchmarking today and discovered that iterating over MA
arrays is 14 times slower than iterating over a Numeric array (or a
python list). One simple solution to this is to add iterator support to
the MaskedArray class. To prove the principle, I patched MaskedArray on
the fly:

    import MA

    def ma__iter__(self):
        "Get an MA iterator."
        def scaler_iter(d, m):
            for i, v in enumerate(d):
                if m[i]:
                    v = masked
                yield v
        def array_iter(d, m, fill_value):
            ss = d.spacesaver()
            tc = d.typecode()
            for i, v in enumerate(d):
                yield MA.array(v, typecode=tc, copy=1, savespace=ss, 
                               mask=m[i], fill_value=fill_value)
        if self._mask is None:
            return iter(self._data)
        if len(self.shape) > 1:
            return array_iter(self._data, self._mask)
        else:
            return scalar_iter(self._data, self._mask, self.fill_value())

    MA.MaskedArray.__iter__ = ma__iter__

Note that this implementation requires generator support.

-- 
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/



From konrad.hinsen at laposte.net  Mon Feb  7 01:48:28 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 01:48:28 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420662E7.9020003@csun.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <ddf1e36c0b0166a093e3643bfb54e678@laposte.net> <8a8a1ec91042a7507db0129c203d816b@embl.de> <420662E7.9020003@csun.edu>
Message-ID: <9d1481068d8a74f9db31251f20287340@laposte.net>

On 06.02.2005, at 19:33, Stephen Walton wrote:

> I'm both largely a lurker here and a Python naif, at least when it 
> comes to the more object oriented aspects.  I'd be interested in 
> seeing snippets of code illustrating the second approach here, as 
> opposed to the array oriented approach with which I'm familiar.
>
You can find lots of examples in my ScientificPython library 
(http://dirac.cnrs-orleans.fr/ScientificPython/). Some examples:

- Geometry. Array people would use arrays of shape (3,) to represent 
points in 3D space. I use objects of class Scientific.Geometry.Vector. 
Array people would also store rotation matrices in arrays of shape (3, 
3), whereas I use the class Scientific.Geometry. 
Transformation.Rotation.

- Function values on a grid. Array people would have an array of x 
values (the grid) plus an array of y values, or a 2D array combining 
both. I use instances of Scientific.Functions.InterpolatingFunction.

- Histograms. Array people would store a histogram in an array of bin 
counts. I use the class Scientific.Statistics.Histogram.Histogram.

The rational behind these choices is the usual one cited for OO 
approaches: the use of abstractions to increase readability and to 
permit alternate implementations.

However, my experience with both programming and physics teaching shows 
that the level of comfort for abstraction levels is not the same for 
everyone. For example, some people (including myself) get a better 
understanding of quantum mechanics when working with quantum states as 
abstractions, whereas others prefer to work with wavefunctions (which 
are the space representations of quantum states). Fortunately there are 
textbooks for both groups.

My impression is that the optimal abstraction level is higher for 
scientists than for engineers, though my data is perhaps a bit to weak 
to back up such a conclusion. It doesn't really matter either.

Konrad.




From konrad.hinsen at laposte.net  Mon Feb  7 01:57:23 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 01:57:23 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42065E31.20906@astraw.com>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com>
Message-ID: <4ad81937f722051f27b37915225a1563@laposte.net>

On 06.02.2005, at 19:13, Andrew Straw wrote:

> While energy is put into re-coding an array package, I'd like to bring 
> up an interesting idea brought up on matplotlib-users by Norbert 
> Nemec: http://sourceforge.net/mailarchive/message.php?msg_id=10528079

My summary of what he is saying is "define flexible enough interfaces, 
and then people can write implementations the fit their needs". Which 
is a very good idea.

Note that all his examples are cases where Python chose to use 
functions where OO principles would suggest methods. If abs(), round(), 
min() etc. were methods from the start, there wouldn't be any problem 
to solve (of course they are not for a very good reason: notational 
habit). The same applies to the math module or the ufunc module in 
Numeric - there would be no need for ufunc if math simply defined 
syntactic sugar for method calls.

I wonder if it is possible to come up with an abstract API for arrays 
that would permit alternate implementations to be chosen as late as 
possible, even at runtime. That could help to get over the 
Numeric/numarray split.

Konrad.




From faltet at carabos.com  Mon Feb  7 08:07:40 2005
From: faltet at carabos.com (Francesc Altet)
Date: Mon Feb  7 08:07:40 2005
Subject: [Numpy-discussion] Proposal for making of Numarray a real Numeric 'NG'
In-Reply-To: <4205C5DE.80504@ee.byu.edu>
References: <200501211413.51663.faltet@carabos.com> <4205C5DE.80504@ee.byu.edu>
Message-ID: <200502071704.56135.faltet@carabos.com>

A Diumenge 06 Febrer 2005 08:23, Travis Oliphant va escriure:
> >The real problem for Numarray: Object Creation Time
> >===================================================
> >
> >On the other hand, the main drawback of Numarray vs Numeric is, in my
> >opinion, its poor performance regarding object creation. This might look
> >like a banal thing at first glance, but it is not in many cases. One
> > example recently reported in this list is:
>
> Ah, and there's the rub.  I don't think this object creation time will
> go away until Numarray's infrastructure becomes essentially like that of
> Numeric.  One tight object all in C.   Getting it there seems harder
> than fixing Numeric, with the additional features of Numarray.

Well, I guess I have to believe in you provided that I'm not an expert
on these issues and you are. However, I think Todd Miller has some
hopes that Numarray can eventually get rid of this object creation
latency. That would be very nice as well.

In any case, and as a person already said before during the message
storm that took place in this distribution list this weekend, I was
rather sceptical when you first announced the project for Numeric 3.0.
But now, and after reading this bunch of messages, I've changed my
mind and think (hope) that your approach can be very beneficial for
unifying the Numeric and numarray split.

I whish you the best of luck in your attempt, and I'll keep an eye on
Numeric 3.0 evolution. It would be wonderful if you finally succeed,
and we can finally have a standard numerical package for Python,
irregardless it will be in the Standard Library or not.

Cheers!,

-- 
>qo<   Francesc Altet ? ? http://www.carabos.com/
V ?V   C?rabos Coop. V. ??Enjoy Data
 ""




From juenglin at cs.pdx.edu  Mon Feb  7 08:38:28 2005
From: juenglin at cs.pdx.edu (Ralf Juengling)
Date: Mon Feb  7 08:38:28 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4ad81937f722051f27b37915225a1563@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	 <42065E31.20906@astraw.com>  <4ad81937f722051f27b37915225a1563@laposte.net>
Message-ID: <1107794238.10581.16.camel@alpspitze.cs.pdx.edu>

On Mon, 2005-02-07 at 01:56, konrad.hinsen at laposte.net wrote:
> Note that all his examples are cases where Python chose to use 
> functions where OO principles would suggest methods. If abs(), round(), 
> min() etc. were methods from the start, there wouldn't be any problem 
> to solve (of course they are not for a very good reason: notational 
> habit). The same applies to the math module or the ufunc module in 
> Numeric - there would be no need for ufunc if math simply defined 
> syntactic sugar for method calls.

No, the idea behind using function calls rather than method calls
is actually a little more than just a difference in syntax. It 
allows for a generic implementation that works in all cases of 
reasonable input, that is, it also works in cases where no efficient
implementation is provided by the input object. A simple example:

>> class DumbList(list):
...     def __len__(self):
...         raise NotImplementedError
...
>>> def len(sequence):
...     try:
...         return sequence.__len__()
...     except:
...         n=0
...         for item in sequence: n+=1
...         return n
...
>>>
>>> len([1,2,3])
3
>>> len(DumbList([1,2,3]))
3


Ralf


> 
> I wonder if it is possible to come up with an abstract API for arrays 
> that would permit alternate implementations to be chosen as late as 
> possible, even at runtime. That could help to get over the 
> Numeric/numarray split.
> 
> Konrad.
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion




From strang at nmr.mgh.harvard.edu  Mon Feb  7 08:42:17 2005
From: strang at nmr.mgh.harvard.edu (Gary Strangman)
Date: Mon Feb  7 08:42:17 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4ad81937f722051f27b37915225a1563@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
Message-ID: <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>

Hi all,

[Long post: a long-time lurker here sharing his numerical 
experiences/input re: "the numeric split" and Numeric3.]

I'm a long-time python user (and follower of this list), a heavy user in 
the numerical realm, and an "incidental" contributor to SciPy (statistical 
functions in stats.py). First, I'm impressed (and indeed amazed) at all 
the hard work folks have put in--usually without pay--to make Numpy and 
Numarray and SciPy and Matplotlib (to name the major players in this 
discussion). I am indebted to each of you. Second, I must say that I think 
this protracted discussion is EXACTLY what the python numerical community 
needs. It appears as though we have critical mass in terms of code and 
interest (and opinions), and just need to bring them all together.

Since the inception of numarray, I've just been standing back and waiting 
to see how this all sorts itself out. My stats functions work for lists 
and numpy arrays. I didn't want to convert them to numarray (given my lack 
of spare time) unless that was going to be the "new path". It appears, 
however, even after all this time, there isn't (quite) a consensus on a 
new path. After the recent message-storm, however, I am very hopeful. I 
see 4 issues at stake here, with the caveat that I'm not the code writer, 
just a user ...

1) Multiarray in Python core: I agree that this (as already stated) is (1) 
mostly irrelevant for heavy-duty numerial folks, BUT (2) is critical to 
provide for python a standardized exchange data format. Being able to 
trivially (i.e., out of the python box) load, save, pickle and load again 
on a new platform N-D array objects would be a big deal for me (and many I 
work with). Such a core object can't favor any particular size array ... 
so it would need to provide good (or excellent) performance on both little 
arrays (a la numpy) and on big arrays (a la numarray). To be in keeping 
with other python objects, it seems this object would need to be tight, 
fast and easily extensible. I *think* this is exactly what Numeric3 is 
intended to do. Getting this right is tricky, but it seems like current 
solutions are EXTREMELY close.

2) Numerical function "packaging": Looking at this from a distance (i.e., 
as a user) numerical packaging is too complex. The python spirit seems to 
call for being a bit more of a splitter (and encapsulator) than a lumper. 
For example, to do web programming in python, one often depends on several 
separate modules (html, xml, cgi, etc) rather than one all-encompassing 
one. To give numerical work the same modular feel (as well as structure 
and insulation from installation headaches), it seems that collections of 
numerical operations should be similarly organized on themes (e.g., 
timeseries analysis, morphology (nd_image?), statistics (stats.py), etc). 
This way, if you're doing timeseries analysis you import the relevant 
modules and go to work ... no worries about installing stuff required for 
morphology or statistics that you don't need. I realize this might require 
(in some cases) more refactoring, but I don't think I'm supporting 
anything *that* different from what already exists. Granted, the notion of 
what's "basic" vs. advanced is relative (e.g., where do you put fft, or 
linear_algebra?). But if made modular and encapsulated (e.g., an fft.py, 
linear_algebra.py, integration.py, morphology.py) and made available both 
individually and as part of one or more suites--see #4 below, it's easier 
to build on existing code rather than reinvent. Interestingly, although 
not obvious, this is how Matlab works too. Your first $500 pays for basic 
array-based functionality (fft, psd, etc). Then there are add-on toolkits 
(at $500 each) specifically for timeseries analysis, imaging, wavelets, 
engineering simulation, etc.

3) Plotting: Until perhaps a year ago, I did almost all my computations in 
python, then saved data out to disk, and read it into matlab to plot it. I 
hated that situation, but it was the only way to quickly and easily look 
at data interactively, with zoom, easy subplotting, etc. Matplotlib has 
all but solved this problem (thanks!!). John indicates that the ultimate 
goal with matplotlib is to provide plotting, not just scientific plotting, 
which is even better! In that case, though, and in keeping with my 
previous comment, perhaps the name matplotlib is a little misleading 
(suggesting scientific plotting only). Again, if I were familiar with 
python but just starting timeseries analysis, I would expect to load my 
data into a (multiarray) python object, import timeseries.py, import 
plotlib.py (i.e., matplotlib) and go to work doing timeseries analysis ... 
be that at LLNL, Wall St, or in my neuro lab.

4) Matlab-like Environment: Both SciPy and Matplotlib have a stated goal 
of creating a matlab-style environment. This is great, as it might help 
wean more folks off of Matlab or IDL and into the python community. 
However, I think that this (as has been suggested ... sorry, I forgot who) 
should be a separate goal from any of the above. Building an environment 
with python is different from providing functionality to python (think 
website design *environment* vs. tools for handling web content ... 
they're different). SciPy, with it's integration goal, plus matplotlib's 
plotting goal would be an outstanding combination to this end.

In sum, I pretty much agree with most previous contributors. With one 
exception. I do agree with Konrad that, theory, the best analysis approach 
is to build a custom class for each domain of investigation. I've even 
done that a couple times. However, of the dozens of researchers (i.e., 
users, not developers) I've talked about this with, almost none have the 
time (or even desire) to design and develop a class-based approach. Their 
goal is data analysis and evaluation. So, they pull the data into python 
(or, more typically, matlab or IDL), analyze it, plot the results, try to 
get a plot that's pretty enough for publication, and then go on to the 
next project ... no time for deciding what functionality should be a 
method or an attribute, subclassing hierarchies, etc. In those rare 
cases where they have the desire, matlab (I don't know about IDL) doesn't 
give you the option, meaning that users (as opposed to developers) 
probably won't go the (unfamiliar and more time consuming) route.

I apologize for the long post that "simply" supports others' opinions, 
particularly when my opinion cannot count for much (after all, I'm not 
likely to be doing much of the coding). But, I did want to express my 
appreciation for ALL the hard work that's been done, and to give the 
strongest encouragement to hashing things out now. I would LOVE to see 
some consensus on (1) what a core multiarray object should look like, 
(2-3) how to imbue python with numerical functionality and plotting for 
generations to come ;-) and (4) to create environments for scientific 
exploration within python. I think we're SOOO close ...

-best
Gary

--------------------------------------------------------------
Gary Strangman, PhD        |  Director, Neural Systems Group
Office: 617-724-0662       |  Massachusetts General Hospital
Fax:    617-726-4078       |  149 13th Street, Ste 10018
strang/@/nmr.mgh.harvard.edu |  Charlestown, MA  02129
http://www.nmr.mgh.harvard.edu/NSG/



From konrad.hinsen at laposte.net  Mon Feb  7 11:41:16 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 11:41:16 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1107794238.10581.16.camel@alpspitze.cs.pdx.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <4ad81937f722051f27b37915225a1563@laposte.net> <1107794238.10581.16.camel@alpspitze.cs.pdx.edu>
Message-ID: <0b7590e6e7b4b325bf3acdaf46aecce7@laposte.net>

On 07.02.2005, at 17:37, Ralf Juengling wrote:

> No, the idea behind using function calls rather than method calls
> is actually a little more than just a difference in syntax. It
> allows for a generic implementation that works in all cases of
> reasonable input, that is, it also works in cases where no efficient
> implementation is provided by the input object. A simple example:

True, but only due to the "add-on" character of arrays in Python. If 
arrays had been part of the initial design, we would most certainly 
have type promotion from lists to arrays in the same way as we have 
type promotion from float to complex.

Note also that this approach is anything but modular or extendable. If 
several modules went that way, they would not be interoperable (which 
is in part the problem with the numpy/numarray split).

There is also a glaring hole in the OO approach in that there is no way 
to add functionality (methods) to an existing object. The only solution 
is subclassing, but then all object creators need to be changed to 
those for the subclass, which is not practical. Unlike other OO 
languages, Python makes it possible to add methods to existing classes, 
but this always looks like a bad hack.

Konrad.




From konrad.hinsen at laposte.net  Mon Feb  7 11:47:12 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 11:47:12 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
Message-ID: <d7673a2db58066510577554360010930@laposte.net>

On 07.02.2005, at 17:41, Gary Strangman wrote:

> This way, if you're doing timeseries analysis you import the relevant  
> modules and go to work ... no worries about installing stuff required  
> for morphology or statistics that you don't need. I realize this might  
> require (in

The amount of installed code is rarely an issue, unused code costs no  
more than disk space. So as long as installation is 100% trouble free,  
I don't mind big packages. They are easier to handle for the  
developers, after all.

> In sum, I pretty much agree with most previous contributors. With one  
> exception. I do agree with Konrad that, theory, the best analysis  
> approach is to build a custom class for each domain of investigation.  
> I've even done that a couple times. However, of the dozens of  
> researchers (i.e., users, not developers) I've talked about this with,  
> almost none have the time (or even desire) to design and develop a  
> class-based approach. Their goal is data

I wasn't thinking about users, but about developers. Users with no  
desire or no competence to develop non-trivial code will take whatever  
is available anyway. But they don't write complex scripts either.

My personal experience as a library developer is that defining the  
appropriate classes is a small investment with a big payoff, but I  
don't mind if anyone else disagrees.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From Chris.Barker at noaa.gov  Mon Feb  7 11:58:22 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Mon Feb  7 11:58:22 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42055FA3.9050909@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	<1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu>	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu>	<c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu>
Message-ID: <4207C816.1000707@noaa.gov>

Travis Oliphant wrote:

> So, it is very concerning to me to see matplotlib try to become what 
> scipy is already trying to become. 

Me too. I'm ecstatic about matplotlib, but disappointed that it has as 
broad a goal as it does (and the matlab-like syntax, I hope to 
contribute more to alternatives).

It's interesting that SciPy and matplotlib really do have much the same 
goal, but taken from different tacks: SciPy focused on the computational 
functionality, and matplotlib on the plotting. However, what's become 
really clear in this discussion is that a good scientific math 
environment really needs both, and they need to be well integrated.

I'm optimistic that we can all work together to get that: One set of 
packages that work together to do everything we all need. As someone 
else on this thread mentioned, most of the individual functionality is 
all there, what's mostly need now is for it all to be packaged well and 
work together well. At the core of that is the numarray/NumPy 
unification: I hope Numeric will get us there.

One question, Travis: Why did you decide to bring numarray stuff into 
the Numeric code base, rather than the other way around? What I've 
gathered is that the only real show-stopper with numarray is array 
creation speed. Is that really impossible to fix without a complete 
re-structuring?

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From Chris.Barker at noaa.gov  Mon Feb  7 11:58:32 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Mon Feb  7 11:58:32 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <55c095f595dfec3e16579094faea14f2@laposte.net> <4203FCD8.8070204@noaa.gov> <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net>
Message-ID: <4207C83D.5040506@noaa.gov>

konrad.hinsen at laposte.net wrote:
>> By the way, here OS-X is much better than other systems, it comes with
>> lapack built in! (Veclib)
> I didn't know it has lapack, I thought it was only BLAS!

I may be wrong about that. It certainly has BLAS, but I didn't patch 
Numeric's setup.py, so I'm not sure how that worked.


> With Apple's Python, it's command line 
> installation (which I don't mind,

Neither do I, so maybe that's why. Frankly, though, fink is command line 
also, and a really ugly one, in my experience!

> I certainly agree on that. I curse Fink about once a week. But in the 
> absence of an alternative, Fink is better than no Fink.

I number of folks seem to think darwinports is a better alternative. I 
think, if  nothing else, they make greater effort to use Apple-supplied 
libraries, when they exist. I still want to try Gentoo. I love it for 
Linux (once the system is installed, anyway!)

> Attributing fault makes little sense in the OpenSource world.

Fault wasn't really the right word. The question that is relevant in 
Open Source is: Where can I contribute to address this problem?

>>  there aren't all that many people using OS-X developing Python 
>> packages, and Apple has, of course, made OS-X just different enough 
>> from other unices to require some work (or take the easy way and use 
>> fink, but then it

> Not really, you can do command-line installation ("python setup.py...") 
> just like on any other Unix, usually without any modification. The 
> problem is that Mac users expect installers.

Anything that you can setup.py --build you can  :

$ bdist_mpkg

And get a nifty Apple-style point and click package. This is thanks to 
Bob Ippolito's Py2App.

However, the big problem is dependencies. Not only are Linux users more 
accustomed to the command line, but Linux distros come with far more 
open source libs. Also, Apple has not provided a standard package 
manager. I can build an rpm (or deb, or ebuild) for a given version of 
Linux, and it fits in with the rest of the system, and users can use the 
same system to inform them about, and help them install, dependencies. 
With OS-X, you're left to fend for yourself, combing the web for 
tarballs, fink or darwinports packages.

> It should be possible to extend distutils for making Mac installers just 
> like it can make RPMs and Windows installers.

It's been done. See above.

> Right. But I still wonder why non-scientific projects (e.g. Python 
> itself, or Linux) don't suffer so much from this problem.

I think they do. I can't say I've seen a distinction. If there is one, 
perhaps it's because much of the programming done on scientific projects 
is done by scientists, rather than programmers, we're even more inclined 
to be building something primarily because we need it.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From oliphant at ee.byu.edu  Mon Feb  7 12:01:26 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Mon Feb  7 12:01:26 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <ddf1e36c0b0166a093e3643bfb54e678@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <ddf1e36c0b0166a093e3643bfb54e678@laposte.net>
Message-ID: <4207C8B9.4010806@ee.byu.edu>

konrad.hinsen at laposte.net wrote:

> On 06.02.2005, at 01:06, Travis Oliphant wrote:
>
>> But MATLAB is a huge package.  SciPy has been trying to do the same  
>> thing.  Plotting was needed and matplotlib did a great thing in  
>> providing excellent plotting.  I've never understood the reason for  
>> matplotlib to remain separate except as evidence of the great schism  
>> that was occuring in the community.
>>
>> In fact, when Eric and I began scipy, I wanted to call it pylab  
>> (you'll notice I have the pylab.sourceforge.net site).  I have fond  
>> ties to MATLAB myself.
>
>
> That's actually one of the non-technical issues I have with SciPy.  
> SciPy "smells" like Matlab. Which is fine for those who know and like  
> Matlab, but not for those (like me) who find Matlab rather 
> incompatible  with their brains. SciPy, like Matlab, is based on the 
> principle that  all operations should be expressed in terms of arrays. 
> My personal  approach is that operations should be expressed in terms 
> of  problem-specific classes in which the internal use of arrays is 
> an  implementation detail.

I think I understand better what Konrad is saying than I did several 
years ago.   SciPy may have started with the idea of being something 
like MATLAB, but in fact, scipy has lot's of class-based approaches:  
polynomials, random variables, gridding, ode solvers, just to name a few.

Basically, scipy just wants code that works and can be useful to 
others.  I don't think you'll find that scipy is opposed to class-based 
solutions at all.   We do tend to like names that are shorter (or at 
least aliases that are shorter, so that command-line interactive use is 
easier). 

Now, I think of scipy really just as a gathering place for Numerical 
algorithms.  I think it is clear that scipy needs to work on it's 
modularity, taking into account some of the great comments that have 
been made on this list if it is to be what it is intended to be.

-Travis










>
> There are arguments for and against both approaches, and I think 
> there  is space (meme space) for both. I just mention this to point 
> out that I  don't expect SciPy to become the one and only scientific 
> Python library  that includes everything. I don't expect to contribute 
> code to SciPy,  for example. I wouldn't mind using it, of course, in 
> the implementation  of my classes, if and when the installation issues 
> become less of a  problem.
>
> Konrad.
> -- 
> ------------------------------------------------------------------------ 
> -------
> Konrad Hinsen
> Laboratoire Leon Brillouin, CEA Saclay,
> 91191 Gif-sur-Yvette Cedex, France
> Tel.: +33-1 69 08 79 25
> Fax: +33-1 69 08 82 61
> E-Mail: khinsen at cea.fr
> ------------------------------------------------------------------------ 
> -------
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion





From Chris.Barker at noaa.gov  Mon Feb  7 12:04:26 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Mon Feb  7 12:04:26 2005
Subject: [Numpy-discussion] scipy, matplotlib, and installation dependencies.(previously:
  Numeric3)
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCGEAFFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCGEAFFJAA.perry@stsci.edu>
Message-ID: <4207C995.7010408@noaa.gov>

Perry Greenfield wrote:
>  Until we do, I'd favor an approach
> that shoots for a smaller, easily installable core, with a simple
> way of asking for optional packages (having users edit setup.py
> is probably asking too much).

It's well worth looking into MacPython's PackMan (AKA PIMP). Jack Jansen 
designed it to be non-platform specific. Indeed, a wxPython GUI from end 
has already been developed. Who knows whether it could ever become the 
definitive CPAN-like tool for Python, but it could be a great way to 
distribute SciPy packages.

For those who have never seen it, with a GUI front-end, you point it at 
a URL (there is a default one), and it gives you a list of packages 
available, flagged as to whether they are installed or not. You click on 
what you want, it gets downloaded and installed. That's it. It works 
very nicely for OS-X, though you do need someone(s) to maintain the 
package repository.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From oliphant at ee.byu.edu  Mon Feb  7 12:16:23 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Mon Feb  7 12:16:23 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
Message-ID: <4207CC4C.8060603@ee.byu.edu>

Gary Strangman wrote:

> To be in keeping with other python objects, it seems this object would 
> need to be tight, fast and easily extensible. I *think* this is 
> exactly what Numeric3 is intended to do. Getting this right is tricky, 
> but it seems like current solutions are EXTREMELY close.

I think it's close too, that's why I'm willing to do this, right now.

> already exists. Granted, the notion of what's "basic" vs. advanced is 
> relative (e.g., where do you put fft, or linear_algebra?). But if made 
> modular and encapsulated (e.g., an fft.py, linear_algebra.py, 
> integration.py, morphology.py) and made available both individually 
> and as part of one or more suites--see #4 below, it's easier to build 
> on existing code rather than reinvent.

There seems to be several users on this list that agree.  I would agree 
too, but would also state that Python also has the idea of "batteries 
included" meaning that the html, cgi, and xml libraries are all 
installed using a common installer.   SciPy has tried to be modular, too 
but have a common installer.  SciPy is an umbrella package with several 
sub-packages.    Sometimes it is hard to be modular when a sub-package 
depends on linear algebra routines for example. 

I think that a basic install should always include linear algebra, 
random numbers, and fft code.  I don't think this should live under a 
Numeric package because if we are to get arrayobjects into the core, 
then we need to develop some other place for these things to live.  That 
is essentially what scipy is trying to be -- a place for these extras 
and additional numeric routines to live.   Perhaps we underestimated the 
problems with using FORTRAN (but I still don't think so given that g77 
is available wherever gcc is). 

With f2py, FORTRAN is incredibly quick to develop Python modules with.  
So, I would have a hard time saying that we should not use FORTRAN.  I 
can conceed that we should separate modules that need FORTRAN from those 
that need only C(++). 

> problem (thanks!!). John indicates that the ultimate goal with 
> matplotlib is to provide plotting, not just scientific plotting, which 
> is even better! In that case, though, and in keeping with my previous 
> comment, perhaps the name matplotlib is a little misleading 
> (suggesting scientific plotting only). Again, if I were familiar with 
> python but just starting timeseries analysis, I would expect to load 
> my data into a (multiarray) python object, import timeseries.py, 
> import plotlib.py (i.e., matplotlib) and go to work doing timeseries 
> analysis ... be that at LLNL, Wall St, or in my neuro lab.

I think matplotlib is great.   SciPy's plotting methods were always 
considered stop-gap (functional but not much else).   I would have much 
rather seen matplotlib integrate in with the scipy effort, though.   
Perhaps these modularity issues would be solved already, had this been 
attempted from the start.

-Travis








From oliphant at ee.byu.edu  Mon Feb  7 12:38:17 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Mon Feb  7 12:38:17 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4207C816.1000707@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	<1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu>	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu>	<c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <4207C816.1000707@noaa.gov>
Message-ID: <4207D18A.1070309@ee.byu.edu>

>
> It's interesting that SciPy and matplotlib really do have much the 
> same goal, but taken from different tacks: SciPy focused on the 
> computational functionality, and matplotlib on the plotting. However, 
> what's become really clear in this discussion is that a good 
> scientific math environment really needs both, and they need to be 
> well integrated.

The problem was that we needed someone like John to join us on the scipy 
effort to get the plotting working well.   Our team lacked someone with 
his skill.  All of us working on SciPy would have loved to work with 
him.  So, I have nothing but respect for John and his efforts.  I just 
wish we at SciPy were better recruiters :-)

> I'm optimistic that we can all work together to get that: One set of 
> packages that work together to do everything we all need. As someone 
> else on this thread mentioned, most of the individual functionality is 
> all there, what's mostly need now is for it all to be packaged well 
> and work together well. At the core of that is the numarray/NumPy 
> unification: I hope Numeric will get us there.

Yes, I think the numarray / Numeric split has been one of the biggest 
problems. 

>
> One question, Travis: Why did you decide to bring numarray stuff into 
> the Numeric code base, rather than the other way around? What I've 
> gathered is that the only real show-stopper with numarray is array 
> creation speed. Is that really impossible to fix without a complete 
> re-structuring?

A great question.    People deserve to hear what I think even if they 
disagree with me, so here is a summary of the issues I'm concerned about 
with numarray.  The basic answer to the question, is that I feel that 
numarray is too different structurally (i.e. the classes and objects 
that get defined) from Numeric and some of these differences are causing 
the speed issues.  I felt it would be too much work
to adapt numarray to the Numeric structure than adapt Numeric to the 
numarray features. 

Here are some specifics.

1)  Numarray did not seem to build on Numeric at all.  It has thrown out 
far too much.  As just one example, the ufunc object in Numeric is a 
pretty good start, but numarray decided to completely change the 
interface for reasons that I do not understand.   One result of this 
decision is that numarray still does not provide a similar C-API for the 
creation of ufuncs that Numeric did.  

2) Basic numarray _ndarray C object is way too big.  numarray added too 
many things to the underlying C-structure of an arrayobject.  I think 
this will have small-array performance implications.

3) While prototyping in Python was a good idea. Numarray should have 
moved the entire object to C and not left so many things to the Python 
level.  I don't think there should be a Python-level arrayobject as the 
basic class (except for RecordArrays).    I think this move must still 
be done to solve the speed issues, and I see this has much harder than 
fixing Numeric which is already all in C.

4) The numarray C-API seems way too big.  There are so many, seemingly 
repeated calls that seem to do the same thing. Are all of these API 
calls really necessary? 

5) Numarray builds fundamentally on Int16, Int32, and Float32 objects.  
I understand the need for this in many applications, but some users will 
still need a way to define arrays based on the c-type that is desired.  
In addition, as the mapping from bit-width to c-type is quite platform 
dependent, this needs to be done more carefully.

I'm not looking to debate these issues, because I agree that other 
opinions may be valid, I could be wrong, and the debate will just 
distract me.   But, fundamentally, my decision was based on a gut-feel 
influenced no doubt by my familiarity and appreciation of the Numeric 
code base.   If I'm wrong it will be apparent in a couple of months. 

-Travis










From Chris.Barker at noaa.gov  Mon Feb  7 12:42:29 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Mon Feb  7 12:42:29 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42042318.2090704@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu>
Message-ID: <4207D285.70103@noaa.gov>

Travis Oliphant wrote:

> Perhaps it would be good to get a list of which platforms actually
> need binaries for people to look more favorably at SciPy.   I can see
> that OS X is on the top of that list, surely somebody could
> distribute a copy of a binary for that platform (I know of people who
> have it installed).

It is MUCH easier to get a complex package working on OS-X than to make
a binary of it. Not because it's hard to make a binary (it's very, very
easy, thanks to Py2App), but because, in general, open source software
ends up being dynamically linked against a bunch of libraries. Getting
those libs installed on your system is a lot easier than figuring out
how to get the package to statically link to them (or install them as
part of your package)

I know this because I spent a few days last week getting a binary of
matplotlib together.

If there is a small group of folks that would like to work on making a
binary SciPy for OS-X, I'd be glad to work with others to do it. I doubt
I'll do it myself, however.

By the way, does anyone know a way to get distutils to statically link a
given library? So far, I've done the trick of making sure it can only
find the *.a, and not a *.dylib, but this is kind of a pain, and feels
like a kludge.

Paul DuBois wrote:
> My experience with binary distributions has convinced me that it is a
> hopeless endeavor in the long run. I understand why people want them
> but until you've tried to provide them to the public and seen the
> number of things that go wrong ... Not to mention that it means you
> have to have someone build on all those systems and you don't have
> them all. And if the person is going to add their other extensions,
> compiled with a different compiler, blooie.

This is Much less of an issue with OS-X than many other systems. Apples 
keeps it pretty tightly controlled, If you build packages without 
linking in a bunch of fink/darwinports/portage stuff!

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From strang at nmr.mgh.harvard.edu  Mon Feb  7 13:32:34 2005
From: strang at nmr.mgh.harvard.edu (Gary Strangman)
Date: Mon Feb  7 13:32:34 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4207CC4C.8060603@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
 <4207CC4C.8060603@ee.byu.edu>
Message-ID: <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>

>> To be in keeping with other python objects, it seems this object would 
>> need to be tight, fast and easily extensible. I *think* this is exactly 
>> what Numeric3 is intended to do. Getting this right is tricky, but it 
>> seems like current solutions are EXTREMELY close.
>
> I think it's close too, that's why I'm willing to do this, right now.

:-)

>> already exists. Granted, the notion of what's "basic" vs. advanced is 
>> relative (e.g., where do you put fft, or linear_algebra?). But if made 
>> modular and encapsulated (e.g., an fft.py, linear_algebra.py, 
>> integration.py, morphology.py) and made available both individually and as 
>> part of one or more suites--see #4 below, it's easier to build on existing 
>> code rather than reinvent.
>
> There seems to be several users on this list that agree.  I would agree too, 
> but would also state that Python also has the idea of "batteries included" 
> meaning that the html, cgi, and xml libraries are all installed using a 
> common installer.   SciPy has tried to be modular, too but have a common 
> installer.  SciPy is an umbrella package with several sub-packages. 
> Sometimes it is hard to be modular when a sub-package depends on linear 
> algebra routines for example.

The "batteries included" is a good point. But I guess I'm not alone in 
the modularity argument.

> I think that a basic install should always include linear algebra, random 
> numbers, and fft code.

+1

> I don't think this should live under a Numeric 
> package because if we are to get arrayobjects into the core, then we need to 
> develop some other place for these things to live.

+1

> That is essentially what 
> scipy is trying to be -- a place for these extras and additional numeric 
> routines to live.

And NOT a matlab-like environment? Or is it trying to be both?

>   Perhaps we underestimated the problems with using FORTRAN 
> (but I still don't think so given that g77 is available wherever gcc is). 
> With f2py, FORTRAN is incredibly quick to develop Python modules with.  So, I 
> would have a hard time saying that we should not use FORTRAN.  I can conceed 
> that we should separate modules that need FORTRAN from those that need only 
> C(++).

Being able to say, "to get these additional functions, you need to get 
FORTRAN" would be fantastic. It would help folks (including myself, who 
dreads complicated installs) if wrestling with the install is going to be 
worth the effort.

Gary



From konrad.hinsen at laposte.net  Mon Feb  7 13:50:26 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 13:50:26 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4207C83D.5040506@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <55c095f595dfec3e16579094faea14f2@laposte.net> <4203FCD8.8070204@noaa.gov> <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net> <4207C83D.5040506@noaa.gov>
Message-ID: <a91c9df3205ba200fdaab4ba667270a0@laposte.net>

On 07.02.2005, at 20:57, Chris Barker wrote:

> I may be wrong about that. It certainly has BLAS, but I didn't patch  
> Numeric's setup.py, so I'm not sure how that worked.

I checked on Apple's developer site: there is an optimized BLAS in  
vecLib, plus the standard CLAPACK.

> Neither do I, so maybe that's why. Frankly, though, fink is command  
> line also, and a really ugly one, in my experience!

There is Fink Commander, which is only 95% as ugly as the Fink command  
line ;-)

> I number of folks seem to think darwinports is a better alternative. I  
> think, if  nothing else, they make greater effort to use  
> Apple-supplied libraries, when they exist. I still want to try Gentoo.  
> I love it for Linux (once the system is installed, anyway!)

I am seriously out of time for playing with installations at the  
moment, but I am always interested to hear about alternatives to Fink.

I like the gentoo approach as well in principle, but I also know that  
compiling tons of stuff on an iBook is a real challenge.

> And get a nifty Apple-style point and click package. This is thanks to  
> Bob Ippolito's Py2App.

I'll have to look at that one!

> open source libs. Also, Apple has not provided a standard package  
> manager. I can build an rpm (or deb, or ebuild) for a given version of  
> Linux,

Yes, that's a big issue. Apple has done a nice job with its  
application-directory system for handling stand-along applications, but  
there is little support for the OpenSource world's building block  
approach. I guess this is why Fink opted for a complete subsystem  
around Debian packages.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From stephen.walton at csun.edu  Mon Feb  7 13:59:33 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Mon Feb  7 13:59:33 2005
Subject: [Numpy-discussion] logical priority
Message-ID: <4207E421.10702@csun.edu>

A probably bonehead question:  is it a known feature of Python that, in 
a logical operation, & has a higher priority than > or < ?  Or is it a 
quirk of numarray?

In [126]: a = arange(25)

In [127]: v=(a>9) & (a<13)

In [128]: v=a>9 & a<13
---------------------------------------------------------------------------
exceptions.RuntimeError                              Traceback (most 
recent call last)

/home/swalton/research/foukal/<console>

/usr/lib/python2.3/site-packages/numarray/generic.py in __nonzero__(self)
    475
    476     def __nonzero__(self):
--> 477         raise RuntimeError("An array doesn't make sense as a 
truth value.  Use any(a) or all(a).")
    478
    479     def __copy__(self):

RuntimeError: An array doesn't make sense as a truth value.  Use any(a) 
or all(a).




From ariciputi at pito.com  Mon Feb  7 14:03:18 2005
From: ariciputi at pito.com (Andrea Riciputi)
Date: Mon Feb  7 14:03:18 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
Message-ID: <2C2E3F9E-7954-11D9-9E63-000A95C0BC68@pito.com>

Hi all,
I'm yet another long-time lurker, and Python/Numeric/SciPy user, and 
I'm very interested in this discussion. I'm completely agree with Gary, 
and I'll look forward to see the results of Travis's effort. I wish him 
best luck for Numeric3, and to all scientific-related Python community 
to keep on developing a striking scientific-environment. As Gary said, 
we are so close. At the moment, I'm stuked with my Ph.D. thesis, I hope 
in the future to be able to give a little contribution.

I also want to thanks all the Python scientific developers for their 
efforts.

Cheers,
  Andrea.

P.S.: I'm a Mac OSX users, and I've all the Python related packages 
(including SciPy, Numeric, matplotlib...) installed via Fink. I've 
never had any problem, could you explain to me what kind of problems 
are you experiencing?


On 7 Feb 2005, at 17:41, Gary Strangman wrote:

> Hi all,
>
> [Long post: a long-time lurker here sharing his numerical 
> experiences/input re: "the numeric split" and Numeric3.]
>
> I'm a long-time python user (and follower of this list), a heavy user 
> in the numerical realm, and an "incidental" contributor to SciPy 
> (statistical functions in stats.py). First, I'm impressed (and indeed 
> amazed) at all the hard work folks have put in--usually without 
> pay--to make Numpy and Numarray and SciPy and Matplotlib (to name the 
> major players in this discussion). I am indebted to each of you. 
> Second, I must say that I think this protracted discussion is EXACTLY 
> what the python numerical community needs. It appears as though we 
> have critical mass in terms of code and interest (and opinions), and 
> just need to bring them all together.




From stephen.walton at csun.edu  Mon Feb  7 14:05:21 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Mon Feb  7 14:05:21 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
Message-ID: <4207E5DF.8020506@csun.edu>

Gary Strangman wrote:

> Being able to say, "to get these additional functions, you need to get 
> FORTRAN" would be fantastic. It would help folks (including myself, 
> who dreads complicated installs) if wrestling with the install is 
> going to be worth the effort. 

I would rather say, "There is a huge amount of free code for doing 
complicated numerical work already developed in Fortran.  F2PY allows 
you to use this existing code base from Python, saving you having to 
rewrite in Python, C++, or what-have-you.  Since g77 is free and easy to 
install, there's no reason to re-invent."

BTW, g95's availability was more-or-less officially announced on Cleve 
Moler's listserv over the weekend.





From konrad.hinsen at laposte.net  Mon Feb  7 14:08:27 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 14:08:27 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <4207E421.10702@csun.edu>
References: <4207E421.10702@csun.edu>
Message-ID: <0a221fd095d46402910135731d02ad05@laposte.net>

On 07.02.2005, at 22:56, Stephen Walton wrote:

> A probably bonehead question:  is it a known feature of Python that,  
> in a logical operation, & has a higher priority than > or < ?  Or is  
> it a quirk of numarray?

Operator priorities are part of the Python language and implemented in  
the parser. Numarray cannot interfere with that.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From stephen.walton at csun.edu  Mon Feb  7 14:30:25 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Mon Feb  7 14:30:25 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <0a221fd095d46402910135731d02ad05@laposte.net>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net>
Message-ID: <4207EBBF.2060808@csun.edu>

konrad.hinsen at laposte.net wrote:

> Operator priorities are part of the Python language and implemented 
> in  the parser. Numarray cannot interfere with that.

In Python, of course, & is a bitwise AND, logical and being the literal 
word "and".  & has lower operator priority than <, but "and" has a 
higher one.

I think the real problem is that "and" doesn't work on numarray bool 
arrays in an element-wise fashion as I expected;  perhaps it should?  
Otherwise constructs like

v = A>10 and A<15

have to be written as

v1=A>10; v2=A<15; v=v1&v2

or as
v = (A>10) & (A<15)




From Chris.Barker at noaa.gov  Mon Feb  7 15:44:10 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Mon Feb  7 15:44:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <2C2E3F9E-7954-11D9-9E63-000A95C0BC68@pito.com>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <2C2E3F9E-7954-11D9-9E63-000A95C0BC68@pito.com>
Message-ID: <4207FD0F.1010908@noaa.gov>

Andrea Riciputi wrote:

> P.S.: I'm a Mac OSX users, and I've all the Python related packages 
> (including SciPy, Numeric, matplotlib...) installed via Fink. I've never 
> had any problem, could you explain to me what kind of problems are you 
> experiencing?

Do you use it with any of the Aqua-based GUIs ? Probably not. My 
experience with fink is that it is a subsystem all it's own. If you do 
everything with fink, it'll work fine. Personally, I'm a Linux geek, so 
I'd probably be happy with an all-fink system, but the only reason I use 
OS-X at all is that I work with other OS-X users, none of whom are Unix 
folks (they mostly come from the old Mac-OS world). I need to be able to 
develop apps that they can just fire up and have working on their stock 
OS-X boxes, and that look like Mac apps.

I've decided that fink-free is the best way to achieve this at the moment.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From oliphant at ee.byu.edu  Mon Feb  7 21:13:10 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Mon Feb  7 21:13:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
Message-ID: <42084A1F.3030007@ee.byu.edu>

> And NOT a matlab-like environment? Or is it trying to be both?

Very loosely a matlab or IDL-like environment.  Nobody has really 
defined what it means to be a MATLAB-like environment.   The idea is 
that someone could do what one can do in MATLAB or IDL, not write code 
in exactly the same way.  A MATLAB translator is an entirely different 
project.

So, I'm not sure.   There is no requirement that contributions "not be 
classed based", or conform to some command in MATLAB. 

Perhaps what you are referring to is the fact that

from scipy import *

makes a lot of command-like interaction possible.  Is this seen as 
MATLAB-like?

 I guess right now, scipy is trying to bring lots of modules under a 
common naming convention.

I think the goals of scipy are very much what people are talking about 
here.   SciPy ultimately becomes what its users and developers want it 
to become.   That is why I'm so unclear as to why we have not received 
more support in its development --- even in the way of ideas as to how 
to make it better --- until the great ideas that have been suggested on 
this list recently.

-Travis




From oliphant at ee.byu.edu  Mon Feb  7 21:19:25 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Mon Feb  7 21:19:25 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <4207EBBF.2060808@csun.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu>
Message-ID: <42084BB7.2060509@ee.byu.edu>

Stephen Walton wrote:

> konrad.hinsen at laposte.net wrote:
>
>> Operator priorities are part of the Python language and implemented 
>> in  the parser. Numarray cannot interfere with that.
>
>
> In Python, of course, & is a bitwise AND, logical and being the 
> literal word "and".  & has lower operator priority than <, but "and" 
> has a higher one.
>
> I think the real problem is that "and" doesn't work on numarray bool 
> arrays in an element-wise fashion as I expected;  perhaps it should?  
> Otherwise constructs like

Currently, I don't think Python allows "over-riding" the "and" 
operation.  It only works on the truth or falseness of the objects.  
Therefore, I don't think it is possible to make it work as you'd expect 
unless Python is changed.





From Fernando.Perez at colorado.edu  Mon Feb  7 21:26:10 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon Feb  7 21:26:10 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42084BB7.2060509@ee.byu.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu>
Message-ID: <42084D42.9050509@colorado.edu>

Travis Oliphant wrote:

> Currently, I don't think Python allows "over-riding" the "and" 
> operation.  It only works on the truth or falseness of the objects.  
> Therefore, I don't think it is possible to make it work as you'd expect 
> unless Python is changed.

You are correct:

In [12]: dis.dis(compile('x and y','<console>','eval'))
   0           0 LOAD_NAME                0 (x)
               3 JUMP_IF_FALSE            4 (to 10)
               6 POP_TOP
               7 LOAD_NAME                1 (y)
         >>   10 RETURN_VALUE


There is no method you can hook in there.  Compare this to '&':

In [13]: dis.dis(compile('x & y','<console>','eval'))
   0           0 LOAD_NAME                0 (x)
               3 LOAD_NAME                1 (y)
               6 BINARY_AND
               7 RETURN_VALUE

Cheers,

f



From NadavH at VisionSense.com  Tue Feb  8 01:08:10 2005
From: NadavH at VisionSense.com (Nadav Horesh)
Date: Tue Feb  8 01:08:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42084A1F.3030007@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu>
Message-ID: <42087FAE.5000303@VisionSense.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050208/959a2d94/attachment.html>

From oliphant at ee.byu.edu  Tue Feb  8 01:23:18 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Tue Feb  8 01:23:18 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42087FAE.5000303@VisionSense.com>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42087FAE.5000303@VisionSense.com>
Message-ID: <420884D2.5020103@ee.byu.edu>

Thanks for your comments.

Nadav Horesh wrote:

> Most of my work is carried using the numarray/Numeric package, and use 
> scipy not so often for three reasons:
>
>    1. nummaray is well equip.
>
It is certainly missing many things.

>    1. scipy is mostly undocumented.
>
SciPy is decently self documented.  And, with  
http://www.scipy.org/livedocs/ 
self-documentation leads to rapid online documentation.

>    1. scipy had bugs in many  of it high-end (=rarely used) routines.
>
Interesting claim.   Yes bugs crop up, but they are typically fixed 
quickly when found.   Basing much of the code on standard compiled 
routines helps to alleviate this problem.

> This is a high barrier for a causal scipy users like me, and thus it 
> hard for me to contribute back. For instance, I wrote a bicoherence 
> routine some time ago, but found no proper place to contribute it. 
> Scipy would be a natural place if it would be more accessible.
> One of the good reasons to provide bare Numeric3 and to put fft, 
> linear algebra, etc. in scipy, is that it would force scipy into the 
> painful process of being user friendly (=accessible)


One very important reason I would like to do it.  It is hard to know if 
scipy is accessible or not if people who have problems don't complain 
but instead walk away or just start their own package.

-Travis





From konrad.hinsen at laposte.net  Tue Feb  8 03:37:28 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Tue Feb  8 03:37:28 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <0b7590e6e7b4b325bf3acdaf46aecce7@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <4ad81937f722051f27b37915225a1563@laposte.net> <1107794238.10581.16.camel@alpspitze.cs.pdx.edu> <0b7590e6e7b4b325bf3acdaf46aecce7@laposte.net>
Message-ID: <e45d15d3bb48ba6869b77ceb91d2e32e@laposte.net>

On 07.02.2005, at 20:40, konrad.hinsen at laposte.net wrote:

> There is also a glaring hole in the OO approach in that there is no  
> way to add functionality (methods) to an existing object. The only  
> solution is subclassing, but then all object creators need to be  
> changed to those for the subclass, which is not practical. Unlike  
> other OO languages, Python makes it possible to add methods to  
> existing classes, but this always looks like a bad hack.

There is something I forgot to mention in this context:

	http://www.python.org/peps/pep-0246.html

If adopted, this would be a big step forward in my opinion.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Tue Feb  8 03:45:16 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Tue Feb  8 03:45:16 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
Message-ID: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>

Is anyone here using NumPy on Opteron machines running Linux in 64 bit  
mode? I am running into problems such as:

Python 2.4 (#4, Jan 18 2005, 18:06:45)
[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> from Numeric import *
 >>> a = arange(4)
 >>> a
array([0, 1, 2, 3])
 >>> a.shape = (2, 2)
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
ValueError: total size of new array must be unchanged

This is with Python 2.4 and Numeric 23.7. Before looking into this  
myself, I thought I'd ask here, perhaps someone has already found a  
fix.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From Alexandre.Fayolle at logilab.fr  Tue Feb  8 04:32:37 2005
From: Alexandre.Fayolle at logilab.fr (Alexandre)
Date: Tue Feb  8 04:32:37 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>
References: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>
Message-ID: <20050208123054.GC14155@crater.logilab.fr>

On Tue, Feb 08, 2005 at 12:44:18PM +0100, konrad.hinsen at laposte.net wrote:
> Is anyone here using NumPy on Opteron machines running Linux in 64 bit  
> mode? I am running into problems such as:
> 
> Python 2.4 (#4, Jan 18 2005, 18:06:45)
> [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from Numeric import *
> >>> a = arange(4)
> >>> a
> array([0, 1, 2, 3])
> >>> a.shape = (2, 2)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> ValueError: total size of new array must be unchanged
> 
> This is with Python 2.4 and Numeric 23.7. Before looking into this  
> myself, I thought I'd ask here, perhaps someone has already found a  
> fix.

I don't have a fix, but, for the record, I can reproduce this on my
alphastation. 

a.typecode() says that the contents of a are Int64.  

-- 
Alexandre Fayolle                              LOGILAB, Paris (France).
http://www.logilab.com   http://www.logilab.fr  http://www.logilab.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050208/5699345f/attachment.sig>

From cjw at sympatico.ca  Tue Feb  8 04:58:37 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Tue Feb  8 04:58:37 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42084BB7.2060509@ee.byu.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu>
Message-ID: <4208B718.4050502@sympatico.ca>

Travis Oliphant wrote:

> Stephen Walton wrote:
>
>> konrad.hinsen at laposte.net wrote:
>>
>>> Operator priorities are part of the Python language and implemented 
>>> in  the parser. Numarray cannot interfere with that.
>>
>>
>>
>> In Python, of course, & is a bitwise AND, logical and being the 
>> literal word "and".  & has lower operator priority than <, but "and" 
>> has a higher one.
>>
>> I think the real problem is that "and" doesn't work on numarray bool 
>> arrays in an element-wise fashion as I expected;  perhaps it should?  
>> Otherwise constructs like
>
>
> Currently, I don't think Python allows "over-riding" the "and" 
> operation.  It only works on the truth or falseness of the objects.  
> Therefore, I don't think it is possible to make it work as you'd 
> expect unless Python is changed.
>
>
There is __nonzero__ which could be treated as allTrue or someTrue, the 
former is probably the better.

Colin W.

>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>




From hoel at gl-group.com  Tue Feb  8 05:16:13 2005
From: hoel at gl-group.com (=?ISO-8859-15?Q?Berthold_H=F6llmann?=)
Date: Tue Feb  8 05:16:13 2005
Subject: [Numpy-discussion] Re: NumPy on 64-bit Linux (Opteron)
In-Reply-To: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net> (konrad hinsen's message of "Tue, 8 Feb 2005 12:44:18 +0100")
References: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>
Message-ID: <se4qgnb3da.fsf@pc047299.dhcp.germanlloyd.org>

konrad.hinsen at laposte.net writes:

> Is anyone here using NumPy on Opteron machines running Linux in 64 bit
> mode? I am running into problems such as:
>
> Python 2.4 (#4, Jan 18 2005, 18:06:45)
> [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> from Numeric import *
>  >>> a = arange(4)
>  >>> a
> array([0, 1, 2, 3])
>  >>> a.shape = (2, 2)
> Traceback (most recent call last):
>    File "<stdin>", line 1, in ?
> ValueError: total size of new array must be unchanged
>
> This is with Python 2.4 and Numeric 23.7. Before looking into this
> myself, I thought I'd ask here, perhaps someone has already found a
> fix.

I do have currently access to an Opteron machine. OS is SuSE SLES9,
with a lot of Python packages from SuSE Professional 9.1. I get:

~> python
Python 2.3.3 (#1, Nov 29 2004, 22:35:24) 
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Numeric as N
>>> a = N.arange(4)
>>> a
array([0, 1, 2, 3])
>>> a.shape = (2,2)
>>> N.__file__
'/usr/lib64/python2.3/site-packages/Numeric/Numeric.pyc'
>>> N.__version__
'23.1'
>>> 

Kind regards

Berthold
-- 
Germanischer Lloyd AG
CAE Development
Vorsetzen 35
20459 Hamburg
Phone: +49(0)40 36149-7374
Fax: +49(0)40 36149-7320
e-mail: hoel at gl-group.com
Internet: http://www.gl-group.com



This e-mail contains confidential information for the exclusive attention of the intended addressee. Any access of third parties to this e-mail is unauthorised. Any use of this e-mail by unintended recipients such as copying, distribution, disclosure etc. is prohibited and may be unlawful. When addressed to our clients the content of this e-mail is subject to the General Terms and Conditions of GL's Group of Companies applicable at the date of this e-mail. 

GL's Group of Companies does not warrant and/or guarantee that this message at the moment of receipt is authentic, correct and its communication free of errors, interruption etc.



From strang at nmr.mgh.harvard.edu  Tue Feb  8 06:57:32 2005
From: strang at nmr.mgh.harvard.edu (Gary Strangman)
Date: Tue Feb  8 06:57:32 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42084A1F.3030007@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
 <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
 <42084A1F.3030007@ee.byu.edu>
Message-ID: <Pine.LNX.4.60.0502080928360.16300@gate.nmr.mgh.harvard.edu>

On Mon, 7 Feb 2005, Travis Oliphant wrote:

>> And NOT a matlab-like environment? Or is it trying to be both?
>
> Very loosely a matlab or IDL-like environment.  Nobody has really defined 
> what it means to be a MATLAB-like environment.   The idea is that someone 
> could do what one can do in MATLAB or IDL, not write code in exactly the same 
> way.  A MATLAB translator is an entirely different project.
>
> So, I'm not sure.   There is no requirement that contributions "not be 
> classed based", or conform to some command in MATLAB. 
> Perhaps what you are referring to is the fact that
>
> from scipy import *
>
> makes a lot of command-like interaction possible.  Is this seen as 
> MATLAB-like?

I was focusing on the word "environment". The current Matlab "environment" 
is heavily graphical and includes separate panels for command history, 
defined variables (along with their memory requirements), a command 
window, etc. Of course it also has a flat namespace. The python 
interpreter is already its own (restricted) environment, and Idle is 
another. Building a Matlab "environment", in this sense of the word, is a 
major undertaking and should be a separate goal (and, IMHO, probably not 
all that useful). Doesn't sound like that's what you meant. My 
misinterpretation.

> I guess right now, scipy is trying to bring lots of modules under a common 
> naming convention.

And this is great. I'd definitely use such a creature. However, it sounds 
like several folks would also appreciate (myself included) access to 
individual modules (or smaller subsets of interdependent modules) because 
of installation troubles. I'm probably not the one to ask about how to 
divvy those up, but perhaps it would be useful to have a "requires 
fortran" group. And a python-only group. And a python-plus-C group. That 
way, users (or "mini-developers") could pick their installation battles 
(on windoze, having ANY compiler can be a significant $$ or installation 
problem). This would make it more transparent as to which routines are 
well-tested and fast (most Fortran-based code), fast (C and Fortran), or 
easy-to-modify (Python pieces). The transparency could also guide folks 
who can/do write C/Fortran code to translate their favorite python-only 
pieces to a faster format. Just a thought. ('course, those dividing lines 
may not make for nice module names ... sigh.)

> I think the goals of scipy are very much what people are talking about here. 
> SciPy ultimately becomes what its users and developers want it to become. 
> That is why I'm so unclear as to why we have not received more support in its 
> development --- even in the way of ideas as to how to make it better --- 
> until the great ideas that have been suggested on this list recently.

Perhaps because SciPy has seemed monolithic, rather than a collection of 
largely independent modules. That's how I've perceived it, and I even have 
a module inside SciPy. ;-) I think that making pieces available (groupings 
of interdependent modules) in addition to everything-under-the-sun would
help with that perception.

-best
Gary



From southey at uiuc.edu  Tue Feb  8 07:07:11 2005
From: southey at uiuc.edu (Bruce Southey)
Date: Tue Feb  8 07:07:11 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux
 (Opteron)
Message-ID: <4542a74a.cb733b75.81ad300@expms6.cites.uiuc.edu>

Hi, 
Yes (Python 2.3.3 [GCC 3.3.3 (SuSE Linux)]) with Numeric 23.7 (clean install). 
 
It is to do with the function: 
extern PyObject * PyArray_Reshape(PyArrayObject *self, PyObject *shape) 
 
the variable s_known is not being assigned correctly (gets set to zero). I 
think that stems from this call that doesn't set the dimensions correctly: 
 
PyArray_As1D(&shape, (char **)&dimensions, &n, PyArray_LONG) 
 
dimensions in the second loop has value 0. 
 
Regards 
Bruce 
 
---- Original message ---- 
>Date: Tue, 8 Feb 2005 12:44:18 +0100 
>From: konrad.hinsen at laposte.net   
>Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)   
>To: numpy-discussion <numpy-discussion at lists.sourceforge.net> 
> 
>Is anyone here using NumPy on Opteron machines running Linux in 64 bit   
>mode? I am running into problems such as: 
> 
>Python 2.4 (#4, Jan 18 2005, 18:06:45) 
>[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2 
>Type "help", "copyright", "credits" or "license" for more information. 
> >>> from Numeric import * 
> >>> a = arange(4) 
> >>> a 
>array([0, 1, 2, 3]) 
> >>> a.shape = (2, 2) 
>Traceback (most recent call last): 
>   File "<stdin>", line 1, in ? 
>ValueError: total size of new array must be unchanged 
> 
>This is with Python 2.4 and Numeric 23.7. Before looking into this   
>myself, I thought I'd ask here, perhaps someone has already found a   
>fix. 
> 
>Konrad. 
>-- 
>------------------------------------------------------------------------  
>------- 
>Konrad Hinsen 
>Laboratoire Leon Brillouin, CEA Saclay, 
>91191 Gif-sur-Yvette Cedex, France 
>Tel.: +33-1 69 08 79 25 
>Fax: +33-1 69 08 82 61 
>E-Mail: khinsen at cea.fr 
>------------------------------------------------------------------------  
>------- 
> 
> 
> 
>------------------------------------------------------- 
>SF email is sponsored by - The IT Product Guide 
>Read honest & candid reviews on hundreds of IT Products from real users. 
>Discover which products truly live up to the hype. Start reading now. 
>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click 
>_______________________________________________ 
>Numpy-discussion mailing list 
>Numpy-discussion at lists.sourceforge.net 
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion 
 



From southey at uiuc.edu  Tue Feb  8 08:05:29 2005
From: southey at uiuc.edu (Bruce Southey)
Date: Tue Feb  8 08:05:29 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux
 (Opteron)
Message-ID: <143ec816.cb7887f4.8344000@expms6.cites.uiuc.edu>

Hi, 
I should also add that on a P3 (same info as below), there is no problem as the 
correct dimensions are returned. 
 
Regards 
Bruce 
 
---- Original message ---- 
>Date: Tue, 8 Feb 2005 09:06:04 -0600 
>From: Bruce Southey <southey at uiuc.edu>   
>Subject: Re: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)   
>To: konrad.hinsen at laposte.net, numpy-discussion 
<numpy-discussion at lists.sourceforge.net> 
> 
>Hi,  
>Yes (Python 2.3.3 [GCC 3.3.3 (SuSE Linux)]) with Numeric 23.7 (clean install).  
>  
>It is to do with the function:  
>extern PyObject * PyArray_Reshape(PyArrayObject *self, PyObject *shape)  
>  
>the variable s_known is not being assigned correctly (gets set to zero). I  
>think that stems from this call that doesn't set the dimensions correctly:  
>  
>PyArray_As1D(&shape, (char **)&dimensions, &n, PyArray_LONG)  
>  
>dimensions in the second loop has value 0.  
>  
>Regards  
>Bruce  
>  
>---- Original message ----  
>>Date: Tue, 8 Feb 2005 12:44:18 +0100  
>>From: konrad.hinsen at laposte.net    
>>Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)    
>>To: numpy-discussion <numpy-discussion at lists.sourceforge.net>  
>>  
>>Is anyone here using NumPy on Opteron machines running Linux in 64 bit    
>>mode? I am running into problems such as:  
>>  
>>Python 2.4 (#4, Jan 18 2005, 18:06:45)  
>>[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2  
>>Type "help", "copyright", "credits" or "license" for more information.  
>> >>> from Numeric import *  
>> >>> a = arange(4)  
>> >>> a  
>>array([0, 1, 2, 3])  
>> >>> a.shape = (2, 2)  
>>Traceback (most recent call last):  
>>   File "<stdin>", line 1, in ?  
>>ValueError: total size of new array must be unchanged  
>>  
>>This is with Python 2.4 and Numeric 23.7. Before looking into this    
>>myself, I thought I'd ask here, perhaps someone has already found a    
>>fix.  
>>  
>>Konrad.  
>>--  
>>------------------------------------------------------------------------   
>>-------  
>>Konrad Hinsen  
>>Laboratoire Leon Brillouin, CEA Saclay,  
>>91191 Gif-sur-Yvette Cedex, France  
>>Tel.: +33-1 69 08 79 25  
>>Fax: +33-1 69 08 82 61  
>>E-Mail: khinsen at cea.fr  
>>------------------------------------------------------------------------   
>>-------  
>>  
>>  
>>  
>>-------------------------------------------------------  
>>SF email is sponsored by - The IT Product Guide  
>>Read honest & candid reviews on hundreds of IT Products from real users.  
>>Discover which products truly live up to the hype. Start reading now.  
>>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click  
>>_______________________________________________  
>>Numpy-discussion mailing list  
>>Numpy-discussion at lists.sourceforge.net  
>>https://lists.sourceforge.net/lists/listinfo/numpy-discussion  
>  
> 
> 
>------------------------------------------------------- 
>SF email is sponsored by - The IT Product Guide 
>Read honest & candid reviews on hundreds of IT Products from real users. 
>Discover which products truly live up to the hype. Start reading now. 
>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click 
>_______________________________________________ 
>Numpy-discussion mailing list 
>Numpy-discussion at lists.sourceforge.net 
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion 
 



From cookedm at physics.mcmaster.ca  Tue Feb  8 08:46:11 2005
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Tue Feb  8 08:46:11 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>
References: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>
Message-ID: <20050208164424.GA1341@arbutus.physics.mcmaster.ca>

On Tue, Feb 08, 2005 at 12:44:18PM +0100, konrad.hinsen at laposte.net wrote:
> Is anyone here using NumPy on Opteron machines running Linux in 64 bit  
> mode? I am running into problems such as:
> 
> Python 2.4 (#4, Jan 18 2005, 18:06:45)
> [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from Numeric import *
> >>> a = arange(4)
> >>> a
> array([0, 1, 2, 3])
> >>> a.shape = (2, 2)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> ValueError: total size of new array must be unchanged
> 
> This is with Python 2.4 and Numeric 23.7. Before looking into this  
> myself, I thought I'd ask here, perhaps someone has already found a  
> fix.

Yes; I submitted a patch (#1108739) to the Numeric patch tracker last week.

Basically, PyArray_Reshape takes an array of dimensions. That has to be
a PyArray_INT (C ints) instead of the PyArray_LONG (C longs) that it has
in the code, because that array is passed to PyArray_FromDimsAndDataAndDescr.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From haase at msg.ucsf.edu  Tue Feb  8 09:36:29 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue Feb  8 09:36:29 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <20050208164424.GA1341@arbutus.physics.mcmaster.ca>
References: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net> <20050208164424.GA1341@arbutus.physics.mcmaster.ca>
Message-ID: <200502080935.47908.haase@msg.ucsf.edu>

(RE: numarray)
Hi, 
On an Opteron (Suse9) I was happy to see that I could allocate the memory for 
an Int32-1024x1024x1024 cube. (On a P4 a get 'MemoryError').
I remember that Todd always said that python itself wasn't 64bit-ready yet.
But why is it needed to support >4GB python-lists if one is only interested in 
large numarrays (or numeric) ?

Just for sake of completeness here is one traceback I got:
>>> import numarray
>>> a = numarray.array(shape=(1024,1024,1024), type=numarray.Int)
>>> a.type()
Int32
>>> a.shape
(1024, 1024, 1024)
>>> len(a)
1024
>>> a
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
numarraycore.py", line 896, in __repr__
    return array_repr(self)
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
numarraycore.py", line 1542, in array_repr
    lst = arrayprint.array2string(
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
arrayprint.py", line 188, in array2string
    separator, prefix)
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
arrayprint.py", line 137, in _array2string
    data = _leading_trailing(a)
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
arrayprint.py", line 111, in _leading_trailing
    l = [_leading_trailing(a[i]) for i in range(
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
arrayprint.py", line 111, in _leading_trailing
    l = [_leading_trailing(a[i]) for i in range(
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
arrayprint.py", line 105, in _leading_trailing
    b = _gen.concatenate((a[:_summaryEdgeItems],
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
generic.py", line 1061, in concatenate
    return _concat(arrs)
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
generic.py", line 1051, in _concat
    dest[ix:ix+_shape0(a)]._copyFrom(a)
numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 
buffersize=0

Thanks 
Sebastian Haase




On Tuesday 08 February 2005 08:44 am, David M. Cooke wrote:
> On Tue, Feb 08, 2005 at 12:44:18PM +0100, konrad.hinsen at laposte.net wrote:
> > Is anyone here using NumPy on Opteron machines running Linux in 64 bit
> > mode? I am running into problems such as:
> >
> > Python 2.4 (#4, Jan 18 2005, 18:06:45)
> > [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> >
> > >>> from Numeric import *
> > >>> a = arange(4)
> > >>> a
> >
> > array([0, 1, 2, 3])
> >
> > >>> a.shape = (2, 2)
> >
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in ?
> > ValueError: total size of new array must be unchanged
> >
> > This is with Python 2.4 and Numeric 23.7. Before looking into this
> > myself, I thought I'd ask here, perhaps someone has already found a
> > fix.
>
> Yes; I submitted a patch (#1108739) to the Numeric patch tracker last week.
>
> Basically, PyArray_Reshape takes an array of dimensions. That has to be
> a PyArray_INT (C ints) instead of the PyArray_LONG (C longs) that it has
> in the code, because that array is passed to
> PyArray_FromDimsAndDataAndDescr.



From konrad.hinsen at laposte.net  Tue Feb  8 09:37:32 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Tue Feb  8 09:37:32 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <20050208164424.GA1341@arbutus.physics.mcmaster.ca>
References: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net> <20050208164424.GA1341@arbutus.physics.mcmaster.ca>
Message-ID: <ba6f727b306b3556c9bf6044083602d4@laposte.net>

On 08.02.2005, at 17:44, David M. Cooke wrote:

> Yes; I submitted a patch (#1108739) to the Numeric patch tracker last  
> week.

Thanks, that patch works fine for me as well.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From haase at msg.ucsf.edu  Tue Feb  8 09:39:20 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue Feb  8 09:39:20 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502080928360.16300@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <42084A1F.3030007@ee.byu.edu> <Pine.LNX.4.60.0502080928360.16300@gate.nmr.mgh.harvard.edu>
Message-ID: <200502080937.59199.haase@msg.ucsf.edu>

Hi,
(just to say something)
I hope "Numeric3" will (magicly) inspire someone to make small-numarray 
creation fast. I really think record-arrays and memmap-arrays of numarray 
should be mentioned more often, just to make Numeric-people realize what they 
miss ;-)

Thanks for "Numerical Python"
Sebastian Haase



From Chris.Barker at noaa.gov  Tue Feb  8 10:28:47 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Tue Feb  8 10:28:47 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42084A1F.3030007@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu>
Message-ID: <42090476.7080604@noaa.gov>

Travis Oliphant wrote:

> Very loosely a matlab or IDL-like environment.  Nobody has really 
> defined what it means to be a MATLAB-like environment. 

Well, I HOPE it means:

An environment in which you can do most of what you can do with MATLAB, 
just as easily, but with a more pythonic approach. I wanted a MATLAB 
like language, I'd use MATLAB (or maybe Octave or Psilab).

> from scipy import *

And I'd like to see this deprecated.

> makes a lot of command-like interaction possible.

I think command-line interaction is quite possible without import *. I'm 
actually kind of disappointed in Numeric, from this perspective. I 
always used to use "from Numeric import *", because Numeric was designed 
to be used this way, which I think is a shame. Far more should be 
methods, rather than functions. For example, there are a bunch of 
functions that take an array as the first argument:

Numeric.take(a, i)

Why in the world is this not:

a.take(i)

If it (and many other functions) were methods, there would be far less 
need for "import *" (I do use import "Numeric as N" to save typing)

In the docs, for many years, there has been the explanation that the 
above approach allows people to use any python sequence, rather than 
just arrays, with these functions. I find that that buys me very little. 
The example given in the docs (a few years ago, is it still there?) is 
transpose:

at = Numeric.transpose(a)

can be passed a Python sequence that has an appropriate structure. 
However, internally, what's going on is something like:

b = Numeric.asarray(a)
at = Numeric.transpose(b)

Is it really so hard to write that extra line? Remember that transpose() 
always returns a NumPy array, regardless of the type of the input. How 
often are you using arbitrary python sequences anyway? In my case, I am 
never using arbitrary python sequences unless I'm writing a function or 
method for an external API, in which case, I almost always put an 
Numeric.asarray() call in at the top.

Well, this turned out to be too much of a rant, but the basic gist was: 
let's make SciPy Pythonic, rather than Matlaby. And the same goes for 
Numeric3.

-Chris















   Is this seen as
> MATLAB-like?
> 
> I guess right now, scipy is trying to bring lots of modules under a 
> common naming convention.
> 
> I think the goals of scipy are very much what people are talking about 
> here.   SciPy ultimately becomes what its users and developers want it 
> to become.   That is why I'm so unclear as to why we have not received 
> more support in its development --- even in the way of ideas as to how 
> to make it better --- until the great ideas that have been suggested on 
> this list recently.
> 
> -Travis
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From haase at msg.ucsf.edu  Tue Feb  8 10:53:38 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue Feb  8 10:53:38 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis" returns 'unexpected' type
Message-ID: <200502081052.40692.haase@msg.ucsf.edu>

Hi,
from the numarray documentation:
<doc>
concatenate(arrs, axis=0)
Returns a new array containing copies of the data contained in all arrays # of 
arrs= (a0, a1, ... an). The arrays ai will be concatenated along the 
specified axis (default=0). All arrays ai must have the same shape along 
every axis except for the one specified in axis. To concatenate arrays along 
a newly created axis, you can use array((a0, ..., an)), as long as all arrays 
have the same shape. 
</doc>

Is it a bug that array((a0,a1)) returns different type that a0 ? 
(type(a0)==type(a1))

>>> a = na.zeros((3,3),na.UInt16)
>>> b = na.zeros((3,3),na.UInt16)
>>> na.concatenate((a,b))
[[0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]]
>>> _.type()
UInt16
>>> na.array((a,b))
[[[0 0 0]
  [0 0 0]
  [0 0 0]]

 [[0 0 0]
  [0 0 0]
  [0 0 0]]]
>>> _.type()
Int32



Thanks,
Sebastian Haase



From cjw at sympatico.ca  Tue Feb  8 10:56:23 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Tue Feb  8 10:56:23 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42090476.7080604@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov>
Message-ID: <42090AE9.7060709@sympatico.ca>

Chris Barker wrote:

> Travis Oliphant wrote:
>
>> Very loosely a matlab or IDL-like environment.  Nobody has really 
>> defined what it means to be a MATLAB-like environment. 
>
>
> Well, I HOPE it means:
>
> An environment in which you can do most of what you can do with 
> MATLAB, just as easily, but with a more pythonic approach. I wanted a 
> MATLAB like language, I'd use MATLAB (or maybe Octave or Psilab).
>
>> from scipy import *
>
>
> And I'd like to see this deprecated.
>
>> makes a lot of command-like interaction possible.
>
>
> I think command-line interaction is quite possible without import *. 
> I'm actually kind of disappointed in Numeric, from this perspective. I 
> always used to use "from Numeric import *", because Numeric was 
> designed to be used this way, which I think is a shame. Far more 
> should be methods, rather than functions. For example, there are a 
> bunch of functions that take an array as the first argument:
>
> Numeric.take(a, i)
>
> Why in the world is this not:
>
> a.take(i)
>
> If it (and many other functions) were methods, there would be far less 
> need for "import *" (I do use import "Numeric as N" to save typing)

Yes, much better.  I use import numarray as _n.  The thinking being that 
there are likely other modules to be imported eg.
import numarray.random as _r.

These other modules no longer need to be concerned with the identifiers 
used in numarray.

>
> In the docs, for many years, there has been the explanation that the 
> above approach allows people to use any python sequence, rather than 
> just arrays, with these functions. I find that that buys me very 
> little. The example given in the docs (a few years ago, is it still 
> there?) is transpose:
>
> at = Numeric.transpose(a)

Instead, why not a.T, as used in PyMatrix, with some help from the class 
structure of NumArray.

Colin W.

>
> can be passed a Python sequence that has an appropriate structure. 
> However, internally, what's going on is something like:
>
> b = Numeric.asarray(a)
> at = Numeric.transpose(b)
>
> Is it really so hard to write that extra line? Remember that 
> transpose() always returns a NumPy array, regardless of the type of 
> the input. How often are you using arbitrary python sequences anyway? 
> In my case, I am never using arbitrary python sequences unless I'm 
> writing a function or method for an external API, in which case, I 
> almost always put an Numeric.asarray() call in at the top.
>
> Well, this turned out to be too much of a rant, but the basic gist 
> was: let's make SciPy Pythonic, rather than Matlaby. And the same goes 
> for Numeric3.
>
> -Chris
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>   Is this seen as
>
>> MATLAB-like?
>>
>> I guess right now, scipy is trying to bring lots of modules under a 
>> common naming convention.
>>
>> I think the goals of scipy are very much what people are talking 
>> about here.   SciPy ultimately becomes what its users and developers 
>> want it to become.   That is why I'm so unclear as to why we have not 
>> received more support in its development --- even in the way of ideas 
>> as to how to make it better --- until the great ideas that have been 
>> suggested on this list recently.
>>
>> -Travis
>>
>>
>>
>> -------------------------------------------------------
>> SF email is sponsored by - The IT Product Guide
>> Read honest & candid reviews on hundreds of IT Products from real users.
>> Discover which products truly live up to the hype. Start reading now.
>> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>




From jmiller at stsci.edu  Tue Feb  8 11:10:10 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Tue Feb  8 11:10:10 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis"
	returns 'unexpected' type
In-Reply-To: <200502081052.40692.haase@msg.ucsf.edu>
References: <200502081052.40692.haase@msg.ucsf.edu>
Message-ID: <1107889739.26021.128.camel@halloween.stsci.edu>

On Tue, 2005-02-08 at 13:52, Sebastian Haase wrote:
> Hi,
> from the numarray documentation:
> <doc>
> concatenate(arrs, axis=0)
> Returns a new array containing copies of the data contained in all arrays # of 
> arrs= (a0, a1, ... an). The arrays ai will be concatenated along the 
> specified axis (default=0). All arrays ai must have the same shape along 
> every axis except for the one specified in axis. To concatenate arrays along 
> a newly created axis, you can use array((a0, ..., an)), as long as all arrays 
> have the same shape. 
> </doc>
> 
> Is it a bug that array((a0,a1)) returns different type that a0 ? 

Yes.  That's not the behavior I see in numarray-CVS though.  CVS looks
OK.

Regards,
Todd





From konrad.hinsen at laposte.net  Tue Feb  8 11:22:49 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Tue Feb  8 11:22:49 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42090476.7080604@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov>
Message-ID: <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net>

On 08.02.2005, at 19:27, Chris Barker wrote:

> Well, I HOPE it means:
>
> An environment in which you can do most of what you can do with  
> MATLAB, just as easily, but with a more pythonic approach. I wanted a  
> MATLAB like language, I'd use MATLAB (or maybe Octave or Psilab).

Me too!

> However, internally, what's going on is something like:
>
> b = Numeric.asarray(a)
> at = Numeric.transpose(b)
>
> Is it really so hard to write that extra line? Remember that  
> transpose()

For interactive work, yes. In particular since arrays have no special  
input notation, you have to type array(...) every time.

> always returns a NumPy array, regardless of the type of the input. How  
> often are you using arbitrary python sequences anyway? In my case, I  
> am

Much less now than in the early days of NumPy, when there was little  
existing code that worked with arrays for obvious reasons. I had a  
small library of numerical operations using lists from the era before  
NumPy, so I was quite happy to be able to interface this easily. But  
that was 10 years ago, none of that code is left unchanged by now.

Another argument for function style code is habit. I suppose few of us  
would want to write x.sin() instead of sin(x). The habits are probably  
less strong for array structure operations, but still for many of us  
functions are more familiar for mathematical stuff.

But I agree that syntactic sugar should be optional and not influence  
basic library design. It is perfectly possible to have an array method  
take() and in addition a syntactic sugar function from some optional  
module for the people who prefer that.

> Well, this turned out to be too much of a rant, but the basic gist  
> was: let's make SciPy Pythonic, rather than Matlaby. And the same goes  
> for Numeric3.

+1

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From strang at nmr.mgh.harvard.edu  Tue Feb  8 11:48:23 2005
From: strang at nmr.mgh.harvard.edu (Gary Strangman)
Date: Tue Feb  8 11:48:23 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
 <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
 <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov>
 <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net>
Message-ID: <Pine.LNX.4.60.0502081435450.16300@gate.nmr.mgh.harvard.edu>

>> b = Numeric.asarray(a)
>> at = Numeric.transpose(b)
>> 
>> Is it really so hard to write that extra line? Remember that transpose()
>
> For interactive work, yes. In particular since arrays have no special input 
> notation, you have to type array(...) every time.

Konrad hits the nail on the head, IMHO. I use interactive mode ALL the 
time (as would many converts from, e.g., Matlab). Lots of additional 
typing would greatly limit the usefulness of interactive mode and, for me, 
discourage use of python for numerics.

> Another argument for function style code is habit. I suppose few of us would 
> want to write x.sin() instead of sin(x). The habits are probably less strong 
> for array structure operations, but still for many of us functions are more 
> familiar for mathematical stuff.

And, if there's any desire to attract folks from (e.g.) Matlab (maybe 
there isn't), a function-style approach is needed. This wouldn't have to 
be the *only* implemented approach, but I would strongly (+10) favor 
having the function notation available. 'course, I majored in math, so 
sin(x) is the only notation I'd recognize. ;-)

-best
Gary



From haase at msg.ucsf.edu  Tue Feb  8 12:06:34 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue Feb  8 12:06:34 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis" returns 'unexpected' type
In-Reply-To: <1107889739.26021.128.camel@halloween.stsci.edu>
References: <200502081052.40692.haase@msg.ucsf.edu> <1107889739.26021.128.camel@halloween.stsci.edu>
Message-ID: <200502081205.22665.haase@msg.ucsf.edu>

On Tuesday 08 February 2005 11:08 am, you wrote:
> On Tue, 2005-02-08 at 13:52, Sebastian Haase wrote:
> > Hi,
> > from the numarray documentation:
> > <doc>
> > concatenate(arrs, axis=0)
> > Returns a new array containing copies of the data contained in all arrays
> > # of arrs= (a0, a1, ... an). The arrays ai will be concatenated along the
> > specified axis (default=0). All arrays ai must have the same shape along
> > every axis except for the one specified in axis. To concatenate arrays
> > along a newly created axis, you can use array((a0, ..., an)), as long as
> > all arrays have the same shape.
> > </doc>
> >
> > Is it a bug that array((a0,a1)) returns different type that a0 ?
>
> Yes.  That's not the behavior I see in numarray-CVS though.  CVS looks
> OK.

Thanks for the info, but after updating from CVS I get a problem in nd_image:
Python 2.2.1 (#1, Feb  3 2005, 06:16:31)
[GCC 2.95.4 20011002 (Debian prerelease)] on linux2
<snip>
  File "<snip>/numarray/nd_image/_ni_support.py", line 35, in ?
    _type_to_num = {
AttributeError: 'module' object has no attribute 'Any'

After changing:
...
import numarray.numerictypes as numarrayt
...
numarrayt.Any
I then complained about 
  File "<snip>/numarray/nd_image/_ni_support.py", line 53, in ?
    _type_to_num[numarray.UInt64] = 9
AttributeError: 'module' object has no attribute 'UInt64'


Any hint ?

Thanks,
Sebastian Haase



From verveer at embl.de  Tue Feb  8 12:47:36 2005
From: verveer at embl.de (Peter Verveer)
Date: Tue Feb  8 12:47:36 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis" returns 'unexpected' type
In-Reply-To: <200502081205.22665.haase@msg.ucsf.edu>
References: <200502081052.40692.haase@msg.ucsf.edu> <1107889739.26021.128.camel@halloween.stsci.edu> <200502081205.22665.haase@msg.ucsf.edu>
Message-ID: <110b816dd825e0d4d513628301fe28fb@embl.de>

> Thanks for the info, but after updating from CVS I get a problem in 
> nd_image:
> Python 2.2.1 (#1, Feb  3 2005, 06:16:31)
> [GCC 2.95.4 20011002 (Debian prerelease)] on linux2
> <snip>
>   File "<snip>/numarray/nd_image/_ni_support.py", line 35, in ?
>     _type_to_num = {
> AttributeError: 'module' object has no attribute 'Any'
>
> After changing:
> ...
> import numarray.numerictypes as numarrayt
> ...
> numarrayt.Any
> I then complained about
>   File "<snip>/numarray/nd_image/_ni_support.py", line 53, in ?
>     _type_to_num[numarray.UInt64] = 9
> AttributeError: 'module' object has no attribute 'UInt64'

Strange, I implemented this yesterday, and it works fine in my setup 
(OS X). I would think you should be able to access numarray.Any and 
numarray.UInt64 without import numarray.numerictypes. Maybe Todd can 
say what the behaviour of numarray should be here.

Peter




From jmiller at stsci.edu  Tue Feb  8 13:12:34 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Tue Feb  8 13:12:34 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis"
	returns 'unexpected' type
In-Reply-To: <110b816dd825e0d4d513628301fe28fb@embl.de>
References: <200502081052.40692.haase@msg.ucsf.edu>
	 <1107889739.26021.128.camel@halloween.stsci.edu>
	 <200502081205.22665.haase@msg.ucsf.edu>
	 <110b816dd825e0d4d513628301fe28fb@embl.de>
Message-ID: <1107897050.26021.150.camel@halloween.stsci.edu>

On Tue, 2005-02-08 at 15:45, Peter Verveer wrote:
> > Thanks for the info, but after updating from CVS I get a problem in 
> > nd_image:
> > Python 2.2.1 (#1, Feb  3 2005, 06:16:31)
> > [GCC 2.95.4 20011002 (Debian prerelease)] on linux2
> > <snip>
> >   File "<snip>/numarray/nd_image/_ni_support.py", line 35, in ?
> >     _type_to_num = {
> > AttributeError: 'module' object has no attribute 'Any'
> >
> > After changing:
> > ...
> > import numarray.numerictypes as numarrayt
> > ...
> > numarrayt.Any
> > I then complained about
> >   File "<snip>/numarray/nd_image/_ni_support.py", line 53, in ?
> >     _type_to_num[numarray.UInt64] = 9
> > AttributeError: 'module' object has no attribute 'UInt64'
> 
> Strange, I implemented this yesterday, and it works fine in my setup 
> (OS X). I would think you should be able to access numarray.Any and 
> numarray.UInt64 without import numarray.numerictypes. Maybe Todd can 
> say what the behaviour of numarray should be here.

Unfortunately, no.  I don't have access to Debian and AFIK UInt64 is
invariantly supported except for MSVC6/win32.  Try this:

>>> import numarray.numinclude as numinc
>>> numinc.hasUInt64
1

If it says 0,  it's probably a Debian thing.

If hasUInt64 is 1,  I'd suggest deleting your site-packages/numarray
directory and re-installing numarray.

Regards,
Todd






From rkern at ucsd.edu  Tue Feb  8 13:12:36 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Tue Feb  8 13:12:36 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4207D285.70103@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu> <4207D285.70103@noaa.gov>
Message-ID: <420817D3.1000500@ucsd.edu>

Chris Barker wrote:

> If there is a small group of folks that would like to work on making a
> binary SciPy for OS-X, I'd be glad to work with others to do it. I doubt
> I'll do it myself, however.

Workin' on it.

http://www.scipy.org/wikis/featurerequests/MacEnthon

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Tue Feb  8 13:30:35 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue Feb  8 13:30:35 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
Message-ID: <42092F3F.1010707@colorado.edu>

Gary Strangman wrote:

> In sum, I pretty much agree with most previous contributors. With one 
> exception. I do agree with Konrad that, theory, the best analysis approach 
> is to build a custom class for each domain of investigation. I've even 
> done that a couple times. However, of the dozens of researchers (i.e., 
> users, not developers) I've talked about this with, almost none have the 
> time (or even desire) to design and develop a class-based approach. Their 
> goal is data analysis and evaluation. So, they pull the data into python 
> (or, more typically, matlab or IDL), analyze it, plot the results, try to 
> get a plot that's pretty enough for publication, and then go on to the 
> next project ... no time for deciding what functionality should be a 
> method or an attribute, subclassing hierarchies, etc. In those rare 
> cases where they have the desire, matlab (I don't know about IDL) doesn't 
> give you the option, meaning that users (as opposed to developers) 
> probably won't go the (unfamiliar and more time consuming) route.

I have to say that in most cases, the approach you describe from colleagues is 
an unfortunately common, but still very shortsighted one (except in cases so 
trivial that it doesn't really matter).  I'm attaching a real-world example of 
a 3d Poisson solver which uses a new algorithm combining some neat ideas.  The 
papers are being only written right now, so the only reference I can offer is 
this:

http://amath.colorado.edu/faculty/fperez/talks/0403_ams_athens.pdf

The point is that anyone should be able to read that script and understand 
what it does.  A few things worth pointing out from that code:

print "Building rho (Right Hand Side)..."
rho = FunctionFromCode(nnod,cutoff,rho_code,norm_type)

print "Building phi (Left Hand Side), analytical solution..."
phi_exact = FunctionFromCode(nnod,cutoff,phi_exact_code,norm_type)

This builds multiwavelet, adaptively decomposed objects which are quite 
complex internally, yet they behave like functions.  Likewise, this line:

print "Computing phi = poisson(rho) explicitly..."
phi  = poisson(rho)

returns another function (even though this one was constructed as a numerical 
solution instead of having an analytical representation).  So checking for 
validity, in this case where we have the exact anwer, is as simple as:

print "Building error function..."
error = phi - phi_exact
enorm = error.norm()

This subtracts two multiwavelet-represented functions in 3d, and returns 
another function of the same kind.  Furthermore, all these objects have 
plot/info methods which you can then use to visualize info or print relevant 
numerical stuff:

print "Norm info (L-%s):" % norm_type
print "rho    = %.3e" % rho.norm()
print "phi    = %.3e" % phi.norm()
print "error  = %.3e" % enorm

Plotting is equally easy:

# Surface plots across z=0.5 for rho and phi, uses MayaVi
rho.plot_surf()
phi.plot_surf()

# Interpolation error for the multiwavelet representation of the rhs/lhs
rho.plot_line(ptype='error',title='Interpolation error for charge density')
phi_exact.plot_line(ptype='error',
                     title='Interpolation error for exact solution')

# Pointwise interpolation error in the Green function's kernel
poisson.plot_kernel_error()

The point is, by making these objects 'smart', the top-level code reads like a 
regular mathematical solution of Poisson's equation, and extracting useful 
info is trivial.  And I can guarantee that in the long run, I've saved myself 
A LOT of time by writing the code in this way, rather than spitting out arrays 
left, right and center, and calling manual plot routines on them all the time. 
  When I need plots for a talk or paper, there is nothing more to do, just 
call the objects.

I should add that having this kind of design in scientific codes makes 
interactive exploration and debugging a very pleasant process: whenever 
something doesn't look right, my objects have a ton of self-diagnostic methods 
which I can quickly use to pinpoint the problem.  And this is not a 
theoretical statement, I have already many times saved myself much grief by 
being able to instantly see where the problem is, thanks to this kind of design.

So while I agree that having plain array access is needed for quick and dirty 
one-off things, basically anything beyond 50-100 lines of code is probably 
best written with a tiny bit of design.  Those 10 minutes invested thinking 
about a design sketch will pay for themselves many times over.

Regards,

f
-------------- next part --------------
A non-text attachment was scrubbed...
Name: demo_poisson_3d.py
Type: application/x-python
Size: 3596 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050208/2ba88da0/attachment.bin>

From jmiller at stsci.edu  Tue Feb  8 13:30:38 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Tue Feb  8 13:30:38 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis"
	returns 'unexpected' type
In-Reply-To: <110b816dd825e0d4d513628301fe28fb@embl.de>
References: <200502081052.40692.haase@msg.ucsf.edu>
	 <1107889739.26021.128.camel@halloween.stsci.edu>
	 <200502081205.22665.haase@msg.ucsf.edu>
	 <110b816dd825e0d4d513628301fe28fb@embl.de>
Message-ID: <1107898168.26021.171.camel@halloween.stsci.edu>

On Tue, 2005-02-08 at 15:45, Peter Verveer wrote:
> > Thanks for the info, but after updating from CVS I get a problem in 
> > nd_image:
> > Python 2.2.1 (#1, Feb  3 2005, 06:16:31)
> > [GCC 2.95.4 20011002 (Debian prerelease)] on linux2
> > <snip>
> >   File "<snip>/numarray/nd_image/_ni_support.py", line 35, in ?
> >     _type_to_num = {
> > AttributeError: 'module' object has no attribute 'Any'
> >
> > After changing:
> > ...
> > import numarray.numerictypes as numarrayt
> > ...
> > numarrayt.Any
> > I then complained about
> >   File "<snip>/numarray/nd_image/_ni_support.py", line 53, in ?
> >     _type_to_num[numarray.UInt64] = 9
> > AttributeError: 'module' object has no attribute 'UInt64'
> 
> Strange, I implemented this yesterday, and it works fine in my setup 
> (OS X). I would think you should be able to access numarray.Any and 
> numarray.UInt64 without import numarray.numerictypes. Maybe Todd can 
> say what the behaviour of numarray should be here.
> 
> Peter

I looked at this some more after getting a clean checkout and discovered
that there's a problem in numarray-CVS (Python-2.2 only) introduced by a
recent bugfix.  I didn't see it because I don't normally test with 2.2.

It's fixed now in CVS,  modulo incomplete bool support for 2.2.

Regards,
Todd






From Chris.Barker at noaa.gov  Tue Feb  8 14:34:32 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Tue Feb  8 14:34:32 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502081435450.16300@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov> <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net> <Pine.LNX.4.60.0502081435450.16300@gate.nmr.mgh.harvard.edu>
Message-ID: <42093E37.9070501@noaa.gov>

Gary Strangman wrote:

> Konrad hits the nail on the head, IMHO. I use interactive mode ALL the 
> time (as would many converts from, e.g., Matlab). Lots of additional 
> typing would greatly limit the usefulness of interactive mode and, for 
> me, discourage use of python for numerics.

We all have different styles, but I was a user of Matlab for years, and 
maybe it's because I'm not a very fast typist, but when I found myself 
writing more than three lines at the prompt, I'd kick myself, and open 
an editor, and write a script. The one thing I found a little lacking 
when I started using python was an easy way to run a script inside an 
interactive session:

execfile("filename.py")

is a bit clunky. However:

$ python -i filename.py

Is not so bad at the command prompt, when all you have to do is hit the 
up-arrow key and run it again.

>> I suppose few of us 
>> would want to write x.sin() instead of sin(x).

Well, that's true for me also, but for structural array stuff, methods 
seem to make more sense. Besides, I'd get used to x.sin(), and like it 
if it kept my namespaces uncluttered!

as for the array() literal: how often do you really need to put in your 
arrays as a literal? I usually calculate, load from a file, generate 
randomly, or use arange(). And if I do type in an array literal, the 
extra typing of "array" is pretty small compared to typing all those 
numbers!

> And, if there's any desire to attract folks from (e.g.) Matlab (maybe 
> there isn't), a function-style approach is needed. This wouldn't have to 
> be the *only* implemented approach, but I would strongly (+10) favor 
> having the function notation available.

This is the approach taken by John with matplotlib. Inside, everything 
is all OO style, but with the "pylab" module creating a bunch of 
matlab-style helper functions. In fact, it's "officially" recommended 
that you use the pylab interface for interactive use, and the OO 
interface for "programming". Unfortunately, the OO interface lacks a 
number of useful features that really has nothing to do with OO vs. 
functional, it's just that the focus has been on easy usability for the 
pylab interface, but not for the OO interface. I've been working to 
change that (mostly just by complaining, but I intend to contribute code 
soon: I promise, really!)

So, in short, I'd like to see both: an internal OO structure, with a 
"interactive" helper module.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From Fernando.Perez at colorado.edu  Tue Feb  8 14:40:35 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue Feb  8 14:40:35 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42093E37.9070501@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov> <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net> <Pine.LNX.4.60.0502081435450.16300@gate.nmr.mgh.harvard.edu> <42093E37.9070501@noaa.gov>
Message-ID: <42093FAF.8050602@colorado.edu>

Chris Barker wrote:

> We all have different styles, but I was a user of Matlab for years, and 
> maybe it's because I'm not a very fast typist, but when I found myself 
> writing more than three lines at the prompt, I'd kick myself, and open 
> an editor, and write a script. The one thing I found a little lacking 
> when I started using python was an easy way to run a script inside an 
> interactive session:
> 
> execfile("filename.py")
> 
> is a bit clunky. However:
> 
> $ python -i filename.py
> 
> Is not so bad at the command prompt, when all you have to do is hit the 
> up-arrow key and run it again.

I know it's bad manners to promote your own stuff, but have you had a look at 
ipython?  The run command in ipython gives you this, plus much, much more 
(debugger access on exceptions, colored tracebacks with detail control, 
automatic profiling, namespace control,...).  My workmode is Xemacs open with 
the files + ipython using 'run foo' to rerun my foo.py script as many times as 
needed.

Sorry for the plug, but since I've spent 3 years tuning ipython precisely to 
make interactive numerical work as pleasant and efficient as possible, I can't 
  resist :)

Regards,

f



From juenglin at cs.pdx.edu  Tue Feb  8 16:42:26 2005
From: juenglin at cs.pdx.edu (Ralf Juengling)
Date: Tue Feb  8 16:42:26 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42090476.7080604@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
	 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
	 <4207CC4C.8060603@ee.byu.edu>
	 <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
	 <42084A1F.3030007@ee.byu.edu>  <42090476.7080604@noaa.gov>
Message-ID: <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>

On Tue, 2005-02-08 at 10:27, Chris Barker wrote:
> Travis Oliphant wrote:
> > from scipy import *
> 
> And I'd like to see this deprecated.

I don't understand what's wrong with this. (Of course, the module
you do the "from XXX import *" with should be designed for this).

This is on of the hallmarks of Python that facilitate rapid 
development: When you want to get familiar with an unfamiliar 
library module, start up python in a shell, do the 
"from XXX import *" thing, and start playing.

Name space pollution is not an issue, because when you write 
programs, you don't do "from XXX import *". 

ralf





From Fernando.Perez at colorado.edu  Tue Feb  8 16:50:11 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue Feb  8 16:50:11 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>	 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>	 <4207CC4C.8060603@ee.byu.edu>	 <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>	 <42084A1F.3030007@ee.byu.edu>  <42090476.7080604@noaa.gov> <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
Message-ID: <42095E13.2020002@colorado.edu>

Ralf Juengling wrote:

> This is on of the hallmarks of Python that facilitate rapid 
> development: When you want to get familiar with an unfamiliar 
> library module, start up python in a shell, do the 
> "from XXX import *" thing, and start playing.

If I may suggest, it tends to be a lot more efficient to explore an unknown 
library by doing

import XXX_LONGNAME_XXX as X

and start tab-completing away on X.<TAB>, or X.get<TAB> for getters, etc. 
This way, you know you can restrict your completions to only the thing you are 
trying to study, instead of all of its names colliding with builtins and globals.

So if anything, interactive _exploration_ of a library is one place not to use 
'from XXX import *'.

Interactive _use_ is an altogether different story.  I constantly use ipython 
in -pylab mode, which pulls all of matplotlib.pylab, and up until a week ago 
did the same with the supplied numeric profile (which used Numeric and 
Gnuplot).   But this is for using names I already know well.  And I still make 
a point, even in these conditions, of having pylab/Numeric available as 
top-level names, so I can do selective tab-completion on them when I need to 
fish around for something I don't know or whose specific name I can't remember.

Just a small comment.

Best,

f



From stephen.walton at csun.edu  Tue Feb  8 17:00:36 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Tue Feb  8 17:00:36 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42084D42.9050509@colorado.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu>
Message-ID: <42096070.5030608@csun.edu>

Fernando Perez wrote:

> Travis Oliphant wrote:
>
>> Currently, I don't think Python allows "over-riding" the "and" operation.
>
>
> You are correct:

What I want must be possible somehow, though:

In [6]: a=[True,False]

In [7]: b=[False,False]

In [8]: a and b
Out[8]: [False, False]

In [9]: a or b
Out[9]: [True, False]

In [10]: not a
Out[10]: False

So, standard Python does apply logical "and" and "or", but not "not", 
element by element to a list or tuple of booleans.  Is there no way to 
get access to this for a numarray bool array?  Can we get Guido to 
change "not" to also operate element-wise on a list of booleans?

I have a really common task in MATLAB I want to have work here.  In 
MATLAB I can extract the overlapping good values from two data sets by doing

f = data1 ~= flagval & data2 ~= flagval;

or even more compactly, since I use IEEE NaN for flagval

f = ~isnan(data1) & ~isnan(data2);

Right now in numarray I have to use

f = (data1 != flagval) & (data2 != flagval)

with the extra parentheses and, confusingly, using bitwise AND instead 
of logical AND.  Even if I use ieeespecial nan as flagval, there's no 
"notisnan(data1)" and no way to say "!isnan(data1)"

Of course, masked arrays are another option for this kind of usage but 
they have their own difficulties:  matplotlib does not at present allow 
one to plot from masked arrays, for example.






From Chris.Barker at noaa.gov  Tue Feb  8 17:08:19 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Tue Feb  8 17:08:19 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>	 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>	 <4207CC4C.8060603@ee.byu.edu>	 <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>	 <42084A1F.3030007@ee.byu.edu>  <42090476.7080604@noaa.gov> <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
Message-ID: <420961B3.1000405@noaa.gov>

Ralf Juengling wrote:
>>>from scipy import *
>>
>>And I'd like to see this deprecated.
> 
> I don't understand what's wrong with this. (Of course, the module
> you do the "from XXX import *" with should be designed for this).

If it's designed for this, then none of the names will clash with 
built-ins, unless intended. However, what about other modules that might 
also be done as "import *"?

It seems to me that the assumption is that whatever module is imported 
that way will be the only one imported that way. This is not always the 
case. Also, what if you need to import a module that happens to have the 
same name as an attribute of the "import *" one. You can do an "import 
as", but only if you know you have a name clash.

> Name space pollution is not an issue, because when you write 
> programs, you don't do "from XXX import *". 

Ha! That's the core problem here. As an example (I don't mean to be 
picking on you, John) matplotlib docs suggest that you do things 
differently when writing programs than playing with the interpreter. 
However, almost all of the examples and docs, and and most of the 
usability effort has gone into the "import *" approach. The same is true 
for Numeric and numarray. Why would someone magically stop using it? 
Besides, I use the interactive interpreter to try out code that I'm then 
going to use in a program. Why have a different syntax?


"Namespaces are one honking great idea -- let's do more of those!"

Note, that's "more", not "less"

OK, I'll stop ranting now.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From Fernando.Perez at colorado.edu  Tue Feb  8 17:14:29 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue Feb  8 17:14:29 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42096070.5030608@csun.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu> <42096070.5030608@csun.edu>
Message-ID: <42096303.5030805@colorado.edu>

Stephen Walton wrote:
> Fernando Perez wrote:
> 
> 
>>Travis Oliphant wrote:
>>
>>
>>>Currently, I don't think Python allows "over-riding" the "and" operation.
>>
>>
>>You are correct:
> 
> 
> What I want must be possible somehow, though:
> 
> In [6]: a=[True,False]
> 
> In [7]: b=[False,False]
> 
> In [8]: a and b
> Out[8]: [False, False]
> 
> In [9]: a or b
> Out[9]: [True, False]
> 
> In [10]: not a
> Out[10]: False
> 
> So, standard Python does apply logical "and" and "or", but not "not", 
> element by element to a list or tuple of booleans.  Is there no way to 
> get access to this for a numarray bool array?  Can we get Guido to 
> change "not" to also operate element-wise on a list of booleans?

I think you are misunderstanding something.  Python does not "apply logical 
"and" and "or", but not "not",  element by element to a list or tuple of 
booleans", as you say.  Look again at the decompiled bytecode:

In [2]: dis.dis(compile('a and b','<>','eval'))
   0           0 LOAD_NAME                0 (a)
               3 JUMP_IF_FALSE            4 (to 10)
               6 POP_TOP
               7 LOAD_NAME                1 (b)
         >>   10 RETURN_VALUE

What it does is execute the JUMP_IF_FALSE bytecode, which is mapped to the 
__nonzero__ special method.  From http://docs.python.org/ref/customization.html:

__nonzero__(  	self)
     Called to implement truth value testing, and the built-in operation 
bool(); should return False or True, or their integer equivalents 0 or 1. When 
this method is not defined, __len__() is called, if it is defined (see below). 
If a class defines neither __len__() nor __nonzero__(), all its instances are 
considered true.

Now, lists don't actually have this method, so a call to __len__ is made, and 
any non-empty list is considered true.  The return value is then whatever is 
on the stack, which will be either be a, or b if a tested false:

In [6]: a
Out[6]: [True, False]

In [7]: a and 0
Out[7]: 0

In [8]: a and 1
Out[8]: 1

In [9]: a and 'hello'
Out[9]: 'hello'


The point is that and/or only operate on one operand at a time, with unary 
functions/methods.  They do NOT call a binary function, the way & does:

In [5]: dis.dis(compile('a & b','<>','eval'))
   0           0 LOAD_NAME                0 (a)
               3 LOAD_NAME                1 (b)
               6 BINARY_AND
               7 RETURN_VALUE

This means that certain things which can be done with & are fundamentally 
impossible with 'and', since you never have both arguments in your hands at 
the same time.

Bytecode analysis can be really useful to understand this kind of subtle 
issues.  I can't think of any language where this approach is as easy to do 
and friendly as in python.

Cheers,

f



From Fernando.Perez at colorado.edu  Tue Feb  8 17:18:31 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue Feb  8 17:18:31 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42096303.5030805@colorado.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu> <42096070.5030608@csun.edu> <42096303.5030805@colorado.edu>
Message-ID: <42096472.7050208@colorado.edu>

Fernando Perez wrote:

> Now, lists don't actually have this method, so a call to __len__ is made, and 
> any non-empty list is considered true.  The return value is then whatever is 
> on the stack, which will be either be a, or b if a tested false:
                                                             ^^^^^^

That should have read TRUE, of course, for an 'a and b' call.  For 'a or b', 
false is the right word.  Sorry about the confusion.

Cheers,

f



From simon at arrowtheory.com  Tue Feb  8 17:42:44 2005
From: simon at arrowtheory.com (Simon Burton)
Date: Tue Feb  8 17:42:44 2005
Subject: [Numpy-discussion] numarray.codegenerator.UfuncModule nary ufuncs
Message-ID: <20050209124102.4632d57b.simon@arrowtheory.com>

It looks like I may need to make n-ary (multi argument) ufuncs.

The use case is something like this: some tricky function operates on single values
but has many parameters (eg. offset and multiply). So the ufunc is called
with an array as the first argument, but just scalars in the other arguments.

I'm wondering how far is numarray.codegenerator.UfuncModule from handling n-ary functions ?
If it's just not on the radar, i'll need to rethink my approach...

bye for now,

Simon.



From juenglin at cs.pdx.edu  Tue Feb  8 17:58:35 2005
From: juenglin at cs.pdx.edu (Ralf Juengling)
Date: Tue Feb  8 17:58:35 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420961B3.1000405@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
	 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
	 <4207CC4C.8060603@ee.byu.edu>
	 <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
	 <42084A1F.3030007@ee.byu.edu>  <42090476.7080604@noaa.gov>
	 <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
	 <420961B3.1000405@noaa.gov>
Message-ID: <1107914212.28976.34.camel@alpspitze.cs.pdx.edu>

On Tue, 2005-02-08 at 17:04, Chris Barker wrote:
> Ha! That's the core problem here. As an example (I don't mean to be 
> picking on you, John) matplotlib docs suggest that you do things 
> differently when writing programs than playing with the interpreter. 
> However, almost all of the examples and docs, and and most of the 
> usability effort has gone into the "import *" approach. The same is true 
> for Numeric and numarray. Why would someone magically stop using it? 
> Besides, I use the interactive interpreter to try out code that I'm then 
> going to use in a program. Why have a different syntax?

Yes, I think it is absurd to ask a user to learn two APIs, one for
interactive use and one for writing programs.

I'm not sure I understand your stance on this, though. You were
complaining in the other posting that matplotlib's "programming API"
would not offer all of the features that are offered by its 
"interactive API". But you don't seem to be offended by the very 
idea of having two APIs.

Ralf





From jdhunter at ace.bsd.uchicago.edu  Tue Feb  8 18:46:10 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Tue Feb  8 18:46:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1107914212.28976.34.camel@alpspitze.cs.pdx.edu> (Ralf
 Juengling's message of "Tue, 08 Feb 2005 17:56:53 -0800")
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	<42065E31.20906@astraw.com>
	<4ad81937f722051f27b37915225a1563@laposte.net>
	<Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
	<4207CC4C.8060603@ee.byu.edu>
	<Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
	<42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov>
	<1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
	<420961B3.1000405@noaa.gov>
	<1107914212.28976.34.camel@alpspitze.cs.pdx.edu>
Message-ID: <m3ll9yv4tr.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Ralf" == Ralf Juengling <juenglin at cs.pdx.edu> writes:

    >> Ha! That's the core problem here. As an example (I don't mean
    >> to be picking on you, John) matplotlib docs suggest that you do
    >> things differently when writing programs than playing with the
    >> interpreter.  However, almost all of the examples and docs, and
    >> and most of the usability effort has gone into the "import *"
    >> approach. The same is true for Numeric and numarray. Why would
    >> someone magically stop using it?  Besides, I use the
    >> interactive interpreter to try out code that I'm then going to
    >> use in a program. Why have a different syntax?

    Ralf> Yes, I think it is absurd to ask a user to learn two APIs,
    Ralf> one for interactive use and one for writing programs.

Just to make sure I'm not being unfairly represented here :-)

matplotlib has a pylab interface which allows you to make plots from
scripts or from the python shell.  Eg you can do
 
  from pylab import *
  plot([1,2,3])
  xlabel('hi mom')

and a figure window will pop up if you've configured your matplotlib
environment for interactive use with a GUI.  Now a lot happens under
the hood when you make these commands -- in particular, the pylab
interface has the concept of the current figure and axes, and these
are created automagically for you if you haven't explicitly created
them already.  Which means the module is realizing and managing GUI
windows, binding events to them, managing their destruction, and so
on.

What we advise is if you are using matplotlib in a GUI application you
are writing -- eg you are explicitly creating wx/gtk/tk/etc windows
and toolbars and menus -- then you shouldn't use the pylab interface.
The reason is that the pylab interface is trying to create and manage
GUI windows, and you might get into a conflict if you are also trying
to create and manage GUI windows.  So as a GUI application developer,
you are responsible for these things and use the matplotlib OO API --
the pylab interface is a thin wrapper to this.

I think the misleading statement is

    >> you do things differently when writing programs than playing
    >> with the interpreter

it would be accurate to say you should not use the pylab interface
when manually embedding matplotlib in a GUI, but you are free to use
the pylab interface or the OO API in python scripts or from the python
shell.

And now we have wandered dangerously off-topic...

JDH



From konrad.hinsen at laposte.net  Wed Feb  9 02:47:28 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Wed Feb  9 02:47:28 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42093E37.9070501@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov> <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net> <Pine.LNX.4.60.0502081435450.16300@gate.nmr.mgh.harvard.edu> <42093E37.9070501@noaa.gov>
Message-ID: <70144ea5aefd645d56d65ad9400575d6@laposte.net>

On Feb 8, 2005, at 23:33, Chris Barker wrote:

> thing I found a little lacking when I started using python was an easy 
> way to run a script inside an interactive session:
>
> execfile("filename.py")
>
> is a bit clunky. However:
>
> $ python -i filename.py
>
> Is not so bad at the command prompt, when all you have to do is hit 
> the up-arrow key and run it again.

Try ipython. "%run filename.py" runs the script and you can recall it 
with the up-arrow key. There are lots of other nice features in 
ipython, once you bring up the courage to go through the manual (66 
pages!).

> Well, that's true for me also, but for structural array stuff, methods 
> seem to make more sense. Besides, I'd get used to x.sin(), and like it 
> if it kept my namespaces uncluttered!

I wonder if you would say the same after reading a bunch of complicated 
formulas in Eiffel code (which requires such notation). I did, and 
didn't find it pleasurable. But I would happily accept methods for 
anything but standard math functions.

> So, in short, I'd like to see both: an internal OO structure, with a 
> "interactive" helper module.

We can agree on that!

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From mdehoon at ims.u-tokyo.ac.jp  Wed Feb  9 03:05:34 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Wed Feb  9 03:05:34 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42055FA3.9050909@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	<1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu>	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu>	<c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu>
Message-ID: <4209EE2B.6090605@ims.u-tokyo.ac.jp>

Travis Oliphant wrote:
> Again,  from my perspective once Numeric3 is working reasonably well, 
> Numeric ceases to exist for me.  Yes others may continue to download it, 
> but I won't be maintaining it anymore.
> 
It is certainly fair for SciPy developers to focus on Numeric3/SciPy and 
stop maintaining Numeric. In open-source development, people get to 
choose what they want to work on. And if creating Numeric3 will lead to 
a more useable SciPy, I'm all for it.

 From the discussion on this list, it seems that there are still users 
who will continue to use Numeric for the time being. It would therefore 
be good if some of those users can become Numeric maintainers for the 
benefit of people who choose to continue using Numeric for now. Is 
anybody else interested in becoming a Numeric maintainer? Since the 
focus will be on maintenance rather than code development, I expect the 
workload to be acceptable. Is anybody else in favor of appointing 
Numeric maintainers?

--Michiel.



From konrad.hinsen at laposte.net  Wed Feb  9 03:12:51 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Wed Feb  9 03:12:51 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42092F3F.1010707@colorado.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <42092F3F.1010707@colorado.edu>
Message-ID: <03bdbcfa147243f9e3d1649781101a41@laposte.net>

On Feb 8, 2005, at 22:29, Fernando Perez wrote:

> it doesn't really matter).  I'm attaching a real-world example of a 3d 
> Poisson solver which uses a new algorithm combining some neat ideas.  
> The papers are being only written right now, so the only reference I 
> can offer is this:

Nice!  Stuff like that is the best advertising for Python in science in 
my opinion. I wonder if it is possible to have a collection of 
references in a well visible place.

> The point is, by making these objects 'smart', the top-level code 
> reads like a regular mathematical solution of Poisson's equation, and 
> extracting useful info is trivial.  And I can guarantee that in the 
> long run, I've saved myself A LOT of time by writing the code in this 
> way, rather than spitting out arrays left, right and center, and

Exactly. To use a buzzword, you make a Problem-Solving Environment in 
the form of Python classes.

> I should add that having this kind of design in scientific codes makes 
> interactive exploration and debugging a very pleasant process: 
> whenever something doesn't look right, my objects have a ton of 
> self-diagnostic methods which I can quickly use to pinpoint the 
> problem.  And this is not a theoretical statement, I have already many 
> times saved myself much grief by being able to instantly see where the 
> problem is, thanks to this kind of design.

My experience (and usage style) is exactly the same.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From strang at nmr.mgh.harvard.edu  Wed Feb  9 05:36:10 2005
From: strang at nmr.mgh.harvard.edu (Gary Strangman)
Date: Wed Feb  9 05:36:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42092F3F.1010707@colorado.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
 <42092F3F.1010707@colorado.edu>
Message-ID: <Pine.LNX.4.60.0502090824050.21865@gate.nmr.mgh.harvard.edu>

> So while I agree that having plain array access is needed for quick and dirty 
> one-off things, basically anything beyond 50-100 lines of code is probably 
> best written with a tiny bit of design.  Those 10 minutes invested thinking 
> about a design sketch will pay for themselves many times over.

I guess this points out different needs and abilities. I unfortunately 
need to do way too many things just once (ahhh, if I only had the luxury 
of doing the same or similar analyses over and over). And though I 
certainly agree that including a bit of design can be a good thing, for my 
abilities this usually boils down to more like 10 hours rather than 10 
minutes of designing, re-designing, debugging the design, adding to the 
design etc. I'm usually happy with it when I'm done, but don't get to 
reuse it more than 1 or 2 times.

To get back to the topic at hand, though, I'd be perfectly satisfied with 
both interfaces (functions and methods). I just worry that the method 
interface would make the multiarray object sufficiently complex and 
unwieldy to (a) induce a performance hit, or (b) prevent the multiarray 
object from getting into the python core. But, maybe I just misunderstand 
the nature of the code that would underly the method interface.

-best
Gary



From konrad.hinsen at laposte.net  Wed Feb  9 06:06:49 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Wed Feb  9 06:06:49 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502090824050.21865@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <42092F3F.1010707@colorado.edu> <Pine.LNX.4.60.0502090824050.21865@gate.nmr.mgh.harvard.edu>
Message-ID: <966d28377eda318746cc75e990063144@laposte.net>

On Feb 9, 2005, at 14:35, Gary Strangman wrote:

> interfaces (functions and methods). I just worry that the method 
> interface would make the multiarray object sufficiently complex and 
> unwieldy to (a) induce a performance hit, or (b) prevent the 
> multiarray object from getting into the python core. But, maybe I just 
> misunderstand the nature of the code that would underly the method 
> interface.

Performance-wise there is no difference between methods and functions. 
And I don't see why a method interface would prevent arrays from 
getting into the core, considering that most Python types have a method 
interface. The string handling stuff was even moved from tunction style 
to method style between Python 1.5 and 2.0.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From jmiller at stsci.edu  Wed Feb  9 06:33:46 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Feb  9 06:33:46 2005
Subject: [Numpy-discussion] numarray.codegenerator.UfuncModule nary
	ufuncs
In-Reply-To: <20050209124102.4632d57b.simon@arrowtheory.com>
References: <20050209124102.4632d57b.simon@arrowtheory.com>
Message-ID: <1107959538.5863.43.camel@jaytmiller.comcast.net>

On Wed, 2005-02-09 at 12:41 +1100, Simon Burton wrote:
> It looks like I may need to make n-ary (multi argument) ufuncs.
> 
> The use case is something like this: some tricky function operates on single values
> but has many parameters (eg. offset and multiply). So the ufunc is called
> with an array as the first argument, but just scalars in the other arguments.
> 
> I'm wondering how far is numarray.codegenerator.UfuncModule from handling n-ary functions ?
> If it's just not on the radar, i'll need to rethink my approach...
> 
> bye for now,
> 
> Simon.

The numarray n-ary ufunc system is very definitely on the radar,  but
still alpha.  Support for the kind of work you're trying to do is in the
code generation API but the ufunc runtime system needs more work for it
to be as fast as it can be.  For large arrays it should already be
pretty good.

There's an example of the nary-ufunc code generation in
numarray/Examples/ufunc/setup_special.py.  I added an example ufunc,
vsdemo(), similar in form to what I think you want,  one array input
parameter and 4 vector input parameters yielding a single value result.

Feedback is welcome.

Regards,
Todd




From faltet at carabos.com  Wed Feb  9 07:38:16 2005
From: faltet at carabos.com (Francesc Altet)
Date: Wed Feb  9 07:38:16 2005
Subject: [Numpy-discussion] Efficiently converting a numarray object to Numeric
Message-ID: <200502091637.32976.faltet@carabos.com>

Hi,

I'm wondering if there is a more efficient way to convert a numarray
object into a Numeric one than:

>>> import Numeric
>>> import numarray
>>> from time import time
>>> a = numarray.arange(100*1000)
>>> t1=time();b=Numeric.array(a);time()-t1
0.43448090553283691

Any suggestions?

-- 
>qo<   Francesc Altet ? ? http://www.carabos.com/
V ?V   C?rabos Coop. V. ??Enjoy Data
 ""




From jmiller at stsci.edu  Wed Feb  9 08:28:33 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Feb  9 08:28:33 2005
Subject: [Numpy-discussion] Efficiently converting a numarray object to
	Numeric
In-Reply-To: <200502091637.32976.faltet@carabos.com>
References: <200502091637.32976.faltet@carabos.com>
Message-ID: <1107966462.5863.55.camel@jaytmiller.comcast.net>

On Wed, 2005-02-09 at 16:37 +0100, Francesc Altet wrote:
> Hi,
> 
> I'm wondering if there is a more efficient way to convert a numarray
> object into a Numeric one than:
> 
> >>> import Numeric
> >>> import numarray
> >>> from time import time
> >>> a = numarray.arange(100*1000)
> >>> t1=time();b=Numeric.array(a);time()-t1
> 0.43448090553283691
> 
> Any suggestions?

Yes!  

import Numeric
import numarray
from time import time

a = numarray.arange(100*1000)

t1=time();b=Numeric.array(a);
print time()-t1

t1 = time();  c=Numeric.fromstring(a._data, typecode='i')
print time()-t1

print b[-10:]
print c[-10:]
      
I get:

1.58109998703
0.00659704208374
[99990 99991 99992 99993 99994 99995 99996 99997 99998 99999]
[99990 99991 99992 99993 99994 99995 99996 99997 99998 99999]

Note that in CVS,  fromstring(a, typecode='i') now also works because
numarray now supplies the buffer protocol as well as consuming buffers.

Regards,
Todd






From jmiller at stsci.edu  Wed Feb  9 08:38:13 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Feb  9 08:38:13 2005
Subject: [Numpy-discussion] Efficiently converting a numarray object to
	Numeric
In-Reply-To: <1107966462.5863.55.camel@jaytmiller.comcast.net>
References: <200502091637.32976.faltet@carabos.com>
	 <1107966462.5863.55.camel@jaytmiller.comcast.net>
Message-ID: <1107966977.5863.58.camel@jaytmiller.comcast.net>

On Wed, 2005-02-09 at 11:27 -0500, Todd Miller wrote:
> On Wed, 2005-02-09 at 16:37 +0100, Francesc Altet wrote:
> > Hi,
> > 
> > I'm wondering if there is a more efficient way to convert a numarray
> > object into a Numeric one than:
> > 
> > >>> import Numeric
> > >>> import numarray
> > >>> from time import time
> > >>> a = numarray.arange(100*1000)
> > >>> t1=time();b=Numeric.array(a);time()-t1
> > 0.43448090553283691
> > 
> > Any suggestions?
> 
> Yes!  
> 
> import Numeric
> import numarray
> from time import time
> 
> a = numarray.arange(100*1000)
> 
> t1=time();b=Numeric.array(a);
> print time()-t1
> 
> t1 = time();  c=Numeric.fromstring(a._data, typecode='i')
> print time()-t1
> 
> print b[-10:]
> print c[-10:]
>       
> I get:
> 
> 1.58109998703
> 0.00659704208374
> [99990 99991 99992 99993 99994 99995 99996 99997 99998 99999]
> [99990 99991 99992 99993 99994 99995 99996 99997 99998 99999]
> 
> Note that in CVS,  fromstring(a, typecode='i') now also works because
> numarray now supplies the buffer protocol as well as consuming buffers.
> 

One more thing is necessary to handle a misbehaved numarray and make its
buffer usable by Numeric:

if not a.is_c_array():
	a = a.copy()

Regards,
Todd






From perry at stsci.edu  Wed Feb  9 08:49:36 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Wed Feb  9 08:49:36 2005
Subject: [Numpy-discussion] Proposal for making of Numarray a real Numeric 'NG'
In-Reply-To: <4205C5DE.80504@ee.byu.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCOEANFJAA.perry@stsci.edu>

I thought I would clarify some historical issues (at least as far as
I recall them) in helping understand how things got the way they are 
today.

Travis Oliphant wrote:
> Francesc Altet wrote:
> 
> >  
> >
> I think numarray has made some incrdedible strides in showing what the 
> numeric object needs to be and in implementing some really neat 
> functionality.   I just think its combination of Python and C code must 
> be redone to overcome the speed issues that have arisen.  My opinion 
> after perusing the numarray code is that it would be easier (for me 
> anyway) to adjust Numeric to support the features of numarray, than to 
> re-write and re-organize the relevant sections of numarray code.  One of 
> the advantages of Numeric is it's tight implementation that added only 
> two fundamental types, both written entirely in C.   I  was hoping that 
> the Python dependencies for the fundamental types would fade as numarray 
> matured, but it appears to me that this is not going to happen.
>
When we started this, one of those that suggested putting much of
the code in Python was Guido himself if I recall correctly (I'd have
to dig up the message he wrote on the subject to see what exactly he
said). That was one of the factors influencing us to go that route
(as well as other mentioned below).
 
> I did not have the time in the past to deal with this.  I wish I had 
> looked at it more closely two years ago.   If I had done this I would 
> have seen how to support the features that Perry wanted without 
> completely re-writing everything.   But, then again, Python 2.2 changed 
> what is possible on the C level and that has had an impact on the 
> discussion.   
> 
Indeed, numarray was started well before Python 2.2 and was another
it wasn't done in C. Knowing what would have been available in 2.2 
would likely have changed the approach used.

> >- Memory-mapped objects: Allow working with on-disk numarray 
> objects like if
> >  they were in-memory.
> >  
> >
> Numeric3 supports this cleanly and old Numeric did too (there was a 
> memory-mapped module), it's just that byteswapping, and alignment had to 
> be done manually.
> 
Just to clarify (it may not be immediately apparent to those who haven't
had to deal with it) many memory mapped files do not use the machine
representation (as is often the case for astronomical data files). 
Memory mapping isn't nearly as useful if one has to create temporaries
to handles these cases.

> >  
> >- RecArrays: Objects that allow to deal with heterogeneous datasets
> >  (tables) in an efficient manner. This ought to be very 
> beneficial in many
> >  fields.
> >  
> >
> Heterogeneous arrays is the big one for old Numeric.  It is a good 
> idea.  In Numeric3 it has required far fewer changes than I had at first 
> imagined.
> 
>
> 
> Numeric has had this in mind for some time.  In fact the early Numeric 
> developers were quite instrumental in getting significant changes into 
> Python istelf, including Complex Objects, Ellipses, and Extended 
> Slicing.   Guido was quite keen on the idea of including Numeric at one 
> point.  Our infighting made him lose interest I think.   So claiming 
> this as an advantage of numarray over Numeric is simply inaccurate.
> 
Actually, as I understand it, Guido had ruled out including Numeric well
before numarray even got started. I can't claim to know exactly his
reasons (I've heard various things such as he looked at the code and 
didn't like it, or Paul Dubois and others advised him against it; I 
can't say exactly why), but I am sure that that decision was made 
before numarray. No doubt the split has prevented any further
consideration of inclusion. 

Perry






From konrad.hinsen at laposte.net  Wed Feb  9 09:15:35 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Wed Feb  9 09:15:35 2005
Subject: [Numpy-discussion] Proposal for making of Numarray a real Numeric 'NG'
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCOEANFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCOEANFJAA.perry@stsci.edu>
Message-ID: <4edd14ca7beb27b727d3984005dda768@laposte.net>

On Feb 9, 2005, at 17:48, Perry Greenfield wrote:

> When we started this, one of those that suggested putting much of
> the code in Python was Guido himself if I recall correctly (I'd have
> to dig up the message he wrote on the subject to see what exactly he

My memory tells me that the NumPy community asked if there was any 
chance to get arrays into the core if the code was decent, and Guido 
replied positively.

> Actually, as I understand it, Guido had ruled out including Numeric 
> well
> before numarray even got started. I can't claim to know exactly his
> reasons (I've heard various things such as he looked at the code and
> didn't like it, or Paul Dubois and others advised him against it; I

There was pretty much a concensus at the time that we shouldn't make 
fools of ourselves by proposing the existing codebase for inclusion in 
the core.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From perry at stsci.edu  Wed Feb  9 09:26:32 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Wed Feb  9 09:26:32 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4207D18A.1070309@ee.byu.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCCEAPFJAA.perry@stsci.edu>

Travis Oliphant wrote:

> >
> > It's interesting that SciPy and matplotlib really do have much the 
> > same goal, but taken from different tacks: SciPy focused on the 
> > computational functionality, and matplotlib on the plotting. However, 
> > what's become really clear in this discussion is that a good 
> > scientific math environment really needs both, and they need to be 
> > well integrated.
> 
> The problem was that we needed someone like John to join us on the scipy 
> effort to get the plotting working well.   Our team lacked someone with 
> his skill.  All of us working on SciPy would have loved to work with 
> him.  So, I have nothing but respect for John and his efforts.  I just 
> wish we at SciPy were better recruiters :-)
>
In John's defense, there was already a scipy-related plotting project
underway (chaco). It wouldn't be surprising for someone to conclude
that an alternative plotting project wouldn't be all that welcome
(and John has already explained why he didn't get involved with 
chaco).
[...]
> 
> 4) The numarray C-API seems way too big.  There are so many, seemingly 
> repeated calls that seem to do the same thing. Are all of these API 
> calls really necessary? 
> 
I'm guessing you are talking all the various ways of accessing elements
in an array. It's not clear to us that these are needed either (I have a 
feeling that no one is using some of the alternatives). The reasons
behind them were to allow C programs a way of accessing arrays with
byteswapped or misaligned values without having to do those operations
explicitly themselves. Various options were provided so that one could
trade off memory usage against speed (e.g., macros vs function call/pixel).
I think the reasons are fairly well given in the manual. I don't know
if that justifies the existence of all of them and we've considered
removing some of them.

Perry




From stephen.walton at csun.edu  Wed Feb  9 10:13:41 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Wed Feb  9 10:13:41 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42096303.5030805@colorado.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu> <42096070.5030608@csun.edu> <42096303.5030805@colorado.edu>
Message-ID: <420A526C.3030804@csun.edu>

Fernando Perez wrote a good deal clarifying how Python's logical 
operators work, for which I'm grateful.  One brief comment:

> Bytecode analysis can be really useful to understand this kind of 
> subtle issues.  I can't think of any language where this approach is 
> as easy to do and friendly as in python.

Pardon me, but this sounds a bit like looking at the assembly language 
Fortran generates in order to understand the subtleties of that language 
:-) .  I know people who used to do that, and were proud of it, but I'd 
rather the language itself didn't behave in surprising ways.  At least 
in Fortran, 'a .and. b' only made sense if a and b were both of type 
LOGICAL;  in fact, it was a compile time error if they weren't.

Anyway, we've wandered afar from the main point:  what is the best way 
to duplicate the functionality of the following MATLAB snippet:

f = ~isnan(data1) & ~isnan(data2);
plot(x,data1(f),x,data2(f))

?




From Fernando.Perez at colorado.edu  Wed Feb  9 10:24:27 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed Feb  9 10:24:27 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <420A526C.3030804@csun.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu> <42096070.5030608@csun.edu> <42096303.5030805@colorado.edu> <420A526C.3030804@csun.edu>
Message-ID: <420A551F.3020206@colorado.edu>

Stephen Walton wrote:
> Fernando Perez wrote a good deal clarifying how Python's logical 
> operators work, for which I'm grateful.  One brief comment:
> 
> 
>>Bytecode analysis can be really useful to understand this kind of 
>>subtle issues.  I can't think of any language where this approach is 
>>as easy to do and friendly as in python.
> 
> 
> Pardon me, but this sounds a bit like looking at the assembly language 
> Fortran generates in order to understand the subtleties of that language 
> :-) .  I know people who used to do that, and were proud of it, but I'd 

Oh, and it is :)  It's just that it's so immediate to do it in python, that it 
actually becomes a normal approach.  I've often used it to clarify behavior or 
performance issues to myself.  I'm not proud of it, it's just a useful trick 
which can come in handy in certain cases.

> rather the language itself didn't behave in surprising ways.  At least 
> in Fortran, 'a .and. b' only made sense if a and b were both of type 
> LOGICAL;  in fact, it was a compile time error if they weren't.

You can argue that this is a wart (I don't think it is), but it was certainly 
done by design:

http://docs.python.org/ref/bitwise.html#l2h-423
http://docs.python.org/ref/Booleans.html#l2h-440

'and' is meant to provie truth checking, with (critically important in many 
applications) short-circuit behavior.  '&' is a binary operation (hence 
necessarily non short-circuiting, due to python's argument evaluation rules), 
most often used for bitwise manipulations.  It seems clear to me that they 
both have a valid use.

> Anyway, we've wandered afar from the main point:  what is the best way 
> to duplicate the functionality of the following MATLAB snippet:
> 
> f = ~isnan(data1) & ~isnan(data2);
> plot(x,data1(f),x,data2(f))

I don't have a clean answer for this one, sorry.

Best,

f



From perry at stsci.edu  Wed Feb  9 10:48:31 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Wed Feb  9 10:48:31 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <420A526C.3030804@csun.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCCEBBFJAA.perry@stsci.edu>

Stephen Walton wrote:

> Anyway, we've wandered afar from the main point:  what is the best way 
> to duplicate the functionality of the following MATLAB snippet:
> 
> f = ~isnan(data1) & ~isnan(data2);
> plot(x,data1(f),x,data2(f))
> 
Strictly speaking there is no operator equivalent (you can always use
the ufunc logical_and: f = logical_and(~isnan(data1), ~isnan(data2)) )
(using isnan for the sake of illustration)

I'm not sure if this is wise advice, I point out to people that 
generally speaking, the arguments to the and operator are typically
boolean arrays and for numarray anyway (I forget if it can be different
for Numeric) you can simply use the '&' bitwise operator to the same
effect. No doubt that such use will eventually be misinterpreted 
and someone will provide a non-boolean argument resulting in confusion
(perhaps a mandatory comment is needed in such cases)

No doubt this is a wart from array user's perspectives.

Perry




From perry at stsci.edu  Wed Feb  9 11:24:32 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Wed Feb  9 11:24:32 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <200502080935.47908.haase@msg.ucsf.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCKEBFFJAA.perry@stsci.edu>

Sebastian Haase wrote:
> (RE: numarray)
> Hi, 
> On an Opteron (Suse9) I was happy to see that I could allocate 
> the memory for 
> an Int32-1024x1024x1024 cube. (On a P4 a get 'MemoryError').
> I remember that Todd always said that python itself wasn't 
> 64bit-ready yet.
> But why is it needed to support >4GB python-lists if one is only 
> interested in 
> large numarrays (or numeric) ?
> 
In short, to support the Python sequence protocol (arrays can be
used anywhere Python code expects sequences transparently). The 
good news is that work is well underway in Python to change the
limit on sequence indices. I think it will take a number of months
for these changes to make it out (in a released version of Python as
well as the necessary updates to numarray). As soon as the changes
are committed to Python CVS, we can start working on updates to 
numarray to support them.

Perry Greenfield




From stephen.walton at csun.edu  Wed Feb  9 11:54:34 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Wed Feb  9 11:54:34 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCCEBBFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCCEBBFJAA.perry@stsci.edu>
Message-ID: <1107978754.21229.4.camel@freyer.sfo.csun.edu>

On Wed, 2005-02-09 at 13:48 -0500, Perry Greenfield wrote:
> Strictly speaking there is no operator equivalent (you can always use
> the ufunc logical_and: f = logical_and(~isnan(data1), ~isnan(data2)) )
> (using isnan for the sake of illustration)

OK, how did I miss in "Learning Python" and the numarray docs that ~ is
bitwise NOT and works on numarray bool arrays?  In point of fact,
"~isnan(data1) & ~isnan(data2)" works fine.  I kept trying !isnan(data1)
which I gather makes no sense.

Thanks for the patient help, all.





From Chris.Barker at noaa.gov  Wed Feb  9 12:22:54 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Wed Feb  9 12:22:54 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1107914212.28976.34.camel@alpspitze.cs.pdx.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>	 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>	 <4207CC4C.8060603@ee.byu.edu>	 <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>	 <42084A1F.3030007@ee.byu.edu>  <42090476.7080604@noaa.gov>	 <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>	 <420961B3.1000405@noaa.gov> <1107914212.28976.34.camel@alpspitze.cs.pdx.edu>
Message-ID: <420A70A7.4020203@noaa.gov>

Ralf Juengling wrote:

> I'm not sure I understand your stance on this, though. You were
> complaining in the other posting that matplotlib's "programming API"
> would not offer all of the features that are offered by its 
> "interactive API". But you don't seem to be offended by the very 
> idea of having two APIs.

Well, I'd prefer to have just one, but understand that others have 
different needs than me, so no, I'm not offended by more than one. I 
would like the OO one to be just as good as the functional one, however. 
If I don't stop reading and writing this thread, I'll never have any 
time to actually contribute to that effort, however.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From haase at msg.ucsf.edu  Wed Feb  9 13:30:36 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed Feb  9 13:30:36 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCKEBFFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCKEBFFJAA.perry@stsci.edu>
Message-ID: <200502091329.09832.haase@msg.ucsf.edu>

On Wednesday 09 February 2005 11:24 am, Perry Greenfield wrote:
> Sebastian Haase wrote:
> > (RE: numarray)
> > Hi,
> > On an Opteron (Suse9) I was happy to see that I could allocate
> > the memory for
> > an Int32-1024x1024x1024 cube. (On a P4 a get 'MemoryError').
> > I remember that Todd always said that python itself wasn't
> > 64bit-ready yet.
> > But why is it needed to support >4GB python-lists if one is only
> > interested in
> > large numarrays (or numeric) ?
>
> In short, to support the Python sequence protocol (arrays can be
> used anywhere Python code expects sequences transparently). 

Perry,
Thanks for the answer - that's about what I remembered.
Is is conceivable to have a compiler flag (#define) to turn all the sequence 
protocol support off ?  (How many places in the numarray code would that be?)

I think the wish to just allocate large arrays and (even if in a very limited 
way!) work with them is clearly shared by many people.
I always felt having a Py-List longer than billons of elements would have a 
rather low priority in comparison.

Thanks again,
Sebastian


> The 
> good news is that work is well underway in Python to change the
> limit on sequence indices. I think it will take a number of months
> for these changes to make it out (in a released version of Python as
> well as the necessary updates to numarray). As soon as the changes
> are committed to Python CVS, we can start working on updates to
> numarray to support them.
>
> Perry Greenfield



From perry at stsci.edu  Wed Feb  9 15:34:39 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Wed Feb  9 15:34:39 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <200502091329.09832.haase@msg.ucsf.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCCEBHFJAA.perry@stsci.edu>

Sebastian Haase wrote:

> Perry,
> Thanks for the answer - that's about what I remembered.
> Is is conceivable to have a compiler flag (#define) to turn all 
> the sequence 
> protocol support off ?  (How many places in the numarray code 
> would that be?)
> 
Offhand, I'm not sure (Todd probably has a much better quick answer
for this question than I do). It is perhaps something that would
allow us to test 64-bit arrays before the Python changes are
ready if it isn't too hard to optionally disable the dependence
on Python sequences. It's worth a quick look at after 1.2 is released.

> I think the wish to just allocate large arrays and (even if in a 
> very limited 
> way!) work with them is clearly shared by many people.
> I always felt having a Py-List longer than billons of elements 
> would have a 
> rather low priority in comparison.
> 
Just to note, we more and more local users who are begining to
have a need for such large arrays too.

Perry




From oliphant at ee.byu.edu  Wed Feb  9 15:48:15 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb  9 15:48:15 2005
Subject: [Numpy-discussion] [Fwd: Re: [Python-Dev] Clarification sought about including a multidimensional
 array object into Python core]
Message-ID: <420AA0F5.2030707@ee.byu.edu>


-------------- next part --------------
An embedded message was scrubbed...
From: unknown sender
Subject: no subject
Date: no date
Size: 38
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050209/3062c877/attachment.mht>

From gvanrossum at gmail.com  Wed Feb  9 17:45:18 2005
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Wed, 9 Feb 2005 14:45:18 -0800
Subject: [Python-Dev] Clarification sought about including a multidimensional array object into Python core
In-Reply-To: <420A8406.4020808@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu>
Message-ID: <ca471dc205020914453f4da355@mail.gmail.com>

> 1) What specifically about Numeric prevented it from being acceptable as
> an addition to the Python core.

It's very long ago, I believe that the authors themselves didn't think
it was good enough. It certainly had a very hackish coding style.

Numarray was supposed to fix all that. I'm sorry to hear that it
hasn't (yet) reached the maturity you find necessary.

> 2) Are there any fixed requirements (other than coding style) before an
> arrayobject would be accepted into the Python core.

The intended user community must accept the code as "best-of-breed".
It seems that the Num* community has some work to do in this respect.

Also (this applies to all code) the code must be stable enough that
the typical Python release cycle (about 18 months between feature
releases) doesn't cause problems.

Finally there must be someone willing to be responsible for
maintenance of the code.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

--------------080201020105040905060300--



From oliphant at ee.byu.edu  Wed Feb  9 15:49:01 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb  9 15:49:01 2005
Subject: [Numpy-discussion] [Fwd: [Python-Dev] Clarification sought about including a multidimensional
 array object into Python core]
Message-ID: <420AA10E.8020200@ee.byu.edu>


-------------- next part --------------
An embedded message was scrubbed...
From: unknown sender
Subject: no subject
Date: no date
Size: 38
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050209/a52ca36e/attachment.mht>

From oliphant at ee.byu.edu  Wed Feb  9 16:43:34 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 09 Feb 2005 14:43:34 -0700
Subject: [Python-Dev] Clarification sought about including a
 multidimensional array object into Python core
Message-ID: <420A8406.4020808@ee.byu.edu>


There has recently been some much-needed discussion on the 
numpy-discussions list run by sourceforge regarding the state of the 
multidimensional array objects available for Python.  It is desired by 
many that there be a single multidimensional array object in the Python 
core to facilitate data transfer and interfacing between multiple packages.

I am a co-author of the current PEP regarding inclusion of the 
multidimensional array object into the core.  However, that PEP is 
sorely outdated.  Currently there are two multidimensional array objects 
that are in use in the Python community:
 
   Numeric --- original arrayobject created by Jim Hugunin and many 
others.  Has been developed and used for 10 years.  An upgrade that adds 
the features of numarray but maintains the same basic structure of 
Numeric called Numeric3 is in development and will be ready for more 
wide-spread use in a couple of weeks.

   Numarray --- in development for about 3 years.  It was billed by some 
as a replacement for Numeric,.  While introducing some new features, it 
still has not covered the full feature set that Numeric had making it 
impossible for all Numeric users to use it.  In addition, it is still 
unacceptably slow for many operations that Numeric does well. 

Scientific users will always have to install more packages in order to 
use Python for their purposes.  However, there is still the desire that 
the basic array object would be common among all Python users.   To 
assist in writing a new PEP, we need clarification from Guido and others 
involved regarding

1) What specifically about Numeric prevented it from being acceptable as 
an addition to the Python core.
2) Are there any fixed requirements (other than coding style) before an 
arrayobject would be accepted into the Python core.

Thanks for your comments.  I think they will help the discussion 
currently taking place.

-Travis Oliphant



_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/oliphant%40ee.byu.edu

--------------080004070407050603040506--



From andrewm at object-craft.com.au  Wed Feb  9 16:25:24 2005
From: andrewm at object-craft.com.au (Andrew McNamara)
Date: Wed Feb  9 16:25:24 2005
Subject: [Numpy-discussion] Numeric vs numarray?
Message-ID: <20050210002435.1C9323C889@coffee.object-craft.com.au>

I know this is a loaded question, but what does numarray offer that
Numeric does not (as they stand today)? A cleaner implementation?

-- 
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/



From oliphant at ee.byu.edu  Wed Feb  9 16:36:34 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb  9 16:36:34 2005
Subject: [Numpy-discussion] Re: [Python-Dev] Clarification sought about including a multidimensional
 array object into Python core
In-Reply-To: <dd28fc2f050209161264f9b601@mail.gmail.com>
References: <420A8406.4020808@ee.byu.edu>	 <ca471dc205020914453f4da355@mail.gmail.com> <dd28fc2f050209161264f9b601@mail.gmail.com>
Message-ID: <420AAC33.807@ee.byu.edu>

David Ascher wrote:

>I've not followed the num* discussion in quite a while, but my
>impression back then was that there wasn't "one" such community. 
>Instead, the technical differences in the approaches required in
>specific fields, regarding things like the relative importance of
>memory profiles, speed, error handling, willingness to require modern
>C++ compilers, etc. made practical compromises quite tricky.
>  
>

I really appreciate comments from those who remember some of the old 
discussions.

There are indeed some different needs.  Most of this, however, is in the 
ufunc object (how do you do math with the arrays).   And, a lot of this 
has been ameliorated with the new concepts of error modes that numarray 
introduced.

There is less argumentation over the basic array object as a memory 
structure.   The biggest argument right now is the design of the object: 
i.e.  a mixture of Python and C (numarray) versus a C-only object 
(Numeric3).

In other words, what I'm saying is that in terms of how the array object 
should be structure, a lot is known.  What is more controversial is 
should the design be built upon Numarray's object structure (a mixture 
of Python and C), or on Numeric's --- all in C

-Travis







From martin at v.loewis.de  Wed Feb  9 16:54:01 2005
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed Feb  9 16:54:01 2005
Subject: [Numpy-discussion] Re: [Python-Dev] Clarification sought about including a	multidimensional
 array object into Python core
In-Reply-To: <420AAC33.807@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu>		<ca471dc205020914453f4da355@mail.gmail.com>	<dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu>
Message-ID: <420AB084.1000008@v.loewis.de>

Travis Oliphant wrote:
> In other words, what I'm saying is that in terms of how the array object 
> should be structure, a lot is known.  What is more controversial is 
> should the design be built upon Numarray's object structure (a mixture 
> of Python and C), or on Numeric's --- all in C

To me, this sounds like an implementation detail. I'm sure it is an
important detail, as I understand all of this is mostly done for
performance reasons. The PEP should list the options, include criteria
for selection, and then propose a choice. People can then discuss
whether the list of options is complete (if not, you need to extend
it), whether the criteria are agreed (they might be not, and there
might be difficult consensus, which the PEP should point out), and
whether the choice is the right one given the criteria (there should
be no debate about this - everybody should agree factually that the
choice meets the criteria best).

Regards,
Martin



From paul at pfdubois.com  Wed Feb  9 17:32:00 2005
From: paul at pfdubois.com (Paul F. Dubois)
Date: Wed Feb  9 17:32:00 2005
Subject: [Numpy-discussion] Numeric life as I see it
In-Reply-To: <420AB084.1000008@v.loewis.de>
References: <420A8406.4020808@ee.byu.edu>		<ca471dc205020914453f4da355@mail.gmail.com>	<dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de>
Message-ID: <420AB928.3090004@pfdubois.com>


Martin v. L?wis wrote:
The PEP should list the options, include criteria
> for selection, and then propose a choice. People can then discuss
> whether the list of options is complete (if not, you need to extend
> it), whether the criteria are agreed (they might be not, and there
> might be difficult consensus, which the PEP should point out), and
> whether the choice is the right one given the criteria (there should
> be no debate about this - everybody should agree factually that the
> choice meets the criteria best).
> 

Unrealistic. I think it is undisputed that there are people with 
irreconcilably different needs. Frankly, we spent many, many months on 
the design of Numeric and it represents a set of compromises already. 
However, the one thing it wouldn't compromise on was speed, even at the 
expense of safety. A community exists that cannot live with this 
compromise. We were told that the Python core could also not live with 
that compromise.

Over the years there was pressure to add safety, convenience, 
flexibility, etc., all sometimes incompatible with speed. Numarray 
represents in some sense the set of compromises in that direction, 
besides its technical innovations. Numeric / Numeric3 represents the 
need for speed camp.

I think it is reasonable to suppose that the need for speed piece can be 
wrapped suitably by the need for safety-flexibility-convenience 
facilities. I believe that hope underlies Travis' plan.

The Nummies (the official set of developers) thought that the Numeric 
code base was an unsuitable basis for further development. There was no 
dissent about that at least. My idea was to get something like what 
Travis is now doing done to replace it. I felt it important to get 
myself out of the picture after five years as the lead developer 
especially since my day job had ceased to involve using Numeric.

However, removing my cork from the bottle released the unresolved 
pressure between these two camps. My plan for transition failed. I 
thought I had consensus on the goal and in fact it wasn't really there. 
Everyone is perfectly good-willed and clever and trying hard to "all 
just get along", but the goal was lost.  Eric Raymond should write a 
book about it called "Bumbled Bazaar".

I hope everyone will still try to achieve that goal. Interoperability of 
all the Numeric-related software (including supporting a 'default' 
plotting package) is required.

Aside: While I am at it, let me reiterate what I have said to the other 
developers privately: there is NO value to inheriting from the array 
class. Don't try to achieve that capability if it costs anything, even 
just effort, because it buys you nothing. Those of you who keep 
remarking on this as if it would simply haven't thought it through IMHO. 
It sounds so intellectually appealing that David Ascher and I had a 
version of Numeric that almost did it before we realized our folly.





From andrewm at object-craft.com.au  Wed Feb  9 18:51:00 2005
From: andrewm at object-craft.com.au (Andrew McNamara)
Date: Wed Feb  9 18:51:00 2005
Subject: [Numpy-discussion] Numeric life as I see it 
In-Reply-To: <420AB928.3090004@pfdubois.com> 
References: <420A8406.4020808@ee.byu.edu> <ca471dc205020914453f4da355@mail.gmail.com> <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com>
Message-ID: <20050210025044.A3FB63C889@coffee.object-craft.com.au>

>However, the one thing it wouldn't compromise on was speed, even at the
>expense of safety.

Yes, because if you don't have speed, then you might as well use the
standard services python offers.

-- 
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/



From oliphant at ee.byu.edu  Wed Feb  9 20:12:05 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb  9 20:12:05 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <420AB928.3090004@pfdubois.com>
References: <420A8406.4020808@ee.byu.edu>		<ca471dc205020914453f4da355@mail.gmail.com>	<dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com>
Message-ID: <420ADE90.9050304@ee.byu.edu>

> Martin v. L?wis wrote:
> The PEP should list the options, include criteria
>
>> for selection, and then propose a choice. People can then discuss
>> whether the list of options is complete (if not, you need to extend
>> it), whether the criteria are agreed (they might be not, and there
>> might be difficult consensus, which the PEP should point out), and
>> whether the choice is the right one given the criteria (there should
>> be no debate about this - everybody should agree factually that the
>> choice meets the criteria best).
>>
>
> Unrealistic. I think it is undisputed that there are people with 
> irreconcilably different needs. Frankly, we spent many, many months on 
> the design of Numeric and it represents a set of compromises already. 
> However, the one thing it wouldn't compromise on was speed, even at 
> the expense of safety. A community exists that cannot live with this 
> compromise. We were told that the Python core could also not live with 
> that compromise.


I'm not sure I agree.  The ufuncobject is the only place where this 
concern existed (should we trip OverFlow, ZeroDivision, etc. errors 
durring array math).   Numarray introduced and implemented the concept 
of error modes that can be pushed and popped.  I believe this is the 
right solution for the ufuncobject.

One question we are pursuing is could the arrayobject get into the core 
without a particular ufunc object.   Most see this as sub-optimal, but 
maybe it is the only way.

>
> Over the years there was pressure to add safety, convenience, 
> flexibility, etc., all sometimes incompatible with speed. Numarray 
> represents in some sense the set of compromises in that direction, 
> besides its technical innovations. Numeric / Numeric3 represents the 
> need for speed camp.

I don't see numarray as representing this at all.  To me, numarray 
represents the desire to have more flexible array types (specifically 
record arrays and maybe character arrays).   I personally don't see 
Numeric3 as in any kind of "need for speed" camp either.  I've never 
liked this distinction, because I don't think it represents a true 
dichotomy.  To me,  the differences between Numeric3 and numarray are 
currently more "architectural" than implementational.    

Perhaps you are referring to the fact that because numarray has several 
portions written in Python it is "more flexible" or "more convenient" 
but slower for small arrays.  If you are saying that then I guess 
Numeric3 is a "need for speed" implementation, and I apologize for not 
understanding. 

>
> I think it is reasonable to suppose that the need for speed piece can 
> be wrapped suitably by the need for safety-flexibility-convenience 
> facilities. I believe that hope underlies Travis' plan.

If the "safety-flexibility-convenience" facilities can be specified, 
then I'm all for one implementation.    Numeric3 design goals do not go 
against any of these ideas intentionally.

>
> The Nummies (the official set of developers) thought that the Numeric 
> code base was an unsuitable basis for further development. There was 
> no dissent about that at least. My idea was to get something like what 
> Travis is now doing done to replace it. I felt it important to get 
> myself out of the picture after five years as the lead developer 
> especially since my day job had ceased to involve using Numeric.

Some of the parts needed to be re-written, but I didn't think that meant 
moving away from the goal to have a single C-type that is the 
arrayobject.   During this process Python 2.2 came out and allowed 
sub-classing from C-types.  As Perry mentioned, and I think needs to be 
emphasized again, this changed things as any benefit from having a 
Python-class for the final basic array type disappeared --- beyond ease 
of prototyping and testing.

>
> However, removing my cork from the bottle released the unresolved 
> pressure between these two camps. My plan for transition failed. I 
> thought I had consensus on the goal and in fact it wasn't really 
> there. Everyone is perfectly good-willed and clever and trying hard to 
> "all just get along", but the goal was lost.  Eric Raymond should 
> write a book about it called "Bumbled Bazaar".

This is an accurate description.  Fortunately, I don't think any 
ill-will exists (assuming I haven't created any with my recent 
activities).  I do want to "get-along."  I just don't want to be silent 
when there are issues that I think I understand.

>
> I hope everyone will still try to achieve that goal. Interoperability 
> of all the Numeric-related software (including supporting a 'default' 
> plotting package) is required.

Utopia is always out of reach :-)

> Aside: While I am at it, let me reiterate what I have said to the 
> other developers privately: there is NO value to inheriting from the 
> array class. Don't try to achieve that capability if it costs 
> anything, even just effort, because it buys you nothing. Those of you 
> who keep remarking on this as if it would simply haven't thought it 
> through IMHO. It sounds so intellectually appealing that David Ascher 
> and I had a version of Numeric that almost did it before we realized 
> our folly.
>
I appreciate some of what Paul is saying here, but I'm not fully 
convinced that this is still true with Python 2.2 and up new-style 
c-types.   The concerns seem to be over the fact that you have to 
re-implement everything in the sub-class because the base-class will 
always return one of its objects instead of a sub-class object.
It seems to me, however,  that if the C methods use the object type 
alloc function when creating new objects then some of this problem is 
avoided (i.e. if the method is called with a sub-class type passed in, 
then a sub-class type gets set).

Have you looked at how Python now allows sub-classing in C?  I'm not an 
expert here, but it seems like a lot of the problems you were discussing 
have been ameliorated.  There are probably still issues, but....

I will know more when I seen what happens with a Matrix Object 
inheriting from a Python C-array object.

I'm wondering if anyone else with more knowledge about new-style c-types 
could help
here and show me the error of my thinking. 

-Travis










From gvanrossum at gmail.com  Wed Feb  9 20:37:06 2005
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Wed Feb  9 20:37:06 2005
Subject: [Numpy-discussion] Re: [Python-Dev] Re: Numeric life as I see it
In-Reply-To: <420ADE90.9050304@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu>
	 <ca471dc205020914453f4da355@mail.gmail.com>
	 <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu>
	 <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com>
	 <420ADE90.9050304@ee.byu.edu>
Message-ID: <ca471dc205020920364ec6fa40@mail.gmail.com>

[Paul]
> > Aside: While I am at it, let me reiterate what I have said to the
> > other developers privately: there is NO value to inheriting from the
> > array class. Don't try to achieve that capability if it costs
> > anything, even just effort, because it buys you nothing. Those of you
> > who keep remarking on this as if it would simply haven't thought it
> > through IMHO. It sounds so intellectually appealing that David Ascher
> > and I had a version of Numeric that almost did it before we realized
> > our folly.

[Travis]
> I appreciate some of what Paul is saying here, but I'm not fully
> convinced that this is still true with Python 2.2 and up new-style
> c-types.   The concerns seem to be over the fact that you have to
> re-implement everything in the sub-class because the base-class will
> always return one of its objects instead of a sub-class object.
> It seems to me, however,  that if the C methods use the object type
> alloc function when creating new objects then some of this problem is
> avoided (i.e. if the method is called with a sub-class type passed in,
> then a sub-class type gets set).

This would severely constrain the __new__ method of the subclass.

> Have you looked at how Python now allows sub-classing in C?  I'm not an
> expert here, but it seems like a lot of the problems you were discussing
> have been ameliorated.  There are probably still issues, but....
> 
> I will know more when I seen what happens with a Matrix Object
> inheriting from a Python C-array object.

And why would a Matrix need to inherit from a C-array? Wouldn't it
make more sense from an OO POV for the Matrix to *have* a C-array
without *being* one?

> I'm wondering if anyone else with more knowledge about new-style c-types
> could help here and show me the error of my thinking.

I'm trying...

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)



From tkorvola at welho.com  Wed Feb  9 22:38:08 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Wed Feb  9 22:38:08 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <20050210002435.1C9323C889@coffee.object-craft.com.au> (Andrew
 McNamara's message of "Thu, 10 Feb 2005 11:24:35 +1100")
References: <20050210002435.1C9323C889@coffee.object-craft.com.au>
Message-ID: <876510q5s1.fsf@welho.com>

Andrew McNamara <andrewm at object-craft.com.au> writes:
> I know this is a loaded question, but what does numarray offer that
> Numeric does not (as they stand today)? A cleaner implementation?

NA_OutputArray and NA_IoArray.  Very convenient for wrapping C
libraries.  NA_InputArray is more or less equivalent to
PyArray_ContiguousFromObject but as far as I know Numeric has nothing
equivalent to the other two.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From matthew.brett at gmail.com  Thu Feb 10 00:16:35 2005
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu Feb 10 00:16:35 2005
Subject: [Numpy-discussion] Numeric life as I see it
In-Reply-To: <420AB928.3090004@pfdubois.com>
References: <420A8406.4020808@ee.byu.edu>
	 <ca471dc205020914453f4da355@mail.gmail.com>
	 <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu>
	 <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com>
Message-ID: <1e2af89e0502100015438bbed6@mail.gmail.com>

> Aside: While I am at it, let me reiterate what I have said to the other
> developers privately: there is NO value to inheriting from the array
> class. Don't try to achieve that capability if it costs anything, even
> just effort, because it buys you nothing. Those of you who keep
> remarking on this as if it would simply haven't thought it through IMHO.
> It sounds so intellectually appealing that David Ascher and I had a
> version of Numeric that almost did it before we realized our folly.

Please forgive a question from someone who _definitely_ has not
thought it through, but would the fact that you cannot inherit from
the array class have any influence on whether it will be accepted in
the python core?

Best,

Matthew



From konrad.hinsen at laposte.net  Thu Feb 10 00:39:23 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Thu Feb 10 00:39:23 2005
Subject: [Numpy-discussion] Re: [Python-Dev] Re: Numeric life as I see it
In-Reply-To: <ca471dc205020920364ec6fa40@mail.gmail.com>
References: <420A8406.4020808@ee.byu.edu>  <ca471dc205020914453f4da355@mail.gmail.com>  <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu>  <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com>  <420ADE90.9050304@ee.byu.edu> <ca471dc205020920364ec6fa40@mail.gmail.com>
Message-ID: <a225db1f60892095a12adb7adc2178e6@laposte.net>

On 10.02.2005, at 05:36, Guido van Rossum wrote:

> And why would a Matrix need to inherit from a C-array? Wouldn't it
> make more sense from an OO POV for the Matrix to *have* a C-array
> without *being* one?

Definitely. Most array operations make no sense on matrices. And  
matrices are limited to two dimensions. Making Matrix a subclass of  
Array would be inheritance for implementation while removing 90% of the  
interface.

On the other hand, a Matrix object is perfectly defined by its  
behaviour and independent of its implementation. One could perfectly  
well implement one using Python lists or dictionaries, even though that  
would be pointless from a performance point of view.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Thu Feb 10 00:46:31 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Thu Feb 10 00:46:31 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <420ADE90.9050304@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu> <ca471dc205020914453f4da355@mail.gmail.com> <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com> <420ADE90.9050304@ee.byu.edu>
Message-ID: <1c3044466186480f55ef45d2c977731b@laposte.net>

On 10.02.2005, at 05:09, Travis Oliphant wrote:

> I'm not sure I agree.  The ufuncobject is the only place where this  
> concern existed (should we trip OverFlow, ZeroDivision, etc. errors  
> durring array math).   Numarray introduced and implemented the concept  
> of error modes that can be pushed and popped.  I believe this is the  
> right solution for the ufuncobject.

Indeed. Note also that the ufunc stuff is less critical to agree on  
than the array data structure. Anyone unhappy with ufuncs could write  
their own module and use it instead. It would be the data structure and  
its access rules that fix the structure of all the code that uses it,  
so that's what needs to be acceptable to everyone.

> One question we are pursuing is could the arrayobject get into the  
> core without a particular ufunc object.   Most see this as  
> sub-optimal, but maybe it is the only way.

Since all the artithmetic operations are in ufunc that would be  
suboptimal solution, but indeed still a workable one.

> I appreciate some of what Paul is saying here, but I'm not fully  
> convinced that this is still true with Python 2.2 and up new-style  
> c-types.   The concerns seem to be over the fact that you have to  
> re-implement everything in the sub-class because the base-class will  
> always return one of its objects instead of a sub-class object.

I'd say that such discussions should be postponed until someone  
proposes a good use for subclassing arrays. Matrices are not one, in my  
opinion.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From oliphant at ee.byu.edu  Thu Feb 10 01:27:00 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 10 01:27:00 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <876510q5s1.fsf@welho.com>
References: <20050210002435.1C9323C889@coffee.object-craft.com.au> <876510q5s1.fsf@welho.com>
Message-ID: <420B2898.8090808@ee.byu.edu>

Timo Korvola wrote:

>Andrew McNamara <andrewm at object-craft.com.au> writes:
>  
>
>>I know this is a loaded question, but what does numarray offer that
>>Numeric does not (as they stand today)? A cleaner implementation?
>>    
>>
>
>NA_OutputArray and NA_IoArray.  Very convenient for wrapping C
>libraries.  NA_InputArray is more or less equivalent to
>PyArray_ContiguousFromObject but as far as I know Numeric has nothing
>equivalent to the other two.
>  
>
I am not convinced these are really useful additional C-API calls.  I've 
done a fair bit of wrapping of C libraries and have never needed 
anything but

PyArray_ContiguousFromObject  and
PyArray_FromOb ject

This idea of "shadowing" ala NA_IOArray is a new one.  Can somebody show 
me an example of it's usefulness that could not be handled by simply a 
PyArray_ContiguousFromObject with an explicit copy back at the end?  I 
don't like this idea of "hiding" what is going on from the user.

This kind of C-API alteration which as far as I can tell has not been 
well justified is an example of some of the problems I see with Numarray.

Numeric3 condenses the Numeric API calls using numarray's idea of the 
requires flag:

PyArray_FromAny(PyObject *op, PyArray_Typecode *typecode, int mindepth, 
int maxdepth, int requires)

typecode is a simple structure with integer elements (type_num, 
itemsize, and fortran).

For some unexplained reason, the mindepth and maxdepth arguments are not 
copied by the new Numarray C-API.  I'm not sure why.  Having them  does 
make it easier to guarantee you are getting the right dimension for the 
array you want rather than having to check after the fact.

These kinds of discussions about numarray's C-API never took place.  In 
my opinion the Numeric C-API is better tested and easier to understand.

-Travis




From oliphant at ee.byu.edu  Thu Feb 10 01:31:21 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 10 01:31:21 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <1c3044466186480f55ef45d2c977731b@laposte.net>
References: <420A8406.4020808@ee.byu.edu> <ca471dc205020914453f4da355@mail.gmail.com> <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com> <420ADE90.9050304@ee.byu.edu> <1c3044466186480f55ef45d2c977731b@laposte.net>
Message-ID: <420B29AE.8030701@ee.byu.edu>

>> One question we are pursuing is could the arrayobject get into the  
>> core without a particular ufunc object.   Most see this as  
>> sub-optimal, but maybe it is the only way.
>
>
> Since all the artithmetic operations are in ufunc that would be  
> suboptimal solution, but indeed still a workable one.


I think replacing basic number operations of the arrayobject should 
simple, so perhaps a default ufunc object could be worked out for 
inclusion.

>
>> I appreciate some of what Paul is saying here, but I'm not fully  
>> convinced that this is still true with Python 2.2 and up new-style  
>> c-types.   The concerns seem to be over the fact that you have to  
>> re-implement everything in the sub-class because the base-class will  
>> always return one of its objects instead of a sub-class object.
>
>
> I'd say that such discussions should be postponed until someone  
> proposes a good use for subclassing arrays. Matrices are not one, in 
> my  opinion.
>
Agreed.  It is is not critical to what I am doing, and I obviously need 
more understanding before tackling such things.  Numeric3 uses the new 
c-type largely because of the nice getsets table which is separate from 
the methods table.  This replaces the rather ugly C-functions getattr 
and setattr.

-Travis





From verveer at embl.de  Thu Feb 10 01:54:06 2005
From: verveer at embl.de (Peter Verveer)
Date: Thu Feb 10 01:54:06 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <420B29AE.8030701@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu> <ca471dc205020914453f4da355@mail.gmail.com> <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com> <420ADE90.9050304@ee.byu.edu> <1c3044466186480f55ef45d2c977731b@laposte.net> <420B29AE.8030701@ee.byu.edu>
Message-ID: <50ac60a36c2add7d708dc02d8bf623a3@embl.de>

On Feb 10, 2005, at 10:30 AM, Travis Oliphant wrote:

>
>>> One question we are pursuing is could the arrayobject get into the  
>>> core without a particular ufunc object.   Most see this as  
>>> sub-optimal, but maybe it is the only way.
>>
>>
>> Since all the artithmetic operations are in ufunc that would be  
>> suboptimal solution, but indeed still a workable one.
>
>
> I think replacing basic number operations of the arrayobject should 
> simple, so perhaps a default ufunc object could be worked out for 
> inclusion.

I agree, getting it in the core is among others, intended to give it 
broad access, not just to hard-core numeric people. For many uses 
(including many of my simpler scripts) you don't need the more exotic 
functionality of ufuncs. You could just do with implementing the 
standard math functions, possibly leaving out things like reduce. That 
would be very easy to implement.

>
>>
>>> I appreciate some of what Paul is saying here, but I'm not fully  
>>> convinced that this is still true with Python 2.2 and up new-style  
>>> c-types.   The concerns seem to be over the fact that you have to  
>>> re-implement everything in the sub-class because the base-class will 
>>>  always return one of its objects instead of a sub-class object.
>>
>>
>> I'd say that such discussions should be postponed until someone  
>> proposes a good use for subclassing arrays. Matrices are not one, in 
>> my  opinion.
>>
> Agreed.  It is is not critical to what I am doing, and I obviously 
> need more understanding before tackling such things.  Numeric3 uses 
> the new c-type largely because of the nice getsets table which is 
> separate from the methods table.  This replaces the rather ugly 
> C-functions getattr and setattr.

I would agree that sub-classing arrays might not be worth the trouble.

Peter




From verveer at embl-heidelberg.de  Thu Feb 10 02:56:53 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Thu Feb 10 02:56:53 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <420B2898.8090808@ee.byu.edu>
References: <20050210002435.1C9323C889@coffee.object-craft.com.au> <876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu>
Message-ID: <b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de>

It is however useful to distinguish between  input and output arrays. 
If your array is just an input, then the PyArray_FromAny function you 
mention below is good enough. If necessary they will make a 
well-behaved copy which can  be discarded afterwards.

However, if you need to modify an array 'in-place', then your 
well-behaved copy needs to be copied back at the end of your function. 
That can be done transparently as numarray does that it. You mention 
that you could explicitly copy back, but that means that I have to 
determine myself if there was a temporary made. And that could also 
happen if the array was byteswapped or misaligned not just if I asked 
for a contiguous array. The Numarray functions do all that for you.

The point is, that if you use an array just as an input, you don't want 
to copy things back, since that is a waste of resources. So you need 
somehow to distinguish between input and output arrays, which numarray 
does by having separate functions.

For me, the NA_OutputArray function has been very useful, I would like 
to see that functionality somehow in Numeric3. The NA_IoArray function 
I don't really need.

Maybe I have understood the Numeric API wrong and you can do this right 
now, I have not used it very much lately.

I know you avoid these problems if you don't modify arrays at all, but 
just return new arrays with your result. However, then you cannot 
properly implement output arguments. Many numarray and Numeric 
functions do _not_ allow you to pass an output array, to modify 
"in-place" as you can do with ufuncs. That annoys me endlessly, it is a 
big source of inefficiency if you have big arrays. A good example are 
the FFT functions that make lots of temporaries if you do a 
multi-dimensional transform. I am digressing now, but maybe output 
arrays could be supported more consistently in Numeric3?

Cheers, Peter

On 10 Feb 2005, at 10:25, Travis Oliphant wrote:

> Timo Korvola wrote:
>
>> Andrew McNamara <andrewm at object-craft.com.au> writes:
>>
>>> I know this is a loaded question, but what does numarray offer that
>>> Numeric does not (as they stand today)? A cleaner implementation?
>>>
>>
>> NA_OutputArray and NA_IoArray.  Very convenient for wrapping C
>> libraries.  NA_InputArray is more or less equivalent to
>> PyArray_ContiguousFromObject but as far as I know Numeric has nothing
>> equivalent to the other two.
>>
> I am not convinced these are really useful additional C-API calls.  
> I've done a fair bit of wrapping of C libraries and have never needed 
> anything but
>
> PyArray_ContiguousFromObject  and
> PyArray_FromOb ject
>
> This idea of "shadowing" ala NA_IOArray is a new one.  Can somebody 
> show me an example of it's usefulness that could not be handled by 
> simply a PyArray_ContiguousFromObject with an explicit copy back at 
> the end?  I don't like this idea of "hiding" what is going on from the 
> user.
>
> This kind of C-API alteration which as far as I can tell has not been 
> well justified is an example of some of the problems I see with 
> Numarray.
>
> Numeric3 condenses the Numeric API calls using numarray's idea of the 
> requires flag:
>
> PyArray_FromAny(PyObject *op, PyArray_Typecode *typecode, int 
> mindepth, int maxdepth, int requires)
>
> typecode is a simple structure with integer elements (type_num, 
> itemsize, and fortran).
>
> For some unexplained reason, the mindepth and maxdepth arguments are 
> not copied by the new Numarray C-API.  I'm not sure why.  Having them  
> does make it easier to guarantee you are getting the right dimension 
> for the array you want rather than having to check after the fact.
>
> These kinds of discussions about numarray's C-API never took place.  
> In my opinion the Numeric C-API is better tested and easier to 
> understand.
>
> -Travis
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real 
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion




From tkorvola at welho.com  Thu Feb 10 05:25:29 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Thu Feb 10 05:25:29 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de> (Peter
 Verveer's message of "Thu, 10 Feb 2005 11:55:36 +0100")
References: <20050210002435.1C9323C889@coffee.object-craft.com.au>
	<876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu>
	<b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de>
Message-ID: <87vf90czto.fsf@welho.com>

Peter Verveer <verveer at embl-heidelberg.de> writes:
> The NA_IoArray function I don't really need.

Depends on what C code you are working with.  Bidirectional array
arguments are not uncommon in libraries.

> Maybe I have understood the Numeric API wrong and you can do this
> right now, I have not used it very much lately.

Perhaps you can but you would have to write it all yourself.  In
Numarray that work has been done for you.  The three functions
NA_{Input,Output,Io}Array are pretty much all you need for accessing C
libraries with the usual (int size, double *data) interface, whichever
way the data goes.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From verveer at embl-heidelberg.de  Thu Feb 10 05:42:00 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Thu Feb 10 05:42:00 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <87vf90czto.fsf@welho.com>
References: <20050210002435.1C9323C889@coffee.object-craft.com.au> <876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu> <b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de> <87vf90czto.fsf@welho.com>
Message-ID: <512022542edd872397b959d4e6624f2c@embl-heidelberg.de>

On 10 Feb 2005, at 14:24, Timo Korvola wrote:

> Peter Verveer <verveer at embl-heidelberg.de> writes:
>> The NA_IoArray function I don't really need.
>
> Depends on what C code you are working with.  Bidirectional array
> arguments are not uncommon in libraries.

I agree, this is just a personal preference of me to rather keep the 
input and output arrays separate arguments, which gives you the same 
functionality plus a bit more flexibility. I would not argue to remove 
NA_IoArray though, if you prefer that.

>
>> Maybe I have understood the Numeric API wrong and you can do this
>> right now, I have not used it very much lately.
>
> Perhaps you can but you would have to write it all yourself.  In
> Numarray that work has been done for you.  The three functions
> NA_{Input,Output,Io}Array are pretty much all you need for accessing C
> libraries with the usual (int size, double *data) interface, whichever
> way the data goes.

Agreed. I don't know what the work would be to extend the Numeric API 
to at least have the OutputArray functionality (I guess IoArray would 
then be easily also added), but it seems worth having it.

Peter




From jeanluc.menut at free.fr  Thu Feb 10 06:21:03 2005
From: jeanluc.menut at free.fr (Jean-Luc Menut)
Date: Thu Feb 10 06:21:03 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
Message-ID: <420B6CB7.1050907@free.fr>

Hello,

I'm want to have the modulus and the phase of a fourier transform of 2D 
array, so I use the numarray FFT to compute the FT of the array (let's 
call it ff) and I use abs(ff) for the modulus and atan(ff.imag/ff.real) 
for the phase.

My problem comes when ff.real = 0, I have either inf or nan as result 
(which is perfectly good and useful). I looking for a function which 
replace atan(ff.imag/ff.real) by 0 when ff.real is 0.

I'm currently using a loop, and I think it's not very good (I call this 
function very often). Since the determination of the phase is something 
really useful and often used, I think there is probably a lot of people 
who had the same problem.

So what solutions do you use?

Thanks,

Jean-Luc Menut



From perry at stsci.edu  Thu Feb 10 06:40:51 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 10 06:40:51 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
In-Reply-To: <420B6CB7.1050907@free.fr>
Message-ID: <NEBBIJKBMLDBLNCEEFOCOEBLFJAA.perry@stsci.edu>

Jean-Luc Menut wrote:

> 
> I'm want to have the modulus and the phase of a fourier transform of 2D 
> array, so I use the numarray FFT to compute the FT of the array (let's 
> call it ff) and I use abs(ff) for the modulus and atan(ff.imag/ff.real) 
> for the phase.
> 
> My problem comes when ff.real = 0, I have either inf or nan as result 
> (which is perfectly good and useful). I looking for a function which 
> replace atan(ff.imag/ff.real) by 0 when ff.real is 0.
> 
> I'm currently using a loop, and I think it's not very good (I call this 
> function very often). Since the determination of the phase is something 
> really useful and often used, I think there is probably a lot of people 
> who had the same problem.
> 
> So what solutions do you use?
> 
Two basic approaches come to mind:

result =  atan(ff.imag/ff.real)
result[where(ff.real == 0)] = 0.

The following is often faster (but uses more memory):

result[ff.real==0] = 0.

The first method generates a list of locations where ff.real is 0 where
the second generates a mask array where true values will be set.

Another approach is to use the functions in the module ieeespecial
to specifically change ieee special values:

import numarray.ieeespecial as ieee
ieee.setnan(result, 0.)

I hope this answers your question.

Perry Greenfield





From ariciputi at pito.com  Thu Feb 10 06:42:53 2005
From: ariciputi at pito.com (Andrea Riciputi)
Date: Thu Feb 10 06:42:53 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
In-Reply-To: <420B6CB7.1050907@free.fr>
References: <420B6CB7.1050907@free.fr>
Message-ID: <61AFD7FA-7B71-11D9-9E63-000A95C0BC68@pito.com>

Why not arctan2(ff.imag, ff.real)?

Cheers,
  Andrea.

On 10 Feb 2005, at 15:16, Jean-Luc Menut wrote:

> Hello,
>
> I'm want to have the modulus and the phase of a fourier transform of 
> 2D array, so I use the numarray FFT to compute the FT of the array 
> (let's call it ff) and I use abs(ff) for the modulus and 
> atan(ff.imag/ff.real) for the phase.
>
> My problem comes when ff.real = 0, I have either inf or nan as result 
> (which is perfectly good and useful). I looking for a function which 
> replace atan(ff.imag/ff.real) by 0 when ff.real is 0.
>
> I'm currently using a loop, and I think it's not very good (I call 
> this function very often). Since the determination of the phase is 
> something really useful and often used, I think there is probably a 
> lot of people who had the same problem.
>
> So what solutions do you use?
>
> Thanks,
>
> Jean-Luc Menut
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real 
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>




From perry at stsci.edu  Thu Feb 10 07:17:09 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 10 07:17:09 2005
Subject: [Numpy-discussion] Numeric life as I see it
In-Reply-To: <420AB928.3090004@pfdubois.com>
Message-ID: <NEBBIJKBMLDBLNCEEFOCKEBNFJAA.perry@stsci.edu>

Paul Dubois wrote:

>
> Aside: While I am at it, let me reiterate what I have said to the other
> developers privately: there is NO value to inheriting from the array
> class. Don't try to achieve that capability if it costs anything, even
> just effort, because it buys you nothing. Those of you who keep
> remarking on this as if it would simply haven't thought it through IMHO.
> It sounds so intellectually appealing that David Ascher and I had a
> version of Numeric that almost did it before we realized our folly.
>
To be contrarian, we did find great benefit (at least initially) for
inheritance for developing the record array and character array classes
since they share so many structural operations (indexing, slicing,
transposes,
concatenation, etc.) with numeric arrays. It's possible that the approach
that Travis is considering doesn't need to use inheritance to accomplish
this (I don't know enough about the details yet), but it sure did save a
lot of duplication of implementation.

I do understand what you are getting at. Any numerical array inheritance
generally forces one to reimplement all ufuncs and such, and that does
make it less useful in that case (though I still wonder if it still isn't
better than the alternatives)

Perry Greenfield





From Alexandre.Fayolle at logilab.fr  Thu Feb 10 07:27:41 2005
From: Alexandre.Fayolle at logilab.fr (Alexandre)
Date: Thu Feb 10 07:27:41 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
In-Reply-To: <420B6CB7.1050907@free.fr>
References: <420B6CB7.1050907@free.fr>
Message-ID: <20050210152551.GA1989@crater.logilab.fr>

On Thu, Feb 10, 2005 at 03:16:23PM +0100, Jean-Luc Menut wrote:
> Hello,
> 
> I'm want to have the modulus and the phase of a fourier transform of 2D 
> array, so I use the numarray FFT to compute the FT of the array (let's 
> call it ff) and I use abs(ff) for the modulus and atan(ff.imag/ff.real) 
> for the phase.
> 
> My problem comes when ff.real = 0, I have either inf or nan as result 
> (which is perfectly good and useful). I looking for a function which 
> replace atan(ff.imag/ff.real) by 0 when ff.real is 0.
> 
> I'm currently using a loop, and I think it's not very good (I call this 
> function very often). Since the determination of the phase is something 
> really useful and often used, I think there is probably a lot of people 
> who had the same problem.
> 
> So what solutions do you use?

Maybe you could try:

from Numeric import *
def phase(ff):
    return (log(ff/abs(ff))/1j).real 

>>> a = array([1, 1j,])
>>> phase(a)
array([ 0.        ,  1.57079633])


-- 
Alexandre Fayolle                              LOGILAB, Paris (France).
http://www.logilab.com   http://www.logilab.fr  http://www.logilab.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050210/b339f54c/attachment.sig>

From jmiller at stsci.edu  Thu Feb 10 08:11:41 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Thu Feb 10 08:11:41 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <512022542edd872397b959d4e6624f2c@embl-heidelberg.de>
References: <20050210002435.1C9323C889@coffee.object-craft.com.au>
	 <876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu>
	 <b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de>
	 <87vf90czto.fsf@welho.com>
	 <512022542edd872397b959d4e6624f2c@embl-heidelberg.de>
Message-ID: <1108051798.28054.176.camel@halloween.stsci.edu>

On Thu, 2005-02-10 at 08:41, Peter Verveer wrote:
> On 10 Feb 2005, at 14:24, Timo Korvola wrote:
> 
> > Peter Verveer <verveer at embl-heidelberg.de> writes:
> >> The NA_IoArray function I don't really need.
> >
> > Depends on what C code you are working with.  Bidirectional array
> > arguments are not uncommon in libraries.
> 
> I agree, this is just a personal preference of me to rather keep the 
> input and output arrays separate arguments, which gives you the same 
> functionality plus a bit more flexibility. I would not argue to remove 
> NA_IoArray though, if you prefer that.

I won't try to explain how the numarray C-API got the way it is because
it's irrelevant.  I agree that the native API is too large and started
encouraging use of the Numeric compatibility API over the summer.  On
the other hand,  I think the numarray problem domain is wider and more
difficult than the one addressed by Numeric.  

If Numeric3 intends to support byteswapped and misaligned data then
additional functionality is clearly required;  whether or not the
functionality is explicitly exposed in the API is another matter;  it is
hidden in our compatibility API so there's no reason Numeric3 can't hide
it too.  But it sounds to me like people have found the numarray high
level API fairly useful so the design features bear discussion even if
the API itself is dropped.

I used NA_IoArray myself in numarray's Numeric compatibility API and
also found it useful in adding f2py support.  I found it necessary to
get Numeric's extra packages (LinearAlgebra, FFT, RandomArray) to pass
their self-tests.   Don't overlook the fact that arrays are typically
passed by reference so whether we like it or not I/O behavior is
normally exposed.  For me,  even for a job as small as the compatibility
layer or f2py support,  NA_IoArray was an asset.  It's just easier to
think about and use and eliminates the chance of getting the shadowing
logic wrong.

The NumArray destructor-based array shadowing works well to keep things
transparent and porting changes low.  You obviously can do the explict
exit copy you've discussed,  but I think that's the kind of unnecessary
and intrusive change we're trying to get away from.  With the best code
coming from experts doing their own thing on their own time,  I don't
think support for numarray's advanced features is going to happen unless
it's as automatic and simple to use as possible.

Regards,
Todd




From verveer at embl-heidelberg.de  Thu Feb 10 08:37:03 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Thu Feb 10 08:37:03 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <1108051798.28054.176.camel@halloween.stsci.edu>
References: <20050210002435.1C9323C889@coffee.object-craft.com.au> <876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu> <b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de> <87vf90czto.fsf@welho.com> <512022542edd872397b959d4e6624f2c@embl-heidelberg.de> <1108051798.28054.176.camel@halloween.stsci.edu>
Message-ID: <56e881bc00532308eb3d5b024e6c1da1@embl-heidelberg.de>

> If Numeric3 intends to support byteswapped and misaligned data then
> additional functionality is clearly required;  whether or not the
> functionality is explicitly exposed in the API is another matter;  it 
> is
> hidden in our compatibility API so there's no reason Numeric3 can't 
> hide
> it too.  But it sounds to me like people have found the numarray high
> level API fairly useful so the design features bear discussion even if
> the API itself is dropped.

Exactly, I found the NA_*Array functions better than the original 
Numeric functions. It seems Travis is moving in a similar direction, 
but you are right that it is going to need extra functionality.

A question I have about the compatibility API in Numarray: In cases 
where a temporary is created, does that get copied back when the 
temporary is destroyed? That seems to be needed when you use an array 
as an output. That would be wasteful if you only use the array as an 
input. Its of course fine to do that, since it is a compability API 
that you cannot change, but it would point to a weakness in the 
original Numeric API, which should be fixed by the Numeric people if 
they want to  move forward.

It is relevant to me since I decided to move the nd_image to the 
Numeric compatibility API, and while doing that found myself wondering 
that in some cases I would be introducing a slight performance penalty.

> I used NA_IoArray myself in numarray's Numeric compatibility API and
> also found it useful in adding f2py support.  I found it necessary to
> get Numeric's extra packages (LinearAlgebra, FFT, RandomArray) to pass
> their self-tests.   Don't overlook the fact that arrays are typically
> passed by reference so whether we like it or not I/O behavior is
> normally exposed.  For me,  even for a job as small as the 
> compatibility
> layer or f2py support,  NA_IoArray was an asset.  It's just easier to
> think about and use and eliminates the chance of getting the shadowing
> logic wrong.

Yes, I can only agree with that.

> The NumArray destructor-based array shadowing works well to keep things
> transparent and porting changes low.  You obviously can do the explict
> exit copy you've discussed,  but I think that's the kind of unnecessary
> and intrusive change we're trying to get away from.  With the best code
> coming from experts doing their own thing on their own time,  I don't
> think support for numarray's advanced features is going to happen 
> unless
> it's as automatic and simple to use as possible.

I can confirm that the high-level API of Numarray works very nicely to 
hide you from things you do not want to know (i.e. byte-swapping) while 
at the same time allowing flexibility that does not seem to be 
completely present in the Numeric API, unless I don't understand the 
latter well enough.

Peter




From histed at MIT.EDU  Thu Feb 10 08:44:00 2005
From: histed at MIT.EDU (Mark Histed)
Date: Thu Feb 10 08:44:00 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <420A526C.3030804@csun.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu> <42096070.5030608@csun.edu> <42096303.5030805@colorado.edu> <420A526C.3030804@csun.edu>
Message-ID: <20050210164145.GB13293@mit.edu>

> Anyway, we've wandered afar from the main point:  what is the best way 
> to duplicate the functionality of the following MATLAB snippet:
> 
> f = ~isnan(data1) & ~isnan(data2);
> plot(x,data1(f),x,data2(f))

I know this has been talked about before on this list, but only in
bits and pieces.  Can I summarize the way I understand the situation
and hopefully others can correct me?  I use MATLAB and Numeric often;
I have rarely used Numarray.

Summary:

For people moving from Matlab to Numeric/Numarray, the recommendation
is to use Numeric's "&" in place of Matlab's "&" to combine logical
arrays for indexing.  Different behavior results if the inputs have
entries > 1, and you have to put parenthesis around any inputs that
use relational operators.
Also, one should use Numeric's "and" in place of Matlab's "&&".

Details:

In MATLAB (R14SP1) there are three logical "and" operators:

"&": a binary operator which performs element-by-element logical AND.
    For its purposes, 0 is FALSE and every other numerical value is TRUE.
"&&": a binary operator which performs logical AND, but does
    short-circuiting, and only works with scalar elements (length-1 arrays)
"bitand()" : a function which takes two integers and does bitwise-and
    on their binary representations

http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_prog/ch12_n11.html#38948

I think that these map onto the python operators in the following way:
MATLAB      Python/Numeric
------      --------------
&&          and
&           ufunc logical_and()
bitand      &

Some inconsistencies in behavior (between Numeric and Matlab) occur
when one uses Numeric's "&" like MATLAB "&".  I can think of:
    * Non-logical {0,1} inputs: Numeric's output is an integer, the
      bitwise and of the inputs, Matlab's output is a logical array,
      either 1 or 0. 
    * Precedence: Numeric's "&" operator is higher prec than its relational
      operators, MATLAB is the reverse

And an inconsistency between MATLAB's "&&" and Numeric's "and":
    * MATLAB returns an error when the arguments to && are not logical
      scalars, Numeric returns the second argument as the expression's
      value. (but Numarray behaves like MATLAB)



Does that seem right?  

Regards, 
Mark

    
  
----------------------------------------------------------------

Matlab example:

>> a = [0,1,2];
>> b = [1,1,63];
>> a & b
ans =
     0     1     1
>> a && b
??? Operands to the || and && operators must be convertible to logical 
    scalar values.
>> bitand(a,b)
ans =
     0     1     2
>> a > 1 & b > 1
ans =
     0     0     1
>> (a > 1) & (b > 1)
ans =
     0     0     1

----------------------------------------------------------------

Numeric example:

Python 2.3.4 (#2, Sep 24 2004, 08:39:09)
[GCC 3.3.4 (Debian 1:3.3.4-12)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Numeric as N
>>> a = N.array([0,1,2]);
>>> b = N.array([1,1,63]);
>>> N.logical_and(a,b)
array([0, 1, 1])
>>> a & b
array([0, 1, 2])
>>> a and b
array([ 1,  1, 63])
>>> a > 1 & b > 1
array([0, 0, 0])
>>> (a > 1) & (b > 1)
array([0, 0, 1])

----------------------------------------------------------------

Numarray example:
>>> import numarray as na
>>> a = na.array([0,1,2])
>>> b = na.array([1,1,63])
>>> na.logical_and(a,b)
array([0, 1, 1], type=Bool)
>>> a and b
RuntimeError: An array doesn't make sense as a truth value.  Use
    sometrue(a) or alltrue(a). 
>>> a & b
array([0, 1, 2])
>>>

----------------------------------------------------------------

On Wed, Feb 09, 2005 at 10:11:56AM -0800, Stephen Walton wrote:
> Fernando Perez wrote a good deal clarifying how Python's logical 
> operators work, for which I'm grateful.  One brief comment:
> 
> >Bytecode analysis can be really useful to understand this kind of 
> >subtle issues.  I can't think of any language where this approach is 
> >as easy to do and friendly as in python.
> 
> Pardon me, but this sounds a bit like looking at the assembly language 
> Fortran generates in order to understand the subtleties of that language 
> :-) .  I know people who used to do that, and were proud of it, but I'd 
> rather the language itself didn't behave in surprising ways.  At least 
> in Fortran, 'a .and. b' only made sense if a and b were both of type 
> LOGICAL;  in fact, it was a compile time error if they weren't.
> 

> 
> ?
> 
> 
> 
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion



From tkorvola at welho.com  Thu Feb 10 09:16:01 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Thu Feb 10 09:16:01 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <420B2898.8090808@ee.byu.edu> (Travis Oliphant's message of
 "Thu, 10 Feb 2005 02:25:44 -0700")
References: <20050210002435.1C9323C889@coffee.object-craft.com.au>
	<876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu>
Message-ID: <87ll9ws5ey.fsf@welho.com>

Travis Oliphant <oliphant at ee.byu.edu> writes:
> Can somebody show me an example of it's usefulness that could not be
> handled by simply a PyArray_ContiguousFromObject with an explicit
> copy back at the end?

It is not that it could not be done, it is the convenience of having
the library do it for you.  Is there a function in the Numeric C API
that one could use for copying the data back?  Preferably one that
would check that the source and the target are different objects
before copying.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From Chris.Barker at noaa.gov  Thu Feb 10 11:08:03 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Thu Feb 10 11:08:03 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <420ADE90.9050304@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu>		<ca471dc205020914453f4da355@mail.gmail.com>	<dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com> <420ADE90.9050304@ee.byu.edu>
Message-ID: <420BB0D8.2000101@noaa.gov>


Travis Oliphant wrote:
> I personally don't see 
> Numeric3 as in any kind of "need for speed" camp either. 

Now I'm confused. As an interested observer, It seems to me that there 
is only one thing that is keeping users of Numeric from switching to 
numarray:

"Performance with many small arrays"

Which seems to boil down to:

"Time to create a new array"

There are other small details, but from following the discussion, that 
seems to be the only show stopper. I understood that the goal of 
Numeric3 was to exactly solve that problem, while still getting all the 
nifty benefits of numarray.

Another nit: "speed" in not one goal. I understand that one of the goals 
of numarray was top performance for very large arrays. In fact, it's 
ironic that someone discovered that it was a safety check in Numeric 
that was causing it to be slower than Numarray with large arrays.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From oliphant at ee.byu.edu  Thu Feb 10 11:22:30 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 10 11:22:30 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <420BB0D8.2000101@noaa.gov>
References: <420A8406.4020808@ee.byu.edu>		<ca471dc205020914453f4da355@mail.gmail.com>	<dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com> <420ADE90.9050304@ee.byu.edu> <420BB0D8.2000101@noaa.gov>
Message-ID: <420BB448.3000900@ee.byu.edu>

Chris Barker wrote:

>
>
> Travis Oliphant wrote:
>
>> I personally don't see Numeric3 as in any kind of "need for speed" 
>> camp either. 
>
>
> Now I'm confused. As an interested observer, It seems to me that there 
> is only one thing that is keeping users of Numeric from switching to 
> numarray:
>
> "Performance with many small arrays"
>
That was the showstopper, but it was just a symptom of the Python-C 
mixture that I would never feel comfortable using as the foundation for 
the basic arrayobject.  There are also lots of little concerns over the 
burgeoning C-API that would make maintainence more difficult.  To me it 
is ironic that people have called Numeric "hackish".  In my view (and 
believe me I mean no disrespect, I know these are challenging problems) 
in comparison to the mix of Python and C code in numarray, Numeric looks 
quite elegant.  Numarray has some nice code in it, don't get me wrong, 
it's just the overall architecture and it's choice of C-API I have 
issues with.

>
> Another nit: "speed" in not one goal. I understand that one of the 
> goals of numarray was top performance for very large arrays. In fact, 
> it's ironic that someone discovered that it was a safety check in 
> Numeric that was causing it to be slower than Numarray with large arrays.

That was known for a long time, which is why scipy disabled the safety 
check.  I think numarray has a better idea for dealing with checks.  One 
that can easily be borrowed in Numeric3.

-Travis





From perry at stsci.edu  Thu Feb 10 11:29:27 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 10 11:29:27 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCCECCFJAA.perry@stsci.edu>

I don't recall if I've made any mention of our (STScI's)
stance on Numeric3 publicly. From what Travis describes,
his goals do appear to encompass all the important goals
we had for numarray (of course, we need to keep on top of
the details to make sure we haven't overlooked anything).
Should he be able to accomplish this and achieve the
performance goals for small arrays, it would be silly for
us not to take advantage of such a unification.

If the long-term future of numarray is that it served to
identify various improvements to the behavior of numeric arrays
for Numeric3, particularly by providing an illustration of
its practicality and usefulness as well as reusable code,
we think that is fine.  We are not wedded to our particular
implementation of these capabilities, nor even to all details
of the interface. We do have a strong need for the ability to
handle misaligned and byte-swapped arrays (which are crucial
for handling memory-mapped arrays and heterogeneous record arrays.)

In our view the issue isn't who owns the array package but
getting one that satisfies the needs of the great majority.
We are not sure how feasible the approach Travis is using is 
so we are taking a wait-and-see position on it. We hope he
is successful in achieving these goals.

On the other hand, that point isn't here yet, so we'll
continue doing what we are doing with numarray and making
it work with scipy. If and when the time comes when Numeric3
looks like a viable alternative to numarray (or is clearly
close enough to illustrate that it will be viable), then we
would begin the effort to substitute Numeric3 for numarray.
And if it doesn't work out, it is in our plan to look at
small array performance improvements in numarray.

We intend to stay engaged in design and interface discussions
with Travis, and encourage any others that have a stake in
numarray capabilities to do so as well.
 
Perry







From oliphant at ee.byu.edu  Thu Feb 10 12:28:23 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 10 12:28:23 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCCECCFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCCECCFJAA.perry@stsci.edu>
Message-ID: <420BC3B5.6030604@ee.byu.edu>

Perry Greenfield wrote

>In our view the issue isn't who owns the array package but
>getting one that satisfies the needs of the great majority.
>We are not sure how feasible the approach Travis is using is 
>so we are taking a wait-and-see position on it. We hope he
>is successful in achieving these goals.
>  
>
I've known it was a risk from the beginning, and I don't expect people 
to believe me until I can "show them the code." 

>We intend to stay engaged in design and interface discussions
>with Travis, and encourage any others that have a stake in
>numarray capabilities to do so as well.
>  
>

Thanks for the email, Perry.

Toward that end, I would really appreciate knowing from anybody who 
understands, which C-API calls are seen as most relevant and new.

For example, after the discussion on this list, I can understand that 
having the ability to "copy-back to a mis-behaved object" can be useful, 
and it has pointed out holes in the Numeric API (holes which we all know 
are there).   I still don't know about having three different API calls 
to do it (instead of just a flag in the requires argument).

Over the next few weeks, I will be putting together a PEP-like 
document.  This PEP will essentially discuss the behavior of the 
arrayobject that we would like to see in the Python core.    I will need 
lots of feedback and help with it, so I will post it to the Numeric page 
as it develops.    There are a couple of unresolved issues that need 
comments.  Hopefully, the PEP will help sort those out.   I think 
though, there are only a couple of issues.  And I'm confident these can 
be worked out satisfactorily.   I have no plans to submit the PEP until 
it has received attention from everybody interested.

Best regards to all,

-Travis Oliphant






From verveer at embl.de  Thu Feb 10 13:09:38 2005
From: verveer at embl.de (Peter Verveer)
Date: Thu Feb 10 13:09:38 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420BC3B5.6030604@ee.byu.edu>
References: <NEBBIJKBMLDBLNCEEFOCCECCFJAA.perry@stsci.edu> <420BC3B5.6030604@ee.byu.edu>
Message-ID: <45892a12a8669ffeccbceaef1974dbd8@embl.de>

> Toward that end, I would really appreciate knowing from anybody who 
> understands, which C-API calls are seen as most relevant and new.

The distinction between input and output (and possible I/O) arrays is 
extremely useful.

> For example, after the discussion on this list, I can understand that 
> having the ability to "copy-back to a mis-behaved object" can be 
> useful, and it has pointed out holes in the Numeric API (holes which 
> we all know are there).

It allows  you easily and effiencly implement output arguments, which I 
think is essential.

>   I still don't know about having three different API calls to do it 
> (instead of just a flag in the requires argument).

How it is finally implemented is not really important, a flag would be 
fine.

> Over the next few weeks, I will be putting together a PEP-like 
> document.  This PEP will essentially discuss the behavior of the 
> arrayobject that we would like to see in the Python core.    I will 
> need lots of feedback and help with it, so I will post it to the 
> Numeric page as it develops.    There are a couple of unresolved 
> issues that need comments.  Hopefully, the PEP will help sort those 
> out.   I think though, there are only a couple of issues.  And I'm 
> confident these can be worked out satisfactorily.   I have no plans to 
> submit the PEP until it has received attention from everybody 
> interested.

Excellent!




From cjw at sympatico.ca  Thu Feb 10 13:46:45 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 10 13:46:45 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420BC3B5.6030604@ee.byu.edu>
References: <NEBBIJKBMLDBLNCEEFOCCECCFJAA.perry@stsci.edu> <420BC3B5.6030604@ee.byu.edu>
Message-ID: <420BD5EA.4030503@sympatico.ca>

Travis Oliphant wrote:

> Perry Greenfield wrote
>
>> In our view the issue isn't who owns the array package but
>> getting one that satisfies the needs of the great majority.
>> We are not sure how feasible the approach Travis is using is so we 
>> are taking a wait-and-see position on it. We hope he
>> is successful in achieving these goals.
>>  
>>
> I've known it was a risk from the beginning, and I don't expect people 
> to believe me until I can "show them the code."
>
>> We intend to stay engaged in design and interface discussions
>> with Travis, and encourage any others that have a stake in
>> numarray capabilities to do so as well.
>>  
>>
>
> Thanks for the email, Perry.
>
> Toward that end, I would really appreciate knowing from anybody who 
> understands, which C-API calls are seen as most relevant and new.
>
> For example, after the discussion on this list, I can understand that 
> having the ability to "copy-back to a mis-behaved object" can be 
> useful, and it has pointed out holes in the Numeric API (holes which 
> we all know are there).   I still don't know about having three 
> different API calls to do it (instead of just a flag in the requires 
> argument).
>
> Over the next few weeks, I will be putting together a PEP-like 
> document.  This PEP will essentially discuss the behavior of the 
> arrayobject that we would like to see in the Python core.    I will 
> need lots of feedback and help with it, so I will post it to the 
> Numeric page as it develops.    There are a couple of unresolved 
> issues that need comments.  Hopefully, the PEP will help sort those 
> out.   I think though, there are only a couple of issues.  And I'm 
> confident these can be worked out satisfactorily.   I have no plans to 
> submit the PEP until it has received attention from everybody interested.
>
> Best regards to all,
>
> -Travis Oliphant

I am glad to see that the suggestion from Martin Loewis regarding the 
PEP approach is being taken up and hope that an early draft will be 
posted to c.l.p,

Colin W.



From bsder at mail.allcaps.org  Thu Feb 10 15:04:12 2005
From: bsder at mail.allcaps.org (Andrew P. Lentvorski, Jr.)
Date: Thu Feb 10 15:04:12 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
In-Reply-To: <420B6CB7.1050907@free.fr>
References: <420B6CB7.1050907@free.fr>
Message-ID: <8a900f8da1886681f7e65d418c4207b5@mail.allcaps.org>

On Feb 10, 2005, at 6:16 AM, Jean-Luc Menut wrote:

> My problem comes when ff.real = 0, I have either inf or nan as result 
> (which is perfectly good and useful). I looking for a function which 
> replace atan(ff.imag/ff.real) by 0 when ff.real is 0.

I would probably use atan2.

However, do you really want to replace atan(ff.imag/ff.real) by 0 when 
ff.real is 0?

I would expect that you want to replace it with +- pi/2 depending upon 
the sign of ff.imag.

-a




From rkern at ucsd.edu  Thu Feb 10 15:53:01 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Thu Feb 10 15:53:01 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
In-Reply-To: <61AFD7FA-7B71-11D9-9E63-000A95C0BC68@pito.com>
References: <420B6CB7.1050907@free.fr> <61AFD7FA-7B71-11D9-9E63-000A95C0BC68@pito.com>
Message-ID: <420BF2F8.6070104@ucsd.edu>

Andrea Riciputi wrote:
> Why not arctan2(ff.imag, ff.real)?

Yes, for the love of all that is holy, please use arctan2()!

For one thing, arctan2() solves other problems besides the divide-by-zero.

In [1]: from numarray import *

In [2]: arctan(-1/-1) == arctan(1/1)
Out[2]: True

In [3]: arctan2(-1,-1) == arctan2(1,1)
Out[3]: False

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From rbastian at club-internet.fr  Fri Feb 11 04:50:24 2005
From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=)
Date: Fri Feb 11 04:50:24 2005
Subject: [Numpy-discussion] Numeric3
Message-ID: <05021105034803.00760@rbastian>

Hello,

>From a user point of view,
in the types module of Python
I wish someting like :

from types import *
....

if type(a)==NumarrayType :
   ...



-- 
Ren? Bastian
http://www.musiques-rb.org : Musique en Python 



From jmiller at stsci.edu  Wed Feb 16 11:12:11 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Feb 16 11:12:11 2005
Subject: [Numpy-discussion] It's so *quiet* I'm testing numpy-discussion
Message-ID: <1108581069.27126.92.camel@halloween.stsci.edu>

If this message actually gets posted,  well, sorry.  It's been so quiet
on the list I thought I'd better check it.

Todd




From jdhunter at ace.bsd.uchicago.edu  Wed Feb 16 11:18:07 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Wed Feb 16 11:18:07 2005
Subject: [Numpy-discussion] It's so *quiet* I'm testing numpy-discussion
In-Reply-To: <1108581069.27126.92.camel@halloween.stsci.edu> (Todd Miller's
 message of "Wed, 16 Feb 2005 14:11:09 -0500")
References: <1108581069.27126.92.camel@halloween.stsci.edu>
Message-ID: <m3sm3wgw99.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Todd" == Todd Miller <jmiller at stsci.edu> writes:

    Todd> If this message actually gets posted, well, sorry.  It's
    Todd> been so quiet on the list I thought I'd better check it.

In neurons, after maximal excitation in the form of an action
potential or Numeric3 announcement, a period of refractory inhibition
follows.  Be alert, though, for post-inhibitory rebound....

JDH



From oliphant at ee.byu.edu  Wed Feb 16 12:44:16 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb 16 12:44:16 2005
Subject: [Numpy-discussion] It's so *quiet* I'm testing numpy-discussion
In-Reply-To: <1108581069.27126.92.camel@halloween.stsci.edu>
References: <1108581069.27126.92.camel@halloween.stsci.edu>
Message-ID: <4213B049.3040709@ee.byu.edu>

I'm busy finishing Numeric3.  It has made quite a bit of progress.  It 
can be checked out of CVS and played with.   Implementing the 
multidimensional indexing magic of numarray is taking a bit of time, 
though.  I'd like to support mixed integer indexing, boolean indexing, 
and slicing too.    Any help on this would be greatly appreciated.

I'm very confident that record arrays are going to work.   The new 
numeric object has a getfield method that lets you interpret subitem 
slices as a new type of array.  The new void * type arrays let the array 
be a collection of anything.  Constructing the fanciness of numarray 
record arrays will just be a matter of an appropriate Python container 
class or subclass.  I think all the requirements to do this are already 
done in Numeric3.

I've added a possible UPDATEIFCOPY flag to the requirements argument in 
PyArray_FromAny that will tell an array to copy back to the original 
upon deletion if this flag is set.

Typecodes are handled by a TypecodeConverter function that takes quite a 
few different types of arguments including an arbitrary Python object 
with a type_num and itemsize attribute.  This allows a lot of 
flexibility in how types are specified.

Multi-byte character and unicode arrays already seem to work reasonably 
well.  There is a lot more testing that needs to be done, of course.

I would also love feedback and/or help on the PEP that I've started.  It 
is online at numeric.scipy.org but that copy may be a few days behind 
the CVS version at sourceforge.

-Travis






From southey at uiuc.edu  Wed Feb 16 12:59:17 2005
From: southey at uiuc.edu (Bruce Southey)
Date: Wed Feb 16 12:59:17 2005
Subject: [Numpy-discussion] random and RandomArray
Message-ID: <24e418bb.cfb24d1c.81ae300@expms6.cites.uiuc.edu>

Hi, 
I was browsing through some of the code and realized that certain random number 
generators occur in both the Python random module and RandomArray.  
 
The random module uses C code to get a scalar uniform random number that is 
modified in Python by other generators to get random numbers from other 
distributions. In RandomArray everything is done in the C code - obviously this 
is way faster especially for arrays. 
 
In the long term, would it make sense to get use the same random number 
generators in both random and RandomArray? 
By this, I am thinking along the lines that both random and RandomArray would 
call the same C code. This would require more changes in random than 
RandomArray but would minimise duplicate code and update RandomArray. 
 
So I was wondering what the thoughts of people on this list are on this before 
even considering this further. 
 
Regards 
Bruce Southey 



From haase at msg.ucsf.edu  Wed Feb 16 13:45:15 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed Feb 16 13:45:15 2005
Subject: [Numpy-discussion] It's so *quiet* I'm testing numpy-discussion
In-Reply-To: <4213B049.3040709@ee.byu.edu>
References: <1108581069.27126.92.camel@halloween.stsci.edu> <4213B049.3040709@ee.byu.edu>
Message-ID: <200502161344.43073.haase@msg.ucsf.edu>

On Wednesday 16 February 2005 12:42 pm, Travis Oliphant wrote:
> I'm busy finishing Numeric3.  It has made quite a bit of progress.  It
> can be checked out of CVS and played with.   Implementing the
> multidimensional indexing magic of numarray is taking a bit of time,
> though.  I'd like to support mixed integer indexing, boolean indexing,
> and slicing too.    Any help on this would be greatly appreciated.
>
> I'm very confident that record arrays are going to work.   The new
> numeric object has a getfield method that lets you interpret subitem
> slices as a new type of array.  The new void * type arrays let the array
> be a collection of anything.  Constructing the fanciness of numarray
> record arrays will just be a matter of an appropriate Python container
> class or subclass.  I think all the requirements to do this are already
> done in Numeric3.
>
Hi Travis,
I'm a committed numarray user since numarray 0.3.6 (I was told not to look 
into Numeric, because it did not have UInt16 which I needed)
Now, What you say about record arrays makes it sound like you are trying to 
"catch up" on numarray by providing these same features in Numeric.
Will you also look into memmap ?  For us this turned into one of the most 
amazing features of numarray: that we can access 500MB files instantaneously.
How about the speed advantage that numarray was having over Numeric for large 
arrays ?

I just wanted to show my interest in this discussion and thank everbody 
(numarray, scipy, Numeric, ...) for the great tools !

Sebastian Haase





> I've added a possible UPDATEIFCOPY flag to the requirements argument in
> PyArray_FromAny that will tell an array to copy back to the original
> upon deletion if this flag is set.
>
> Typecodes are handled by a TypecodeConverter function that takes quite a
> few different types of arguments including an arbitrary Python object
> with a type_num and itemsize attribute.  This allows a lot of
> flexibility in how types are specified.
>
> Multi-byte character and unicode arrays already seem to work reasonably
> well.  There is a lot more testing that needs to be done, of course.
>
> I would also love feedback and/or help on the PEP that I've started.  It
> is online at numeric.scipy.org but that copy may be a few days behind
> the CVS version at sourceforge.
>
> -Travis



From mdehoon at ims.u-tokyo.ac.jp  Wed Feb 16 18:44:14 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Wed Feb 16 18:44:14 2005
Subject: [Numpy-discussion] random and RandomArray
In-Reply-To: <24e418bb.cfb24d1c.81ae300@expms6.cites.uiuc.edu>
References: <24e418bb.cfb24d1c.81ae300@expms6.cites.uiuc.edu>
Message-ID: <421404CF.7080707@ims.u-tokyo.ac.jp>

For what it's worth, I think that this is a good idea. I'd rather have 
one really good random number package than two separate ones.

--Michiel.

Bruce Southey wrote:
> Hi, 
> I was browsing through some of the code and realized that certain random number 
> generators occur in both the Python random module and RandomArray.  
>  
> The random module uses C code to get a scalar uniform random number that is 
> modified in Python by other generators to get random numbers from other 
> distributions. In RandomArray everything is done in the C code - obviously this 
> is way faster especially for arrays. 
>  
> In the long term, would it make sense to get use the same random number 
> generators in both random and RandomArray? 
> By this, I am thinking along the lines that both random and RandomArray would 
> call the same C code. This would require more changes in random than 
> RandomArray but would minimise duplicate code and update RandomArray. 
>  
> So I was wondering what the thoughts of people on this list are on this before 
> even considering this further. 
>  
> Regards 
> Bruce Southey 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 
> 




From rkern at ucsd.edu  Wed Feb 16 20:06:17 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Wed Feb 16 20:06:17 2005
Subject: [Numpy-discussion] random and RandomArray
In-Reply-To: <24e418bb.cfb24d1c.81ae300@expms6.cites.uiuc.edu>
References: <24e418bb.cfb24d1c.81ae300@expms6.cites.uiuc.edu>
Message-ID: <421417F5.3010703@ucsd.edu>

Bruce Southey wrote:
> Hi, 
> I was browsing through some of the code and realized that certain random number 
> generators occur in both the Python random module and RandomArray.  
>  
> The random module uses C code to get a scalar uniform random number that is 
> modified in Python by other generators to get random numbers from other 
> distributions. In RandomArray everything is done in the C code - obviously this 
> is way faster especially for arrays. 
>  
> In the long term, would it make sense to get use the same random number 
> generators in both random and RandomArray? 

I think that it is certainly feasible and desirable that when/if a 
multiarray object enters the standard library that the standard random 
module be extended to produce arrays as well.

I don't think it's terribly worthwhile to hack the random module to 
expose its PRNG so that we use it's implementation without duplicating 
code. I think the result will be quite fragile, and won't be useful 
until Python 2.5.

I do think it would be extremely worthwhile to implement the Mersenne 
Twister for numarray/scipy. I promised some time ago to look into this, 
but I have not gotten around to it, unfortunately.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From oliphant at ee.byu.edu  Wed Feb 16 23:26:15 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb 16 23:26:15 2005
Subject: [Numpy-discussion] Multiarray PEP
Message-ID: <421446E3.7080202@ee.byu.edu>

I am looking for feedback on the PEP.   Of particular interest is the 
specification of multidimensional indexing that I've outlined.  I think 
it is mostly the same as numarray (I'd love some feedback to be sure 
about that), except for the fact that X[K] where K is a a single integer 
index array can do 1-d indexing similar to MATLAB.  The equivalent 
numarray indexing is available as X[K,].    Now that I've specified what 
is to happen, I think it won't be very difficult to code. 

I would also like some help on resolving the type issue.   Is it 
important to have hierarchial classes defined (probably in Python) that 
can be used to check and/or specify type or are functions that check for 
various properties, fine.   Right now x.type is specified as an 
attribute but the attribute could be replaced by a method.  What to 
return is the biggest question.

I've never had a problem with just the typecode characters although 
aliases (like Int16 or Short) which might be more meaningful are nice.   
Are shadow-classes important in this case?  To me they look a little 
like extra bulk with no gain.  I'd love to be educated as to why it is 
worth the extra effort to have a bunch of instances lying around.  I 
don't feel very strongly about this though, so if somebody else does, 
I'm willing to go for it.

I've designed Numeric3 so that it is not difficult to change how 
typecodes are specified on the Python level.   We can really accommodate 
whatever.

<Opening can ....>

What is the opinion of everybody?

-Travis




From verveer at embl-heidelberg.de  Thu Feb 17 01:27:17 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Thu Feb 17 01:27:17 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <421446E3.7080202@ee.byu.edu>
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <d8c7c031fc63e548680c24b1b66ea6f3@embl-heidelberg.de>

> I would also like some help on resolving the type issue.   Is it 
> important to have hierarchial classes defined (probably in Python) 
> that can be used to check and/or specify type or are functions that 
> check for various properties, fine.   Right now x.type is specified as 
> an attribute but the attribute could be replaced by a method.  What to 
> return is the biggest question.
>
> I've never had a problem with just the typecode characters although 
> aliases (like Int16 or Short) which might be more meaningful are nice. 
>   Are shadow-classes important in this case?  To me they look a little 
> like extra bulk with no gain.  I'd love to be educated as to why it is 
> worth the extra effort to have a bunch of instances lying around.  I 
> don't feel very strongly about this though, so if somebody else does, 
> I'm willing to go for it.
>
> I've designed Numeric3 so that it is not difficult to change how 
> typecodes are specified on the Python level.   We can really 
> accommodate whatever.

I am used to the numarray convention of using Int16 and so on, which to 
me are more meaningful and far more easy to remember then typecode 
characters. These could be aliases for character codes, which for most 
purposes would be fine for me. The only practical problem that comes to 
mind now is that if you print a type, you would get its character code, 
which I would find inconvenient as I generally don't use them. By using 
a python class you can use any string representation you like. But I 
suppose you could just provide a function/method to return a string 
representation for an alias to solve that.

Peter




From magnus at hetland.org  Thu Feb 17 01:35:20 2005
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Thu Feb 17 01:35:20 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <421446E3.7080202@ee.byu.edu>
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <20050217093428.GA7324@idi.ntnu.no>

Travis Oliphant <oliphant at ee.byu.edu>:
>
> 
> I am looking for feedback on the PEP.   Of particular interest is the 
> specification of multidimensional indexing that I've outlined.  I think 
> it is mostly the same as numarray (I'd love some feedback to be sure 
> about that), except for the fact that X[K] where K is a a single integer 
> index array can do 1-d indexing similar to MATLAB.  The equivalent 
> numarray indexing is available as X[K,].    Now that I've specified what 
> is to happen, I think it won't be very difficult to code. 

Sorry if I'm a bit quick to respond here (haven't read the PEP yet; I
seem to have lost the URL) but ... is this related to the following
issue (which I struggled with for a while back in 2002)?

  http://sourceforge.net/mailarchive/message.php?msg_id=2700193

IMO, special-casing (if that's what you're proposing) is a Bad
Think(tm), as general code will then have to add special-casing to
avoid tripping over it... (But I may just be confused here :)

- M

-- 
Magnus Lie Hetland       Fallen flower I see / Returning to its branch
http://hetland.org       Ah! a butterfly.           [Arakida Moritake]



From magnus at hetland.org  Thu Feb 17 01:37:15 2005
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Thu Feb 17 01:37:15 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <20050217093428.GA7324@idi.ntnu.no>
References: <421446E3.7080202@ee.byu.edu> <20050217093428.GA7324@idi.ntnu.no>
Message-ID: <20050217093641.GB7324@idi.ntnu.no>

Magnus Lie Hetland <magnus at hetland.org>:
>
> Bad Think(tm)

... or Thing, as the case may be ;)

-- 
Magnus Lie Hetland       Fallen flower I see / Returning to its branch
http://hetland.org       Ah! a butterfly.           [Arakida Moritake]



From cjw at sympatico.ca  Thu Feb 17 05:39:26 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 17 05:39:26 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <d8c7c031fc63e548680c24b1b66ea6f3@embl-heidelberg.de>
References: <421446E3.7080202@ee.byu.edu> <d8c7c031fc63e548680c24b1b66ea6f3@embl-heidelberg.de>
Message-ID: <42149E40.9000800@sympatico.ca>

Peter Verveer wrote:

>> I would also like some help on resolving the type issue.   Is it 
>> important to have hierarchial classes defined (probably in Python) 
>> that can be used to check and/or specify type or are functions that 
>> check for various properties, fine.   Right now x.type is specified 
>> as an attribute but the attribute could be replaced by a method.  
>> What to return is the biggest question.
>>
>> I've never had a problem with just the typecode characters although 
>> aliases (like Int16 or Short) which might be more meaningful are 
>> nice.   Are shadow-classes important in this case?  To me they look a 
>> little like extra bulk with no gain.  I'd love to be educated as to 
>> why it is worth the extra effort to have a bunch of instances lying 
>> around.  I don't feel very strongly about this though, so if somebody 
>> else does, I'm willing to go for it.
>>
>> I've designed Numeric3 so that it is not difficult to change how 
>> typecodes are specified on the Python level.   We can really 
>> accommodate whatever.
>
>
> I am used to the numarray convention of using Int16 and so on, which 
> to me are more meaningful and far more easy to remember then typecode 
> characters. These could be aliases for character codes, which for most 
> purposes would be fine for me. The only practical problem that comes 
> to mind now is that if you print a type, you would get its character 
> code, which I would find inconvenient as I generally don't use them. 
> By using a python class you can use any string representation you 
> like. But I suppose you could just provide a function/method to return 
> a string representation for an alias to solve that.
>
> Peter
>
I agree, the use of numerictypes is the better way to go.  By all means 
provide a tool of some sort to permit onversion from or to the C codes.

Colin W.



From barrett at stsci.edu  Thu Feb 17 05:58:19 2005
From: barrett at stsci.edu (Paul Barrett)
Date: Thu Feb 17 05:58:19 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <421446E3.7080202@ee.byu.edu>
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <4214A2A3.9030401@stsci.edu>

Travis Oliphant wrote:
> 
> I am looking for feedback on the PEP.   Of particular interest is the 
> specification of multidimensional indexing that I've outlined.  I think 
> it is mostly the same as numarray (I'd love some feedback to be sure 
> about that), except for the fact that X[K] where K is a a single integer 
> index array can do 1-d indexing similar to MATLAB.  The equivalent 
> numarray indexing is available as X[K,].    Now that I've specified what 
> is to happen, I think it won't be very difficult to code.

Your indexing implementation appears reasonable.

> I would also like some help on resolving the type issue.   Is it 
> important to have hierarchial classes defined (probably in Python) that 
> can be used to check and/or specify type or are functions that check for 
> various properties, fine.   Right now x.type is specified as an 
> attribute but the attribute could be replaced by a method.  What to 
> return is the biggest question.

I don't see the need for hierarchical type classes, though it might ease 
the comparison with Python types.

> I've never had a problem with just the typecode characters although 
> aliases (like Int16 or Short) which might be more meaningful are nice.   
> Are shadow-classes important in this case?  To me they look a little 
> like extra bulk with no gain.  I'd love to be educated as to why it is 
> worth the extra effort to have a bunch of instances lying around.  I 
> don't feel very strongly about this though, so if somebody else does, 
> I'm willing to go for it.

I prefer Int16, UInt16, Float32, Float64, etc. for specifying types. 
They are unambiguous and the scheme is easily extendable.  I also don't 
mind character code aliases for named types as long as they are a 
shorthand for the them and are not used routinely when doing a repr.

  -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218



From tkorvola at welho.com  Thu Feb 17 06:53:26 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Thu Feb 17 06:53:26 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <421446E3.7080202@ee.byu.edu> (Travis Oliphant's message of
 "Thu, 17 Feb 2005 00:25:23 -0700")
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <87oeej6xxh.fsf@welho.com>

Travis Oliphant <oliphant at ee.byu.edu> writes:
> except for the fact that X[K] where K is a a single
> integer index array can do 1-d indexing similar to MATLAB.

I find the current Numarray behaviour more logical.  Is 1-D indexing
really useful enough to require a shorter and less clear notation
than X.flat[K]?

Another one that I don't like is capping the indices.  Is it useful or
just a potential source of obscure errors?  Throwing an IndexError for
out of range indices would seem safer.

Perhaps only one ellipsis should be allowed for clarity?  Is it useful
to allow more if they are only equivalent to colons?

Integers scalars are not on the list where you describe how different
types are interpreted in advanced indexing.  I suppose the intention
is to treat them as rank 0 arrays, broadcastable to any shape.

The rank of a basic index expression can only be zero if all
dimensions are indexed and all the indices are integers.  Are you
really suggesting that a rank 0 array rather than a scalar be returned
in that case?

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From southey at uiuc.edu  Thu Feb 17 07:28:19 2005
From: southey at uiuc.edu (Bruce Southey)
Date: Thu Feb 17 07:28:19 2005
Subject: [Numpy-discussion] random and RandomArray
Message-ID: <a27efba0.d017d219.81aed00@expms6.cites.uiuc.edu>

Hi, 
The Mersenne Twister is available in Python as: 
 
/Python-2.4/Modules/_randommodule.c 
 
This file contains the outdated information: 
"The code in this module was based on a download from: 
          http://www.math.keio.ac.jp/~matumoto/MT2002/emt19937ar.html" 
 
The redirected link is titled: Mersenne Twister with improved initialization 
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html 
 
A brief look at randlib() suggests that it is written so that different uniform 
generators could be used. Could randf() be rewritten to have the default 
(current status) function and a second that just to link to the random module 
library to use the Mersenne Twister?  
 
Okay, I do know it is not that easy (the seed code would also need to change so 
account for which generator is being used) but may be sufficient. 
 
Regards 
Bruce 
 
---- Original message ---- 
>Date: Wed, 16 Feb 2005 20:05:09 -0800 
>From: Robert Kern <rkern at ucsd.edu>   
>Subject: Re: [Numpy-discussion] random and RandomArray   
>To: numpy-discussion at lists.sourceforge.net 
> 
>Bruce Southey wrote: 
>> Hi,  
>> I was browsing through some of the code and realized that certain random 
number  
>> generators occur in both the Python random module and RandomArray.   
>>   
>> The random module uses C code to get a scalar uniform random number that is  
>> modified in Python by other generators to get random numbers from other  
>> distributions. In RandomArray everything is done in the C code - obviously 
this  
>> is way faster especially for arrays.  
>>   
>> In the long term, would it make sense to get use the same random number  
>> generators in both random and RandomArray?  
> 
>I think that it is certainly feasible and desirable that when/if a  
>multiarray object enters the standard library that the standard random  
>module be extended to produce arrays as well. 
> 
>I don't think it's terribly worthwhile to hack the random module to  
>expose its PRNG so that we use it's implementation without duplicating  
>code. I think the result will be quite fragile, and won't be useful  
>until Python 2.5. 
> 
>I do think it would be extremely worthwhile to implement the Mersenne  
>Twister for numarray/scipy. I promised some time ago to look into this,  
>but I have not gotten around to it, unfortunately. 
> 
>--  
>Robert Kern 
>rkern at ucsd.edu 
> 
>"In the fields of hell where the grass grows high 
>  Are the graves of dreams allowed to die." 
>   -- Richard Harter 
> 
> 
>------------------------------------------------------- 
>SF email is sponsored by - The IT Product Guide 
>Read honest & candid reviews on hundreds of IT Products from real users. 
>Discover which products truly live up to the hype. Start reading now. 
>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click 
>_______________________________________________ 
>Numpy-discussion mailing list 
>Numpy-discussion at lists.sourceforge.net 
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion 
 



From rkern at ucsd.edu  Thu Feb 17 07:45:19 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Thu Feb 17 07:45:19 2005
Subject: [Numpy-discussion] random and RandomArray
In-Reply-To: <a27efba0.d017d219.81aed00@expms6.cites.uiuc.edu>
References: <a27efba0.d017d219.81aed00@expms6.cites.uiuc.edu>
Message-ID: <4214BBBE.1060704@ucsd.edu>

Bruce Southey wrote:
> Hi, 
> The Mersenne Twister is available in Python as: 
>  
> /Python-2.4/Modules/_randommodule.c 
>  
> This file contains the outdated information: 
> "The code in this module was based on a download from: 
>           http://www.math.keio.ac.jp/~matumoto/MT2002/emt19937ar.html" 
>  
> The redirected link is titled: Mersenne Twister with improved initialization 
> http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html 
>  
> A brief look at randlib() suggests that it is written so that different uniform 
> generators could be used. Could randf() be rewritten to have the default 
> (current status) function and a second that just to link to the random module 
> library to use the Mersenne Twister?  
>  
> Okay, I do know it is not that easy (the seed code would also need to change so 
> account for which generator is being used) but may be sufficient. 

All the functions from _randommodule.c that we would need to use are 
declared static. I'm fairly sure this precludes trying to "link" to 
_randommodule.so to use these functions. The standard way to do such a 
thing would be for _randommodule.c to expose function pointers as 
PyCObjects, but that would require modification of _randommodule.c.

The path of least resistence is to just duplicate the code. It's short, 
clean, well-tested, and widely-used. It should not present a real 
maintenance problem. Certainly no more of one trying to share the 
implementation with _randommodule.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From faltet at carabos.com  Thu Feb 17 07:58:11 2005
From: faltet at carabos.com (Francesc Altet)
Date: Thu Feb 17 07:58:11 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <4214A2A3.9030401@stsci.edu>
References: <421446E3.7080202@ee.byu.edu> <4214A2A3.9030401@stsci.edu>
Message-ID: <200502171657.09055.faltet@carabos.com>

A Dijous 17 Febrer 2005 14:56, Paul Barrett va escriure:
> I prefer Int16, UInt16, Float32, Float64, etc. for specifying types.
> They are unambiguous and the scheme is easily extendable.  I also don't
> mind character code aliases for named types as long as they are a
> shorthand for the them and are not used routinely when doing a repr.

I completely agree with that point of view too.

-- 
>qo<   Francesc Altet ? ? http://www.carabos.com/
V ?V   C?rabos Coop. V. ??Enjoy Data
 ""




From klimek at grc.nasa.gov  Thu Feb 17 08:14:15 2005
From: klimek at grc.nasa.gov (Bob Klimek)
Date: Thu Feb 17 08:14:15 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <200502171657.09055.faltet@carabos.com>
References: <421446E3.7080202@ee.byu.edu> <4214A2A3.9030401@stsci.edu> <2005
	02171657.09055.faltet@carabos.com>
Message-ID: <4214C3DC.6010909@grc.nasa.gov>

Francesc Altet wrote:

>A Dijous 17 Febrer 2005 14:56, Paul Barrett va escriure:
>  
>
>>I prefer Int16, UInt16, Float32, Float64, etc. for specifying types.
>>They are unambiguous and the scheme is easily extendable.  I also don't
>>mind character code aliases for named types as long as they are a
>>shorthand for the them and are not used routinely when doing a repr.
>>    
>>
>
>I completely agree with that point of view too.
>
>  
>
I also agree.

Bob




From perry at stsci.edu  Thu Feb 17 08:29:22 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 17 08:29:22 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <421446E3.7080202@ee.byu.edu>
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <4867e3b08186795e72c4d52e2a02ae17@stsci.edu>

I'm generally going to refrain from making comments since it usually is 
obvious what we prefer. But there may be exceptions (perhaps a reminder 
of past arguments, a desire to change numarray behavior, or clarify 
what the issue is).

On Feb 17, 2005, at 2:25 AM, Travis Oliphant wrote:

>
> I am looking for feedback on the PEP.   Of particular interest is the 
> specification of multidimensional indexing that I've outlined.  I 
> think it is mostly the same as numarray (I'd love some feedback to be 
> sure about that), except for the fact that X[K] where K is a a single 
> integer index array can do 1-d indexing similar to MATLAB.  The 
> equivalent numarray indexing is available as X[K,].    Now that I've 
> specified what is to happen, I think it won't be very difficult to 
> code.
Could you clarify this with an example? I'm not quite sure what 
alternatives you are addressing.

> I would also like some help on resolving the type issue.   Is it 
> important to have hierarchial classes defined (probably in Python) 
> that can be used to check and/or specify type or are functions that 
> check for various properties, fine.   Right now x.type is specified as 
> an attribute but the attribute could be replaced by a method.  What to 
> return is the biggest question.

I'll just note that the hierarchial approach allows simpler tests for 
kinds of types (e.g., is the type an integer type) rather than doing 
something like ntype in [Int16, Int8,...].

Thanks, Perry




From tkorvola at welho.com  Thu Feb 17 08:54:11 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Thu Feb 17 08:54:11 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <4214A2A3.9030401@stsci.edu> (Paul Barrett's message of "Thu,
 17 Feb 2005 08:56:51 -0500")
References: <421446E3.7080202@ee.byu.edu> <4214A2A3.9030401@stsci.edu>
Message-ID: <87braj6scr.fsf@welho.com>

Paul Barrett <barrett at stsci.edu> writes:
> I prefer Int16, UInt16, Float32, Float64, etc. for specifying
> types.

The C based types have their uses: if you want to call external
libraries that use double or int from Python, you will want to work
with arrays of double or int without having to care how many bits they
happen to be on the current platform.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From verveer at embl-heidelberg.de  Thu Feb 17 08:58:27 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Thu Feb 17 08:58:27 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <87braj6scr.fsf@welho.com>
References: <421446E3.7080202@ee.byu.edu> <4214A2A3.9030401@stsci.edu> <87braj6scr.fsf@welho.com>
Message-ID: <1921b9ac31a272057b946aec6a3d49d8@embl-heidelberg.de>

On 17 Feb 2005, at 17:53, Timo Korvola wrote:

> Paul Barrett <barrett at stsci.edu> writes:
>> I prefer Int16, UInt16, Float32, Float64, etc. for specifying
>> types.
>
> The C based types have their uses: if you want to call external
> libraries that use double or int from Python, you will want to work
> with arrays of double or int without having to care how many bits they
> happen to be on the current platform.

Agreed, I find them very useful for such cases. Normally I use the 
exact specifications, but I agree that the C based types should be 
around. Fortunately that should be easy to support.




From faltet at carabos.com  Thu Feb 17 10:01:22 2005
From: faltet at carabos.com (Francesc Altet)
Date: Thu Feb 17 10:01:22 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <1921b9ac31a272057b946aec6a3d49d8@embl-heidelberg.de>
References: <421446E3.7080202@ee.byu.edu> <87braj6scr.fsf@welho.com> <1921b9ac31a272057b946aec6a3d49d8@embl-heidelberg.de>
Message-ID: <200502171900.56294.faltet@carabos.com>

A Dijous 17 Febrer 2005 17:56, Peter Verveer va escriure:
> On 17 Feb 2005, at 17:53, Timo Korvola wrote:
> Agreed, I find them very useful for such cases. Normally I use the
> exact specifications, but I agree that the C based types should be
> around. Fortunately that should be easy to support.

Yes, I think so. Why not .type for Python types (and numarray
compatibility) and .typecode for C types (and Numeric compatibility)?.
numarray supports both right now without problems.

Besides, in numarray.records module, you may find the following map:

numfmt = {'i1':num.Int8,  'u1':num.UInt8,
          'i2':num.Int16, 'u2':num.UInt16,
          'i4':num.Int32, 'u4':num.UInt32,
          'i8':num.Int64, 'u8':num.UInt64,
          'f4':num.Float32, 'f8':num.Float64,
          'c8':num.Complex32, 'c16':num.Complex64,
          'b1':num.Bool}

which I find very useful and intuitive, but I guess it is incompatible
with current Numeric convention (that I find quite confusing, anyway).

Cheers,

-- 
>qo<   Francesc Altet ? ? http://www.carabos.com/
V ?V   C?rabos Coop. V. ??Enjoy Data
 ""




From oliphant at ee.byu.edu  Thu Feb 17 10:53:22 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 17 10:53:22 2005
Subject: [Numpy-discussion] Response to PEP suggestions
Message-ID: <4214E7FE.9090403@ee.byu.edu>

I'm glad to get the feedback.

1) Types

I like Francesc's suggestion that .typecode return a code and .type 
return a Python class.   What is the attitude and opinion regarding the 
use of attributes or methods for
this kind of thing?  It always seems to me so arbitrary as to what is an 
attribute or what
is a method.

There will definitely be support for the nummary-style type 
specification.   Something like that will be how they print (I like the 
'i4', 'f4', specification a bit better though). There will also be 
support for specification in terms of a c-type.  The typecodes will 
still be there, underneath.

One thing has always bothered me though.  Why is a double complex type 
Complex64? and a float complex type Complex32.  This seems to break the 
idea that the number at the end specifies a bit width.   Why don't we 
just call it Complex64 and Complex128?  Can we change this?

I'm also glad that some recognize the problems with always requiring 
specification of types in terms of bit-width or byte-widths as these are 
not the same across platforms.  For some types (like Int8 or Int16) this 
is not a problem.   But what about long double?  On an intel machine 
long double is Float96 while on a PowerPC it is Float128.   Wouldn't it 
just be easier to specify LDouble or 'g' then special-case your code?

Problems also exist when you are interfacing with hardware or other C or 
Fortran code.  You know you want single-precision floating point.  You 
don't know or care what the bit-width is.    I think with the Integer 
types the bit-width specification is more important than floating point 
types.  In sum, I think it is important to have the ability to specify 
it both ways.   When printing the array, it's probably better if it 
gives bit-width information.  I like the way numarray prints arrays.


2) Multidimensional array indexing.

Sometimes it is useful to select out of an array some elements based on 
it's linear (flattened) index in the array.   MATLAB, for example, will 
allow you to take a three-dimensional array and index it with a single 
integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...

What I'm proposing would have X[K] essentially equivalent to X.flat[K].  
The problem with always requiring the use of X.flat[K] is that X.flat 
does not work for discontiguous arrays.   It could be made to work if 
X.flat returned some kind of specially-marked array, which would then 
have to be checked every time indexing occurred for any array.  Or, 
there maybe someway to have X.flat return an "indexable iterator" for X 
which may be a more Pythonic thing to do anyway.  That could solve the 
problem and solve the discontiguous X.flat problem as well. 

If we can make X.flat[K] work for discontiguous arrays, then I would be 
very happy to not special-case the single index array but always treat 
it as a 1-tuple of integer index arrays.

Capping indexes was proposed because of what numarray does.   I can only 
think that the benefit would be that you don't have to check for and 
raise an error in the middle of an indexing loop or pre-scan the 
indexes.  But, I suppose this is unavoidalbe, anyway.  Currently Numeric 
allows specifying indexes that are too high in slices. It just chops 
them.  Python allows this too, for slices.  So, I guess I'm just 
specifying Python behavior.  Of course indexing with an integer that is 
too large or too small will raise errors:

In Python:

a = [1,2,3,4,5]
a[:20]   works
a[20] raises an error.


3)  Always returning rank-0 arrays.

This may be a bit controversial as it is a bit of a change.  But, my 
experience is that quite a bit of extra code is written to check whether 
or not a calculation returns a Python-scalar (because these don't have 
the same methods as arrays).  In particular len(a) does not work if a is 
a scalar, but len(b) works if b is a rank-0 array (numeric scalar).   
Rank-0 arrays are scalars.  

When Python needs a scalar it will generally ask the object if it can 
turn itself into an int or a float.   A notable exception is indexing in 
a list (where Python needs an integer and won't ask the object to 
convert if it can).  But int(b) always returns a Python integer if the 
array has only 1 element. 

I'd like to know what reasons people can think of for ever returning 
Python scalars  unless explicitly asked for.


Thanks for the suggestions.

-Travis








From Fernando.Perez at colorado.edu  Thu Feb 17 11:12:21 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu Feb 17 11:12:21 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <4214EC72.6060106@colorado.edu>

Just a small comment:

Travis Oliphant wrote:

> Capping indexes was proposed because of what numarray does.   I can only 
> think that the benefit would be that you don't have to check for and 
> raise an error in the middle of an indexing loop or pre-scan the 
> indexes.  But, I suppose this is unavoidalbe, anyway.  Currently Numeric 
> allows specifying indexes that are too high in slices. It just chops 
> them.  Python allows this too, for slices.  So, I guess I'm just 
> specifying Python behavior.  Of course indexing with an integer that is 
> too large or too small will raise errors:
> 
> In Python:
> 
> a = [1,2,3,4,5]
> a[:20]   works
> a[20] raises an error.

This feature is extremely useful.  Just yesterday, I needed some code to check 
whether the first character in a (potentially empty) string was one of a 
certain list.  I couldn't use .startswith(), because I'd have to call it for 
each test, and I didn't feel like writing a regexp (since I abandoned Perl 
coding, I've mostly forgotten them and I need to look up the syntax every 
time).  The danger with:

if mystr[0] in ('a','b','c'): ...

is that if mystr is empty, this blows up.  Thanks to Python's acceptance of 
invalid indices in slices, I used

if mystr[0:1] in ('a','b','c'): ...

This is exactly identical to the first case, except that if the string is 
empty, mystr[0:1] returns ''.  No need for extra checks, no need for a 
try/except block, nothing.  Clean, elegant and to the point.

Given that python supports these semantics for all their sliceable containers, 
I'd very much vote for numerix doing the same.

There are cases where it makes sense for numerix arrays to have different 
semantics from python lists.  The slice-as-view is probably the best example. 
  But unless there is a strong reason to do so, I think numerix arrays should 
not deviate from python sequence behavior.

Regards,

f



From dd55 at cornell.edu  Thu Feb 17 11:39:18 2005
From: dd55 at cornell.edu (Darren Dale)
Date: Thu Feb 17 11:39:18 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214EC72.6060106@colorado.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4214EC72.6060106@colorado.edu>
Message-ID: <200502171438.06720.dd55@cornell.edu>

On Thursday 17 February 2005 02:11 pm, Fernando Perez wrote:
>  The danger with:
>
> if mystr[0] in ('a','b','c'): ...
>
> is that if mystr is empty, this blows up.  Thanks to Python's acceptance of
> invalid indices in slices, I used
>
> if mystr[0:1] in ('a','b','c'): ...
>

Has Numeric or numarray considered allowing assignment to invalid indices? I 
sometimes miss being able to do this:
a=zeros(2)
a[1:0]=1

in order to get this:
[[0,0],
 [1,0]]

I guess it is not consistent with the list operations in python, but it is a 
really handy shortcut for constructing arrays interactively.


Darren



From oliphant at ee.byu.edu  Thu Feb 17 11:48:21 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 17 11:48:21 2005
Subject: [Numpy-discussion] Expanding an array by assignment
In-Reply-To: <200502171438.06720.dd55@cornell.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4214EC72.6060106@colorado.edu> <200502171438.06720.dd55@cornell.edu>
Message-ID: <4214F4C8.5080100@ee.byu.edu>

Darren Dale wrote:

>Has Numeric or numarray considered allowing assignment to invalid indices? I 
>sometimes miss being able to do this:
>a=zeros(2)
>a[1:0]=1
>
>in order to get this:
>[[0,0],
> [1,0]]
>
>I guess it is not consistent with the list operations in python, but it is a 
>really handy shortcut for constructing arrays interactively.
>  
>

I'm not sure what you meant by this.  Did you mean being able to expand 
an array only by assignment?

i.e.

a = zeros(2)  a (2,1) array

a[1,0] = 1    (resizes a behind the scenes to a 2x2 array and then sets 
the 1,0 element)?

MATLAB does display this behavior.  I'm not sure how difficult it would 
be to support it or if it would be worth it or not.  What does IDL do? 

-Travis




From Fernando.Perez at colorado.edu  Thu Feb 17 12:01:15 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu Feb 17 12:01:15 2005
Subject: [Numpy-discussion] Expanding an array by assignment
In-Reply-To: <4214F4C8.5080100@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4214EC72.6060106@colorado.edu> <200502171438.06720.dd55@cornell.edu> <4214F4C8.5080100@ee.byu.edu>
Message-ID: <4214F6F5.8030608@colorado.edu>

Travis Oliphant wrote:

> I'm not sure what you meant by this.  Did you mean being able to expand 
> an array only by assignment?
> 
> i.e.
> 
> a = zeros(2)  a (2,1) array
> 
> a[1,0] = 1    (resizes a behind the scenes to a 2x2 array and then sets 
> the 1,0 element)?

Mmmh.  I'm not sure I like the idea of an assignment triggering a silent 
resize/reshape event.  Explicit is better than implicit and all that...

I could see this kind of magical behavior easily causing silent, extremely 
hard to find bugs in a big program.

I may be missing something, but I'd be -1 on this.

The 'invalid indices in slices' is basically just sytnactic sugar for a 
try/except block, and it's well-documented behavior in the base language, 
across all its sequence types:

In [2]: ll=[]

In [3]: tt=()

In [4]: ss=''

In [5]: ll[0:1]
Out[5]: []

In [6]: tt[0:1]
Out[6]: ()

In [7]: ss[0:1]
Out[7]: ''

So in my view at least, this behavior of python isn't a good justification for 
a silent resize/reshape (which could, I'm sure, be also potentially explosive 
memory-wise) in numerix arrays.

Regards,

f



From jswhit at fastmail.fm  Thu Feb 17 12:14:16 2005
From: jswhit at fastmail.fm (Jeff Whitaker)
Date: Thu Feb 17 12:14:16 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <4214FAD7.3030109@fastmail.fm>

Travis Oliphant wrote:

>
> 2) Multidimensional array indexing.
>
> Sometimes it is useful to select out of an array some elements based 
> on it's linear (flattened) index in the array.   MATLAB, for example, 
> will allow you to take a three-dimensional array and index it with a 
> single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...
>
> What I'm proposing would have X[K] essentially equivalent to X.flat[K].  

Maybe I'm missing something, but in Numeric and Numarray right now

 >>> from Numeric import *
 >>> a = reshape(arange(9),(3,3))
 >>> print a
[[0 1 2]
 [3 4 5]
 [6 7 8]]
 >>> a[1]
array([3, 4, 5])
 >>> a.flat[1]
1

so a[K] and a.flat[K] are very different things for multidimensional 
arrays.  I think it would be a bad idea to change this now - it would 
certainly break a lot of my code.

Or are you only talking about the case when K is a rank-1 index array 
and not a scalar?

-Jeff


-- 
Jeffrey S. Whitaker         Phone  : (303)497-6313
Meteorologist               FAX    : (303)497-6449
NOAA/OAR/CDC  R/CDC1        Email  : Jeffrey.S.Whitaker at noaa.gov
325 Broadway                Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web    : http://tinyurl.com/5telg




From oliphant at ee.byu.edu  Thu Feb 17 12:27:17 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 17 12:27:17 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214FAD7.3030109@fastmail.fm>
References: <4214E7FE.9090403@ee.byu.edu> <4214FAD7.3030109@fastmail.fm>
Message-ID: <4214FDCD.7000901@ee.byu.edu>

Jeff Whitaker wrote:

> >>> from Numeric import *
> >>> a = reshape(arange(9),(3,3))
> >>> print a
> [[0 1 2]
> [3 4 5]
> [6 7 8]]
> >>> a[1]
> array([3, 4, 5])
> >>> a.flat[1]
> 1
>
> so a[K] and a.flat[K] are very different things for multidimensional 
> arrays.  I think it would be a bad idea to change this now - it would 
> certainly break a lot of my code.

>
> Or are you only talking about the case when K is a rank-1 index array 
> and not a scalar?

a[scalar] will behave exactly the same as now.

We are just talking about the case where K is an index array (I should 
clarify that it must be rank-1 or bigger so that K being a rank-0 array 
wold behave just like the a[scalar] case).

This is basically new behavior that numarray has started supporting.  I 
just think numarray missed an important case of flattened indexing that 
MATLAB supports.  My current proposal would distinguish between 
single-index array cases and tuple-index array cases. 

I'm still thinking about the X.flat possibility.  Basically, I think 
that direction would requires a new "generic array view" or something 
like that.  It may be worth trying, but I'm not sure I want to go that 
direction right now until I convince more people to come on board with 
Numeric3.

-Travis





From tkorvola at welho.com  Thu Feb 17 13:01:19 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Thu Feb 17 13:01:19 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu> (Travis Oliphant's message of
 "Thu, 17 Feb 2005 11:52:46 -0700")
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <87zmy26gx6.fsf@welho.com>

Travis Oliphant <oliphant at ee.byu.edu> writes:
> Currently Numeric allows specifying indexes that are too high in
> slices. It just chops them.  Python allows this too, for slices.

Yes, since foo[:] just means foo.__getitem__(slice(sys.maxint)).
Slices even have a nice method for normalizing the indices.

> So, I guess I'm just specifying Python behavior.

Not really: your specification causes an element with a different
index to be returned, whereas the usual slice behaviour only causes
out-of-range indices to be omitted from the result.

> 3)  Always returning rank-0 arrays.
>
> This may be a bit controversial as it is a bit of a change.

Indeed.  So you really do intend that if foo=array([1,2]), foo[0]
should evaluate to array(1) rather than 1?

> But, my experience is that quite a bit of extra code is written to
> check whether or not a calculation returns a Python-scalar

I suppose this may be necessary for code which operates on arrays of
somewhat arbitrary rank and would not know without looking whether,
e.g., foo[0] is a scalar or an array of positive rank.

> In particular len(a) does not work if a is a scalar,

Depends on what kinds of scalars are supported.  What about object
arrays?

> but len(b) works if b is a rank-0 array

It does?  In Numarray len(b) raises ValueError and size(b) returns 1.
To me this would seem the correct behaviour.

> When Python needs a scalar it will generally ask the object if it can
> turn itself into an int or a float.

Hence this change might not be as incompatible as it seems, although
users of object arrays would be in for some surprises.

> I'd like to know what reasons people can think of for ever returning
> Python scalars  unless explicitly asked for.

It would be more consistent with the usual container semantics and
less likely to break existing code.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From theller at python.net  Thu Feb 17 13:15:10 2005
From: theller at python.net (Thomas Heller)
Date: Thu Feb 17 13:15:10 2005
Subject: [Numpy-discussion] Re: Response to PEP suggestions
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <zmy2gago.fsf@python.net>

I don't think I can help with the discussion, but I would be interested
to read this PEP - where is it?

TIA,

Thomas




From perry at stsci.edu  Thu Feb 17 13:18:18 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 17 13:18:18 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214FDCD.7000901@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4214FAD7.3030109@fastmail.fm> <4214FDCD.7000901@ee.byu.edu>
Message-ID: <28c54bbc706b589b401f14cb4672c918@stsci.edu>

On Feb 17, 2005, at 3:25 PM, Travis Oliphant wrote:

>
> This is basically new behavior that numarray has started supporting.  
> I just think numarray missed an important case of flattened indexing 
> that MATLAB supports.  My current proposal would distinguish between 
> single-index array cases and tuple-index array cases.
> I'm still thinking about the X.flat possibility.  Basically, I think 
> that direction would requires a new "generic array view" or something 
> like that.  It may be worth trying, but I'm not sure I want to go that 
> direction right now until I convince more people to come on board with 
> Numeric3.
>
It was new behavior in the sense that Numeric didn't support 
multidimensional array takes and puts at the time. For a long time it 
was the only kind of array indexing IDL supported (1-d and implicit 
.flat of multidimensional arrays). Speaking only for myself, I found 
the .flat semantics more often as unwanted behavior than convenient. 
I'd rather keep the numarray behavior in this case and make the .flat 
case explicit (but I understand the difficulty of that approach). There 
is the possibility of a custom index option (but ugly I suppose)

X[K, flatten]

where flatten is a special object that indexing recognizes as signaling 
a different interpretation to indexing.

Perry




From perry at stsci.edu  Thu Feb 17 13:29:13 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 17 13:29:13 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <aad8596ded4601bd9eed29e34f8efbe8@stsci.edu>

On Feb 17, 2005, at 1:52 PM, Travis Oliphant wrote:

>
> I'm glad to get the feedback.
>
> 1) Types
>
>
[...]
>
> One thing has always bothered me though.  Why is a double complex type 
> Complex64? and a float complex type Complex32.  This seems to break 
> the idea that the number at the end specifies a bit width.   Why don't 
> we just call it Complex64 and Complex128?  Can we change this?
>
My recollection is that is how we originally did it until we saw how it 
was done in Numeric, but maybe my memory is screwed up. I'm happy with 
real bit widths.

>
> Problems also exist when you are interfacing with hardware or other C 
> or Fortran code.  You know you want single-precision floating point.  
> You don't know or care what the bit-width is.    I think with the 
> Integer types the bit-width specification is more important than 
> floating point types.  In sum, I think it is important to have the 
> ability to specify it both ways.

I'd agree that supporting both is ideal. Sometimes you really want a 
specific bit-width and don't care what the platform default is, and 
others you are tied to the platform default for one reason or another.
>
> 3)  Always returning rank-0 arrays.
>
> This may be a bit controversial as it is a bit of a change.  But, my 
> experience is that quite a bit of extra code is written to check 
> whether or not a calculation returns a Python-scalar (because these 
> don't have the same methods as arrays).  In particular len(a) does not 
> work if a is a scalar, but len(b) works if b is a rank-0 array 
> (numeric scalar).   Rank-0 arrays are scalars.
> When Python needs a scalar it will generally ask the object if it can 
> turn itself into an int or a float.   A notable exception is indexing 
> in a list (where Python needs an integer and won't ask the object to 
> convert if it can).  But int(b) always returns a Python integer if the 
> array has only 1 element.
> I'd like to know what reasons people can think of for ever returning 
> Python scalars  unless explicitly asked for.
>
I'm not sure this is an important issue for us (either way) so long as 
the overhead for rank-0 arrays is not much higher than for scalars (for 
numarray it was an issue). But there are those that argue (Konrad as an 
example if I remember correctly) that the definitions of rank and such 
mean len(rank-0) should not be 1 and that one should not be able to 
index rank-0 arrays. I know that the argument has been made that this 
helps support generic programming (not having to check between scalars 
and arrays), but every time I ask for specific examples I've found that 
there are simple alternatives to solve this problem or that type checks 
are still necessary because there is no control over what users may 
supply as arguments. If this is the reason, could it be motivated with 
a couple examples to show why it is the only reasonable alternative? 
(Then you can use it to slay all subsequent whiners).

Perry




From cookedm at physics.mcmaster.ca  Thu Feb 17 13:32:19 2005
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu Feb 17 13:32:19 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu> (Travis Oliphant's message of
 "Thu, 17 Feb 2005 11:52:46 -0700")
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>

Travis Oliphant <oliphant at ee.byu.edu> writes:

> I'm glad to get the feedback.
>
> 1) Types
>
> I like Francesc's suggestion that .typecode return a code and .type
> return a Python class.   What is the attitude and opinion regarding
> the use of attributes or methods for
> this kind of thing?  It always seems to me so arbitrary as to what is
> an attribute or what
> is a method.

If it's an intrinisic attribute (heh) of the object, I usually try to
make it an attribute. So I'd make these attributes.

> There will definitely be support for the nummary-style type
> specification.   Something like that will be how they print (I like
> the 'i4', 'f4', specification a bit better though). There will also be
> support for specification in terms of a c-type.  The typecodes will
> still be there, underneath.

+1. I think labelling types with their sizes at some level is necessary
for cross-platform compatibility (more below).

> One thing has always bothered me though.  Why is a double complex type
> Complex64? and a float complex type Complex32.  This seems to break
> the idea that the number at the end specifies a bit width.   Why don't
> we just call it Complex64 and Complex128?  Can we change this?

Or rename to ComplexFloat32 and ComplexFloat64?

> I'm also glad that some recognize the problems with always requiring
> specification of types in terms of bit-width or byte-widths as these
> are not the same across platforms.  For some types (like Int8 or
> Int16) this is not a problem.   But what about long double?  On an
> intel machine long double is Float96 while on a PowerPC it is
> Float128.   Wouldn't it just be easier to specify LDouble or 'g' then
> special-case your code?

One problem to consider (and where I first ran into these type of
things) is when pickling. A pickle containing an array of Int isn't
portable, if the two machines have a different idea of what an Int is
(Int32 or Int64, for instance). Another reason to keep the byte-width.

LDouble, for instance, should probably be an alias to Float96 on
Intel, and Float128 on PPC, and pickle accordingly.

> Problems also exist when you are interfacing with hardware or other C
> or Fortran code.  You know you want single-precision floating point.
> You don't know or care what the bit-width is.    I think with the
> Integer types the bit-width specification is more important than
> floating point types.  In sum, I think it is important to have the
> ability to specify it both ways.   When printing the array, it's
> probably better if it gives bit-width information.  I like the way
> numarray prints arrays.

Do you mean adding bit-width info to str()? repr() definitely needs
it, and it should be included in all cases, I think.

You also run into that sizeof(Python integer) isn't necessarily
sizeof(C int) (a Python int being a C long), espically on 64-bit systems.

I come from a C background, so things like Float64, etc., look wrong.
I think more in terms of single- and double-precision, so I think
adding some more descriptive types:

CInt         (would be either Int32 or Int64, depending on the platform)
CFloat       (can't do Float, for backwards-compatibility reasons)
CDouble      (could just be Double)
CLong        (or Long)
CLongLong    (or LongLong)

That could make it easier to match types in Python code to types in C
extensions.

Oh, and the Python types int and float should be allowed (especially
if you want this to go in the core!).

And a Fortran integer could be something else, but I think that's
more of a SciPy problem than Numeric or numarray. It could add
FInteger and FBoolean, for instance.

> 2) Multidimensional array indexing.
>
> Sometimes it is useful to select out of an array some elements based
> on it's linear (flattened) index in the array.   MATLAB, for example,
> will allow you to take a three-dimensional array and index it with a
> single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...
>
> What I'm proposing would have X[K] essentially equivalent to
> X.flat[K].  The problem with always requiring the use of X.flat[K] is
> that X.flat does not work for discontiguous arrays.   It could be made
> to work if X.flat returned some kind of specially-marked array, which
> would then have to be checked every time indexing occurred for any
> array.  Or, there maybe someway to have X.flat return an "indexable
> iterator" for X which may be a more Pythonic thing to do anyway.  That
> could solve the problem and solve the discontiguous X.flat problem as
> well.
>
> If we can make X.flat[K] work for discontiguous arrays, then I would
> be very happy to not special-case the single index array but always
> treat it as a 1-tuple of integer index arrays.

Right now, I find X.flat to be pretty useless, as you need a
contiguous array. I'm +1 on making X.flat work in all cases (contiguous
and discontiguous). Either

a) X.flat returns a contiguous 1-dimensional array (like ravel(X)),
   which may be a copy of X

or

b) X.flat returns a "flat-indexable" view of X

I'd argue for b), as I feel that attributes should operate as views,
not as potential copies. To me, attributes "feel like" they do no
work, so making a copy by mere dereferencing would be suprising.

If a), I'd rather flat() be a method (or have a ravel() method).


I think overloading X[K] starts to run into trouble: too many special
cases.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From Fernando.Perez at colorado.edu  Thu Feb 17 14:06:11 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu Feb 17 14:06:11 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>
Message-ID: <4215152B.2070108@colorado.edu>

David M. Cooke wrote:

> Right now, I find X.flat to be pretty useless, as you need a
> contiguous array. I'm +1 on making X.flat work in all cases (contiguous
> and discontiguous). Either
> 
> a) X.flat returns a contiguous 1-dimensional array (like ravel(X)),
>    which may be a copy of X
> 
> or
> 
> b) X.flat returns a "flat-indexable" view of X
> 
> I'd argue for b), as I feel that attributes should operate as views,
> not as potential copies. To me, attributes "feel like" they do no
> work, so making a copy by mere dereferencing would be suprising.

+1 on b).

I have quite a bit of code doing things like

try:
   b=a.flat
except:
   b=ravel(a)

which feels silly.  It would be nice for .flat to be a guaranteed, indexable, 
no-copy view of the original (even if it's non-contiguous).  Granted, the 
indexing will be costlier for a non-contiguous object, but for the external 
users this provides a clean API.

Regards,

f



From rkern at ucsd.edu  Thu Feb 17 14:15:21 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Thu Feb 17 14:15:21 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4215152B.2070108@colorado.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu>
Message-ID: <42151728.7010307@ucsd.edu>

Fernando Perez wrote:

> I have quite a bit of code doing things like
> 
> try:
>   b=a.flat
> except:
>   b=ravel(a)
> 
> which feels silly.  It would be nice for .flat to be a guaranteed, 
> indexable, no-copy view of the original (even if it's non-contiguous).  
> Granted, the indexing will be costlier for a non-contiguous object, but 
> for the external users this provides a clean API.

Why not just do

   b = ravel(a)

?

That should work in both cases just fine.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From rowen at cesmail.net  Thu Feb 17 14:17:19 2005
From: rowen at cesmail.net (Russell E. Owen)
Date: Thu Feb 17 14:17:19 2005
Subject: [Numpy-discussion] Re: Multiarray PEP
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <rowen-8046F3.14024817022005@sea.gmane.org>

In article <421446E3.7080202 at ee.byu.edu>,
 Travis Oliphant <oliphant at ee.byu.edu> wrote:

> I've never had a problem with just the typecode characters although 
> aliases (like Int16 or Short) which might be more meaningful are nice.   
> Are shadow-classes important in this case?  To me they look a little 
> like extra bulk with no gain.  I'd love to be educated as to why it is 
> worth the extra effort to have a bunch of instances lying around. 

Instances can give you useful information. For example:
- the number of bytes needed to represent a value
- the min and max values for that type (numarray doesn't do this yet, 
but I really hope it'll be added someday).

I personally dislike character codes because they are uninformative and 
because they meaningless without the documentation.

-- Russell




From Fernando.Perez at colorado.edu  Thu Feb 17 14:19:23 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu Feb 17 14:19:23 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151728.7010307@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu>
Message-ID: <42151848.301@colorado.edu>

Robert Kern wrote:
> Why not just do
> 
>    b = ravel(a)
> 
> ?
> 
> That should work in both cases just fine.

Because I thouhgt that ravel would make a copy regardless.  But I may be 
wrong, much of this is very old code, when I was just picking up Numeric.  At 
the time, I felt it might be best to avoid the unnecessary copies if possible.

Best,

f



From rlw at stsci.edu  Thu Feb 17 14:22:22 2005
From: rlw at stsci.edu (Rick White)
Date: Thu Feb 17 14:22:22 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>
Message-ID: <Pine.GSO.4.44.0502171715260.9373-100000@sundog.stsci.edu>

On Thu, 17 Feb 2005, David M. Cooke wrote:

> I come from a C background, so things like Float64, etc., look wrong.
> I think more in terms of single- and double-precision, so I think
> adding some more descriptive types:
>
> CInt         (would be either Int32 or Int64, depending on the platform)
> CFloat       (can't do Float, for backwards-compatibility reasons)
> CDouble      (could just be Double)
> CLong        (or Long)
> CLongLong    (or LongLong)
>
> That could make it easier to match types in Python code to types in C
> extensions.

Good choice of names.

> Oh, and the Python types int and float should be allowed (especially
> if you want this to go in the core!).

Say, I like that idea.  And maybe, like float and int, the numeric types
could be callable to construct numeric arrays of that type, e.g.,

a = numeric3.Int16([1,2,3])

- Rick




From duncan at enthought.com  Thu Feb 17 14:23:17 2005
From: duncan at enthought.com (Duncan Child)
Date: Thu Feb 17 14:23:17 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <87zmy26gx6.fsf@welho.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>
Message-ID: <42151919.60607@enthought.com>

Here are a couple of issues that are important to me that might be 
relevant to the design discussion.

I have attached some code that illustrates part of the pain we have 
experienced developing libraries of algorithms that can handle both 
arrays and scalars. The attached library is the reusable part. The other 
part of this problem is that we have lots of logic sprinkled throughout 
our algorithms to enable them to handle both arrays and scalars.

Secondly, I have just been bitten by this declaration which suggests 
that the new Numeric might handle default values better:.

_vp_mod = zeros(num_pts)

It would be less surprising to someone developing numeric algorithms if 
functions like this defaulted to creating a double precision array 
rather than integers.

Regards,

Duncan

>  
>
>>3)  Always returning rank-0 arrays.
>>
>>This may be a bit controversial as it is a bit of a change.
>>    
>>
>
>Indeed.  So you really do intend that if foo=array([1,2]), foo[0]
>should evaluate to array(1) rather than 1?
>
>  
>
import scipy
from scipy import take, amin, amax, arange, asarray, PyObject, mean, \
                  product, shape, array, Float64, nonzero


"""
The following safe_ methods were written to handle both arrays amd 
scalars to
save the developer of numerical methods having to clutter their code 
with tests
to determine the type of the data.
"""

def safe_take(a,indices):
    # Slice the input if it is an array but not if it is a scalar
    try:
        a = take(a,indices)
    except ValueError:
        # a is scalar
        pass
    return a
   
def safe_copy(a):
    # Return a copy for both scalar and array input
    try:
        b = a.copy()
    except AttributeError:
        # a is a scalar
        b = a
    return b

# Note: if x is a scalar and y = asarray(x), amin(y) FAILS but min(y) works
# Note: BUT IF z=convert(y,frac,frac), THEN min(z) FAILS!!!
def safe_min(a):
    # Return the minimum of the input array or the input if it is a scalar
    try:
        safemin = amin(a)
    except:
        safemin = a
    return safemin

def safe_max(a):
    # Return the maximum of the input array or the input if it is a scalar
    try:
        safemax = amax(a)
    except:
        safemax = a
    return safemax

def safe_mean(a):
    # Return the mean of the input array or the input if it is a scalar
    try:
        safemean = mean(a)
    except:
        safemean = a
    return safemean

def safe_len(a):
    # Return the length of the input array or 1 if it is a scalar
    try:
        safelen = len(a)
    except:
        safelen = 1
    return safelen

def safe_flat(a):
    # Return a flat version of the input array or input if it is a scalar
    try:
        safeflat = a.flat
    except:
        safeflat = a
    return safeflat




From stephen.walton at csun.edu  Thu Feb 17 14:27:16 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Thu Feb 17 14:27:16 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4215152B.2070108@colorado.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu>
Message-ID: <421519F2.1070607@csun.edu>

Fernando Perez wrote:

> It would be nice for .flat to be a guaranteed, indexable, no-copy view 
> of the original (even if it's non-contiguous).

Dare I point out that a.flat works for non-contiguous arrays in numarray?

In [4]: a.iscontiguous()
Out[4]: 0

In [5]: a.flat
Out[5]:
array([ 0,  5, 10, 15, 20,  1,  6, 11, 16, 21,  2,  7, 12, 17, 22,  3,
        8, 13, 18, 23,  4,  9, 14, 19, 24])





From rkern at ucsd.edu  Thu Feb 17 14:28:16 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Thu Feb 17 14:28:16 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151848.301@colorado.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu> <42151848.301@colorado.edu>
Message-ID: <42151A4A.9070200@ucsd.edu>

Fernando Perez wrote:
> Robert Kern wrote:
> 
>> Why not just do
>>
>>    b = ravel(a)
>>
>> ?
>>
>> That should work in both cases just fine.
> 
> 
> Because I thouhgt that ravel would make a copy regardless.  But I may be 
> wrong, much of this is very old code, when I was just picking up 
> Numeric.  At the time, I felt it might be best to avoid the unnecessary 
> copies if possible.

Nope.

In [1]: a = arange(8)

In [2]: a.shape = (2,4)

In [3]: a
Out[3]: NumPy array, format: long
[[0 1 2 3]
  [4 5 6 7]]

In [4]: ravel(a)[3] = 10

In [5]: a
Out[5]: NumPy array, format: long
[[ 0  1  2 10]
  [ 4  5  6  7]]

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Thu Feb 17 14:33:14 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu Feb 17 14:33:14 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151A4A.9070200@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu> <42151848.301@colorado.edu> <42151A4A.9070200@ucsd.edu>
Message-ID: <42151B82.3040507@colorado.edu>

Robert Kern wrote:

>>Because I thouhgt that ravel would make a copy regardless.  But I may be 
>>wrong, much of this is very old code, when I was just picking up 
>>Numeric.  At the time, I felt it might be best to avoid the unnecessary 
>>copies if possible.
> 
> 
> Nope.

[...]

OK, thanks for the clarification.  I guess if I had a nice, easy to use 
interactive shell I could have figured it out on my own.  You'll have to sell 
me that shiny one you seem to be using :)

Best,

f



From tim.hochberg at cox.net  Thu Feb 17 14:45:09 2005
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Thu Feb 17 14:45:09 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151A4A.9070200@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu> <42151848.301@colorado.edu> <42151A4A.9070200@ucsd.edu>
Message-ID: <42151E39.9000409@cox.net>

Robert Kern wrote:

> Fernando Perez wrote:
>
>> Robert Kern wrote:
>>
>>> Why not just do
>>>
>>>    b = ravel(a)
>>>
>>> ?
>>>
>>> That should work in both cases just fine.
>>
>>
>>
>> Because I thouhgt that ravel would make a copy regardless.  But I may 
>> be wrong, much of this is very old code, when I was just picking up 
>> Numeric.  At the time, I felt it might be best to avoid the 
>> unnecessary copies if possible.
>
>
> Nope.
>
> In [1]: a = arange(8)
>
> In [2]: a.shape = (2,4)
>
> In [3]: a
> Out[3]: NumPy array, format: long
> [[0 1 2 3]
>  [4 5 6 7]]
>
> In [4]: ravel(a)[3] = 10
>
> In [5]: a
> Out[5]: NumPy array, format: long
> [[ 0  1  2 10]
>  [ 4  5  6  7]]


Ick, that's horrible. Functions that sometimes copy and sometimes don't 
are generally bad news IMO. This is just a way to introduce nasty, 
invisible bugs. The exceptions are things like asarray that are explicit 
about their variable behaviour.

I'd be much happier if flat never made copies, but always worked by some 
sort of deep juju, while ravel always made copies.

-tim





From stephen.walton at csun.edu  Thu Feb 17 15:34:21 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Thu Feb 17 15:34:21 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151919.60607@enthought.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42151919.60607@enthought.com>
Message-ID: <421529C0.9070605@csun.edu>

Duncan Child wrote:

> # Note: if x is a scalar and y = asarray(x), amin(y) FAILS but min(y) 
> works
> # Note: BUT IF z=convert(y,frac,frac), THEN min(z) FAILS!!

My quick check shows that amin(y), min(y), and amin(x) all work after 
"from scipy import *".  Only min(x) doesn't, and it is a Python built-in 
which throws an "iteration over non-sequence" exception when called with 
a scalar.   Should Numeric 3 bump amin and similar functions up into 
itself?  How many others need a min-like function which works with both 
arrays and scalars?  In numarray, min is a method.  Is this better?




From verveer at embl.de  Thu Feb 17 15:54:27 2005
From: verveer at embl.de (Peter Verveer)
Date: Thu Feb 17 15:54:27 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <3a2f1080766ca2dad8bfcaf6ca46ee0b@embl.de>

On Feb 17, 2005, at 7:52 PM, Travis Oliphant wrote:

>
> I'm glad to get the feedback.
>
> 1) Types
>
> I like Francesc's suggestion that .typecode return a code and .type 
> return a Python class.   What is the attitude and opinion regarding 
> the use of attributes or methods for
> this kind of thing?  It always seems to me so arbitrary as to what is 
> an attribute or what
> is a method.

I don't think it really matters. Attributes seem natural, shape is an 
attribute for instance, so why not type? In the end, I don't care.

> There will definitely be support for the nummary-style type 
> specification.   Something like that will be how they print (I like 
> the 'i4', 'f4', specification a bit better though). There will also be 
> support for specification in terms of a c-type.  The typecodes will 
> still be there, underneath.

sounds fine to me.

>
> One thing has always bothered me though.  Why is a double complex type 
> Complex64? and a float complex type Complex32.  This seems to break 
> the idea that the number at the end specifies a bit width.   Why don't 
> we just call it Complex64 and Complex128?  Can we change this?

I actually find the current approach natural. You specify the width of 
the real and the imaginary type, which are some  kind of a double type. 
Again, in the end I would not care.

> I'm also glad that some recognize the problems with always requiring 
> specification of types in terms of bit-width or byte-widths as these 
> are not the same across platforms.  For some types (like Int8 or 
> Int16) this is not a problem.   But what about long double?  On an 
> intel machine long double is Float96 while on a PowerPC it is 
> Float128.   Wouldn't it just be easier to specify LDouble or 'g' then 
> special-case your code?

long double is a bit of a special case. I guess I would probably not 
use it anyway. The point is indeed that having things like LDouble is 
'a good thing'.

> Problems also exist when you are interfacing with hardware or other C 
> or Fortran code.  You know you want single-precision floating point.  
> You don't know or care what the bit-width is.    I think with the 
> Integer types the bit-width specification is more important than 
> floating point types.  In sum, I think it is important to have the 
> ability to specify it both ways.

I completely agree with this. I probably dont care for floating point, 
it is good enough to distinguish between single and double precision. 
Integer types are a different story, you want to be a bit more precise 
then. Having both solves the problem quite well.

>   When printing the array, it's probably better if it gives bit-width 
> information.  I like the way numarray prints arrays.

Agreed.

>
>
> 2) Multidimensional array indexing.
>
> Sometimes it is useful to select out of an array some elements based 
> on it's linear (flattened) index in the array.   MATLAB, for example, 
> will allow you to take a three-dimensional array and index it with a 
> single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...
>
> What I'm proposing would have X[K] essentially equivalent to 
> X.flat[K].  The problem with always requiring the use of X.flat[K] is 
> that X.flat does not work for discontiguous arrays.   It could be made 
> to work if X.flat returned some kind of specially-marked array, which 
> would then have to be checked every time indexing occurred for any 
> array.  Or, there maybe someway to have X.flat return an "indexable 
> iterator" for X which may be a more Pythonic thing to do anyway.  That 
> could solve the problem and solve the discontiguous X.flat problem as 
> well.

But possibly slow, and that we want to avoid.

> If we can make X.flat[K] work for discontiguous arrays, then I would 
> be very happy to not special-case the single index array but always 
> treat it as a 1-tuple of integer index arrays.

Speed will be an issue.

> Capping indexes was proposed because of what numarray does.   I can 
> only think that the benefit would be that you don't have to check for 
> and raise an error in the middle of an indexing loop or pre-scan the 
> indexes.  But, I suppose this is unavoidalbe, anyway.  Currently 
> Numeric allows specifying indexes that are too high in slices. It just 
> chops them.  Python allows this too, for slices.  So, I guess I'm just 
> specifying Python behavior.  Of course indexing with an integer that 
> is too large or too small will raise errors:
>
> In Python:
>
> a = [1,2,3,4,5]
> a[:20]   works
> a[20] raises an error.

Probably better to stick to Python behavior.




From verveer at embl.de  Thu Feb 17 16:00:10 2005
From: verveer at embl.de (Peter Verveer)
Date: Thu Feb 17 16:00:10 2005
Subject: [Numpy-discussion] Re: Multiarray PEP
In-Reply-To: <rowen-8046F3.14024817022005@sea.gmane.org>
References: <421446E3.7080202@ee.byu.edu> <rowen-8046F3.14024817022005@sea.gmane.org>
Message-ID: <cc426cd57565fdad7a682ce82a3ef96b@embl.de>

On Feb 17, 2005, at 11:02 PM, Russell E. Owen wrote:

> In article <421446E3.7080202 at ee.byu.edu>,
>  Travis Oliphant <oliphant at ee.byu.edu> wrote:
>
>> I've never had a problem with just the typecode characters although
>> aliases (like Int16 or Short) which might be more meaningful are nice.
>> Are shadow-classes important in this case?  To me they look a little
>> like extra bulk with no gain.  I'd love to be educated as to why it is
>> worth the extra effort to have a bunch of instances lying around.
>
> Instances can give you useful information. For example:
> - the number of bytes needed to represent a value
> - the min and max values for that type (numarray doesn't do this yet,
> but I really hope it'll be added someday).

+1 on this! This would be very useful...




From verveer at embl.de  Thu Feb 17 16:02:18 2005
From: verveer at embl.de (Peter Verveer)
Date: Thu Feb 17 16:02:18 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <Pine.GSO.4.44.0502171715260.9373-100000@sundog.stsci.edu>
References: <Pine.GSO.4.44.0502171715260.9373-100000@sundog.stsci.edu>
Message-ID: <8c7fe7fab3cba13e35398e4b26bbca77@embl.de>

>> Oh, and the Python types int and float should be allowed (especially
>> if you want this to go in the core!).
>
> Say, I like that idea.  And maybe, like float and int, the numeric 
> types
> could be callable to construct numeric arrays of that type, e.g.,
>
> a = numeric3.Int16([1,2,3])

That is a good idea. Seems easy to implement, they would just be 
aliases.




From cjw at sympatico.ca  Thu Feb 17 17:01:27 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 17 17:01:27 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>
Message-ID: <42153E36.7050808@sympatico.ca>

David M. Cooke wrote:

>Travis Oliphant <oliphant at ee.byu.edu> writes:
>
>  
>
>>I'm glad to get the feedback.
>>
>>1) Types
>>
>>I like Francesc's suggestion that .typecode return a code and .type
>>return a Python class.   What is the attitude and opinion regarding
>>the use of attributes or methods for
>>this kind of thing?  It always seems to me so arbitrary as to what is
>>an attribute or what
>>is a method.
>>    
>>
>
>If it's an intrinisic attribute (heh) of the object, I usually try to
>make it an attribute. So I'd make these attributes.
>
>  
>
>>There will definitely be support for the nummary-style type
>>specification.   Something like that will be how they print (I like
>>the 'i4', 'f4', specification a bit better though). There will also be
>>support for specification in terms of a c-type.  The typecodes will
>>still be there, underneath.
>>    
>>
>
>+1. I think labelling types with their sizes at some level is necessary
>for cross-platform compatibility (more below).
>
>  
>
>>One thing has always bothered me though.  Why is a double complex type
>>Complex64? and a float complex type Complex32.  This seems to break
>>the idea that the number at the end specifies a bit width.   Why don't
>>we just call it Complex64 and Complex128?  Can we change this?
>>    
>>
>
>Or rename to ComplexFloat32 and ComplexFloat64?
>
>  
>
>>I'm also glad that some recognize the problems with always requiring
>>specification of types in terms of bit-width or byte-widths as these
>>are not the same across platforms.  For some types (like Int8 or
>>Int16) this is not a problem.   But what about long double?  On an
>>intel machine long double is Float96 while on a PowerPC it is
>>Float128.   Wouldn't it just be easier to specify LDouble or 'g' then
>>special-case your code?
>>    
>>
>
>One problem to consider (and where I first ran into these type of
>things) is when pickling. A pickle containing an array of Int isn't
>portable, if the two machines have a different idea of what an Int is
>(Int32 or Int64, for instance). Another reason to keep the byte-width.
>
>LDouble, for instance, should probably be an alias to Float96 on
>Intel, and Float128 on PPC, and pickle accordingly.
>
>  
>
>>Problems also exist when you are interfacing with hardware or other C
>>or Fortran code.  You know you want single-precision floating point.
>>You don't know or care what the bit-width is.    I think with the
>>Integer types the bit-width specification is more important than
>>floating point types.  In sum, I think it is important to have the
>>ability to specify it both ways.   When printing the array, it's
>>probably better if it gives bit-width information.  I like the way
>>numarray prints arrays.
>>    
>>
>
>Do you mean adding bit-width info to str()? repr() definitely needs
>it, and it should be included in all cases, I think.
>
>You also run into that sizeof(Python integer) isn't necessarily
>sizeof(C int) (a Python int being a C long), espically on 64-bit systems.
>
>I come from a C background, so things like Float64, etc., look wrong.
>I think more in terms of single- and double-precision, so I think
>adding some more descriptive types:
>
>CInt         (would be either Int32 or Int64, depending on the platform)
>CFloat       (can't do Float, for backwards-compatibility reasons)
>CDouble      (could just be Double)
>CLong        (or Long)
>CLongLong    (or LongLong)
>
>That could make it easier to match types in Python code to types in C
>extensions.
>  
>
I guess the issue revolves around the characteristics of the target 
users, if most are C aficionados then the above has merit.  However, 
this doesn't provide for the Int8's or the Int16's.  Neither does it 
provide for a bit array, which would be suitable for Booleans.

My guess is that most users would not be from a C background and so 
something along the lines of numerictypes makes sense.

>Oh, and the Python types int and float should be allowed (especially
>if you want this to go in the core!).
>
>And a Fortran integer could be something else, but I think that's
>more of a SciPy problem than Numeric or numarray. It could add
>FInteger and FBoolean, for instance.
>
>  
>
>>2) Multidimensional array indexing.
>>
>>Sometimes it is useful to select out of an array some elements based
>>on it's linear (flattened) index in the array.   MATLAB, for example,
>>will allow you to take a three-dimensional array and index it with a
>>single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...
>>
>>What I'm proposing would have X[K] essentially equivalent to
>>X.flat[K].  The problem with always requiring the use of X.flat[K] is
>>that X.flat does not work for discontiguous arrays.   It could be made
>>to work if X.flat returned some kind of specially-marked array, which
>>would then have to be checked every time indexing occurred for any
>>array.  Or, there maybe someway to have X.flat return an "indexable
>>iterator" for X which may be a more Pythonic thing to do anyway.  That
>>could solve the problem and solve the discontiguous X.flat problem as
>>well.
>>
>>If we can make X.flat[K] work for discontiguous arrays, then I would
>>be very happy to not special-case the single index array but always
>>treat it as a 1-tuple of integer index arrays.
>>    
>>
>
>Right now, I find X.flat to be pretty useless, as you need a
>contiguous array. I'm +1 on making X.flat work in all cases (contiguous
>and discontiguous). Either
>
>a) X.flat returns a contiguous 1-dimensional array (like ravel(X)),
>   which may be a copy of X
>
>or
>
>b) X.flat returns a "flat-indexable" view of X
>
>I'd argue for b), as I feel that attributes should operate as views,
>not as potential copies. To me, attributes "feel like" they do no
>work, so making a copy by mere dereferencing would be suprising.
>
>If a), I'd rather flat() be a method (or have a ravel() method).
>
>
>I think overloading X[K] starts to run into trouble: too many special
>cases.
>  
>
As someone else said, the draft PEP needs to have a much clearer 
statement of what datatype K is and just what X[K] would mean.

Colin W.




From cjw at sympatico.ca  Thu Feb 17 17:14:40 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 17 17:14:40 2005
Subject: [Numpy-discussion] Re: Multiarray PEP
In-Reply-To: <rowen-8046F3.14024817022005@sea.gmane.org>
References: <421446E3.7080202@ee.byu.edu> <rowen-8046F3.14024817022005@sea.gmane.org>
Message-ID: <42154124.6080406@sympatico.ca>

Russell E. Owen wrote:

>In article <421446E3.7080202 at ee.byu.edu>,
> Travis Oliphant <oliphant at ee.byu.edu> wrote:
>
>  
>
>>I've never had a problem with just the typecode characters although 
>>aliases (like Int16 or Short) which might be more meaningful are nice.   
>>Are shadow-classes important in this case?  To me they look a little 
>>like extra bulk with no gain.  I'd love to be educated as to why it is 
>>worth the extra effort to have a bunch of instances lying around. 
>>    
>>
>
>Instances can give you useful information. For example:
>- the number of bytes needed to represent a value
>  
>
 >>> import numarray.numarraycore as _n
 >>> a= _n.arange(9, shape= (3, 3))
 >>> print 'bytes per element:', a.itemsize()
bytes per element: 4

Alternatively:
 >>> a._type.bytes
4

>- the min and max values for that type (numarray doesn't do this yet, 
>but I really hope it'll be added someday).
>  
>
This could come from the type instance but it's not too difficult to 
determine for a given type.

>I personally dislike character codes because they are uninformative and 
>because they meaningless without the documentation.
>
>  
>
(+1)
Colin W.



From cjw at sympatico.ca  Thu Feb 17 17:29:23 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 17 17:29:23 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <3a2f1080766ca2dad8bfcaf6ca46ee0b@embl.de>
References: <4214E7FE.9090403@ee.byu.edu> <3a2f1080766ca2dad8bfcaf6ca46ee0b@embl.de>
Message-ID: <421544C0.70404@sympatico.ca>

Peter Verveer wrote:

>
> On Feb 17, 2005, at 7:52 PM, Travis Oliphant wrote:
>
>>
>> I'm glad to get the feedback.
>> [snip]
>
>> 2) Multidimensional array indexing.
>>
>> Sometimes it is useful to select out of an array some elements based 
>> on it's linear (flattened) index in the array.   MATLAB, for example, 
>> will allow you to take a three-dimensional array and index it with a 
>> single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...
>>
>> What I'm proposing would have X[K] essentially equivalent to 
>> X.flat[K].  The problem with always requiring the use of X.flat[K] is 
>> that X.flat does not work for discontiguous arrays.   It could be 
>> made to work if X.flat returned some kind of specially-marked array, 
>> which would then have to be checked every time indexing occurred for 
>> any array.  Or, there maybe someway to have X.flat return an 
>> "indexable iterator" for X which may be a more Pythonic thing to do 
>> anyway.  That could solve the problem and solve the discontiguous 
>> X.flat problem as well.
>
>
> But possibly slow, and that we want to avoid.

Currently, numarray returns an array with a reduced rank:

 >>> a
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])
 >>> a[1]
array([3, 4, 5])
 >>> a[:,1]
array([1, 4, 7])
 >>>

Is this to be abandoned?

Colin W.



From cookedm at physics.mcmaster.ca  Thu Feb 17 17:55:20 2005
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu Feb 17 17:55:20 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42153E36.7050808@sympatico.ca>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <42153E36.7050808@sympatico.ca>
Message-ID: <20050218015333.GA18265@arbutus.physics.mcmaster.ca>

On Thu, Feb 17, 2005 at 08:00:38PM -0500, Colin J. Williams wrote:
> David M. Cooke wrote:
> >I come from a C background, so things like Float64, etc., look wrong.
> >I think more in terms of single- and double-precision, so I think
> >adding some more descriptive types:
> >
> >CInt         (would be either Int32 or Int64, depending on the platform)
> >CFloat       (can't do Float, for backwards-compatibility reasons)
> >CDouble      (could just be Double)
> >CLong        (or Long)
> >CLongLong    (or LongLong)
> >
> >That could make it easier to match types in Python code to types in C
> >extensions.
> > 
> >
> I guess the issue revolves around the characteristics of the target 
> users, if most are C aficionados then the above has merit.  However, 
> this doesn't provide for the Int8's or the Int16's.  Neither does it 
> provide for a bit array, which would be suitable for Booleans.
> 
> My guess is that most users would not be from a C background and so 
> something along the lines of numerictypes makes sense.

I'm thinking that CInt, etc., would be aliases for Int32 or Int64 (or
whatever makes sense on the platform), at least at the Python level.

The idea is if you're writing wrapper code for external routines, you
want to use the types used in the routine, which most likely will vary
platform-by-platform. In that case you *don't* want to hardcode Int32,
etc., because that's not the right type for all platforms. I've run into
enough of these bugs, since I'm using a 64-bit Athlon64 Linux system now
as my main system (and Numeric still has some problems internally in
this respect).

It's partly documentation: you're asserting that an array is an array of
C ints, whatever a C int is. Passing that to a routine that takes C ints
shouldn't require conversion, or fail because of mismatched min-max int
ranges.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From cjw at sympatico.ca  Thu Feb 17 18:51:20 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 17 18:51:20 2005
Subject: [Numpy-discussion] Some comments on the draft PEP (Rev 1.8)
Message-ID: <42155814.4040908@sympatico.ca>

Basic Types

    There is no mention of the record.  This was one of the useful ideas 
from COBOL and carried over by PL/1.  See numarray.record.py.

    I suggest it is worth continuing.

Sequence and Mapping 2)  The advanced indexing seems to offer masked 
extraction from an array, some examples would help.

                                     3)  I've always wondered about the 
rank 0 array.  If such values are returned as scalars then the 
processing inside the array package is increased slightly.  If they are 
returned as an array then there is more work to be done in the Python 
code outside the the array package.  My preference is to return a scalar.

Iterator.   A scalar appears to be much better in this case.

Methods.  Why not use properties for those methods which don't require 
arguments?

                Some of the maybes are number oriented but the basic 
Array is intended to have a wider scope than just numbers.

Implementation Plan   The usual convention is that class/type names 
start with a capital, thus Array, rather than array.  This should clash 
with a name on the existing array module.  It is good to see that the 
Array is to be sub-classable.  This is one of the weaknesses of 
numarray.  'efficiencies' meant that Python's introspection is not fully 
available in NumArray.

C Structures              Is NOTSWAPPED, based on system order, going to 
give trouble if data is exported from one system to another?

Type characters         My feeling is that the numarray.numerictypes 
descriptors are clearer for the user who has not memorized the 
typecodes.  It takes a little more space overhead but simplifies the 
Array repr.  With an editor, such as PythonWin, one can enter 
'MN.numerictypes.'.  The editor responds with a drop down list of 
available types - MN is an abbreviated code for the imported modules 
name (I don't use from X import *).  The programmer does not have to 
remember the non mnemonic letter codes.

One final request.             In the instance constructor, please avoid 
the use of 'type' as a parameter name if the constructor is implemented 
in Python.  This hides the builtin type(), which is sometimes useful in 
debugging or exploring code.

I hope that this is of some help.

Colin W.






From oliphant at ee.byu.edu  Thu Feb 17 21:26:13 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 17 21:26:13 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151E39.9000409@cox.net>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu> <42151848.301@colorado.edu> <42151A4A.9070200@ucsd.edu> <42151E39.9000409@cox.net>
Message-ID: <42157BF9.9090302@ee.byu.edu>

> Ick, that's horrible. Functions that sometimes copy and sometimes 
> don't are generally bad news IMO. This is just a way to introduce 
> nasty, invisible bugs. The exceptions are things like asarray that are 
> explicit about their variable behaviour.
>
> I'd be much happier if flat never made copies, but always worked by 
> some sort of deep juju, while ravel always made copies.


I tend to agree (though) there is precedence for "return a copy only if 
you have to" at least on the C-level.

What I suggest is that attributes never return copies while methods 
return copies when necessary.

In that vein, I am proposing making X.flat an array iterator  and 
allowing array iterators to be indexed and set as if they were 1-d 
arrays with the underlying array being changed.  This is actually an 
easy change with the current code base.  Will it break any code?  

There maybe some X.flats that need to be changed to ravel.  But it seems 
like a really good idea.

-Travis





From verveer at embl-heidelberg.de  Fri Feb 18 00:59:29 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Fri Feb 18 00:59:29 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42157BF9.9090302@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu> <42151848.301@colorado.edu> <42151A4A.9070200@ucsd.edu> <42151E39.9000409@cox.net> <42157BF9.9090302@ee.byu.edu>
Message-ID: <720e1fa453a30163c6d98b20324121df@embl-heidelberg.de>

> In that vein, I am proposing making X.flat an array iterator  and 
> allowing array iterators to be indexed and set as if they were 1-d 
> arrays with the underlying array being changed.  This is actually an 
> easy change with the current code base.  Will it break any code?
> There maybe some X.flats that need to be changed to ravel.  But it 
> seems like a really good idea.

That iterator would only work on the Python level I suppose, that would 
be effectively indistinguishable from an 1D array. But how would that 
work when passed to a function implemented in C? The function needs to 
know somehow that non-contiguous arrays need to be treated as 1D. That 
means having to code for such a special case (not a good idea), or you 
are back to making a copy.




From konrad.hinsen at laposte.net  Fri Feb 18 06:09:11 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 06:09:11 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <4bf0e99f809d592b068270a200b6deb0@laposte.net>

On Feb 17, 2005, at 19:52, Travis Oliphant wrote:

> I like Francesc's suggestion that .typecode return a code and .type 
> return a Python class.   What is the attitude and opinion regarding 
> the use of attributes or methods for this kind of thing?  It always 
> seems to me so arbitrary as to what is an attribute or what is a 
> method.

My view of pythonicity is that retrieving a value should be written as 
attribute access. Methods are more appropriate if there are arguments 
(no choice then anyway) or side effects. So I'd have .type as an 
attribute.

BTW, as the goal is inclusion into the Python core, why not

1) Use Python type objects for array creation and as the values of the 
.type
     attribute.
2) Implement scalar types for those array element types that currently
     have no Python scalar equivalent (e.g. UInt16).
3) Implement the same set of attributes of methods for scalar types and
    arrays.

Then the distinction between scalars and rank-0 arrays would become a 
minor implementation detail rather than a topic of heated debate. In 
different words, I propose that the PEP should include unification of 
scalars and arrays such that for all practical purposes scalars *are* 
rank-0 arrays.

> One thing has always bothered me though.  Why is a double complex type 
> Complex64? and a float complex type Complex32.  This seems to break 
> the idea that the number at the end specifies a bit width.   Why don't 
> we just call it Complex64 and Complex128?  Can we change this?

+1

> PowerPC it is Float128.   Wouldn't it just be easier to specify 
> LDouble or 'g' then special-case your code?

Definitely.

> Sometimes it is useful to select out of an array some elements based 
> on it's linear (flattened) index in the array.   MATLAB, for example, 
> will allow you to take a three-dimensional array and index it with a 
> single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...

Could you give an example where this would be useful? To me this looks 
like a feature that MATLAB inherited from Fortran, which had it for 
efficiency reasons in a time when compilers were not so good at 
optimizing index expressions.

I don't lile the "special case" status of such a construct either. It 
could lead to unpleasant bugs that would be hard to find by those who 
are not aware of the special case. I'd say that special cases need 
special justifications - and I don't see one here.

> discontiguous arrays.   It could be made to work if X.flat returned 
> some kind of specially-marked array, which would then have to be 
> checked every time indexing occurred for any array.  Or, there maybe 
> someway to have X.flat return an

I much prefer that approach, assuming there is a real use for this 
feature.

> Capping indexes was proposed because of what numarray does.   I can 
> only think that the benefit would be that you don't have to check for 
> and raise an error in the middle of an indexing loop or pre-scan the 
> indexes.  But, I suppose this is unavoidalbe, anyway.  Currently 
> Numeric allows specifying indexes that are too high in slices. It just 
> chops them.  Python allows this too, for slices.  So, I guess I'm just 
> specifying Python behavior.  Of course indexing with an integer that 
> is too large or too small will raise errors:

I am all for imitating Python list behaviour in arrays, but we should 
also watch out for pitfalls. Array index expressions are in general 
much more complex than list index expressions, so the risk of 
introducing bugs is also much higher, which might well justify a 
somewhat incompatible approach.

> This may be a bit controversial as it is a bit of a change.  But, my 
> experience is that quite a bit of extra code is written to check 
> whether or not a calculation returns a Python-scalar (because these 
> don't have the same methods as arrays).  In

The only method I can think of is typecode(). But if more array 
functionality is migrated to arrays, this might become more serious.

> When Python needs a scalar it will generally ask the object if it can 
> turn itself into an int or a float.   A notable exception is indexing 
> in a list (where Python needs an integer and won't ask the object to 
> convert if it can).  But int(b) always returns a Python integer if the 
> array has only 1 element.

Still, this is a major point in practice. There was a Numeric release 
at some point in history that always returned rank-0 array objects (I 
don't remember if by design or by mistake), and it broke lots of my 
code because I was using array elements as indices.

Another compatibility issue is routines in C modules that insist on 
scalar arguments.

As I outlined above, I'd prefer a solution in which the distinction 
disappears from the Python programmer's point of view, even if scalars 
and rank-0 arrays remain distinct in the implementation (which is 
reasonable for performance reasons).

> I'd like to know what reasons people can think of for ever returning 
> Python scalars  unless explicitly asked for.

Other than the pragmatic ones, consistency: arrays are container 
structures that store elements of particular types. You should get out 
what you put in.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From konrad.hinsen at laposte.net  Fri Feb 18 06:31:17 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 06:31:17 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151919.60607@enthought.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42151919.60607@enthought.com>
Message-ID: <ce60847bc35769e2b8287b737e998ee6@laposte.net>

On Feb 17, 2005, at 23:22, Duncan Child wrote:

> I have attached some code that illustrates part of the pain we have 
> experienced developing libraries of algorithms that can handle both 
> arrays and scalars. The attached library is the reusable part. The 
> other part of this problem is that we have lots of logic sprinkled 
> throughout our algorithms to enable them to handle both arrays and 
> scalars.

See comments below...

> Secondly, I have just been bitten by this declaration which suggests 
> that the new Numeric might handle default values better:.
>
> _vp_mod = zeros(num_pts)
>
> It would be less surprising to someone developing numeric algorithms 
> if functions like this defaulted to creating a double precision array 
> rather than integers.

If you want a function that returns float arrays, it is trivial to 
write and adds negligible overhead:

	def float_array(array_spec):
		return Numeric.array(array_spec, Numeric.Float)

No need to interfere with Numeric's principle of "smallest usable 
type", which fits well into the Python type promotion hierarchy.

More generally, I don't think defaults should be chosen with a 
particular application in mind. Arrays are a general and widely useful 
datatype in many domains. I use integer arrays as much as float arrays, 
even though my applications qualify as "numeric".

> """
> The following safe_ methods were written to handle both arrays amd 
> scalars to
> save the developer of numerical methods having to clutter their code 
> with tests
> to determine the type of the data.
> """
>
> def safe_take(a,indices):
>    # Slice the input if it is an array but not if it is a scalar

This is a very bad example. Your function does not interpret scalars as 
rank-0 arrays (for which take() would fail), but as something 
completely different.

>   def safe_copy(a):
>    # Return a copy for both scalar and array input

That is a semantically reasonable application, but also one for which a 
simple and standard solution already exists: copy.copy().

> def safe_min(a):
>    # Return the minimum of the input array or the input if it is a 
> scalar

I would argue that this is not a good example either, as "minimum over 
an array" implies a reduction operation which is not defined for a 
scalar. On the other hand, the operation you define certainly makes 
sense.

> def safe_len(a):
>    # Return the length of the input array or 1 if it is a scalar

That implies that a scalar is somehow equivalent to a rank-1 array of 
length 1, which is not the case.

Actually, all of your examples look like an attempt to recreate Matlab 
behaviour. But Python is not Matlab!

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From barrett at stsci.edu  Fri Feb 18 06:56:19 2005
From: barrett at stsci.edu (Paul Barrett)
Date: Fri Feb 18 06:56:19 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <28c54bbc706b589b401f14cb4672c918@stsci.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4214FAD7.3030109@fastmail.fm> <4214FDCD.7000901@ee.byu.edu> <28c54bbc706b589b401f14cb4672c918@stsci.edu>
Message-ID: <421601EA.9080503@stsci.edu>

Perry Greenfield wrote:

>
> On Feb 17, 2005, at 3:25 PM, Travis Oliphant wrote:
>
>>
>> This is basically new behavior that numarray has started supporting.  
>> I just think numarray missed an important case of flattened indexing 
>> that MATLAB supports.  My current proposal would distinguish between 
>> single-index array cases and tuple-index array cases.
>> I'm still thinking about the X.flat possibility.  Basically, I think 
>> that direction would requires a new "generic array view" or something 
>> like that.  It may be worth trying, but I'm not sure I want to go 
>> that direction right now until I convince more people to come on 
>> board with Numeric3.
>>
> It was new behavior in the sense that Numeric didn't support 
> multidimensional array takes and puts at the time. For a long time it 
> was the only kind of array indexing IDL supported (1-d and implicit 
> .flat of multidimensional arrays). Speaking only for myself, I found 
> the .flat semantics more often as unwanted behavior than convenient. 
> I'd rather keep the numarray behavior in this case and make the .flat 
> case explicit (but I understand the difficulty of that approach). 
> There is the possibility of a custom index option (but ugly I suppose)
>
> X[K, flatten]
>
> where flatten is a special object that indexing recognizes as 
> signaling a different interpretation to indexing.


+1 on  Travis's suggestion of X.flat[K] behavior, since I think it is 
explicit and intuitive.

- 1 on X[K. flatten]

I think in the long term that the X.flat[K] proposal should be pursued, 
even for non-contiguous arrays. Since we have decided that slice-as-view 
behavior is the default, then I believe the user should not have to 
worry whether the internal structure of an array is continguous or not.

 -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218




From barrett at stsci.edu  Fri Feb 18 07:02:16 2005
From: barrett at stsci.edu (Paul Barrett)
Date: Fri Feb 18 07:02:16 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <87oeej6xxh.fsf@welho.com>
References: <421446E3.7080202@ee.byu.edu> <87oeej6xxh.fsf@welho.com>
Message-ID: <4216034A.7020705@stsci.edu>

Timo Korvola wrote:

>Travis Oliphant <oliphant at ee.byu.edu> writes:
>  
>
>>except for the fact that X[K] where K is a a single
>>integer index array can do 1-d indexing similar to MATLAB.
>>    
>>
[snip,  snip]

>Perhaps only one ellipsis should be allowed for clarity?  Is it useful
>to allow more if they are only equivalent to colons?
>  
>
This is the case anyway, since the ellipsis operator is greedy and 
swallows up all unspecified indices. Any later ellipsis operators should 
be ignored, otherwise this situation is ambiguous.

 -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218




From barrett at stsci.edu  Fri Feb 18 07:14:13 2005
From: barrett at stsci.edu (Paul Barrett)
Date: Fri Feb 18 07:14:13 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <87zmy26gx6.fsf@welho.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>
Message-ID: <42160609.3070402@stsci.edu>

Timo Korvola wrote:

[snip, snip]

>>3)  Always returning rank-0 arrays.
>>
>>This may be a bit controversial as it is a bit of a change.
>>    
>>
>
>Indeed.  So you really do intend that if foo=array([1,2]), foo[0]
>should evaluate to array(1) rather than 1?
>
>  
>
>>But, my experience is that quite a bit of extra code is written to
>>check whether or not a calculation returns a Python-scalar
>>    
>>
>
>I suppose this may be necessary for code which operates on arrays of
>somewhat arbitrary rank and would not know without looking whether,
>e.g., foo[0] is a scalar or an array of positive rank.
>
>  
>
>>In particular len(a) does not work if a is a scalar,
>>    
>>
>
>Depends on what kinds of scalars are supported.  What about object
>arrays?
>
>  
>
>>but len(b) works if b is a rank-0 array
>>
How about proposing a PEP to extend Python's scalar behavior, so the 
len(a) works for either scalars or arrays.  Though I haven't thought 
this through in great detail, it would appear to be a transparent 
addition for most Python users who would never use this.  At the same 
time, we should also consider other behavior that would unify (or smear) 
the behavior of Python scalars and arrays.

 -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218




From rkern at ucsd.edu  Fri Feb 18 07:24:22 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Fri Feb 18 07:24:22 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42160609.3070402@stsci.edu>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42160609.3070402@stsci.edu>
Message-ID: <42160877.1070402@ucsd.edu>

Paul Barrett wrote:

> How about proposing a PEP to extend Python's scalar behavior, so the 
> len(a) works for either scalars or arrays.  Though I haven't thought 
> this through in great detail, it would appear to be a transparent 
> addition for most Python users who would never use this.  At the same 
> time, we should also consider other behavior that would unify (or smear) 
> the behavior of Python scalars and arrays.

I'm sure there are any number of people who use len(x) as a way to test 
the sequenceness of x. While it might be okay for rank-0 arrays, 
extending this to builtin ints and floats may not be a good idea. I'm 
pretty sure this would get rejected.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From konrad.hinsen at laposte.net  Fri Feb 18 08:00:14 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 08:00:14 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42160877.1070402@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42160609.3070402@stsci.edu> <42160877.1070402@ucsd.edu>
Message-ID: <82562984f3b12d2e39ddc05df631d47c@laposte.net>

On Feb 18, 2005, at 16:23, Robert Kern wrote:

> I'm sure there are any number of people who use len(x) as a way to 
> test the sequenceness of x. While it might be okay for rank-0 arrays, 
> extending this to builtin ints and floats may not be a good idea. I'm 
> pretty sure this would get rejected.

For arrays, len(x) == x.shape[0], so len(x) should fail for rank-0 
arrays anyway. As it does in Numeric.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From perry at stsci.edu  Fri Feb 18 08:13:18 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Fri Feb 18 08:13:18 2005
Subject: [Numpy-discussion] Some comments on the draft PEP (Rev 1.8)
In-Reply-To: <42155814.4040908@sympatico.ca>
References: <42155814.4040908@sympatico.ca>
Message-ID: <622f2e967fe6a234c016983cb64f6918@stsci.edu>

On Feb 17, 2005, at 9:51 PM, Colin J. Williams wrote:
>
> One final request.             In the instance constructor, please 
> avoid the use of 'type' as a parameter name if the constructor is 
> implemented in Python.  This hides the builtin type(), which is 
> sometimes useful in debugging or exploring code.
>

It shouldn't be a problem for those using the constructor in their 
code. It is a problem if one is writing a function where one wants to 
use the same parameter name (to retain an identical case) in which case 
one is masking the built-in (and thus it is necessary to alias the 
built-in). If type is not to be used I think an alternative to typecode 
should be available as that would be a misnomer if the underlying type 
is not a character. One possible alternative is "atype". Any other 
suggestions?

Perry




From tkorvola at welho.com  Fri Feb 18 08:20:19 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Fri Feb 18 08:20:19 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <4216034A.7020705@stsci.edu> (Paul Barrett's message of "Fri,
 18 Feb 2005 10:01:30 -0500")
References: <421446E3.7080202@ee.byu.edu> <87oeej6xxh.fsf@welho.com>
	<4216034A.7020705@stsci.edu>
Message-ID: <877jl5g7sb.fsf@welho.com>

Paul Barrett <barrett at stsci.edu> writes:
> Any later ellipsis operators should be ignored, otherwise this
> situation is ambiguous.

The PEP says they'll turn into colons, which is not the same as being
ignored.  It only describes advanced indexing though, which is very
different from basic indexing.  In advanced indexing an ellipsis would
pick diagonal entries, e.g., a[NewAxis, ...][[0], ...] would return
the diagonal of a.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From duncan at enthought.com  Fri Feb 18 08:50:19 2005
From: duncan at enthought.com (Duncan Child)
Date: Fri Feb 18 08:50:19 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <ce60847bc35769e2b8287b737e998ee6@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42151919.60607@enthought.com> <ce60847bc35769e2b8287b737e998ee6@laposte.net>
Message-ID: <42161CA3.40506@enthought.com>

konrad.hinsen at laposte.net wrote:

> On Feb 17, 2005, at 23:22, Duncan Child wrote:
>
>> I have attached some code that illustrates part of the pain we have 
>> experienced developing libraries of algorithms that can handle both 
>> arrays and scalars. The attached library is the reusable part. The 
>> other part of this problem is that we have lots of logic sprinkled 
>> throughout our algorithms to enable them to handle both arrays and 
>> scalars.
>
>
> See comments below...
>
>> Secondly, I have just been bitten by this declaration which suggests 
>> that the new Numeric might handle default values better:.
>>
>> _vp_mod = zeros(num_pts)
>>
>> It would be less surprising to someone developing numeric algorithms 
>> if functions like this defaulted to creating a double precision array 
>> rather than integers.
>
>
> If you want a function that returns float arrays, it is trivial to 
> write and adds negligible overhead:
>
>     def float_array(array_spec):
>         return Numeric.array(array_spec, Numeric.Float)
>
> No need to interfere with Numeric's principle of "smallest usable 
> type", which fits well into the Python type promotion hierarchy.
>
> More generally, I don't think defaults should be chosen with a 
> particular application in mind. Arrays are a general and widely useful 
> datatype in many domains. I use integer arrays as much as float 
> arrays, even though my applications qualify as "numeric".

What I meant was that accidentally ommitting the na.Float in the 
declaration below introduced a hard to find bug in my code.

_vp_mod = na.zeros(num_pts, na.Float)

I had not heard of Numeric's "smallest usable type" principle. Even so, 
I would argue that for doing signal processing the smallest usable type 
is floating point :-)

> snipped >>
>
> Actually, all of your examples look like an attempt to recreate Matlab 
> behaviour. But Python is not Matlab!

Good point, and this code was actually written by developers who were 
porting libraries of Matlab code. I thought the examples illustrated a 
more general problem that was created by Numeric handling scalars 
differently to arrays.

In another post you said:

"... as the goal is inclusion into the Python core .... I propose that 
the PEP should include unification of scalars and arrays such that for 
all practical purposes scalars *are* rank-0 arrays. "

So I think we are in broad agreement.

Regards

Duncan




>
> Konrad.
> -- 
> ---------------------------------------------------------------------
> Konrad Hinsen
> Laboratoire L?on Brillouin, CEA Saclay,
> 91191 Gif-sur-Yvette Cedex, France
> Tel.: +33-1 69 08 79 25
> Fax: +33-1 69 08 82 61
> E-Mail: hinsen at llb.saclay.cea.fr
> ---------------------------------------------------------------------
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_ide95&alloc_id396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>





From rkern at ucsd.edu  Fri Feb 18 09:28:23 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Fri Feb 18 09:28:23 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <82562984f3b12d2e39ddc05df631d47c@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42160609.3070402@stsci.edu> <42160877.1070402@ucsd.edu> <82562984f3b12d2e39ddc05df631d47c@laposte.net>
Message-ID: <42162562.9080102@ucsd.edu>

konrad.hinsen at laposte.net wrote:
> On Feb 18, 2005, at 16:23, Robert Kern wrote:
> 
>> I'm sure there are any number of people who use len(x) as a way to 
>> test the sequenceness of x. While it might be okay for rank-0 arrays, 
>> extending this to builtin ints and floats may not be a good idea. I'm 
>> pretty sure this would get rejected.
> 
> 
> For arrays, len(x) == x.shape[0], so len(x) should fail for rank-0 
> arrays anyway. As it does in Numeric.

I'm not averse to len(x) returning 0 when given a rank-0 array. I see it 
as giving up one consistency (that scalar-like objects don't have 
lengths) for another (arrays having a common set of operations that one 
can expect regardless of rank or shape).

My objection was to extending that set of operations to other standard 
objects where they make less sense. Although the len(x) 
sequenceness-test is a reasonably common idiom, it's not expected to be 
foolproof against any input. However, the test shouldn't stop working on 
core objects that are already there.

In any case, len(x) is probably one of the less-common operations one 
would want to perform seamlessly on scalar and rank-n outputs from 
Numeric functions. x.typecode() and x.shape would probably top my list.

Might our PEP efforts be better spent locating and fixing the places in 
core Python where rank-0 arrays won't be accepted as core ints and 
floats? List/tuple indexing is one. Extension code that demands an 
actual int object where it could be looser might already be considered 
deficient considering the int/long unification.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From oliphant at ee.byu.edu  Fri Feb 18 09:42:29 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 09:42:29 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <877jl5g7sb.fsf@welho.com>
References: <421446E3.7080202@ee.byu.edu> <87oeej6xxh.fsf@welho.com>	<4216034A.7020705@stsci.edu> <877jl5g7sb.fsf@welho.com>
Message-ID: <421628D2.4020608@ee.byu.edu>

Timo Korvola wrote:

>Paul Barrett <barrett at stsci.edu> writes:
>  
>
>>Any later ellipsis operators should be ignored, otherwise this
>>situation is ambiguous.
>>    
>>
>
>The PEP says they'll turn into colons, which is not the same as being
>ignored. 
>
This is the current default   behavior for Numeric standard slicing, so 
I see this as a continuation
of previous behavior.

I guess it should be more accurately stated, however, because two 
ellipses right next to each other will always be the same as one ellipse. 
Only if they are separated by something else will another ellipse be 
interpreted as a single colon.

I'm happy to raise an error instead if people have not been using this 
"feature"

-Travis




From konrad.hinsen at laposte.net  Fri Feb 18 09:51:19 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 09:51:19 2005
Subject: [Numpy-discussion] Multiarray PEP: iterators
Message-ID: <d9952a7aa4628a84a86727a753cfd909@laposte.net>

The PEP says:

   Iterator

       An iterator will be defined that will walk through any array,
       returning a rank-0 array at each step.  Rank-0 arrays act like
       the appropriate Python scalar and will be converted to one
       whenever Python asks the object explicitly to try and do so.
       Order of the iteration is the same for contiguous and
       discontiguous arrays.  The last index always varies the fastest


At the moment, iteration over a rank-N array yields rank-(N-1) arrays 
(indexing along the first dimension). I consider this vastly more 
useful than iterating over the rank-0 elements and thus ignoring the 
array structure completely.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From oliphant at ee.byu.edu  Fri Feb 18 10:00:24 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 10:00:24 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4bf0e99f809d592b068270a200b6deb0@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net>
Message-ID: <42162CEC.3090607@ee.byu.edu>

konrad.hinsen at laposte.net wrote:

> My view of pythonicity is that retrieving a value should be written as 
> attribute access. Methods are more appropriate if there are arguments 
> (no choice then anyway) or side effects. So I'd have .type as an 
> attribute.

That's my view as well.

>
> BTW, as the goal is inclusion into the Python core, why not
>
> 1) Use Python type objects for array creation and as the values of the 
> .type
>     attribute.

Do you mean have register each of the 21 different types of arrays as a 
new type object?   Hmm.   That is an interesting idea.   I'm a little 
worried about implications for having the arrays behave together, though 
it would make it more straightforward to define specific mixed 
operations.    This does deserve a little more thought.

> 2) Implement scalar types for those array element types that currently
>     have no Python scalar equivalent (e.g. UInt16).

Do you think this would fly with the Python folks.  Counting the 
suggestion above, we would be encouraging the creation of 39 new types 
to the Python core.  My current count shows the current number of types 
as 35 so we would basically double that.   This doesn't have to matter, 
but I'd have to hear how Guido feels about something like that.

> 3) Implement the same set of attributes of methods for scalar types and
>    arrays.

That would be ideal.   But, I'm not sure what kind of chance we have 
with that.

-Travis




From Fernando.Perez at colorado.edu  Fri Feb 18 10:03:13 2005
From: Fernando.Perez at colorado.edu (Fernando.Perez at colorado.edu)
Date: Fri Feb 18 10:03:13 2005
Subject: [Numpy-discussion] Multiarray PEP: iterators
In-Reply-To: <d9952a7aa4628a84a86727a753cfd909@laposte.net>
References: <d9952a7aa4628a84a86727a753cfd909@laposte.net>
Message-ID: <1108749739.42162dab6219e@webmail.colorado.edu>

Quoting konrad.hinsen at laposte.net:

> At the moment, iteration over a rank-N array yields rank-(N-1) arrays
> (indexing along the first dimension). I consider this vastly more
> useful than iterating over the rank-0 elements and thus ignoring the
> array structure completely.

Completely agreed.  I'm sure I'm not the only one who has code like

for row in matrix:
  do_something_with(row)

Furthermore, this is consistent with python's behavior for nested lists:

In [3]: nested=[['first row',1,2],['second row',3,4]]

In [4]: for row in nested:
   ...:     print row
   ...:
['first row', 1, 2]
['second row', 3, 4]

Python does not flatten nested structures when iterating over them, neither
should numerix.  I think the addition (which Travis seems keen on) the .flat
iterator attribute is the right approach here.  Normal iterators would maintain
structure (and behave like pyhon's for nested lists), while .flat would return
an element-wise iterator for cases where such a thing is needed (which do
exist, obviously).

Regards,

f



From tkorvola at welho.com  Fri Feb 18 11:03:18 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Fri Feb 18 11:03:18 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42162562.9080102@ucsd.edu> (Robert Kern's message of "Fri, 18
 Feb 2005 09:26:58 -0800")
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>
	<42160609.3070402@stsci.edu> <42160877.1070402@ucsd.edu>
	<82562984f3b12d2e39ddc05df631d47c@laposte.net>
	<42162562.9080102@ucsd.edu>
Message-ID: <87wtt5elnw.fsf@welho.com>

Robert Kern <rkern at ucsd.edu> writes:
> I'm not averse to len(x) returning 0 when given a rank-0 array. I see
> it as giving up one consistency (that scalar-like objects don't have
> lengths) for another (arrays having a common set of operations that
> one can expect regardless of rank or shape).

That would be akin to making 0/0 return 0.  It is possible to create
arrays of length zero, e.g, by indexing with an empty slice, and these
should not be confused with rank zero arrays.

> Might our PEP efforts be better spent locating and fixing the places
> in core Python where rank-0 arrays won't be accepted as core ints and
> floats?

Sounds useful.

> List/tuple indexing is one.

Rank 0 arrays should also be kept in mind when defining array indexing
in the PEP.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From oliphant at ee.byu.edu  Fri Feb 18 11:13:16 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 11:13:16 2005
Subject: [Numpy-discussion] Proposed X.flat solution
Message-ID: <42163DED.7080808@ee.byu.edu>

O.K.  here is my X.flat solution.

I've decided to make use of the fancy-new, numarray-inspired idea of the 
UPDATEIFCOPY flag for array creation.  Basically, X.flat will create a 
new array if X is discontiguous but with the UPDATEIFCOPY flag set so 
that upon destruction the contents will get copied back to X. 

So, it will seem like the X.flat array is updating the original array 
(provided it get's dereferenced someday (which it should because you 
rarely) keep references to X.flat unless you really meant ravel(X).

This is the simplest solution.  It's not quite as fancy as the indexable 
iterator (which could still be implemented at some point) but it lets me 
move forward.

-Travis




From Chris.Barker at noaa.gov  Fri Feb 18 11:14:18 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Fri Feb 18 11:14:18 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <42163E14.9050700@noaa.gov>

Travis Oliphant wrote:

> I'm still thinking about the X.flat possibility.

There was a discussion about his a while back, but I don't remember what 
conclusions were reached (if any). The technical details are beyond me, 
but it seems quite clear to me that A.flat() should be supported for all 
arrays. Otherwise, it makes it very hard to write generic code. What's 
the point of having discontiguous arrays if you can't use them 
everywhere you can use contiguous ones?


Fernando Perez wrote:
> Granted, the indexing will be costlier for a non-contiguous object, but 
> for the external users this provides a clean API.

Right. A number of things are costlier for non-contiguous arrays. If you 
need to optimize, you can make sure your arrays are contiguous where it 
matters. If you don't, it should "just work"


> But Python is not Matlab!
> 
> Konrad.

hear! hear!. If I wanted Matlab, I'd use Matlab (or Octave, or Psilab)







-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From rkern at ucsd.edu  Fri Feb 18 11:14:21 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Fri Feb 18 11:14:21 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <87wtt5elnw.fsf@welho.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>	<42160609.3070402@stsci.edu> <42160877.1070402@ucsd.edu>	<82562984f3b12d2e39ddc05df631d47c@laposte.net>	<42162562.9080102@ucsd.edu> <87wtt5elnw.fsf@welho.com>
Message-ID: <42163E66.9040908@ucsd.edu>

Timo Korvola wrote:
> Robert Kern <rkern at ucsd.edu> writes:
> 
>>I'm not averse to len(x) returning 0 when given a rank-0 array. I see
>>it as giving up one consistency (that scalar-like objects don't have
>>lengths) for another (arrays having a common set of operations that
>>one can expect regardless of rank or shape).
> 
> 
> That would be akin to making 0/0 return 0.  It is possible to create
> arrays of length zero, e.g, by indexing with an empty slice, and these
> should not be confused with rank zero arrays.

Fair enough. I'm not averse to len(x) raising an exception when x is a 
rank-0 array either.  :-)

>>Might our PEP efforts be better spent locating and fixing the places
>>in core Python where rank-0 arrays won't be accepted as core ints and
>>floats?
> 
> 
> Sounds useful.
> 
> 
>>List/tuple indexing is one.
> 
> 
> Rank 0 arrays should also be kept in mind when defining array indexing
> in the PEP.

Hopefully, this would drop out cleanly from the regular array-indexing 
and not require too much in the way of special-casing.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From rlw at stsci.edu  Fri Feb 18 11:32:21 2005
From: rlw at stsci.edu (Rick White)
Date: Fri Feb 18 11:32:21 2005
Subject: [Numpy-discussion] Proposed X.flat solution
In-Reply-To: <42163DED.7080808@ee.byu.edu>
Message-ID: <Pine.GSO.4.44.0502181424000.1023-100000@sundog.stsci.edu>

On Fri, 18 Feb 2005, Travis Oliphant wrote:

> O.K.  here is my X.flat solution.
>
> I've decided to make use of the fancy-new, numarray-inspired idea of the
> UPDATEIFCOPY flag for array creation.  Basically, X.flat will create a
> new array if X is discontiguous but with the UPDATEIFCOPY flag set so
> that upon destruction the contents will get copied back to X.
>
> So, it will seem like the X.flat array is updating the original array
> (provided it get's dereferenced someday (which it should because you
> rarely) keep references to X.flat unless you really meant ravel(X).
>
> This is the simplest solution.  It's not quite as fancy as the indexable
> iterator (which could still be implemented at some point) but it lets me
> move forward.

Hmm.  It seems to me that if it is not a live view you can run into
serious problems.  What happens if you do this:

a = zeros((10,10))
b = a[::2,::2].flat
b[4] = 1
a[0,8] = 2
del b

Doesn't your behavior lead to a[0,8] being overwritten to 1 by the copy
back from b?




From barrett at stsci.edu  Fri Feb 18 11:52:20 2005
From: barrett at stsci.edu (Paul Barrett)
Date: Fri Feb 18 11:52:20 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42163E66.9040908@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>	<42160609.3070402@stsci.edu> <42160877.1070402@ucsd.edu>	<82562984f3b12d2e39ddc05df631d47c@laposte.net>	<42162562.9080102@ucsd.edu> <87wtt5elnw.fsf@welho.com> <42163E66.9040908@ucsd.edu>
Message-ID: <4216473A.3090909@stsci.edu>

Robert Kern wrote:

> Timo Korvola wrote:
>
>> Robert Kern <rkern at ucsd.edu> writes:
>>
>>> I'm not averse to len(x) returning 0 when given a rank-0 array. I see
>>> it as giving up one consistency (that scalar-like objects don't have
>>> lengths) for another (arrays having a common set of operations that
>>> one can expect regardless of rank or shape).
>>
>>
>>
>> That would be akin to making 0/0 return 0.  It is possible to create
>> arrays of length zero, e.g, by indexing with an empty slice, and these
>> should not be confused with rank zero arrays.
>
>
> Fair enough. I'm not averse to len(x) raising an exception when x is a 
> rank-0 array either.  :-)

Or it could return 'None', i.e. undefined.

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218




From oliphant at ee.byu.edu  Fri Feb 18 12:29:21 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 12:29:21 2005
Subject: [Numpy-discussion] Proposed X.flat solution
In-Reply-To: <Pine.GSO.4.44.0502181424000.1023-100000@sundog.stsci.edu>
References: <Pine.GSO.4.44.0502181424000.1023-100000@sundog.stsci.edu>
Message-ID: <42164FF6.5010801@ee.byu.edu>

>Hmm.  It seems to me that if it is not a live view you can run into
>serious problems.  What happens if you do this:
>
>a = zeros((10,10))
>b = a[::2,::2].flat
>b[4] = 1
>a[0,8] = 2
>del b
>
>Doesn't your behavior lead to a[0,8] being overwritten to 1 by the copy
>back from b?
>  
>
Well in this particular case since a is contiguous to begin with, b is 
not a copy so the copy-back would not occur.

But, you bring up a valid point that if "a" were discontiguous which 
forced a copy, then any later modifications to "a" would be ignored.
Now, this could be fixed by making "a" read-only for the duration of 
existence of the flattened reference. 

In fact, it probably stands to reason that whenever, UPDATEIFCOPY is 
used on the C-level, the to-be-copied-to array is flagged readonly for the
duration of existence of the "reference."

Besides that point, to clarify what we are talking about here:

1)  The PEP has been modified so that X[K] is interpreted as X[K,]  for 
integer arrays, and the previously-discussed distinction disappears.

2)  It is desired that X.flat[K]  provide the flattened behavior.   How 
should this be done?

     a) having X.flat return an indexable iterator (with an __array__ 
method that will automatically return an update-if-copy array on
         request from asarray() or C-level equivalent)

     b) having X.flat return an update-if-copy array (with X flagged as 
readonly while a reference to X.flat exists)

     c) it shouldn't be done and X.flat should just raise an error if X 
is not contiguous.

Please chime in and "vote" for one of the three solutions (or another 
one not covered) for 2 if you haven't already expressed your view.

By the way, I think it's been settled that X.flat will not behave the 
same as ravel(X) (which should probably be a method---X.ravel() ).

-Travis




From cookedm at physics.mcmaster.ca  Fri Feb 18 13:12:10 2005
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Fri Feb 18 13:12:10 2005
Subject: [Numpy-discussion] Proposed X.flat solution
In-Reply-To: <42164FF6.5010801@ee.byu.edu> (Travis Oliphant's message of
 "Fri, 18 Feb 2005 13:28:38 -0700")
References: <Pine.GSO.4.44.0502181424000.1023-100000@sundog.stsci.edu>
	<42164FF6.5010801@ee.byu.edu>
Message-ID: <qnkhdk9po93.fsf@arbutus.physics.mcmaster.ca>

Travis Oliphant <oliphant at ee.byu.edu> writes:

>>Hmm.  It seems to me that if it is not a live view you can run into
>>serious problems.  What happens if you do this:
>>
>>a = zeros((10,10))
>>b = a[::2,::2].flat
>>b[4] = 1
>>a[0,8] = 2
>>del b
>>
>>Doesn't your behavior lead to a[0,8] being overwritten to 1 by the copy
>>back from b?
>>
>>
> Well in this particular case since a is contiguous to begin with, b is
> not a copy so the copy-back would not occur.
>
> But, you bring up a valid point that if "a" were discontiguous which
> forced a copy, then any later modifications to "a" would be ignored.
> Now, this could be fixed by making "a" read-only for the duration of
> existence of the flattened reference.

> In fact, it probably stands to reason that whenever, UPDATEIFCOPY is
> used on the C-level, the to-be-copied-to array is flagged readonly for
> the duration of existence of the "reference."

Ugh. I realize I'm going to have to sit down and think on this first,
and look at code. My gut feeling is UPDATEIFCOPY and read-write-locks
(which is essentially what you're proposing) will make for some odd,
hard-to-track down, errors.

> Besides that point, to clarify what we are talking about here:
>
> 1)  The PEP has been modified so that X[K] is interpreted as X[K,]
> for integer arrays, and the previously-discussed distinction
> disappears.
>
> 2)  It is desired that X.flat[K]  provide the flattened behavior.
> How should this be done?
>
>      a) having X.flat return an indexable iterator (with an __array__
> method that will automatically return an update-if-copy array on
>          request from asarray() or C-level equivalent)

I'm +1. This feels to be the simplest (in sense of use, not necessarily
implementation :-), and the obvious use. Here, X.flat is *always* a
valid view of X.

>      b) having X.flat return an update-if-copy array (with X flagged
> as readonly while a reference to X.flat exists)

-0, for the locking reasons I mention above. For this usage, I'd make
it a method instead.

>      c) it shouldn't be done and X.flat should just raise an error if
> X is not contiguous.

If this one, then I'd say that X.flat is useless and should be
removed. ravel() (or X.ravel()) should return a view or copy depending
on whether X is contiguous or not. A copy could be forced by a keyword
argument: ravel(X, copy=True)

While we're at it, I'd argue that "ravel" is a bad word for this
usage -- it's not obvious what it does. "flatten" might be a better word.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From oliphant at ee.byu.edu  Fri Feb 18 13:25:22 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 13:25:22 2005
Subject: [Numpy-discussion] Proposed X.flat solution
In-Reply-To: <qnkhdk9po93.fsf@arbutus.physics.mcmaster.ca>
References: <Pine.GSO.4.44.0502181424000.1023-100000@sundog.stsci.edu>	<42164FF6.5010801@ee.byu.edu> <qnkhdk9po93.fsf@arbutus.physics.mcmaster.ca>
Message-ID: <42165CF3.8070104@ee.byu.edu>

David M. Cooke wrote:

>>In fact, it probably stands to reason that whenever, UPDATEIFCOPY is
>>used on the C-level, the to-be-copied-to array is flagged readonly for
>>the duration of existence of the "reference."
>>    
>>
>
>Ugh. I realize I'm going to have to sit down and think on this first,
>and look at code. My gut feeling is UPDATEIFCOPY and read-write-locks
>(which is essentially what you're proposing) will make for some odd,
>hard-to-track down, errors.
>  
>
The problem is with this idea of UPDATEIFCOPY which is borrowed from 
numarray and which users over there seem to like.   It just seems to
me that if an array is going to be "automatically" copied back to 
another array (this is currently done in numarray), then the 
array-to-be-copied to
needs to be flagged read-only until the copy is complete.  Then, an 
error will be raised whenever the users tries to do something that would
cause "hard-to-track-down" bugs (like write to an array that is going to 
be clobbered by a copy-back).  

If the UPDATEIFCOPY is only used for C-extensions where you need to get 
a nice array from a potentially "bad" one (e.g. contiguous from 
discontiguous) and the thinking is that you would not be using the 
original one while you played with the copy.  But, if the C-extension 
calls arbitrary code (as is sometimes done), then this is not 
guaranteed.  So, I think, if you are going to have UPDATEIFCOPY, you 
need to set the original array readonly until the copy is complete.


>  
>
>>     b) having X.flat return an update-if-copy array (with X flagged
>>as readonly while a reference to X.flat exists)
>>    
>>
>
>  
>
Yeah, I don't like this anymore either. 


I like X.flatten() better than X.ravel() too. 

-Travis





From konrad.hinsen at laposte.net  Fri Feb 18 13:44:27 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 13:44:27 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42162CEC.3090607@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu>
Message-ID: <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net>

On 18.02.2005, at 18:59, Travis Oliphant wrote:

> Do you mean have register each of the 21 different types of arrays as  
> a new type object?   Hmm.   That is an interesting idea.   I'm a  
> little worried

Yes. It only costs a bit of memory for the type objects, so why not?

>> 2) Implement scalar types for those array element types that currently
>>     have no Python scalar equivalent (e.g. UInt16).
>
> Do you think this would fly with the Python folks.  Counting the  
> suggestion above, we would be encouraging the creation of 39 new types  
> to the Python core.  My current count shows the current number of  
> types as 35 so

Those new types would (if all goes well) be part of the standard  
library, but not built-in types. Compared to the number of types and  
classes in the standard library, the addition is not so big. There  
wouldn't be literals either. Anyone who doesn't use the array module  
could thus safely ignore the existence of those types.

Implementation-wise, the new types could well be rank-0 arrays  
internally and thus add nearly no overhead.

>  we would basically double that.   This doesn't have to matter, but  
> I'd have to hear how Guido feels about something like that.

Of course!

>> 3) Implement the same set of attributes of methods for scalar types  
>> and
>>    arrays.
>
> That would be ideal.   But, I'm not sure what kind of chance we have  
> with that.

Me neither. It might depend on clever presentation of the project.  
Perhaps some bribing could help ;-)

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From stephen.walton at csun.edu  Fri Feb 18 13:59:24 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Fri Feb 18 13:59:24 2005
Subject: [Numpy-discussion] Multiarray PEP: iterators
In-Reply-To: <1108749739.42162dab6219e@webmail.colorado.edu>
References: <d9952a7aa4628a84a86727a753cfd909@laposte.net> <1108749739.42162dab6219e@webmail.colorado.edu>
Message-ID: <421664DF.1090409@csun.edu>

Fernando.Perez at colorado.edu wrote:

>Completely agreed.  I'm sure I'm not the only one who has code like
>
>for row in matrix:
>  do_something_with(row)
>  
>
You're not.  I vote +1 for keeping this behavior.




From stephen.walton at csun.edu  Fri Feb 18 14:03:14 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Fri Feb 18 14:03:14 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42161CA3.40506@enthought.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42151919.60607@enthought.com> <ce60847bc35769e2b8287b737e998ee6@laposte.net> <42161CA3.40506@enthought.com>
Message-ID: <421665E1.6000008@csun.edu>

Duncan Child wrote:

> In another post [Konrad Hinsen] said:
>
> "... as the goal is inclusion into the Python core .... I propose that 
> the PEP should include unification of scalars and arrays such that for 
> all practical purposes scalars *are* rank-0 arrays. "
>
> So I think we are in broad agreement.

It seems to me that actually getting the above behavior is going to 
require Guido van Rossum et al. to change how scalars behave in the 
Python core.  As I said, right now min(scalar) in stock Python raises a 
TypeError on the grounds that scalars can't be iterated over.




From oliphant at ee.byu.edu  Fri Feb 18 14:07:24 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 14:07:24 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net>
Message-ID: <421666F3.6050702@ee.byu.edu>

>> Do you mean have register each of the 21 different types of arrays 
>> as  a new type object?   Hmm.   That is an interesting idea.   I'm a  
>> little worried
>
> Yes. It only costs a bit of memory for the type objects, so why not?

Let me see if i understand you here.  Since the type of the array is 
bascially handled by the
PyArray_Descr*  structure.   Are you suggesting turning that into a 
full-fledged PythonObject with
new type?   I had thought of this before since it's basically calling 
for us to do something like that.  Or do you have something else in mind?

-Travis




From oliphant at ee.byu.edu  Fri Feb 18 14:46:10 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 14:46:10 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4bf0e99f809d592b068270a200b6deb0@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net>
Message-ID: <42166FF3.9050908@ee.byu.edu>

>> When Python needs a scalar it will generally ask the object if it can 
>> turn itself into an int or a float.   A notable exception is indexing 
>> in a list (where Python needs an integer and won't ask the object to 
>> convert if it can).  But int(b) always returns a Python integer if 
>> the array has only 1 element.
>
>
> Still, this is a major point in practice. There was a Numeric release 
> at some point in history that always returned rank-0 array objects (I 
> don't remember if by design or by mistake), and it broke lots of my 
> code because I was using array elements as indices.
>
I posted a question to python-dev about changing the internals of Python 
to support asking objects
to convert to ints in slicing. 

While open to the idea for Python 3000, Guido does not seem favorable to 
the idea for Python 2.X The problem, Guido mentions was that float-like 
objects can convert to ints by truncation and he doesn't want to allow 
floats to be used as indexes.   He feels it would break too much code.

Using this line of reasoning, then, arrays should not be used as indexes 
unless they are explicitly converted to integers:  int(a)

I have proposed a second solution that asks if a special check could be 
made for rank-0 arrayobjects (of integer type) if and when they are 
allowed in the core.


I think Konrad's valid point regarding consistency is that to the user 
it looks like he is making an array of integers

a = array([1,2,3,4])

so it is confusing (the first time) if

a[0] fails to act like an integer when requested for slicing an array.

Of course underneath, a is not an array of integers (it is an array of 
homogeneous c-ints converted from the Python integer and so why should 
a[0] be a Python integer). 

This is the problem.  We want different things to act the same all the 
time when fundamentally they are different.   Python allows this in many 
cases, but doesn't seem to be fully transparent in this regard.

When I first started with Python, as a MATLAB user I was confused by the 
fact that lists, tuples, and arrays were all different things that had 
some commonality.  I was much happier when I just decided to let them be 
different and write my code accordingly.   (Interestingly enough since 
that time MATLAB has added other types to their language as well --- 
which don't always get along). 

Here I think we should just let rank-0 arrays and Python scalars be 
different things and let people know that instead of trying to mask the 
situation which ultimately confuses things.

-Travis





From oliphant at ee.byu.edu  Fri Feb 18 15:05:11 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 15:05:11 2005
Subject: [Numpy-discussion] rank-0 arrays ideas
Message-ID: <4216747C.7090504@ee.byu.edu>

 From the current  PEP:

 
    Proposed Solution:

        The solution proposed by this PEP is to fix the places in Python
        that could use rank-0 arrayobjects to allow for them before raising
        an exception (this will be in the core after-all).

        A Python scalar will never be returned unless explicitly
        requested.  I think this is the cleanest, easiest to understand
        and code for solution.  It may require some explicity conversion
        to int here and there, but it seems like a case of "explicit is
        better than implicit"

        Possible compromises:
            -  a compromise could be made for OBJECT arrays and
               perhaps LONG arrays if needed.

            - a special flag could be defined which is the default
              when an array of integers is constructed and which when
              set rank-0 array returns behave differently.

        It is also proposed that slicing, indexing and len() do
        not work (i.e. they raise an error) for rank-0 arrays.


Comments:

-Travis




From tkorvola at welho.com  Fri Feb 18 15:22:14 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Fri Feb 18 15:22:14 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <421665E1.6000008@csun.edu> (Stephen Walton's message of "Fri,
 18 Feb 2005 14:02:09 -0800")
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>
	<42151919.60607@enthought.com>
	<ce60847bc35769e2b8287b737e998ee6@laposte.net>
	<42161CA3.40506@enthought.com> <421665E1.6000008@csun.edu>
Message-ID: <87wtt5e9oz.fsf@welho.com>

Stephen Walton <stephen.walton at csun.edu> writes:
> As I said, right now min(scalar) in stock Python raises
> a TypeError on the grounds that scalars can't be iterated over.

This is not very different from Numarray, where min(rank_0_array)
raises ValueError because min is applied to an empty sequence.
Iteration normally applies to the first dimenson but rank 0 arrays
have no dimensions.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From dd55 at cornell.edu  Fri Feb 18 15:28:23 2005
From: dd55 at cornell.edu (Darren Dale)
Date: Fri Feb 18 15:28:23 2005
Subject: [Numpy-discussion] rank-0 arrays ideas
In-Reply-To: <4216747C.7090504@ee.byu.edu>
References: <4216747C.7090504@ee.byu.edu>
Message-ID: <200502181826.38518.dd55@cornell.edu>

On Friday 18 February 2005 06:04 pm, Travis Oliphant wrote:
>  From the current  PEP:
>
>
>     Proposed Solution:
>
>         The solution proposed by this PEP is to fix the places in Python
>         that could use rank-0 arrayobjects to allow for them before raising
>         an exception (this will be in the core after-all).
>
>         A Python scalar will never be returned unless explicitly
>         requested.  I think this is the cleanest, easiest to understand
>         and code for solution.  It may require some explicity conversion
>         to int here and there, but it seems like a case of "explicit is
>         better than implicit"
>
>         Possible compromises:
>             -  a compromise could be made for OBJECT arrays and
>                perhaps LONG arrays if needed.
>
>             - a special flag could be defined which is the default
>               when an array of integers is constructed and which when
>               set rank-0 array returns behave differently.
>
>         It is also proposed that slicing, indexing and len() do
>         not work (i.e. they raise an error) for rank-0 arrays.
>
>
> Comments:

I have one:

The following script measures the time it takes for in-place array operations, 
comparing 0-D arrays with scalars. The results are 

0.81 0D double
0.85 0D int
0.11 Scalar float
0.12 Scalar int

Would Numeric3 address this issue?

---

import Numeric as N
import time


a=N.zeros(100000,'d')

one=N.array(1,'d')
d=time.clock()
for i in N.arange(100000):
    a[i]+=one
d = time.clock()-d
print d,'0D double'

one=N.array(1)
d=time.clock()
for i in N.arange(100000):
    a[i]+=one
d = time.clock()-d
print d,'0D int'

d=time.clock()
for i in N.arange(100000):
    a[i]+=1.
d = time.clock()-d
print d,'Scalar float'

d=time.clock()
for i in N.arange(100000):
    a[i]+=1
d = time.clock()-d
print d,'Scalar int'

-- 

Darren



From rkern at ucsd.edu  Fri Feb 18 20:38:22 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Fri Feb 18 20:38:22 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net>
Message-ID: <4216C29F.2090105@ucsd.edu>

konrad.hinsen at laposte.net wrote:
> On 18.02.2005, at 18:59, Travis Oliphant wrote:

>> Do you think this would fly with the Python folks.  Counting the  
>> suggestion above, we would be encouraging the creation of 39 new 
>> types  to the Python core.  My current count shows the current number 
>> of  types as 35 so
> 
> 
> Those new types would (if all goes well) be part of the standard  
> library, but not built-in types. Compared to the number of types and  
> classes in the standard library, the addition is not so big. There  
> wouldn't be literals either. Anyone who doesn't use the array module  
> could thus safely ignore the existence of those types.

I don't see the problem that this approach would solve. It doesn't solve 
the list/tuple indexing problem by itself. Even if the types are part of 
the standard library, they won't be bona-fide ints, so the indexing code 
would still have to be modified to check for them.

I *do* like the idea of the typecode objects, however they are 
implemented, to be able to act as constructors.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From konrad.hinsen at laposte.net  Fri Feb 18 23:45:24 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 23:45:24 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <421665E1.6000008@csun.edu>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42151919.60607@enthought.com> <ce60847bc35769e2b8287b737e998ee6@laposte.net> <42161CA3.40506@enthought.com> <421665E1.6000008@csun.edu>
Message-ID: <a6e5c7b31d2d89c06eef4ed47a30b2e1@laposte.net>

On 18.02.2005, at 23:02, Stephen Walton wrote:

> Duncan Child wrote:
>
>> In another post [Konrad Hinsen] said:
>>
>> "... as the goal is inclusion into the Python core .... I propose  
>> that the PEP should include unification of scalars and arrays such  
>> that for all practical purposes scalars *are* rank-0 arrays. "
>>
>> So I think we are in broad agreement.
>
> It seems to me that actually getting the above behavior is going to  
> require Guido van Rossum et al. to change how scalars behave in the  
> Python core.  As I said, right now min(scalar) in stock Python raises  
> a TypeError on the grounds that scalars can't be iterated over.

That's fine - scalars (like rank-0 arrays) cannot be iterated over.

However, there would indeed have to be changes to scalars. For example,  
1.shape would have to return ().

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Sat Feb 19 00:00:29 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 00:00:29 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <421666F3.6050702@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <421666F3.6050702@ee.byu.edu>
Message-ID: <6eef976ba50ce690a5215cad728e228e@laposte.net>

On 18.02.2005, at 23:06, Travis Oliphant wrote:

> Let me see if i understand you here.  Since the type of the array is  
> bascially handled by the
> PyArray_Descr*  structure.   Are you suggesting turning that into a  
> full-fledged PythonObject with
> new type?   I had thought of this before since it's basically calling  
> for us to do something like that.  Or do you have something else in  
> mind?

I was thinking of the scalar types than of the arrays. What I am  
proposing is to have a Int16, Int32, UInt 16 etc. as Python scalar  
types (though not built-in types) and use the type objects to create  
arrays and to identify the type of array elements. Those types would be  
part of a type hierarchy for type testing.

One could then either have a single array type that stores the type  
object for the elements internally, or have different Python types for  
every kind of array (i.e. Int16Array, Int32Array). I don't see any  
important difference there, so I have no clear opinion on this.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Sat Feb 19 00:13:25 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 00:13:25 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42166FF3.9050908@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42166FF3.9050908@ee.byu.edu>
Message-ID: <18bd99d55abc05ceadd854a1a66d75ec@laposte.net>

On 18.02.2005, at 23:45, Travis Oliphant wrote:

> While open to the idea for Python 3000, Guido does not seem favorable  
> to the idea for Python 2.X The problem, Guido mentions was that  
> float-like objects can convert to ints by truncation and he doesn't  
> want to allow floats to be used as indexes.   He feels it would break  
> too much code.

I can agree with that. Basically the problem is the type hierarchy that  
is too simple (but many if not most programming language suffer from  
that problem). Type conversions are all handled in the same way, which  
doesn't give enough flexibility. But we won't change that of course.

> Of course underneath, a is not an array of integers (it is an array of  
> homogeneous c-ints converted from the Python integer and so why should  
> a[0] be a Python integer).

That would basically mean to change the status of arrays from  
generalized sequence objects to something different. Why not, but then  
this should be made clear to the programmer, in particular by having a  
different printed representation for rank-0 arrays and scalars. It also  
means adding some conversion functions, e.g. for extracting a Python  
Object from a rank-0 Python Object array.

Still, I am worried about two aspects:

1) The amount of confusion this generates among users. The distinction  
between scalars and rank-0 arrays has no utility for the programmer, it  
exists only for implementation and political reasons. I am not looking  
forward to explaining this in my Python courses for beginners.

2) Compatibility with existing code. I am not sure I will convert my  
code to such conventions any time soon, because it requires inspecting  
every single indexing operation in its particular context to see if the  
index could be a rank-0 integer array. There is no way to spot those  
cases by textual analysis. So this change could reduce acceptance to  
the point where there is no interest in pursuing the project any more.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Sat Feb 19 00:15:28 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 00:15:28 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4216C29F.2090105@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu>
Message-ID: <01b322c0be695e314111300a363aaae6@laposte.net>

On 19.02.2005, at 05:37, Robert Kern wrote:

> I don't see the problem that this approach would solve. It doesn't  
> solve the list/tuple indexing problem by itself. Even if the types are  
> part of the standard library, they won't be bona-fide ints, so the  
> indexing code would still have to be modified to check for them.

Yes, but it could check for "integer type" (using the type hierarchy)  
rather than convert everything to an integer with the problem that  
Guido pointed out.

However, my original motivation was consistency of usage. Python has  
type objects to specify types, so I'd rather use them than introduce  
another way to specify the type of array elements.

> I *do* like the idea of the typecode objects, however they are  
> implemented, to be able to act as constructors.

That is an interesting idea as well, though a slightly different one.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From cjw at sympatico.ca  Sat Feb 19 05:45:28 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Sat Feb 19 05:45:28 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <18bd99d55abc05ceadd854a1a66d75ec@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42166FF3.9050908@ee.byu.edu> <18bd99d55abc05ceadd854a1a66d75ec@laposte.net>
Message-ID: <421742C2.1010601@sympatico.ca>

konrad.hinsen at laposte.net wrote:

> On 18.02.2005, at 23:45, Travis Oliphant wrote:
>
>> While open to the idea for Python 3000, Guido does not seem 
>> favorable  to the idea for Python 2.X The problem, Guido mentions was 
>> that  float-like objects can convert to ints by truncation and he 
>> doesn't  want to allow floats to be used as indexes.   He feels it 
>> would break  too much code.
>
>
> I can agree with that. Basically the problem is the type hierarchy 
> that  is too simple (but many if not most programming language suffer 
> from  that problem). Type conversions are all handled in the same way, 
> which  doesn't give enough flexibility. But we won't change that of 
> course.
>
>> Of course underneath, a is not an array of integers (it is an array 
>> of  homogeneous c-ints converted from the Python integer and so why 
>> should  a[0] be a Python integer).
>
>
> That would basically mean to change the status of arrays from  
> generalized sequence objects to something different. Why not, but 
> then  this should be made clear to the programmer, in particular by 
> having a  different printed representation for rank-0 arrays and 
> scalars. It also  means adding some conversion functions, e.g. for 
> extracting a Python  Object from a rank-0 Python Object array.
>
> Still, I am worried about two aspects:
>
> 1) The amount of confusion this generates among users. The 
> distinction  between scalars and rank-0 arrays has no utility for the 
> programmer, it  exists only for implementation and political reasons. 
> I am not looking  forward to explaining this in my Python courses for 
> beginners.

(+1)
If we consider an array as a sequence of objects  of a fixed type, 
numeric or other, it makes sense that when a single object is returned 
then an object of that type be returned, coverted if necessary for Int8 
etc.  Returning a zero rank array is an historical pain.

It might make sense if all traditional Python objects were of zero rank  
I can see no merit in that.

>
> 2) Compatibility with existing code. I am not sure I will convert my  
> code to such conventions any time soon, because it requires 
> inspecting  every single indexing operation in its particular context 
> to see if the  index could be a rank-0 integer array. There is no way 
> to spot those  cases by textual analysis. So this change could reduce 
> acceptance to  the point where there is no interest in pursuing the 
> project any more.
>
I thought that the intent of Numeric 3 was to produce the best - a new 
start, without being overly concerned about compatibility.

I was glad to see the proposal to abandon "ravel" ( a hangover from 
APL?).  Words should have a clear generally accepted meaning.  For 
"ravel" dictionary.com offers:

       1. To separate the fibers or threads of (cloth, for example);
          unravel.
       2. To clarify by separating the aspects of.
       3. To tangle or complicate.

Colin W.




From rkern at ucsd.edu  Sat Feb 19 06:29:25 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Sat Feb 19 06:29:25 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <01b322c0be695e314111300a363aaae6@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net>
Message-ID: <42174D07.7070806@ucsd.edu>

konrad.hinsen at laposte.net wrote:
> On 19.02.2005, at 05:37, Robert Kern wrote:
> 
>> I don't see the problem that this approach would solve. It doesn't  
>> solve the list/tuple indexing problem by itself. Even if the types 
>> are  part of the standard library, they won't be bona-fide ints, so 
>> the  indexing code would still have to be modified to check for them.
> 
> 
> Yes, but it could check for "integer type" (using the type hierarchy)  
> rather than convert everything to an integer with the problem that  
> Guido pointed out.

Except that these types probably can't be derived from the builtin int. 
The C layouts would have to be compatible. They'd probably have to be a 
separate hierarchy.

At that, rank-0 arrays would have to become a special case because their 
value will have to be reflected by x->ob_ival. And how that happens is 
going to depend on their actual C type. We'll be inheriting methods that 
we can't use, and general arrays, even if the C types are compatible, 
can't be used in place of a bona fide PyIntObject.

I would prefer a single type of array object that can store different 
kinds of values.

> However, my original motivation was consistency of usage. Python has  
> type objects to specify types, so I'd rather use them than introduce  
> another way to specify the type of array elements.

True. However, if we introduce a bona fide TypeObject hierarchy for 
numerical scalars that *can* be profitably used outside of the array 
context, it's *going* to be used outside of the array context. If it 
gets into the standard library, it can't just be a large number 
hierarchy for our use; it will have to be *the* number hierarchy for 
Python and include PyLongObjects and decimals and rationals.

And that's probably a bit more than we care to deal with to get 
multiarrays into the core.

On the other hand, the list/tuple indexing issue won't go away until the 
PEP is accepted and integrated into the core. And it won't be accepted 
until Numeric3 has had some life of it's own outside the standard library.

Bugger.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From cjw at sympatico.ca  Sat Feb 19 07:15:23 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Sat Feb 19 07:15:23 2005
Subject: [Numpy-discussion] Numeric3 PEP
Message-ID: <421757E9.10206@sympatico.ca>

The PEP has:

     In Python there will be a hierchial set of classes defined:

      GenericArray (type classes)
        BooleanArray  (Bool)

        NumericArray  (XX is the bit-width of the type)
          SignedArray
          UnsignedArray
	  IntegerArray
            SignedIntegerArray   (IntXX)
	    UnsignedIntegerArray (UIntXX)
	  FloatArray    (FloatXX)
	  ComplexArray  (FloatXX)

	FlexibleArray
          CharacterArray
            StringArray  (String)
            UnicodeArray (Unicode)
          VoidArray (Void)

        ObjectArray
  

It seems that the record type is to be handled under the VoidArray.  I 
hope that it will permit the setting and reading of fields.

For example, if recs is an instance of an Array of records, then it 
should be possible to write:

     >>> recs[22, 5].sex= 'F'
     >>> recs[22, 5].sex
    F

It is not clear, from the current draft, what is intended for 
ObjectArray.  Can these be any Python objects tuple, list etc. or 
instances of any user defined class?

There is also a tofile method ( I would prefer toFile), does this mean 
that pickling would be used for the object instances?

The term WRITABLE is used, is this different from "mutable", the term 
used by Python?

"Methods such as x.r(), x.i(), and x.flatten() are proposed.".  Why not 
use properties, x.r, x.i and x.flatten.  The parentheses appear redundant.

Colin W.

  




From tkorvola at welho.com  Sat Feb 19 08:57:16 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Sat Feb 19 08:57:16 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <421742C2.1010601@sympatico.ca> (Colin J. Williams's message of
 "Sat, 19 Feb 2005 08:44:34 -0500")
References: <4214E7FE.9090403@ee.byu.edu>
	<4bf0e99f809d592b068270a200b6deb0@laposte.net>
	<42166FF3.9050908@ee.byu.edu>
	<18bd99d55abc05ceadd854a1a66d75ec@laposte.net>
	<421742C2.1010601@sympatico.ca>
Message-ID: <87vf8olc92.fsf@welho.com>

"Colin J. Williams" <cjw at sympatico.ca> writes:
> Returning a zero rank array is an historical pain.

The historical pain is returning a scalar: that is what both Numeric
and Numarray currently do.  Returning a zero rank array would be a new
pain to replace that.

> It might make sense if all traditional Python objects were of zero
> rank  I can see no merit in that.

Pushing arrays that deep into the core language would be natural for a
language inteded for numerical linear algebra but perhaps not for a
general purpose language which people also use for web services and
whatnot.

> I was glad to see the proposal to abandon "ravel" ( a hangover from
> APL?).

I thought all APL builtins were denoted by weird special characters
rather than any readable or pronouncable names.  But I don't see ravel
actually being abandoned, as the PEP does not discuss functions much.
One reason for preferring functions to methods and attributes is that
functions can be made to work with scalars and generic sequences.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From cjw at sympatico.ca  Sat Feb 19 11:43:23 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Sat Feb 19 11:43:23 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <87vf8olc92.fsf@welho.com>
References: <4214E7FE.9090403@ee.byu.edu>	<4bf0e99f809d592b068270a200b6deb0@laposte.net>	<42166FF3.9050908@ee.byu.edu>	<18bd99d55abc05ceadd854a1a66d75ec@laposte.net>	<421742C2.1010601@sympatico.ca> <87vf8olc92.fsf@welho.com>
Message-ID: <421796C2.4080808@sympatico.ca>

Timo Korvola wrote:

>"Colin J. Williams" <cjw at sympatico.ca> writes:
>  
>
> [snip]
>
>  
>
>>I was glad to see the proposal to abandon "ravel" ( a hangover from
>>APL?).
>>    
>>
 From 05-02-18 16:24 PM

    Yeah, I don't like this anymore either.

    I like X.flatten() better than X.ravel() too.
    -Travis

I suggest that X.flatten would be even better.

>
>I thought all APL builtins were denoted by weird special characters
>rather than any readable or pronouncable names.  But I don't see ravel
>actually being abandoned, as the PEP does not discuss functions much.
>One reason for preferring functions to methods and attributes is that
>functions can be made to work with scalars and generic sequences.
>
Yes, most of these weird special operators had names, one of these was 
ravel.  Incidentally, 'shape' is also probably inherited from Iverson's APL.

Colin W.



From sdhyok at gmail.com  Sat Feb 19 12:00:10 2005
From: sdhyok at gmail.com (Daehyok Shin)
Date: Sat Feb 19 12:00:10 2005
Subject: [Numpy-discussion] How to subclass NumArray?
Message-ID: <371840ef0502191159306b8a58@mail.gmail.com>

How can I create a simple subclass inheriting everything from NumArray?

-- 
Daehyok Shin (Peter)
Geography Department
University of North Carolina-Chapel Hill
USA



From oliphant at ee.byu.edu  Sat Feb 19 12:54:22 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb 19 12:54:22 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <421757E9.10206@sympatico.ca>
References: <421757E9.10206@sympatico.ca>
Message-ID: <4217A73D.8040900@ee.byu.edu>

>
> It seems that the record type is to be handled under the VoidArray.  I 
> hope that it will permit the setting and reading of fields.

Exactly.  A Python record class can make use of the basic array structure. 

>
> For example, if recs is an instance of an Array of records, then it 
> should be possible to write:
>
>     >>> recs[22, 5].sex= 'F'
>     >>> recs[22, 5].sex
>    F

This will need to be handled by the record class specifically (it will 
make use of a void array). I do not see a need to clutter the general 
array c-type with this.   

Along that idea, the current design is based heavily on Numeric which 
places all arraytypes under a single ArrayType.    It "knows" about the 
type of each array instead of asking the type what it can do. I know 
numarray made strides in a different direction (but I'm not sure how far 
it got, since there is still this basic set of types). 

I am very willing to consider the possibility of a more top-down design 
in which a basic array c-type had no information about what was in the 
array (it just managed its shape, its memory usage, and the size of the 
memory being used).   Then, subtypes could be created which handled 
type-specific needs. 

This seems to be a more logical direction to pursue, but it is a bit of 
a switch and so carries even more risk.   Someone from numarray could 
help here, perhaps.

>
> It is not clear, from the current draft, what is intended for 
> ObjectArray.  Can these be any Python objects tuple, list etc. or 
> instances of any user defined class?

It is the familiar "O" type from Numeric.  Yes, these objects can be any 
Python object whatever.  Numeric already handles this (and numarray 
recently added them).

>
> There is also a tofile method ( I would prefer toFile), does this mean 
> that pickling would be used for the object instances?

The naming convention is lowercase (camelCase is reserved for 
ClassInstances).   I have not thought that far, but probably...

>
> The term WRITABLE is used, is this different from "mutable", the term 
> used by Python?

MUTABLE is a better term.   Do numarray folks agree?

>
> "Methods such as x.r(), x.i(), and x.flatten() are proposed.".  Why 
> not use properties, x.r, x.i and x.flatten.  The parentheses appear 
> redundant.

The distinction is that attributes do not return copies, but methods 
might.  Thus, the idea is that x.i() would return zeros if the array 
were not complex, but x.imag would raise and error.   I like the idea of 
attributes being intrinsic properties of an array, while methods could 
conceivably return or do anything.

Thanks for your continued help with clarification and improvement to the 
PEP.

-Travis




From oliphant at ee.byu.edu  Sat Feb 19 13:09:24 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb 19 13:09:24 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <18bd99d55abc05ceadd854a1a66d75ec@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42166FF3.9050908@ee.byu.edu> <18bd99d55abc05ceadd854a1a66d75ec@laposte.net>
Message-ID: <4217AACC.6040804@ee.byu.edu>

> I can agree with that. Basically the problem is the type hierarchy 
> that  is too simple (but many if not most programming language suffer 
> from  that problem). Type conversions are all handled in the same way, 
> which  doesn't give enough flexibility. But we won't change that of 
> course.

Guido might in Python 3000....

>
> That would basically mean to change the status of arrays from  
> generalized sequence objects to something different. Why not, but 
> then  this should be made clear to the programmer, in particular by 
> having a  different printed representation for rank-0 arrays and 
> scalars. It also  means adding some conversion functions, e.g. for 
> extracting a Python  Object from a rank-0 Python Object array.

I guess I don't consider arrays as generalized sequence objects.  Except 
for the Python Object array which might be.     Still, my views on this 
are really not central as I want to help get something together that a 
large majority of us can be satisfied with.  I'm really not interested 
in intentionally making something that is difficult to use and I am 
willing to bend on many things (especially if work-arounds are possible). 

It seems to me that the rank-0 array scalar problem is fundamental to 
the situation of trying to craft a Numerical-environment on top of a 
general-purpose one.    I don't see a "right solution" that would please 
all parties short of altering all Python scalars --- and that would 
likely upset an entirely different crowd.  I suppose that is why we have 
the compromise we do now. 

> Still, I am worried about two aspects:
>
> 1) The amount of confusion this generates among users. The 
> distinction  between scalars and rank-0 arrays has no utility for the 
> programmer, it  exists only for implementation and political reasons. 
> I am not looking  forward to explaining this in my Python courses for 
> beginners.

The only utility might be speed because it is not too hard to see that 
rank-0 arrays that behave as part of a generic system of 
multidimensional arrays might carry quite a bit of baggage that is not 
needed if all you are every want is a scalar.    This distinction may 
not be possible to get rid of. 

To put it more clearly:  Is it possible to define a scalar that 
interacts seamlessly with a system of multidimensional arrays without 
slowing slowing it down for usage in other contexts?  

I don't know the answer, but it sure seems to be no.  I've looked at 
Piddle (PERL's equivalent to Numeric)  and they seem to do exactly the 
same thing (have a rank-0 array that is very different from the Perl 
scalar). 


Added to our problem is that we do not have much control over the 
definition of fundamental scalars in Python.   Guido has suggested that 
he may be willing to allow integers to get methods (how many methods I'm 
not sure --- I didn't push him.  He mentioned the possibility of adding 
a rank-inquiry method for example).   It would pleasantly surprise me if 
he were willing to give scalars all of the methods and attributes of 
arrays.

>
> 2) Compatibility with existing code. I am not sure I will convert my  
> code to such conventions any time soon, because it requires 
> inspecting  every single indexing operation in its particular context 
> to see if the  index could be a rank-0 integer array. There is no way 
> to spot those  cases by textual analysis. So this change could reduce 
> acceptance to  the point where there is no interest in pursuing the 
> project any more.

A healthy use of int()  in indexing operations could fix this, but yes, 
I see compatibility as an issue.  I don't want to create 
incompatibilities if we can avoid it.    On the other hand, I don't want 
to continue with a serious design flaw just for the sake of 
incompatibility either (I'm still trying to figure out if it is a flaw 
or not, it sure seems like a hack).

Thanks for your continued help and advice.

-Travis





From oliphant at ee.byu.edu  Sat Feb 19 13:20:26 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb 19 13:20:26 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42174D07.7070806@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu>
Message-ID: <4217AD44.1090302@ee.byu.edu>

>>
>> Yes, but it could check for "integer type" (using the type 
>> hierarchy)  rather than convert everything to an integer with the 
>> problem that  Guido pointed out.
>
>
> Except that these types probably can't be derived from the builtin 
> int. The C layouts would have to be compatible. They'd probably have 
> to be a separate hierarchy.

On the python-dev list someone (Bob Ippolito) suggested inheriting 
rank-0 arrays from the Python Int Type.  I don't see how this can be 
even be done for all of the integer types (unless the Python Int Type is 
changed to hold the largest possible integer (long long)).   

>
> At that, rank-0 arrays would have to become a special case because 
> their value will have to be reflected by x->ob_ival. And how that 
> happens is going to depend on their actual C type. We'll be inheriting 
> methods that we can't use, and general arrays, even if the C types are 
> compatible, can't be used in place of a bona fide PyIntObject.
>
> I would prefer a single type of array object that can store different 
> kinds of values.

I see the same problems that Robert is talking about.  Do we really want 
to special-case all array code to handle rank-0 arrays differently?   
That seems to be opening up a very big can of worms.   Is the only way 
to solve this problem to handle rank-0 arrays in a separate hierarchy?  
I have doubts that such a system would even work.

>
> On the other hand, the list/tuple indexing issue won't go away until 
> the PEP is accepted and integrated into the core. And it won't be 
> accepted until Numeric3 has had some life of it's own outside the 
> standard library.


I agree with Robert's assesment --- bugger.  I'm really annoyed that 
such a relatively simple think like rank-0 arrays versus Python's 
already-defined scalars could be such a potential show-stopper.

Hoping-it-won't-be

-Travis





From oliphant at ee.byu.edu  Sat Feb 19 13:46:15 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb 19 13:46:15 2005
Subject: [Numpy-discussion] Poll on a Rank-0 arrays: please give +1, -1, or 0
Message-ID: <4217B374.2060702@ee.byu.edu>

We need to get some opinions regarding the recent discussion on rank-0 
arrays.   Please give your  vote +1 (like), -1 (hate), or 0 (don't 
care)  for **each** of the following possibilities.  Feel free to 
clarify your vote if you need to.


1)  Rank-0 arrays are always returned from uarray operations, Python 
scalars available on request only.   (*If* arrayobject get's into Python 
core, Guido has agreed to let rank-0 integer arrays be used as
index variables).  Rank-0 arrays are made to resmbe scalar of



2) Rank-0 arrays are never returned from uarray operations (unless using 
asarray on a scalar), and when a rank-0 array naturally appears in the 
calculation, an appropriate Python scalar is returned (note that this 
would lose precision for long doubles unless a new Python object was 
created).



3) The current compromise is continued and for some types a Python 
scalar is returned, while for other types a  rank-0 array is returned



4) Rank-0 arrays are made equivalent to Python scalars and a new Python 
scalar for each fundamental data type supported by uarray is constructed 
(rank-0 arrays would still probably be used internally, but users would 
not have to know this).   The new Python-scalars would inherit from an 
existing Python scalar where appropriate and would have the same 
attributes and methods of uarrays (very likely at least initially they 
would be seemlessly converted to rank-0 arrays when "mixed" operations 
occur).









From jmiller at stsci.edu  Sat Feb 19 13:58:18 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Sat Feb 19 13:58:18 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <4217A73D.8040900@ee.byu.edu>
References: <421757E9.10206@sympatico.ca>  <4217A73D.8040900@ee.byu.edu>
Message-ID: <1108850232.3450.1.camel@jaytmiller.comcast.net>

On Sat, 2005-02-19 at 13:53 -0700, Travis Oliphant wrote:
> >
> > The term WRITABLE is used, is this different from "mutable", the term 
> > used by Python?
> 
> MUTABLE is a better term.   Do numarray folks agree?
> 

Well, no.  I've never heard of page of "mutable" memory.  It's no biggie
though.

Regards,
Todd




From tim.hochberg at cox.net  Sat Feb 19 14:53:23 2005
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Sat Feb 19 14:53:23 2005
Subject: [Numpy-discussion] Poll on a Rank-0 arrays: please give +1, -1,
 or 0
In-Reply-To: <4217B374.2060702@ee.byu.edu>
References: <4217B374.2060702@ee.byu.edu>
Message-ID: <4217C335.9060002@cox.net>

Travis Oliphant wrote:

>
> We need to get some opinions regarding the recent discussion on rank-0 
> arrays.   Please give your  vote +1 (like), -1 (hate), or 0 (don't 
> care)  for **each** of the following possibilities.  Feel free to 
> clarify your vote if you need to.
>
>
> 1)  Rank-0 arrays are always returned from uarray operations, Python 
> scalars available on request only.   (*If* arrayobject get's into 
> Python core, Guido has agreed to let rank-0 integer arrays be used as
> index variables).  Rank-0 arrays are made to resmbe scalar of

[Something appears to be scrambled here, but I think I get the gist of it]

0.

This seems most self consistent option. It will, however, break some 
unknown amount of code since I'm sure some stuff relys on Python int / 
float behaviour that will be somewhat different than rank-0 array 
behaviour. One example is multiplying two rank-0 integers  won't 
overflow into longs as would two python integers. I think it will also 
feel odd: people expect scalars when they index a rank one array.

Even though Guido agreed to let rank-0 arrays work as index variables, 
the index issue will still be problematic. Even assuming that 
arrayobject makes it into the core, it won't, and shouldn't happen, till 
we have some experience as a community using and polishing it. In the 
meantime, indexing won't work, which will make it harder to use. Meaning 
a smaller community and less likelihood that it goes into the core. It's 
chicken and an egg problem.

>
>
>
> 2) Rank-0 arrays are never returned from uarray operations (unless 
> using asarray on a scalar), and when a rank-0 array naturally appears 
> in the calculation, an appropriate Python scalar is returned (note 
> that this would lose precision for long doubles unless a new Python 
> object was created).

-1 unless a new object is created. -0 in that case.

>
>
>
> 3) The current compromise is continued and for some types a Python 
> scalar is returned, while for other types a  rank-0 array is returned

0. This is a bit of a mess, but has the advantage of backwards 
compatibility. Also we have a fair amount of experience with it and know 
that it works at least fairly well.

>
> 4) Rank-0 arrays are made equivalent to Python scalars and a new 
> Python scalar for each fundamental data type supported by uarray is 
> constructed (rank-0 arrays would still probably be used internally, 
> but users would not have to know this).   The new Python-scalars would 
> inherit from an existing Python scalar where appropriate and would 
> have the same attributes and methods of uarrays (very likely at least 
> initially they would be seemlessly converted to rank-0 arrays when 
> "mixed" operations occur).

+1 assuming the details can be worked out. I think this will feel more 
natural in acutal use than the first option. It also should avoid the 
indexing problem that hamper that option.

-tim


>
>
>
>
>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>





From konrad.hinsen at laposte.net  Sat Feb 19 14:55:32 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 14:55:32 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <421742C2.1010601@sympatico.ca>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42166FF3.9050908@ee.byu.edu> <18bd99d55abc05ceadd854a1a66d75ec@laposte.net> <421742C2.1010601@sympatico.ca>
Message-ID: <6dc26b212e60c7ab35735a669ba6101b@laposte.net>

On 19.02.2005, at 14:44, Colin J. Williams wrote:

>> 2) Compatibility with existing code. I am not sure I will convert my   
>> code to such conventions any time soon, because it requires  
>> inspecting  every single indexing operation in its particular context  
>> to see if the  index could be a rank-0 integer array. There is no way  
>> to spot those  cases by textual analysis. So this change could reduce  
>> acceptance to  the point where there is no interest in pursuing the  
>> project any more.
>>
> I thought that the intent of Numeric 3 was to produce the best - a new  
> start, without being overly concerned about compatibility.

It all depends on where "overly" starts.

Let's be pragmatic: the majority of potential Numeric 3 users will be  
the current users of Numeric and numarray. If they don't accept Numeric  
3 because it's a pain, no amount of nice design will help.

> I was glad to see the proposal to abandon "ravel" ( a hangover from  
> APL?).

Fine, but that's the kind of change that doesn't hurt much: a name  
change can be made with a text editor. Changing the behaviour of  
fundamental operations (indexing) is a different thing.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Sat Feb 19 14:57:23 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 14:57:23 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42174D07.7070806@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu>
Message-ID: <292015fbc60118037b345c59d0c10355@laposte.net>

On 19.02.2005, at 15:28, Robert Kern wrote:

> Except that these types probably can't be derived from the builtin  
> int. The C layouts would have to be compatible. They'd probably have  
> to be a separate hierarchy.

They could all derive from a common (yet-to-be-written) base class that  
has no data layout at all.

> True. However, if we introduce a bona fide TypeObject hierarchy for  
> numerical scalars that *can* be profitably used outside of the array  
> context, it's *going* to be used outside of the array context. If it  
> gets into the

True, though I expect its use to be limited to the numeric community.

>  standard library, it can't just be a large number hierarchy for our  
> use; it will have to be *the* number hierarchy for Python and include  
> PyLongObjects and decimals and rationals.

That would be preferable indeed.

> And that's probably a bit more than we care to deal with to get  
> multiarrays into the core.

It all depends on the reaction of the Python developer community. We  
won't know before asking.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Sat Feb 19 15:03:09 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 15:03:09 2005
Subject: [Numpy-discussion] Poll on a Rank-0 arrays: please give +1, -1, or 0
In-Reply-To: <4217B374.2060702@ee.byu.edu>
References: <4217B374.2060702@ee.byu.edu>
Message-ID: <d59a6837a68b8a99b10d8f9fdc780e9f@laposte.net>

On 19.02.2005, at 22:45, Travis Oliphant wrote:

> 1)  Rank-0 arrays are always returned from uarray operations, Python 
> scalars available on request only.   (*If* arrayobject get's into 
> Python core, Guido has agreed to let rank-0 integer arrays be used as
> index variables).  Rank-0 arrays are made to resmbe scalar of

-1

I consider this a kludge that might be acceptable if no other solution 
is politically doable, but not a desirable feature.


> 2) Rank-0 arrays are never returned from uarray operations (unless 
> using asarray on a scalar), and when a rank-0 array naturally appears 
> in the calculation, an appropriate Python scalar is returned (note 
> that this would lose precision for long doubles unless a new Python 
> object was created).

+1

provided that
1) a special long double type is created
2) the type promotion rules handle scalars in some reasonable way

> 3) The current compromise is continued and for some types a Python 
> scalar is returned, while for other types a  rank-0 array is returned

+1

It works well enough in practice.

> 4) Rank-0 arrays are made equivalent to Python scalars and a new 
> Python scalar for each fundamental data type supported by uarray is 
> constructed (rank-0 arrays would still probably be used internally, 
> but users would not have to know this).   The new Python-scalars would 
> inherit from an existing Python scalar where appropriate and would 
> have the same attributes and methods of uarrays (very likely at least 
> initially they would be seemlessly converted to rank-0 arrays when 
> "mixed" operations occur).

+1

This is still my first choice.

Konrad.




From tim.hochberg at cox.net  Sat Feb 19 15:10:14 2005
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Sat Feb 19 15:10:14 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <4217A73D.8040900@ee.byu.edu>
References: <421757E9.10206@sympatico.ca> <4217A73D.8040900@ee.byu.edu>
Message-ID: <4217C71E.8050002@cox.net>

Hi Travis,

First off, let me say that I'm encouraged to see some action towards 
unifying Numeric/Numarray the split has been somewhat dismaying. Thank 
you for your efforts in this regard.

I'd like to lobby against flatten(), r() and i(). To a large extent, 
these duplicate the functionality of flat, real and imag. And, these 
three methods are defined to sometimes return copies and sometimes 
return views. That type of interface is a recipe for errors and should 
only be used as a last resort. Fortunately in this case there are better 
alternatives.
 
Flatten() is not necessary now that flat will be faux array with a view 
to the original [I believe you are calling it an indexable iterator].  I 
would, however,  recomend that A.flat.copy() work. In that case, A.flat 
would be used when no copy was desired, and A.flat.copy() when a copy 
was desired.  I don't find the copy when discontiguous case useful 
enough to deserve it's own function and it's error prone as I'll discuss 
more below. 

r()  appears to be around just for symmetry with i() since A.r() will 
always be the same as A.real. That leaves i(). My opinion is that this 
case would be better served by returning a read-only array of zeros when 
operating on a real array. This array could even be a faux-array that 
doesn't allocate any space, although that may be a project for another day.

If it's really deemed necessary to have these functions in addition to 
their attribute brethren, I recomend that they always return copies 
rather than varying their behaviour depending on the situation. The 
problem with methods that sometimes return a copy, is that it won't be 
long before someone types:

    def foobar(a)
       flat_view = a.flatten()
       # lots of code
       flat_view[some_index] = some_new_number

This works until someone passes in a discontiguous array, at which point 
it fails mysteriously. This type of problem tends to be somewhat 
resistant to unit tests, since tests often involve only contiguous arrays.

Regards,

-tim




From rkern at ucsd.edu  Sat Feb 19 15:56:16 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Sat Feb 19 15:56:16 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <292015fbc60118037b345c59d0c10355@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu> <292015fbc60118037b345c59d0c10355@laposte.net>
Message-ID: <4217D1E8.2070602@ucsd.edu>

konrad.hinsen at laposte.net wrote:
> On 19.02.2005, at 15:28, Robert Kern wrote:
> 
>> Except that these types probably can't be derived from the builtin  
>> int. The C layouts would have to be compatible. They'd probably have  
>> to be a separate hierarchy.
> 
> 
> They could all derive from a common (yet-to-be-written) base class that  
> has no data layout at all.

We then end up with the same chicken-egg problem as accepting rank-0 
integer arrays as indices. It won't work until it's in the core. If I'm 
understanding your proposition correctly, it also creates another 
problem: rank-n arrays would then pass this check, although they shouldn't.

>> True. However, if we introduce a bona fide TypeObject hierarchy for  
>> numerical scalars that *can* be profitably used outside of the array  
>> context, it's *going* to be used outside of the array context. If it  
>> gets into the
> 
> 
> True, though I expect its use to be limited to the numeric community.

I expect so, too. However, when considering additions to the standard 
library, python-dev has to assume otherwise. If it's going to be so 
limited in application, then something so large shouldn't be in the 
standard library.

>>  standard library, it can't just be a large number hierarchy for our  
>> use; it will have to be *the* number hierarchy for Python and include  
>> PyLongObjects and decimals and rationals.
> 
> 
> That would be preferable indeed.
> 
>> And that's probably a bit more than we care to deal with to get  
>> multiarrays into the core.
> 
> 
> It all depends on the reaction of the Python developer community. We  
> won't know before asking.

I think it would be great to have a more thorough number hierarchy in 
the standard library. So would some others. See PEPs 228 and 242. 
However, I think that the issue is orthogonal getting an multiarray 
object into the standard library. I'm not convinced that it actually 
solves the problems with getting multiarrays into the core. Now, we may 
have different priorities, so we have different thresholds of 
"problem-ness."

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From cjw at sympatico.ca  Sat Feb 19 16:40:10 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Sat Feb 19 16:40:10 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4217D1E8.2070602@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu> <292015fbc60118037b345c59d0c10355@laposte.net> <4217D1E8.2070602@ucsd.edu>
Message-ID: <4217DC34.8000105@sympatico.ca>

Robert Kern wrote:

> konrad.hinsen at laposte.net wrote:
>
>>
>>
>> It all depends on the reaction of the Python developer community. We  
>> won't know before asking.
>
>
> I think it would be great to have a more thorough number hierarchy in 
> the standard library. So would some others. See PEPs 228 and 242. 
> However, I think that the issue is orthogonal getting an multiarray 
> object into the standard library. I'm not convinced that it actually 
> solves the problems with getting multiarrays into the core. Now, we 
> may have different priorities, so we have different thresholds of 
> "problem-ness."
>
PEP 228 is under consideration (since 2000):


          Numerical Python Issues

    People who use Numerical Python do so for high-performance vector
    operations.  Therefore, NumPy should keep its hardware based
    numeric model.

*Unresolved Issues*

    Which number literals will be exact, and which inexact?

    How do we deal with IEEE 754 operations? (probably, isnan/isinf should
    be methods)

    On 64-bit machines, comparisons between ints and floats may be
    broken when the comparison involves conversion to float.  Ditto
    for comparisons between longs and floats.  This can be dealt with
    by avoiding the conversion to float.  (Due to Andrew Koenig.)


  

For PEP 242 the status is:

    This PEP has been closed by the author.  The kinds module will not
    be added to the standard library.

    There was no opposition to the proposal but only mild interest in
    using it, not enough to justify adding the module to the standard
    library.  Instead, it will be made available as a separate
    distribution item at the Numerical Python site.  At the next
    release of Numerical Python, it will no longer be a part of the
    Numeric distribution.
  

It seems to be up to the numerical folk to make proposals.

Colin W.

  




From rkern at ucsd.edu  Sat Feb 19 17:03:26 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Sat Feb 19 17:03:26 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4217DC34.8000105@sympatico.ca>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu> <292015fbc60118037b345c59d0c10355@laposte.net> <4217D1E8.2070602@ucsd.edu> <4217DC34.8000105@sympatico.ca>
Message-ID: <4217E1A9.6010202@ucsd.edu>

Colin J. Williams wrote:
> Robert Kern wrote:
> 
>> konrad.hinsen at laposte.net wrote:
>>
>>>
>>>
>>> It all depends on the reaction of the Python developer community. We  
>>> won't know before asking.
>>
>>
>>
>> I think it would be great to have a more thorough number hierarchy in 
>> the standard library. So would some others. See PEPs 228 and 242. 
>> However, I think that the issue is orthogonal getting an multiarray 
>> object into the standard library. I'm not convinced that it actually 
>> solves the problems with getting multiarrays into the core. Now, we 
>> may have different priorities, so we have different thresholds of 
>> "problem-ness."
>>
> PEP 228 is under consideration (since 2000):
> 
> 
>          Numerical Python Issues
> 
>    People who use Numerical Python do so for high-performance vector
>    operations.  Therefore, NumPy should keep its hardware based
>    numeric model.

Note that the recommendation is that Numeric ignore PEP's number model. 
That PEP points *away* from things like Int32 and Float64.

[snip]

> For PEP 242 the status is:
> 
>    This PEP has been closed by the author.  The kinds module will not
>    be added to the standard library.
> 
>    There was no opposition to the proposal but only mild interest in
>    using it, not enough to justify adding the module to the standard
>    library.  Instead, it will be made available as a separate
>    distribution item at the Numerical Python site.  At the next
>    release of Numerical Python, it will no longer be a part of the
>    Numeric distribution.
>  
> 
> It seems to be up to the numerical folk to make proposals.

Note also that PEP 242 was retracted before people got really interested 
    (by which I mean "interested enough to implement") in other number 
types like decimal and rationals. While historically these proposal have 
come from the NumPy community (which I'm distinguishing from "numerical 
folk"), in the future they will need to intimately involve a much larger 
group of people.

Of course, the NumPy community is a subset of "numerical folk," so we 
are naturally interested in how numbers are represented in Python. I'm 
not saying we shouldn't be or that such a proposal shouldn't come from 
this community. In general, such a thing would be of great use to this 
community. However, I don't see how it would help, in specific, the 
addition of multiarray objects to the standard library, nor do I think 
that such should wait upon the acceptance and implementation of such a 
proposal.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From oliphant at ee.byu.edu  Sat Feb 19 19:52:20 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb 19 19:52:20 2005
Subject: [Numpy-discussion] PEP updated
Message-ID: <4218091A.4010404@ee.byu.edu>

I've updated the PEP to conform to what I think is the best hybrid 
solution propsed and that is to implement a tree of PythonTypes in C 
whose leaves are (to the Python user) new rank-0 arrays.   This is
more work to implement (but I don't think a great deal of work see 
below), and I think it will lead to the best results under our current 
constraints.

Internally, the PyArray_Type will still deal with rank-0 arrays (in fact 
the new Python scalars will be converted to them internally rather than 
special-case a bunch of code).

I don't think we should try to solve the general number-hierchy in 
Python at this time.  We will implement the array-number hierarchy 
only.    If this leads to others getting interested in the general 
problem, then that may come, but now we have to focus on our objective 
of getting the arrayobject accepted into Python.

Note that these new rank-0 Python scalars will be simple, have the same 
methods and attributes as Python arrays (likely-implemented once in the 
GenericArrType by converting the object to a rank-0 array).
Also, they will inherit (multiple-inheritance is possible in C) from the 
appropriate Python Type where an exact-match is available.

-Travis







From konrad.hinsen at laposte.net  Sun Feb 20 01:00:26 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sun Feb 20 01:00:26 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4217D1E8.2070602@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu> <292015fbc60118037b345c59d0c10355@laposte.net> <4217D1E8.2070602@ucsd.edu>
Message-ID: <b78086cf38e1648a7fc0055e405a6f7c@laposte.net>

On 20.02.2005, at 00:55, Robert Kern wrote:

>> They could all derive from a common (yet-to-be-written) base class  
>> that  has no data layout at all.
>
> We then end up with the same chicken-egg problem as accepting rank-0  
> integer arrays as indices. It won't work until it's in the core. If  
> I'm

True. We could have a patch to Python for testing purposes before such  
a decision, but this is not an ideal situation.

> understanding your proposition correctly, it also creates another  
> problem: rank-n arrays would then pass this check, although they  
> shouldn't.

No. My proposal is to have new Python scalar types, which are distinct  
from the array type(s). The new scalar types would use rank-0 arrays as  
their internal representation, but that would be an implementation  
detail not visible to users.

> I expect so, too. However, when considering additions to the standard  
> library, python-dev has to assume otherwise. If it's going to be so  
> limited in application, then something so large shouldn't be in the  
> standard library.

I am not so sure about this. There is some very domain-specific stuff  
in the standard library.

> I think it would be great to have a more thorough number hierarchy in  
> the standard library. So would some others. See PEPs 228 and 242.  
> However, I think that the issue is orthogonal getting an multiarray  
> object into the standard library. I'm not convinced that it actually  
> solves the problems with

The common point is just the community that is interested in this.  
However, there might be a wider interest in a re-working of the number  
hierarchy. There have been additions to the number hierarchy that don't  
come from the numerics community, such as the decimal number type. I  
think that a more open number hierarchy, into which modules could add  
their own types, could make it into the core if it doesn't cause any  
compatibility problems. In fact, this may be easier to argue for than  
having array methods on all scalar types.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From rkern at ucsd.edu  Sun Feb 20 01:52:09 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Sun Feb 20 01:52:09 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <b78086cf38e1648a7fc0055e405a6f7c@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu> <292015fbc60118037b345c59d0c10355@laposte.net> <4217D1E8.2070602@ucsd.edu> <b78086cf38e1648a7fc0055e405a6f7c@laposte.net>
Message-ID: <42185DA7.3040507@ucsd.edu>

konrad.hinsen at laposte.net wrote:
> On 20.02.2005, at 00:55, Robert Kern wrote:

>> understanding your proposition correctly, it also creates another  
>> problem: rank-n arrays would then pass this check, although they  
>> shouldn't.
> 
> 
> No. My proposal is to have new Python scalar types, which are distinct  
> from the array type(s). The new scalar types would use rank-0 arrays as  
> their internal representation, but that would be an implementation  
> detail not visible to users.

*Ahh!* Enlightenment dawns. That makes a *hell* of a lot more sense than 
what I thought you were proposing. Thank you for bearing with my babbling.

I withdraw my objections.

I do have one comment, though. I still prefer the idea that arrays, 
including rank-0 arrays, be containers. So I would suggest that there 
only be a handful of new rank-0 types, int-like, float-like, 
complex-like, object, and maybe a couple more. The differences between 
the type objects would be solely for inheritance reasons. Different 
precisions would be handled like they are for rank-n arrays.

>> I expect so, too. However, when considering additions to the standard  
>> library, python-dev has to assume otherwise. If it's going to be so  
>> limited in application, then something so large shouldn't be in the  
>> standard library.
> 
> 
> I am not so sure about this. There is some very domain-specific stuff  
> in the standard library.

I think my point was that a number hierarchy is something that *ought* 
to have much wider applicability. If the implementation *doesn't*, then 
it shouldn't be in the standard library. That's why I prefer the 
arrangement I described above. It isn't a real number hierarchy and 
doesn't purport to be one. It's just an implementation detail[1] to make 
multiarrays place nicer with the rest of the Python universe.

>> I think it would be great to have a more thorough number hierarchy in  
>> the standard library. So would some others. See PEPs 228 and 242.  
>> However, I think that the issue is orthogonal getting an multiarray  
>> object into the standard library. I'm not convinced that it actually  
>> solves the problems with
> 
> 
> The common point is just the community that is interested in this.  
> However, there might be a wider interest in a re-working of the number  
> hierarchy. There have been additions to the number hierarchy that don't  
> come from the numerics community, such as the decimal number type. I  
> think that a more open number hierarchy, into which modules could add  
> their own types, could make it into the core if it doesn't cause any  
> compatibility problems. In fact, this may be easier to argue for than  
> having array methods on all scalar types.

Agreed. I think, though, that what Travis proposes in "PEP Updated" is 
going to be easiest of all to swallow.

[1] Isn't it amazing how many design problems go away by claiming that 
"it's just an implementation detail?"  :-)

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From tkorvola at welho.com  Sun Feb 20 05:40:23 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Sun Feb 20 05:40:23 2005
Subject: [Numpy-discussion] PEP updated
In-Reply-To: <4218091A.4010404@ee.byu.edu> (Travis Oliphant's message of
 "Sat, 19 Feb 2005 20:50:50 -0700")
References: <4218091A.4010404@ee.byu.edu>
Message-ID: <87u0o7pcyp.fsf@welho.com>

Travis Oliphant <oliphant at ee.byu.edu> writes:
> I've updated the PEP to conform to what I think is the best hybrid
> solution propsed and that is to implement a tree of PythonTypes in C
> whose leaves are (to the Python user) new rank-0 arrays.

For the purpose of differentiating between basic and advanced
indexing, are rank 0 arrays considered scalars or arrays?  Does their
presence among indices trigger advanced indexing?

For sequence behaviour len, indexing and iteration should work
consistently, i.e., "a[i] for i in xrange(len(a))" and "x for x in a"
should identically generate the contents of a.  Currently indexing and
iteration occur along the first dimension, which for rank 0 arrays
does not exist.  I understand that there is no burning desire to
completely switch to Matlab-style one-dimensional indexing.  The
alternatives are then giving rank 0 arrays exceptional semantics
inconsistent with other arrays or raising an exception if len or
iteration is applied to a rank 0 array.  Without exceptional semantics
the only index expression that makes sense is a[()].  I would favor
consistency here and raise an exception but there is one matter that
raises some doubt: do programs expect that if __len__ and __iter__
exist they will not fail?

Converting scalars back to arrays: they should be rank 0.  Rank 1
would be Matlabish nonsense, except that Matlab is even more
nonsensical and uses rank 2.  But when is such conversion needed?  If
it is only for operations with other arrays then broadcasting is
likely to occur immediately.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From perry at stsci.edu  Sun Feb 20 07:58:32 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Sun Feb 20 07:58:32 2005
Subject: [Numpy-discussion] Poll on a Rank-0 arrays: please give +1, -1, or 0
In-Reply-To: <4217B374.2060702@ee.byu.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCGEDHFJAA.perry@stsci.edu>

I'm going to refrain from a numerical rating but rather add a few comments
and questions
> 1)  Rank-0 arrays are always returned from uarray operations, Python
> scalars available on request only.   (*If* arrayobject get's into Python
> core, Guido has agreed to let rank-0 integer arrays be used as
> index variables).  Rank-0 arrays are made to resmbe scalar of
>
A minor question, in this and other cases where rank-0 is being used, what
will repr show? For example, if I do:

>>> x = arange(2)
>>> x[0]

do I get '0' or 'array(0)' ?

I suppose the former is fine if rank-0 is usable in every way as a scalar
and no one really needs to know the difference, but I troubles me a bit to
hide this fact (with repr) if it isn't quite the same.

Along those lines (for this case and the last) I worry that there are
potential gotchas that haven't been discovered yet. How confident are you
that there aren't any in trying to integrate rank-0 with Python scalars and
their usage. It may be one of those things that is hard to know unless the
work of trying to implement it is actually done. You've thought about this a
lot more than I have.
>
>
> 2) Rank-0 arrays are never returned from uarray operations (unless using
> asarray on a scalar), and when a rank-0 array naturally appears in the
> calculation, an appropriate Python scalar is returned (note that this
> would lose precision for long doubles unless a new Python object was
> created).
>
As Konrad mentions, as long as there is some means of handling long doubles,
I find scalars perfectly acceptable. I tend to think that is what most user
assume they are getting.
>
>
> 3) The current compromise is continued and for some types a Python
> scalar is returned, while for other types a  rank-0 array is returned
>
I've never really liked this behavior.
>
>
> 4) Rank-0 arrays are made equivalent to Python scalars and a new Python
> scalar for each fundamental data type supported by uarray is constructed
> (rank-0 arrays would still probably be used internally, but users would
> not have to know this).   The new Python-scalars would inherit from an
> existing Python scalar where appropriate and would have the same
> attributes and methods of uarrays (very likely at least initially they
> would be seemlessly converted to rank-0 arrays when "mixed" operations
> occur).
>
Sounds good if there are no gotchas but may be a lot of work (coding and
political). Any performance issues regarding rank-0 vs scalars? Will users
pay a penalty for using rank-0 in scalar-like expressions (no apparent array
usage other than the rank-0 values)?

Perry





From jdhunter at ace.bsd.uchicago.edu  Sun Feb 20 09:50:25 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Sun Feb 20 09:50:25 2005
Subject: [Numpy-discussion] PEP updated
In-Reply-To: <4218091A.4010404@ee.byu.edu> (Travis Oliphant's message of
 "Sat, 19 Feb 2005 20:50:50 -0700")
References: <4218091A.4010404@ee.byu.edu>
Message-ID: <m37jl3uo6x.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Travis" == Travis Oliphant <oliphant at ee.byu.edu> writes:

    Travis> I've updated the PEP to conform to what I think is the
    Travis> best hybrid solution propsed and that is to implement a
    Travis> tree of PythonTypes in C whose leaves are (to the Python
    Travis> user) new rank-0 arrays.  This is more work to implement
    Travis> (but I don't think a great deal of work see below), and I
    Travis> think it will lead to the best results under our current
    Travis> constraints.

Hi Travis, 

One issue I didn't see addressed in the PEP is the handling of special
values for floating point numbers, eg inf and nan.

Will Numeric3 be able to handle something like

  >>> nan = float('nan')
  >>> inf = float('inf')
  >>> n.array([inf, nan])

or otherwise support these special values?  A common request I get in
matplotlib is to be able to handle float arrays with nan, but I've
hesitated tackling this because of a lack of consistent support for
these values in python, Numeric and numarray.  In this particular
example, Numeric 23.7 raises a traceback in the repr pipeline, and
numarray appears to work properly.

Apologies if I missed a discussion of this topic already on this list
-- searching for nan in my mail reader keeps matching Fernando, and
inf matches listinfo....

Thanks,
JDH



From cjw at sympatico.ca  Sun Feb 20 13:04:00 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Sun Feb 20 13:04:00 2005
Subject: [Numpy-discussion] Matlab is a tool for doing numerical computations with matrices and
 vectors.
Message-ID: <4218FAD8.6060804@sympatico.ca>

The subject line is from http://www.math.utah.edu/lab/ms/matlab/matlab.html

I've seen little use of the words "matrix" or "vector" in the numarray 
stuff.  These are one or two dimensional structures.

The Numeric/numarray focus is on multidimensional arrays.

Some years ago,  Huaiyu Zhu made a number of postings advocating the 
introduction of a Python-based system permitting the usual matrix 
operations.  It seems that the culmination of these efforts was Matpy 
(http://matpy.sourceforge.net/), which is not currently active.

I was impressed with the case made by Huaiyu Zhu and have attempted to 
carry on in this direction.  An outline of this work is at: 
http://www3.sympatico.ca/cjw/IntroToPyMatrix.pdf.

This work was based on numarray, partly because it was generally 
considered to be the way ahead and partly because it purported to 
provide a class structure.  numarray has proved to be a bit of a moving 
target.

I am continuing this work in the hope that some class-based structures 
will come out of the current Numeric3 discussions.  The hope is also 
that there is a need out there for matrix operations, as distinct from 
array operations.

Colin W.



From oliphant at ee.byu.edu  Sun Feb 20 14:02:26 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sun Feb 20 14:02:26 2005
Subject: [Numpy-discussion] Matlab is a tool for doing numerical computations
 with matrices and vectors.
In-Reply-To: <4218FAD8.6060804@sympatico.ca>
References: <4218FAD8.6060804@sympatico.ca>
Message-ID: <421908B0.90406@ee.byu.edu>

> The Numeric/numarray focus is on multidimensional arrays.
>
> Some years ago,  Huaiyu Zhu made a number of postings advocating the 
> introduction of a Python-based system permitting the usual matrix 
> operations.  It seems that the culmination of these 

> efforts was Matpy (http://matpy.sourceforge.net/), which is not 
> currently active.


I remember his work.   I really liked many of his suggestions, though it 
took him a while to recognize that a Matrix class has been distributed 
with Numeric from very early on.    Yes, it needed work, and a few of 
his ideas were picked up on and included in Numeric's Matrix object. 

I would like to see a standard Matrix class (along with the record-array 
class) also be included with the universal array.  

By the way, I've been calling the array object uarray in the PEP (kind 
of parallel to ufunc), but if others have better alternatives, I'm 
certainly not against them.  The problem with array is that it is taken 
by the arraymodule, which I don't want to try and replace at this point 
(I don't want to worry about backward compatibility with it...). 

I also like the name ndarray though.  I haven't heard what others think.

-Travis





From oliphant at ee.byu.edu  Sun Feb 20 14:24:11 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sun Feb 20 14:24:11 2005
Subject: [Numpy-discussion] Poll on a Rank-0 arrays: please give +1, -1,
 or 0
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCGEDHFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCGEDHFJAA.perry@stsci.edu>
Message-ID: <42190DC7.6080305@ee.byu.edu>

Perry Greenfield wrote:

>I'm going to refrain from a numerical rating but rather add a few comments
>and questions
>  
>
Great, works just as well.

>do I get '0' or 'array(0)' ?
>  
>
I'm thinking '0' here since we want "array" scalars to look and act like 
Python scalars.

>I suppose the former is fine if rank-0 is usable in every way as a scalar
>and no one really needs to know the difference, but I troubles me a bit to
>hide this fact (with repr) if it isn't quite the same.
>  
>
Well, I suppose repr could show something else (i.e. how you would 
create one) while str could print it like a scalar.  

>Along those lines (for this case and the last) I worry that there are
>potential gotchas that haven't been discovered yet. How confident are you
>that there aren't any in trying to integrate rank-0 with Python scalars and
>their usage. It may be one of those things that is hard to know unless the
>work of trying to implement it is actually done. You've thought about this a
>
Well, there may always be potential gotchas.  But, I'm extremely 
confident that the new Python scalars will work seemlessly with array 
objects.  The will work in Python as scalars pretty much anyplace a 
subtype is allowed (PyInt_Check will return True, for example).   There 
are a few places in Python where for performance reasons, exact integers 
are special-cased.  The new array-type scalars would not benefit from 
those places in the code.

>>2) Rank-0 arrays are never returned from uarray operations (unless using
>>asarray on a scalar), and when a rank-0 array naturally appears in the
>>calculation, an appropriate Python scalar is returned (note that this
>>would lose precision for long doubles unless a new Python object was
>>created).
>>
>>    
>>
>As Konrad mentions, as long as there is some means of handling long doubles,
>I find scalars perfectly acceptable. I tend to think that is what most user
>assume they are getting.
>  
>
Then, I think solution 4 is the right one....

>>    
>>
>Sounds good if there are no gotchas but may be a lot of work (coding and
>political). Any performance issues regarding rank-0 vs scalars? Will users
>pay a penalty for using rank-0 in scalar-like expressions (no apparent array
>usage other than the rank-0 values)?
>  
>
I see three performance issues:

1) the parts of Python that special-case for exact integer arguments.

2) if array scalars are internally converted to rank-0 arrays and then 
passed through the ufunc machinery to implement all operations (which 
will be the default implementation), then this will be a bit slower (not 
sure how much slower) --- especially after special casing for contiguous 
arguments in the ufunc code (PEP to come later).   

3) If the type hierarchy is too deep, then a few more pointer 
dereferences would be needed to find the function to call. 

Power users looking for a speed boost on scalar operations would have to 
manually get an exact Python integer.   So, I don't think array scalars 
would "take over" every need for a scalar, but they would be scalars 
that act as a smooth intermediary between N-d arrays *and* Python scalars.

I think by introducing this parallel set of array scalars we decrease 
the political issues.  We are not asking non-numeric Python users to 
change all beloved integers to something more generic.   We are just 
introducing a bunch of "array-behaved" scalar objects that will also 
inherit from Python builtin types where appropriate.


-Travis






From tim.hochberg at cox.net  Sun Feb 20 15:25:23 2005
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Sun Feb 20 15:25:23 2005
Subject: [Numpy-discussion] More PEP comments
Message-ID: <42191C33.8080203@cox.net>

 From the PEP:

>      integers in index arrays can be negative to mean selection from the end
>      of the array.  Indices that are too large will be taken to mean the
>      last element.  Indicies that are too negative will mean
>      the first element.
>
Here's another little piece of the PEP that I feel should be 
reconsidered. In the absence of a compelling reason to do otherwise, the 
indices in index arrays should behave just as if one were using that 
index directly as in index tuple. And, index tuples should match the 
behaviour Python lists. In other words:

alist[i] == anarray[i] == anarray(array([i]))[0]

In order to make this equivalence work, both anarray[i] and 
anarray(array([i])) need to raise index errors when i is too large or 
too negative. And for consistency this rule should continue to apply 
when the index is a tuple rather than a simple integer index.

This is more or less how numarray works and I think that's the right 
choice. [However, I did notice that numarray 1.1 produced nonsensical 
answers when index arrays with values that were too negative, when they 
were too positive they worked as I would expect and produced an error].

-tim










From aisaac at american.edu  Sun Feb 20 16:17:04 2005
From: aisaac at american.edu (Alan G Isaac)
Date: Sun Feb 20 16:17:04 2005
Subject: [Numpy-discussion] Matlab is a tool for doing numerical computations with matrices and vectors.
In-Reply-To: <4218FAD8.6060804@sympatico.ca>
References: <4218FAD8.6060804@sympatico.ca>
Message-ID: <Mahogany-0.66.0-572-20050220-191912.00@american.edu>

On Sun, 20 Feb 2005, "Colin J. Williams" apparently wrote:
> The hope is also that there is a need out there for matrix 
> operations, as distinct from array operations.

Absolutely.
I use Numeric's Matrix class all the time
(and long for additional functionality).
I am uncertain why numarray did not adopt
most of this class (especially its 
attribute list).

Cheers,
Alan Isaac







From juenglin at cs.pdx.edu  Sun Feb 20 23:32:00 2005
From: juenglin at cs.pdx.edu (Ralf Juengling)
Date: Sun Feb 20 23:32:00 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <4217C71E.8050002@cox.net>
References: <421757E9.10206@sympatico.ca> <4217A73D.8040900@ee.byu.edu> <4217C71E.8050002@cox.net>
Message-ID: <42198CDB.5020600@cs.pdx.edu>

Tim Hochberg wrote:
> I'd like to lobby against flatten(), r() and i(). To a large extent, 
> these duplicate the functionality of flat, real and imag. And, these 
> three methods are defined to sometimes return copies and sometimes 
> return views. That type of interface is a recipe for errors and should 
> only be used as a last resort. 

I feel the same.


> Flatten() is not necessary now that flat will be faux array with a view 
> to the original [I believe you are calling it an indexable iterator].  I 
> would, however,  recomend that A.flat.copy() work. In that case, A.flat 
> would be used when no copy was desired, and A.flat.copy() when a copy 
> was desired.  

Good idea.

ralf




From rkern at ucsd.edu  Mon Feb 21 04:28:04 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Mon Feb 21 04:28:04 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <4217C71E.8050002@cox.net>
References: <421757E9.10206@sympatico.ca> <4217A73D.8040900@ee.byu.edu> <4217C71E.8050002@cox.net>
Message-ID: <4219D37C.8000504@ucsd.edu>

Tim Hochberg wrote:
> 
> Hi Travis,
> 
> First off, let me say that I'm encouraged to see some action towards 
> unifying Numeric/Numarray the split has been somewhat dismaying. Thank 
> you for your efforts in this regard.
> 
> I'd like to lobby against flatten(), r() and i(). To a large extent, 
> these duplicate the functionality of flat, real and imag. And, these 
> three methods are defined to sometimes return copies and sometimes 
> return views. That type of interface is a recipe for errors and should 
> only be used as a last resort. 

There is, however, a blisteringly common use case for such an interface: 
you are using the result directly in an expression such that it is only 
going to be read and never written to. In that case, you want it to 
never fail (except in truly pathological cases like being out of 
memory), and you want it to be as efficient as possible and so never 
produce a copy where you can produce a view.

So, I think we need three interfaces for each of this kind of attribute:

1) Getting a view. If a view cannot be obtained, raise an error. Never copy.

2) Getting a copy. Never return a view.

3) Getting *something* the most efficient way possible. Caller beware.

While I lean towards making the syntaxes for the first two The Most 
Obvious Ways To Do It, I think it may be rather important to keep the 
syntax of the third convenient and short, particularly since it is that 
case that usually occurs in the middle of already-complicated expressions.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From tim.hochberg at cox.net  Mon Feb 21 07:20:02 2005
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Mon Feb 21 07:20:02 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <4219D37C.8000504@ucsd.edu>
References: <421757E9.10206@sympatico.ca> <4217A73D.8040900@ee.byu.edu> <4217C71E.8050002@cox.net> <4219D37C.8000504@ucsd.edu>
Message-ID: <4219FBE3.1000906@cox.net>

Robert Kern wrote:

> Tim Hochberg wrote:
>
>>
>> Hi Travis,
>>
>> First off, let me say that I'm encouraged to see some action towards 
>> unifying Numeric/Numarray the split has been somewhat dismaying. 
>> Thank you for your efforts in this regard.
>>
>> I'd like to lobby against flatten(), r() and i(). To a large extent, 
>> these duplicate the functionality of flat, real and imag. And, these 
>> three methods are defined to sometimes return copies and sometimes 
>> return views. That type of interface is a recipe for errors and 
>> should only be used as a last resort. 
>
>
> There is, however, a blisteringly common use case for such an 
> interface: you are using the result directly in an expression such 
> that it is only going to be read and never written to. In that case, 
> you want it to never fail (except in truly pathological cases like 
> being out of memory), and you want it to be as efficient as possible 
> and so never produce a copy where you can produce a view.
>
> So, I think we need three interfaces for each of this kind of attribute:
>
> 1) Getting a view. If a view cannot be obtained, raise an error. Never 
> copy.

The proposal for flat is to always return a view, if the array is not 
contiguous a special view-of-a-discontiguous-array will be returned. 
This special object obviously be less efficient to index than a 
contiguous array, but if implemented carefully could probably be made 
more efficient than a copy plus indexing for one-shot uses (i.e., in an 
expression).

>
> 2) Getting a copy. Never return a view.
>
> 3) Getting *something* the most efficient way possible. Caller beware.

By this you mean return a view if contiguous, a copy if not, correct?

> While I lean towards making the syntaxes for the first two The Most 
> Obvious Ways To Do It, I think it may be rather important to keep the 
> syntax of the third convenient and short, particularly since it is 
> that case that usually occurs in the middle of already-complicated 
> expressions.

Personally I don't find this all that compelling. Primarily because the 
blanket statement that (3) is more efficient than (1) is extremely 
suspect. In can many, if not most, cases (1) [as clarified above] will 
be more efficient than (3) anyway. Our disagreement here may stem from 
having different understandings of the proposed behaviour for flat.

There are cases where you probably *do* want to copy if discontiguous, 
but not in expressions. I'm thinking of cases where you are going to be 
reusing some flattened slice of a larger array multiple times, and you 
plan to use it read only (already that's sounding pretty rare though). 
In that case, my preferred spelling is:

a_contig_flat_array = ascontiguous(an_array).flat

In other words, I'd like to segregate all of the sometimes-copy 
behaviour into functions called asXXX, so that it's easy to see and 
easier to debug when it goes wrong. Of course, ascontiguous doesn't 
exist at present, as far as I can tell, but it'd be easy enough to add:

def ascontiguous(a):
    a = asarray(a)
    if not a.iscontiguous():
        a = a.copy()
    return a


Regards,

-tim







From cjw at sympatico.ca  Mon Feb 21 10:22:05 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Mon Feb 21 10:22:05 2005
Subject: [Numpy-discussion] Matlab is a tool for doing numerical computations
 with matrices and vectors.
In-Reply-To: <421908B0.90406@ee.byu.edu>
References: <4218FAD8.6060804@sympatico.ca> <421908B0.90406@ee.byu.edu>
Message-ID: <421A26A5.7070306@sympatico.ca>

Travis Oliphant wrote:

>
>> The Numeric/numarray focus is on multidimensional arrays.
>>
>> Some years ago, Huaiyu Zhu made a number of postings advocating the 
>> introduction of a Python-based system permitting the usual matrix 
>> operations. It seems that the culmination of these 
>
>
>> efforts was Matpy (http://matpy.sourceforge.net/), which is not 
>> currently active.
>
>
>
> I remember his work. I really liked many of his suggestions, though it 
> took him a while to recognize that a Matrix class has been distributed 
> with Numeric from very early on. 

numpy.pdf dated 03-07-18 has

    "For those users, the Matrix class provides a more intuitive
    interface. We defer discussion of the Matrix class until later."


Page 51 of that document has:

    "Subclassing
    Subclassing Numeric arrays is not possible due to a limitation of
    Python. The approach taken in the Masked
    Array facility (?Masked Arrays? on page 97) is one answer.
    UserArray.py, described below, can be subclassed,
    but this is often unsatisfactory unless you put in a similar effort
    to that in MA."

On the same page there is:

    "Matrix.py
    The Matrix.py python module defines a class Matrix which is a
    subclass of UserArray. The only differences
    between Matrix instances and UserArray instances is that the *
    operator on Matrix performs a
    matrix multiplication, as opposed to element-wise multiplication,
    and that the power operator ** is disallowed
    for Matrix instances."

In view of the above, I can understand why Huaiyu Zhu took a while. His 
proposal was much more ambitious.

Yes, I know that the power operator is implemented and that there is a 
random matrix but I hope that some attention is given to the 
functionality PyMatrix. I recognize that the implementation has some 
weakneses.

> Yes, it needed work, and a few of his ideas were picked up on and 
> included in Numeric's Matrix object.

I suggest that this overstates what was picked up.

> I would like to see a standard Matrix class (along with the 
> record-array class) also be included with the universal array.

Good, on both scores. I hope that the PEP will set out these ideas.

Colin W.




From tchur at optushome.com.au  Tue Feb 22 16:58:39 2005
From: tchur at optushome.com.au (Tim Churches)
Date: Tue Feb 22 16:58:39 2005
Subject: [Numpy-discussion] Broken setup.py in Numeric 23.7
In-Reply-To: <4219FBE3.1000906@cox.net>
References: <421757E9.10206@sympatico.ca> <4217A73D.8040900@ee.byu.edu> <4217C71E.8050002@cox.net> <4219D37C.8000504@ucsd.edu> <4219FBE3.1000906@cox.net>
Message-ID: <421BD507.3050107@optushome.com.au>

A small plea: exciting work on Numeric3 not withstanding, would it be 
possible to fix setup.py in Numeric 23.7 so that it doesn't fall over on 
Linux? A few versions ago, if BLAS and LAPACK weren't installed or 
weren't found, NumPy would automatically build with its own versions of 
BLAS_lite and LAPACK_lite. Now it just falls over, and it is very 
inconvenient to have to instruct users of our app, which depends on 
Numeric, how to patch setup.py, and/or provide a replacement setup.py.

Also, the new home page for Numpy, to which http://numpy.sf.net is 
redirected, does not have a link to the SorceForge project page for 
Numpy from which released tarballs or zip file can be downloaded. That's 
another source of grief for users, who may not know to go to 
http://www.sf.net/projects/numpy

Thanks,

Tim C





From Sebastien.deMentendeHorne at electrabel.com  Wed Feb 23 03:18:02 2005
From: Sebastien.deMentendeHorne at electrabel.com (Sebastien.deMentendeHorne at electrabel.com)
Date: Wed Feb 23 03:18:02 2005
Subject: [Numpy-discussion] PEP updated
Message-ID: <035965348644D511A38C00508BF7EAEB145CB330@seacex03.eib.electrabel.be>

> For sequence behaviour len, indexing and iteration should work
> consistently, i.e., "a[i] for i in xrange(len(a))" and "x for x in a"
> should identically generate the contents of a.  Currently indexing and
> iteration occur along the first dimension, which for rank 0 arrays
> does not exist.  I understand that there is no burning desire to
> completely switch to Matlab-style one-dimensional indexing.  The
> alternatives are then giving rank 0 arrays exceptional semantics
> inconsistent with other arrays or raising an exception if len or
> iteration is applied to a rank 0 array.  Without exceptional semantics
> the only index expression that makes sense is a[()].  I would favor
> consistency here and raise an exception but there is one matter that
> raises some doubt: do programs expect that if __len__ and __iter__
> exist they will not fail?
> 

Looking at the code I wrote that must deal with special scalar vs array
cases, I noticed that an important feature that the rank-0/scalar should
have is a correct indexing behaviour with:
 a[..., NewAxis] or a[NewAxis] or a[NewAxis, ...]

By the way, isn't it easier to find a solution to this problem by looking at
specific pieces of code that currently needs an ugly case for scalar ?
Moreover, should this rank-0 array behaviour of scalar be extended to other
"scalar" objects ? Should we be able to use:
 p = (3,4)
 a = p[NewAxis]
 a.shape = (1,)
 a.typecode() ==> 'O'
?



From gerard.vermeulen at grenoble.cnrs.fr  Wed Feb 23 09:52:50 2005
From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen)
Date: Wed Feb 23 09:52:50 2005
Subject: [Numpy-discussion] ANN: PyQwt-4.2 released
Message-ID: <20050223184715.6f4d1d07.gerard.vermeulen@grenoble.cnrs.fr>

What is PyQwt?

- it is a set of Python bindings for the Qwt C++ class library which
  extends the Qt framework with widgets for scientific, engineering
  and financial applications.   It provides a widget to plot
  2-dimensional data and various widgets to display and control
  bounded or unbounded floating point values.

- it requires and extends PyQt, a set of Python bindings for Qt.

- it supports the use of PyQt, Qt, Qwt, the Numerical Python extensions
  (either Numeric, or numarray or both) and optionally SciPy in a GUI
  Python application or in an interactive Python session.

- it runs on POSIX, MacOS/X and Windows platforms (practically any
  platform supported by Qt and Python).

The home page of PyQwt is http://pyqwt.sourceforge.net.

PyQwt-4.2 is a maintenance release to keep up with the recent releases
of PyQt-3.14 and its code generator SIP-4.2.

PyQwt-4.2 supports:

1. Python-2.4 downto -2.1.
2. PyQt-3.14 downto -3.10.
3. SIP-4.2 downto -4.0, and SIP-3.11 downto -3.10.
4. Qt-3.3.4 downto -2.3.0.
5. Qwt-4.2.0.


Have fun -- Gerard Vermeulen



From jmiller at stsci.edu  Wed Feb 23 12:46:45 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Feb 23 12:46:45 2005
Subject: [Numpy-discussion] ANN: numarray-1.2.2
Message-ID: <1109191504.19455.3.camel@halloween.stsci.edu>

Release Notes for numarray-1.2.2

Numarray is an array processing package designed to efficiently
manipulate large multi-dimensional arrays.  Numarray is modelled after
Numeric and features c-code generated from python template scripts, the
capacity to operate directly on arrays in files, arrays of heterogeneous
records, string arrays, and in-place operation on 
array memory mapped onto files.

I. ENHANCEMENTS

1. Support for user defined universal functions with arbitrary numbers
of parameters.  See numarray/Examples/ufunc/setup_special.py

2. Additional modifications to support numarray operation with
scipy_base.

3. Import time roughly halved thanks to Rory Yorke.

4. Improved sorting functions from Chuck Harris.

5. Simplified array() and fromfile() from Rory Yorke.

6. Port of Numeric's dotblas extension for faster matrix
multiplication when linked against ATLAS, LAPACK, BLAS, etc.

7. Improvements to array indexing speed from Francesc Alted and Tim
Hochberg.

II. BUGS FIXED / CLOSED

1120138	Document ieeespecial	  
1119411	No file named "LinearAlgebra2/setup.py"
1119265	'print' fails on masked arrays	  
1111765	python setup.py bdist_rpm fails (1.2a)
1110655	Problem with the ufunc .sum() and a rank-0 array.
1110607	f2py file paths on windows	  
1108374	count method in string.array wrong
1102874	remove use of basestring in fromfile for Python 2.2
1100435	Remove NUL bytes in nd_image/test.py
1099974	.sum() gives wrong results	
1091474	log(0) should be -inf	
1089116	CharArray maxLen bug	
1088817	Convolution with complex kernels gives wrong answer
1087378	LinAlgError is a string, not an Exception
1087304	Add toUpper() and toLower to CharArray
1087158	Float64 == records.CharType
1075802	masked_print_option produces incorrect output
1075795	float() and int() don't work on masked arrays
1075794	correlate2d should do autocorrelations
1075793	convolve2d/correlate2d should create output arrays
1075789	resized() pads with nulls not blanks: numarray 1.2.2
1069032	A little bug in setup.py
1068768	KeyboardInterrupt shows strange behavior
1067238	memory leak in determinant
1056649	Memory leak with dot and transpose
1052488	Modify sum() result type for boolean arrays
1050292	Side effects of putmask
1047272	Update license references
1045518	A disconnected numarray rant
1044885	complex number ieee oddities
1044788	wrong typecode
1038528	Making PyArray_CanCastSafely safer on 64-bit machines
1038397	numarray sum() has a possible memory leak
1038064	tofile() of zero length array
1037038	boolean cummulative operator leak
1036327	Buffer overflow in sum and add_Int64_reduce
1035701	numerical array from string list loops
1033860	Wrong array indexing when byteorder is changed
1031883	MA __setslice__ bug
1031356	Sequences with Python longs --> Int64
1030023	problem with matrixmultiply
1029547	IndexError in 4.8 Index Arrays example
1028782	matrixmultiply reference counting
1028738	atlas/blas library listings in addons.py
1025160	gcc error during install: duplicate case value
1022523	chararray concatentation problem
1018252	bad memory leak in reduce/accumulate

See
http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse
for more details.

III. CAUTIONS

This release should be backward binary compatible with numarray-1.1.1.

WHERE
-----------

Numarray-1.2.2 windows executable installers, source code, and manual is
here:

http://sourceforge.net/project/showfiles.php?group_id=1369

Numarray is hosted by Source Forge in the same project which hosts
Numeric:

http://sourceforge.net/projects/numpy/

The web page for Numarray information is at:

http://stsdas.stsci.edu/numarray/index.html

Trackers for Numarray Bugs, Feature Requests, Support, and Patches are
at the Source Forge project for NumPy at:

http://sourceforge.net/tracker/?group_id=1369

REQUIREMENTS
------------------------------

numarray-1.2.2 requires Python 2.2.2 or greater.  


AUTHORS, LICENSE
------------------------------

Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC
Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science
Institute.  We'd like to acknowledge the assitance of Francesc Alted,
Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh,
Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu
Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else
who has contributed with comments and feedback.

Numarray is made available under a BSD-style License.  See
LICENSE.txt in the source distribution for details.

-- 
Todd Miller             jmiller at stsci.edu





From NadavH at VisionSense.com  Thu Feb 24 06:58:01 2005
From: NadavH at VisionSense.com (Nadav Horesh)
Date: Thu Feb 24 06:58:01 2005
Subject: [Numpy-discussion] Incompatibility betwewn numarray 1.2.2  and Numeric
Message-ID: <421DE929.9020803@VisionSense.com>

Numeric and numarray arrays became incopatible ---
Is it a bug or a design change?


In [10]: import Numeric as N

In [11]: A = N.arange(6.0)

In [12]: import numarray as n

In [13]: a = n.arange(6.0)

In [14]: a+A                  # numarray op Numeric
---------------------------------------------------------------------------
exceptions.ValueError                                Traceback (most 
recent call last)

/tmp/numarray-1.2.2/<console>

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
__add__(self, operand)
    933             return operand.__radd__(self)
    934         else:
--> 935             return ufunc.add(self, operand)
    936    
    937     def __radd__(self, operand):

/usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
_cache_miss2(self, n1, n2, out)
    918
    919     def _cache_miss2(self, n1, n2, out):
--> 920         (in1, in2), inform, scalar = _inputcheck(n1, n2)
    921        
    922         mode, win1, win2, wout, cfunc, ufargs = \

/usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
_inputcheck(*inargs)    374             # see if it can be made into an 
array
    375             try:
--> 376                 inarg = _nc.array(inarg)
    377             except TypeError:
    378                 raise TypeError(

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
array(sequence, typecode, copy, savespace, type, shape)
    399     if shape is None:
    400         shape=()
--> 401     return fromlist([sequence],type,shape)
    402
    403 def asarray(seq, type=None, typecode=None):

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
fromlist(seq, type, shape, check_overflow, typecode)
    245     tshape = _frontseqshape(seq)
    246     if shape is not None and _gen.product(shape) != 
_gen.product(tshape):
--> 247         raise ValueError("shape incompatible with sequence")
    248     ndim = len(tshape)
    249     if ndim <= 0:

ValueError: shape incompatible with sequence

In [15]: A+a     # The other way around
---------------------------------------------------------------------------
exceptions.ValueError                                Traceback (most 
recent call last)

/tmp/numarray-1.2.2/<console>

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
__radd__(self, operand)
    940             return operand.__add__(self)
    941         else:
--> 942             r = ufunc.add(operand, self)
    943             if isinstance(r, NumArray):
    944                 r.__class__ = self.__class__

/usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
_cache_miss2(self, n1, n2, out)
    918
    919     def _cache_miss2(self, n1, n2, out):
--> 920         (in1, in2), inform, scalar = _inputcheck(n1, n2)
    921        
    922         mode, win1, win2, wout, cfunc, ufargs = \

/usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
_inputcheck(*inargs)    374             # see if it can be made into an 
array
    375             try:
--> 376                 inarg = _nc.array(inarg)
    377             except TypeError:
    378                 raise TypeError(

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
array(sequence, typecode, copy, savespace, type, shape)
    399     if shape is None:
    400         shape=()
--> 401     return fromlist([sequence],type,shape)
    402
    403 def asarray(seq, type=None, typecode=None):

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
fromlist(seq, type, shape, check_overflow, typecode)
    245     tshape = _frontseqshape(seq)
    246     if shape is not None and _gen.product(shape) != 
_gen.product(tshape):
--> 247         raise ValueError("shape incompatible with sequence")
    248     ndim = len(tshape)
    249     if ndim <= 0:

ValueError: shape incompatible with sequence


--------------------------

  Nadav.



From jmiller at stsci.edu  Thu Feb 24 08:05:35 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Thu Feb 24 08:05:35 2005
Subject: [Numpy-discussion] Incompatibility betwewn numarray 1.2.2  and
	Numeric
In-Reply-To: <421DE929.9020803@VisionSense.com>
References: <421DE929.9020803@VisionSense.com>
Message-ID: <1109261019.16689.23.camel@halloween.stsci.edu>

It's a bug.

Thanks,
Todd

On Thu, 2005-02-24 at 09:48, Nadav Horesh wrote:
> Numeric and numarray arrays became incopatible ---
> Is it a bug or a design change?
> 
> 
> In [10]: import Numeric as N
> 
> In [11]: A = N.arange(6.0)
> 
> In [12]: import numarray as n
> 
> In [13]: a = n.arange(6.0)
> 
> In [14]: a+A                  # numarray op Numeric
> ---------------------------------------------------------------------------
> exceptions.ValueError                                Traceback (most 
> recent call last)
> 
> /tmp/numarray-1.2.2/<console>
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> __add__(self, operand)
>     933             return operand.__radd__(self)
>     934         else:
> --> 935             return ufunc.add(self, operand)
>     936    
>     937     def __radd__(self, operand):
> 
> /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
> _cache_miss2(self, n1, n2, out)
>     918
>     919     def _cache_miss2(self, n1, n2, out):
> --> 920         (in1, in2), inform, scalar = _inputcheck(n1, n2)
>     921        
>     922         mode, win1, win2, wout, cfunc, ufargs = \
> 
> /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
> _inputcheck(*inargs)    374             # see if it can be made into an 
> array
>     375             try:
> --> 376                 inarg = _nc.array(inarg)
>     377             except TypeError:
>     378                 raise TypeError(
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> array(sequence, typecode, copy, savespace, type, shape)
>     399     if shape is None:
>     400         shape=()
> --> 401     return fromlist([sequence],type,shape)
>     402
>     403 def asarray(seq, type=None, typecode=None):
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> fromlist(seq, type, shape, check_overflow, typecode)
>     245     tshape = _frontseqshape(seq)
>     246     if shape is not None and _gen.product(shape) != 
> _gen.product(tshape):
> --> 247         raise ValueError("shape incompatible with sequence")
>     248     ndim = len(tshape)
>     249     if ndim <= 0:
> 
> ValueError: shape incompatible with sequence
> 
> In [15]: A+a     # The other way around
> ---------------------------------------------------------------------------
> exceptions.ValueError                                Traceback (most 
> recent call last)
> 
> /tmp/numarray-1.2.2/<console>
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> __radd__(self, operand)
>     940             return operand.__add__(self)
>     941         else:
> --> 942             r = ufunc.add(operand, self)
>     943             if isinstance(r, NumArray):
>     944                 r.__class__ = self.__class__
> 
> /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
> _cache_miss2(self, n1, n2, out)
>     918
>     919     def _cache_miss2(self, n1, n2, out):
> --> 920         (in1, in2), inform, scalar = _inputcheck(n1, n2)
>     921        
>     922         mode, win1, win2, wout, cfunc, ufargs = \
> 
> /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
> _inputcheck(*inargs)    374             # see if it can be made into an 
> array
>     375             try:
> --> 376                 inarg = _nc.array(inarg)
>     377             except TypeError:
>     378                 raise TypeError(
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> array(sequence, typecode, copy, savespace, type, shape)
>     399     if shape is None:
>     400         shape=()
> --> 401     return fromlist([sequence],type,shape)
>     402
>     403 def asarray(seq, type=None, typecode=None):
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> fromlist(seq, type, shape, check_overflow, typecode)
>     245     tshape = _frontseqshape(seq)
>     246     if shape is not None and _gen.product(shape) != 
> _gen.product(tshape):
> --> 247         raise ValueError("shape incompatible with sequence")
>     248     ndim = len(tshape)
>     249     if ndim <= 0:
> 
> ValueError: shape incompatible with sequence
> 
> 
> --------------------------
> 
>   Nadav.
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 




From gerard.vermeulen at grenoble.cnrs.fr  Sun Feb 27 07:13:05 2005
From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen)
Date: Sun Feb 27 07:13:05 2005
Subject: [Numpy-discussion] ANN: PyQwt3D-0.1 released
Message-ID: <20050227160647.1ebdf98d.gerard.vermeulen@grenoble.cnrs.fr>

What is PyQwt3D?

- it is a set of Python bindings for the QwtPlot3D C++ class library
  which extends the Qt framework with widgets for 3D data visualization.
  PyQwt3D inherits the snappy feel from QwtPlot3D.
  The examples at http://pyqwt.sourceforge.net/pyqwt3d-examples.html
  show how easy it is to make a 3D plot and how to save a 3D plot to
  an image or an (E)PS/PDF file.

- it requires and extends PyQt, a set of Python bindings for Qt.

- it supports the use of PyQt, Qt, Qwt, the Numerical Python extensions
  (either Numeric, or numarray or both) and optionally SciPy in a GUI
  Python application or in an interactive Python session.

- it runs on POSIX, MacOS/X and Windows platforms (practically any
  platform supported by Qt and Python).

The home page of PyQwt3D is http://pyqwt.sourceforge.net.


PyQwt3D-0.1 supports:

1. Python-2.4 downto -2.3.
2. PyQt-3.14 downto -3.12.
3. SIP-4.2 downto -4.0, but SIP-4.2 is recommended.
   PyQwt3D does *not* support SIP-3.x (I tried, but failed).
4. Qt-3.3.4 downto -2.3.0.
5. QwtPlot3D-0.2.4-beta.


Have fun -- Gerard Vermeulen



From jmiller at stsci.edu  Tue Feb  1 05:09:08 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Tue Feb  1 05:09:08 2005
Subject: [Numpy-discussion] Numeric array's actual data address?
In-Reply-To: <5.2.0.4.2.20050131165659.12d54640@blue-cove.com>
References: <5.2.0.4.2.20050131165659.12d54640@blue-cove.com>
Message-ID: <1107263206.5023.22.camel@jaytmiller.comcast.net>

On Mon, 2005-01-31 at 17:04 -0800, Ray S wrote:
> If I have an array N:
> 
>  >>> N = Numeric.zeros((1000,), Numeric.Float)
>  >>> repr(N.__copy__)
> '<built-in method __copy__ of array object at 0x00809F00>'
> 
> What is the actual address of the first element? 

In C,  look at a->data.

> Or, as an offset from the 
> object?

This doesn't really make sense given the layout of the objects and how
the memory is stored.  You should probably read over the section on the
C-API in the Numeric and numarray manuals.  The Numeric manual,  here,
is probably the best place to start because it is the simplest:

http://www.pfdubois.com/numpy/numpy.pdf

The numarray manual,  here:

http://prdownloads.sourceforge.net/numpy/numarray-1.1.pdf?download

has a section on a Numeric compatibility layer which is very similar.

> numarray gives us that:
>  >>> N = numarray.zeros((1000,), numarray.Float)
>  >>> N.info()
> class: <class 'numarray.numarraycore.NumArray'>
> shape: (1000,)
> strides: (8,)
> byteoffset: 0
> bytestride: 8
> itemsize: 8
> aligned: 1
> contiguous: 1
> data: <memory at 009d67b8 with size:8000 held by object 009d6798 aliasing 
> object  00000000>
> byteorder: little
> byteswap: 0
> type: Float64
> 
> In numarray, the offset appears to be 20.
> If I try to use memmove() to fill a Numeric array it faults when using an 
> offset of 20...

The offset of 0x20 is unique to the numarray.memory memory object.  It
has nothing to do with Numeric arrays so it could only work by accident.

Note that the "data" section of numarray's info() is referring to a
buffer object which is stored in the _data attribute.  I just changed
the info() to avoid this confusion in the future by renaming "data" to
"buffer".  From the overall perspective of the array object,  the
"working data pointer" (a->data in C) is the sum of the byteoffset and
_data base pointer.  

In response to this post,  I modified info() to the following:

>>> import numarray as na
>>> a = na.arange(10)
>>> a.info()
class: <class 'numarray.numarraycore.NumArray'>
shape: (10,)
strides: (4,)
byteoffset: 0
bytestride: 4
itemsize: 4
aligned: 1
contiguous: 1
buffer: <memory at 0xb7c07f68 with size:0x00000028 held by object 0xb7f8d5a0 aliasing object 0x00000000>
fragile data pointer: 0xb7c07f68 (DEBUG ONLY)
byteorder: 'little'
byteswap: 0
type: Int32

I think the "fragile data pointer" is generally useful information,  but
not completely dependable so I gave it the garish name it has.
Comments?

Regards,
Todd




From rays at san.rr.com  Tue Feb  1 08:24:53 2005
From: rays at san.rr.com (RJ)
Date: Tue Feb  1 08:24:53 2005
Subject: [Numpy-discussion] Numeric array's actual data address?
In-Reply-To: <1107263206.5023.22.camel@jaytmiller.comcast.net>
References: <5.2.0.4.2.20050131165659.12d54640@blue-cove.com>
 <5.2.0.4.2.20050131165659.12d54640@blue-cove.com>
Message-ID: <5.2.1.1.2.20050201070034.04bf2520@pop-server.san.rr.com>

Thanks Todd, Travis,

Yes, Travis, I _was_ using ctypes' memmove() with numarray to great benefit. I had also understood that the address of the numarray data was stable.

Originally, I was trying to move data from a hardware A/D driver DLL call to Python as quickly as possible and did some benchmarking, so as per Thomas Heller's suggestion on the ctypes list I used memmove():
http://sourceforge.net/mailarchive/forum.php?thread_id=6166311&forum_id=24606
 from ctypes import *
 import array 
 src = array.array("i", range(32))
 dst = (c_int * 32)()
 memmove(dst, *src.buffer_info())
memmove() is orders of magnitude faster than map() or assignments.
I then tested and found numarray faster for the rest of the tasks (FFT etc.) http://sourceforge.net/mailarchive/forum.php?thread_id=6205658&forum_id=24606
and so parsed the info() output and plugged the data[0] address into the call.

Thomas Heller (in the above) suggested he make a mod to ctypes to accept Python objects which implement the buffer interface as function parameters, which would allow Numeric use once implemented.
I like numarray's breadth of methods so I used it at the time, but in another try at speed-squeezing yesterday afternoon I found Numeric's FFT and subtraction to be >30% faster in this case, so I switched that code over (this increase includes the use of map() with Numeric to read the A/D in another thread). Using memmove() with Numeric would speed up the reader thread once again.

At 08:06 AM 2/1/2005 -0500, Todd Miller wrote:
>> 
>> What is the actual address of the first element? 
>
>In C,  look at a->data.

I had read the Numeric API and looked at the PyObject structure, as Travis then suggested, but my question then is: if the offset from the "array object at 0x..." (object address value) to the array[0] address is not fixed and must be read from the pointer in the PyObject structure, can we get that pointer's value directly from Python or ctypes? 
ctypes pointer() "allows" poking in and directly reading memory: 
"It is also possible to use indexes different from 0, but you must know what you're doing when you use this: You access or change arbitrary memory locations when you do this."
http://starship.python.net/crew/theller/ctypes/tutorial.html
So, could I use ctypes' pointer(offset) to read the structure's pointer-to-data[0]-value and then use it in memmove()? 
I'll cross-post this to ctypes-users...

>I think the "fragile data pointer" is generally useful information,  but
>not completely dependable so I gave it the garish name it has.
>Comments?

It's quite reasonable. 
Is the data pointer value really often changed for contiguous arrays in small stand-alone Python apps? If so, I may stick with map() to avoid having to re-read the pointer before each data buffer transfer.

If anyone is interested I could post some relevant code snips from the office...

Thanks for the help,
Ray
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050201/74d07089/attachment-0001.html>

From oliphant at ee.byu.edu  Thu Feb  3 03:46:23 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb  3 03:46:23 2005
Subject: [Numpy-discussion] Numeric3 status
Message-ID: <42020E7D.3040109@ee.byu.edu>

I've gotten Numeric3 (equivalent to Numeric 30.0) into a state where 
multiarray compiles.

It is far from production ready.  Several sections of the code still 
need to be cleaned up to avoid segfaults.  But, the general outline is 
there.   As this is very alpha code (though based on a very mature 
code-base) it is named Numeric3 so it does not clobber your current 
Numeric installation just in case you accidentally do an install.    You 
will have to make sure the correct multiarray is being loaded through 
your sys.path list (or just point your sys.path to the build location of 
Numeric3) if you want to play with it. 

No array_number methods have been loaded yet (there is an interface to 
load your own if you like).   I expect to have ufuncs working with the 
new arrayobject by the end of the month.  Until then, arrayobjects are 
purely structural beasts.

If anybody would like to comment on the design or direction of Numeric 
30.0, your comments are welcome.  If you would like to checkout the code 
through CVS you may do that using

cvs -d:pserver:anonymous at cvs.sourceforge.net:/cvsroot/numpy login

press <Enter> when prompted for a password.
 
cvs -z3 -d:pserver:anonymous at cvs.sourceforge.net:/cvsroot/numpy co -P 
/Numeric3/

A rough design document is available by following the link at  
http://www.scipy.org/wikis/numdesign/num3news

The major goal is:

Numeric 30.0 will have all the important new features of numarray as 
well as maintain or improve on the speed of Numeric for all array types.

Besides this, Numeric 30.0 will not include any FFT's, LinearAlgebra,  
or Random Numbers (these are all provided much more proficiently by 
SciPy).  Masked Arrays, and Matrices will be provided as standard 
subclasses in Python.   Ideally, RecordArrays will be provided in the 
same way.

I will not re-invent code that I can borrow from past versions of 
Numeric and or numarray.

Best wishes,

-Travis Oliphant








From mdehoon at ims.u-tokyo.ac.jp  Thu Feb  3 04:22:18 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Thu Feb  3 04:22:18 2005
Subject: [Numpy-discussion] Numeric3 status
In-Reply-To: <42020E7D.3040109@ee.byu.edu>
References: <42020E7D.3040109@ee.byu.edu>
Message-ID: <420216CE.9080502@ims.u-tokyo.ac.jp>

Travis Oliphant wrote:
> Besides this, Numeric 30.0 will not include any FFT's, LinearAlgebra,  
> or Random Numbers (these are all provided much more proficiently by 
> SciPy).

I use Numeric's LinearAlgebra and RandomArray quite a lot and I will 
really miss these if they are absent in Numeric 30.0. While it is true 
that these can be provided by SciPy, the drawback is that SciPy is 
harder to install. Since linear algebra, random numbers, and possibly 
FFTs are such basic numerical routines, I think they should be included 
in a Numeric instead of requiring users to download and install both 
Numeric and SciPy. Especially for new users, that may be a bridge too far.

--Michiel.



From konrad.hinsen at laposte.net  Thu Feb  3 05:14:37 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Thu Feb  3 05:14:37 2005
Subject: [Numpy-discussion] Numeric3 status
In-Reply-To: <420216CE.9080502@ims.u-tokyo.ac.jp>
References: <42020E7D.3040109@ee.byu.edu> <420216CE.9080502@ims.u-tokyo.ac.jp>
Message-ID: <450914d39d01fcf682ee304547f081bb@laposte.net>

On 03.02.2005, at 13:19, Michiel Jan Laurens de Hoon wrote:

> I use Numeric's LinearAlgebra and RandomArray quite a lot and I will  
> really miss these if they are absent in Numeric 30.0. While it is true  
> that these can be provided by SciPy, the drawback is that SciPy is  
> harder to install. Since linear algebra, random numbers, and possibly  
> FFTs are such

I agree completely. In addition, there is compatibilty: lots of code  
out there uses the FFT and LinearAlgebra interfaces that have been part  
of Numeric for many years. numarray has them as well (it just requires  
a change in the import statements).

While I like many aspects of your Numeric 3.0 project, I fear that it  
will just create a third nearly-but-not-fully-compatible choice for an  
array module.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From hyclak at math.ohiou.edu  Thu Feb  3 06:47:38 2005
From: hyclak at math.ohiou.edu (Matt Hyclak)
Date: Thu Feb  3 06:47:38 2005
Subject: [Numpy-discussion] Problems compiling against ATLAS on x86_64
Message-ID: <20050203144231.GB24701@math.ohiou.edu>

I'm about ready to pull out my hair trying to get this stuff to compile for
a faculty member. I'm running RHEL 3 AS on a dual opteron machine. ATLAS
compiles fine as I go through xconfig, but I can't for the life of me get
Numeric to compile against it. I've tried ATLAS 3.6.0 and 3.7.8. I'm
trying to build Numeric 23.7. I keep getting the following error:

gcc -shared build/temp.linux-x86_64-2.2/lapack_litemodule.o
-L/usr/lib64/atlas -llapack -lcblas -lf77blas -latlas -lg2c -o
build/lib.linux-x86_64-2.2/lapack_lite.so
/usr/bin/ld: /usr/lib64/atlas/liblapack.a(dgesv.o): relocation R_X86_64_32
can not be used when making a shared object; recompile with -fPIC
/usr/lib64/atlas/liblapack.a: could not read symbols: Bad value
collect2: ld returned 1 exit status

I built ATLAS with -fPIC, but that didn't seem to help. Could someone
*please* point me in the right direction before I start hitting things?

Thanks!

Matt

-- 
Matt Hyclak
Department of Mathematics 
Department of Social Work
Ohio University
(740) 593-1263





From jdhunter at ace.bsd.uchicago.edu  Thu Feb  3 07:22:48 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Thu Feb  3 07:22:48 2005
Subject: [Numpy-discussion] Numeric3 status
In-Reply-To: <450914d39d01fcf682ee304547f081bb@laposte.net> (konrad.hinsen@laposte.net's
 message of "Thu, 3 Feb 2005 14:12:01 +0100")
References: <42020E7D.3040109@ee.byu.edu> <420216CE.9080502@ims.u-tokyo.ac.jp>
	<450914d39d01fcf682ee304547f081bb@laposte.net>
Message-ID: <m3vf99of39.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "konrad" == konrad hinsen <konrad.hinsen at laposte.net> writes:

    konrad> I agree completely. In addition, there is compatibilty:
    konrad> lots of code out there uses the FFT and LinearAlgebra
    konrad> interfaces that have been part of Numeric for many
    konrad> years. numarray has them as well (it just requires a
    konrad> change in the import statements).

I too would be *very disappointed* to see LinearAlgebra, RandomArray
and FFT leave Numeric.  matplotlib requires all three.  Numeric is
lightweight and easy to install, and although the scipy install is
significantly easier than it was in yesteryear, it still has higher
barrier to entry.  I am much happier requiring only Numeric than
Numeric and scipy.

As for the change in import statements, one thing that would be nice
to see in Numeric3 is a package structure for these extra modules that
mirrors numarray's.  This is what the matplotlib numerix module
currently does, and the in the various __init__ files imports either
the Numeric or numarray symbols depending on the configuration
settings.  This seems most sensible since the longer term goal is
integration.

    peds-pc311:~/python/projects/matplotlib/lib/matplotlib> find numerix/ | grep -v CVS
    numerix/
    numerix/mlab
    numerix/mlab/__init__.py
    numerix/linear_algebra
    numerix/linear_algebra/__init__.py
    numerix/__init__.py
    numerix/fft
    numerix/fft/__init__.py
    numerix/_na_imports.py
    numerix/random_array
    numerix/random_array/__init__.py
    numerix/_nc_imports.py


Finally, breaking both backward compatibility with Numeric, and
forward compatibility with numarray, seems ill-advised.

JDH



From faltet at carabos.com  Thu Feb  3 09:10:15 2005
From: faltet at carabos.com (Francesc Altet)
Date: Thu Feb  3 09:10:15 2005
Subject: [Numpy-discussion] pyrex numarray
In-Reply-To: <20050201121626.6150f9ff.simon@arrowtheory.com>
References: <20050201121626.6150f9ff.simon@arrowtheory.com>
Message-ID: <200502031805.23278.faltet@carabos.com>

A Dimarts 01 Febrer 2005 02:16, Simon Burton va escriure:
> Has anyone considered using pyrex to implement numarray ?

None that I'm aware of. But this issue has been already discussed in
this list at least a couple of times before. See, for example:

http://sourceforge.net/mailarchive/message.php?msg_id=8080203

> I don't know a lot of the details but it seems to me that pyrex could
> unify numarray's python/c source code mixture and smooth the transition
> from python (==untyped pyrex) code to c (==typed pyrex) code. It would also
> help clueless users like me understand, and perhaps contribute to, the
> codebase.

Well, after the past discussions, I've got the feeling that the main
problems to do that are:

1.- Construct general Ufuncs that can handle many different
  combinations of arrays and types. I think that this is currently
  done through a use of some C preprocesor than creates specific C
  files from kind of template (correct me if I'm wrong there). Do the
  same thing with Pyrex may potentially need such a preprocessor as
  well.
  
2.- Recoding the numarray beast in Pyrex could be *major* task

3.- The lack of experience with Pyrex (at least some time ago)

Perhaps I miss something still more important, but my guess is that
these reasons are the real stoppers. While problem 3 can be quite
easily surmounted, problem 1 probably is the big one (except if one
finds some easy way to deal with it). Although now that I think, one
can always take advantage of the files coming from the current
numarray C preprocessor and use Pyrex as the glue with Python. Problem
2 is also worrying but a combined effort could contribute to alleviate
it.

However, I do think that is probably worth the effort to concentrate
the efforts resolving first the remaining problems with numarray
performance (mainly array creation time), than go and switch to yet
another implementation that would take far more time to implement.

My 2 cents,

-- 
>qo<   Francesc Altet ? ? http://www.carabos.com/
V ?V   C?rabos Coop. V. ??Enjoy Data
 ""




From focke at slac.stanford.edu  Thu Feb  3 09:34:34 2005
From: focke at slac.stanford.edu (Warren Focke)
Date: Thu Feb  3 09:34:34 2005
Subject: [Numpy-discussion] pyrex numarray
In-Reply-To: <200502031805.23278.faltet@carabos.com>
References: <20050201121626.6150f9ff.simon@arrowtheory.com>
 <200502031805.23278.faltet@carabos.com>
Message-ID: <Pine.LNX.4.58.0502030924340.20901@auriga.slac.stanford.edu>


On Thu, 3 Feb 2005, Francesc Altet wrote:

> A Dimarts 01 Febrer 2005 02:16, Simon Burton va escriure:
> > Has anyone considered using pyrex to implement numarray ?
>
> None that I'm aware of. But this issue has been already discussed in
> this list at least a couple of times before. See, for example:
>
> http://sourceforge.net/mailarchive/message.php?msg_id=8080203

Tim Hochberg worked on a psyco version, look at:

http://sourceforge.net/mailarchive/message.php?msg_id=3845401

maybe that could be used as a start.

Warren Focke





From perry at stsci.edu  Thu Feb  3 09:55:47 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb  3 09:55:47 2005
Subject: [Numpy-discussion] pyrex numarray
In-Reply-To: <200502031805.23278.faltet@carabos.com>
References: <20050201121626.6150f9ff.simon@arrowtheory.com> <200502031805.23278.faltet@carabos.com>
Message-ID: <285db149e78a418f0bad7d50e3515ae9@stsci.edu>

On Feb 3, 2005, at 12:05 PM, Francesc Altet wrote:

> A Dimarts 01 Febrer 2005 02:16, Simon Burton va escriure:
>> Has anyone considered using pyrex to implement numarray ?
>
> None that I'm aware of. But this issue has been already discussed in
> this list at least a couple of times before. See, for example:
>
> http://sourceforge.net/mailarchive/message.php?msg_id=8080203
>
>> I don't know a lot of the details but it seems to me that pyrex could
>> unify numarray's python/c source code mixture and smooth the 
>> transition
>> from python (==untyped pyrex) code to c (==typed pyrex) code. It 
>> would also
>> help clueless users like me understand, and perhaps contribute to, the
>> codebase.
>
> Well, after the past discussions, I've got the feeling that the main
> problems to do that are:
>
> 1.- Construct general Ufuncs that can handle many different
>   combinations of arrays and types. I think that this is currently
>   done through a use of some C preprocesor than creates specific C
>   files from kind of template (correct me if I'm wrong there). Do the
>   same thing with Pyrex may potentially need such a preprocessor as
>   well.

I meant to respond this earlier but left it hanging due to other 
distractions. Francesc points out that this has been raised in the 
past, and he also correctly mentions the biggest issue. How do you 
handle all the possible numeric types? If handling all that results in 
large amounts of C or psuedo C code, what is pyrex buying you, really? 
I think there are various aspects of arrays that get confused sometime. 
One is thinking that pyrex would make it easy to code a C loop over an 
array in a fairly simple way. And it would--for a case where you have 
fixed the types of the arrays being operated on. But what about all 
possible types for arguments? That's a different story (at least I 
think so; if someone has a clever idea that solves these problems let's 
hear it).

Perry




From oliphant at ee.byu.edu  Thu Feb  3 11:53:07 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb  3 11:53:07 2005
Subject: [Numpy-discussion] Numeric3
Message-ID: <420280AD.8080309@ee.byu.edu>

John Hunter wrote:

>>>>>> "konrad" == konrad hinsen <konrad.hinsen at laposte.net> writes:
>>>>>>           
>>>>>
>
>    konrad> I agree completely. In addition, there is compatibilty:
>    konrad> lots of code out there uses the FFT and LinearAlgebra
>    konrad> interfaces that have been part of Numeric for many
>    konrad> years. numarray has them as well (it just requires a
>    konrad> change in the import statements).
>
> I too would be *very disappointed* to see LinearAlgebra, RandomArray
> and FFT leave Numeric.  matplotlib requires all three.  Numeric is
> lightweight and easy to install, and although the scipy install is
> significantly easier than it was in yesteryear, it still has higher
> barrier to entry.  I am much happier requiring only Numeric than
> Numeric and scipy.
>  
>

This is all a matter of semantics and packaging.

My point is that it is silly to have a really nice wrapping of fft, 
linear algebra, and random number routines in SciPy and then repeat them 
somewhat for Numeric (or vice-versa).

So to satisfy this complaint, the standard Numeric3 install is actually 
a "slimmed-down" scipy.

> As for the change in import statements, one thing that would be nice
> to see in Numeric3 is a package structure for these extra modules that
> mirrors numarray's. 
> This is what the matplotlib numerix module
> currently does, and the in the various __init__ files imports either
> the Numeric or numarray symbols depending on the configuration
> settings.  This seems most sensible since the longer term goal is
> integration.
>  
>
I'm not that concerned about what names we give things,  but why not the 
names that scipy has.

Or scipy should change it's names.

There was some discussion about this on the scipy list.  Nobody had a 
clear winner in terms of "naming" conventions.
SciPy's stated goals for some time have been to provide numerical 
operations for arrays.  If you want to call this super-structure 
numerix, then fine.

SciPy adjusted for those that don't want to install everything.  It is 
very easy to pick and choose the packages you want to install.  There 
appears to be a hesitancy on the part of some to adopt something like 
scipy.  I'm not sure whether it's scipy itself, or just the idea of 
scipy, that some people find difficult to adjust to.   

We talk about some version of the arrayobject getting into the core.  
Guido is not going to accept FFT's and LinearAlgebra (or maybe not even 
ufuncs).  Towards this end, I prefer  to separate the ideas of the 
arrayobject from the things you can do with it, which is why I believe 
scipy (with appropriate slimmed down versions) is a better place to 
discuss fft's, linear algebra, and the like.  

Old code can be handled with compatibility layers.

-Travis










From rays at blue-cove.com  Thu Feb  3 12:25:47 2005
From: rays at blue-cove.com (Ray S)
Date: Thu Feb  3 12:25:47 2005
Subject: [Numpy-discussion] Re: [ctypes-users] copying/slicing ctypes arrays, (c_ulong
 *n)() - to Numeric
In-Reply-To: <d5vhbijx.fsf@python.net>
References: <5.2.0.4.2.20050202134734.137b2930@blue-cove.com>
 <5.2.1.1.2.20050201081943.05264720@blue-cove.com>
 <5.2.1.1.2.20041222161345.09a30140@blue-cove.com>
 <5.2.1.1.2.20041216074849.090f3010@blue-cove.com>
 <5.2.0.4.2.20041215100117.0dff97b0@blue-cove.com>
 <5.2.1.1.2.20041214202542.05f02920@pop-server.san.rr.com>
 <5.2.1.1.2.20041214202542.05f02920@pop-server.san.rr.com>
 <5.2.0.4.2.20041215100117.0dff97b0@blue-cove.com>
 <5.2.1.1.2.20041216074849.090f3010@blue-cove.com>
 <5.2.1.1.2.20041222161345.09a30140@blue-cove.com>
 <5.2.1.1.2.20050201081943.05264720@blue-cove.com>
 <5.2.0.4.2.20050202134734.137b2930@blue-cove.com>
Message-ID: <5.2.0.4.2.20050203104517.1448f1a8@blue-cove.com>

At 07:32 PM 2/3/2005 +0100, you wrote:

>Don't be confused that the buffer() object says <read-only buffer ...>!
>The buffer call only asks for readable memory..., but ctypes doesn't
>care about the readonly attribute - it will happily write into this
>memory.

Hi Thomas,

Yes, I was thinking of what the shell error said upon assignment...

Upon adding to some working code all is well:

 >>> import Numeric, ctypes, string
 >>> N = Numeric.zeros((10,), Numeric.Float)
 >>> buf = buffer(N)
 >>> buf
<read-only buffer for 0x008F9C28, ptr 0x008D7780, size 80 at 0x008FE220>
 >>> int(string.split(repr(buf))[5][:-1], 16)
9271168

## numarray version
# nAddress = int(string.split(repr(N._data))[2], 16)

## Numeric version
NAddress = int(string.split(repr(buffer(N)))[5][:-1], 16)

## Load DLL here...
## do this to get data from the USB A/D's DLL
usb.GetData(usb.Sn, (bufferInsertPos * N.itemsize()) + NAddress,
                                     ctypes.byref( (types.c_long * 
buffersize)() ) )

Which is faster than getting data into a ctypes array (c_ulong *n)() and 
then doing memmove() to Numeric - one less step.

Maybe this snip would be of help to some others, although more so to numpy 
people.

Of course, the Python array works the same:
 >>> a = array.array('l',[1,2,3])
 >>> int(string.split(repr(buffer(a)))[5][:-1], 16)
8380408

>If this is too confusing, and this may well be, ctypes could expose a
>memory() function which would insist on read-write memory, but apart
>from that do the same that buffer does:

No, not confusing, just not clear to a non-expert C person that ctypes 
ignores where Python is read-only. A simple note in the tutorial would be 
fine.
Some over at numpy were also unaware of memmove()s' existence in the new 
releases, and seemed interested.

Thanks again,
Ray




From oliphant at ee.byu.edu  Thu Feb  3 13:31:26 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb  3 13:31:26 2005
Subject: [Numpy-discussion] Numeric3 and speed of new-style classes
Message-ID: <42029763.9090700@ee.byu.edu>


I fixed a major oversight in Numeric3 and now it works pretty well (it 
is basically old Numeric with a bunch of new types, no math (yet)), 
using new-style classes, and some checks for alignment when 
dereferencing data.

I did some simple array creation timing tests.  An example of the 
results is given below.  The bottom line is that new-style classes do 
nothing but speed up (a little bit) basic object creation. 


FYI:


First test (Numeric3):    .196081876755

import sys
sys.path.insert(0,"../build/lib.linux-i686-2.3/")

import time

import multiarray

start = time.time()
for k in range(100000):
    g = multiarray.arraytype((4,3),'d')
print time.time() - start


Second test (Numeric):  .203589916229

import Numeric
import time


start = time.time()
for k in range(100000):
    g = Numeric.empty((4,3),'d')
print time.time() - start













From Chris.Barker at noaa.gov  Thu Feb  3 15:13:26 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Thu Feb  3 15:13:26 2005
Subject: [Numpy-discussion] pyrex numarray
In-Reply-To: <285db149e78a418f0bad7d50e3515ae9@stsci.edu>
References: <20050201121626.6150f9ff.simon@arrowtheory.com> <200502031805.23278.faltet@carabos.com> <285db149e78a418f0bad7d50e3515ae9@stsci.edu>
Message-ID: <4202AF4E.8060207@noaa.gov>

Perry Greenfield wrote:
> How do you handle all 
> the possible numeric types? If handling all that results in large 
> amounts of C or psuedo C code, what is pyrex buying you, really?

There was a quote from Greg Ewing that went something along the lines of:

"I am not re-implementing C++ templates for Pyrex!"

Which I think summarized his take on this.

This begs the question for me:

Why not use C++ templates?

I know this was brought up very early in numarray development, 
specifically the idea of using Blitz++ (or something similar) as the 
basis for the new NumPy. I'm pretty sure it was rejected primarily on 
the grounds of limited good support for C++ templates on a wide variety 
of compilers. I think that's sad, as by now template support is much better.

The other idea bandied about is to make Psyco multi-array aware. That 
would be cool!

-Chris

---
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From mdehoon at ims.u-tokyo.ac.jp  Thu Feb  3 20:16:07 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Thu Feb  3 20:16:07 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420280AD.8080309@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu>
Message-ID: <4202F6A0.3@ims.u-tokyo.ac.jp>

Travis Oliphant wrote:
> John Hunter wrote:
>> I too would be *very disappointed* to see LinearAlgebra, RandomArray
>> and FFT leave Numeric.  matplotlib requires all three.  Numeric is
>> lightweight and easy to install, and although the scipy install is
>> significantly easier than it was in yesteryear, it still has higher
>> barrier to entry.  I am much happier requiring only Numeric than
>> Numeric and scipy.
>>  
>>
> 
> This is all a matter of semantics and packaging.
> 
Well it is mainly a matter of being able to install what you need, and 
not having to spend time figuring out how to install what you don't 
need. Let me give you an example from my own field (computational 
biology). I am one of the maintainers of Biopython, which uses 
LinearAlgebra and RandomArray. Many of our users are not very familiar 
with Python. Even installing Numerical Python sometimes causes problems, 
and I'm sure we have lost users in the past because of that. SciPy, in 
my experience, doesn't build out of the box. If Biopython required 
SciPy, many users will run into installation problems, most of them will 
give up and return to using Microsoft Excel for their numerical 
analysis, medical research will come to a standstill, and we'll all die.

But seriously, I think that there are advantages to Numerical Python and 
to SciPy, and they both have a reason to exist. Numerical Python has the 
advantage that it is easy to install and contains most (if not all) of 
the numerical routines users will need. SciPy, on the other hand, is 
harder to install (e.g. because it doesn't use ANSI-C only), but 
therefore can also offer pieces of software that are not available in 
Numerical Python.

Many users, however, will never need the additional capabilities that 
are in SciPy. Moving LinearAlgebra, RandomArray, and FFT to SciPy will 
make their life more difficult. I don't see an advantage for SciPy users 
either, because they need to install Numerical Python anyway.

So I would suggest the following demarcation line between Numerical 
Python and SciPy:

Stuff that is needed by lots of users (linear algebra, FFT, random 
numbers, special functions) and is available in ANSI-C (so its 
installation is straightforward and won't cause problems to users who 
don't need it) should go into Numerical Python.

Stuff that is needed by fewer users or is not available in ANSI-C should 
go into SciPy. SciPy, with its binary installers, can provide a real 
benefit to users who need extension modules written in Fortran/C++/whatnot.

To address the issue you raised about duplicated efforts in numpy and 
SciPy: What does SciPy do better in terms of linear algebra, FFT, or 
random numbers, than numpy? If so, is it possible to move the relevant 
SciPy code pieces to numpy (without affecting numpy's ease of installation)?

--Michiel.



From rkern at ucsd.edu  Thu Feb  3 20:46:48 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Thu Feb  3 20:46:48 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420280AD.8080309@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu>
Message-ID: <4202FE20.8040302@ucsd.edu>

Travis Oliphant wrote:

> We talk about some version of the arrayobject getting into the core.  
> Guido is not going to accept FFT's and LinearAlgebra (or maybe not even 
> ufuncs).  Towards this end, I prefer  to separate the ideas of the 
> arrayobject from the things you can do with it, which is why I believe 
> scipy (with appropriate slimmed down versions) is a better place to 
> discuss fft's, linear algebra, and the like. 

I think this is the point that some people are missing. Numeric3 is 
intended to be the multiarray object proposed to go into the standard 
library. FFTs will not go into the standard library. QR decompositions 
will not go into the standard library.

One can always support the extra packages externally.

So for people who need LinearAlgebra, et al., their current situation is 
this:

   ThirdPartyPackage depends on Numeric and a standard Python
   distribution.

Their new situation would be this:

   ThirdPartyPackage depends on NumericExtras and a standard Python
   distribution.

If you need to depend on these modules with their current interfaces, 
then someone needs to step up to support it. This support probably must 
come from the people who need the current interfaces. For far too long, 
the responsibility of maintaining Numeric and its extras has been 
saddled on people who don't have (current) interest in them.

That said, it might be a better use of energy to make the build process 
of a lite scipy easier and standardized.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From konrad.hinsen at laposte.net  Fri Feb  4 01:13:14 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb  4 01:13:14 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4202FE20.8040302@ucsd.edu>
References: <420280AD.8080309@ee.byu.edu> <4202FE20.8040302@ucsd.edu>
Message-ID: <35b6b589e88d3db534f8bf07f8c75dae@laposte.net>

On 04.02.2005, at 05:46, Robert Kern wrote:

> I think this is the point that some people are missing. Numeric3 is  
> intended to be the multiarray object proposed to go into the standard  
> library. FFTs

Wasn't numarray as well?


> So for people who need LinearAlgebra, et al., their current situation  
> is this:
>
>   ThirdPartyPackage depends on Numeric and a standard Python
>   distribution.
>
> Their new situation would be this:
>
>   ThirdPartyPackage depends on NumericExtras and a standard Python
>   distribution.

Fine. But before then, Numeric3 will get a lot less testing than it  
should if it can be used only by power-users who write all their  
scientific code themselves.

> If you need to depend on these modules with their current interfaces,  
> then someone needs to step up to support it. This support probably  
> must come from the people who need the current interfaces. For far too  
> long, the responsibility of maintaining Numeric and its extras has  
> been saddled on people who don't have (current) interest in them.

There is no responsibility at all in the OpenSource world. I am not  
criticizing the idea of making Numeric3 a subset of current Numeric, I  
am just pointing out that it will be much less used (and tested) as a  
result.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Fri Feb  4 01:19:12 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb  4 01:19:12 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4202F6A0.3@ims.u-tokyo.ac.jp>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
Message-ID: <1f7baacd417e55debdde0351e463cf10@laposte.net>

On 04.02.2005, at 05:14, Michiel Jan Laurens de Hoon wrote:

> give you an example from my own field (computational biology). I am  
> one of the maintainers of Biopython, which uses LinearAlgebra and  
> RandomArray. Many of our users are not very familiar with Python. Even  
> installing Numerical Python sometimes causes problems, and I'm sure we  
> have lost users in the past because of that. SciPy, in my experience,

My experience with users of MMTK, nMOLDYN, and DomainFinder is exactly  
the same. Installation problems are the #1 source for support  
questions, in particular under Windows and Irix.

There are a couple of nice things I could use and recommend to my  
users, e.g. SciPy or Atlas-accelerated LinearAlgebra. I don't because I  
know it will generate more support questions, which are already eating  
up too much of my time. I don't use features of very recent Python  
versions for the same reason.

> But seriously, I think that there are advantages to Numerical Python  
> and to SciPy, and they both have a reason to exist. Numerical Python  
> has the

Exactly.

> So I would suggest the following demarcation line between Numerical  
> Python and SciPy:
>
> Stuff that is needed by lots of users (linear algebra, FFT, random  
> numbers, special functions) and is available in ANSI-C (so its  
> installation is straightforward and won't cause problems to users who  
> don't need it) should go into Numerical Python.
>
> Stuff that is needed by fewer users or is not available in ANSI-C  
> should go into SciPy. SciPy, with its binary installers, can provide a  
> real benefit to users who need extension modules written in  
> Fortran/C++/whatnot.

That sounds like a good division line.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From Vincent.Picavet at inria.fr  Fri Feb  4 01:38:59 2005
From: Vincent.Picavet at inria.fr (Vincent Picavet)
Date: Fri Feb  4 01:38:59 2005
Subject: [Numpy-discussion] Load numarray with specific shape cause error
Message-ID: <20050204103751.20ee5979@hazard.inria.fr>

Hi,
Here is what I launch in the Python interpreter :
----
import numarray
simulation_file = "/path/to/my/file.bin"
a = numarray.fromfile(simulation_file, 'Float32',(3144, 33, 65))
a.sum() 
----

And here I get the following error :

----
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File
"/nfs/data2/picavet/workspace/softs/python/lib/python2.4/site-packages/
numarray/numarraycore.py", line 1133, in sum    return
ufunc.add.reduce(ufunc.add.areduce(self, type=type).flat, type=type)
libnumarray.error: add_Float64_reduce: access beyond buffer.
offset=25151 buffersize=17160
----

Just after that, I execute :
----
a.sum()
588108255.50844395
a.mean()
87.206215933326803
----

... and as you can see I got no error the second time.
The binary file I try to load is a raw file of 3144*33*64
4-Bytes float elements.

In fact I just saw that if I use the shape (65,33,3144) when calling
fromfile(), everything works just fine. And after having called a.sum()
the first time, the shape of the array has been changed.

----
>>> a.getshape()
(3144, 65, 33)
>>> a.sum()
Traceback (most recent call last):
[snip, same as above]
>>> a.getshape()
(33, 65, 3144)
----

Any idea why I can't load an array with such a shape ?

I use Python 2.4 and the latest version of numarray.
Python 2.4 (#1, Jan 27 2005, 15:21:15)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2

Thanks,
Vincent



-- 
Vincent Picavet,		Tel : +33 (0) 1 39 63 52 14	
INRIA Rocquencourt,		E-mail : vincent.picavet at inria.fr
Projet CLIME			http://www-rocq.inria.fr/clime	



From jmiller at stsci.edu  Fri Feb  4 03:01:50 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Fri Feb  4 03:01:50 2005
Subject: [Numpy-discussion] Load numarray with specific shape cause
	error
In-Reply-To: <20050204103751.20ee5979@hazard.inria.fr>
References: <20050204103751.20ee5979@hazard.inria.fr>
Message-ID: <1107514898.4962.18.camel@jaytmiller.comcast.net>

On Fri, 2005-02-04 at 10:37 +0100, Vincent Picavet wrote:
> Hi,
> Here is what I launch in the Python interpreter :
> ----
> import numarray
> simulation_file = "/path/to/my/file.bin"
> a = numarray.fromfile(simulation_file, 'Float32',(3144, 33, 65))
> a.sum() 
> ----
> 
> And here I get the following error :
> 
> ----
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File
> "/nfs/data2/picavet/workspace/softs/python/lib/python2.4/site-packages/
> numarray/numarraycore.py", line 1133, in sum    return
> ufunc.add.reduce(ufunc.add.areduce(self, type=type).flat, type=type)
> libnumarray.error: add_Float64_reduce: access beyond buffer.
> offset=25151 buffersize=17160
> ----
> 
> Just after that, I execute :
> ----
> a.sum()
> 588108255.50844395
> a.mean()
> 87.206215933326803
> ----
> 
> ... and as you can see I got no error the second time.
> The binary file I try to load is a raw file of 3144*33*64
> 4-Bytes float elements.
> 
> In fact I just saw that if I use the shape (65,33,3144) when calling
> fromfile(), everything works just fine. And after having called a.sum()
> the first time, the shape of the array has been changed.
> 
> ----
> >>> a.getshape()
> (3144, 65, 33)
> >>> a.sum()
> Traceback (most recent call last):
> [snip, same as above]
> >>> a.getshape()
> (33, 65, 3144)
> ----
> 
> Any idea why I can't load an array with such a shape ?

Yes.  You're hitting a known bug which I'm working on now in preparation
for numarray-1.2.  

In a nutshell,  sum() exposes surprisingly complex code in numarray in
order to use higher precision without making a temporary copy of the
array.  This should be fixed when numarray-1.2 is released,  hopefully
very soon.

For now,  there are two work arounds:

# use simpler "same type" summing code at lower precision
a.sum('Float32')  

# create a higher precision temporary and sum that
a.astype('Float64').sum()  

Regards,
Todd




From jmiller at stsci.edu  Fri Feb  4 04:24:59 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Fri Feb  4 04:24:59 2005
Subject: [Numpy-discussion] Numeric3 and speed of new-style classes
In-Reply-To: <42029763.9090700@ee.byu.edu>
References: <42029763.9090700@ee.byu.edu>
Message-ID: <1107519864.4962.74.camel@jaytmiller.comcast.net>

On Thu, 2005-02-03 at 14:28 -0700, Travis Oliphant wrote:
> 
> I fixed a major oversight in Numeric3 and now it works pretty well (it 
> is basically old Numeric with a bunch of new types, no math (yet)), 
> using new-style classes, and some checks for alignment when 
> dereferencing data.
> 
> I did some simple array creation timing tests.  An example of the 
> results is given below.  The bottom line is that new-style classes do 
> nothing but speed up (a little bit) basic object creation. 

I think you've done a real service here,  bounding the cost of new-style
classes,  but I'm not convinced we have a bottom line,  at least not in
the sense of my own FUD about new style classes.  I'm not contesting the
technical direction you're taking with Numeric3 (I hope it works,  but
the details are up to you),  but I am contesting the idea that new style
classes do nothing but speed up object creation.

I think among the promises of new style classes are the ability to
subclass in both C and Python,  and my thinking is that if you actually
do *that*,  you pay for it through the nose.  So,  I'm trying to dispell
the notion that new style classes are pure bliss;  I think they need to
be used either with insight (say a lot more than C++ OOP) or sidestepped
altogether (as, IMHO, you have done).  Using them,  you're up against
more than just faking OOP in C.

I think one of the flaws currently in numarray is that I tried to push a
Python class hierarchy down (or half-way down) into C.   IMO,  that's
where some of the cost of new style classes would be incurred.  The cost
of the object __dict__ I mostly expected to be optimized out,  perhaps
by freelist magic or some lazy construction technique.  If it's the
latter.

As far as state in the arrayobject goes, I *did* flatten the numarray
class heirarchy.  But I didn't flatten the numarray class hierarchy in
terms of constructors and methods,  and my sense is that there is a
significant cost there.   Calling up the class hierarchy uses Python
method calls (maybe an error) which themselves require the construction,
parsing, and destruction of tuple objects and at least some of the
objects the tuples contain.  

So,  I'd assert that unless you solve *that* problem,  using new style
classes and some form of inheritance,  saying that new style classes do
nothing but speed things up is stretching the truth.  To really disprove
the hidden costs in my FUD,  I'd want to see a C basetype and
inheritance by a Python subclass.

If you actually use them for anything,  I think you wind up paying
Python function call overheads.  That, anyway,  is a problem to solve
(or sidestep as you have done) for numarray.

AFICT, among the things you're doing with Numeric3 (which I applaud) is
basically throwing out inheritance from the fundamental array type;
squashing all the constructor functionality for NDArray and NumArray
into one type was more than I had the nerve to do.  So, I know you're
adding the ability to subclass,  but you're not actually using it,  so I
think it's stretching the truth to say it speeds things up.

In numarray,  we use it,  in both C and Python,  and that's maybe not a
good thing.  

If you can pull it off and still cleanly support the features of
numarray,  it will be a triumph of KISS.  But that's not a given.






From jmiller at stsci.edu  Fri Feb  4 06:07:05 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Fri Feb  4 06:07:05 2005
Subject: [Numpy-discussion] Just ignore me.  That was an accidental send.
In-Reply-To: <1107519864.4962.74.camel@jaytmiller.comcast.net>
References: <42029763.9090700@ee.byu.edu>
	 <1107519864.4962.74.camel@jaytmiller.comcast.net>
Message-ID: <1107525958.2505.23.camel@halloween.stsci.edu>

Please ignore that last post.   It consists of unfinished thoughts which
were sent accidentally.  Doing numarray isn't easy.

Regards,
Todd

On Fri, 2005-02-04 at 07:24, Todd Miller wrote:
> On Thu, 2005-02-03 at 14:28 -0700, Travis Oliphant wrote:
> > 
> > I fixed a major oversight in Numeric3 and now it works pretty well (it 
> > is basically old Numeric with a bunch of new types, no math (yet)), 
> > using new-style classes, and some checks for alignment when 
> > dereferencing data.
> > 
> > I did some simple array creation timing tests.  An example of the 
> > results is given below.  The bottom line is that new-style classes do 
> > nothing but speed up (a little bit) basic object creation. 
> 
> I think you've done a real service here,  bounding the cost of new-style
> classes,  but I'm not convinced we have a bottom line,  at least not in
> the sense of my own FUD about new style classes.  I'm not contesting the
> technical direction you're taking with Numeric3 (I hope it works,  but
> the details are up to you),  but I am contesting the idea that new style
> classes do nothing but speed up object creation.
> 
> I think among the promises of new style classes are the ability to
> subclass in both C and Python,  and my thinking is that if you actually
> do *that*,  you pay for it through the nose.  So,  I'm trying to dispell
> the notion that new style classes are pure bliss;  I think they need to
> be used either with insight (say a lot more than C++ OOP) or sidestepped
> altogether (as, IMHO, you have done).  Using them,  you're up against
> more than just faking OOP in C.
> 
> I think one of the flaws currently in numarray is that I tried to push a
> Python class hierarchy down (or half-way down) into C.   IMO,  that's
> where some of the cost of new style classes would be incurred.  The cost
> of the object __dict__ I mostly expected to be optimized out,  perhaps
> by freelist magic or some lazy construction technique.  If it's the
> latter.
> 
> As far as state in the arrayobject goes, I *did* flatten the numarray
> class heirarchy.  But I didn't flatten the numarray class hierarchy in
> terms of constructors and methods,  and my sense is that there is a
> significant cost there.   Calling up the class hierarchy uses Python
> method calls (maybe an error) which themselves require the construction,
> parsing, and destruction of tuple objects and at least some of the
> objects the tuples contain.  
> 
> So,  I'd assert that unless you solve *that* problem,  using new style
> classes and some form of inheritance,  saying that new style classes do
> nothing but speed things up is stretching the truth.  To really disprove
> the hidden costs in my FUD,  I'd want to see a C basetype and
> inheritance by a Python subclass.
> 
> If you actually use them for anything,  I think you wind up paying
> Python function call overheads.  That, anyway,  is a problem to solve
> (or sidestep as you have done) for numarray.
> 
> AFICT, among the things you're doing with Numeric3 (which I applaud) is
> basically throwing out inheritance from the fundamental array type;
> squashing all the constructor functionality for NDArray and NumArray
> into one type was more than I had the nerve to do.  So, I know you're
> adding the ability to subclass,  but you're not actually using it,  so I
> think it's stretching the truth to say it speeds things up.
> 
> In numarray,  we use it,  in both C and Python,  and that's maybe not a
> good thing.  
> 
> If you can pull it off and still cleanly support the features of
> numarray,  it will be a triumph of KISS.  But that's not a given.
> 
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 




From hyclak at math.ohiou.edu  Fri Feb  4 06:40:44 2005
From: hyclak at math.ohiou.edu (Matt Hyclak)
Date: Fri Feb  4 06:40:44 2005
Subject: [Numpy-discussion] Re: Problems compiling against ATLAS on x86_64
In-Reply-To: <20050203144231.GB24701@math.ohiou.edu>
References: <20050203144231.GB24701@math.ohiou.edu>
Message-ID: <20050204143935.GE24701@math.ohiou.edu>

On Thu, Feb 03, 2005 at 09:42:31AM -0500, Matt Hyclak enlightened us:
> I'm about ready to pull out my hair trying to get this stuff to compile for
> a faculty member. I'm running RHEL 3 AS on a dual opteron machine. ATLAS
> compiles fine as I go through xconfig, but I can't for the life of me get
> Numeric to compile against it. I've tried ATLAS 3.6.0 and 3.7.8. I'm
> trying to build Numeric 23.7. I keep getting the following error:
> 
> gcc -shared build/temp.linux-x86_64-2.2/lapack_litemodule.o
> -L/usr/lib64/atlas -llapack -lcblas -lf77blas -latlas -lg2c -o
> build/lib.linux-x86_64-2.2/lapack_lite.so
> /usr/bin/ld: /usr/lib64/atlas/liblapack.a(dgesv.o): relocation R_X86_64_32
> can not be used when making a shared object; recompile with -fPIC
> /usr/lib64/atlas/liblapack.a: could not read symbols: Bad value
> collect2: ld returned 1 exit status
> 
> I built ATLAS with -fPIC, but that didn't seem to help. Could someone
> *please* point me in the right direction before I start hitting things?
> 

Sorry for replying to myself. Just for the archives, the problem seems to be
the static lapack/blas libraries provided with RHEL3 are not compiled with
-fPIC. I ripped open the rpm and rebuild it with an -fPIC added in and all
at least compiles now. I'll leave it up to my faculty to tell me whether or
not it works :-)

Thanks,
Matt

-- 
Matt Hyclak
Department of Mathematics 
Department of Social Work
Ohio University
(740) 593-1263



From stephen.walton at csun.edu  Fri Feb  4 10:43:20 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Fri Feb  4 10:43:20 2005
Subject: [Numpy-discussion] Re: Problems compiling against ATLAS on x86_64
In-Reply-To: <20050204143935.GE24701@math.ohiou.edu>
References: <20050203144231.GB24701@math.ohiou.edu> <20050204143935.GE24701@math.ohiou.edu>
Message-ID: <4203C213.5020805@csun.edu>

Matt Hyclak wrote:

>Sorry for replying to myself. Just for the archives, the problem seems to be
>the static lapack/blas libraries provided with RHEL3...
>  
>
Common problem which I just posted to scipy-devel about.  In most 
variants of RH and Fedora, the RH-provided lapack RPM is only there to 
satisfy Octave's dependency.  If you're not using Octave, you'll be 
happiest uninstalling both it and the RedHat provided lapack (rpm -e 
lapack octave)





From oliphant at ee.byu.edu  Fri Feb  4 10:47:45 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb  4 10:47:45 2005
Subject: [Numpy-discussion] Just ignore me.  That was an accidental send.
In-Reply-To: <1107525958.2505.23.camel@halloween.stsci.edu>
References: <42029763.9090700@ee.byu.edu>	 <1107519864.4962.74.camel@jaytmiller.comcast.net> <1107525958.2505.23.camel@halloween.stsci.edu>
Message-ID: <4203C31A.8020607@ee.byu.edu>

Todd Miller wrote:

>Please ignore that last post.   It consists of unfinished thoughts which
>were sent accidentally.  Doing numarray isn't easy.
>  
>
Unfinished or not, there were some good comments here, worth hearing. 

>>I think you've done a real service here,  bounding the cost of new-style
>>classes,  but I'm not convinced we have a bottom line,  at least not in
>>the sense of my own FUD about new style classes.  I'm not contesting the
>>technical direction you're taking with Numeric3 (I hope it works,  but
>>the details are up to you),  but I am contesting the idea that new style
>>classes do nothing but speed up object creation.
>>    
>>
Yes, your point about subclassing new-types causing speed slow downs is 
a very valid one and that has not been tested at all.  But, for my 
purposes, I was just interested to know if allowing a record Array to 
subclass from Numeric3 would cause slow-downs to the rest of Numeric.   
It appears that this will not be the case.  Clearly any type that 
subclasses from Numeric3 might still have speed issues, but those were 
less of a concern for me, right now.

>>So,  I'd assert that unless you solve *that* problem,  using new style
>>classes and some form of inheritance,  saying that new style classes do
>>nothing but speed things up is stretching the truth.  To really disprove
>>the hidden costs in my FUD,  I'd want to see a C basetype and
>>inheritance by a Python subclass.
>>    
>>
Sure.  But, my hope is that only RecordArrays would need to subclass in 
Python (or potentially in C).   These might incur some creation 
"slowdown" but it would be inconsequential to the rest of Numeric.

>>If you can pull it off and still cleanly support the features of
>>numarray,  it will be a triumph of KISS.  But that's not a given.
>>
>>    
>>
Clearly it's not a given, but I have high hopes.  I really think it can 
be done. 

-Travis




From oliphant at ee.byu.edu  Fri Feb  4 11:20:44 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb  4 11:20:44 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1f7baacd417e55debdde0351e463cf10@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>
Message-ID: <4203CAD0.6050609@ee.byu.edu>

konrad.hinsen at laposte.net wrote:

> On 04.02.2005, at 05:14, Michiel Jan Laurens de Hoon wrote:
>
>> give you an example from my own field (computational biology). I am  
>> one of the maintainers of Biopython, which uses LinearAlgebra and  
>> RandomArray. Many of our users are not very familiar with Python. 
>> Even  installing Numerical Python sometimes causes problems, and I'm 
>> sure we  have lost users in the past because of that. SciPy, in my 
>> experience,
>
>
> My experience with users of MMTK, nMOLDYN, and DomainFinder is 
> exactly  the same. Installationproblems are the #1 source for support  
> questions, in particular under Windows and Irix.
>
O.K.   I can see that there are several out there who belive that SciPy 
is sufficiently hard to install that they are concerned about requiring 
it for their math-using packages (despite the provided binary 
distributions, and the work that continues on making it easier to 
install).   I'm very glad to hear these comments, as I constantly wonder 
why people seem to be duplicating SciPy's efforts instead of helping 
SciPy.   I really would like to help make SciPy into something that many 
people can use.  We have done a lot of work in trying to make SciPy 
modular, just for this reason.

While I can appreciate that installation issues are a major headache 
(one reason I support the idea of selling binary-only copies to 
customers), I think that this is an issue no matter what packages you 
decide to deliver, so I don't think it negates my comments that we 
should package numeric routines under the same framework (right now, 
since SciPy already exists, why not there?)

I'm glad to hear at least one specific regarding installation, and that 
is the use of FORTRAN code in SciPy.   I would be very interested to 
know which platforms people have installation difficulties caused by the 
use of FORTRAN.   The biggest problem, I see, is not using FORTRAN, but 
trying to support all the different FORTRAN compilers that might get 
used.  Pearu has done a magnificient job in this difficult area, 
already.   

I very much want to get at the root of why people are staying aloof from 
SciPy, and fix the problems if we can.  SciPy is very much an open 
project.  It is what it is because of the people involved.   Those of us 
developing it so far are pretty easy going and are willing to make 
changes to make others more satisfied with the result.

>> So I would suggest the following demarcation line between Numerical  
>> Python and SciPy:
>>
>> Stuff that is needed by lots of users (linear algebra, FFT, random  
>> numbers, special functions) and is available in ANSI-C (so its  
>> installation is straightforward and won't cause problems to users 
>> who  don't need it) should go into Numerical Python.
>>
If this is an important distinction, why not just place this division in 
SciPy itself?   SciPy's goal is not to be "hard to install", or "only 
for power users."    If these installation issues are real and cannot be 
fixed with better installers, then those of us at SciPy would be very 
glad to see an "easy-to-install" scipy sub-package, that fits into a 
single framework.   

In other words,  why aren't you helping make SciPy better, instead of 
just re-creating what it is doing.  Are you really saying that "scipy is 
beyond hope for me to help?"      Even if you can't get in and change 
SciPy, offering concrete suggestions such as  the one just offered 
"FORTRAN code is too hard to install and so there should be subset of 
SciPy in ANSI-C only" is great.   Then, we can discuss this issue, 
really get to the root of it, and fix the problem.

It would appear that so few actually want to do that.  I actually 
understand, because it sometimes means giving up a treasured notion that 
doesn't fit into the picture, and this happens to me over and over 
again.    Perhaps it is unavoidable.

In sum, I would prefer to see people rally behind SciPy and fix it to be 
a package (or collection of packages) that all can use.  If that doesn't 
happen, I certainly won't fight people who want to make Numeric Packages 
instead.   It just will not be a high priority for me and so won't 
appear in my workflow. 

-Travis






From Fernando.Perez at colorado.edu  Fri Feb  4 11:39:03 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri Feb  4 11:39:03 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203CAD0.6050609@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu>
Message-ID: <4203CF33.3090309@colorado.edu>

Travis Oliphant wrote:

> O.K.   I can see that there are several out there who belive that SciPy 
> is sufficiently hard to install that they are concerned about requiring 
> it for their math-using packages (despite the provided binary 
> distributions, and the work that continues on making it easier to 
> install).   I'm very glad to hear these comments, as I constantly wonder 
> why people seem to be duplicating SciPy's efforts instead of helping 
> SciPy.   I really would like to help make SciPy into something that many 
> people can use.  We have done a lot of work in trying to make SciPy 
> modular, just for this reason.

I can only speak for a linux (specifically Fedora3) environment.  But I'd 
venture that much of this fear is based on the old lore of epic battles fought 
by many (myself included) against scipy cvs.  It is also true that _very 
recently_ there have been the many -no-f2c problems, due to the complexities 
of handling this flag.

An aside for Fedora3 users, the blas/lapack supplied packages are broken:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=138683

Specifically, see my comment 15 there.   As of today (2005/02/04), redhat has 
completely ignored this.  If you want a functional default (non-atlas) lapack, 
you must either build it yourself or grab it from rawhide, at least the -28 
revision.

However, as we've discussed recently on the scipy list, with current CVS, and 
my scripts for rpm-building out of Pearu's binary ATLAS/lapack packages, 
building a fully installable, sub-architecture-specific RPM for scipy requires 
the following:

/path/to/scipy-cvs/ >  python setup.py bdist_rpm --release=$YOUR_ARCH_TAG

That's it.  Period.  One single command, and you get a properly tagged RPM 
which can be handled automatically on a network of machines with multiple 
architectures (you obviously will have to build one for each arch).

What I'm trying to say is that, thanks to all the recent work on this problem, 
its MUCH EASIER than most people think, at least under linux.

For Win32, Joe Cooper's Enthon releases basically solve the problem in one 
clean swoop, and they include not only scipy, but also a ton of other stuff, 
some of which is _a lot_ harder to build than scipy (ever built VTK?).

For OSX, I understand that R. Kern is working on a MacEnthon, which will 
hopefully solve the problem in a similar way.

So I fully agree with Travis that it would probably take a lot less work, and 
benefit our overall combined goals much more, if people spent a bit of their 
time in helping solve the install/distribution issues at the scipy core rather 
than reinventing their own side packages.  That's been my approach, and Pearu 
has been _extremely_ receptive to all my patches and requests, to the point 
where now scipy flies in autopilot for me, even keeping it up to date with raw 
CVS.

Just my $.02

Cheers,

f



From perry at stsci.edu  Fri Feb  4 12:01:38 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Fri Feb  4 12:01:38 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203CAD0.6050609@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu>
Message-ID: <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>

On Feb 4, 2005, at 2:19 PM, Travis Oliphant wrote:

> konrad.hinsen at laposte.net wrote:
>
>> On 04.02.2005, at 05:14, Michiel Jan Laurens de Hoon wrote:
>>
>>> give you an example from my own field (computational biology). I am  
>>> one of the maintainers of Biopython, which uses LinearAlgebra and  
>>> RandomArray. Many of our users are not very familiar with Python. 
>>> Even  installing Numerical Python sometimes causes problems, and I'm 
>>> sure we  have lost users in the past because of that. SciPy, in my 
>>> experience,
>>
>>
>> My experience with users of MMTK, nMOLDYN, and DomainFinder is 
>> exactly  the same. Installationproblems are the #1 source for support 
>>  questions, in particular under Windows and Irix.
>>
> O.K.   I can see that there are several out there who belive that 
> SciPy is sufficiently hard to install that they are concerned about 
> requiring it for their math-using packages (despite the provided 
> binary distributions, and the work that continues on making it easier 
> to install).   I'm
[...]
This sort of leads back to the original point, that is, putting 
something in the Python Standard Library. I suspect that for most 
scientists and engineers if they have to install anything extra at all 
to use arrays practially, they really don't care if it is in the core 
or not. The current Numeric (or numarray) is a fairly straightforward 
install now and if the issue is having users avoid this extra install, 
putting a reduced functionality array package in the core is not going 
to address this issue. On the other hand, it is likely to have the 
general Python community more likely to use arrays as an interchange 
format, which is a separate benefit (a point I stressed at the last 
scipy conference).

In other words, we should be clear about what should go into the core 
and what the sought-after benefit is. I don't think that has been done 
yet. If scipy is to be the solution for all scientists and engineers, 
why should Numeric (or numarray) go into the core? They still will have 
to install scipy (or some extra package) anyway.

Perry




From konrad.hinsen at laposte.net  Fri Feb  4 12:08:07 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb  4 12:08:07 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203CAD0.6050609@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu>
Message-ID: <55c095f595dfec3e16579094faea14f2@laposte.net>

On Feb 4, 2005, at 20:19, Travis Oliphant wrote:

> O.K.   I can see that there are several out there who belive that 
> SciPy is sufficiently hard to install that they are concerned about 
> requiring it for their math-using

It's not just belief, it's my own experience, admittedly not very 
recent (I last checked about a year ago).

> packages (despite the provided binary distributions, and the work that 
> continues on

For Windows and Debian - that covers none of the machines I have ever 
used in my scientific career. Moreover, my experience with binary 
installations for Linux in general is not so good. Hardly anyone uses 
an unmodified standard distribution, there's nearly always something 
that doesn't fit.

> While I can appreciate that installation issues are a major headache 
> (one reason I support the idea of selling binary-only copies to 
> customers), I think that this is an issue no matter what packages you 
> decide to deliver, so I don't think it negates my

In principle, yes, but there are significant differences in "mean 
expected trouble". Some libraries are moving targets (GTK comes to 
mind), and dependencies on them are a big source of fragility. With 
only C code and no external dependencies, there is rarely ever a 
problem, and that's why Numeric is a low-risk package.

There's also integration with other stuff to be considered. Take the 
installation instructions for MaxOS X for example. They start by 
recommending not to use the Fink package. Good advice, because I never 
got it to work. Fink has enormous package interdependencies, so usually 
you need to install or upgrade 10 or more packages, and the chance for 
one of them not installing correctly is quite high.

But then, following the instructions puts SciPy with the Apple-provided 
python. Fine if SciPy is all you want, but I have dozens of other 
Python packages installed for the Fink Python, which in general is the 
better choice because there is a very complete set of easy-to-install 
Python packages from Fink.

The net result is that I still don't have SciPy on my Mac, which is my 
main development machine.

> people have installation difficulties caused by the use of FORTRAN.   
> The biggest problem, I see, is not using FORTRAN, but trying to 
> support all the different FORTRAN compilers that might get used.  
> Pearu has done a magnificient job in

There's also the need to have a Fortran compiler. Installing g77 is not 
exactly an easy task for the average user.

> If this is an important distinction, why not just place this division 
> in SciPy itself?   SciPy's goal is not to be "hard to install", or 
> "only for power users."    If these installation issues are real and 
> cannot be fixed with better installers, then those of us at SciPy 
> would be very glad to see an "easy-to-install" scipy sub-package, that 
> fits into a single framework.

That sounds like a very good idea.

> In other words,  why aren't you helping make SciPy better, instead of 
> just re-creating what it is doing.  Are you really saying that "scipy 
> is beyond hope for me to

Because in the short run that's the path of least effort, or least 
pain: I actually like coding, but I strongly dislike sorting out 
installation problems. I dislike even more sorting out someone else's 
installation problems, which is what I would have to do if my published 
code depended on SciPy.

This is in fact not just a SciPy issue. Installation problems and the 
fragility of package interdependencies are the #1 problem in the 
OpenSource world in my opinion.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From oliphant at ee.byu.edu  Fri Feb  4 14:23:08 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb  4 14:23:08 2005
Subject: [Numpy-discussion] Optimization question for ufuncs
Message-ID: <4203F5A3.3020402@ee.byu.edu>

I've been thinking lately about ufuncs and I would love to hear the 
opinion of others.

I like what numarray has done with the temporary buffer ideas so that 
full copies are never made if they are just going to be thrown away.   
This has led to other thoughts about possible improvements to the ufunc 
object to support "ufunc chaining" so that array operations on 
expressions don't have to create any temporary copies (using buffers 
instead) --- I think I remember the numarray guys thinking along these 
lines as well.

Regardless, there is always an inner for loop (for each type) that 
performs the requested operation.   The question I have is whether to 
assume unit strides for the inner loop.  The current Numeric ufunc inner 
loops allow for discontiguous memory to be accessed during the loop 
(non-unit strides).   I'm not sure what numarray does, I think it only 
allows for unit strides and uses temporary buffers to support 
discontiguous arrays.

Is this requirement for unit-strides on the inner loop a good one?  Does 
it allow faster code to be compiled?   Is it part of the reason that 
numarray is a little faster on large arrays?

I am not an optimization expert, though I've read a bit as of late.  I'm 
just wondering what the experts on this list think about unit-strides 
versus non unit-strides on the inner loop?

Thanks,

Travis O.




From oliphant at ee.byu.edu  Fri Feb  4 14:32:06 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb  4 14:32:06 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
Message-ID: <4203F7CE.8070403@ee.byu.edu>

>> O.K.   I can see that there are several out there who belive that 
>> SciPy is sufficiently hard to install that they are concerned about 
>> requiring it for their math-using packages (despite the provided 
>> binary distributions, and the work that continues on making it easier 
>> to install).   I'm
>
> [...]
> This sort of leads back to the original point, that is, putting 
> something in the Python Standard Library. I suspect that for most 
> scientists and engineers if they have to install anything extra at all 
> to use arrays practially, they really don't care if it is in the core 
> or not. The current Numeric (or numarray) is a fairly straightforward 
> install now and if the issue is having users avoid this extra install, 
> putting a reduced functionality array package in the core is not going 
> to address this issue. On the other hand, it is likely to have the 
> general Python community more likely to use arrays as an interchange 
> format, which is a separate benefit (a point I stressed at the last 
> scipy conference).
>
As Perry suggested, I personally don't care whether it is in the Python 
core or not.  The only reason I ever wanted to put it into the core is 
because of the PIL.  When I asked the creator of the PIL, why he did not 
use Numeric Python, his response was that it wasn't in the core.    
Right now, I think we would benefit if the basic multidimensional array 
object were in the core.  It would be nicer if we could agree on how 
basic operations were to behave so math could be done with them, and put 
that in there as well.

Perry makes a great point about the benefit of using arrays as an 
interchange format...  There are people out there who use the array  
object in Python for just that.   It would be great if there was one 
multidimensional array object used by the entire Python community for 
applications requiring such a structure, but it wouldn't remove the need 
for "more stuff" to do science and so my interest in getting such an 
object into the Python core has waned considerably over the years. 

I'm more interested now in uniting a fractured community (or at least 
making interoperability easier). 

-Travis








From Chris.Barker at noaa.gov  Fri Feb  4 14:53:45 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Fri Feb  4 14:53:45 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <55c095f595dfec3e16579094faea14f2@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <55c095f595dfec3e16579094faea14f2@laposte.net>
Message-ID: <4203FCD8.8070204@noaa.gov>

konrad.hinsen at laposte.net wrote:

>> SciPy is sufficiently hard to install that they are concerned about 
>> requiring it for their math-using

> It's not just belief, it's my own experience,

And mine. I love the idea of SciPy, and have been following the project 
since the beginning, but have never really used it. I've tried a number 
of times, and it's been a pain.

> admittedly not very recent 

me neither.

> For Windows and Debian - that covers none of the machines

I use Gentoo and OS-X. I'm used to weird installs (or I wouldn't use 
Gentoo), but my colleagues all use OS-X, and It I can't give them an 
easy installer, forget it.

 > My experience with binary
> installations for Linux in general is not so good.

Mine neither, but source installs are far easier there than any other 
system! Probably because a lot of developers use Linux, but also because 
it's a very developer and open-source friendly system. Right now I'm 
working on making a binary for matplotlib on OS-X. No one would have a 
Linux box without libpng and libfreetype, but OS-X doesn't have those 
out of the box.

> In principle, yes, but there are significant differences in "mean 
> expected trouble". Some libraries are moving targets (GTK comes to 
> mind), and dependencies on them are a big source of fragility.

Exactly, and things get weird with overlapping dependencies. PyGTK 
requires X11, which has libfreetype. matplotlib requires freetype, but 
can be used with or without pyGTK....
> With only 
> C code and no external dependencies, there is rarely ever a problem, and 
> that's why Numeric is a low-risk package.

Except that it's setup.py has been broken for the last few releases. 
That's been a problem for a number of users.

By the way, here OS-X is much better than other systems, it comes with 
lapack built in! (Veclib)

 > Fine if SciPy is all you want, but I have dozens of other Python
> packages installed for the Fink Python, which in general is the better 
> choice because there is a very complete set of easy-to-install Python 
> packages from Fink.

Actually, I disagree. Apple's Python is a better choice (you can't run 
Aqua applications from a fink python, at least I don't think so), but we 
do need to do more to provide easy to install packages for it. (there 
are a lot)

Here Fink drives me crazy. It's a system-within-a-system. I want 
everything on my Mac to fit in well with my Mac. If there was a 
fink-like system (I haven't yet tried darwinports or Gentoo) that didn't 
duplicate Apples supplied stuff with incompatible versions (Python, 
X11), I'd be much happier.

> The net result is that I still don't have SciPy on my Mac

Me neither, though I don't think it's SciPy's fault. The core problem is 
  that there aren't all that many people using OS-X developing Python 
packages, and Apple has, of course, made OS-X just different enough from 
other unices to require some work (or take the easy way and use fink, 
but then it doesn't play with the rest of the Mac). Ironically, this 
should be even harder on Windows, but there are enough Windows users 
that someone usually gets around making a binary.

>> people have installation difficulties caused by the use of FORTRAN.   
>> The biggest problem, I see, is not using FORTRAN, but trying to 
>> support all the different FORTRAN compilers that might get used.

What's the distinction? Again, Apple doesn't provide f77. That's been a 
stopper for me.

> Because in the short run that's the path of least effort, or least pain: 

This is a key issue with all open-source projects (at least ones that 
aren't funder). We all do the easiest thing to get the functionality we 
need. If we combined all our duplicate efforts, we'd be in great shape, 
but an any given moment, I mostly need to get my job done, not make a 
better SciPy, or whatever.

> This is in fact not just a SciPy issue. Installation problems and the 
> fragility of package interdependencies are the #1 problem in the 
> OpenSource world in my opinion.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From Chris.Barker at noaa.gov  Fri Feb  4 14:58:56 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Fri Feb  4 14:58:56 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203F7CE.8070403@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu> <4203F7CE.8070403@ee.byu.edu>
Message-ID: <4203FDFF.4060904@noaa.gov>

Travis Oliphant wrote:
> The only reason I ever wanted to put it into the core is 
> because of the PIL.  When I asked the creator of the PIL, why he did not 
> use Numeric Python, his response was that it wasn't in the core.    

Same with wxPython. If you draw a line with wxPython, and pass in a 
(N,2) array for the point coordinates, it's slower than a list of 
tuples. With numarray, a LOT slower. If the core multiarray type were in 
the standard lib, wxPython would have code to use them and be much 
faster for this kind of thing.

> Perry makes a great point about the benefit of using arrays as an 
> interchange format...  There are people out there who use the array  
> object in Python for just that.   It would be great if there was one 
> multidimensional array object used by the entire Python community for 
> applications requiring such a structure, but it wouldn't remove the need 
> for "more stuff" to do science

No, but I'd also love to see at least the basic math operations in the 
core too. There are so many questions on c.l.p (and other places) that 
just cry out for a NumPy solution!

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From verveer at embl.de  Fri Feb  4 16:44:50 2005
From: verveer at embl.de (Peter Verveer)
Date: Fri Feb  4 16:44:50 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203CAD0.6050609@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu>
Message-ID: <3ba3b109f1f344e31b758074cbc7f64c@embl.de>

On Feb 4, 2005, at 8:19 PM, Travis Oliphant wrote:

> konrad.hinsen at laposte.net wrote:
>
>> On 04.02.2005, at 05:14, Michiel Jan Laurens de Hoon wrote:
>>
>>> give you an example from my own field (computational biology). I am  
>>> one of the maintainers of Biopython, which uses LinearAlgebra and  
>>> RandomArray. Many of our users are not very familiar with Python. 
>>> Even  installing Numerical Python sometimes causes problems, and I'm 
>>> sure we  have lost users in the past because of that. SciPy, in my 
>>> experience,
>>
>>
>> My experience with users of MMTK, nMOLDYN, and DomainFinder is 
>> exactly  the same. Installationproblems are the #1 source for support 
>>  questions, in particular under Windows and Irix.
>>
> O.K.   I can see that there are several out there who belive that 
> SciPy is sufficiently hard to install that they are concerned about 
> requiring it for their math-using packages (despite the provided 
> binary distributions, and the work that continues on making it easier 
> to install).

I have in the past tried to install SciPy on OSX and I had big problems 
with it. Motivated by the recent discussions on the list I tried again 
today, and found it much improved, but still quite involved. It would 
still be a pain to distribute software that has a dependency on SciPy 
to 'average users' that would have trouble installing SciPy. There does 
not seem to be a binary distribution on scipy.org for OS X, which would 
solve that problem. This is not unimportant, since I guess OS X is 
gaining popularity in scientific computing.

>   I'm very glad to hear these comments, as I constantly wonder why 
> people seem to be duplicating SciPy's efforts instead of helping 
> SciPy.

I agree with that, but fact is that the split between Numarray and 
Numeric prevented me from contributing to SciPy. I have developed for 
numarray (the image processing package nd_image, part of numarray, is 
done by me) because I felt numarray was much better then Numeric for 
that application, and that also ruled out SciPy. There seems now to be 
an effort going to either try to merge the two or to bring Numeric up 
to numarray standard. Thats great, and if that works out I will try to 
rewrite my package so it works with both. Would  be happy to contribute 
it to SciPy. But I would make sure it remains a package that can be 
installed separately from  SciPy if required.

> I really would like to help make SciPy into something that many people 
> can use.  We have done a lot of work in trying to make SciPy modular, 
> just for this reason.

Making SciPy modular is in my opinion a very good approach. Why do I 
need to install all of it, if I really  only  need a few parts? If I 
can install only the subset I need, the likelihood that I will have an 
easier job doing that will increase. That would make it also possible 
to have sub-sets that are free of dependencies such as FORTRAN.

> In sum, I would prefer to see people rally behind SciPy and fix it to 
> be a package (or collection of packages) that all can use.  If that 
> doesn't happen, I certainly won't fight people who want to make 
> Numeric Packages instead.   It just will not be a high priority for me 
> and so won't appear in my workflow.

On the other hand, I would think it would be a good idea to write 
software that does not  have dependencies on big packages such as SciPy 
when possible, but only on Numeric/Numarray. That does not exclude them 
from being included in SciPy. It seems to me that it would be a 
desirable property if a package can be installed just with Numeric (or 
numarray), with an appropriate sub-set of SciPy, or come with the big 
SciPy package. I would rather see a "collection of SciPy packages" then 
a "SciPy package".

Just my 2 cents, although currently I am using numarray exclusively, I 
am glad to see that there is some discussion going on to get the main 
python numerical packages to work together.

Peter





From oliphant at ee.byu.edu  Fri Feb  4 17:37:02 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb  4 17:37:02 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <3ba3b109f1f344e31b758074cbc7f64c@embl.de>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>
Message-ID: <42042318.2090704@ee.byu.edu>

>
> I have in the past tried to install SciPy on OSX and I had big 
> problems with it. Motivated by the recent discussions on the list I 
> tried again today, and found it much improved, but still quite 
> involved. It would still be a pain to distribute software that has a 
> dependency on SciPy to 'average users' that would have trouble 
> installing SciPy. There does not seem to be a binary distribution on 
> scipy.org for OS X, which would solve that problem. This is not 
> unimportant, since I guess OS X is gaining popularity in scientific 
> computing.

A binary for OS X is a desireable thing.  I think more of a stink should 
be made, so that one gets provided by default.  I know that Robert Kern 
has worked on such a thing and it is not out of the question, but is on 
the horizon.   More users asking for it would help.

>>   I'm very glad to hear these comments, as I constantly wonder why 
>> people seem to be duplicating SciPy's efforts instead of helping SciPy.
>
>
> I agree with that, but fact is that the split between Numarray and 
> Numeric prevented me from contributing to SciPy. I have developed for 
> numarray (the image processing package nd_image, part of numarray, is 
> done by me) because I felt numarray was much better then Numeric for 
> that application, and that also ruled out SciPy. There seems now to be 
> an effort going to either try to merge the two or to bring Numeric up 
> to numarray standard. Thats great, and if that works out I will try to 
> rewrite my package so it works with both. Would  be happy to 
> contribute it to SciPy. But I would make sure it remains a package 
> that can be installed separately from  SciPy if required.
>
"What scipy is" is quite open to interpretation.  The idea of scipy is 
just to have a super-package where different packages can live and have 
some kind of inter-operability.   It is made so that sub-packages can be 
easily added and built upon request.   We try hard to make sure that the 
only real dependency is scipy_base which should not be hard to install 
anywhere (does not require ATLAS or FORTRAN). 

It would be great to add nd_image as a sub-package to SciPy.    In fact, 
nd_image was one of the things that most alarmed me about the emerging 
split in the community, as it reproduces some of what is already in 
SciPy (N-D convolution)  while adding some great things that I've always 
wanted in SciPy (morphology).

I believe scipy can be easily installed as a collection of packages if 
that is desired.  Pearu has done an incredible job with scipy_disutils 
to make that process simple.   Making a single super package helps users 
see that they have only a part of what's available (but maybe all they 
need), and you get the added benefit of standard documentation 
efforts.    Nobody has taken up the ball to actually try and distribute 
individual packages of scipy, but it is a welcome effort.   
www.scipy.org has places where such packages could live.  Enthought has 
mainly been interested in distributing "batteries-included" packages for 
windows because it is what their customers need.  Other are welcome to 
pick up the slack.

Perhaps it would be good to get a list of which platforms actually need 
binaries for people to look more favorably at SciPy.   I can see that OS 
X is on the top of that list, surely somebody could distribute a copy of 
a binary for that platform (I know of people who have it installed).     
Also, scipy is still version 0.3 not because the code is unstable but 
because the layout of the packages is still open to change.  Come and 
discuss your visions of what it should look like with us on the 
scipy-dev at scipy.net  mailing list.

> Making SciPy modular is in my opinion a very good approach. Why do I 
> need to install all of it, if I really  only  need a few parts? If I 
> can install only the subset I need, the likelihood that I will have an 
> easier job doing that will increase. That would make it also possible 
> to have sub-sets that are free of dependencies such as FORTRAN.

>
> On the other hand, I would think it would be a good idea to write 
> software that does not  have dependencies on big packages such as 
> SciPy when possible, but only on Numeric/Numarray. That does not 
> exclude them from being included in SciPy. It seems to me that it 
> would be a desirable property if a package can be installed just with 
> Numeric (or numarray), with an appropriate sub-set of SciPy, or come 
> with the big SciPy package. I would rather see a "collection of SciPy 
> packages" then a "SciPy package".

I'd like to see more of the development work that goes on happen under a 
common standard, that is all.  SciPy has done a lot of work to provide 
an infrastructure within which to distribute packages.  The field is 
built, but people don't seem to want to come.    I understand that the 
Numeric / Numarray dichotomy is a big source of this problem.  That is 
the whole reason I'm devoting a big portion of my time to the problem 
right now.  

My very ambitious goal with Numeric3 is to replace both Numeric and 
Numarray (heavily borrowing code from each).  When I say replace them, 
I'm talking about the array object and umath module.  I'm not talking 
about the other packages that use them.  I want those to live under a 
common standard (i.e. scipy).   It would be fine with me if that common 
standard include a scipy_lite or something like that which was easy to 
install, if that is desired.   But, if the problem is really just 
availability of binary copies, then why don't we solve that instead of 
worrying about installation on platforms that nobody uses.   In 
addition, even though I've developed a lot of scipy, I'm willing to 
throw it all out or alter it considerably in order to satisfy the 
demands that exist for package developers to want to put their weight 
behind something like SciPy.   So, don't equate history with an 
unwillingness to cooperate. 

I think you'll find the people involved with scipy to be incredibly 
willing to cooperate to achieve mutual success.  If you have users of a 
package that demand a certain binary of scipy to be installed, then 
bring them over.  I can almost guarantee you that a copy of scipy will 
be available for them. 

While I do recognize the value of repeated work.  When it comes to 
interoperability, single standards seem to work best.  So, the longer 
this unnecessary division of resources goes on, the farther behind 
Python will get for doing anything but niche work. 

-Travis






From paul at pfdubois.com  Fri Feb  4 20:04:21 2005
From: paul at pfdubois.com (Paul F. Dubois)
Date: Fri Feb  4 20:04:21 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42042318.2090704@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu>
Message-ID: <42044565.1030902@pfdubois.com>

My experience with binary distributions has convinced me that it is a 
hopeless endeavor in the long run. I understand why people want them but 
until you've tried to provide them to the public and seen the number of 
things that go wrong ... Not to mention that it means you have to have 
someone build on all those systems and you don't have them all. And if 
the person is going to add their other extensions, compiled with a 
different compiler, blooie.


Travis Oliphant wrote:
> 
>>
>> I have in the past tried to install SciPy on OSX and I had big 
>> problems with it. Motivated by the recent discussions on the list I 
>> tried again today, and found it much improved, but still quite 
>> involved. It would still be a pain to distribute software that has a 
>> dependency on SciPy to 'average users' that would have trouble 
>> installing SciPy. There does not seem to be a binary distribution on 
>> scipy.org for OS X, which would solve that problem. This is not 
>> unimportant, since I guess OS X is gaining popularity in scientific 
>> computing.
> 
> 
> A binary for OS X is a desireable thing.  I think more of a stink should 
> be made, so that one gets provided by default.  I know that Robert Kern 
> has worked on such a thing and it is not out of the question, but is on 
> the horizon.   More users asking for it would help.
> 
>>>   I'm very glad to hear these comments, as I constantly wonder why 
>>> people seem to be duplicating SciPy's efforts instead of helping SciPy.
>>
>>
>>
>> I agree with that, but fact is that the split between Numarray and 
>> Numeric prevented me from contributing to SciPy. I have developed for 
>> numarray (the image processing package nd_image, part of numarray, is 
>> done by me) because I felt numarray was much better then Numeric for 
>> that application, and that also ruled out SciPy. There seems now to be 
>> an effort going to either try to merge the two or to bring Numeric up 
>> to numarray standard. Thats great, and if that works out I will try to 
>> rewrite my package so it works with both. Would  be happy to 
>> contribute it to SciPy. But I would make sure it remains a package 
>> that can be installed separately from  SciPy if required.
>>
> "What scipy is" is quite open to interpretation.  The idea of scipy is 
> just to have a super-package where different packages can live and have 
> some kind of inter-operability.   It is made so that sub-packages can be 
> easily added and built upon request.   We try hard to make sure that the 
> only real dependency is scipy_base which should not be hard to install 
> anywhere (does not require ATLAS or FORTRAN).
> It would be great to add nd_image as a sub-package to SciPy.    In fact, 
> nd_image was one of the things that most alarmed me about the emerging 
> split in the community, as it reproduces some of what is already in 
> SciPy (N-D convolution)  while adding some great things that I've always 
> wanted in SciPy (morphology).
> 
> I believe scipy can be easily installed as a collection of packages if 
> that is desired.  Pearu has done an incredible job with scipy_disutils 
> to make that process simple.   Making a single super package helps users 
> see that they have only a part of what's available (but maybe all they 
> need), and you get the added benefit of standard documentation 
> efforts.    Nobody has taken up the ball to actually try and distribute 
> individual packages of scipy, but it is a welcome effort.   
> www.scipy.org has places where such packages could live.  Enthought has 
> mainly been interested in distributing "batteries-included" packages for 
> windows because it is what their customers need.  Other are welcome to 
> pick up the slack.
> 
> Perhaps it would be good to get a list of which platforms actually need 
> binaries for people to look more favorably at SciPy.   I can see that OS 
> X is on the top of that list, surely somebody could distribute a copy of 
> a binary for that platform (I know of people who have it installed).     
> Also, scipy is still version 0.3 not because the code is unstable but 
> because the layout of the packages is still open to change.  Come and 
> discuss your visions of what it should look like with us on the 
> scipy-dev at scipy.net  mailing list.
> 
>> Making SciPy modular is in my opinion a very good approach. Why do I 
>> need to install all of it, if I really  only  need a few parts? If I 
>> can install only the subset I need, the likelihood that I will have an 
>> easier job doing that will increase. That would make it also possible 
>> to have sub-sets that are free of dependencies such as FORTRAN.
> 
> 
>>
>> On the other hand, I would think it would be a good idea to write 
>> software that does not  have dependencies on big packages such as 
>> SciPy when possible, but only on Numeric/Numarray. That does not 
>> exclude them from being included in SciPy. It seems to me that it 
>> would be a desirable property if a package can be installed just with 
>> Numeric (or numarray), with an appropriate sub-set of SciPy, or come 
>> with the big SciPy package. I would rather see a "collection of SciPy 
>> packages" then a "SciPy package".
> 
> 
> I'd like to see more of the development work that goes on happen under a 
> common standard, that is all.  SciPy has done a lot of work to provide 
> an infrastructure within which to distribute packages.  The field is 
> built, but people don't seem to want to come.    I understand that the 
> Numeric / Numarray dichotomy is a big source of this problem.  That is 
> the whole reason I'm devoting a big portion of my time to the problem 
> right now. 
> My very ambitious goal with Numeric3 is to replace both Numeric and 
> Numarray (heavily borrowing code from each).  When I say replace them, 
> I'm talking about the array object and umath module.  I'm not talking 
> about the other packages that use them.  I want those to live under a 
> common standard (i.e. scipy).   It would be fine with me if that common 
> standard include a scipy_lite or something like that which was easy to 
> install, if that is desired.   But, if the problem is really just 
> availability of binary copies, then why don't we solve that instead of 
> worrying about installation on platforms that nobody uses.   In 
> addition, even though I've developed a lot of scipy, I'm willing to 
> throw it all out or alter it considerably in order to satisfy the 
> demands that exist for package developers to want to put their weight 
> behind something like SciPy.   So, don't equate history with an 
> unwillingness to cooperate.
> I think you'll find the people involved with scipy to be incredibly 
> willing to cooperate to achieve mutual success.  If you have users of a 
> package that demand a certain binary of scipy to be installed, then 
> bring them over.  I can almost guarantee you that a copy of scipy will 
> be available for them.
> While I do recognize the value of repeated work.  When it comes to 
> interoperability, single standards seem to work best.  So, the longer 
> this unnecessary division of resources goes on, the farther behind 
> Python will get for doing anything but niche work.
> -Travis
> 
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 



From konrad.hinsen at laposte.net  Sat Feb  5 02:09:41 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb  5 02:09:41 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203FCD8.8070204@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <55c095f595dfec3e16579094faea14f2@laposte.net> <4203FCD8.8070204@noaa.gov>
Message-ID: <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net>

On 04.02.2005, at 23:53, Chris Barker wrote:

>> With only C code and no external dependencies, there is rarely ever a 
>> problem, and that's why Numeric is a low-risk package.
>
> Except that it's setup.py has been broken for the last few releases. 
> That's been a problem for a number of users.

Indeed.

> By the way, here OS-X is much better than other systems, it comes with 
> lapack built in! (Veclib)

I didn't know it has lapack, I thought it was only BLAS!

> Actually, I disagree. Apple's Python is a better choice (you can't run 
> Aqua applications from a fink python, at least I don't think so), but 
> we do need to do more to provide easy to install packages for it. 
> (there are a lot)

I don't care for Aqua as much as I care for all those nice Python 
packages I use every day. With Apple's Python, it's command line 
installation (which I don't mind, but many of my users do), or 
PackageManager for a select few packages. And I can't my own without 
proving my own Web server as well, so package contribution is 
effectively discouraged.

> Here Fink drives me crazy. It's a system-within-a-system. I want 
> everything

I certainly agree on that. I curse Fink about once a week. But in the 
absence of an alternative, Fink is better than no Fink.

>  on my Mac to fit in well with my Mac. If there was a fink-like system 
> (I haven't yet tried darwinports or Gentoo) that didn't duplicate 
> Apples supplied stuff with incompatible versions (Python, X11), I'd be 
> much happier.

Agreed. But there isn't.

>> The net result is that I still don't have SciPy on my Mac
>
> Me neither, though I don't think it's SciPy's fault. The core problem 
> is  that

Attributing fault makes little sense in the OpenSource world.

>  there aren't all that many people using OS-X developing Python 
> packages, and Apple has, of course, made OS-X just different enough 
> from other unices to require some work (or take the easy way and use 
> fink, but then it

Not really, you can do command-line installation ("python setup.py...") 
just like on any other Unix, usually without any modification. The 
problem is that Mac users expect installers.

It should be possible to extend distutils for making Mac installers 
just like it can make RPMs and Windows installers. I suppose nobody 
looked into this because the Apple Python community is divided into 
those who come from the Unix world and go for Fink, and those who come 
from pre-10 MacOS and go for MacPython plus PackageManager.

> What's the distinction? Again, Apple doesn't provide f77. That's been 
> a stopper for me.

I guess Apple's point of view is that you should buy a compiler. But 
they could easily add g77 to their developer CD, so there is indeed 
little excuse.

> This is a key issue with all open-source projects (at least ones that 
> aren't funder). We all do the easiest thing to get the functionality 
> we need. If we combined all our duplicate efforts, we'd be in great 
> shape, but an any given moment, I mostly need to get my job done, not 
> make a better SciPy, or whatever.

Right. But I still wonder why non-scientific projects (e.g. Python 
itself, or Linux) don't suffer so much from this problem. The larger 
potential developer base is probably just one part of the answer.

BTW, there is a scientific study of the Python development process that 
most Pythoneers are probably not aware of but which I think is quite 
interesting: "A Methodological Framework for Socio-Cognitive  Analyses 
of Collaborative Design of Open Source Software" by W. Sack, F. 
D?tienne, J-M. Burkhardt, F. Barcellini,  N. Ducheneaut, and D. 
Mahendran. Easy to find via Google.

Konrad.




From jochen at fhi-berlin.mpg.de  Sat Feb  5 04:15:46 2005
From: jochen at fhi-berlin.mpg.de (=?iso-8859-1?Q?Jochen_K=FCpper?=)
Date: Sat Feb  5 04:15:46 2005
Subject: The need for a unified numerical array object (was: [Numpy-discussion] Numeric3)
In-Reply-To: <4203F7CE.8070403@ee.byu.edu> (Travis Oliphant's message of "Fri,
	04 Feb 2005 15:31:42 -0700")
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	<1f7baacd417e55debdde0351e463cf10@laposte.net>
	<4203CAD0.6050609@ee.byu.edu>
	<f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
	<4203F7CE.8070403@ee.byu.edu>
Message-ID: <9ezmyjursi.fsf_-_@gowron.rz-berlin.mpg.de>

Travis Oliphant <oliphant at ee.byu.edu> writes:

> I'm more interested now in uniting a fractured community (or at
> least making interoperability easier).

All,

although I am a bit too busy these days I wanna add my 1.5ct:

Few years ago I had the hope I could do (almost) all my calculations
in python, maybe writing a few limited extension modules in C. 

That hope is cured for now. Generally, all the necessary pieces are
there, but they are scattered over many packages: Numeric/Numarray,
scipy, pygsl, plus stuff like pyQt/pyGTK/wxWindows/... But getting
everything up and running for myself and colleagues is just to
demanding, esp. if system administrators are busy as well and
reluctant to mess up their system or work schedule.

Therefore I do almost all calculations in C++ right now. Compilers are
there, BLAS, LAPACK and even GSL are typically provided already,
otherwise admins do not hesitate much to install the latest binary
packages, which are available. It takes me a little longer to develop
code, but I know I can run it on any system I find and I easily save
the time their.

I still do small stuff in Python and use Numarray for it, but
everything that requires anything else I simply write in C++.

What I would love to see is a twofold system for numerics in Python:

An arrayobject plus simple ufuncs like min/max/sin/exp/... in the
Python core.

A "single" scientific Python project that provides multiple packages
using that arrayobject. There should be packages, for example, with
  a) Python+C only to provide basic linear algebra, FFT, ... with
     everything included (no external dependencies)
  b) FFTW, GSL, ... wrappers that still are only Python + C, but
     require external libraries to be installed.
  c) An "expert" package which wraps the most advanced libraries for
     each field, i.e. LAPACK. Here is should be possible to install
     each wrapper separately.

If would be nice if the subset a) of b) would use an identical
interface as a), similar for c) <- b).

Moreover I would see it advantageous to have three independent
packages here, so I could tell a sysadmin: "Please install
SciencePython_simple" or "Please install SciencePython_wrappers"
instead of "Please install SciencePython with options a, b, c".
Somehow these sys-mins feel that is easier.


Anyway, I appreciate all of yours work and I do understand that first
of all you have to solve your own problems. However, if this scattered
community manages to unify and simplify (lower entrance barriers)
their approach to Scientific calculations with Python, that will allow
it to grow much further than ever possible with the current situation,
that somehow limits itself to double experts: scientists with *good*
computer science knowledge or vice versa.

Time will tell, but I wish you all the luck you need! For me new
projects will come up constantly, maybe even a larger one will be in
Python again sometime.

Greetings, 
Jochen
-- 
Einigkeit und Recht und Freiheit                http://www.Jochen-Kuepper.de
    Libert?, ?galit?, Fraternit?                GnuPG key: CC1B0B4D
        (Part 3 you find in my messages before fall 2003.)




From mdehoon at ims.u-tokyo.ac.jp  Sat Feb  5 06:24:17 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Sat Feb  5 06:24:17 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4203CAD0.6050609@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu>
Message-ID: <4204D6F1.50106@ims.u-tokyo.ac.jp>

Travis Oliphant wrote:
> O.K.   I can see that there are several out there who belive that
> SciPy is sufficiently hard to install that they are concerned about
> requiring it for their math-using packages (despite the provided
> binary distributions, and the work that continues on making it easier
> to install).

Relying on binary distributions should be a method of last resort. You
can never keep up with all the possible operating systems x Python
versions x Numeric versions. I run Python on four computers here
(Windows, Mac OS X, Linux, and Unix), and only for Linux there is a
SciPy binary installer available (on Windows I am using Python 2.4, for
which there is no installer). Now SciPy binary installers can be made
for all these platforms, but I am sure that you have better ways to
spend your time.

We need to think carefully about what the goal of SciPy actually is. I
am worried that SciPy is turning into a goal in itself. To my mind, the
goal of Numerical Python and SciPy should be to help users perform
numerical calculations. Numerical Python does that by making commonly
needed numerical routines available as a convenient package. SciPy can
do that by making less-needed routines available either as a binary
installer or as source code. I don't see how moving well-functioning 
code from Numerical Python to SciPy will help any user.

--Michiel.




From verveer at embl.de  Sat Feb  5 08:01:44 2005
From: verveer at embl.de (Peter Verveer)
Date: Sat Feb  5 08:01:44 2005
Subject: The need for a unified numerical array object (was: [Numpy-discussion] Numeric3)
In-Reply-To: <9ezmyjursi.fsf_-_@gowron.rz-berlin.mpg.de>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu> <4203F7CE.8070403@ee.byu.edu> <9ezmyjursi.fsf_-_@gowron.rz-berlin.mpg.de>
Message-ID: <f8ad905b157df63154e45d42d9b58987@embl.de>

On Feb 5, 2005, at 1:15 PM, Jochen K?pper wrote:

> Travis Oliphant <oliphant at ee.byu.edu> writes:
>
>> I'm more interested now in uniting a fractured community (or at
>> least making interoperability easier).
> I still do small stuff in Python and use Numarray for it, but
> everything that requires anything else I simply write in C++.
>
> What I would love to see is a twofold system for numerics in Python:
>
> An arrayobject plus simple ufuncs like min/max/sin/exp/... in the
> Python core.

Yes, I am starting to think that this approach is the only one that 
would cure the split we currently have between Numarray and 
Numeric/SciPy. Provide this, and make sure that Numarray/SciPy can use 
its Python/C API's and we can take it from there.

> A "single" scientific Python project that provides multiple packages
> using that arrayobject. There should be packages, for example, with
>   a) Python+C only to provide basic linear algebra, FFT, ... with
>      everything included (no external dependencies)

That will be needed and should be easy to provide.  Basically these 
exist already.

>   b) FFTW, GSL, ... wrappers that still are only Python + C, but
>      require external libraries to be installed.

Don't forget that these packages tend to be using the GPL license. That 
should be fine, but means that such a package needs to be GPL-ed and 
separate from the rest.

>   c) An "expert" package which wraps the most advanced libraries for
>      each field, i.e. LAPACK. Here is should be possible to install
>      each wrapper separately.
>
> If would be nice if the subset a) of b) would use an identical
> interface as a), similar for c) <- b).
>
> Moreover I would see it advantageous to have three independent
> packages here, so I could tell a sysadmin: "Please install
> SciencePython_simple" or "Please install SciencePython_wrappers"
> instead of "Please install SciencePython with options a, b, c".
> Somehow these sys-mins feel that is easier.

Agreed they should be independent and I dont see why this could not be 
the case.

I would add another requirement to such packages: good documentation.

> Anyway, I appreciate all of yours work and I do understand that first
> of all you have to solve your own problems. However, if this scattered
> community manages to unify and simplify (lower entrance barriers)
> their approach to Scientific calculations with Python, that will allow
> it to grow much further than ever possible with the current situation,
> that somehow limits itself to double experts: scientists with *good*
> computer science knowledge or vice versa.
>
> Time will tell, but I wish you all the luck you need! For me new
> projects will come up constantly, maybe even a larger one will be in
> Python again sometime.

I can only agree with all what you wrote. My experience with Python has 
been the other way around: I have been leaving C/C++ behind for Python. 
But at the same time I have found myself limiting my code to a small 
sub-set of all the good stuff that is out there (basically I only use 
Numarray). I would like to see that change and what you outlined above 
seems to me the best way to go forward.

Peter




From verveer at embl.de  Sat Feb  5 08:28:48 2005
From: verveer at embl.de (Peter Verveer)
Date: Sat Feb  5 08:28:48 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42042318.2090704@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu>
Message-ID: <c533ee29143e3286b075927540f5844b@embl.de>

> "What scipy is" is quite open to interpretation.  The idea of scipy is 
> just to have a super-package where different packages can live and 
> have some kind of inter-operability.   It is made so that sub-packages 
> can be easily added and built upon request.   We try hard to make sure 
> that the only real dependency is scipy_base which should not be hard 
> to install anywhere (does not require ATLAS or FORTRAN).
> It would be great to add nd_image as a sub-package to SciPy.    In 
> fact, nd_image was one of the things that most alarmed me about the 
> emerging split in the community, as it reproduces some of what is 
> already in SciPy (N-D convolution)  while adding some great things 
> that I've always wanted in SciPy (morphology).

What I find unfortunate is that due to the differences between the 
packages, nd_image cannot be compiled for both Numeric and Numarray at 
the moment. I did not forsee that the split between the packages would 
exist for so long. I do however not agree that it is an example of 
duplication of work. The N-D convolutions are based on a 
filter-framework that is an essential part of nd_image, which would 
have to be implemented anyway (e.g. the morphology operators are also 
based on it.) So one should be too quick to judge if something is 
duplication and a wast of resources.

> I'd like to see more of the development work that goes on happen under 
> a common standard, that is all.  SciPy has done a lot of work to 
> provide an infrastructure within which to distribute packages.  The 
> field is built, but people don't seem to want to come.

A common standard is a very good idea. But right now I don't find SciPy 
attractive as a framework, because 1) it is too big and not easily 
installed. 2) it is not very well documented. Thus, I prefer to write 
my package against a smaller code-base, in this case Numarray, but it 
could have also been Numeric. That has the advantage that people can 
install it more easily, while it still can be included in things like 
SciPy if desired.

>  I understand that the Numeric / Numarray dichotomy is a big source of 
> this problem.  That is the whole reason I'm devoting a big portion of 
> my time to the problem right now.

I can only agree with that. Regardless if I want to use SciPy or not, 
or in whatever form, I would like to see this problem go away, so that 
my software can become available for everybody.

> My very ambitious goal with Numeric3 is to replace both Numeric and 
> Numarray (heavily borrowing code from each).  When I say replace them, 
> I'm talking about the array object and umath module.  I'm not talking 
> about the other packages that use them.

I have to admit that I was very sceptical when I read your announcement 
for Numeric3, since I thought it would not be a good idea to have yet 
another array package. However, now it is clearer to me what you mean 
to do, and I think this is in fact exactly how I would like to see it 
happen. To me it would be perfect if there is some array package that 
forms a minimal basis to build on. I would program the nd_image package 
to that and offer the result for inclusion in both Numarray and SciPy, 
and allow it to be installed standalone. But it seems to me that there 
is a danger for the "yet another package" effect to occur. I think I 
will remain sceptical unless you achieve three things: 1) It has the 
most important improvements that numarray  has. 2) It has a good API 
that can be used to write packages that work with Numeric3/SciPy and 
Numarray (the latter probably will not go away). 3) Inclusion in the 
main Python tree, so that it is guaranteed to be available.

> I want those to live under a common standard (i.e. scipy).   It would 
> be fine with me if that common standard include a scipy_lite or 
> something like that which was easy to install, if that is desired.

Jochem K?pper just outlined very well how it could look like: A small 
core, plus a common project with packages at different levels. I think 
it is a very good idea, and probably similar to what SciPy is trying to 
do now. But he suggests an explicit division between independent 
packages: basic packages, packages with external library dependencies 
like FFTW, and advanced packages. Maybe something like that should be 
set up if we get an arraybobject into the Python core.

BTW it was mentioned before that it would be a problem to remove 
packages like LinearAlgebra and FFT from the core Numeric. matplotlib 
was mentioned as an example of a package that depends on them. I think 
that points however to a fundamental problem with matplotlib: why does 
a plotting library need FFTs and linear algebra? So I don't think 
anybody can really argue that things like an FFT should be in a core 
array package.

> But, if the problem is really just availability of binary copies, then 
> why don't we solve that instead of worrying about installation on 
> platforms that nobody uses.   In addition, even though I've developed 
> a lot of scipy, I'm willing to throw it all out or alter it 
> considerably in order to satisfy the demands that exist for package 
> developers to want to put their weight behind something like SciPy.   
> So, don't equate history with an unwillingness to cooperate.

That sounds great. If you go the route of a separate Numeric3 that goes 
into the python core and keep a common (simple) API at both Python and 
C level, then I will support that by rewriting nd_image against it. The 
result should be able to replace the current nd_image package in 
numarray, and hopefully be included in SciPy.

> While I do recognize the value of repeated work.  When it comes to 
> interoperability, single standards seem to work best.  So, the longer 
> this unnecessary division of resources goes on, the farther behind 
> Python will get for doing anything but niche work.

Agreed about the single standard thing. But I am not willing to just 
'join' the SciPy project to achieve it (at least for nd_image). I am 
however very interested in the possibility of writing against a small 
high-quality array package that is included in the pyhton core. That 
would be all the standard I need. If you manage to make SciPy into a 
useful larger standard on top of that, great, more power to all of us!

Peter




From rays at san.rr.com  Sat Feb  5 08:28:49 2005
From: rays at san.rr.com (RJ)
Date: Sat Feb  5 08:28:49 2005
Subject: The need for a unified numerical array object (was:
  [Numpy-discussion] Numeric3)
In-Reply-To: <9ezmyjursi.fsf_-_@gowron.rz-berlin.mpg.de>
References: <4203F7CE.8070403@ee.byu.edu>
 <420280AD.8080309@ee.byu.edu>
 <4202F6A0.3@ims.u-tokyo.ac.jp>
 <1f7baacd417e55debdde0351e463cf10@laposte.net>
 <4203CAD0.6050609@ee.byu.edu>
 <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
 <4203F7CE.8070403@ee.byu.edu>
Message-ID: <5.2.1.1.2.20050205080759.026eb700@pop-server.san.rr.com>

At 01:15 PM 2/5/2005 +0100, Jochen K?pper wrote:
>Travis Oliphant <oliphant at ee.byu.edu> writes:
>
>Moreover I would see it advantageous to have three independent
>packages here, so I could tell a sysadmin: "Please install
>SciencePython_simple" or "Please install SciencePython_wrappers"
>instead of "Please install SciencePython with options a, b, c".
>Somehow these sys-mins feel that is easier.

I've used Python for a couple of years now, but spent 5 years before in Perl (and not interested in going back); but the one thing I miss from Perl is CPAN. When I needed a module, I typed CPAN <module> and the proper version magically installed, with dependances. And there are a lot of modules, all available in one place.
I would also like to see more of the PHP-style user-contribution online docs. I also maintain some PHP database code and find it very time-saving to have all the comments and examples right there, and it allows users to take some of the burden of docs off of the developers. They also fill in some of the knowledge assumptions the language developers skim past, which might not be so obvious to novices.

I find that there is a lot of Python resources available for most any task, including the numpy family, but finding and installing the right thing for the local install and task at hand does take some searching and reading.

No complaints! just my experience to date.
Thanks to all,
Ray Schumacher
http://rjs.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050205/e34f22b7/attachment-0001.html>

From jdhunter at ace.bsd.uchicago.edu  Sat Feb  5 09:11:21 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Sat Feb  5 09:11:21 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <c533ee29143e3286b075927540f5844b@embl.de> (Peter Verveer's
 message of "Sat, 5 Feb 2005 17:28:09 +0100")
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	<1f7baacd417e55debdde0351e463cf10@laposte.net>
	<4203CAD0.6050609@ee.byu.edu>
	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>
	<42042318.2090704@ee.byu.edu>
	<c533ee29143e3286b075927540f5844b@embl.de>
Message-ID: <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Peter" == Peter Verveer <verveer at embl.de> writes:

    Peter> BTW it was mentioned before that it would be a problem to
    Peter> remove packages like LinearAlgebra and FFT from the core
    Peter> Numeric. matplotlib was mentioned as an example of a
    Peter> package that depends on them. I think that points however
    Peter> to a fundamental problem with matplotlib: why does a
    Peter> plotting library need FFTs and linear algebra? So I don't
    Peter> think anybody can really argue that things like an FFT
    Peter> should be in a core array package.

The dependence on fft arises from the original mission of matplotlib,
which was to provide a matlab compatible python plotting.  matlab has
functions like psd, csd and cohere, which in addition to computing the
spectral density and coherence functions, generate the plots.  So I
implemented Welch's Average Periodogram on top of fft to compute
spectral densities for these plotting commands.  

I use Matrix a lot for matrix multiplications to do text rotations and
layout.  If Matrix disappeared, it can be easily replace by wrapping
dot -- Todd did that already for the numarray equivalent in the
numerix module.

The matplotlib pylab module aspires to provide a matlab like
environment -- the focus is on plotting of course -- but as I've
needed things for my own work I've added commands to the
matplotlib.mlab module to provide matlab commands in python that do
require linear algebra, eg polyfit and polyval.  Basically
matplotlib.mlab and matplotlib.pylab extend Numeric's MLab.  This
section of the code could of course be folded into scipy.mlab,
Numeric.mlab, or wherever appropriate, but in my view it is already in
a pretty sensible place, in a module that attempts to provide a matlab
environment for python users.

If linear algebra, fft and random numbers disappeared from Numeric,
matplotlib would survive.  As you say, they don't go to the core of
plotting.  I could raise an exception if the user called psd and
didn't have scipy installed.  And everyone else who has written code
based on the functionality of Numeric's extra modules could rewrite
their code as well.  It would mean more work for me and less
functionality for some matplotlib users, but would be manageable.

We've put a fair amount of work in making Numeric and numarray work
transparently in matplotlib at the python and extension code level.
Having a third array package in the wild which is incompatible with
either is an unattractive option for me, but one which I can deal with
if need be.  And if the goal is to get an array package into the
standard library that has the core functionality of Numeric and
numarray, and Numeric3 is the most sensible means to that end, then I
am all for it.

JDH



From verveer at embl.de  Sat Feb  5 09:43:52 2005
From: verveer at embl.de (Peter Verveer)
Date: Sat Feb  5 09:43:52 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu>
Message-ID: <ce075fa4fe288fb3129ab3c4122c94b2@embl.de>

I did not realize fully what matplotlib intents to do, so it seems I 
picked a bad example, my apologies for that...

But I think your answers point out quite well that it may be worth to 
go this route. Sure, people will need to adapt their packages, but it 
will be worth it for all of us in the long run.

I do agree with your remarks about possible problems with having a 
third incompatible package. But I think the goal would be to get that 
package in the Python core, and to make actually sure that it is not 
incompatible with numarray and scipy.  I suppose when it does go into 
the core, everybody will have to be compatible to some level.

Peter

On Feb 5, 2005, at 6:00 PM, John Hunter wrote:

>>>>>> "Peter" == Peter Verveer <verveer at embl.de> writes:
>
>     Peter> BTW it was mentioned before that it would be a problem to
>     Peter> remove packages like LinearAlgebra and FFT from the core
>     Peter> Numeric. matplotlib was mentioned as an example of a
>     Peter> package that depends on them. I think that points however
>     Peter> to a fundamental problem with matplotlib: why does a
>     Peter> plotting library need FFTs and linear algebra? So I don't
>     Peter> think anybody can really argue that things like an FFT
>     Peter> should be in a core array package.
>
> The dependence on fft arises from the original mission of matplotlib,
> which was to provide a matlab compatible python plotting.  matlab has
> functions like psd, csd and cohere, which in addition to computing the
> spectral density and coherence functions, generate the plots.  So I
> implemented Welch's Average Periodogram on top of fft to compute
> spectral densities for these plotting commands.
>
> I use Matrix a lot for matrix multiplications to do text rotations and
> layout.  If Matrix disappeared, it can be easily replace by wrapping
> dot -- Todd did that already for the numarray equivalent in the
> numerix module.
>
> The matplotlib pylab module aspires to provide a matlab like
> environment -- the focus is on plotting of course -- but as I've
> needed things for my own work I've added commands to the
> matplotlib.mlab module to provide matlab commands in python that do
> require linear algebra, eg polyfit and polyval.  Basically
> matplotlib.mlab and matplotlib.pylab extend Numeric's MLab.  This
> section of the code could of course be folded into scipy.mlab,
> Numeric.mlab, or wherever appropriate, but in my view it is already in
> a pretty sensible place, in a module that attempts to provide a matlab
> environment for python users.
>
> If linear algebra, fft and random numbers disappeared from Numeric,
> matplotlib would survive.  As you say, they don't go to the core of
> plotting.  I could raise an exception if the user called psd and
> didn't have scipy installed.  And everyone else who has written code
> based on the functionality of Numeric's extra modules could rewrite
> their code as well.  It would mean more work for me and less
> functionality for some matplotlib users, but would be manageable.
>
> We've put a fair amount of work in making Numeric and numarray work
> transparently in matplotlib at the python and extension code level.
> Having a third array package in the wild which is incompatible with
> either is an unattractive option for me, but one which I can deal with
> if need be.  And if the goal is to get an array package into the
> standard library that has the core functionality of Numeric and
> numarray, and Numeric3 is the most sensible means to that end, then I
> am all for it.
>
> JDH
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion




From strawman at astraw.com  Sat Feb  5 13:30:14 2005
From: strawman at astraw.com (Andrew Straw)
Date: Sat Feb  5 13:30:14 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <55c095f595dfec3e16579094faea14f2@laposte.net> <4203FCD8.8070204@noaa.gov> <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net>
Message-ID: <04B12B1A-77BD-11D9-BC02-000D93476BAC@astraw.com>

Just a minor point, but I think the bdist_mpkg extension to distutils 
by Bob Ippolitto as part of py2app actually solves a lot of the 
problems you describe for Mac OS X.  
http://undefined.org/python/#py2app

On Feb 5, 2005, at 2:09 AM, konrad.hinsen at laposte.net wrote:

> On 04.02.2005, at 23:53, Chris Barker wrote:
>
>> Actually, I disagree. Apple's Python is a better choice (you can't 
>> run Aqua applications from a fink python, at least I don't think so), 
>> but we do need to do more to provide easy to install packages for it. 
>> (there are a lot)
>
> I don't care for Aqua as much as I care for all those nice Python 
> packages I use every day. With Apple's Python, it's command line 
> installation (which I don't mind, but many of my users do), or 
> PackageManager for a select few packages. And I can't my own without 
> proving my own Web server as well, so package contribution is 
> effectively discouraged.
>
>>  there aren't all that many people using OS-X developing Python 
>> packages, and Apple has, of course, made OS-X just different enough 
>> from other unices to require some work (or take the easy way and use 
>> fink, but then it
>
> Not really, you can do command-line installation ("python 
> setup.py...") just like on any other Unix, usually without any 
> modification. The problem is that Mac users expect installers.
>
> It should be possible to extend distutils for making Mac installers 
> just like it can make RPMs and Windows installers. I suppose nobody 
> looked into this because the Apple Python community is divided into 
> those who come from the Unix world and go for Fink, and those who come 
> from pre-10 MacOS and go for MacPython plus PackageManager.




From oliphant at ee.byu.edu  Sat Feb  5 16:00:09 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb  5 16:00:09 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <c533ee29143e3286b075927540f5844b@embl.de>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de>
Message-ID: <42055DAF.5020601@ee.byu.edu>

Peter Verveer wrote:

>
> What I find unfortunate is that due to the differences between the 
> packages, nd_image cannot be compiled for both Numeric and Numarray at 
> the moment. I did not forsee that the split between the packages would 
> exist for so long. I do however not agree that it is an example of 
> duplication of work. The N-D convolutions are based on a 
> filter-framework that is an essential part of nd_image, which would 
> have to be implemented anyway (e.g. the morphology operators are also 
> based on it.) So one should be too quick to judge if something is 
> duplication and a wast of resources.


This is exactly why I am dissatisfied with the way numarray has been 
advertised (I'm not blaming anyone here, I recognize the part that I 
played in it).  Third-party package providers starting to build on top 
of numarray before it was a clear replacement.   I don't see anything in 
nd_image that could not have sat on top of Numeric from the 
beginning.    I don't see why it needed to use numarray specific things 
at all.   By the way, nd_image it is a very nice piece of work that I 
have been admiring.  

> common standard is a very good idea. But right now I don't find SciPy 
> attractive as a framework, because 1) it is too big and not easily 
> installed. 2) it is not very well documented. Thus, I prefer to write 
> my package against a smaller code-base, in this case Numarray, but it 
> could have also been Numeric. That has the advantage that people can 
> install it more easily, while it still can be included in things like 
> SciPy if desired.

What is too big about it?  Which packages would you like to see not 
included?   Is it the dependence on Fortran that frightens, or the 
dependence on ATLAS (not a requirement by the way)?   How many realize 
that you can turn off installation of packages in the setup.py file by a 
simple switch?

The point is that nd_image should live perfectly well as a scipy package 
and be installed and distributed separately if that is desired as 
well.   But, it would present a uniform face to others looking to do 
scientific work with Python.  This appearance of multiple packages has 
only amplified the problem that scipy was trying to solve.  If scipy is 
solving it badly, then lets fix it.     I do appreciate the comments 
that are helping me understand at least what needs to be fixed.  But, 
everybody has always been welcome to come and help fix things --- it is 
quite easy to get CVS write access to the scipy core.

Matplotlib as well is reinventing stuff already available elsewhere.  
SciPy has always tried to bring a matlab-like environment to the user.  
That's always been my goal.  It is frustrating to see so many with the 
same goal work so independently of each other.    Sometimes I guess it 
is necessary so that new ideas can be tried out.   But, it sure seems to 
happen more often than I would like with Python and Science / Engineering.

Is it ownership that is a concern?  None of us involved with scipy have 
an alterior motive for trying to bring packages together except the 
creation of a single standard and infrastructure for scientific 
computing with Python.    I keep hearing that people like the idea of a 
small subset of packages, but that is exactly what scipy has always been 
trying to be.   Perhaps those involved with scipy have not voiced our 
intentions enough or loud enough. 

> I can only agree with that. Regardless if I want to use SciPy or not, 
> or in whatever form, I would like to see this problem go away, so that 
> my software can become available for everybody.

I'm glad to hear this. 

>> My very ambitious goal with Numeric3 is to replace both Numeric and 
>> Numarray (heavily borrowing code from each).  When I say replace 
>> them, I'm talking about the array object and umath module.  I'm not 
>> talking about the other packages that use them.
>
>
> I have to admit that I was very sceptical when I read your 
> announcement for Numeric3, since I thought it would not be a good idea 
> to have yet another array package. 

I've never seen it as yet another array package.  I've always seen it as 
a Numeric upgrade with the hope of getting closer to numarray (and even 
replace it if possible).    In fact, since I'm currently the maintainer 
of Numeric, those who use Numeric should perhaps be the most concerned 
because Numeric3 will replace Numeric (and will therefore try not to 
break old code too much --- though there may be a few changes).   I can 
guarantee that SciPy will work with Numeric3.   We've always told scipy 
users that if they use SciPy they won't have to worry about the 
arrayobject issue because scipy will shield them from it (at least on 
the Python level).

    Of course Numeric will still be there and if someone else wants to 
maintain it, they are welcome to.  I'm just serving notice that any 
resources I have will be devoted to Numeric3, and as soon as Numeric3 is 
out of alpha I would actually like to call it Numeric (verson 30.0).

> But it seems to me that there is a danger for the "yet another 
> package" effect to occur. I think I will remain sceptical unless you 
> achieve three things: 1) It has the most important improvements that 
> numarray  has. 2) It has a good API that can be used to write packages 
> that work with Numeric3/SciPy and Numarray (the latter probably will 
> not go away). 3) Inclusion in the main Python tree, so that it is 
> guaranteed to be available.

Thanks for your advice.  All encouragement is appreciated. 

1)  The design document is posted.  Please let me know what "the most 
important improvements" are.

2)  It will have an improved API and I would also like to support a lot 
of the numarray API as well (although I I don't understand the need for 
many of the API calls numarray allows and see much culling that needs to 
be done --- much input is appreciated here as to which are the most 
important API calls.  I will probably use nd_image as an example of what 
to support).

3) Inclusion in the main Python tree will rely on Guido feeling like 
there is a consensus.  I'm hopeful here and will try to pursue this -- 
but I need to finish code first.

> Jochem K?pper just outlined very well how it could look like: A small 
> core, plus a common project with packages at different levels. I think 
> it is a very good idea, and probably similar to what SciPy is trying 
> to do now. But he suggests an explicit division between independent 
> packages: basic packages, packages with external library dependencies 
> like FFTW, and advanced packages. Maybe something like that should be 
> set up if we get an arraybobject into the Python core.

Sounds great.  SciPy has been trying to do exactly this, but we need 
ideas --- especially from package developers who understand the issues 
--- as to how to set it up correctly.  We've already re-factored a 
couple of times.  We could do it again if we needed to, so that the 
infrastructure had the right feel.   A lot of this is already in place.  
I don't think many recognize some of the work that has already been 
done.  This is not an easy task. 

>
> BTW it was mentioned before that it would be a problem to remove 
> packages like LinearAlgebra and FFT from the core Numeric. matplotlib 
> was mentioned as an example of a package that depends on them. I think 
> that points however to a fundamental problem with matplotlib: why does 
> a plotting library need FFTs and linear algebra? So I don't think 
> anybody can really argue that things like an FFT should be in a core 
> array package.

My point exactly.  Having these things in Numeric/ Numarray actually 
encourages the creation of multiple separate packages as everybody tries 
to add just there little extra bit on top.

Plotting is potentially problematic because there are a lot of ways to 
plot.  I think we need to define interfaces in this regard and adapters 
so that commands that would throw up a plot could use several different 
plotting methods to do it.   I'm not favoring any plotting technique, so 
don't pre-guess me.  My ideas of plotting are probably very similiar to 
John's with matplotlib.  His work is another that I'm disappointed is 
not part of scipy and has led me to my current craziness with Numeric3 :-)

>
> Agreed about the single standard thing. But I am not willing to just 
> 'join' the SciPy project to achieve it (at least for nd_image). I am 
> however very interested in the possibility of writing against a small 
> high-quality array package that is included in the pyhton core. That 
> would be all the standard I need. If you manage to make SciPy into a 
> useful larger standard on top of that, great, more power to all of us!

Why not?  Your goals are not at odds with ours.   O.K. it may be that 
more work is required to re-write nd_image against the Numeric C-API 
than you'd like --- that's why Numeric3 is going to try and support the 
numarray C-API as well.

Thanks for your valuable feedback.   I appreciate the time it took you 
to provide it.   I hope we can work more together in the future.

Best regards,

-Travis





From oliphant at ee.byu.edu  Sat Feb  5 16:08:08 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb  5 16:08:08 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	<1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu>	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu>	<c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu>
Message-ID: <42055FA3.9050909@ee.byu.edu>

>The dependence on fft arises from the original mission of matplotlib,
>which was to provide a matlab compatible python plotting.  matlab has
>functions like psd, csd and cohere, which in addition to computing the
>spectral density and coherence functions, generate the plots.  So I
>implemented Welch's Average Periodogram on top of fft to compute
>spectral densities for these plotting commands.  
>  
>
Matlab-like environment is very much what scipy has always tried to be. 

>The matplotlib pylab module aspires to provide a matlab like
>environment -- the focus is on plotting of course -- but as I've
>needed things for my own work I've added commands to the
>matplotlib.mlab module to provide matlab commands in python that do
>require linear algebra, eg polyfit and polyval.  Basically
>matplotlib.mlab and matplotlib.pylab extend Numeric's MLab.  This
>section of the code could of course be folded into scipy.mlab,
>Numeric.mlab, or wherever appropriate, but in my view it is already in
>a pretty sensible place, in a module that attempts to provide a matlab
>environment for python users.
>  
>
But MATLAB is a huge package.  SciPy has been trying to do the same 
thing.  Plotting was needed and matplotlib did a great thing in 
providing excellent plotting.  I've never understood the reason for 
matplotlib to remain separate except as evidence of the great schism 
that was occuring in the community.

In fact, when Eric and I began scipy, I wanted to call it pylab (you'll 
notice I have the pylab.sourceforge.net site).  I have fond ties to 
MATLAB myself.    Eric chose the scipy name which is probably better.  
So, it is very concerning to me to see matplotlib try to become what 
scipy is already trying to become.   I think it is very confusing to 
newcomers.

>We've put a fair amount of work in making Numeric and numarray work
>transparently in matplotlib at the python and extension code level.
>Having a third array package in the wild which is incompatible with
>either is an unattractive option for me, but one which I can deal with
>if need be.  And if the goal is to get an array package into the
>standard library that has the core functionality of Numeric and
>numarray, and Numeric3 is the most sensible means to that end, then I
>am all for it.
>  
>
Again,  from my perspective once Numeric3 is working reasonably well, 
Numeric ceases to exist for me.  Yes others may continue to download it, 
but I won't be maintaining it anymore.

-Travis




From perry at stsci.edu  Sat Feb  5 17:49:10 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Sat Feb  5 17:49:10 2005
Subject: [Numpy-discussion] scipy, matplotlib, and installation dependencies.(previously:  Numeric3)
In-Reply-To: <42055FA3.9050909@ee.byu.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCGEAFFJAA.perry@stsci.edu>

Travis Oliphant wrote:
> >
> But MATLAB is a huge package.  SciPy has been trying to do the same 
> thing.  Plotting was needed and matplotlib did a great thing in 
> providing excellent plotting.  I've never understood the reason for 
> matplotlib to remain separate except as evidence of the great schism 
> that was occuring in the community.
> 
> In fact, when Eric and I began scipy, I wanted to call it pylab (you'll 
> notice I have the pylab.sourceforge.net site).  I have fond ties to 
> MATLAB myself.    Eric chose the scipy name which is probably better.  
> So, it is very concerning to me to see matplotlib try to become what 
> scipy is already trying to become.   I think it is very confusing to 
> newcomers.
> 
I think you are reading far too much into the overlap between
matplotlib and scipy. Yes, matplotlib duplicates some numerical
functionality, but the plotting and image display aspects of 
matplotlib are far and away the primary focus. Comparitively
little of it is devoted to numerical libraries. Likewise, 
although scipy has had plotting capability, we all recognize 
(at least I think we do) that it left a lot to be desired and 
was considered by most a stopgap until a better plotting solution
was found.

So this leads to the question: why wasn't matplotlib developed 
for scipy? You'll have to ask John for a definitive answer, but I 
think I can guess. As you may have noticed, many are fearful of 
building lots of dependencies into their library or package. That's
understandable. The more there are, the greater the chances that
installation will be too painful. Scipy has many such dependencies.
Until it has a reputation of being an easy install, it's unlikely 
lots of packages with a much narrower focus will depend on it.
Hence all the requests for something along the scale of what Numeric
packages rather than just all of scipy.

There are two competing forces at work here. One is to bundle
everything together and do the work of making sure it all plays
together nicely thus avoiding the pain of installing N different
packages independently (what scipy is trying to do). That's great
as far as it works easily, but with so many dependencies, more 
risk there is of one of them causing unsophiscated users (and 
even the sophisticated) stumbling on installation. The other 
force is for minimal dependencies. It makes for less headaches
for the package maintainer, but for users that have to install
several components, it can be more of a hassle.

Can scipy be made to be a painless install for most users? I'm 
not sure we know the answer yet. Until we do, I'd favor an approach
that shoots for a smaller, easily installable core, with a simple
way of asking for optional packages (having users edit setup.py
is probably asking too much). When packages prove that they are
reliably installed under all circumstances, then they can be added
to the core distribution, otherwise, it is understood that the 
optional packages may be tricker to install. The fewer you need,
the better. It seems to me that is what many would like to see,
but maybe I'm misrepresenting the most common point of view.

Perry



> >We've put a fair amount of work in making Numeric and numarray work
> >transparently in matplotlib at the python and extension code level.
> >Having a third array package in the wild which is incompatible with
> >either is an unattractive option for me, but one which I can deal with
> >if need be.  And if the goal is to get an array package into the
> >standard library that has the core functionality of Numeric and
> >numarray, and Numeric3 is the most sensible means to that end, then I
> >am all for it.
> >  
> >
> Again,  from my perspective once Numeric3 is working reasonably well, 
> Numeric ceases to exist for me.  Yes others may continue to download it, 
> but I won't be maintaining it anymore.
> 
> -Travis
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 




From southey at uiuc.edu  Sat Feb  5 19:28:10 2005
From: southey at uiuc.edu (Bruce Southey)
Date: Sat Feb  5 19:28:10 2005
Subject: [Numpy-discussion] Numeric3
Message-ID: <4760baf2.ca2b9b41.81db100@expms6.cites.uiuc.edu>

Hi,
I am not the developer of any of Numpy (in any form) so take my thoughts 'with a
grain of salt' because I am not doing the coding. I am really very thankful that
people have developed Numeric and numarray for Python!

Many of the comments I fully agree with especially on installation. Large
packages, like SciPy, require time and effort to ensure every thing works. Also,
various platforms lack compilers (i.e. MS Windows are not by default and free
Fortran 90 compilers for Linux). A user without 'system admin' experience or
access can not be expected get these compilers installed especially if the
dependencies don't compile or are incorrectly compiled.

With all the arguments that have been presented so far, I fail to see any need
for at least a third Python numerical option, especially one that may not
support many needs. This discussion shows that there doesn't seem to be the
'maturity' of the code base to justify a push towards the Python Standard
Library. By maturity, I mean that there are still too many differences between
Numeric and numarray that, while appear to be solvable, are not trivial to
address at the present time. (This is also one of the main reasons why many
different versions of the same thing exist.) 

I would like to see a single NumPy option that everyone can use. If numarray or
Numeric3 does that then great as I will support either. However, I do fall in
the numarray camp as I do tend to work with large matrices (plus features like
Masked Arrays are a big plus).

Regards
Bruce Southey



From oliphant at ee.byu.edu  Sat Feb  5 23:24:16 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb  5 23:24:16 2005
Subject: [Numpy-discussion] Proposal for making of Numarray a real Numeric
 'NG'
In-Reply-To: <200501211413.51663.faltet@carabos.com>
References: <200501211413.51663.faltet@carabos.com>
Message-ID: <4205C5DE.80504@ee.byu.edu>

Francesc Altet wrote:

>Hi List,
>
>I would like to make a formal proposal regarding with the subject of
>previous discussions in that list. This message is a bit long, but I've
>tried my best to expose my thoughts as clearly as possible.
>  
>
I did not have time to respond to this mail, but it is very good.  I  
will be placing some of its comments in the scipy site.

>It's worth remembering that Numeric has been a major breakthrough in
>introducing the capability to deal with large (homogeneous) datasets in
>Python in a very efficient mannner. In my opinion Numarray is, generally
>speaking, a very good package as well with many interesting new features
>that lack Numeric. Between the main advantages of Numarray vs Numeric I can
>list the next (although I can be a bit misleaded here because of my own user
>cases of both libraries):
>  
>
I think numarray has made some incrdedible strides in showing what the 
numeric object needs to be and in implementing some really neat 
functionality.   I just think its combination of Python and C code must 
be redone to overcome the speed issues that have arisen.  My opinion 
after perusing the numarray code is that it would be easier (for me 
anyway) to adjust Numeric to support the features of numarray, than to 
re-write and re-organize the relevant sections of numarray code.  One of 
the advantages of Numeric is it's tight implementation that added only 
two fundamental types, both written entirely in C.   I  was hoping that 
the Python dependencies for the fundamental types would fade as numarray 
matured, but it appears to me that this is not going to happen.

I did not have the time in the past to deal with this.  I wish I had 
looked at it more closely two years ago.   If I had done this I would 
have seen how to support the features that Perry wanted without 
completely re-writing everything.   But, then again, Python 2.2 changed 
what is possible on the C level and that has had an impact on the 
discussion.   

>- Memory-mapped objects: Allow working with on-disk numarray objects like if
>  they were in-memory.
>  
>
Numeric3 supports this cleanly and old Numeric did too (there was a 
memory-mapped module), it's just that byteswapping, and alignment had to 
be done manually.

>  
>- RecArrays: Objects that allow to deal with heterogeneous datasets
>  (tables) in an efficient manner. This ought to be very beneficial in many
>  fields.
>  
>
Heterogeneous arrays is the big one for old Numeric.  It is a good 
idea.  In Numeric3 it has required far fewer changes than I had at first 
imagined.

>  
>- CharArrays: Allow to work with large amounts of fixed and variable length
>  strings. I see this implementation much more powerful that Numeric.
>  
>
Also a good idea, and comees along for the ride with in Numeric3.   
Numeric had CHAR arrays but a vision was never specified  for how to 
make them more useful.  This change would have been a good step towards 
heterogeneous arrays.

>  
>- Index arrays within subscripts: e.g. if ind = array([4, 4, 0, 2])
>  and x = 2*arange(6), x[inx] results in array([8, 8, 0, 4])
>
>  
>
For scipy this was implemented on top of Numeric (so it is in Numeric3 
too),  the multidimensional version needs to be worked on, still.

>- New design interface: We should not forget that numarray has been designed
>  from the ground with Python Library integration in mind (or at least, this
>  is my impression). So, it should have more chances (if there is some hope)
>  to enter in the Standard Library than Numeric.
>  
>

Numeric has had this in mind for some time.  In fact the early Numeric 
developers were quite instrumental in getting significant changes into 
Python istelf, including Complex Objects, Ellipses, and Extended 
Slicing.   Guido was quite keen on the idea of including Numeric at one 
point.  Our infighting made him lose interest I think.   So claiming 
this as an advantage of numarray over Numeric is simply inaccurate.

>The real problem for Numarray: Object Creation Time
>===================================================
>
>On the other hand, the main drawback of Numarray vs Numeric is, in my
>opinion, its poor performance regarding object creation. This might look
>like a banal thing at first glance, but it is not in many cases. One example
>recently reported in this list is:
>  
>
Ah, and there's the rub.  I don't think this object creation time will 
go away until Numarray's infrastructure becomes essentially like that of 
Numeric.  One tight object all in C.   Getting it there seems harder 
than fixing Numeric, with the additional features of Numarray.

Thanks for these comments.   It is very good to hear what the most 
important features for users are.

-Travis





From verveer at embl.de  Sun Feb  6 05:28:45 2005
From: verveer at embl.de (Peter Verveer)
Date: Sun Feb  6 05:28:45 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42055DAF.5020601@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <42055DAF.5020601@ee.byu.edu>
Message-ID: <12fab106c08dbd04c949d9b7c5f96e3c@embl.de>

> This is exactly why I am dissatisfied with the way numarray has been 
> advertised (I'm not blaming anyone here, I recognize the part that I 
> played in it).  Third-party package providers starting to build on top 
> of numarray before it was a clear replacement.   I don't see anything 
> in nd_image that could not have sat on top of Numeric from the 
> beginning.    I don't see why it needed to use numarray specific 
> things at all.   By the way, nd_image it is a very nice piece of work 
> that I have been admiring.

Thanks. nd_image could sit easily on top of Numeric, it sits on top of 
numarray simply because I felt numarray was the better package. And 
this is still the case. There is however nothing inherent in nd_image 
that requires numarray. In fact, I am contemplating factoring out the 
image processing functionality into a separate C library, on which 
python packages then would be based.


>> common standard is a very good idea. But right now I don't find SciPy 
>> attractive as a framework, because 1) it is too big and not easily 
>> installed. 2) it is not very well documented. Thus, I prefer to write 
>> my package against a smaller code-base, in this case Numarray, but it 
>> could have also been Numeric. That has the advantage that people can 
>> install it more easily, while it still can be included in things like 
>> SciPy if desired.
>
> What is too big about it?  Which packages would you like to see not 
> included?

I think I was unfair to call scipy 'big'. There is nothing inherently 
bad about being big. The problem is rather installation, I suppose. 
Also I would like to see modularity, if I only want say, least-squares 
fitting, I should be able to just take that.

> Is it the dependence on Fortran that frightens, or the dependence on 
> ATLAS (not a requirement by the way)?   How many realize that you can 
> turn off installation of packages in the setup.py file by a simple 
> switch?

Fortran or Atlas do not scare me. But if they make it more difficult to 
have some average user to install scipy, just so they can use some of 
my software that depends on it, then there is a problem. Fear of such 
problems keeps me away from scipy. I can point somebody to numarray and 
say type 'python setup.py install' and be sure it works. I know, this a 
bit unfair, scipy aims to be more then numarray is now, but it would be 
nice to have functionality that is in scipy available with the same 
ease.

> The point is that nd_image should live perfectly well as a scipy 
> package and be installed and distributed separately if that is desired 
> as well.

It should live perfectly well as a scipy package because it should be 
dependent only on a basic array package. Then scipy would be welcome to 
include it. That is now not possible because of the split between 
numarray or numeric. If scipy and numarray would share the basic array 
package, nd_image would work perfectly fine with both.

> Is it ownership that is a concern?  None of us involved with scipy 
> have an alterior motive for trying to bring packages together except 
> the creation of a single standard and infrastructure for scientific 
> computing with Python.    I keep hearing that people like the idea of 
> a small subset of packages, but that is exactly what scipy has always 
> been trying to be.   Perhaps those involved with scipy have not voiced 
> our intentions enough or loud enough.

Maybe this is the problem. My impression of scipy has been that it 
tries to be a matlab-type of enviroment, i.e., including all in a 
single install. Nothing wrong with that, but also not what I really 
need, certainly not if it is difficult to install. I just need to be 
able get the packages that I need to write my programs in python, just 
like you would just link a library if you would do it in C...

So, although I do applaud the scipy effort to provide a complete 
enviroment, I would prefer to see it based on a set of packages that 
are as independent as possible, so that I can just stick to those if I 
want. I think you have been saying that SciPy has been designed with 
that goal in mind, so maybe I should investigate how easy it is to take 
out things I  need.

>> But it seems to me that there is a danger for the "yet another 
>> package" effect to occur. I think I will remain sceptical unless you 
>> achieve three things: 1) It has the most important improvements that 
>> numarray  has. 2) It has a good API that can be used to write 
>> packages that work with Numeric3/SciPy and Numarray (the latter 
>> probably will not go away). 3) Inclusion in the main Python tree, so 
>> that it is guaranteed to be available.
>
> Thanks for your advice.  All encouragement is appreciated.
> 1)  The design document is posted.  Please let me know what "the most 
> important improvements" are.

I will have a look and give my comments. On first reading it seems to 
fix all that I did not like in Numeric.

>
> 2)  It will have an improved API and I would also like to support a 
> lot of the numarray API as well (although I I don't understand the 
> need for many of the API calls numarray allows and see much culling 
> that needs to be done --- much input is appreciated here as to which 
> are the most important API calls.  I will probably use nd_image as an 
> example of what to support).

I think if Numeric3 is intended to be a basic package that might go 
into the Pyhton core, that its API should be as small as possible. Dont 
forget that numarray already includes a Numeric API, so writing 
packages that compile on both should be feasible. I personally prefer 
to switch to whatever the API in the core may be, over seeing multiple 
API's in such a core package. Having a set of multiple APIs will not 
help getting it accepted in the core, it should be kept as simple as 
possible.

>> Jochem K?pper just outlined very well how it could look like: A small 
>> core, plus a common project with packages at different levels. I 
>> think it is a very good idea, and probably similar to what SciPy is 
>> trying to do now. But he suggests an explicit division between 
>> independent packages: basic packages, packages with external library 
>> dependencies like FFTW, and advanced packages. Maybe something like 
>> that should be set up if we get an arraybobject into the Python core.
>
> Sounds great.  SciPy has been trying to do exactly this, but we need 
> ideas --- especially from package developers who understand the issues 
> --- as to how to set it up correctly.  We've already re-factored a 
> couple of times.  We could do it again if we needed to, so that the 
> infrastructure had the right feel.   A lot of this is already in 
> place.  I don't think many recognize some of the work that has already 
> been done.  This is not an easy task.

I think a crucial difference is that there would be different packages 
that need to be installed separately, which increasing levels of 
dependency and difficulties. Enviroments such as scipy could be build 
on top of it. At the same time it would make people like me that dont 
want to install enviroments that do everything, happier.
>
> Plotting is potentially problematic because there are a lot of ways to 
> plot.  I think we need to define interfaces in this regard and 
> adapters so that commands that would throw up a plot could use several 
> different plotting methods to do it.   I'm not favoring any plotting 
> technique, so don't pre-guess me.  My ideas of plotting are probably 
> very similiar to John's with matplotlib.  His work is another that I'm 
> disappointed is not part of scipy and has led me to my current 
> craziness with Numeric3 :-)

With things like plotting you get into the realm of user interfaces. In 
my opinion such things should not mix at all with packages that 
implement numerical functionality. Obviously an environment like scipy  
needs to have plotting, but again, if I don't want it, I should not 
have to download and install it. So within a structure that consists of 
different levels of packages, plotting and displaying should probably 
consist within their own set of packages.

>> Agreed about the single standard thing. But I am not willing to just 
>> 'join' the SciPy project to achieve it (at least for nd_image). I am 
>> however very interested in the possibility of writing against a small 
>> high-quality array package that is included in the pyhton core. That 
>> would be all the standard I need. If you manage to make SciPy into a 
>> useful larger standard on top of that, great, more power to all of 
>> us!
>
> Why not?  Your goals are not at odds with ours.   O.K. it may be that 
> more work is required to re-write nd_image against the Numeric C-API 
> than you'd like --- that's why Numeric3 is going to try and support 
> the numarray C-API as well.

Rewriting is not the problem, that can be done fairly easily, as long 
as it can be done such that the result compiles for both numarray and 
numeric. I am not sure if it makes sense to have multiple APIs into 
core package, keep it simple.

> Thanks for your valuable feedback.   I appreciate the time it took you 
> to provide it.   I hope we can work more together in the future.

For now I will watch and see what happens :-) nd_image remains a 
numarray package for now, but if it becomes feasible to make it work 
for both numarray and numeric I will give that a try. If some package 
makes into the python core, then I will support that and remove all 
dependencies that might exist on other packages.

Cheers, Peter




From konrad.hinsen at laposte.net  Sun Feb  6 06:46:25 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sun Feb  6 06:46:25 2005
Subject: The need for a unified numerical array object (was: [Numpy-discussion] Numeric3)
In-Reply-To: <5.2.1.1.2.20050205080759.026eb700@pop-server.san.rr.com>
References: <4203F7CE.8070403@ee.byu.edu> <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu> <4203F7CE.8070403@ee.byu.edu> <5.2.1.1.2.20050205080759.026eb700@pop-server.san.rr.com>
Message-ID: <55c61cbfde4feb91b4d8da721d1878a3@laposte.net>

On 05.02.2005, at 17:28, RJ wrote:

>  I've used Python for a couple of years now, but spent 5 years before 
> in Perl (and not interested in going back); but the one thing I miss 
> from Perl is CPAN. When I needed a module, I typed CPAN <module> and 
> the proper version magically installed, with dependances. And there 
> are a lot of modules, all available in one place.

How does CPAN handle external dependencies? For example, what does CPAN 
do with a GTK interface for Perl when it finds that GTK is not 
available on the machine? Or when the package requires a Fortran 
compiler and none is installed?

Konrad.




From dd55 at cornell.edu  Sun Feb  6 08:17:30 2005
From: dd55 at cornell.edu (Darren Dale)
Date: Sun Feb  6 08:17:30 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4760baf2.ca2b9b41.81db100@expms6.cites.uiuc.edu>
References: <4760baf2.ca2b9b41.81db100@expms6.cites.uiuc.edu>
Message-ID: <200502061116.35095.dd55@cornell.edu>

Hi,

As a relatively new user, I would like to contribute my experience to the 
discussion. Late in a graduate program, I caught the bug and wanted to try 
out python. Being established in Matlab, I was hoping to find a similar 
collection of capabilities for Python, something that I could experiment with 
and start learning fairly quickly. When I discovered Matplotlib, I felt home 
free. Without too much effort, I had a plot on screen and felt 'ok, this is 
feasible.' With the recent effort from Matplotlib and IPython, this should be 
even more true for newcomers.

On the downside, I have been somewhat confused by the state of things 
concerning Numeric and numarray. I think I can trace this confusion back to 
the pfdubios.com/numpy site (linked from numpy.sourceforge.net), which is the 
first hit I get doing a google search for either numpy or numerical python. 
"If you are new to Numerical Python, please use Numarray. The older module, 
Numeric, is unsupported." There is no date associated with the site, but it 
still advertises release 22.0. In the near future, I strongly suggest editing 
websites for misleading information.

So now I have both Numeric and numarray on my system, linked to the full, 
atlas-tuned, lapack and blas libraries. I started out using numarray, 
thinking it was in my long term interest, but I guess this is open to debate. 
I was also hesitant to use scipy, because it relied on Numeric. Being an 
uninformed newbie, I just wasnt able to make an informed decision (I am still 
struggling with it).

A lot of what I have read recently is very encouraging to me. I like the idea 
of bridging the API gap between numarray and Numeric. I like the idea of a 
multidimensional array getting into the core, something the community can 
agree on. I dont care what array package I am using, I just want it to be the 
accepted standard. I would like to hear what the numarray guys have to say 
about Numeric3.

Finally, I like the idea of scipy. I hope the community will come to an 
agreement and work within a framework that will result in the most efficient 
use of everyones time, the most logical interplay between packages, the least 
redundancy, and therefore the most gentle barrier to newcomers who want to 
experiment with scientific computing in Python.

Deepest regards,
Darren



From konrad.hinsen at laposte.net  Sun Feb  6 08:42:21 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sun Feb  6 08:42:21 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42055FA3.9050909@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu>
Message-ID: <ddf1e36c0b0166a093e3643bfb54e678@laposte.net>

On 06.02.2005, at 01:06, Travis Oliphant wrote:

> But MATLAB is a huge package.  SciPy has been trying to do the same  
> thing.  Plotting was needed and matplotlib did a great thing in  
> providing excellent plotting.  I've never understood the reason for  
> matplotlib to remain separate except as evidence of the great schism  
> that was occuring in the community.
>
> In fact, when Eric and I began scipy, I wanted to call it pylab  
> (you'll notice I have the pylab.sourceforge.net site).  I have fond  
> ties to MATLAB myself.

That's actually one of the non-technical issues I have with SciPy.  
SciPy "smells" like Matlab. Which is fine for those who know and like  
Matlab, but not for those (like me) who find Matlab rather incompatible  
with their brains. SciPy, like Matlab, is based on the principle that  
all operations should be expressed in terms of arrays. My personal  
approach is that operations should be expressed in terms of  
problem-specific classes in which the internal use of arrays is an  
implementation detail.

There are arguments for and against both approaches, and I think there  
is space (meme space) for both. I just mention this to point out that I  
don't expect SciPy to become the one and only scientific Python library  
that includes everything. I don't expect to contribute code to SciPy,  
for example. I wouldn't mind using it, of course, in the implementation  
of my classes, if and when the installation issues become less of a  
problem.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From rays at san.rr.com  Sun Feb  6 08:50:24 2005
From: rays at san.rr.com (RJ)
Date: Sun Feb  6 08:50:24 2005
Subject: The need for a unified numerical array object (was:
  [Numpy-discussion] Numeric3)
In-Reply-To: <55c61cbfde4feb91b4d8da721d1878a3@laposte.net>
References: <5.2.1.1.2.20050205080759.026eb700@pop-server.san.rr.com>
 <4203F7CE.8070403@ee.byu.edu>
 <420280AD.8080309@ee.byu.edu>
 <4202F6A0.3@ims.u-tokyo.ac.jp>
 <1f7baacd417e55debdde0351e463cf10@laposte.net>
 <4203CAD0.6050609@ee.byu.edu>
 <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
 <4203F7CE.8070403@ee.byu.edu>
 <5.2.1.1.2.20050205080759.026eb700@pop-server.san.rr.com>
Message-ID: <5.2.1.1.2.20050206081454.033838e0@pop-server.san.rr.com>

At 10:40 AM 2/6/2005 +0100, konrad.hinsen at laposte.net wrote:
>On 05.02.2005, at 17:28, RJ wrote:
>
>How does CPAN handle external dependencies? For example, what does CPAN do with a GTK interface for Perl when it finds that GTK is not available on the machine? Or when the package requires a Fortran compiler and none is installed?

It can fail for truly external requirements; module/bundle authors are largely responsible for declaring dependencies, and of course, they must be via CPAN, such as
http://cpan.org/authors/id/M/ML/MLEHMANN/Gtk-Perl-0.7009.tar.gz
among 7,582 others.

I mainly used ActivePerl's Perl Package Manager but apparently there is now a CPANPLUS.

http://aspn.activestate.com/ASPN/docs/ActivePerl/lib/CPAN.html
is the best resource for the guts of it.
In the above link, see:
The four CPAN::* Classes: Author, Bundle, Module, Distribution
and
POPULATE AN INSTALLATION WITH LOTS OF MODULES

A page search for "depend" points out what I think you're interested in, such as
" Please note, CPAN.pm does not know the dependency tree in advance and cannot sort the queue of things to install in a topologically correct order. It resolves perfectly well IFF all modules declare the prerequisites correctly with the PREREQ_PM attribute to MakeMaker. For bundles which fail and you need to install often, it is recommended sort the Bundle definition file manually. It is planned to improve the metadata situation for dependencies on CPAN in general, but this will still take some time."

At least here I didn't have to read any ($q*=2)+=$f=!fork;
As a second-place "winner" described:
"the winning program was not nearly as confusing as mine---it was actually very simple. If you fixed the indentation, it was quite straightforward. My program is difficult to understand even if you fix it up with good formatting and reasonable variable names. In the solution file I provided, I expanded it in this way, and it still took me about 2500 words to explain what was going on."

Ray Schumacher 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050206/33eda4f9/attachment-0001.html>

From verveer at embl.de  Sun Feb  6 08:56:24 2005
From: verveer at embl.de (Peter Verveer)
Date: Sun Feb  6 08:56:24 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <ddf1e36c0b0166a093e3643bfb54e678@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <ddf1e36c0b0166a093e3643bfb54e678@laposte.net>
Message-ID: <8a8a1ec91042a7507db0129c203d816b@embl.de>

On Feb 6, 2005, at 5:41 PM, konrad.hinsen at laposte.net wrote:

> On 06.02.2005, at 01:06, Travis Oliphant wrote:
>
>> But MATLAB is a huge package.  SciPy has been trying to do the same 
>> thing.  Plotting was needed and matplotlib did a great thing in 
>> providing excellent plotting.  I've never understood the reason for 
>> matplotlib to remain separate except as evidence of the great schism 
>> that was occuring in the community.
>>
>> In fact, when Eric and I began scipy, I wanted to call it pylab 
>> (you'll notice I have the pylab.sourceforge.net site).  I have fond 
>> ties to MATLAB myself.
>
> That's actually one of the non-technical issues I have with SciPy. 
> SciPy "smells" like Matlab. Which is fine for those who know and like 
> Matlab, but not for those (like me) who find Matlab rather 
> incompatible with their brains. SciPy, like Matlab, is based on the 
> principle that all operations should be expressed in terms of arrays. 
> My personal approach is that operations should be expressed in terms 
> of problem-specific classes in which the internal use of arrays is an 
> implementation detail.
>
> There are arguments for and against both approaches, and I think there 
> is space (meme space) for both. I just mention this to point out that 
> I don't expect SciPy to become the one and only scientific Python 
> library that includes everything. I don't expect to contribute code to 
> SciPy, for example. I wouldn't mind using it, of course, in the 
> implementation of my classes, if and when the installation issues 
> become less of a problem.

That more or less sums up my approach too. I tend to program in a mix 
of these two approaches.

I also think there is space for both approaches. But when code goes 
into a matlab-like package it may become less accessible to people who 
prefer to work in the more programming oriented style. The other way 
around is better: if it is accessible to all in the form of a more or 
less independent package, it can be used for both approaches. This is 
why I am also not going to program specifically for SciPy, but I would 
program my packages to be 'scipy-ready' if that is made easy...

Peter




From jdhunter at ace.bsd.uchicago.edu  Sun Feb  6 09:00:22 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Sun Feb  6 09:00:22 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42055DAF.5020601@ee.byu.edu> (Travis Oliphant's message of
 "Sat, 05 Feb 2005 16:58:39 -0700")
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	<1f7baacd417e55debdde0351e463cf10@laposte.net>
	<4203CAD0.6050609@ee.byu.edu>
	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>
	<42042318.2090704@ee.byu.edu>
	<c533ee29143e3286b075927540f5844b@embl.de>
	<42055DAF.5020601@ee.byu.edu>
Message-ID: <m3zmyhr5us.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Travis" == Travis Oliphant <oliphant at ee.byu.edu> writes:

    Travis> Matplotlib as well is reinventing stuff already available
    Travis> elsewhere.  SciPy has always tried to bring a matlab-like
    Travis> environment to the user.  That's always been my goal.  It
    Travis> is frustrating to see so many with the same goal work so
    Travis> independently of each other.  Sometimes I guess it is
    Travis> necessary so that new ideas can be tried out.  But, it
    Travis> sure seems to happen more often than I would like with
    Travis> Python and Science / Engineering.

True there has been some reinvention, but as Perry pointed out 95% of
what we do on matplotlib is plotting.  The "matlab-like" environment
is created primarily by simply importing symbols from Numeric, MLab,
FFT and so on.  Along the way I've added *a few* analysis functions,
eg matplotlib.mlab's hist, psd, polyfit, but these were usually to
support some plotting feature or just to have some fun with linear
algebra. 

The other thing to emphasize vis-a-vis the matplotlib duplicating
scipy effort discussion is that the matlab-like environment is only
one thing matplotlib tries to do, and in terms of size of the code
base it is a small part of it.  We also provide an API to embed
matplotlib in every major python GUI (WX, GTK, Tk, FLTK, and QT)
including basic event handling.  We support dynamic images, eg for
building data monitoring applications (performance can be an issue
here, since matplotlib is too slow for some applications).  And we try
to appeal, with some success, to non-scientists -- financial plots,
simple pie and bar charts for business types, webapp developers...  I
would like to see it become a comprehensive 2D plotting toolkit, not
just a toolkit for scientists and engineers.

Thus, is scipy the best place for it?

    Travis> Is it ownership that is a concern?  None of us involved
    Travis> with scipy have an alterior motive for trying to bring
    Travis> packages together except the creation of a single standard
    Travis> and infrastructure for scientific computing with Python.

Ownership is a concern, but perhaps not in the sense you mean it.  I
think I pointed out before that when I started looking at plotting
solutions, I seriously considered putting my effort into
chaco/kiva/agg.  I choose not to, because it is a complex package that
was hard for me to get my head around.  I was something of a python
newbie at the time, though I had coded extensively in other languages.
I didn't really get how all the parts worked together, didn't know
much or anything about swig, agg, quartz and freetype, etc.  It's a
daunting code base to walk up to, and I don't say this pejoratively
because it's an amazing piece of work, it's just a fact.

By writing my own plotting library, I owned the code in the sense that
I fully understood it, knew how the pieces fit together, knew how to
extend and fix it.  A newbie walking up to matplotlib today might have
the same reaction I did to chaco -- they might rather roll their own
plotting solution tailored to their own needs.  I call it the curse of
python -- coding in python is so fun and easy that it facilitates if
not encourages you to roll your own solutions.

If that is right, then it really argues for making libraries as
modular as possible, so that people who want to reuse just some part
of the code, eg the chaco freetype engine, or the matplotlib mathtext
engine or W3C font manager, or the scipy constrained optimizer, can do
so.  In practice, as library developers, we want people to use (and
contribute) to our code, not to take out some juicy morsel and use it
as the basis to build a competing project.  So in addition to the
extra work it takes to make a library truly modular (matplotlib
certainly isn't) there may be a disincentive to do it.

But if it is true that new developers will shy away from largish and
complex projects, and I think scipy, chaco and matplotlib are all in
that category, then we might all benefit by making our packages more
modular.  To me, this basically means that there is an easy,
advertised and documented way with examples on how to take a piece of
the library out and plug it into your own project.  I don't know that
any of us really have the time to do it, since everyone I know is
maximally committed already to providing what they provide in
conjunction with their paying work.  Or maybe scipy already does this
and I'm not aware of it.

    Travis> I keep hearing that people like the idea of a small subset
    Travis> of packages, but that is exactly what scipy has always
    Travis> been trying to be.  Perhaps those involved with scipy have
    Travis> not voiced our intentions enough or loud enough.

I think there are two ideas about what scipy should be.  Many view it
as an awesome repository of high performance numerical algorithms that
are array aware.  You seem to view it as an integrated matlab-like
environment, eg with plotting.  I tend toward the former view in part
because I was unsatisfied with the plotting capabilities, but as you
note there are ways around that, and I'm open minded about integration
if that is the best solution.

An integrated matlab like environment, as you suggested earlier, is a
big job, and in my view should be a project that is separate from
scipy *and* matplotlib.  I don't view scipy and matplotlib as
competitors in providing a matlab-like environment, but as two
components that interoperate well and can be used together with other
software to build such an environment.  There is really very little
duplication of effort in scipy and matplotlib, and we both share a
goal of providing a matlab-like environment.  

In my view, building a matlab-like environment should be an effort
separate from scipy, matplotlib, and chaco, which incorporate scipy's
algorithms, matplotlib and/or chaco's 2D vis, mayavi's 3D vis, the
ipython shell, a good code editor, a notebook interface, and a bit
more.  Ie, we should all focus on our core competencies :-) .  A bit
like what envisage appears to be doing but envisage is a framework and
this is an application -- perhaps envisage is the framework that will
build this application.

JDH



From verveer at embl.de  Sun Feb  6 09:15:25 2005
From: verveer at embl.de (Peter Verveer)
Date: Sun Feb  6 09:15:25 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <m3zmyhr5us.fsf@peds-pc311.bsd.uchicago.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <42055DAF.5020601@ee.byu.edu> <m3zmyhr5us.fsf@peds-pc311.bsd.uchicago.edu>
Message-ID: <a4e68c9cb13cf98afba94ed7d6bd2bc4@embl.de>

> I think there are two ideas about what scipy should be.  Many view it
> as an awesome repository of high performance numerical algorithms that
> are array aware.  You seem to view it as an integrated matlab-like
> environment, eg with plotting.  I tend toward the former view in part
> because I was unsatisfied with the plotting capabilities, but as you
> note there are ways around that, and I'm open minded about integration
> if that is the best solution.

To me it seems that there are two things people want and they find more 
or less important depending on the way they work. On the one hand there 
is a need for a repository of numerical algorithms that anybody can use 
and everybody needs.  On the other hand there is a desire by a large 
group of people, but not all, to have a matlab-like environment. It 
seems to me these things should be kept separate. I am not going to 
tell anybody what their code should look like, but if I need to 
download and install a complete environment to get to a repository of 
code, I rather avoid that.




From paul at pfdubois.com  Sun Feb  6 09:52:10 2005
From: paul at pfdubois.com (Paul F. Dubois)
Date: Sun Feb  6 09:52:10 2005
Subject: [Numpy-discussion] Numerical site taken down
In-Reply-To: <200502061116.35095.dd55@cornell.edu>
References: <4760baf2.ca2b9b41.81db100@expms6.cites.uiuc.edu> <200502061116.35095.dd55@cornell.edu>
Message-ID: <4206562E.4000207@pfdubois.com>

My hosting of the Numerical Python website is now ended. Recent 
complaints that it was out of date and confusing are correct. I have 
long ago asked the current maintainers to change the page at 
numpy.sf.net which currently redirects to this page. I didn't want to 
just take it down and have nothing so I left it; and of course no good 
deed goes unpunished.

pfdubois.com/numpy now refers you to the sf.net/projects/python page.


Darren Dale wrote:
> Hi,
> 
> As a relatively new user, I would like to contribute my experience to the 
> discussion. Late in a graduate program, I caught the bug and wanted to try 
> out python. Being established in Matlab, I was hoping to find a similar 
> collection of capabilities for Python, something that I could experiment with 
> and start learning fairly quickly. When I discovered Matplotlib, I felt home 
> free. Without too much effort, I had a plot on screen and felt 'ok, this is 
> feasible.' With the recent effort from Matplotlib and IPython, this should be 
> even more true for newcomers.
> 
> On the downside, I have been somewhat confused by the state of things 
> concerning Numeric and numarray. I think I can trace this confusion back to 
> the pfdubios.com/numpy site (linked from numpy.sourceforge.net), which is the 
> first hit I get doing a google search for either numpy or numerical python. 
> "If you are new to Numerical Python, please use Numarray. The older module, 
> Numeric, is unsupported." There is no date associated with the site, but it 
> still advertises release 22.0. In the near future, I strongly suggest editing 
> websites for misleading information.
> 
> So now I have both Numeric and numarray on my system, linked to the full, 
> atlas-tuned, lapack and blas libraries. I started out using numarray, 
> thinking it was in my long term interest, but I guess this is open to debate. 
> I was also hesitant to use scipy, because it relied on Numeric. Being an 
> uninformed newbie, I just wasnt able to make an informed decision (I am still 
> struggling with it).
> 
> A lot of what I have read recently is very encouraging to me. I like the idea 
> of bridging the API gap between numarray and Numeric. I like the idea of a 
> multidimensional array getting into the core, something the community can 
> agree on. I dont care what array package I am using, I just want it to be the 
> accepted standard. I would like to hear what the numarray guys have to say 
> about Numeric3.
> 
> Finally, I like the idea of scipy. I hope the community will come to an 
> agreement and work within a framework that will result in the most efficient 
> use of everyones time, the most logical interplay between packages, the least 
> redundancy, and therefore the most gentle barrier to newcomers who want to 
> experiment with scientific computing in Python.
> 
> Deepest regards,
> Darren
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 



From strawman at astraw.com  Sun Feb  6 10:13:22 2005
From: strawman at astraw.com (Andrew Straw)
Date: Sun Feb  6 10:13:22 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4202F6A0.3@ims.u-tokyo.ac.jp>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
Message-ID: <42065E31.20906@astraw.com>

While energy is put into re-coding an array package, I'd like to bring 
up an interesting idea brought up on matplotlib-users by Norbert Nemec: 
http://sourceforge.net/mailarchive/message.php?msg_id=10528079 

Norbert's idea is a solution to the problem of a numeric package 
overriding builtins when doing "from Numeric import *". Currently 
Numeric (23.6) overrides sum() on my system, and numarray (1.2a) 
overrides round(), sum(), and abs(). The respective mlab modules bring 
in min() and max().

(I think we all agree that "from blah import *" is a naughty thing to 
do, but people are going to do it, especially for interactive work, 
especially those who have come from namespace-less languages such as 
matlab.)

Here's Norbert's original email, which I think summarizes the idea nicely.

Norbert Nemec wrote:

> Look at these one by one:
> * abs() already calls an __abs__() method. The clean way to extend it, would 
> therefore be to give arrays such a method. This should solve the problem 
> completely.
> 
> * round() does not seem to be extendable in Python. Maybe we should propose to 
> change Python itself to introduce a __round__ method? That would only be 
> straightforward.
> 
> * min, max and sum are all based on iterating over a sequence. Maybe, one 
> should again have __min__, __max__ and __sum__ which should then be checked 
> by the builtin before falling back to iterating over the sequence? I could 
> imagine many kinds of containers that could optimize these operations. So 
> this would again be a detail that should be changed in Python itself. If the 
> builtin min() function would then also pass on keyword arguments, that would 
> solve our problem completely and thoroughly.
> 
> Does anybody have experience with discussions in the Python forum to estimate 
> how realistic such a PEP would be?
>





From stephen.walton at csun.edu  Sun Feb  6 10:34:24 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Sun Feb  6 10:34:24 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <8a8a1ec91042a7507db0129c203d816b@embl.de>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <ddf1e36c0b0166a093e3643bfb54e678@laposte.net> <8a8a1ec91042a7507db0129c203d816b@embl.de>
Message-ID: <420662E7.9020003@csun.edu>

Peter Verveer wrote:

>
> On Feb 6, 2005, at 5:41 PM, konrad.hinsen at laposte.net wrote:
>
>> My personal approach is that operations should be expressed in terms 
>> of problem-specific classes in which the internal use of arrays is an 
>> implementation detail.
>>
>> There are arguments for and against both approaches, and I think 
>> there is space (meme space) for both.
>
>
> That more or less sums up my approach too. I tend to program in a mix 
> of these two approaches.

I'm both largely a lurker here and a Python naif, at least when it comes 
to the more object oriented aspects.  I'd be interested in seeing 
snippets of code illustrating the second approach here, as opposed to 
the array oriented approach with which I'm familiar.




From verveer at embl.de  Sun Feb  6 14:52:10 2005
From: verveer at embl.de (Peter Verveer)
Date: Sun Feb  6 14:52:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420662E7.9020003@csun.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <ddf1e36c0b0166a093e3643bfb54e678@laposte.net> <8a8a1ec91042a7507db0129c203d816b@embl.de> <420662E7.9020003@csun.edu>
Message-ID: <9b3f26bae489530a547c30d6afc183e0@embl.de>

On Feb 6, 2005, at 7:33 PM, Stephen Walton wrote:

> Peter Verveer wrote:
>
>>
>> On Feb 6, 2005, at 5:41 PM, konrad.hinsen at laposte.net wrote:
>>
>>> My personal approach is that operations should be expressed in terms 
>>> of problem-specific classes in which the internal use of arrays is 
>>> an implementation detail.
>>>
>>> There are arguments for and against both approaches, and I think 
>>> there is space (meme space) for both.
>>
>>
>> That more or less sums up my approach too. I tend to program in a mix 
>> of these two approaches.
>
> I'm both largely a lurker here and a Python naif, at least when it 
> comes to the more object oriented aspects.  I'd be interested in 
> seeing snippets of code illustrating the second approach here, as 
> opposed to the array oriented approach with which I'm familiar.

I guess it it a bit hard just to give a snippet, but maybe I can 
explain what it means for me. I use numarray mainly as part of 
implementing command-line programs that perform specific data analysis 
tasks. So python with numarray replaced for me what used to done in C 
or Fortran. To make the code reusable and flexible I use the 
programming constructs that Python offers, like classes and such. 
Obviously array orient numerics remains a part of that, you have to get 
the actual calculations done using the array functions. I usually have 
the array part still exposed at some level, but from what Konrad said, 
it appears he takes that a step further and he hides the actual 
mechanics of doing the calculations with arrays completely away.

I suppose most people that use python+numarray/numeric will do this to 
differents extents, but I guess many people like to have a more 
interactive enviroment, where you can type your commands and see the 
results straight away, which I guess matlab is very good at. I don't do 
that much, I prefer to design and write a program that does the task 
that needs to be done. Python+numarray provide a great programming 
language compared to C or Fortran in terms of programmer productivity. 
In this light, it may be understandable that I am not so much 
interested in having a python based numerical enviroment as much as in 
having a good repository of numeric algorithms that I can use in my 
programs.

Peter




From mdehoon at ims.u-tokyo.ac.jp  Sun Feb  6 19:58:22 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Sun Feb  6 19:58:22 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <m3zmyhr5us.fsf@peds-pc311.bsd.uchicago.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	<1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu>	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu>	<c533ee29143e3286b075927540f5844b@embl.de>	<42055DAF.5020601@ee.byu.edu> <m3zmyhr5us.fsf@peds-pc311.bsd.uchicago.edu>
Message-ID: <4206E720.60908@ims.u-tokyo.ac.jp>

>"Travis" == Travis Oliphant <oliphant at ee.byu.edu> writes:
> 
> 
>     Travis> Matplotlib as well is reinventing stuff already available
>     Travis> elsewhere.  SciPy has always tried to bring a matlab-like
>     Travis> environment to the user.  That's always been my goal.  It
>     Travis> is frustrating to see so many with the same goal work so
>     Travis> independently of each other.  Sometimes I guess it is
>     Travis> necessary so that new ideas can be tried out.  But, it
>     Travis> sure seems to happen more often than I would like with
>     Travis> Python and Science / Engineering.
> 
Speaking of code reinvention, it seems that SciPy contains some very 
useful code that overlaps with existing Python or Numerical Python code. 
Such code would reach more users if it were better integrated with 
existing code outside of SciPy. For example, I would love to see SciPy's 
additions to distutils integrated with Python's distutils, e.g. if that 
would allow users to easily compile fortran code.



From andrewm at object-craft.com.au  Mon Feb  7 00:24:28 2005
From: andrewm at object-craft.com.au (Andrew McNamara)
Date: Mon Feb  7 00:24:28 2005
Subject: [Numpy-discussion] MA and iterators
Message-ID: <20050207082300.5D00D3C889@coffee.object-craft.com.au>

I did some benchmarking today and discovered that iterating over MA
arrays is 14 times slower than iterating over a Numeric array (or a
python list). One simple solution to this is to add iterator support to
the MaskedArray class. To prove the principle, I patched MaskedArray on
the fly:

    import MA

    def ma__iter__(self):
        "Get an MA iterator."
        def scaler_iter(d, m):
            for i, v in enumerate(d):
                if m[i]:
                    v = masked
                yield v
        def array_iter(d, m, fill_value):
            ss = d.spacesaver()
            tc = d.typecode()
            for i, v in enumerate(d):
                yield MA.array(v, typecode=tc, copy=1, savespace=ss, 
                               mask=m[i], fill_value=fill_value)
        if self._mask is None:
            return iter(self._data)
        if len(self.shape) > 1:
            return array_iter(self._data, self._mask)
        else:
            return scalar_iter(self._data, self._mask, self.fill_value())

    MA.MaskedArray.__iter__ = ma__iter__

Note that this implementation requires generator support.

-- 
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/



From konrad.hinsen at laposte.net  Mon Feb  7 01:48:28 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 01:48:28 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420662E7.9020003@csun.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <ddf1e36c0b0166a093e3643bfb54e678@laposte.net> <8a8a1ec91042a7507db0129c203d816b@embl.de> <420662E7.9020003@csun.edu>
Message-ID: <9d1481068d8a74f9db31251f20287340@laposte.net>

On 06.02.2005, at 19:33, Stephen Walton wrote:

> I'm both largely a lurker here and a Python naif, at least when it 
> comes to the more object oriented aspects.  I'd be interested in 
> seeing snippets of code illustrating the second approach here, as 
> opposed to the array oriented approach with which I'm familiar.
>
You can find lots of examples in my ScientificPython library 
(http://dirac.cnrs-orleans.fr/ScientificPython/). Some examples:

- Geometry. Array people would use arrays of shape (3,) to represent 
points in 3D space. I use objects of class Scientific.Geometry.Vector. 
Array people would also store rotation matrices in arrays of shape (3, 
3), whereas I use the class Scientific.Geometry. 
Transformation.Rotation.

- Function values on a grid. Array people would have an array of x 
values (the grid) plus an array of y values, or a 2D array combining 
both. I use instances of Scientific.Functions.InterpolatingFunction.

- Histograms. Array people would store a histogram in an array of bin 
counts. I use the class Scientific.Statistics.Histogram.Histogram.

The rational behind these choices is the usual one cited for OO 
approaches: the use of abstractions to increase readability and to 
permit alternate implementations.

However, my experience with both programming and physics teaching shows 
that the level of comfort for abstraction levels is not the same for 
everyone. For example, some people (including myself) get a better 
understanding of quantum mechanics when working with quantum states as 
abstractions, whereas others prefer to work with wavefunctions (which 
are the space representations of quantum states). Fortunately there are 
textbooks for both groups.

My impression is that the optimal abstraction level is higher for 
scientists than for engineers, though my data is perhaps a bit to weak 
to back up such a conclusion. It doesn't really matter either.

Konrad.




From konrad.hinsen at laposte.net  Mon Feb  7 01:57:23 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 01:57:23 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42065E31.20906@astraw.com>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com>
Message-ID: <4ad81937f722051f27b37915225a1563@laposte.net>

On 06.02.2005, at 19:13, Andrew Straw wrote:

> While energy is put into re-coding an array package, I'd like to bring 
> up an interesting idea brought up on matplotlib-users by Norbert 
> Nemec: http://sourceforge.net/mailarchive/message.php?msg_id=10528079

My summary of what he is saying is "define flexible enough interfaces, 
and then people can write implementations the fit their needs". Which 
is a very good idea.

Note that all his examples are cases where Python chose to use 
functions where OO principles would suggest methods. If abs(), round(), 
min() etc. were methods from the start, there wouldn't be any problem 
to solve (of course they are not for a very good reason: notational 
habit). The same applies to the math module or the ufunc module in 
Numeric - there would be no need for ufunc if math simply defined 
syntactic sugar for method calls.

I wonder if it is possible to come up with an abstract API for arrays 
that would permit alternate implementations to be chosen as late as 
possible, even at runtime. That could help to get over the 
Numeric/numarray split.

Konrad.




From faltet at carabos.com  Mon Feb  7 08:07:40 2005
From: faltet at carabos.com (Francesc Altet)
Date: Mon Feb  7 08:07:40 2005
Subject: [Numpy-discussion] Proposal for making of Numarray a real Numeric 'NG'
In-Reply-To: <4205C5DE.80504@ee.byu.edu>
References: <200501211413.51663.faltet@carabos.com> <4205C5DE.80504@ee.byu.edu>
Message-ID: <200502071704.56135.faltet@carabos.com>

A Diumenge 06 Febrer 2005 08:23, Travis Oliphant va escriure:
> >The real problem for Numarray: Object Creation Time
> >===================================================
> >
> >On the other hand, the main drawback of Numarray vs Numeric is, in my
> >opinion, its poor performance regarding object creation. This might look
> >like a banal thing at first glance, but it is not in many cases. One
> > example recently reported in this list is:
>
> Ah, and there's the rub.  I don't think this object creation time will
> go away until Numarray's infrastructure becomes essentially like that of
> Numeric.  One tight object all in C.   Getting it there seems harder
> than fixing Numeric, with the additional features of Numarray.

Well, I guess I have to believe in you provided that I'm not an expert
on these issues and you are. However, I think Todd Miller has some
hopes that Numarray can eventually get rid of this object creation
latency. That would be very nice as well.

In any case, and as a person already said before during the message
storm that took place in this distribution list this weekend, I was
rather sceptical when you first announced the project for Numeric 3.0.
But now, and after reading this bunch of messages, I've changed my
mind and think (hope) that your approach can be very beneficial for
unifying the Numeric and numarray split.

I whish you the best of luck in your attempt, and I'll keep an eye on
Numeric 3.0 evolution. It would be wonderful if you finally succeed,
and we can finally have a standard numerical package for Python,
irregardless it will be in the Standard Library or not.

Cheers!,

-- 
>qo<   Francesc Altet ? ? http://www.carabos.com/
V ?V   C?rabos Coop. V. ??Enjoy Data
 ""




From juenglin at cs.pdx.edu  Mon Feb  7 08:38:28 2005
From: juenglin at cs.pdx.edu (Ralf Juengling)
Date: Mon Feb  7 08:38:28 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4ad81937f722051f27b37915225a1563@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	 <42065E31.20906@astraw.com>  <4ad81937f722051f27b37915225a1563@laposte.net>
Message-ID: <1107794238.10581.16.camel@alpspitze.cs.pdx.edu>

On Mon, 2005-02-07 at 01:56, konrad.hinsen at laposte.net wrote:
> Note that all his examples are cases where Python chose to use 
> functions where OO principles would suggest methods. If abs(), round(), 
> min() etc. were methods from the start, there wouldn't be any problem 
> to solve (of course they are not for a very good reason: notational 
> habit). The same applies to the math module or the ufunc module in 
> Numeric - there would be no need for ufunc if math simply defined 
> syntactic sugar for method calls.

No, the idea behind using function calls rather than method calls
is actually a little more than just a difference in syntax. It 
allows for a generic implementation that works in all cases of 
reasonable input, that is, it also works in cases where no efficient
implementation is provided by the input object. A simple example:

>> class DumbList(list):
...     def __len__(self):
...         raise NotImplementedError
...
>>> def len(sequence):
...     try:
...         return sequence.__len__()
...     except:
...         n=0
...         for item in sequence: n+=1
...         return n
...
>>>
>>> len([1,2,3])
3
>>> len(DumbList([1,2,3]))
3


Ralf


> 
> I wonder if it is possible to come up with an abstract API for arrays 
> that would permit alternate implementations to be chosen as late as 
> possible, even at runtime. That could help to get over the 
> Numeric/numarray split.
> 
> Konrad.
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion




From strang at nmr.mgh.harvard.edu  Mon Feb  7 08:42:17 2005
From: strang at nmr.mgh.harvard.edu (Gary Strangman)
Date: Mon Feb  7 08:42:17 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4ad81937f722051f27b37915225a1563@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
Message-ID: <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>

Hi all,

[Long post: a long-time lurker here sharing his numerical 
experiences/input re: "the numeric split" and Numeric3.]

I'm a long-time python user (and follower of this list), a heavy user in 
the numerical realm, and an "incidental" contributor to SciPy (statistical 
functions in stats.py). First, I'm impressed (and indeed amazed) at all 
the hard work folks have put in--usually without pay--to make Numpy and 
Numarray and SciPy and Matplotlib (to name the major players in this 
discussion). I am indebted to each of you. Second, I must say that I think 
this protracted discussion is EXACTLY what the python numerical community 
needs. It appears as though we have critical mass in terms of code and 
interest (and opinions), and just need to bring them all together.

Since the inception of numarray, I've just been standing back and waiting 
to see how this all sorts itself out. My stats functions work for lists 
and numpy arrays. I didn't want to convert them to numarray (given my lack 
of spare time) unless that was going to be the "new path". It appears, 
however, even after all this time, there isn't (quite) a consensus on a 
new path. After the recent message-storm, however, I am very hopeful. I 
see 4 issues at stake here, with the caveat that I'm not the code writer, 
just a user ...

1) Multiarray in Python core: I agree that this (as already stated) is (1) 
mostly irrelevant for heavy-duty numerial folks, BUT (2) is critical to 
provide for python a standardized exchange data format. Being able to 
trivially (i.e., out of the python box) load, save, pickle and load again 
on a new platform N-D array objects would be a big deal for me (and many I 
work with). Such a core object can't favor any particular size array ... 
so it would need to provide good (or excellent) performance on both little 
arrays (a la numpy) and on big arrays (a la numarray). To be in keeping 
with other python objects, it seems this object would need to be tight, 
fast and easily extensible. I *think* this is exactly what Numeric3 is 
intended to do. Getting this right is tricky, but it seems like current 
solutions are EXTREMELY close.

2) Numerical function "packaging": Looking at this from a distance (i.e., 
as a user) numerical packaging is too complex. The python spirit seems to 
call for being a bit more of a splitter (and encapsulator) than a lumper. 
For example, to do web programming in python, one often depends on several 
separate modules (html, xml, cgi, etc) rather than one all-encompassing 
one. To give numerical work the same modular feel (as well as structure 
and insulation from installation headaches), it seems that collections of 
numerical operations should be similarly organized on themes (e.g., 
timeseries analysis, morphology (nd_image?), statistics (stats.py), etc). 
This way, if you're doing timeseries analysis you import the relevant 
modules and go to work ... no worries about installing stuff required for 
morphology or statistics that you don't need. I realize this might require 
(in some cases) more refactoring, but I don't think I'm supporting 
anything *that* different from what already exists. Granted, the notion of 
what's "basic" vs. advanced is relative (e.g., where do you put fft, or 
linear_algebra?). But if made modular and encapsulated (e.g., an fft.py, 
linear_algebra.py, integration.py, morphology.py) and made available both 
individually and as part of one or more suites--see #4 below, it's easier 
to build on existing code rather than reinvent. Interestingly, although 
not obvious, this is how Matlab works too. Your first $500 pays for basic 
array-based functionality (fft, psd, etc). Then there are add-on toolkits 
(at $500 each) specifically for timeseries analysis, imaging, wavelets, 
engineering simulation, etc.

3) Plotting: Until perhaps a year ago, I did almost all my computations in 
python, then saved data out to disk, and read it into matlab to plot it. I 
hated that situation, but it was the only way to quickly and easily look 
at data interactively, with zoom, easy subplotting, etc. Matplotlib has 
all but solved this problem (thanks!!). John indicates that the ultimate 
goal with matplotlib is to provide plotting, not just scientific plotting, 
which is even better! In that case, though, and in keeping with my 
previous comment, perhaps the name matplotlib is a little misleading 
(suggesting scientific plotting only). Again, if I were familiar with 
python but just starting timeseries analysis, I would expect to load my 
data into a (multiarray) python object, import timeseries.py, import 
plotlib.py (i.e., matplotlib) and go to work doing timeseries analysis ... 
be that at LLNL, Wall St, or in my neuro lab.

4) Matlab-like Environment: Both SciPy and Matplotlib have a stated goal 
of creating a matlab-style environment. This is great, as it might help 
wean more folks off of Matlab or IDL and into the python community. 
However, I think that this (as has been suggested ... sorry, I forgot who) 
should be a separate goal from any of the above. Building an environment 
with python is different from providing functionality to python (think 
website design *environment* vs. tools for handling web content ... 
they're different). SciPy, with it's integration goal, plus matplotlib's 
plotting goal would be an outstanding combination to this end.

In sum, I pretty much agree with most previous contributors. With one 
exception. I do agree with Konrad that, theory, the best analysis approach 
is to build a custom class for each domain of investigation. I've even 
done that a couple times. However, of the dozens of researchers (i.e., 
users, not developers) I've talked about this with, almost none have the 
time (or even desire) to design and develop a class-based approach. Their 
goal is data analysis and evaluation. So, they pull the data into python 
(or, more typically, matlab or IDL), analyze it, plot the results, try to 
get a plot that's pretty enough for publication, and then go on to the 
next project ... no time for deciding what functionality should be a 
method or an attribute, subclassing hierarchies, etc. In those rare 
cases where they have the desire, matlab (I don't know about IDL) doesn't 
give you the option, meaning that users (as opposed to developers) 
probably won't go the (unfamiliar and more time consuming) route.

I apologize for the long post that "simply" supports others' opinions, 
particularly when my opinion cannot count for much (after all, I'm not 
likely to be doing much of the coding). But, I did want to express my 
appreciation for ALL the hard work that's been done, and to give the 
strongest encouragement to hashing things out now. I would LOVE to see 
some consensus on (1) what a core multiarray object should look like, 
(2-3) how to imbue python with numerical functionality and plotting for 
generations to come ;-) and (4) to create environments for scientific 
exploration within python. I think we're SOOO close ...

-best
Gary

--------------------------------------------------------------
Gary Strangman, PhD        |  Director, Neural Systems Group
Office: 617-724-0662       |  Massachusetts General Hospital
Fax:    617-726-4078       |  149 13th Street, Ste 10018
strang/@/nmr.mgh.harvard.edu |  Charlestown, MA  02129
http://www.nmr.mgh.harvard.edu/NSG/



From konrad.hinsen at laposte.net  Mon Feb  7 11:41:16 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 11:41:16 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1107794238.10581.16.camel@alpspitze.cs.pdx.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <4ad81937f722051f27b37915225a1563@laposte.net> <1107794238.10581.16.camel@alpspitze.cs.pdx.edu>
Message-ID: <0b7590e6e7b4b325bf3acdaf46aecce7@laposte.net>

On 07.02.2005, at 17:37, Ralf Juengling wrote:

> No, the idea behind using function calls rather than method calls
> is actually a little more than just a difference in syntax. It
> allows for a generic implementation that works in all cases of
> reasonable input, that is, it also works in cases where no efficient
> implementation is provided by the input object. A simple example:

True, but only due to the "add-on" character of arrays in Python. If 
arrays had been part of the initial design, we would most certainly 
have type promotion from lists to arrays in the same way as we have 
type promotion from float to complex.

Note also that this approach is anything but modular or extendable. If 
several modules went that way, they would not be interoperable (which 
is in part the problem with the numpy/numarray split).

There is also a glaring hole in the OO approach in that there is no way 
to add functionality (methods) to an existing object. The only solution 
is subclassing, but then all object creators need to be changed to 
those for the subclass, which is not practical. Unlike other OO 
languages, Python makes it possible to add methods to existing classes, 
but this always looks like a bad hack.

Konrad.




From konrad.hinsen at laposte.net  Mon Feb  7 11:47:12 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 11:47:12 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
Message-ID: <d7673a2db58066510577554360010930@laposte.net>

On 07.02.2005, at 17:41, Gary Strangman wrote:

> This way, if you're doing timeseries analysis you import the relevant  
> modules and go to work ... no worries about installing stuff required  
> for morphology or statistics that you don't need. I realize this might  
> require (in

The amount of installed code is rarely an issue, unused code costs no  
more than disk space. So as long as installation is 100% trouble free,  
I don't mind big packages. They are easier to handle for the  
developers, after all.

> In sum, I pretty much agree with most previous contributors. With one  
> exception. I do agree with Konrad that, theory, the best analysis  
> approach is to build a custom class for each domain of investigation.  
> I've even done that a couple times. However, of the dozens of  
> researchers (i.e., users, not developers) I've talked about this with,  
> almost none have the time (or even desire) to design and develop a  
> class-based approach. Their goal is data

I wasn't thinking about users, but about developers. Users with no  
desire or no competence to develop non-trivial code will take whatever  
is available anyway. But they don't write complex scripts either.

My personal experience as a library developer is that defining the  
appropriate classes is a small investment with a big payoff, but I  
don't mind if anyone else disagrees.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From Chris.Barker at noaa.gov  Mon Feb  7 11:58:22 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Mon Feb  7 11:58:22 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42055FA3.9050909@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	<1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu>	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu>	<c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu>
Message-ID: <4207C816.1000707@noaa.gov>

Travis Oliphant wrote:

> So, it is very concerning to me to see matplotlib try to become what 
> scipy is already trying to become. 

Me too. I'm ecstatic about matplotlib, but disappointed that it has as 
broad a goal as it does (and the matlab-like syntax, I hope to 
contribute more to alternatives).

It's interesting that SciPy and matplotlib really do have much the same 
goal, but taken from different tacks: SciPy focused on the computational 
functionality, and matplotlib on the plotting. However, what's become 
really clear in this discussion is that a good scientific math 
environment really needs both, and they need to be well integrated.

I'm optimistic that we can all work together to get that: One set of 
packages that work together to do everything we all need. As someone 
else on this thread mentioned, most of the individual functionality is 
all there, what's mostly need now is for it all to be packaged well and 
work together well. At the core of that is the numarray/NumPy 
unification: I hope Numeric will get us there.

One question, Travis: Why did you decide to bring numarray stuff into 
the Numeric code base, rather than the other way around? What I've 
gathered is that the only real show-stopper with numarray is array 
creation speed. Is that really impossible to fix without a complete 
re-structuring?

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From Chris.Barker at noaa.gov  Mon Feb  7 11:58:32 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Mon Feb  7 11:58:32 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <55c095f595dfec3e16579094faea14f2@laposte.net> <4203FCD8.8070204@noaa.gov> <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net>
Message-ID: <4207C83D.5040506@noaa.gov>

konrad.hinsen at laposte.net wrote:
>> By the way, here OS-X is much better than other systems, it comes with
>> lapack built in! (Veclib)
> I didn't know it has lapack, I thought it was only BLAS!

I may be wrong about that. It certainly has BLAS, but I didn't patch 
Numeric's setup.py, so I'm not sure how that worked.


> With Apple's Python, it's command line 
> installation (which I don't mind,

Neither do I, so maybe that's why. Frankly, though, fink is command line 
also, and a really ugly one, in my experience!

> I certainly agree on that. I curse Fink about once a week. But in the 
> absence of an alternative, Fink is better than no Fink.

I number of folks seem to think darwinports is a better alternative. I 
think, if  nothing else, they make greater effort to use Apple-supplied 
libraries, when they exist. I still want to try Gentoo. I love it for 
Linux (once the system is installed, anyway!)

> Attributing fault makes little sense in the OpenSource world.

Fault wasn't really the right word. The question that is relevant in 
Open Source is: Where can I contribute to address this problem?

>>  there aren't all that many people using OS-X developing Python 
>> packages, and Apple has, of course, made OS-X just different enough 
>> from other unices to require some work (or take the easy way and use 
>> fink, but then it

> Not really, you can do command-line installation ("python setup.py...") 
> just like on any other Unix, usually without any modification. The 
> problem is that Mac users expect installers.

Anything that you can setup.py --build you can  :

$ bdist_mpkg

And get a nifty Apple-style point and click package. This is thanks to 
Bob Ippolito's Py2App.

However, the big problem is dependencies. Not only are Linux users more 
accustomed to the command line, but Linux distros come with far more 
open source libs. Also, Apple has not provided a standard package 
manager. I can build an rpm (or deb, or ebuild) for a given version of 
Linux, and it fits in with the rest of the system, and users can use the 
same system to inform them about, and help them install, dependencies. 
With OS-X, you're left to fend for yourself, combing the web for 
tarballs, fink or darwinports packages.

> It should be possible to extend distutils for making Mac installers just 
> like it can make RPMs and Windows installers.

It's been done. See above.

> Right. But I still wonder why non-scientific projects (e.g. Python 
> itself, or Linux) don't suffer so much from this problem.

I think they do. I can't say I've seen a distinction. If there is one, 
perhaps it's because much of the programming done on scientific projects 
is done by scientists, rather than programmers, we're even more inclined 
to be building something primarily because we need it.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From oliphant at ee.byu.edu  Mon Feb  7 12:01:26 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Mon Feb  7 12:01:26 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <ddf1e36c0b0166a093e3643bfb54e678@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu> <c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <ddf1e36c0b0166a093e3643bfb54e678@laposte.net>
Message-ID: <4207C8B9.4010806@ee.byu.edu>

konrad.hinsen at laposte.net wrote:

> On 06.02.2005, at 01:06, Travis Oliphant wrote:
>
>> But MATLAB is a huge package.  SciPy has been trying to do the same  
>> thing.  Plotting was needed and matplotlib did a great thing in  
>> providing excellent plotting.  I've never understood the reason for  
>> matplotlib to remain separate except as evidence of the great schism  
>> that was occuring in the community.
>>
>> In fact, when Eric and I began scipy, I wanted to call it pylab  
>> (you'll notice I have the pylab.sourceforge.net site).  I have fond  
>> ties to MATLAB myself.
>
>
> That's actually one of the non-technical issues I have with SciPy.  
> SciPy "smells" like Matlab. Which is fine for those who know and like  
> Matlab, but not for those (like me) who find Matlab rather 
> incompatible  with their brains. SciPy, like Matlab, is based on the 
> principle that  all operations should be expressed in terms of arrays. 
> My personal  approach is that operations should be expressed in terms 
> of  problem-specific classes in which the internal use of arrays is 
> an  implementation detail.

I think I understand better what Konrad is saying than I did several 
years ago.   SciPy may have started with the idea of being something 
like MATLAB, but in fact, scipy has lot's of class-based approaches:  
polynomials, random variables, gridding, ode solvers, just to name a few.

Basically, scipy just wants code that works and can be useful to 
others.  I don't think you'll find that scipy is opposed to class-based 
solutions at all.   We do tend to like names that are shorter (or at 
least aliases that are shorter, so that command-line interactive use is 
easier). 

Now, I think of scipy really just as a gathering place for Numerical 
algorithms.  I think it is clear that scipy needs to work on it's 
modularity, taking into account some of the great comments that have 
been made on this list if it is to be what it is intended to be.

-Travis










>
> There are arguments for and against both approaches, and I think 
> there  is space (meme space) for both. I just mention this to point 
> out that I  don't expect SciPy to become the one and only scientific 
> Python library  that includes everything. I don't expect to contribute 
> code to SciPy,  for example. I wouldn't mind using it, of course, in 
> the implementation  of my classes, if and when the installation issues 
> become less of a  problem.
>
> Konrad.
> -- 
> ------------------------------------------------------------------------ 
> -------
> Konrad Hinsen
> Laboratoire Leon Brillouin, CEA Saclay,
> 91191 Gif-sur-Yvette Cedex, France
> Tel.: +33-1 69 08 79 25
> Fax: +33-1 69 08 82 61
> E-Mail: khinsen at cea.fr
> ------------------------------------------------------------------------ 
> -------
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion





From Chris.Barker at noaa.gov  Mon Feb  7 12:04:26 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Mon Feb  7 12:04:26 2005
Subject: [Numpy-discussion] scipy, matplotlib, and installation dependencies.(previously:
  Numeric3)
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCGEAFFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCGEAFFJAA.perry@stsci.edu>
Message-ID: <4207C995.7010408@noaa.gov>

Perry Greenfield wrote:
>  Until we do, I'd favor an approach
> that shoots for a smaller, easily installable core, with a simple
> way of asking for optional packages (having users edit setup.py
> is probably asking too much).

It's well worth looking into MacPython's PackMan (AKA PIMP). Jack Jansen 
designed it to be non-platform specific. Indeed, a wxPython GUI from end 
has already been developed. Who knows whether it could ever become the 
definitive CPAN-like tool for Python, but it could be a great way to 
distribute SciPy packages.

For those who have never seen it, with a GUI front-end, you point it at 
a URL (there is a default one), and it gives you a list of packages 
available, flagged as to whether they are installed or not. You click on 
what you want, it gets downloaded and installed. That's it. It works 
very nicely for OS-X, though you do need someone(s) to maintain the 
package repository.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From oliphant at ee.byu.edu  Mon Feb  7 12:16:23 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Mon Feb  7 12:16:23 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
Message-ID: <4207CC4C.8060603@ee.byu.edu>

Gary Strangman wrote:

> To be in keeping with other python objects, it seems this object would 
> need to be tight, fast and easily extensible. I *think* this is 
> exactly what Numeric3 is intended to do. Getting this right is tricky, 
> but it seems like current solutions are EXTREMELY close.

I think it's close too, that's why I'm willing to do this, right now.

> already exists. Granted, the notion of what's "basic" vs. advanced is 
> relative (e.g., where do you put fft, or linear_algebra?). But if made 
> modular and encapsulated (e.g., an fft.py, linear_algebra.py, 
> integration.py, morphology.py) and made available both individually 
> and as part of one or more suites--see #4 below, it's easier to build 
> on existing code rather than reinvent.

There seems to be several users on this list that agree.  I would agree 
too, but would also state that Python also has the idea of "batteries 
included" meaning that the html, cgi, and xml libraries are all 
installed using a common installer.   SciPy has tried to be modular, too 
but have a common installer.  SciPy is an umbrella package with several 
sub-packages.    Sometimes it is hard to be modular when a sub-package 
depends on linear algebra routines for example. 

I think that a basic install should always include linear algebra, 
random numbers, and fft code.  I don't think this should live under a 
Numeric package because if we are to get arrayobjects into the core, 
then we need to develop some other place for these things to live.  That 
is essentially what scipy is trying to be -- a place for these extras 
and additional numeric routines to live.   Perhaps we underestimated the 
problems with using FORTRAN (but I still don't think so given that g77 
is available wherever gcc is). 

With f2py, FORTRAN is incredibly quick to develop Python modules with.  
So, I would have a hard time saying that we should not use FORTRAN.  I 
can conceed that we should separate modules that need FORTRAN from those 
that need only C(++). 

> problem (thanks!!). John indicates that the ultimate goal with 
> matplotlib is to provide plotting, not just scientific plotting, which 
> is even better! In that case, though, and in keeping with my previous 
> comment, perhaps the name matplotlib is a little misleading 
> (suggesting scientific plotting only). Again, if I were familiar with 
> python but just starting timeseries analysis, I would expect to load 
> my data into a (multiarray) python object, import timeseries.py, 
> import plotlib.py (i.e., matplotlib) and go to work doing timeseries 
> analysis ... be that at LLNL, Wall St, or in my neuro lab.

I think matplotlib is great.   SciPy's plotting methods were always 
considered stop-gap (functional but not much else).   I would have much 
rather seen matplotlib integrate in with the scipy effort, though.   
Perhaps these modularity issues would be solved already, had this been 
attempted from the start.

-Travis








From oliphant at ee.byu.edu  Mon Feb  7 12:38:17 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Mon Feb  7 12:38:17 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4207C816.1000707@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	<1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu>	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu>	<c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu> <4207C816.1000707@noaa.gov>
Message-ID: <4207D18A.1070309@ee.byu.edu>

>
> It's interesting that SciPy and matplotlib really do have much the 
> same goal, but taken from different tacks: SciPy focused on the 
> computational functionality, and matplotlib on the plotting. However, 
> what's become really clear in this discussion is that a good 
> scientific math environment really needs both, and they need to be 
> well integrated.

The problem was that we needed someone like John to join us on the scipy 
effort to get the plotting working well.   Our team lacked someone with 
his skill.  All of us working on SciPy would have loved to work with 
him.  So, I have nothing but respect for John and his efforts.  I just 
wish we at SciPy were better recruiters :-)

> I'm optimistic that we can all work together to get that: One set of 
> packages that work together to do everything we all need. As someone 
> else on this thread mentioned, most of the individual functionality is 
> all there, what's mostly need now is for it all to be packaged well 
> and work together well. At the core of that is the numarray/NumPy 
> unification: I hope Numeric will get us there.

Yes, I think the numarray / Numeric split has been one of the biggest 
problems. 

>
> One question, Travis: Why did you decide to bring numarray stuff into 
> the Numeric code base, rather than the other way around? What I've 
> gathered is that the only real show-stopper with numarray is array 
> creation speed. Is that really impossible to fix without a complete 
> re-structuring?

A great question.    People deserve to hear what I think even if they 
disagree with me, so here is a summary of the issues I'm concerned about 
with numarray.  The basic answer to the question, is that I feel that 
numarray is too different structurally (i.e. the classes and objects 
that get defined) from Numeric and some of these differences are causing 
the speed issues.  I felt it would be too much work
to adapt numarray to the Numeric structure than adapt Numeric to the 
numarray features. 

Here are some specifics.

1)  Numarray did not seem to build on Numeric at all.  It has thrown out 
far too much.  As just one example, the ufunc object in Numeric is a 
pretty good start, but numarray decided to completely change the 
interface for reasons that I do not understand.   One result of this 
decision is that numarray still does not provide a similar C-API for the 
creation of ufuncs that Numeric did.  

2) Basic numarray _ndarray C object is way too big.  numarray added too 
many things to the underlying C-structure of an arrayobject.  I think 
this will have small-array performance implications.

3) While prototyping in Python was a good idea. Numarray should have 
moved the entire object to C and not left so many things to the Python 
level.  I don't think there should be a Python-level arrayobject as the 
basic class (except for RecordArrays).    I think this move must still 
be done to solve the speed issues, and I see this has much harder than 
fixing Numeric which is already all in C.

4) The numarray C-API seems way too big.  There are so many, seemingly 
repeated calls that seem to do the same thing. Are all of these API 
calls really necessary? 

5) Numarray builds fundamentally on Int16, Int32, and Float32 objects.  
I understand the need for this in many applications, but some users will 
still need a way to define arrays based on the c-type that is desired.  
In addition, as the mapping from bit-width to c-type is quite platform 
dependent, this needs to be done more carefully.

I'm not looking to debate these issues, because I agree that other 
opinions may be valid, I could be wrong, and the debate will just 
distract me.   But, fundamentally, my decision was based on a gut-feel 
influenced no doubt by my familiarity and appreciation of the Numeric 
code base.   If I'm wrong it will be apparent in a couple of months. 

-Travis










From Chris.Barker at noaa.gov  Mon Feb  7 12:42:29 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Mon Feb  7 12:42:29 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42042318.2090704@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu>
Message-ID: <4207D285.70103@noaa.gov>

Travis Oliphant wrote:

> Perhaps it would be good to get a list of which platforms actually
> need binaries for people to look more favorably at SciPy.   I can see
> that OS X is on the top of that list, surely somebody could
> distribute a copy of a binary for that platform (I know of people who
> have it installed).

It is MUCH easier to get a complex package working on OS-X than to make
a binary of it. Not because it's hard to make a binary (it's very, very
easy, thanks to Py2App), but because, in general, open source software
ends up being dynamically linked against a bunch of libraries. Getting
those libs installed on your system is a lot easier than figuring out
how to get the package to statically link to them (or install them as
part of your package)

I know this because I spent a few days last week getting a binary of
matplotlib together.

If there is a small group of folks that would like to work on making a
binary SciPy for OS-X, I'd be glad to work with others to do it. I doubt
I'll do it myself, however.

By the way, does anyone know a way to get distutils to statically link a
given library? So far, I've done the trick of making sure it can only
find the *.a, and not a *.dylib, but this is kind of a pain, and feels
like a kludge.

Paul DuBois wrote:
> My experience with binary distributions has convinced me that it is a
> hopeless endeavor in the long run. I understand why people want them
> but until you've tried to provide them to the public and seen the
> number of things that go wrong ... Not to mention that it means you
> have to have someone build on all those systems and you don't have
> them all. And if the person is going to add their other extensions,
> compiled with a different compiler, blooie.

This is Much less of an issue with OS-X than many other systems. Apples 
keeps it pretty tightly controlled, If you build packages without 
linking in a bunch of fink/darwinports/portage stuff!

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From strang at nmr.mgh.harvard.edu  Mon Feb  7 13:32:34 2005
From: strang at nmr.mgh.harvard.edu (Gary Strangman)
Date: Mon Feb  7 13:32:34 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4207CC4C.8060603@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
 <4207CC4C.8060603@ee.byu.edu>
Message-ID: <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>

>> To be in keeping with other python objects, it seems this object would 
>> need to be tight, fast and easily extensible. I *think* this is exactly 
>> what Numeric3 is intended to do. Getting this right is tricky, but it 
>> seems like current solutions are EXTREMELY close.
>
> I think it's close too, that's why I'm willing to do this, right now.

:-)

>> already exists. Granted, the notion of what's "basic" vs. advanced is 
>> relative (e.g., where do you put fft, or linear_algebra?). But if made 
>> modular and encapsulated (e.g., an fft.py, linear_algebra.py, 
>> integration.py, morphology.py) and made available both individually and as 
>> part of one or more suites--see #4 below, it's easier to build on existing 
>> code rather than reinvent.
>
> There seems to be several users on this list that agree.  I would agree too, 
> but would also state that Python also has the idea of "batteries included" 
> meaning that the html, cgi, and xml libraries are all installed using a 
> common installer.   SciPy has tried to be modular, too but have a common 
> installer.  SciPy is an umbrella package with several sub-packages. 
> Sometimes it is hard to be modular when a sub-package depends on linear 
> algebra routines for example.

The "batteries included" is a good point. But I guess I'm not alone in 
the modularity argument.

> I think that a basic install should always include linear algebra, random 
> numbers, and fft code.

+1

> I don't think this should live under a Numeric 
> package because if we are to get arrayobjects into the core, then we need to 
> develop some other place for these things to live.

+1

> That is essentially what 
> scipy is trying to be -- a place for these extras and additional numeric 
> routines to live.

And NOT a matlab-like environment? Or is it trying to be both?

>   Perhaps we underestimated the problems with using FORTRAN 
> (but I still don't think so given that g77 is available wherever gcc is). 
> With f2py, FORTRAN is incredibly quick to develop Python modules with.  So, I 
> would have a hard time saying that we should not use FORTRAN.  I can conceed 
> that we should separate modules that need FORTRAN from those that need only 
> C(++).

Being able to say, "to get these additional functions, you need to get 
FORTRAN" would be fantastic. It would help folks (including myself, who 
dreads complicated installs) if wrestling with the install is going to be 
worth the effort.

Gary



From konrad.hinsen at laposte.net  Mon Feb  7 13:50:26 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 13:50:26 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4207C83D.5040506@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <55c095f595dfec3e16579094faea14f2@laposte.net> <4203FCD8.8070204@noaa.gov> <3a3eead6a9c806e61c480f761dc6b6cd@laposte.net> <4207C83D.5040506@noaa.gov>
Message-ID: <a91c9df3205ba200fdaab4ba667270a0@laposte.net>

On 07.02.2005, at 20:57, Chris Barker wrote:

> I may be wrong about that. It certainly has BLAS, but I didn't patch  
> Numeric's setup.py, so I'm not sure how that worked.

I checked on Apple's developer site: there is an optimized BLAS in  
vecLib, plus the standard CLAPACK.

> Neither do I, so maybe that's why. Frankly, though, fink is command  
> line also, and a really ugly one, in my experience!

There is Fink Commander, which is only 95% as ugly as the Fink command  
line ;-)

> I number of folks seem to think darwinports is a better alternative. I  
> think, if  nothing else, they make greater effort to use  
> Apple-supplied libraries, when they exist. I still want to try Gentoo.  
> I love it for Linux (once the system is installed, anyway!)

I am seriously out of time for playing with installations at the  
moment, but I am always interested to hear about alternatives to Fink.

I like the gentoo approach as well in principle, but I also know that  
compiling tons of stuff on an iBook is a real challenge.

> And get a nifty Apple-style point and click package. This is thanks to  
> Bob Ippolito's Py2App.

I'll have to look at that one!

> open source libs. Also, Apple has not provided a standard package  
> manager. I can build an rpm (or deb, or ebuild) for a given version of  
> Linux,

Yes, that's a big issue. Apple has done a nice job with its  
application-directory system for handling stand-along applications, but  
there is little support for the OpenSource world's building block  
approach. I guess this is why Fink opted for a complete subsystem  
around Debian packages.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From stephen.walton at csun.edu  Mon Feb  7 13:59:33 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Mon Feb  7 13:59:33 2005
Subject: [Numpy-discussion] logical priority
Message-ID: <4207E421.10702@csun.edu>

A probably bonehead question:  is it a known feature of Python that, in 
a logical operation, & has a higher priority than > or < ?  Or is it a 
quirk of numarray?

In [126]: a = arange(25)

In [127]: v=(a>9) & (a<13)

In [128]: v=a>9 & a<13
---------------------------------------------------------------------------
exceptions.RuntimeError                              Traceback (most 
recent call last)

/home/swalton/research/foukal/<console>

/usr/lib/python2.3/site-packages/numarray/generic.py in __nonzero__(self)
    475
    476     def __nonzero__(self):
--> 477         raise RuntimeError("An array doesn't make sense as a 
truth value.  Use any(a) or all(a).")
    478
    479     def __copy__(self):

RuntimeError: An array doesn't make sense as a truth value.  Use any(a) 
or all(a).




From ariciputi at pito.com  Mon Feb  7 14:03:18 2005
From: ariciputi at pito.com (Andrea Riciputi)
Date: Mon Feb  7 14:03:18 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
Message-ID: <2C2E3F9E-7954-11D9-9E63-000A95C0BC68@pito.com>

Hi all,
I'm yet another long-time lurker, and Python/Numeric/SciPy user, and 
I'm very interested in this discussion. I'm completely agree with Gary, 
and I'll look forward to see the results of Travis's effort. I wish him 
best luck for Numeric3, and to all scientific-related Python community 
to keep on developing a striking scientific-environment. As Gary said, 
we are so close. At the moment, I'm stuked with my Ph.D. thesis, I hope 
in the future to be able to give a little contribution.

I also want to thanks all the Python scientific developers for their 
efforts.

Cheers,
  Andrea.

P.S.: I'm a Mac OSX users, and I've all the Python related packages 
(including SciPy, Numeric, matplotlib...) installed via Fink. I've 
never had any problem, could you explain to me what kind of problems 
are you experiencing?


On 7 Feb 2005, at 17:41, Gary Strangman wrote:

> Hi all,
>
> [Long post: a long-time lurker here sharing his numerical 
> experiences/input re: "the numeric split" and Numeric3.]
>
> I'm a long-time python user (and follower of this list), a heavy user 
> in the numerical realm, and an "incidental" contributor to SciPy 
> (statistical functions in stats.py). First, I'm impressed (and indeed 
> amazed) at all the hard work folks have put in--usually without 
> pay--to make Numpy and Numarray and SciPy and Matplotlib (to name the 
> major players in this discussion). I am indebted to each of you. 
> Second, I must say that I think this protracted discussion is EXACTLY 
> what the python numerical community needs. It appears as though we 
> have critical mass in terms of code and interest (and opinions), and 
> just need to bring them all together.




From stephen.walton at csun.edu  Mon Feb  7 14:05:21 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Mon Feb  7 14:05:21 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
Message-ID: <4207E5DF.8020506@csun.edu>

Gary Strangman wrote:

> Being able to say, "to get these additional functions, you need to get 
> FORTRAN" would be fantastic. It would help folks (including myself, 
> who dreads complicated installs) if wrestling with the install is 
> going to be worth the effort. 

I would rather say, "There is a huge amount of free code for doing 
complicated numerical work already developed in Fortran.  F2PY allows 
you to use this existing code base from Python, saving you having to 
rewrite in Python, C++, or what-have-you.  Since g77 is free and easy to 
install, there's no reason to re-invent."

BTW, g95's availability was more-or-less officially announced on Cleve 
Moler's listserv over the weekend.





From konrad.hinsen at laposte.net  Mon Feb  7 14:08:27 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Mon Feb  7 14:08:27 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <4207E421.10702@csun.edu>
References: <4207E421.10702@csun.edu>
Message-ID: <0a221fd095d46402910135731d02ad05@laposte.net>

On 07.02.2005, at 22:56, Stephen Walton wrote:

> A probably bonehead question:  is it a known feature of Python that,  
> in a logical operation, & has a higher priority than > or < ?  Or is  
> it a quirk of numarray?

Operator priorities are part of the Python language and implemented in  
the parser. Numarray cannot interfere with that.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From stephen.walton at csun.edu  Mon Feb  7 14:30:25 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Mon Feb  7 14:30:25 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <0a221fd095d46402910135731d02ad05@laposte.net>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net>
Message-ID: <4207EBBF.2060808@csun.edu>

konrad.hinsen at laposte.net wrote:

> Operator priorities are part of the Python language and implemented 
> in  the parser. Numarray cannot interfere with that.

In Python, of course, & is a bitwise AND, logical and being the literal 
word "and".  & has lower operator priority than <, but "and" has a 
higher one.

I think the real problem is that "and" doesn't work on numarray bool 
arrays in an element-wise fashion as I expected;  perhaps it should?  
Otherwise constructs like

v = A>10 and A<15

have to be written as

v1=A>10; v2=A<15; v=v1&v2

or as
v = (A>10) & (A<15)




From Chris.Barker at noaa.gov  Mon Feb  7 15:44:10 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Mon Feb  7 15:44:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <2C2E3F9E-7954-11D9-9E63-000A95C0BC68@pito.com>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <2C2E3F9E-7954-11D9-9E63-000A95C0BC68@pito.com>
Message-ID: <4207FD0F.1010908@noaa.gov>

Andrea Riciputi wrote:

> P.S.: I'm a Mac OSX users, and I've all the Python related packages 
> (including SciPy, Numeric, matplotlib...) installed via Fink. I've never 
> had any problem, could you explain to me what kind of problems are you 
> experiencing?

Do you use it with any of the Aqua-based GUIs ? Probably not. My 
experience with fink is that it is a subsystem all it's own. If you do 
everything with fink, it'll work fine. Personally, I'm a Linux geek, so 
I'd probably be happy with an all-fink system, but the only reason I use 
OS-X at all is that I work with other OS-X users, none of whom are Unix 
folks (they mostly come from the old Mac-OS world). I need to be able to 
develop apps that they can just fire up and have working on their stock 
OS-X boxes, and that look like Mac apps.

I've decided that fink-free is the best way to achieve this at the moment.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From oliphant at ee.byu.edu  Mon Feb  7 21:13:10 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Mon Feb  7 21:13:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
Message-ID: <42084A1F.3030007@ee.byu.edu>

> And NOT a matlab-like environment? Or is it trying to be both?

Very loosely a matlab or IDL-like environment.  Nobody has really 
defined what it means to be a MATLAB-like environment.   The idea is 
that someone could do what one can do in MATLAB or IDL, not write code 
in exactly the same way.  A MATLAB translator is an entirely different 
project.

So, I'm not sure.   There is no requirement that contributions "not be 
classed based", or conform to some command in MATLAB. 

Perhaps what you are referring to is the fact that

from scipy import *

makes a lot of command-like interaction possible.  Is this seen as 
MATLAB-like?

 I guess right now, scipy is trying to bring lots of modules under a 
common naming convention.

I think the goals of scipy are very much what people are talking about 
here.   SciPy ultimately becomes what its users and developers want it 
to become.   That is why I'm so unclear as to why we have not received 
more support in its development --- even in the way of ideas as to how 
to make it better --- until the great ideas that have been suggested on 
this list recently.

-Travis




From oliphant at ee.byu.edu  Mon Feb  7 21:19:25 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Mon Feb  7 21:19:25 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <4207EBBF.2060808@csun.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu>
Message-ID: <42084BB7.2060509@ee.byu.edu>

Stephen Walton wrote:

> konrad.hinsen at laposte.net wrote:
>
>> Operator priorities are part of the Python language and implemented 
>> in  the parser. Numarray cannot interfere with that.
>
>
> In Python, of course, & is a bitwise AND, logical and being the 
> literal word "and".  & has lower operator priority than <, but "and" 
> has a higher one.
>
> I think the real problem is that "and" doesn't work on numarray bool 
> arrays in an element-wise fashion as I expected;  perhaps it should?  
> Otherwise constructs like

Currently, I don't think Python allows "over-riding" the "and" 
operation.  It only works on the truth or falseness of the objects.  
Therefore, I don't think it is possible to make it work as you'd expect 
unless Python is changed.





From Fernando.Perez at colorado.edu  Mon Feb  7 21:26:10 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon Feb  7 21:26:10 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42084BB7.2060509@ee.byu.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu>
Message-ID: <42084D42.9050509@colorado.edu>

Travis Oliphant wrote:

> Currently, I don't think Python allows "over-riding" the "and" 
> operation.  It only works on the truth or falseness of the objects.  
> Therefore, I don't think it is possible to make it work as you'd expect 
> unless Python is changed.

You are correct:

In [12]: dis.dis(compile('x and y','<console>','eval'))
   0           0 LOAD_NAME                0 (x)
               3 JUMP_IF_FALSE            4 (to 10)
               6 POP_TOP
               7 LOAD_NAME                1 (y)
         >>   10 RETURN_VALUE


There is no method you can hook in there.  Compare this to '&':

In [13]: dis.dis(compile('x & y','<console>','eval'))
   0           0 LOAD_NAME                0 (x)
               3 LOAD_NAME                1 (y)
               6 BINARY_AND
               7 RETURN_VALUE

Cheers,

f



From NadavH at VisionSense.com  Tue Feb  8 01:08:10 2005
From: NadavH at VisionSense.com (Nadav Horesh)
Date: Tue Feb  8 01:08:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42084A1F.3030007@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu>
Message-ID: <42087FAE.5000303@VisionSense.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050208/959a2d94/attachment-0001.html>

From oliphant at ee.byu.edu  Tue Feb  8 01:23:18 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Tue Feb  8 01:23:18 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42087FAE.5000303@VisionSense.com>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42087FAE.5000303@VisionSense.com>
Message-ID: <420884D2.5020103@ee.byu.edu>

Thanks for your comments.

Nadav Horesh wrote:

> Most of my work is carried using the numarray/Numeric package, and use 
> scipy not so often for three reasons:
>
>    1. nummaray is well equip.
>
It is certainly missing many things.

>    1. scipy is mostly undocumented.
>
SciPy is decently self documented.  And, with  
http://www.scipy.org/livedocs/ 
self-documentation leads to rapid online documentation.

>    1. scipy had bugs in many  of it high-end (=rarely used) routines.
>
Interesting claim.   Yes bugs crop up, but they are typically fixed 
quickly when found.   Basing much of the code on standard compiled 
routines helps to alleviate this problem.

> This is a high barrier for a causal scipy users like me, and thus it 
> hard for me to contribute back. For instance, I wrote a bicoherence 
> routine some time ago, but found no proper place to contribute it. 
> Scipy would be a natural place if it would be more accessible.
> One of the good reasons to provide bare Numeric3 and to put fft, 
> linear algebra, etc. in scipy, is that it would force scipy into the 
> painful process of being user friendly (=accessible)


One very important reason I would like to do it.  It is hard to know if 
scipy is accessible or not if people who have problems don't complain 
but instead walk away or just start their own package.

-Travis





From konrad.hinsen at laposte.net  Tue Feb  8 03:37:28 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Tue Feb  8 03:37:28 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <0b7590e6e7b4b325bf3acdaf46aecce7@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <4ad81937f722051f27b37915225a1563@laposte.net> <1107794238.10581.16.camel@alpspitze.cs.pdx.edu> <0b7590e6e7b4b325bf3acdaf46aecce7@laposte.net>
Message-ID: <e45d15d3bb48ba6869b77ceb91d2e32e@laposte.net>

On 07.02.2005, at 20:40, konrad.hinsen at laposte.net wrote:

> There is also a glaring hole in the OO approach in that there is no  
> way to add functionality (methods) to an existing object. The only  
> solution is subclassing, but then all object creators need to be  
> changed to those for the subclass, which is not practical. Unlike  
> other OO languages, Python makes it possible to add methods to  
> existing classes, but this always looks like a bad hack.

There is something I forgot to mention in this context:

	http://www.python.org/peps/pep-0246.html

If adopted, this would be a big step forward in my opinion.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Tue Feb  8 03:45:16 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Tue Feb  8 03:45:16 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
Message-ID: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>

Is anyone here using NumPy on Opteron machines running Linux in 64 bit  
mode? I am running into problems such as:

Python 2.4 (#4, Jan 18 2005, 18:06:45)
[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> from Numeric import *
 >>> a = arange(4)
 >>> a
array([0, 1, 2, 3])
 >>> a.shape = (2, 2)
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
ValueError: total size of new array must be unchanged

This is with Python 2.4 and Numeric 23.7. Before looking into this  
myself, I thought I'd ask here, perhaps someone has already found a  
fix.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From Alexandre.Fayolle at logilab.fr  Tue Feb  8 04:32:37 2005
From: Alexandre.Fayolle at logilab.fr (Alexandre)
Date: Tue Feb  8 04:32:37 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>
References: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>
Message-ID: <20050208123054.GC14155@crater.logilab.fr>

On Tue, Feb 08, 2005 at 12:44:18PM +0100, konrad.hinsen at laposte.net wrote:
> Is anyone here using NumPy on Opteron machines running Linux in 64 bit  
> mode? I am running into problems such as:
> 
> Python 2.4 (#4, Jan 18 2005, 18:06:45)
> [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from Numeric import *
> >>> a = arange(4)
> >>> a
> array([0, 1, 2, 3])
> >>> a.shape = (2, 2)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> ValueError: total size of new array must be unchanged
> 
> This is with Python 2.4 and Numeric 23.7. Before looking into this  
> myself, I thought I'd ask here, perhaps someone has already found a  
> fix.

I don't have a fix, but, for the record, I can reproduce this on my
alphastation. 

a.typecode() says that the contents of a are Int64.  

-- 
Alexandre Fayolle                              LOGILAB, Paris (France).
http://www.logilab.com   http://www.logilab.fr  http://www.logilab.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050208/5699345f/attachment-0001.sig>

From cjw at sympatico.ca  Tue Feb  8 04:58:37 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Tue Feb  8 04:58:37 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42084BB7.2060509@ee.byu.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu>
Message-ID: <4208B718.4050502@sympatico.ca>

Travis Oliphant wrote:

> Stephen Walton wrote:
>
>> konrad.hinsen at laposte.net wrote:
>>
>>> Operator priorities are part of the Python language and implemented 
>>> in  the parser. Numarray cannot interfere with that.
>>
>>
>>
>> In Python, of course, & is a bitwise AND, logical and being the 
>> literal word "and".  & has lower operator priority than <, but "and" 
>> has a higher one.
>>
>> I think the real problem is that "and" doesn't work on numarray bool 
>> arrays in an element-wise fashion as I expected;  perhaps it should?  
>> Otherwise constructs like
>
>
> Currently, I don't think Python allows "over-riding" the "and" 
> operation.  It only works on the truth or falseness of the objects.  
> Therefore, I don't think it is possible to make it work as you'd 
> expect unless Python is changed.
>
>
There is __nonzero__ which could be treated as allTrue or someTrue, the 
former is probably the better.

Colin W.

>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>




From hoel at gl-group.com  Tue Feb  8 05:16:13 2005
From: hoel at gl-group.com (=?ISO-8859-15?Q?Berthold_H=F6llmann?=)
Date: Tue Feb  8 05:16:13 2005
Subject: [Numpy-discussion] Re: NumPy on 64-bit Linux (Opteron)
In-Reply-To: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net> (konrad hinsen's message of "Tue, 8 Feb 2005 12:44:18 +0100")
References: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>
Message-ID: <se4qgnb3da.fsf@pc047299.dhcp.germanlloyd.org>

konrad.hinsen at laposte.net writes:

> Is anyone here using NumPy on Opteron machines running Linux in 64 bit
> mode? I am running into problems such as:
>
> Python 2.4 (#4, Jan 18 2005, 18:06:45)
> [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> from Numeric import *
>  >>> a = arange(4)
>  >>> a
> array([0, 1, 2, 3])
>  >>> a.shape = (2, 2)
> Traceback (most recent call last):
>    File "<stdin>", line 1, in ?
> ValueError: total size of new array must be unchanged
>
> This is with Python 2.4 and Numeric 23.7. Before looking into this
> myself, I thought I'd ask here, perhaps someone has already found a
> fix.

I do have currently access to an Opteron machine. OS is SuSE SLES9,
with a lot of Python packages from SuSE Professional 9.1. I get:

~> python
Python 2.3.3 (#1, Nov 29 2004, 22:35:24) 
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Numeric as N
>>> a = N.arange(4)
>>> a
array([0, 1, 2, 3])
>>> a.shape = (2,2)
>>> N.__file__
'/usr/lib64/python2.3/site-packages/Numeric/Numeric.pyc'
>>> N.__version__
'23.1'
>>> 

Kind regards

Berthold
-- 
Germanischer Lloyd AG
CAE Development
Vorsetzen 35
20459 Hamburg
Phone: +49(0)40 36149-7374
Fax: +49(0)40 36149-7320
e-mail: hoel at gl-group.com
Internet: http://www.gl-group.com



This e-mail contains confidential information for the exclusive attention of the intended addressee. Any access of third parties to this e-mail is unauthorised. Any use of this e-mail by unintended recipients such as copying, distribution, disclosure etc. is prohibited and may be unlawful. When addressed to our clients the content of this e-mail is subject to the General Terms and Conditions of GL's Group of Companies applicable at the date of this e-mail. 

GL's Group of Companies does not warrant and/or guarantee that this message at the moment of receipt is authentic, correct and its communication free of errors, interruption etc.



From strang at nmr.mgh.harvard.edu  Tue Feb  8 06:57:32 2005
From: strang at nmr.mgh.harvard.edu (Gary Strangman)
Date: Tue Feb  8 06:57:32 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42084A1F.3030007@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
 <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
 <42084A1F.3030007@ee.byu.edu>
Message-ID: <Pine.LNX.4.60.0502080928360.16300@gate.nmr.mgh.harvard.edu>

On Mon, 7 Feb 2005, Travis Oliphant wrote:

>> And NOT a matlab-like environment? Or is it trying to be both?
>
> Very loosely a matlab or IDL-like environment.  Nobody has really defined 
> what it means to be a MATLAB-like environment.   The idea is that someone 
> could do what one can do in MATLAB or IDL, not write code in exactly the same 
> way.  A MATLAB translator is an entirely different project.
>
> So, I'm not sure.   There is no requirement that contributions "not be 
> classed based", or conform to some command in MATLAB. 
> Perhaps what you are referring to is the fact that
>
> from scipy import *
>
> makes a lot of command-like interaction possible.  Is this seen as 
> MATLAB-like?

I was focusing on the word "environment". The current Matlab "environment" 
is heavily graphical and includes separate panels for command history, 
defined variables (along with their memory requirements), a command 
window, etc. Of course it also has a flat namespace. The python 
interpreter is already its own (restricted) environment, and Idle is 
another. Building a Matlab "environment", in this sense of the word, is a 
major undertaking and should be a separate goal (and, IMHO, probably not 
all that useful). Doesn't sound like that's what you meant. My 
misinterpretation.

> I guess right now, scipy is trying to bring lots of modules under a common 
> naming convention.

And this is great. I'd definitely use such a creature. However, it sounds 
like several folks would also appreciate (myself included) access to 
individual modules (or smaller subsets of interdependent modules) because 
of installation troubles. I'm probably not the one to ask about how to 
divvy those up, but perhaps it would be useful to have a "requires 
fortran" group. And a python-only group. And a python-plus-C group. That 
way, users (or "mini-developers") could pick their installation battles 
(on windoze, having ANY compiler can be a significant $$ or installation 
problem). This would make it more transparent as to which routines are 
well-tested and fast (most Fortran-based code), fast (C and Fortran), or 
easy-to-modify (Python pieces). The transparency could also guide folks 
who can/do write C/Fortran code to translate their favorite python-only 
pieces to a faster format. Just a thought. ('course, those dividing lines 
may not make for nice module names ... sigh.)

> I think the goals of scipy are very much what people are talking about here. 
> SciPy ultimately becomes what its users and developers want it to become. 
> That is why I'm so unclear as to why we have not received more support in its 
> development --- even in the way of ideas as to how to make it better --- 
> until the great ideas that have been suggested on this list recently.

Perhaps because SciPy has seemed monolithic, rather than a collection of 
largely independent modules. That's how I've perceived it, and I even have 
a module inside SciPy. ;-) I think that making pieces available (groupings 
of interdependent modules) in addition to everything-under-the-sun would
help with that perception.

-best
Gary



From southey at uiuc.edu  Tue Feb  8 07:07:11 2005
From: southey at uiuc.edu (Bruce Southey)
Date: Tue Feb  8 07:07:11 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux
 (Opteron)
Message-ID: <4542a74a.cb733b75.81ad300@expms6.cites.uiuc.edu>

Hi, 
Yes (Python 2.3.3 [GCC 3.3.3 (SuSE Linux)]) with Numeric 23.7 (clean install). 
 
It is to do with the function: 
extern PyObject * PyArray_Reshape(PyArrayObject *self, PyObject *shape) 
 
the variable s_known is not being assigned correctly (gets set to zero). I 
think that stems from this call that doesn't set the dimensions correctly: 
 
PyArray_As1D(&shape, (char **)&dimensions, &n, PyArray_LONG) 
 
dimensions in the second loop has value 0. 
 
Regards 
Bruce 
 
---- Original message ---- 
>Date: Tue, 8 Feb 2005 12:44:18 +0100 
>From: konrad.hinsen at laposte.net   
>Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)   
>To: numpy-discussion <numpy-discussion at lists.sourceforge.net> 
> 
>Is anyone here using NumPy on Opteron machines running Linux in 64 bit   
>mode? I am running into problems such as: 
> 
>Python 2.4 (#4, Jan 18 2005, 18:06:45) 
>[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2 
>Type "help", "copyright", "credits" or "license" for more information. 
> >>> from Numeric import * 
> >>> a = arange(4) 
> >>> a 
>array([0, 1, 2, 3]) 
> >>> a.shape = (2, 2) 
>Traceback (most recent call last): 
>   File "<stdin>", line 1, in ? 
>ValueError: total size of new array must be unchanged 
> 
>This is with Python 2.4 and Numeric 23.7. Before looking into this   
>myself, I thought I'd ask here, perhaps someone has already found a   
>fix. 
> 
>Konrad. 
>-- 
>------------------------------------------------------------------------  
>------- 
>Konrad Hinsen 
>Laboratoire Leon Brillouin, CEA Saclay, 
>91191 Gif-sur-Yvette Cedex, France 
>Tel.: +33-1 69 08 79 25 
>Fax: +33-1 69 08 82 61 
>E-Mail: khinsen at cea.fr 
>------------------------------------------------------------------------  
>------- 
> 
> 
> 
>------------------------------------------------------- 
>SF email is sponsored by - The IT Product Guide 
>Read honest & candid reviews on hundreds of IT Products from real users. 
>Discover which products truly live up to the hype. Start reading now. 
>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click 
>_______________________________________________ 
>Numpy-discussion mailing list 
>Numpy-discussion at lists.sourceforge.net 
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion 
 



From southey at uiuc.edu  Tue Feb  8 08:05:29 2005
From: southey at uiuc.edu (Bruce Southey)
Date: Tue Feb  8 08:05:29 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux
 (Opteron)
Message-ID: <143ec816.cb7887f4.8344000@expms6.cites.uiuc.edu>

Hi, 
I should also add that on a P3 (same info as below), there is no problem as the 
correct dimensions are returned. 
 
Regards 
Bruce 
 
---- Original message ---- 
>Date: Tue, 8 Feb 2005 09:06:04 -0600 
>From: Bruce Southey <southey at uiuc.edu>   
>Subject: Re: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)   
>To: konrad.hinsen at laposte.net, numpy-discussion 
<numpy-discussion at lists.sourceforge.net> 
> 
>Hi,  
>Yes (Python 2.3.3 [GCC 3.3.3 (SuSE Linux)]) with Numeric 23.7 (clean install).  
>  
>It is to do with the function:  
>extern PyObject * PyArray_Reshape(PyArrayObject *self, PyObject *shape)  
>  
>the variable s_known is not being assigned correctly (gets set to zero). I  
>think that stems from this call that doesn't set the dimensions correctly:  
>  
>PyArray_As1D(&shape, (char **)&dimensions, &n, PyArray_LONG)  
>  
>dimensions in the second loop has value 0.  
>  
>Regards  
>Bruce  
>  
>---- Original message ----  
>>Date: Tue, 8 Feb 2005 12:44:18 +0100  
>>From: konrad.hinsen at laposte.net    
>>Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)    
>>To: numpy-discussion <numpy-discussion at lists.sourceforge.net>  
>>  
>>Is anyone here using NumPy on Opteron machines running Linux in 64 bit    
>>mode? I am running into problems such as:  
>>  
>>Python 2.4 (#4, Jan 18 2005, 18:06:45)  
>>[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2  
>>Type "help", "copyright", "credits" or "license" for more information.  
>> >>> from Numeric import *  
>> >>> a = arange(4)  
>> >>> a  
>>array([0, 1, 2, 3])  
>> >>> a.shape = (2, 2)  
>>Traceback (most recent call last):  
>>   File "<stdin>", line 1, in ?  
>>ValueError: total size of new array must be unchanged  
>>  
>>This is with Python 2.4 and Numeric 23.7. Before looking into this    
>>myself, I thought I'd ask here, perhaps someone has already found a    
>>fix.  
>>  
>>Konrad.  
>>--  
>>------------------------------------------------------------------------   
>>-------  
>>Konrad Hinsen  
>>Laboratoire Leon Brillouin, CEA Saclay,  
>>91191 Gif-sur-Yvette Cedex, France  
>>Tel.: +33-1 69 08 79 25  
>>Fax: +33-1 69 08 82 61  
>>E-Mail: khinsen at cea.fr  
>>------------------------------------------------------------------------   
>>-------  
>>  
>>  
>>  
>>-------------------------------------------------------  
>>SF email is sponsored by - The IT Product Guide  
>>Read honest & candid reviews on hundreds of IT Products from real users.  
>>Discover which products truly live up to the hype. Start reading now.  
>>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click  
>>_______________________________________________  
>>Numpy-discussion mailing list  
>>Numpy-discussion at lists.sourceforge.net  
>>https://lists.sourceforge.net/lists/listinfo/numpy-discussion  
>  
> 
> 
>------------------------------------------------------- 
>SF email is sponsored by - The IT Product Guide 
>Read honest & candid reviews on hundreds of IT Products from real users. 
>Discover which products truly live up to the hype. Start reading now. 
>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click 
>_______________________________________________ 
>Numpy-discussion mailing list 
>Numpy-discussion at lists.sourceforge.net 
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion 
 



From cookedm at physics.mcmaster.ca  Tue Feb  8 08:46:11 2005
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Tue Feb  8 08:46:11 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>
References: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net>
Message-ID: <20050208164424.GA1341@arbutus.physics.mcmaster.ca>

On Tue, Feb 08, 2005 at 12:44:18PM +0100, konrad.hinsen at laposte.net wrote:
> Is anyone here using NumPy on Opteron machines running Linux in 64 bit  
> mode? I am running into problems such as:
> 
> Python 2.4 (#4, Jan 18 2005, 18:06:45)
> [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from Numeric import *
> >>> a = arange(4)
> >>> a
> array([0, 1, 2, 3])
> >>> a.shape = (2, 2)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> ValueError: total size of new array must be unchanged
> 
> This is with Python 2.4 and Numeric 23.7. Before looking into this  
> myself, I thought I'd ask here, perhaps someone has already found a  
> fix.

Yes; I submitted a patch (#1108739) to the Numeric patch tracker last week.

Basically, PyArray_Reshape takes an array of dimensions. That has to be
a PyArray_INT (C ints) instead of the PyArray_LONG (C longs) that it has
in the code, because that array is passed to PyArray_FromDimsAndDataAndDescr.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From haase at msg.ucsf.edu  Tue Feb  8 09:36:29 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue Feb  8 09:36:29 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <20050208164424.GA1341@arbutus.physics.mcmaster.ca>
References: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net> <20050208164424.GA1341@arbutus.physics.mcmaster.ca>
Message-ID: <200502080935.47908.haase@msg.ucsf.edu>

(RE: numarray)
Hi, 
On an Opteron (Suse9) I was happy to see that I could allocate the memory for 
an Int32-1024x1024x1024 cube. (On a P4 a get 'MemoryError').
I remember that Todd always said that python itself wasn't 64bit-ready yet.
But why is it needed to support >4GB python-lists if one is only interested in 
large numarrays (or numeric) ?

Just for sake of completeness here is one traceback I got:
>>> import numarray
>>> a = numarray.array(shape=(1024,1024,1024), type=numarray.Int)
>>> a.type()
Int32
>>> a.shape
(1024, 1024, 1024)
>>> len(a)
1024
>>> a
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
numarraycore.py", line 896, in __repr__
    return array_repr(self)
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
numarraycore.py", line 1542, in array_repr
    lst = arrayprint.array2string(
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
arrayprint.py", line 188, in array2string
    separator, prefix)
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
arrayprint.py", line 137, in _array2string
    data = _leading_trailing(a)
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
arrayprint.py", line 111, in _leading_trailing
    l = [_leading_trailing(a[i]) for i in range(
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
arrayprint.py", line 111, in _leading_trailing
    l = [_leading_trailing(a[i]) for i in range(
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
arrayprint.py", line 105, in _leading_trailing
    b = _gen.concatenate((a[:_summaryEdgeItems],
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
generic.py", line 1061, in concatenate
    return _concat(arrs)
  File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/
generic.py", line 1051, in _concat
    dest[ix:ix+_shape0(a)]._copyFrom(a)
numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 
buffersize=0

Thanks 
Sebastian Haase




On Tuesday 08 February 2005 08:44 am, David M. Cooke wrote:
> On Tue, Feb 08, 2005 at 12:44:18PM +0100, konrad.hinsen at laposte.net wrote:
> > Is anyone here using NumPy on Opteron machines running Linux in 64 bit
> > mode? I am running into problems such as:
> >
> > Python 2.4 (#4, Jan 18 2005, 18:06:45)
> > [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> >
> > >>> from Numeric import *
> > >>> a = arange(4)
> > >>> a
> >
> > array([0, 1, 2, 3])
> >
> > >>> a.shape = (2, 2)
> >
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in ?
> > ValueError: total size of new array must be unchanged
> >
> > This is with Python 2.4 and Numeric 23.7. Before looking into this
> > myself, I thought I'd ask here, perhaps someone has already found a
> > fix.
>
> Yes; I submitted a patch (#1108739) to the Numeric patch tracker last week.
>
> Basically, PyArray_Reshape takes an array of dimensions. That has to be
> a PyArray_INT (C ints) instead of the PyArray_LONG (C longs) that it has
> in the code, because that array is passed to
> PyArray_FromDimsAndDataAndDescr.



From konrad.hinsen at laposte.net  Tue Feb  8 09:37:32 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Tue Feb  8 09:37:32 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <20050208164424.GA1341@arbutus.physics.mcmaster.ca>
References: <1b6a69c8ce4621c4cfd22edfd7685a75@laposte.net> <20050208164424.GA1341@arbutus.physics.mcmaster.ca>
Message-ID: <ba6f727b306b3556c9bf6044083602d4@laposte.net>

On 08.02.2005, at 17:44, David M. Cooke wrote:

> Yes; I submitted a patch (#1108739) to the Numeric patch tracker last  
> week.

Thanks, that patch works fine for me as well.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From haase at msg.ucsf.edu  Tue Feb  8 09:39:20 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue Feb  8 09:39:20 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502080928360.16300@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <42084A1F.3030007@ee.byu.edu> <Pine.LNX.4.60.0502080928360.16300@gate.nmr.mgh.harvard.edu>
Message-ID: <200502080937.59199.haase@msg.ucsf.edu>

Hi,
(just to say something)
I hope "Numeric3" will (magicly) inspire someone to make small-numarray 
creation fast. I really think record-arrays and memmap-arrays of numarray 
should be mentioned more often, just to make Numeric-people realize what they 
miss ;-)

Thanks for "Numerical Python"
Sebastian Haase



From Chris.Barker at noaa.gov  Tue Feb  8 10:28:47 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Tue Feb  8 10:28:47 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42084A1F.3030007@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu>
Message-ID: <42090476.7080604@noaa.gov>

Travis Oliphant wrote:

> Very loosely a matlab or IDL-like environment.  Nobody has really 
> defined what it means to be a MATLAB-like environment. 

Well, I HOPE it means:

An environment in which you can do most of what you can do with MATLAB, 
just as easily, but with a more pythonic approach. I wanted a MATLAB 
like language, I'd use MATLAB (or maybe Octave or Psilab).

> from scipy import *

And I'd like to see this deprecated.

> makes a lot of command-like interaction possible.

I think command-line interaction is quite possible without import *. I'm 
actually kind of disappointed in Numeric, from this perspective. I 
always used to use "from Numeric import *", because Numeric was designed 
to be used this way, which I think is a shame. Far more should be 
methods, rather than functions. For example, there are a bunch of 
functions that take an array as the first argument:

Numeric.take(a, i)

Why in the world is this not:

a.take(i)

If it (and many other functions) were methods, there would be far less 
need for "import *" (I do use import "Numeric as N" to save typing)

In the docs, for many years, there has been the explanation that the 
above approach allows people to use any python sequence, rather than 
just arrays, with these functions. I find that that buys me very little. 
The example given in the docs (a few years ago, is it still there?) is 
transpose:

at = Numeric.transpose(a)

can be passed a Python sequence that has an appropriate structure. 
However, internally, what's going on is something like:

b = Numeric.asarray(a)
at = Numeric.transpose(b)

Is it really so hard to write that extra line? Remember that transpose() 
always returns a NumPy array, regardless of the type of the input. How 
often are you using arbitrary python sequences anyway? In my case, I am 
never using arbitrary python sequences unless I'm writing a function or 
method for an external API, in which case, I almost always put an 
Numeric.asarray() call in at the top.

Well, this turned out to be too much of a rant, but the basic gist was: 
let's make SciPy Pythonic, rather than Matlaby. And the same goes for 
Numeric3.

-Chris















   Is this seen as
> MATLAB-like?
> 
> I guess right now, scipy is trying to bring lots of modules under a 
> common naming convention.
> 
> I think the goals of scipy are very much what people are talking about 
> here.   SciPy ultimately becomes what its users and developers want it 
> to become.   That is why I'm so unclear as to why we have not received 
> more support in its development --- even in the way of ideas as to how 
> to make it better --- until the great ideas that have been suggested on 
> this list recently.
> 
> -Travis
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From haase at msg.ucsf.edu  Tue Feb  8 10:53:38 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue Feb  8 10:53:38 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis" returns 'unexpected' type
Message-ID: <200502081052.40692.haase@msg.ucsf.edu>

Hi,
from the numarray documentation:
<doc>
concatenate(arrs, axis=0)
Returns a new array containing copies of the data contained in all arrays # of 
arrs= (a0, a1, ... an). The arrays ai will be concatenated along the 
specified axis (default=0). All arrays ai must have the same shape along 
every axis except for the one specified in axis. To concatenate arrays along 
a newly created axis, you can use array((a0, ..., an)), as long as all arrays 
have the same shape. 
</doc>

Is it a bug that array((a0,a1)) returns different type that a0 ? 
(type(a0)==type(a1))

>>> a = na.zeros((3,3),na.UInt16)
>>> b = na.zeros((3,3),na.UInt16)
>>> na.concatenate((a,b))
[[0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]]
>>> _.type()
UInt16
>>> na.array((a,b))
[[[0 0 0]
  [0 0 0]
  [0 0 0]]

 [[0 0 0]
  [0 0 0]
  [0 0 0]]]
>>> _.type()
Int32



Thanks,
Sebastian Haase



From cjw at sympatico.ca  Tue Feb  8 10:56:23 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Tue Feb  8 10:56:23 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42090476.7080604@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov>
Message-ID: <42090AE9.7060709@sympatico.ca>

Chris Barker wrote:

> Travis Oliphant wrote:
>
>> Very loosely a matlab or IDL-like environment.  Nobody has really 
>> defined what it means to be a MATLAB-like environment. 
>
>
> Well, I HOPE it means:
>
> An environment in which you can do most of what you can do with 
> MATLAB, just as easily, but with a more pythonic approach. I wanted a 
> MATLAB like language, I'd use MATLAB (or maybe Octave or Psilab).
>
>> from scipy import *
>
>
> And I'd like to see this deprecated.
>
>> makes a lot of command-like interaction possible.
>
>
> I think command-line interaction is quite possible without import *. 
> I'm actually kind of disappointed in Numeric, from this perspective. I 
> always used to use "from Numeric import *", because Numeric was 
> designed to be used this way, which I think is a shame. Far more 
> should be methods, rather than functions. For example, there are a 
> bunch of functions that take an array as the first argument:
>
> Numeric.take(a, i)
>
> Why in the world is this not:
>
> a.take(i)
>
> If it (and many other functions) were methods, there would be far less 
> need for "import *" (I do use import "Numeric as N" to save typing)

Yes, much better.  I use import numarray as _n.  The thinking being that 
there are likely other modules to be imported eg.
import numarray.random as _r.

These other modules no longer need to be concerned with the identifiers 
used in numarray.

>
> In the docs, for many years, there has been the explanation that the 
> above approach allows people to use any python sequence, rather than 
> just arrays, with these functions. I find that that buys me very 
> little. The example given in the docs (a few years ago, is it still 
> there?) is transpose:
>
> at = Numeric.transpose(a)

Instead, why not a.T, as used in PyMatrix, with some help from the class 
structure of NumArray.

Colin W.

>
> can be passed a Python sequence that has an appropriate structure. 
> However, internally, what's going on is something like:
>
> b = Numeric.asarray(a)
> at = Numeric.transpose(b)
>
> Is it really so hard to write that extra line? Remember that 
> transpose() always returns a NumPy array, regardless of the type of 
> the input. How often are you using arbitrary python sequences anyway? 
> In my case, I am never using arbitrary python sequences unless I'm 
> writing a function or method for an external API, in which case, I 
> almost always put an Numeric.asarray() call in at the top.
>
> Well, this turned out to be too much of a rant, but the basic gist 
> was: let's make SciPy Pythonic, rather than Matlaby. And the same goes 
> for Numeric3.
>
> -Chris
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>   Is this seen as
>
>> MATLAB-like?
>>
>> I guess right now, scipy is trying to bring lots of modules under a 
>> common naming convention.
>>
>> I think the goals of scipy are very much what people are talking 
>> about here.   SciPy ultimately becomes what its users and developers 
>> want it to become.   That is why I'm so unclear as to why we have not 
>> received more support in its development --- even in the way of ideas 
>> as to how to make it better --- until the great ideas that have been 
>> suggested on this list recently.
>>
>> -Travis
>>
>>
>>
>> -------------------------------------------------------
>> SF email is sponsored by - The IT Product Guide
>> Read honest & candid reviews on hundreds of IT Products from real users.
>> Discover which products truly live up to the hype. Start reading now.
>> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>




From jmiller at stsci.edu  Tue Feb  8 11:10:10 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Tue Feb  8 11:10:10 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis"
	returns 'unexpected' type
In-Reply-To: <200502081052.40692.haase@msg.ucsf.edu>
References: <200502081052.40692.haase@msg.ucsf.edu>
Message-ID: <1107889739.26021.128.camel@halloween.stsci.edu>

On Tue, 2005-02-08 at 13:52, Sebastian Haase wrote:
> Hi,
> from the numarray documentation:
> <doc>
> concatenate(arrs, axis=0)
> Returns a new array containing copies of the data contained in all arrays # of 
> arrs= (a0, a1, ... an). The arrays ai will be concatenated along the 
> specified axis (default=0). All arrays ai must have the same shape along 
> every axis except for the one specified in axis. To concatenate arrays along 
> a newly created axis, you can use array((a0, ..., an)), as long as all arrays 
> have the same shape. 
> </doc>
> 
> Is it a bug that array((a0,a1)) returns different type that a0 ? 

Yes.  That's not the behavior I see in numarray-CVS though.  CVS looks
OK.

Regards,
Todd





From konrad.hinsen at laposte.net  Tue Feb  8 11:22:49 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Tue Feb  8 11:22:49 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42090476.7080604@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov>
Message-ID: <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net>

On 08.02.2005, at 19:27, Chris Barker wrote:

> Well, I HOPE it means:
>
> An environment in which you can do most of what you can do with  
> MATLAB, just as easily, but with a more pythonic approach. I wanted a  
> MATLAB like language, I'd use MATLAB (or maybe Octave or Psilab).

Me too!

> However, internally, what's going on is something like:
>
> b = Numeric.asarray(a)
> at = Numeric.transpose(b)
>
> Is it really so hard to write that extra line? Remember that  
> transpose()

For interactive work, yes. In particular since arrays have no special  
input notation, you have to type array(...) every time.

> always returns a NumPy array, regardless of the type of the input. How  
> often are you using arbitrary python sequences anyway? In my case, I  
> am

Much less now than in the early days of NumPy, when there was little  
existing code that worked with arrays for obvious reasons. I had a  
small library of numerical operations using lists from the era before  
NumPy, so I was quite happy to be able to interface this easily. But  
that was 10 years ago, none of that code is left unchanged by now.

Another argument for function style code is habit. I suppose few of us  
would want to write x.sin() instead of sin(x). The habits are probably  
less strong for array structure operations, but still for many of us  
functions are more familiar for mathematical stuff.

But I agree that syntactic sugar should be optional and not influence  
basic library design. It is perfectly possible to have an array method  
take() and in addition a syntactic sugar function from some optional  
module for the people who prefer that.

> Well, this turned out to be too much of a rant, but the basic gist  
> was: let's make SciPy Pythonic, rather than Matlaby. And the same goes  
> for Numeric3.

+1

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From strang at nmr.mgh.harvard.edu  Tue Feb  8 11:48:23 2005
From: strang at nmr.mgh.harvard.edu (Gary Strangman)
Date: Tue Feb  8 11:48:23 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
 <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
 <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov>
 <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net>
Message-ID: <Pine.LNX.4.60.0502081435450.16300@gate.nmr.mgh.harvard.edu>

>> b = Numeric.asarray(a)
>> at = Numeric.transpose(b)
>> 
>> Is it really so hard to write that extra line? Remember that transpose()
>
> For interactive work, yes. In particular since arrays have no special input 
> notation, you have to type array(...) every time.

Konrad hits the nail on the head, IMHO. I use interactive mode ALL the 
time (as would many converts from, e.g., Matlab). Lots of additional 
typing would greatly limit the usefulness of interactive mode and, for me, 
discourage use of python for numerics.

> Another argument for function style code is habit. I suppose few of us would 
> want to write x.sin() instead of sin(x). The habits are probably less strong 
> for array structure operations, but still for many of us functions are more 
> familiar for mathematical stuff.

And, if there's any desire to attract folks from (e.g.) Matlab (maybe 
there isn't), a function-style approach is needed. This wouldn't have to 
be the *only* implemented approach, but I would strongly (+10) favor 
having the function notation available. 'course, I majored in math, so 
sin(x) is the only notation I'd recognize. ;-)

-best
Gary



From haase at msg.ucsf.edu  Tue Feb  8 12:06:34 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue Feb  8 12:06:34 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis" returns 'unexpected' type
In-Reply-To: <1107889739.26021.128.camel@halloween.stsci.edu>
References: <200502081052.40692.haase@msg.ucsf.edu> <1107889739.26021.128.camel@halloween.stsci.edu>
Message-ID: <200502081205.22665.haase@msg.ucsf.edu>

On Tuesday 08 February 2005 11:08 am, you wrote:
> On Tue, 2005-02-08 at 13:52, Sebastian Haase wrote:
> > Hi,
> > from the numarray documentation:
> > <doc>
> > concatenate(arrs, axis=0)
> > Returns a new array containing copies of the data contained in all arrays
> > # of arrs= (a0, a1, ... an). The arrays ai will be concatenated along the
> > specified axis (default=0). All arrays ai must have the same shape along
> > every axis except for the one specified in axis. To concatenate arrays
> > along a newly created axis, you can use array((a0, ..., an)), as long as
> > all arrays have the same shape.
> > </doc>
> >
> > Is it a bug that array((a0,a1)) returns different type that a0 ?
>
> Yes.  That's not the behavior I see in numarray-CVS though.  CVS looks
> OK.

Thanks for the info, but after updating from CVS I get a problem in nd_image:
Python 2.2.1 (#1, Feb  3 2005, 06:16:31)
[GCC 2.95.4 20011002 (Debian prerelease)] on linux2
<snip>
  File "<snip>/numarray/nd_image/_ni_support.py", line 35, in ?
    _type_to_num = {
AttributeError: 'module' object has no attribute 'Any'

After changing:
...
import numarray.numerictypes as numarrayt
...
numarrayt.Any
I then complained about 
  File "<snip>/numarray/nd_image/_ni_support.py", line 53, in ?
    _type_to_num[numarray.UInt64] = 9
AttributeError: 'module' object has no attribute 'UInt64'


Any hint ?

Thanks,
Sebastian Haase



From verveer at embl.de  Tue Feb  8 12:47:36 2005
From: verveer at embl.de (Peter Verveer)
Date: Tue Feb  8 12:47:36 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis" returns 'unexpected' type
In-Reply-To: <200502081205.22665.haase@msg.ucsf.edu>
References: <200502081052.40692.haase@msg.ucsf.edu> <1107889739.26021.128.camel@halloween.stsci.edu> <200502081205.22665.haase@msg.ucsf.edu>
Message-ID: <110b816dd825e0d4d513628301fe28fb@embl.de>

> Thanks for the info, but after updating from CVS I get a problem in 
> nd_image:
> Python 2.2.1 (#1, Feb  3 2005, 06:16:31)
> [GCC 2.95.4 20011002 (Debian prerelease)] on linux2
> <snip>
>   File "<snip>/numarray/nd_image/_ni_support.py", line 35, in ?
>     _type_to_num = {
> AttributeError: 'module' object has no attribute 'Any'
>
> After changing:
> ...
> import numarray.numerictypes as numarrayt
> ...
> numarrayt.Any
> I then complained about
>   File "<snip>/numarray/nd_image/_ni_support.py", line 53, in ?
>     _type_to_num[numarray.UInt64] = 9
> AttributeError: 'module' object has no attribute 'UInt64'

Strange, I implemented this yesterday, and it works fine in my setup 
(OS X). I would think you should be able to access numarray.Any and 
numarray.UInt64 without import numarray.numerictypes. Maybe Todd can 
say what the behaviour of numarray should be here.

Peter




From jmiller at stsci.edu  Tue Feb  8 13:12:34 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Tue Feb  8 13:12:34 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis"
	returns 'unexpected' type
In-Reply-To: <110b816dd825e0d4d513628301fe28fb@embl.de>
References: <200502081052.40692.haase@msg.ucsf.edu>
	 <1107889739.26021.128.camel@halloween.stsci.edu>
	 <200502081205.22665.haase@msg.ucsf.edu>
	 <110b816dd825e0d4d513628301fe28fb@embl.de>
Message-ID: <1107897050.26021.150.camel@halloween.stsci.edu>

On Tue, 2005-02-08 at 15:45, Peter Verveer wrote:
> > Thanks for the info, but after updating from CVS I get a problem in 
> > nd_image:
> > Python 2.2.1 (#1, Feb  3 2005, 06:16:31)
> > [GCC 2.95.4 20011002 (Debian prerelease)] on linux2
> > <snip>
> >   File "<snip>/numarray/nd_image/_ni_support.py", line 35, in ?
> >     _type_to_num = {
> > AttributeError: 'module' object has no attribute 'Any'
> >
> > After changing:
> > ...
> > import numarray.numerictypes as numarrayt
> > ...
> > numarrayt.Any
> > I then complained about
> >   File "<snip>/numarray/nd_image/_ni_support.py", line 53, in ?
> >     _type_to_num[numarray.UInt64] = 9
> > AttributeError: 'module' object has no attribute 'UInt64'
> 
> Strange, I implemented this yesterday, and it works fine in my setup 
> (OS X). I would think you should be able to access numarray.Any and 
> numarray.UInt64 without import numarray.numerictypes. Maybe Todd can 
> say what the behaviour of numarray should be here.

Unfortunately, no.  I don't have access to Debian and AFIK UInt64 is
invariantly supported except for MSVC6/win32.  Try this:

>>> import numarray.numinclude as numinc
>>> numinc.hasUInt64
1

If it says 0,  it's probably a Debian thing.

If hasUInt64 is 1,  I'd suggest deleting your site-packages/numarray
directory and re-installing numarray.

Regards,
Todd






From rkern at ucsd.edu  Tue Feb  8 13:12:36 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Tue Feb  8 13:12:36 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4207D285.70103@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <1f7baacd417e55debdde0351e463cf10@laposte.net> <4203CAD0.6050609@ee.byu.edu> <3ba3b109f1f344e31b758074cbc7f64c@embl.de> <42042318.2090704@ee.byu.edu> <4207D285.70103@noaa.gov>
Message-ID: <420817D3.1000500@ucsd.edu>

Chris Barker wrote:

> If there is a small group of folks that would like to work on making a
> binary SciPy for OS-X, I'd be glad to work with others to do it. I doubt
> I'll do it myself, however.

Workin' on it.

http://www.scipy.org/wikis/featurerequests/MacEnthon

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Tue Feb  8 13:30:35 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue Feb  8 13:30:35 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
Message-ID: <42092F3F.1010707@colorado.edu>

Gary Strangman wrote:

> In sum, I pretty much agree with most previous contributors. With one 
> exception. I do agree with Konrad that, theory, the best analysis approach 
> is to build a custom class for each domain of investigation. I've even 
> done that a couple times. However, of the dozens of researchers (i.e., 
> users, not developers) I've talked about this with, almost none have the 
> time (or even desire) to design and develop a class-based approach. Their 
> goal is data analysis and evaluation. So, they pull the data into python 
> (or, more typically, matlab or IDL), analyze it, plot the results, try to 
> get a plot that's pretty enough for publication, and then go on to the 
> next project ... no time for deciding what functionality should be a 
> method or an attribute, subclassing hierarchies, etc. In those rare 
> cases where they have the desire, matlab (I don't know about IDL) doesn't 
> give you the option, meaning that users (as opposed to developers) 
> probably won't go the (unfamiliar and more time consuming) route.

I have to say that in most cases, the approach you describe from colleagues is 
an unfortunately common, but still very shortsighted one (except in cases so 
trivial that it doesn't really matter).  I'm attaching a real-world example of 
a 3d Poisson solver which uses a new algorithm combining some neat ideas.  The 
papers are being only written right now, so the only reference I can offer is 
this:

http://amath.colorado.edu/faculty/fperez/talks/0403_ams_athens.pdf

The point is that anyone should be able to read that script and understand 
what it does.  A few things worth pointing out from that code:

print "Building rho (Right Hand Side)..."
rho = FunctionFromCode(nnod,cutoff,rho_code,norm_type)

print "Building phi (Left Hand Side), analytical solution..."
phi_exact = FunctionFromCode(nnod,cutoff,phi_exact_code,norm_type)

This builds multiwavelet, adaptively decomposed objects which are quite 
complex internally, yet they behave like functions.  Likewise, this line:

print "Computing phi = poisson(rho) explicitly..."
phi  = poisson(rho)

returns another function (even though this one was constructed as a numerical 
solution instead of having an analytical representation).  So checking for 
validity, in this case where we have the exact anwer, is as simple as:

print "Building error function..."
error = phi - phi_exact
enorm = error.norm()

This subtracts two multiwavelet-represented functions in 3d, and returns 
another function of the same kind.  Furthermore, all these objects have 
plot/info methods which you can then use to visualize info or print relevant 
numerical stuff:

print "Norm info (L-%s):" % norm_type
print "rho    = %.3e" % rho.norm()
print "phi    = %.3e" % phi.norm()
print "error  = %.3e" % enorm

Plotting is equally easy:

# Surface plots across z=0.5 for rho and phi, uses MayaVi
rho.plot_surf()
phi.plot_surf()

# Interpolation error for the multiwavelet representation of the rhs/lhs
rho.plot_line(ptype='error',title='Interpolation error for charge density')
phi_exact.plot_line(ptype='error',
                     title='Interpolation error for exact solution')

# Pointwise interpolation error in the Green function's kernel
poisson.plot_kernel_error()

The point is, by making these objects 'smart', the top-level code reads like a 
regular mathematical solution of Poisson's equation, and extracting useful 
info is trivial.  And I can guarantee that in the long run, I've saved myself 
A LOT of time by writing the code in this way, rather than spitting out arrays 
left, right and center, and calling manual plot routines on them all the time. 
  When I need plots for a talk or paper, there is nothing more to do, just 
call the objects.

I should add that having this kind of design in scientific codes makes 
interactive exploration and debugging a very pleasant process: whenever 
something doesn't look right, my objects have a ton of self-diagnostic methods 
which I can quickly use to pinpoint the problem.  And this is not a 
theoretical statement, I have already many times saved myself much grief by 
being able to instantly see where the problem is, thanks to this kind of design.

So while I agree that having plain array access is needed for quick and dirty 
one-off things, basically anything beyond 50-100 lines of code is probably 
best written with a tiny bit of design.  Those 10 minutes invested thinking 
about a design sketch will pay for themselves many times over.

Regards,

f
-------------- next part --------------
A non-text attachment was scrubbed...
Name: demo_poisson_3d.py
Type: application/x-python
Size: 3596 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050208/2ba88da0/attachment-0001.bin>

From jmiller at stsci.edu  Tue Feb  8 13:30:38 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Tue Feb  8 13:30:38 2005
Subject: [Numpy-discussion] numarray: "concatenate along new axis"
	returns 'unexpected' type
In-Reply-To: <110b816dd825e0d4d513628301fe28fb@embl.de>
References: <200502081052.40692.haase@msg.ucsf.edu>
	 <1107889739.26021.128.camel@halloween.stsci.edu>
	 <200502081205.22665.haase@msg.ucsf.edu>
	 <110b816dd825e0d4d513628301fe28fb@embl.de>
Message-ID: <1107898168.26021.171.camel@halloween.stsci.edu>

On Tue, 2005-02-08 at 15:45, Peter Verveer wrote:
> > Thanks for the info, but after updating from CVS I get a problem in 
> > nd_image:
> > Python 2.2.1 (#1, Feb  3 2005, 06:16:31)
> > [GCC 2.95.4 20011002 (Debian prerelease)] on linux2
> > <snip>
> >   File "<snip>/numarray/nd_image/_ni_support.py", line 35, in ?
> >     _type_to_num = {
> > AttributeError: 'module' object has no attribute 'Any'
> >
> > After changing:
> > ...
> > import numarray.numerictypes as numarrayt
> > ...
> > numarrayt.Any
> > I then complained about
> >   File "<snip>/numarray/nd_image/_ni_support.py", line 53, in ?
> >     _type_to_num[numarray.UInt64] = 9
> > AttributeError: 'module' object has no attribute 'UInt64'
> 
> Strange, I implemented this yesterday, and it works fine in my setup 
> (OS X). I would think you should be able to access numarray.Any and 
> numarray.UInt64 without import numarray.numerictypes. Maybe Todd can 
> say what the behaviour of numarray should be here.
> 
> Peter

I looked at this some more after getting a clean checkout and discovered
that there's a problem in numarray-CVS (Python-2.2 only) introduced by a
recent bugfix.  I didn't see it because I don't normally test with 2.2.

It's fixed now in CVS,  modulo incomplete bool support for 2.2.

Regards,
Todd






From Chris.Barker at noaa.gov  Tue Feb  8 14:34:32 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Tue Feb  8 14:34:32 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502081435450.16300@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov> <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net> <Pine.LNX.4.60.0502081435450.16300@gate.nmr.mgh.harvard.edu>
Message-ID: <42093E37.9070501@noaa.gov>

Gary Strangman wrote:

> Konrad hits the nail on the head, IMHO. I use interactive mode ALL the 
> time (as would many converts from, e.g., Matlab). Lots of additional 
> typing would greatly limit the usefulness of interactive mode and, for 
> me, discourage use of python for numerics.

We all have different styles, but I was a user of Matlab for years, and 
maybe it's because I'm not a very fast typist, but when I found myself 
writing more than three lines at the prompt, I'd kick myself, and open 
an editor, and write a script. The one thing I found a little lacking 
when I started using python was an easy way to run a script inside an 
interactive session:

execfile("filename.py")

is a bit clunky. However:

$ python -i filename.py

Is not so bad at the command prompt, when all you have to do is hit the 
up-arrow key and run it again.

>> I suppose few of us 
>> would want to write x.sin() instead of sin(x).

Well, that's true for me also, but for structural array stuff, methods 
seem to make more sense. Besides, I'd get used to x.sin(), and like it 
if it kept my namespaces uncluttered!

as for the array() literal: how often do you really need to put in your 
arrays as a literal? I usually calculate, load from a file, generate 
randomly, or use arange(). And if I do type in an array literal, the 
extra typing of "array" is pretty small compared to typing all those 
numbers!

> And, if there's any desire to attract folks from (e.g.) Matlab (maybe 
> there isn't), a function-style approach is needed. This wouldn't have to 
> be the *only* implemented approach, but I would strongly (+10) favor 
> having the function notation available.

This is the approach taken by John with matplotlib. Inside, everything 
is all OO style, but with the "pylab" module creating a bunch of 
matlab-style helper functions. In fact, it's "officially" recommended 
that you use the pylab interface for interactive use, and the OO 
interface for "programming". Unfortunately, the OO interface lacks a 
number of useful features that really has nothing to do with OO vs. 
functional, it's just that the focus has been on easy usability for the 
pylab interface, but not for the OO interface. I've been working to 
change that (mostly just by complaining, but I intend to contribute code 
soon: I promise, really!)

So, in short, I'd like to see both: an internal OO structure, with a 
"interactive" helper module.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From Fernando.Perez at colorado.edu  Tue Feb  8 14:40:35 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue Feb  8 14:40:35 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42093E37.9070501@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov> <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net> <Pine.LNX.4.60.0502081435450.16300@gate.nmr.mgh.harvard.edu> <42093E37.9070501@noaa.gov>
Message-ID: <42093FAF.8050602@colorado.edu>

Chris Barker wrote:

> We all have different styles, but I was a user of Matlab for years, and 
> maybe it's because I'm not a very fast typist, but when I found myself 
> writing more than three lines at the prompt, I'd kick myself, and open 
> an editor, and write a script. The one thing I found a little lacking 
> when I started using python was an easy way to run a script inside an 
> interactive session:
> 
> execfile("filename.py")
> 
> is a bit clunky. However:
> 
> $ python -i filename.py
> 
> Is not so bad at the command prompt, when all you have to do is hit the 
> up-arrow key and run it again.

I know it's bad manners to promote your own stuff, but have you had a look at 
ipython?  The run command in ipython gives you this, plus much, much more 
(debugger access on exceptions, colored tracebacks with detail control, 
automatic profiling, namespace control,...).  My workmode is Xemacs open with 
the files + ipython using 'run foo' to rerun my foo.py script as many times as 
needed.

Sorry for the plug, but since I've spent 3 years tuning ipython precisely to 
make interactive numerical work as pleasant and efficient as possible, I can't 
  resist :)

Regards,

f



From juenglin at cs.pdx.edu  Tue Feb  8 16:42:26 2005
From: juenglin at cs.pdx.edu (Ralf Juengling)
Date: Tue Feb  8 16:42:26 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42090476.7080604@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
	 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
	 <4207CC4C.8060603@ee.byu.edu>
	 <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
	 <42084A1F.3030007@ee.byu.edu>  <42090476.7080604@noaa.gov>
Message-ID: <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>

On Tue, 2005-02-08 at 10:27, Chris Barker wrote:
> Travis Oliphant wrote:
> > from scipy import *
> 
> And I'd like to see this deprecated.

I don't understand what's wrong with this. (Of course, the module
you do the "from XXX import *" with should be designed for this).

This is on of the hallmarks of Python that facilitate rapid 
development: When you want to get familiar with an unfamiliar 
library module, start up python in a shell, do the 
"from XXX import *" thing, and start playing.

Name space pollution is not an issue, because when you write 
programs, you don't do "from XXX import *". 

ralf





From Fernando.Perez at colorado.edu  Tue Feb  8 16:50:11 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue Feb  8 16:50:11 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>	 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>	 <4207CC4C.8060603@ee.byu.edu>	 <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>	 <42084A1F.3030007@ee.byu.edu>  <42090476.7080604@noaa.gov> <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
Message-ID: <42095E13.2020002@colorado.edu>

Ralf Juengling wrote:

> This is on of the hallmarks of Python that facilitate rapid 
> development: When you want to get familiar with an unfamiliar 
> library module, start up python in a shell, do the 
> "from XXX import *" thing, and start playing.

If I may suggest, it tends to be a lot more efficient to explore an unknown 
library by doing

import XXX_LONGNAME_XXX as X

and start tab-completing away on X.<TAB>, or X.get<TAB> for getters, etc. 
This way, you know you can restrict your completions to only the thing you are 
trying to study, instead of all of its names colliding with builtins and globals.

So if anything, interactive _exploration_ of a library is one place not to use 
'from XXX import *'.

Interactive _use_ is an altogether different story.  I constantly use ipython 
in -pylab mode, which pulls all of matplotlib.pylab, and up until a week ago 
did the same with the supplied numeric profile (which used Numeric and 
Gnuplot).   But this is for using names I already know well.  And I still make 
a point, even in these conditions, of having pylab/Numeric available as 
top-level names, so I can do selective tab-completion on them when I need to 
fish around for something I don't know or whose specific name I can't remember.

Just a small comment.

Best,

f



From stephen.walton at csun.edu  Tue Feb  8 17:00:36 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Tue Feb  8 17:00:36 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42084D42.9050509@colorado.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu>
Message-ID: <42096070.5030608@csun.edu>

Fernando Perez wrote:

> Travis Oliphant wrote:
>
>> Currently, I don't think Python allows "over-riding" the "and" operation.
>
>
> You are correct:

What I want must be possible somehow, though:

In [6]: a=[True,False]

In [7]: b=[False,False]

In [8]: a and b
Out[8]: [False, False]

In [9]: a or b
Out[9]: [True, False]

In [10]: not a
Out[10]: False

So, standard Python does apply logical "and" and "or", but not "not", 
element by element to a list or tuple of booleans.  Is there no way to 
get access to this for a numarray bool array?  Can we get Guido to 
change "not" to also operate element-wise on a list of booleans?

I have a really common task in MATLAB I want to have work here.  In 
MATLAB I can extract the overlapping good values from two data sets by doing

f = data1 ~= flagval & data2 ~= flagval;

or even more compactly, since I use IEEE NaN for flagval

f = ~isnan(data1) & ~isnan(data2);

Right now in numarray I have to use

f = (data1 != flagval) & (data2 != flagval)

with the extra parentheses and, confusingly, using bitwise AND instead 
of logical AND.  Even if I use ieeespecial nan as flagval, there's no 
"notisnan(data1)" and no way to say "!isnan(data1)"

Of course, masked arrays are another option for this kind of usage but 
they have their own difficulties:  matplotlib does not at present allow 
one to plot from masked arrays, for example.






From Chris.Barker at noaa.gov  Tue Feb  8 17:08:19 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Tue Feb  8 17:08:19 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>	 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>	 <4207CC4C.8060603@ee.byu.edu>	 <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>	 <42084A1F.3030007@ee.byu.edu>  <42090476.7080604@noaa.gov> <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
Message-ID: <420961B3.1000405@noaa.gov>

Ralf Juengling wrote:
>>>from scipy import *
>>
>>And I'd like to see this deprecated.
> 
> I don't understand what's wrong with this. (Of course, the module
> you do the "from XXX import *" with should be designed for this).

If it's designed for this, then none of the names will clash with 
built-ins, unless intended. However, what about other modules that might 
also be done as "import *"?

It seems to me that the assumption is that whatever module is imported 
that way will be the only one imported that way. This is not always the 
case. Also, what if you need to import a module that happens to have the 
same name as an attribute of the "import *" one. You can do an "import 
as", but only if you know you have a name clash.

> Name space pollution is not an issue, because when you write 
> programs, you don't do "from XXX import *". 

Ha! That's the core problem here. As an example (I don't mean to be 
picking on you, John) matplotlib docs suggest that you do things 
differently when writing programs than playing with the interpreter. 
However, almost all of the examples and docs, and and most of the 
usability effort has gone into the "import *" approach. The same is true 
for Numeric and numarray. Why would someone magically stop using it? 
Besides, I use the interactive interpreter to try out code that I'm then 
going to use in a program. Why have a different syntax?


"Namespaces are one honking great idea -- let's do more of those!"

Note, that's "more", not "less"

OK, I'll stop ranting now.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From Fernando.Perez at colorado.edu  Tue Feb  8 17:14:29 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue Feb  8 17:14:29 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42096070.5030608@csun.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu> <42096070.5030608@csun.edu>
Message-ID: <42096303.5030805@colorado.edu>

Stephen Walton wrote:
> Fernando Perez wrote:
> 
> 
>>Travis Oliphant wrote:
>>
>>
>>>Currently, I don't think Python allows "over-riding" the "and" operation.
>>
>>
>>You are correct:
> 
> 
> What I want must be possible somehow, though:
> 
> In [6]: a=[True,False]
> 
> In [7]: b=[False,False]
> 
> In [8]: a and b
> Out[8]: [False, False]
> 
> In [9]: a or b
> Out[9]: [True, False]
> 
> In [10]: not a
> Out[10]: False
> 
> So, standard Python does apply logical "and" and "or", but not "not", 
> element by element to a list or tuple of booleans.  Is there no way to 
> get access to this for a numarray bool array?  Can we get Guido to 
> change "not" to also operate element-wise on a list of booleans?

I think you are misunderstanding something.  Python does not "apply logical 
"and" and "or", but not "not",  element by element to a list or tuple of 
booleans", as you say.  Look again at the decompiled bytecode:

In [2]: dis.dis(compile('a and b','<>','eval'))
   0           0 LOAD_NAME                0 (a)
               3 JUMP_IF_FALSE            4 (to 10)
               6 POP_TOP
               7 LOAD_NAME                1 (b)
         >>   10 RETURN_VALUE

What it does is execute the JUMP_IF_FALSE bytecode, which is mapped to the 
__nonzero__ special method.  From http://docs.python.org/ref/customization.html:

__nonzero__(  	self)
     Called to implement truth value testing, and the built-in operation 
bool(); should return False or True, or their integer equivalents 0 or 1. When 
this method is not defined, __len__() is called, if it is defined (see below). 
If a class defines neither __len__() nor __nonzero__(), all its instances are 
considered true.

Now, lists don't actually have this method, so a call to __len__ is made, and 
any non-empty list is considered true.  The return value is then whatever is 
on the stack, which will be either be a, or b if a tested false:

In [6]: a
Out[6]: [True, False]

In [7]: a and 0
Out[7]: 0

In [8]: a and 1
Out[8]: 1

In [9]: a and 'hello'
Out[9]: 'hello'


The point is that and/or only operate on one operand at a time, with unary 
functions/methods.  They do NOT call a binary function, the way & does:

In [5]: dis.dis(compile('a & b','<>','eval'))
   0           0 LOAD_NAME                0 (a)
               3 LOAD_NAME                1 (b)
               6 BINARY_AND
               7 RETURN_VALUE

This means that certain things which can be done with & are fundamentally 
impossible with 'and', since you never have both arguments in your hands at 
the same time.

Bytecode analysis can be really useful to understand this kind of subtle 
issues.  I can't think of any language where this approach is as easy to do 
and friendly as in python.

Cheers,

f



From Fernando.Perez at colorado.edu  Tue Feb  8 17:18:31 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue Feb  8 17:18:31 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42096303.5030805@colorado.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu> <42096070.5030608@csun.edu> <42096303.5030805@colorado.edu>
Message-ID: <42096472.7050208@colorado.edu>

Fernando Perez wrote:

> Now, lists don't actually have this method, so a call to __len__ is made, and 
> any non-empty list is considered true.  The return value is then whatever is 
> on the stack, which will be either be a, or b if a tested false:
                                                             ^^^^^^

That should have read TRUE, of course, for an 'a and b' call.  For 'a or b', 
false is the right word.  Sorry about the confusion.

Cheers,

f



From simon at arrowtheory.com  Tue Feb  8 17:42:44 2005
From: simon at arrowtheory.com (Simon Burton)
Date: Tue Feb  8 17:42:44 2005
Subject: [Numpy-discussion] numarray.codegenerator.UfuncModule nary ufuncs
Message-ID: <20050209124102.4632d57b.simon@arrowtheory.com>

It looks like I may need to make n-ary (multi argument) ufuncs.

The use case is something like this: some tricky function operates on single values
but has many parameters (eg. offset and multiply). So the ufunc is called
with an array as the first argument, but just scalars in the other arguments.

I'm wondering how far is numarray.codegenerator.UfuncModule from handling n-ary functions ?
If it's just not on the radar, i'll need to rethink my approach...

bye for now,

Simon.



From juenglin at cs.pdx.edu  Tue Feb  8 17:58:35 2005
From: juenglin at cs.pdx.edu (Ralf Juengling)
Date: Tue Feb  8 17:58:35 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420961B3.1000405@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
	 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
	 <4207CC4C.8060603@ee.byu.edu>
	 <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
	 <42084A1F.3030007@ee.byu.edu>  <42090476.7080604@noaa.gov>
	 <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
	 <420961B3.1000405@noaa.gov>
Message-ID: <1107914212.28976.34.camel@alpspitze.cs.pdx.edu>

On Tue, 2005-02-08 at 17:04, Chris Barker wrote:
> Ha! That's the core problem here. As an example (I don't mean to be 
> picking on you, John) matplotlib docs suggest that you do things 
> differently when writing programs than playing with the interpreter. 
> However, almost all of the examples and docs, and and most of the 
> usability effort has gone into the "import *" approach. The same is true 
> for Numeric and numarray. Why would someone magically stop using it? 
> Besides, I use the interactive interpreter to try out code that I'm then 
> going to use in a program. Why have a different syntax?

Yes, I think it is absurd to ask a user to learn two APIs, one for
interactive use and one for writing programs.

I'm not sure I understand your stance on this, though. You were
complaining in the other posting that matplotlib's "programming API"
would not offer all of the features that are offered by its 
"interactive API". But you don't seem to be offended by the very 
idea of having two APIs.

Ralf





From jdhunter at ace.bsd.uchicago.edu  Tue Feb  8 18:46:10 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Tue Feb  8 18:46:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1107914212.28976.34.camel@alpspitze.cs.pdx.edu> (Ralf
 Juengling's message of "Tue, 08 Feb 2005 17:56:53 -0800")
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
	<42065E31.20906@astraw.com>
	<4ad81937f722051f27b37915225a1563@laposte.net>
	<Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
	<4207CC4C.8060603@ee.byu.edu>
	<Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>
	<42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov>
	<1107909680.28976.17.camel@alpspitze.cs.pdx.edu>
	<420961B3.1000405@noaa.gov>
	<1107914212.28976.34.camel@alpspitze.cs.pdx.edu>
Message-ID: <m3ll9yv4tr.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Ralf" == Ralf Juengling <juenglin at cs.pdx.edu> writes:

    >> Ha! That's the core problem here. As an example (I don't mean
    >> to be picking on you, John) matplotlib docs suggest that you do
    >> things differently when writing programs than playing with the
    >> interpreter.  However, almost all of the examples and docs, and
    >> and most of the usability effort has gone into the "import *"
    >> approach. The same is true for Numeric and numarray. Why would
    >> someone magically stop using it?  Besides, I use the
    >> interactive interpreter to try out code that I'm then going to
    >> use in a program. Why have a different syntax?

    Ralf> Yes, I think it is absurd to ask a user to learn two APIs,
    Ralf> one for interactive use and one for writing programs.

Just to make sure I'm not being unfairly represented here :-)

matplotlib has a pylab interface which allows you to make plots from
scripts or from the python shell.  Eg you can do
 
  from pylab import *
  plot([1,2,3])
  xlabel('hi mom')

and a figure window will pop up if you've configured your matplotlib
environment for interactive use with a GUI.  Now a lot happens under
the hood when you make these commands -- in particular, the pylab
interface has the concept of the current figure and axes, and these
are created automagically for you if you haven't explicitly created
them already.  Which means the module is realizing and managing GUI
windows, binding events to them, managing their destruction, and so
on.

What we advise is if you are using matplotlib in a GUI application you
are writing -- eg you are explicitly creating wx/gtk/tk/etc windows
and toolbars and menus -- then you shouldn't use the pylab interface.
The reason is that the pylab interface is trying to create and manage
GUI windows, and you might get into a conflict if you are also trying
to create and manage GUI windows.  So as a GUI application developer,
you are responsible for these things and use the matplotlib OO API --
the pylab interface is a thin wrapper to this.

I think the misleading statement is

    >> you do things differently when writing programs than playing
    >> with the interpreter

it would be accurate to say you should not use the pylab interface
when manually embedding matplotlib in a GUI, but you are free to use
the pylab interface or the OO API in python scripts or from the python
shell.

And now we have wandered dangerously off-topic...

JDH



From konrad.hinsen at laposte.net  Wed Feb  9 02:47:28 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Wed Feb  9 02:47:28 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42093E37.9070501@noaa.gov>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <4207CC4C.8060603@ee.byu.edu> <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu> <42084A1F.3030007@ee.byu.edu> <42090476.7080604@noaa.gov> <4d2732eec6dbb0caec65eb1e2621c8e7@laposte.net> <Pine.LNX.4.60.0502081435450.16300@gate.nmr.mgh.harvard.edu> <42093E37.9070501@noaa.gov>
Message-ID: <70144ea5aefd645d56d65ad9400575d6@laposte.net>

On Feb 8, 2005, at 23:33, Chris Barker wrote:

> thing I found a little lacking when I started using python was an easy 
> way to run a script inside an interactive session:
>
> execfile("filename.py")
>
> is a bit clunky. However:
>
> $ python -i filename.py
>
> Is not so bad at the command prompt, when all you have to do is hit 
> the up-arrow key and run it again.

Try ipython. "%run filename.py" runs the script and you can recall it 
with the up-arrow key. There are lots of other nice features in 
ipython, once you bring up the courage to go through the manual (66 
pages!).

> Well, that's true for me also, but for structural array stuff, methods 
> seem to make more sense. Besides, I'd get used to x.sin(), and like it 
> if it kept my namespaces uncluttered!

I wonder if you would say the same after reading a bunch of complicated 
formulas in Eiffel code (which requires such notation). I did, and 
didn't find it pleasurable. But I would happily accept methods for 
anything but standard math functions.

> So, in short, I'd like to see both: an internal OO structure, with a 
> "interactive" helper module.

We can agree on that!

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From mdehoon at ims.u-tokyo.ac.jp  Wed Feb  9 03:05:34 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Wed Feb  9 03:05:34 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42055FA3.9050909@ee.byu.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	<1f7baacd417e55debdde0351e463cf10@laposte.net>	<4203CAD0.6050609@ee.byu.edu>	<3ba3b109f1f344e31b758074cbc7f64c@embl.de>	<42042318.2090704@ee.byu.edu>	<c533ee29143e3286b075927540f5844b@embl.de> <m37jlnc57g.fsf@peds-pc311.bsd.uchicago.edu> <42055FA3.9050909@ee.byu.edu>
Message-ID: <4209EE2B.6090605@ims.u-tokyo.ac.jp>

Travis Oliphant wrote:
> Again,  from my perspective once Numeric3 is working reasonably well, 
> Numeric ceases to exist for me.  Yes others may continue to download it, 
> but I won't be maintaining it anymore.
> 
It is certainly fair for SciPy developers to focus on Numeric3/SciPy and 
stop maintaining Numeric. In open-source development, people get to 
choose what they want to work on. And if creating Numeric3 will lead to 
a more useable SciPy, I'm all for it.

 From the discussion on this list, it seems that there are still users 
who will continue to use Numeric for the time being. It would therefore 
be good if some of those users can become Numeric maintainers for the 
benefit of people who choose to continue using Numeric for now. Is 
anybody else interested in becoming a Numeric maintainer? Since the 
focus will be on maintenance rather than code development, I expect the 
workload to be acceptable. Is anybody else in favor of appointing 
Numeric maintainers?

--Michiel.



From konrad.hinsen at laposte.net  Wed Feb  9 03:12:51 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Wed Feb  9 03:12:51 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42092F3F.1010707@colorado.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <42092F3F.1010707@colorado.edu>
Message-ID: <03bdbcfa147243f9e3d1649781101a41@laposte.net>

On Feb 8, 2005, at 22:29, Fernando Perez wrote:

> it doesn't really matter).  I'm attaching a real-world example of a 3d 
> Poisson solver which uses a new algorithm combining some neat ideas.  
> The papers are being only written right now, so the only reference I 
> can offer is this:

Nice!  Stuff like that is the best advertising for Python in science in 
my opinion. I wonder if it is possible to have a collection of 
references in a well visible place.

> The point is, by making these objects 'smart', the top-level code 
> reads like a regular mathematical solution of Poisson's equation, and 
> extracting useful info is trivial.  And I can guarantee that in the 
> long run, I've saved myself A LOT of time by writing the code in this 
> way, rather than spitting out arrays left, right and center, and

Exactly. To use a buzzword, you make a Problem-Solving Environment in 
the form of Python classes.

> I should add that having this kind of design in scientific codes makes 
> interactive exploration and debugging a very pleasant process: 
> whenever something doesn't look right, my objects have a ton of 
> self-diagnostic methods which I can quickly use to pinpoint the 
> problem.  And this is not a theoretical statement, I have already many 
> times saved myself much grief by being able to instantly see where the 
> problem is, thanks to this kind of design.

My experience (and usage style) is exactly the same.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From strang at nmr.mgh.harvard.edu  Wed Feb  9 05:36:10 2005
From: strang at nmr.mgh.harvard.edu (Gary Strangman)
Date: Wed Feb  9 05:36:10 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <42092F3F.1010707@colorado.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>
 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>
 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>
 <42092F3F.1010707@colorado.edu>
Message-ID: <Pine.LNX.4.60.0502090824050.21865@gate.nmr.mgh.harvard.edu>

> So while I agree that having plain array access is needed for quick and dirty 
> one-off things, basically anything beyond 50-100 lines of code is probably 
> best written with a tiny bit of design.  Those 10 minutes invested thinking 
> about a design sketch will pay for themselves many times over.

I guess this points out different needs and abilities. I unfortunately 
need to do way too many things just once (ahhh, if I only had the luxury 
of doing the same or similar analyses over and over). And though I 
certainly agree that including a bit of design can be a good thing, for my 
abilities this usually boils down to more like 10 hours rather than 10 
minutes of designing, re-designing, debugging the design, adding to the 
design etc. I'm usually happy with it when I'm done, but don't get to 
reuse it more than 1 or 2 times.

To get back to the topic at hand, though, I'd be perfectly satisfied with 
both interfaces (functions and methods). I just worry that the method 
interface would make the multiarray object sufficiently complex and 
unwieldy to (a) induce a performance hit, or (b) prevent the multiarray 
object from getting into the python core. But, maybe I just misunderstand 
the nature of the code that would underly the method interface.

-best
Gary



From konrad.hinsen at laposte.net  Wed Feb  9 06:06:49 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Wed Feb  9 06:06:49 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <Pine.LNX.4.60.0502090824050.21865@gate.nmr.mgh.harvard.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp> <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net> <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu> <42092F3F.1010707@colorado.edu> <Pine.LNX.4.60.0502090824050.21865@gate.nmr.mgh.harvard.edu>
Message-ID: <966d28377eda318746cc75e990063144@laposte.net>

On Feb 9, 2005, at 14:35, Gary Strangman wrote:

> interfaces (functions and methods). I just worry that the method 
> interface would make the multiarray object sufficiently complex and 
> unwieldy to (a) induce a performance hit, or (b) prevent the 
> multiarray object from getting into the python core. But, maybe I just 
> misunderstand the nature of the code that would underly the method 
> interface.

Performance-wise there is no difference between methods and functions. 
And I don't see why a method interface would prevent arrays from 
getting into the core, considering that most Python types have a method 
interface. The string handling stuff was even moved from tunction style 
to method style between Python 1.5 and 2.0.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From jmiller at stsci.edu  Wed Feb  9 06:33:46 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Feb  9 06:33:46 2005
Subject: [Numpy-discussion] numarray.codegenerator.UfuncModule nary
	ufuncs
In-Reply-To: <20050209124102.4632d57b.simon@arrowtheory.com>
References: <20050209124102.4632d57b.simon@arrowtheory.com>
Message-ID: <1107959538.5863.43.camel@jaytmiller.comcast.net>

On Wed, 2005-02-09 at 12:41 +1100, Simon Burton wrote:
> It looks like I may need to make n-ary (multi argument) ufuncs.
> 
> The use case is something like this: some tricky function operates on single values
> but has many parameters (eg. offset and multiply). So the ufunc is called
> with an array as the first argument, but just scalars in the other arguments.
> 
> I'm wondering how far is numarray.codegenerator.UfuncModule from handling n-ary functions ?
> If it's just not on the radar, i'll need to rethink my approach...
> 
> bye for now,
> 
> Simon.

The numarray n-ary ufunc system is very definitely on the radar,  but
still alpha.  Support for the kind of work you're trying to do is in the
code generation API but the ufunc runtime system needs more work for it
to be as fast as it can be.  For large arrays it should already be
pretty good.

There's an example of the nary-ufunc code generation in
numarray/Examples/ufunc/setup_special.py.  I added an example ufunc,
vsdemo(), similar in form to what I think you want,  one array input
parameter and 4 vector input parameters yielding a single value result.

Feedback is welcome.

Regards,
Todd




From faltet at carabos.com  Wed Feb  9 07:38:16 2005
From: faltet at carabos.com (Francesc Altet)
Date: Wed Feb  9 07:38:16 2005
Subject: [Numpy-discussion] Efficiently converting a numarray object to Numeric
Message-ID: <200502091637.32976.faltet@carabos.com>

Hi,

I'm wondering if there is a more efficient way to convert a numarray
object into a Numeric one than:

>>> import Numeric
>>> import numarray
>>> from time import time
>>> a = numarray.arange(100*1000)
>>> t1=time();b=Numeric.array(a);time()-t1
0.43448090553283691

Any suggestions?

-- 
>qo<   Francesc Altet ? ? http://www.carabos.com/
V ?V   C?rabos Coop. V. ??Enjoy Data
 ""




From jmiller at stsci.edu  Wed Feb  9 08:28:33 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Feb  9 08:28:33 2005
Subject: [Numpy-discussion] Efficiently converting a numarray object to
	Numeric
In-Reply-To: <200502091637.32976.faltet@carabos.com>
References: <200502091637.32976.faltet@carabos.com>
Message-ID: <1107966462.5863.55.camel@jaytmiller.comcast.net>

On Wed, 2005-02-09 at 16:37 +0100, Francesc Altet wrote:
> Hi,
> 
> I'm wondering if there is a more efficient way to convert a numarray
> object into a Numeric one than:
> 
> >>> import Numeric
> >>> import numarray
> >>> from time import time
> >>> a = numarray.arange(100*1000)
> >>> t1=time();b=Numeric.array(a);time()-t1
> 0.43448090553283691
> 
> Any suggestions?

Yes!  

import Numeric
import numarray
from time import time

a = numarray.arange(100*1000)

t1=time();b=Numeric.array(a);
print time()-t1

t1 = time();  c=Numeric.fromstring(a._data, typecode='i')
print time()-t1

print b[-10:]
print c[-10:]
      
I get:

1.58109998703
0.00659704208374
[99990 99991 99992 99993 99994 99995 99996 99997 99998 99999]
[99990 99991 99992 99993 99994 99995 99996 99997 99998 99999]

Note that in CVS,  fromstring(a, typecode='i') now also works because
numarray now supplies the buffer protocol as well as consuming buffers.

Regards,
Todd






From jmiller at stsci.edu  Wed Feb  9 08:38:13 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Feb  9 08:38:13 2005
Subject: [Numpy-discussion] Efficiently converting a numarray object to
	Numeric
In-Reply-To: <1107966462.5863.55.camel@jaytmiller.comcast.net>
References: <200502091637.32976.faltet@carabos.com>
	 <1107966462.5863.55.camel@jaytmiller.comcast.net>
Message-ID: <1107966977.5863.58.camel@jaytmiller.comcast.net>

On Wed, 2005-02-09 at 11:27 -0500, Todd Miller wrote:
> On Wed, 2005-02-09 at 16:37 +0100, Francesc Altet wrote:
> > Hi,
> > 
> > I'm wondering if there is a more efficient way to convert a numarray
> > object into a Numeric one than:
> > 
> > >>> import Numeric
> > >>> import numarray
> > >>> from time import time
> > >>> a = numarray.arange(100*1000)
> > >>> t1=time();b=Numeric.array(a);time()-t1
> > 0.43448090553283691
> > 
> > Any suggestions?
> 
> Yes!  
> 
> import Numeric
> import numarray
> from time import time
> 
> a = numarray.arange(100*1000)
> 
> t1=time();b=Numeric.array(a);
> print time()-t1
> 
> t1 = time();  c=Numeric.fromstring(a._data, typecode='i')
> print time()-t1
> 
> print b[-10:]
> print c[-10:]
>       
> I get:
> 
> 1.58109998703
> 0.00659704208374
> [99990 99991 99992 99993 99994 99995 99996 99997 99998 99999]
> [99990 99991 99992 99993 99994 99995 99996 99997 99998 99999]
> 
> Note that in CVS,  fromstring(a, typecode='i') now also works because
> numarray now supplies the buffer protocol as well as consuming buffers.
> 

One more thing is necessary to handle a misbehaved numarray and make its
buffer usable by Numeric:

if not a.is_c_array():
	a = a.copy()

Regards,
Todd






From perry at stsci.edu  Wed Feb  9 08:49:36 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Wed Feb  9 08:49:36 2005
Subject: [Numpy-discussion] Proposal for making of Numarray a real Numeric 'NG'
In-Reply-To: <4205C5DE.80504@ee.byu.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCOEANFJAA.perry@stsci.edu>

I thought I would clarify some historical issues (at least as far as
I recall them) in helping understand how things got the way they are 
today.

Travis Oliphant wrote:
> Francesc Altet wrote:
> 
> >  
> >
> I think numarray has made some incrdedible strides in showing what the 
> numeric object needs to be and in implementing some really neat 
> functionality.   I just think its combination of Python and C code must 
> be redone to overcome the speed issues that have arisen.  My opinion 
> after perusing the numarray code is that it would be easier (for me 
> anyway) to adjust Numeric to support the features of numarray, than to 
> re-write and re-organize the relevant sections of numarray code.  One of 
> the advantages of Numeric is it's tight implementation that added only 
> two fundamental types, both written entirely in C.   I  was hoping that 
> the Python dependencies for the fundamental types would fade as numarray 
> matured, but it appears to me that this is not going to happen.
>
When we started this, one of those that suggested putting much of
the code in Python was Guido himself if I recall correctly (I'd have
to dig up the message he wrote on the subject to see what exactly he
said). That was one of the factors influencing us to go that route
(as well as other mentioned below).
 
> I did not have the time in the past to deal with this.  I wish I had 
> looked at it more closely two years ago.   If I had done this I would 
> have seen how to support the features that Perry wanted without 
> completely re-writing everything.   But, then again, Python 2.2 changed 
> what is possible on the C level and that has had an impact on the 
> discussion.   
> 
Indeed, numarray was started well before Python 2.2 and was another
it wasn't done in C. Knowing what would have been available in 2.2 
would likely have changed the approach used.

> >- Memory-mapped objects: Allow working with on-disk numarray 
> objects like if
> >  they were in-memory.
> >  
> >
> Numeric3 supports this cleanly and old Numeric did too (there was a 
> memory-mapped module), it's just that byteswapping, and alignment had to 
> be done manually.
> 
Just to clarify (it may not be immediately apparent to those who haven't
had to deal with it) many memory mapped files do not use the machine
representation (as is often the case for astronomical data files). 
Memory mapping isn't nearly as useful if one has to create temporaries
to handles these cases.

> >  
> >- RecArrays: Objects that allow to deal with heterogeneous datasets
> >  (tables) in an efficient manner. This ought to be very 
> beneficial in many
> >  fields.
> >  
> >
> Heterogeneous arrays is the big one for old Numeric.  It is a good 
> idea.  In Numeric3 it has required far fewer changes than I had at first 
> imagined.
> 
>
> 
> Numeric has had this in mind for some time.  In fact the early Numeric 
> developers were quite instrumental in getting significant changes into 
> Python istelf, including Complex Objects, Ellipses, and Extended 
> Slicing.   Guido was quite keen on the idea of including Numeric at one 
> point.  Our infighting made him lose interest I think.   So claiming 
> this as an advantage of numarray over Numeric is simply inaccurate.
> 
Actually, as I understand it, Guido had ruled out including Numeric well
before numarray even got started. I can't claim to know exactly his
reasons (I've heard various things such as he looked at the code and 
didn't like it, or Paul Dubois and others advised him against it; I 
can't say exactly why), but I am sure that that decision was made 
before numarray. No doubt the split has prevented any further
consideration of inclusion. 

Perry






From konrad.hinsen at laposte.net  Wed Feb  9 09:15:35 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Wed Feb  9 09:15:35 2005
Subject: [Numpy-discussion] Proposal for making of Numarray a real Numeric 'NG'
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCOEANFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCOEANFJAA.perry@stsci.edu>
Message-ID: <4edd14ca7beb27b727d3984005dda768@laposte.net>

On Feb 9, 2005, at 17:48, Perry Greenfield wrote:

> When we started this, one of those that suggested putting much of
> the code in Python was Guido himself if I recall correctly (I'd have
> to dig up the message he wrote on the subject to see what exactly he

My memory tells me that the NumPy community asked if there was any 
chance to get arrays into the core if the code was decent, and Guido 
replied positively.

> Actually, as I understand it, Guido had ruled out including Numeric 
> well
> before numarray even got started. I can't claim to know exactly his
> reasons (I've heard various things such as he looked at the code and
> didn't like it, or Paul Dubois and others advised him against it; I

There was pretty much a concensus at the time that we shouldn't make 
fools of ourselves by proposing the existing codebase for inclusion in 
the core.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From perry at stsci.edu  Wed Feb  9 09:26:32 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Wed Feb  9 09:26:32 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <4207D18A.1070309@ee.byu.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCCEAPFJAA.perry@stsci.edu>

Travis Oliphant wrote:

> >
> > It's interesting that SciPy and matplotlib really do have much the 
> > same goal, but taken from different tacks: SciPy focused on the 
> > computational functionality, and matplotlib on the plotting. However, 
> > what's become really clear in this discussion is that a good 
> > scientific math environment really needs both, and they need to be 
> > well integrated.
> 
> The problem was that we needed someone like John to join us on the scipy 
> effort to get the plotting working well.   Our team lacked someone with 
> his skill.  All of us working on SciPy would have loved to work with 
> him.  So, I have nothing but respect for John and his efforts.  I just 
> wish we at SciPy were better recruiters :-)
>
In John's defense, there was already a scipy-related plotting project
underway (chaco). It wouldn't be surprising for someone to conclude
that an alternative plotting project wouldn't be all that welcome
(and John has already explained why he didn't get involved with 
chaco).
[...]
> 
> 4) The numarray C-API seems way too big.  There are so many, seemingly 
> repeated calls that seem to do the same thing. Are all of these API 
> calls really necessary? 
> 
I'm guessing you are talking all the various ways of accessing elements
in an array. It's not clear to us that these are needed either (I have a 
feeling that no one is using some of the alternatives). The reasons
behind them were to allow C programs a way of accessing arrays with
byteswapped or misaligned values without having to do those operations
explicitly themselves. Various options were provided so that one could
trade off memory usage against speed (e.g., macros vs function call/pixel).
I think the reasons are fairly well given in the manual. I don't know
if that justifies the existence of all of them and we've considered
removing some of them.

Perry




From stephen.walton at csun.edu  Wed Feb  9 10:13:41 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Wed Feb  9 10:13:41 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <42096303.5030805@colorado.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu> <42096070.5030608@csun.edu> <42096303.5030805@colorado.edu>
Message-ID: <420A526C.3030804@csun.edu>

Fernando Perez wrote a good deal clarifying how Python's logical 
operators work, for which I'm grateful.  One brief comment:

> Bytecode analysis can be really useful to understand this kind of 
> subtle issues.  I can't think of any language where this approach is 
> as easy to do and friendly as in python.

Pardon me, but this sounds a bit like looking at the assembly language 
Fortran generates in order to understand the subtleties of that language 
:-) .  I know people who used to do that, and were proud of it, but I'd 
rather the language itself didn't behave in surprising ways.  At least 
in Fortran, 'a .and. b' only made sense if a and b were both of type 
LOGICAL;  in fact, it was a compile time error if they weren't.

Anyway, we've wandered afar from the main point:  what is the best way 
to duplicate the functionality of the following MATLAB snippet:

f = ~isnan(data1) & ~isnan(data2);
plot(x,data1(f),x,data2(f))

?




From Fernando.Perez at colorado.edu  Wed Feb  9 10:24:27 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed Feb  9 10:24:27 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <420A526C.3030804@csun.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu> <42096070.5030608@csun.edu> <42096303.5030805@colorado.edu> <420A526C.3030804@csun.edu>
Message-ID: <420A551F.3020206@colorado.edu>

Stephen Walton wrote:
> Fernando Perez wrote a good deal clarifying how Python's logical 
> operators work, for which I'm grateful.  One brief comment:
> 
> 
>>Bytecode analysis can be really useful to understand this kind of 
>>subtle issues.  I can't think of any language where this approach is 
>>as easy to do and friendly as in python.
> 
> 
> Pardon me, but this sounds a bit like looking at the assembly language 
> Fortran generates in order to understand the subtleties of that language 
> :-) .  I know people who used to do that, and were proud of it, but I'd 

Oh, and it is :)  It's just that it's so immediate to do it in python, that it 
actually becomes a normal approach.  I've often used it to clarify behavior or 
performance issues to myself.  I'm not proud of it, it's just a useful trick 
which can come in handy in certain cases.

> rather the language itself didn't behave in surprising ways.  At least 
> in Fortran, 'a .and. b' only made sense if a and b were both of type 
> LOGICAL;  in fact, it was a compile time error if they weren't.

You can argue that this is a wart (I don't think it is), but it was certainly 
done by design:

http://docs.python.org/ref/bitwise.html#l2h-423
http://docs.python.org/ref/Booleans.html#l2h-440

'and' is meant to provie truth checking, with (critically important in many 
applications) short-circuit behavior.  '&' is a binary operation (hence 
necessarily non short-circuiting, due to python's argument evaluation rules), 
most often used for bitwise manipulations.  It seems clear to me that they 
both have a valid use.

> Anyway, we've wandered afar from the main point:  what is the best way 
> to duplicate the functionality of the following MATLAB snippet:
> 
> f = ~isnan(data1) & ~isnan(data2);
> plot(x,data1(f),x,data2(f))

I don't have a clean answer for this one, sorry.

Best,

f



From perry at stsci.edu  Wed Feb  9 10:48:31 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Wed Feb  9 10:48:31 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <420A526C.3030804@csun.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCCEBBFJAA.perry@stsci.edu>

Stephen Walton wrote:

> Anyway, we've wandered afar from the main point:  what is the best way 
> to duplicate the functionality of the following MATLAB snippet:
> 
> f = ~isnan(data1) & ~isnan(data2);
> plot(x,data1(f),x,data2(f))
> 
Strictly speaking there is no operator equivalent (you can always use
the ufunc logical_and: f = logical_and(~isnan(data1), ~isnan(data2)) )
(using isnan for the sake of illustration)

I'm not sure if this is wise advice, I point out to people that 
generally speaking, the arguments to the and operator are typically
boolean arrays and for numarray anyway (I forget if it can be different
for Numeric) you can simply use the '&' bitwise operator to the same
effect. No doubt that such use will eventually be misinterpreted 
and someone will provide a non-boolean argument resulting in confusion
(perhaps a mandatory comment is needed in such cases)

No doubt this is a wart from array user's perspectives.

Perry




From perry at stsci.edu  Wed Feb  9 11:24:32 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Wed Feb  9 11:24:32 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <200502080935.47908.haase@msg.ucsf.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCKEBFFJAA.perry@stsci.edu>

Sebastian Haase wrote:
> (RE: numarray)
> Hi, 
> On an Opteron (Suse9) I was happy to see that I could allocate 
> the memory for 
> an Int32-1024x1024x1024 cube. (On a P4 a get 'MemoryError').
> I remember that Todd always said that python itself wasn't 
> 64bit-ready yet.
> But why is it needed to support >4GB python-lists if one is only 
> interested in 
> large numarrays (or numeric) ?
> 
In short, to support the Python sequence protocol (arrays can be
used anywhere Python code expects sequences transparently). The 
good news is that work is well underway in Python to change the
limit on sequence indices. I think it will take a number of months
for these changes to make it out (in a released version of Python as
well as the necessary updates to numarray). As soon as the changes
are committed to Python CVS, we can start working on updates to 
numarray to support them.

Perry Greenfield




From stephen.walton at csun.edu  Wed Feb  9 11:54:34 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Wed Feb  9 11:54:34 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCCEBBFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCCEBBFJAA.perry@stsci.edu>
Message-ID: <1107978754.21229.4.camel@freyer.sfo.csun.edu>

On Wed, 2005-02-09 at 13:48 -0500, Perry Greenfield wrote:
> Strictly speaking there is no operator equivalent (you can always use
> the ufunc logical_and: f = logical_and(~isnan(data1), ~isnan(data2)) )
> (using isnan for the sake of illustration)

OK, how did I miss in "Learning Python" and the numarray docs that ~ is
bitwise NOT and works on numarray bool arrays?  In point of fact,
"~isnan(data1) & ~isnan(data2)" works fine.  I kept trying !isnan(data1)
which I gather makes no sense.

Thanks for the patient help, all.





From Chris.Barker at noaa.gov  Wed Feb  9 12:22:54 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Wed Feb  9 12:22:54 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <1107914212.28976.34.camel@alpspitze.cs.pdx.edu>
References: <420280AD.8080309@ee.byu.edu> <4202F6A0.3@ims.u-tokyo.ac.jp>	 <42065E31.20906@astraw.com> <4ad81937f722051f27b37915225a1563@laposte.net>	 <Pine.LNX.4.60.0502070855170.9809@gate.nmr.mgh.harvard.edu>	 <4207CC4C.8060603@ee.byu.edu>	 <Pine.LNX.4.60.0502071620220.9823@gate.nmr.mgh.harvard.edu>	 <42084A1F.3030007@ee.byu.edu>  <42090476.7080604@noaa.gov>	 <1107909680.28976.17.camel@alpspitze.cs.pdx.edu>	 <420961B3.1000405@noaa.gov> <1107914212.28976.34.camel@alpspitze.cs.pdx.edu>
Message-ID: <420A70A7.4020203@noaa.gov>

Ralf Juengling wrote:

> I'm not sure I understand your stance on this, though. You were
> complaining in the other posting that matplotlib's "programming API"
> would not offer all of the features that are offered by its 
> "interactive API". But you don't seem to be offended by the very 
> idea of having two APIs.

Well, I'd prefer to have just one, but understand that others have 
different needs than me, so no, I'm not offended by more than one. I 
would like the OO one to be just as good as the functional one, however. 
If I don't stop reading and writing this thread, I'll never have any 
time to actually contribute to that effort, however.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From haase at msg.ucsf.edu  Wed Feb  9 13:30:36 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed Feb  9 13:30:36 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCKEBFFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCKEBFFJAA.perry@stsci.edu>
Message-ID: <200502091329.09832.haase@msg.ucsf.edu>

On Wednesday 09 February 2005 11:24 am, Perry Greenfield wrote:
> Sebastian Haase wrote:
> > (RE: numarray)
> > Hi,
> > On an Opteron (Suse9) I was happy to see that I could allocate
> > the memory for
> > an Int32-1024x1024x1024 cube. (On a P4 a get 'MemoryError').
> > I remember that Todd always said that python itself wasn't
> > 64bit-ready yet.
> > But why is it needed to support >4GB python-lists if one is only
> > interested in
> > large numarrays (or numeric) ?
>
> In short, to support the Python sequence protocol (arrays can be
> used anywhere Python code expects sequences transparently). 

Perry,
Thanks for the answer - that's about what I remembered.
Is is conceivable to have a compiler flag (#define) to turn all the sequence 
protocol support off ?  (How many places in the numarray code would that be?)

I think the wish to just allocate large arrays and (even if in a very limited 
way!) work with them is clearly shared by many people.
I always felt having a Py-List longer than billons of elements would have a 
rather low priority in comparison.

Thanks again,
Sebastian


> The 
> good news is that work is well underway in Python to change the
> limit on sequence indices. I think it will take a number of months
> for these changes to make it out (in a released version of Python as
> well as the necessary updates to numarray). As soon as the changes
> are committed to Python CVS, we can start working on updates to
> numarray to support them.
>
> Perry Greenfield



From perry at stsci.edu  Wed Feb  9 15:34:39 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Wed Feb  9 15:34:39 2005
Subject: [Numpy-discussion] NumPy on 64-bit Linux (Opteron)
In-Reply-To: <200502091329.09832.haase@msg.ucsf.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCCEBHFJAA.perry@stsci.edu>

Sebastian Haase wrote:

> Perry,
> Thanks for the answer - that's about what I remembered.
> Is is conceivable to have a compiler flag (#define) to turn all 
> the sequence 
> protocol support off ?  (How many places in the numarray code 
> would that be?)
> 
Offhand, I'm not sure (Todd probably has a much better quick answer
for this question than I do). It is perhaps something that would
allow us to test 64-bit arrays before the Python changes are
ready if it isn't too hard to optionally disable the dependence
on Python sequences. It's worth a quick look at after 1.2 is released.

> I think the wish to just allocate large arrays and (even if in a 
> very limited 
> way!) work with them is clearly shared by many people.
> I always felt having a Py-List longer than billons of elements 
> would have a 
> rather low priority in comparison.
> 
Just to note, we more and more local users who are begining to
have a need for such large arrays too.

Perry




From oliphant at ee.byu.edu  Wed Feb  9 15:48:15 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb  9 15:48:15 2005
Subject: [Numpy-discussion] [Fwd: Re: [Python-Dev] Clarification sought about including a multidimensional
 array object into Python core]
Message-ID: <420AA0F5.2030707@ee.byu.edu>


-------------- next part --------------
An embedded message was scrubbed...
From: unknown sender
Subject: no subject
Date: no date
Size: 38
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050209/3062c877/attachment-0001.mht>

From gvanrossum at gmail.com  Wed Feb  9 17:45:18 2005
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Wed, 9 Feb 2005 14:45:18 -0800
Subject: [Python-Dev] Clarification sought about including a multidimensional array object into Python core
In-Reply-To: <420A8406.4020808@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu>
Message-ID: <ca471dc205020914453f4da355@mail.gmail.com>

> 1) What specifically about Numeric prevented it from being acceptable as
> an addition to the Python core.

It's very long ago, I believe that the authors themselves didn't think
it was good enough. It certainly had a very hackish coding style.

Numarray was supposed to fix all that. I'm sorry to hear that it
hasn't (yet) reached the maturity you find necessary.

> 2) Are there any fixed requirements (other than coding style) before an
> arrayobject would be accepted into the Python core.

The intended user community must accept the code as "best-of-breed".
It seems that the Num* community has some work to do in this respect.

Also (this applies to all code) the code must be stable enough that
the typical Python release cycle (about 18 months between feature
releases) doesn't cause problems.

Finally there must be someone willing to be responsible for
maintenance of the code.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

--------------080201020105040905060300--



From oliphant at ee.byu.edu  Wed Feb  9 15:49:01 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb  9 15:49:01 2005
Subject: [Numpy-discussion] [Fwd: [Python-Dev] Clarification sought about including a multidimensional
 array object into Python core]
Message-ID: <420AA10E.8020200@ee.byu.edu>


-------------- next part --------------
An embedded message was scrubbed...
From: unknown sender
Subject: no subject
Date: no date
Size: 38
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050209/a52ca36e/attachment-0001.mht>

From oliphant at ee.byu.edu  Wed Feb  9 16:43:34 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed, 09 Feb 2005 14:43:34 -0700
Subject: [Python-Dev] Clarification sought about including a
 multidimensional array object into Python core
Message-ID: <420A8406.4020808@ee.byu.edu>


There has recently been some much-needed discussion on the 
numpy-discussions list run by sourceforge regarding the state of the 
multidimensional array objects available for Python.  It is desired by 
many that there be a single multidimensional array object in the Python 
core to facilitate data transfer and interfacing between multiple packages.

I am a co-author of the current PEP regarding inclusion of the 
multidimensional array object into the core.  However, that PEP is 
sorely outdated.  Currently there are two multidimensional array objects 
that are in use in the Python community:
 
   Numeric --- original arrayobject created by Jim Hugunin and many 
others.  Has been developed and used for 10 years.  An upgrade that adds 
the features of numarray but maintains the same basic structure of 
Numeric called Numeric3 is in development and will be ready for more 
wide-spread use in a couple of weeks.

   Numarray --- in development for about 3 years.  It was billed by some 
as a replacement for Numeric,.  While introducing some new features, it 
still has not covered the full feature set that Numeric had making it 
impossible for all Numeric users to use it.  In addition, it is still 
unacceptably slow for many operations that Numeric does well. 

Scientific users will always have to install more packages in order to 
use Python for their purposes.  However, there is still the desire that 
the basic array object would be common among all Python users.   To 
assist in writing a new PEP, we need clarification from Guido and others 
involved regarding

1) What specifically about Numeric prevented it from being acceptable as 
an addition to the Python core.
2) Are there any fixed requirements (other than coding style) before an 
arrayobject would be accepted into the Python core.

Thanks for your comments.  I think they will help the discussion 
currently taking place.

-Travis Oliphant



_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/oliphant%40ee.byu.edu

--------------080004070407050603040506--



From andrewm at object-craft.com.au  Wed Feb  9 16:25:24 2005
From: andrewm at object-craft.com.au (Andrew McNamara)
Date: Wed Feb  9 16:25:24 2005
Subject: [Numpy-discussion] Numeric vs numarray?
Message-ID: <20050210002435.1C9323C889@coffee.object-craft.com.au>

I know this is a loaded question, but what does numarray offer that
Numeric does not (as they stand today)? A cleaner implementation?

-- 
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/



From oliphant at ee.byu.edu  Wed Feb  9 16:36:34 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb  9 16:36:34 2005
Subject: [Numpy-discussion] Re: [Python-Dev] Clarification sought about including a multidimensional
 array object into Python core
In-Reply-To: <dd28fc2f050209161264f9b601@mail.gmail.com>
References: <420A8406.4020808@ee.byu.edu>	 <ca471dc205020914453f4da355@mail.gmail.com> <dd28fc2f050209161264f9b601@mail.gmail.com>
Message-ID: <420AAC33.807@ee.byu.edu>

David Ascher wrote:

>I've not followed the num* discussion in quite a while, but my
>impression back then was that there wasn't "one" such community. 
>Instead, the technical differences in the approaches required in
>specific fields, regarding things like the relative importance of
>memory profiles, speed, error handling, willingness to require modern
>C++ compilers, etc. made practical compromises quite tricky.
>  
>

I really appreciate comments from those who remember some of the old 
discussions.

There are indeed some different needs.  Most of this, however, is in the 
ufunc object (how do you do math with the arrays).   And, a lot of this 
has been ameliorated with the new concepts of error modes that numarray 
introduced.

There is less argumentation over the basic array object as a memory 
structure.   The biggest argument right now is the design of the object: 
i.e.  a mixture of Python and C (numarray) versus a C-only object 
(Numeric3).

In other words, what I'm saying is that in terms of how the array object 
should be structure, a lot is known.  What is more controversial is 
should the design be built upon Numarray's object structure (a mixture 
of Python and C), or on Numeric's --- all in C

-Travis







From martin at v.loewis.de  Wed Feb  9 16:54:01 2005
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed Feb  9 16:54:01 2005
Subject: [Numpy-discussion] Re: [Python-Dev] Clarification sought about including a	multidimensional
 array object into Python core
In-Reply-To: <420AAC33.807@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu>		<ca471dc205020914453f4da355@mail.gmail.com>	<dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu>
Message-ID: <420AB084.1000008@v.loewis.de>

Travis Oliphant wrote:
> In other words, what I'm saying is that in terms of how the array object 
> should be structure, a lot is known.  What is more controversial is 
> should the design be built upon Numarray's object structure (a mixture 
> of Python and C), or on Numeric's --- all in C

To me, this sounds like an implementation detail. I'm sure it is an
important detail, as I understand all of this is mostly done for
performance reasons. The PEP should list the options, include criteria
for selection, and then propose a choice. People can then discuss
whether the list of options is complete (if not, you need to extend
it), whether the criteria are agreed (they might be not, and there
might be difficult consensus, which the PEP should point out), and
whether the choice is the right one given the criteria (there should
be no debate about this - everybody should agree factually that the
choice meets the criteria best).

Regards,
Martin



From paul at pfdubois.com  Wed Feb  9 17:32:00 2005
From: paul at pfdubois.com (Paul F. Dubois)
Date: Wed Feb  9 17:32:00 2005
Subject: [Numpy-discussion] Numeric life as I see it
In-Reply-To: <420AB084.1000008@v.loewis.de>
References: <420A8406.4020808@ee.byu.edu>		<ca471dc205020914453f4da355@mail.gmail.com>	<dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de>
Message-ID: <420AB928.3090004@pfdubois.com>


Martin v. L?wis wrote:
The PEP should list the options, include criteria
> for selection, and then propose a choice. People can then discuss
> whether the list of options is complete (if not, you need to extend
> it), whether the criteria are agreed (they might be not, and there
> might be difficult consensus, which the PEP should point out), and
> whether the choice is the right one given the criteria (there should
> be no debate about this - everybody should agree factually that the
> choice meets the criteria best).
> 

Unrealistic. I think it is undisputed that there are people with 
irreconcilably different needs. Frankly, we spent many, many months on 
the design of Numeric and it represents a set of compromises already. 
However, the one thing it wouldn't compromise on was speed, even at the 
expense of safety. A community exists that cannot live with this 
compromise. We were told that the Python core could also not live with 
that compromise.

Over the years there was pressure to add safety, convenience, 
flexibility, etc., all sometimes incompatible with speed. Numarray 
represents in some sense the set of compromises in that direction, 
besides its technical innovations. Numeric / Numeric3 represents the 
need for speed camp.

I think it is reasonable to suppose that the need for speed piece can be 
wrapped suitably by the need for safety-flexibility-convenience 
facilities. I believe that hope underlies Travis' plan.

The Nummies (the official set of developers) thought that the Numeric 
code base was an unsuitable basis for further development. There was no 
dissent about that at least. My idea was to get something like what 
Travis is now doing done to replace it. I felt it important to get 
myself out of the picture after five years as the lead developer 
especially since my day job had ceased to involve using Numeric.

However, removing my cork from the bottle released the unresolved 
pressure between these two camps. My plan for transition failed. I 
thought I had consensus on the goal and in fact it wasn't really there. 
Everyone is perfectly good-willed and clever and trying hard to "all 
just get along", but the goal was lost.  Eric Raymond should write a 
book about it called "Bumbled Bazaar".

I hope everyone will still try to achieve that goal. Interoperability of 
all the Numeric-related software (including supporting a 'default' 
plotting package) is required.

Aside: While I am at it, let me reiterate what I have said to the other 
developers privately: there is NO value to inheriting from the array 
class. Don't try to achieve that capability if it costs anything, even 
just effort, because it buys you nothing. Those of you who keep 
remarking on this as if it would simply haven't thought it through IMHO. 
It sounds so intellectually appealing that David Ascher and I had a 
version of Numeric that almost did it before we realized our folly.





From andrewm at object-craft.com.au  Wed Feb  9 18:51:00 2005
From: andrewm at object-craft.com.au (Andrew McNamara)
Date: Wed Feb  9 18:51:00 2005
Subject: [Numpy-discussion] Numeric life as I see it 
In-Reply-To: <420AB928.3090004@pfdubois.com> 
References: <420A8406.4020808@ee.byu.edu> <ca471dc205020914453f4da355@mail.gmail.com> <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com>
Message-ID: <20050210025044.A3FB63C889@coffee.object-craft.com.au>

>However, the one thing it wouldn't compromise on was speed, even at the
>expense of safety.

Yes, because if you don't have speed, then you might as well use the
standard services python offers.

-- 
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/



From oliphant at ee.byu.edu  Wed Feb  9 20:12:05 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb  9 20:12:05 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <420AB928.3090004@pfdubois.com>
References: <420A8406.4020808@ee.byu.edu>		<ca471dc205020914453f4da355@mail.gmail.com>	<dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com>
Message-ID: <420ADE90.9050304@ee.byu.edu>

> Martin v. L?wis wrote:
> The PEP should list the options, include criteria
>
>> for selection, and then propose a choice. People can then discuss
>> whether the list of options is complete (if not, you need to extend
>> it), whether the criteria are agreed (they might be not, and there
>> might be difficult consensus, which the PEP should point out), and
>> whether the choice is the right one given the criteria (there should
>> be no debate about this - everybody should agree factually that the
>> choice meets the criteria best).
>>
>
> Unrealistic. I think it is undisputed that there are people with 
> irreconcilably different needs. Frankly, we spent many, many months on 
> the design of Numeric and it represents a set of compromises already. 
> However, the one thing it wouldn't compromise on was speed, even at 
> the expense of safety. A community exists that cannot live with this 
> compromise. We were told that the Python core could also not live with 
> that compromise.


I'm not sure I agree.  The ufuncobject is the only place where this 
concern existed (should we trip OverFlow, ZeroDivision, etc. errors 
durring array math).   Numarray introduced and implemented the concept 
of error modes that can be pushed and popped.  I believe this is the 
right solution for the ufuncobject.

One question we are pursuing is could the arrayobject get into the core 
without a particular ufunc object.   Most see this as sub-optimal, but 
maybe it is the only way.

>
> Over the years there was pressure to add safety, convenience, 
> flexibility, etc., all sometimes incompatible with speed. Numarray 
> represents in some sense the set of compromises in that direction, 
> besides its technical innovations. Numeric / Numeric3 represents the 
> need for speed camp.

I don't see numarray as representing this at all.  To me, numarray 
represents the desire to have more flexible array types (specifically 
record arrays and maybe character arrays).   I personally don't see 
Numeric3 as in any kind of "need for speed" camp either.  I've never 
liked this distinction, because I don't think it represents a true 
dichotomy.  To me,  the differences between Numeric3 and numarray are 
currently more "architectural" than implementational.    

Perhaps you are referring to the fact that because numarray has several 
portions written in Python it is "more flexible" or "more convenient" 
but slower for small arrays.  If you are saying that then I guess 
Numeric3 is a "need for speed" implementation, and I apologize for not 
understanding. 

>
> I think it is reasonable to suppose that the need for speed piece can 
> be wrapped suitably by the need for safety-flexibility-convenience 
> facilities. I believe that hope underlies Travis' plan.

If the "safety-flexibility-convenience" facilities can be specified, 
then I'm all for one implementation.    Numeric3 design goals do not go 
against any of these ideas intentionally.

>
> The Nummies (the official set of developers) thought that the Numeric 
> code base was an unsuitable basis for further development. There was 
> no dissent about that at least. My idea was to get something like what 
> Travis is now doing done to replace it. I felt it important to get 
> myself out of the picture after five years as the lead developer 
> especially since my day job had ceased to involve using Numeric.

Some of the parts needed to be re-written, but I didn't think that meant 
moving away from the goal to have a single C-type that is the 
arrayobject.   During this process Python 2.2 came out and allowed 
sub-classing from C-types.  As Perry mentioned, and I think needs to be 
emphasized again, this changed things as any benefit from having a 
Python-class for the final basic array type disappeared --- beyond ease 
of prototyping and testing.

>
> However, removing my cork from the bottle released the unresolved 
> pressure between these two camps. My plan for transition failed. I 
> thought I had consensus on the goal and in fact it wasn't really 
> there. Everyone is perfectly good-willed and clever and trying hard to 
> "all just get along", but the goal was lost.  Eric Raymond should 
> write a book about it called "Bumbled Bazaar".

This is an accurate description.  Fortunately, I don't think any 
ill-will exists (assuming I haven't created any with my recent 
activities).  I do want to "get-along."  I just don't want to be silent 
when there are issues that I think I understand.

>
> I hope everyone will still try to achieve that goal. Interoperability 
> of all the Numeric-related software (including supporting a 'default' 
> plotting package) is required.

Utopia is always out of reach :-)

> Aside: While I am at it, let me reiterate what I have said to the 
> other developers privately: there is NO value to inheriting from the 
> array class. Don't try to achieve that capability if it costs 
> anything, even just effort, because it buys you nothing. Those of you 
> who keep remarking on this as if it would simply haven't thought it 
> through IMHO. It sounds so intellectually appealing that David Ascher 
> and I had a version of Numeric that almost did it before we realized 
> our folly.
>
I appreciate some of what Paul is saying here, but I'm not fully 
convinced that this is still true with Python 2.2 and up new-style 
c-types.   The concerns seem to be over the fact that you have to 
re-implement everything in the sub-class because the base-class will 
always return one of its objects instead of a sub-class object.
It seems to me, however,  that if the C methods use the object type 
alloc function when creating new objects then some of this problem is 
avoided (i.e. if the method is called with a sub-class type passed in, 
then a sub-class type gets set).

Have you looked at how Python now allows sub-classing in C?  I'm not an 
expert here, but it seems like a lot of the problems you were discussing 
have been ameliorated.  There are probably still issues, but....

I will know more when I seen what happens with a Matrix Object 
inheriting from a Python C-array object.

I'm wondering if anyone else with more knowledge about new-style c-types 
could help
here and show me the error of my thinking. 

-Travis










From gvanrossum at gmail.com  Wed Feb  9 20:37:06 2005
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Wed Feb  9 20:37:06 2005
Subject: [Numpy-discussion] Re: [Python-Dev] Re: Numeric life as I see it
In-Reply-To: <420ADE90.9050304@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu>
	 <ca471dc205020914453f4da355@mail.gmail.com>
	 <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu>
	 <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com>
	 <420ADE90.9050304@ee.byu.edu>
Message-ID: <ca471dc205020920364ec6fa40@mail.gmail.com>

[Paul]
> > Aside: While I am at it, let me reiterate what I have said to the
> > other developers privately: there is NO value to inheriting from the
> > array class. Don't try to achieve that capability if it costs
> > anything, even just effort, because it buys you nothing. Those of you
> > who keep remarking on this as if it would simply haven't thought it
> > through IMHO. It sounds so intellectually appealing that David Ascher
> > and I had a version of Numeric that almost did it before we realized
> > our folly.

[Travis]
> I appreciate some of what Paul is saying here, but I'm not fully
> convinced that this is still true with Python 2.2 and up new-style
> c-types.   The concerns seem to be over the fact that you have to
> re-implement everything in the sub-class because the base-class will
> always return one of its objects instead of a sub-class object.
> It seems to me, however,  that if the C methods use the object type
> alloc function when creating new objects then some of this problem is
> avoided (i.e. if the method is called with a sub-class type passed in,
> then a sub-class type gets set).

This would severely constrain the __new__ method of the subclass.

> Have you looked at how Python now allows sub-classing in C?  I'm not an
> expert here, but it seems like a lot of the problems you were discussing
> have been ameliorated.  There are probably still issues, but....
> 
> I will know more when I seen what happens with a Matrix Object
> inheriting from a Python C-array object.

And why would a Matrix need to inherit from a C-array? Wouldn't it
make more sense from an OO POV for the Matrix to *have* a C-array
without *being* one?

> I'm wondering if anyone else with more knowledge about new-style c-types
> could help here and show me the error of my thinking.

I'm trying...

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)



From tkorvola at welho.com  Wed Feb  9 22:38:08 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Wed Feb  9 22:38:08 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <20050210002435.1C9323C889@coffee.object-craft.com.au> (Andrew
 McNamara's message of "Thu, 10 Feb 2005 11:24:35 +1100")
References: <20050210002435.1C9323C889@coffee.object-craft.com.au>
Message-ID: <876510q5s1.fsf@welho.com>

Andrew McNamara <andrewm at object-craft.com.au> writes:
> I know this is a loaded question, but what does numarray offer that
> Numeric does not (as they stand today)? A cleaner implementation?

NA_OutputArray and NA_IoArray.  Very convenient for wrapping C
libraries.  NA_InputArray is more or less equivalent to
PyArray_ContiguousFromObject but as far as I know Numeric has nothing
equivalent to the other two.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From matthew.brett at gmail.com  Thu Feb 10 00:16:35 2005
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu Feb 10 00:16:35 2005
Subject: [Numpy-discussion] Numeric life as I see it
In-Reply-To: <420AB928.3090004@pfdubois.com>
References: <420A8406.4020808@ee.byu.edu>
	 <ca471dc205020914453f4da355@mail.gmail.com>
	 <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu>
	 <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com>
Message-ID: <1e2af89e0502100015438bbed6@mail.gmail.com>

> Aside: While I am at it, let me reiterate what I have said to the other
> developers privately: there is NO value to inheriting from the array
> class. Don't try to achieve that capability if it costs anything, even
> just effort, because it buys you nothing. Those of you who keep
> remarking on this as if it would simply haven't thought it through IMHO.
> It sounds so intellectually appealing that David Ascher and I had a
> version of Numeric that almost did it before we realized our folly.

Please forgive a question from someone who _definitely_ has not
thought it through, but would the fact that you cannot inherit from
the array class have any influence on whether it will be accepted in
the python core?

Best,

Matthew



From konrad.hinsen at laposte.net  Thu Feb 10 00:39:23 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Thu Feb 10 00:39:23 2005
Subject: [Numpy-discussion] Re: [Python-Dev] Re: Numeric life as I see it
In-Reply-To: <ca471dc205020920364ec6fa40@mail.gmail.com>
References: <420A8406.4020808@ee.byu.edu>  <ca471dc205020914453f4da355@mail.gmail.com>  <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu>  <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com>  <420ADE90.9050304@ee.byu.edu> <ca471dc205020920364ec6fa40@mail.gmail.com>
Message-ID: <a225db1f60892095a12adb7adc2178e6@laposte.net>

On 10.02.2005, at 05:36, Guido van Rossum wrote:

> And why would a Matrix need to inherit from a C-array? Wouldn't it
> make more sense from an OO POV for the Matrix to *have* a C-array
> without *being* one?

Definitely. Most array operations make no sense on matrices. And  
matrices are limited to two dimensions. Making Matrix a subclass of  
Array would be inheritance for implementation while removing 90% of the  
interface.

On the other hand, a Matrix object is perfectly defined by its  
behaviour and independent of its implementation. One could perfectly  
well implement one using Python lists or dictionaries, even though that  
would be pointless from a performance point of view.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Thu Feb 10 00:46:31 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Thu Feb 10 00:46:31 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <420ADE90.9050304@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu> <ca471dc205020914453f4da355@mail.gmail.com> <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com> <420ADE90.9050304@ee.byu.edu>
Message-ID: <1c3044466186480f55ef45d2c977731b@laposte.net>

On 10.02.2005, at 05:09, Travis Oliphant wrote:

> I'm not sure I agree.  The ufuncobject is the only place where this  
> concern existed (should we trip OverFlow, ZeroDivision, etc. errors  
> durring array math).   Numarray introduced and implemented the concept  
> of error modes that can be pushed and popped.  I believe this is the  
> right solution for the ufuncobject.

Indeed. Note also that the ufunc stuff is less critical to agree on  
than the array data structure. Anyone unhappy with ufuncs could write  
their own module and use it instead. It would be the data structure and  
its access rules that fix the structure of all the code that uses it,  
so that's what needs to be acceptable to everyone.

> One question we are pursuing is could the arrayobject get into the  
> core without a particular ufunc object.   Most see this as  
> sub-optimal, but maybe it is the only way.

Since all the artithmetic operations are in ufunc that would be  
suboptimal solution, but indeed still a workable one.

> I appreciate some of what Paul is saying here, but I'm not fully  
> convinced that this is still true with Python 2.2 and up new-style  
> c-types.   The concerns seem to be over the fact that you have to  
> re-implement everything in the sub-class because the base-class will  
> always return one of its objects instead of a sub-class object.

I'd say that such discussions should be postponed until someone  
proposes a good use for subclassing arrays. Matrices are not one, in my  
opinion.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From oliphant at ee.byu.edu  Thu Feb 10 01:27:00 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 10 01:27:00 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <876510q5s1.fsf@welho.com>
References: <20050210002435.1C9323C889@coffee.object-craft.com.au> <876510q5s1.fsf@welho.com>
Message-ID: <420B2898.8090808@ee.byu.edu>

Timo Korvola wrote:

>Andrew McNamara <andrewm at object-craft.com.au> writes:
>  
>
>>I know this is a loaded question, but what does numarray offer that
>>Numeric does not (as they stand today)? A cleaner implementation?
>>    
>>
>
>NA_OutputArray and NA_IoArray.  Very convenient for wrapping C
>libraries.  NA_InputArray is more or less equivalent to
>PyArray_ContiguousFromObject but as far as I know Numeric has nothing
>equivalent to the other two.
>  
>
I am not convinced these are really useful additional C-API calls.  I've 
done a fair bit of wrapping of C libraries and have never needed 
anything but

PyArray_ContiguousFromObject  and
PyArray_FromOb ject

This idea of "shadowing" ala NA_IOArray is a new one.  Can somebody show 
me an example of it's usefulness that could not be handled by simply a 
PyArray_ContiguousFromObject with an explicit copy back at the end?  I 
don't like this idea of "hiding" what is going on from the user.

This kind of C-API alteration which as far as I can tell has not been 
well justified is an example of some of the problems I see with Numarray.

Numeric3 condenses the Numeric API calls using numarray's idea of the 
requires flag:

PyArray_FromAny(PyObject *op, PyArray_Typecode *typecode, int mindepth, 
int maxdepth, int requires)

typecode is a simple structure with integer elements (type_num, 
itemsize, and fortran).

For some unexplained reason, the mindepth and maxdepth arguments are not 
copied by the new Numarray C-API.  I'm not sure why.  Having them  does 
make it easier to guarantee you are getting the right dimension for the 
array you want rather than having to check after the fact.

These kinds of discussions about numarray's C-API never took place.  In 
my opinion the Numeric C-API is better tested and easier to understand.

-Travis




From oliphant at ee.byu.edu  Thu Feb 10 01:31:21 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 10 01:31:21 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <1c3044466186480f55ef45d2c977731b@laposte.net>
References: <420A8406.4020808@ee.byu.edu> <ca471dc205020914453f4da355@mail.gmail.com> <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com> <420ADE90.9050304@ee.byu.edu> <1c3044466186480f55ef45d2c977731b@laposte.net>
Message-ID: <420B29AE.8030701@ee.byu.edu>

>> One question we are pursuing is could the arrayobject get into the  
>> core without a particular ufunc object.   Most see this as  
>> sub-optimal, but maybe it is the only way.
>
>
> Since all the artithmetic operations are in ufunc that would be  
> suboptimal solution, but indeed still a workable one.


I think replacing basic number operations of the arrayobject should 
simple, so perhaps a default ufunc object could be worked out for 
inclusion.

>
>> I appreciate some of what Paul is saying here, but I'm not fully  
>> convinced that this is still true with Python 2.2 and up new-style  
>> c-types.   The concerns seem to be over the fact that you have to  
>> re-implement everything in the sub-class because the base-class will  
>> always return one of its objects instead of a sub-class object.
>
>
> I'd say that such discussions should be postponed until someone  
> proposes a good use for subclassing arrays. Matrices are not one, in 
> my  opinion.
>
Agreed.  It is is not critical to what I am doing, and I obviously need 
more understanding before tackling such things.  Numeric3 uses the new 
c-type largely because of the nice getsets table which is separate from 
the methods table.  This replaces the rather ugly C-functions getattr 
and setattr.

-Travis





From verveer at embl.de  Thu Feb 10 01:54:06 2005
From: verveer at embl.de (Peter Verveer)
Date: Thu Feb 10 01:54:06 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <420B29AE.8030701@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu> <ca471dc205020914453f4da355@mail.gmail.com> <dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com> <420ADE90.9050304@ee.byu.edu> <1c3044466186480f55ef45d2c977731b@laposte.net> <420B29AE.8030701@ee.byu.edu>
Message-ID: <50ac60a36c2add7d708dc02d8bf623a3@embl.de>

On Feb 10, 2005, at 10:30 AM, Travis Oliphant wrote:

>
>>> One question we are pursuing is could the arrayobject get into the  
>>> core without a particular ufunc object.   Most see this as  
>>> sub-optimal, but maybe it is the only way.
>>
>>
>> Since all the artithmetic operations are in ufunc that would be  
>> suboptimal solution, but indeed still a workable one.
>
>
> I think replacing basic number operations of the arrayobject should 
> simple, so perhaps a default ufunc object could be worked out for 
> inclusion.

I agree, getting it in the core is among others, intended to give it 
broad access, not just to hard-core numeric people. For many uses 
(including many of my simpler scripts) you don't need the more exotic 
functionality of ufuncs. You could just do with implementing the 
standard math functions, possibly leaving out things like reduce. That 
would be very easy to implement.

>
>>
>>> I appreciate some of what Paul is saying here, but I'm not fully  
>>> convinced that this is still true with Python 2.2 and up new-style  
>>> c-types.   The concerns seem to be over the fact that you have to  
>>> re-implement everything in the sub-class because the base-class will 
>>>  always return one of its objects instead of a sub-class object.
>>
>>
>> I'd say that such discussions should be postponed until someone  
>> proposes a good use for subclassing arrays. Matrices are not one, in 
>> my  opinion.
>>
> Agreed.  It is is not critical to what I am doing, and I obviously 
> need more understanding before tackling such things.  Numeric3 uses 
> the new c-type largely because of the nice getsets table which is 
> separate from the methods table.  This replaces the rather ugly 
> C-functions getattr and setattr.

I would agree that sub-classing arrays might not be worth the trouble.

Peter




From verveer at embl-heidelberg.de  Thu Feb 10 02:56:53 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Thu Feb 10 02:56:53 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <420B2898.8090808@ee.byu.edu>
References: <20050210002435.1C9323C889@coffee.object-craft.com.au> <876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu>
Message-ID: <b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de>

It is however useful to distinguish between  input and output arrays. 
If your array is just an input, then the PyArray_FromAny function you 
mention below is good enough. If necessary they will make a 
well-behaved copy which can  be discarded afterwards.

However, if you need to modify an array 'in-place', then your 
well-behaved copy needs to be copied back at the end of your function. 
That can be done transparently as numarray does that it. You mention 
that you could explicitly copy back, but that means that I have to 
determine myself if there was a temporary made. And that could also 
happen if the array was byteswapped or misaligned not just if I asked 
for a contiguous array. The Numarray functions do all that for you.

The point is, that if you use an array just as an input, you don't want 
to copy things back, since that is a waste of resources. So you need 
somehow to distinguish between input and output arrays, which numarray 
does by having separate functions.

For me, the NA_OutputArray function has been very useful, I would like 
to see that functionality somehow in Numeric3. The NA_IoArray function 
I don't really need.

Maybe I have understood the Numeric API wrong and you can do this right 
now, I have not used it very much lately.

I know you avoid these problems if you don't modify arrays at all, but 
just return new arrays with your result. However, then you cannot 
properly implement output arguments. Many numarray and Numeric 
functions do _not_ allow you to pass an output array, to modify 
"in-place" as you can do with ufuncs. That annoys me endlessly, it is a 
big source of inefficiency if you have big arrays. A good example are 
the FFT functions that make lots of temporaries if you do a 
multi-dimensional transform. I am digressing now, but maybe output 
arrays could be supported more consistently in Numeric3?

Cheers, Peter

On 10 Feb 2005, at 10:25, Travis Oliphant wrote:

> Timo Korvola wrote:
>
>> Andrew McNamara <andrewm at object-craft.com.au> writes:
>>
>>> I know this is a loaded question, but what does numarray offer that
>>> Numeric does not (as they stand today)? A cleaner implementation?
>>>
>>
>> NA_OutputArray and NA_IoArray.  Very convenient for wrapping C
>> libraries.  NA_InputArray is more or less equivalent to
>> PyArray_ContiguousFromObject but as far as I know Numeric has nothing
>> equivalent to the other two.
>>
> I am not convinced these are really useful additional C-API calls.  
> I've done a fair bit of wrapping of C libraries and have never needed 
> anything but
>
> PyArray_ContiguousFromObject  and
> PyArray_FromOb ject
>
> This idea of "shadowing" ala NA_IOArray is a new one.  Can somebody 
> show me an example of it's usefulness that could not be handled by 
> simply a PyArray_ContiguousFromObject with an explicit copy back at 
> the end?  I don't like this idea of "hiding" what is going on from the 
> user.
>
> This kind of C-API alteration which as far as I can tell has not been 
> well justified is an example of some of the problems I see with 
> Numarray.
>
> Numeric3 condenses the Numeric API calls using numarray's idea of the 
> requires flag:
>
> PyArray_FromAny(PyObject *op, PyArray_Typecode *typecode, int 
> mindepth, int maxdepth, int requires)
>
> typecode is a simple structure with integer elements (type_num, 
> itemsize, and fortran).
>
> For some unexplained reason, the mindepth and maxdepth arguments are 
> not copied by the new Numarray C-API.  I'm not sure why.  Having them  
> does make it easier to guarantee you are getting the right dimension 
> for the array you want rather than having to check after the fact.
>
> These kinds of discussions about numarray's C-API never took place.  
> In my opinion the Numeric C-API is better tested and easier to 
> understand.
>
> -Travis
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real 
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion




From tkorvola at welho.com  Thu Feb 10 05:25:29 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Thu Feb 10 05:25:29 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de> (Peter
 Verveer's message of "Thu, 10 Feb 2005 11:55:36 +0100")
References: <20050210002435.1C9323C889@coffee.object-craft.com.au>
	<876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu>
	<b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de>
Message-ID: <87vf90czto.fsf@welho.com>

Peter Verveer <verveer at embl-heidelberg.de> writes:
> The NA_IoArray function I don't really need.

Depends on what C code you are working with.  Bidirectional array
arguments are not uncommon in libraries.

> Maybe I have understood the Numeric API wrong and you can do this
> right now, I have not used it very much lately.

Perhaps you can but you would have to write it all yourself.  In
Numarray that work has been done for you.  The three functions
NA_{Input,Output,Io}Array are pretty much all you need for accessing C
libraries with the usual (int size, double *data) interface, whichever
way the data goes.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From verveer at embl-heidelberg.de  Thu Feb 10 05:42:00 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Thu Feb 10 05:42:00 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <87vf90czto.fsf@welho.com>
References: <20050210002435.1C9323C889@coffee.object-craft.com.au> <876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu> <b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de> <87vf90czto.fsf@welho.com>
Message-ID: <512022542edd872397b959d4e6624f2c@embl-heidelberg.de>

On 10 Feb 2005, at 14:24, Timo Korvola wrote:

> Peter Verveer <verveer at embl-heidelberg.de> writes:
>> The NA_IoArray function I don't really need.
>
> Depends on what C code you are working with.  Bidirectional array
> arguments are not uncommon in libraries.

I agree, this is just a personal preference of me to rather keep the 
input and output arrays separate arguments, which gives you the same 
functionality plus a bit more flexibility. I would not argue to remove 
NA_IoArray though, if you prefer that.

>
>> Maybe I have understood the Numeric API wrong and you can do this
>> right now, I have not used it very much lately.
>
> Perhaps you can but you would have to write it all yourself.  In
> Numarray that work has been done for you.  The three functions
> NA_{Input,Output,Io}Array are pretty much all you need for accessing C
> libraries with the usual (int size, double *data) interface, whichever
> way the data goes.

Agreed. I don't know what the work would be to extend the Numeric API 
to at least have the OutputArray functionality (I guess IoArray would 
then be easily also added), but it seems worth having it.

Peter




From jeanluc.menut at free.fr  Thu Feb 10 06:21:03 2005
From: jeanluc.menut at free.fr (Jean-Luc Menut)
Date: Thu Feb 10 06:21:03 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
Message-ID: <420B6CB7.1050907@free.fr>

Hello,

I'm want to have the modulus and the phase of a fourier transform of 2D 
array, so I use the numarray FFT to compute the FT of the array (let's 
call it ff) and I use abs(ff) for the modulus and atan(ff.imag/ff.real) 
for the phase.

My problem comes when ff.real = 0, I have either inf or nan as result 
(which is perfectly good and useful). I looking for a function which 
replace atan(ff.imag/ff.real) by 0 when ff.real is 0.

I'm currently using a loop, and I think it's not very good (I call this 
function very often). Since the determination of the phase is something 
really useful and often used, I think there is probably a lot of people 
who had the same problem.

So what solutions do you use?

Thanks,

Jean-Luc Menut



From perry at stsci.edu  Thu Feb 10 06:40:51 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 10 06:40:51 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
In-Reply-To: <420B6CB7.1050907@free.fr>
Message-ID: <NEBBIJKBMLDBLNCEEFOCOEBLFJAA.perry@stsci.edu>

Jean-Luc Menut wrote:

> 
> I'm want to have the modulus and the phase of a fourier transform of 2D 
> array, so I use the numarray FFT to compute the FT of the array (let's 
> call it ff) and I use abs(ff) for the modulus and atan(ff.imag/ff.real) 
> for the phase.
> 
> My problem comes when ff.real = 0, I have either inf or nan as result 
> (which is perfectly good and useful). I looking for a function which 
> replace atan(ff.imag/ff.real) by 0 when ff.real is 0.
> 
> I'm currently using a loop, and I think it's not very good (I call this 
> function very often). Since the determination of the phase is something 
> really useful and often used, I think there is probably a lot of people 
> who had the same problem.
> 
> So what solutions do you use?
> 
Two basic approaches come to mind:

result =  atan(ff.imag/ff.real)
result[where(ff.real == 0)] = 0.

The following is often faster (but uses more memory):

result[ff.real==0] = 0.

The first method generates a list of locations where ff.real is 0 where
the second generates a mask array where true values will be set.

Another approach is to use the functions in the module ieeespecial
to specifically change ieee special values:

import numarray.ieeespecial as ieee
ieee.setnan(result, 0.)

I hope this answers your question.

Perry Greenfield





From ariciputi at pito.com  Thu Feb 10 06:42:53 2005
From: ariciputi at pito.com (Andrea Riciputi)
Date: Thu Feb 10 06:42:53 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
In-Reply-To: <420B6CB7.1050907@free.fr>
References: <420B6CB7.1050907@free.fr>
Message-ID: <61AFD7FA-7B71-11D9-9E63-000A95C0BC68@pito.com>

Why not arctan2(ff.imag, ff.real)?

Cheers,
  Andrea.

On 10 Feb 2005, at 15:16, Jean-Luc Menut wrote:

> Hello,
>
> I'm want to have the modulus and the phase of a fourier transform of 
> 2D array, so I use the numarray FFT to compute the FT of the array 
> (let's call it ff) and I use abs(ff) for the modulus and 
> atan(ff.imag/ff.real) for the phase.
>
> My problem comes when ff.real = 0, I have either inf or nan as result 
> (which is perfectly good and useful). I looking for a function which 
> replace atan(ff.imag/ff.real) by 0 when ff.real is 0.
>
> I'm currently using a loop, and I think it's not very good (I call 
> this function very often). Since the determination of the phase is 
> something really useful and often used, I think there is probably a 
> lot of people who had the same problem.
>
> So what solutions do you use?
>
> Thanks,
>
> Jean-Luc Menut
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real 
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>




From perry at stsci.edu  Thu Feb 10 07:17:09 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 10 07:17:09 2005
Subject: [Numpy-discussion] Numeric life as I see it
In-Reply-To: <420AB928.3090004@pfdubois.com>
Message-ID: <NEBBIJKBMLDBLNCEEFOCKEBNFJAA.perry@stsci.edu>

Paul Dubois wrote:

>
> Aside: While I am at it, let me reiterate what I have said to the other
> developers privately: there is NO value to inheriting from the array
> class. Don't try to achieve that capability if it costs anything, even
> just effort, because it buys you nothing. Those of you who keep
> remarking on this as if it would simply haven't thought it through IMHO.
> It sounds so intellectually appealing that David Ascher and I had a
> version of Numeric that almost did it before we realized our folly.
>
To be contrarian, we did find great benefit (at least initially) for
inheritance for developing the record array and character array classes
since they share so many structural operations (indexing, slicing,
transposes,
concatenation, etc.) with numeric arrays. It's possible that the approach
that Travis is considering doesn't need to use inheritance to accomplish
this (I don't know enough about the details yet), but it sure did save a
lot of duplication of implementation.

I do understand what you are getting at. Any numerical array inheritance
generally forces one to reimplement all ufuncs and such, and that does
make it less useful in that case (though I still wonder if it still isn't
better than the alternatives)

Perry Greenfield





From Alexandre.Fayolle at logilab.fr  Thu Feb 10 07:27:41 2005
From: Alexandre.Fayolle at logilab.fr (Alexandre)
Date: Thu Feb 10 07:27:41 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
In-Reply-To: <420B6CB7.1050907@free.fr>
References: <420B6CB7.1050907@free.fr>
Message-ID: <20050210152551.GA1989@crater.logilab.fr>

On Thu, Feb 10, 2005 at 03:16:23PM +0100, Jean-Luc Menut wrote:
> Hello,
> 
> I'm want to have the modulus and the phase of a fourier transform of 2D 
> array, so I use the numarray FFT to compute the FT of the array (let's 
> call it ff) and I use abs(ff) for the modulus and atan(ff.imag/ff.real) 
> for the phase.
> 
> My problem comes when ff.real = 0, I have either inf or nan as result 
> (which is perfectly good and useful). I looking for a function which 
> replace atan(ff.imag/ff.real) by 0 when ff.real is 0.
> 
> I'm currently using a loop, and I think it's not very good (I call this 
> function very often). Since the determination of the phase is something 
> really useful and often used, I think there is probably a lot of people 
> who had the same problem.
> 
> So what solutions do you use?

Maybe you could try:

from Numeric import *
def phase(ff):
    return (log(ff/abs(ff))/1j).real 

>>> a = array([1, 1j,])
>>> phase(a)
array([ 0.        ,  1.57079633])


-- 
Alexandre Fayolle                              LOGILAB, Paris (France).
http://www.logilab.com   http://www.logilab.fr  http://www.logilab.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050210/b339f54c/attachment-0001.sig>

From jmiller at stsci.edu  Thu Feb 10 08:11:41 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Thu Feb 10 08:11:41 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <512022542edd872397b959d4e6624f2c@embl-heidelberg.de>
References: <20050210002435.1C9323C889@coffee.object-craft.com.au>
	 <876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu>
	 <b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de>
	 <87vf90czto.fsf@welho.com>
	 <512022542edd872397b959d4e6624f2c@embl-heidelberg.de>
Message-ID: <1108051798.28054.176.camel@halloween.stsci.edu>

On Thu, 2005-02-10 at 08:41, Peter Verveer wrote:
> On 10 Feb 2005, at 14:24, Timo Korvola wrote:
> 
> > Peter Verveer <verveer at embl-heidelberg.de> writes:
> >> The NA_IoArray function I don't really need.
> >
> > Depends on what C code you are working with.  Bidirectional array
> > arguments are not uncommon in libraries.
> 
> I agree, this is just a personal preference of me to rather keep the 
> input and output arrays separate arguments, which gives you the same 
> functionality plus a bit more flexibility. I would not argue to remove 
> NA_IoArray though, if you prefer that.

I won't try to explain how the numarray C-API got the way it is because
it's irrelevant.  I agree that the native API is too large and started
encouraging use of the Numeric compatibility API over the summer.  On
the other hand,  I think the numarray problem domain is wider and more
difficult than the one addressed by Numeric.  

If Numeric3 intends to support byteswapped and misaligned data then
additional functionality is clearly required;  whether or not the
functionality is explicitly exposed in the API is another matter;  it is
hidden in our compatibility API so there's no reason Numeric3 can't hide
it too.  But it sounds to me like people have found the numarray high
level API fairly useful so the design features bear discussion even if
the API itself is dropped.

I used NA_IoArray myself in numarray's Numeric compatibility API and
also found it useful in adding f2py support.  I found it necessary to
get Numeric's extra packages (LinearAlgebra, FFT, RandomArray) to pass
their self-tests.   Don't overlook the fact that arrays are typically
passed by reference so whether we like it or not I/O behavior is
normally exposed.  For me,  even for a job as small as the compatibility
layer or f2py support,  NA_IoArray was an asset.  It's just easier to
think about and use and eliminates the chance of getting the shadowing
logic wrong.

The NumArray destructor-based array shadowing works well to keep things
transparent and porting changes low.  You obviously can do the explict
exit copy you've discussed,  but I think that's the kind of unnecessary
and intrusive change we're trying to get away from.  With the best code
coming from experts doing their own thing on their own time,  I don't
think support for numarray's advanced features is going to happen unless
it's as automatic and simple to use as possible.

Regards,
Todd




From verveer at embl-heidelberg.de  Thu Feb 10 08:37:03 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Thu Feb 10 08:37:03 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <1108051798.28054.176.camel@halloween.stsci.edu>
References: <20050210002435.1C9323C889@coffee.object-craft.com.au> <876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu> <b4003613e3b6b56a0adab7a28456d184@embl-heidelberg.de> <87vf90czto.fsf@welho.com> <512022542edd872397b959d4e6624f2c@embl-heidelberg.de> <1108051798.28054.176.camel@halloween.stsci.edu>
Message-ID: <56e881bc00532308eb3d5b024e6c1da1@embl-heidelberg.de>

> If Numeric3 intends to support byteswapped and misaligned data then
> additional functionality is clearly required;  whether or not the
> functionality is explicitly exposed in the API is another matter;  it 
> is
> hidden in our compatibility API so there's no reason Numeric3 can't 
> hide
> it too.  But it sounds to me like people have found the numarray high
> level API fairly useful so the design features bear discussion even if
> the API itself is dropped.

Exactly, I found the NA_*Array functions better than the original 
Numeric functions. It seems Travis is moving in a similar direction, 
but you are right that it is going to need extra functionality.

A question I have about the compatibility API in Numarray: In cases 
where a temporary is created, does that get copied back when the 
temporary is destroyed? That seems to be needed when you use an array 
as an output. That would be wasteful if you only use the array as an 
input. Its of course fine to do that, since it is a compability API 
that you cannot change, but it would point to a weakness in the 
original Numeric API, which should be fixed by the Numeric people if 
they want to  move forward.

It is relevant to me since I decided to move the nd_image to the 
Numeric compatibility API, and while doing that found myself wondering 
that in some cases I would be introducing a slight performance penalty.

> I used NA_IoArray myself in numarray's Numeric compatibility API and
> also found it useful in adding f2py support.  I found it necessary to
> get Numeric's extra packages (LinearAlgebra, FFT, RandomArray) to pass
> their self-tests.   Don't overlook the fact that arrays are typically
> passed by reference so whether we like it or not I/O behavior is
> normally exposed.  For me,  even for a job as small as the 
> compatibility
> layer or f2py support,  NA_IoArray was an asset.  It's just easier to
> think about and use and eliminates the chance of getting the shadowing
> logic wrong.

Yes, I can only agree with that.

> The NumArray destructor-based array shadowing works well to keep things
> transparent and porting changes low.  You obviously can do the explict
> exit copy you've discussed,  but I think that's the kind of unnecessary
> and intrusive change we're trying to get away from.  With the best code
> coming from experts doing their own thing on their own time,  I don't
> think support for numarray's advanced features is going to happen 
> unless
> it's as automatic and simple to use as possible.

I can confirm that the high-level API of Numarray works very nicely to 
hide you from things you do not want to know (i.e. byte-swapping) while 
at the same time allowing flexibility that does not seem to be 
completely present in the Numeric API, unless I don't understand the 
latter well enough.

Peter




From histed at MIT.EDU  Thu Feb 10 08:44:00 2005
From: histed at MIT.EDU (Mark Histed)
Date: Thu Feb 10 08:44:00 2005
Subject: [Numpy-discussion] logical priority
In-Reply-To: <420A526C.3030804@csun.edu>
References: <4207E421.10702@csun.edu> <0a221fd095d46402910135731d02ad05@laposte.net> <4207EBBF.2060808@csun.edu> <42084BB7.2060509@ee.byu.edu> <42084D42.9050509@colorado.edu> <42096070.5030608@csun.edu> <42096303.5030805@colorado.edu> <420A526C.3030804@csun.edu>
Message-ID: <20050210164145.GB13293@mit.edu>

> Anyway, we've wandered afar from the main point:  what is the best way 
> to duplicate the functionality of the following MATLAB snippet:
> 
> f = ~isnan(data1) & ~isnan(data2);
> plot(x,data1(f),x,data2(f))

I know this has been talked about before on this list, but only in
bits and pieces.  Can I summarize the way I understand the situation
and hopefully others can correct me?  I use MATLAB and Numeric often;
I have rarely used Numarray.

Summary:

For people moving from Matlab to Numeric/Numarray, the recommendation
is to use Numeric's "&" in place of Matlab's "&" to combine logical
arrays for indexing.  Different behavior results if the inputs have
entries > 1, and you have to put parenthesis around any inputs that
use relational operators.
Also, one should use Numeric's "and" in place of Matlab's "&&".

Details:

In MATLAB (R14SP1) there are three logical "and" operators:

"&": a binary operator which performs element-by-element logical AND.
    For its purposes, 0 is FALSE and every other numerical value is TRUE.
"&&": a binary operator which performs logical AND, but does
    short-circuiting, and only works with scalar elements (length-1 arrays)
"bitand()" : a function which takes two integers and does bitwise-and
    on their binary representations

http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_prog/ch12_n11.html#38948

I think that these map onto the python operators in the following way:
MATLAB      Python/Numeric
------      --------------
&&          and
&           ufunc logical_and()
bitand      &

Some inconsistencies in behavior (between Numeric and Matlab) occur
when one uses Numeric's "&" like MATLAB "&".  I can think of:
    * Non-logical {0,1} inputs: Numeric's output is an integer, the
      bitwise and of the inputs, Matlab's output is a logical array,
      either 1 or 0. 
    * Precedence: Numeric's "&" operator is higher prec than its relational
      operators, MATLAB is the reverse

And an inconsistency between MATLAB's "&&" and Numeric's "and":
    * MATLAB returns an error when the arguments to && are not logical
      scalars, Numeric returns the second argument as the expression's
      value. (but Numarray behaves like MATLAB)



Does that seem right?  

Regards, 
Mark

    
  
----------------------------------------------------------------

Matlab example:

>> a = [0,1,2];
>> b = [1,1,63];
>> a & b
ans =
     0     1     1
>> a && b
??? Operands to the || and && operators must be convertible to logical 
    scalar values.
>> bitand(a,b)
ans =
     0     1     2
>> a > 1 & b > 1
ans =
     0     0     1
>> (a > 1) & (b > 1)
ans =
     0     0     1

----------------------------------------------------------------

Numeric example:

Python 2.3.4 (#2, Sep 24 2004, 08:39:09)
[GCC 3.3.4 (Debian 1:3.3.4-12)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Numeric as N
>>> a = N.array([0,1,2]);
>>> b = N.array([1,1,63]);
>>> N.logical_and(a,b)
array([0, 1, 1])
>>> a & b
array([0, 1, 2])
>>> a and b
array([ 1,  1, 63])
>>> a > 1 & b > 1
array([0, 0, 0])
>>> (a > 1) & (b > 1)
array([0, 0, 1])

----------------------------------------------------------------

Numarray example:
>>> import numarray as na
>>> a = na.array([0,1,2])
>>> b = na.array([1,1,63])
>>> na.logical_and(a,b)
array([0, 1, 1], type=Bool)
>>> a and b
RuntimeError: An array doesn't make sense as a truth value.  Use
    sometrue(a) or alltrue(a). 
>>> a & b
array([0, 1, 2])
>>>

----------------------------------------------------------------

On Wed, Feb 09, 2005 at 10:11:56AM -0800, Stephen Walton wrote:
> Fernando Perez wrote a good deal clarifying how Python's logical 
> operators work, for which I'm grateful.  One brief comment:
> 
> >Bytecode analysis can be really useful to understand this kind of 
> >subtle issues.  I can't think of any language where this approach is 
> >as easy to do and friendly as in python.
> 
> Pardon me, but this sounds a bit like looking at the assembly language 
> Fortran generates in order to understand the subtleties of that language 
> :-) .  I know people who used to do that, and were proud of it, but I'd 
> rather the language itself didn't behave in surprising ways.  At least 
> in Fortran, 'a .and. b' only made sense if a and b were both of type 
> LOGICAL;  in fact, it was a compile time error if they weren't.
> 

> 
> ?
> 
> 
> 
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion



From tkorvola at welho.com  Thu Feb 10 09:16:01 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Thu Feb 10 09:16:01 2005
Subject: [Numpy-discussion] Numeric vs numarray?
In-Reply-To: <420B2898.8090808@ee.byu.edu> (Travis Oliphant's message of
 "Thu, 10 Feb 2005 02:25:44 -0700")
References: <20050210002435.1C9323C889@coffee.object-craft.com.au>
	<876510q5s1.fsf@welho.com> <420B2898.8090808@ee.byu.edu>
Message-ID: <87ll9ws5ey.fsf@welho.com>

Travis Oliphant <oliphant at ee.byu.edu> writes:
> Can somebody show me an example of it's usefulness that could not be
> handled by simply a PyArray_ContiguousFromObject with an explicit
> copy back at the end?

It is not that it could not be done, it is the convenience of having
the library do it for you.  Is there a function in the Numeric C API
that one could use for copying the data back?  Preferably one that
would check that the source and the target are different objects
before copying.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From Chris.Barker at noaa.gov  Thu Feb 10 11:08:03 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Thu Feb 10 11:08:03 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <420ADE90.9050304@ee.byu.edu>
References: <420A8406.4020808@ee.byu.edu>		<ca471dc205020914453f4da355@mail.gmail.com>	<dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com> <420ADE90.9050304@ee.byu.edu>
Message-ID: <420BB0D8.2000101@noaa.gov>


Travis Oliphant wrote:
> I personally don't see 
> Numeric3 as in any kind of "need for speed" camp either. 

Now I'm confused. As an interested observer, It seems to me that there 
is only one thing that is keeping users of Numeric from switching to 
numarray:

"Performance with many small arrays"

Which seems to boil down to:

"Time to create a new array"

There are other small details, but from following the discussion, that 
seems to be the only show stopper. I understood that the goal of 
Numeric3 was to exactly solve that problem, while still getting all the 
nifty benefits of numarray.

Another nit: "speed" in not one goal. I understand that one of the goals 
of numarray was top performance for very large arrays. In fact, it's 
ironic that someone discovered that it was a safety check in Numeric 
that was causing it to be slower than Numarray with large arrays.

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From oliphant at ee.byu.edu  Thu Feb 10 11:22:30 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 10 11:22:30 2005
Subject: [Numpy-discussion] Re: Numeric life as I see it
In-Reply-To: <420BB0D8.2000101@noaa.gov>
References: <420A8406.4020808@ee.byu.edu>		<ca471dc205020914453f4da355@mail.gmail.com>	<dd28fc2f050209161264f9b601@mail.gmail.com> <420AAC33.807@ee.byu.edu> <420AB084.1000008@v.loewis.de> <420AB928.3090004@pfdubois.com> <420ADE90.9050304@ee.byu.edu> <420BB0D8.2000101@noaa.gov>
Message-ID: <420BB448.3000900@ee.byu.edu>

Chris Barker wrote:

>
>
> Travis Oliphant wrote:
>
>> I personally don't see Numeric3 as in any kind of "need for speed" 
>> camp either. 
>
>
> Now I'm confused. As an interested observer, It seems to me that there 
> is only one thing that is keeping users of Numeric from switching to 
> numarray:
>
> "Performance with many small arrays"
>
That was the showstopper, but it was just a symptom of the Python-C 
mixture that I would never feel comfortable using as the foundation for 
the basic arrayobject.  There are also lots of little concerns over the 
burgeoning C-API that would make maintainence more difficult.  To me it 
is ironic that people have called Numeric "hackish".  In my view (and 
believe me I mean no disrespect, I know these are challenging problems) 
in comparison to the mix of Python and C code in numarray, Numeric looks 
quite elegant.  Numarray has some nice code in it, don't get me wrong, 
it's just the overall architecture and it's choice of C-API I have 
issues with.

>
> Another nit: "speed" in not one goal. I understand that one of the 
> goals of numarray was top performance for very large arrays. In fact, 
> it's ironic that someone discovered that it was a safety check in 
> Numeric that was causing it to be slower than Numarray with large arrays.

That was known for a long time, which is why scipy disabled the safety 
check.  I think numarray has a better idea for dealing with checks.  One 
that can easily be borrowed in Numeric3.

-Travis





From perry at stsci.edu  Thu Feb 10 11:29:27 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 10 11:29:27 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <f064f705d441f70ca8ff56e4e3859ac4@stsci.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCCECCFJAA.perry@stsci.edu>

I don't recall if I've made any mention of our (STScI's)
stance on Numeric3 publicly. From what Travis describes,
his goals do appear to encompass all the important goals
we had for numarray (of course, we need to keep on top of
the details to make sure we haven't overlooked anything).
Should he be able to accomplish this and achieve the
performance goals for small arrays, it would be silly for
us not to take advantage of such a unification.

If the long-term future of numarray is that it served to
identify various improvements to the behavior of numeric arrays
for Numeric3, particularly by providing an illustration of
its practicality and usefulness as well as reusable code,
we think that is fine.  We are not wedded to our particular
implementation of these capabilities, nor even to all details
of the interface. We do have a strong need for the ability to
handle misaligned and byte-swapped arrays (which are crucial
for handling memory-mapped arrays and heterogeneous record arrays.)

In our view the issue isn't who owns the array package but
getting one that satisfies the needs of the great majority.
We are not sure how feasible the approach Travis is using is 
so we are taking a wait-and-see position on it. We hope he
is successful in achieving these goals.

On the other hand, that point isn't here yet, so we'll
continue doing what we are doing with numarray and making
it work with scipy. If and when the time comes when Numeric3
looks like a viable alternative to numarray (or is clearly
close enough to illustrate that it will be viable), then we
would begin the effort to substitute Numeric3 for numarray.
And if it doesn't work out, it is in our plan to look at
small array performance improvements in numarray.

We intend to stay engaged in design and interface discussions
with Travis, and encourage any others that have a stake in
numarray capabilities to do so as well.
 
Perry







From oliphant at ee.byu.edu  Thu Feb 10 12:28:23 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 10 12:28:23 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCCECCFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCCECCFJAA.perry@stsci.edu>
Message-ID: <420BC3B5.6030604@ee.byu.edu>

Perry Greenfield wrote

>In our view the issue isn't who owns the array package but
>getting one that satisfies the needs of the great majority.
>We are not sure how feasible the approach Travis is using is 
>so we are taking a wait-and-see position on it. We hope he
>is successful in achieving these goals.
>  
>
I've known it was a risk from the beginning, and I don't expect people 
to believe me until I can "show them the code." 

>We intend to stay engaged in design and interface discussions
>with Travis, and encourage any others that have a stake in
>numarray capabilities to do so as well.
>  
>

Thanks for the email, Perry.

Toward that end, I would really appreciate knowing from anybody who 
understands, which C-API calls are seen as most relevant and new.

For example, after the discussion on this list, I can understand that 
having the ability to "copy-back to a mis-behaved object" can be useful, 
and it has pointed out holes in the Numeric API (holes which we all know 
are there).   I still don't know about having three different API calls 
to do it (instead of just a flag in the requires argument).

Over the next few weeks, I will be putting together a PEP-like 
document.  This PEP will essentially discuss the behavior of the 
arrayobject that we would like to see in the Python core.    I will need 
lots of feedback and help with it, so I will post it to the Numeric page 
as it develops.    There are a couple of unresolved issues that need 
comments.  Hopefully, the PEP will help sort those out.   I think 
though, there are only a couple of issues.  And I'm confident these can 
be worked out satisfactorily.   I have no plans to submit the PEP until 
it has received attention from everybody interested.

Best regards to all,

-Travis Oliphant






From verveer at embl.de  Thu Feb 10 13:09:38 2005
From: verveer at embl.de (Peter Verveer)
Date: Thu Feb 10 13:09:38 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420BC3B5.6030604@ee.byu.edu>
References: <NEBBIJKBMLDBLNCEEFOCCECCFJAA.perry@stsci.edu> <420BC3B5.6030604@ee.byu.edu>
Message-ID: <45892a12a8669ffeccbceaef1974dbd8@embl.de>

> Toward that end, I would really appreciate knowing from anybody who 
> understands, which C-API calls are seen as most relevant and new.

The distinction between input and output (and possible I/O) arrays is 
extremely useful.

> For example, after the discussion on this list, I can understand that 
> having the ability to "copy-back to a mis-behaved object" can be 
> useful, and it has pointed out holes in the Numeric API (holes which 
> we all know are there).

It allows  you easily and effiencly implement output arguments, which I 
think is essential.

>   I still don't know about having three different API calls to do it 
> (instead of just a flag in the requires argument).

How it is finally implemented is not really important, a flag would be 
fine.

> Over the next few weeks, I will be putting together a PEP-like 
> document.  This PEP will essentially discuss the behavior of the 
> arrayobject that we would like to see in the Python core.    I will 
> need lots of feedback and help with it, so I will post it to the 
> Numeric page as it develops.    There are a couple of unresolved 
> issues that need comments.  Hopefully, the PEP will help sort those 
> out.   I think though, there are only a couple of issues.  And I'm 
> confident these can be worked out satisfactorily.   I have no plans to 
> submit the PEP until it has received attention from everybody 
> interested.

Excellent!




From cjw at sympatico.ca  Thu Feb 10 13:46:45 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 10 13:46:45 2005
Subject: [Numpy-discussion] Numeric3
In-Reply-To: <420BC3B5.6030604@ee.byu.edu>
References: <NEBBIJKBMLDBLNCEEFOCCECCFJAA.perry@stsci.edu> <420BC3B5.6030604@ee.byu.edu>
Message-ID: <420BD5EA.4030503@sympatico.ca>

Travis Oliphant wrote:

> Perry Greenfield wrote
>
>> In our view the issue isn't who owns the array package but
>> getting one that satisfies the needs of the great majority.
>> We are not sure how feasible the approach Travis is using is so we 
>> are taking a wait-and-see position on it. We hope he
>> is successful in achieving these goals.
>>  
>>
> I've known it was a risk from the beginning, and I don't expect people 
> to believe me until I can "show them the code."
>
>> We intend to stay engaged in design and interface discussions
>> with Travis, and encourage any others that have a stake in
>> numarray capabilities to do so as well.
>>  
>>
>
> Thanks for the email, Perry.
>
> Toward that end, I would really appreciate knowing from anybody who 
> understands, which C-API calls are seen as most relevant and new.
>
> For example, after the discussion on this list, I can understand that 
> having the ability to "copy-back to a mis-behaved object" can be 
> useful, and it has pointed out holes in the Numeric API (holes which 
> we all know are there).   I still don't know about having three 
> different API calls to do it (instead of just a flag in the requires 
> argument).
>
> Over the next few weeks, I will be putting together a PEP-like 
> document.  This PEP will essentially discuss the behavior of the 
> arrayobject that we would like to see in the Python core.    I will 
> need lots of feedback and help with it, so I will post it to the 
> Numeric page as it develops.    There are a couple of unresolved 
> issues that need comments.  Hopefully, the PEP will help sort those 
> out.   I think though, there are only a couple of issues.  And I'm 
> confident these can be worked out satisfactorily.   I have no plans to 
> submit the PEP until it has received attention from everybody interested.
>
> Best regards to all,
>
> -Travis Oliphant

I am glad to see that the suggestion from Martin Loewis regarding the 
PEP approach is being taken up and hope that an early draft will be 
posted to c.l.p,

Colin W.



From bsder at mail.allcaps.org  Thu Feb 10 15:04:12 2005
From: bsder at mail.allcaps.org (Andrew P. Lentvorski, Jr.)
Date: Thu Feb 10 15:04:12 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
In-Reply-To: <420B6CB7.1050907@free.fr>
References: <420B6CB7.1050907@free.fr>
Message-ID: <8a900f8da1886681f7e65d418c4207b5@mail.allcaps.org>

On Feb 10, 2005, at 6:16 AM, Jean-Luc Menut wrote:

> My problem comes when ff.real = 0, I have either inf or nan as result 
> (which is perfectly good and useful). I looking for a function which 
> replace atan(ff.imag/ff.real) by 0 when ff.real is 0.

I would probably use atan2.

However, do you really want to replace atan(ff.imag/ff.real) by 0 when 
ff.real is 0?

I would expect that you want to replace it with +- pi/2 depending upon 
the sign of ff.imag.

-a




From rkern at ucsd.edu  Thu Feb 10 15:53:01 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Thu Feb 10 15:53:01 2005
Subject: [Numpy-discussion] calcul of phase (dividing by 0)
In-Reply-To: <61AFD7FA-7B71-11D9-9E63-000A95C0BC68@pito.com>
References: <420B6CB7.1050907@free.fr> <61AFD7FA-7B71-11D9-9E63-000A95C0BC68@pito.com>
Message-ID: <420BF2F8.6070104@ucsd.edu>

Andrea Riciputi wrote:
> Why not arctan2(ff.imag, ff.real)?

Yes, for the love of all that is holy, please use arctan2()!

For one thing, arctan2() solves other problems besides the divide-by-zero.

In [1]: from numarray import *

In [2]: arctan(-1/-1) == arctan(1/1)
Out[2]: True

In [3]: arctan2(-1,-1) == arctan2(1,1)
Out[3]: False

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From rbastian at club-internet.fr  Fri Feb 11 04:50:24 2005
From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=)
Date: Fri Feb 11 04:50:24 2005
Subject: [Numpy-discussion] Numeric3
Message-ID: <05021105034803.00760@rbastian>

Hello,

>From a user point of view,
in the types module of Python
I wish someting like :

from types import *
....

if type(a)==NumarrayType :
   ...



-- 
Ren? Bastian
http://www.musiques-rb.org : Musique en Python 



From jmiller at stsci.edu  Wed Feb 16 11:12:11 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Feb 16 11:12:11 2005
Subject: [Numpy-discussion] It's so *quiet* I'm testing numpy-discussion
Message-ID: <1108581069.27126.92.camel@halloween.stsci.edu>

If this message actually gets posted,  well, sorry.  It's been so quiet
on the list I thought I'd better check it.

Todd




From jdhunter at ace.bsd.uchicago.edu  Wed Feb 16 11:18:07 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Wed Feb 16 11:18:07 2005
Subject: [Numpy-discussion] It's so *quiet* I'm testing numpy-discussion
In-Reply-To: <1108581069.27126.92.camel@halloween.stsci.edu> (Todd Miller's
 message of "Wed, 16 Feb 2005 14:11:09 -0500")
References: <1108581069.27126.92.camel@halloween.stsci.edu>
Message-ID: <m3sm3wgw99.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Todd" == Todd Miller <jmiller at stsci.edu> writes:

    Todd> If this message actually gets posted, well, sorry.  It's
    Todd> been so quiet on the list I thought I'd better check it.

In neurons, after maximal excitation in the form of an action
potential or Numeric3 announcement, a period of refractory inhibition
follows.  Be alert, though, for post-inhibitory rebound....

JDH



From oliphant at ee.byu.edu  Wed Feb 16 12:44:16 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb 16 12:44:16 2005
Subject: [Numpy-discussion] It's so *quiet* I'm testing numpy-discussion
In-Reply-To: <1108581069.27126.92.camel@halloween.stsci.edu>
References: <1108581069.27126.92.camel@halloween.stsci.edu>
Message-ID: <4213B049.3040709@ee.byu.edu>

I'm busy finishing Numeric3.  It has made quite a bit of progress.  It 
can be checked out of CVS and played with.   Implementing the 
multidimensional indexing magic of numarray is taking a bit of time, 
though.  I'd like to support mixed integer indexing, boolean indexing, 
and slicing too.    Any help on this would be greatly appreciated.

I'm very confident that record arrays are going to work.   The new 
numeric object has a getfield method that lets you interpret subitem 
slices as a new type of array.  The new void * type arrays let the array 
be a collection of anything.  Constructing the fanciness of numarray 
record arrays will just be a matter of an appropriate Python container 
class or subclass.  I think all the requirements to do this are already 
done in Numeric3.

I've added a possible UPDATEIFCOPY flag to the requirements argument in 
PyArray_FromAny that will tell an array to copy back to the original 
upon deletion if this flag is set.

Typecodes are handled by a TypecodeConverter function that takes quite a 
few different types of arguments including an arbitrary Python object 
with a type_num and itemsize attribute.  This allows a lot of 
flexibility in how types are specified.

Multi-byte character and unicode arrays already seem to work reasonably 
well.  There is a lot more testing that needs to be done, of course.

I would also love feedback and/or help on the PEP that I've started.  It 
is online at numeric.scipy.org but that copy may be a few days behind 
the CVS version at sourceforge.

-Travis






From southey at uiuc.edu  Wed Feb 16 12:59:17 2005
From: southey at uiuc.edu (Bruce Southey)
Date: Wed Feb 16 12:59:17 2005
Subject: [Numpy-discussion] random and RandomArray
Message-ID: <24e418bb.cfb24d1c.81ae300@expms6.cites.uiuc.edu>

Hi, 
I was browsing through some of the code and realized that certain random number 
generators occur in both the Python random module and RandomArray.  
 
The random module uses C code to get a scalar uniform random number that is 
modified in Python by other generators to get random numbers from other 
distributions. In RandomArray everything is done in the C code - obviously this 
is way faster especially for arrays. 
 
In the long term, would it make sense to get use the same random number 
generators in both random and RandomArray? 
By this, I am thinking along the lines that both random and RandomArray would 
call the same C code. This would require more changes in random than 
RandomArray but would minimise duplicate code and update RandomArray. 
 
So I was wondering what the thoughts of people on this list are on this before 
even considering this further. 
 
Regards 
Bruce Southey 



From haase at msg.ucsf.edu  Wed Feb 16 13:45:15 2005
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed Feb 16 13:45:15 2005
Subject: [Numpy-discussion] It's so *quiet* I'm testing numpy-discussion
In-Reply-To: <4213B049.3040709@ee.byu.edu>
References: <1108581069.27126.92.camel@halloween.stsci.edu> <4213B049.3040709@ee.byu.edu>
Message-ID: <200502161344.43073.haase@msg.ucsf.edu>

On Wednesday 16 February 2005 12:42 pm, Travis Oliphant wrote:
> I'm busy finishing Numeric3.  It has made quite a bit of progress.  It
> can be checked out of CVS and played with.   Implementing the
> multidimensional indexing magic of numarray is taking a bit of time,
> though.  I'd like to support mixed integer indexing, boolean indexing,
> and slicing too.    Any help on this would be greatly appreciated.
>
> I'm very confident that record arrays are going to work.   The new
> numeric object has a getfield method that lets you interpret subitem
> slices as a new type of array.  The new void * type arrays let the array
> be a collection of anything.  Constructing the fanciness of numarray
> record arrays will just be a matter of an appropriate Python container
> class or subclass.  I think all the requirements to do this are already
> done in Numeric3.
>
Hi Travis,
I'm a committed numarray user since numarray 0.3.6 (I was told not to look 
into Numeric, because it did not have UInt16 which I needed)
Now, What you say about record arrays makes it sound like you are trying to 
"catch up" on numarray by providing these same features in Numeric.
Will you also look into memmap ?  For us this turned into one of the most 
amazing features of numarray: that we can access 500MB files instantaneously.
How about the speed advantage that numarray was having over Numeric for large 
arrays ?

I just wanted to show my interest in this discussion and thank everbody 
(numarray, scipy, Numeric, ...) for the great tools !

Sebastian Haase





> I've added a possible UPDATEIFCOPY flag to the requirements argument in
> PyArray_FromAny that will tell an array to copy back to the original
> upon deletion if this flag is set.
>
> Typecodes are handled by a TypecodeConverter function that takes quite a
> few different types of arguments including an arbitrary Python object
> with a type_num and itemsize attribute.  This allows a lot of
> flexibility in how types are specified.
>
> Multi-byte character and unicode arrays already seem to work reasonably
> well.  There is a lot more testing that needs to be done, of course.
>
> I would also love feedback and/or help on the PEP that I've started.  It
> is online at numeric.scipy.org but that copy may be a few days behind
> the CVS version at sourceforge.
>
> -Travis



From mdehoon at ims.u-tokyo.ac.jp  Wed Feb 16 18:44:14 2005
From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon)
Date: Wed Feb 16 18:44:14 2005
Subject: [Numpy-discussion] random and RandomArray
In-Reply-To: <24e418bb.cfb24d1c.81ae300@expms6.cites.uiuc.edu>
References: <24e418bb.cfb24d1c.81ae300@expms6.cites.uiuc.edu>
Message-ID: <421404CF.7080707@ims.u-tokyo.ac.jp>

For what it's worth, I think that this is a good idea. I'd rather have 
one really good random number package than two separate ones.

--Michiel.

Bruce Southey wrote:
> Hi, 
> I was browsing through some of the code and realized that certain random number 
> generators occur in both the Python random module and RandomArray.  
>  
> The random module uses C code to get a scalar uniform random number that is 
> modified in Python by other generators to get random numbers from other 
> distributions. In RandomArray everything is done in the C code - obviously this 
> is way faster especially for arrays. 
>  
> In the long term, would it make sense to get use the same random number 
> generators in both random and RandomArray? 
> By this, I am thinking along the lines that both random and RandomArray would 
> call the same C code. This would require more changes in random than 
> RandomArray but would minimise duplicate code and update RandomArray. 
>  
> So I was wondering what the thoughts of people on this list are on this before 
> even considering this further. 
>  
> Regards 
> Bruce Southey 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 
> 




From rkern at ucsd.edu  Wed Feb 16 20:06:17 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Wed Feb 16 20:06:17 2005
Subject: [Numpy-discussion] random and RandomArray
In-Reply-To: <24e418bb.cfb24d1c.81ae300@expms6.cites.uiuc.edu>
References: <24e418bb.cfb24d1c.81ae300@expms6.cites.uiuc.edu>
Message-ID: <421417F5.3010703@ucsd.edu>

Bruce Southey wrote:
> Hi, 
> I was browsing through some of the code and realized that certain random number 
> generators occur in both the Python random module and RandomArray.  
>  
> The random module uses C code to get a scalar uniform random number that is 
> modified in Python by other generators to get random numbers from other 
> distributions. In RandomArray everything is done in the C code - obviously this 
> is way faster especially for arrays. 
>  
> In the long term, would it make sense to get use the same random number 
> generators in both random and RandomArray? 

I think that it is certainly feasible and desirable that when/if a 
multiarray object enters the standard library that the standard random 
module be extended to produce arrays as well.

I don't think it's terribly worthwhile to hack the random module to 
expose its PRNG so that we use it's implementation without duplicating 
code. I think the result will be quite fragile, and won't be useful 
until Python 2.5.

I do think it would be extremely worthwhile to implement the Mersenne 
Twister for numarray/scipy. I promised some time ago to look into this, 
but I have not gotten around to it, unfortunately.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From oliphant at ee.byu.edu  Wed Feb 16 23:26:15 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Wed Feb 16 23:26:15 2005
Subject: [Numpy-discussion] Multiarray PEP
Message-ID: <421446E3.7080202@ee.byu.edu>

I am looking for feedback on the PEP.   Of particular interest is the 
specification of multidimensional indexing that I've outlined.  I think 
it is mostly the same as numarray (I'd love some feedback to be sure 
about that), except for the fact that X[K] where K is a a single integer 
index array can do 1-d indexing similar to MATLAB.  The equivalent 
numarray indexing is available as X[K,].    Now that I've specified what 
is to happen, I think it won't be very difficult to code. 

I would also like some help on resolving the type issue.   Is it 
important to have hierarchial classes defined (probably in Python) that 
can be used to check and/or specify type or are functions that check for 
various properties, fine.   Right now x.type is specified as an 
attribute but the attribute could be replaced by a method.  What to 
return is the biggest question.

I've never had a problem with just the typecode characters although 
aliases (like Int16 or Short) which might be more meaningful are nice.   
Are shadow-classes important in this case?  To me they look a little 
like extra bulk with no gain.  I'd love to be educated as to why it is 
worth the extra effort to have a bunch of instances lying around.  I 
don't feel very strongly about this though, so if somebody else does, 
I'm willing to go for it.

I've designed Numeric3 so that it is not difficult to change how 
typecodes are specified on the Python level.   We can really accommodate 
whatever.

<Opening can ....>

What is the opinion of everybody?

-Travis




From verveer at embl-heidelberg.de  Thu Feb 17 01:27:17 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Thu Feb 17 01:27:17 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <421446E3.7080202@ee.byu.edu>
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <d8c7c031fc63e548680c24b1b66ea6f3@embl-heidelberg.de>

> I would also like some help on resolving the type issue.   Is it 
> important to have hierarchial classes defined (probably in Python) 
> that can be used to check and/or specify type or are functions that 
> check for various properties, fine.   Right now x.type is specified as 
> an attribute but the attribute could be replaced by a method.  What to 
> return is the biggest question.
>
> I've never had a problem with just the typecode characters although 
> aliases (like Int16 or Short) which might be more meaningful are nice. 
>   Are shadow-classes important in this case?  To me they look a little 
> like extra bulk with no gain.  I'd love to be educated as to why it is 
> worth the extra effort to have a bunch of instances lying around.  I 
> don't feel very strongly about this though, so if somebody else does, 
> I'm willing to go for it.
>
> I've designed Numeric3 so that it is not difficult to change how 
> typecodes are specified on the Python level.   We can really 
> accommodate whatever.

I am used to the numarray convention of using Int16 and so on, which to 
me are more meaningful and far more easy to remember then typecode 
characters. These could be aliases for character codes, which for most 
purposes would be fine for me. The only practical problem that comes to 
mind now is that if you print a type, you would get its character code, 
which I would find inconvenient as I generally don't use them. By using 
a python class you can use any string representation you like. But I 
suppose you could just provide a function/method to return a string 
representation for an alias to solve that.

Peter




From magnus at hetland.org  Thu Feb 17 01:35:20 2005
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Thu Feb 17 01:35:20 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <421446E3.7080202@ee.byu.edu>
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <20050217093428.GA7324@idi.ntnu.no>

Travis Oliphant <oliphant at ee.byu.edu>:
>
> 
> I am looking for feedback on the PEP.   Of particular interest is the 
> specification of multidimensional indexing that I've outlined.  I think 
> it is mostly the same as numarray (I'd love some feedback to be sure 
> about that), except for the fact that X[K] where K is a a single integer 
> index array can do 1-d indexing similar to MATLAB.  The equivalent 
> numarray indexing is available as X[K,].    Now that I've specified what 
> is to happen, I think it won't be very difficult to code. 

Sorry if I'm a bit quick to respond here (haven't read the PEP yet; I
seem to have lost the URL) but ... is this related to the following
issue (which I struggled with for a while back in 2002)?

  http://sourceforge.net/mailarchive/message.php?msg_id=2700193

IMO, special-casing (if that's what you're proposing) is a Bad
Think(tm), as general code will then have to add special-casing to
avoid tripping over it... (But I may just be confused here :)

- M

-- 
Magnus Lie Hetland       Fallen flower I see / Returning to its branch
http://hetland.org       Ah! a butterfly.           [Arakida Moritake]



From magnus at hetland.org  Thu Feb 17 01:37:15 2005
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Thu Feb 17 01:37:15 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <20050217093428.GA7324@idi.ntnu.no>
References: <421446E3.7080202@ee.byu.edu> <20050217093428.GA7324@idi.ntnu.no>
Message-ID: <20050217093641.GB7324@idi.ntnu.no>

Magnus Lie Hetland <magnus at hetland.org>:
>
> Bad Think(tm)

... or Thing, as the case may be ;)

-- 
Magnus Lie Hetland       Fallen flower I see / Returning to its branch
http://hetland.org       Ah! a butterfly.           [Arakida Moritake]



From cjw at sympatico.ca  Thu Feb 17 05:39:26 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 17 05:39:26 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <d8c7c031fc63e548680c24b1b66ea6f3@embl-heidelberg.de>
References: <421446E3.7080202@ee.byu.edu> <d8c7c031fc63e548680c24b1b66ea6f3@embl-heidelberg.de>
Message-ID: <42149E40.9000800@sympatico.ca>

Peter Verveer wrote:

>> I would also like some help on resolving the type issue.   Is it 
>> important to have hierarchial classes defined (probably in Python) 
>> that can be used to check and/or specify type or are functions that 
>> check for various properties, fine.   Right now x.type is specified 
>> as an attribute but the attribute could be replaced by a method.  
>> What to return is the biggest question.
>>
>> I've never had a problem with just the typecode characters although 
>> aliases (like Int16 or Short) which might be more meaningful are 
>> nice.   Are shadow-classes important in this case?  To me they look a 
>> little like extra bulk with no gain.  I'd love to be educated as to 
>> why it is worth the extra effort to have a bunch of instances lying 
>> around.  I don't feel very strongly about this though, so if somebody 
>> else does, I'm willing to go for it.
>>
>> I've designed Numeric3 so that it is not difficult to change how 
>> typecodes are specified on the Python level.   We can really 
>> accommodate whatever.
>
>
> I am used to the numarray convention of using Int16 and so on, which 
> to me are more meaningful and far more easy to remember then typecode 
> characters. These could be aliases for character codes, which for most 
> purposes would be fine for me. The only practical problem that comes 
> to mind now is that if you print a type, you would get its character 
> code, which I would find inconvenient as I generally don't use them. 
> By using a python class you can use any string representation you 
> like. But I suppose you could just provide a function/method to return 
> a string representation for an alias to solve that.
>
> Peter
>
I agree, the use of numerictypes is the better way to go.  By all means 
provide a tool of some sort to permit onversion from or to the C codes.

Colin W.



From barrett at stsci.edu  Thu Feb 17 05:58:19 2005
From: barrett at stsci.edu (Paul Barrett)
Date: Thu Feb 17 05:58:19 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <421446E3.7080202@ee.byu.edu>
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <4214A2A3.9030401@stsci.edu>

Travis Oliphant wrote:
> 
> I am looking for feedback on the PEP.   Of particular interest is the 
> specification of multidimensional indexing that I've outlined.  I think 
> it is mostly the same as numarray (I'd love some feedback to be sure 
> about that), except for the fact that X[K] where K is a a single integer 
> index array can do 1-d indexing similar to MATLAB.  The equivalent 
> numarray indexing is available as X[K,].    Now that I've specified what 
> is to happen, I think it won't be very difficult to code.

Your indexing implementation appears reasonable.

> I would also like some help on resolving the type issue.   Is it 
> important to have hierarchial classes defined (probably in Python) that 
> can be used to check and/or specify type or are functions that check for 
> various properties, fine.   Right now x.type is specified as an 
> attribute but the attribute could be replaced by a method.  What to 
> return is the biggest question.

I don't see the need for hierarchical type classes, though it might ease 
the comparison with Python types.

> I've never had a problem with just the typecode characters although 
> aliases (like Int16 or Short) which might be more meaningful are nice.   
> Are shadow-classes important in this case?  To me they look a little 
> like extra bulk with no gain.  I'd love to be educated as to why it is 
> worth the extra effort to have a bunch of instances lying around.  I 
> don't feel very strongly about this though, so if somebody else does, 
> I'm willing to go for it.

I prefer Int16, UInt16, Float32, Float64, etc. for specifying types. 
They are unambiguous and the scheme is easily extendable.  I also don't 
mind character code aliases for named types as long as they are a 
shorthand for the them and are not used routinely when doing a repr.

  -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218



From tkorvola at welho.com  Thu Feb 17 06:53:26 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Thu Feb 17 06:53:26 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <421446E3.7080202@ee.byu.edu> (Travis Oliphant's message of
 "Thu, 17 Feb 2005 00:25:23 -0700")
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <87oeej6xxh.fsf@welho.com>

Travis Oliphant <oliphant at ee.byu.edu> writes:
> except for the fact that X[K] where K is a a single
> integer index array can do 1-d indexing similar to MATLAB.

I find the current Numarray behaviour more logical.  Is 1-D indexing
really useful enough to require a shorter and less clear notation
than X.flat[K]?

Another one that I don't like is capping the indices.  Is it useful or
just a potential source of obscure errors?  Throwing an IndexError for
out of range indices would seem safer.

Perhaps only one ellipsis should be allowed for clarity?  Is it useful
to allow more if they are only equivalent to colons?

Integers scalars are not on the list where you describe how different
types are interpreted in advanced indexing.  I suppose the intention
is to treat them as rank 0 arrays, broadcastable to any shape.

The rank of a basic index expression can only be zero if all
dimensions are indexed and all the indices are integers.  Are you
really suggesting that a rank 0 array rather than a scalar be returned
in that case?

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From southey at uiuc.edu  Thu Feb 17 07:28:19 2005
From: southey at uiuc.edu (Bruce Southey)
Date: Thu Feb 17 07:28:19 2005
Subject: [Numpy-discussion] random and RandomArray
Message-ID: <a27efba0.d017d219.81aed00@expms6.cites.uiuc.edu>

Hi, 
The Mersenne Twister is available in Python as: 
 
/Python-2.4/Modules/_randommodule.c 
 
This file contains the outdated information: 
"The code in this module was based on a download from: 
          http://www.math.keio.ac.jp/~matumoto/MT2002/emt19937ar.html" 
 
The redirected link is titled: Mersenne Twister with improved initialization 
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html 
 
A brief look at randlib() suggests that it is written so that different uniform 
generators could be used. Could randf() be rewritten to have the default 
(current status) function and a second that just to link to the random module 
library to use the Mersenne Twister?  
 
Okay, I do know it is not that easy (the seed code would also need to change so 
account for which generator is being used) but may be sufficient. 
 
Regards 
Bruce 
 
---- Original message ---- 
>Date: Wed, 16 Feb 2005 20:05:09 -0800 
>From: Robert Kern <rkern at ucsd.edu>   
>Subject: Re: [Numpy-discussion] random and RandomArray   
>To: numpy-discussion at lists.sourceforge.net 
> 
>Bruce Southey wrote: 
>> Hi,  
>> I was browsing through some of the code and realized that certain random 
number  
>> generators occur in both the Python random module and RandomArray.   
>>   
>> The random module uses C code to get a scalar uniform random number that is  
>> modified in Python by other generators to get random numbers from other  
>> distributions. In RandomArray everything is done in the C code - obviously 
this  
>> is way faster especially for arrays.  
>>   
>> In the long term, would it make sense to get use the same random number  
>> generators in both random and RandomArray?  
> 
>I think that it is certainly feasible and desirable that when/if a  
>multiarray object enters the standard library that the standard random  
>module be extended to produce arrays as well. 
> 
>I don't think it's terribly worthwhile to hack the random module to  
>expose its PRNG so that we use it's implementation without duplicating  
>code. I think the result will be quite fragile, and won't be useful  
>until Python 2.5. 
> 
>I do think it would be extremely worthwhile to implement the Mersenne  
>Twister for numarray/scipy. I promised some time ago to look into this,  
>but I have not gotten around to it, unfortunately. 
> 
>--  
>Robert Kern 
>rkern at ucsd.edu 
> 
>"In the fields of hell where the grass grows high 
>  Are the graves of dreams allowed to die." 
>   -- Richard Harter 
> 
> 
>------------------------------------------------------- 
>SF email is sponsored by - The IT Product Guide 
>Read honest & candid reviews on hundreds of IT Products from real users. 
>Discover which products truly live up to the hype. Start reading now. 
>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click 
>_______________________________________________ 
>Numpy-discussion mailing list 
>Numpy-discussion at lists.sourceforge.net 
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion 
 



From rkern at ucsd.edu  Thu Feb 17 07:45:19 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Thu Feb 17 07:45:19 2005
Subject: [Numpy-discussion] random and RandomArray
In-Reply-To: <a27efba0.d017d219.81aed00@expms6.cites.uiuc.edu>
References: <a27efba0.d017d219.81aed00@expms6.cites.uiuc.edu>
Message-ID: <4214BBBE.1060704@ucsd.edu>

Bruce Southey wrote:
> Hi, 
> The Mersenne Twister is available in Python as: 
>  
> /Python-2.4/Modules/_randommodule.c 
>  
> This file contains the outdated information: 
> "The code in this module was based on a download from: 
>           http://www.math.keio.ac.jp/~matumoto/MT2002/emt19937ar.html" 
>  
> The redirected link is titled: Mersenne Twister with improved initialization 
> http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html 
>  
> A brief look at randlib() suggests that it is written so that different uniform 
> generators could be used. Could randf() be rewritten to have the default 
> (current status) function and a second that just to link to the random module 
> library to use the Mersenne Twister?  
>  
> Okay, I do know it is not that easy (the seed code would also need to change so 
> account for which generator is being used) but may be sufficient. 

All the functions from _randommodule.c that we would need to use are 
declared static. I'm fairly sure this precludes trying to "link" to 
_randommodule.so to use these functions. The standard way to do such a 
thing would be for _randommodule.c to expose function pointers as 
PyCObjects, but that would require modification of _randommodule.c.

The path of least resistence is to just duplicate the code. It's short, 
clean, well-tested, and widely-used. It should not present a real 
maintenance problem. Certainly no more of one trying to share the 
implementation with _randommodule.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From faltet at carabos.com  Thu Feb 17 07:58:11 2005
From: faltet at carabos.com (Francesc Altet)
Date: Thu Feb 17 07:58:11 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <4214A2A3.9030401@stsci.edu>
References: <421446E3.7080202@ee.byu.edu> <4214A2A3.9030401@stsci.edu>
Message-ID: <200502171657.09055.faltet@carabos.com>

A Dijous 17 Febrer 2005 14:56, Paul Barrett va escriure:
> I prefer Int16, UInt16, Float32, Float64, etc. for specifying types.
> They are unambiguous and the scheme is easily extendable.  I also don't
> mind character code aliases for named types as long as they are a
> shorthand for the them and are not used routinely when doing a repr.

I completely agree with that point of view too.

-- 
>qo<   Francesc Altet ? ? http://www.carabos.com/
V ?V   C?rabos Coop. V. ??Enjoy Data
 ""




From klimek at grc.nasa.gov  Thu Feb 17 08:14:15 2005
From: klimek at grc.nasa.gov (Bob Klimek)
Date: Thu Feb 17 08:14:15 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <200502171657.09055.faltet@carabos.com>
References: <421446E3.7080202@ee.byu.edu> <4214A2A3.9030401@stsci.edu> <2005
	02171657.09055.faltet@carabos.com>
Message-ID: <4214C3DC.6010909@grc.nasa.gov>

Francesc Altet wrote:

>A Dijous 17 Febrer 2005 14:56, Paul Barrett va escriure:
>  
>
>>I prefer Int16, UInt16, Float32, Float64, etc. for specifying types.
>>They are unambiguous and the scheme is easily extendable.  I also don't
>>mind character code aliases for named types as long as they are a
>>shorthand for the them and are not used routinely when doing a repr.
>>    
>>
>
>I completely agree with that point of view too.
>
>  
>
I also agree.

Bob




From perry at stsci.edu  Thu Feb 17 08:29:22 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 17 08:29:22 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <421446E3.7080202@ee.byu.edu>
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <4867e3b08186795e72c4d52e2a02ae17@stsci.edu>

I'm generally going to refrain from making comments since it usually is 
obvious what we prefer. But there may be exceptions (perhaps a reminder 
of past arguments, a desire to change numarray behavior, or clarify 
what the issue is).

On Feb 17, 2005, at 2:25 AM, Travis Oliphant wrote:

>
> I am looking for feedback on the PEP.   Of particular interest is the 
> specification of multidimensional indexing that I've outlined.  I 
> think it is mostly the same as numarray (I'd love some feedback to be 
> sure about that), except for the fact that X[K] where K is a a single 
> integer index array can do 1-d indexing similar to MATLAB.  The 
> equivalent numarray indexing is available as X[K,].    Now that I've 
> specified what is to happen, I think it won't be very difficult to 
> code.
Could you clarify this with an example? I'm not quite sure what 
alternatives you are addressing.

> I would also like some help on resolving the type issue.   Is it 
> important to have hierarchial classes defined (probably in Python) 
> that can be used to check and/or specify type or are functions that 
> check for various properties, fine.   Right now x.type is specified as 
> an attribute but the attribute could be replaced by a method.  What to 
> return is the biggest question.

I'll just note that the hierarchial approach allows simpler tests for 
kinds of types (e.g., is the type an integer type) rather than doing 
something like ntype in [Int16, Int8,...].

Thanks, Perry




From tkorvola at welho.com  Thu Feb 17 08:54:11 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Thu Feb 17 08:54:11 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <4214A2A3.9030401@stsci.edu> (Paul Barrett's message of "Thu,
 17 Feb 2005 08:56:51 -0500")
References: <421446E3.7080202@ee.byu.edu> <4214A2A3.9030401@stsci.edu>
Message-ID: <87braj6scr.fsf@welho.com>

Paul Barrett <barrett at stsci.edu> writes:
> I prefer Int16, UInt16, Float32, Float64, etc. for specifying
> types.

The C based types have their uses: if you want to call external
libraries that use double or int from Python, you will want to work
with arrays of double or int without having to care how many bits they
happen to be on the current platform.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From verveer at embl-heidelberg.de  Thu Feb 17 08:58:27 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Thu Feb 17 08:58:27 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <87braj6scr.fsf@welho.com>
References: <421446E3.7080202@ee.byu.edu> <4214A2A3.9030401@stsci.edu> <87braj6scr.fsf@welho.com>
Message-ID: <1921b9ac31a272057b946aec6a3d49d8@embl-heidelberg.de>

On 17 Feb 2005, at 17:53, Timo Korvola wrote:

> Paul Barrett <barrett at stsci.edu> writes:
>> I prefer Int16, UInt16, Float32, Float64, etc. for specifying
>> types.
>
> The C based types have their uses: if you want to call external
> libraries that use double or int from Python, you will want to work
> with arrays of double or int without having to care how many bits they
> happen to be on the current platform.

Agreed, I find them very useful for such cases. Normally I use the 
exact specifications, but I agree that the C based types should be 
around. Fortunately that should be easy to support.




From faltet at carabos.com  Thu Feb 17 10:01:22 2005
From: faltet at carabos.com (Francesc Altet)
Date: Thu Feb 17 10:01:22 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <1921b9ac31a272057b946aec6a3d49d8@embl-heidelberg.de>
References: <421446E3.7080202@ee.byu.edu> <87braj6scr.fsf@welho.com> <1921b9ac31a272057b946aec6a3d49d8@embl-heidelberg.de>
Message-ID: <200502171900.56294.faltet@carabos.com>

A Dijous 17 Febrer 2005 17:56, Peter Verveer va escriure:
> On 17 Feb 2005, at 17:53, Timo Korvola wrote:
> Agreed, I find them very useful for such cases. Normally I use the
> exact specifications, but I agree that the C based types should be
> around. Fortunately that should be easy to support.

Yes, I think so. Why not .type for Python types (and numarray
compatibility) and .typecode for C types (and Numeric compatibility)?.
numarray supports both right now without problems.

Besides, in numarray.records module, you may find the following map:

numfmt = {'i1':num.Int8,  'u1':num.UInt8,
          'i2':num.Int16, 'u2':num.UInt16,
          'i4':num.Int32, 'u4':num.UInt32,
          'i8':num.Int64, 'u8':num.UInt64,
          'f4':num.Float32, 'f8':num.Float64,
          'c8':num.Complex32, 'c16':num.Complex64,
          'b1':num.Bool}

which I find very useful and intuitive, but I guess it is incompatible
with current Numeric convention (that I find quite confusing, anyway).

Cheers,

-- 
>qo<   Francesc Altet ? ? http://www.carabos.com/
V ?V   C?rabos Coop. V. ??Enjoy Data
 ""




From oliphant at ee.byu.edu  Thu Feb 17 10:53:22 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 17 10:53:22 2005
Subject: [Numpy-discussion] Response to PEP suggestions
Message-ID: <4214E7FE.9090403@ee.byu.edu>

I'm glad to get the feedback.

1) Types

I like Francesc's suggestion that .typecode return a code and .type 
return a Python class.   What is the attitude and opinion regarding the 
use of attributes or methods for
this kind of thing?  It always seems to me so arbitrary as to what is an 
attribute or what
is a method.

There will definitely be support for the nummary-style type 
specification.   Something like that will be how they print (I like the 
'i4', 'f4', specification a bit better though). There will also be 
support for specification in terms of a c-type.  The typecodes will 
still be there, underneath.

One thing has always bothered me though.  Why is a double complex type 
Complex64? and a float complex type Complex32.  This seems to break the 
idea that the number at the end specifies a bit width.   Why don't we 
just call it Complex64 and Complex128?  Can we change this?

I'm also glad that some recognize the problems with always requiring 
specification of types in terms of bit-width or byte-widths as these are 
not the same across platforms.  For some types (like Int8 or Int16) this 
is not a problem.   But what about long double?  On an intel machine 
long double is Float96 while on a PowerPC it is Float128.   Wouldn't it 
just be easier to specify LDouble or 'g' then special-case your code?

Problems also exist when you are interfacing with hardware or other C or 
Fortran code.  You know you want single-precision floating point.  You 
don't know or care what the bit-width is.    I think with the Integer 
types the bit-width specification is more important than floating point 
types.  In sum, I think it is important to have the ability to specify 
it both ways.   When printing the array, it's probably better if it 
gives bit-width information.  I like the way numarray prints arrays.


2) Multidimensional array indexing.

Sometimes it is useful to select out of an array some elements based on 
it's linear (flattened) index in the array.   MATLAB, for example, will 
allow you to take a three-dimensional array and index it with a single 
integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...

What I'm proposing would have X[K] essentially equivalent to X.flat[K].  
The problem with always requiring the use of X.flat[K] is that X.flat 
does not work for discontiguous arrays.   It could be made to work if 
X.flat returned some kind of specially-marked array, which would then 
have to be checked every time indexing occurred for any array.  Or, 
there maybe someway to have X.flat return an "indexable iterator" for X 
which may be a more Pythonic thing to do anyway.  That could solve the 
problem and solve the discontiguous X.flat problem as well. 

If we can make X.flat[K] work for discontiguous arrays, then I would be 
very happy to not special-case the single index array but always treat 
it as a 1-tuple of integer index arrays.

Capping indexes was proposed because of what numarray does.   I can only 
think that the benefit would be that you don't have to check for and 
raise an error in the middle of an indexing loop or pre-scan the 
indexes.  But, I suppose this is unavoidalbe, anyway.  Currently Numeric 
allows specifying indexes that are too high in slices. It just chops 
them.  Python allows this too, for slices.  So, I guess I'm just 
specifying Python behavior.  Of course indexing with an integer that is 
too large or too small will raise errors:

In Python:

a = [1,2,3,4,5]
a[:20]   works
a[20] raises an error.


3)  Always returning rank-0 arrays.

This may be a bit controversial as it is a bit of a change.  But, my 
experience is that quite a bit of extra code is written to check whether 
or not a calculation returns a Python-scalar (because these don't have 
the same methods as arrays).  In particular len(a) does not work if a is 
a scalar, but len(b) works if b is a rank-0 array (numeric scalar).   
Rank-0 arrays are scalars.  

When Python needs a scalar it will generally ask the object if it can 
turn itself into an int or a float.   A notable exception is indexing in 
a list (where Python needs an integer and won't ask the object to 
convert if it can).  But int(b) always returns a Python integer if the 
array has only 1 element. 

I'd like to know what reasons people can think of for ever returning 
Python scalars  unless explicitly asked for.


Thanks for the suggestions.

-Travis








From Fernando.Perez at colorado.edu  Thu Feb 17 11:12:21 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu Feb 17 11:12:21 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <4214EC72.6060106@colorado.edu>

Just a small comment:

Travis Oliphant wrote:

> Capping indexes was proposed because of what numarray does.   I can only 
> think that the benefit would be that you don't have to check for and 
> raise an error in the middle of an indexing loop or pre-scan the 
> indexes.  But, I suppose this is unavoidalbe, anyway.  Currently Numeric 
> allows specifying indexes that are too high in slices. It just chops 
> them.  Python allows this too, for slices.  So, I guess I'm just 
> specifying Python behavior.  Of course indexing with an integer that is 
> too large or too small will raise errors:
> 
> In Python:
> 
> a = [1,2,3,4,5]
> a[:20]   works
> a[20] raises an error.

This feature is extremely useful.  Just yesterday, I needed some code to check 
whether the first character in a (potentially empty) string was one of a 
certain list.  I couldn't use .startswith(), because I'd have to call it for 
each test, and I didn't feel like writing a regexp (since I abandoned Perl 
coding, I've mostly forgotten them and I need to look up the syntax every 
time).  The danger with:

if mystr[0] in ('a','b','c'): ...

is that if mystr is empty, this blows up.  Thanks to Python's acceptance of 
invalid indices in slices, I used

if mystr[0:1] in ('a','b','c'): ...

This is exactly identical to the first case, except that if the string is 
empty, mystr[0:1] returns ''.  No need for extra checks, no need for a 
try/except block, nothing.  Clean, elegant and to the point.

Given that python supports these semantics for all their sliceable containers, 
I'd very much vote for numerix doing the same.

There are cases where it makes sense for numerix arrays to have different 
semantics from python lists.  The slice-as-view is probably the best example. 
  But unless there is a strong reason to do so, I think numerix arrays should 
not deviate from python sequence behavior.

Regards,

f



From dd55 at cornell.edu  Thu Feb 17 11:39:18 2005
From: dd55 at cornell.edu (Darren Dale)
Date: Thu Feb 17 11:39:18 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214EC72.6060106@colorado.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4214EC72.6060106@colorado.edu>
Message-ID: <200502171438.06720.dd55@cornell.edu>

On Thursday 17 February 2005 02:11 pm, Fernando Perez wrote:
>  The danger with:
>
> if mystr[0] in ('a','b','c'): ...
>
> is that if mystr is empty, this blows up.  Thanks to Python's acceptance of
> invalid indices in slices, I used
>
> if mystr[0:1] in ('a','b','c'): ...
>

Has Numeric or numarray considered allowing assignment to invalid indices? I 
sometimes miss being able to do this:
a=zeros(2)
a[1:0]=1

in order to get this:
[[0,0],
 [1,0]]

I guess it is not consistent with the list operations in python, but it is a 
really handy shortcut for constructing arrays interactively.


Darren



From oliphant at ee.byu.edu  Thu Feb 17 11:48:21 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 17 11:48:21 2005
Subject: [Numpy-discussion] Expanding an array by assignment
In-Reply-To: <200502171438.06720.dd55@cornell.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4214EC72.6060106@colorado.edu> <200502171438.06720.dd55@cornell.edu>
Message-ID: <4214F4C8.5080100@ee.byu.edu>

Darren Dale wrote:

>Has Numeric or numarray considered allowing assignment to invalid indices? I 
>sometimes miss being able to do this:
>a=zeros(2)
>a[1:0]=1
>
>in order to get this:
>[[0,0],
> [1,0]]
>
>I guess it is not consistent with the list operations in python, but it is a 
>really handy shortcut for constructing arrays interactively.
>  
>

I'm not sure what you meant by this.  Did you mean being able to expand 
an array only by assignment?

i.e.

a = zeros(2)  a (2,1) array

a[1,0] = 1    (resizes a behind the scenes to a 2x2 array and then sets 
the 1,0 element)?

MATLAB does display this behavior.  I'm not sure how difficult it would 
be to support it or if it would be worth it or not.  What does IDL do? 

-Travis




From Fernando.Perez at colorado.edu  Thu Feb 17 12:01:15 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu Feb 17 12:01:15 2005
Subject: [Numpy-discussion] Expanding an array by assignment
In-Reply-To: <4214F4C8.5080100@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4214EC72.6060106@colorado.edu> <200502171438.06720.dd55@cornell.edu> <4214F4C8.5080100@ee.byu.edu>
Message-ID: <4214F6F5.8030608@colorado.edu>

Travis Oliphant wrote:

> I'm not sure what you meant by this.  Did you mean being able to expand 
> an array only by assignment?
> 
> i.e.
> 
> a = zeros(2)  a (2,1) array
> 
> a[1,0] = 1    (resizes a behind the scenes to a 2x2 array and then sets 
> the 1,0 element)?

Mmmh.  I'm not sure I like the idea of an assignment triggering a silent 
resize/reshape event.  Explicit is better than implicit and all that...

I could see this kind of magical behavior easily causing silent, extremely 
hard to find bugs in a big program.

I may be missing something, but I'd be -1 on this.

The 'invalid indices in slices' is basically just sytnactic sugar for a 
try/except block, and it's well-documented behavior in the base language, 
across all its sequence types:

In [2]: ll=[]

In [3]: tt=()

In [4]: ss=''

In [5]: ll[0:1]
Out[5]: []

In [6]: tt[0:1]
Out[6]: ()

In [7]: ss[0:1]
Out[7]: ''

So in my view at least, this behavior of python isn't a good justification for 
a silent resize/reshape (which could, I'm sure, be also potentially explosive 
memory-wise) in numerix arrays.

Regards,

f



From jswhit at fastmail.fm  Thu Feb 17 12:14:16 2005
From: jswhit at fastmail.fm (Jeff Whitaker)
Date: Thu Feb 17 12:14:16 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <4214FAD7.3030109@fastmail.fm>

Travis Oliphant wrote:

>
> 2) Multidimensional array indexing.
>
> Sometimes it is useful to select out of an array some elements based 
> on it's linear (flattened) index in the array.   MATLAB, for example, 
> will allow you to take a three-dimensional array and index it with a 
> single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...
>
> What I'm proposing would have X[K] essentially equivalent to X.flat[K].  

Maybe I'm missing something, but in Numeric and Numarray right now

 >>> from Numeric import *
 >>> a = reshape(arange(9),(3,3))
 >>> print a
[[0 1 2]
 [3 4 5]
 [6 7 8]]
 >>> a[1]
array([3, 4, 5])
 >>> a.flat[1]
1

so a[K] and a.flat[K] are very different things for multidimensional 
arrays.  I think it would be a bad idea to change this now - it would 
certainly break a lot of my code.

Or are you only talking about the case when K is a rank-1 index array 
and not a scalar?

-Jeff


-- 
Jeffrey S. Whitaker         Phone  : (303)497-6313
Meteorologist               FAX    : (303)497-6449
NOAA/OAR/CDC  R/CDC1        Email  : Jeffrey.S.Whitaker at noaa.gov
325 Broadway                Office : Skaggs Research Cntr 1D-124
Boulder, CO, USA 80303-3328 Web    : http://tinyurl.com/5telg




From oliphant at ee.byu.edu  Thu Feb 17 12:27:17 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 17 12:27:17 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214FAD7.3030109@fastmail.fm>
References: <4214E7FE.9090403@ee.byu.edu> <4214FAD7.3030109@fastmail.fm>
Message-ID: <4214FDCD.7000901@ee.byu.edu>

Jeff Whitaker wrote:

> >>> from Numeric import *
> >>> a = reshape(arange(9),(3,3))
> >>> print a
> [[0 1 2]
> [3 4 5]
> [6 7 8]]
> >>> a[1]
> array([3, 4, 5])
> >>> a.flat[1]
> 1
>
> so a[K] and a.flat[K] are very different things for multidimensional 
> arrays.  I think it would be a bad idea to change this now - it would 
> certainly break a lot of my code.

>
> Or are you only talking about the case when K is a rank-1 index array 
> and not a scalar?

a[scalar] will behave exactly the same as now.

We are just talking about the case where K is an index array (I should 
clarify that it must be rank-1 or bigger so that K being a rank-0 array 
wold behave just like the a[scalar] case).

This is basically new behavior that numarray has started supporting.  I 
just think numarray missed an important case of flattened indexing that 
MATLAB supports.  My current proposal would distinguish between 
single-index array cases and tuple-index array cases. 

I'm still thinking about the X.flat possibility.  Basically, I think 
that direction would requires a new "generic array view" or something 
like that.  It may be worth trying, but I'm not sure I want to go that 
direction right now until I convince more people to come on board with 
Numeric3.

-Travis





From tkorvola at welho.com  Thu Feb 17 13:01:19 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Thu Feb 17 13:01:19 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu> (Travis Oliphant's message of
 "Thu, 17 Feb 2005 11:52:46 -0700")
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <87zmy26gx6.fsf@welho.com>

Travis Oliphant <oliphant at ee.byu.edu> writes:
> Currently Numeric allows specifying indexes that are too high in
> slices. It just chops them.  Python allows this too, for slices.

Yes, since foo[:] just means foo.__getitem__(slice(sys.maxint)).
Slices even have a nice method for normalizing the indices.

> So, I guess I'm just specifying Python behavior.

Not really: your specification causes an element with a different
index to be returned, whereas the usual slice behaviour only causes
out-of-range indices to be omitted from the result.

> 3)  Always returning rank-0 arrays.
>
> This may be a bit controversial as it is a bit of a change.

Indeed.  So you really do intend that if foo=array([1,2]), foo[0]
should evaluate to array(1) rather than 1?

> But, my experience is that quite a bit of extra code is written to
> check whether or not a calculation returns a Python-scalar

I suppose this may be necessary for code which operates on arrays of
somewhat arbitrary rank and would not know without looking whether,
e.g., foo[0] is a scalar or an array of positive rank.

> In particular len(a) does not work if a is a scalar,

Depends on what kinds of scalars are supported.  What about object
arrays?

> but len(b) works if b is a rank-0 array

It does?  In Numarray len(b) raises ValueError and size(b) returns 1.
To me this would seem the correct behaviour.

> When Python needs a scalar it will generally ask the object if it can
> turn itself into an int or a float.

Hence this change might not be as incompatible as it seems, although
users of object arrays would be in for some surprises.

> I'd like to know what reasons people can think of for ever returning
> Python scalars  unless explicitly asked for.

It would be more consistent with the usual container semantics and
less likely to break existing code.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From theller at python.net  Thu Feb 17 13:15:10 2005
From: theller at python.net (Thomas Heller)
Date: Thu Feb 17 13:15:10 2005
Subject: [Numpy-discussion] Re: Response to PEP suggestions
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <zmy2gago.fsf@python.net>

I don't think I can help with the discussion, but I would be interested
to read this PEP - where is it?

TIA,

Thomas




From perry at stsci.edu  Thu Feb 17 13:18:18 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 17 13:18:18 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214FDCD.7000901@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4214FAD7.3030109@fastmail.fm> <4214FDCD.7000901@ee.byu.edu>
Message-ID: <28c54bbc706b589b401f14cb4672c918@stsci.edu>

On Feb 17, 2005, at 3:25 PM, Travis Oliphant wrote:

>
> This is basically new behavior that numarray has started supporting.  
> I just think numarray missed an important case of flattened indexing 
> that MATLAB supports.  My current proposal would distinguish between 
> single-index array cases and tuple-index array cases.
> I'm still thinking about the X.flat possibility.  Basically, I think 
> that direction would requires a new "generic array view" or something 
> like that.  It may be worth trying, but I'm not sure I want to go that 
> direction right now until I convince more people to come on board with 
> Numeric3.
>
It was new behavior in the sense that Numeric didn't support 
multidimensional array takes and puts at the time. For a long time it 
was the only kind of array indexing IDL supported (1-d and implicit 
.flat of multidimensional arrays). Speaking only for myself, I found 
the .flat semantics more often as unwanted behavior than convenient. 
I'd rather keep the numarray behavior in this case and make the .flat 
case explicit (but I understand the difficulty of that approach). There 
is the possibility of a custom index option (but ugly I suppose)

X[K, flatten]

where flatten is a special object that indexing recognizes as signaling 
a different interpretation to indexing.

Perry




From perry at stsci.edu  Thu Feb 17 13:29:13 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Thu Feb 17 13:29:13 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <aad8596ded4601bd9eed29e34f8efbe8@stsci.edu>

On Feb 17, 2005, at 1:52 PM, Travis Oliphant wrote:

>
> I'm glad to get the feedback.
>
> 1) Types
>
>
[...]
>
> One thing has always bothered me though.  Why is a double complex type 
> Complex64? and a float complex type Complex32.  This seems to break 
> the idea that the number at the end specifies a bit width.   Why don't 
> we just call it Complex64 and Complex128?  Can we change this?
>
My recollection is that is how we originally did it until we saw how it 
was done in Numeric, but maybe my memory is screwed up. I'm happy with 
real bit widths.

>
> Problems also exist when you are interfacing with hardware or other C 
> or Fortran code.  You know you want single-precision floating point.  
> You don't know or care what the bit-width is.    I think with the 
> Integer types the bit-width specification is more important than 
> floating point types.  In sum, I think it is important to have the 
> ability to specify it both ways.

I'd agree that supporting both is ideal. Sometimes you really want a 
specific bit-width and don't care what the platform default is, and 
others you are tied to the platform default for one reason or another.
>
> 3)  Always returning rank-0 arrays.
>
> This may be a bit controversial as it is a bit of a change.  But, my 
> experience is that quite a bit of extra code is written to check 
> whether or not a calculation returns a Python-scalar (because these 
> don't have the same methods as arrays).  In particular len(a) does not 
> work if a is a scalar, but len(b) works if b is a rank-0 array 
> (numeric scalar).   Rank-0 arrays are scalars.
> When Python needs a scalar it will generally ask the object if it can 
> turn itself into an int or a float.   A notable exception is indexing 
> in a list (where Python needs an integer and won't ask the object to 
> convert if it can).  But int(b) always returns a Python integer if the 
> array has only 1 element.
> I'd like to know what reasons people can think of for ever returning 
> Python scalars  unless explicitly asked for.
>
I'm not sure this is an important issue for us (either way) so long as 
the overhead for rank-0 arrays is not much higher than for scalars (for 
numarray it was an issue). But there are those that argue (Konrad as an 
example if I remember correctly) that the definitions of rank and such 
mean len(rank-0) should not be 1 and that one should not be able to 
index rank-0 arrays. I know that the argument has been made that this 
helps support generic programming (not having to check between scalars 
and arrays), but every time I ask for specific examples I've found that 
there are simple alternatives to solve this problem or that type checks 
are still necessary because there is no control over what users may 
supply as arguments. If this is the reason, could it be motivated with 
a couple examples to show why it is the only reasonable alternative? 
(Then you can use it to slay all subsequent whiners).

Perry




From cookedm at physics.mcmaster.ca  Thu Feb 17 13:32:19 2005
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu Feb 17 13:32:19 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu> (Travis Oliphant's message of
 "Thu, 17 Feb 2005 11:52:46 -0700")
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>

Travis Oliphant <oliphant at ee.byu.edu> writes:

> I'm glad to get the feedback.
>
> 1) Types
>
> I like Francesc's suggestion that .typecode return a code and .type
> return a Python class.   What is the attitude and opinion regarding
> the use of attributes or methods for
> this kind of thing?  It always seems to me so arbitrary as to what is
> an attribute or what
> is a method.

If it's an intrinisic attribute (heh) of the object, I usually try to
make it an attribute. So I'd make these attributes.

> There will definitely be support for the nummary-style type
> specification.   Something like that will be how they print (I like
> the 'i4', 'f4', specification a bit better though). There will also be
> support for specification in terms of a c-type.  The typecodes will
> still be there, underneath.

+1. I think labelling types with their sizes at some level is necessary
for cross-platform compatibility (more below).

> One thing has always bothered me though.  Why is a double complex type
> Complex64? and a float complex type Complex32.  This seems to break
> the idea that the number at the end specifies a bit width.   Why don't
> we just call it Complex64 and Complex128?  Can we change this?

Or rename to ComplexFloat32 and ComplexFloat64?

> I'm also glad that some recognize the problems with always requiring
> specification of types in terms of bit-width or byte-widths as these
> are not the same across platforms.  For some types (like Int8 or
> Int16) this is not a problem.   But what about long double?  On an
> intel machine long double is Float96 while on a PowerPC it is
> Float128.   Wouldn't it just be easier to specify LDouble or 'g' then
> special-case your code?

One problem to consider (and where I first ran into these type of
things) is when pickling. A pickle containing an array of Int isn't
portable, if the two machines have a different idea of what an Int is
(Int32 or Int64, for instance). Another reason to keep the byte-width.

LDouble, for instance, should probably be an alias to Float96 on
Intel, and Float128 on PPC, and pickle accordingly.

> Problems also exist when you are interfacing with hardware or other C
> or Fortran code.  You know you want single-precision floating point.
> You don't know or care what the bit-width is.    I think with the
> Integer types the bit-width specification is more important than
> floating point types.  In sum, I think it is important to have the
> ability to specify it both ways.   When printing the array, it's
> probably better if it gives bit-width information.  I like the way
> numarray prints arrays.

Do you mean adding bit-width info to str()? repr() definitely needs
it, and it should be included in all cases, I think.

You also run into that sizeof(Python integer) isn't necessarily
sizeof(C int) (a Python int being a C long), espically on 64-bit systems.

I come from a C background, so things like Float64, etc., look wrong.
I think more in terms of single- and double-precision, so I think
adding some more descriptive types:

CInt         (would be either Int32 or Int64, depending on the platform)
CFloat       (can't do Float, for backwards-compatibility reasons)
CDouble      (could just be Double)
CLong        (or Long)
CLongLong    (or LongLong)

That could make it easier to match types in Python code to types in C
extensions.

Oh, and the Python types int and float should be allowed (especially
if you want this to go in the core!).

And a Fortran integer could be something else, but I think that's
more of a SciPy problem than Numeric or numarray. It could add
FInteger and FBoolean, for instance.

> 2) Multidimensional array indexing.
>
> Sometimes it is useful to select out of an array some elements based
> on it's linear (flattened) index in the array.   MATLAB, for example,
> will allow you to take a three-dimensional array and index it with a
> single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...
>
> What I'm proposing would have X[K] essentially equivalent to
> X.flat[K].  The problem with always requiring the use of X.flat[K] is
> that X.flat does not work for discontiguous arrays.   It could be made
> to work if X.flat returned some kind of specially-marked array, which
> would then have to be checked every time indexing occurred for any
> array.  Or, there maybe someway to have X.flat return an "indexable
> iterator" for X which may be a more Pythonic thing to do anyway.  That
> could solve the problem and solve the discontiguous X.flat problem as
> well.
>
> If we can make X.flat[K] work for discontiguous arrays, then I would
> be very happy to not special-case the single index array but always
> treat it as a 1-tuple of integer index arrays.

Right now, I find X.flat to be pretty useless, as you need a
contiguous array. I'm +1 on making X.flat work in all cases (contiguous
and discontiguous). Either

a) X.flat returns a contiguous 1-dimensional array (like ravel(X)),
   which may be a copy of X

or

b) X.flat returns a "flat-indexable" view of X

I'd argue for b), as I feel that attributes should operate as views,
not as potential copies. To me, attributes "feel like" they do no
work, so making a copy by mere dereferencing would be suprising.

If a), I'd rather flat() be a method (or have a ravel() method).


I think overloading X[K] starts to run into trouble: too many special
cases.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From Fernando.Perez at colorado.edu  Thu Feb 17 14:06:11 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu Feb 17 14:06:11 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>
Message-ID: <4215152B.2070108@colorado.edu>

David M. Cooke wrote:

> Right now, I find X.flat to be pretty useless, as you need a
> contiguous array. I'm +1 on making X.flat work in all cases (contiguous
> and discontiguous). Either
> 
> a) X.flat returns a contiguous 1-dimensional array (like ravel(X)),
>    which may be a copy of X
> 
> or
> 
> b) X.flat returns a "flat-indexable" view of X
> 
> I'd argue for b), as I feel that attributes should operate as views,
> not as potential copies. To me, attributes "feel like" they do no
> work, so making a copy by mere dereferencing would be suprising.

+1 on b).

I have quite a bit of code doing things like

try:
   b=a.flat
except:
   b=ravel(a)

which feels silly.  It would be nice for .flat to be a guaranteed, indexable, 
no-copy view of the original (even if it's non-contiguous).  Granted, the 
indexing will be costlier for a non-contiguous object, but for the external 
users this provides a clean API.

Regards,

f



From rkern at ucsd.edu  Thu Feb 17 14:15:21 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Thu Feb 17 14:15:21 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4215152B.2070108@colorado.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu>
Message-ID: <42151728.7010307@ucsd.edu>

Fernando Perez wrote:

> I have quite a bit of code doing things like
> 
> try:
>   b=a.flat
> except:
>   b=ravel(a)
> 
> which feels silly.  It would be nice for .flat to be a guaranteed, 
> indexable, no-copy view of the original (even if it's non-contiguous).  
> Granted, the indexing will be costlier for a non-contiguous object, but 
> for the external users this provides a clean API.

Why not just do

   b = ravel(a)

?

That should work in both cases just fine.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From rowen at cesmail.net  Thu Feb 17 14:17:19 2005
From: rowen at cesmail.net (Russell E. Owen)
Date: Thu Feb 17 14:17:19 2005
Subject: [Numpy-discussion] Re: Multiarray PEP
References: <421446E3.7080202@ee.byu.edu>
Message-ID: <rowen-8046F3.14024817022005@sea.gmane.org>

In article <421446E3.7080202 at ee.byu.edu>,
 Travis Oliphant <oliphant at ee.byu.edu> wrote:

> I've never had a problem with just the typecode characters although 
> aliases (like Int16 or Short) which might be more meaningful are nice.   
> Are shadow-classes important in this case?  To me they look a little 
> like extra bulk with no gain.  I'd love to be educated as to why it is 
> worth the extra effort to have a bunch of instances lying around. 

Instances can give you useful information. For example:
- the number of bytes needed to represent a value
- the min and max values for that type (numarray doesn't do this yet, 
but I really hope it'll be added someday).

I personally dislike character codes because they are uninformative and 
because they meaningless without the documentation.

-- Russell




From Fernando.Perez at colorado.edu  Thu Feb 17 14:19:23 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu Feb 17 14:19:23 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151728.7010307@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu>
Message-ID: <42151848.301@colorado.edu>

Robert Kern wrote:
> Why not just do
> 
>    b = ravel(a)
> 
> ?
> 
> That should work in both cases just fine.

Because I thouhgt that ravel would make a copy regardless.  But I may be 
wrong, much of this is very old code, when I was just picking up Numeric.  At 
the time, I felt it might be best to avoid the unnecessary copies if possible.

Best,

f



From rlw at stsci.edu  Thu Feb 17 14:22:22 2005
From: rlw at stsci.edu (Rick White)
Date: Thu Feb 17 14:22:22 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>
Message-ID: <Pine.GSO.4.44.0502171715260.9373-100000@sundog.stsci.edu>

On Thu, 17 Feb 2005, David M. Cooke wrote:

> I come from a C background, so things like Float64, etc., look wrong.
> I think more in terms of single- and double-precision, so I think
> adding some more descriptive types:
>
> CInt         (would be either Int32 or Int64, depending on the platform)
> CFloat       (can't do Float, for backwards-compatibility reasons)
> CDouble      (could just be Double)
> CLong        (or Long)
> CLongLong    (or LongLong)
>
> That could make it easier to match types in Python code to types in C
> extensions.

Good choice of names.

> Oh, and the Python types int and float should be allowed (especially
> if you want this to go in the core!).

Say, I like that idea.  And maybe, like float and int, the numeric types
could be callable to construct numeric arrays of that type, e.g.,

a = numeric3.Int16([1,2,3])

- Rick




From duncan at enthought.com  Thu Feb 17 14:23:17 2005
From: duncan at enthought.com (Duncan Child)
Date: Thu Feb 17 14:23:17 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <87zmy26gx6.fsf@welho.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>
Message-ID: <42151919.60607@enthought.com>

Here are a couple of issues that are important to me that might be 
relevant to the design discussion.

I have attached some code that illustrates part of the pain we have 
experienced developing libraries of algorithms that can handle both 
arrays and scalars. The attached library is the reusable part. The other 
part of this problem is that we have lots of logic sprinkled throughout 
our algorithms to enable them to handle both arrays and scalars.

Secondly, I have just been bitten by this declaration which suggests 
that the new Numeric might handle default values better:.

_vp_mod = zeros(num_pts)

It would be less surprising to someone developing numeric algorithms if 
functions like this defaulted to creating a double precision array 
rather than integers.

Regards,

Duncan

>  
>
>>3)  Always returning rank-0 arrays.
>>
>>This may be a bit controversial as it is a bit of a change.
>>    
>>
>
>Indeed.  So you really do intend that if foo=array([1,2]), foo[0]
>should evaluate to array(1) rather than 1?
>
>  
>
import scipy
from scipy import take, amin, amax, arange, asarray, PyObject, mean, \
                  product, shape, array, Float64, nonzero


"""
The following safe_ methods were written to handle both arrays amd 
scalars to
save the developer of numerical methods having to clutter their code 
with tests
to determine the type of the data.
"""

def safe_take(a,indices):
    # Slice the input if it is an array but not if it is a scalar
    try:
        a = take(a,indices)
    except ValueError:
        # a is scalar
        pass
    return a
   
def safe_copy(a):
    # Return a copy for both scalar and array input
    try:
        b = a.copy()
    except AttributeError:
        # a is a scalar
        b = a
    return b

# Note: if x is a scalar and y = asarray(x), amin(y) FAILS but min(y) works
# Note: BUT IF z=convert(y,frac,frac), THEN min(z) FAILS!!!
def safe_min(a):
    # Return the minimum of the input array or the input if it is a scalar
    try:
        safemin = amin(a)
    except:
        safemin = a
    return safemin

def safe_max(a):
    # Return the maximum of the input array or the input if it is a scalar
    try:
        safemax = amax(a)
    except:
        safemax = a
    return safemax

def safe_mean(a):
    # Return the mean of the input array or the input if it is a scalar
    try:
        safemean = mean(a)
    except:
        safemean = a
    return safemean

def safe_len(a):
    # Return the length of the input array or 1 if it is a scalar
    try:
        safelen = len(a)
    except:
        safelen = 1
    return safelen

def safe_flat(a):
    # Return a flat version of the input array or input if it is a scalar
    try:
        safeflat = a.flat
    except:
        safeflat = a
    return safeflat




From stephen.walton at csun.edu  Thu Feb 17 14:27:16 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Thu Feb 17 14:27:16 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4215152B.2070108@colorado.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu>
Message-ID: <421519F2.1070607@csun.edu>

Fernando Perez wrote:

> It would be nice for .flat to be a guaranteed, indexable, no-copy view 
> of the original (even if it's non-contiguous).

Dare I point out that a.flat works for non-contiguous arrays in numarray?

In [4]: a.iscontiguous()
Out[4]: 0

In [5]: a.flat
Out[5]:
array([ 0,  5, 10, 15, 20,  1,  6, 11, 16, 21,  2,  7, 12, 17, 22,  3,
        8, 13, 18, 23,  4,  9, 14, 19, 24])





From rkern at ucsd.edu  Thu Feb 17 14:28:16 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Thu Feb 17 14:28:16 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151848.301@colorado.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu> <42151848.301@colorado.edu>
Message-ID: <42151A4A.9070200@ucsd.edu>

Fernando Perez wrote:
> Robert Kern wrote:
> 
>> Why not just do
>>
>>    b = ravel(a)
>>
>> ?
>>
>> That should work in both cases just fine.
> 
> 
> Because I thouhgt that ravel would make a copy regardless.  But I may be 
> wrong, much of this is very old code, when I was just picking up 
> Numeric.  At the time, I felt it might be best to avoid the unnecessary 
> copies if possible.

Nope.

In [1]: a = arange(8)

In [2]: a.shape = (2,4)

In [3]: a
Out[3]: NumPy array, format: long
[[0 1 2 3]
  [4 5 6 7]]

In [4]: ravel(a)[3] = 10

In [5]: a
Out[5]: NumPy array, format: long
[[ 0  1  2 10]
  [ 4  5  6  7]]

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Thu Feb 17 14:33:14 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu Feb 17 14:33:14 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151A4A.9070200@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu> <42151848.301@colorado.edu> <42151A4A.9070200@ucsd.edu>
Message-ID: <42151B82.3040507@colorado.edu>

Robert Kern wrote:

>>Because I thouhgt that ravel would make a copy regardless.  But I may be 
>>wrong, much of this is very old code, when I was just picking up 
>>Numeric.  At the time, I felt it might be best to avoid the unnecessary 
>>copies if possible.
> 
> 
> Nope.

[...]

OK, thanks for the clarification.  I guess if I had a nice, easy to use 
interactive shell I could have figured it out on my own.  You'll have to sell 
me that shiny one you seem to be using :)

Best,

f



From tim.hochberg at cox.net  Thu Feb 17 14:45:09 2005
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Thu Feb 17 14:45:09 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151A4A.9070200@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu> <42151848.301@colorado.edu> <42151A4A.9070200@ucsd.edu>
Message-ID: <42151E39.9000409@cox.net>

Robert Kern wrote:

> Fernando Perez wrote:
>
>> Robert Kern wrote:
>>
>>> Why not just do
>>>
>>>    b = ravel(a)
>>>
>>> ?
>>>
>>> That should work in both cases just fine.
>>
>>
>>
>> Because I thouhgt that ravel would make a copy regardless.  But I may 
>> be wrong, much of this is very old code, when I was just picking up 
>> Numeric.  At the time, I felt it might be best to avoid the 
>> unnecessary copies if possible.
>
>
> Nope.
>
> In [1]: a = arange(8)
>
> In [2]: a.shape = (2,4)
>
> In [3]: a
> Out[3]: NumPy array, format: long
> [[0 1 2 3]
>  [4 5 6 7]]
>
> In [4]: ravel(a)[3] = 10
>
> In [5]: a
> Out[5]: NumPy array, format: long
> [[ 0  1  2 10]
>  [ 4  5  6  7]]


Ick, that's horrible. Functions that sometimes copy and sometimes don't 
are generally bad news IMO. This is just a way to introduce nasty, 
invisible bugs. The exceptions are things like asarray that are explicit 
about their variable behaviour.

I'd be much happier if flat never made copies, but always worked by some 
sort of deep juju, while ravel always made copies.

-tim





From stephen.walton at csun.edu  Thu Feb 17 15:34:21 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Thu Feb 17 15:34:21 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151919.60607@enthought.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42151919.60607@enthought.com>
Message-ID: <421529C0.9070605@csun.edu>

Duncan Child wrote:

> # Note: if x is a scalar and y = asarray(x), amin(y) FAILS but min(y) 
> works
> # Note: BUT IF z=convert(y,frac,frac), THEN min(z) FAILS!!

My quick check shows that amin(y), min(y), and amin(x) all work after 
"from scipy import *".  Only min(x) doesn't, and it is a Python built-in 
which throws an "iteration over non-sequence" exception when called with 
a scalar.   Should Numeric 3 bump amin and similar functions up into 
itself?  How many others need a min-like function which works with both 
arrays and scalars?  In numarray, min is a method.  Is this better?




From verveer at embl.de  Thu Feb 17 15:54:27 2005
From: verveer at embl.de (Peter Verveer)
Date: Thu Feb 17 15:54:27 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <3a2f1080766ca2dad8bfcaf6ca46ee0b@embl.de>

On Feb 17, 2005, at 7:52 PM, Travis Oliphant wrote:

>
> I'm glad to get the feedback.
>
> 1) Types
>
> I like Francesc's suggestion that .typecode return a code and .type 
> return a Python class.   What is the attitude and opinion regarding 
> the use of attributes or methods for
> this kind of thing?  It always seems to me so arbitrary as to what is 
> an attribute or what
> is a method.

I don't think it really matters. Attributes seem natural, shape is an 
attribute for instance, so why not type? In the end, I don't care.

> There will definitely be support for the nummary-style type 
> specification.   Something like that will be how they print (I like 
> the 'i4', 'f4', specification a bit better though). There will also be 
> support for specification in terms of a c-type.  The typecodes will 
> still be there, underneath.

sounds fine to me.

>
> One thing has always bothered me though.  Why is a double complex type 
> Complex64? and a float complex type Complex32.  This seems to break 
> the idea that the number at the end specifies a bit width.   Why don't 
> we just call it Complex64 and Complex128?  Can we change this?

I actually find the current approach natural. You specify the width of 
the real and the imaginary type, which are some  kind of a double type. 
Again, in the end I would not care.

> I'm also glad that some recognize the problems with always requiring 
> specification of types in terms of bit-width or byte-widths as these 
> are not the same across platforms.  For some types (like Int8 or 
> Int16) this is not a problem.   But what about long double?  On an 
> intel machine long double is Float96 while on a PowerPC it is 
> Float128.   Wouldn't it just be easier to specify LDouble or 'g' then 
> special-case your code?

long double is a bit of a special case. I guess I would probably not 
use it anyway. The point is indeed that having things like LDouble is 
'a good thing'.

> Problems also exist when you are interfacing with hardware or other C 
> or Fortran code.  You know you want single-precision floating point.  
> You don't know or care what the bit-width is.    I think with the 
> Integer types the bit-width specification is more important than 
> floating point types.  In sum, I think it is important to have the 
> ability to specify it both ways.

I completely agree with this. I probably dont care for floating point, 
it is good enough to distinguish between single and double precision. 
Integer types are a different story, you want to be a bit more precise 
then. Having both solves the problem quite well.

>   When printing the array, it's probably better if it gives bit-width 
> information.  I like the way numarray prints arrays.

Agreed.

>
>
> 2) Multidimensional array indexing.
>
> Sometimes it is useful to select out of an array some elements based 
> on it's linear (flattened) index in the array.   MATLAB, for example, 
> will allow you to take a three-dimensional array and index it with a 
> single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...
>
> What I'm proposing would have X[K] essentially equivalent to 
> X.flat[K].  The problem with always requiring the use of X.flat[K] is 
> that X.flat does not work for discontiguous arrays.   It could be made 
> to work if X.flat returned some kind of specially-marked array, which 
> would then have to be checked every time indexing occurred for any 
> array.  Or, there maybe someway to have X.flat return an "indexable 
> iterator" for X which may be a more Pythonic thing to do anyway.  That 
> could solve the problem and solve the discontiguous X.flat problem as 
> well.

But possibly slow, and that we want to avoid.

> If we can make X.flat[K] work for discontiguous arrays, then I would 
> be very happy to not special-case the single index array but always 
> treat it as a 1-tuple of integer index arrays.

Speed will be an issue.

> Capping indexes was proposed because of what numarray does.   I can 
> only think that the benefit would be that you don't have to check for 
> and raise an error in the middle of an indexing loop or pre-scan the 
> indexes.  But, I suppose this is unavoidalbe, anyway.  Currently 
> Numeric allows specifying indexes that are too high in slices. It just 
> chops them.  Python allows this too, for slices.  So, I guess I'm just 
> specifying Python behavior.  Of course indexing with an integer that 
> is too large or too small will raise errors:
>
> In Python:
>
> a = [1,2,3,4,5]
> a[:20]   works
> a[20] raises an error.

Probably better to stick to Python behavior.




From verveer at embl.de  Thu Feb 17 16:00:10 2005
From: verveer at embl.de (Peter Verveer)
Date: Thu Feb 17 16:00:10 2005
Subject: [Numpy-discussion] Re: Multiarray PEP
In-Reply-To: <rowen-8046F3.14024817022005@sea.gmane.org>
References: <421446E3.7080202@ee.byu.edu> <rowen-8046F3.14024817022005@sea.gmane.org>
Message-ID: <cc426cd57565fdad7a682ce82a3ef96b@embl.de>

On Feb 17, 2005, at 11:02 PM, Russell E. Owen wrote:

> In article <421446E3.7080202 at ee.byu.edu>,
>  Travis Oliphant <oliphant at ee.byu.edu> wrote:
>
>> I've never had a problem with just the typecode characters although
>> aliases (like Int16 or Short) which might be more meaningful are nice.
>> Are shadow-classes important in this case?  To me they look a little
>> like extra bulk with no gain.  I'd love to be educated as to why it is
>> worth the extra effort to have a bunch of instances lying around.
>
> Instances can give you useful information. For example:
> - the number of bytes needed to represent a value
> - the min and max values for that type (numarray doesn't do this yet,
> but I really hope it'll be added someday).

+1 on this! This would be very useful...




From verveer at embl.de  Thu Feb 17 16:02:18 2005
From: verveer at embl.de (Peter Verveer)
Date: Thu Feb 17 16:02:18 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <Pine.GSO.4.44.0502171715260.9373-100000@sundog.stsci.edu>
References: <Pine.GSO.4.44.0502171715260.9373-100000@sundog.stsci.edu>
Message-ID: <8c7fe7fab3cba13e35398e4b26bbca77@embl.de>

>> Oh, and the Python types int and float should be allowed (especially
>> if you want this to go in the core!).
>
> Say, I like that idea.  And maybe, like float and int, the numeric 
> types
> could be callable to construct numeric arrays of that type, e.g.,
>
> a = numeric3.Int16([1,2,3])

That is a good idea. Seems easy to implement, they would just be 
aliases.




From cjw at sympatico.ca  Thu Feb 17 17:01:27 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 17 17:01:27 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca>
Message-ID: <42153E36.7050808@sympatico.ca>

David M. Cooke wrote:

>Travis Oliphant <oliphant at ee.byu.edu> writes:
>
>  
>
>>I'm glad to get the feedback.
>>
>>1) Types
>>
>>I like Francesc's suggestion that .typecode return a code and .type
>>return a Python class.   What is the attitude and opinion regarding
>>the use of attributes or methods for
>>this kind of thing?  It always seems to me so arbitrary as to what is
>>an attribute or what
>>is a method.
>>    
>>
>
>If it's an intrinisic attribute (heh) of the object, I usually try to
>make it an attribute. So I'd make these attributes.
>
>  
>
>>There will definitely be support for the nummary-style type
>>specification.   Something like that will be how they print (I like
>>the 'i4', 'f4', specification a bit better though). There will also be
>>support for specification in terms of a c-type.  The typecodes will
>>still be there, underneath.
>>    
>>
>
>+1. I think labelling types with their sizes at some level is necessary
>for cross-platform compatibility (more below).
>
>  
>
>>One thing has always bothered me though.  Why is a double complex type
>>Complex64? and a float complex type Complex32.  This seems to break
>>the idea that the number at the end specifies a bit width.   Why don't
>>we just call it Complex64 and Complex128?  Can we change this?
>>    
>>
>
>Or rename to ComplexFloat32 and ComplexFloat64?
>
>  
>
>>I'm also glad that some recognize the problems with always requiring
>>specification of types in terms of bit-width or byte-widths as these
>>are not the same across platforms.  For some types (like Int8 or
>>Int16) this is not a problem.   But what about long double?  On an
>>intel machine long double is Float96 while on a PowerPC it is
>>Float128.   Wouldn't it just be easier to specify LDouble or 'g' then
>>special-case your code?
>>    
>>
>
>One problem to consider (and where I first ran into these type of
>things) is when pickling. A pickle containing an array of Int isn't
>portable, if the two machines have a different idea of what an Int is
>(Int32 or Int64, for instance). Another reason to keep the byte-width.
>
>LDouble, for instance, should probably be an alias to Float96 on
>Intel, and Float128 on PPC, and pickle accordingly.
>
>  
>
>>Problems also exist when you are interfacing with hardware or other C
>>or Fortran code.  You know you want single-precision floating point.
>>You don't know or care what the bit-width is.    I think with the
>>Integer types the bit-width specification is more important than
>>floating point types.  In sum, I think it is important to have the
>>ability to specify it both ways.   When printing the array, it's
>>probably better if it gives bit-width information.  I like the way
>>numarray prints arrays.
>>    
>>
>
>Do you mean adding bit-width info to str()? repr() definitely needs
>it, and it should be included in all cases, I think.
>
>You also run into that sizeof(Python integer) isn't necessarily
>sizeof(C int) (a Python int being a C long), espically on 64-bit systems.
>
>I come from a C background, so things like Float64, etc., look wrong.
>I think more in terms of single- and double-precision, so I think
>adding some more descriptive types:
>
>CInt         (would be either Int32 or Int64, depending on the platform)
>CFloat       (can't do Float, for backwards-compatibility reasons)
>CDouble      (could just be Double)
>CLong        (or Long)
>CLongLong    (or LongLong)
>
>That could make it easier to match types in Python code to types in C
>extensions.
>  
>
I guess the issue revolves around the characteristics of the target 
users, if most are C aficionados then the above has merit.  However, 
this doesn't provide for the Int8's or the Int16's.  Neither does it 
provide for a bit array, which would be suitable for Booleans.

My guess is that most users would not be from a C background and so 
something along the lines of numerictypes makes sense.

>Oh, and the Python types int and float should be allowed (especially
>if you want this to go in the core!).
>
>And a Fortran integer could be something else, but I think that's
>more of a SciPy problem than Numeric or numarray. It could add
>FInteger and FBoolean, for instance.
>
>  
>
>>2) Multidimensional array indexing.
>>
>>Sometimes it is useful to select out of an array some elements based
>>on it's linear (flattened) index in the array.   MATLAB, for example,
>>will allow you to take a three-dimensional array and index it with a
>>single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...
>>
>>What I'm proposing would have X[K] essentially equivalent to
>>X.flat[K].  The problem with always requiring the use of X.flat[K] is
>>that X.flat does not work for discontiguous arrays.   It could be made
>>to work if X.flat returned some kind of specially-marked array, which
>>would then have to be checked every time indexing occurred for any
>>array.  Or, there maybe someway to have X.flat return an "indexable
>>iterator" for X which may be a more Pythonic thing to do anyway.  That
>>could solve the problem and solve the discontiguous X.flat problem as
>>well.
>>
>>If we can make X.flat[K] work for discontiguous arrays, then I would
>>be very happy to not special-case the single index array but always
>>treat it as a 1-tuple of integer index arrays.
>>    
>>
>
>Right now, I find X.flat to be pretty useless, as you need a
>contiguous array. I'm +1 on making X.flat work in all cases (contiguous
>and discontiguous). Either
>
>a) X.flat returns a contiguous 1-dimensional array (like ravel(X)),
>   which may be a copy of X
>
>or
>
>b) X.flat returns a "flat-indexable" view of X
>
>I'd argue for b), as I feel that attributes should operate as views,
>not as potential copies. To me, attributes "feel like" they do no
>work, so making a copy by mere dereferencing would be suprising.
>
>If a), I'd rather flat() be a method (or have a ravel() method).
>
>
>I think overloading X[K] starts to run into trouble: too many special
>cases.
>  
>
As someone else said, the draft PEP needs to have a much clearer 
statement of what datatype K is and just what X[K] would mean.

Colin W.




From cjw at sympatico.ca  Thu Feb 17 17:14:40 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 17 17:14:40 2005
Subject: [Numpy-discussion] Re: Multiarray PEP
In-Reply-To: <rowen-8046F3.14024817022005@sea.gmane.org>
References: <421446E3.7080202@ee.byu.edu> <rowen-8046F3.14024817022005@sea.gmane.org>
Message-ID: <42154124.6080406@sympatico.ca>

Russell E. Owen wrote:

>In article <421446E3.7080202 at ee.byu.edu>,
> Travis Oliphant <oliphant at ee.byu.edu> wrote:
>
>  
>
>>I've never had a problem with just the typecode characters although 
>>aliases (like Int16 or Short) which might be more meaningful are nice.   
>>Are shadow-classes important in this case?  To me they look a little 
>>like extra bulk with no gain.  I'd love to be educated as to why it is 
>>worth the extra effort to have a bunch of instances lying around. 
>>    
>>
>
>Instances can give you useful information. For example:
>- the number of bytes needed to represent a value
>  
>
 >>> import numarray.numarraycore as _n
 >>> a= _n.arange(9, shape= (3, 3))
 >>> print 'bytes per element:', a.itemsize()
bytes per element: 4

Alternatively:
 >>> a._type.bytes
4

>- the min and max values for that type (numarray doesn't do this yet, 
>but I really hope it'll be added someday).
>  
>
This could come from the type instance but it's not too difficult to 
determine for a given type.

>I personally dislike character codes because they are uninformative and 
>because they meaningless without the documentation.
>
>  
>
(+1)
Colin W.



From cjw at sympatico.ca  Thu Feb 17 17:29:23 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 17 17:29:23 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <3a2f1080766ca2dad8bfcaf6ca46ee0b@embl.de>
References: <4214E7FE.9090403@ee.byu.edu> <3a2f1080766ca2dad8bfcaf6ca46ee0b@embl.de>
Message-ID: <421544C0.70404@sympatico.ca>

Peter Verveer wrote:

>
> On Feb 17, 2005, at 7:52 PM, Travis Oliphant wrote:
>
>>
>> I'm glad to get the feedback.
>> [snip]
>
>> 2) Multidimensional array indexing.
>>
>> Sometimes it is useful to select out of an array some elements based 
>> on it's linear (flattened) index in the array.   MATLAB, for example, 
>> will allow you to take a three-dimensional array and index it with a 
>> single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...
>>
>> What I'm proposing would have X[K] essentially equivalent to 
>> X.flat[K].  The problem with always requiring the use of X.flat[K] is 
>> that X.flat does not work for discontiguous arrays.   It could be 
>> made to work if X.flat returned some kind of specially-marked array, 
>> which would then have to be checked every time indexing occurred for 
>> any array.  Or, there maybe someway to have X.flat return an 
>> "indexable iterator" for X which may be a more Pythonic thing to do 
>> anyway.  That could solve the problem and solve the discontiguous 
>> X.flat problem as well.
>
>
> But possibly slow, and that we want to avoid.

Currently, numarray returns an array with a reduced rank:

 >>> a
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])
 >>> a[1]
array([3, 4, 5])
 >>> a[:,1]
array([1, 4, 7])
 >>>

Is this to be abandoned?

Colin W.



From cookedm at physics.mcmaster.ca  Thu Feb 17 17:55:20 2005
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu Feb 17 17:55:20 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42153E36.7050808@sympatico.ca>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <42153E36.7050808@sympatico.ca>
Message-ID: <20050218015333.GA18265@arbutus.physics.mcmaster.ca>

On Thu, Feb 17, 2005 at 08:00:38PM -0500, Colin J. Williams wrote:
> David M. Cooke wrote:
> >I come from a C background, so things like Float64, etc., look wrong.
> >I think more in terms of single- and double-precision, so I think
> >adding some more descriptive types:
> >
> >CInt         (would be either Int32 or Int64, depending on the platform)
> >CFloat       (can't do Float, for backwards-compatibility reasons)
> >CDouble      (could just be Double)
> >CLong        (or Long)
> >CLongLong    (or LongLong)
> >
> >That could make it easier to match types in Python code to types in C
> >extensions.
> > 
> >
> I guess the issue revolves around the characteristics of the target 
> users, if most are C aficionados then the above has merit.  However, 
> this doesn't provide for the Int8's or the Int16's.  Neither does it 
> provide for a bit array, which would be suitable for Booleans.
> 
> My guess is that most users would not be from a C background and so 
> something along the lines of numerictypes makes sense.

I'm thinking that CInt, etc., would be aliases for Int32 or Int64 (or
whatever makes sense on the platform), at least at the Python level.

The idea is if you're writing wrapper code for external routines, you
want to use the types used in the routine, which most likely will vary
platform-by-platform. In that case you *don't* want to hardcode Int32,
etc., because that's not the right type for all platforms. I've run into
enough of these bugs, since I'm using a 64-bit Athlon64 Linux system now
as my main system (and Numeric still has some problems internally in
this respect).

It's partly documentation: you're asserting that an array is an array of
C ints, whatever a C int is. Passing that to a routine that takes C ints
shouldn't require conversion, or fail because of mismatched min-max int
ranges.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From cjw at sympatico.ca  Thu Feb 17 18:51:20 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Thu Feb 17 18:51:20 2005
Subject: [Numpy-discussion] Some comments on the draft PEP (Rev 1.8)
Message-ID: <42155814.4040908@sympatico.ca>

Basic Types

    There is no mention of the record.  This was one of the useful ideas 
from COBOL and carried over by PL/1.  See numarray.record.py.

    I suggest it is worth continuing.

Sequence and Mapping 2)  The advanced indexing seems to offer masked 
extraction from an array, some examples would help.

                                     3)  I've always wondered about the 
rank 0 array.  If such values are returned as scalars then the 
processing inside the array package is increased slightly.  If they are 
returned as an array then there is more work to be done in the Python 
code outside the the array package.  My preference is to return a scalar.

Iterator.   A scalar appears to be much better in this case.

Methods.  Why not use properties for those methods which don't require 
arguments?

                Some of the maybes are number oriented but the basic 
Array is intended to have a wider scope than just numbers.

Implementation Plan   The usual convention is that class/type names 
start with a capital, thus Array, rather than array.  This should clash 
with a name on the existing array module.  It is good to see that the 
Array is to be sub-classable.  This is one of the weaknesses of 
numarray.  'efficiencies' meant that Python's introspection is not fully 
available in NumArray.

C Structures              Is NOTSWAPPED, based on system order, going to 
give trouble if data is exported from one system to another?

Type characters         My feeling is that the numarray.numerictypes 
descriptors are clearer for the user who has not memorized the 
typecodes.  It takes a little more space overhead but simplifies the 
Array repr.  With an editor, such as PythonWin, one can enter 
'MN.numerictypes.'.  The editor responds with a drop down list of 
available types - MN is an abbreviated code for the imported modules 
name (I don't use from X import *).  The programmer does not have to 
remember the non mnemonic letter codes.

One final request.             In the instance constructor, please avoid 
the use of 'type' as a parameter name if the constructor is implemented 
in Python.  This hides the builtin type(), which is sometimes useful in 
debugging or exploring code.

I hope that this is of some help.

Colin W.






From oliphant at ee.byu.edu  Thu Feb 17 21:26:13 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Thu Feb 17 21:26:13 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151E39.9000409@cox.net>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu> <42151848.301@colorado.edu> <42151A4A.9070200@ucsd.edu> <42151E39.9000409@cox.net>
Message-ID: <42157BF9.9090302@ee.byu.edu>

> Ick, that's horrible. Functions that sometimes copy and sometimes 
> don't are generally bad news IMO. This is just a way to introduce 
> nasty, invisible bugs. The exceptions are things like asarray that are 
> explicit about their variable behaviour.
>
> I'd be much happier if flat never made copies, but always worked by 
> some sort of deep juju, while ravel always made copies.


I tend to agree (though) there is precedence for "return a copy only if 
you have to" at least on the C-level.

What I suggest is that attributes never return copies while methods 
return copies when necessary.

In that vein, I am proposing making X.flat an array iterator  and 
allowing array iterators to be indexed and set as if they were 1-d 
arrays with the underlying array being changed.  This is actually an 
easy change with the current code base.  Will it break any code?  

There maybe some X.flats that need to be changed to ravel.  But it seems 
like a really good idea.

-Travis





From verveer at embl-heidelberg.de  Fri Feb 18 00:59:29 2005
From: verveer at embl-heidelberg.de (Peter Verveer)
Date: Fri Feb 18 00:59:29 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42157BF9.9090302@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <qnkoeeiq3fk.fsf@arbutus.physics.mcmaster.ca> <4215152B.2070108@colorado.edu> <42151728.7010307@ucsd.edu> <42151848.301@colorado.edu> <42151A4A.9070200@ucsd.edu> <42151E39.9000409@cox.net> <42157BF9.9090302@ee.byu.edu>
Message-ID: <720e1fa453a30163c6d98b20324121df@embl-heidelberg.de>

> In that vein, I am proposing making X.flat an array iterator  and 
> allowing array iterators to be indexed and set as if they were 1-d 
> arrays with the underlying array being changed.  This is actually an 
> easy change with the current code base.  Will it break any code?
> There maybe some X.flats that need to be changed to ravel.  But it 
> seems like a really good idea.

That iterator would only work on the Python level I suppose, that would 
be effectively indistinguishable from an 1D array. But how would that 
work when passed to a function implemented in C? The function needs to 
know somehow that non-contiguous arrays need to be treated as 1D. That 
means having to code for such a special case (not a good idea), or you 
are back to making a copy.




From konrad.hinsen at laposte.net  Fri Feb 18 06:09:11 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 06:09:11 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <4bf0e99f809d592b068270a200b6deb0@laposte.net>

On Feb 17, 2005, at 19:52, Travis Oliphant wrote:

> I like Francesc's suggestion that .typecode return a code and .type 
> return a Python class.   What is the attitude and opinion regarding 
> the use of attributes or methods for this kind of thing?  It always 
> seems to me so arbitrary as to what is an attribute or what is a 
> method.

My view of pythonicity is that retrieving a value should be written as 
attribute access. Methods are more appropriate if there are arguments 
(no choice then anyway) or side effects. So I'd have .type as an 
attribute.

BTW, as the goal is inclusion into the Python core, why not

1) Use Python type objects for array creation and as the values of the 
.type
     attribute.
2) Implement scalar types for those array element types that currently
     have no Python scalar equivalent (e.g. UInt16).
3) Implement the same set of attributes of methods for scalar types and
    arrays.

Then the distinction between scalars and rank-0 arrays would become a 
minor implementation detail rather than a topic of heated debate. In 
different words, I propose that the PEP should include unification of 
scalars and arrays such that for all practical purposes scalars *are* 
rank-0 arrays.

> One thing has always bothered me though.  Why is a double complex type 
> Complex64? and a float complex type Complex32.  This seems to break 
> the idea that the number at the end specifies a bit width.   Why don't 
> we just call it Complex64 and Complex128?  Can we change this?

+1

> PowerPC it is Float128.   Wouldn't it just be easier to specify 
> LDouble or 'g' then special-case your code?

Definitely.

> Sometimes it is useful to select out of an array some elements based 
> on it's linear (flattened) index in the array.   MATLAB, for example, 
> will allow you to take a three-dimensional array and index it with a 
> single integer based on it's Fortran-order:  x(1,1,1),  x(2,1,1), ...

Could you give an example where this would be useful? To me this looks 
like a feature that MATLAB inherited from Fortran, which had it for 
efficiency reasons in a time when compilers were not so good at 
optimizing index expressions.

I don't lile the "special case" status of such a construct either. It 
could lead to unpleasant bugs that would be hard to find by those who 
are not aware of the special case. I'd say that special cases need 
special justifications - and I don't see one here.

> discontiguous arrays.   It could be made to work if X.flat returned 
> some kind of specially-marked array, which would then have to be 
> checked every time indexing occurred for any array.  Or, there maybe 
> someway to have X.flat return an

I much prefer that approach, assuming there is a real use for this 
feature.

> Capping indexes was proposed because of what numarray does.   I can 
> only think that the benefit would be that you don't have to check for 
> and raise an error in the middle of an indexing loop or pre-scan the 
> indexes.  But, I suppose this is unavoidalbe, anyway.  Currently 
> Numeric allows specifying indexes that are too high in slices. It just 
> chops them.  Python allows this too, for slices.  So, I guess I'm just 
> specifying Python behavior.  Of course indexing with an integer that 
> is too large or too small will raise errors:

I am all for imitating Python list behaviour in arrays, but we should 
also watch out for pitfalls. Array index expressions are in general 
much more complex than list index expressions, so the risk of 
introducing bugs is also much higher, which might well justify a 
somewhat incompatible approach.

> This may be a bit controversial as it is a bit of a change.  But, my 
> experience is that quite a bit of extra code is written to check 
> whether or not a calculation returns a Python-scalar (because these 
> don't have the same methods as arrays).  In

The only method I can think of is typecode(). But if more array 
functionality is migrated to arrays, this might become more serious.

> When Python needs a scalar it will generally ask the object if it can 
> turn itself into an int or a float.   A notable exception is indexing 
> in a list (where Python needs an integer and won't ask the object to 
> convert if it can).  But int(b) always returns a Python integer if the 
> array has only 1 element.

Still, this is a major point in practice. There was a Numeric release 
at some point in history that always returned rank-0 array objects (I 
don't remember if by design or by mistake), and it broke lots of my 
code because I was using array elements as indices.

Another compatibility issue is routines in C modules that insist on 
scalar arguments.

As I outlined above, I'd prefer a solution in which the distinction 
disappears from the Python programmer's point of view, even if scalars 
and rank-0 arrays remain distinct in the implementation (which is 
reasonable for performance reasons).

> I'd like to know what reasons people can think of for ever returning 
> Python scalars  unless explicitly asked for.

Other than the pragmatic ones, consistency: arrays are container 
structures that store elements of particular types. You should get out 
what you put in.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From konrad.hinsen at laposte.net  Fri Feb 18 06:31:17 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 06:31:17 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42151919.60607@enthought.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42151919.60607@enthought.com>
Message-ID: <ce60847bc35769e2b8287b737e998ee6@laposte.net>

On Feb 17, 2005, at 23:22, Duncan Child wrote:

> I have attached some code that illustrates part of the pain we have 
> experienced developing libraries of algorithms that can handle both 
> arrays and scalars. The attached library is the reusable part. The 
> other part of this problem is that we have lots of logic sprinkled 
> throughout our algorithms to enable them to handle both arrays and 
> scalars.

See comments below...

> Secondly, I have just been bitten by this declaration which suggests 
> that the new Numeric might handle default values better:.
>
> _vp_mod = zeros(num_pts)
>
> It would be less surprising to someone developing numeric algorithms 
> if functions like this defaulted to creating a double precision array 
> rather than integers.

If you want a function that returns float arrays, it is trivial to 
write and adds negligible overhead:

	def float_array(array_spec):
		return Numeric.array(array_spec, Numeric.Float)

No need to interfere with Numeric's principle of "smallest usable 
type", which fits well into the Python type promotion hierarchy.

More generally, I don't think defaults should be chosen with a 
particular application in mind. Arrays are a general and widely useful 
datatype in many domains. I use integer arrays as much as float arrays, 
even though my applications qualify as "numeric".

> """
> The following safe_ methods were written to handle both arrays amd 
> scalars to
> save the developer of numerical methods having to clutter their code 
> with tests
> to determine the type of the data.
> """
>
> def safe_take(a,indices):
>    # Slice the input if it is an array but not if it is a scalar

This is a very bad example. Your function does not interpret scalars as 
rank-0 arrays (for which take() would fail), but as something 
completely different.

>   def safe_copy(a):
>    # Return a copy for both scalar and array input

That is a semantically reasonable application, but also one for which a 
simple and standard solution already exists: copy.copy().

> def safe_min(a):
>    # Return the minimum of the input array or the input if it is a 
> scalar

I would argue that this is not a good example either, as "minimum over 
an array" implies a reduction operation which is not defined for a 
scalar. On the other hand, the operation you define certainly makes 
sense.

> def safe_len(a):
>    # Return the length of the input array or 1 if it is a scalar

That implies that a scalar is somehow equivalent to a rank-1 array of 
length 1, which is not the case.

Actually, all of your examples look like an attempt to recreate Matlab 
behaviour. But Python is not Matlab!

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From barrett at stsci.edu  Fri Feb 18 06:56:19 2005
From: barrett at stsci.edu (Paul Barrett)
Date: Fri Feb 18 06:56:19 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <28c54bbc706b589b401f14cb4672c918@stsci.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4214FAD7.3030109@fastmail.fm> <4214FDCD.7000901@ee.byu.edu> <28c54bbc706b589b401f14cb4672c918@stsci.edu>
Message-ID: <421601EA.9080503@stsci.edu>

Perry Greenfield wrote:

>
> On Feb 17, 2005, at 3:25 PM, Travis Oliphant wrote:
>
>>
>> This is basically new behavior that numarray has started supporting.  
>> I just think numarray missed an important case of flattened indexing 
>> that MATLAB supports.  My current proposal would distinguish between 
>> single-index array cases and tuple-index array cases.
>> I'm still thinking about the X.flat possibility.  Basically, I think 
>> that direction would requires a new "generic array view" or something 
>> like that.  It may be worth trying, but I'm not sure I want to go 
>> that direction right now until I convince more people to come on 
>> board with Numeric3.
>>
> It was new behavior in the sense that Numeric didn't support 
> multidimensional array takes and puts at the time. For a long time it 
> was the only kind of array indexing IDL supported (1-d and implicit 
> .flat of multidimensional arrays). Speaking only for myself, I found 
> the .flat semantics more often as unwanted behavior than convenient. 
> I'd rather keep the numarray behavior in this case and make the .flat 
> case explicit (but I understand the difficulty of that approach). 
> There is the possibility of a custom index option (but ugly I suppose)
>
> X[K, flatten]
>
> where flatten is a special object that indexing recognizes as 
> signaling a different interpretation to indexing.


+1 on  Travis's suggestion of X.flat[K] behavior, since I think it is 
explicit and intuitive.

- 1 on X[K. flatten]

I think in the long term that the X.flat[K] proposal should be pursued, 
even for non-contiguous arrays. Since we have decided that slice-as-view 
behavior is the default, then I believe the user should not have to 
worry whether the internal structure of an array is continguous or not.

 -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218




From barrett at stsci.edu  Fri Feb 18 07:02:16 2005
From: barrett at stsci.edu (Paul Barrett)
Date: Fri Feb 18 07:02:16 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <87oeej6xxh.fsf@welho.com>
References: <421446E3.7080202@ee.byu.edu> <87oeej6xxh.fsf@welho.com>
Message-ID: <4216034A.7020705@stsci.edu>

Timo Korvola wrote:

>Travis Oliphant <oliphant at ee.byu.edu> writes:
>  
>
>>except for the fact that X[K] where K is a a single
>>integer index array can do 1-d indexing similar to MATLAB.
>>    
>>
[snip,  snip]

>Perhaps only one ellipsis should be allowed for clarity?  Is it useful
>to allow more if they are only equivalent to colons?
>  
>
This is the case anyway, since the ellipsis operator is greedy and 
swallows up all unspecified indices. Any later ellipsis operators should 
be ignored, otherwise this situation is ambiguous.

 -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218




From barrett at stsci.edu  Fri Feb 18 07:14:13 2005
From: barrett at stsci.edu (Paul Barrett)
Date: Fri Feb 18 07:14:13 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <87zmy26gx6.fsf@welho.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>
Message-ID: <42160609.3070402@stsci.edu>

Timo Korvola wrote:

[snip, snip]

>>3)  Always returning rank-0 arrays.
>>
>>This may be a bit controversial as it is a bit of a change.
>>    
>>
>
>Indeed.  So you really do intend that if foo=array([1,2]), foo[0]
>should evaluate to array(1) rather than 1?
>
>  
>
>>But, my experience is that quite a bit of extra code is written to
>>check whether or not a calculation returns a Python-scalar
>>    
>>
>
>I suppose this may be necessary for code which operates on arrays of
>somewhat arbitrary rank and would not know without looking whether,
>e.g., foo[0] is a scalar or an array of positive rank.
>
>  
>
>>In particular len(a) does not work if a is a scalar,
>>    
>>
>
>Depends on what kinds of scalars are supported.  What about object
>arrays?
>
>  
>
>>but len(b) works if b is a rank-0 array
>>
How about proposing a PEP to extend Python's scalar behavior, so the 
len(a) works for either scalars or arrays.  Though I haven't thought 
this through in great detail, it would appear to be a transparent 
addition for most Python users who would never use this.  At the same 
time, we should also consider other behavior that would unify (or smear) 
the behavior of Python scalars and arrays.

 -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218




From rkern at ucsd.edu  Fri Feb 18 07:24:22 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Fri Feb 18 07:24:22 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42160609.3070402@stsci.edu>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42160609.3070402@stsci.edu>
Message-ID: <42160877.1070402@ucsd.edu>

Paul Barrett wrote:

> How about proposing a PEP to extend Python's scalar behavior, so the 
> len(a) works for either scalars or arrays.  Though I haven't thought 
> this through in great detail, it would appear to be a transparent 
> addition for most Python users who would never use this.  At the same 
> time, we should also consider other behavior that would unify (or smear) 
> the behavior of Python scalars and arrays.

I'm sure there are any number of people who use len(x) as a way to test 
the sequenceness of x. While it might be okay for rank-0 arrays, 
extending this to builtin ints and floats may not be a good idea. I'm 
pretty sure this would get rejected.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From konrad.hinsen at laposte.net  Fri Feb 18 08:00:14 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 08:00:14 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42160877.1070402@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42160609.3070402@stsci.edu> <42160877.1070402@ucsd.edu>
Message-ID: <82562984f3b12d2e39ddc05df631d47c@laposte.net>

On Feb 18, 2005, at 16:23, Robert Kern wrote:

> I'm sure there are any number of people who use len(x) as a way to 
> test the sequenceness of x. While it might be okay for rank-0 arrays, 
> extending this to builtin ints and floats may not be a good idea. I'm 
> pretty sure this would get rejected.

For arrays, len(x) == x.shape[0], so len(x) should fail for rank-0 
arrays anyway. As it does in Numeric.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From perry at stsci.edu  Fri Feb 18 08:13:18 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Fri Feb 18 08:13:18 2005
Subject: [Numpy-discussion] Some comments on the draft PEP (Rev 1.8)
In-Reply-To: <42155814.4040908@sympatico.ca>
References: <42155814.4040908@sympatico.ca>
Message-ID: <622f2e967fe6a234c016983cb64f6918@stsci.edu>

On Feb 17, 2005, at 9:51 PM, Colin J. Williams wrote:
>
> One final request.             In the instance constructor, please 
> avoid the use of 'type' as a parameter name if the constructor is 
> implemented in Python.  This hides the builtin type(), which is 
> sometimes useful in debugging or exploring code.
>

It shouldn't be a problem for those using the constructor in their 
code. It is a problem if one is writing a function where one wants to 
use the same parameter name (to retain an identical case) in which case 
one is masking the built-in (and thus it is necessary to alias the 
built-in). If type is not to be used I think an alternative to typecode 
should be available as that would be a misnomer if the underlying type 
is not a character. One possible alternative is "atype". Any other 
suggestions?

Perry




From tkorvola at welho.com  Fri Feb 18 08:20:19 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Fri Feb 18 08:20:19 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <4216034A.7020705@stsci.edu> (Paul Barrett's message of "Fri,
 18 Feb 2005 10:01:30 -0500")
References: <421446E3.7080202@ee.byu.edu> <87oeej6xxh.fsf@welho.com>
	<4216034A.7020705@stsci.edu>
Message-ID: <877jl5g7sb.fsf@welho.com>

Paul Barrett <barrett at stsci.edu> writes:
> Any later ellipsis operators should be ignored, otherwise this
> situation is ambiguous.

The PEP says they'll turn into colons, which is not the same as being
ignored.  It only describes advanced indexing though, which is very
different from basic indexing.  In advanced indexing an ellipsis would
pick diagonal entries, e.g., a[NewAxis, ...][[0], ...] would return
the diagonal of a.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From duncan at enthought.com  Fri Feb 18 08:50:19 2005
From: duncan at enthought.com (Duncan Child)
Date: Fri Feb 18 08:50:19 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <ce60847bc35769e2b8287b737e998ee6@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42151919.60607@enthought.com> <ce60847bc35769e2b8287b737e998ee6@laposte.net>
Message-ID: <42161CA3.40506@enthought.com>

konrad.hinsen at laposte.net wrote:

> On Feb 17, 2005, at 23:22, Duncan Child wrote:
>
>> I have attached some code that illustrates part of the pain we have 
>> experienced developing libraries of algorithms that can handle both 
>> arrays and scalars. The attached library is the reusable part. The 
>> other part of this problem is that we have lots of logic sprinkled 
>> throughout our algorithms to enable them to handle both arrays and 
>> scalars.
>
>
> See comments below...
>
>> Secondly, I have just been bitten by this declaration which suggests 
>> that the new Numeric might handle default values better:.
>>
>> _vp_mod = zeros(num_pts)
>>
>> It would be less surprising to someone developing numeric algorithms 
>> if functions like this defaulted to creating a double precision array 
>> rather than integers.
>
>
> If you want a function that returns float arrays, it is trivial to 
> write and adds negligible overhead:
>
>     def float_array(array_spec):
>         return Numeric.array(array_spec, Numeric.Float)
>
> No need to interfere with Numeric's principle of "smallest usable 
> type", which fits well into the Python type promotion hierarchy.
>
> More generally, I don't think defaults should be chosen with a 
> particular application in mind. Arrays are a general and widely useful 
> datatype in many domains. I use integer arrays as much as float 
> arrays, even though my applications qualify as "numeric".

What I meant was that accidentally ommitting the na.Float in the 
declaration below introduced a hard to find bug in my code.

_vp_mod = na.zeros(num_pts, na.Float)

I had not heard of Numeric's "smallest usable type" principle. Even so, 
I would argue that for doing signal processing the smallest usable type 
is floating point :-)

> snipped >>
>
> Actually, all of your examples look like an attempt to recreate Matlab 
> behaviour. But Python is not Matlab!

Good point, and this code was actually written by developers who were 
porting libraries of Matlab code. I thought the examples illustrated a 
more general problem that was created by Numeric handling scalars 
differently to arrays.

In another post you said:

"... as the goal is inclusion into the Python core .... I propose that 
the PEP should include unification of scalars and arrays such that for 
all practical purposes scalars *are* rank-0 arrays. "

So I think we are in broad agreement.

Regards

Duncan




>
> Konrad.
> -- 
> ---------------------------------------------------------------------
> Konrad Hinsen
> Laboratoire L?on Brillouin, CEA Saclay,
> 91191 Gif-sur-Yvette Cedex, France
> Tel.: +33-1 69 08 79 25
> Fax: +33-1 69 08 82 61
> E-Mail: hinsen at llb.saclay.cea.fr
> ---------------------------------------------------------------------
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_ide95&alloc_id396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>





From rkern at ucsd.edu  Fri Feb 18 09:28:23 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Fri Feb 18 09:28:23 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <82562984f3b12d2e39ddc05df631d47c@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42160609.3070402@stsci.edu> <42160877.1070402@ucsd.edu> <82562984f3b12d2e39ddc05df631d47c@laposte.net>
Message-ID: <42162562.9080102@ucsd.edu>

konrad.hinsen at laposte.net wrote:
> On Feb 18, 2005, at 16:23, Robert Kern wrote:
> 
>> I'm sure there are any number of people who use len(x) as a way to 
>> test the sequenceness of x. While it might be okay for rank-0 arrays, 
>> extending this to builtin ints and floats may not be a good idea. I'm 
>> pretty sure this would get rejected.
> 
> 
> For arrays, len(x) == x.shape[0], so len(x) should fail for rank-0 
> arrays anyway. As it does in Numeric.

I'm not averse to len(x) returning 0 when given a rank-0 array. I see it 
as giving up one consistency (that scalar-like objects don't have 
lengths) for another (arrays having a common set of operations that one 
can expect regardless of rank or shape).

My objection was to extending that set of operations to other standard 
objects where they make less sense. Although the len(x) 
sequenceness-test is a reasonably common idiom, it's not expected to be 
foolproof against any input. However, the test shouldn't stop working on 
core objects that are already there.

In any case, len(x) is probably one of the less-common operations one 
would want to perform seamlessly on scalar and rank-n outputs from 
Numeric functions. x.typecode() and x.shape would probably top my list.

Might our PEP efforts be better spent locating and fixing the places in 
core Python where rank-0 arrays won't be accepted as core ints and 
floats? List/tuple indexing is one. Extension code that demands an 
actual int object where it could be looser might already be considered 
deficient considering the int/long unification.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From oliphant at ee.byu.edu  Fri Feb 18 09:42:29 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 09:42:29 2005
Subject: [Numpy-discussion] Multiarray PEP
In-Reply-To: <877jl5g7sb.fsf@welho.com>
References: <421446E3.7080202@ee.byu.edu> <87oeej6xxh.fsf@welho.com>	<4216034A.7020705@stsci.edu> <877jl5g7sb.fsf@welho.com>
Message-ID: <421628D2.4020608@ee.byu.edu>

Timo Korvola wrote:

>Paul Barrett <barrett at stsci.edu> writes:
>  
>
>>Any later ellipsis operators should be ignored, otherwise this
>>situation is ambiguous.
>>    
>>
>
>The PEP says they'll turn into colons, which is not the same as being
>ignored. 
>
This is the current default   behavior for Numeric standard slicing, so 
I see this as a continuation
of previous behavior.

I guess it should be more accurately stated, however, because two 
ellipses right next to each other will always be the same as one ellipse. 
Only if they are separated by something else will another ellipse be 
interpreted as a single colon.

I'm happy to raise an error instead if people have not been using this 
"feature"

-Travis




From konrad.hinsen at laposte.net  Fri Feb 18 09:51:19 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 09:51:19 2005
Subject: [Numpy-discussion] Multiarray PEP: iterators
Message-ID: <d9952a7aa4628a84a86727a753cfd909@laposte.net>

The PEP says:

   Iterator

       An iterator will be defined that will walk through any array,
       returning a rank-0 array at each step.  Rank-0 arrays act like
       the appropriate Python scalar and will be converted to one
       whenever Python asks the object explicitly to try and do so.
       Order of the iteration is the same for contiguous and
       discontiguous arrays.  The last index always varies the fastest


At the moment, iteration over a rank-N array yields rank-(N-1) arrays 
(indexing along the first dimension). I consider this vastly more 
useful than iterating over the rank-0 elements and thus ignoring the 
array structure completely.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Laboratoire L?on Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: hinsen at llb.saclay.cea.fr
---------------------------------------------------------------------




From oliphant at ee.byu.edu  Fri Feb 18 10:00:24 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 10:00:24 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4bf0e99f809d592b068270a200b6deb0@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net>
Message-ID: <42162CEC.3090607@ee.byu.edu>

konrad.hinsen at laposte.net wrote:

> My view of pythonicity is that retrieving a value should be written as 
> attribute access. Methods are more appropriate if there are arguments 
> (no choice then anyway) or side effects. So I'd have .type as an 
> attribute.

That's my view as well.

>
> BTW, as the goal is inclusion into the Python core, why not
>
> 1) Use Python type objects for array creation and as the values of the 
> .type
>     attribute.

Do you mean have register each of the 21 different types of arrays as a 
new type object?   Hmm.   That is an interesting idea.   I'm a little 
worried about implications for having the arrays behave together, though 
it would make it more straightforward to define specific mixed 
operations.    This does deserve a little more thought.

> 2) Implement scalar types for those array element types that currently
>     have no Python scalar equivalent (e.g. UInt16).

Do you think this would fly with the Python folks.  Counting the 
suggestion above, we would be encouraging the creation of 39 new types 
to the Python core.  My current count shows the current number of types 
as 35 so we would basically double that.   This doesn't have to matter, 
but I'd have to hear how Guido feels about something like that.

> 3) Implement the same set of attributes of methods for scalar types and
>    arrays.

That would be ideal.   But, I'm not sure what kind of chance we have 
with that.

-Travis




From Fernando.Perez at colorado.edu  Fri Feb 18 10:03:13 2005
From: Fernando.Perez at colorado.edu (Fernando.Perez at colorado.edu)
Date: Fri Feb 18 10:03:13 2005
Subject: [Numpy-discussion] Multiarray PEP: iterators
In-Reply-To: <d9952a7aa4628a84a86727a753cfd909@laposte.net>
References: <d9952a7aa4628a84a86727a753cfd909@laposte.net>
Message-ID: <1108749739.42162dab6219e@webmail.colorado.edu>

Quoting konrad.hinsen at laposte.net:

> At the moment, iteration over a rank-N array yields rank-(N-1) arrays
> (indexing along the first dimension). I consider this vastly more
> useful than iterating over the rank-0 elements and thus ignoring the
> array structure completely.

Completely agreed.  I'm sure I'm not the only one who has code like

for row in matrix:
  do_something_with(row)

Furthermore, this is consistent with python's behavior for nested lists:

In [3]: nested=[['first row',1,2],['second row',3,4]]

In [4]: for row in nested:
   ...:     print row
   ...:
['first row', 1, 2]
['second row', 3, 4]

Python does not flatten nested structures when iterating over them, neither
should numerix.  I think the addition (which Travis seems keen on) the .flat
iterator attribute is the right approach here.  Normal iterators would maintain
structure (and behave like pyhon's for nested lists), while .flat would return
an element-wise iterator for cases where such a thing is needed (which do
exist, obviously).

Regards,

f



From tkorvola at welho.com  Fri Feb 18 11:03:18 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Fri Feb 18 11:03:18 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42162562.9080102@ucsd.edu> (Robert Kern's message of "Fri, 18
 Feb 2005 09:26:58 -0800")
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>
	<42160609.3070402@stsci.edu> <42160877.1070402@ucsd.edu>
	<82562984f3b12d2e39ddc05df631d47c@laposte.net>
	<42162562.9080102@ucsd.edu>
Message-ID: <87wtt5elnw.fsf@welho.com>

Robert Kern <rkern at ucsd.edu> writes:
> I'm not averse to len(x) returning 0 when given a rank-0 array. I see
> it as giving up one consistency (that scalar-like objects don't have
> lengths) for another (arrays having a common set of operations that
> one can expect regardless of rank or shape).

That would be akin to making 0/0 return 0.  It is possible to create
arrays of length zero, e.g, by indexing with an empty slice, and these
should not be confused with rank zero arrays.

> Might our PEP efforts be better spent locating and fixing the places
> in core Python where rank-0 arrays won't be accepted as core ints and
> floats?

Sounds useful.

> List/tuple indexing is one.

Rank 0 arrays should also be kept in mind when defining array indexing
in the PEP.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From oliphant at ee.byu.edu  Fri Feb 18 11:13:16 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 11:13:16 2005
Subject: [Numpy-discussion] Proposed X.flat solution
Message-ID: <42163DED.7080808@ee.byu.edu>

O.K.  here is my X.flat solution.

I've decided to make use of the fancy-new, numarray-inspired idea of the 
UPDATEIFCOPY flag for array creation.  Basically, X.flat will create a 
new array if X is discontiguous but with the UPDATEIFCOPY flag set so 
that upon destruction the contents will get copied back to X. 

So, it will seem like the X.flat array is updating the original array 
(provided it get's dereferenced someday (which it should because you 
rarely) keep references to X.flat unless you really meant ravel(X).

This is the simplest solution.  It's not quite as fancy as the indexable 
iterator (which could still be implemented at some point) but it lets me 
move forward.

-Travis




From Chris.Barker at noaa.gov  Fri Feb 18 11:14:18 2005
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Fri Feb 18 11:14:18 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4214E7FE.9090403@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu>
Message-ID: <42163E14.9050700@noaa.gov>

Travis Oliphant wrote:

> I'm still thinking about the X.flat possibility.

There was a discussion about his a while back, but I don't remember what 
conclusions were reached (if any). The technical details are beyond me, 
but it seems quite clear to me that A.flat() should be supported for all 
arrays. Otherwise, it makes it very hard to write generic code. What's 
the point of having discontiguous arrays if you can't use them 
everywhere you can use contiguous ones?


Fernando Perez wrote:
> Granted, the indexing will be costlier for a non-contiguous object, but 
> for the external users this provides a clean API.

Right. A number of things are costlier for non-contiguous arrays. If you 
need to optimize, you can make sure your arrays are contiguous where it 
matters. If you don't, it should "just work"


> But Python is not Matlab!
> 
> Konrad.

hear! hear!. If I wanted Matlab, I'd use Matlab (or Octave, or Psilab)







-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From rkern at ucsd.edu  Fri Feb 18 11:14:21 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Fri Feb 18 11:14:21 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <87wtt5elnw.fsf@welho.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>	<42160609.3070402@stsci.edu> <42160877.1070402@ucsd.edu>	<82562984f3b12d2e39ddc05df631d47c@laposte.net>	<42162562.9080102@ucsd.edu> <87wtt5elnw.fsf@welho.com>
Message-ID: <42163E66.9040908@ucsd.edu>

Timo Korvola wrote:
> Robert Kern <rkern at ucsd.edu> writes:
> 
>>I'm not averse to len(x) returning 0 when given a rank-0 array. I see
>>it as giving up one consistency (that scalar-like objects don't have
>>lengths) for another (arrays having a common set of operations that
>>one can expect regardless of rank or shape).
> 
> 
> That would be akin to making 0/0 return 0.  It is possible to create
> arrays of length zero, e.g, by indexing with an empty slice, and these
> should not be confused with rank zero arrays.

Fair enough. I'm not averse to len(x) raising an exception when x is a 
rank-0 array either.  :-)

>>Might our PEP efforts be better spent locating and fixing the places
>>in core Python where rank-0 arrays won't be accepted as core ints and
>>floats?
> 
> 
> Sounds useful.
> 
> 
>>List/tuple indexing is one.
> 
> 
> Rank 0 arrays should also be kept in mind when defining array indexing
> in the PEP.

Hopefully, this would drop out cleanly from the regular array-indexing 
and not require too much in the way of special-casing.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From rlw at stsci.edu  Fri Feb 18 11:32:21 2005
From: rlw at stsci.edu (Rick White)
Date: Fri Feb 18 11:32:21 2005
Subject: [Numpy-discussion] Proposed X.flat solution
In-Reply-To: <42163DED.7080808@ee.byu.edu>
Message-ID: <Pine.GSO.4.44.0502181424000.1023-100000@sundog.stsci.edu>

On Fri, 18 Feb 2005, Travis Oliphant wrote:

> O.K.  here is my X.flat solution.
>
> I've decided to make use of the fancy-new, numarray-inspired idea of the
> UPDATEIFCOPY flag for array creation.  Basically, X.flat will create a
> new array if X is discontiguous but with the UPDATEIFCOPY flag set so
> that upon destruction the contents will get copied back to X.
>
> So, it will seem like the X.flat array is updating the original array
> (provided it get's dereferenced someday (which it should because you
> rarely) keep references to X.flat unless you really meant ravel(X).
>
> This is the simplest solution.  It's not quite as fancy as the indexable
> iterator (which could still be implemented at some point) but it lets me
> move forward.

Hmm.  It seems to me that if it is not a live view you can run into
serious problems.  What happens if you do this:

a = zeros((10,10))
b = a[::2,::2].flat
b[4] = 1
a[0,8] = 2
del b

Doesn't your behavior lead to a[0,8] being overwritten to 1 by the copy
back from b?




From barrett at stsci.edu  Fri Feb 18 11:52:20 2005
From: barrett at stsci.edu (Paul Barrett)
Date: Fri Feb 18 11:52:20 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42163E66.9040908@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>	<42160609.3070402@stsci.edu> <42160877.1070402@ucsd.edu>	<82562984f3b12d2e39ddc05df631d47c@laposte.net>	<42162562.9080102@ucsd.edu> <87wtt5elnw.fsf@welho.com> <42163E66.9040908@ucsd.edu>
Message-ID: <4216473A.3090909@stsci.edu>

Robert Kern wrote:

> Timo Korvola wrote:
>
>> Robert Kern <rkern at ucsd.edu> writes:
>>
>>> I'm not averse to len(x) returning 0 when given a rank-0 array. I see
>>> it as giving up one consistency (that scalar-like objects don't have
>>> lengths) for another (arrays having a common set of operations that
>>> one can expect regardless of rank or shape).
>>
>>
>>
>> That would be akin to making 0/0 return 0.  It is possible to create
>> arrays of length zero, e.g, by indexing with an empty slice, and these
>> should not be confused with rank zero arrays.
>
>
> Fair enough. I'm not averse to len(x) raising an exception when x is a 
> rank-0 array either.  :-)

Or it could return 'None', i.e. undefined.

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218




From oliphant at ee.byu.edu  Fri Feb 18 12:29:21 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 12:29:21 2005
Subject: [Numpy-discussion] Proposed X.flat solution
In-Reply-To: <Pine.GSO.4.44.0502181424000.1023-100000@sundog.stsci.edu>
References: <Pine.GSO.4.44.0502181424000.1023-100000@sundog.stsci.edu>
Message-ID: <42164FF6.5010801@ee.byu.edu>

>Hmm.  It seems to me that if it is not a live view you can run into
>serious problems.  What happens if you do this:
>
>a = zeros((10,10))
>b = a[::2,::2].flat
>b[4] = 1
>a[0,8] = 2
>del b
>
>Doesn't your behavior lead to a[0,8] being overwritten to 1 by the copy
>back from b?
>  
>
Well in this particular case since a is contiguous to begin with, b is 
not a copy so the copy-back would not occur.

But, you bring up a valid point that if "a" were discontiguous which 
forced a copy, then any later modifications to "a" would be ignored.
Now, this could be fixed by making "a" read-only for the duration of 
existence of the flattened reference. 

In fact, it probably stands to reason that whenever, UPDATEIFCOPY is 
used on the C-level, the to-be-copied-to array is flagged readonly for the
duration of existence of the "reference."

Besides that point, to clarify what we are talking about here:

1)  The PEP has been modified so that X[K] is interpreted as X[K,]  for 
integer arrays, and the previously-discussed distinction disappears.

2)  It is desired that X.flat[K]  provide the flattened behavior.   How 
should this be done?

     a) having X.flat return an indexable iterator (with an __array__ 
method that will automatically return an update-if-copy array on
         request from asarray() or C-level equivalent)

     b) having X.flat return an update-if-copy array (with X flagged as 
readonly while a reference to X.flat exists)

     c) it shouldn't be done and X.flat should just raise an error if X 
is not contiguous.

Please chime in and "vote" for one of the three solutions (or another 
one not covered) for 2 if you haven't already expressed your view.

By the way, I think it's been settled that X.flat will not behave the 
same as ravel(X) (which should probably be a method---X.ravel() ).

-Travis




From cookedm at physics.mcmaster.ca  Fri Feb 18 13:12:10 2005
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Fri Feb 18 13:12:10 2005
Subject: [Numpy-discussion] Proposed X.flat solution
In-Reply-To: <42164FF6.5010801@ee.byu.edu> (Travis Oliphant's message of
 "Fri, 18 Feb 2005 13:28:38 -0700")
References: <Pine.GSO.4.44.0502181424000.1023-100000@sundog.stsci.edu>
	<42164FF6.5010801@ee.byu.edu>
Message-ID: <qnkhdk9po93.fsf@arbutus.physics.mcmaster.ca>

Travis Oliphant <oliphant at ee.byu.edu> writes:

>>Hmm.  It seems to me that if it is not a live view you can run into
>>serious problems.  What happens if you do this:
>>
>>a = zeros((10,10))
>>b = a[::2,::2].flat
>>b[4] = 1
>>a[0,8] = 2
>>del b
>>
>>Doesn't your behavior lead to a[0,8] being overwritten to 1 by the copy
>>back from b?
>>
>>
> Well in this particular case since a is contiguous to begin with, b is
> not a copy so the copy-back would not occur.
>
> But, you bring up a valid point that if "a" were discontiguous which
> forced a copy, then any later modifications to "a" would be ignored.
> Now, this could be fixed by making "a" read-only for the duration of
> existence of the flattened reference.

> In fact, it probably stands to reason that whenever, UPDATEIFCOPY is
> used on the C-level, the to-be-copied-to array is flagged readonly for
> the duration of existence of the "reference."

Ugh. I realize I'm going to have to sit down and think on this first,
and look at code. My gut feeling is UPDATEIFCOPY and read-write-locks
(which is essentially what you're proposing) will make for some odd,
hard-to-track down, errors.

> Besides that point, to clarify what we are talking about here:
>
> 1)  The PEP has been modified so that X[K] is interpreted as X[K,]
> for integer arrays, and the previously-discussed distinction
> disappears.
>
> 2)  It is desired that X.flat[K]  provide the flattened behavior.
> How should this be done?
>
>      a) having X.flat return an indexable iterator (with an __array__
> method that will automatically return an update-if-copy array on
>          request from asarray() or C-level equivalent)

I'm +1. This feels to be the simplest (in sense of use, not necessarily
implementation :-), and the obvious use. Here, X.flat is *always* a
valid view of X.

>      b) having X.flat return an update-if-copy array (with X flagged
> as readonly while a reference to X.flat exists)

-0, for the locking reasons I mention above. For this usage, I'd make
it a method instead.

>      c) it shouldn't be done and X.flat should just raise an error if
> X is not contiguous.

If this one, then I'd say that X.flat is useless and should be
removed. ravel() (or X.ravel()) should return a view or copy depending
on whether X is contiguous or not. A copy could be forced by a keyword
argument: ravel(X, copy=True)

While we're at it, I'd argue that "ravel" is a bad word for this
usage -- it's not obvious what it does. "flatten" might be a better word.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca



From oliphant at ee.byu.edu  Fri Feb 18 13:25:22 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 13:25:22 2005
Subject: [Numpy-discussion] Proposed X.flat solution
In-Reply-To: <qnkhdk9po93.fsf@arbutus.physics.mcmaster.ca>
References: <Pine.GSO.4.44.0502181424000.1023-100000@sundog.stsci.edu>	<42164FF6.5010801@ee.byu.edu> <qnkhdk9po93.fsf@arbutus.physics.mcmaster.ca>
Message-ID: <42165CF3.8070104@ee.byu.edu>

David M. Cooke wrote:

>>In fact, it probably stands to reason that whenever, UPDATEIFCOPY is
>>used on the C-level, the to-be-copied-to array is flagged readonly for
>>the duration of existence of the "reference."
>>    
>>
>
>Ugh. I realize I'm going to have to sit down and think on this first,
>and look at code. My gut feeling is UPDATEIFCOPY and read-write-locks
>(which is essentially what you're proposing) will make for some odd,
>hard-to-track down, errors.
>  
>
The problem is with this idea of UPDATEIFCOPY which is borrowed from 
numarray and which users over there seem to like.   It just seems to
me that if an array is going to be "automatically" copied back to 
another array (this is currently done in numarray), then the 
array-to-be-copied to
needs to be flagged read-only until the copy is complete.  Then, an 
error will be raised whenever the users tries to do something that would
cause "hard-to-track-down" bugs (like write to an array that is going to 
be clobbered by a copy-back).  

If the UPDATEIFCOPY is only used for C-extensions where you need to get 
a nice array from a potentially "bad" one (e.g. contiguous from 
discontiguous) and the thinking is that you would not be using the 
original one while you played with the copy.  But, if the C-extension 
calls arbitrary code (as is sometimes done), then this is not 
guaranteed.  So, I think, if you are going to have UPDATEIFCOPY, you 
need to set the original array readonly until the copy is complete.


>  
>
>>     b) having X.flat return an update-if-copy array (with X flagged
>>as readonly while a reference to X.flat exists)
>>    
>>
>
>  
>
Yeah, I don't like this anymore either. 


I like X.flatten() better than X.ravel() too. 

-Travis





From konrad.hinsen at laposte.net  Fri Feb 18 13:44:27 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 13:44:27 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42162CEC.3090607@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu>
Message-ID: <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net>

On 18.02.2005, at 18:59, Travis Oliphant wrote:

> Do you mean have register each of the 21 different types of arrays as  
> a new type object?   Hmm.   That is an interesting idea.   I'm a  
> little worried

Yes. It only costs a bit of memory for the type objects, so why not?

>> 2) Implement scalar types for those array element types that currently
>>     have no Python scalar equivalent (e.g. UInt16).
>
> Do you think this would fly with the Python folks.  Counting the  
> suggestion above, we would be encouraging the creation of 39 new types  
> to the Python core.  My current count shows the current number of  
> types as 35 so

Those new types would (if all goes well) be part of the standard  
library, but not built-in types. Compared to the number of types and  
classes in the standard library, the addition is not so big. There  
wouldn't be literals either. Anyone who doesn't use the array module  
could thus safely ignore the existence of those types.

Implementation-wise, the new types could well be rank-0 arrays  
internally and thus add nearly no overhead.

>  we would basically double that.   This doesn't have to matter, but  
> I'd have to hear how Guido feels about something like that.

Of course!

>> 3) Implement the same set of attributes of methods for scalar types  
>> and
>>    arrays.
>
> That would be ideal.   But, I'm not sure what kind of chance we have  
> with that.

Me neither. It might depend on clever presentation of the project.  
Perhaps some bribing could help ;-)

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From stephen.walton at csun.edu  Fri Feb 18 13:59:24 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Fri Feb 18 13:59:24 2005
Subject: [Numpy-discussion] Multiarray PEP: iterators
In-Reply-To: <1108749739.42162dab6219e@webmail.colorado.edu>
References: <d9952a7aa4628a84a86727a753cfd909@laposte.net> <1108749739.42162dab6219e@webmail.colorado.edu>
Message-ID: <421664DF.1090409@csun.edu>

Fernando.Perez at colorado.edu wrote:

>Completely agreed.  I'm sure I'm not the only one who has code like
>
>for row in matrix:
>  do_something_with(row)
>  
>
You're not.  I vote +1 for keeping this behavior.




From stephen.walton at csun.edu  Fri Feb 18 14:03:14 2005
From: stephen.walton at csun.edu (Stephen Walton)
Date: Fri Feb 18 14:03:14 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42161CA3.40506@enthought.com>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42151919.60607@enthought.com> <ce60847bc35769e2b8287b737e998ee6@laposte.net> <42161CA3.40506@enthought.com>
Message-ID: <421665E1.6000008@csun.edu>

Duncan Child wrote:

> In another post [Konrad Hinsen] said:
>
> "... as the goal is inclusion into the Python core .... I propose that 
> the PEP should include unification of scalars and arrays such that for 
> all practical purposes scalars *are* rank-0 arrays. "
>
> So I think we are in broad agreement.

It seems to me that actually getting the above behavior is going to 
require Guido van Rossum et al. to change how scalars behave in the 
Python core.  As I said, right now min(scalar) in stock Python raises a 
TypeError on the grounds that scalars can't be iterated over.




From oliphant at ee.byu.edu  Fri Feb 18 14:07:24 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 14:07:24 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net>
Message-ID: <421666F3.6050702@ee.byu.edu>

>> Do you mean have register each of the 21 different types of arrays 
>> as  a new type object?   Hmm.   That is an interesting idea.   I'm a  
>> little worried
>
> Yes. It only costs a bit of memory for the type objects, so why not?

Let me see if i understand you here.  Since the type of the array is 
bascially handled by the
PyArray_Descr*  structure.   Are you suggesting turning that into a 
full-fledged PythonObject with
new type?   I had thought of this before since it's basically calling 
for us to do something like that.  Or do you have something else in mind?

-Travis




From oliphant at ee.byu.edu  Fri Feb 18 14:46:10 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 14:46:10 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4bf0e99f809d592b068270a200b6deb0@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net>
Message-ID: <42166FF3.9050908@ee.byu.edu>

>> When Python needs a scalar it will generally ask the object if it can 
>> turn itself into an int or a float.   A notable exception is indexing 
>> in a list (where Python needs an integer and won't ask the object to 
>> convert if it can).  But int(b) always returns a Python integer if 
>> the array has only 1 element.
>
>
> Still, this is a major point in practice. There was a Numeric release 
> at some point in history that always returned rank-0 array objects (I 
> don't remember if by design or by mistake), and it broke lots of my 
> code because I was using array elements as indices.
>
I posted a question to python-dev about changing the internals of Python 
to support asking objects
to convert to ints in slicing. 

While open to the idea for Python 3000, Guido does not seem favorable to 
the idea for Python 2.X The problem, Guido mentions was that float-like 
objects can convert to ints by truncation and he doesn't want to allow 
floats to be used as indexes.   He feels it would break too much code.

Using this line of reasoning, then, arrays should not be used as indexes 
unless they are explicitly converted to integers:  int(a)

I have proposed a second solution that asks if a special check could be 
made for rank-0 arrayobjects (of integer type) if and when they are 
allowed in the core.


I think Konrad's valid point regarding consistency is that to the user 
it looks like he is making an array of integers

a = array([1,2,3,4])

so it is confusing (the first time) if

a[0] fails to act like an integer when requested for slicing an array.

Of course underneath, a is not an array of integers (it is an array of 
homogeneous c-ints converted from the Python integer and so why should 
a[0] be a Python integer). 

This is the problem.  We want different things to act the same all the 
time when fundamentally they are different.   Python allows this in many 
cases, but doesn't seem to be fully transparent in this regard.

When I first started with Python, as a MATLAB user I was confused by the 
fact that lists, tuples, and arrays were all different things that had 
some commonality.  I was much happier when I just decided to let them be 
different and write my code accordingly.   (Interestingly enough since 
that time MATLAB has added other types to their language as well --- 
which don't always get along). 

Here I think we should just let rank-0 arrays and Python scalars be 
different things and let people know that instead of trying to mask the 
situation which ultimately confuses things.

-Travis





From oliphant at ee.byu.edu  Fri Feb 18 15:05:11 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Fri Feb 18 15:05:11 2005
Subject: [Numpy-discussion] rank-0 arrays ideas
Message-ID: <4216747C.7090504@ee.byu.edu>

 From the current  PEP:

 
    Proposed Solution:

        The solution proposed by this PEP is to fix the places in Python
        that could use rank-0 arrayobjects to allow for them before raising
        an exception (this will be in the core after-all).

        A Python scalar will never be returned unless explicitly
        requested.  I think this is the cleanest, easiest to understand
        and code for solution.  It may require some explicity conversion
        to int here and there, but it seems like a case of "explicit is
        better than implicit"

        Possible compromises:
            -  a compromise could be made for OBJECT arrays and
               perhaps LONG arrays if needed.

            - a special flag could be defined which is the default
              when an array of integers is constructed and which when
              set rank-0 array returns behave differently.

        It is also proposed that slicing, indexing and len() do
        not work (i.e. they raise an error) for rank-0 arrays.


Comments:

-Travis




From tkorvola at welho.com  Fri Feb 18 15:22:14 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Fri Feb 18 15:22:14 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <421665E1.6000008@csun.edu> (Stephen Walton's message of "Fri,
 18 Feb 2005 14:02:09 -0800")
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com>
	<42151919.60607@enthought.com>
	<ce60847bc35769e2b8287b737e998ee6@laposte.net>
	<42161CA3.40506@enthought.com> <421665E1.6000008@csun.edu>
Message-ID: <87wtt5e9oz.fsf@welho.com>

Stephen Walton <stephen.walton at csun.edu> writes:
> As I said, right now min(scalar) in stock Python raises
> a TypeError on the grounds that scalars can't be iterated over.

This is not very different from Numarray, where min(rank_0_array)
raises ValueError because min is applied to an empty sequence.
Iteration normally applies to the first dimenson but rank 0 arrays
have no dimensions.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From dd55 at cornell.edu  Fri Feb 18 15:28:23 2005
From: dd55 at cornell.edu (Darren Dale)
Date: Fri Feb 18 15:28:23 2005
Subject: [Numpy-discussion] rank-0 arrays ideas
In-Reply-To: <4216747C.7090504@ee.byu.edu>
References: <4216747C.7090504@ee.byu.edu>
Message-ID: <200502181826.38518.dd55@cornell.edu>

On Friday 18 February 2005 06:04 pm, Travis Oliphant wrote:
>  From the current  PEP:
>
>
>     Proposed Solution:
>
>         The solution proposed by this PEP is to fix the places in Python
>         that could use rank-0 arrayobjects to allow for them before raising
>         an exception (this will be in the core after-all).
>
>         A Python scalar will never be returned unless explicitly
>         requested.  I think this is the cleanest, easiest to understand
>         and code for solution.  It may require some explicity conversion
>         to int here and there, but it seems like a case of "explicit is
>         better than implicit"
>
>         Possible compromises:
>             -  a compromise could be made for OBJECT arrays and
>                perhaps LONG arrays if needed.
>
>             - a special flag could be defined which is the default
>               when an array of integers is constructed and which when
>               set rank-0 array returns behave differently.
>
>         It is also proposed that slicing, indexing and len() do
>         not work (i.e. they raise an error) for rank-0 arrays.
>
>
> Comments:

I have one:

The following script measures the time it takes for in-place array operations, 
comparing 0-D arrays with scalars. The results are 

0.81 0D double
0.85 0D int
0.11 Scalar float
0.12 Scalar int

Would Numeric3 address this issue?

---

import Numeric as N
import time


a=N.zeros(100000,'d')

one=N.array(1,'d')
d=time.clock()
for i in N.arange(100000):
    a[i]+=one
d = time.clock()-d
print d,'0D double'

one=N.array(1)
d=time.clock()
for i in N.arange(100000):
    a[i]+=one
d = time.clock()-d
print d,'0D int'

d=time.clock()
for i in N.arange(100000):
    a[i]+=1.
d = time.clock()-d
print d,'Scalar float'

d=time.clock()
for i in N.arange(100000):
    a[i]+=1
d = time.clock()-d
print d,'Scalar int'

-- 

Darren



From rkern at ucsd.edu  Fri Feb 18 20:38:22 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Fri Feb 18 20:38:22 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net>
Message-ID: <4216C29F.2090105@ucsd.edu>

konrad.hinsen at laposte.net wrote:
> On 18.02.2005, at 18:59, Travis Oliphant wrote:

>> Do you think this would fly with the Python folks.  Counting the  
>> suggestion above, we would be encouraging the creation of 39 new 
>> types  to the Python core.  My current count shows the current number 
>> of  types as 35 so
> 
> 
> Those new types would (if all goes well) be part of the standard  
> library, but not built-in types. Compared to the number of types and  
> classes in the standard library, the addition is not so big. There  
> wouldn't be literals either. Anyone who doesn't use the array module  
> could thus safely ignore the existence of those types.

I don't see the problem that this approach would solve. It doesn't solve 
the list/tuple indexing problem by itself. Even if the types are part of 
the standard library, they won't be bona-fide ints, so the indexing code 
would still have to be modified to check for them.

I *do* like the idea of the typecode objects, however they are 
implemented, to be able to act as constructors.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From konrad.hinsen at laposte.net  Fri Feb 18 23:45:24 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Fri Feb 18 23:45:24 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <421665E1.6000008@csun.edu>
References: <4214E7FE.9090403@ee.byu.edu> <87zmy26gx6.fsf@welho.com> <42151919.60607@enthought.com> <ce60847bc35769e2b8287b737e998ee6@laposte.net> <42161CA3.40506@enthought.com> <421665E1.6000008@csun.edu>
Message-ID: <a6e5c7b31d2d89c06eef4ed47a30b2e1@laposte.net>

On 18.02.2005, at 23:02, Stephen Walton wrote:

> Duncan Child wrote:
>
>> In another post [Konrad Hinsen] said:
>>
>> "... as the goal is inclusion into the Python core .... I propose  
>> that the PEP should include unification of scalars and arrays such  
>> that for all practical purposes scalars *are* rank-0 arrays. "
>>
>> So I think we are in broad agreement.
>
> It seems to me that actually getting the above behavior is going to  
> require Guido van Rossum et al. to change how scalars behave in the  
> Python core.  As I said, right now min(scalar) in stock Python raises  
> a TypeError on the grounds that scalars can't be iterated over.

That's fine - scalars (like rank-0 arrays) cannot be iterated over.

However, there would indeed have to be changes to scalars. For example,  
1.shape would have to return ().

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Sat Feb 19 00:00:29 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 00:00:29 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <421666F3.6050702@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <421666F3.6050702@ee.byu.edu>
Message-ID: <6eef976ba50ce690a5215cad728e228e@laposte.net>

On 18.02.2005, at 23:06, Travis Oliphant wrote:

> Let me see if i understand you here.  Since the type of the array is  
> bascially handled by the
> PyArray_Descr*  structure.   Are you suggesting turning that into a  
> full-fledged PythonObject with
> new type?   I had thought of this before since it's basically calling  
> for us to do something like that.  Or do you have something else in  
> mind?

I was thinking of the scalar types than of the arrays. What I am  
proposing is to have a Int16, Int32, UInt 16 etc. as Python scalar  
types (though not built-in types) and use the type objects to create  
arrays and to identify the type of array elements. Those types would be  
part of a type hierarchy for type testing.

One could then either have a single array type that stores the type  
object for the elements internally, or have different Python types for  
every kind of array (i.e. Int16Array, Int32Array). I don't see any  
important difference there, so I have no clear opinion on this.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Sat Feb 19 00:13:25 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 00:13:25 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42166FF3.9050908@ee.byu.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42166FF3.9050908@ee.byu.edu>
Message-ID: <18bd99d55abc05ceadd854a1a66d75ec@laposte.net>

On 18.02.2005, at 23:45, Travis Oliphant wrote:

> While open to the idea for Python 3000, Guido does not seem favorable  
> to the idea for Python 2.X The problem, Guido mentions was that  
> float-like objects can convert to ints by truncation and he doesn't  
> want to allow floats to be used as indexes.   He feels it would break  
> too much code.

I can agree with that. Basically the problem is the type hierarchy that  
is too simple (but many if not most programming language suffer from  
that problem). Type conversions are all handled in the same way, which  
doesn't give enough flexibility. But we won't change that of course.

> Of course underneath, a is not an array of integers (it is an array of  
> homogeneous c-ints converted from the Python integer and so why should  
> a[0] be a Python integer).

That would basically mean to change the status of arrays from  
generalized sequence objects to something different. Why not, but then  
this should be made clear to the programmer, in particular by having a  
different printed representation for rank-0 arrays and scalars. It also  
means adding some conversion functions, e.g. for extracting a Python  
Object from a rank-0 Python Object array.

Still, I am worried about two aspects:

1) The amount of confusion this generates among users. The distinction  
between scalars and rank-0 arrays has no utility for the programmer, it  
exists only for implementation and political reasons. I am not looking  
forward to explaining this in my Python courses for beginners.

2) Compatibility with existing code. I am not sure I will convert my  
code to such conventions any time soon, because it requires inspecting  
every single indexing operation in its particular context to see if the  
index could be a rank-0 integer array. There is no way to spot those  
cases by textual analysis. So this change could reduce acceptance to  
the point where there is no interest in pursuing the project any more.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Sat Feb 19 00:15:28 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 00:15:28 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4216C29F.2090105@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu>
Message-ID: <01b322c0be695e314111300a363aaae6@laposte.net>

On 19.02.2005, at 05:37, Robert Kern wrote:

> I don't see the problem that this approach would solve. It doesn't  
> solve the list/tuple indexing problem by itself. Even if the types are  
> part of the standard library, they won't be bona-fide ints, so the  
> indexing code would still have to be modified to check for them.

Yes, but it could check for "integer type" (using the type hierarchy)  
rather than convert everything to an integer with the problem that  
Guido pointed out.

However, my original motivation was consistency of usage. Python has  
type objects to specify types, so I'd rather use them than introduce  
another way to specify the type of array elements.

> I *do* like the idea of the typecode objects, however they are  
> implemented, to be able to act as constructors.

That is an interesting idea as well, though a slightly different one.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From cjw at sympatico.ca  Sat Feb 19 05:45:28 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Sat Feb 19 05:45:28 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <18bd99d55abc05ceadd854a1a66d75ec@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42166FF3.9050908@ee.byu.edu> <18bd99d55abc05ceadd854a1a66d75ec@laposte.net>
Message-ID: <421742C2.1010601@sympatico.ca>

konrad.hinsen at laposte.net wrote:

> On 18.02.2005, at 23:45, Travis Oliphant wrote:
>
>> While open to the idea for Python 3000, Guido does not seem 
>> favorable  to the idea for Python 2.X The problem, Guido mentions was 
>> that  float-like objects can convert to ints by truncation and he 
>> doesn't  want to allow floats to be used as indexes.   He feels it 
>> would break  too much code.
>
>
> I can agree with that. Basically the problem is the type hierarchy 
> that  is too simple (but many if not most programming language suffer 
> from  that problem). Type conversions are all handled in the same way, 
> which  doesn't give enough flexibility. But we won't change that of 
> course.
>
>> Of course underneath, a is not an array of integers (it is an array 
>> of  homogeneous c-ints converted from the Python integer and so why 
>> should  a[0] be a Python integer).
>
>
> That would basically mean to change the status of arrays from  
> generalized sequence objects to something different. Why not, but 
> then  this should be made clear to the programmer, in particular by 
> having a  different printed representation for rank-0 arrays and 
> scalars. It also  means adding some conversion functions, e.g. for 
> extracting a Python  Object from a rank-0 Python Object array.
>
> Still, I am worried about two aspects:
>
> 1) The amount of confusion this generates among users. The 
> distinction  between scalars and rank-0 arrays has no utility for the 
> programmer, it  exists only for implementation and political reasons. 
> I am not looking  forward to explaining this in my Python courses for 
> beginners.

(+1)
If we consider an array as a sequence of objects  of a fixed type, 
numeric or other, it makes sense that when a single object is returned 
then an object of that type be returned, coverted if necessary for Int8 
etc.  Returning a zero rank array is an historical pain.

It might make sense if all traditional Python objects were of zero rank  
I can see no merit in that.

>
> 2) Compatibility with existing code. I am not sure I will convert my  
> code to such conventions any time soon, because it requires 
> inspecting  every single indexing operation in its particular context 
> to see if the  index could be a rank-0 integer array. There is no way 
> to spot those  cases by textual analysis. So this change could reduce 
> acceptance to  the point where there is no interest in pursuing the 
> project any more.
>
I thought that the intent of Numeric 3 was to produce the best - a new 
start, without being overly concerned about compatibility.

I was glad to see the proposal to abandon "ravel" ( a hangover from 
APL?).  Words should have a clear generally accepted meaning.  For 
"ravel" dictionary.com offers:

       1. To separate the fibers or threads of (cloth, for example);
          unravel.
       2. To clarify by separating the aspects of.
       3. To tangle or complicate.

Colin W.




From rkern at ucsd.edu  Sat Feb 19 06:29:25 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Sat Feb 19 06:29:25 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <01b322c0be695e314111300a363aaae6@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net>
Message-ID: <42174D07.7070806@ucsd.edu>

konrad.hinsen at laposte.net wrote:
> On 19.02.2005, at 05:37, Robert Kern wrote:
> 
>> I don't see the problem that this approach would solve. It doesn't  
>> solve the list/tuple indexing problem by itself. Even if the types 
>> are  part of the standard library, they won't be bona-fide ints, so 
>> the  indexing code would still have to be modified to check for them.
> 
> 
> Yes, but it could check for "integer type" (using the type hierarchy)  
> rather than convert everything to an integer with the problem that  
> Guido pointed out.

Except that these types probably can't be derived from the builtin int. 
The C layouts would have to be compatible. They'd probably have to be a 
separate hierarchy.

At that, rank-0 arrays would have to become a special case because their 
value will have to be reflected by x->ob_ival. And how that happens is 
going to depend on their actual C type. We'll be inheriting methods that 
we can't use, and general arrays, even if the C types are compatible, 
can't be used in place of a bona fide PyIntObject.

I would prefer a single type of array object that can store different 
kinds of values.

> However, my original motivation was consistency of usage. Python has  
> type objects to specify types, so I'd rather use them than introduce  
> another way to specify the type of array elements.

True. However, if we introduce a bona fide TypeObject hierarchy for 
numerical scalars that *can* be profitably used outside of the array 
context, it's *going* to be used outside of the array context. If it 
gets into the standard library, it can't just be a large number 
hierarchy for our use; it will have to be *the* number hierarchy for 
Python and include PyLongObjects and decimals and rationals.

And that's probably a bit more than we care to deal with to get 
multiarrays into the core.

On the other hand, the list/tuple indexing issue won't go away until the 
PEP is accepted and integrated into the core. And it won't be accepted 
until Numeric3 has had some life of it's own outside the standard library.

Bugger.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From cjw at sympatico.ca  Sat Feb 19 07:15:23 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Sat Feb 19 07:15:23 2005
Subject: [Numpy-discussion] Numeric3 PEP
Message-ID: <421757E9.10206@sympatico.ca>

The PEP has:

     In Python there will be a hierchial set of classes defined:

      GenericArray (type classes)
        BooleanArray  (Bool)

        NumericArray  (XX is the bit-width of the type)
          SignedArray
          UnsignedArray
	  IntegerArray
            SignedIntegerArray   (IntXX)
	    UnsignedIntegerArray (UIntXX)
	  FloatArray    (FloatXX)
	  ComplexArray  (FloatXX)

	FlexibleArray
          CharacterArray
            StringArray  (String)
            UnicodeArray (Unicode)
          VoidArray (Void)

        ObjectArray
  

It seems that the record type is to be handled under the VoidArray.  I 
hope that it will permit the setting and reading of fields.

For example, if recs is an instance of an Array of records, then it 
should be possible to write:

     >>> recs[22, 5].sex= 'F'
     >>> recs[22, 5].sex
    F

It is not clear, from the current draft, what is intended for 
ObjectArray.  Can these be any Python objects tuple, list etc. or 
instances of any user defined class?

There is also a tofile method ( I would prefer toFile), does this mean 
that pickling would be used for the object instances?

The term WRITABLE is used, is this different from "mutable", the term 
used by Python?

"Methods such as x.r(), x.i(), and x.flatten() are proposed.".  Why not 
use properties, x.r, x.i and x.flatten.  The parentheses appear redundant.

Colin W.

  




From tkorvola at welho.com  Sat Feb 19 08:57:16 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Sat Feb 19 08:57:16 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <421742C2.1010601@sympatico.ca> (Colin J. Williams's message of
 "Sat, 19 Feb 2005 08:44:34 -0500")
References: <4214E7FE.9090403@ee.byu.edu>
	<4bf0e99f809d592b068270a200b6deb0@laposte.net>
	<42166FF3.9050908@ee.byu.edu>
	<18bd99d55abc05ceadd854a1a66d75ec@laposte.net>
	<421742C2.1010601@sympatico.ca>
Message-ID: <87vf8olc92.fsf@welho.com>

"Colin J. Williams" <cjw at sympatico.ca> writes:
> Returning a zero rank array is an historical pain.

The historical pain is returning a scalar: that is what both Numeric
and Numarray currently do.  Returning a zero rank array would be a new
pain to replace that.

> It might make sense if all traditional Python objects were of zero
> rank  I can see no merit in that.

Pushing arrays that deep into the core language would be natural for a
language inteded for numerical linear algebra but perhaps not for a
general purpose language which people also use for web services and
whatnot.

> I was glad to see the proposal to abandon "ravel" ( a hangover from
> APL?).

I thought all APL builtins were denoted by weird special characters
rather than any readable or pronouncable names.  But I don't see ravel
actually being abandoned, as the PEP does not discuss functions much.
One reason for preferring functions to methods and attributes is that
functions can be made to work with scalars and generic sequences.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From cjw at sympatico.ca  Sat Feb 19 11:43:23 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Sat Feb 19 11:43:23 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <87vf8olc92.fsf@welho.com>
References: <4214E7FE.9090403@ee.byu.edu>	<4bf0e99f809d592b068270a200b6deb0@laposte.net>	<42166FF3.9050908@ee.byu.edu>	<18bd99d55abc05ceadd854a1a66d75ec@laposte.net>	<421742C2.1010601@sympatico.ca> <87vf8olc92.fsf@welho.com>
Message-ID: <421796C2.4080808@sympatico.ca>

Timo Korvola wrote:

>"Colin J. Williams" <cjw at sympatico.ca> writes:
>  
>
> [snip]
>
>  
>
>>I was glad to see the proposal to abandon "ravel" ( a hangover from
>>APL?).
>>    
>>
 From 05-02-18 16:24 PM

    Yeah, I don't like this anymore either.

    I like X.flatten() better than X.ravel() too.
    -Travis

I suggest that X.flatten would be even better.

>
>I thought all APL builtins were denoted by weird special characters
>rather than any readable or pronouncable names.  But I don't see ravel
>actually being abandoned, as the PEP does not discuss functions much.
>One reason for preferring functions to methods and attributes is that
>functions can be made to work with scalars and generic sequences.
>
Yes, most of these weird special operators had names, one of these was 
ravel.  Incidentally, 'shape' is also probably inherited from Iverson's APL.

Colin W.



From sdhyok at gmail.com  Sat Feb 19 12:00:10 2005
From: sdhyok at gmail.com (Daehyok Shin)
Date: Sat Feb 19 12:00:10 2005
Subject: [Numpy-discussion] How to subclass NumArray?
Message-ID: <371840ef0502191159306b8a58@mail.gmail.com>

How can I create a simple subclass inheriting everything from NumArray?

-- 
Daehyok Shin (Peter)
Geography Department
University of North Carolina-Chapel Hill
USA



From oliphant at ee.byu.edu  Sat Feb 19 12:54:22 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb 19 12:54:22 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <421757E9.10206@sympatico.ca>
References: <421757E9.10206@sympatico.ca>
Message-ID: <4217A73D.8040900@ee.byu.edu>

>
> It seems that the record type is to be handled under the VoidArray.  I 
> hope that it will permit the setting and reading of fields.

Exactly.  A Python record class can make use of the basic array structure. 

>
> For example, if recs is an instance of an Array of records, then it 
> should be possible to write:
>
>     >>> recs[22, 5].sex= 'F'
>     >>> recs[22, 5].sex
>    F

This will need to be handled by the record class specifically (it will 
make use of a void array). I do not see a need to clutter the general 
array c-type with this.   

Along that idea, the current design is based heavily on Numeric which 
places all arraytypes under a single ArrayType.    It "knows" about the 
type of each array instead of asking the type what it can do. I know 
numarray made strides in a different direction (but I'm not sure how far 
it got, since there is still this basic set of types). 

I am very willing to consider the possibility of a more top-down design 
in which a basic array c-type had no information about what was in the 
array (it just managed its shape, its memory usage, and the size of the 
memory being used).   Then, subtypes could be created which handled 
type-specific needs. 

This seems to be a more logical direction to pursue, but it is a bit of 
a switch and so carries even more risk.   Someone from numarray could 
help here, perhaps.

>
> It is not clear, from the current draft, what is intended for 
> ObjectArray.  Can these be any Python objects tuple, list etc. or 
> instances of any user defined class?

It is the familiar "O" type from Numeric.  Yes, these objects can be any 
Python object whatever.  Numeric already handles this (and numarray 
recently added them).

>
> There is also a tofile method ( I would prefer toFile), does this mean 
> that pickling would be used for the object instances?

The naming convention is lowercase (camelCase is reserved for 
ClassInstances).   I have not thought that far, but probably...

>
> The term WRITABLE is used, is this different from "mutable", the term 
> used by Python?

MUTABLE is a better term.   Do numarray folks agree?

>
> "Methods such as x.r(), x.i(), and x.flatten() are proposed.".  Why 
> not use properties, x.r, x.i and x.flatten.  The parentheses appear 
> redundant.

The distinction is that attributes do not return copies, but methods 
might.  Thus, the idea is that x.i() would return zeros if the array 
were not complex, but x.imag would raise and error.   I like the idea of 
attributes being intrinsic properties of an array, while methods could 
conceivably return or do anything.

Thanks for your continued help with clarification and improvement to the 
PEP.

-Travis




From oliphant at ee.byu.edu  Sat Feb 19 13:09:24 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb 19 13:09:24 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <18bd99d55abc05ceadd854a1a66d75ec@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42166FF3.9050908@ee.byu.edu> <18bd99d55abc05ceadd854a1a66d75ec@laposte.net>
Message-ID: <4217AACC.6040804@ee.byu.edu>

> I can agree with that. Basically the problem is the type hierarchy 
> that  is too simple (but many if not most programming language suffer 
> from  that problem). Type conversions are all handled in the same way, 
> which  doesn't give enough flexibility. But we won't change that of 
> course.

Guido might in Python 3000....

>
> That would basically mean to change the status of arrays from  
> generalized sequence objects to something different. Why not, but 
> then  this should be made clear to the programmer, in particular by 
> having a  different printed representation for rank-0 arrays and 
> scalars. It also  means adding some conversion functions, e.g. for 
> extracting a Python  Object from a rank-0 Python Object array.

I guess I don't consider arrays as generalized sequence objects.  Except 
for the Python Object array which might be.     Still, my views on this 
are really not central as I want to help get something together that a 
large majority of us can be satisfied with.  I'm really not interested 
in intentionally making something that is difficult to use and I am 
willing to bend on many things (especially if work-arounds are possible). 

It seems to me that the rank-0 array scalar problem is fundamental to 
the situation of trying to craft a Numerical-environment on top of a 
general-purpose one.    I don't see a "right solution" that would please 
all parties short of altering all Python scalars --- and that would 
likely upset an entirely different crowd.  I suppose that is why we have 
the compromise we do now. 

> Still, I am worried about two aspects:
>
> 1) The amount of confusion this generates among users. The 
> distinction  between scalars and rank-0 arrays has no utility for the 
> programmer, it  exists only for implementation and political reasons. 
> I am not looking  forward to explaining this in my Python courses for 
> beginners.

The only utility might be speed because it is not too hard to see that 
rank-0 arrays that behave as part of a generic system of 
multidimensional arrays might carry quite a bit of baggage that is not 
needed if all you are every want is a scalar.    This distinction may 
not be possible to get rid of. 

To put it more clearly:  Is it possible to define a scalar that 
interacts seamlessly with a system of multidimensional arrays without 
slowing slowing it down for usage in other contexts?  

I don't know the answer, but it sure seems to be no.  I've looked at 
Piddle (PERL's equivalent to Numeric)  and they seem to do exactly the 
same thing (have a rank-0 array that is very different from the Perl 
scalar). 


Added to our problem is that we do not have much control over the 
definition of fundamental scalars in Python.   Guido has suggested that 
he may be willing to allow integers to get methods (how many methods I'm 
not sure --- I didn't push him.  He mentioned the possibility of adding 
a rank-inquiry method for example).   It would pleasantly surprise me if 
he were willing to give scalars all of the methods and attributes of 
arrays.

>
> 2) Compatibility with existing code. I am not sure I will convert my  
> code to such conventions any time soon, because it requires 
> inspecting  every single indexing operation in its particular context 
> to see if the  index could be a rank-0 integer array. There is no way 
> to spot those  cases by textual analysis. So this change could reduce 
> acceptance to  the point where there is no interest in pursuing the 
> project any more.

A healthy use of int()  in indexing operations could fix this, but yes, 
I see compatibility as an issue.  I don't want to create 
incompatibilities if we can avoid it.    On the other hand, I don't want 
to continue with a serious design flaw just for the sake of 
incompatibility either (I'm still trying to figure out if it is a flaw 
or not, it sure seems like a hack).

Thanks for your continued help and advice.

-Travis





From oliphant at ee.byu.edu  Sat Feb 19 13:20:26 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb 19 13:20:26 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42174D07.7070806@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu>
Message-ID: <4217AD44.1090302@ee.byu.edu>

>>
>> Yes, but it could check for "integer type" (using the type 
>> hierarchy)  rather than convert everything to an integer with the 
>> problem that  Guido pointed out.
>
>
> Except that these types probably can't be derived from the builtin 
> int. The C layouts would have to be compatible. They'd probably have 
> to be a separate hierarchy.

On the python-dev list someone (Bob Ippolito) suggested inheriting 
rank-0 arrays from the Python Int Type.  I don't see how this can be 
even be done for all of the integer types (unless the Python Int Type is 
changed to hold the largest possible integer (long long)).   

>
> At that, rank-0 arrays would have to become a special case because 
> their value will have to be reflected by x->ob_ival. And how that 
> happens is going to depend on their actual C type. We'll be inheriting 
> methods that we can't use, and general arrays, even if the C types are 
> compatible, can't be used in place of a bona fide PyIntObject.
>
> I would prefer a single type of array object that can store different 
> kinds of values.

I see the same problems that Robert is talking about.  Do we really want 
to special-case all array code to handle rank-0 arrays differently?   
That seems to be opening up a very big can of worms.   Is the only way 
to solve this problem to handle rank-0 arrays in a separate hierarchy?  
I have doubts that such a system would even work.

>
> On the other hand, the list/tuple indexing issue won't go away until 
> the PEP is accepted and integrated into the core. And it won't be 
> accepted until Numeric3 has had some life of it's own outside the 
> standard library.


I agree with Robert's assesment --- bugger.  I'm really annoyed that 
such a relatively simple think like rank-0 arrays versus Python's 
already-defined scalars could be such a potential show-stopper.

Hoping-it-won't-be

-Travis





From oliphant at ee.byu.edu  Sat Feb 19 13:46:15 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb 19 13:46:15 2005
Subject: [Numpy-discussion] Poll on a Rank-0 arrays: please give +1, -1, or 0
Message-ID: <4217B374.2060702@ee.byu.edu>

We need to get some opinions regarding the recent discussion on rank-0 
arrays.   Please give your  vote +1 (like), -1 (hate), or 0 (don't 
care)  for **each** of the following possibilities.  Feel free to 
clarify your vote if you need to.


1)  Rank-0 arrays are always returned from uarray operations, Python 
scalars available on request only.   (*If* arrayobject get's into Python 
core, Guido has agreed to let rank-0 integer arrays be used as
index variables).  Rank-0 arrays are made to resmbe scalar of



2) Rank-0 arrays are never returned from uarray operations (unless using 
asarray on a scalar), and when a rank-0 array naturally appears in the 
calculation, an appropriate Python scalar is returned (note that this 
would lose precision for long doubles unless a new Python object was 
created).



3) The current compromise is continued and for some types a Python 
scalar is returned, while for other types a  rank-0 array is returned



4) Rank-0 arrays are made equivalent to Python scalars and a new Python 
scalar for each fundamental data type supported by uarray is constructed 
(rank-0 arrays would still probably be used internally, but users would 
not have to know this).   The new Python-scalars would inherit from an 
existing Python scalar where appropriate and would have the same 
attributes and methods of uarrays (very likely at least initially they 
would be seemlessly converted to rank-0 arrays when "mixed" operations 
occur).









From jmiller at stsci.edu  Sat Feb 19 13:58:18 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Sat Feb 19 13:58:18 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <4217A73D.8040900@ee.byu.edu>
References: <421757E9.10206@sympatico.ca>  <4217A73D.8040900@ee.byu.edu>
Message-ID: <1108850232.3450.1.camel@jaytmiller.comcast.net>

On Sat, 2005-02-19 at 13:53 -0700, Travis Oliphant wrote:
> >
> > The term WRITABLE is used, is this different from "mutable", the term 
> > used by Python?
> 
> MUTABLE is a better term.   Do numarray folks agree?
> 

Well, no.  I've never heard of page of "mutable" memory.  It's no biggie
though.

Regards,
Todd




From tim.hochberg at cox.net  Sat Feb 19 14:53:23 2005
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Sat Feb 19 14:53:23 2005
Subject: [Numpy-discussion] Poll on a Rank-0 arrays: please give +1, -1,
 or 0
In-Reply-To: <4217B374.2060702@ee.byu.edu>
References: <4217B374.2060702@ee.byu.edu>
Message-ID: <4217C335.9060002@cox.net>

Travis Oliphant wrote:

>
> We need to get some opinions regarding the recent discussion on rank-0 
> arrays.   Please give your  vote +1 (like), -1 (hate), or 0 (don't 
> care)  for **each** of the following possibilities.  Feel free to 
> clarify your vote if you need to.
>
>
> 1)  Rank-0 arrays are always returned from uarray operations, Python 
> scalars available on request only.   (*If* arrayobject get's into 
> Python core, Guido has agreed to let rank-0 integer arrays be used as
> index variables).  Rank-0 arrays are made to resmbe scalar of

[Something appears to be scrambled here, but I think I get the gist of it]

0.

This seems most self consistent option. It will, however, break some 
unknown amount of code since I'm sure some stuff relys on Python int / 
float behaviour that will be somewhat different than rank-0 array 
behaviour. One example is multiplying two rank-0 integers  won't 
overflow into longs as would two python integers. I think it will also 
feel odd: people expect scalars when they index a rank one array.

Even though Guido agreed to let rank-0 arrays work as index variables, 
the index issue will still be problematic. Even assuming that 
arrayobject makes it into the core, it won't, and shouldn't happen, till 
we have some experience as a community using and polishing it. In the 
meantime, indexing won't work, which will make it harder to use. Meaning 
a smaller community and less likelihood that it goes into the core. It's 
chicken and an egg problem.

>
>
>
> 2) Rank-0 arrays are never returned from uarray operations (unless 
> using asarray on a scalar), and when a rank-0 array naturally appears 
> in the calculation, an appropriate Python scalar is returned (note 
> that this would lose precision for long doubles unless a new Python 
> object was created).

-1 unless a new object is created. -0 in that case.

>
>
>
> 3) The current compromise is continued and for some types a Python 
> scalar is returned, while for other types a  rank-0 array is returned

0. This is a bit of a mess, but has the advantage of backwards 
compatibility. Also we have a fair amount of experience with it and know 
that it works at least fairly well.

>
> 4) Rank-0 arrays are made equivalent to Python scalars and a new 
> Python scalar for each fundamental data type supported by uarray is 
> constructed (rank-0 arrays would still probably be used internally, 
> but users would not have to know this).   The new Python-scalars would 
> inherit from an existing Python scalar where appropriate and would 
> have the same attributes and methods of uarrays (very likely at least 
> initially they would be seemlessly converted to rank-0 arrays when 
> "mixed" operations occur).

+1 assuming the details can be worked out. I think this will feel more 
natural in acutal use than the first option. It also should avoid the 
indexing problem that hamper that option.

-tim


>
>
>
>
>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>





From konrad.hinsen at laposte.net  Sat Feb 19 14:55:32 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 14:55:32 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <421742C2.1010601@sympatico.ca>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42166FF3.9050908@ee.byu.edu> <18bd99d55abc05ceadd854a1a66d75ec@laposte.net> <421742C2.1010601@sympatico.ca>
Message-ID: <6dc26b212e60c7ab35735a669ba6101b@laposte.net>

On 19.02.2005, at 14:44, Colin J. Williams wrote:

>> 2) Compatibility with existing code. I am not sure I will convert my   
>> code to such conventions any time soon, because it requires  
>> inspecting  every single indexing operation in its particular context  
>> to see if the  index could be a rank-0 integer array. There is no way  
>> to spot those  cases by textual analysis. So this change could reduce  
>> acceptance to  the point where there is no interest in pursuing the  
>> project any more.
>>
> I thought that the intent of Numeric 3 was to produce the best - a new  
> start, without being overly concerned about compatibility.

It all depends on where "overly" starts.

Let's be pragmatic: the majority of potential Numeric 3 users will be  
the current users of Numeric and numarray. If they don't accept Numeric  
3 because it's a pain, no amount of nice design will help.

> I was glad to see the proposal to abandon "ravel" ( a hangover from  
> APL?).

Fine, but that's the kind of change that doesn't hurt much: a name  
change can be made with a text editor. Changing the behaviour of  
fundamental operations (indexing) is a different thing.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Sat Feb 19 14:57:23 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 14:57:23 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <42174D07.7070806@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu>
Message-ID: <292015fbc60118037b345c59d0c10355@laposte.net>

On 19.02.2005, at 15:28, Robert Kern wrote:

> Except that these types probably can't be derived from the builtin  
> int. The C layouts would have to be compatible. They'd probably have  
> to be a separate hierarchy.

They could all derive from a common (yet-to-be-written) base class that  
has no data layout at all.

> True. However, if we introduce a bona fide TypeObject hierarchy for  
> numerical scalars that *can* be profitably used outside of the array  
> context, it's *going* to be used outside of the array context. If it  
> gets into the

True, though I expect its use to be limited to the numeric community.

>  standard library, it can't just be a large number hierarchy for our  
> use; it will have to be *the* number hierarchy for Python and include  
> PyLongObjects and decimals and rationals.

That would be preferable indeed.

> And that's probably a bit more than we care to deal with to get  
> multiarrays into the core.

It all depends on the reaction of the Python developer community. We  
won't know before asking.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From konrad.hinsen at laposte.net  Sat Feb 19 15:03:09 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sat Feb 19 15:03:09 2005
Subject: [Numpy-discussion] Poll on a Rank-0 arrays: please give +1, -1, or 0
In-Reply-To: <4217B374.2060702@ee.byu.edu>
References: <4217B374.2060702@ee.byu.edu>
Message-ID: <d59a6837a68b8a99b10d8f9fdc780e9f@laposte.net>

On 19.02.2005, at 22:45, Travis Oliphant wrote:

> 1)  Rank-0 arrays are always returned from uarray operations, Python 
> scalars available on request only.   (*If* arrayobject get's into 
> Python core, Guido has agreed to let rank-0 integer arrays be used as
> index variables).  Rank-0 arrays are made to resmbe scalar of

-1

I consider this a kludge that might be acceptable if no other solution 
is politically doable, but not a desirable feature.


> 2) Rank-0 arrays are never returned from uarray operations (unless 
> using asarray on a scalar), and when a rank-0 array naturally appears 
> in the calculation, an appropriate Python scalar is returned (note 
> that this would lose precision for long doubles unless a new Python 
> object was created).

+1

provided that
1) a special long double type is created
2) the type promotion rules handle scalars in some reasonable way

> 3) The current compromise is continued and for some types a Python 
> scalar is returned, while for other types a  rank-0 array is returned

+1

It works well enough in practice.

> 4) Rank-0 arrays are made equivalent to Python scalars and a new 
> Python scalar for each fundamental data type supported by uarray is 
> constructed (rank-0 arrays would still probably be used internally, 
> but users would not have to know this).   The new Python-scalars would 
> inherit from an existing Python scalar where appropriate and would 
> have the same attributes and methods of uarrays (very likely at least 
> initially they would be seemlessly converted to rank-0 arrays when 
> "mixed" operations occur).

+1

This is still my first choice.

Konrad.




From tim.hochberg at cox.net  Sat Feb 19 15:10:14 2005
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Sat Feb 19 15:10:14 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <4217A73D.8040900@ee.byu.edu>
References: <421757E9.10206@sympatico.ca> <4217A73D.8040900@ee.byu.edu>
Message-ID: <4217C71E.8050002@cox.net>

Hi Travis,

First off, let me say that I'm encouraged to see some action towards 
unifying Numeric/Numarray the split has been somewhat dismaying. Thank 
you for your efforts in this regard.

I'd like to lobby against flatten(), r() and i(). To a large extent, 
these duplicate the functionality of flat, real and imag. And, these 
three methods are defined to sometimes return copies and sometimes 
return views. That type of interface is a recipe for errors and should 
only be used as a last resort. Fortunately in this case there are better 
alternatives.
 
Flatten() is not necessary now that flat will be faux array with a view 
to the original [I believe you are calling it an indexable iterator].  I 
would, however,  recomend that A.flat.copy() work. In that case, A.flat 
would be used when no copy was desired, and A.flat.copy() when a copy 
was desired.  I don't find the copy when discontiguous case useful 
enough to deserve it's own function and it's error prone as I'll discuss 
more below. 

r()  appears to be around just for symmetry with i() since A.r() will 
always be the same as A.real. That leaves i(). My opinion is that this 
case would be better served by returning a read-only array of zeros when 
operating on a real array. This array could even be a faux-array that 
doesn't allocate any space, although that may be a project for another day.

If it's really deemed necessary to have these functions in addition to 
their attribute brethren, I recomend that they always return copies 
rather than varying their behaviour depending on the situation. The 
problem with methods that sometimes return a copy, is that it won't be 
long before someone types:

    def foobar(a)
       flat_view = a.flatten()
       # lots of code
       flat_view[some_index] = some_new_number

This works until someone passes in a discontiguous array, at which point 
it fails mysteriously. This type of problem tends to be somewhat 
resistant to unit tests, since tests often involve only contiguous arrays.

Regards,

-tim




From rkern at ucsd.edu  Sat Feb 19 15:56:16 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Sat Feb 19 15:56:16 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <292015fbc60118037b345c59d0c10355@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu> <292015fbc60118037b345c59d0c10355@laposte.net>
Message-ID: <4217D1E8.2070602@ucsd.edu>

konrad.hinsen at laposte.net wrote:
> On 19.02.2005, at 15:28, Robert Kern wrote:
> 
>> Except that these types probably can't be derived from the builtin  
>> int. The C layouts would have to be compatible. They'd probably have  
>> to be a separate hierarchy.
> 
> 
> They could all derive from a common (yet-to-be-written) base class that  
> has no data layout at all.

We then end up with the same chicken-egg problem as accepting rank-0 
integer arrays as indices. It won't work until it's in the core. If I'm 
understanding your proposition correctly, it also creates another 
problem: rank-n arrays would then pass this check, although they shouldn't.

>> True. However, if we introduce a bona fide TypeObject hierarchy for  
>> numerical scalars that *can* be profitably used outside of the array  
>> context, it's *going* to be used outside of the array context. If it  
>> gets into the
> 
> 
> True, though I expect its use to be limited to the numeric community.

I expect so, too. However, when considering additions to the standard 
library, python-dev has to assume otherwise. If it's going to be so 
limited in application, then something so large shouldn't be in the 
standard library.

>>  standard library, it can't just be a large number hierarchy for our  
>> use; it will have to be *the* number hierarchy for Python and include  
>> PyLongObjects and decimals and rationals.
> 
> 
> That would be preferable indeed.
> 
>> And that's probably a bit more than we care to deal with to get  
>> multiarrays into the core.
> 
> 
> It all depends on the reaction of the Python developer community. We  
> won't know before asking.

I think it would be great to have a more thorough number hierarchy in 
the standard library. So would some others. See PEPs 228 and 242. 
However, I think that the issue is orthogonal getting an multiarray 
object into the standard library. I'm not convinced that it actually 
solves the problems with getting multiarrays into the core. Now, we may 
have different priorities, so we have different thresholds of 
"problem-ness."

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From cjw at sympatico.ca  Sat Feb 19 16:40:10 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Sat Feb 19 16:40:10 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4217D1E8.2070602@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu> <292015fbc60118037b345c59d0c10355@laposte.net> <4217D1E8.2070602@ucsd.edu>
Message-ID: <4217DC34.8000105@sympatico.ca>

Robert Kern wrote:

> konrad.hinsen at laposte.net wrote:
>
>>
>>
>> It all depends on the reaction of the Python developer community. We  
>> won't know before asking.
>
>
> I think it would be great to have a more thorough number hierarchy in 
> the standard library. So would some others. See PEPs 228 and 242. 
> However, I think that the issue is orthogonal getting an multiarray 
> object into the standard library. I'm not convinced that it actually 
> solves the problems with getting multiarrays into the core. Now, we 
> may have different priorities, so we have different thresholds of 
> "problem-ness."
>
PEP 228 is under consideration (since 2000):


          Numerical Python Issues

    People who use Numerical Python do so for high-performance vector
    operations.  Therefore, NumPy should keep its hardware based
    numeric model.

*Unresolved Issues*

    Which number literals will be exact, and which inexact?

    How do we deal with IEEE 754 operations? (probably, isnan/isinf should
    be methods)

    On 64-bit machines, comparisons between ints and floats may be
    broken when the comparison involves conversion to float.  Ditto
    for comparisons between longs and floats.  This can be dealt with
    by avoiding the conversion to float.  (Due to Andrew Koenig.)


  

For PEP 242 the status is:

    This PEP has been closed by the author.  The kinds module will not
    be added to the standard library.

    There was no opposition to the proposal but only mild interest in
    using it, not enough to justify adding the module to the standard
    library.  Instead, it will be made available as a separate
    distribution item at the Numerical Python site.  At the next
    release of Numerical Python, it will no longer be a part of the
    Numeric distribution.
  

It seems to be up to the numerical folk to make proposals.

Colin W.

  




From rkern at ucsd.edu  Sat Feb 19 17:03:26 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Sat Feb 19 17:03:26 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4217DC34.8000105@sympatico.ca>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu> <292015fbc60118037b345c59d0c10355@laposte.net> <4217D1E8.2070602@ucsd.edu> <4217DC34.8000105@sympatico.ca>
Message-ID: <4217E1A9.6010202@ucsd.edu>

Colin J. Williams wrote:
> Robert Kern wrote:
> 
>> konrad.hinsen at laposte.net wrote:
>>
>>>
>>>
>>> It all depends on the reaction of the Python developer community. We  
>>> won't know before asking.
>>
>>
>>
>> I think it would be great to have a more thorough number hierarchy in 
>> the standard library. So would some others. See PEPs 228 and 242. 
>> However, I think that the issue is orthogonal getting an multiarray 
>> object into the standard library. I'm not convinced that it actually 
>> solves the problems with getting multiarrays into the core. Now, we 
>> may have different priorities, so we have different thresholds of 
>> "problem-ness."
>>
> PEP 228 is under consideration (since 2000):
> 
> 
>          Numerical Python Issues
> 
>    People who use Numerical Python do so for high-performance vector
>    operations.  Therefore, NumPy should keep its hardware based
>    numeric model.

Note that the recommendation is that Numeric ignore PEP's number model. 
That PEP points *away* from things like Int32 and Float64.

[snip]

> For PEP 242 the status is:
> 
>    This PEP has been closed by the author.  The kinds module will not
>    be added to the standard library.
> 
>    There was no opposition to the proposal but only mild interest in
>    using it, not enough to justify adding the module to the standard
>    library.  Instead, it will be made available as a separate
>    distribution item at the Numerical Python site.  At the next
>    release of Numerical Python, it will no longer be a part of the
>    Numeric distribution.
>  
> 
> It seems to be up to the numerical folk to make proposals.

Note also that PEP 242 was retracted before people got really interested 
    (by which I mean "interested enough to implement") in other number 
types like decimal and rationals. While historically these proposal have 
come from the NumPy community (which I'm distinguishing from "numerical 
folk"), in the future they will need to intimately involve a much larger 
group of people.

Of course, the NumPy community is a subset of "numerical folk," so we 
are naturally interested in how numbers are represented in Python. I'm 
not saying we shouldn't be or that such a proposal shouldn't come from 
this community. In general, such a thing would be of great use to this 
community. However, I don't see how it would help, in specific, the 
addition of multiarray objects to the standard library, nor do I think 
that such should wait upon the acceptance and implementation of such a 
proposal.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From oliphant at ee.byu.edu  Sat Feb 19 19:52:20 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sat Feb 19 19:52:20 2005
Subject: [Numpy-discussion] PEP updated
Message-ID: <4218091A.4010404@ee.byu.edu>

I've updated the PEP to conform to what I think is the best hybrid 
solution propsed and that is to implement a tree of PythonTypes in C 
whose leaves are (to the Python user) new rank-0 arrays.   This is
more work to implement (but I don't think a great deal of work see 
below), and I think it will lead to the best results under our current 
constraints.

Internally, the PyArray_Type will still deal with rank-0 arrays (in fact 
the new Python scalars will be converted to them internally rather than 
special-case a bunch of code).

I don't think we should try to solve the general number-hierchy in 
Python at this time.  We will implement the array-number hierarchy 
only.    If this leads to others getting interested in the general 
problem, then that may come, but now we have to focus on our objective 
of getting the arrayobject accepted into Python.

Note that these new rank-0 Python scalars will be simple, have the same 
methods and attributes as Python arrays (likely-implemented once in the 
GenericArrType by converting the object to a rank-0 array).
Also, they will inherit (multiple-inheritance is possible in C) from the 
appropriate Python Type where an exact-match is available.

-Travis







From konrad.hinsen at laposte.net  Sun Feb 20 01:00:26 2005
From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net)
Date: Sun Feb 20 01:00:26 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <4217D1E8.2070602@ucsd.edu>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu> <292015fbc60118037b345c59d0c10355@laposte.net> <4217D1E8.2070602@ucsd.edu>
Message-ID: <b78086cf38e1648a7fc0055e405a6f7c@laposte.net>

On 20.02.2005, at 00:55, Robert Kern wrote:

>> They could all derive from a common (yet-to-be-written) base class  
>> that  has no data layout at all.
>
> We then end up with the same chicken-egg problem as accepting rank-0  
> integer arrays as indices. It won't work until it's in the core. If  
> I'm

True. We could have a patch to Python for testing purposes before such  
a decision, but this is not an ideal situation.

> understanding your proposition correctly, it also creates another  
> problem: rank-n arrays would then pass this check, although they  
> shouldn't.

No. My proposal is to have new Python scalar types, which are distinct  
from the array type(s). The new scalar types would use rank-0 arrays as  
their internal representation, but that would be an implementation  
detail not visible to users.

> I expect so, too. However, when considering additions to the standard  
> library, python-dev has to assume otherwise. If it's going to be so  
> limited in application, then something so large shouldn't be in the  
> standard library.

I am not so sure about this. There is some very domain-specific stuff  
in the standard library.

> I think it would be great to have a more thorough number hierarchy in  
> the standard library. So would some others. See PEPs 228 and 242.  
> However, I think that the issue is orthogonal getting an multiarray  
> object into the standard library. I'm not convinced that it actually  
> solves the problems with

The common point is just the community that is interested in this.  
However, there might be a wider interest in a re-working of the number  
hierarchy. There have been additions to the number hierarchy that don't  
come from the numerics community, such as the decimal number type. I  
think that a more open number hierarchy, into which modules could add  
their own types, could make it into the core if it doesn't cause any  
compatibility problems. In fact, this may be easier to argue for than  
having array methods on all scalar types.

Konrad.
--
------------------------------------------------------------------------ 
-------
Konrad Hinsen
Laboratoire Leon Brillouin, CEA Saclay,
91191 Gif-sur-Yvette Cedex, France
Tel.: +33-1 69 08 79 25
Fax: +33-1 69 08 82 61
E-Mail: khinsen at cea.fr
------------------------------------------------------------------------ 
-------




From rkern at ucsd.edu  Sun Feb 20 01:52:09 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Sun Feb 20 01:52:09 2005
Subject: [Numpy-discussion] Response to PEP suggestions
In-Reply-To: <b78086cf38e1648a7fc0055e405a6f7c@laposte.net>
References: <4214E7FE.9090403@ee.byu.edu> <4bf0e99f809d592b068270a200b6deb0@laposte.net> <42162CEC.3090607@ee.byu.edu> <64c4c87f7fd6e461bdcd8fbe11610899@laposte.net> <4216C29F.2090105@ucsd.edu> <01b322c0be695e314111300a363aaae6@laposte.net> <42174D07.7070806@ucsd.edu> <292015fbc60118037b345c59d0c10355@laposte.net> <4217D1E8.2070602@ucsd.edu> <b78086cf38e1648a7fc0055e405a6f7c@laposte.net>
Message-ID: <42185DA7.3040507@ucsd.edu>

konrad.hinsen at laposte.net wrote:
> On 20.02.2005, at 00:55, Robert Kern wrote:

>> understanding your proposition correctly, it also creates another  
>> problem: rank-n arrays would then pass this check, although they  
>> shouldn't.
> 
> 
> No. My proposal is to have new Python scalar types, which are distinct  
> from the array type(s). The new scalar types would use rank-0 arrays as  
> their internal representation, but that would be an implementation  
> detail not visible to users.

*Ahh!* Enlightenment dawns. That makes a *hell* of a lot more sense than 
what I thought you were proposing. Thank you for bearing with my babbling.

I withdraw my objections.

I do have one comment, though. I still prefer the idea that arrays, 
including rank-0 arrays, be containers. So I would suggest that there 
only be a handful of new rank-0 types, int-like, float-like, 
complex-like, object, and maybe a couple more. The differences between 
the type objects would be solely for inheritance reasons. Different 
precisions would be handled like they are for rank-n arrays.

>> I expect so, too. However, when considering additions to the standard  
>> library, python-dev has to assume otherwise. If it's going to be so  
>> limited in application, then something so large shouldn't be in the  
>> standard library.
> 
> 
> I am not so sure about this. There is some very domain-specific stuff  
> in the standard library.

I think my point was that a number hierarchy is something that *ought* 
to have much wider applicability. If the implementation *doesn't*, then 
it shouldn't be in the standard library. That's why I prefer the 
arrangement I described above. It isn't a real number hierarchy and 
doesn't purport to be one. It's just an implementation detail[1] to make 
multiarrays place nicer with the rest of the Python universe.

>> I think it would be great to have a more thorough number hierarchy in  
>> the standard library. So would some others. See PEPs 228 and 242.  
>> However, I think that the issue is orthogonal getting an multiarray  
>> object into the standard library. I'm not convinced that it actually  
>> solves the problems with
> 
> 
> The common point is just the community that is interested in this.  
> However, there might be a wider interest in a re-working of the number  
> hierarchy. There have been additions to the number hierarchy that don't  
> come from the numerics community, such as the decimal number type. I  
> think that a more open number hierarchy, into which modules could add  
> their own types, could make it into the core if it doesn't cause any  
> compatibility problems. In fact, this may be easier to argue for than  
> having array methods on all scalar types.

Agreed. I think, though, that what Travis proposes in "PEP Updated" is 
going to be easiest of all to swallow.

[1] Isn't it amazing how many design problems go away by claiming that 
"it's just an implementation detail?"  :-)

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From tkorvola at welho.com  Sun Feb 20 05:40:23 2005
From: tkorvola at welho.com (Timo Korvola)
Date: Sun Feb 20 05:40:23 2005
Subject: [Numpy-discussion] PEP updated
In-Reply-To: <4218091A.4010404@ee.byu.edu> (Travis Oliphant's message of
 "Sat, 19 Feb 2005 20:50:50 -0700")
References: <4218091A.4010404@ee.byu.edu>
Message-ID: <87u0o7pcyp.fsf@welho.com>

Travis Oliphant <oliphant at ee.byu.edu> writes:
> I've updated the PEP to conform to what I think is the best hybrid
> solution propsed and that is to implement a tree of PythonTypes in C
> whose leaves are (to the Python user) new rank-0 arrays.

For the purpose of differentiating between basic and advanced
indexing, are rank 0 arrays considered scalars or arrays?  Does their
presence among indices trigger advanced indexing?

For sequence behaviour len, indexing and iteration should work
consistently, i.e., "a[i] for i in xrange(len(a))" and "x for x in a"
should identically generate the contents of a.  Currently indexing and
iteration occur along the first dimension, which for rank 0 arrays
does not exist.  I understand that there is no burning desire to
completely switch to Matlab-style one-dimensional indexing.  The
alternatives are then giving rank 0 arrays exceptional semantics
inconsistent with other arrays or raising an exception if len or
iteration is applied to a rank 0 array.  Without exceptional semantics
the only index expression that makes sense is a[()].  I would favor
consistency here and raise an exception but there is one matter that
raises some doubt: do programs expect that if __len__ and __iter__
exist they will not fail?

Converting scalars back to arrays: they should be rank 0.  Rank 1
would be Matlabish nonsense, except that Matlab is even more
nonsensical and uses rank 2.  But when is such conversion needed?  If
it is only for operations with other arrays then broadcasting is
likely to occur immediately.

-- 
	Timo Korvola		<URL:http://www.iki.fi/tkorvola>



From perry at stsci.edu  Sun Feb 20 07:58:32 2005
From: perry at stsci.edu (Perry Greenfield)
Date: Sun Feb 20 07:58:32 2005
Subject: [Numpy-discussion] Poll on a Rank-0 arrays: please give +1, -1, or 0
In-Reply-To: <4217B374.2060702@ee.byu.edu>
Message-ID: <NEBBIJKBMLDBLNCEEFOCGEDHFJAA.perry@stsci.edu>

I'm going to refrain from a numerical rating but rather add a few comments
and questions
> 1)  Rank-0 arrays are always returned from uarray operations, Python
> scalars available on request only.   (*If* arrayobject get's into Python
> core, Guido has agreed to let rank-0 integer arrays be used as
> index variables).  Rank-0 arrays are made to resmbe scalar of
>
A minor question, in this and other cases where rank-0 is being used, what
will repr show? For example, if I do:

>>> x = arange(2)
>>> x[0]

do I get '0' or 'array(0)' ?

I suppose the former is fine if rank-0 is usable in every way as a scalar
and no one really needs to know the difference, but I troubles me a bit to
hide this fact (with repr) if it isn't quite the same.

Along those lines (for this case and the last) I worry that there are
potential gotchas that haven't been discovered yet. How confident are you
that there aren't any in trying to integrate rank-0 with Python scalars and
their usage. It may be one of those things that is hard to know unless the
work of trying to implement it is actually done. You've thought about this a
lot more than I have.
>
>
> 2) Rank-0 arrays are never returned from uarray operations (unless using
> asarray on a scalar), and when a rank-0 array naturally appears in the
> calculation, an appropriate Python scalar is returned (note that this
> would lose precision for long doubles unless a new Python object was
> created).
>
As Konrad mentions, as long as there is some means of handling long doubles,
I find scalars perfectly acceptable. I tend to think that is what most user
assume they are getting.
>
>
> 3) The current compromise is continued and for some types a Python
> scalar is returned, while for other types a  rank-0 array is returned
>
I've never really liked this behavior.
>
>
> 4) Rank-0 arrays are made equivalent to Python scalars and a new Python
> scalar for each fundamental data type supported by uarray is constructed
> (rank-0 arrays would still probably be used internally, but users would
> not have to know this).   The new Python-scalars would inherit from an
> existing Python scalar where appropriate and would have the same
> attributes and methods of uarrays (very likely at least initially they
> would be seemlessly converted to rank-0 arrays when "mixed" operations
> occur).
>
Sounds good if there are no gotchas but may be a lot of work (coding and
political). Any performance issues regarding rank-0 vs scalars? Will users
pay a penalty for using rank-0 in scalar-like expressions (no apparent array
usage other than the rank-0 values)?

Perry





From jdhunter at ace.bsd.uchicago.edu  Sun Feb 20 09:50:25 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Sun Feb 20 09:50:25 2005
Subject: [Numpy-discussion] PEP updated
In-Reply-To: <4218091A.4010404@ee.byu.edu> (Travis Oliphant's message of
 "Sat, 19 Feb 2005 20:50:50 -0700")
References: <4218091A.4010404@ee.byu.edu>
Message-ID: <m37jl3uo6x.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Travis" == Travis Oliphant <oliphant at ee.byu.edu> writes:

    Travis> I've updated the PEP to conform to what I think is the
    Travis> best hybrid solution propsed and that is to implement a
    Travis> tree of PythonTypes in C whose leaves are (to the Python
    Travis> user) new rank-0 arrays.  This is more work to implement
    Travis> (but I don't think a great deal of work see below), and I
    Travis> think it will lead to the best results under our current
    Travis> constraints.

Hi Travis, 

One issue I didn't see addressed in the PEP is the handling of special
values for floating point numbers, eg inf and nan.

Will Numeric3 be able to handle something like

  >>> nan = float('nan')
  >>> inf = float('inf')
  >>> n.array([inf, nan])

or otherwise support these special values?  A common request I get in
matplotlib is to be able to handle float arrays with nan, but I've
hesitated tackling this because of a lack of consistent support for
these values in python, Numeric and numarray.  In this particular
example, Numeric 23.7 raises a traceback in the repr pipeline, and
numarray appears to work properly.

Apologies if I missed a discussion of this topic already on this list
-- searching for nan in my mail reader keeps matching Fernando, and
inf matches listinfo....

Thanks,
JDH



From cjw at sympatico.ca  Sun Feb 20 13:04:00 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Sun Feb 20 13:04:00 2005
Subject: [Numpy-discussion] Matlab is a tool for doing numerical computations with matrices and
 vectors.
Message-ID: <4218FAD8.6060804@sympatico.ca>

The subject line is from http://www.math.utah.edu/lab/ms/matlab/matlab.html

I've seen little use of the words "matrix" or "vector" in the numarray 
stuff.  These are one or two dimensional structures.

The Numeric/numarray focus is on multidimensional arrays.

Some years ago,  Huaiyu Zhu made a number of postings advocating the 
introduction of a Python-based system permitting the usual matrix 
operations.  It seems that the culmination of these efforts was Matpy 
(http://matpy.sourceforge.net/), which is not currently active.

I was impressed with the case made by Huaiyu Zhu and have attempted to 
carry on in this direction.  An outline of this work is at: 
http://www3.sympatico.ca/cjw/IntroToPyMatrix.pdf.

This work was based on numarray, partly because it was generally 
considered to be the way ahead and partly because it purported to 
provide a class structure.  numarray has proved to be a bit of a moving 
target.

I am continuing this work in the hope that some class-based structures 
will come out of the current Numeric3 discussions.  The hope is also 
that there is a need out there for matrix operations, as distinct from 
array operations.

Colin W.



From oliphant at ee.byu.edu  Sun Feb 20 14:02:26 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sun Feb 20 14:02:26 2005
Subject: [Numpy-discussion] Matlab is a tool for doing numerical computations
 with matrices and vectors.
In-Reply-To: <4218FAD8.6060804@sympatico.ca>
References: <4218FAD8.6060804@sympatico.ca>
Message-ID: <421908B0.90406@ee.byu.edu>

> The Numeric/numarray focus is on multidimensional arrays.
>
> Some years ago,  Huaiyu Zhu made a number of postings advocating the 
> introduction of a Python-based system permitting the usual matrix 
> operations.  It seems that the culmination of these 

> efforts was Matpy (http://matpy.sourceforge.net/), which is not 
> currently active.


I remember his work.   I really liked many of his suggestions, though it 
took him a while to recognize that a Matrix class has been distributed 
with Numeric from very early on.    Yes, it needed work, and a few of 
his ideas were picked up on and included in Numeric's Matrix object. 

I would like to see a standard Matrix class (along with the record-array 
class) also be included with the universal array.  

By the way, I've been calling the array object uarray in the PEP (kind 
of parallel to ufunc), but if others have better alternatives, I'm 
certainly not against them.  The problem with array is that it is taken 
by the arraymodule, which I don't want to try and replace at this point 
(I don't want to worry about backward compatibility with it...). 

I also like the name ndarray though.  I haven't heard what others think.

-Travis





From oliphant at ee.byu.edu  Sun Feb 20 14:24:11 2005
From: oliphant at ee.byu.edu (Travis Oliphant)
Date: Sun Feb 20 14:24:11 2005
Subject: [Numpy-discussion] Poll on a Rank-0 arrays: please give +1, -1,
 or 0
In-Reply-To: <NEBBIJKBMLDBLNCEEFOCGEDHFJAA.perry@stsci.edu>
References: <NEBBIJKBMLDBLNCEEFOCGEDHFJAA.perry@stsci.edu>
Message-ID: <42190DC7.6080305@ee.byu.edu>

Perry Greenfield wrote:

>I'm going to refrain from a numerical rating but rather add a few comments
>and questions
>  
>
Great, works just as well.

>do I get '0' or 'array(0)' ?
>  
>
I'm thinking '0' here since we want "array" scalars to look and act like 
Python scalars.

>I suppose the former is fine if rank-0 is usable in every way as a scalar
>and no one really needs to know the difference, but I troubles me a bit to
>hide this fact (with repr) if it isn't quite the same.
>  
>
Well, I suppose repr could show something else (i.e. how you would 
create one) while str could print it like a scalar.  

>Along those lines (for this case and the last) I worry that there are
>potential gotchas that haven't been discovered yet. How confident are you
>that there aren't any in trying to integrate rank-0 with Python scalars and
>their usage. It may be one of those things that is hard to know unless the
>work of trying to implement it is actually done. You've thought about this a
>
Well, there may always be potential gotchas.  But, I'm extremely 
confident that the new Python scalars will work seemlessly with array 
objects.  The will work in Python as scalars pretty much anyplace a 
subtype is allowed (PyInt_Check will return True, for example).   There 
are a few places in Python where for performance reasons, exact integers 
are special-cased.  The new array-type scalars would not benefit from 
those places in the code.

>>2) Rank-0 arrays are never returned from uarray operations (unless using
>>asarray on a scalar), and when a rank-0 array naturally appears in the
>>calculation, an appropriate Python scalar is returned (note that this
>>would lose precision for long doubles unless a new Python object was
>>created).
>>
>>    
>>
>As Konrad mentions, as long as there is some means of handling long doubles,
>I find scalars perfectly acceptable. I tend to think that is what most user
>assume they are getting.
>  
>
Then, I think solution 4 is the right one....

>>    
>>
>Sounds good if there are no gotchas but may be a lot of work (coding and
>political). Any performance issues regarding rank-0 vs scalars? Will users
>pay a penalty for using rank-0 in scalar-like expressions (no apparent array
>usage other than the rank-0 values)?
>  
>
I see three performance issues:

1) the parts of Python that special-case for exact integer arguments.

2) if array scalars are internally converted to rank-0 arrays and then 
passed through the ufunc machinery to implement all operations (which 
will be the default implementation), then this will be a bit slower (not 
sure how much slower) --- especially after special casing for contiguous 
arguments in the ufunc code (PEP to come later).   

3) If the type hierarchy is too deep, then a few more pointer 
dereferences would be needed to find the function to call. 

Power users looking for a speed boost on scalar operations would have to 
manually get an exact Python integer.   So, I don't think array scalars 
would "take over" every need for a scalar, but they would be scalars 
that act as a smooth intermediary between N-d arrays *and* Python scalars.

I think by introducing this parallel set of array scalars we decrease 
the political issues.  We are not asking non-numeric Python users to 
change all beloved integers to something more generic.   We are just 
introducing a bunch of "array-behaved" scalar objects that will also 
inherit from Python builtin types where appropriate.


-Travis






From tim.hochberg at cox.net  Sun Feb 20 15:25:23 2005
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Sun Feb 20 15:25:23 2005
Subject: [Numpy-discussion] More PEP comments
Message-ID: <42191C33.8080203@cox.net>

 From the PEP:

>      integers in index arrays can be negative to mean selection from the end
>      of the array.  Indices that are too large will be taken to mean the
>      last element.  Indicies that are too negative will mean
>      the first element.
>
Here's another little piece of the PEP that I feel should be 
reconsidered. In the absence of a compelling reason to do otherwise, the 
indices in index arrays should behave just as if one were using that 
index directly as in index tuple. And, index tuples should match the 
behaviour Python lists. In other words:

alist[i] == anarray[i] == anarray(array([i]))[0]

In order to make this equivalence work, both anarray[i] and 
anarray(array([i])) need to raise index errors when i is too large or 
too negative. And for consistency this rule should continue to apply 
when the index is a tuple rather than a simple integer index.

This is more or less how numarray works and I think that's the right 
choice. [However, I did notice that numarray 1.1 produced nonsensical 
answers when index arrays with values that were too negative, when they 
were too positive they worked as I would expect and produced an error].

-tim










From aisaac at american.edu  Sun Feb 20 16:17:04 2005
From: aisaac at american.edu (Alan G Isaac)
Date: Sun Feb 20 16:17:04 2005
Subject: [Numpy-discussion] Matlab is a tool for doing numerical computations with matrices and vectors.
In-Reply-To: <4218FAD8.6060804@sympatico.ca>
References: <4218FAD8.6060804@sympatico.ca>
Message-ID: <Mahogany-0.66.0-572-20050220-191912.00@american.edu>

On Sun, 20 Feb 2005, "Colin J. Williams" apparently wrote:
> The hope is also that there is a need out there for matrix 
> operations, as distinct from array operations.

Absolutely.
I use Numeric's Matrix class all the time
(and long for additional functionality).
I am uncertain why numarray did not adopt
most of this class (especially its 
attribute list).

Cheers,
Alan Isaac







From juenglin at cs.pdx.edu  Sun Feb 20 23:32:00 2005
From: juenglin at cs.pdx.edu (Ralf Juengling)
Date: Sun Feb 20 23:32:00 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <4217C71E.8050002@cox.net>
References: <421757E9.10206@sympatico.ca> <4217A73D.8040900@ee.byu.edu> <4217C71E.8050002@cox.net>
Message-ID: <42198CDB.5020600@cs.pdx.edu>

Tim Hochberg wrote:
> I'd like to lobby against flatten(), r() and i(). To a large extent, 
> these duplicate the functionality of flat, real and imag. And, these 
> three methods are defined to sometimes return copies and sometimes 
> return views. That type of interface is a recipe for errors and should 
> only be used as a last resort. 

I feel the same.


> Flatten() is not necessary now that flat will be faux array with a view 
> to the original [I believe you are calling it an indexable iterator].  I 
> would, however,  recomend that A.flat.copy() work. In that case, A.flat 
> would be used when no copy was desired, and A.flat.copy() when a copy 
> was desired.  

Good idea.

ralf




From rkern at ucsd.edu  Mon Feb 21 04:28:04 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Mon Feb 21 04:28:04 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <4217C71E.8050002@cox.net>
References: <421757E9.10206@sympatico.ca> <4217A73D.8040900@ee.byu.edu> <4217C71E.8050002@cox.net>
Message-ID: <4219D37C.8000504@ucsd.edu>

Tim Hochberg wrote:
> 
> Hi Travis,
> 
> First off, let me say that I'm encouraged to see some action towards 
> unifying Numeric/Numarray the split has been somewhat dismaying. Thank 
> you for your efforts in this regard.
> 
> I'd like to lobby against flatten(), r() and i(). To a large extent, 
> these duplicate the functionality of flat, real and imag. And, these 
> three methods are defined to sometimes return copies and sometimes 
> return views. That type of interface is a recipe for errors and should 
> only be used as a last resort. 

There is, however, a blisteringly common use case for such an interface: 
you are using the result directly in an expression such that it is only 
going to be read and never written to. In that case, you want it to 
never fail (except in truly pathological cases like being out of 
memory), and you want it to be as efficient as possible and so never 
produce a copy where you can produce a view.

So, I think we need three interfaces for each of this kind of attribute:

1) Getting a view. If a view cannot be obtained, raise an error. Never copy.

2) Getting a copy. Never return a view.

3) Getting *something* the most efficient way possible. Caller beware.

While I lean towards making the syntaxes for the first two The Most 
Obvious Ways To Do It, I think it may be rather important to keep the 
syntax of the third convenient and short, particularly since it is that 
case that usually occurs in the middle of already-complicated expressions.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From tim.hochberg at cox.net  Mon Feb 21 07:20:02 2005
From: tim.hochberg at cox.net (Tim Hochberg)
Date: Mon Feb 21 07:20:02 2005
Subject: [Numpy-discussion] Re: Numeric3 PEP
In-Reply-To: <4219D37C.8000504@ucsd.edu>
References: <421757E9.10206@sympatico.ca> <4217A73D.8040900@ee.byu.edu> <4217C71E.8050002@cox.net> <4219D37C.8000504@ucsd.edu>
Message-ID: <4219FBE3.1000906@cox.net>

Robert Kern wrote:

> Tim Hochberg wrote:
>
>>
>> Hi Travis,
>>
>> First off, let me say that I'm encouraged to see some action towards 
>> unifying Numeric/Numarray the split has been somewhat dismaying. 
>> Thank you for your efforts in this regard.
>>
>> I'd like to lobby against flatten(), r() and i(). To a large extent, 
>> these duplicate the functionality of flat, real and imag. And, these 
>> three methods are defined to sometimes return copies and sometimes 
>> return views. That type of interface is a recipe for errors and 
>> should only be used as a last resort. 
>
>
> There is, however, a blisteringly common use case for such an 
> interface: you are using the result directly in an expression such 
> that it is only going to be read and never written to. In that case, 
> you want it to never fail (except in truly pathological cases like 
> being out of memory), and you want it to be as efficient as possible 
> and so never produce a copy where you can produce a view.
>
> So, I think we need three interfaces for each of this kind of attribute:
>
> 1) Getting a view. If a view cannot be obtained, raise an error. Never 
> copy.

The proposal for flat is to always return a view, if the array is not 
contiguous a special view-of-a-discontiguous-array will be returned. 
This special object obviously be less efficient to index than a 
contiguous array, but if implemented carefully could probably be made 
more efficient than a copy plus indexing for one-shot uses (i.e., in an 
expression).

>
> 2) Getting a copy. Never return a view.
>
> 3) Getting *something* the most efficient way possible. Caller beware.

By this you mean return a view if contiguous, a copy if not, correct?

> While I lean towards making the syntaxes for the first two The Most 
> Obvious Ways To Do It, I think it may be rather important to keep the 
> syntax of the third convenient and short, particularly since it is 
> that case that usually occurs in the middle of already-complicated 
> expressions.

Personally I don't find this all that compelling. Primarily because the 
blanket statement that (3) is more efficient than (1) is extremely 
suspect. In can many, if not most, cases (1) [as clarified above] will 
be more efficient than (3) anyway. Our disagreement here may stem from 
having different understandings of the proposed behaviour for flat.

There are cases where you probably *do* want to copy if discontiguous, 
but not in expressions. I'm thinking of cases where you are going to be 
reusing some flattened slice of a larger array multiple times, and you 
plan to use it read only (already that's sounding pretty rare though). 
In that case, my preferred spelling is:

a_contig_flat_array = ascontiguous(an_array).flat

In other words, I'd like to segregate all of the sometimes-copy 
behaviour into functions called asXXX, so that it's easy to see and 
easier to debug when it goes wrong. Of course, ascontiguous doesn't 
exist at present, as far as I can tell, but it'd be easy enough to add:

def ascontiguous(a):
    a = asarray(a)
    if not a.iscontiguous():
        a = a.copy()
    return a


Regards,

-tim







From cjw at sympatico.ca  Mon Feb 21 10:22:05 2005
From: cjw at sympatico.ca (Colin J. Williams)
Date: Mon Feb 21 10:22:05 2005
Subject: [Numpy-discussion] Matlab is a tool for doing numerical computations
 with matrices and vectors.
In-Reply-To: <421908B0.90406@ee.byu.edu>
References: <4218FAD8.6060804@sympatico.ca> <421908B0.90406@ee.byu.edu>
Message-ID: <421A26A5.7070306@sympatico.ca>

Travis Oliphant wrote:

>
>> The Numeric/numarray focus is on multidimensional arrays.
>>
>> Some years ago, Huaiyu Zhu made a number of postings advocating the 
>> introduction of a Python-based system permitting the usual matrix 
>> operations. It seems that the culmination of these 
>
>
>> efforts was Matpy (http://matpy.sourceforge.net/), which is not 
>> currently active.
>
>
>
> I remember his work. I really liked many of his suggestions, though it 
> took him a while to recognize that a Matrix class has been distributed 
> with Numeric from very early on. 

numpy.pdf dated 03-07-18 has

    "For those users, the Matrix class provides a more intuitive
    interface. We defer discussion of the Matrix class until later."


Page 51 of that document has:

    "Subclassing
    Subclassing Numeric arrays is not possible due to a limitation of
    Python. The approach taken in the Masked
    Array facility (?Masked Arrays? on page 97) is one answer.
    UserArray.py, described below, can be subclassed,
    but this is often unsatisfactory unless you put in a similar effort
    to that in MA."

On the same page there is:

    "Matrix.py
    The Matrix.py python module defines a class Matrix which is a
    subclass of UserArray. The only differences
    between Matrix instances and UserArray instances is that the *
    operator on Matrix performs a
    matrix multiplication, as opposed to element-wise multiplication,
    and that the power operator ** is disallowed
    for Matrix instances."

In view of the above, I can understand why Huaiyu Zhu took a while. His 
proposal was much more ambitious.

Yes, I know that the power operator is implemented and that there is a 
random matrix but I hope that some attention is given to the 
functionality PyMatrix. I recognize that the implementation has some 
weakneses.

> Yes, it needed work, and a few of his ideas were picked up on and 
> included in Numeric's Matrix object.

I suggest that this overstates what was picked up.

> I would like to see a standard Matrix class (along with the 
> record-array class) also be included with the universal array.

Good, on both scores. I hope that the PEP will set out these ideas.

Colin W.




From tchur at optushome.com.au  Tue Feb 22 16:58:39 2005
From: tchur at optushome.com.au (Tim Churches)
Date: Tue Feb 22 16:58:39 2005
Subject: [Numpy-discussion] Broken setup.py in Numeric 23.7
In-Reply-To: <4219FBE3.1000906@cox.net>
References: <421757E9.10206@sympatico.ca> <4217A73D.8040900@ee.byu.edu> <4217C71E.8050002@cox.net> <4219D37C.8000504@ucsd.edu> <4219FBE3.1000906@cox.net>
Message-ID: <421BD507.3050107@optushome.com.au>

A small plea: exciting work on Numeric3 not withstanding, would it be 
possible to fix setup.py in Numeric 23.7 so that it doesn't fall over on 
Linux? A few versions ago, if BLAS and LAPACK weren't installed or 
weren't found, NumPy would automatically build with its own versions of 
BLAS_lite and LAPACK_lite. Now it just falls over, and it is very 
inconvenient to have to instruct users of our app, which depends on 
Numeric, how to patch setup.py, and/or provide a replacement setup.py.

Also, the new home page for Numpy, to which http://numpy.sf.net is 
redirected, does not have a link to the SorceForge project page for 
Numpy from which released tarballs or zip file can be downloaded. That's 
another source of grief for users, who may not know to go to 
http://www.sf.net/projects/numpy

Thanks,

Tim C





From Sebastien.deMentendeHorne at electrabel.com  Wed Feb 23 03:18:02 2005
From: Sebastien.deMentendeHorne at electrabel.com (Sebastien.deMentendeHorne at electrabel.com)
Date: Wed Feb 23 03:18:02 2005
Subject: [Numpy-discussion] PEP updated
Message-ID: <035965348644D511A38C00508BF7EAEB145CB330@seacex03.eib.electrabel.be>

> For sequence behaviour len, indexing and iteration should work
> consistently, i.e., "a[i] for i in xrange(len(a))" and "x for x in a"
> should identically generate the contents of a.  Currently indexing and
> iteration occur along the first dimension, which for rank 0 arrays
> does not exist.  I understand that there is no burning desire to
> completely switch to Matlab-style one-dimensional indexing.  The
> alternatives are then giving rank 0 arrays exceptional semantics
> inconsistent with other arrays or raising an exception if len or
> iteration is applied to a rank 0 array.  Without exceptional semantics
> the only index expression that makes sense is a[()].  I would favor
> consistency here and raise an exception but there is one matter that
> raises some doubt: do programs expect that if __len__ and __iter__
> exist they will not fail?
> 

Looking at the code I wrote that must deal with special scalar vs array
cases, I noticed that an important feature that the rank-0/scalar should
have is a correct indexing behaviour with:
 a[..., NewAxis] or a[NewAxis] or a[NewAxis, ...]

By the way, isn't it easier to find a solution to this problem by looking at
specific pieces of code that currently needs an ugly case for scalar ?
Moreover, should this rank-0 array behaviour of scalar be extended to other
"scalar" objects ? Should we be able to use:
 p = (3,4)
 a = p[NewAxis]
 a.shape = (1,)
 a.typecode() ==> 'O'
?



From gerard.vermeulen at grenoble.cnrs.fr  Wed Feb 23 09:52:50 2005
From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen)
Date: Wed Feb 23 09:52:50 2005
Subject: [Numpy-discussion] ANN: PyQwt-4.2 released
Message-ID: <20050223184715.6f4d1d07.gerard.vermeulen@grenoble.cnrs.fr>

What is PyQwt?

- it is a set of Python bindings for the Qwt C++ class library which
  extends the Qt framework with widgets for scientific, engineering
  and financial applications.   It provides a widget to plot
  2-dimensional data and various widgets to display and control
  bounded or unbounded floating point values.

- it requires and extends PyQt, a set of Python bindings for Qt.

- it supports the use of PyQt, Qt, Qwt, the Numerical Python extensions
  (either Numeric, or numarray or both) and optionally SciPy in a GUI
  Python application or in an interactive Python session.

- it runs on POSIX, MacOS/X and Windows platforms (practically any
  platform supported by Qt and Python).

The home page of PyQwt is http://pyqwt.sourceforge.net.

PyQwt-4.2 is a maintenance release to keep up with the recent releases
of PyQt-3.14 and its code generator SIP-4.2.

PyQwt-4.2 supports:

1. Python-2.4 downto -2.1.
2. PyQt-3.14 downto -3.10.
3. SIP-4.2 downto -4.0, and SIP-3.11 downto -3.10.
4. Qt-3.3.4 downto -2.3.0.
5. Qwt-4.2.0.


Have fun -- Gerard Vermeulen



From jmiller at stsci.edu  Wed Feb 23 12:46:45 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Wed Feb 23 12:46:45 2005
Subject: [Numpy-discussion] ANN: numarray-1.2.2
Message-ID: <1109191504.19455.3.camel@halloween.stsci.edu>

Release Notes for numarray-1.2.2

Numarray is an array processing package designed to efficiently
manipulate large multi-dimensional arrays.  Numarray is modelled after
Numeric and features c-code generated from python template scripts, the
capacity to operate directly on arrays in files, arrays of heterogeneous
records, string arrays, and in-place operation on 
array memory mapped onto files.

I. ENHANCEMENTS

1. Support for user defined universal functions with arbitrary numbers
of parameters.  See numarray/Examples/ufunc/setup_special.py

2. Additional modifications to support numarray operation with
scipy_base.

3. Import time roughly halved thanks to Rory Yorke.

4. Improved sorting functions from Chuck Harris.

5. Simplified array() and fromfile() from Rory Yorke.

6. Port of Numeric's dotblas extension for faster matrix
multiplication when linked against ATLAS, LAPACK, BLAS, etc.

7. Improvements to array indexing speed from Francesc Alted and Tim
Hochberg.

II. BUGS FIXED / CLOSED

1120138	Document ieeespecial	  
1119411	No file named "LinearAlgebra2/setup.py"
1119265	'print' fails on masked arrays	  
1111765	python setup.py bdist_rpm fails (1.2a)
1110655	Problem with the ufunc .sum() and a rank-0 array.
1110607	f2py file paths on windows	  
1108374	count method in string.array wrong
1102874	remove use of basestring in fromfile for Python 2.2
1100435	Remove NUL bytes in nd_image/test.py
1099974	.sum() gives wrong results	
1091474	log(0) should be -inf	
1089116	CharArray maxLen bug	
1088817	Convolution with complex kernels gives wrong answer
1087378	LinAlgError is a string, not an Exception
1087304	Add toUpper() and toLower to CharArray
1087158	Float64 == records.CharType
1075802	masked_print_option produces incorrect output
1075795	float() and int() don't work on masked arrays
1075794	correlate2d should do autocorrelations
1075793	convolve2d/correlate2d should create output arrays
1075789	resized() pads with nulls not blanks: numarray 1.2.2
1069032	A little bug in setup.py
1068768	KeyboardInterrupt shows strange behavior
1067238	memory leak in determinant
1056649	Memory leak with dot and transpose
1052488	Modify sum() result type for boolean arrays
1050292	Side effects of putmask
1047272	Update license references
1045518	A disconnected numarray rant
1044885	complex number ieee oddities
1044788	wrong typecode
1038528	Making PyArray_CanCastSafely safer on 64-bit machines
1038397	numarray sum() has a possible memory leak
1038064	tofile() of zero length array
1037038	boolean cummulative operator leak
1036327	Buffer overflow in sum and add_Int64_reduce
1035701	numerical array from string list loops
1033860	Wrong array indexing when byteorder is changed
1031883	MA __setslice__ bug
1031356	Sequences with Python longs --> Int64
1030023	problem with matrixmultiply
1029547	IndexError in 4.8 Index Arrays example
1028782	matrixmultiply reference counting
1028738	atlas/blas library listings in addons.py
1025160	gcc error during install: duplicate case value
1022523	chararray concatentation problem
1018252	bad memory leak in reduce/accumulate

See
http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse
for more details.

III. CAUTIONS

This release should be backward binary compatible with numarray-1.1.1.

WHERE
-----------

Numarray-1.2.2 windows executable installers, source code, and manual is
here:

http://sourceforge.net/project/showfiles.php?group_id=1369

Numarray is hosted by Source Forge in the same project which hosts
Numeric:

http://sourceforge.net/projects/numpy/

The web page for Numarray information is at:

http://stsdas.stsci.edu/numarray/index.html

Trackers for Numarray Bugs, Feature Requests, Support, and Patches are
at the Source Forge project for NumPy at:

http://sourceforge.net/tracker/?group_id=1369

REQUIREMENTS
------------------------------

numarray-1.2.2 requires Python 2.2.2 or greater.  


AUTHORS, LICENSE
------------------------------

Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC
Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science
Institute.  We'd like to acknowledge the assitance of Francesc Alted,
Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh,
Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu
Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else
who has contributed with comments and feedback.

Numarray is made available under a BSD-style License.  See
LICENSE.txt in the source distribution for details.

-- 
Todd Miller             jmiller at stsci.edu





From NadavH at VisionSense.com  Thu Feb 24 06:58:01 2005
From: NadavH at VisionSense.com (Nadav Horesh)
Date: Thu Feb 24 06:58:01 2005
Subject: [Numpy-discussion] Incompatibility betwewn numarray 1.2.2  and Numeric
Message-ID: <421DE929.9020803@VisionSense.com>

Numeric and numarray arrays became incopatible ---
Is it a bug or a design change?


In [10]: import Numeric as N

In [11]: A = N.arange(6.0)

In [12]: import numarray as n

In [13]: a = n.arange(6.0)

In [14]: a+A                  # numarray op Numeric
---------------------------------------------------------------------------
exceptions.ValueError                                Traceback (most 
recent call last)

/tmp/numarray-1.2.2/<console>

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
__add__(self, operand)
    933             return operand.__radd__(self)
    934         else:
--> 935             return ufunc.add(self, operand)
    936    
    937     def __radd__(self, operand):

/usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
_cache_miss2(self, n1, n2, out)
    918
    919     def _cache_miss2(self, n1, n2, out):
--> 920         (in1, in2), inform, scalar = _inputcheck(n1, n2)
    921        
    922         mode, win1, win2, wout, cfunc, ufargs = \

/usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
_inputcheck(*inargs)    374             # see if it can be made into an 
array
    375             try:
--> 376                 inarg = _nc.array(inarg)
    377             except TypeError:
    378                 raise TypeError(

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
array(sequence, typecode, copy, savespace, type, shape)
    399     if shape is None:
    400         shape=()
--> 401     return fromlist([sequence],type,shape)
    402
    403 def asarray(seq, type=None, typecode=None):

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
fromlist(seq, type, shape, check_overflow, typecode)
    245     tshape = _frontseqshape(seq)
    246     if shape is not None and _gen.product(shape) != 
_gen.product(tshape):
--> 247         raise ValueError("shape incompatible with sequence")
    248     ndim = len(tshape)
    249     if ndim <= 0:

ValueError: shape incompatible with sequence

In [15]: A+a     # The other way around
---------------------------------------------------------------------------
exceptions.ValueError                                Traceback (most 
recent call last)

/tmp/numarray-1.2.2/<console>

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
__radd__(self, operand)
    940             return operand.__add__(self)
    941         else:
--> 942             r = ufunc.add(operand, self)
    943             if isinstance(r, NumArray):
    944                 r.__class__ = self.__class__

/usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
_cache_miss2(self, n1, n2, out)
    918
    919     def _cache_miss2(self, n1, n2, out):
--> 920         (in1, in2), inform, scalar = _inputcheck(n1, n2)
    921        
    922         mode, win1, win2, wout, cfunc, ufargs = \

/usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
_inputcheck(*inargs)    374             # see if it can be made into an 
array
    375             try:
--> 376                 inarg = _nc.array(inarg)
    377             except TypeError:
    378                 raise TypeError(

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
array(sequence, typecode, copy, savespace, type, shape)
    399     if shape is None:
    400         shape=()
--> 401     return fromlist([sequence],type,shape)
    402
    403 def asarray(seq, type=None, typecode=None):

/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
fromlist(seq, type, shape, check_overflow, typecode)
    245     tshape = _frontseqshape(seq)
    246     if shape is not None and _gen.product(shape) != 
_gen.product(tshape):
--> 247         raise ValueError("shape incompatible with sequence")
    248     ndim = len(tshape)
    249     if ndim <= 0:

ValueError: shape incompatible with sequence


--------------------------

  Nadav.



From jmiller at stsci.edu  Thu Feb 24 08:05:35 2005
From: jmiller at stsci.edu (Todd Miller)
Date: Thu Feb 24 08:05:35 2005
Subject: [Numpy-discussion] Incompatibility betwewn numarray 1.2.2  and
	Numeric
In-Reply-To: <421DE929.9020803@VisionSense.com>
References: <421DE929.9020803@VisionSense.com>
Message-ID: <1109261019.16689.23.camel@halloween.stsci.edu>

It's a bug.

Thanks,
Todd

On Thu, 2005-02-24 at 09:48, Nadav Horesh wrote:
> Numeric and numarray arrays became incopatible ---
> Is it a bug or a design change?
> 
> 
> In [10]: import Numeric as N
> 
> In [11]: A = N.arange(6.0)
> 
> In [12]: import numarray as n
> 
> In [13]: a = n.arange(6.0)
> 
> In [14]: a+A                  # numarray op Numeric
> ---------------------------------------------------------------------------
> exceptions.ValueError                                Traceback (most 
> recent call last)
> 
> /tmp/numarray-1.2.2/<console>
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> __add__(self, operand)
>     933             return operand.__radd__(self)
>     934         else:
> --> 935             return ufunc.add(self, operand)
>     936    
>     937     def __radd__(self, operand):
> 
> /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
> _cache_miss2(self, n1, n2, out)
>     918
>     919     def _cache_miss2(self, n1, n2, out):
> --> 920         (in1, in2), inform, scalar = _inputcheck(n1, n2)
>     921        
>     922         mode, win1, win2, wout, cfunc, ufargs = \
> 
> /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
> _inputcheck(*inargs)    374             # see if it can be made into an 
> array
>     375             try:
> --> 376                 inarg = _nc.array(inarg)
>     377             except TypeError:
>     378                 raise TypeError(
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> array(sequence, typecode, copy, savespace, type, shape)
>     399     if shape is None:
>     400         shape=()
> --> 401     return fromlist([sequence],type,shape)
>     402
>     403 def asarray(seq, type=None, typecode=None):
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> fromlist(seq, type, shape, check_overflow, typecode)
>     245     tshape = _frontseqshape(seq)
>     246     if shape is not None and _gen.product(shape) != 
> _gen.product(tshape):
> --> 247         raise ValueError("shape incompatible with sequence")
>     248     ndim = len(tshape)
>     249     if ndim <= 0:
> 
> ValueError: shape incompatible with sequence
> 
> In [15]: A+a     # The other way around
> ---------------------------------------------------------------------------
> exceptions.ValueError                                Traceback (most 
> recent call last)
> 
> /tmp/numarray-1.2.2/<console>
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> __radd__(self, operand)
>     940             return operand.__add__(self)
>     941         else:
> --> 942             r = ufunc.add(operand, self)
>     943             if isinstance(r, NumArray):
>     944                 r.__class__ = self.__class__
> 
> /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
> _cache_miss2(self, n1, n2, out)
>     918
>     919     def _cache_miss2(self, n1, n2, out):
> --> 920         (in1, in2), inform, scalar = _inputcheck(n1, n2)
>     921        
>     922         mode, win1, win2, wout, cfunc, ufargs = \
> 
> /usr/local/lib/python2.4/site-packages/numarray/ufunc.py in 
> _inputcheck(*inargs)    374             # see if it can be made into an 
> array
>     375             try:
> --> 376                 inarg = _nc.array(inarg)
>     377             except TypeError:
>     378                 raise TypeError(
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> array(sequence, typecode, copy, savespace, type, shape)
>     399     if shape is None:
>     400         shape=()
> --> 401     return fromlist([sequence],type,shape)
>     402
>     403 def asarray(seq, type=None, typecode=None):
> 
> /usr/local/lib/python2.4/site-packages/numarray/numarraycore.py in 
> fromlist(seq, type, shape, check_overflow, typecode)
>     245     tshape = _frontseqshape(seq)
>     246     if shape is not None and _gen.product(shape) != 
> _gen.product(tshape):
> --> 247         raise ValueError("shape incompatible with sequence")
>     248     ndim = len(tshape)
>     249     if ndim <= 0:
> 
> ValueError: shape incompatible with sequence
> 
> 
> --------------------------
> 
>   Nadav.
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 




From gerard.vermeulen at grenoble.cnrs.fr  Sun Feb 27 07:13:05 2005
From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen)
Date: Sun Feb 27 07:13:05 2005
Subject: [Numpy-discussion] ANN: PyQwt3D-0.1 released
Message-ID: <20050227160647.1ebdf98d.gerard.vermeulen@grenoble.cnrs.fr>

What is PyQwt3D?

- it is a set of Python bindings for the QwtPlot3D C++ class library
  which extends the Qt framework with widgets for 3D data visualization.
  PyQwt3D inherits the snappy feel from QwtPlot3D.
  The examples at http://pyqwt.sourceforge.net/pyqwt3d-examples.html
  show how easy it is to make a 3D plot and how to save a 3D plot to
  an image or an (E)PS/PDF file.

- it requires and extends PyQt, a set of Python bindings for Qt.

- it supports the use of PyQt, Qt, Qwt, the Numerical Python extensions
  (either Numeric, or numarray or both) and optionally SciPy in a GUI
  Python application or in an interactive Python session.

- it runs on POSIX, MacOS/X and Windows platforms (practically any
  platform supported by Qt and Python).

The home page of PyQwt3D is http://pyqwt.sourceforge.net.


PyQwt3D-0.1 supports:

1. Python-2.4 downto -2.3.
2. PyQt-3.14 downto -3.12.
3. SIP-4.2 downto -4.0, but SIP-4.2 is recommended.
   PyQwt3D does *not* support SIP-3.x (I tried, but failed).
4. Qt-3.3.4 downto -2.3.0.
5. QwtPlot3D-0.2.4-beta.


Have fun -- Gerard Vermeulen